I am working on the file "BayesianAnalysisWithPython/Coin Flip.ipynb".
The first task is:
Chapter 2
Coin flip pymc3
Can you write Python code for it?

np.random.seed(123)
n_experiments = 4
theta_real = 0.35
data = stats.bernoulli.rvs(p=theta_real, size=n_experiments)
print(data)

XX = np.linspace(0,1,100)
plt.plot(XX, stats.beta(1,1).pdf(XX))

with pm.Model() as our_first_model:
    theta = pm.Beta('theta', alpha=1, beta=1)
    y = pm.Bernoulli('y', p=theta, observed=data)
    start = pm.find_MAP()
    step = pm.Metropolis()
    trace = pm.sample(1000, step=step, start=start, chains=4, compute_convergence_checks=True)


I am working on the file "BayesianDataAnalysis/Poisson_disease.ipynb".
The first task is:
Page 45 example
Using a uniform Prior
Can you write Python code for it?

mu = pymc.Uniform('mu', 0, 1e5)
deaths = pymc.Poisson('deaths', mu = 2.0*mu, observed=True, value=[3])
model = pymc.MCMC((mu, deaths))

model.sample(10000, burn=100, burn_till_tuned=True)

print(model.summary())

pymc.Matplot.plot(model)


I am working on the file "BayesianDataAnalysis/Poisson_disease.ipynb".
The first task is:
This does not match the example very well. The example is centered around 0.9
Using a conjugate prior (gamma)
Can you write Python code for it?

mu = pymc.Gamma('mu', 3.0, 5.0)
deaths = pymc.Poisson('deaths', mu = 2.0*mu, observed=True, value=[3])
model = pymc.MCMC((mu, deaths))
model.sample(10000, burn=100, burn_till_tuned=True)
print(model.summary())
pymc.Matplot.plot(model)


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
< 20 seconds on my older laptop. That’s pretty good considering that NUTS is having a really hard time. Further below we make this even faster. To make it really fly, we probably want to run the Neural Network on the GPU.
As samples are more convenient to work with, we can very quickly draw samples from the variational posterior using sample_vp() (this is just sampling from Normal distributions, so not at all the same like MCMC):
Can you write Python code for it?

%%time

with neural_network:
    trace = pm.variational.sample_vp(v_params, draws=5000)


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
Model specification
A neural network is quite simple. The basic unit is a perceptron which is nothing more than logistic regression. We use many of these in parallel and then stack them up to get hidden layers. Here we will use 2 hidden layers with 5 neurons each which is sufficient for such a simple problem.
Can you write Python code for it?

# Trick: Turn inputs and outputs into shared variables.
# It's still the same thing, but we can later change the values of the shared variable
# (to switch in the test-data later) and pymc3 will just use the new data.
# Kind-of like a pointer we can redirect.
# For more info, see: http://deeplearning.net/software/theano/library/compile/shared.html
ann_input = theano.shared(X_train)
ann_output = theano.shared(Y_train)

n_hidden = 5

# Initialize random weights between each layer
init_1 = np.random.randn(X.shape[1], n_hidden).astype(floatX)
init_2 = np.random.randn(n_hidden, n_hidden).astype(floatX)
init_out = np.random.randn(n_hidden).astype(floatX)

with pm.Model() as neural_network:
    # Weights from input to hidden layer
    weights_in_1 = pm.Normal('w_in_1', 0, sd=1,
                             shape=(X.shape[1], n_hidden),
                             testval=init_1)

    # Weights from 1st to 2nd layer
    weights_1_2 = pm.Normal('w_1_2', 0, sd=1,
                            shape=(n_hidden, n_hidden),
                            testval=init_2)

    # Weights from hidden layer to output
    weights_2_out = pm.Normal('w_2_out', 0, sd=1,
                              shape=(n_hidden,),
                              testval=init_out)

    # Build neural-network using tanh activation function
    act_1 = pm.math.tanh(pm.math.dot(ann_input,
                                     weights_in_1))
    act_2 = pm.math.tanh(pm.math.dot(act_1,
                                     weights_1_2))
    act_out = pm.math.sigmoid(pm.math.dot(act_2,
                                          weights_2_out))

    # Binary classification -> Bernoulli likelihood
    out = pm.Bernoulli('out',
                       act_out,
                       observed=ann_output)


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
Now that we trained our model, lets predict on the hold-out set using a posterior predictive check (PPC). We use sample_ppc() <http://pymc-devs.github.io/pymc3/api.html#pymc3.sampling.sample_ppc>__ to generate new data (in this case class predictions) from the posterior (sampled from the variational estimation).
Can you write Python code for it?

# Replace shared variables with testing set
ann_input.set_value(X_test)
ann_output.set_value(Y_test)

# Creater posterior predictive samples
ppc = pm.sample_ppc(trace, model=neural_network, samples=500)

# Use probability of > 0.5 to assume prediction of class 1
pred = ppc['out'].mean(axis=0) > 0.5

fig, ax = plt.subplots()
ax.scatter(X_test[pred==0, 0], X_test[pred==0, 1])
ax.scatter(X_test[pred==1, 0], X_test[pred==1, 1], color='r')
sns.despine()
ax.set(title='Predicted labels in testing set', xlabel='X', ylabel='Y');

print('Accuracy = {}%'.format((Y_test == pred).mean() * 100))


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
Plotting the objective function (ELBO) we can see that the optimization slowly improves the fit over time.
Can you write Python code for it?


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
Probability surface
Can you write Python code for it?

cmap = sns.diverging_palette(250, 12, s=85, l=25, as_cmap=True)
fig, ax = plt.subplots(figsize=(10, 6))
contour = ax.contourf(*grid, ppc['out'].mean(axis=0).reshape(100, 100), cmap=cmap)
ax.scatter(X_test[pred==0, 0], X_test[pred==0, 1])
ax.scatter(X_test[pred==1, 0], X_test[pred==1, 1], color='r')
cbar = plt.colorbar(contour, ax=ax)
_ = ax.set(xlim=(-3, 3), ylim=(-3, 3), xlabel='X', ylabel='Y');
cbar.ax.set_ylabel('Posterior predictive mean probability of class label = 0');


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
That’s not so bad. The Normal priors help regularize the weights. Usually we would add a constant b to the inputs but I omitted it here to keep the code cleaner.
Variational Inference: Scaling model complexity
We could now just run a MCMC sampler like NUTS <http://pymc-devs.github.io/pymc3/api.html#nuts>__ which works pretty well in this case but as I already mentioned, this will become very slow as we scale our model up to deeper architectures with more layers.
Instead, we will use the brand-new ADVI variational inference algorithm which was recently added to PyMC3. This is much faster and will scale better. Note, that this is a mean-field approximation so we ignore correlations in the posterior.
Can you write Python code for it?

%%time

with neural_network:
    # Run ADVI which returns posterior means, standard deviations, and the evidence lower bound (ELBO)
    v_params = pm.variational.advi(n=50000)


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
Uncertainty in predicted value
So far, everything I showed we could have done with a non-Bayesian Neural Network. The mean of the posterior predictive for each class-label should be identical to maximum likelihood predicted values. However, we can also look at the standard deviation of the posterior predictive to get a sense for the uncertainty in our predictions. Here is what that looks like:
Can you write Python code for it?

cmap = sns.cubehelix_palette(light=1, as_cmap=True)
fig, ax = plt.subplots(figsize=(10, 6))
contour = ax.contourf(*grid, ppc['out'].std(axis=0).reshape(100, 100), cmap=cmap)
ax.scatter(X_test[pred==0, 0], X_test[pred==0, 1])
ax.scatter(X_test[pred==1, 0], X_test[pred==1, 1], color='r')
cbar = plt.colorbar(contour, ax=ax)
_ = ax.set(xlim=(-3, 3), ylim=(-3, 3), xlabel='X', ylabel='Y');
cbar.ax.set_ylabel('Uncertainty (posterior predictive standard deviation)');


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
We can see that very close to the decision boundary, our uncertainty as to which label to predict is highest. You can imagine that associating predictions with uncertainty is a critical property for many applications like health care. To further maximize accuracy, we might want to train the model primarily on samples from that high-uncertainty region.
Mini-batch ADVI: Scaling data size
So far, we have trained our model on all data at once. Obviously this won’t scale to something like ImageNet. Moreover, training on mini-batches of data (stochastic gradient descent) avoids local minima and can lead to faster convergence.
Fortunately, ADVI can be run on mini-batches as well. It just requires some setting up:
Can you write Python code for it?

from six.moves import zip

# Set back to original data to retrain
ann_input.set_value(X_train)
ann_output.set_value(Y_train)

# Tensors and RV that will be using mini-batches
minibatch_tensors = [ann_input, ann_output]
minibatch_RVs = [out]

# Generator that returns mini-batches in each iteration
def create_minibatch(data):
    rng = np.random.RandomState(0)

    while True:
        # Return random data samples of set size 100 each iteration
        ixs = rng.randint(len(data), size=50)
        yield data[ixs]

minibatches = zip(
    create_minibatch(X_train),
    create_minibatch(Y_train),
)

total_size = len(Y_train)


I am working on the file "BayesianNetwork/Neural Networks in PyMC3.ipynb".
The first task is:
While the above might look a bit daunting, I really like the design. Especially the fact that you define a generator allows for great flexibility. In principle, we could just pool from a database there and not have to keep all the data in RAM.
Lets pass those to advi_minibatch():
Can you write Python code for it?

%%time

with neural_network:
    # Run advi_minibatch
    v_params = pm.variational.advi_minibatch(
        n=50000, minibatch_tensors=minibatch_tensors,
        minibatch_RVs=minibatch_RVs, minibatches=minibatches,
        total_size=total_size, learning_rate=1e-2, epsilon=1.0
    )

with neural_network:
    trace = pm.variational.sample_vp(v_params, draws=5000)

plt.plot(v_params.elbo_vals)
plt.ylabel('ELBO')
plt.xlabel('iteration')
sns.despine()


I am working on the file "BayesianNetworks.ipynb".
The first task is:
Building an inference engine
Can you write Python code for it?

#index_names = [a for a in set(['A', 'S', 'T', 'L', 'B', 'E', 'X', 'D'])]
index_names = ['A', 'S', 'T', 'L', 'B', 'E', 'X', 'D']
visible_names = ['X', 'A', 'D']
parents = random_parents(index_names)
cardinalities = random_cardinalities(index_names)
states = random_states(index_names, cardinalities)

show_dag_image(index_names, parents)

families = make_families(index_names, parents)
visibles = make_visibles(index_names, visible_names)

Clique, elim_seq = make_cliques(families, cardinalities, visibles=None)

gamma = 0.01
theta = make_random_cp_tables(index_names, cardinalities, parents, gamma)

index_names = ['A', 'S', 'T', 'L', 'B', 'E', 'X', 'D']
parents = {'A':[], 'S':[], 'T':['A'], 'L':['S'], 'B':['S'], 'E':['T','L'], 'X':['E'], 'D':['B','E']}
show_dag_image(index_names, parents)
cardinalities = [2,2,2,2,2,2,2,2]
states = states_from_cardinalities(index_names, cardinalities)

visible_names = ['A', 'X', 'D']
visibles = index_names_to_num(index_names, visible_names)
## Generate random potentials
gamma = 0.01
theta = random_cp_tables(index_names, cardinalities, parents, gamma)

#X = random_observations(cardinalities, visibles)

eng = Engine(index_names, parents, states, theta, visible_names)
eng.propagate_observation({})

eng.sample_table(visible_names, num_of_samples=1000 )

eng.propagate_observation()
eng.marginal_table(visible_names)

X = eng.sample_table(visible_names, num_of_samples=1000)

X

E_S_new = eng.compute_ESS()

P = multiply(eng.theta, range(len(cardinalities)))
E_S = P

E_S_new = eng.propagate(X)

P = multiply(eng.theta, range(len(cardinalities)))
Px = marginalize(P, visibles, cardinalities)
P /= Px
E_S = X*P


I am working on the file "BayesianNetworks.ipynb".
The first task is:
Creation of the Inference Engine
Can you write Python code for it?

eng = Engine(index_names, parents, states, theta)


I am working on the file "BayesianNetworks.ipynb".
The first task is:
Example Query: Given a banana is chosen, what is the probability that Alice took it from Box1?
We need to compute the posterior $$p(\text{Box}| \text{Fruit} = \text{banana}) = \frac{ p(\text{Fruit} = \text{banana}|\text{Box}) p(\text{Box})}{p(\text{Fruit} = \text{banana})}$$
BJN implements two methods for calculating the answer for this query.

propagate_observation
propagate_table

In the first method, propagate_observation, a single observation is represented by a catalog with key-value pairs where keys are observed variable names and values are observed states. This is the standard method for inference in Bayesian networks.
Can you write Python code for it?

#obs = {'Fruit': 'Orange', 'Box': 'Box1'}
#obs = {'Fruit': 'Orange'}
obs = {'Fruit':'Banana'}
eng = Engine(index_names, parents, states, theta)
eng.propagate_observation(obs)

marg_names = ['Box']
print(normalize(eng.compute_marginal(marg_names)))

# Independent verification
#print(normalize(multiply(theta, [0,1])[:,1]))
print(normalize(multiply(theta, [0,1])[:,2]))


I am working on the file "BayesianNetworks.ipynb".
The first task is:
Example: A Simple chain
Can you write Python code for it?

## Define the model
index_names = ['i', 'j', 'k']
cardinalities = [10, 20, 3]
parents = {'i': [], 'j': ['k'], 'k': ['i']}

show_dag_image(index_names, parents)

#parents = {'k': [], 'i': ['k'], 'j': ['k']}
visible_names = ['i','j']
visibles = index_names_to_num(index_names, visible_names)


I am working on the file "BayesianNetworks.ipynb".
The first task is:
Example: Chest Clinic
The chest clinic model (originally known also as Asia) is a famous toy medical expert system example from the classical 1988 paper of Lauritzen and Spiegelhalter.
https://www.jstor.org/stable/pdf/2345762.pdf
Shortness-of-breath (dyspnoea) may be due to tuberculosis, lung cancer or
 bronchitis, or none of them, or more than one of them. A recent visit to Asia
 increases the chances of tuberculosis, while smoking is known to be a risk factor
 for both lung cancer and bronchitis. The results of a single chest X-ray do not
 discriminate between lung cancer and tuberculosis, as neither does the presence
 or absence of dyspnoea.
Also see: http://www.bnlearn.com/documentation/man/asia.html
In this toy domain, a patient can have three diseases:

T: Tuberculosis
L: Lung Cancer
B: Bronchitis

We also have the following possible a-priori effects on contracting a disease:

A: There is a Tuberculosis epidemic in Asia, so a visit may have an effect on Tuberculosis
S: Smoking history is known to have an effect both on Lung cancer and Bronchitis

Logical variables may be present, for example
* E: Either T or L or both
Symptoms or medical test results

D: The patient can show Dyspnea: Difficult breathing; shortness of breath, depending on T,L or B
X: X Ray gives a positive response if either Lung Cancer or Tuberclosis, or both. B does not have a direct efect on the outcome of the X-Ray
Can you write Python code for it?

# [A][S][T|A][L|S][B|S][E|T:L][X|E][D|B:E]

index_names = ['A', 'S', 'T', 'L', 'B', 'E', 'X', 'D']
parents = {'A':[], 'S':[], 'T':['A'], 'L':['S'], 'B':['S'], 'E':['T','L'], 'X':['E'], 'D':['B','E']}

## A Method for systematically entering the conditional probability tables

# P(A = Yes) = 0.01
# P(S = Yes) = 0.5
# P(T=Positive | A=Yes) = 0.05
# P(T=Positive | A=No) = 0.01
# P(L=Positive | S=Yes) = 0.1
# P(L=Positive | S=No) = 0.01
# P(B=Positive | S=Yes) = 0.6
# P(B=Positive | S=No) = 0.3
# P(E=True | L=Positive, T=Positive) = 1
# P(E=True | L=Positive, T=Negative) = 1
# P(E=True | L=Negative, T=Positive) = 1
# P(E=True | L=Negative, T=Negative) = 0
# P(X=Positive | E=True) = 0.98
# P(X=Positive | E=False) = 0.05
# P(D=Positive | E=True, B=Positive) = 0.9
# P(D=Positive | E=True, B=Negative) = 0.7
# P(D=Positive | E=False, B=Positive) = 0.8
# P(D=Positive | E=False, B=Negative) = 0.1

states = {'A':['No', 'Yes'],
          'S':['No', 'Yes'],
          'T':['Negative','Positive'],
          'L':['Negative','Positive'],
          'B':['Negative','Positive'],
          'E':['False', 'True'],
          'X':['Negative','Positive'],
          'D':['Negative','Positive']}

cardinalities = cardinalities_from_states(index_names, states)

# Conditional Probability Tables
cp_tables = {('A',): [0.99, 0.01],
             ('S',): [0.5, 0.5],
             ('T','A'): [[0.99, 0.01],[0.95,0.05]],
             ('L','S'): [[0.99, 0.01],[0.9,0.1]],
             ('B','S'): [[0.7,0.3],[0.4, 0.6]],
             ('E','T','L'): [[[1.,0.],[0.,1.]] , [[0.,1.],[0.,1.]]],
             ('X','E'): [[0.95, 0.05], [0.02, 0.98]],
             ('D','B','E'):[[[0.9,0.1],[0.2,0.8]],[[0.3,0.7],[0.1,0.9]]]
            }

# Todo: write a converter from a standard bn format

theta = make_cp_tables(index_names, cardinalities, cp_tables)
show_dag_image(index_names, parents)

eng = Engine(index_names, parents, states, theta)

eng.propagate_observation({'X': 'Positive', 'D':'Positive',...


I am working on the file "BayesianNetworks.ipynb".
The first task is:
Example: Two dice
Two fair die are thrown. Their sum comes up $9$. What is the face value of the first dice?
Can you write Python code for it?

from itertools import product
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt

## Define the model
index_names = ['Die1', 'Die2', 'Sum']

parents = {'Die1': [], 'Die2': [], 'Sum': ['Die1', 'Die2']}

states = {'Die1': list(range(1,7)), 'Die2': list(range(1,7)), 'Sum': list(range(2,13))}
cardinalities = cardinalities_from_states(index_names, states)

show_dag_image(index_names, parents)

S = np.zeros(cardinalities)

for i,j in product(range(1,7),repeat=2):
    S[i-1,j-1,i+j-2] = 1

cp_tables = {('Die1',): [1/6, 1/6,1/6,1/6,1/6,1/6],
             ('Die2',): [1/6, 1/6,1/6,1/6,1/6,1/6],
             ('Sum','Die1','Die2'): S
    }

theta = make_cp_tables(index_names, cardinalities, cp_tables )
eng = Engine(index_names, parents, states, theta)

eng.propagate_observation({'Sum':8})
marg = eng.compute_marginal(['Die1', 'Die2'], normalization=True)
marg

vname = 'Die2'
eng.propagate_observation({'Sum':9})
marg = eng.singleton_marginals([vname], states)
a = [x for x in marg[vname].keys()]
b = [marg[vname][x] for x in marg[vname].keys()]

plt.title(vname)
plt.bar(a, normalize(b))
plt.ylim([0, 1])
plt.show()

ev_names = ['Sum']
ev_counts = {(2,):1, (3,):1, (4,):1}
C = counts_to_table(ev_names, ev_counts, index_names, states)
eng.propagate_table(C)

marg_names = ['Die1']
#idx = index_names_to_num(index_names, ['Box'])
print(normalize(eng.compute_marginal(marg_names)))

#marg_names = ['Sum','Die2', 'Die1']
marg_names = ['Die2', 'Die1']
eng.propagate_observation({'Sum':9})
clamped = eng.values_clamped
table_to_counts(eng.compute_marginal(marg_names), marg_names, index_names, states, clamped)

eng.compute_marginal(marg_names)

marg_names = ['Sum','Die2', 'Die1']
eng.propagate_observation({'Sum': 11})
eng.marginal_table(marg_names, normalization=True)

marg_names = ['Sum','Die2', 'Die1']
T = eng.singleton_marginals(marg_names, normalization=True)
T


I am working on the file "BayesianNetworks.ipynb".
The first task is:
In the second method, propagate_table, we assume that we have obtained a collection
of observations and we are given the counts of each possible configuration.
Can you write Python code for it?

ev_names = ['Fruit']
ev_counts = {('Banana',): 1}

ev_table = counts_to_table(ev_names, ev_counts, index_names, states)
eng = Engine(index_names, parents, states, theta, visible_names=ev_names)
eng.propagate_table(ev_table)

marg_names = ['Box']
#idx = index_names_to_num(index_names, ['Box'])
print(normalize(eng.compute_marginal(marg_names)))

# Independent verification
print(normalize(multiply(theta, [0,1])[:,2]))


I am working on the file "BayesianNetworks.ipynb".
The first task is:
Samplers
Can you write Python code for it?

def sample_indices(index_names, parents, cardinalities, theta, num_of_samples=1):
    '''
    Sample directly the indices given a Bayesian network
    '''
    N = len(index_names)
    order = topological_order(index_names, parents)
    X = []

    for count in range(num_of_samples):
        x = [[]]*N
        for n in order:
            varname = index_names[n]

            idx = index_names_to_num(index_names,parents[varname])

            j = [0]*N
            for i in idx:
                j[i] = x[i]

            I_n = cardinalities[n]
            j[n] = tuple(range(I_n))
            #print(j)
            #print(theta[n][j])
            x[n] = np.random.choice(I_n, p=theta[n][j].flatten())
            #print(x)
        X.append(x)
    return X

def sample_states(var_names, states, index_names, parents, theta, num_of_samples=1):
    """
    Returns a dict with keys as state_name tuples and values as counts.
    This function generates each sample separately, so if
    num_of_samples is large, consider using sample_counts

    """
    N = len(index_names)
    order = topological_order(index_names, parents)

    X = dict()
    nums = index_names_to_num(index_names,var_names)
    cardinalities = cardinalities_from_states(index_names, states)

    shape = clique_shape(cardinalities, nums)

    for count in range(num_of_samples):
        x = [[]]*N
        for n in order:
            varname = index_names[n]

            idx = index_names_to_num(index_names,parents[varname])

            j = [0]*N
            for i in idx:
                j[i] = x[i]

            I_n = cardinalities[n]
            j[n] = tuple(range(I_n))
            #print(j)
            #print(theta[n][j])
            x[n] = np.random.choice(I_n, p=theta[n][j].flatten())
            #print(x)

        key = tuple((states[index_names[n]][x[n]] for n in nums))
        X[key] = X.get(key, 0) + 1

    return X

def counts_to_table(var_names, ev_counts, index_names, states):
    """
    Given...


I am working on the file "BayesianNetworks.ipynb".
The first task is:
Spanning tree and graph traversal
Can you write Python code for it?

def mst(E, N):
    """
    Generate a Spanning Tree of a graph with N nodes by Kruskal's algorithm,
    given preordered edge set E with each edge as (weight, v1, v2)

    For a minimum spanning tree, use
    E.sort()
    mst(E, N)

    For a maximum spanning tree, use
    E.sort(reverse=True)
    mst(E, N)
    """

    parent = list(range(N))
    spanning_tree = {i:[] for i in range(N)}

    def find_v(vertex):
        v = vertex
        while parent[v] != v:
            v = parent[v]
        return v

    def union(v1, v2):
        root1 = find_v(v1)
        root2 = find_v(v2)
        if root1 != root2:
            parent[root2] = root1

    for edge in E:
        weight, v1, v2 = edge
        p1, p2 = find_v(v1), find_v(v2)
        if p1 != p2:
            union(p1, p2)
            spanning_tree[v1].append(v2)
            spanning_tree[v2].append(v1)

    return spanning_tree

def bfs(adj_list, root):
    """
        Breadth-first search starting from the root

        adj_list : A list of lists where adj_list[n] denotes the set of nodes that can be reached from node n

        Returns a BFS order, and a BFS tree as an array parent[i]
        The root node has parent[rootnode] = -1
    """
    N = len(adj_list)
    visited = [False]*N
    parent = [-1]*N

    queue = [root]
    order = []
    while queue:
        v = queue.pop(0)
        if not visited[v]:
            visited[v] = True
            for w in adj_list[v]:
                if not visited[w]:
                    parent[w] = v
                    queue.append(w)
            order.append(v)

    return order, parent

def is_leaf(i, parent):
    return  not (i in parent)

def is_root(i, parent):
    return parent[i] == -1

def make_list_receive_from(parent):
    lst = [[] for i in range(len(parent)) ]
    for i,p in enumerate(parent):
        if p!= -1:
            lst[p].append(i)

    return lst


I am working on the file "BayesianNetworks.ipynb".
The first task is:
The advantage of using propagate_table is that if several observations are given on
the same variables, as is often the case in applications, the expected sufficient statistics can be computed in one collect-distribute iteration instead of a separate one for each observation.
Can you write Python code for it?

ev_names = ['Fruit']
ev_counts = {('Orange',):3, ('Banana',):1}
#ev_counts = {('Orange',):2, ('Banana',):5, ('Apple',):12}
#ev_counts = {('Orange',):1}
C = counts_to_table(ev_names, ev_counts, index_names, states)

eng.propagate_table(C)

marg_names = ['Box']
#idx = index_names_to_num(index_names, ['Box'])
print(normalize(eng.compute_marginal(marg_names)))

eng.marginal_table(marg_names)

ev_names = ['Fruit','Box']
ev_counts = {('Apple','Box1'):12, ('Orange','Box2'):2, ('Banana','Box2'):5}

eng = Engine(index_names, parents, states, theta, visible_names=ev_names)
C = counts_to_table(ev_names, ev_counts, index_names, states)
eng.propagate_table(C)

marg_names = ['Fruit', 'Box']
#idx = index_names_to_num(index_names, ['Box'])
print(normalize(eng.compute_marginal(marg_names)))

var_names = ['Fruit', 'Box']
ev_counts = sample_states(var_names, states, index_names, parents, theta, num_of_samples=1000)
C = counts_to_table(var_names, ev_counts, index_names, states)
#eng = Engine(index_names, visible_names, parents, cardinalities, theta)

#C = evidence_to_counts(X, index_names)
print(ev_counts)
print(C)


I am working on the file "BayesianNetworks.ipynb".
The first task is:
https://github.com/jupyter-widgets/tutorial/blob/master/notebooks/04.00-widget-list.ipynb
Can you write Python code for it?

%matplotlib inline

import matplotlib as mpl
import matplotlib.pyplot as plt

import numpy as np

from ipywidgets import interact, interactive, fixed
import ipywidgets as widgets
from IPython.display import clear_output, display, HTML
from matplotlib import rc

mpl.rc('font',**{'family':'sans-serif','sans-serif':['Helvetica']})
mpl.rc('text', usetex=True)

fig = plt.figure(figsize=(5,5))
ax = plt.gca()
ln = plt.Line2D([0],[0])

ax.add_line(ln)
ax.set_xlim([-1,1])
ax.set_ylim([-1,1])
ax.set_axis_off()

plt.close(fig)

def set_line(th):
    ln.set_xdata([np.cos(th), -np.cos(th)])
    ln.set_ydata([np.sin(th), -np.sin(th)])
    display(fig)

interact(set_line, th=(0.0, 2*np.pi,0.01))

widgets.IntSlider(
    value=7,
    min=0,
    max=10,
    step=1,
    description='Test:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='d'
)

widgets.FloatSlider(
    value=7.5,
    min=0,
    max=10.0,
    step=0.1,
    description='Test:',
    disabled=False,
    continuous_update=False,
    orientation='vertical',
    readout=True,
    readout_format='.2f',
)

w = widgets.IntRangeSlider(
    value=[5, 7],
    min=0,
    max=10,
    step=1,
    description='Test:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='d',
)
display(w)

w.get_interact_value()

w = widgets.BoundedIntText(
    value=7,
    min=0,
    max=10,
    step=1,
    description='Text:',
    disabled=False
)

display(w)

w.value

accordion = widgets.Accordion(children=[widgets.IntSlider(), widgets.Text()])
accordion.set_title(0, 'Slider')
accordion.set_title(1, 'Text')
accordion

tab_nest = widgets.Tab()
tab_nest.children = [accordion, accordion]
tab_nest.set_title(0, 'An accordion')
tab_nest.set_title(1, 'Copy of the accordion')
tab_nest

items = [widgets.Label(str(i)) for i in range(4)]
w2 = widgets.HBox(items)
display(w2)

items = [widgets.Label(str(i)) for i in range(4)]
left_box = widgets.VBox([items[0],...


I am working on the file "BayesianOptimization/BayesianOptimization.ipynb".
The first task is:
Acquisition Function
With acquisition function we determine where to sample next from our GP prior to best achieve our optimization objective. This functions yields an automatic optimized choice between exploration (where GP posterior variance is high) and exploitation (where the mean of GP is high). We will choose new set of parameters where exploitation is high (high GP mean) and also exploration is high (high GP uncertainty).
Different options exist for the choice of acquisition fucntion:
- Expected improvement: $a_{EI}(x,{x_n,y_n},\theta) = \sigma_{GP}(x,{x_n,y_n},\theta)\left[\gamma(x)\Phi\left(\gamma(x)\right) + \mathcal{N}(\gamma(x); 0, 1)\right]$
where
$\gamma(x)=\frac{f(x_{best}) - \mu_{GP}(x,{x_n,y_n},\theta)}{\sigma_{GP}(x,{x_n,y_n},\theta)}$
$\Phi$ is the normal cumulative distribution and $x{best}$ is the location of the lowest posterior mean._
In conlusion, each time we get new observed point the question is where to look next to get the optimum (here assume maximum) point in the underlying unseen function; and by finding the optimum point (min or max) of an acquizition function we will choose between exploitation/exploration dillema. The good thing about EI acquisition function is that, it has no parameter of itself and the choice is automatic for optimum point which will be the next point of sampling from the undelying function. Further on we will use LBFGS algorithem to find the min of the acquisition function (or negative of ac function for its maximum) but this can be also part of the OB itself.
Can you write Python code for it?

def expected_improvement(x, xp, yp, gp_params, kappa=0.0, n_params = 1):

    xpredict = np.asarray(x).reshape(-1, n_params)

    GP_mu, GP_sigma = GPR(xpredict, xp, yp, gp_params)
    GP_sigma = GP_sigma.reshape(-1,1)
    f_best = np.max(yp)

    #print n_params, xp.shape,yp.shape,xpredict.shape
    with np.errstate(divide='ignore'):
        gamma_x = (GP_mu - f_best) / GP_sigma
        EIx = GP_sigma * (gamma_x * norm.cdf(gamma_x) + norm.pdf(gamma_x))
        EIx[GP_sigma == 0.0] == 0
    return -1*EIx # negative because we will use a minimizer to find the maximum

def upper_confidence_bound(x, xp, yp, gp_params, kappa=0.0, n_params = 1):
    xpredict = np.asarray(x).reshape(-1, n_params)
    GP_mu, GP_sigma = GPR(xpredict, xp, yp, gp_params)
    GP_sigma = GP_sigma.reshape(-1,n_params)
    return -1*(GP_mu + kappa * GP_sigma)

def sample_next_hyperparameter(acquisition_func, xp, yp, gp_params, bounds, kappa=0.0, n_restarts=10):
    '''n_restarts: integer.
            Number of times to run the minimizer with different starting points.'''
    best_x = None
    best_acquisition_value = 1
    n_params = bounds.shape[0]
    for starting_point in np.random.uniform(bounds[:, 0], bounds[:, 1], size=(n_restarts, n_params)):
        res = minimize(fun=acquisition_func,
                       x0=starting_point.reshape(-1, n_params),
                       bounds=bounds,
                       method='L-BFGS-B',
                       args=(xp,yp, gp_params,kappa, n_params))

        if res.fun < best_acquisition_value:
            best_acquisition_value = res.fun
            best_x = res.x

    return best_x


I am working on the file "BayesianOptimization/BayesianOptimization.ipynb".
The first task is:
Optimizing a Neural Network's Hyperparameters with Bayesian Optimization
Choosing the best configuration of a neural network (it's hidden layer depth, learning rate, batch size, ...) can be seen as an optimization which can be also targeted with bayesian optimization using gaussian processes regressors. Through out this section we will test BO's power in this problem.
First we define the initial network from TensorFlow MNIST tutorial:
Can you write Python code for it?

# Getting the data
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Create a 2 layer fully connected network
def multilayer_FCN(x, weights, biases):
    # Hidden layer with RELU activation
    layer_1 = tf.add(tf.matmul(x, weights['w1']), biases['b1'])
    layer_1 = tf.nn.relu(layer_1)
    # output layer with RELU activation
    layer_2 = tf.add(tf.matmul(layer_1, weights['w2']), biases['b2'])
    #layer_2 = tf.nn.relu(layer_2)
    return layer_2

# define a function which  will construct and train a network with our hyperparameter set
def train_network(learning_rate=0.5,batch_size=100,training_epochs=1,hidden_n=500):

    x = tf.placeholder(tf.float32, [None, 784])
    weights = {
        'w1': tf.Variable(tf.random_normal([784, hidden_n])),
        'w2': tf.Variable(tf.random_normal([hidden_n, 10])),
    }
    biases = {
        'b1': tf.Variable(tf.random_normal([hidden_n])),
        'b2': tf.Variable(tf.random_normal([10])),
    }

    y = tf.placeholder(tf.float32, [None, 10])

    # Construct model
    pred = multilayer_FCN(x, weights, biases)

    # Define loss and optimizer
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))
    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    # Initializing the variables
    init = tf.global_variables_initializer()

    # Launch the graph
    with tf.Session() as sess:
        sess.run(init)

        # Training cycle
        for epoch in range(training_epochs):
            avg_cost = 0.
            total_batch = int(mnist.train.num_examples/batch_size/2)
            # Loop over all batches
            for i in range(total_batch):
                batch_x, batch_y = mnist.train.next_batch(batch_size)
                # Run optimization op (backprop) and cost op (to get loss value)
                _, c = sess.run([optimizer, cost], feed_dict={x: batch_x,
                                          ...


I am working on the file "BayesianOptimization/BayesianOptimization.ipynb".
The first task is:
Shall Optimize!
Can you write Python code for it?

def BayesianOptimization(f, bounds,max_iter=10,n_pre_samples=2,
                         gp_params=(0.5, 1., 0.001),
                         fit_GP_every=0,dis_every=1,plot_res = 0, kappa = 0.0):
    x_list = []
    y_list = []

    n_param = bounds.shape[0]

    #randomly sample the function to be optimized
    for j, params in enumerate(np.random.uniform(bounds[:, 0], bounds[:, 1], (n_pre_samples, bounds.shape[0]))):
        x_list.append(params)
        y_list.append(f(params))
        #if dis_every and i%dis_every==0: print 'Generating sample %d ...'%i

    xp = np.array(x_list)
    yp = np.array(y_list)

    for i in range(max_iter):

        if fit_GP_every and i%fit_GP_every==0: gp_params =  fit_GP(xp,yp, gp_params) # sigmaf, l, sigman

        next_sample = sample_next_hyperparameter(expected_improvement, xp, yp,gp_params, bounds=bounds,n_restarts=20)
        #next_sample = sample_next_hyperparameter(upper_confidence_bound, xp, yp,gp_params, bounds=bounds,kappa=kappa)

        # avoid very close points
        while np.any(np.abs(next_sample - xp) <= np.finfo(float).eps):
            next_sample = np.random.uniform(bounds[:, 0], bounds[:, 1], bounds.shape[0])

        cv_score = f(next_sample)

        # save previous values for plotting
        prev_xp = xp
        prev_yp = yp

        # Updates
        x_list.append(next_sample)
        y_list.append(cv_score)

        xp = np.array(x_list)
        yp = np.array(y_list)

        if dis_every and i%dis_every==0:
            print 'Iter. # %d - Best Results: '%(i+1),
            for j,val in enumerate(next_sample):
                #print 'parameter_%d = %2.2f,'%(j,val),
                print 'parameter_%d = %2.2f,'%(j+1,xp[np.argmax(yp),j]),
            print ' val=%2.2f'%max(yp)
            if plot_res:
                if xp.shape[1] == 1: # 2D plots
                    fig = plt.figure(figsize=(8,8))
                    param_choices = np.arange(bounds[0,0],bounds[0,1],0.1).reshape(-1,xp.shape[1])
                   ...


I am working on the file "BayesianOptimization/BayesianOptimization.ipynb".
The first task is:
We will begin by a simple 1D optimization problem for a function that we dont know the closed form of it (for presentation purposes we will know thee analytic optimum point of our function). Then we will try with a 2D function and finally we will apply our be then prepared tools in a real neural network case. Let's begin with our fun! but first we need our building parts.
GP Tools
Here we will include code for all the GP tools that we will use further in the notebook. For closer explanation you can see my notebook on gaussian process regression.
Can you write Python code for it?

# %load GPR.py
# GP squared exponential with its own hyperparameters
def get_kernel(X1,X2,sigmaf,l,sigman):
    k = lambda x1,x2,sigmaf,l,sigman:(sigmaf**2)*np.exp(-(1/float(2*(l**2)))*np.dot((x1-x2),(x1-x2).T)) + (sigman**2);
    K = np.zeros((X1.shape[0],X2.shape[0]))
    for i in range(0,X1.shape[0]):
        for j in range(0,X2.shape[0]):
            if i==j:
                K[i,j] = k(X1[i,:],X2[j,:],sigmaf,l,sigman);
            else:
                K[i,j] = k(X1[i,:],X2[j,:],sigmaf,l,0);
    return K

# finding optimized parameters for GP by maximizing closed-form log p(y|x,theta)
def fit_GP(x,y, gp_params):
    # log p(y|x,theta) = 0.5y'K^-1y-0.5log|K|-0.5nlog2pi
    # sigman > measurement noise
    # sigmaf > maximum covariance value
    # l > lenght-scale of our GP with squared exponential kernel

    bounds = [(0.001,2),(0.001,1),(0.001,1)]
    logpxtheta = lambda p: 0.5*(np.dot(y.T,np.dot(inv(get_kernel(x,x,p[0],p[1],p[2])),y)) + np.log(det(get_kernel(x,x,p[0],p[1],p[2]))) + x.shape[0]*np.log(2*np.pi)).reshape(-1,1)[0];

    res = minimize(fun=logpxtheta,
                       x0=gp_params,
                       bounds=bounds,
                       method='L-BFGS-B')
    new_gp_params = res['x']
    #print 'new GP parameters', new_gp_params
    return new_gp_params

#putting everything about GP to use here
def GPR(x_predict,x,y,gp_params):

    pdim = x_predict.shape[0]

    sigmaf, l, sigman = gp_params

    K = get_kernel(x, x, sigmaf, l, sigman)
    K_s = get_kernel(x_predict, x, sigmaf, l, 0)
    K_ss = get_kernel(x_predict, x_predict, sigmaf, l, sigman)

    #print K.shape,K_s.shape, K_ss.shape, y.shape
    y_predict_mean = np.dot(np.dot(K_s,inv(K)),y).reshape(pdim,-1)
    y_predict_var = np.diag(K_ss - np.dot(K_s,(np.dot(inv(K),K_s.T))))#.reshape(-1,1)
    #print y_predict_mean.shape
    return (y_predict_mean, y_predict_var)


I am working on the file "Bayesian_LogReg.ipynb".
The first task is:
Finally, we can find a confidence interval for this quantity. This may be the best part about Bayesian statistics: we get to interpret confidence intervals the way we've always wanted to interpret them. We are 95% confident that the odds ratio lies within our interval!
Can you write Python code for it?

lb, ub = np.percentile(b, 2.5), np.percentile(b, 97.5)

print("P(%.3f < O.R. < %.3f) = 0.95"%(np.exp(3*lb),np.exp(3*ub)))


I am working on the file "Bayesian_LogReg.ipynb".
The first task is:
So how do age and education affect the probability of making more than $$50K?$ To answer this question, we can show how the probability of making more than $50K changes with age for a few different education levels. Here, we assume that the number of hours worked per week is fixed at 50. PyMC3 gives us a convenient way to plot the posterior predictive distribution. We need to give the function a linear model and a set of points to evaluate. We will pass in three different linear models: one with educ == 12 (finished high school), one with educ == 16 (finished undergrad) and one with educ == 19 (three years of grad school).
Can you write Python code for it?

# Linear model with hours == 50 and educ == 12
lm = lambda x, samples: 1 / (1 + np.exp(-(samples['Intercept'] +
                                          samples['age']*x +
                                          samples['age2']*np.square(x) +
                                          samples['educ']*12 +
                                          samples['hours']*50)))

# Linear model with hours == 50 and educ == 16
lm2 = lambda x, samples: 1 / (1 + np.exp(-(samples['Intercept'] +
                                          samples['age']*x +
                                          samples['age2']*np.square(x) +
                                          samples['educ']*16 +
                                          samples['hours']*50)))

# Linear model with hours == 50 and educ == 19
lm3 = lambda x, samples: 1 / (1 + np.exp(-(samples['Intercept'] +
                                          samples['age']*x +
                                          samples['age2']*np.square(x) +
                                          samples['educ']*19 +
                                          samples['hours']*50)))


I am working on the file "Bayesian_LogReg.ipynb".
The first task is:
Some results
One of the major benefits that makes Bayesian data analysis worth the extra computational effort in many circumstances is that we can be explicit about our uncertainty. Maximum likelihood returns a number, but how certain can we be that we found the right number? Instead, Bayesian inference returns a distribution over parameter values.
Below, I overlay a kernel density estimate onto a 2-D histogram of samples from $p(β_{1},β_{3}|D)$.
Can you write Python code for it?

plt.figure(figsize=(9,7))
plt.hist2d(trace['age'], trace['educ'], alpha=.5, cmap="Reds", bins=25)
plt.colorbar()
seaborn.kdeplot(trace['age'], trace['educ'])
plt.xlabel("beta_age")
plt.ylabel("beta_educ")
plt.show()


I am working on the file "Bayesian_Rugby.ipynb".
The first task is:
Conclusion

'All models are wrong, some are useful'
Model correctly predicts that Ireland would win 4 games
Model incorrectly predicted that England would come out on top
In reality it was very close
Recommendation: Don't use this model to bet on the Six Nations next years

Want to learn more?

Thomas Wiecki Blog on all things Bayesian
Twitter: @springcoil

Probabilistic Programming for Hackers -- IPython Notebook book on Bayesian stats using PyMC2

Doing Bayesian Data Analysis -- Great book by Kruschke.

Get PyMC3 alpha
Zalando Example
Can you write Python code for it?

from IPython.display import Image
Image(filename='the-most-interesting-man-in-the-world-meme-generator-i-don-t-give-many-talks-but-when-i-do-i-do-q-a-e38222 copy.jpg')


I am working on the file "Bayesian_Rugby.ipynb".
The first task is:
Diagnostics
Let's see if/how the model converged.
The home parameter looks good, and indicates that home field advantage amounts to goals per game at the intercept.
We can see that it converges just like the model for the Premier League in the other tutorial.
I wonder and this is left as a question if all field sports have models of this form that converge.
Can you write Python code for it?


I am working on the file "Bayesian_Rugby.ipynb".
The first task is:
Simulating a season
We would like to now simulate a season. Just to see what happens.
Can you write Python code for it?

def simulate_season():
    """
    Simulate a season once, using one random draw from the mcmc chain.
    """
    num_samples = atts.trace().shape[0]
    draw = np.random.randint(0, num_samples)
    atts_draw = pd.DataFrame({'att': atts.trace()[draw, :],})
    defs_draw = pd.DataFrame({'def': defs.trace()[draw, :],})
    home_draw = home.trace()[draw]
    intercept_draw = intercept.trace()[draw]
    season = df.copy()
    season = pd.merge(season, atts_draw, left_on='i_home', right_index=True)
    season = pd.merge(season, defs_draw, left_on='i_home', right_index=True)
    season = season.rename(columns = {'att': 'att_home', 'def': 'def_home'})
    season = pd.merge(season, atts_draw, left_on='i_away', right_index=True)
    season = pd.merge(season, defs_draw, left_on='i_away', right_index=True)
    season = season.rename(columns = {'att': 'att_away', 'def': 'def_away'})
    season['home'] = home_draw
    season['intercept'] = intercept_draw
    season['home_theta'] = season.apply(lambda x: math.exp(x['intercept'] +
                                                           x['home'] +
                                                           x['att_home'] +
                                                           x['def_away']), axis=1)
    season['away_theta'] = season.apply(lambda x: math.exp(x['intercept'] +
                                                           x['att_away'] +
                                                           x['def_home']), axis=1)
    season['home_goals'] = season.apply(lambda x: np.random.poisson(x['home_theta']), axis=1)
    season['away_goals'] = season.apply(lambda x: np.random.poisson(x['away_theta']), axis=1)
    season['home_outcome'] = season.apply(lambda x: 'win' if x['home_goals'] > x['away_goals'] else
                                                    'loss' if x['home_goals'] < x['away_goals'] else 'draw', axis=1)
    season['away_outcome'] = season.apply(lambda x: 'win' if x['home_goals'] < x['away_goals'] else
                                    ...


I am working on the file "Bayesian_Rugby.ipynb".
The first task is:
Simulation

We are going to simulate 1000 seasons
Can you write Python code for it?

simuls = simulate_seasons(1000)

def fig1():
    ax = simuls.points[simuls.team == 'Ireland'].hist()
    median = simuls.points[simuls.team == 'Ireland'].median()
    ax.set_title('Ireland: 2015 points, 1000 simulations')
    ax.plot([median, median], ax.get_ylim())
    plt.annotate('Median: %s' % median, xy=(median + 1, ax.get_ylim()[1]-10))

fig1()


I am working on the file "Bayesian_Rugby.ipynb".
The first task is:
Six Nations Rugby

Rugby is a physical sport popular worldwide.
Six Nations consists of Italy, Ireland, Scotland, England, France and Wales
Game consists of scoring tries (similar to touch downs) or kicking the goal.
Average player is something like 100kg and 1.82m tall.
Paul O'Connell the Irish captain is Height: 6' 6" (1.98 m) Weight: 243 lbs (110 kg)

They compete for this!

Significant this year because the World Cup occurs in 2015.

Photo: Hostelrome

Motivation

Your estimate of the strength of a team depends on your estimates of the other strengths
Ireland are a stronger team than Italy for example - but by how much?
Source for Results 2014 are Wikipedia.
I handcrafted these results
Small data

Can you write Python code for it?

DATA_DIR = os.path.join(os.getcwd(), 'data/')

#The results_2014 is a handcrafted results table from Wikipedia
data_file = DATA_DIR + 'results_2014.csv'
df = pd.read_csv(data_file, sep=',')
df.tail()

teams = df.home_team.unique()
teams = pd.DataFrame(teams, columns=['team'])
teams['i'] = teams.index
teams.head()


I am working on the file "Bayesian_Rugby.ipynb".
The first task is:
So what have we learned so far, we've got 1000 simulations of Ireland and their median points in the table is 8.
In Rugby you get 2 points per win, and there are 5 games per year. So this model predicted that Ireland would win most of the time 4 games.
Can you write Python code for it?

def fig2():
    ax = simuls.gf[simuls.team == 'Ireland'].hist(figsize=(7,5))
    median = simuls.gf[simuls.team == 'Ireland'].median()
    ax.set_title('Ireland: 2015 scores for, 1000 simulations')
    ax.plot([median, median], ax.get_ylim())
    plt.annotate('Median: %s' % median, xy=(median + 1, ax.get_ylim()[1]-10))

fig2()


I am working on the file "Bayesian_Rugby.ipynb".
The first task is:
We can see in this analysis that home advantage gives about 0.55 points advantage.
We also see not too much auto-correlation, so this looks quite good plot wise.
We see here how probabilistic programming allows us to quantify our uncertainty about certain parameters.
Can you write Python code for it?

pymc.Matplot.plot(atts)
# We can plot all of the parameters, just to see.

observed_season = DATA_DIR + 'table_2014.csv'
df_observed = pd.read_csv(observed_season)
df_observed.loc[df_observed.QR.isnull(), 'QR'] = ''
df_avg = pd.DataFrame({'avg_att': atts.stats()['mean'],
                       'avg_def': defs.stats()['mean']},
                      index=teams.team.values)

df_new = df_avg.merge(df_observed, left_index=True, right_on='team', how='left')

#Bring in the new csv file
df_results = pd.read_csv('df_avg_data_edited.csv', sep=';')

df_results.tail()

def fig3():
    fig, ax = plt.subplots(figsize=(12,12))
    for outcome in ['winners', 'wooden_spoon', 'triple_crown', 'NaN']:
        ax.plot(df_results.avg_att[df_results.QR == outcome],
    df_results.avg_def[df_results.QR == outcome], 'o', label=outcome)
    for label, x, y in zip(df_results.team.values, df_results.avg_att.values, df_results.avg_def.values):
        ax.annotate(label, xy=(x,y), xytext = (-5,5), textcoords = 'offset points')
        ax.set_title('Attack vs Defense avg effect: 13-14 Six Nations')
        ax.set_xlabel('Avg attack effect')
        ax.set_ylabel('Avg defense effect')
        ax.legend()

fig3()


I am working on the file "Bayesian_Rugby.ipynb".
The first task is:
What happened in reality?

Well Ireland actually scored 119 points, so the model over predicted this!
We call this 'shrinkage' in the literature.
All models are wrong, but some are useful

What are the predictions of the model?

So let us look at the winning team on average.
We do a simulation and we'll assign probability of 'winning' to the team
We used the MCMC to do this.
Can you write Python code for it?

g = simuls.groupby('team')
df_champs = pd.DataFrame({'percent_champs': g.champion.mean()})
df_champs = df_champs.sort_index(by='percent_champs')
df_champs = df_champs[df_champs.percent_champs > .05]
df_champs = df_champs.reset_index()

fig, ax = plt.subplots(figsize=(8,6))
ax.barh(df_champs.index.values, df_champs.percent_champs.values)

for i, row in df_champs.iterrows():
    label = "{0:.1f}%".format(100 * row['percent_champs'])
    ax.annotate(label, xy=(row['percent_champs'], i), xytext = (3, 10), textcoords = 'offset points')
ax.set_ylabel('Team')
ax.set_title('% of Simulated Seasons In Which Team Finished Winners')
_= ax.set_yticks(df_champs.index + .5)
_= ax.set_yticklabels(df_champs['team'].values)


I am working on the file "Bayesian_bootstrap.ipynb".
The first task is:
Below we apply the -Bayesian, but could just as well be classical- bootstrap method to a linear regression by bootstrapping the data.
This is not the only way to apply the bootstrap. One could fix the regressors/covariates $x_i$, as well as the regression coefficients, $\mathbf{\beta}$, of the linear fit to the original dataset, and then bootstrap the residuals $y_i - \hat{y}_i(\mathbf{\beta})$. The former approach is expected to be more conservative (give larger confidence intervals) than the latter, which is more model dependent in that it fixes the coefficients and thereby implicitly assumes the linear model is essentially correct and only the random variation/noise of the data needs to be bootstrapped.
For more on this see the corresponding section in the original work:
https://books.google.co.uk/books?id=gLlpIUxRntoC&lpg=PA113&ots=A8xyY7Lcz3&dq=regression%20bootstrap%20data%20or%20residuals&pg=PA113#v=onepage&q=regression%20bootstrap%20data%20or%20residuals&f=false
For a short, concise and clear explanation of the pros and cons of each way of bootstrapping regression models see:
http://www.stat.cmu.edu/~cshalizi/uADA/13/lectures/which-bootstrap-when.pdf
Can you write Python code for it?

slopes = linregres_posterior[:,0]
slopemean = slopes.mean()
slopeCI = CI(slopes)
print "The mean slope and its 95% CI are:\t{0:.4g}\t\t[{1:.4g}, {2:.4g}]".format(slopemean,slopeCI[0],slopeCI[1])

intercepts = linregres_posterior[:,1]
interceptmean = intercept.mean()
interceptCI = CI(intercepts)
print "The mean intercept and its 95% CI are:\t{0:.4g}\t\t[{1:.4g}, {2:.4g}]".format(interceptmean,interceptCI[0],
                                                                                interceptCI[1])

# Plot the data points
plt.scatter(data['order'].values, data['height_cm'].values)

# The linear function we will use to plot fit coefficients
def linfit(x,slope,intercept):
    return slope*x + intercept

x = data['order'].values
y = data['height_cm'].values

# Choose linear regressions for 10 of the bootstrap samples at random and plot them
ids = npr.randint(0, linregres_posterior.shape[0], 10)
otherfits = [linfit(x, linregres_posterior[i,0], linregres_posterior[i,1]) for i in ids]
for i in otherfits:
    plt.plot(x, i, color='#BBBBBB')

# The fit to the original data
slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
plt.plot(x, linfit(x, slope, intercept), color='black', linewidth=2)

plt.xlim(0,x.max()+1)
plt.show()


I am working on the file "Bayesian_bootstrap.ipynb".
The first task is:
Define a function to compute confidence intervals and use it
Can you write Python code for it?

def CI(sample, alpha=0.05):
    """Returns the 100*(1-alpha)% confidence level interval for a test statistic computed on a bootstrap sample."""
    sample.sort()
    num_samples = sample.shape[0]
    low_ci = sample[int((alpha / 2.0) * num_samples)]
    high_ci = sample[int((1 - alpha / 2.0) * num_samples)]
    return [low_ci, high_ci]

meanCI = CI(posterior_mean, alpha=0.05)

print "The mean of the posterior is:\t{0:.4g}".format(posterior_mean.mean())
print "With confidence interval:\t[{0:.4g}, {1:.4g}]".format(meanCI[0],meanCI[1])
#print posterior_median.mean(), CI(posterior_median)

fig,ax =plt.subplots(2,1, sharex=True)
ax[0].hist(height_data, color='blue')
ax[0].set_xlabel('Heights of American Presidents (in cm)')
ax[0].set_ylabel('Frequency')
ax[1].hist(posterior_mean, color='blue')
ax[1].set_xlabel('Bayesian Bootstrap posterior of the mean (95% CI in red)')
ax[1].set_ylabel('Frequency')
ax[1].plot([meanCI[0], meanCI[1]], [0, 0], 'r', linewidth=8)
plt.show()

from scipy import stats

x = data['order'].values
y = data['height_cm'].values

slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
print slope
print intercept

def bayes_bstrp1(data, statistic, nbstrp, samplesize):
    """Implements the Bayesian bootstrap method."""
    """Input can be a 1D Numpy array, or for test statistics of two variables: a Pandas DataFrame with two columns: x,y"""

    def Dirichlet_sample(m,n):
        """Returns a matrix of values drawn from a Dirichlet distribution with parameters = 1.
        'm' rows of values, with 'n' Dirichlet draws in each one."""
        Dirichlet_params = np.ones(m*n) # Set Dirichlet distribution parameters
        # https://en.wikipedia.org/wiki/Dirichlet_distribution#Gamma_distribution
        Dirichlet_weights = np.asarray([random.gammavariate(a,1) for a in Dirichlet_params]) # Draw from Gamma distrib
        Dirichlet_weights = Dirichlet_weights.reshape(m,n) # Fold them (row by row) into a matrix
        row_sums = Dirichlet_weights.sum(axis=1)
        Dirichlet_weights =...


I am working on the file "Bayesian_bootstrap.ipynb".
The first task is:
From this plot and the confidence interval on the slope we can confidently say that there is no evidence for a correlation between the two variables.
Can you write Python code for it?

from statsmodels.nonparametric.smoothers_lowess import lowess

# for some odd reason this loess function takes the y values as the first argument and x as second
test_df = data[['height_cm', 'order']]

posterior_loess = bayes_bstrp1(test_df, lowess, nbstrp=100, samplesize=60)
print posterior_loess

x = data['order'].values
y = data['height_cm'].values

# To see all the loess curves found:
#for i in posterior_loess:
#    plt.plot(i[:,0], i[:,1], color='#BBBBBB')

ids = npr.randint(0, posterior_loess.shape[0], 20)
for i in ids:
    plt.plot(posterior_loess[i,:,0], posterior_loess[i,:,1], color='#BBBBBB')

plt.scatter(x, y)

original_loess = lowess(y, x)
plt.plot(original_loess[:,0], original_loess[:,1], color='black', linewidth=2)

plt.xlim(0,x.max()+1)
plt.show()


I am working on the file "Bayesian_bootstrap.ipynb".
The first task is:
Test both the weighted statistic method and the weighted sampling methods
Can you write Python code for it?

height_data = data['height_cm'].values

posterior_mean = bayes_bstrp(height_data, np.mean, nbstrp=10000, samplesize=1000)
print posterior_mean

posterior_median = bayes_bstrp(height_data, np.median, nbstrp=10000, samplesize=1000)
print posterior_median


I am working on the file "Bayesian_bootstrap.ipynb".
The first task is:
The Bayesian bootstrap (with a Dirichlet prior)
(See:
http://sumsar.net/blog/2015/04/the-non-parametric-bootstrap-as-a-bayesian-model/
and
http://projecteuclid.org/euclid.aos/1176345338
)
Can you write Python code for it?

def bayes_bstrp(data, statistic, nbstrp, samplesize):
    """Implements the Bayesian bootstrap method."""

    def Dirichlet_sample(m,n):
        """Returns a matrix of values drawn from a Dirichlet distribution with parameters = 1.
        'm' rows of values, with 'n' Dirichlet draws in each one."""
        # Draw from Gamma distribution
        Dirichlet_params = np.ones(m*n) # Set Dirichlet distribution parameters
        # https://en.wikipedia.org/wiki/Dirichlet_distribution#Gamma_distribution
        Dirichlet_weights = np.asarray([random.gammavariate(a,1) for a in Dirichlet_params])
        Dirichlet_weights = Dirichlet_weights.reshape(m,n) # Fold them (row by row) into a matrix
        row_sums = Dirichlet_weights.sum(axis=1)
        Dirichlet_weights = Dirichlet_weights / row_sums[:, np.newaxis] # Reweight each row to be normalised to 1
        return Dirichlet_weights

    Dirich_wgts_matrix = Dirichlet_sample(nbstrp, data.shape[0]) #Generate sample of Dirichlet weights

    # If statistic can be directly computed using the weights (such as the mean), do this since it will be faster.
    if statistic==np.mean or statistic==np.average:
        results = np.asarray([np.average(data, weights=Dirich_wgts_matrix[i]) for i in xrange(nbstrp)])
        return results

    # Otherwise resort to sampling according to the Dirichlet weights and computing the statistic
    else:
        results = np.zeros(nbstrp)
        for i in xrange(nbstrp): #Sample from data according to Dirichlet weights
            weighted_sample = np.random.choice(data, samplesize, replace=True, p = Dirich_wgts_matrix[i])
            results[i] = statistic(weighted_sample) #Compute the statistic for each sample
        return results


I am working on the file "Beam Splitter Leonhart.ipynb".
The first task is:
Note: this is different from van Enk (NotesBS.pdf) by the sign and a factor of i. It agrees with several other papers in the reference folder. TODO: sort this out.
Now to let the outputs interfere again after a phase shift in path 2:
Can you write Python code for it?

def phaseshifter(phi):
    """ I believe this shifts one arm by phi
    doesn't seem to be working though"""
    shifter = tensor(identity(N),exp(1j*phi)*identity(N))
    return shifter

def stateplot(state):
    plt.plot(real(state.full()),"bo")
    plt.plot(imag(state.full()),"ro")

philist = linspace(0,2*pi,20)

input2 = [phaseshifter(pi/8)*psi1 for phi in philist]
out2 = bs.dag() * input2
plt.plot(philist,expect(n1,out2),label="$n_1$")
plt.plot(philist,expect(n2,out2),label="$n_2$")
plt.legend()

stateplot(test3[6])

expect(n2,out2)

expect(n1,out2)

stateplot(psi3[0])

# Explore the phase-shifted version of the 1 photon output by hard-coding the state

psi3 = [1/sqrt(2)*(exp(1j*phi)*tp(0,1) + tp(1,0)) for phi in philist]
test3 = bs.dag()*psi3
plt.plot(philist,expect(n1,test3),label="$n_1$")
plt.plot(philist,expect(n2,test3),label="$n_2$")
plt.legend()

(tp(1,0)).full()


I am working on the file "Beam Splitter Leonhart.ipynb".
The first task is:
These all agree: so far so good.
Next, try a single photon input:
Can you write Python code for it?


I am working on the file "Beam_equation.ipynb".
The first task is:
$$\Delta(x) = x^{3} \left(- \frac{L w}{12 E I} - \frac{w x_{0}}{6 E I}\right) + x^{2} \left(\frac{L w x_{0}}{4 E I} + \frac{w x_{0}^{2}}{4 E I}\right) + x \left(\frac{L^{3} w}{24 E I} - \frac{L w x_{0}^{2}}{4 E I} - \frac{w x_{0}^{3}}{6 E I}\right) + y_{0} - \frac{L^{3} w x_{0}}{24 E I} + \frac{L w x_{0}^{3}}{12 E I} + \frac{w x^{4}}{24 E I} + \frac{w x_{0}^{4}}{24 E I}$$
$$\Delta(x) = \frac{w}{2 EI}\left(\frac{x^{4}}{12} - \left(\frac{L}{6 } + \frac{x_{0}}{3}\right)x^{3}  + \left(\frac{L  x_{0}}{2} + \frac{x_{0}^{2}}{2}\right) x^{2}  + \left(\frac{L^{3}}{12} - \frac{L x_{0}^{2}}{2} - \frac{x_{0}^{3}}{3}\right) x \right)  +\frac{w}{2 E I}\left(- \frac{L^{3}  x_{0}}{12} + \frac{L x_{0}^{3}}{6}  + \frac{ x_{0}^{4}}{12}\right)+ y_{0}$$
$$\Delta(x) = \frac{w}{2 EI}\left(\frac{x^{4}}{12} - \frac{1}{3}\left(\frac{L}{2} + x_{0}\right)x^{3}  + \frac{1}{2}\left(L  x_{0} + x_{0}^{2}\right) x^{2}  + \left(\frac{L^{3}}{12} - \frac{L x_{0}^{2}}{2} - \frac{x_{0}^{3}}{3}\right) x \right)  +\frac{w}{2 E I}\frac{1}{6}\left(- \frac{L^{3}  x_{0}}{2} + L x_{0}^{3}  + \frac{ x_{0}^{4}}{2}\right)+ y_{0}$$
The following was not used due to another approach being found.
The following calculations were done in an effort to determine what the loacation of the supports needed to be if they did not move in the drawing when the beam deflects.  This is necessary due to the simplifiaction made where the ends of the beam centerline do not move when the bem deflects.
Can you write Python code for it?


I am working on the file "BeanStalkSeries.ipynb".
The first task is:
Lets start with a Pentagonal Dodecahedron and build it from Es + e3s.
Can you write Python code for it?

PD = 3 * root2 * (ø ** 2 + 1)
print(PD)

E  = e = E_vol # shorthand (E3 = E * ø_up ** 3, e3 = E * ø_down ** 3, E = e)
e3 = e * ø_down ** 3
PD = 348 * E + 84 * e3
print(PD)


I am working on the file "BeanStalkSeries.ipynb".
The first task is:
Now lets import the tetravolume.py module, which in turn has dependencies, to get these volumes directly, based on edge lengths.  I'll use the edges given in Fig. 986.411 of Synergetics, spoking out from the point C at the center of any RT diamond, and/or values computed by David Koski.
First, lets get a color coded version of the E module...

Figure 2: Dissected E-mod with color-coded vertexes (Koski with vZome)

The black hub is at the center of the RT, as shown here...

Figure 3: RT center is the black hub (Koski with vZome)

The edges of the Fum module, tetrahedron Black-Orange-Yellow-Blue will be... (note R=1, D=2):
Can you write Python code for it?

# Edges needed for Fum and Emod
e0 = Black_Yellow  = root3 * ø_down
e1 = Black_Blue    = mpfr(1)  # raddfius of RT = 1 (same as unit-radius sphere)
e2 = Black_Orange  = 1/(rt2(ø**2+1)/2)
e3 = Yellow_Blue   = (3 - root5)/2
e4 = Blue_Orange   = (ø**-1)*(1/rt2(ø**2+1))
e5 = Orange_Yellow = rt2(Yellow_Blue**2 - Blue_Orange**2)

e6 = Black_Red  = rt2((5 - root5)/2)
e7 = Blue_Red   = 1/ø
e8 = Red_Yellow = rt2(5 - 2 * root5)

#print(e3 ** 2 + e7 ** 2)
#print(e8 ** 2)
#assert e3 ** 2 + e7 ** 2 == e8 ** 2  # check
#assert e4 ** 2 + e5 ** 2 == e3 ** 2  # check

# not needed for this computation
e9  = Black_Green = 20/(5 * root2 * ø**2) # Sfactor
e10 = Purple_Green = ø ** -4

for e in range(11):
    val = "e" + str(e)
    length = eval(val)
    print("Edge {:3} = {:40.37}".format(val, length))

import tetravolume as tv # has to be in your path, stored on Github with this JN
# D = 1 in this module, so final volume need to be divided by 8 to match R=1 (D=2)
# see Fig. 986.411A in Synergetics

Fum_vol = tv.Tetrahedron(e0,e1,e2,e3,e4,e5).ivm_volume()/8
E_vol   = tv.Tetrahedron(e1,e0,e6,e3,e8,e7).ivm_volume()/8
print("Fum volume (in tetravolumes): {:40.38}".format( Fum_vol ))
print("E volume (in tetravolumes)  : {:40.38}".format(  E_vol ))

Fe     = (ø**-7) * (rt2(2)/8)
Fi     = (ø**-6) * (rt2(2)/8)
Fo     = ((5-rt2(5))/5) * (ø**-4) * (rt2(2)/8)
Fum    = (rt2(5)/5) * (ø**-4)*(rt2(2)/8)
Fe_Fi  = (ø**-5) * (rt2(2)/8)
Fo_Fum = (ø**-4) * (rt2(2)/8)
print("Fe:    {:40.38}".format(Fe))
print("Fi:    {:40.38}".format(Fi))
print("Fo:    {:40.38}".format(Fo))
print("Fum:   {:40.38}".format(Fum))
print("E_vol: {:40.38}".format((Fe_Fi) + (Fo_Fum)))
print("E_vol: {:40.38}".format((ø**-3)*(rt2(2)/8)))


I am working on the file "BeanStalkSeries.ipynb".
The first task is:
RT3, on the other hand, has a volume we may express as:
Can you write Python code for it?


I am working on the file "BeanStalkSeries.ipynb".
The first task is:
Recall RT3 is the Rhombic Triacontahedron we get by intersecting the two Platonic duals:  Icosahedron (Icosa) and Pentagonal Dodecahedron (PD), with the former having edges = 2R and volume ~18.51 (5 * rt2(2) * ø ** 2).
It turns out that if we shave a Fum3 off an E3, and multiply by 120, we get the PD's volume.
Put another way: RT3 - PD leaves a volume of 120 Fum3 volumes.  In other words 120 * (E3 - Fum3) = PD.
Can you write Python code for it?

E3 = E_vol * ø_up ** 3
Fum3 = Fum_vol * ø_up ** 3
print(E3)
print(Fum3)
print(RT3 - PD)
print(120 * Fum3)


I am working on the file "BeautifulSoup/Improving Performance by Parsing Only Part of the Document.ipynb".
The first task is:
pare a tags with specified contecnt inside it
Can you write Python code for it?

linksToBob = SoupStrainer('a', href=re.compile('bob.com/'))
[tag for tag in BeautifulSoup(doc,"lxml", parse_only=linksToBob)]

mentionsOfBob = SoupStrainer(text=re.compile("Bob"))
[text for text in BeautifulSoup(doc,"lxml", parse_only=mentionsOfBob)]


I am working on the file "BeautifulSoup/Improving Performance by Parsing Only Part of the Document.ipynb".
The first task is:
parse only a tag
Can you write Python code for it?

links = SoupStrainer('a')
[tag for tag in BeautifulSoup(doc,"lxml",parse_only=links)]


I am working on the file "BeautifulSoup/Improving Performance by Parsing Only Part of the Document.ipynb".
The first task is:
specify criterion for tags
Can you write Python code for it?

allCaps = SoupStrainer(text=lambda t:t.upper()==t)
[text for text in BeautifulSoup(doc,"lxml", parse_only=allCaps)]


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
Activity Stream and Top Styles
These two screen captures above show a heavy user's profile as it appeared on April (figure 1, above) and August of 2009 (figure 2, above). Aside from the aesthetic changes, several key changes stand out. Most notable is the introduction of a new "activity" stream (figure 3, below). This feature is a sort of automated social awareness stream of an individual's UGC across the site, likely inspired by the success of Facebook around this time. Furthermore, while statistics were previously available behind the "Statistics" tab on the old site, RateBeer replaced the simple reverse chronological listing of reviews with an interesting summary of the user's  contributions by beer style. This beer style chart, standing clearly out on its white background, also draws attention to a more prominent link to the full page of statistics.
In their MovieLens study, Breece et al. explain that "the collective effort model posits that people will socially loaf less when they perceive that their contribution is important to the group...if they think they are unique, they should be more motivated to contribute." [3] The top styles chart in particular, and the increased emphasis on each user's contribution characteristics over the plain listing of reviews more generally, appears to be designed to better represent the uniqueness of each contributor. These features allow "contributors to stand out," as Preece and Schneiderman put it.
Can you write Python code for it?


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
Average Review Length
Based on the results below, we notice that, across the board, the average review length very slightly increased after the profile-page change. However, the increase is very small (smaller than 1 word), and seems to reflect that the profile page change didn't markedly impact review length.
We note that the standard deviation for high-usage users is significantly smaller for this variable (perhaps due to longtime users have an anchoring affect to what reviews "should" be or having less reviews in the second period).
Can you write Python code for it?

# average review lengths before/after change
print 'all users review length avg'
reviews_all = group_time_split(users_split_active, 'review/text', lambda x: float(x))
reviews_all = map(lambda x: [[(len(review.split()),  x[0][review]) for review in x[0].keys()], [(len(review.split()),  x[1][review]) for review in x[1].keys()]], reviews_all)
avg_review_len = avg_from_user_array_tuple(reviews_all)
styles_plot = plot_diff(avg_review_len)
plt.xlabel('User ID (sorted by pre-split contributions)')
plt.ylabel('avg review length after - avg review length before')
plt.title('Average Review Length Plus-Minus')
print

print 'top users review length avg'
reviews_top = group_time_split(users_top_active, 'review/text', lambda x: float(x))
reviews_top = map(lambda x: [[(len(review.split()),  x[0][review]) for review in x[0].keys()], [(len(review.split()),  x[1][review]) for review in x[1].keys()]], reviews_top)
avg_review_len = avg_from_user_array_tuple(reviews_top)
styles_plot = plot_diff(avg_review_len, lambda x: x, False)
print

print 'low users review length avg'
reviews_low = group_time_split(users_low_active, 'review/text', lambda x: float(x))
reviews_low = map(lambda x: [[(len(review.split()),  x[0][review]) for review in x[0].keys()], [(len(review.split()),  x[1][review]) for review in x[1].keys()]], reviews_low)
avg_review_len = avg_from_user_array_tuple(reviews_low)
styles_plot = plot_diff(avg_review_len, lambda x: x, False)
print

print 'sampled users review length avg'
reviews_samp = group_time_split(users_sampled_active, 'review/text', lambda x: float(x))
reviews_samp = map(lambda x: [[(len(review.split()),  x[0][review]) for review in x[0].keys()], [(len(review.split()),  x[1][review]) for review in x[1].keys()]], reviews_samp)
avg_review_len = avg_from_user_array_tuple(reviews_samp)
styles_plot = plot_diff(avg_review_len, lambda x: x, False)
print


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
Explore Binary Classifier
Even though no clear linear correlation is apparent to us based on our analysis so far of the select features, we thought it'd be worthwhile to test different linear classification methods with our selected feature (especially considering the potential non-linear correlation we've noticed). By doing so, we would be able to evaluate how predictive our set of features really is (based on an actual model evaluation accuracy score).
Specifically, we were interested in whether our identified list of features could be used to predict whether a user's review contribution would increase or decrease followiing the change in the website. As we were attempting to create a positive/negative classifier, we first created a balanced pool of data to work from -- where there were an equal number of positive and negative samples (where positive/negative is determined by the user's review count plus-minus).  Following this, we split the data pool 80/20 into a training and test set. We chose to perform cross_validation (n=4) to evaluate different models and used a standard mean/standard deviation scaling to normalize our feature values.
For the purposes of this assignment, we looked at 4 different sklearn classifiers: rbf SVM, linear SVM, dual-l2 logistic regression, non-dual-l2 logistic regression. From a cross-validation score, the model which performed best was the rbf SVM algorithm.
Overall, this resulted in a 62% accuracy (consistent with the output from the other classifiers) -- a score that is not very good and not that much better than random. This leads us to believe that the features we selected are not strongly predictive of future review contributions to the RateBeer website.
Can you write Python code for it?

bal_data_pool = balance_pool(data_pool)
test, train = [], []
create_train_test(bal_data_pool, test, train)

train = [[user[0] for user in train], [1 if user[1] > 0 else -1 for user in train]]
test = [[user[0] for user in test], [1 if user[1] > 0 else -1 for user in test]]

from sklearn.preprocessing import StandardScaler
from sklearn import svm, neighbors, qda, metrics, cross_validation
from sklearn.pipeline import Pipeline

clf = Pipeline([('Scaler', StandardScaler()),
#                 ('Log-Reg', linear_model.LogisticRegression(penalty='l2', dual=True))])
#                     ('Log-Reg', linear_model.LogisticRegression(penalty='l2', dual=False))])
#                     ('SVC-linear', svm.SVC(kernel='linear'))])
                    ('SVC-rbf', svm.SVC(kernel='rbf'))])

cv = cross_validation.KFold(len(train[0]), n_folds=4, shuffle=True)
scores = cross_validation.cross_val_score(clf, train[0], train[1], cv=cv)
print "Cross Validation Scores:", scores
print "Average Cross Validation Score:", np.average(scores)
print
# peforms test on selected model
clf = clf.fit(train[0], train[1])
predicted = clf.predict(test[0])
print "Model Accuracy:", metrics.accuracy_score(test[1], predicted)
print "Confusion Matrix"
print metrics.confusion_matrix(test[1], predicted)


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
Explore Potential  Correlation
Based on the earlier high-level analysis performed, it seemed apparent that some of the features visibly exhibited patterns suggesting some form of correlation. In order to further explore this idea, we performed a correlation analysis (focusing on linear correlation) on each "pre-split" feature -- total number of reviews, number of styles reviewed, average overall rating, number of distinct ratings, average review length, and number of distinct review lengths.
For each of these features, we calculated the pearson correlation coefficient against both the number of post-split reviews and the review count plus-minus for each user.
Setting up Feature Vectors
Can you write Python code for it?

# build feature vectors
x, y_after, y_diff = [], [], []
for user in user_vals_split:
    pre_char = user[0]
    style_map = groupby_key_count(pre_char, 'beer/style')
    rating_map = groupby_key_count(pre_char, 'review/overall')
    review_map = groupby_key_count(pre_char, 'review/text', lambda x : len(x.split()))
    total_count = len(pre_char)
    style_count = len(style_map.keys())
    avg_rating = avg_from_map(rating_map)
    rating_count = len(rating_map.keys())
    avg_review = avg_from_map(review_map)
    review_count = len(review_map.keys())
    # throw away points that don't have rating / review
    if (avg_rating is not None and avg_review is not None):
        x.append([total_count, style_count, avg_rating, rating_count, avg_review, review_count])
        y_after.append(len(user[1]))
        y_diff.append(len(user[1])-len(user[0]))
data_pool = zip(x,y_diff)

def balance_pool(pool):
    """
    Given a pool of year-key formatted unbalanced outcome data, return a balanced set where positive outcomes have equal number of occurances to negative outcomes
    The larger set is randomly sampled to reduce its size by using reservoir_sample
    """
    newpool = []
    neg = []
    pos = []
    for user in pool:
        if user[-1] < 0:
            neg.append(user)
        else:
            pos.append(user)
    minlen = min(len(pos), len(neg))
    for elem in reservoir_sample(neg, minlen):
        newpool.append(elem)
    for elem in reservoir_sample(pos, minlen):
        newpool.append(elem)
    return newpool

def create_train_test(pool, test, train):
    """
    Split the data pool created in create_data_pool randomly into a 80/20 split between training data and testing data
    Shuffles all the years and randomly splits 80/20 between training and test
    Should only be ran once to randomly split train/test data as it will return different results between runs
    """
    random.shuffle(pool)
    ind = int(len(pool) * 0.8)
    train += pool[:ind]
    test += pool[ind:]


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
Helper functions
Can you write Python code for it?

"""
Helper to return unicode string as ascii with special characters removed.
"""
def clean_unicode(instr):
    return unicodedata.normalize('NFKD', instr).encode('ascii', 'ignore')

"""
Helper to parse and yield data from file to json
"""
def parse_json(filename, normalize=True):
    with open(filename) as f:
        for line in f:
            out = json.loads(line, encoding='latin-1')
            if normalize:
                temp = {}
                for key in out:
                    try:
                        key = clean_unicode(key)
                    except:
                        key = key
                    try:
                        temp[key] = clean_unicode(out[key])
                    except:
                        temp[key] = out[key]
                out = temp
            yield out
    f.close()

# via http://matplotlib.org/examples/pylab_examples/finance_work2.html
def moving_average(x, n, type='simple'):
    """
    compute an n period moving average.

    type is 'simple' | 'exponential'

    """
    x = np.asarray(x)
    if type=='simple':
        weights = np.ones(n)
    else:
        weights = np.exp(np.linspace(-1., 0., n))

    weights /= weights.sum()

    a =  np.convolve(x, weights, mode='full')[:len(x)]
    a[:n] = a[n]
    return a


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
High-level Feature Trends
As we brainstormed features which may reflect a user's engagement level, we eventually settled on the following variables: number of reviews contributed, review diversity (types of beers being reviewed), average review rating (score), and average review length (word count).
Specifically, we were interested in seeing how each of these variables changed "pre-split" to "post-split" based on the user's relative amount of reviews contributed "pre-split".
In order to look at these features, we sorted the user dataset based on the number of reviews each user contributed "pre-split" and calculated a plus-minus value for each of the variables we looked at (plus-minus calculated as value_post_split - value_pre_split). We then plotted these values for the entire population and calculated mean, median, and standard deviation.
Over the course of our analysis, we noticed that high-usage users typically exhibited more extreme changes in plus-minus. For this reason, we also performed the mean/median/stdev analysis for the top 5% of users. In order to provide a control for these high-usage users, we paired the high-usage statistics with the numbers for a ramdomly sampled group of the same size from the entire dataset as well as the "long tail" of 95% other users.
Can you write Python code for it?

# sample "top" 5% and "low" 95% of users
users_top = user_vals_split[int(math.floor(len(user_vals_split)*0.95)):]
users_low = user_vals_split[:int(math.floor(len(user_vals_split)*0.95))]

# sample "random" user group to be equal to the "top" group
sample_size = len(users_top)
users_sampled = reservoir_sample(user_vals_split, sample_size)

"""
Plot utility to plot the users's plus minus
Prints out the mean, median, and standard deviation for the plus minus
"""
def plot_diff(user_list, func=lambda x: x, plot=True):
    user_pre = [func(user[0]) for user in user_list]
    user_post = [func(user[1]) for user in user_list]

    np_pre = np.array(user_pre)
    np_post = np.array(user_post)
    np_pm = np_post - np_pre
    print "Mean p/m: ", np.mean(np_pm)
    print "Median p/m: ", np.median(np_pm)
    print "Std Dev p/m: ", np.std(np_pm)

    ind = np.arange(len(np_pm))
    if (plot):
        fig, ax = plt.subplots()
        scatt = ax.scatter(ind, np_pm, c="grey",s=10,edgecolor='none')
        mean = ax.axhline(y=np.mean(np_pm),xmin=0,xmax=ind[len(ind) - 1],c="blue",linewidth=0.9, linestyle = '-', zorder=1, label='mean')
        median = ax.axhline(y=np.median(np_pm),xmin=0,xmax=ind[len(ind) - 1],c="red",linewidth=0.9,linestyle = '--', zorder=2, label='median')
        plt.legend(loc=3, ncol=2)


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
Introduction
RateBeer.com 2009 User Profile Redesign
In an effort to confirm our hypothesis, we identified RateBeer redesigns that adhered to the RTLF principles. To pinpoint these redesigns, and to allow for a comparison of the site before and after the redesign, we scanned page archives captured by the Way Back Machine service (archive.org). We quickly realized that many site redesigns were incremental, and that large, site wide redesigns were comparatively rare occerence. One particular redesign did stick out, however, and that was the update to user profile pages launched by RateBeer administrators beginning on May 3, 2009 (forum posts from that time suggest the feature was rolled out of a period of a few weeks)[4].
[4] "Love the New Format on the Userpage..." RateBeer.com, 3 May 2009. Web. 28 Apr. 2015. http://www.ratebeer.com/forums/love-the-new-format-on-the-userpage_115257.htm.
RateBeer.com Profile Pages

April 2009 profile page (user MiP): https://web.archive.org/web/20090425001049/http://www.ratebeer.com/View-User-10185.htm
August 2009 profile page (user MiP): https://web.archive.org/web/20090822202711/http://www.ratebeer.com/View-User-10185.htm
Can you write Python code for it?


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
Pearson Correlation Coefficient
Based on our results for the Correlation Coefficient, only one of our features exibits a value suggesting a good linear fit: the correlation coefficient between number of pre-split reviews and the number of post-split reviews (a correlation coefficient of ~0.84).
Can you write Python code for it?

# calculate pearson correlation coefficient for each variable
# use to predict plus-minus and actual number of reviews after
import scipy.stats
np_x = np.array(x)
np_y_after = np.array(y_after)
np_y_diff = np.array(y_diff)
"""
Index to Pre-Split Feature Mapping
0 - total number of reviews
1 - number of styles reviewed
2 - average overall rating
3 - number of distinct ratings
4 - average review length
5 - number of distinct review lengths
"""
print "Pearson Correlation Coefficients Against Post-Split Number of Reviews"
for i in xrange(len(np_x[0])):
    print "pearson for char", i, ":", scipy.stats.pearsonr(np_x[:,i], np_y_after)
print
print "Pearson Correlation Coefficients Against Number of Reviews Plus-Minus"
for i in xrange(len(np_x[0])):
    print "Difference -- pearson for char", i, ":", scipy.stats.pearsonr(np_x[:,i], np_y_diff)


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
RateBeer Beer Style Review Trends by Month (cont.)
This second chart display only the "other" beer review totals, separated from the top 10 styles. This graph also adds a linear model, fit to the monthly "other" reviews data, which summarizes the general trend in review totals within this category. The flat trend, seen on the charts above and below, in contributions of "other" style beer reviews before and after the profile page redesign again contradicts our hypothesis that the diversity of contributions would increase following the redesign.
Can you write Python code for it?

plot_other_styles_monthly(months, other_counts, other_style_trend_line, current_palette, N_styles)


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
RateBeer Reviews by Date
This collection of functions produces a summary of the total number of reviews per day within the sample dataset. This aggregation is conducted on the pre-sorted data set, prepared during the parsing stage. This dataset is intended to help review the site-wide trends in UGC generation before and after the launch of the new RateBeer profiles in May 2009.
Can you write Python code for it?

# Sorted data doesn't fit comfortably in memory. This function provides access to an iterator over the data.
def sorted_data_iter():
    return parse_json(SORTED_FILE)

timestamp_keyfunc = lambda r: r.get('review/timeUnix')
date_keyfunc = lambda r: date.fromtimestamp(r.get('review/timeUnix'))

def groupby_daily(sorted_reviews):
    reviews_by_date = {}
    for key, group in groupby(sorted_reviews, date_keyfunc):
        reviews_by_date[key] = len(list(group))
    return reviews_by_date

reviews_by_date = groupby_daily(sorted_data_iter())

dates = sorted(reviews_by_date.keys())
review_counts = [reviews_by_date[k] for k in dates]

# Moving average of review counts by day
ma_days = 45
ma = moving_average(review_counts, ma_days, 'exponential')

# Linear trendline of review counts by day
polynomial_order = 3
trend_line = np.polyfit(range(len(dates)), review_counts, polynomial_order)

"""
Groups reviews by month
"""
user_keyfunc = lambda r: r.get('user/profileName')

def month_keyfunc(review):
    # Return the first day of the month for any day in a month
    review_date = date.fromtimestamp(review.get('review/timeUnix'))
    return date(review_date.year, review_date.month, 1)

def groupby_monthly(sorted_reviews):
    monthly_user_count = {}
    for key, group in groupby(sorted_reviews, month_keyfunc):
        user_sorted = sorted(group, key=user_keyfunc)
        monthly_user_count[key] = len(list(groupby(user_sorted, user_keyfunc)))
    return monthly_user_count

monthly_user_count = groupby_monthly(sorted_data_iter())
months = sorted(monthly_user_count.keys())
user_counts = [monthly_user_count[m] for m in months]

"""
Groups reviews by month, then by beer/style.
"""
beer_style_keyfunc = lambda r: r.get(u'beer/style')

def groupby_styles_monthly(sorted_reviews):
    monthly_beer_count = {}
    all_styles = {}
    for key_month, group in groupby(sorted_reviews, month_keyfunc):
        style_sorted = sorted(group, key=beer_style_keyfunc)
        monthly_beer_count[key_month] = {}
        for key_style, subgroup in...


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
RateBeer Reviews over Time
The chart below plots the total quantity of reviews by the RateBeer community before and after the launch of the new profile page in May 2009. The x-axis represents individual days within our sample data set, while the y-axis represents the total number of reviews contributed by RateBeer users on that date.  The launch date of the new profile pages is annotated on the x-axis for reference.
In an effort to better portray the trend of reviews over the time period of our sample set, which included all reviews +/- 1 year from the launch of the new profile pages, we also calculated a 45-day moving average of daily review totals. The moving average (red trend line) provides a cleaner depiction of the fluctuations in contribution of new reviews over time. The overall trend over the full time period is best represented by the linear model we fit to the full dataset (blue trendline).
These results indicate that, contrary to our hypothesis, there was no measurable increase in UGC corresponding with the launch of the new RateBeer user profile pages. If anything, the linear model suggets that reviews were trending somewhat downwards towards the end of our sample set.
Note: The chart below shows a clear gap in the sample set over a period of time in late December 2008. While this appeared to be a processing error at first glance, further investigation suggested that it reflects a period of downtime resulting from attacks on the RateBeer site around that time. [1]
[1]: "Bottoms Up." Bottoms Up Oddbits Ratebeer Hacked Sam Loses Money New Beers Coming Our Way Comments. Bottoms Up, 6 Nov. 2008. Web. 28 Apr. 2015. http://www.ibabuzz.com/bottomsup/2008/11/06/oddbits-ratebeer-hacked-sam-loses-money-new-beers-coming-our-way/.
Can you write Python code for it?

plot_reviews(dates, review_counts, ma, ma_days, trend_line, polynomial_order, current_palette)

def plot_users_monthly(months, user_counts, current_palette):
    fig, axes = plt.subplots(1, figsize=(16, 10))

    axes.plot(months, user_counts, color=current_palette[3], lw=1, label='Unique users')
    axes.fill_between(months, 0, user_counts, facecolor=current_palette[2], alpha=0.8)

    axes.annotate("Profile redesign",
              xy=(LAUNCH_DATE, 0),
              xytext=(LAUNCH_DATE, 500),
              arrowprops=dict(arrowstyle="->"),
              fontsize='large'
             )

    fig.autofmt_xdate()
    axes.set_title('RateBeer Unique Users by Month')
    axes.set_xlabel('Months')
    axes.set_ylabel('Users')
    axes.fmt_xdata = mdates.DateFormatter('%Y-%m')
    plt.show()


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
Review Counts
Based on the results below, we notice that, across the board, user contribution (as measured by review count) seem to have decreased after the profile-page change (counter what we would have predicted based on the increased behavioral incentive on the profile page). This includes users who didn't contribute in the second period at all (reflecting a lack of user retention).
We notice that, on average, high-usage users seem to have skewed the overall average review count downwards as the average "high-usage" user had a review count decrease of 76 as compared to ~3 for both the long-tail and the random sample group (which makes sense considering a finite cap in number of beers to be reviewed). We do notice that there does appear to be a fair number of "mid-usage" users who did have the number of reviews contributed increase after the change.
Can you write Python code for it?

# review counts
# all users stats
print 'all user review counts'
plot_diff(user_vals_split, lambda x: len(x))
plt.xlabel('User ID (sorted by pre-split contributions)')
plt.ylabel('review after - reviews before')
plt.title('Review Count Plus-Minus')
print

# top users stats
print 'top user review counts', len(users_top)
plot_diff(users_top, lambda x: len(x), False)
print

# low users stats
print 'low user review counts', len(users_low)
plot_diff(users_low, lambda x: len(x), False)
print

# sampled users stats
print 'sampled user review counts', len(users_sampled)
plot_diff(users_sampled, lambda x: len(x), False)
print

# remove users who did not post reviews after the split (for rest of metrics)
users_split_active = [user for user in user_vals_split if len(user[1]) > 0]
users_top_active = users_split_active[int(math.floor(len(users_split_active)*0.95)):]
users_low_active = users_split_active[:int(math.floor(len(users_split_active)*0.95))]
users_sampled_active = reservoir_sample(users_split_active, len(users_top_active))


I am working on the file "BeerDataAnalysis.ipynb".
The first task is:
User Level Results
In addition to looking at site-level impact, we also hypothesized that the website changes may have altered the way that individual users engaged with and contributed to the overall community. This portion of the analysis looks at trends in select features and identifies how a user's contribution level before the remodel correlated with observed contribution changes after.
We conclude this section by building a classification model to predict (based on a user's contributions before the "split") how the number of reviews contributed by the user change after the "split" [frome hereon referred to as "pre-split" and "post-split"].We test several different linear classifiers and evaluate how predictive the features identified really are.
For the purposes of this analysis, we focused on the following user feature variables:
* Number of Reviews Posted - Users more commited to the community will post more reviews
* Review Diversity - Users who are more engaged will post more and more diverse information
    * Types of Beers - Count
    * Rating Scores - Number of Different, Average
    * Review Length - Number of Different, Average
Data Parsing
Can you write Python code for it?

# import statements
import unicodedata
import random
import calendar
import math
import operator

"""
A variety of helper methods to group data structures accordingto
"""
def groupby_key(data, key_str):
    key_map = {}
    for datum in data:
        key = datum.get(key_str)
        key_map[key] = key_map.setdefault(key, [])
        key_map[key].append(datum)
    return key_map

def groupby_key_count(data, key_str, func=None):
    key_map = {}
    for datum in data:
        key = datum.get(key_str)
        try:
            key = func(key)
        except:
            key = key
        key_map[key] = key_map.setdefault(key, 0) + 1
    return key_map

def group_time_split(user_array, key, func=None):
    return [[groupby_key_count(time_per, key, func) for time_per in user] for user in user_array]

"""
Reservoir sampling given an iterable input and k for number of items to be sampled
"""
def reservoir_sample(iterator, k):
    iterator = iter(iterator)
    # fill the reservoir to start
    result = [next(iterator) for _ in range(k)]
    n = k
    for item in iterator:
        n += 1
        s = random.randint(0, n)
        if s < k:
            result[s] = item
    return result
"""
Calculates the average using dictionary keys as "values" and dictionary values as "counts"
"""
def avg_from_map(keymap):
    count, total, a = 0, 0, None
    for key in keymap:
        if (key):
            count += keymap[key]
            total += key * keymap[key]
    if (count):
        a = total / count
    return a
"""
average from user tuple array
"""
def avg_from_user_array_tuple(array):
    out = []
    for user in array:
        user_avg = []
        for time in user:
            count, total, a = 0, 0, None
            for item in time:
                count += item[1]
                total += item[0] * item[1]
            if count > 0:
                a = total/count
            user_avg.append(a)
        out.append(user_avg)
    return out

# Parse and read in dataset from reduced_data.txt (as produced by our parsing...


I am working on the file "Beer_User_Analyze.ipynb".
The first task is:
Average Number of Beer Styles Reviewed
Based on the results below, we notice that, across the board, the average number of beer styles reviewed decreased after the profile-page change (consistent with the way that the new page encourages users to specialize). However, part of this change may have to do with the overall decrease in number of reviews, so it doesn't appear conclusive. We notice that, on average, high-usage users have a steeper drop in style count than the whole and random sample.
Looking at the plot, we notice that there does appear to be a distinct pattern of having "low-contribution" users be more likely to have a positive plus-minus in this variable than "high-contribution" users. This makes sense as "low-usage" users have more beers available to them to review while "high-usage" users have less or the fact that "high-usage" users are more aware of their profile page. This seems to support the inconclusive nature of this feature.
Can you write Python code for it?

# average number of styles reviewed before/after change
print 'all users style counts'
styles_all = group_time_split(users_split_active, 'beer/style')
# want number of styles
styles_all = map(lambda x: [len(x[0].keys()), len(x[1].keys())], styles_all)
plot_diff(styles_all)
plt.xlabel('User ID (sorted by pre-split contributions)')
plt.ylabel('avg beer styles after - avg beer styles before')
plt.title('Average Beer Style Count Plus-Minus')
print

print 'top users style counts'
styles_top = group_time_split(users_top_active, 'beer/style')
# want number of styles
styles_top = map(lambda x: [len(x[0].keys()), len(x[1].keys())], styles_top)
plot_diff(styles_top, lambda x: x, False)
print

print 'low users style counts'
styles_low = group_time_split(users_low_active, 'beer/style')
# want number of styles
styles_low = map(lambda x: [len(x[0].keys()), len(x[1].keys())], styles_low)
plot_diff(styles_low, lambda x: x, False)
print

print 'sample users style counts'
styles_samp = group_time_split(users_sampled_active, 'beer/style')
# want number of styles
styles_samp = map(lambda x: [len(x[0].keys()), len(x[1].keys())], styles_samp)
plot_diff(styles_samp, lambda x: x, False)
print


I am working on the file "Beer_User_Analyze.ipynb".
The first task is:
Average Overall Rating
Based on the results below, we notice that, across the board, the average overall rating very slightly increased after the profile-page change. However, the increase is very small (smaller than 1%), and seems to reflect that the profile page change didn't markedly impact ratings.
We note that the standard deviation for high-usage users is significantly smaller for this variable (perhaps due to longtime users have an anchoring affect to what ratings "should" be or having less reviews in the second period).
Can you write Python code for it?

# average overall ratings before/after change
print 'all users rating avg'
rates_all = group_time_split(users_split_active, 'review/overall', lambda x: float(x))
rates_all = map(lambda x: [[(rating,  x[0][rating]) for rating in x[0].keys()], [(rating,  x[1][rating]) for rating in x[1].keys()]], rates_all)
avg_rates = avg_from_user_array_tuple(rates_all)
plot_diff(avg_rates)
plt.xlabel('User ID (sorted by pre-split contributions)')
plt.ylabel('avg overall rating after - avg overall rating before')
plt.title('Average Overall Rating Plus-Minus (Max Rating = 1)')
print

print 'top users rating avg'
rates_top = group_time_split(users_top_active, 'review/overall', lambda x: float(x))
rates_top = map(lambda x: [[(rating,  x[0][rating]) for rating in x[0].keys()], [(rating,  x[1][rating]) for rating in x[1].keys()]], rates_top)
avg_rates = avg_from_user_array_tuple(rates_top)
plot_diff(avg_rates, lambda x: x, False)
print

print 'low users rating avg'
rates_low = group_time_split(users_low_active, 'review/overall', lambda x: float(x))
rates_low = map(lambda x: [[(rating,  x[0][rating]) for rating in x[0].keys()], [(rating,  x[1][rating]) for rating in x[1].keys()]], rates_low)
avg_rates = avg_from_user_array_tuple(rates_low)
plot_diff(avg_rates, lambda x: x, False)
print

print 'sampled users rating avg'
rates_samp = group_time_split(users_sampled_active, 'review/overall', lambda x: float(x))
rates_samp = map(lambda x: [[(rating,  x[0][rating]) for rating in x[0].keys()], [(rating,  x[1][rating]) for rating in x[1].keys()]], rates_samp)
avg_rates = avg_from_user_array_tuple(rates_samp)
styles_plot = plot_diff(avg_rates, lambda x: x, False)
print