I am working on the file "Lab1.ipynb".
The first task is:
Why Do We Pass in None?
A note about ‘None’ and fluid-sized dimensions:
You may notice that the first dimension of shape of data_placeholder is ‘None’. data_placeholder should have shape (num_tweets, vocab_size). However, we don’t know how many tweets we are going to be passing in at a time, num_tweets is unknown. Its possible that we only want to pass in 1 tweet at a time, or 30, or 1,000. Thankfully, TensorFlow allows us to specify placeholders with fluid-sized dimensions. We can use None to specify some fluid dimension of our shape. When our data eventually gets passed in as a numpy array, TensorFlow can figure out what the value of the fluid-size dimension should be.
Step 3: Define Network Parameters
Let’s now define and initialize our network parameters.
We'll our model parameters using tf.Variable. When you create a tf.Variable you pass a Tensor as its initial value to the Variable() constructor. A Tensor is a term for any N-dimensional matrix. There are a ton of different initial Tensor value functions you can use (full list). All these functions take a list argument that determines their shape. Here we use tf.truncated_normal for our weights, and tf.zeros for our biases. Its important that the shape of these parameters are compatible. We’ll be matrix-multiplying the weights, so the last dimension of the previous weight matrix must equal the first dimension of the next weight matrix. Notice this pattern in the following tensor initialization code. Lastly, notice the size of the tensor for our last weights. We are predicting a vector of size n_classes so our network needs to end with n_classes nodes.
Can you write Python code for it?

# Define Network Parameters

# Here, we can define how many units will be in each hidden layer.
n_hidden_units_h0 = 512
n_hidden_units_h1 = 256

# Weights going from input to first hidden layer.
# The first value passed into tf.Variable is initialization of the variable.
# We initialize our weights to be sampled from a truncated normal, as this does something
# called symmetry breaking and keeps the neural network from getting stuck at the start.
# Since the weight matrix multiplies the previous layer to get inputs to the next layer,
# its size is prev_layer_size x next_layer_size.
h0_weights = tf.Variable(
    tf.truncated_normal([vocab_size, n_hidden_units_h0]),
    name='h0_weights')
h0_biases = tf.Variable(tf.zeros([n_hidden_units_h0]),
                     name='h0_biases')

'''TODO: Set up variables for the weights going into the second hidden layer.
You can check out the tf.Variable API here: https://www.tensorflow.org/api_docs/python/tf/Variable.
''';
h1_weights = #todo
h1_biases = #todo

# Weights going into the output layer.
out_weights = tf.Variable(
    tf.truncated_normal([n_hidden_units_h1, n_classes]),
    name='out_weights')
out_biases = tf.Variable(tf.zeros([n_classes]),
                     name='out_biases')


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
Group by play type
Can you write Python code for it?


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
Let's take a cursory glance at the data to see what we're working with.
Can you write Python code for it?


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
Look at the number of yards gained by a FirstDown
Can you write Python code for it?


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
Now all of the objects are encoded the way we'd like them to be
Can you write Python code for it?


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
THIS IS SOME MORE REFORMATTING SHIT I'M DOING FOR NOW. PROLLY GONNA KEEP IT
Can you write Python code for it?

df['PassOutcome'].replace(['Complete', 'Incomplete Pass'], [1, 0], inplace=True)

df = df[df["PlayType"] != 'Quarter End']
df = df[df["PlayType"] != 'Two Minute Warning']
df = df[df["PlayType"] != 'End of Game']


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
There's a lot of data that we don't care about. For example, 'PassAttempt' is a binary attribute, but there's also an attribute called 'PlayType' which is set to 'Pass' for a passing play.
We define a list of the columns which we're not interested in, and then we delete them
Can you write Python code for it?

columns_to_delete = ['Unnamed: 0', 'Date', 'time', 'TimeUnder',
                     'PosTeamScore', 'PassAttempt', 'RushAttempt',
                     'DefTeamScore', 'Season', 'PlayAttempted']

#Iterate through and delete the columns we don't want
for col in columns_to_delete:
    if col in df:
        del df[col]


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
We can eliminate combos who didn't have at least 10 receptions together, and then re-sample the data. This will remove noise from QB-receiver combos who have very high or low completion rates because they've played very little together.
Can you write Python code for it?

size = 10
corr = df.corr()
fig, ax = plt.subplots(figsize=(size, size))
ax.matshow(corr)
plt.xticks(range(len(corr.columns)), corr.columns);
for tick in ax.get_xticklabels():
    tick.set_rotation(90)
plt.yticks(range(len(corr.columns)), corr.columns);


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
We can then grab a list of the remaining column names
Can you write Python code for it?


I am working on the file "Lab1/.ipynb_checkpoints/Lab1-Travis-checkpoint.ipynb".
The first task is:
We define four lists based on the types of features we're using.
Binary features are separated from the other categorical features so that they can be stored in less space
Can you write Python code for it?

continuous_features = ['TimeSecs', 'PlayTimeDiff', 'yrdln', 'yrdline100',
                       'ydstogo', 'ydsnet', 'Yards.Gained', 'Penalty.Yards',
                       'ScoreDiff', 'AbsScoreDiff']

ordinal_features = ['Drive', 'qtr', 'down']
binary_features = ['GoalToGo', 'FirstDown','sp', 'Touchdown', 'Safety', 'Fumble']
categorical_features = df.columns.difference(continuous_features).difference(ordinal_features)


I am working on the file "Lab1/Lab1-Travis-Copy1.ipynb".
The first task is:
At this point, lots of things are encoded as objects, or with excesively large data types
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis-Copy1.ipynb".
The first task is:
Let's take a cursory glance at the data to see what we're working with.
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis-Copy1.ipynb".
The first task is:
Now all of the objects are encoded the way we'd like them to be
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis-Copy1.ipynb".
The first task is:
Now we can start to take a look at what's in each of our columns
Can you write Python code for it?

df.describe()

import matplotlib.pyplot as plt
import warnings
warnings.simplefilter('ignore', DeprecationWarning)
#Embed figures in the Jupyter Notebook
%matplotlib inline

#Use GGPlot style for matplotlib
plt.style.use('ggplot')

pass_plays = df[df['PlayType'] == "Pass"]
pass_plays_grouped = pass_plays.groupby(by=['Passer'])


I am working on the file "Lab1/Lab1-Travis-Copy1.ipynb".
The first task is:
There's a lot of data that we don't care about. For example, 'PassAttempt' is a binary attribute, but there's also an attribute called 'PlayType' which is set to 'Pass' for a passing play.
We define a list of the columns which we're not interested in, and then we delete them
Can you write Python code for it?

columns_to_delete = ['Unnamed: 0', 'Date', 'time',
                     'PosTeamScore', 'PassAttempt', 'RushAttempt',
                     'DefTeamScore', 'Season', 'PlayAttempted']

#Iterate through and delete the columns we don't want
for col in columns_to_delete:
    if col in df:
        del df[col]


I am working on the file "Lab1/Lab1-Travis-Copy1.ipynb".
The first task is:
We can also extract the highest-completion percentage combos.
Here we take the top-10 most reliable QB-receiver pairs.
Can you write Python code for it?

import seaborn as sns
%matplotlib inline

# df_dropped = df.dropna()
# df_dropped.info()
selected_types = df.select_dtypes(exclude=["object"])

useful_attributes = df[['FieldGoalDistance','ydstogo']]
print(useful_attributes)

sns.heatmap(corr)

cluster_corr = sns.clustermap(corr)
plt.setp(cluster_corr.ax_heatmap.yaxis.get_majorticklabels(), rotation=0)
# plt.xticks(rotation=90)

fg_analysis = df[['FieldGoalDistance','FieldGoalResult', 'PlayType']]

fg_analysis = fg_analysis[fg_analysis['FieldGoalResult'] != -1]
fg_grouped = fg_analysis.groupby(by=["FieldGoalResult"])
print(fg_grouped.sum()/fg_grouped.count())

sns.violinplot(x="FieldGoalResult", y="FieldGoalDistance", data=fg_analysis, inner="quart")

fg_analysis = fg_analysis[fg_analysis['FieldGoalResult'] != "Blocked"]
fg_analysis = fg_analysis[fg_analysis['PlayType'] == "Field Goal"]
sns.violinplot(x = "PlayType", y="FieldGoalDistance",  hue="FieldGoalResult", data=fg_analysis, inner="quart", split = True)

pass_analysis = df[df.PlayType == 'Pass']
pass_analysis = pass_analysis[['PassOutcome','PassLength','PassLocation']]
# print(pass_analysis)

pass_analysis = pass_analysis[pass_analysis.PassLength != -1]
pa_grouped = pass_analysis.groupby(by=['PassLength'])
print(pa_grouped.count())
# pass_analysis['SuccessfulPass'] = pd.cut(df.PassOutcome,[0,1,2],2,labels=['Complete','Incomplete'])

pass_analysis.info()

# Draw a nested violinplot and split the violins for easier comparison
# sns.violinplot(x="PassLocation", y="SuccessfulPass", hue="PassLength", data=pass_analysis, split=True,
#                inner="quart")
# sns.despine(left=True)

pass_info = pd.crosstab([pass_analysis['PassLength'],pass_analysis['PassLocation'] ],
                       pass_analysis.PassOutcome.astype(bool))
print(pass_info)

pass_info.plot(kind='bar', stacked=True)

df.RunGap.value_counts()

pass_rate = pass_info.div(pass_info.sum(1).astype(float),
                             axis=0) # normalize the value

# print pass_rate
pass_rate.plot(kind='barh',
                  ...


I am working on the file "Lab1/Lab1-Travis-Copy1.ipynb".
The first task is:
We can eliminate combos who didn't have at least 10 receptions together, and then re-sample the data. This will remove noise from QB-receiver combos who have very high or low completion rates because they've played very little together.
Can you write Python code for it?

size = 10
corr = df.corr()
fig, ax = plt.subplots(figsize=(size, size))
ax.matshow(corr)
plt.xticks(range(len(corr.columns)), corr.columns)
for tick in ax.get_xticklabels():
    tick.set_rotation(90)
plt.yticks(range(len(corr.columns)), corr.columns)


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
Group by play type
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
Look at the number of yards gained by a FirstDown
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
Now all of the objects are encoded the way we'd like them to be
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
Now we can start to take a look at what's in each of our columns
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
THIS IS SOME MORE REFORMATTING SHIT I'M DOING FOR NOW. PROLLY GONNA KEEP IT
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
Temporary simple data replacement so that we can cast to integers (instead of objects)
Can you write Python code for it?

df.info()

df = df.replace(to_replace=np.nan,value=-1)


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
There's a lot of data that we don't care about. For example, 'PassAttempt' is a binary attribute, but there's also an attribute called 'PlayType' which is set to 'Pass' for a passing play.
We define a list of the columns which we're not interested in, and then we delete them
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
We can also extract the highest-completion percentage combos.
Here we take the top-10 most reliable QB-receiver pairs.
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
We can then grab a list of the remaining column names
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
We define four lists based on the types of features we're using.
Binary features are separated from the other categorical features so that they can be stored in less space
Can you write Python code for it?


I am working on the file "Lab1/Lab1-Travis.ipynb".
The first task is:
We then cast all of the columns to the appropriate underlying data types
Can you write Python code for it?

df[continuous_features] = df[continuous_features].astype(np.float64)
df[ordinal_features] = df[ordinal_features].astype(np.int64)
df[binary_features] = df[binary_features].astype(np.int8)


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
At this point, lots of things are encoded as objects, or with excesively large data types
Can you write Python code for it?


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
It looks like they gave up a lot more passing yards than rushing yards.
Let's see what QBs caused the Denver defense the most trouble.
Can you write Python code for it?

pass_plays = df[df.PlayType == "Pass"]
pass_plays_against_den = pass_plays[pass_plays.DefensiveTeam == "DEN"]
pass_plays_against_den_grouped = pass_plays_against_den.groupby("Passer")
qbs_yards_against_den = pass_plays_against_den_grouped["Yards.Gained"].sum()
qbs_yards_against_den.sort_values(inplace=True, ascending=False)
qbs_yards_against_den.plot(kind='barh')


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
Let's take a cursory glance at the data to see what we're working with.
Can you write Python code for it?


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
Now let's take a look at defenses. Which defenses allowed the fewest overall yards?
Can you write Python code for it?

def_play_groups = df.groupby("DefensiveTeam")

def_yards_allowed = def_play_groups["Yards.Gained"].sum()
def_yards_allowed.sort_values(inplace=True)
def_yards_allowed.plot(kind='barh')


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
Now we can start to take a look at what's in each of our columns
Can you write Python code for it?


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
THIS IS SOME MORE REFORMATTING SHIT I'M DOING FOR NOW. PROLLY GONNA KEEP IT
Can you write Python code for it?

df['PassOutcome'].replace(['Complete', 'Incomplete Pass'], [1, 0], inplace=True)


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
There's a lot of data that we don't care about. For example, 'PassAttempt' is a binary attribute, but there's also an attribute called 'PlayType' which is set to 'Pass' for a passing play.
We define a list of the columns which we're not interested in, and then we delete them
Can you write Python code for it?


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
We can also extract the highest-completion percentage combos.
Here we take the top-10 most reliable QB-receiver pairs.
Can you write Python code for it?

completion_rate.sort_values(inplace=True, ascending = False)
completion_rate = pd.Series(completion_rate)
completion_rate[0:10].plot(kind='barh')


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
We can also group by both passer and receiver, to check for highly effective QB-Receiver combos.
Can you write Python code for it?

pass_plays_grouped = pass_plays.groupby(by=['Passer', 'Receiver'])
completion_rate = pass_plays_grouped.PassOutcome.sum() / pass_plays_grouped.PassOutcome.count()
completion_rate_sampled = completion_rate.sample(10)
completion_rate_sampled.sort_values(inplace=True)
completion_rate_sampled.plot(kind='barh')


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
We can take a random sample of passers and show their completion rate:
Can you write Python code for it?

completion_rate = pass_plays_grouped.PassOutcome.sum() / pass_plays_grouped.PassOutcome.count()
completion_rate_sampled = completion_rate.sample(10)
completion_rate_sampled.sort_values(inplace=True)
completion_rate_sampled.plot(kind='barh')


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
We can then grab a list of the remaining column names
Can you write Python code for it?


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
We define four lists based on the types of features we're using.
Binary features are separated from the other categorical features so that they can be stored in less space
Can you write Python code for it?


I am working on the file "Lab1/Lab1_Playground.ipynb".
The first task is:
We then cast all of the columns to the appropriate underlying data types
Can you write Python code for it?


I am working on the file "Lab1/MAT491-Lab1.ipynb".
The first task is:
Q1. Graph the curve
$x= 26 \sin^3{t}$,
$y = 13\cos{t} - 5\cos{2t} - 2\cos{4t}$,
$0 \le t \le 2\pi.$
Q2. Graph the curve
$x=\mathrm{e}^t \cos{t} - \mathrm{e}^t \sin{t}$,
$y = \mathrm{e}^t \sin{t}$,
$0 \le t \le \pi.$
Hint: Use np.exp(t) for $\mathrm{e}^t$.
Polar Coordinates
Example: Sketch the curve $r = 1 + \sin{\theta}$.
Can you write Python code for it?

# define array from 0 to pi 100 elements
theta = np.linspace(0, 2*np.pi, 100)

# define $r = 1 + \sin{\theta}$
r = 1 + np.sin(theta)

# add subplot in polar coordinates
ax = plt.subplot(111, polar=True)

# plot the graph
ax.plot(theta, r)
plt.show()


I am working on the file "Lab1/lab1.ipynb".
The first task is:
Analysis
We can see from the chart that standard deviation grows to some moment and then falls down
Experiments with other source sequences have shown that the point with maximal standard deviation moves to right with sequence size growth
This can mean that before that moment we can gather data which is good for entropy estimate and after that we have not enough numbers for estimate
For example, when block size is equal to sequence length, entropy will be zero but source generates different pseudorandom numbers
Estimate
Entropy of the source should be calculated as
H = lim(n to infinity) Hn/n
where n is a block size and Hn is an entropy of the sequence with size of block equal to n
Here are following conserns:
- we cannot get n big enough without sequence size increase;
- it's impossible to get very big sequence;
- estimate based on single block size will have undefined error
We can use least squares method to estimate the mean entropies (will be denoted as Hn') using linear model
Hn' = k * n + c
Here k will be an estimate of H because following equality will take a place when n is big enough
Hn'/n = k + c/n -> k = H
Can you write Python code for it?

count = standard_deviation.argmax()
a, b = lstsq(vstack([BITS_RANGE[:count], ones(count)]).T, means[:count])[0]
approximation_error = array([means[:count], a*BITS_RANGE[:count]+b]).std(axis=0)
show_entropies([shannon_entropies[:count], renyi_entropies2[:count], renyi_entropies3[:count], means[:count]], labels=['Shannon', 'Renyi (2)', 'Renyi (3)', 'Mean'], errors={'x': BITS_RANGE[:count], 'y': a*BITS_RANGE[:count]+b, 'yerr': approximation_error, 'label': 'Linear approximation'}, count=count)

show_entropies([shannon_entropies_chaos[:count], means[:count]], labels=['Uniform entropy', 'Mean'], errors={'x': BITS_RANGE[:count], 'y': a*BITS_RANGE[:count]+b, 'yerr': approximation_error, 'label': 'Linear approximation'}, count=count)


I am working on the file "Lab1/lab1.ipynb".
The first task is:
Get same entropies for chaotic sequence
Sequences of uniformly distributed numbers with same binary sizes as existent sequences have the biggest possible entropies
Can you write Python code for it?

shannon_entropies_chaos = [n*log(2) for n in BITS_RANGE]
renyi_entropies2_chaos = [-(n*log(2)-2*n*log(2)) for n in BITS_RANGE]
renyi_entropies3_chaos = [-(n*log(2)-3*n*log(2))/2 for n in BITS_RANGE]

show_entropies([shannon_entropies, renyi_entropies2, renyi_entropies3, means, shannon_entropies_chaos], labels=['Shannon', 'Renyi (2)', 'Renyi (3)', 'Mean', 'Uniform entropy'])


I am working on the file "Lab1/lab1.ipynb".
The first task is:
Results
Here are estimate of the source entropy and value of chaotic source
Can you write Python code for it?

print('Source entropy estimate is', a)
print('Chaotic entropy is', log(2))

(array(shannon_entropies[1:]) - shannon_entropies[:-1])[:count][:6]


I am working on the file "Lab2.ipynb".
The first task is:
1.1.3 Stochastic gradient descent (10 points)
Write a function sgd_iter(x_train, t_train, w, b) that performs one iteration of stochastic gradient descent (SGD), and returns the new weights. It should go through the trainingset once in randomized order, call logreg_gradient(x, t, w, b) for each datapoint to get the gradients, and update the parameters using a small learning rate (e.g. 1E-4). Note that in this case we're maximizing the likelihood function, so we should actually performing gradient ascent... For more information about SGD, see Bishop 5.2.4 or an online source (i.e. https://en.wikipedia.org/wiki/Stochastic_gradient_descent)
Can you write Python code for it?

def compute_log_likelihood(x, t, w, b):
    log_q = x.dot(w) + b
    a = np.max(log_q, axis=1)
    log_Z = a + np.log(np.sum(np.exp((log_q.T - a).T)))
    #print log_Z
    log_likelihood = np.sum(np.array([log_q[index, value] for (index,), value in np.ndenumerate(t)]) - log_Z)
    return log_likelihood

def sgd_iter(x_train, t_train, w, b, lr=0.001):
    """
    Don't need to return w and b. They are modified by reference
    """
    indexes = np.arange(len(x_train))
    np.random.shuffle(indexes)

    for index in indexes:
        grad_w, delta = logreg_gradient(x_train[index], t_train[index], w, b)
        w += lr * grad_w
        b += lr * delta

# Test
w, b = init_weights(num_of_features, num_of_class)
print 'Likelihood:', compute_log_likelihood(x_train, t_train, w, b)
sgd_iter(x_train, t_train, w, b)
print 'Likelihood:', compute_log_likelihood(x_train, t_train, w, b)

print w.shape, b.shape


I am working on the file "Lab2.ipynb".
The first task is:
1.2.2 Visualize weights (10 points)
Visualize the resulting parameters $\bW$ after a few iterations through the training set, by treating each column of $\bW$ as an image. If you want, you can use or edit the plot_digits(...) above.
Can you write Python code for it?


I am working on the file "Lab2.ipynb".
The first task is:
In multiclass logistic regression, the conditional probability of class label $j$ given the image $\bx$ for some datapoint is given by:
$ \log p(t = j \;|\; \bx, \bb, \bW) = \log q_j - \log Z$
where $\log q_j = \bw_j^T \bx + b_j$ (the log of the unnormalized probability of the class $j$), and $Z = \sum_k q_k$ is the normalizing factor. $\bw_j$ is the $j$-th column of $\bW$ (a matrix of size $784 \times 10$) corresponding to the class label, $b_j$ is the $j$-th element of $\bb$.
Given an input image, the multiclass logistic regression model first computes the intermediate vector $\log \bq$ (of size $10 \times 1$), using $\log q_j = \bw_j^T \bx + b_j$, containing the unnormalized log-probabilities per class.
The unnormalized probabilities are then normalized by $Z$ such that $\sum_j p_j = \sum_j \exp(\log p_j) = 1$. This is done by $\log p_j = \log q_j - \log Z$ where $Z = \sum_j \exp(\log q_j)$. This is known as the softmax transformation, and is also used as a last layer of many classifcation neural network models, to ensure that the output of the network is a normalized distribution, regardless of the values of second-to-last layer ($\log \bq$)
Warning: when computing $\log Z$, you are likely to encounter numerical problems. Save yourself countless hours of debugging and learn the log-sum-exp trick.
The network's output $\log \bp$ of size $10 \times 1$ then contains the conditional log-probabilities $\log p(t = j \;|\; \bx, \bb, \bW)$ for each digit class $j$. In summary, the computations are done in this order:
$\bx \rightarrow \log \bq \rightarrow Z \rightarrow \log \bp$
Given some dataset with $N$ independent, identically distributed datapoints, the log-likelihood is given by:
$ \mathcal{L}(\bb, \bW) = \sum_{n=1}^N \mathcal{L}^{(n)}$
where we use $\mathcal{L}^{(n)}$ to denote the partial log-likelihood evaluated over a single datapoint. It is important to see that the log-probability of the class label $t^{(n)}$ given the image, is given by the $t^{(n)}$-th element of the network's output $\log \bp$,...
Can you write Python code for it?

def logreg_gradient(x, t, W, b):
    log_q = W.T.dot(x) + b

    # Log-sum-exp trick
    a = np.max(log_q)
    logZ = a + np.log(np.sum(np.exp(log_q - a)))
    Z = np.exp(logZ)
    grad_b = -np.exp(log_q) / Z
    grad_b[t] += 1
    grad_w = x[np.newaxis].T.dot(grad_b[np.newaxis])
    return grad_w, grad_b

def init_weights(num_of_features, num_of_class):
    w = (np.random.random_sample((num_of_features, num_of_class)) - 0.5) * 0.001
    b = (np.random.random_sample(num_of_class) - 0.5) * 0.001
    return w, b

np.random.seed(0)
num_of_features = x_train.shape[1]
num_of_class = 10

# Test
w, b = init_weights(num_of_features, num_of_class)

random_item_id = np.random.randint(x_train.shape[0])
grad_w, grad_b = logreg_gradient(x_train[random_item_id], t_train[random_item_id], w, b)
print grad_w.shape, grad_b.shape


I am working on the file "Lab2.ipynb".
The first task is:
Part 1. Multiclass logistic regression
Scenario: you have a friend with one big problem: she's completely blind. You decided to help her: she has a special smartphone for blind people, and you are going to develop a mobile phone app that can do machine vision using the mobile camera: converting a picture (from the camera) to the meaning of the image. You decide to start with an app that can read handwritten digits, i.e. convert an image of handwritten digits to text (e.g. it would enable her to read precious handwritten phone numbers).
A key building block for such an app would be a function predict_digit(x) that returns the digit class of an image patch $\bx$. Since hand-coding this function is highly non-trivial, you decide to solve this problem using machine learning, such that the internal parameters of this function are automatically learned using machine learning techniques.
The dataset you're going to use for this is the MNIST handwritten digits dataset (http://yann.lecun.com/exdb/mnist/). You can load the data from mnist.pkl.gz we provided, using:
Can you write Python code for it?

def load_mnist():
    f = gzip.open('mnist.pkl.gz', 'rb')
    data = cPickle.load(f)
    f.close()
    return data

(x_train, t_train), (x_valid, t_valid), (x_test, t_test) = load_mnist()


I am working on the file "Lab2.ipynb".
The first task is:
Solutions
Can you write Python code for it?

import math
import pickle as p
import tensorflow as tf
import numpy as np
import utils

# set variables
tweet_size = 20
hidden_size = 100
vocab_size = 7597
batch_size = 64

# this just makes sure that all our following operations will be placed in the right graph.
tf.reset_default_graph()

# create a session variable that we can run later.
session = tf.Session()

# make placeholders for data we'll feed in
tweets = tf.placeholder(tf.float32, [None, tweet_size, vocab_size])
labels = tf.placeholder(tf.float32, [None])

# make the lstm cells, and wrap them in MultiRNNCell for multiple layers
lstm_cell = tf.contrib.rnn.LSTMCell(hidden_size)
multi_lstm_cells = tf.contrib.rnn.MultiRNNCell(cells=[lstm_cell] * 2, state_is_tuple=True)

# define the op that runs the LSTM, across time, on the data
_, final_state = tf.nn.dynamic_rnn(multi_lstm_cells, tweets, dtype=tf.float32)

# a useful function that takes an input and what size we want the output
# to be, and multiples the input by a weight matrix plus bias (also creating
# these variables)
def linear(input_, output_size, name, init_bias=0.0):
    shape = input_.get_shape().as_list()
    with tf.variable_scope(name):
        W = tf.get_variable("weight_matrix", [shape[-1], output_size], tf.float32, tf.random_normal_initializer(stddev=1.0 / math.sqrt(shape[-1])))
    if init_bias is None:
        return tf.matmul(input_, W)
    with tf.variable_scope(name):
        b = tf.get_variable("bias", [output_size], initializer=tf.constant_initializer(init_bias))
    return tf.matmul(input_, W) + b

# define that our final sentiment logit is a linear function of the final state
# of the LSTM
sentiment = linear(final_state[-1][-1], 1, name="output")

sentiment = tf.squeeze(sentiment, [1])

# define cross entropy loss function
loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=sentiment, labels=labels)
loss = tf.reduce_mean(loss)

# round our actual probabilities to compute error
prob = tf.nn.sigmoid(sentiment)
prediction = tf.to_float(tf.greater_equal(prob, 0.5))
pred_err =...


I am working on the file "Lab2.ipynb".
The first task is:
Step 2: Create Placeholders
We need to create placeholders for variable data that we will feed in ourselves (aka our tweets). Placeholders allow us to incorporate this data into the graph even though we don't know what it is yet.
Can you write Python code for it?

# the placeholder for tweets has first dimension batch_size for each tweet in a batch,
# second dimension tweet_size for each word in the tweet, and third dimension vocab_size
# since each word itself is represented by a one-hot vector of size vocab_size.
# Note that we use 'None' instead of batch_size for the first dimsension.  This allows us
# to deal with variable batch sizes
tweets = tf.placeholder(tf.float32, [None, tweet_size, vocab_size])

'''TODO: create a placeholder for the labels (our predictions).
   This should be a 1D vector with size = None,
   since we are predicting one value for each tweet in the batch,
   but we want to be able to deal with variable batch sizes.''';
labels = #todo


I am working on the file "Lab2.ipynb".
The first task is:
Step 3: Build the LSTM Layers
We want to feed the input sequence, word by word, into an LSTM layer, or multiple LSTM layers (we could also call this an LSTM encoder).  At each "timestep", we feed in the next word, and the LSTM updates its cell state. The final LSTM cell state can then be fed through a final classification layer(s) to get our sentiment prediction.
Now let's make our LSTM layer. The steps for this are:
1. Create a LSTM Cell using tf.contrib.rnn.LSTMCell

Wrap a couple of these cells in tf.nn.rnn_cell.MultiRNNCell to create a multiple LSTM layers.

Define the operation to run these layers with dynamic_rnn.
Can you write Python code for it?

'''TODO: create an LSTM Cell using BasicLSTMCell.  Note that this creates a *layer* of LSTM
   cells, not just a single one.''';
lstm_cell = #todo

'''TODO: create three LSTM layers by wrapping three instances of
   lstm_cell from above in tf.contrib.rnn_cell.MultiRNNCell. Note that
   you can create multiple cells by doing [lstm_cell] * 2. Also note
   that you should use state_is_tuple=True as an argument.  This will allow
   us to access the part of the cell state that we need later on.''';
multi_lstm_cells = #todo

'''TODO: define the operation to create the RNN graph across time.
   tf.nn.dynamic_rnn dynamically constructs the graph when it is executed,
   and returns the final cell state.''';
_, final_state = #todo


I am working on the file "Lab2.ipynb".
The first task is:
Step 4: Classification Layer
Now we have the final state of the LSTM layers after feeding in the tweet word by word. We can take this final state and feed it into a simple classfication layer that takes the cell state, multiplies it by some weight matrix (with bias) and outputs a single value corresponding to whether it thinks the tweet is overall positive or not.
Can you write Python code for it?

## We define this function that creates a weight matrix + bias parameter
## and uses them to do a matrix multiplication.
def linear(input_, output_size, name, init_bias=0.0):
    shape = input_.get_shape().as_list()
    with tf.variable_scope(name):
        W = tf.get_variable("weights", [shape[-1], output_size], tf.float32, tf.random_normal_initializer(stddev=1.0 / math.sqrt(shape[-1])))
    if init_bias is None:
        return tf.matmul(input_, W)
    with tf.variable_scope(name):
        b = tf.get_variable("bias", [output_size], initializer=tf.constant_initializer(init_bias))
    return tf.matmul(input_, W) + b

'''TODO: pass the final state into this linear function to multiply it
   by the weights and add bias to get our output.

   {Quick note that we need to feed in final_state[-1][-1] into linear since
   final_state is actually a tuple consisting of the cell state
   (used internally for the cell to keep track of things)
   as well as the hidden state (the output of the cell), and one of these
   tuples for each layer. We want the hidden state for the last layer, so we use
   final_state[-1][-1]}''';

sentiment = #todo


I am working on the file "Lab2.ipynb".
The first task is:
Step 5: Define Loss
Now we define a loss function that we'll use to determine the difference between what we predicted and what's actually correct.  We'll want to use cross entropy, since we can take into account what probability the model gave to the a tweet being positive.
The output we just got from the linear classification layer is called a 'logit' -- the raw value before transforming it into a probability between 0 and 1.  We can feed these logits to  tf.nn.sigmoid_cross_entropy_with_logits, which will take the sigmoid of these logits (making them between 0 and 1) and then calculate the cross-entropy with the ground truth labels.
Can you write Python code for it?

sentiment = tf.squeeze(sentiment, [1])

'''TODO: define our loss function.
   We will use tf.nn.sigmoid_cross_entropy_with_logits, which will compare our
   sigmoid-ed prediction (sentiment from above) to the ground truth (labels).''';

loss = #todo

# our loss with sigmoid_cross_entropy_with_logits gives us a loss for each
# example in the batch.  We take the mean of all these losses.
loss = tf.reduce_mean(loss)

# to get actual results like 'positive' or 'negative' ,
# we round the prediction probability to 0 or 1.
prediction = tf.to_float(tf.greater_equal(sentiment, 0.5))

# calculate the error based on which predictions were actually correct.
pred_err = tf.to_float(tf.not_equal(prediction, labels))
pred_err = tf.reduce_sum(pred_err)


I am working on the file "Lab2.ipynb".
The first task is:
Step 6: Train
Now we define the operation that actually changes the weights by minimizing the loss.
tf.train.AdamOptimizer is just a gradient descent algorithm that uses a variable learning rate to converge faster and more effectively.
We want to specify this optimizer and then call the minimize function, the optimizer knows it wants to minimize the loss we defined above.
Can you write Python code for it?

'''Define the operation that specifies the AdamOptimizer and tells
   it to minimize the loss.''';
optimizer = #todo


I am working on the file "Lab2.ipynb".
The first task is:
We got 91.9% of accuracy on validation set without model selection:
Can you write Python code for it?

# Test
prediction = predict_mlperceptron(W, b, V, a, x_valid)
correctness = t_valid == prediction
accuracy = len(correctness[correctness == True]) / float(len(t_valid))
print accuracy

figure = plt.figure()
figure.set_size_inches((16, 10))
plot_digits(V.T, numcols=5)


I am working on the file "Lab3.ipynb".
The first task is:
1. Sampling from the Gaussian process prior (30 points)
We will implement Gaussian process regression using the kernel function in Bishop Eqn. 6.63.
1.1 k_n_m( xn, xm, thetas ) (10 points)
To start, implement function "k_n_m( xn, xm, thetas )" that takes scalars $\xn$ and $\xm$, and a vector of $4$ thetas, and computes the kernel function Bishop Eqn. 6.63 (10 points).
Can you write Python code for it?

def k_n_m(xn, xm, thetas):
    return thetas[0] * np.exp(-thetas[1] * (xn - xm) ** 2/ 2) + thetas[2] + thetas[3] * xn * xm


I am working on the file "Lab3.ipynb".
The first task is:
2. Predictive distribution (35 points)
So far we have sampled mean functions from the prior.  We can draw actual data $\t$ two ways.  The first way is generatively, by first sampling $\y | \K$, then sampling $\t | \y, \beta$ (Eqns 6.60 followed by 6.59).  The second way is to integrate over $\y$ (the mean draw) and directly sample $\t | \K, \beta$ using Eqn 6.61.    This is the generative process for $\t$.  Note that we have not specified a distribution over inputs $\x$;  this is because Gaussian processes are conditional models.  Because of this we are free to generate locations $\x$ when playing around with the GP; obviously a dataset will give us input-output pairs.
Once we have data, we are interested in the predictive distribution (note: the prior is the predictive distribution when there is no data).  Consider the joint distribution for $N+1$ targets, given by Eqn 6.64.  Its covariance matrix is composed of block components $\CN$, $\k$, and $c$.  The covariance matrix $CN$ for $\tN$ is $\CN = \KN + \eyeN / \beta$.  We have just made explicit the size $N$ of the matrix; $N$ is the number of training points.  The kernel vector $\k$ is a $N$ by $1$ vector of kernel function evaluations between the training input data and the test input vector.  The scalar $c$ is a kernel evaluation at the test input.
2.1 gp_predictive_distribution(...) (10 points)
Write a function "gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C = None)" that computes  Eqns 6.66 and 6.67, except allow for an arbitrary number of test points (not just one) and now the kernel matrix is for training data.  By having C as an optional parameter, we can avoid computing it more than once (for this problem it is unimportant, but for real problems this is an issue).  The function should compute $\C$, $\k$, and $c$ and the mean and noise functions.  Do not forget: the computeK function computes $\K$, not $\C$! (10 points)
Can you write Python code for it?

def computeC(x_train, theta, beta):
    K_t = computeK(x_train, x_train, theta)
    return K_t + np.identity(len(x_train)) / beta

def gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C = None):
    if not C:
        C = computeC(x_train, theta, beta)
    k = computeK(x_train, x_test, theta)
    Cinv = np.linalg.inv(C)
    means = k.T.dot(Cinv).dot(t_train)
    c = computeK(x_test, x_test, theta) + 1.0 / beta
    covs = c - k.T.dot(Cinv).dot(k)
    return means, covs


I am working on the file "Lab3.ipynb".
The first task is:
2.2 gp_log_likelihood(...) (10 points)
Later, to learn the hyperparameters, we will need to compute the log-likelihood of the of the training data.  Implicitly, this is conditioned on the value setting for $\thetav$.  Write a function "gp_log_likelihood( x_train, t_train, theta, C = None, invC = None )", where C and invC can be stored and reused.  (10 points)
Can you write Python code for it?

def gp_log_likelihood(x_train, t_train, theta, beta, C = None, invC = None):
    if not C:
        C = computeC(x_train, theta, beta)
    if not invC:
        invC = np.linalg.inv(C)

    return -(np.log(np.linalg.det(C)) + t_train.T.dot(invC).dot(t_train) + len(t_train) * np.log(2 * np.pi)) / 2


I am working on the file "Lab3.ipynb".
The first task is:
2.3 Plotting (10 points)
Repeat the 6 plots above, but this time conditioned on the training points.  Use the sinuosoidal data generator to create 2 training points where x is sampled uniformly between $-1$ and $1$.  For these plots, feel free to use the provided function "gp_plot".  Make sure you put the parameters in the title and this time also the log-likelihood. (10 points)  Try to understand the two types of uncertainty!  If you do not use "gp_plot", please add a fill between for the model and target noise.
Can you write Python code for it?

def gp_plot( x_test, y_test, mu_test, var_test, x_train, t_train, theta, beta ):
    # x_test:   the test data
    # y_test:   the true function at x_test
    # mu_test:  predictive mean at x_test
    # var_test: predictive covariance at x_test
    # t_train:  the training values
    # theta:    the kernel parameters
    # beta:     the precision (known)

    # the reason for the manipulation is to allow plots separating model and data stddevs.
    std_total = np.sqrt(np.diag(var_test))         # includes all uncertainty, model and target noise
    std_model = np.sqrt( std_total**2 - 1.0/beta ) # remove data noise to get model uncertainty in stddev
    std_combo = std_model + np.sqrt( 1.0/beta )    # add stddev (note: not the same as full)

    pp.plot( x_test, y_test, 'b', lw=3)
    pp.plot( x_test, mu_test, 'k--', lw=2 )
    pp.fill_between( x_test, mu_test+2*std_combo,mu_test-2*std_combo, color='k', alpha=0.25 )
    pp.fill_between( x_test, mu_test+2*std_model,mu_test-2*std_model, color='r', alpha=0.25 )
    pp.plot( x_train, t_train, 'ro', ms=10 )

beta  = 1.0 / pow(sigma,2)

def plot1(x_train):
    fig = plt.figure()
    y_train = true_mean_function(x_train)
    t_train = add_noise(y_train, sigma)
    x_test = np.linspace(-1, 1, N_test)
    y_test = true_mean_function(x_test)

    fig.set_size_inches(14, 8)
    for row, col in product(range(2), range(3)):
        idx = row * 3 + col
        plt.subplot(2, 3, idx + 1)
        means, sigmas = gp_predictive_distribution(x_train, t_train, x_test, thetas[idx], beta)
        gp_plot(x_test, y_test, means, sigmas, x_train, t_train, thetas[idx], beta)
        plt.title(str(thetas[idx]))

    plt.show()

x_train = np.random.uniform(-1, 1, 2)
plot1(x_train)


I am working on the file "Lab3.ipynb".
The first task is:
2.4 More ploting (5 points)
Repeat the 6 plots above, but this time conditioned a new set of 10 training points. (5 points)
Can you write Python code for it?

x_train = np.random.uniform(-1, 1, 10)
plot1(x_train)


I am working on the file "Lab3.ipynb".
The first task is:
3. Learning the hyperparameters (45 points)
Learning the values of the parameter $\thetav$ can be very tricky for Gaussian processes in general, but when the data is univariate like ours, we can visualize the fit and see how plausible it looks.
3.1 Derivatives (5 points)
Maximum likelihood or MAP learning is the most common way of setting the parameters, though a fully Bayesian approach is possible too.  We will look at ML today.  For this, we start with the dervivative of the log-likelihood with respect to the parameters $\thetav$; this is Eqn 6.70.  This, in turn, requires the derivative of the kernel matrix $\CN$ wrt $\thetav$.  This is the matrix of element-wise derivatives of the kernel function.  Write the derivatives for $\theta_0$ to $\theta_3$ for our kernel function (5 points).
Answer
$\dfrac{\partial k(x_n,x_m)}{\partial\theta_0} = \exp(-\dfrac{\theta_1}{2}||x_n - x_m||^2)$
$\dfrac{\partial k(x_n,x_m)}{\partial\theta_1} = -\dfrac{1}{2}||x_n - x_m||^2 \theta_0 \exp(-\dfrac{\theta_1}{2}||x_n - x_m||^2)$
$\dfrac{\partial k(x_n,x_m)}{\partial\theta_2} = 1$
$\dfrac{\partial k(x_n,x_m)}{\partial\theta_3} = x_n^Tx_m$
3.2 Questions (5 points)
Which parameters in $\thetav$ are unconstrained, that is, where any positive/ negative values are valid? (5 points)
Answer
We start by decomposing our kernal function into a sum of 3 kernal functions:
$k(x_n,x_m)= \theta_0 k_1(x_n,x_m) + \theta_2  k_2(x_n,x_m) + \theta_3 k_3(x_n,x_m)$
As we can see $\theta_0$, $\theta_2$, and $\theta_3$ are constants and therefore should be >0 for the kernal to be valid. On the other hand, $\theta_1$ plays a role of a bandwidth in $k_1$ and is not constrained.
3.3 More derivatives (5 points)
For parameters that are constrained to be positive, the usual approach is to use the exponential of the free-parameter in the kernel function, but perform gradient ascent on the unconstrained values.  Consider the case  $\theta_i = \exp( \phi_i)$, where $\phi_i$ is unconstrained.  Write the derivative for $\phi_i$ in terms of the derivatives you...
Can you write Python code for it?

from operator import itemgetter

theta = [0, 1, 2, 4, 8]

x_train = np.random.uniform(-1, 1, 10)
y_train = true_mean_function(x_train)
t_train = add_noise(y_train, sigma)

results = []
# 27 combinations
for t in product(theta, repeat=4):
    results.append(tuple([gp_log_likelihood(x_train, t_train, t, beta)]) + t)

results = sorted(results, key=itemgetter(0))
for loglkl, t0, t1, t2, t3 in results:
    print('log-likelihood: %f thetas: %s' % (loglkl, (t0, t1, t2, t3)))
print 'Best Log Likelihood: %f for thetas: (%d,%d,%d,%d)' % results[-1]
print 'Worse Log Likelihood: %f for thetas: (%d,%d,%d,%d)' % results[0]

fig = plt.figure()
fig.set_size_inches(14, 8)

x_test = np.linspace(-1, 1, N_test)
y_test = true_mean_function(x_test)

plt.subplot(1, 2, 1)
means, sigmas = gp_predictive_distribution(x_train, t_train, x_test, np.array(results[-1][1:]), beta)
gp_plot(x_test, y_test, means, sigmas, x_train, t_train, np.array(results[-1]), beta)
plt.title('Best likelihood %.2f, theta = (%d, %d, %d, %d)' % results[-1])

plt.subplot(1, 2, 2)
means, sigmas = gp_predictive_distribution(x_train, t_train, x_test, np.array(results[0][1:]), beta)
gp_plot(x_test, y_test, means, sigmas, x_train, t_train, np.array(results[0]), beta)
plt.title('Worse likelihood %.2f, theta = (%d, %d, %d, %d)' % results[0])
plt.show()


I am working on the file "Lab3.ipynb".
The first task is:
Obtained likelihood turns to be better than the one in grid search
Can you write Python code for it?

fig = plt.figure()
fig.set_size_inches(14, 8)

means, sigmas = gp_predictive_distribution(x_train, t_train, x_test, result.x, beta)
gp_plot(x_test, y_test, means, sigmas, x_train, t_train, result.x, beta)
plt.show()

plt.plot(np.arange(len(log_likelihoods)), log_likelihoods)
plt.ylabel('Log likelihood')
plt.xlabel('Iteration')
plt.show()


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Although the grid search returned the optimal step count and rate, it is meaningful to visualize the grid that was generated, to get an idea for how much better these particular hyperparameters are than the other possible combinations in the grid.
Can you write Python code for it?


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Architecture Comparison
The following getDifference function is used to create a tuple of the range of possible differences of mean for two sets of scores (cost or auc_roc), with 95% confidence. We use the second of the two confidence interval tests proposed in the ICA3 reversed assignment, because the datasets cannot be assumed to be independent, so the binomial approximation to the normal distribution does not hold.
Can you write Python code for it?

def getDifference(cost1,cost2,z_val=2.26,size=10):
    cost1 = np.asarray(cost1)
    cost2 = np.asarray(cost2)
    diff12 = cost1 - cost2
    sigma12 = np.sqrt(np.sum(diff12*diff12) * 1/(size-1))
    d12 = (np.mean(diff12) + 1/(np.sqrt(size)) * z_val * sigma12, np.mean(diff12) - 1/(np.sqrt(size)) * z_val * sigma12)
    return d12


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Calculating Costs for a Model
The following code performs cross validation on a model with a given step count and learning rate. This will be used as part of the grid search, as well as for evaluating the final classifier after the gridsearch is complete.
Can you write Python code for it?

def get_scores_for_model(model_fn, X, y, steps=1000, learning_rate=0.05, num_splits = 10):

    auc = []
    costs = []

    for train_index, test_index in KFold(n_splits=num_splits).split(X, y):
        classifier = learn.TensorFlowEstimator(model_fn=model_fn,
                                               n_classes=7, batch_size=1000,
                                               steps=steps, learning_rate=learning_rate)
        classifier.fit(X[train_index], y[train_index])
        yhat = classifier.predict(X[test_index])

        costs.append(cost(y[test_index], yhat))
        auc.append(auc_of_roc(y[test_index], yhat, levels=range(0,7)))

    return costs, auc


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Finally, we produce AUC_ROC scores for the random forest classifier.
Can you write Python code for it?

#Pipeline for cost evaluation
clf = Pipeline([('sca',StandardScaler()),
                ('clf',RandomForestClassifier(class_weight=ClassWeights, n_estimators=250))])

per_fold_eval_criteria = cross_val_score(estimator=clf,
                                    X=X,
                                    y=y,
                                    cv=cv,
                                    scoring=auc_roc_scorer,
                                    n_jobs=-1
                                   )

RF_auc = per_fold_eval_criteria


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
First Deep Learning Architecture
The first deep learning architecture will be adapted from a model designed by PayPal that is used for anomaly detection. Because the vast majority of football plays are either runs or passes, and there are only a few anomalous plays, like onside kicks, etc., it stands to reason that an anomaly detection architecture would perform well for this classification task.
The architecture is quite simple: it consists of a set of 6 fully connected layers of 700 neurons, followed by a hyperbolic tangent activation function and then a single fully connected layer for output. We will adapt this model slightly to allow for the embedding of the team attributes. We will split the data into embedding and non-embedding data, run each subset of data through 6 fully connected layers of 700 neurons, and then combine their output as the input into a single final layer, used for classification. A simple drawing of the architecture is shown below.

The original talk about this architecture can be found here.

Defining the Model
Can you write Python code for it?

def deep_model_1(X, y):

    #Embeddings layer
    teamembeddings = layers.stack(X[:,11:75], layers.fully_connected, [700 for _ in range(6)])
    teamembeddings = tf.nn.tanh(teamembeddings)

    #Non-embeddings features
    otherfeatures = X[:,0:10]
    otherfeatures = layers.stack(otherfeatures, layers.fully_connected, [700 for _ in range(6)])

    tensors = tf.concat(1, [teamembeddings, otherfeatures])
    tensors = tf.nn.tanh(tensors)

    pred,loss = learn.models.logistic_regression(tensors, y)

    return pred, loss


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Grid Searching on the Model
A grid search is performed on the model to find the approximately optimal step count and learning rate for the TensorFlowEstimator.
Can you write Python code for it?

costs, optimal_steps, optimal_rate = grid_search(deep_model_2, [250,500,1000,1500,2000], [.05, .01, .005, .001, .0005])
print((optimal_steps, optimal_rate))


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Modeling
Before we build any models, we define a cost function in Python below, which is used to test all of our forthcoming models. It computes the item-wise product of a confusion matrix and our cost matrix, and returns the sum of all of the elements in the resulting matrix. We also define a function to calculate area under roc curve for a multiclass classification problem.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
from sklearn.metrics import roc_curve, auc,make_scorer
from scipy import interp

import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

cost_mat =     [[0  ,.1  , 1   , .15 , 0.15, .1 , 1   ],
                [.1 , 0  , 1   , 0.15, 0.15, 0.1, 1   ],
                [1  , 1  , 0   , 1   , 1   , 1  , 0.75],
                [.25,0.25, 1   , 0   , 1   ,0.15,  1  ],
                [0.4, 0.4, 1   , 1   , 0   , 1  ,  1  ],
                [0.4, 0.4, 1   , 0.1 , 1   , 0  ,  1  ],
                [1  , 1  , 0.25, 1   , 1   , 1  ,  0  ]]

def cost(Y, yhat):
    return np.sum(np.multiply(confusion_matrix(Y,yhat), cost_mat))

def auc_of_roc(Y,yhat, levels=['Pass', 'Run', 'Kickoff', 'Punt', 'Extra Point', 'Field Goal', 'Onside Kick']):

    mean_tpr = 0.0
    mean_fpr = np.linspace(0, 1, 100)
    for c in levels:
        tempY = [x==c for x in Y]
        tempYhat = [x==c for x in yhat]

        fpr, tpr, thresholds = roc_curve(tempY, tempYhat)
        mean_tpr += interp(mean_fpr, fpr, tpr)
        mean_tpr[0] = 0.0

        roc_auc = auc(fpr, tpr)

    mean_tpr /= len(levels)
    mean_tpr[-1] = 1.0
    mean_auc = auc(mean_fpr, mean_tpr)

    return mean_auc

#For use in the final deployment section
scorer = make_scorer(cost)
auc_roc_scorer = make_scorer(auc_of_roc)


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Neural Network Embeddings
We will use neural network embeddings from TensorFlow for the posteam and DefensiveTeam. However, we will be building these embeddings manually using one-hot encoding and additional fully-connected layers in each of the deep architectures. The following Python function was used for one-hot encoding, and was adapted from the website referenced in the code.
Can you write Python code for it?

from sklearn.feature_extraction import DictVectorizer

#Simple function for 1 hot encoding
def encode_onehot(df, cols):
    """
    One-hot encoding is applied to columns specified in a pandas DataFrame.

    Modified from: https://gist.github.com/kljensen/5452382

    Details:

    http://en.wikipedia.org/wiki/One-hot
    http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

    @param df pandas DataFrame
    @param cols a list of columns to encode
    @return a DataFrame with one-hot encoding
    """
    vec = DictVectorizer()

    vec_data = pd.DataFrame(vec.fit_transform(df[cols].to_dict(outtype='records')).toarray())
    vec_data.columns = vec.get_feature_names()
    vec_data.index = df.index

    df = df.drop(cols, axis=1)
    df = df.join(vec_data)
    return df

df = encode_onehot(df, cols=['posteam', 'DefensiveTeam'])


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Next, we produce cost scores for the random forest classifier.
Can you write Python code for it?

#Pipeline for cost evaluation
clf = Pipeline([('sca',StandardScaler()),
                ('clf',RandomForestClassifier(class_weight=ClassWeights, n_estimators=250))])

per_fold_eval_criteria = cross_val_score(estimator=clf,
                                    X=X,
                                    y=y,
                                    cv=cv,
                                    scoring=scorer,
                                    n_jobs=-1
                                   )

RFCosts = per_fold_eval_criteria


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Second Deep Learning Architecture
The second deep learning architecture will be adapted from a model designed by O'Shea Research that is used for radio modulation recognition. Because the majority of the football data is ordinal numerical data, an architecture that uses similar data for a classification task poses as a candidate model for the football play classification task.
The architecture consists of a set of two convolutional reLu neurons, followed by a dense relu and dense softmax activation function. The result is a single fully connected layer for output. We will adapt this model slightly to allow for the embedding of the team attributes. We will split the data into embedding and non-embedding data, run each subset of data through the convolutional neurons, combine their output and run it through the dense neurons, and take a logistic regression for the input into a single final layer, used for classification. A simple drawing of the architecture is shown below.

The original discussion about this architecture can be found here.

Defining the Model
Can you write Python code for it?

def deep_model_2(X, y):

    #Embeddings layer
    teamembeddings = layers.stack(X[:,11:75], layers.fully_connected, [200,1,3], activation_fn=tf.nn.relu)
    teamembeddings = layers.stack(teamembeddings, layers.fully_connected, [50,2,3], activation_fn=tf.nn.relu)

    #Non-embeddings features
    otherfeatures = X[:,0:10]
    otherfeatures = layers.stack(otherfeatures, layers.fully_connected, [50,1,3], activation_fn=tf.nn.relu)
    otherfeatures = layers.stack(otherfeatures, layers.fully_connected, [12,2,3], activation_fn=tf.nn.relu)

    #combine the team and play data
    tensors = tf.concat(1, [teamembeddings, otherfeatures])

    tensors = layers.stack(tensors, layers.fully_connected, [100], activation_fn=tf.nn.relu)
    tensors = layers.stack(tensors, layers.fully_connected, [7], activation_fn=tf.nn.softmax)

    """  # This section is doing all layers before combining team and play data
    #Embeddings layer
    teamembeddings = layers.stack(X[:,11:75], layers.fully_connected, [200,1,3], activation_fn=tf.nn.relu)
    teamembeddings = layers.stack(teamembeddings, layers.fully_connected, [50,2,3], activation_fn=tf.nn.relu)

    teamembeddings = layers.stack(teamembeddings, layers.fully_connected, [100], activation_fn=tf.nn.relu)
    teamembeddings = layers.stack(teamembeddings, layers.fully_connected, [7], activation_fn=tf.nn.softmax)

    #Non-embeddings features
    otherfeatures = X[:,0:10]

    otherfeatures = layers.stack(otherfeatures, layers.fully_connected, [50,1,3], activation_fn=tf.nn.relu)
    otherfeatures = layers.stack(otherfeatures, layers.fully_connected, [12,2,3], activation_fn=tf.nn.relu)

    otherfeatures = layers.stack(otherfeatures, layers.fully_connected, [100], activation_fn=tf.nn.relu)
    otherfeatures = layers.stack(otherfeatures, layers.fully_connected, [7], activation_fn=tf.nn.softmax)

    #combine the team and play data
    tensors = tf.concat(1, [teamembeddings, otherfeatures])
    """

    pred, loss =...


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
The above 3 confidence intervals show that the first two architectures performed similarly, as the model 1 to model 2 difference confidence interval contains zero. However, the third architecture, which we created specifically because of how poorly the first two architectures performed, did significantly outperform both of the first two models, as zero does not appear in either of the confidence intervals including model 3. Therefore, it can be concluded that, with respect to cost, our "home-grown" model 3 does significantly outperform the other architectures.
Area Under Multiclass ROC Curve Difference
The same process is now used for confidence intervals for the auc_roc metric.
Can you write Python code for it?

d_one_two = np.array(getDifference(auc_roc_model_1, auc_roc_model_2))
d_one_three = np.array(getDifference(auc_roc_model_1, auc_roc_model_3))
d_two_three = np.array(getDifference(auc_roc_model_2, auc_roc_model_3))

print("Average Model 1 vs Model 2 Difference:", d_one_two)
print("Average Model 1 vs Model 3 Difference:", d_one_three)
print("Average Model 2 vs Model 3 Difference:", d_two_three)


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
The above grid search shows that, while there is no obvious pattern with respect to the performance, it seems that the major diagonal generally has the lowest costs. Interestingly, the lower-left tile has a significantly higher score than all of the others. This is likely due to overlearning, as it comes from a section in the grid with the highest possible step count.
The optimal parameter pair, of 500 steps and a learning rate of .001, is only marginally better than the second-best option, which falls at 1000 steps and a learning rate of .005.
Calculating Costs and Area under ROC Curve Scores for the Model
With the optimal step count and learning rate, the costs of the model built with the given step count and rate are computed.
Can you write Python code for it?

costs_model_1, auc_roc_model_1 = get_scores_for_model(deep_model_1, X, y, optimal_steps, optimal_rate)

print(costs_model_1)
print(auc_roc_model_1)


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
The confidence intervals for area under ROC curve provide the same insight as the confidence intervals for the cost function. The first 2 models are statistically similar with 95% confidence, but the third model statistically outperforms both of the first 2, with 95% confidence, because zero does not fall in the ROC difference confidence interval for model 3 vs the other 2 models.
With the ROC curve score's confirmation, we can confidently declare that architecture 3 is better for this classification task than models 1 and 2. However, in our deployment section, we will address the possibility that even our best deep learning architecture may be outperformed by conventional machine learning algorithms which were explored in Project 2.
Deployment
Given the performances of the three above deep learning models for classifying NFL play types, we beleive that model 3 would be the best choice, as it, with statistical significance, outperformed both other models with respect to both cost and AUC_ROC score. However, the costs for the models generated, as well as the AUC_ROC scores, seemed to, in general be inferior to the scores from the conventional algorithms.
In our previous report, we declared random forests to be our preffered model, due to their superior performance, and ease of use / computational inexpense. Therefore, for the purpose of comparison, we will compare the performance of random forests to the performance of our best deep learning architecture.
Note that we (mistakenly) used an incorrect cross-validation strategy for Lab 2, so random forest scores must be re-generated. The code below, to generate the new scores, is adapted from Lab 2 for use with our new cross validation strategy.
First, we perform some setup that is needed to use the original SKLearn library for classification.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

#Building the class weight map
PlayTypes = df.PlayType.value_counts().index.tolist()
Costs = [sum(x) for x in cost_mat]
y = df.PlayType.values

ClassWeights = dict(zip(PlayTypes, Costs))


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
The cost and auc scores for the random forest are saved below so that they don't need to be re-computed.
Can you write Python code for it?


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
The costs and auc scores computed above are hard-coded below for later use, so that they don't need to be computed again.
Can you write Python code for it?


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
The following are descriptions of the remaining data columns in the play-by-play dataset. Note that the one-hot encoded columns do not follow the structure listed below, but for the sake of readability they are presented as if they were not one-hot encoded.

GameID (nominal): A unique integer which identifies each game played
Drive (ordinal): The number of the drive during a game when the play occurred (indexed at one, so the first drive of the game has Drive 1 and the nth drive has Drive n)
qtr (interval): The quarter of the game when the play occurred
down (interval): The down when the play occurred (1st, 2nd, 3rd, or 4th)
TimeSecs (interval): The remaining game time, in seconds, when the play began
yrdline100 (ratio): The absolute yard-line on the field where the play started (from 0 to 100, where 0 is the defensive end zone and 100 is the offensive end zone of the team with the ball)
ydstogo (ratio): The number of yards from the line of scrimmage to the first-down line
ydsnet (ratio): The number of yards from the beginning of the drive to the current line of scrimmage
GoalToGo (nominal): A binary attribute whose value is 1 if there is no first down line (the end-zone is the first down line) or 0 if there is a normal first down line
posteam (nominal): A 2-or-3 character code representing the team on offense
PosTeamScore (ratio): The score of the team with possesion of the ball
DefensiveTeam (nominal): A 2-or-3 character code representing the team on defense
ScoreDiff: (ratio) The difference in score between the offensive and defensive at the time of the play.
PlayType: (nominal) An attribute that identifies the type of play (i.e. Kickoff, Run, Pass, Sack, etc)

Performance Metrics
The value of a classifier will be evaulated using the following cost matrix. Costs in the matrix which are set to 1 represent play predictions that would never actually occur in the context of a football game. For example, if we predicted a pass play and a kickoff occurs, then the classifier has a significant flaw.
Bolded...
Can you write Python code for it?

from sklearn.model_selection import KFold

#Using a 10-fold sequential split.
#Note that this cv object is unused, but is here for reference
cv = KFold(n_splits=10)

y,levels = pd.factorize(df.PlayType.values)
X = df.drop('PlayType', 1).values.astype(np.float32)

num_classes = len(levels)


I am working on the file "Lab3/Lab3.ipynb".
The first task is:
Third Deep Learning Architecture
Due to the poor performance of the two architectures we used in the previous two deep models, we have elected to build our own model to test on the data. This third deep learning architecture was primarily the product of trial and error. Many different cofigurations of layers and activation functions were tried, and in the given time, this the setup that resulted in the lowest cost for the model.
We explored a number of different architectures, of varying complexity, including an LSTM network from TensorFlow, and we discovered that, generally speaking, simple networks which look like just a multi-layer perceptron seem to provide the lowest cost. We tuned the sizes of the hidden layers with trial and error, and we discovered that the network shown below is the most efficient.
Note that the team embeddings are not shown in the main drawing, as they are computed using a very small network prior to the main MLP. Interestingly, we found that the smallest networks were the best for the team embeddings. We beleive that this is due to overlearning that may occur if the embeddings have too much weight in the overall network architecture.
The architecture consists of 6 fully connected layers. The first 3 layers have 1000 nodes, followed by a 500 node layer, a 200 node layer, and another 1000 node layer. This is followed by a relu activation function. The output of this network is then used for classifcation.
<img src="NetworkDrawings/network3.png">
Can you write Python code for it?

def deep_model_3(X, y):

    #Embeddings layer
    teamembeddings = layers.stack(X[:,11:75], layers.fully_connected, [20,4])
    #teamembeddings = tf.nn.relu(teamembeddings)

    #Non-embeddings features
    otherfeatures = X[:,0:10]

    #Concatenate the embeddings with the non-embeddings
    tensors = tf.concat(1, [teamembeddings, otherfeatures])

    #[500,200,100,500][1000,1000,1000,500,200,1000]
    tensors = layers.stack(tensors, layers.fully_connected, [1000,1000,1000,500,200,1000])

    #Relu activation function
    tensors = tf.nn.relu(tensors)

    pred, loss = learn.models.logistic_regression(tensors, y)

    return pred, loss


I am working on the file "Lab4/lab4.ipynb".
The first task is:
$$
\dot{u} = a \cdot \left( b \cdot v - u \right)
$$
Can you write Python code for it?

def Ut(v, u):
    return a * (b * v - u)


I am working on the file "Lab4/lab4.ipynb".
The first task is:
Critical values
$$
\begin{cases}
v_{cr} = \frac{b - k_2}{2 \cdot k_1} \
u_{cr} = b \cdot v_{cr}
\end{cases}
$$
Linearization
$$
\begin{cases}
\dot{v} = k_1 \cdot v^2 + k_2 \cdot v + k_3 - u + I \
\dot{u} = a \cdot \left( b \cdot v - u \right)
\end{cases}
$$
Jacobian
$$
J =
\begin{bmatrix}
2 \cdot k_1 \cdot v + k_2 & -1 \
a \cdot b & -a
\end{bmatrix}
$$
Linearized Jacobian
$$
J_l =
\begin{bmatrix}
b & -1 \
a \cdot b & -a
\end{bmatrix}
$$
Characteristics polynomial
$$
\begin{vmatrix}
b - \lambda & -1 \
a \cdot b & -a - \lambda
\end{vmatrix}
= \left( - b + \lambda \right) \cdot \left( a + \lambda \right) + a \cdot b
= - b \cdot a + \lambda \cdot a - b \cdot \lambda + \lambda \cdot \lambda + a \cdot b
= \lambda^2 + \lambda \cdot \left( a - b \right) = 0
$$
Finally
$$
\lambda \cdot \left( \lambda + a - b \right) = 0
$$
Solutions
$$
\begin{cases}
\lambda_1 = 0 \
\lambda_2 = b - a
\end{cases}
$$
Can you write Python code for it?

lambdas = 0, b - a; lambdas

(lambdas[0] + lambdas[1]) / 2, lambdas[0] - (lambdas[0] + lambdas[1]) / 2


I am working on the file "Lab4/lab4.ipynb".
The first task is:
Following calculations are done with NumPy library and contain real eigenvalues and normed eigenvectors
Can you write Python code for it?


I am working on the file "Lab4/lab4.ipynb".
The first task is:
If they're almost equal we've done
Can you write Python code for it?

assert np.allclose(left[0], right[0]) and np.allclose(left[1], right[1])


I am working on the file "Lab4/lab4.ipynb".
The first task is:
Separatrix
Let's take a look at plots with
$$\left( v_0, u_0 \right) = \left( v_{crit} \pm e_v, u_{crit} \pm e_u \right)$$
Can you write Python code for it?

multiplier = 10.
delta_v = multiplier * e_x_[1]
delta_u_unstable = multiplier * e_y_[1]
delta_v, delta_u_unstable

get_plot(I_critical, (v_critical + delta_v, u_critical + delta_u_unstable))

get_plot(I_critical, (v_critical - delta_v, u_critical - delta_u_unstable))


I am working on the file "Lab4/lab4.ipynb".
The first task is:
These are the values of the right part of the solved equation
$$
\begin{bmatrix}
b & -1 \
a \cdot b & -a
\end{bmatrix}
\cdot e_{1, 2}
$$
Can you write Python code for it?


I am working on the file "Lab5.ipynb".
The first task is:
$$
p(\verb+Bronchitis+ = 1 | \verb+Influenza+ = 1, \verb+Smokes+ = 1) = 0.99 \\
p(\verb+Bronchitis+ = 1 | \verb+Influenza+ = 1, \verb+Smokes+ = 0) = 0.9 \\
p(\verb+Bronchitis+ = 1 | \verb+Influenza+ = 0, \verb+Smokes+ = 1) = 0.7 \\
p(\verb+Bronchitis+ = 1 | \verb+Influenza+ = 0, \verb+Smokes+ = 0) = 0.0001
$$
Can you write Python code for it?


I am working on the file "Lab5.ipynb".
The first task is:
$$
p(\verb+Coughing+ = 1| \verb+Bronchitis+ = 1) = 0.8 \\
p(\verb+Coughing+ = 1| \verb+Bronchitis+ = 0) = 0.07
$$
Can you write Python code for it?

def init_f4(B, C):
    # Order: B, C
    f4_weights = np.empty((2, 2))

    f4_weights[1, 1] = 0.8
    f4_weights[0, 1] = 0.07

    f4_weights[:, 0] = 1 - f4_weights[:, 1]

    f4 = Factor('f4', f4_weights, [B, C])
    return f4


I am working on the file "Lab5.ipynb".
The first task is:
$$
p(\verb+Fever+ = 1| \verb+Influenza+ = 1) = 0.9 \\
p(\verb+Fever+ = 1| \verb+Influenza+ = 0) = 0.05
$$
Can you write Python code for it?

def init_f2(I, F):
    # Order: I, F
    f2_weights = np.empty((2, 2))

    f2_weights[1, 1]  = 0.9
    f2_weights[0, 1] = 0.05

    f2_weights[:, 0] = 1 - f2_weights[:, 1]

    f2 = Factor('f2', f2_weights, [I, F])
    return f2


I am working on the file "Lab5.ipynb".
The first task is:
$$
p(\verb+Wheezing+ = 1| \verb+Bronchitis+ = 1) = 0.6 \\
p(\verb+Wheezing+ = 1| \verb+Bronchitis+ = 0) = 0.001
$$
Can you write Python code for it?

def init_f5(B, W):
    # Order: B, W
    f5_weights = np.empty((2, 2))

    f5_weights[1, 1] = 0.6
    f5_weights[0, 1] = 0.001

    f5_weights[:, 0] = 1 - f5_weights[:, 1]

    f5 = Factor('f5', f5_weights, [B, W])
    return f5


I am working on the file "Lab5.ipynb".
The first task is:
1.1 Instantiate network (10 points)
Convert the directed graphical model ("Bayesian Network") shown below to a factor graph. Instantiate this graph by creating Variable and Factor instances and linking them according to the graph structure.
To instantiate the factor graph, first create the Variable nodes and then create Factor nodes, passing a list of neighbour Variables to each Factor.
Use the following prior and conditional probabilities.
$$
p(\verb+Influenza+) = 0.05 \\
p(\verb+Smokes+) = 0.2 \\
$$
$$
p(\verb+SoreThroat+ = 1 | \verb+Influenza+ = 1) = 0.3 \\
p(\verb+SoreThroat+ = 1 | \verb+Influenza+ = 0) = 0.001 \\
p(\verb+Fever+ = 1| \verb+Influenza+ = 1) = 0.9 \\
p(\verb+Fever+ = 1| \verb+Influenza+ = 0) = 0.05 \\
p(\verb+Bronchitis+ = 1 | \verb+Influenza+ = 1, \verb+Smokes+ = 1) = 0.99 \\
p(\verb+Bronchitis+ = 1 | \verb+Influenza+ = 1, \verb+Smokes+ = 0) = 0.9 \\
p(\verb+Bronchitis+ = 1 | \verb+Influenza+ = 0, \verb+Smokes+ = 1) = 0.7 \\
p(\verb+Bronchitis+ = 1 | \verb+Influenza+ = 0, \verb+Smokes+ = 0) = 0.0001 \\
p(\verb+Coughing+ = 1| \verb+Bronchitis+ = 1) = 0.8 \\
p(\verb+Coughing+ = 1| \verb+Bronchitis+ = 0) = 0.07 \\
p(\verb+Wheezing+ = 1| \verb+Bronchitis+ = 1) = 0.6 \\
p(\verb+Wheezing+ = 1| \verb+Bronchitis+ = 0) = 0.001 \\
$$
Can you write Python code for it?

from IPython.core.display import Image
Image(filename='bn.png')


I am working on the file "Lab5.ipynb".
The first task is:
1.3 Variable to factor messages (10 points)
Write a method send_sp_message(self, other) for the Variable class, that checks if all the information required to pass a message to Variable var is present, computes the message and sends it to factor.
Can you write Python code for it?

def variable_send_sp_msg(self, other):
    assert isinstance(other, Factor)
    # checking if the target factor node has the current node in its neighbours
    assert self in other.neighbours
    # computing the message
    mes = np.prod(self.get_neighbour_messages(other),axis=0) * self.observed_state
    self.send_msg_proc(other,mes)

Variable.send_sp_msg = variable_send_sp_msg


I am working on the file "Lab5.ipynb".
The first task is:
1.4 Compute marginal (10 points)
Later in this assignment, we will implement message passing schemes to do inference. Once the message passing has completed, we will want to compute local marginals for each variable.
Write the method marginal for the Variable class, that computes a marginal distribution over that node.
Can you write Python code for it?

def variable_marginal(self, Z=None):
    """
    Compute the marginal distribution of this Variable.
    It is assumed that message passing has completed when this function is called.
    Args:
        Z: an optional normalization constant can be passed in. If None is passed, Z is computed.
    Returns: marginal, Z. The first is a numpy array containing the normalized marginal distribution.
     Z is either equal to the input Z, or computed in this function (if Z=None was passed).
    """
    mar = np.prod(self.in_msgs.values(),axis=0) * self.observed_state
    if Z==None:Z=sum(mar)
    return mar/Z, Z

Variable.marginal = variable_marginal


I am working on the file "Lab5.ipynb".
The first task is:
1.6 Inference Engine (10 points)
Write a function sum_product(node_list) that runs the sum-product message passing algorithm on a tree-structured factor graph with given nodes. The input parameter node_list is a list of all Node instances in the graph, which is assumed to be ordered correctly. That is, the list starts with a leaf node, which can always send a message. Subsequent nodes in node_list should be capable of sending a message when the pending messages of preceding nodes in the list have been sent. The sum-product algorithm then proceeds by passing over the list from beginning to end, sending all pending messages at the nodes it encounters. Then, in reverse order, the algorithm traverses the list again and again sends all pending messages at each node as it is encountered. For this to work, you must initialize pending messages for all the leaf nodes, e.g. influenza_prior.pending.add(influenza), where influenza_prior is a Factor node corresponding the the prior, influenza is a Variable node and the only connection of influenza_prior goes to influenza.
Can you write Python code for it?

def apply_algorithm(node_list, func):
    for node in node_list:
        for other in list(node.pending):
            func(node, other)

def sum_product(node, other):
    node.send_sp_msg(other)

def configure_experiment():
    variables = init_variables()

    I, S, ST, F, B, C, W = variables

    f1 = init_f1(I, ST)
    f2 = init_f2(I, F)
    f3 = init_f3(I, S, B)
    f4 = init_f4(B, C)
    f5 = init_f5(B, W)
    f6 = init_f6(S)
    f7 = init_f7(I)

    f6.pending.update([S])
    f7.pending.update([I])

    ST.pending.update([f1])
    F.pending.update([f2])
    C.pending.update([f4])
    W.pending.update([f5])

    return (I, S, ST, F, B, C, W), (f1, f2, f3, f4, f5, f6, f7)

def print_marginals(variables):
    for variable in variables:
        marginal, Z = variable.marginal(None)
        print variable, marginal

variables, factors = configure_experiment()

I, S, ST, F, B, C, W = variables
f1, f2, f3, f4, f5, f6, f7 = factors

node_list = [f6, f7, W, C, F, f4, f5, S, f2, B, f3, I, f1, ST]

print '-----Forward pass-----'
apply_algorithm(node_list, sum_product)

ST.pending.update([f1])

print '-----Backward pass-----'
apply_algorithm(reversed(node_list), sum_product)

print '-----Marginals-----'
print_marginals(variables)


I am working on the file "Lab5.ipynb".
The first task is:
1.7 Observed variables and probabilistic queries (15 points)
We will now use the inference engine to answer probabilistic queries. That is, we will set certain variables to observed values, and obtain the marginals over latent variables. We have already provided functions set_observed and set_latent that manage a member of Variable called observed_state. Modify the Variable.send_msg and Variable.marginal routines that you wrote before, to use observed_state so as to get the required marginals when some nodes are observed.
Can you write Python code for it?

variables, factors = configure_experiment()

I, S, ST, F, B, C, W = variables
f1, f2, f3, f4, f5, f6, f7 = factors

B.set_observed(1)

node_list = [f6, f7, W, C, F, f4, f5, S, f2, B, f3, I, f1, ST]

print 'Forward pass'
apply_algorithm(node_list, sum_product)

ST.pending.update([f1])

print 'Backward pass'
apply_algorithm(reversed(node_list), sum_product)

print_marginals(variables)


I am working on the file "Lab5.ipynb".
The first task is:
1.8 Sum-product and MAP states (5 points)
A maximum a posteriori state (MAP-state) is an assignment of all latent variables that maximizes the probability of latent variables given observed variables:
$$
\mathbf{x}{\verb+MAP+} = \arg\max {\mathbf{x}} p(\mathbf{x} | \mathbf{y})
$$
Could we use the sum-product algorithm to obtain a MAP state? If yes, how? If no, why not?
Answer: It's possible to compute MAP by factorizing $p(x|y)$ into components and considering all possible combinations of its components, for example:
$$
p(x|y) = p(x_1)p(y_1|x_1)p(x_3|x_1)p(x_2|y_1)
$$
in the case of a bayesian network $y_1 \rightarrow x_2$, $x_1 \rightarrow x_3$, $y_1 \rightarrow x_1$, and $y_1$ is observed. By considering all possibilites of states $p(x_1)$, $p(y_1|x_1)$, etc and computing them using sum-product algorithm, we can compute MAP but it's not efficient, that's why we need max-sum.
Technically, we could compute conditional probabilities using sum-product by using set_observed() methods, which we used in the previous task. The main inefficiency is caused by the fact that we will be forced to recompute messages once the combination of initially observed states changes.
Part 2: The max-sum algorithm
Next, we implement the max-sum algorithm as described in section 8.4.5 of Bishop.
2.1 Factor to variable messages (10 points)
Implement the function Factor.send_ms_msg that sends Factor -> Variable messages for the max-sum algorithm. It is analogous to the Factor.send_sp_msg function you implemented before.
Can you write Python code for it?

def factor_send_ms_msg(self, other):
    assert isinstance(other,Variable)
    if not other in self.neighbours:
        raise Exception('The specified node is not a neighbour')

    factor_indexes = range(len(self.neighbours))
    factor_indexes.remove(self.neighbours.index(other))

    mes = self.get_neighbour_messages(other)

    # computing messages
    mes = np.expand_dims(np.add.reduce(np.ix_(*mes)), self.neighbours.index(other))
    mes = np.apply_over_axes(np.amax, np.log(self.f)+mes,factor_indexes).squeeze()

    # sending the message
    self.send_msg_proc(other,mes)

Factor.send_ms_msg = factor_send_ms_msg


I am working on the file "Lab5.ipynb".
The first task is:
2.2 Variable to factor messages (10 points)
Implement the Variable.send_ms_msg function that sends Variable -> Factor messages for the max-sum algorithm.
Can you write Python code for it?

def variable_send_ms_msg(self, other):
    assert isinstance(other, Factor)
    # checking if the target factor node has the current node in its neighbours
    assert self in other.neighbours
    # computing the message
    mes = np.sum(self.get_neighbour_messages(other),axis=0)+np.log(self.observed_state)
    self.send_msg_proc(other, mes)

Variable.send_ms_msg = variable_send_ms_msg


I am working on the file "Lab5.ipynb".
The first task is:
3.2 Loopy max-sum (10 points)
Implement the loopy max-sum algorithm, by passing messages from randomly chosen nodes iteratively until no more pending messages are created or a maximum number of iterations is reached.
Think of a good way to initialize the messages in the graph.
Can you write Python code for it?

def denoise(img, niter=10):
    from itertools import product
    Y, X, fYX, fXR, fXB = create_factor_graph(img)
    for i in range(niter):
        fXX = np.hstack((fXR.flatten(), fXB.flatten()))
        np.random.shuffle(fXX)

        # Preordered, first observed variables, then factors between observed variables and
        # corresponding laten variables, then all latent variables and then factors between
        # latents in the random order.
        node_list = np.hstack((Y.flatten(), fYX.flatten(), X.flatten(), fXX)).tolist()

        apply_algorithm(node_list, max_sum)
    result = np.zeros_like(img)
    for y, x in product(range(img.shape[0]), range(img.shape[1])):
        result[y, x] = X[y, x].map_state()
    return result

imshow(denoise(noise_test_im))
show()

imshow(denoise(noise_im, niter=10))
show()


I am working on the file "Lab5.ipynb".
The first task is:
Part 3: Image Denoising and Loopy BP
Next, we will use a loopy version of max-sum to perform denoising on a binary image. The model itself is discussed in Bishop 8.3.3, but we will use loopy max-sum instead of Iterative Conditional Modes as Bishop does.
The following code creates some toy data. im is a quite large binary image, test_im is a smaller synthetic binary image. Noisy versions are also provided.
Can you write Python code for it?

from pylab import imread, gray
# Load the image and binarize
im = np.mean(imread('dalmatian1.png'), axis=2) > 0.5
imshow(im)
gray()

# Add some noise
noise = np.random.rand(*im.shape) > 0.9
noise_im = np.logical_xor(noise, im)
figure()
imshow(noise_im)

test_im = np.zeros((10,10))
#test_im[5:8, 3:8] = 1.0
#test_im[5,5] = 1.0
figure()
imshow(test_im)

# Add some noise
noise = np.random.rand(*test_im.shape) > 0.9
noise_test_im = np.logical_xor(noise, test_im)
figure()
imshow(noise_test_im)

show()


I am working on the file "Lab5.ipynb".
The first task is:
We have transformed the bayesian network to a factor graph as following:

where:
$$
f_1 = p(ST|I)\\
f_2 = p(F|I)\\
f_3 = p(B|I,S)\\
f_4 = p(C|B)\\
f_5 = p(W|B)\\
f_6 = p(S)\\
f_7 = p(I)
$$
Can you write Python code for it?

# Variables
def init_variables():
    I = Variable('Influenza', 2)
    S = Variable('Smokes', 2)
    ST = Variable('SoreThroat', 2)
    F = Variable('Fever', 2)
    B = Variable('Bronchitits', 2)
    C = Variable('Coughing', 2)
    W = Variable('Wheezing', 2)
    return I, S, ST, F, B, C, W


I am working on the file "Lab5/Lab5-Task1.ipynb".
The first task is:
3-Gram Generator
A for loop to generate the time duration and set up the path for output
We can specify a folder structure having different number of files.
And similarly we can define the output folder based on the input folder
Also showing the data in a tabular format.
Following is just a test result we did using small amount of data.
Running the same code with better RAM provides much better performance.
Can you write Python code for it?

for dirPath in subDirList:
    outputPath = dirPath.replace("input","output3")

    start_time = time.time()

    data_rdd = sc.textFile(dirPath)

    test = data_rdd.filter(lambda y : y.strip()!="")\
        .map(lambda x : x.replace('\t','').lower().split(">"))\
        .map(lambda (x,y): (x,regex.sub('',y).strip().replace("j","i").replace("v","u").split(" ")))\
        .flatMap(lambda (x,y): [(pair,x[1:]+" |"+str(1+y.index(pair[0]))+"."+str(1+y.index(pair[1]))+"."+str(1+y.index(pair[2]))+"| ") for pair in combinations(y,3)])\
        .filter(lambda (x,y): x[0]!="" and x[1]!="" and x[2]!="")\
        .flatMap(lambda (x,y): [(lemma,y) for lemma in product(getLemmasList(x[0]),getLemmasList(x[1]),getLemmasList(x[2]))])\
        .reduceByKey(lambda x,y : x + ", "+y).sortByKey(True)

    print("Input Directory Path :" + dirPath)
    print("Ouput Directory Path :" + outputPath)
    print("Time taken for "+ dirPath[-1:] +" files %s" % (time.time() - start_time))

    test = test.map(lambda (x,y):("{"+x[0]+","+x[1]+","+x[2]+"}",y))

    test.take(5)

    df = sqlContext.createDataFrame(test, ['n-gram (n =3)', 'Location'])

    df.show()

    df.coalesce(1).write.option("header", "true").csv(outputPath+"/result.csv")


I am working on the file "Lab5/Lab5-Task1.ipynb".
The first task is:
A function to read the string and generate the lemma dictionary
Can you write Python code for it?

def buildLemmaDict(x):
    lemmas = x.split(",")

    lemmaWords = []

    for lemma in lemmas:
        if lemma!="":
            lemmaWords.append(lemma)

    lemmaDic = [(lemmas[0],list(set(lemmaWords)))]

    return(lemmaDic)


I am working on the file "Lab5/Lab5-Task1.ipynb".
The first task is:
Collect the rdd as a map to get the dictionary
Can you write Python code for it?


I am working on the file "Lab5/Lab5-Task1.ipynb".
The first task is:
Create a rdd from the above file to apply a common function "buildLemmaDict"
Can you write Python code for it?

lemmaDictionary_rdd = (lemma_rdd.flatMap(lambda x : buildLemmaDict(x)))


I am working on the file "Lab5/Lab5-Task1.ipynb".
The first task is:
Generate a pyspark Context
Can you write Python code for it?


I am working on the file "Lab5/Lab5-Task1.ipynb".
The first task is:
Scale up Graph results for 2-Gram and 3-Gram
The values used are the actual data we collected from various number of files
Can you write Python code for it?


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
Now we compute the test error (% predicted survival outcomes == actual outcomes) and display the decision tree for good measure.
Can you write Python code for it?

accuracy = truth_and_predictions_rdd.filter(lambda v_p: v_p[0] == v_p[1]).count() / float(test_count)
print('Accuracy =', accuracy)
print(model.toDebugString())


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
Sample the data
We point the context at a CSV file on disk. The result is a RDD, not the content of the file. This is a Spark transformation.
Can you write Python code for it?


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
Split for training and test
We split the transformed data into a training (70%) and test set (30%), and print the total number of items in each segment.
Can you write Python code for it?

training_rdd, test_rdd = labeled_points_rdd.randomSplit([0.7, 0.3], seed = 0)

training_count = training_rdd.count()
test_count = test_rdd.count()

training_count, test_count


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
Train and test a logistic regression classifier
For a simple comparison, we also train and test a LogisticRegressionWithSGD model.
Can you write Python code for it?

model = LogisticRegressionWithSGD.train(training_rdd)

predictions_rdd = model.predict(test_rdd.map(lambda x: x.features))

labels_and_predictions_rdd = test_rdd.map(lambda lp: lp.label).zip(predictions_rdd)

accuracy = labels_and_predictions_rdd.filter(lambda v_p: v_p[0] == v_p[1]).count() / float(test_count)
print('Accuracy =', accuracy)


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
We apply the function to all rows.
Can you write Python code for it?


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
We bundle our predictions with the ground truth outcome for each passenger in the test set.
Can you write Python code for it?

truth_and_predictions_rdd = test_rdd.map(lambda lp: lp.label).zip(predictions_rdd)


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
We query RDD for the number of lines in the file. The call here causes the file to be read and the result computed. This is a Spark action.
Can you write Python code for it?


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
We query for the first five rows of the RDD. Even though the data is small, we shouldn't get into the habit of pulling the entire dataset into the notebook. Many datasets that we might want to work with using Spark will be much too large to fit in memory of a single machine.
Can you write Python code for it?


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
We see a header row followed by a set of data rows. We filter out the header to define a new RDD containing only the data rows.
Can you write Python code for it?

header = raw_rdd.first()
data_rdd = raw_rdd.filter(lambda line: line != header)


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
We see that the first value in every row is a passenger number. The next three values are the passenger attributes we might use to predict passenger survival: ticket class, age group, and gender. The final value is the survival ground truth.
Create labeled points (i.e., feature vectors and ground truth)
Now we define a function to turn the passenger attributions into structured LabeledPoint objects.
Can you write Python code for it?

def row_to_labeled_point(line):
    '''
    Builds a LabelPoint consisting of:

    survival (truth): 0=no, 1=yes
    ticket class: 0=1st class, 1=2nd class, 2=3rd class
    age group: 0=child, 1=adults
    gender: 0=man, 1=woman
    '''
    passenger_id, klass, age, sex, survived = [segs.strip('"') for segs in line.split(',')]
    klass = int(klass[0]) - 1

    if (age not in ['adults', 'child'] or
        sex not in ['man', 'women'] or
        survived not in ['yes', 'no']):
        raise RuntimeError('unknown value')

    features = [
        klass,
        (1 if age == 'adults' else 0),
        (1 if sex == 'women' else 0)
    ]
    return LabeledPoint(1 if survived == 'yes' else 0, features)


I am working on the file "Lab5/Welcome+to+Spark+with+Python.ipynb".
The first task is:
We take a random sample of the data rows to better understand the possible values.
Can you write Python code for it?


I am working on the file "Lab6/lab6.ipynb".
The first task is:
Task 2.2
Can you write Python code for it?

sequence = sequences[0]
sequence_shifted = sequence[:]
sequence_shifted.insert(0, 0)
sequence_shifted.pop(-1)

plt.title(r'$x\left( n \right), x\left( n+1 \right)$')
plt.plot(sequence_shifted, sequence)
plt.show()


I am working on the file "Lab6/lab6.ipynb".
The first task is:
Task 4.1
Can you write Python code for it?

xs, ys, titles = [], [], []
for uv_sequence in uv_sequences:
    xs.append(uv_sequence[0][0])
    ys.append(uv_sequence[0][1])
    titles.append(r'$\varepsilon=%s$' % uv_sequence[1])
draw_plots(xs, ys, titles)


I am working on the file "Lab6/lab6.ipynb".
The first task is:
Task 4.2
Can you write Python code for it?

xs, ys, titles = [], [], []
for xy_sequence in xy_sequences:
    xs.append(xy_sequence[0][0])
    ys.append(xy_sequence[0][1])
    titles.append(r'$\varepsilon=%s$' % xy_sequence[1])
draw_plots(xs, ys, titles)


I am working on the file "Lab6/lab6.ipynb".
The first task is:
Task 4.3
Can you write Python code for it?

xs, ys, titles = [], [], []
for uv_sequence in uv_sequences:
    vn = [x for x in uv_sequence[0][1] if x != 0]
    titles.append(r'$\varepsilon=%s$' % uv_sequence[1])
    ys.append([np.log(np.abs(item)) for item in vn])
    xs.append(list(range(len(ys[-1]))))
draw_plots(xs, ys, titles)


I am working on the file "Lab6/lab6.ipynb".
The first task is:
Task 5
Systems are starting synchronization at the moment $\varepsilon = 0.17$ — their link looks less stochastic.
With $\varepsilon = 0.25$ we can see synchronized behavior.
Task 6
Can you write Python code for it?

plt.title(r'$\varepsilon=%s$' % uv_sequences[5][1])
sequence = np.log(np.abs([x for x in uv_sequences[5][0][1] if x != 0]))
x = np.array(list(range(len(sequence))))
a, b = np.linalg.lstsq(np.vstack([x, np.ones_like(sequence)]).T, sequence)[0]

plt.plot(x, sequence, label='Sequence')
plt.plot(x, a * x + b, label='Approximation')
plt.legend()
plt.show()

a, 1/a


I am working on the file "Lab9 - TOOLS.ipynb".
The first task is:
$\chi^2$ is minimized at $\sigma=2.42$ and $\bar{x}=99.92$
The uncertainty in the mean is given by: $$\delta \bar{x}=\frac{1}{\sqrt{N}}=\frac{1}{\sqrt{9}}=0.333$$
I've had a hard time trying to find the uncertainty in sigma, unfortunately. Using propagation of error blows it up since each of the nine terms must have their own partial derivatives calculated.
There is no obvious covariance between the mean and the standard deviation, As we can see by the argument in the previous section. There is no single line for which $\frac{\partial^2\chi^2}{\partial m^2}=\frac{\partial^2\chi^2}{\partial b^2}=0$.
Can you write Python code for it?

plt.plot(x,y)
plt.plot(x,(np.exp(-((x-mean[meanfit])**2)/(2.0*(sig[sigfit])**2)))/(sig[sigfit]*np.sqrt(2.0*np.pi)))
plt.show()

plt.imshow(chi2,vmax=.2)
plt.colorbar()
plt.contour(chi2,levels=np.arange(0,.03,.005))
plt.show()


I am working on the file "Lab9 - TOOLS.ipynb".
The first task is:
Consult the internet to "reverse engineer" what is going on in the example here.  Put comments in the code (using #) to explain what is happening in most of the lines.  You should have enough comments such that you could use this to fit your own data with your own function.
Carefully explain what the printed outputs mean.  Hint:  What is the significance of the next cell's output?
The "Coefficients output" is the parameters of the gaussian fit, first the mean, then the standard deviation. The next output is the variance matrix, showing the best estimates for the variance coefficients of each parameter.
Can you write Python code for it?


I am working on the file "Lab9 - TOOLS.ipynb".
The first task is:
Consult the internet to "reverse engineer" what is going on in the example here. Carefully explain all the outputs of the cell above.
The first array consists of the fit parameters that were calculated from x and y. Since the function was fed a polynomial degree argument of 1, there are two parameters returned.
Because cov was set to true, the other output is the covariance matrix for the parameters. The values on the main diagonal are the variance estimates for each parameter.
$$\left| \begin{array}{cc}
 0.0004177 & -0.00167676 \
 -0.00167676 & 0.00822924 \end{array} \right|$$
Part 2a.  Fitting Other Curves
Now repeat the exercise by fitting a Gaussian distribution to the data below, perhaps representing results from one of our previous labs.
Can you write Python code for it?

x=np.array([95.,96.,97.,98.,99.,100.,101.,102.,103.])
yraw=np.array([1.,2.,3.,4.,5.,6.,5.,4.,3.])
y=yraw/(np.sum(yraw))

# x=np.random.normal(loc=100.,scale=10.,size=1000)
# hist, bin_edges = np.histogram(x, density=True)
# bin_centers = (bin_edges[:-1] + bin_edges[1:])/2

plt.plot(x,y)
plt.show()

def gauss(x, *p):
    mu, sigma = p
    return (1/(sigma*2.506))*np.exp(-(x-mu)**2/(2.*sigma**2))

mean=np.arange(98.,102.,0.01)
sig=np.arange(1.0,6.0,0.01)
chi2=np.zeros([mean.size,sig.size])
for i in range(mean.size):
    for j in range(sig.size):
        chi2[i,j]=sum(((y-(np.exp(-((x-mean[i])**2)/(2.0*(sig[j])**2)))/(sig[j]*np.sqrt(2.0*np.pi)))**2)/1.0**2)

meanfit,sigfit=np.unravel_index(np.argmin(chi2), (mean.size,sig.size))
print(mean[meanfit])
print(sig[sigfit])
print(chi2[meanfit][sigfit])


I am working on the file "Lab9 - TOOLS.ipynb".
The first task is:
Try a range of slopes and intercepts, and calculate $\chi^2$ values for each set.
Can you write Python code for it?

m=np.arange(-1,4.0,0.1)
b=np.arange(-5.0,5.0,0.1)
chi2=np.zeros([m.size,b.size])
for i in range(m.size):
    for j in range(b.size):
        chi2[i,j]=sum(((y-m[i]*x-b[j])**2)/dely**2)