I am working on the file "MachineLearning/DataScience-Python3/MatPlotLib.ipynb".
The first task is:
Save it to a File
Can you write Python code for it?

plt.plot(x, norm.pdf(x))
plt.plot(x, norm.pdf(x, 1.0, 0.5))
plt.savefig('C:\\Users\\Frank\\MyPlot.png', format='png')


I am working on the file "MachineLearning/DataScience-Python3/MatPlotLib.ipynb".
The first task is:
XKCD Style :)
Can you write Python code for it?

plt.xkcd()

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
plt.xticks([])
plt.yticks([])
ax.set_ylim([-30, 10])

data = np.ones(100)
data[70:] -= np.arange(30)

plt.annotate(
    'THE DAY I REALIZED\nI COULD COOK BACON\nWHENEVER I WANTED',
    xy=(70, 1), arrowprops=dict(arrowstyle='->'), xytext=(15, -10))

plt.plot(data)

plt.xlabel('time')
plt.ylabel('my overall health')


I am working on the file "MachineLearning/DataScience-Python3/PCA.ipynb".
The first task is:
Let's see how much information we've managed to preserve:
Can you write Python code for it?


I am working on the file "MachineLearning/DataScience-Python3/PCA.ipynb".
The first task is:
So, this tells us our data set has 150 samples (individual flowers) in it. It has 4 dimensions - called features here, and three distinct Iris species that each flower is classified into.
While we can visualize 2 or even 3 dimensions of data pretty easily, visualizing 4D data isn't something our brains can do. So let's distill this down to 2 dimensions, and see how well it works:
Can you write Python code for it?

X = iris.data
pca = PCA(n_components=2, whiten=True).fit(X)
X_pca = pca.transform(X)


I am working on the file "MachineLearning/DataScience-Python3/PCA.ipynb".
The first task is:
That's pretty cool. Although we have thrown away two of our four dimensions, PCA has chosen the remaining two dimensions well enough that we've captured 92% of the variance in our data in a single dimension alone! The second dimension just gives us an additional 5%; altogether we've only really lost less than 3% of the variance in our data by projecting it down to two dimensions.
As promised, now that we have a 2D representation of our data, we can plot it:
Can you write Python code for it?

%matplotlib inline
from pylab import *

colors = cycle('rgb')
target_ids = range(len(iris.target_names))
pl.figure()
for i, c, label in zip(target_ids, colors, iris.target_names):
    pl.scatter(X_pca[iris.target == i, 0], X_pca[iris.target == i, 1],
        c=c, label=label)
pl.legend()
pl.show()


I am working on the file "MachineLearning/DataScience-Python3/SVC.ipynb".
The first task is:
By setting up a dense mesh of points in the grid and classifying all of them, we can render the regions of each cluster as distinct colors:
Can you write Python code for it?

def plotPredictions(clf):
    xx, yy = np.meshgrid(np.arange(0, 250000, 10),
                     np.arange(10, 70, 0.5))
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

    plt.figure(figsize=(8, 6))
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, cmap=plt.cm.Paired, alpha=0.8)
    plt.scatter(X[:,0], X[:,1], c=y.astype(np.float))
    plt.show()

plotPredictions(svc)


I am working on the file "MachineLearning/DataScience-Python3/SimilarMovies.ipynb".
The first task is:
And, sort these new results by similarity score. That's more like it!
Can you write Python code for it?


I am working on the file "MachineLearning/DataScience-Python3/SimilarMovies.ipynb".
The first task is:
Let's get rid of any movies rated by fewer than 100 people, and check the top-rated ones that are left:
Can you write Python code for it?

popularMovies = movieStats['rating']['size'] >= 100
movieStats[popularMovies].sort_values([('rating', 'mean')], ascending=False)[:15]


I am working on the file "MachineLearning/DataScience-Python3/SimilarMovies.ipynb".
The first task is:
Now the amazing pivot_table function on a DataFrame will construct a user / movie rating matrix. Note how NaN indicates missing data - movies that specific users didn't rate.
Can you write Python code for it?

movieRatings = ratings.pivot_table(index=['user_id'],columns=['title'],values='rating')
movieRatings.head()


I am working on the file "MachineLearning/DataScience-Python3/SimilarMovies.ipynb".
The first task is:
Our results are probably getting messed up by movies that have only been viewed by a handful of people who also happened to like Star Wars. So we need to get rid of movies that were only watched by a few people that are producing spurious results. Let's construct a new DataFrame that counts up how many ratings exist for each movie, and also the average rating while we're at it - that could also come in handy later.
Can you write Python code for it?

import numpy as np
movieStats = ratings.groupby('title').agg({'rating': [np.size, np.mean]})
movieStats.head()


I am working on the file "MachineLearning/DataScience-Python3/SimilarMovies.ipynb".
The first task is:
Pandas' corrwith function makes it really easy to compute the pairwise correlation of Star Wars' vector of user rating with every other movie! After that, we'll drop any results that have no data, and construct a new DataFrame of movies and their correlation score (similarity) to Star Wars:
Can you write Python code for it?

similarMovies = movieRatings.corrwith(starWarsRatings)
similarMovies = similarMovies.dropna()
df = pd.DataFrame(similarMovies)
df.head(10)


I am working on the file "MachineLearning/DataScience-Python3/TopPages.ipynb".
The first task is:
Here's the full path to the log file I'm analyzing; change this if you want to run this stuff yourself:
Can you write Python code for it?


I am working on the file "MachineLearning/DataScience-Python3/TopPages.ipynb".
The first task is:
Hm. The 'request' part of the line is supposed to look something like this:
GET /blog/ HTTP/1.1
There should be an HTTP action, the URL, and the protocol. But it seems that's not always happening. Let's print out requests that don't contain three items:
Can you write Python code for it?

URLCounts = {}

with open(logPath, "r") as f:
    for line in (l.rstrip() for l in f):
        match= format_pat.match(line)
        if match:
            access = match.groupdict()
            request = access['request']
            fields = request.split()
            if (len(fields) != 3):
                print(fields)


I am working on the file "MachineLearning/DataScience-Python3/TopPages.ipynb".
The first task is:
It worked! But, the results don't really make sense. What we really want is pages accessed by real humans looking for news from our little news site. What the heck is xmlrpc.php? A look at the log itself turns up a lot of entries like this:
46.166.139.20 - - [05/Dec/2015:05:19:35 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
I'm not entirely sure what the script does, but it points out that we're not just processing GET actions. We don't want POSTS, so let's filter those out:
Can you write Python code for it?

URLCounts = {}

with open(logPath, "r") as f:
    for line in (l.rstrip() for l in f):
        match= format_pat.match(line)
        if match:
            access = match.groupdict()
            request = access['request']
            fields = request.split()
            if (len(fields) == 3):
                (action, URL, protocol) = fields
                if (action == 'GET'):
                    if URLCounts.has_key(URL):
                        URLCounts[URL] = URLCounts[URL] + 1
                    else:
                        URLCounts[URL] = 1

results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)

for result in results[:20]:
    print(result + ": " + str(URLCounts[result]))


I am working on the file "MachineLearning/DataScience-Python3/TopPages.ipynb".
The first task is:
Now we'll whip up a little script to extract the URL in each access, and use a dictionary to count up the number of times each one appears. Then we'll sort it and print out the top 20 pages. What could go wrong?
Can you write Python code for it?

URLCounts = {}

with open(logPath, "r") as f:
    for line in (l.rstrip() for l in f):
        match= format_pat.match(line)
        if match:
            access = match.groupdict()
            request = access['request']
            (action, URL, protocol) = request.split()
            if URLCounts.has_key(URL):
                URLCounts[URL] = URLCounts[URL] + 1
            else:
                URLCounts[URL] = 1

results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)

for result in results[:20]:
    print(result + ": " + str(URLCounts[result]))


I am working on the file "MachineLearning/DataScience-Python3/TopPages.ipynb".
The first task is:
Now, our new problem is that we're getting a bunch of hits on things that aren't web pages. We're not interested in those, so let's filter out any URL that doesn't end in / (all of the pages on my site are accessed in that manner - again this is applying knowledge about my data to the analysis!)
Can you write Python code for it?

URLCounts = {}

with open(logPath, "r") as f:
    for line in (l.rstrip() for l in f):
        match= format_pat.match(line)
        if match:
            access = match.groupdict()
            agent = access['user_agent']
            if (not('bot' in agent or 'spider' in agent or
                    'Bot' in agent or 'Spider' in agent or
                    'W3 Total Cache' in agent or agent =='-')):
                request = access['request']
                fields = request.split()
                if (len(fields) == 3):
                    (action, URL, protocol) = fields
                    if (URL.endswith("/")):
                        if (action == 'GET'):
                            if URLCounts.has_key(URL):
                                URLCounts[URL] = URLCounts[URL] + 1
                            else:
                                URLCounts[URL] = 1

results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)

for result in results[:20]:
    print(result + ": " + str(URLCounts[result]))


I am working on the file "MachineLearning/DataScience-Python3/TopPages.ipynb".
The first task is:
Yikes! In addition to '-', there are also a million different web robots accessing the site and polluting my data. Filtering out all of them is really hard, but getting rid of the ones significantly polluting my data in this case should be a matter of getting rid of '-', anything containing "bot" or "spider", and W3 Total Cache.
Can you write Python code for it?

URLCounts = {}

with open(logPath, "r") as f:
    for line in (l.rstrip() for l in f):
        match= format_pat.match(line)
        if match:
            access = match.groupdict()
            agent = access['user_agent']
            if (not('bot' in agent or 'spider' in agent or
                    'Bot' in agent or 'Spider' in agent or
                    'W3 Total Cache' in agent or agent =='-')):
                request = access['request']
                fields = request.split()
                if (len(fields) == 3):
                    (action, URL, protocol) = fields
                    if (action == 'GET'):
                        if URLCounts.has_key(URL):
                            URLCounts[URL] = URLCounts[URL] + 1
                        else:
                            URLCounts[URL] = 1

results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)

for result in results[:20]:
    print(result + ": " + str(URLCounts[result]))


I am working on the file "MachineLearning/DataScience-Python3/TrainTest.ipynb".
The first task is:
And against our test data:
Can you write Python code for it?

testx = np.array(testX)
testy = np.array(testY)

axes = plt.axes()
axes.set_xlim([0,7])
axes.set_ylim([0, 200])
plt.scatter(testx, testy)
plt.plot(xp, p4(xp), c='r')
plt.show()


I am working on the file "MachineLearning/DataScience-Python3/TrainTest.ipynb".
The first task is:
And our test dataset:
Can you write Python code for it?


I am working on the file "MachineLearning/DataScience-Python3/TrainTest.ipynb".
The first task is:
Doesn't look that bad when you just eyeball it, but the r-squared score on the test data is kind of horrible! This tells us that our model isn't all that great...
Can you write Python code for it?

from sklearn.metrics import r2_score

r2 = r2_score(testy, p4(testx))

print(r2)


I am working on the file "MachineLearning/DataScience-Python3/TrainTest.ipynb".
The first task is:
Now we'll split the data in two - 80% of it will be used for "training" our model, and the other 20% for testing it. This way we can avoid overfitting.
Can you write Python code for it?

trainX = pageSpeeds[:80]
testX = pageSpeeds[80:]

trainY = purchaseAmount[:80]
testY = purchaseAmount[80:]


I am working on the file "MachineLearning/DataScience-Python3/TrainTest.ipynb".
The first task is:
Now we'll try to fit an 8th-degree polynomial to this data (which is almost certainly overfitting, given what we know about how it was generated!)
Can you write Python code for it?

x = np.array(trainX)
y = np.array(trainY)

p4 = np.poly1d(np.polyfit(x, y, 8))


I am working on the file "MachineLearning/DecisionTree.ipynb".
The first task is:
<H2> Load CSV file </H2>
Can you write Python code for it?

import pandas as pd

df = pd.read_csv('../data/SliceQuality.csv', header=0)
df.head(5) # return the first 5 rows


I am working on the file "MachineLearning/DecisionTree.ipynb".
The first task is:

Decission Tree classifier

Can you write Python code for it?

from sklearn import tree
clf = tree.DecisionTreeClassifier()

clf = clf.fit(X,Y) # so easy!


I am working on the file "MachineLearning/DecisionTree.ipynb".
The first task is:

Display decission tree

Can you write Python code for it?

from IPython.display import Image
from sklearn.externals.six import StringIO
import pydot

dot_data = StringIO()
tree.export_graphviz(clf, out_file = dot_data, feature_names = features)
graph = pydot.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())


I am working on the file "MachineLearning/DecisionTree.ipynb".
The first task is:

Ensemble learning using random forest

We use a random forest of 10 decission trees to predict
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=10)
clf = clf.fit(X,Y)

# predict


I am working on the file "MachineLearning/DecisionTree.ipynb".
The first task is:
Let's parse everything in numbers
Can you write Python code for it?

myEducation = {'BS':0, 'MS':1, 'PhD':2}
df['Level of Education'] = df['Level of Education'].map(myEducation)
df.head(5)

myOutPut = {'Y':1, 'N':0}
for key in ['Employed?', 'Top-tier school', 'Interned', 'Hired']:
    df[key] = df[key].map(myOutPut)
df.head(5)

# separates features and target columns
features = list(df.columns[:6])
features

df['Years Experience']

X = df[features]
Y = df['Hired']


I am working on the file "MachineLearning/K-means_clustering.ipynb".
The first task is:
<H2> Normalizing the data</H2>
To account for the differences mean values of the data, we normalize the vector
Can you write Python code for it?

from sklearn.preprocessing import scale

x = np.random.normal(loc=10, scale=2, size=10)
print('[Data]-> Mean = %2.4f, StDev = %2.4f'% (x.mean(), x.std()))
y = scale(x)
print('[Norm]-> Mean = %2.4f, StDev = %2.4f'% (y.mean(), y.std()) )

mymodel.fit(scale(data))
plt.scatter(data[:,0], data[:,1], c=mymodel.labels_.astype(float));

mymodel.score(data) #???


I am working on the file "MachineLearning/KMC.ipynb".
The first task is:
Consider the following data set consisting of the scores of two variables on each of 17 experiments
Can you write Python code for it?

x = np.loadtxt("data.txt", comments='//')
x.shape
print(x.shape)

# Plot 2 measurements
#for i in x:
#    plt.plot(i[0],i[1], 'ko');
plt.scatter(x[:,0], x[:,1], color='black')
plt.xlim(-0.5, 10);
plt.ylim(-0.5, 10);
plt.xlabel('Measurement 1');
plt.ylabel('Measurement 2');


I am working on the file "MachineLearning/KMC.ipynb".
The first task is:
This data set is to be grouped into two clusters.
As a first step in finding a sensible initial partition, let the values of the measurements 1 and 2 of the two individuals
Can you write Python code for it?

centroid1 = x[0] # first experiment
centroid2 = x[3] # fourth experiment
print(centroid1, centroid2)

# calculate Euclidean distances from centroid 1
sample1= list()
for i, obs in enumerate(x):
    dist = distance.euclidean(obs, centroid1)
    sample1.append(dist)
    print("%2d -> %6f" %(i, dist))

# calculate Euclidean distances from centroid 2
sample2 = list()
for i, obs in enumerate(x):
    dist = distance.euclidean(obs, centroid2)
    sample2.append(dist)
    print("%2d -> %6f" %(i, dist))

group1 = x[np.array(sample1)<=np.array(sample2)]
print(group1)

group2 = x[np.array(sample2)<np.array(sample1)]
print(group2)

# Replot first categorization
# Plot 2 measurements
for i in group1:
    plt.plot(i[0],i[1], 'o',color='magenta');
for i in group2:
    plt.plot(i[0],i[1], 'o', color='cyan');
plt.plot(centroid1[0], centroid1[1], '*', color='magenta', ms=12)
plt.plot(centroid2[0], centroid2[1], '*', color='cyan', ms=12)
plt.xlim(-0.5, 10);
plt.ylim(-0.5, 10);
plt.xlabel('Measurement 1');
plt.ylabel('Measurement 2');

# recalculate centroids
centroid1 = np.average(group1, axis=0)
centroid2 = np.average(group2, axis=0)
print(centroid1, centroid2)

# calculate Euclidean distances from new centroid 1
sample1= list()
for i, obs in enumerate(x):
    dist = distance.euclidean(obs, centroid1)
    sample1.append(dist)
    print("%2d -> %6f" %(i, dist))

sample2= list()
for i, obs in enumerate(x):
    dist = distance.euclidean(obs, centroid2)
    sample2.append(dist)
    print("%2d -> %6f" %(i, dist))

# reasign the groups
group1 = x[np.array(sample1)<=np.array(sample2)]
print(group1)

group2 = x[np.array(sample2)<np.array(sample1)]
print(group2)

# Replot first categorization with new centrodis
# Plot 2 measurements
for i in group1:
    plt.plot(i[0],i[1], 'o', color='magenta');
for i in group2:
    plt.plot(i[0],i[1], 'o', color='cyan');
plt.plot(centroid1[0], centroid1[1], '*', color='magenta', ms=12)
plt.plot(centroid2[0], centroid2[1], '*', color='cyan', ms=12)
plt.xlim(-0.5, 10);
plt.ylim(-0.5,...


I am working on the file "MachineLearning/NaiveBayesanClassifier.ipynb".
The first task is:
We now test with the classifier with the training data
Can you write Python code for it?

def predict(idx):
    if myclassifier.predict( X[idx]):
        print('Cell %2d is GAD67  mice'%idx)
    else:
        print('Cell %2d is CB57BL mice'%idx)

# test with training data (similar to myclassifier.score(X,Y) )
for i in range(df.shape[0]):
    predict(i)


I am working on the file "MachineLearning/NaiveBayesanClassifier.ipynb".
The first task is:
We use pandas to split up the matrix into the feature vectors we're interested in. We will also to convert textual category data (Strain, Gender) into an ordinal number that we can work with.
Can you write Python code for it?

pd.Categorical(df.Strain).codes # CB57BL is zero, GAD67 is one

df['Gender'] = pd.Categorical(df.Gender).codes
df['Strain'] = pd.Categorical(df.Strain).codes
df.head()

df.shape # as with NumPy the number of rows first

df.iloc[[0]].values[0] # get a row as NumPy array

# create X and Y
Y = df['Strain'].values

del df['Strain'] # remove Strain
X = [ df.iloc[[i]].values[0] for i in range(df.shape[0]) ]
len(X)==len(Y)

X[0] # data from CB57BL


I am working on the file "MachineLearning/PCA.ipynb".
The first task is:

Plot everything

Can you write Python code for it?

fig = plt.figure(figsize = (4,4))
ax = fig.add_subplot(111)
xlabel = 'Component 1 (%2.2f %% $\sigma^2$)'%var1
ylabel = 'Component 2 (%2.2f %% $\sigma^2$)'%var2
ax.set_xlabel(xlabel, fontsize = 12)
ax.set_ylabel(ylabel, fontsize = 12)
ax.set_title('Two component analysis', fontsize = 15)
mytargets = np.unique(df['target'].values).tolist()
colors = ['r', 'g', 'b']
for target, color in zip(mytargets,colors):
    mytarget = finalDf['target'] == target
    ax.scatter(finalDf.loc[mytarget, 'PC1']
               , finalDf.loc[mytarget, 'PC2']
               , c = color
               , s = 10)
ax.legend(mytargets)
#ax.grid()


I am working on the file "MachineLearning/PCA.ipynb".
The first task is:
<H2>Principal component analysis into two dimensions </H2>
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA(n_components = 2)
principalComponents = pca.fit_transform(stdx)
pcDf = pd.DataFrame(data = principalComponents, columns =['PC1', 'PC2'])
finalDf = pd.concat([pcDf, df['target']], axis=1)
finalDf.head()

var1, var2 = pca.explained_variance_ratio_
print('The first component contains %2.4f %% of the variance'%var1)
print('The second component containts %2.4f %% of the variance'%var2)
print('Total variance explained %2.4f %% '%(var1+var2))


I am working on the file "MachineLearning/PCA_new.ipynb".
The first task is:
Gaussian Distribution
The function that describes a Gaussian or a normal distribution is:
\begin{equation}
\mathcal{N}(x;\mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp{\frac{(x-\mu)^2}{2\sigma^2}}
\end{equation}
The independent variable x is a random variable X, two parameters mean $\mu$ and variance $\sigma^2$, which is simply the square of the standard deviation. The man of a Gaussian distribution shifts the center of the Gaussian, whereas the standard deviation is a measure of the spread of the Gaussian.
The probability density function (pdf) tell us the probability of observing an input $x$ given a normal distribution. Because it is a probability distribution, all values are negative and their sum is one.
Can you write Python code for it?

mu = 0
sigma = 1 # std
rv = norm(loc = mu, scale = sigma)

x = np.linspace(-4,15, 200)

plt.plot(x, rv.pdf(x), label=f'$\mu$ = {mu} $\sigma^2$ = {sigma**2:}')
plt.hist(rv.rvs(10000), density=True, alpha=0.5);

mu = 6
sigma = 2.
rv = norm(loc = mu, scale = sigma)

plt.plot(x, rv.pdf(x), label=f'$\mu$ = {mu} $\sigma^2$ = {sigma**2:}')
plt.hist(rv.rvs(10000), density=True, alpha=0.5);

plt.xlabel('x'); plt.ylabel('PDF')
plt.legend()


I am working on the file "MachineLearning/PCA_new.ipynb".
The first task is:
Linear transformation
We can transform the covariance matrix with
Can you write Python code for it?


I am working on the file "MachineLearning/PCA_new.ipynb".
The first task is:
The covariance is always measured between two dimensions. If we have datasets with more than two dimensions
there is more than one covariance to be calculated, and we can use a matrix.
The covariance can be calculated as entries in a covariance matrix, which is a square matrix. The diagonal of the
matrix are the variances of the random variables, and the other entries the covariances.
\begin{equation}
\Sigma = \begin{pmatrix} \text{var}(x) & \text{cov}(x,y) \ \text{cov}(y,x) &\text{var}(y)\end{pmatrix}
\end{equation}
Can you write Python code for it?

mycov(x,y) #x_var = 0.99, y_var = 3.64 (almost 4)


I am working on the file "MachineLearning/PCA_new.ipynb".
The first task is:
The covariance is ultimately with the correlation coefficient. The correlation coefficient is the normlized
version of the covariance, and ragnes between -1 and 1.
\begin{align}
\rho = \frac{\text{cov}(x_1,x_2)}{\sqrt{\sigma_1^2 \sigma_2^2}}.
\end{align}
We can use the correlation coefficient to calculate a new covariance matrix.
Can you write Python code for it?

C = np.cov(X.T)
X = np.random.multivariate_normal(mean = [0, 0], cov = C, size = 700)

plt.figure(figsize=(4,4))

plt.scatter(X[:,0], X[:,1], s=3, color='k')
plt.xticks(np.arange(-4,15, 4))
plt.yticks(np.arange(-4,15, 4))
plt.xlabel('x'), plt.ylabel('y', rotation = 0)
plt.grid('on')

rho = 0.8
var1, var2 = 2, 6
cov = rho*np.sqrt(var1*var2)
C = np.array([[var1, cov],[cov, var2]])
X = np.random.multivariate_normal(mean = [0, 0], cov = C, size = 700)

plt.figure(figsize=(4,4))

plt.scatter(X[:,0], X[:,1], s=3, color='k')
plt.xticks(np.arange(-4,15, 4))
plt.yticks(np.arange(-4,15, 4))
plt.xlabel('x'), plt.ylabel('y', rotation = 0)
plt.grid('on')


I am working on the file "MachineLearning/PCA_new.ipynb".
The first task is:
The covariance matrix can be computed from a data set with zero mean with:
\begin{equation}
C = \frac{XX^T}{n-1}
\end{equation}
If X is the column matrix of variables
Can you write Python code for it?

X.shape # matrix of 500 observations and 2 dimensions (features)

np.matmul(X.T,X)/(len(x)-1) # didn't substract mean!

F= X - np.mean(X, axis=0)

np.matmul(F.T,F)/(len(x)-1) # now mean substracted!

np.cov(x,y) # numpy substracts the mean

np.cov(X.T)


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
Boosting
Can you write Python code for it?

ab_reg = ensemble.AdaBoostRegressor(base_estimator=None,  # default is DT
                                    n_estimators=50,  # number to try before stopping
                                    learning_rate=1.0,  # decrease influence of each additional estimator
                                    loss='linear')  # also ‘square’, ‘exponential’

model = ab_reg.fit(train, test)
print(model.score(train, test))


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
Discrete Target Data (classification problem)
Logistic Regression
For this type of problem the target data takes on discretet values, for example the target be 'yes' or 'no' for a disease, or equivalently '1' or '0'
Can you write Python code for it?

n_samples = 100
train = np.random.normal(size=n_samples) # make random data set
target = (train > 0).astype(np.float) # make half the values true/1/yes
train[train > 0] *= 2
train += .3 * np.random.normal(size=n_samples) # create abiguity near the transition

train = np.reshape(train, [n_samples, 1])
target = np.reshape(target, [n_samples, 1])

plt.plot(train, target, 'o')
plt.ylim([-.1,1.1])
plt.xlabel('Predictor')
plt.ylabel('Target')


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
However, using k = 1 is overfitting the data. We will try using 15 now.  It will increase the number of incorrect train data that are mislabeled, but should increase the test error rate
Can you write Python code for it?

from sklearn import neighbors
nbrs = neighbors.KNeighborsClassifier(n_neighbors=15)
nbrs_fit = nbrs.fit(train, target)

nbrs_pred = nbrs_fit.predict(train)

print((nbrs_pred!=target).sum()/100.)

pred = nbrs_fit.predict(points)

for i in range(num_classes):
    plt.plot(points[pred==i][:,0],points[pred==i][:,1],'o')


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
Linear Regression
Linear regression fits the data to the follow equation
$\hat{y}i=\beta_0+\sum_j^p\beta_jx{i,j}=\beta_0+\mathbf{x}^T\beta$
where $i=1,...,n$ and $p$ is the number of predictors
The error produced by this equation is
$\epsilon_i=\hat{y_i}-y_i$
For linear regression, the goal is the sum of the errors by using least squares
$\min\left|\sum_i^n\epsilon_i^2\right|$
Can you write Python code for it?

from sklearn import linear_model
lin_reg = linear_model.LinearRegression()

model = lin_reg.fit(train, test)
print(model.score(train, test))

plt.plot(model.predict(train), test, 'ro')
plt.plot([0,50],[0,50], 'k', linewidth = 2)
plt.xlabel('Predicted Value')
plt.ylabel('True Value')
plt.axis('square')
plt.ylim([-10, 55])
plt.xlim([-10, 55])


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
Now we can transform the data to use in the models
Can you write Python code for it?


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
Now we can use this in our models.  We now only have 2 features, thus building the models will be far simplier.
Can you write Python code for it?

lin_reg = linear_model.LinearRegression()

model = lin_reg.fit(new_boston, test)
print(model.score(new_boston, test))

plt.plot(model.predict(new_boston), test, 'ro')
plt.plot([0,50],[0,50], 'k', linewidth = 2)
plt.xlabel('Predicted Value')
plt.ylabel('True Value')
plt.axis('square')
plt.ylim([-10, 55])
plt.xlim([-10, 55])


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
Polynomial Fits
While linear regression is nice, there is often a need for higher order parameters and cross terms, such as $x^2$ or $x_1x_2$.  In this case the following programs can be used to add the polynomial terms.
Can you write Python code for it?

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

model = make_pipeline(PolynomialFeatures(3), ridge_reg)
model.fit(train, test)
print(model.score(train, test))

plt.plot(model.predict(train), test, 'ro')
plt.plot([0,50],[0,50], 'k', linewidth = 2)
plt.xlabel('Predicted Value')
plt.ylabel('True Value')
plt.axis('square')
plt.ylim([-10, 55])
plt.xlim([-10, 55])


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
The data above shows that the predictor can estimate the target.  However, if we fit the data with a linear fit, the solution is rather ambigous.
Can you write Python code for it?

lin_reg = linear_model.LinearRegression()

model = lin_reg.fit(train, target)

plt.plot(train, target, 'o')
plt.plot(train, model.coef_*train + model.intercept_, 'r-', linewidth=2)
plt.ylim([-.1,1.1])
plt.xlabel('Predictor')
plt.ylabel('Target')


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
The nice about this method is that it is fairly simple to excute and understand.  However it is only accurate when classes are well defined locally, which may not always be the case.
K-nearest neighbors
The next method that could be used is k nearest neighbors, or knn.  This method can also work very well for continues data, but I think it is easier to visualize and explain as a classification model.
This method works on the idea that if you look to the k train data points nearest to the point x, then you can determine the class based on the most common class of the nearest points.  Often, euclidean distance is used for this, but other methods of deciding the nearest neighbor is possible, such as weighting the strength of nearest neighbor based on its distance.
If k == 1, this is equivalent to using a step function.
Can you write Python code for it?

from sklearn import neighbors
nbrs = neighbors.KNeighborsClassifier(n_neighbors=1)
nbrs_fit = nbrs.fit(train, target)

nbrs_pred = nbrs_fit.predict(train)

print((nbrs_pred!=target).sum()/100.)

for group in range(num_classes):
    plt.plot(train[target==group][:,0],train[target==group][:,1], 'o')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')

pred = nbrs_fit.predict(points)

for i in range(num_classes):
    plt.plot(points[pred==i][:,0],points[pred==i][:,1],'o')


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
The previous code uses 5-fold cross-validation to predict the values for the boston data with a linear model.
PCA
Sometimes, there are too many features to describe the target, maybe some features are duplicates or combinations of other features, or maybe some features just dont correlate well with the target.  In these cases it is good to reduce the feature space, because the lower the dimension the easier it is to create a model and the easier it is for us to comprehend with our simple brains.
A common way to reduce the feature space is to use principle component analysis.  PCA computes the directions that best describe the data.  Here is a 2-d exmaple
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/f/f5/GaussianScatterPCA.svg/720px-GaussianScatterPCA.svg.png">
This shows that pca will change from x,y to vectors that better describe the data.  PCA in sklearn is determined by SVD and can be used with the following commands
Can you write Python code for it?

from sklearn.decomposition import PCA
pca = PCA(n_components=None) #returns the same number of features as the original set
pca_fit = pca.fit(boston.data)
print(pca.explained_variance_ratio_)


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
The printed values show the significance of the principle compnent.  Thus, we can reduce the feature space by say only keeping the principle components that explain more than 90% of the data, which means we would only keep the first two principle compenents here.
Can you write Python code for it?

pca = PCA(n_components=3) #returns 3 componenets
pca_fit = pca.fit(boston.data)
print(pca.explained_variance_ratio_)


I am working on the file "MachineLearning/scikit-learn/sklearn_overview.ipynb".
The first task is:
This used the gaussian kernel.
Organizing Your Data
Train/test data sets
A crucial part of machine learning is obtaining a training data set to build the model and a testing data set to verify the success of the model.  The purpose of a new test data set that was not used in building the model is that if the model is tested against the training data, the model may be overfit.  For example, if the model fits the train data perfectly, there is a good chance that the predictions on new data are poor.
<img src="https://upload.wikimedia.org/wikipedia/commons/6/68/Overfitted_Data.png">
The above figure shows how a high order polynomial can overfit a simple line.  Thus, it is important to split the data into train and test data, and a strong model will have a low test data error rate.
To do this with sklearn, there is a package to split data sets into train and test
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

boston = load_boston()

X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target,
                                                    train_size=0.8, test_size=0.2)


I am working on the file "MachineLearningLibrary/Cluster/kmeans_example.ipynb".
The first task is:
Cluster via KMeans
Can you write Python code for it?

trial_clusters = [2, 3, 4, 6, 8, 10, 15, 20]
inertia_lst = []

for i in trial_clusters:
    kmeans = KMeans(n_clusters=i, init='kmeans++', max_iter=100)
    kmeans.fit(X)
    inertia_lst.append(kmeans.inertia)

plt.figure()
plt.plot(trial_clusters, inertia_lst, 'o--')
plt.xlim(min(trial_clusters) - 1, max(trial_clusters) + 1)
plt.ylabel('inertia')
plt.xlabel('n_clusters')


I am working on the file "MachineLearningLibrary/NeuralNetworks/CNN_MNIST_Keras_Tensorflow.ipynb".
The first task is:
Build CNN Model...
Can you write Python code for it?

from keras.models import Sequential
from keras.layers import Conv2D, MaxPool2D, Dense, Dropout, Flatten

img_shape = X_train[0].shape
print(img_shape)

model = Sequential()

# Convolutional Section...
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=img_shape))
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPool2D((2, 2)))
model.add(Dropout(rate=0.25))
model.add(Conv2D(64, (3, 3), activation='relu', input_shape=img_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPool2D((2, 2)))
model.add(Dropout(rate=0.25))

# Fully Connected Section...
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(rate=0.25))
model.add(Dense(10, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.summary()


I am working on the file "MachineLearningLibrary/NeuralNetworks/CNN_MNIST_Keras_Tensorflow.ipynb".
The first task is:
Image Processing...
Reshape (28, 28) to (28, 28, 1)... and Normalized Image Data... from uint8 to float32 over the range [0,1]
Can you write Python code for it?

X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], X_train.shape[2], 1).astype('float32') / 255.
X_test = X_test.reshape(X_test.shape[0], X_test.shape[1], X_test.shape[2], 1).astype('float32') / 255.

print(X_train.shape)


I am working on the file "MachineLearningLibrary/NeuralNetworks/CNN_MNIST_Keras_Tensorflow.ipynb".
The first task is:
Import the MNIST dataset using the keras api
Can you write Python code for it?

from keras.datasets import mnist
# load data...
(X_train, y_train), (X_test, y_test) = mnist.load_data()
# check dimensions...
print('Train: ', X_train.shape, y_train.shape)
print('Test: ', X_test.shape, y_test.shape)


I am working on the file "MachineLearningLibrary/NeuralNetworks/CNN_MNIST_Keras_Tensorflow.ipynb".
The first task is:
Looks like we have 60k images of 28, 28 pixels.  These images are single-channel, i.e. black and white... If these were color images, then we would see dimensions of (60000, 28, 28, 3)... 3 channels for Red-Green-Blue (RGB) or Blue-Green-Red (BGR), depending on the order of the color channels...
Show an image and check data...
Can you write Python code for it?

# select a number [0, 60000)...
idx = 1000

# plot image...
plt.figure()
plt.title('Number: %s'%y_train[idx])
plt.imshow(X_train[idx], cmap='gray')


I am working on the file "MachineLearningLibrary/NeuralNetworks/CNN_MNIST_Keras_Tensorflow.ipynb".
The first task is:
The code below will create an OpenCV popup window... the window can be closed using the 'esc' key... and we can draw in the window by holding the left-mouse button and moving the mouse within the window...
Can you write Python code for it?

def record_location(event, x, y, flags, param):
    '''callback function that draws a circle at the point x, y...'''
    if flags == cv2.EVENT_FLAG_LBUTTON and event == cv2.EVENT_MOUSEMOVE:
        cv2.circle(img, (x,y), 10, (255, 255, 255), -1)

img = np.zeros((256, 256, 3), np.uint8)

cv2.namedWindow('image')
cv2.setMouseCallback('image', record_location)
while(1):
    cv2.imshow('image',img)
    k = cv2.waitKey(1) & 0xFF
    if k == ord('m'):
        mode = not mode
    elif k == 27:
        break
cv2.destroyAllWindows()

# copy one color channel and normalize values...
_img = img[:,:,0] / 255.0
# resize image to (28, 28)
_img = cv2.resize(_img, (28, 28), interpolation=cv2.INTER_AREA).reshape(1, 28, 28, 1)

p = model.predict(_img)
print(p)
plt.figure()
plt.title('Guess: %s' %p.argmax())
plt.imshow(_img[0][:,:,0], cmap='gray')


I am working on the file "MachineLearningLibrary/NeuralNetworks/CNN_MNIST_Keras_Tensorflow.ipynb".
The first task is:
Train model...
Can you write Python code for it?

n_epochs = 2
model.fit(X_train, y_train, batch_size=32, epochs=n_epochs, verbose=True)

loss, accuracy = model.evaluate(X_test, y_test, batch_size=32)
print('Test Accuracy: ', accuracy)

# save model for retrieval at later date...
model.save('./MNIST_CNN')


I am working on the file "MachineLearningLibrary/NeuralNetworks/CNN_MNIST_Keras_Tensorflow.ipynb".
The first task is:
now we have explicity created a one-channel dataset... and normalized it between [0, 1]... alternatively, you might normalize it more correctly as Gaussian distributed about zero with a variance of one... this would help with training but for this example, as you'll see, it doesn't really matter...
Encode numbers from 0-9 into 10-dimensional vectors... this is called one-hot encoding... i.e. 0 -> [1, 0, 0, ..., 0] and 1 -> [0, 1, 0, 0, ..., 0], etc.
Can you write Python code for it?

# import to_categorial function that does the one-hot encoding...
from keras.utils import to_categorical

# encode both training and testing data...
y_train = to_categorical(y_train, 10)
y_test = to_categorical(y_test, 10)

y_train[0]


I am working on the file "Machine_Learning_in_Python.ipynb".
The first task is:
And each one of these data points has a label, 0 through 9.
Machine learning time
Can you write Python code for it?

half_length = len(data_set.data) // 2
train_set = {
    'data': data_set.data[:half_length],
    'target': data_set.target[:half_length],
}
test_set = {
    'data': data_set.data[half_length:],
    'target': data_set.target[half_length:]
}

from sklearn.ensemble import RandomForestClassifier

classifier = RandomForestClassifier()
classifier.fit(train_set['data'], train_set['target'])


I am working on the file "Machine_Learning_in_Python.ipynb".
The first task is:
Bad classifier
Can you write Python code for it?

import numpy as np

# Totally random data, nothing can be learned here.
rows = 1000
features = 64
data = np.random.random((rows, features))
labels = np.random.randint(0, 2, rows)

from sklearn.ensemble import RandomForestClassifier

# Train it on all the data, such a bad idea!
bad_classifier = RandomForestClassifier()
bad_classifier.fit(data, labels)

probabilities = bad_classifier.predict_proba(data)
p, r, _ = metrics.precision_recall_curve(labels, probabilities[:,1])
plt.plot(r, p)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.show()


I am working on the file "Machine_Learning_in_Python.ipynb".
The first task is:
Let's poke around and see what is in the data set.
Can you write Python code for it?


I am working on the file "Machine_Learning_in_Python.ipynb".
The first task is:
The right way to do it
Can you write Python code for it?

# Divide our data in half
half_length = len(data) // 2
train_data = data[:half_length]
train_labels = labels[:half_length]
test_data = data[half_length:]
test_labels = labels[half_length:]

# train on half of it
good_classifier = RandomForestClassifier()
good_classifier.fit(train_data, train_labels)

# evaluate
probabilities = good_classifier.predict_proba(test_data)
p, r, _ = metrics.precision_recall_curve(test_labels, probabilities[:,1])
plt.plot(r, p)
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.show()


I am working on the file "Machine_Learning_in_Python.ipynb".
The first task is:
Well, that is a bit hard to grok. Let's see if we can get a better view.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (4.0, 4.0)

def show_image(image_data):
    plt.imshow(image_data, cmap=plt.cm.gray_r, interpolation='nearest')
    plt.show()

show_image(data_set.images[0])
data_set.images[0]


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
And the test set contains 10,000 images labels:
Can you write Python code for it?


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
As the model trains, the loss and accuracy metrics are displayed. This model reaches an accuracy of about 0.91 (or 91%) on the training data.
Save trained weights
As the model is trained fully, save the model weights to a file for future use.
Can you write Python code for it?


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Basic classification: Classify images of clothing

    View on TensorFlow.org

    Run in Google Colab

    View source on GitHub

    Download notebook

This guide trains a neural network model to classify images of clothing, like sneakers and shirts. It's okay if you don't understand all the details; this is a fast-paced overview of a complete TensorFlow program with the details explained as you go.
This guide uses tf.keras, a high-level API to build and train models in TensorFlow.
Can you write Python code for it?

import tensorflow as tf
print('Using tensorflow version: {} ...'.format(tf.__version__))
print('Visible devices for tensorflow: {} ...'.format(tf.config.list_physical_devices()))

from __future__ import absolute_import, division, print_function, unicode_literals

# tf.keras
import tensorflow.keras as keras

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt

print('Using keras version: {} ...'.format(keras.__version__))

WEIGHTS_DIR = './weights/'
WEIGHTS_FILENAME = 'fashionMNIST_classification.h5'
WEIGHTS_FILE ='{}/{}'.format(WEIGHTS_DIR, WEIGHTS_FILENAME)


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Build the model
Building the neural network requires configuring the layers of the model, then compiling the model.
Set up the layers
The basic building block of a neural network is the layer. Layers extract representations from the data fed into them. Hopefully, these representations are meaningful for the problem at hand.
Most of deep learning consists of chaining together simple layers. Most layers, such as tf.keras.layers.Dense, have parameters that are learned during training.
Can you write Python code for it?

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10)
])


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Each label is an integer between 0 and 9:
Can you write Python code for it?


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Explore the data
Let's explore the format of the dataset before training the model. The following shows there are 60,000 images in the training set, with each image represented as 28 x 28 pixels:
Can you write Python code for it?


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Feed the model
To start training,  call the model.fit method—so called because it "fits" the model to the training data:
Can you write Python code for it?

model.fit(train_images, train_labels, epochs=10, batch_size=256)


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Graph this to look at the full set of 10 class predictions.
Can you write Python code for it?

def plot_image(i, predictions_array, true_label, img):
  predictions_array, true_label, img = predictions_array, true_label[i], img[i]
  plt.grid(False)
  plt.xticks([])
  plt.yticks([])

  plt.imshow(img, cmap=plt.cm.binary)

  predicted_label = np.argmax(predictions_array)
  if predicted_label == true_label:
    color = 'blue'
  else:
    color = 'red'

  plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
                                100*np.max(predictions_array),
                                class_names[true_label]),
                                color=color)

def plot_value_array(i, predictions_array, true_label):
  predictions_array, true_label = predictions_array, true_label[i]
  plt.grid(False)
  plt.xticks(range(10))
  plt.yticks([])
  thisplot = plt.bar(range(10), predictions_array, color="#777777")
  plt.ylim([0, 1])
  predicted_label = np.argmax(predictions_array)

  thisplot[predicted_label].set_color('red')
  thisplot[true_label].set_color('blue')


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Import the Fashion MNIST dataset
This guide uses the Fashion MNIST dataset which contains 70,000 grayscale images in 10 categories. The images show individual articles of clothing at low resolution (28 by 28 pixels), as seen here:

    Figure 1. Fashion-MNIST samples (by Zalando, MIT License).

Fashion MNIST is intended as a drop-in replacement for the classic MNIST dataset—often used as the "Hello, World" of machine learning programs for computer vision. The MNIST dataset contains images of handwritten digits (0, 1, 2, etc.) in a format identical to that of the articles of clothing you'll use here.
This guide uses Fashion MNIST for variety, and because it's a slightly more challenging problem than regular MNIST. Both datasets are relatively small and are used to verify that an algorithm works as expected. They're good starting points to test and debug code.
Here, 60,000 images are used to train the network and 10,000 images to evaluate how accurately the network learned to classify images. You can access the Fashion MNIST directly from TensorFlow. Import and load the Fashion MNIST data directly from TensorFlow:
Can you write Python code for it?

(train_images, train_labels), (test_images, test_labels) = keras.datasets.fashion_mnist.load_data()


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
It turns out that the accuracy on the test dataset is a little less than the accuracy on the training dataset. This gap between training accuracy and test accuracy represents overfitting. Overfitting is when a machine learning model performs worse on new, previously unseen inputs than on the training data. An overfitted model "memorizes" the training data—with less accuracy on testing data. For more information, see the following:
*   Demonstrate overfitting
*   Strategies to prevent overfitting
Make predictions
With the model trained, you can use it to make predictions about some images.
The model's linear outputs, logits. Attach a softmax layer to convert the logits to probabilities, which are easier to interpret.
Can you write Python code for it?

probability_model = tf.keras.Sequential([model,
                                         tf.keras.layers.Softmax()])

predictions = probability_model.predict(test_images)


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Likewise, there are 60,000 labels in the training set:
Can you write Python code for it?


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Loading the dataset returns four NumPy arrays:

The train_images and train_labels arrays are the training set—the data the model uses to learn.
The model is tested against the test set, the test_images, and test_labels arrays.

The images are 28x28 NumPy arrays, with pixel values ranging from 0 to 255. The labels are an array of integers, ranging from 0 to 9. These correspond to the class of clothing the image represents:

    Label
    Class

    0
    T-shirt/top

    1
    Trouser

    2
    Pullover

    3
    Dress

    4
    Coat

    5
    Sandal

    6
    Shirt

    7
    Sneaker

    8
    Bag

    9
    Ankle boot

Each image is mapped to a single label. Since the class names are not included with the dataset, store them here to use later when plotting the images:
Can you write Python code for it?

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Now predict the correct label for this image:
Can you write Python code for it?

predictions_single = probability_model.predict(img)

print(predictions_single)

plot_value_array(1, predictions_single[0], test_labels)
_ = plt.xticks(range(10), class_names, rotation=45)


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Preprocess the data
The data must be preprocessed before training the network. If you inspect the first image in the training set, you will see that the pixel values fall in the range of 0 to 255:
Can you write Python code for it?

plt.figure()
plt.imshow(train_images[0])
plt.colorbar()
plt.grid(False)
plt.show()


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
So, the model is most confident that this image is an ankle boot, or class_names[9]. Examining the test label shows that this classification is correct:
Can you write Python code for it?


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
The first layer in this network, tf.keras.layers.Flatten, transforms the format of the images from a two-dimensional array (of 28 by 28 pixels) to a one-dimensional array (of 28 * 28 = 784 pixels). Think of this layer as unstacking rows of pixels in the image and lining them up. This layer has no parameters to learn; it only reformats the data.
After the pixels are flattened, the network consists of a sequence of two tf.keras.layers.Dense layers. These are densely connected, or fully connected, neural layers. The first Dense layer has 128 nodes (or neurons). The second (and last) layer is a 10-node softmax layer that returns an array of 10 probability scores that sum to 1. Each node contains a score that indicates the probability that the current image belongs to one of the 10 classes.
Compile the model
Before the model is ready for training, it needs a few more settings. These are added during the model's compile step:

Loss function —This measures how accurate the model is during training. You want to minimize this function to "steer" the model in the right direction.
Optimizer —This is how the model is updated based on the data it sees and its loss function.
Metrics —Used to monitor the training and testing steps. The following example uses accuracy, the fraction of the images that are correctly classified.
Can you write Python code for it?

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
There are 10,000 images in the test set. Again, each image is represented as 28 x 28 pixels:
Can you write Python code for it?


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Train the model
Training the neural network model requires the following steps:

Feed the training data to the model. In this example, the training data is in the train_images and train_labels arrays.
The model learns to associate images and labels.
You ask the model to make predictions about a test set—in this example, the test_images array.
Verify that the predictions match the labels from the test_labels array.

Download and load pre-trained model weights
Can you write Python code for it?

import os
if not os.path.exists(WEIGHTS_FILE):
  ! mkdir {WEIGHTS_DIR}
  ! wget https://github.com/rahulremanan/python_tutorial/blob/master/Machine_Vision/00_Install_Instructions/notebook/weights/fashionMNIST_classification.h5?raw=true -O {WEIGHTS_FILE}

if os.path.exists(WEIGHTS_FILE):
  model.load_weights(WEIGHTS_FILE)
  print ('Loaded pre-trained weights ...')


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
Visually check prediction on a single image
And the model predicts a label as expected.
Can you write Python code for it?

plt.figure(figsize=(6,3))
plt.subplot(1,2,1)
plot_image(i, predictions_single, test_labels, test_images)
plt.subplot(1,2,2)
plot_value_array(i, predictions[i],  test_labels)
plt.show()


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
keras.Model.predict returns a list of lists—one list for each image in the batch of data. Grab the predictions for our (only) image in the batch:
Can you write Python code for it?


I am working on the file "Machine_Vision/00_Install_Instructions/notebook/Tensorflow_fashionMNIST_classification.ipynb".
The first task is:
tf.keras models are optimized to make predictions on a batch, or collection, of examples at once. Accordingly, even though you're using a single image, you need to add it to a list:
Can you write Python code for it?

# Add the image to a batch where it's the only member.
img = (np.expand_dims(img,0))

print(img.shape)


I am working on the file "Machine_Vision/02_Object_Prediction/notebook/prediction_cats_dogs.ipynb".
The first task is:
Function to generate predictions
Can you write Python code for it?

def predict(model, img, target_size):
  print ("Running prediction model on the image file ...")
  if img.size != target_size:
    img = img.resize(target_size)

  _x_ = image.img_to_array(img)
  _x_ = np.expand_dims(_x_, axis=0)
  _x_ = preprocess_input(_x_)
  preds = model.predict(_x_)
  probabilities = model.predict(_x_, batch_size=1).flatten()
  prediction = labels[np.argmax(probabilities)]
  return preds[0], prediction


I am working on the file "Machine_Vision/02_Object_Prediction/notebook/prediction_cats_dogs.ipynb".
The first task is:
Function to plot prediction accuracy
Can you write Python code for it?

def plot_preds(image, preds, labels, timestr):
  output_loc = args.output_dir[0]
  output_file_preds = os.path.join(output_loc+"//preds_out_"+timestr+".png")
  fig = plt.figure()
  plt.axis('on')
  labels = labels
  plt.barh([0, 1], preds, alpha=0.5)
  plt.yticks([0, 1], labels)
  plt.xlabel('Probability')
  plt.xlim(0,1.01)
  plt.tight_layout()
  fig.savefig(output_file_preds, dpi=fig.dpi)


I am working on the file "Machine_Vision/02_Object_Prediction/notebook/prediction_cats_dogs.ipynb".
The first task is:
Generating predictions and plotting classification accuracy
Test image 1
Visualize input image
Can you write Python code for it?

from IPython.display import Image as PyImage
from IPython.core.display import HTML
PyImage(args.image[0])

if args.image is not None:
    img = Image.open(args.image[0])
    preds = predict(model, img, target_size)
    print (preds[1] + "\t" + "\t".join(map(lambda x: "%.2f" % x, preds[0])))
    print (str(preds[1]))
    timestr = generate_timestamp()
    plot_preds(img, preds[0], labels, timestr)

image_path_pred1 = os.path.join('./preds_out_'+timestr+'.png')
PyImage(image_path_pred1)


I am working on the file "Machine_Vision/02_Object_Prediction/notebook/prediction_cats_dogs.ipynb".
The first task is:
Initialize parameters for generating predictions
Can you write Python code for it?

import types
args=types.SimpleNamespace()
args.config_file = ['./trained_cats_dogs.config']
args.weights_file = ['./trained_cats_dogs_epochs30_weights.model']
args.labels_file = ['./trained_labels.json']
args.output_dir = ['./']
args.image = ['./cat_01.jpg']
args.image_url = ['https://github.com/rahulremanan/python_tutorial/raw/master/Machine_Vision/02_Object_Prediction/test_images/dog_01.jpg']


I am working on the file "Machine_Vision/02_Object_Prediction/notebook/prediction_cats_dogs.ipynb".
The first task is:
Load the trained deep learning model and load model weights
Can you write Python code for it?

model, labels = load_prediction_model(args)


I am working on the file "Machine_Vision/02_Object_Prediction/notebook/prediction_cats_dogs.ipynb".
The first task is:
Specify input size for the image to generate predictions
Can you write Python code for it?


I am working on the file "Machine_Vision/02_Object_Prediction/notebook/prediction_cats_dogs.ipynb".
The first task is:
Test image 2
Can you write Python code for it?

PyImage(url = args.image_url[0])

if args.image_url is not None:
    response = requests.get(args.image_url[0])
    img = Image.open(BytesIO(response.content))
    preds = predict(model, img, target_size)
    print (preds[1] + "\t" + "\t".join(map(lambda x: "%.2f" % x, preds[0])))
    print (str(preds[1]))
    timestr = generate_timestamp()
    plot_preds(img, preds[0], labels, timestr)

image_path_pred2 = os.path.join('./preds_out_'+timestr+'.png')
PyImage(image_path_pred2)

def class_activation_map(INPUT_IMG_FILE=None,
                         PRE_PROCESSOR=None,
                         LABEL_DECODER=None,
                         MODEL=None,
                         LABELS=None,
                         IM_WIDTH=299,
                         IM_HEIGHT=299,
                         CONV_LAYER='conv_7b',
                         URL_MODE=False,
                         FILE_MODE=False,
                         EVAL_STEPS=1,
                         HEATMAP_SHAPE=[8,8],
                         BENCHMARK=True):

  if INPUT_IMG_FILE == None:
    print ('No input file specified to generate predictions ...')
    return

  if URL_MODE:
    response = requests.get(INPUT_IMG_FILE)
    img = Image.open(BytesIO(response.content))
    img = img.resize((IM_WIDTH, IM_HEIGHT))
  elif FILE_MODE:
    img = INPUT_IMG_FILE
  else:
    img = image.load_img(INPUT_IMG_FILE, target_size=(IM_WIDTH, IM_HEIGHT))

  x = img

  if not FILE_MODE:
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    if PRE_PROCESSOR !=None:
      preprocess_input = PRE_PROCESSOR
      x = preprocess_input(x)

  model = MODEL
  if model == None:
    print ('No input model specified to generate predictions ...')
    return
  labels = LABELS

  heatmaps = []
  heatmap_sum = np.empty(HEATMAP_SHAPE, float)

  last_conv_layer = model.get_layer(CONV_LAYER)
  feature_size = tensor_featureSizeExtractor(last_conv_layer)

  model_input = model.input
  model_output = model.output
  last_conv_layer_out = last_conv_layer.output
...


I am working on the file "Machine_Vision/02_Object_Prediction/notebook/prediction_cats_dogs.ipynb".
The first task is:
Visualizing the model architecture -- Inception version 3
Can you write Python code for it?

from keras.utils import plot_model
import pydot
import graphviz # apt-get install -y graphviz libgraphviz-dev && pip3 install pydot graphviz
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

output_dir = './'
plot_model(model, to_file= output_dir + '/model_summary_plot.png')
SVG(model_to_dot(model).create(prog='dot', format='svg'))


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_Fashion_MNIST.ipynb".
The first task is:
Bayesian inference
The inference step is repeated over and over again to obtain the model uncertainty associated with each prediction class. Unlike in the regular deep-learning architecture, each inference step returns a different set of probabilities for each class. The final accuracy is calculated as the class-wise mean of all the probabilities. The model uncertainty is numerically represented as the class-wise standard deviation of all the probabilities.
Can you write Python code for it?

with strategy.scope():
  bayesian_model = fashionMNIST_model(x_train,enable_bayesian_inference=True)
  bayesian_model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
                         loss=tf.keras.losses.sparse_categorical_crossentropy,
                         metrics=[tf.keras.metrics.sparse_categorical_accuracy])
bayesian_model.load_weights(WEIGHTS_FILE)

preds=[]
num_bayesian_inference_steps=10
for i in tqdm(range(num_bayesian_inference_steps)):
  preds.append(bayesian_model.predict(x_test[:16].astype(np.float32)))
mean_preds=np.mean(np.asarray(preds),axis=0)
stdev_preds=np.std(np.asarray(preds),axis=0)


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_Fashion_MNIST.ipynb".
The first task is:
Specify variables
Can you write Python code for it?


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_Fashion_MNIST.ipynb".
The first task is:
Training the fashion MNIST Bayesian deep-learning model
Can you write Python code for it?

bayesian_model.fit(x_train.astype(np.float32),y_train.astype(np.float32),
                   epochs=5,
                   steps_per_epoch=60,
                   validation_data=(x_test.astype(np.float32),y_test.astype(np.float32)),
                   validation_freq=1)

bayesian_model.save_weights(WEIGHTS_FILE,overwrite=True)


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_Fashion_MNIST.ipynb".
The first task is:
Using the TPU
To use the TPU for training and inference, first the TPU device needs to be initialized. Then the model has to be built and compiled specifically to use the TPU.
Can you write Python code for it?

tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
print('Running on TPU ', tpu.cluster_spec().as_dict()['worker'])

tf.config.experimental_connect_to_cluster(tpu)
tf.tpu.experimental.initialize_tpu_system(tpu)

strategy = tf.distribute.experimental.TPUStrategy(tpu)
print("REPLICAS: ", strategy.num_replicas_in_sync)

with strategy.scope():
  bayesian_model = fashionMNIST_model(x_train,enable_bayesian_inference=True)
  bayesian_model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),
                         loss=tf.keras.losses.sparse_categorical_crossentropy,
                         metrics=[tf.keras.metrics.sparse_categorical_accuracy])


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_Fashion_MNIST.ipynb".
The first task is:
Visualize predictions
Can you write Python code for it?

def plot_predictions(images,ground_truths,
                     preds_acc,preds_stdev=None,
                     label_names=None,
                     enable_bayesian_inference=True):
  n = images.shape[0]
  nc = int(np.ceil(n / 4))
  f, axes = pyplot.subplots(nc, 4)
  for i in range(nc * 4):
    y = i // 4
    x = i % 4
    axes[x, y].axis('off')

    label = label_names[np.argmax(preds_acc[i])]
    ground_truth=label_names[ground_truths[i]]
    accuracy = np.max(preds_acc[i])
    if enable_bayesian_inference and preds_stdev is not None:
      confidence = preds_stdev[i][np.argmax(preds_acc[i])]

    if i > n:
      continue
    axes[x, y].imshow(images[i])
    if enable_bayesian_inference and preds_stdev is not None:
      axes[x, y].text(0.5,0.5, '\nLabel (Actual): {} ({})'.format(label,ground_truth) +
                               '\nAccuracy: {}, \nUncertainty: {}\n'.format(str(round(accuracy,2)),
                                                                            str(round(confidence,2))),
                      fontsize=10)
    else:
      axes[x, y].text(0.5,0.5, '\nLabel: {}'.format(label) +
                               '\nAccuracy: {} \n'.format(str(round(accuracy,2))),
                      fontsize=10)
    pyplot.gcf().set_size_inches(16,16)

plot_predictions(np.squeeze(x_test[:16]), y_test[:16],
                 mean_preds,stdev_preds,
                 label_names=LABEL_NAMES,
                 enable_bayesian_inference=True)


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Authenticate Kaggle by uploading kaggle.json file
Can you write Python code for it?

from google.colab import files
upload = files.upload()

!mkdir ~/.kaggle

!cp kaggle.json ~/.kaggle/
!chmod 600 ~/.kaggle/kaggle.json


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Create datagenerator
Can you write Python code for it?

class data_generator:

    def create_train(dataset_info, batch_size, shape, augument=True):
        assert shape[2] == 3
        while True:
            random_indexes = np.random.choice(len(dataset_info), batch_size)
            batch_images = np.empty((batch_size, shape[0], shape[1], shape[2]))
            batch_labels = np.zeros((batch_size, 28))
            for i, idx in enumerate(random_indexes):
                image = data_generator.load_image(
                    dataset_info[idx]['path'], shape)
                if augument:
                    image = data_generator.augment(image)
                batch_images[i] = image
                batch_labels[i][dataset_info[idx]['labels']] = 1
            yield batch_images, batch_labels

    def load_image(path, shape):
        R = np.array(Image.open(path+'_red.png'))
        G = np.array(Image.open(path+'_green.png'))
        B = np.array(Image.open(path+'_blue.png'))
        Y = np.array(Image.open(path+'_yellow.png'))

        image = np.stack((
            R/2 + Y/2,
            G/2 + Y/2,
            B),-1)

        image = cv2.resize(image, (shape[0], shape[1]))
        image = np.divide(image, 255)
        return image

    def augment(image):
        augment_img = iaa.Sequential([
            iaa.OneOf([
                iaa.Affine(rotate=0),
                iaa.Affine(rotate=90),
                iaa.Affine(rotate=180),
                iaa.Affine(rotate=270),
                iaa.Fliplr(0.5),
                iaa.Flipud(0.5),
            ])], random_order=True)

        image_aug = augment_img.augment_image(image)
        return image_aug


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Create model
Can you write Python code for it?

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential, load_model
from keras.layers import Activation
from keras.layers import Dropout
from keras.layers import Flatten
from keras.layers import Dense
from keras.layers import Input
from keras.layers import BatchNormalization
from keras.layers import Conv2D
from keras.models import Model
from keras.applications import InceptionResNetV2
from keras.callbacks import ModelCheckpoint
from keras.callbacks import LambdaCallback
from keras.callbacks import Callback
from keras import metrics
from keras.optimizers import Adam
from keras import backend as K
import tensorflow as tf
import keras

def create_model(input_shape, n_out):

    pretrain_model = InceptionResNetV2(
        include_top=False,
        weights='imagenet',
        input_shape=input_shape)

    input_tensor = Input(shape=input_shape)
    bn = BatchNormalization()(input_tensor)
    x = pretrain_model(bn)
    x = Conv2D(128, kernel_size=(1,1), activation='relu')(x)
    x = Flatten()(x)
    x = Dropout(0.5)(x)
    x = Dense(512, activation='relu')(x)
    x = Dropout(0.5)(x)
    output = Dense(n_out, activation='sigmoid')(x)
    model = Model(input_tensor, output)

    return model

def f1(y_true, y_pred):
    tp = K.sum(K.cast(y_true*y_pred, 'float'), axis=0)
    fp = K.sum(K.cast((1-y_true)*y_pred, 'float'), axis=0)
    fn = K.sum(K.cast(y_true*(1-y_pred), 'float'), axis=0)

    p = tp / (tp + fp + K.epsilon())
    r = tp / (tp + fn + K.epsilon())

    f1 = 2*p*r / (p+r+K.epsilon())
    f1 = tf.where(tf.is_nan(f1), tf.zeros_like(f1), f1)
    return K.mean(f1)

def show_history(history):
    fig, ax = plt.subplots(1, 3, figsize=(15,5))
    ax[0].set_title('loss')
    ax[0].plot(history.epoch, history.history["loss"], label="Train loss")
    ax[0].plot(history.epoch, history.history["val_loss"], label="Validation loss")
    ax[1].set_title('f1')
    ax[1].plot(history.epoch, history.history["f1"], label="Train f1")
    ax[1].plot(history.epoch,...


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Download dataset
Ensure that you have joined the competition on your kaggle account
Can you write Python code for it?

!kaggle competitions download -c human-protein-atlas-image-classification


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Load dataset info
Can you write Python code for it?

path_to_train = './human_protein_atlas/train/'
data = pd.read_csv('human_protein_atlas/train.csv')

train_dataset_info = []
for name, labels in zip(data['Id'], data['Target'].str.split(' ')):
    train_dataset_info.append({
        'path':os.path.join(path_to_train, name),
        'labels':np.array([int(label) for label in labels])})
train_dataset_info = np.array(train_dataset_info)

from sklearn.model_selection import train_test_split
train_ids, test_ids, train_targets, test_target = train_test_split(
    data['Id'], data['Target'], test_size=0.2, random_state=42)


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Notebook
This notebook uses an inception resnet v2 model from the kernel Pretrained InceptionResNetV2 base classifier
. We are using a pre-designed architecture that has been trained and does well on general image classification. We add a few more layers that are custom to our specific task.
Can you write Python code for it?

import os, sys, math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from PIL import Image
import cv2
from imgaug import augmenters as iaa
from tqdm import tqdm

import warnings
warnings.filterwarnings("ignore")

INPUT_SHAPE = (299,299,3)
BATCH_SIZE = 10


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Show data
Can you write Python code for it?

# create train datagen
input_shape = (299, 299, 3)
train_datagen = data_generator.create_train(
    train_dataset_info, 5, input_shape, augument=True)

images, labels = next(train_datagen)

fig, ax = plt.subplots(1,5,figsize=(25,5))
for i in range(5):
    ax[i].imshow(images[i])
print('min: {0}, max: {1}'.format(images.min(), images.max()))


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Train model
Can you write Python code for it?

checkpointer = ModelCheckpoint(
    './InceptionResNetV2.model', monitor = 'val_f1',
    verbose=2, save_best_only=True)

# no data augmentation training
train_generator = data_generator.create_train(
    train_dataset_info[train_ids.index], BATCH_SIZE, INPUT_SHAPE, augument=False)
validation_generator = data_generator.create_train(
    train_dataset_info[test_ids.index], 256, INPUT_SHAPE, augument=False)

model.layers[2].trainable = False

model.compile(
    loss='categorical_crossentropy',
    optimizer=Adam(1e-3),
    metrics=['acc', f1])

history = model.fit_generator(
    train_generator,
    steps_per_epoch=100,
    validation_data=next(validation_generator),
    epochs=1,
    verbose=1,
    callbacks=[checkpointer])

# To prevent error during training, custom f1 scoring object must be defined
from keras.utils.generic_utils import get_custom_objects

get_custom_objects().update({'f1': f1})

# Take a checkpoint model and load into keras
from keras.models import load_model
checkpointer_savepath = './InceptionResNetV2.model'
model = load_model(checkpointer_savepath)

show_history(history)


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Bayesian_hyperparameter_optimization.ipynb".
The first task is:
Unzip files
Can you write Python code for it?

!mkdir ./human_protein_atlas/
!mkdir ./human_protein_atlas/train
!mkdir ./human_protein_atlas/test

!mv train.csv ./human_protein_atlas/train.csv

!unzip -q ./train.zip -d ./human_protein_atlas/train

!unzip -q ./test.zip -d ./human_protein_atlas/test


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Monte_Carlo_Dropout_MNIST_Example.ipynb".
The first task is:
Compile model
Can you write Python code for it?

model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=['acc'])


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Monte_Carlo_Dropout_MNIST_Example.ipynb".
The first task is:
Load model weights
Can you write Python code for it?

if os.path.exists(MODEL_PATH):
  model.load_weights(MODEL_PATH)
  print ('Loaded model weights from: {}'.format(MODEL_PATH))


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Monte_Carlo_Dropout_MNIST_Example.ipynb".
The first task is:
Run Bayesian inference
Can you write Python code for it?

if bayesian_network:
  out = bayesianInference(model, X_test, y_test)
  print ('\n')
  print ('\nValidation accuracy: {} ...'.format(out[0]))
  print ('Validation uncertainty: {} ...'.format(out[1]))
else:
  (_, acc) = model.evaluate(x=X_test,
                            y=y_test,
                            batch_size=args.batch_size)
  print('\nValidation accuracy: {}'.format(acc))

if download_weights:
  from google.colab import files
  files.download(MODEL_PATH)


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Monte_Carlo_Dropout_MNIST_Example.ipynb".
The first task is:
Save model weights
Can you write Python code for it?


I am working on the file "Machine_Vision/07_Bayesian_deep_learning/notebook/Monte_Carlo_Dropout_MNIST_Example.ipynb".
The first task is:
Train a Bayesian network
Can you write Python code for it?

model.fit(x=X_train,
          y=y_train,
          batch_size=batch_size,
          epochs=epochs,
          validation_data=(X_test,
                           y_test),
          callbacks=[TensorBoard(log_dir=os.path.join(TENSORBOARD_DIR,
                                                      model.name),
                                 write_images=True)])


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
As usually, is a good idea to display our instances and the clusters they belong to, to have a first approximation to how well our algorithm is behaving on our data:
Can you write Python code for it?

colormarkers = [ ['red','s'], ['greenyellow','o'], ['blue','x']]
step = .01
margin = .1
sl_min, sl_max = X_train4[:, 0].min()-margin, X_train4[:, 0].max() + margin
sw_min, sw_max = X_train4[:, 1].min()-margin, X_train4[:, 1].max() + margin
sl, sw  = np.meshgrid(
    np.arange(sl_min, sl_max, step),
np.arange(sw_min, sw_max, step)
    )
Zs = clf_sepal.predict(np.c_[sl.ravel(), sw.ravel()]).reshape(sl.shape)
centroids_s = clf_sepal.cluster_centers_


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
Create training and testing sets, and scale values, as usual
Can you write Python code for it?

X_train_boston=boston.data
y_train_boston=boston.target
X_train_boston = StandardScaler().fit_transform(X_train_boston)
y_train_boston = StandardScaler().fit_transform(y_train_boston)


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
Display the data points and the calculated regions
Can you write Python code for it?

plt.figure(1)
plt.clf()
plt.imshow(Zs, interpolation='nearest',
           extent=(sl.min(), sl.max(), sw.min(), sw.max()),
           cmap= plt.cm.Pastel1, aspect='auto', origin='lower')
for j in [0,1,2]:
    px = X_train4[:, 0][y_train == j]
    py = X_train4[:, 1][y_train == j]
    plt.scatter(px, py, c=colormarkers[j][0], marker= colormarkers[j][1])
plt.scatter(centroids_s[:, 0], centroids_s[:, 1],marker='*',linewidths=3, color='black', zorder=10)
plt.title('K-means clustering on the Iris dataset using Sepal dimensions\nCentroids are marked with stars')
plt.xlim(sl_min, sl_max)
plt.ylim(sw_min, sw_max)
plt.xlabel("Sepal length")
plt.ylabel("Sepal width")
plt.show()


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
Evaluating our results
We want to be a little more formal when we talk about a good classifier. What does
that mean? The performance of a classifier is a measure of its effectiveness. The
simplest performance measure is accuracy: given a classifier and an evaluation
dataset, it measures the proportion of instances correctly classified by the classifier.
First, let's test the accuracy on the training set:
Can you write Python code for it?

from sklearn import metrics
y_train_pred = clf.predict(X_train)
print metrics.accuracy_score(y_train, y_train_pred)


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
If our classifier is right, this Iris flower is a setosa. Probably, you have noticed that
we are predicting a class from the possible three classes but that linear models are
essentially binary: something is missing. You are right. Our prediction procedure
combines the result of the three binary classifiers and selects the class in which it is
more confident. In this case, we will select the boundary line whose distance to the
instance is longer. We can check that using the classifier decision_function method:
Can you write Python code for it?


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
One of the main advantages of cross-validation is reducing the variance of the evaluation measures. Note that when you have a training and testing set, you result will depend on how you created them. Evaluation within machine learning generally assumes that the distribution of classes on your training and testing sets are similar. If this is not the case, you may get results that are not an accurate measure of the actual classifier’s performance. Cross-validation allows us to mitigate this risk: since we are averaging on k different models built on k different datasets, we are reducing variance and probably producing more realistic performance scores for our models.
Scikit-learn includes a linear model method called linear_model.SGDRegressor , which tries to minimize least squares loss using stochastic gradient descent. We will use it to find the model for Boston house values, and predict a new house value, given its 13 learning features.
Can you write Python code for it?

import sklearn
from sklearn import linear_model
clf_sgd = linear_model.SGDRegressor(loss='squared_loss', penalty=None, random_state=33)
train_and_evaluate(clf_sgd, X_train_boston, y_train_boston,5)


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
Repeat the experiment, using petal dimensions
Can you write Python code for it?

clf_petal = cluster.KMeans(init='k-means++', n_clusters=3, random_state=33)
clf_petal.fit(X_train4[:,2:4])

print (y_train4[clf_petal.labels_==0])

print (y_train4[clf_petal.labels_==1])

print (y_train4[clf_petal.labels_==2])


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
The SGDClassifier initialization function allows several parameters. For the
moment, we will use the default values, but keep in mind that these parameters
could be very important, especially when you face more real-world tasks, where the
number of instances (or even the number of attributes) could be very large. The fit
function is probably the most important one in scikit-learn. It receives the training
data and the training classes, and builds the classifier. Every supervised learning
method in scikit-learn implements this function.
What does the classifier look like in our linear model method? As we have already
said, every future classification decision depends just on a hyperplane. That
hyperplane is, then, our model. The coef_ attribute of the clf object (consider, for
the moment, only the first row of the matrices), now has the coefficients of the linear
boundary and the intercept_ attribute, the point of intersection of the line with the
y axis. Let's print them:
Can you write Python code for it?

# print learned coeficients
print clf.coef_
print clf.intercept_


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
Using NumPy's indexing capabilities, we can display the actual target classes for each cluster, just to compare the built clusters with our flower type classes...
Can you write Python code for it?

print (y_train[clf_sepal.labels_==0])

print (y_train[clf_sepal.labels_==1])

print (y_train[clf_sepal.labels_==2])


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
Wait, every performance measure is better using just two attributes. It is possible that less features give better results? Although at a first glance this seems contradictory, we will see in future notebooks that selecting the right subset of features, a process called feature selection, could actually improve the performance of our algorithms.
Supervised Learning: Regression
In every example we have seen so far the output we aimed at predicting belonged to a discrete set. For classification, the set was the target class, while for the clustering algorithm the set included different calculated clusters. What if we want to predict a value extracted from the real line?. In this case, we are trying to solve a regression problem.
To show how regression works in scikit-learn, we will apply to a (very) simple and well-know problem: trying to predict the price of a house given some of its. As the dataset, we will use the Boston house-prices dataset (find the dataset description and attributes here.
Can you write Python code for it?

from sklearn.datasets import load_boston
boston = load_boston()
print ('Boston dataset shape:{}'.format(boston.data.shape))

print (boston.feature_names)


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
We have computed precision, recall, and f1-score for each class and their average
values. What we can see in this table is:
* The classifier obtained 1.0 precision and recall in the setosa class. This means that for precision, 100 percent of the instances that are classified as setosa are really setosa instances, and for recall, that 100 percent of the setosa instances were classified as setosa.
* On the other hand, in the versicolor class, the results are not as good: we have a precision of 0.43, that is, only 43 percent of the instances that are classified as versicolor are really versicolor instances. Also, for versicolor, we have a recall of 0.27, that is, only 27 percent of the versicolor instances are correctly classified.
Now, we can see that our method (as we expected) is very good at predicting
setosa, while it suffers when it has to separate the versicolor or virginica
classes. The support value shows how many instances of each class we had in the
testing set.
Another useful metric (especially for multi-class problems) is the confusion matrix:
in its (i, j) cell, it shows the number of class instances i that were predicted to
be in class j. A good classifier will accumulate the values on the confusion matrix
diagonal, where correctly classified instances belong.
Can you write Python code for it?


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
We obtained an accuracy of 68 percent in our testing set. Usually, accuracy on the
testing set is lower than the accuracy on the training set, since the model is actually
modeling the training set, not the testing set. Our goal will always be to produce
models that avoid overfitting when trained over a training set, so they have enough
generalization power to also correctly model the unseen data.
Accuracy on the test set is a good performance measure when the number of instances
of each class is similar, that is, we have a uniform distribution of classes. But if you
have a skewed distribution (say, 99 percent of the instances belong to one class), a
classifier that always predicts the majority class could have an excellent performance in
terms of accuracy despite the fact that it is an extremely naive method.
Within scikit-learn, there are several evaluation functions; we will show three
popular ones: precision, recall, and F1-score (or f-measure). They assume a binary
classification problem and two classes—a positive one and a negative one. In our
example, the positive class could be Iris setosa, while the other two will be combined
into one negative class.

Precision: This computes the proportion of instances predicted as positives that were correctly evaluated (it measures how right our classifier is when it says that an instance is positive).
Recall: This counts the proportion of positive instances that were correctly evaluated (measuring how right our classifier is when faced with a positive instance).
F1-score: This is the harmonic mean of precision and recall, and tries to combine both in a single number.

We can define these measures in terms of True and False, and Positives and Negatives:
                    <pre>
                                 | <b>Prediction: Positive           </b>|<b> Prediction: Negative </b>
                    Target cass: | Positive True Positive  (TP)   | False Negative (FN)
                    Target cass: | Negative False Positive (FP)   | True Negative (TN)
            ...
Can you write Python code for it?

print metrics.classification_report(y_test, y_pred, target_names=iris.target_names)


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
We obtained an array with the k scores. We can calculate the mean and the standard
error to obtain a final figure:
Can you write Python code for it?

from scipy.stats import sem

def mean_score(scores):
    """Print the empirical mean score and standard error of the mean."""
    return ("Mean score: {0:.3f} (+/-{1:.3f})").format(
        np.mean(scores), sem(scores))

print mean_score(scores)


I am working on the file "Maching Learning using Scikit-Learn/Getting-Started.ipynb".
The first task is:
You probably noted the penalty=None parameter when we called the method. The penalization parameter for linear regression methods is introduced to avoid overfitting. It does this by penalizing those hyperplanes having some of their coefficients too large, seeking hyperplanes where each feature contributes more or less the same to the predicted value. This parameter is generally the L2 norm (the squared sums of the coefficients) or the L1 norm (the sum of the absolute value of the coefficients). Let us see how our model works if we introduce a L2 penalty:
Can you write Python code for it?

clf_sgd1 = linear_model.SGDRegressor(loss='squared_loss', penalty='l2', random_state=33)
train_and_evaluate(clf_sgd1, X_train_boston, y_train_boston,folds=5)


I am working on the file "MagDipole.ipynb".
The first task is:
3D plot of field lines and data

Color bar in nT
Spatial distance in meter
Profile data only reflects shape of anomaly and positivity
Can you write Python code for it?

fig = plt.figure()
ax = fig.gca(projection='3d')

# plot field lines
for lx,ly,lz in zip(linex,liney,linez):
    ax.plot(lx,ly,lz,'-',markersize=1)

# plot map
ax.scatter(x1,y1,z1,s=2,alpha=0.3)
Bt = Ba1.reshape(xi.shape)*1e9 # contour and color scale in nT
c = ax.contourf(xi,yi,Bt,alpha=1,zdir='z',offset=z-max(radii)*2,cmap='jet',
                  levels=np.linspace(Bt.min(),Bt.max(),50,endpoint=True))
fig.colorbar(c)

# auto-scaling for profile plot
ptpmax = np.max((Ba2.ptp(),Ba3.ptp())) # dynamic range
autoscaling = np.max(radii) / ptpmax

# plot x-profile
ax.scatter(x2,y2,z2,s=2,c='black',alpha=0.3)
ax.plot(x2,Ba2*autoscaling,zs=ymax,c='black',zdir='y')

# plot y-profile
ax.scatter(x3,y3,z3,s=2,c='black',alpha=0.3)
ax.plot(y3,Ba3*autoscaling,zs=xmin,c='black',zdir='x')

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')

ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_zlim(z-max(radii)*2, max(radii)*1.5)


I am working on the file "MagDipole.ipynb".
The first task is:
Define the Earth's magnetic field $B_0$
$B_0$ is used to calcualte the total field anomaly, which is the projection of the anomalous vector field onto the earth's field (inner product).
* B0: the magnitude of the earth's field
* Binc: inclination of the earth's field
* Bdec: declination of the earth's field
Can you write Python code for it?

# geomagnetic field
B0, Binc, Bdec = 53600e-9, 90., 0. # in Tesla, degree, degree
B0x = B0*np.cos(np.radians(Binc))*np.sin(np.radians(Bdec))
B0y = B0*np.cos(np.radians(Binc))*np.cos(np.radians(Bdec))
B0z = -B0*np.sin(np.radians(Binc))


I am working on the file "MagDipole.ipynb".
The first task is:
Define the observations
Four data plots will be generated in the figure: magnetic anomalous field map (contour) at a certain elevation, magnetic anomalous field data (curve) along a x-profile and a y-profile, and the magnetic field lines of the dipole.
* xmin, xmax, ymin, ymax: the outer bounds of the survey grid
* z: elevation at which the data map is measured
* profile_x, profile_y: x-coordinate of y-profile, y-coordinate of x-profile
* h: grid interval
* radii: how far the field lines expand; can plot multiple layers if given (r1, r2, ...)
* Nazi: number of azimuth angles for the field line
Can you write Python code for it?

# set observation grid
xmin, xmax, ymin, ymax, z = -5., 5., -5., 5., 1. # x, y bounds and elevation
profile_x = 0. # x-coordinate of y-profile
profile_y = 0. # y-coordinate of x-profile
h = 0.2 # grid interval
radii = (2., 5.) # how many layers of field lines for plotting
Nazi = 10 # number of azimuth


I am working on the file "MainCodeDoc.ipynb".
The first task is:
2. Extract Data
Can you write Python code for it?


I am working on the file "MainCodeDoc.ipynb".
The first task is:
3. Select Data
We start by filtering the columns that were considered the most important based on a data analysis made prior to the development of the regression.
We filter by the relevant columns.
Can you write Python code for it?

df1 = gdf[['AIRLINE_ID','ORIGIN', 'DEST', 'DEP_TIME','ARR_TIME','DEP_DELAY','ARR_DELAY','CANCELLED','DIVERTED','ACTUAL_ELAPSED_TIME']]


I am working on the file "MainCodeDoc.ipynb".
The first task is:
4. Sample Data
We now get a random sample Data from the big data set
Can you write Python code for it?

sampledf = df2.sample(n=10000)
print("Sample Size:" + str(len(sampledf.axes[0])))


I am working on the file "MainCodeDoc.ipynb".
The first task is:
5. Clean Data
Having the sample data, we proceed to clean the data and make sure there are no invalid data in the dataset
First, we trim the string columns to avoid any unexpected error
Can you write Python code for it?

sampledf["AIRLINE_ID"] = sampledf.apply(lambda row: str.strip(str(row.AIRLINE_ID)), axis=1)
sampledf["ORIGIN"] = sampledf.apply(lambda row: str.strip(str(row.ORIGIN)), axis=1)
sampledf["DEST"] = sampledf.apply(lambda row: str.strip(str(row.DEST)), axis=1)


I am working on the file "MainCodeDoc.ipynb".
The first task is:
6. Add new Columns
We can now calculate the new columns we need for the regression. Based on a prior data analysis we found that the period of the day is relevant for the delay and flight time of a flight.
So, we first calculate the flight periods columns, where Morning is from 6 to 12 , Afternoon is from 12 to 19, Night is from 19 to 24, and Dawn is from 24 to 6
Can you write Python code for it?

sampledf["Morning"]   = sampledf.apply(lambda row: 1 if(not row.CANCELLED and int(row.DEP_TIME) >= 600 and int(row.DEP_TIME) < 1200) else 0, axis=1)
sampledf["Afternoon"] = sampledf.apply(lambda row: 1 if(not row.CANCELLED and int(row.DEP_TIME) >= 1200 and int(row.DEP_TIME) < 1900) else 0, axis=1)
sampledf["Night"]     = sampledf.apply(lambda row: 1 if(not row.CANCELLED and int(row.DEP_TIME) >= 1900 and int(row.DEP_TIME) < 2400) else 0, axis=1)
sampledf["Dawn"]      = sampledf.apply(lambda row: 1 if(not row.CANCELLED and int(row.DEP_TIME) >= 2400 and int(row.DEP_TIME) < 600) else 0, axis=1)


I am working on the file "MainCodeDoc.ipynb".
The first task is:
9. 50-50 Data
We now split the data set to make sure that 50% of the rows are delayed flights and 50% are non-delayed flights
We do it for the training data set
Can you write Python code for it?

trainingDFDelayed =  trainingDF[trainingDF.Delayed == 1].head(500)
trainingDFNotDelayed = trainingDF[trainingDF.Delayed == 0].head(500)

allTraining = [trainingDFNotDelayed,trainingDFDelayed]
trainingDF = pd.concat(allTraining)

print("Training Size:" + str(len(trainingDF.axes[0])))


I am working on the file "MainCodeDoc.ipynb".
The first task is:
And we make sure all variables are an integer for the regression
Can you write Python code for it?

trainingDF = trainingDF.applymap(np.int)
testDF = testDF.applymap(np.int)


I am working on the file "MainCodeDoc.ipynb".
The first task is:
Finally, we clean any invalid Data - any flight that has negative time
Can you write Python code for it?

sampledf = sampledf[sampledf.ACTUAL_ELAPSED_TIME >= 0]


I am working on the file "MainCodeDoc.ipynb".
The first task is:
Predicted Total Time Graph (Testing set is used here)
Now we create a testing graph where we can see how much if the data is under a specific difference of time.
Can you write Python code for it?

#Create dataframe with the difference ranges
roicTime = pd.DataFrame({"Values":range(int(subsetPredictTime["Difference"].min()),int(subsetPredictTime["Difference"].max()),10)})
roicTime["Percentage"] = roicTime.apply(lambda row: len(subsetPredictTime[subsetPredictTime.Difference < row.Values]["Difference"]) / len(subsetPredictTime["Difference"]) * 100, axis=1 )

plt.plot(roicTime.Values,roicTime.Percentage)
plt.xlabel("Time Difference between actual and predicted (min)")
plt.ylabel("% of Data covered")
plt.title('Predict Time Validation Graph')
plt.show()


I am working on the file "MainCodeDoc.ipynb".
The first task is:
Using the dummy variables, we create a table for the regression by concatenating all of the dummy columns and the dependant variable
Can you write Python code for it?

dummyDf = pd.DataFrame()
dummyDf = pd.concat([originDummy,destDummy,airlineDummy,sampledf['Morning'], sampledf['Afternoon'], sampledf['Night'],sampledf['Delayed'],sampledf['ACTUAL_ELAPSED_TIME']], axis=1)
dummyDf.head()


I am working on the file "MainCodeDoc.ipynb".
The first task is:
VALIDATION AND TESTING
Validate with Test Data -- Delayed Prediction
Using the regression, we predict the outcome for the testing data set for the delay prediction.
Can you write Python code for it?

#Copy of the testing data set
validateDataDelay = testDF.copy()

#Get a subset of the data without the validation data
subsetPredictDelay = validateDataDelay[validateDataDelay.columns.difference(['Delayed','ACTUAL_ELAPSED_TIME'])]

#Predict the outcome with the regression and put the result in a new column
subsetPredictDelay['Calculated_Delay'] = subsetPredictDelay.apply(lambda row: (row * resultDelayed.params).sum(),axis=1)

#Add the real outcome in a new column
subsetPredictDelay["Real_Delayed"] = testDF["Delayed"]

subsetPredictDelay.head()[["Real_Delayed","Calculated_Delay"]]


I am working on the file "MainCodeDoc.ipynb".
The first task is:
Validate with Test Data -- Predicted Total Time (Flight time + taxi)
Using the regression, we predict the outcome for the testing data set for the flight time prediction.
Can you write Python code for it?

#Copy of the testing data set
validateDataTime = testDF.copy()

subsetPredictTime = validateDataTime[validateDataTime.columns.difference(['Delayed','ACTUAL_ELAPSED_TIME'])]
subsetPredictTime["const"] = 1
subsetPredictTime['Calculated'] = subsetPredictTime.apply(lambda row: (row * resultTime.params).sum(),axis=1)

subsetPredictTime["ACTUAL_ELAPSED_TIME"] = validateDataTime["ACTUAL_ELAPSED_TIME"]
subsetPredictTime["Difference"] = subsetPredictTime.apply(lambda row: abs(row.ACTUAL_ELAPSED_TIME - row.Calculated), axis=1)

subsetPredictTime.head()[["ACTUAL_ELAPSED_TIME","Calculated","Difference"]]