I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
The problem is that variables are just names referring to an object. Assigning the object of a variable to an other does not create a copy of this object. It creates a new variable B which refers to the same object A refers to. Hence, there is only one object but two variables are referring to this one. This holds only for mutable objects like lists, sets, dictionaries and so on.
There are also immutable objects like inits, floats, strings, tuple and so on.
Can you write Python code for it?


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
We will change variable B if we change the inner object of variable A. This is because we have a copy of the first object (outer list) and just set a reference of the inner object. This brings us back to the problem we discussed above.
Deepcopy
To avoid this we have the possibility to make a deepcopy of our object assigned to a variable.
Can you write Python code for it?

import copy
A = [[5]]
B = copy.deepcopy(A)
A[0][0] = 1
print "A: ", A
print "B: ", B


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Why do I need this?
OOP facilitates some desirable features in software projects:

code organization
encapsulation
DRY principle
modularization

How we will use it
Our project will make extensive use of object orientation and inheritance.

classes use each other to run the model
each process knows which species to update, makes modularization possible
simulation class manages everything, registers processes
states/processes inherit from base class

Decorators
Some helpful syntax
Nested Functions
Functions can contain other functions:
Can you write Python code for it?

def outer_function():
    print "out here"
    def inner_1():
        return "in room 1"

    def inner_2():
        return "in room 2"

    print inner_1()
    print inner_2()

outer_function()


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
But if you change the value of variable A as it is done blow, variable B changes, too.
Can you write Python code for it?


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Decorator Syntax
Can you write Python code for it?

@greetings
def shout(sentence):
    print(sentence.upper())

shout("Am I polite, or what?!")


@greetings
def shout(sentence):
    print sentence.upper()

shout("Am I polite, or what?!")


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Dictionaries
Creating dictionaries
Can you write Python code for it?

# dictionary comprehension
name_space = "a","b","c","d","e"
dictionary = {name_space[x]: x**2 for x in range(len(name_space))}
print "dictionary comprehension: ", dictionary

# creating a dictionary using zip function
dictionary = dict(zip(['A', 'B', 'C'], [1, 2, 3]))
print "dictionary: ", dictionary

# default dictionary
from collections import defaultdict
default_dict = defaultdict(int)
default_dict['a'] += 1  # increasing the value of key "a" directly during the initialisation of the key
print "default dictionary: ", default_dict

# normal dictionary initialisation
no_default = dict()
no_default['a'] += 1


# dictionary comprehension
name_space = "a","b","c","d","e"
dictionary = {name_space[x]: x**2 for x in range(len(name_space))}
print("dictionary comprehension: ", dictionary)

# creating a dictionary using zip function
dictionary = dict(zip(['A', 'B', 'C'], [1, 2, 3]))
print("dictionary: ", dictionary)

# default dictionary
from collections import defaultdict
default_dict = defaultdict(list)
default_dict['a'].append(12)  # if a value has not been set for 'a', initialize with empty list
print("default dictionary: ", default_dict)

# normal dictionary initialisation
no_default = dict()
no_default['a'].append(12)


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Flatting Lists
Can you write Python code for it?

import itertools
a_list = [[1,2],[3,4],[5,6]]

a_flatted_list = [x for i in a_list for x in i]
print "a_flatted_list: ", a_flatted_list


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
How will we use it?
Can you write Python code for it?

class Molecule(object):
    def __init__(self, mass):
        self.mass = mass

    @property
    def mass(self):
        return self.__mass

    @mass.setter
    def mass(self, value):
        if not isinstance(value, int):
            raise TypeError("Mass must be Integer.")
        self.__mass = value

m = Molecule(1)

m.mass = "two"


class Molecule(object):
    def __init__(self, mass):
        self.__mass = mass

    def get_mass(self):
        return self.__mass

    def set_mass(sefl, value):
        if not isinstance(value, int):
            raise TypeError("Mass must be Integer.")
        self.__mass = value

m = Molecule(12)
m.set_mass('13')

class Molecule(object):
    def __init__(self, mass):
        self.mass = mass

    @property
    def mass(self):
        print("getter")
        return self.__mass

    @mass.setter
    def mass(self, value):
        print("setter")
        if not isinstance(value, int):
            raise TypeError("Mass must be Integer.")
        self.__mass = value

m = Molecule(1)

m.mass

m.mass = "two"


class Molecule(object):
    def __init__(self, mass):
        self.mass = mass

    @property
    def mass(self):
        return self.__mass

    @mass.setter
    def mass(self, value):
        if not isinstance(value, int):
            raise TypeError("Mass must be Integer.")
        self.__mass = value

m = Molecule(1)

m.mass = "20"


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
However, some operations directly create new objects.
Can you write Python code for it?

A = [[2,3,1]]
B = A
A = A + [4]
print B
print sorted(A[0])
print B


A = [[2,3,1]]
B = A
A = A + [4]
print(B)
print(sorted(A[0]))
print(B)


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
If we apply this operation on a variable referring to an immutable object - a new object is created, but if we apply this to an object referring to a mutable object like lists, sets and so on we are just changing the object.
Can you write Python code for it?

A = [1,2,3]  # assigning a mutable list object to A
B = A
A += [1]
print "A: ", A
print "B: ", B


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
In this case the += operation on a list is equivalent to the A.extend([1]) operation.
Copy
The copy operation can be used to make copies of an object. Like
Can you write Python code for it?

import copy
A = [5]
B = copy.copy(A)
A[0] = 1
print "A: ", A
print "B: ", B


import copy
A = [5]
B = copy.copy(A)
A[0] = 1
print("A: ", A)
print("B: ", B)


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Inheritance
Classes can inherit traits (and even skills) from other classes:
Can you write Python code for it?

class Dachshund(Dog):
    color = "brownish"
    def __init__(self, name):
        super(Dachshund, self).__init__(name, 20)

d = Dachshund('bello')
d.bark()


class Dachshund(Dog):
    color = "brownish"
    def __init__(self, name):
        super().__init__(name, 20)

d = Dachshund('bello')
d.bark()


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Instances
Classes can be used to create objects with certain attributes and behavior:
Can you write Python code for it?

class Dog:
    color = "yellow"
    def __init__(self, name, size):
        self.name = name
        self.size = size
    def bark(self):
        print('whoof, whoof!')

b = Dog("blitzer", 100)
b.bark()


class Dog(object):
    color = "yellow"
    def __init__(self, name, size):
            self.name = name
            self.size = size
    def bark(self):
        print 'whoof, whoof!'

b = Dog("blitzer", 100)
b.bark()


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Inverting dictionaries
Can you write Python code for it?

dictionary = dict(a=1, b=2, c=3)  # normal dict generation
print "dictionary: ", dictionary

# inverting dictionary using zip function
inverted_dict = dict(zip(dictionary.values(), dictionary.keys()))
print "inverted dict: ", inverted_dict


dictionary = dict(a=1, b=2, c=3)  # normal dict generation
print("dictionary: ", dictionary)

# inverting dictionary using zip function
inverted_dict = dict(zip(dictionary.values(), dictionary.keys()))
print("inverted dict: ", inverted_dict)


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Mating
Can you write Python code for it?

class Dog(object):
    color = "yellow"
    def __init__(self, name, size):
            self.name = name
            self.size = size

    def __add__(self, dog):
        if not isinstance(dog, Dog):
            raise TypeError
        return Dog(self.name + " " + dog.name, (self.size + dog.size)/2)

    def bark(self):
        print 'whoof, whoof!'

b = Dog("blitzer", 100)
dog = b + p
dog.bark()
print dog.name
print dog.size


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Ordered dictionaries
Can you write Python code for it?

from collections import OrderedDict
ordered_dict = OrderedDict((name_space[x], x**2) for x in range(len(name_space)))
print("ordered dictionary: ", ordered_dict)


from collections import OrderedDict
ordered_dict = OrderedDict((name_space[x], x**2) for x in range(len(name_space)))
print "ordered dictionary: ", ordered_dict


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Permutations
Can you write Python code for it?

import itertools
for p in itertools.permutations([1,2,3]):
    print "permutations: ", p


import itertools
for p in itertools.permutations([1,2,3]):
    print("permutations: ", p)


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Pypi
Python has a very active development community and the Python package index contains a myriad of packages for every need.
These can be install by using pip which is installed with newer python versions:
pip install numpy
Object Orientation

Programming Paradigms

Object orientation in Python

everything is a object (even functions)

You can define your own objects by using the class keyword:
Can you write Python code for it?

class Dog(object):
    color = "yellow"
    def __init__(self, name):
            self.name = name

Dog


class Dog:
    color = "yellow"
    def __init__(self, name):
        self.name = name

Dog


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Shadowing
Classes can alter functionality of their parents:
Can you write Python code for it?

class Pekingese(Dog):
    color = "brownish"
    def __init__(self, name):
        super().__init__(name, 15)

    def bark(self):
        print('whif.')

d = Pekingese('tiny')
d.bark()


class Pekingese(Dog):
    color = "brownish"
    def __init__(self, name):
        super(Pekingese, self).__init__(name, 15)

    def bark(self):
        print 'whif.'

d = Pekingese('tiny')
d.bark()


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
So we end up with two objects and two variables referring to these objects, respectively.
But we have to pay attention when we use operations like +=. The behaviour of this kind of operations depends on the object it is applied on.
Can you write Python code for it?

A = ((1,2,3))  # assigning a immutable tuple object to A
B = A
A += (3,4,5)
print "A: ", A
print "B: ", B


A = ((1,2,3))  # assigning a immutable tuple object to A
B = A
A += (3,4,5)
print("A: ", A)
print("B: ", B)


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Special methods
Can you write Python code for it?

class Pekingese(Dog):
    color = "brownish"
    def __init__(self, name):
        super().__init__(name, 15)

    def __len__(self):
        return self.size

    def bark(self):
        print('whif.')

p = Pekingese("wang")
len(p)


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
The copy operation creates a copy of the object (in this case a list) assigned to variable A and set references of the inner objects. The inner object (integer 1) is immutable, hence, the variable B will not change if we change A. But if we have something like
Can you write Python code for it?

A = [[5]]
B = copy.copy(A)
A[0][0] = 1
print "A: ", A
print "B: ", B


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
The problem is that variables are just names referring to an object. Assigning the object of a variable to an other does not create a copy of this object. It creates a new variable B which refers to the same object A refers to. Hence, there is only one object but two variables are referring to this one. This holds only for mutable objects like lists, sets, dictionaries and so on.
There are also immutable objects like inits, floats, strings, tuple and so on.
Can you write Python code for it?


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
We will change variable B if we change the inner object of variable A. This is because we have a copy of the first object (outer list) and just set a reference of the inner object. This brings us back to the problem we discussed above.
Deepcopy
To avoid this we have the possibility to make a deepcopy of our object assigned to a variable.
Can you write Python code for it?

import copy
A = [[5]]
B = copy.deepcopy(A)
A[0][0] = 1
print "A: ", A
print "B: ", B


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Why do I need this?
OOP facilitates some desirable features in software projects:

code organization
encapsulation
DRY principle
modularization

How we will use it
Our project will make extensive use of object orientation and inheritance.

classes use each other to run the model
each process knows which species to update, makes modularization possible
simulation class manages everything, registers processes
states/processes inherit from base class

Decorators
Some helpful syntax
Nested Functions
Functions can contain other functions:
Can you write Python code for it?

def outer_function():
    print("out here")
    def inner_1():
        return "in room 1"

    def inner_2():
        return "in room 2"

    print(inner_1())
    print(inner_2())

outer_function()


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
With this operation we are creating a new object of the first object and for all nested objects, too. This means that the entire structure is copied and we end up with two objects (containing arbitray nested objects) assigned to two variables.
Zipping unzipping lists and iterables
Can you write Python code for it?

A = [1, 2, 3]
B = ['a', 'b', 'c']
z = zip(A, B)
print "zipping: ", z
print "unzipping: ", zip(*z)


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
Wrappers
Decorators are functions wrapped around functions:
Can you write Python code for it?

def greetings(some_function):
    def wrapper(args):
        print("Hello!")
        some_function(args)
        print("Goodbye!")
    return wrapper

def say_something(sentence):
    print("Say: {0}".format(sentence))

polite_talker = greetings(say_something)

polite_talker("Good to see you")


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
getter and setter methods

Further Tricks
Some python specialties you should know about
Copy
By Reference/Value
As you might know you can assign values of one variable to an other like:
Can you write Python code for it?


I am working on the file "03_advanced/advanced_python_tricks.ipynb".
The first task is:
try to call inner_1

Wrappers
Decorators are functions wrapped around functions:
Can you write Python code for it?

def greetings(some_function):
    def wrapper(args):
        print "Hello!"
        some_function(args)
        print "Goodbye!"
    return wrapper

def say_something(sentence):
    print "Say: {0}".format(sentence)

polite_talker = greetings(say_something)

polite_talker("Good to see you")


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
2. Scikit
Scikit is a machine learning library for Python built upon numpy and matplotlib. It provides functions for classification, regression, clustering and other common analytics tasks.
Can you write Python code for it?

from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=3)
results = kmeans.fit_predict(data[['SepalLength', 'SepalWidth', 'PetalLength', 'PetalWidth']])

data_kmeans=pd.concat([data, pd.Series(results, name="ClusterId")], axis=1)
data_kmeans.head()


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
3. Pilot Approach
We will now use RADICAL-Pilot to compute the distance function, as a simple representation of how the above example can be executed as a task-parallel application.
Can you write Python code for it?

import os, sys
import commands
import radical.pilot as rp
os.environ["RADICAL_PILOT_DBURL"]="mongodb://ec2-54-221-194-147.compute-1.amazonaws.com:24242/giannis"

def print_details(detail_object):
    if type(detail_object)==str:
        detail_object = ast.literal_eval(detail_object)
    for i in detail_object:
        detail_object[i]=str(detail_object[i])
    #print str(detail_object)
    return pd.DataFrame(detail_object.values(),
             index=detail_object.keys(),
             columns=["Value"])

session = rp.Session()
c = rp.Context('ssh')
c.user_id = "radical"
session.add_context(c)
pmgr = rp.PilotManager(session=session)
umgr = rp.UnitManager (session=session,
                       scheduler=rp.SCHED_DIRECT_SUBMISSION)
print "Session id: %s Pilot Manager: %s" % (session.uid, str(pmgr.as_dict()))
pdesc = rp.ComputePilotDescription ()
pdesc.resource = "local.localhost_anaconda"
pdesc.runtime  = 10
pdesc.cores    = 16
pdesc.cleanup  = False
pilot = pmgr.submit_pilots(pdesc)
umgr = rp.UnitManager (session=session,
                       scheduler=rp.SCHED_DIRECT_SUBMISSION)
umgr.add_pilots(pilot)


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
Convert DataFrame to Tuple for MLlib
Can you write Python code for it?

data_spark_tuple = data_spark.map(lambda a: (a[0],a[1],a[2],a[3]))

from pyspark.mllib.clustering import KMeans, KMeansModel
clusters = KMeans.train(data_spark_tuple, 3, maxIterations=10,
                        runs=10, initializationMode="random")

# Evaluate clustering by computing Within Set Sum of Squared Errors
def error(point):
    center = clusters.centers[clusters.predict(point)]
    return sqrt(sum([x**2 for x in (point - center)]))

WSSSE = data_spark_tuple.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print("Within Set Sum of Squared Error = " + str(WSSSE))


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
Helper Function for computing new centroids as mean of all points assigned to a cluster
Can you write Python code for it?

def compute_new_centroids(distances):
    df = pd.DataFrame(distances)
    df[4] =  df[4].astype(int)
    df = df.groupby(4)[0,1,2,3].mean()
    centroids_np = df.as_matrix()
    return centroids_np


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
In the following we evaluate the resulting fit (commonly referred to as the model), using the sum of squared errors and a pair plot. The following pair plot shows the scatter-plot between each of the four features. Clusters for the different species are indicated by different colors.
Can you write Python code for it?

print "Sum of squared error: %.1f"%kmeans.inertia_

sns.pairplot(data_kmeans, vars=["SepalLength", "SepalWidth", "PetalLength", "PetalWidth"], hue="ClusterId");


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
Load and parse the data in a Spark DataFrame.
Can you write Python code for it?

data_spark=sqlCtx.createDataFrame(data)
data_spark_without_class=data_spark.select('SepalLength', 'SepalWidth', 'PetalLength', 'PetalWidth').show()


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
Print out final centroids computed
Can you write Python code for it?


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
Running Mapper Function as an External Process
Can you write Python code for it?

for i in range(10):
    distances =!/opt/anaconda/bin/python mapper.py points.csv clusters.csv
    distances_np =  np.array(eval(" ".join(distances)))
    new_centroids = compute_new_centroids(distances_np)
    new_centroids_df = pd.DataFrame(new_centroids, columns=["SepalLength", "SepalWidth", "PetalLength", "PetalWidth"])
    new_centroids_df.to_csv("clusters.csv")


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
Running Mapper Function inside RADICAL-Pilot
Helper function to read output from completed compute units after it has been executed inside the Pilot.
Can you write Python code for it?

import urlparse

def get_output(compute_unit):
    working_directory=compute_unit.as_dict()['working_directory']
    path = urlparse.urlparse(working_directory).path
    output=open(os.path.join(path, "STDOUT")).read()
    return output


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
Spark MLLib
In the following we utilize the Spark MLlib KMeans implementation. See http://spark.apache.org/docs/latest/mllib-clustering.html#k-means
We use Pilot-Spark to startup Spark.
Can you write Python code for it?

from numpy import array
from math import sqrt

%run ../env.py
%run ../util/init_spark.py

from pilot_hadoop import PilotComputeService as PilotSparkComputeService

try:
    sc
except:
    pilotcompute_description = {
        "service_url": "yarn-client://sc15.radical-cybertools.org",
        "number_of_processes": 5
    }
    pilot_spark = PilotSparkComputeService.create_pilot(pilotcompute_description=pilotcompute_description)

    sc = pilot_spark.get_spark_context()
    sqlCtx=SQLContext(sc)


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
Stop Pilot-Job
Can you write Python code for it?


I am working on the file "03_analytics/Kmeans.ipynb".
The first task is:
This is the main application loop. The distance computation is executed inside a ComputeUnit.  See mapper.py for code. Data is read from files and written to stdout. We execute 10 iterations of KMeans.
Can you write Python code for it?

for i in range(10):
    cudesc = rp.ComputeUnitDescription()
    cudesc.executable  = "/opt/anaconda/bin/python"
    cudesc.arguments   = [os.path.join(os.getcwd(), "mapper.py"),
                          os.path.join(os.getcwd(), "points.csv"),
                          os.path.join(os.getcwd(), "clusters.csv")]
    cu_set = umgr.submit_units([cudesc])
    umgr.wait_units()
    output = get_output(cu_set[0])
    distances_np =  np.array(eval(output))
    new_centroids = compute_new_centroids(distances_np)
    new_centroids_df = pd.DataFrame(new_centroids, columns=["SepalLength", "SepalWidth", "PetalLength", "PetalWidth"])
    new_centroids_df.to_csv("clusters.csv")
    print "Finished iteration: %d"%(i)


I am working on the file "03_aprendizaje_supervisado.ipynb".
The first task is:
Aprendizaje Supervisado

suponiendo que $f(x) = a x^2 + b x + c$

Entrenamiento

minimizar $\sum_i (y_i - (a x_i^2 + b x_i + c))^2$
Can you write Python code for it?

import numpy as np
X = np.vstack((x**2, x, np.ones_like(x))).T
coef = np.linalg.lstsq(X, y)[0]
print(coef)

pylab.plot(x, y, 'o')
pylab.plot(x, np.dot(X, coef))


I am working on the file "03_aprendizaje_supervisado.ipynb".
The first task is:
TFIDF
 buen dia microtc , excelente dia, odio el trafico
Can you write Python code for it?

from microtc.textmodel import TextModel
import numpy as np
import pandas as pd
M = np.zeros((11, len(docs)))
model = TextModel(docs, token_list=[-1])
words = {}
for x in docs:
    for i in x.split(): words[model.dictionary.doc2bow([i])[0][0]] = i
[[M.__setitem__((x[0], k), "%0.4f" % x[1]) for x in model[d]] for k, d in enumerate(docs)]
M = np.concatenate((np.atleast_2d([words[x] for x in range(11)]).T, M), axis=1)
pd.DataFrame(M, columns=['Palabras'] + ["Doc. %s" % (x + 1) for x in range(len(docs))])


I am working on the file "03_basics_collections_ex3.ipynb".
The first task is:
3.2)
Can you write Python code for it?

gvars = graph.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)
init_assign_ops = [graph.get_operation_by_name(gvar.op.name + "/Assign")
                   for gvar in gvars]

init_assign_ops


I am working on the file "03_basics_collections_ex3.ipynb".
The first task is:
3.4)
Can you write Python code for it?

init_val_ops = [init_assign_op.inputs[1]
                for init_assign_op in init_assign_ops]

with tf.Session(graph=graph) as sess:
    print(sess.run(init_val_ops))


I am working on the file "03_basics_collections_ex3.ipynb".
The first task is:
Exercise 3

3.1) Create a graph with four variables named "x1", "x2", "x3" and "x4", with initial values 1.0, 2.0, 3.0 and 4.0 respectively, then write some code that prints the name of every operation in the graph.
3.2) Notice that for each Variable, TensorFlow actually created 4 operations:
* the variable itself,
* its initial value,
* an assignment operation to assign the initial value to the variable,
* and a read operation that you can safely ignore for now (for details, check out mrry's great answer to this question).
Get the collection of global variables in the graph, and for each one of them use get_operation_by_name() to find its corresponding /Assign operation (just append "/Assign" to the variable's name).
Hint: each object in the collection of global variables is actually a Tensor, not an Operation (it represents the variable's output, i.e., its value), so its name ends with ":0". You can get the Operation through the Tensor's op attribute: its name will not end with ":0"
3.3) Add a tf.group() to your graph, containing all the assignment operations you got in question 3.2. Congratulations! You have just reimplemented tf.global_variables_initializer().
Start a Session(), run your group operation, then evaluate each variable and print out the result.
3.4) For each assignment operation you fetched earlier, get its second input and store it in a list. Next, start a session and evaluate that list (using sess.run()). Print out the result: you should see [1.0, 2.0, 3.0, 4.0]. Can you guess why?
Try not to peek at the solution below before you have done the exercise! :)

Exercise 3 - Solution
3.1)
Can you write Python code for it?

graph = tf.Graph()
with graph.as_default():
    x1 = tf.Variable(1.0, name="x1")
    x2 = tf.Variable(2.0, name="x2")
    x3 = tf.Variable(3.0, name="x3")
    x4 = tf.Variable(4.0, name="x4")


I am working on the file "03_basics_collections_ex3.ipynb".
The first task is:
From notebook 2 on variables:
Can you write Python code for it?

>>> graph = tf.Graph()
>>> with graph.as_default():
...     x = tf.Variable(100)
...     c = tf.constant(5)
...     increment_op = tf.assign(x, x + c)
...


I am working on the file "03_basics_collections_ex3.ipynb".
The first task is:
Naming Operations
Can you write Python code for it?

>>> graph = tf.Graph()
>>> with graph.as_default():
...     a = tf.constant(3, name='a')
...     b = tf.constant(5, name='b')
...     s = tf.add(a, b, name='s')
...

>>> graph.get_operations()


I am working on the file "03_basics_collections_ex3.ipynb".
The first task is:
Navigating the Graph
Can you write Python code for it?

>>> graph = tf.Graph()
>>> with graph.as_default():
...     a = tf.constant(3)
...     b = tf.constant(5)
...     s = a + b
...
>>> graph.get_operations()

>>> graph.get_operation_by_name("add") is s.op

>>> graph.get_tensor_by_name("add:0") is s

>>> list(s.op.inputs)

>>> list(s.op.outputs)


I am working on the file "03_bootstrap_replicates.ipynb".
The first task is:
For further experiments Sepal Length variable will be used. Additionally, we will focus on bootstrap technique used for mean.
Can you write Python code for it?

# calculating the mean from data
np.mean(sepalLength)


I am working on the file "03_bootstrap_replicates.ipynb".
The first task is:
It would be interesting to repeat call of boostrap_replicate_1d function many times to see how the mean value changes. For that reason another function is defined:
def draw_bs_reps(data, func, size=1):
    """Draw bootstrap replicates."""
    # Initialize array of replicates: bs_replicates
    bs_replicates = np.empty(size)

    # Generate replicates
    for i in range(size):
        bs_replicates[i] = bootstrap_replicate_1d(data, func)

    return bs_replicates

In the example we will call the boostrap_replicate_1d function 10,000 times.
Can you write Python code for it?

bs_replicates = draw_bs_reps(sepalLength, np.mean, size=10000)

# histogram plot will show us how the mean value of sepal length changes when the experiment is repeated 10,000 times
plt.hist(bs_replicates, bins=30, normed=True, edgecolor='black')
plt.xlabel("Mean sepal length [cm]")
plt.ylabel('PDF');

# calculating 95% confidence interval for the mean based on boostrap technique
conf_intervals = np.percentile(bs_replicates, [2.5, 97.5])
conf_intervals


I am working on the file "03_bootstrap_replicates.ipynb".
The first task is:
Use of numpy.random.choice for bootstraping
For the boostraping technique we will use numpy.random.choice function that will randomly select the data from a given variable forming a new data set. Then a function of choice can be used to calculate required statistics. In our case it will be mean value.
The code of the function:
def bootstrap_replicate_1d(data, func):
    '''Generate bootstrap replicate of 1D data'''
    bs_sample = np.random.choice(data, len(data))
    return func(bs_sample)

To show the impact of the function on the mean value, the function is called three times in the cells below:
Can you write Python code for it?

bootstrap_replicate_1d(sepalLength, np.mean)

bootstrap_replicate_1d(sepalLength, np.mean)

bootstrap_replicate_1d(sepalLength, np.mean)


I am working on the file "03_classification.ipynb".
The first task is:
2. Data Augmentation
Can you write Python code for it?

from scipy.ndimage.interpolation import shift

def shift_image(image, dx, dy):
    image = image.reshape((28, 28))
    shifted_image = shift(image, [dy, dx], cval=0, mode="constant")
    return shifted_image.reshape([-1])

image = X_train[1000]
shifted_image_down = shift_image(image, 0, 5)
shifted_image_left = shift_image(image, -5, 0)

plt.figure(figsize=(12,3))
plt.subplot(131)
plt.title("Original", fontsize=14)
plt.imshow(image.reshape(28, 28), interpolation="nearest", cmap="Greys")
plt.subplot(132)
plt.title("Shifted down", fontsize=14)
plt.imshow(shifted_image_down.reshape(28, 28), interpolation="nearest", cmap="Greys")
plt.subplot(133)
plt.title("Shifted left", fontsize=14)
plt.imshow(shifted_image_left.reshape(28, 28), interpolation="nearest", cmap="Greys")
plt.show()

X_train_augmented = [image for image in X_train]
y_train_augmented = [label for label in y_train]

for dx, dy in ((1, 0), (-1, 0), (0, 1), (0, -1)):
    for image, label in zip(X_train, y_train):
        X_train_augmented.append(shift_image(image, dx, dy))
        y_train_augmented.append(label)

X_train_augmented = np.array(X_train_augmented)
y_train_augmented = np.array(y_train_augmented)

shuffle_idx = np.random.permutation(len(X_train_augmented))
X_train_augmented = X_train_augmented[shuffle_idx]
y_train_augmented = y_train_augmented[shuffle_idx]

knn_clf = KNeighborsClassifier(**grid_search.best_params_)

knn_clf.fit(X_train_augmented, y_train_augmented)

y_pred = knn_clf.predict(X_test)
accuracy_score(y_test, y_pred)


I am working on the file "03_classification.ipynb".
The first task is:
And this is the resulting plain text:
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
Binary classifier
Can you write Python code for it?

y_train_5 = (y_train == 5)
y_test_5 = (y_test == 5)

from sklearn.linear_model import SGDClassifier

sgd_clf = SGDClassifier(random_state=42)
sgd_clf.fit(X_train, y_train_5)

sgd_clf.predict([some_digit])

from sklearn.model_selection import cross_val_score
cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring="accuracy")

from sklearn.model_selection import StratifiedKFold
from sklearn.base import clone

skfolds = StratifiedKFold(n_splits=3, random_state=42)

for train_index, test_index in skfolds.split(X_train, y_train_5):
    clone_clf = clone(sgd_clf)
    X_train_folds = X_train[train_index]
    y_train_folds = (y_train_5[train_index])
    X_test_fold = X_train[test_index]
    y_test_fold = (y_train_5[test_index])

    clone_clf.fit(X_train_folds, y_train_folds)
    y_pred = clone_clf.predict(X_test_fold)
    n_correct = sum(y_pred == y_test_fold)
    print(n_correct / len(y_pred))

from sklearn.base import BaseEstimator
class Never5Classifier(BaseEstimator):
    def fit(self, X, y=None):
        pass
    def predict(self, X):
        return np.zeros((len(X), 1), dtype=bool)

never_5_clf = Never5Classifier()
cross_val_score(never_5_clf, X_train, y_train_5, cv=3, scoring="accuracy")

from sklearn.model_selection import cross_val_predict

y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)

from sklearn.metrics import confusion_matrix

confusion_matrix(y_train_5, y_train_pred)

y_train_perfect_predictions = y_train_5

confusion_matrix(y_train_5, y_train_perfect_predictions)

from sklearn.metrics import precision_score, recall_score

precision_score(y_train_5, y_train_pred)

4344 / (4344 + 1307)

recall_score(y_train_5, y_train_pred)

4344 / (4344 + 1077)

from sklearn.metrics import f1_score
f1_score(y_train_5, y_train_pred)

4344 / (4344 + (1077 + 1307)/2)

y_scores = sgd_clf.decision_function([some_digit])
y_scores

threshold = 0
y_some_digit_pred = (y_scores > threshold)

y_some_digit_pred

threshold = 200000
y_some_digit_pred = (y_scores > threshold)
y_some_digit_pred

y_scores =...


y_train_5 = (y_train == 5)
y_test_5 = (y_test == 5)


I am working on the file "03_classification.ipynb".
The first task is:
By simply augmenting the data, we got a 0.5% accuracy boost. :)
3. Tackle the Titanic dataset
The goal is to predict whether or not a passenger survived based on attributes such as their age, sex, passenger class, where they embarked and so on.
First, login to Kaggle and go to the Titanic challenge to download train.csv and test.csv. Save them to the datasets/titanic directory.
Next, let's load the data:
Can you write Python code for it?

import os

TITANIC_PATH = os.path.join("datasets", "titanic")

import pandas as pd

def load_titanic_data(filename, titanic_path=TITANIC_PATH):
    csv_path = os.path.join(titanic_path, filename)
    return pd.read_csv(csv_path)

train_data = load_titanic_data("train.csv")
test_data = load_titanic_data("test.csv")


I am working on the file "03_classification.ipynb".
The first task is:
By simply augmenting the data, we got a 0.5% accuracy boost. :)
3. Tackle the Titanic dataset
The goal is to predict whether or not a passenger survived based on attributes such as their age, sex, passenger class, where they embarked and so on.
Let's load the data:
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
Cool! Now we have a nice preprocessing pipeline that takes the raw data and outputs numerical input features that we can feed to any Machine Learning model we want.
Can you write Python code for it?

X_train = preprocess_pipeline.fit_transform(train_data)
X_train


I am working on the file "03_classification.ipynb".
The first task is:
Exercise solutions
1. An MNIST Classifier With Over 97% Accuracy
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

param_grid = [{'weights': ["uniform", "distance"], 'n_neighbors': [3, 4, 5]}]

knn_clf = KNeighborsClassifier()
grid_search = GridSearchCV(knn_clf, param_grid, cv=5, verbose=3)
grid_search.fit(X_train, y_train)

grid_search.best_params_

grid_search.best_score_

from sklearn.metrics import accuracy_score

y_pred = grid_search.predict(X_test)
accuracy_score(y_test, y_pred)


I am working on the file "03_classification.ipynb".
The first task is:
Extra material
Dummy (ie. random) classifier
Can you write Python code for it?

from sklearn.dummy import DummyClassifier
dmy_clf = DummyClassifier()
y_probas_dmy = cross_val_predict(dmy_clf, X_train, y_train_5, cv=3, method="predict_proba")
y_scores_dmy = y_probas_dmy[:, 1]

fprr, tprr, thresholdsr = roc_curve(y_train_5, y_scores_dmy)
plot_roc_curve(fprr, tprr)


I am working on the file "03_classification.ipynb".
The first task is:
Great! Now let's write a function that takes an email as input and returns its content as plain text, whatever its format is:
Can you write Python code for it?

def email_to_text(email):
    html = None
    for part in email.walk():
        ctype = part.get_content_type()
        if not ctype in ("text/plain", "text/html"):
            continue
        try:
            content = part.get_content()
        except: # in case of encoding issues
            content = str(part.get_payload())
        if ctype == "text/plain":
            return content
        else:
            html = content
    if html:
        return html_to_plain_text(html)

print(email_to_text(sample_html_spam)[:100], "...")


I am working on the file "03_classification.ipynb".
The first task is:
Great, our model is trained, let's use it to make predictions on the test set:
Can you write Python code for it?

X_test = preprocess_pipeline.transform(test_data)
y_pred = svm_clf.predict(X_test)


I am working on the file "03_classification.ipynb".
The first task is:
KNN classifier
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn_clf = KNeighborsClassifier(n_jobs=-1, weights='distance', n_neighbors=4)
knn_clf.fit(X_train, y_train)

y_knn_pred = knn_clf.predict(X_test)

from sklearn.metrics import accuracy_score
accuracy_score(y_test, y_knn_pred)

from scipy.ndimage.interpolation import shift
def shift_digit(digit_array, dx, dy, new=0):
    return shift(digit_array.reshape(28, 28), [dy, dx], cval=new).reshape(784)

plot_digit(shift_digit(some_digit, 5, 1, new=100))

X_train_expanded = [X_train]
y_train_expanded = [y_train]
for dx, dy in ((1, 0), (-1, 0), (0, 1), (0, -1)):
    shifted_images = np.apply_along_axis(shift_digit, axis=1, arr=X_train, dx=dx, dy=dy)
    X_train_expanded.append(shifted_images)
    y_train_expanded.append(y_train)

X_train_expanded = np.concatenate(X_train_expanded)
y_train_expanded = np.concatenate(y_train_expanded)
X_train_expanded.shape, y_train_expanded.shape

knn_clf.fit(X_train_expanded, y_train_expanded)

y_knn_expanded_pred = knn_clf.predict(X_test)

accuracy_score(y_test, y_knn_expanded_pred)

ambiguous_digit = X_test[2589]
knn_clf.predict_proba([ambiguous_digit])

plot_digit(ambiguous_digit)


I am working on the file "03_classification.ipynb".
The first task is:
Let's build the pipeline for the numerical attributes:
Warning: Since Scikit-Learn 0.20, the sklearn.preprocessing.Imputer class was replaced by the sklearn.impute.SimpleImputer class.
Can you write Python code for it?

from sklearn.pipeline import Pipeline
try:
    from sklearn.impute import SimpleImputer # Scikit-Learn 0.20+
except ImportError:
    from sklearn.preprocessing import Imputer as SimpleImputer

num_pipeline = Pipeline([
        ("select_numeric", DataFrameSelector(["Age", "SibSp", "Parch", "Fare"])),
        ("imputer", SimpleImputer(strategy="median")),
    ])

num_pipeline.fit_transform(train_data)


I am working on the file "03_classification.ipynb".
The first task is:
Let's look at one example of ham and one example of spam, to get a feel of what the data looks like:
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
Let's not forget to get the labels:
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
Let's throw in some stemming! For this to work, you need to install the Natural Language Toolkit (NLTK). It's as simple as running the following command (don't forget to activate your virtualenv first; if you don't have one, you will likely need administrator rights, or use the --user option):
$ pip3 install nltk
Can you write Python code for it?

try:
    import nltk

    stemmer = nltk.PorterStemmer()
    for word in ("Computations", "Computation", "Computing", "Computed", "Compute", "Compulsive"):
        print(word, "=>", stemmer.stem(word))
except ImportError:
    print("Error: stemming requires the NLTK module.")
    stemmer = None


I am working on the file "03_classification.ipynb".
The first task is:
Let's try this transformer on a few emails:
Can you write Python code for it?

X_few = X_train[:3]
X_few_wordcounts = EmailToWordCounterTransformer().fit_transform(X_few)
X_few_wordcounts


I am working on the file "03_classification.ipynb".
The first task is:
MNIST
Can you write Python code for it?

from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')
mnist

X, y = mnist["data"], mnist["target"]
X.shape

y.shape

28*28

%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt

some_digit = X[36000]
some_digit_image = some_digit.reshape(28, 28)
plt.imshow(some_digit_image, cmap = matplotlib.cm.binary,
           interpolation="nearest")
plt.axis("off")

save_fig("some_digit_plot")
plt.show()

def plot_digit(data):
    image = data.reshape(28, 28)
    plt.imshow(image, cmap = matplotlib.cm.binary,
               interpolation="nearest")
    plt.axis("off")

# EXTRA
def plot_digits(instances, images_per_row=10, **options):
    size = 28
    images_per_row = min(len(instances), images_per_row)
    images = [instance.reshape(size,size) for instance in instances]
    n_rows = (len(instances) - 1) // images_per_row + 1
    row_images = []
    n_empty = n_rows * images_per_row - len(instances)
    images.append(np.zeros((size, size * n_empty)))
    for row in range(n_rows):
        rimages = images[row * images_per_row : (row + 1) * images_per_row]
        row_images.append(np.concatenate(rimages, axis=1))
    image = np.concatenate(row_images, axis=0)
    plt.imshow(image, cmap = matplotlib.cm.binary, **options)
    plt.axis("off")

plt.figure(figsize=(9,9))
example_images = np.r_[X[:12000:600], X[13000:30600:600], X[30600:60000:590]]
plot_digits(example_images, images_per_row=10)
save_fig("more_digits_plot")
plt.show()

y[36000]

X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]

import numpy as np

shuffle_index = np.random.permutation(60000)
X_train, y_train = X_train[shuffle_index], y_train[shuffle_index]


I am working on the file "03_classification.ipynb".
The first task is:
MNIST
Warning: fetch_mldata() is deprecated since Scikit-Learn 0.20. You should use fetch_openml() instead. However, it returns the unsorted MNIST dataset, whereas fetch_mldata() returned the dataset sorted by target (the training set and the test test were sorted separately). In general, this is fine, but if you want to get the exact same results as before, you need to sort the dataset using the following function:
Can you write Python code for it?

def sort_by_target(mnist):
    reorder_train = np.array(sorted([(target, i) for i, target in enumerate(mnist.target[:60000])]))[:, 1]
    reorder_test = np.array(sorted([(target, i) for i, target in enumerate(mnist.target[60000:])]))[:, 1]
    mnist.data[:60000] = mnist.data[reorder_train]
    mnist.target[:60000] = mnist.target[reorder_train]
    mnist.data[60000:] = mnist.data[reorder_test + 60000]
    mnist.target[60000:] = mnist.target[reorder_test + 60000]

try:
    from sklearn.datasets import fetch_openml
    mnist = fetch_openml('mnist_784', version=1, cache=True, as_frame=False)
    mnist.target = mnist.target.astype(np.int8) # fetch_openml() returns targets as strings
    sort_by_target(mnist) # fetch_openml() returns an unsorted dataset
except ImportError:
    from sklearn.datasets import fetch_mldata
    mnist = fetch_mldata('MNIST original')
mnist["data"], mnist["target"]

mnist.data.shape

X, y = mnist["data"], mnist["target"]
X.shape

y.shape

28*28

some_digit = X[36000]
some_digit_image = some_digit.reshape(28, 28)
plt.imshow(some_digit_image, cmap = mpl.cm.binary,
           interpolation="nearest")
plt.axis("off")

save_fig("some_digit_plot")
plt.show()

def plot_digit(data):
    image = data.reshape(28, 28)
    plt.imshow(image, cmap = mpl.cm.binary,
               interpolation="nearest")
    plt.axis("off")

# EXTRA
def plot_digits(instances, images_per_row=10, **options):
    size = 28
    images_per_row = min(len(instances), images_per_row)
    images = [instance.reshape(size,size) for instance in instances]
    n_rows = (len(instances) - 1) // images_per_row + 1
    row_images = []
    n_empty = n_rows * images_per_row - len(instances)
    images.append(np.zeros((size, size * n_empty)))
    for row in range(n_rows):
        rimages = images[row * images_per_row : (row + 1) * images_per_row]
        row_images.append(np.concatenate(rimages, axis=1))
    image = np.concatenate(row_images, axis=0)
    plt.imshow(image, cmap = mpl.cm.binary, **options)
   ...


I am working on the file "03_classification.ipynb".
The first task is:
Multiclass classification
Can you write Python code for it?

sgd_clf.fit(X_train, y_train)
sgd_clf.predict([some_digit])

some_digit_scores = sgd_clf.decision_function([some_digit])
some_digit_scores

np.argmax(some_digit_scores)

sgd_clf.classes_

sgd_clf.classes_[5]

from sklearn.multiclass import OneVsOneClassifier
ovo_clf = OneVsOneClassifier(SGDClassifier(random_state=42))
ovo_clf.fit(X_train, y_train)
ovo_clf.predict([some_digit])

len(ovo_clf.estimators_)

forest_clf.fit(X_train, y_train)
forest_clf.predict([some_digit])

forest_clf.predict_proba([some_digit])

cross_val_score(sgd_clf, X_train, y_train, cv=3, scoring="accuracy")

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring="accuracy")

y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, cv=3)
conf_mx = confusion_matrix(y_train, y_train_pred)
conf_mx

def plot_confusion_matrix(matrix):
    """If you prefer color and a colorbar"""
    fig = plt.figure(figsize=(8,8))
    ax = fig.add_subplot(111)
    cax = ax.matshow(matrix)
    fig.colorbar(cax)

plt.matshow(conf_mx, cmap=plt.cm.gray)
save_fig("confusion_matrix_plot", tight_layout=False)
plt.show()

row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx / row_sums

np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
save_fig("confusion_matrix_errors_plot", tight_layout=False)
plt.show()

cl_a, cl_b = 3, 5
X_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)]
X_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)]
X_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)]
X_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)]

plt.figure(figsize=(8,8))
plt.subplot(221); plot_digits(X_aa[:25], images_per_row=5)
plt.subplot(222); plot_digits(X_ab[:25], images_per_row=5)
plt.subplot(223); plot_digits(X_ba[:25], images_per_row=5)
plt.subplot(224); plot_digits(X_bb[:25], images_per_row=5)
save_fig("error_analysis_digits_plot")
plt.show()


sgd_clf.fit(X_train, y_train)
sgd_clf.predict([some_digit])

some_digit_scores = sgd_clf.decision_function([some_digit])
some_digit_scores

np.argmax(some_digit_scores)

sgd_clf.classes_

sgd_clf.classes_[5]

from sklearn.multiclass import OneVsOneClassifier
ovo_clf = OneVsOneClassifier(SGDClassifier(max_iter=5, tol=-np.infty, random_state=42))
ovo_clf.fit(X_train, y_train)
ovo_clf.predict([some_digit])

len(ovo_clf.estimators_)

forest_clf.fit(X_train, y_train)
forest_clf.predict([some_digit])

forest_clf.predict_proba([some_digit])

cross_val_score(sgd_clf, X_train, y_train, cv=3, scoring="accuracy")

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))
cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring="accuracy")

y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, cv=3)
conf_mx = confusion_matrix(y_train, y_train_pred)
conf_mx

def plot_confusion_matrix(matrix):
    """If you prefer color and a colorbar"""
    fig = plt.figure(figsize=(8,8))
    ax = fig.add_subplot(111)
    cax = ax.matshow(matrix)
    fig.colorbar(cax)

plt.matshow(conf_mx, cmap=plt.cm.gray)
save_fig("confusion_matrix_plot", tight_layout=False)
plt.show()

row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx / row_sums

np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
save_fig("confusion_matrix_errors_plot", tight_layout=False)
plt.show()

cl_a, cl_b = 3, 5
X_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)]
X_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)]
X_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)]
X_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)]

plt.figure(figsize=(8,8))
plt.subplot(221); plot_digits(X_aa[:25], images_per_row=5)
plt.subplot(222); plot_digits(X_ab[:25], images_per_row=5)
plt.subplot(223); plot_digits(X_ba[:25], images_per_row=5)
plt.subplot(224); plot_digits(X_bb[:25],...


I am working on the file "03_classification.ipynb".
The first task is:
Multilabel classification
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

y_train_large = (y_train >= 7)
y_train_odd = (y_train % 2 == 1)
y_multilabel = np.c_[y_train_large, y_train_odd]

knn_clf = KNeighborsClassifier()
knn_clf.fit(X_train, y_multilabel)

knn_clf.predict([some_digit])


I am working on the file "03_classification.ipynb".
The first task is:
Next, let's load all the emails:
Can you write Python code for it?

HAM_DIR = os.path.join(SPAM_PATH, "easy_ham")
SPAM_DIR = os.path.join(SPAM_PATH, "spam")
ham_filenames = [name for name in sorted(os.listdir(HAM_DIR)) if len(name) > 20]
spam_filenames = [name for name in sorted(os.listdir(SPAM_DIR)) if len(name) > 20]

len(ham_filenames)

len(spam_filenames)


I am working on the file "03_classification.ipynb".
The first task is:
Note: a few hyperparameters will have a different default value in future versions of Scikit-Learn, so a warning is issued if you do not set them explicitly. This is why we set max_iter=5 and tol=-np.infty, to get the same results as in the book, while avoiding the warnings.
Can you write Python code for it?

from sklearn.linear_model import SGDClassifier

sgd_clf = SGDClassifier(max_iter=5, tol=-np.infty, random_state=42)
sgd_clf.fit(X_train, y_train_5)

sgd_clf.predict([some_digit])

from sklearn.model_selection import cross_val_score
cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring="accuracy")

from sklearn.model_selection import StratifiedKFold
from sklearn.base import clone

skfolds = StratifiedKFold(n_splits=3, random_state=42, shuffle=True)

for train_index, test_index in skfolds.split(X_train, y_train_5):
    clone_clf = clone(sgd_clf)
    X_train_folds = X_train[train_index]
    y_train_folds = (y_train_5[train_index])
    X_test_fold = X_train[test_index]
    y_test_fold = (y_train_5[test_index])

    clone_clf.fit(X_train_folds, y_train_folds)
    y_pred = clone_clf.predict(X_test_fold)
    n_correct = sum(y_pred == y_test_fold)
    print(n_correct / len(y_pred))

from sklearn.base import BaseEstimator
class Never5Classifier(BaseEstimator):
    def fit(self, X, y=None):
        pass
    def predict(self, X):
        return np.zeros((len(X), 1), dtype=bool)

never_5_clf = Never5Classifier()
cross_val_score(never_5_clf, X_train, y_train_5, cv=3, scoring="accuracy")

from sklearn.model_selection import cross_val_predict

y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)

from sklearn.metrics import confusion_matrix

confusion_matrix(y_train_5, y_train_pred)

y_train_perfect_predictions = y_train_5

confusion_matrix(y_train_5, y_train_perfect_predictions)

from sklearn.metrics import precision_score, recall_score

precision_score(y_train_5, y_train_pred)

4344 / (4344 + 1307)

recall_score(y_train_5, y_train_pred)

4344 / (4344 + 1077)

from sklearn.metrics import f1_score
f1_score(y_train_5, y_train_pred)

4344 / (4344 + (1077 + 1307)/2)

y_scores = sgd_clf.decision_function([some_digit])
y_scores

threshold = 0
y_some_digit_pred = (y_scores > threshold)

y_some_digit_pred

threshold = 200000
y_some_digit_pred = (y_scores > threshold)
y_some_digit_pred

y_scores =...


I am working on the file "03_classification.ipynb".
The first task is:
Note: there was an issue in Scikit-Learn 0.19.0 (fixed in 0.19.1) where the result of cross_val_predict() was incorrect in the binary classification case when using method="decision_function", as in the code above. The resulting array had an extra first dimension full of 0s. Just in case you are using 0.19.0, we need to add this small hack to work around this issue:
Can you write Python code for it?

y_scores.shape

# hack to work around issue #9589 in Scikit-Learn 0.19.0
if y_scores.ndim == 2:
    y_scores = y_scores[:, 1]

from sklearn.metrics import precision_recall_curve

precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)

def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):
    plt.plot(thresholds, precisions[:-1], "b--", label="Precision", linewidth=2)
    plt.plot(thresholds, recalls[:-1], "g-", label="Recall", linewidth=2)
    plt.xlabel("Threshold", fontsize=16)
    plt.legend(loc="upper left", fontsize=16)
    plt.ylim([0, 1])

plt.figure(figsize=(8, 4))
plot_precision_recall_vs_threshold(precisions, recalls, thresholds)
plt.xlim([-700000, 700000])
save_fig("precision_recall_vs_threshold_plot")
plt.show()

(y_train_pred == (y_scores > 0)).all()

y_train_pred_90 = (y_scores > 70000)

precision_score(y_train_5, y_train_pred_90)

recall_score(y_train_5, y_train_pred_90)

def plot_precision_vs_recall(precisions, recalls):
    plt.plot(recalls, precisions, "b-", linewidth=2)
    plt.xlabel("Recall", fontsize=16)
    plt.ylabel("Precision", fontsize=16)
    plt.axis([0, 1, 0, 1])

plt.figure(figsize=(8, 6))
plot_precision_vs_recall(precisions, recalls)
save_fig("precision_vs_recall_plot")
plt.show()


I am working on the file "03_classification.ipynb".
The first task is:
Note: we set n_estimators=10 to avoid a warning about the fact that its default value will be set to 100 in Scikit-Learn 0.22.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
forest_clf = RandomForestClassifier(n_estimators=10, random_state=42)
y_probas_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3,
                                    method="predict_proba")

y_scores_forest = y_probas_forest[:, 1] # score = proba of positive class
fpr_forest, tpr_forest, thresholds_forest = roc_curve(y_train_5,y_scores_forest)

plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, "b:", linewidth=2, label="SGD")
plot_roc_curve(fpr_forest, tpr_forest, "Random Forest")
plt.legend(loc="lower right", fontsize=16)
save_fig("roc_curve_comparison_plot")
plt.show()

roc_auc_score(y_train_5, y_scores_forest)

y_train_pred_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3)
precision_score(y_train_5, y_train_pred_forest)

recall_score(y_train_5, y_train_pred_forest)


I am working on the file "03_classification.ipynb".
The first task is:
Now let's take a quick look at all the categorical attributes:
Can you write Python code for it?

train_data["Pclass"].value_counts()

train_data["Sex"].value_counts()

train_data["Embarked"].value_counts()


I am working on the file "03_classification.ipynb".
The first task is:
Now we can build the pipeline for the categorical attributes:
Can you write Python code for it?

cat_pipeline = Pipeline([
        ("select_cat", DataFrameSelector(["Pclass", "Sex", "Embarked"])),
        ("imputer", MostFrequentImputer()),
        ("cat_encoder", CategoricalEncoder(encoding='onehot-dense')),
    ])

cat_pipeline.fit_transform(train_data)


I am working on the file "03_classification.ipynb".
The first task is:
Okay, before we learn too much about the data, let's not forget to split it into a training set and a test set:
Can you write Python code for it?

import numpy as np
from sklearn.model_selection import train_test_split

X = np.array(ham_emails + spam_emails)
y = np.array([0] * len(ham_emails) + [1] * len(spam_emails))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)


I am working on the file "03_classification.ipynb".
The first task is:
Okay, over 73% accuracy, clearly better than random chance, but it's not a great score. Looking at the leaderboard for the Titanic competition on Kaggle, you can see that you need to reach above 80% accuracy to be within the top 10% Kagglers. Some reached 100%, but since you can easily find the list of victims of the Titanic, it seems likely that there was little Machine Learning involved in their performance! ;-) So let's try to build a model that reaches 80% accuracy.
Let's try a RandomForestClassifier:
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

forest_clf = RandomForestClassifier(n_estimators=100, random_state=42)
forest_scores = cross_val_score(forest_clf, X_train, y_train, cv=10)
forest_scores.mean()


from sklearn.ensemble import RandomForestClassifier

forest_clf = RandomForestClassifier(random_state=42)
scores = cross_val_score(forest_clf, X_train, y_train, cv=10)
scores.mean()


I am working on the file "03_classification.ipynb".
The first task is:
Okay, the Age, Cabin and Embarked attributes are sometimes null (less than 891 non-null), especially the Cabin (77% are null). We will ignore the Cabin for now and focus on the rest. The Age attribute has about 19% null values, so we will need to decide what to do with them. Replacing null values with the median age seems reasonable.
The Name and Ticket attributes may have some value, but they will be a bit tricky to convert into useful numbers that a model can consume. So for now, we will ignore them.
Let's take a look at the numerical attributes:
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
Over 98.7%, not bad for a first try! :) However, remember that we are using the "easy" dataset. You can try with the harder datasets, the results won't be so amazing. You would have to try multiple models, select the best ones and fine-tune them using cross-validation, and so on.
But you get the picture, so let's stop now, and just print out the precision/recall we get on the test set:
Can you write Python code for it?

from sklearn.metrics import precision_score, recall_score

X_test_transformed = preprocess_pipeline.transform(X_test)

log_clf = LogisticRegression(solver="liblinear", random_state=42)
log_clf.fit(X_train_transformed, y_train)

y_pred = log_clf.predict(X_test_transformed)

print("Precision: {:.2f}%".format(100 * precision_score(y_test, y_pred)))
print("Recall: {:.2f}%".format(100 * recall_score(y_test, y_pred)))


I am working on the file "03_classification.ipynb".
The first task is:
ROC curves
Can you write Python code for it?

from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)

def plot_roc_curve(fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], 'k--')
    plt.axis([0, 1, 0, 1])
    plt.xlabel('False Positive Rate', fontsize=16)
    plt.ylabel('True Positive Rate', fontsize=16)

plt.figure(figsize=(8, 6))
plot_roc_curve(fpr, tpr)
save_fig("roc_curve_plot")
plt.show()

from sklearn.metrics import roc_auc_score

roc_auc_score(y_train_5, y_scores)


from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)

def plot_roc_curve(fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0, 1], [0, 1], 'k--')
    plt.axis([0, 1, 0, 1])
    plt.xlabel('False Positive Rate', fontsize=16)
    plt.ylabel('True Positive Rate', fontsize=16)

plt.figure(figsize=(8, 6))
plot_roc_curve(fpr, tpr)
save_fig("roc_curve_plot")
plt.show()

from sklearn.metrics import roc_auc_score

roc_auc_score(y_train_5, y_scores)

from sklearn.ensemble import RandomForestClassifier
forest_clf = RandomForestClassifier(random_state=42)
y_probas_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3,
                                    method="predict_proba")

y_scores_forest = y_probas_forest[:, 1] # score = proba of positive class
fpr_forest, tpr_forest, thresholds_forest = roc_curve(y_train_5,y_scores_forest)

plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, "b:", linewidth=2, label="SGD")
plot_roc_curve(fpr_forest, tpr_forest, "Random Forest")
plt.legend(loc="lower right", fontsize=16)
save_fig("roc_curve_comparison_plot")
plt.show()

roc_auc_score(y_train_5, y_scores_forest)

y_train_pred_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3)
precision_score(y_train_5, y_train_pred_forest)

recall_score(y_train_5, y_train_pred_forest)


I am working on the file "03_classification.ipynb".
The first task is:
Some emails are actually multipart, with images and attachments (which can have their own attachments). Let's look at the various types of structures we have:
Can you write Python code for it?

def get_email_structure(email):
    if isinstance(email, str):
        return email
    payload = email.get_payload()
    if isinstance(payload, list):
        return "multipart({})".format(", ".join([
            get_email_structure(sub_email)
            for sub_email in payload
        ]))
    else:
        return email.get_content_type()

from collections import Counter

def structures_counter(emails):
    structures = Counter()
    for email in emails:
        structure = get_email_structure(email)
        structures[structure] += 1
    return structures

structures_counter(ham_emails).most_common()

structures_counter(spam_emails).most_common()


I am working on the file "03_classification.ipynb".
The first task is:
That's much better!
Instead of just looking at the mean accuracy across the 10 cross-validation folds, let's plot all 10 scores for each model, along with a box plot highlighting the lower and upper quartiles, and "whiskers" showing the extent of the scores (thanks to Nevin Yilmaz for suggesting this visualization). Note that the boxplot() function detects outliers (called "fliers") and does not include them within the whiskers. Specifically, if the lower quartile is $Q_1$ and the upper quartile is $Q_3$, then the interquartile range $IQR = Q_3 - Q_1$ (this is the box's height), and any score lower than $Q_1 - 1.5 \times IQR$ is a flier, and so is any score greater than $Q3 + 1.5 \times IQR$.
Can you write Python code for it?

plt.figure(figsize=(8, 4))
plt.plot([1]*10, svm_scores, ".")
plt.plot([2]*10, forest_scores, ".")
plt.boxplot([svm_scores, forest_scores], labels=("SVM","Random Forest"))
plt.ylabel("Accuracy", fontsize=14)
plt.show()


I am working on the file "03_classification.ipynb".
The first task is:
That's much better!
To improve this result further, you could:
* Compare many more models and tune hyperparameters using cross validation and grid search,
* Do more feature engineering, for example:
  * replace SibSp and Parch with their sum,
  * try to identify parts of names that correlate well with the Survived attribute (e.g. if the name contains "Countess", then survival seems more likely),
* try to convert numerical attributes to categorical attributes: for example, different age groups had very different survival rates (see below), so it may help to create an age bucket category and use it instead of the age. Similarly, it may be useful to have a special category for people traveling alone since only 30% of them survived (see below).
Can you write Python code for it?

train_data["AgeBucket"] = train_data["Age"] // 15 * 15
train_data[["AgeBucket", "Survived"]].groupby(['AgeBucket']).mean()

train_data["RelativesOnboard"] = train_data["SibSp"] + train_data["Parch"]
train_data[["RelativesOnboard", "Survived"]].groupby(['RelativesOnboard']).mean()


I am working on the file "03_classification.ipynb".
The first task is:
The Embarked attribute tells us where the passenger embarked: C=Cherbourg, Q=Queenstown, S=Southampton.
Now let's build our preprocessing pipelines. We will reuse the DataframeSelector we built in the previous chapter to select specific attributes from the DataFrame:
Can you write Python code for it?

from sklearn.base import BaseEstimator, TransformerMixin

# A class to select numerical or categorical columns
# since Scikit-Learn doesn't handle DataFrames yet
class DataFrameSelector(BaseEstimator, TransformerMixin):
    def __init__(self, attribute_names):
        self.attribute_names = attribute_names
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X[self.attribute_names]


I am working on the file "03_classification.ipynb".
The first task is:
The Embarked attribute tells us where the passenger embarked: C=Cherbourg, Q=Queenstown, S=Southampton.
The CategoricalEncoder class will allow us to convert categorical attributes to one-hot vectors. It will soon be added to Scikit-Learn, and in the meantime you can use the code below (copied from Pull Request #9151).
Can you write Python code for it?

# Definition of the CategoricalEncoder class, copied from PR #9151.
# Just run this cell, or copy it to your code, no need to try to
# understand every line.

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.utils import check_array
from sklearn.preprocessing import LabelEncoder
from scipy import sparse

class CategoricalEncoder(BaseEstimator, TransformerMixin):
    """Encode categorical features as a numeric array.
    The input to this transformer should be a matrix of integers or strings,
    denoting the values taken on by categorical (discrete) features.
    The features can be encoded using a one-hot aka one-of-K scheme
    (``encoding='onehot'``, the default) or converted to ordinal integers
    (``encoding='ordinal'``).
    This encoding is needed for feeding categorical data to many scikit-learn
    estimators, notably linear models and SVMs with the standard kernels.
    Read more in the :ref:`User Guide <preprocessing_categorical_features>`.
    Parameters
    ----------
    encoding : str, 'onehot', 'onehot-dense' or 'ordinal'
        The type of encoding to use (default is 'onehot'):
        - 'onehot': encode the features using a one-hot aka one-of-K scheme
          (or also called 'dummy' encoding). This creates a binary column for
          each category and returns a sparse matrix.
        - 'onehot-dense': the same as 'onehot' but returns a dense array
          instead of a sparse matrix.
        - 'ordinal': encode the features as ordinal integers. This results in
          a single column of integers (0 to n_categories - 1) per feature.
    categories : 'auto' or a list of lists/arrays of values.
        Categories (unique values) per feature:
        - 'auto' : Determine categories automatically from the training data.
        - list : ``categories[i]`` holds the categories expected in the ith
          column. The passed categories are sorted before encoding the data
          (used categories can be found in the ``categories_`` attribute).
    dtype : number type,...


I am working on the file "03_classification.ipynb".
The first task is:
The attributes have the following meaning:
* Survived: that's the target, 0 means the passenger did not survive, while 1 means he/she survived.
* Pclass: passenger class.
* Name, Sex, Age: self-explanatory
* SibSp: how many siblings & spouses of the passenger aboard the Titanic.
* Parch: how many children & parents of the passenger aboard the Titanic.
* Ticket: ticket id
* Fare: price paid (in pounds)
* Cabin: passenger's cabin number
* Embarked: where the passenger embarked the Titanic
Let's get more info to see how much data is missing:
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
The data is already split into a training set and a test set. However, the test data does not contain the labels: your goal is to train the best model you can using the training data, then make your predictions on the test data and upload them to Kaggle to see your final score.
Let's take a peek at the top few rows of the training set:
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
This looks about right!
Now we have the word counts, and we need to convert them to vectors. For this, we will build another transformer whose fit() method will build the vocabulary (an ordered list of the most common words) and whose transform() method will use the vocabulary to convert word counts to vectors. The output is a sparse matrix.
Can you write Python code for it?

from scipy.sparse import csr_matrix

class WordCounterToVectorTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, vocabulary_size=1000):
        self.vocabulary_size = vocabulary_size
    def fit(self, X, y=None):
        total_count = Counter()
        for word_count in X:
            for word, count in word_count.items():
                total_count[word] += min(count, 10)
        most_common = total_count.most_common()[:self.vocabulary_size]
        self.most_common_ = most_common
        self.vocabulary_ = {word: index + 1 for index, (word, count) in enumerate(most_common)}
        return self
    def transform(self, X, y=None):
        rows = []
        cols = []
        data = []
        for row, word_count in enumerate(X):
            for word, count in word_count.items():
                rows.append(row)
                cols.append(self.vocabulary_.get(word, 0))
                data.append(count)
        return csr_matrix((data, (rows, cols)), shape=(len(X), self.vocabulary_size + 1))

vocab_transformer = WordCounterToVectorTransformer(vocabulary_size=10)
X_few_vectors = vocab_transformer.fit_transform(X_few_wordcounts)
X_few_vectors

X_few_vectors.toarray()


I am working on the file "03_classification.ipynb".
The first task is:
To improve this result further, you could:
* Compare many more models and tune hyperparameters using cross validation and grid search,
* Do more feature engineering, for example:
  * replace SibSp and Parch with their sum,
  * try to identify parts of names that correlate well with the Survived attribute (e.g. if the name contains "Countess", then survival seems more likely),
* try to convert numerical attributes to categorical attributes: for example, different age groups had very different survival rates (see below), so it may help to create an age bucket category and use it instead of the age. Similarly, it may be useful to have a special category for people traveling alone since only 30% of them survived (see below).
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
Warning: earlier versions of the book used the LabelBinarizer or CategoricalEncoder classes to convert each categorical value to a one-hot vector. It is now preferable to use the OneHotEncoder class. Since Scikit-Learn 0.20 it can handle string categorical inputs (see PR #10521), not just integer categorical inputs. If you are using an older version of Scikit-Learn, you can import the new version from future_encoders.py:
Can you write Python code for it?

try:
    from sklearn.preprocessing import OrdinalEncoder # just to raise an ImportError if Scikit-Learn < 0.20
    from sklearn.preprocessing import OneHotEncoder
except ImportError:
    from future_encoders import OneHotEncoder # Scikit-Learn < 0.20


I am working on the file "03_classification.ipynb".
The first task is:
Warning: the following cell may take a very long time (possibly hours depending on your hardware).
Can you write Python code for it?

y_train_knn_pred = cross_val_predict(knn_clf, X_train, y_multilabel, cv=3)
f1_score(y_multilabel, y_train_knn_pred, average="macro")


y_train_knn_pred = cross_val_predict(knn_clf, X_train, y_multilabel, cv=3, n_jobs=-1)
f1_score(y_multilabel, y_train_knn_pred, average="macro")


I am working on the file "03_classification.ipynb".
The first task is:
We are now ready to train a classifier. Let's start with an SVC:
Can you write Python code for it?

from sklearn.svm import SVC

svm_clf = SVC()
svm_clf.fit(X_train, y_train)


from sklearn.svm import SVC

svm_clf = SVC(gamma="auto")
svm_clf.fit(X_train, y_train)


I am working on the file "03_classification.ipynb".
The first task is:
We are ready to put all this together into a transformer that we will use to convert emails to word counters. Note that we split sentences into words using Python's split() method, which uses whitespaces for word boundaries. This works for many written languages, but not all. For example, Chinese and Japanese scripts generally don't use spaces between words, and Vietnamese often uses spaces even between syllables. It's okay in this exercise, because the dataset is (mostly) in English.
Can you write Python code for it?

from sklearn.base import BaseEstimator, TransformerMixin

class EmailToWordCounterTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, strip_headers=True, lower_case=True, remove_punctuation=True,
                 replace_urls=True, replace_numbers=True, stemming=True):
        self.strip_headers = strip_headers
        self.lower_case = lower_case
        self.remove_punctuation = remove_punctuation
        self.replace_urls = replace_urls
        self.replace_numbers = replace_numbers
        self.stemming = stemming
    def fit(self, X, y=None):
        return self
    def transform(self, X, y=None):
        X_transformed = []
        for email in X:
            text = email_to_text(email) or ""
            if self.lower_case:
                text = text.lower()
            if self.replace_urls and url_extractor is not None:
                urls = list(set(url_extractor.find_urls(text)))
                urls.sort(key=lambda url: len(url), reverse=True)
                for url in urls:
                    text = text.replace(url, " URL ")
            if self.replace_numbers:
                text = re.sub(r'\d+(?:\.\d*(?:[eE]\d+))?', 'NUMBER', text)
            if self.remove_punctuation:
                text = re.sub(r'\W+', ' ', text, flags=re.M)
            word_counts = Counter(text.split())
            if self.stemming and stemmer is not None:
                stemmed_word_counts = Counter()
                for word, count in word_counts.items():
                    stemmed_word = stemmer.stem(word)
                    stemmed_word_counts[stemmed_word] += count
                word_counts = stemmed_word_counts
            X_transformed.append(word_counts)
        return np.array(X_transformed)


I am working on the file "03_classification.ipynb".
The first task is:
We can use Python's email module to parse these emails (this handles headers, encoding, and so on):
Can you write Python code for it?

import email
import email.policy

def load_email(is_spam, filename, spam_path=SPAM_PATH):
    directory = "spam" if is_spam else "easy_ham"
    with open(os.path.join(spam_path, directory, filename), "rb") as f:
        return email.parser.BytesParser(policy=email.policy.default).parse(f)

ham_emails = [load_email(is_spam=False, filename=name) for name in ham_filenames]
spam_emails = [load_email(is_spam=True, filename=name) for name in spam_filenames]


I am working on the file "03_classification.ipynb".
The first task is:
We will also need a way to replace URLs with the word "URL". For this, we could use hard core regular expressions but we will just use the urlextract library. You can install it with the following command (don't forget to activate your virtualenv first; if you don't have one, you will likely need administrator rights, or use the --user option):
$ pip install urlextract
Can you write Python code for it?

try:
    import urlextract # may require an Internet connection to download root domain names

    url_extractor = urlextract.URLExtract()
    print(url_extractor.find_urls("Will it detect github.com and https://youtu.be/7Pq-S557XQU?t=3m32s"))
except ImportError:
    print("Error: replacing URLs requires the urlextract module.")
    url_extractor = None


I am working on the file "03_classification.ipynb".
The first task is:
We will also need a way to replace URLs with the word "URL". For this, we could use hard core regular expressions but we will just use the urlextract library. You can install it with the following command (don't forget to activate your virtualenv first; if you don't have one, you will likely need administrator rights, or use the --user option):
$ pip3 install urlextract
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
We will also need an imputer for the string categorical columns (the regular Imputer does not work on those):
Can you write Python code for it?

# Inspired from stackoverflow.com/questions/25239958
class MostFrequentImputer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        self.most_frequent = pd.Series([X[c].value_counts().index[0] for c in X],
                                       index=X.columns)
        return self
    def transform(self, X, y=None):
        return X.fillna(self.most_frequent)


I am working on the file "03_classification.ipynb".
The first task is:
What does this matrix mean? Well, the 64 in the third row, first column, means that the third email contains 64 words that are not part of the vocabulary. The 1 next to it means that the first word in the vocabulary is present once in this email. The 2 next to it means that the second word is present twice, and so on. You can look at the vocabulary to know which words we are talking about. The first word is "of", the second word is "and", etc.
Can you write Python code for it?


I am working on the file "03_classification.ipynb".
The first task is:
Yikes, only 38% Survived. :(  That's close enough to 40%, so accuracy will be a reasonable metric to evaluate our model.
The mean Fare was £32.20, which does not seem so expensive (but it was probably a lot of money back then).
The mean Age was less than 30 years old.

Let's check that the target is indeed 0 or 1:
Can you write Python code for it?


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
And if desired, we can get a matrix form of the data with the values property
Can you write Python code for it?

df_categorical_exploded.values

df_categorical_exploded['m35_off'].plot(figsize=(10,5));

kitchen_columns = ['m{}_on'.format(ii) for ii in (15,16,17,18,19,51)]

start = datetime(2009, 2, 2, 10)
end = datetime(2009, 2, 2, 11)

df_categorical_exploded[(df_categorical_exploded.index > start) & (df_categorical_exploded.index < end)][kitchen_columns].plot(figsize=(10,15), subplots=True);

start = datetime(2009, 2, 2, 15)
end = datetime(2009, 2, 2, 17)

df_categorical_exploded[(df_categorical_exploded.index > start) & (df_categorical_exploded.index < end)][kitchen_columns].plot(figsize=(10,15), subplots=True);


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Create a datetime column (currently date and time are separate, and are also strings). The following
should achieve this:
python
df['datetime'] = pd.to_datetime(df[['date', 'time']].apply(lambda row: ' '.join(row), axis=1))
The code above will fail, however, but this is expected behaviour. If we investigate the traceback for the error, we will discover that the final row in the file is from the year 22009. This is a typo, clearly, and it should be from the year 2009, so we will first replace this value, and then parse the dates.
Can you write Python code for it?

#df.ix[df.date.str.startswith('22009'), 'date'] = '2009-02-03'
df.loc[df.date.str.startswith('22009'), 'date'] = '2009-02-03'
df['datetime'] = pd.to_datetime(df[['date', 'time']].apply(lambda row: ' '.join(row), axis=1))

df.dtypes


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Create the output dataframe
Can you write Python code for it?

df_annotation_exploded = pd.get_dummies(df_annotation.annotation)

df_annotation_exploded.head(50)

paired = pd.concat(dfs)

left = paired.index[:-1:2]
right = paired.index[1::2]

print(df_annotation_exploded.mean())
for ll, rr in zip(left, right):
    l = paired.loc[ll]
    r = paired.loc[rr]

    assert l.annotation == r.annotation

    annotation = l.annotation
    begin = l.name
    end = r.name

    # Another advantage of using datetime index: can slice with time ranges
    df_annotation_exploded.loc[begin:end, annotation] = 1

df_annotation_exploded.head(50)


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Download the dataset, and unzip it using the following commands in shell
shell
wget http://casas.wsu.edu/datasets/twor.2009.zip
unzip twor.2009.zip
or directly in python
python
from subprocess import call
call(('wget', url));
call(('unzip', zipfile);
Can you write Python code for it?

#from subprocess import call
#call(('wget', url));
#call(('unzip', zipfile));

from IPython.display import Image
Image("twor.2009/sensorlayout2.png")

column_headings = ('date', 'time', 'sensor', 'value', 'annotation', 'state')

df = pd.read_csv(
    filename,
    delim_whitespace=True,  # Note, the file is delimited by both space and tab characters
    names=column_headings
)

df.head()

df.columns
#df.sensor

df.dtypes

df.time[0]

df['datetime'] = pd.to_datetime(df[['date', 'time']].apply(lambda row: ' '.join(row), axis=1))


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
It's important to ensure that the expected format of the data is consistent in this dataset. This means that there should be
One way to do this is to group by the annotation label and to print some statistics about the data.
Can you write Python code for it?

for annotation, group in df_annotation.groupby('annotation'):
    counts = group.state.value_counts()

    if counts.begin == counts.end:
        print('             {}: equal counts ({}, {})'.format(
            annotation,
            counts.begin,
            counts.end
        ))

    else:
        print(' *** WARNING {}: inconsistent annotation counts with {} begin and {} end'.format(
            annotation,
            counts.begin,
            counts.end
        ))


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Merging categorical and numeric data together
Since we have two dataframes (one for categorical data and one for numeric data). For any analysis, it would be useful to have these two in a unified data frame.
First we will remind ourselves what the dataframes look like, and then these will be unified into one format.
Can you write Python code for it?


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Merging the full dataset
Can you write Python code for it?

dataset = pd.merge(
    df_joined,
    df_annotation_exploded,
    left_index=True,
    right_index=True,
    how='outer'
)

data_cols = df_joined.columns
annotation_cols = df_annotation_exploded.columns

dataset[data_cols] = dataset[data_cols].fillna(0)
dataset[annotation_cols] = dataset[annotation_cols].ffill()

dataset.head()

dataset[data_cols].head()

dataset[annotation_cols].head()

dataset.loc[dataset.Meal_Preparation == 1][kitchen_columns + ['AD1-A']].head()


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Note, however, that since the value data was obtained from file that it is still in string format. We can convert these str data types to floating point data types easily as follows:
python
df_numeric.value.map(float)
However, if we do this, we will discover that there is one record that holds a circumspect value (0.509695F). This is clearly another mistake in the dataset. We can remove the F from the string easily. It is not difficult to make the next bit of code more robust to other data types (eg by applying regular expressions to the strings), but here we will simply slice up until the last character to remove the F:
Can you write Python code for it?

f_inds = df_numeric.value.str.endswith('F')

df_numeric.loc[f_inds, 'value'] = df_numeric.loc[f_inds, 'value'].str[:-1]

df_numeric.loc[f_inds]


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Note, that in merging the dataframes, we now have a time-ordered dataframe. This is one of the advantages of using datetimes as the index type in dataframes since pandas will understand precisely how to merge the two datasets.
Annotations
So far we have extracted the categorical and the numeric data from the dataframe. The annotations/labels have not yet been considered. This section will focus on these data.
Since this part of the notebook will focus only on the annotations, the annotation dataframe will slice these columns exclusively.
Finally, because the annotations have begin and end times, we will need to fill in the time between begin and end with 1's. The ideal output therefore will be that we produce a matrix. Each column of the matrix will correspond to a particular activity. The duration of the activity is captured with a series of 1's. This is similar to the techniques we used to create the categorical dataframe, however, in that we only recorded that a sensor turned on and off, and did not track its continuous state.
Can you write Python code for it?

annotation_inds = pd.notnull(df.annotation)

df_annotation = df.loc[annotation_inds][['annotation', 'state']]

# There are some duplicated indices. Remove with
df_annotation = df_annotation.groupby(level=0).first()

df_annotation.head()


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Numeric columns
We have extracted matrix representation of the categorical data. Now, we will do the same for the numeric data.
Can you write Python code for it?

numeric_inds = df.sensor.str.startswith("A")

df_numeric = df.loc[numeric_inds][['sensor', 'value']]

df_numeric.head()

np.asarray(df_numeric.value)

df_numeric.value.astype(float)


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Our ambition is to create a matrix where each column corresponds to the combinations of sensors and values that are availabele in the data. For example, one column would correspond to the state of M35 being ON, and another column will correspond to M35 being OFF. The reason for having two columns to represent the ON and OFF states is that diffferent information may be achieved by the combinations. For example, a sensor turning on may correspond to somebody entering a room, but correspondingly, a sensor turning off may correspond to somebody leaving the room.
We will achieve the matrix representation by creating a new column that has the sensor and value columns concatenated, and then we will use the get_dummies function provided by pandas to create the representat that we desire.
Can you write Python code for it?

df_categorical.loc[:, 'sensor_value'] = df_categorical[['sensor', 'value']].apply(
    lambda row: '{}_{}'.format(*row).lower(),
    axis=1
)

df_categorical.head()

df_categorical_exploded = pd.get_dummies(df_categorical.sensor_value)

df_categorical_exploded.head()


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
Querying consecutive annotations, we can print the pair of annotations that have
Can you write Python code for it?

def filter_annotations(anns):
    left = iter(anns.index[:-1])
    right = iter(anns.index[1:])

    inds = []
    for ii, (ll, rr) in enumerate(zip(left, right)):
        try:
            l = anns.loc[ll]
            r = anns.loc[rr]

            if l.state == 'begin' and r.state == 'end':
                inds.extend([ll, rr])

        except ValueError:
            print(ii)
            print(l)
            print()
            print(r)
            print()
            print()

            asdf

    return anns.loc[inds, :]

dfs = []
for annotation, group in df_annotation.groupby('annotation'):
    print('{:>30} - {}'.format(annotation, group.size))
    dfs.append(filter_annotations(group))


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
There are only three numeric sensor types, as we can see with the unique member function:
Can you write Python code for it?

unique_keys = df_numeric.sensor.unique()

unique_keys


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
We can now inspect the unique sensor, activity and value values:
Can you write Python code for it?

df.sensor.unique()

df.annotation.unique()

df.state.unique()

df.value.unique()


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
We can now map all data to floating point numbers
Can you write Python code for it?


I am working on the file "03_data_transformation_and_integration/01_wrangling_casas.ipynb".
The first task is:
We can see here that the unique values contains both numbers (eg 2.82231) and strings (ON, OFF). This is because the data recorded by all sensors is contained in one column. The next few steps will be to extract the non-numeric (ie categorical) data from the column.
Extracting the categorical data
We can extract the categorical dataframe (noting that no categorical sensor starts with the letter A):
The regular expression ^[^A] returns true for strings that do not begin with A -- the first ^ is the symbol for the start of the sequence, and the second ^ when contained within square brackets is a 'not match' operation.
Since we only need the sensor name and the value, the data frames that we will deal with in this and the next section will slice these columns only (and doesn't consider the annotation columns).
Can you write Python code for it?

categorical_inds = df.sensor.str.match(r"^[^A]")

df_categorical = df.loc[categorical_inds][['sensor', 'value']]

df_categorical.head()

df_categorical.value.value_counts()

for val in ('O', 'OF', 'OFFF', 'ONF'):
    df_categorical.loc[df_categorical.value == val, 'value'] = 'OFF';

df_categorical.value.value_counts()


I am working on the file "03_data_transformation_and_integration/02_worldbank.ipynb".
The first task is:
GDP per Capita
Data from: http://api.worldbank.org/v2/en/indicator/NY.GDP.PCAP.CD?downloadformat=csv
Can you write Python code for it?

indicator = 'NY.GDP.PCAP.CD' if DOWNLOAD_DATA else 'API_NY.GDP.PCAP.CD_DS2_en_csv_v2_713080'
df3 = get_worldbank_dataframe(indicator)
df3.head()

indicator = 'GDP per capita'
df3r = rearrange_dataframe(df3, indicator)
df3r.sort_index(ascending=True, inplace=True)
df3c = df3r.loc[(slice(None), countries),:]
df3c.head()

plot_indicator(df3c, indicator)

df4 = df1c.merge(df2c, left_index=True, right_index=True).merge(df3c, left_index=True, right_index=True)
df4.head()

x = 'GDP per capita'
y = 'Life expectency at birth'
s = 'Total population'

fig, ax = plt.subplots(figsize=[15 ,10])
for c in countries:
    g = df4.loc['1980', c]
    ax.plot(g[x], g[y], marker='o', linestyle='', ms=g[s] / 1e7, label=c)
lgnd = ax.legend()
for i in range(len(countries)):
    lgnd.legendHandles[i]._legmarker.set_markersize(20)
ax.set_xlabel(x)
ax.set_ylabel(y)

ax.set_xlim([0, df4[x].max()])
ax.set_ylim([df4[y].min()*0.9, df4[y].max()*1.1])
ax.set_title('1980')

def wb_scatter(df, year):
    current_palette = sns.color_palette()
    rng = float(df.index.levels[0].max()) - float(df.index.levels[0].min())
    cur = float(year) - float(df.index.levels[0].min())
    alpha = (cur / rng) / 2
    for i, c in enumerate(countries):
        g = df.loc[year, c]
        ax.plot(g[x], g[y], marker='o', linestyle='', color=current_palette[i], ms=g[s] / 1e7, label=c, alpha=alpha)
    ax.set_xlabel(x)
    ax.set_ylabel(y)
    ax.set_xlim([0, df[x].max()])
    ax.set_ylim([df[y].min()*0.9, df[y].max()*1.1])
    ax.set_title(year)

# Animated version
fig, ax = plt.subplots(figsize=[15 ,10])

do_legend = True

for year in df4.index.levels[0].values:
    wb_scatter(df4, year)

    if do_legend:
        lgnd = ax.legend()
        for i in range(len(countries)):
            lgnd.legendHandles[i]._legmarker.set_markersize(20)
            lgnd.legendHandles[i]._legmarker.set_alpha(1.0)
        do_legend = False

    display.clear_output(wait=True)
    display.display(plt.gcf())
    sleep(0.01)


I am working on the file "03_data_transformation_and_integration/02_worldbank.ipynb".
The first task is:
Total Population
Data from: http://data.worldbank.org/indicator/SP.POP.TOTL
Can you write Python code for it?

indicator = 'SP.POP.TOTL' if DOWNLOAD_DATA else 'API_SP.POP.TOTL_DS2_en_csv_v2_713131'
df2 = get_worldbank_dataframe(indicator)
df2.head()

indicator = 'Total population'
df2r = rearrange_dataframe(df2, indicator)
df2r.sort_index(ascending=True, inplace=True)
df2c = df2r.loc[(slice(None), countries),:]
df2c.head()

plot_indicator(df2c, indicator)


I am working on the file "03_data_transformation_and_integration/04_survey_demo.ipynb".
The first task is:
It seems there are four people who filled out the first form but not the second. And one person who used a different ID on the second form 'MM' instead of 'MJM'...So we can fix that discrepancy and then either omit the four respondess (by using outer join) or included them but with the data from the second form missing (by using inner join).
Can you write Python code for it?

dodgy_ids = ['MM']
replacement_ids = ['MJM']

replace = lambda x: x if x not in dodgy_ids else replacement_ids[dodgy_ids.index(x)]

df1_corrected = df1.assign(NID = [replace(ii) for ii in df1.index]).set_index('NID')
df2_corrected = df2.assign(NID = [replace(ii) for ii in df2.index]).set_index('NID')

#df1.drop(columns=['NID'], inplace=True)
#df2_corrected = df2
#df1_corrected = df1

resultset1_ids = set(df1_corrected.index.unique())
resultset2_ids = set(df2_corrected.index.unique())

intersection = resultset1_ids.intersection(resultset2_ids)
difference = resultset1_ids.difference(resultset2_ids)

print(intersection)
print(difference)


I am working on the file "03_data_transformation_and_integration/04_survey_demo.ipynb".
The first task is:
Let's get the data from the first survey:
Can you write Python code for it?

columns = ['Datetime', 'ID', 'Course', 'Python_Experience', 'Favourite_Language']
df1 = pd.read_csv('Data Fusion.csv', names=columns, skiprows=1)

df1.set_index('ID', inplace=True)
df1.head()

len(df1)

df1.dtypes
#type(df1.Datetime[0])

df1['Datetime'] = pd.to_datetime(df1['Datetime'])

df1.index.is_unique

## if non-unique idex we can test inspect the duplicate indicies
indices = df1.groupby(level=0)

duplicate_indices = []

for index, gi in indices:
    if len(gi) > 1:
        print("Repeated index: ", index)
        duplicate_indices.append(gi)

duplicate_indices[0]

df1 = df1.loc[~df1.index.duplicated(keep='first')]
print(df1.index.is_unique)
df1.head()

df1.index.is_unique


I am working on the file "03_data_transformation_and_integration/04_survey_demo.ipynb".
The first task is:
Now that the data sets are fused we can interogate them and continue the exploration...
Can you write Python code for it?

plt.figure(figsize=(12,6))
plt.subplot(1,2,1)
plt.scatter(joined.Python_Experience, joined.Ethics_Importance)
plt.xlabel('Python_Experience')
plt.ylabel('Ethics_Importance');

plt.subplot(1,2,2)
plt.scatter(joined.Python_Experience, joined.Female_Developers)
plt.xlabel('Python_Experience')
plt.ylabel('Female_Developers');

joined.Favourite_Language.value_counts()

joined.Favourite_Movie.value_counts()

gi = 6
groups = list(joined.groupby('Course'))

print("There are %d courses." %len(groups))

plt.title(groups[gi][0])
groups[gi][1].Favourite_Language.value_counts().plot.pie()
plt.axis('off');

joined.loc[joined['Course'] == 'Digital Health']
#joined[joined[['Course']] == 'Digital Health']#[['Favourite_Movie']]

def plot_movie_pie(group_on='Course', gi='Computer Science'):

    plt.title(gi)
    joined.loc[joined[group_on] == gi, 'Favourite_Movie'].value_counts().plot.pie()
    plt.axis('off');

plot_movie_pie() #gi='Engineering Mathematics')


I am working on the file "03_data_transformation_and_integration/04_survey_demo.ipynb".
The first task is:
We need to deal with the duplication in the Course variable. We take the most recently input value of 'Course' (if not null).
Can you write Python code for it?

joined.loc[joined.Course_x != joined.Course_y]

most_recent = lambda row: row.Course_x if ((row.Datetime_x > row.Datetime_y) & (pd.notnull(row.Course_x))) else row.Course_y
joined = joined.assign(Course = [ci if ci==cj else most_recent(joined.iloc[ii]) for ii, (ci,cj) in enumerate(zip(joined.Course_x, joined.Course_y))])

joined.drop(columns=['Course_x', 'Course_y'], inplace=True)
joined.head()

joined.Course.value_counts()


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
A derivatives position consists of "data only" and not instantiated model or valuation objects. The necessary model and valuation objects are instantiated during the portfolio valuation.
Can you write Python code for it?

put = derivatives_position(
                name='put',  # name of position
                quantity=1,  # number of instruments
                underlyings=['gbm'],  # relevant risk factors
                mar_env=me,  # market environment
                otype='American single',  # the option type
                payoff_func='np.maximum(40. - instrument_values, 0)')
                    # the payoff funtion


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Based on this new risk factor model object, a European call option is defined.
Can you write Python code for it?

call_jump = derivatives_position(
                name='call_jump',
                quantity=3,
                underlyings=['jd'],
                mar_env=me_jump,
                otype='European single',
                payoff_func='np.maximum(maturity_value - 36., 0)')