I am working on the file "Section 2 - Advance Python/Chapter S2.11 Multiprocessing and Threading/threading-tutorial.ipynb".
The first task is:
The two functions worker_with() and worker_no_with() manager the lock in equivalent ways.
$ python threading_lock_with.py
(Thread-1  ) Lock acquired via with
(Thread-2  ) Lock acquired directly
Synchronizing Threads
In addition to using Events, another way of synchronizing threads is through using a Condition object. Because the Condition uses a Lock, it can be tied to a shared resource. This allows threads to wait for the resource to be updated. In this example, the consume() threads wait() for the Condition to be set before continuing. The producer() thread is responsible for setting the condition and notifying the other threads that they can continue.
Can you write Python code for it?

# %load threading_condition.py
import logging
import threading
import time

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s (%(threadName)-2s) %(message)s',
                    )

def consumer(cond):
    """wait for the condition and use the resource"""
    logging.debug('Starting consumer thread')
    t = threading.currentThread()
    with cond:
        cond.wait()
        logging.debug('Resource is available to consumer')

def producer(cond):
    """set up the resource to be used by the consumer"""
    logging.debug('Starting producer thread')
    with cond:
        logging.debug('Making resource available')
        cond.notifyAll()

condition = threading.Condition()
c1 = threading.Thread(name='c1', target=consumer, args=(condition,))
c2 = threading.Thread(name='c2', target=consumer, args=(condition,))
p = threading.Thread(name='p', target=producer, args=(condition,))

c1.start()
time.sleep(2)
c2.start()
time.sleep(2)
p.start()


I am working on the file "Section 2 - Advance Python/Chapter S2.11 Multiprocessing and Threading/threading-tutorial.ipynb".
The first task is:
Waiting for the daemon thread to exit using join() means it has a chance to produce its "Exiting" message.
$ python threading_daemon_join.py
(daemon    ) Starting
(non-daemon) Starting
(non-daemon) Exiting
(daemon    ) Exiting
By default, join() blocks indefinitely. It is also possible to pass a timeout argument (a float representing the number of seconds to wait for the thread to become inactive). If the thread does not complete within the timeout period, join() returns anyway.
Can you write Python code for it?

# %load threading_daemon_join_timeout.py
import threading
import time
import logging

logging.basicConfig(level=logging.DEBUG,
                    format='(%(threadName)-10s) %(message)s',
                    )

def daemon():
    logging.debug('Starting')
    time.sleep(2)
    logging.debug('Exiting')

d = threading.Thread(name='daemon', target=daemon)
d.setDaemon(True)

def non_daemon():
    logging.debug('Starting')
    logging.debug('Exiting')

t = threading.Thread(name='non-daemon', target=non_daemon)

d.start()
t.start()

d.join(1)
print('d.isAlive()', d.isAlive())
t.join()
print('d.isAlive()', d.isAlive())


I am working on the file "Section 2 - Advance Python/Chapter S2.12 - Weak Reference, Copy/Weak Reference.ipynb".
The first task is:
Reference Callbacks
The ref constructor accepts an optional callback function that is invoked when the referenced object is deleted.
Can you write Python code for it?

import weakref

class ExpensiveObject:

    def __del__(self):
        print('(Deleting {})'.format(self))

def callback(reference):
    """Invoked when referenced object is deleted"""
    print('callback({!r})'.format(reference))

obj = ExpensiveObject()
r = weakref.ref(obj, callback)

print('obj:', obj)
print('ref:', r)
print('r():', r())

print('deleting obj')
del obj
print('r():', r())


I am working on the file "Section 2 - Advance Python/Chapter S2.12 - Weak Reference, Copy/Weak Reference.ipynb".
The first task is:
Weak Reference:
Non-permanent References to Objects
What is Strong reference
When memory is referenced by a variable, its called strong reference. What that means is that untill all the strong references are removed from the memory location, it is not removed.
Can you write Python code for it?

a = 101
b = a
print(a, id(a))
print(b, id(b))

del(a)
# print(a, id(a))
print(b, id(b))

del(b)
# print(a, id(a))
# print(b, id(b))

import sys

class MayaHello:
    def __init__(self, lang):
        self.lang = lang

    def hello(self):
        if(self.lang == 'hi'):
            return "नमस्ते"
        elif(self.lang == 'de'):
            return "Hallo"
        else:
            return ("Hello")

    def __del__(self):
        print("self destruct initiated")
        del self.lang

a = MayaHello('hi')
print(a.hello())
sys.stdout.flush()
b = a
print(b.hello())
sys.stdout.flush()

print(a.hello())


I am working on the file "Section 2 - Advance Python/Chapter S2.12 - Weak Reference, Copy/copy.ipynb".
The first task is:
Deep Copies
The deep copy created by deepcopy() is a new container populated with copies of the contents of the original object. To make a deep copy of a list, a new list is constructed, the elements of the original list are copied, and then those copies are appended to the new list.
Replacing the call to copy() with deepcopy() makes the difference in the output apparent.
Can you write Python code for it?

import copy

class MyTry:
    def __init__(self):
        self.lst = [1,2,3,4,5]

a = MyTry()
dup = copy.deepcopy(a)
a.lst.append(6)
print(a.lst, dup.lst)
print(id(a), id(dup))
print(id(a.lst), id(dup.lst))

lst = [1,2,3,4,5]
dup_lst = copy.deepcopy(lst)
print(id(lst), id(dup_lst))

import copy

class MyTry:
    def __init__(self):
        self.lst = [1,2,3,4,[5]]

a = MyTry()
dup = copy.deepcopy(a)
a.lst.append(6)
print(a.lst, dup.lst)
print(id(a), id(dup))
print(id(a.lst), id(dup.lst))


I am working on the file "Section 2 - Advance Python/Chapter S2.14 - Memory Management/01. Garbage Collection - Introduction.ipynb".
The first task is:
The data of good lives even after we deleted it ...

lets take another example, to validate it further
The living proof
Can you write Python code for it?

a = ["test"]
b = []
b.append(a)
print(id(a))
print(id(b[0]))
print(b)
del(a)
print(id(b[0]))
print(b)

import gc
print(gc.garbage)
del(b)
gc.collect()
print(gc.garbage)

import gc
print ("Garbage collection thresholds: ", gc.get_threshold())


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
!!! Easter Eggs !!!
Can you write Python code for it?

from __future__ import braces

import __hello__


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
!!! ouch moment !!!
Can you write Python code for it?

ori = [1, 2, 3, [4], 5, 6]
dup = ori[:]
print(id(ori))
print(id(dup))

ori[3].append(10)
print(ori)
print(dup)
print(id(ori[3]))
print(id(dup[4]))
print(id(ori))
print(id(dup))


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Array Transpose using zip
Can you write Python code for it?


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Both the variables are still pointing to same list, thus change in one will change another also.
Can you write Python code for it?


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Descriptor
http://users.rcn.com/python/download/Descriptor.htm
Iterators
iter() can take a callable argument
Can you write Python code for it?

def seek_next_line(f):
    """
    The iter(callable, until_value) function repeatedly calls
    callable and yields its result until until_value is returned.
    """
    for c in iter(lambda: f.read(1),'\n'):
        pass


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Dictionaries
Reverse the key values in unique dictionary
Can you write Python code for it?

states_capitals = {'MP': 'Bhopal', 'UP': 'Lucknow', 'Rajasthan': 'Jaipur'}


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Exception
Re-raising exceptions:
Can you write Python code for it?

# Python 2 syntax
try:
    some_operation()
except SomeError, e:
    if is_fatal(e):
        raise
    handle_nonfatal(e)

def some_operation():
    raise Exception

def is_fatal(e):
    return True

# Python 3 syntax
try:
    some_operation()
except Exception as e:
    if is_fatal(e):
        raise
    handle_nonfatal(e)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Finally returns the ultimate return
Can you write Python code for it?

def dum_dum():
    try:
        return '`dum dum` returning from try'
    finally:
        return '`dum dum` returning from finally'

print(dum_dum())


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Function arguments
Can you write Python code for it?

def letsEcho():
    test = "Hello"
    print(test)

letsEcho.test = "Welcome"
print(letsEcho.test)
letsEcho()
print(letsEcho.test)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Functions
default arguments
Dangerous mutable default arguments
Can you write Python code for it?

def foo(x=[]):
    x.append(1)
    print(x)

foo()
foo()
foo()

# instead use:
def fun(x=None):
    if x is None:
        x = []
    x.append(1)
    print(x)

fun()
fun()
fun()


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Generators
Sending values into generator functions
https://www.python.org/dev/peps/pep-0342/, also please reaad http://www.dabeaz.com/coroutines/
Can you write Python code for it?

def mygen():
    """Yield 5 until something else is passed back via send()"""
    a = 5
    while True:
        f = (yield a) #yield a and possibly get f in return
        if f is not None:
            a = f  #store the new value

g = mygen()
print(next(g))
print(next(g))
g.send(7)
print(next(g))
print(next(g))
g.send(17)
print(next(g))
print(next(g))


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
I/O
with
open multiple files in a single with.
Can you write Python code for it?

try:
    with open('a', 'w') as a, open('b', 'w') as b:
        pass
except IOError as e:
    print ('Operation failed: %s' % e.strerror)

#### write file using `print`

with open("outfile.txt" , "w+") as outFile:
    print('Modern Standard Hindi is a standardised and sanskritised register of the Hindustani language.', file=outFile)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Infinite Recursion
Can you write Python code for it?


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Integer
Negative round
round is a function to round off the numbers and its normal usage is as follows
Can you write Python code for it?


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Join list of strings
Can you write Python code for it?

list_cities = ["Bhopal", "New Delhi", "Agra", "Mumbai", "Aligarh", "Hyderabad"]

# Lets join the list of string in string using `join`
str_cities = ", ".join(list_cities)
print(str_cities)

list_cities = ("Bhopal", "New Delhi", "Agra", "Mumbai", "Aligarh", "Hyderabad")

# Lets join the list of string in string using `join`
str_cities = ", ".join(list_cities)
print(str_cities)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Lets encrypt our code using cot13
Can you write Python code for it?

import codecs
s   = 'The Zen of Python, by Tim Peters'
enc = codecs.getencoder( "rot-13" )
dec = codecs.getdecoder("rot-13")
os  = enc( s )[0]
print(os)
print(dec(os)[0])

import this


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Lets update code to handle this situation
Can you write Python code for it?

flattened_list = [si for i in l1 for si in (i if isinstance(i, list) else [i])]
print(flattened_list)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
List / Tuple
tuple / list unpacking
Can you write Python code for it?

a, b, *remaining = (1, 2, 3, 4, 5, "test")
print(a, b)
print(remaining)

a, b, *remaining = [1, 2, 3, 4, 5, "test"]
print(a, b)
print(remaining)

first,*middle,last = (1, 2, 3, 4, 5, 6, 7, 8)

print(first, last)
print(middle)

first,*middle,last = [1, 2, 3, 4, 5, 6, 7, 8]

print(first, last)
print(middle)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Method 1:
Can you write Python code for it?

capitals_states = dict(zip(*list(zip(*states_capitals.items()))[::-1]))
print(capitals_states)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Method 2:
Can you write Python code for it?

capitals_states = dict([v, k] for k, v in states_capitals.items())
print(capitals_states)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Method 3:
Can you write Python code for it?

capitals_states = dict(zip(states_capitals.values(), states_capitals.keys()))
print(capitals_states)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Method 4:
Can you write Python code for it?

capitals_states = {states_capitals[k] : k for k in states_capitals}
print(capitals_states)


flattened_list = []
for x in l:
    for y in x:
        flattened_list.append(y)
print(flattened_list)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Method 5:
Can you write Python code for it?

from functools import reduce
flattened_list = reduce(lambda x, y: x + y, l)
print(flattened_list)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Method 6:
Can you write Python code for it?

import operator
flattened_list = reduce(operator.add, l)
print(flattened_list)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
NOTE: this method will fail if any of the element is non list item as shown in the below example
Can you write Python code for it?


flattened_list = [y for x in l1 for y in x]
print(flattened_list)


flattened_list = []
for x in l1:
    for y in x:
        flattened_list.append(y)
print(flattened_list)


from itertools import chain
try:
    flattened_list = list(chain(*l1))
    print(flattened_list)
except:
    print("Error !!!")

from itertools import chain

flattened_list = list(chain(*l2))
print(flattened_list)


import operator
flattened_list = reduce(operator.add, l1)
print(flattened_list)


flattened_list = reduce(lambda x, y: x + y, l1)
print(flattened_list)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
OOPS
Attributes
Dynamically added attributes
Can you write Python code for it?

class Test():
    def __getattribute__(self, name):
        f = lambda: " ".join([name, name[::-1]])
        return f

t = Test()
# New attribute created at runtime
t.rev()


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Reverse the list
built-in keyword reversed allows the list to be reversed.
Can you write Python code for it?

lst = [1, 2, 3, 4, 53]
print(list(reversed(lst)))

print(lst[::-1])


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
Search substring in string
Can you write Python code for it?

print("ash" in "ashwini")

print("ash" is ['a', 's', 'h'])

print("ash" is ('a', 's', 'h'))

print("ash" is 'ash')

### Implicit concatenation without "+" operator

name = "Mayank" " " "Johri"
print(name)

try:
    name = "Mayank" " " "Johri" ' .'
    print(name)
except SyntaxError:
    pass


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
String
Multi line strings
In python we can have multiple ways to achieve multi line strings.

Using triple quotes
Can you write Python code for it?

txt = """The Supreme Lord said: The indestructible, transcendental living
entity is called Brahman  and his eternal nature is called the
self. Action pertaining to the development of these  material
bodies is called karma, or fruitive activities."""

print(txt)


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
The second parameter defines the decimal number to which the number to rounded of. But if we provide a -ve number to it then it starts rounding of the number itself instead of decimal digit as shown in the below example
Can you write Python code for it?


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
operators
Chaining comparison operators
Can you write Python code for it?


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
pow power - pow() can calculate (x ** y) % z
Can you write Python code for it?


I am working on the file "Section 2 - Advance Python/Chapter S2.901 - Python Hidden Gems/Python Hidden Treasure.ipynb".
The first task is:
rescue using Deep copy
Can you write Python code for it?

from copy import deepcopy
ori = [1, 2, 3, [4], 5, 6]
dup = deepcopy(ori)
print(ori)
print(dup)
print(id(ori[3]))
print(id(dup[3]))
print(id(ori))
print(id(dup))
ori[3].append(10)
print(ori)
print(dup)
print(id(ori[3]))
print(id(dup[3]))
print(id(ori))
print(id(dup))


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/2 Regression Analysis.ipynb".
The first task is:
Linear Algorithms

Linear Regression
Logistic Regression
Gradient Descent
Linear Discriminant Analysis

Nonlinear Algorithms

Classification And Regression Trees
Naive Bayes
K-Nearest Neighbors
Learning Vector Quantization
Support Vector Machines
Bagging and Random Forest
Boosting and AdaBoost

Linear Regression
Linear regression is an modeling approach to describe the relationship between a scalar dependent variable Y and one or more explanatory variables (or independent variables) denoted X.
In simple linear regression only one explanatory variable is present. In multiple linear regression more than one explanatory variable are present.
In linear regression, the relationships are modeled using linear predictor functions whose unknown model parameters are estimated from the data and are called linear models.
Example of linear regression

Why use linear regression?

widely used
runs fast
easy to use (not a lot of tuning required)
highly interpretable
basis for many other methods

Important Points:

There must be linear relationship between independent and dependent variables
Multiple regression suffers from multicollinearity, autocorrelation, heteroskedasticity.
Linear Regression is very sensitive to Outliers. It can terribly affect the regression line and eventually the forecasted values.
Multicollinearity can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable
In case of multiple independent variables, we can go with forward selection, backward elimination and step wise approach for selection of most significant independent variables.

Simple Linear Regression
Multiple Linear Regression
Polynomial Regression
A regression equation is a polynomial regression equation if the power of independent variable is more than 1. The equation below represents a polynomial equation:
y=a+b*x^2
In this regression technique, the best fit line is not a straight line. It is rather a curve that...
Can you write Python code for it?

from sklearn import datasets
from sklearn.model_selection import cross_val_predict
from sklearn import linear_model
import matplotlib.pyplot as plt

lr = linear_model.LinearRegression()
boston = datasets.load_boston()
y = boston.target

# cross_val_predict returns an array of the same size as `y` where each entry
# is a prediction obtained by cross validation:
predicted = cross_val_predict(lr, boston.data, y, cv=10)

fig, ax = plt.subplots()
ax.scatter(y, predicted)
ax.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=4)
ax.set_xlabel('Measured')
ax.set_ylabel('Predicted')
plt.show()


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Classification/1. Linear Classifiers - support vector machines.ipynb".
The first task is:
Example
Can you write Python code for it?

print(__doc__)

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets

def make_meshgrid(x, y, h=.02):
    """Create a mesh of points to plot in

    Parameters
    ----------
    x: data to base x-axis meshgrid on
    y: data to base y-axis meshgrid on
    h: stepsize for meshgrid, optional

    Returns
    -------
    xx, yy : ndarray
    """
    x_min, x_max = x.min() - 1, x.max() + 1
    y_min, y_max = y.min() - 1, y.max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    return xx, yy

def plot_contours(ax, clf, xx, yy, **params):
    """Plot the decision boundaries for a classifier.

    Parameters
    ----------
    ax: matplotlib axes object
    clf: a classifier
    xx: meshgrid ndarray
    yy: meshgrid ndarray
    params: dictionary of params to pass to contourf, optional
    """
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    out = ax.contourf(xx, yy, Z, **params)
    return out

# import some data to play with
iris = datasets.load_iris()
# Take the first two features. We could avoid this by using a two-dim dataset
X = iris.data[:, :2]
y = iris.target

# we create an instance of SVM and fit out data. We do not scale our
# data since we want to plot the support vectors
C = 1.0  # SVM regularization parameter
models = (svm.SVC(kernel='linear', C=C),
          svm.LinearSVC(C=C),
          svm.SVC(kernel='rbf', gamma=0.7, C=C),
          svm.SVC(kernel='poly', degree=3, C=C))
models = (clf.fit(X, y) for clf in models)

# title for the plots
titles = ('SVC with linear kernel',
          'LinearSVC (linear kernel)',
          'SVC with RBF kernel',
          'SVC with polynomial (degree 3) kernel')

# Set-up 2x2 grid for plotting.
fig, sub = plt.subplots(2, 2)
plt.subplots_adjust(wspace=0.4, hspace=0.4)

X0, X1 = X[:, 0], X[:, 1]
xx, yy = make_meshgrid(X0, X1)

for clf, title, ax in zip(models, titles, sub.flatten()):
    plot_contours(ax, clf, xx, yy,
                 ...


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Classification/1. Linear Classifiers - support vector machines.ipynb".
The first task is:
RBF kernel (Radial Basis Function)
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Classification/1. Linear Classifiers - support vector machines.ipynb".
The first task is:
There are several support vector machine implementations in scikit-learn . The most commonly used ones
are svm.SVC , svm.NuSVC and svm.LinearSVC ; “SVC” stands for Support Vector Classifier.
Using kernels
Classes are not always separable by a hyperplane, so it would be desirable to have a decision function that is
not linear but that may be for instance polynomial or exponential:
Linear kernel
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Classification/3. Gaussian Naive Bayes Classification.ipynb".
The first task is:
Quantitative Measurement of Performance
Can you write Python code for it?

matches = (predicted == expected)
print(matches)

print(matches.sum())
print(len(matches))
qmp = matches.sum() / float(len(matches))
print(qmp)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Classification/4. Model Selection via Validation - Comparing the models.ipynb".
The first task is:
Hyperparameter optimization with cross-validation
Can you write Python code for it?

from sklearn.datasets import load_diabetes
data = load_diabetes()
X, y = data.data, data.target
print(X.shape)

from sklearn.linear_model import Ridge, Lasso

for Model in [Ridge, Lasso]:
    model = Model()
    print('%s: %s' % (Model.__name__, cross_val_score(model, X, y).mean()))


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Classification/4. Model Selection via Validation - Comparing the models.ipynb".
The first task is:
We can use different splitting strategies, such as random splitting (There exists many different cross-validation strategies in scikit-learn. They are often useful to take in account non iid datasets)
Can you write Python code for it?

from sklearn.model_selection import ShuffleSplit
cv = ShuffleSplit(n_splits=5)
cross_val_score(clf, X, y, cv=cv)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Classification/7. Decision Trees.ipynb".
The first task is:
After being fitted, the model can then be used to predict the class of samples:
Can you write Python code for it?

clf.predict(iris.data[:1, :])

clf.predict_proba(iris.data[:1, :])


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Classification/7. Decision Trees.ipynb".
The first task is:
Regression
Decision trees can also be applied to regression problems, using the DecisionTreeRegressor class.
As in the classification setting, the fit method will take as argument arrays X and y, only that in this case y is expected to have floating point values instead of integer values:
Can you write Python code for it?

from sklearn import tree
X = [[0, 0], [2, 2]]
y = [0.5, 2.5]
clf = tree.DecisionTreeRegressor()
clf = clf.fit(X, y)
clf.predict([[1, 1]])

print(__doc__)

# Import the necessary modules and libraries
import numpy as np
from sklearn.tree import DecisionTreeRegressor
import matplotlib.pyplot as plt

# Create a random dataset
rng = np.random.RandomState(1)
X = np.sort(5 * rng.rand(80, 1), axis=0)
y = np.sin(X).ravel()
y[::5] += 3 * (0.5 - rng.rand(16))

# Fit regression model
regr_1 = DecisionTreeRegressor(max_depth=3)
regr_2 = DecisionTreeRegressor(max_depth=9)
regr_1.fit(X, y)
regr_2.fit(X, y)

# Predict
X_test = np.arange(0.0, 5.0, 0.01)[:, np.newaxis]
y_1 = regr_1.predict(X_test)
y_2 = regr_2.predict(X_test)

# Plot the results
plt.figure()
plt.scatter(X, y, s=20, edgecolor="black",
            c="darkorange", label="data")
plt.plot(X_test, y_1, color="cornflowerblue",
         label="max_depth=2", linewidth=2)
plt.plot(X_test, y_2, color="yellowgreen", label="max_depth=5", linewidth=2)
plt.xlabel("data")
plt.ylabel("target")
plt.title("Decision Tree Regression")
plt.legend()
plt.show()

# Quadratic training set + noise
np.random.seed(42)
m = 200
X = np.random.rand(m, 1)
y = 4 * (X - 0.5) ** 2
y = y + np.random.randn(m, 1) / 10

# To support both python 2 and python 3
from __future__ import division, print_function, unicode_literals

# Common imports
import numpy as np
import os

# to make this notebook's output stable across runs
np.random.seed(42)

# To plot pretty figures
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12

# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "decision_trees"

def image_path(fig_id):
    return os.path.join("files", fig_id)

def save_fig(fig_id, tight_layout=True):
    print("Saving figure", fig_id)
    if tight_layout:
        plt.tight_layout()
    plt.savefig(image_path(fig_id) + ".png", format='png', dpi=300)

from sklearn.tree...


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
Consider Storing All Observations
Generally, it is a good idea to keep track of all the observations.
This will allow you to:
Provide a context for further time series analysis to understand new changes in the data.
Train a new model in the future on the most recent data.
Back-test new and different models to see if performance can be improved.
For small applications, perhaps you could store the raw observations in a file alongside your model.
It may also be desirable to store the model coefficients and required lag data and last observation in plain text for easy review.
For larger applications, perhaps a database system could be used to store the observations.
Example 3: Old Faithful Geyser
Old Faithful Geyser Data
Description:
 Waiting time between eruptions and the duration of the eruption
 for the Old Faithful geyser in Yellowstone National Park, Wyoming,
 USA.

 A data frame with observations on 2 variables.

eruptions  numeric  Eruption time in mins
waiting    numeric  Waiting time to next eruption
References:
 Hardle, W. (1991) Smoothing Techniques with Implementation in S.
 New York: Springer.

 Azzalini, A. and Bowman, A. W. (1990). A look at some data on the
 Old Faithful geyser. Applied Statistics 39, 357-365.
Can you write Python code for it?

# Required Packages
import csv
import sys
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn import datasets, linear_model

def get_data(file_name):
    """ Get Data from CSV file

    Reads the csv file and returns the lists of attributes.
    """
    data = pd.read_csv(file_name).sort_values("eruptions")
    print(type(data))
    print(data)
    x_parameter = []
    y_parameter = []
    for x, y in zip(data['eruptions'],data['waiting']):
        x_parameter.append([float(x)])
        y_parameter.append(float(y))
    return x_parameter, y_parameter

A = get_data("../files/OldFaithfulGeyserData.csv")

plt.figure()
plt.plot(A[0], A[1])
plt.grid(True)
plt.show()

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(A[0], A[1])

dist = 3
print ('waiting time will be %.2f sec' % model.predict([[dist]]))


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
Finalize and Save Time Series Forecast Model
Once the model is selected, we must finalize it.
This means save the salient information learned by the model so that we do not have to re-create it every time a prediction is needed.
This involves first training the model on all available data and then saving the model to file.
The statsmodels implementations of time series models do provide built-in capability to save and load models by calling save() and load() on the fit ARResults object.
For example, the code below will train an AR(6) model on the entire Female Births dataset and save it using the built-in save() function, which will essentially pickle the ARResults object.
The differenced training data must also be saved, both the for the lag variables needed to make a prediction, and for knowledge of the number of observations seen, required by the predict() function of the ARResults object.
Finally, we need to be able to transform the differenced dataset back into the original form. To do this, we must keep track of the last actual observation. This is so that the predicted differenced value can be added to it.
Can you write Python code for it?

# fit an AR model and save the whole model to file
from pandas import Series
from statsmodels.tsa.ar_model import AR
from statsmodels.tsa.ar_model import ARResults
import numpy

# create a difference transform of the dataset
def difference(dataset):
	diff = list()
	for i in range(1, len(dataset)):
		value = dataset[i] - dataset[i - 1]
		diff.append(value)
	return numpy.array(diff)

# load dataset
series = pd.read_csv('../files/daily-total-female-births-in-cal.csv', header=0)
series=series.set_index('Date')

X = difference(series.values)
# fit model
model = AR(X)
model_fit = model.fit(maxlag=6, disp=False)
# save model to file
model_fit.save('ar_model.pkl')
# save the differenced dataset
numpy.save('ar_data.npy', X)
# save the last ob
numpy.save('ar_obs.npy', [series.values[-1]])


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
Model Selection
You must select a model.
This is where the bulk of the effort will be in preparing the data, performing analysis, and ultimately selecting a model and model hyperparameters that best capture the relationships in the data.
In this case, we can arbitrarily select an autoregression model (AR) with a lag of 6 on the differenced dataset.
We can demonstrate this model below.
First, the data is transformed by differencing, with each observation transformed as:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
Next, the AR(6) model is trained on 66% of the historical data. The regression coefficients learned by the model are extracted and used to make predictions in a rolling manner across the test dataset.
As each time step in the test dataset is executed, the prediction is made using the coefficients and stored. The actual observation for the time step is then made available and stored to be used as a lag variable for future predictions.
Can you write Python code for it?

from pandas import Series
from matplotlib import pyplot
from statsmodels.tsa.ar_model import AR
from sklearn.metrics import mean_squared_error
import numpy

# create a difference transform of the dataset
def difference(dataset):
	diff = list()
	for i in range(1, len(dataset)):
		value = dataset[i] - dataset[i - 1]
		diff.append(value)
	return numpy.array(diff)

# Make a prediction give regression coefficients and lag obs
def predict(coef, history):
	yhat = coef[0]
	for i in range(1, len(coef)):
		yhat += coef[i] * history[-i]
	return yhat

series = pd.read_csv('../files/daily-total-female-births-in-cal.csv', header=0)
series=series.set_index('Date')

# split dataset
X = difference(series.values)

size = int(len(X) * 0.66)
train, test = X[0:size], X[size:]
# train autoregression
model = AR(train)
model_fit = model.fit(maxlag=6, disp=False)
window = model_fit.k_ar
coef = model_fit.params
# walk forward over time steps in test
history = [train[i] for i in range(len(train))]
predictions = list()
for t in range(len(test)):
	yhat = predict(coef, history)
	obs = test[t]
	predictions.append(yhat)
	history.append(obs)
error = mean_squared_error(test, predictions)
print('Test MSE: %.3f' % error)
# plot
pyplot.plot(test)
pyplot.plot(predictions, color='red')
pyplot.show()


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
So, you see that their is a bit of error in our predicted values from the original values.
Example 2: Daily Female Births in California
This dataset describes the number of daily female births in California in 1959. The units are a count and there are 365 observations. The source of the dataset is credited to Newton (1988).
URL: https://datamarket.com/data/set/235k/daily-total-female-births-in-california-1959#!ds=235k&display=line
Step 1 - Load the dataset
Can you write Python code for it?

from pandas import Series
from matplotlib import pyplot
series = pd.read_csv('../files/daily-total-female-births-in-cal.csv', header=0)
series=series.set_index('Date')
print(series.tail())
series.plot()
plt.show()


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
The coefficients are saved in the local file man_model.npy, the lag history is saved in the file man_data.npy, and the last observation is saved in the file man_obs.npy.
These values can then be loaded again as follows:
Can you write Python code for it?

# load the manually saved model from file
import numpy
coef = numpy.load('man_model.npy')
print(coef)
lag = numpy.load('man_data.npy')
print(lag)
last_ob = numpy.load('man_obs.npy')
print(last_ob)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
Update Forecast Model
Our work is not done.
Once the next real observation is made available, we must update the data associated with the model.
Specifically, we must update:
The differenced training dataset used as inputs to make the subsequent prediction.
The last observation, providing a context for the predicted differenced value.
Let’s assume the next actual observation in the series was 48.
The new observation must first be differenced with the last observation. It can then be stored in the list of differenced observations. Finally, the value can be stored as the last observation.
In the case of the stored AR model, we can update the ar_data.npy and ar_obs.npy files. The complete example is listed below:
Can you write Python code for it?

# update the data for the AR model with a new obs
import numpy
# get real observation
observation = 48
# load the saved data
data = numpy.load('ar_data.npy')
last_ob = numpy.load('ar_obs.npy')
# update and save differenced observation
diffed = observation - last_ob[0]
data = numpy.append(data, [diffed], axis=0)
numpy.save('ar_data.npy', data)
# update and save real observation
last_ob[0] = observation
numpy.save('ar_obs.npy', last_ob)

# update the data for the manual model with a new obs
import numpy
# get real observation
observation = 48
# update and save differenced observation
lag = numpy.load('man_data.npy')
last_ob = numpy.load('man_obs.npy')
diffed = observation - last_ob[0]
lag = numpy.append(lag[1:], [diffed], axis=0)
numpy.save('man_data.npy', lag)
# update and save real observation
last_ob[0] = observation
numpy.save('man_obs.npy', last_ob)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
We can also use a similar trick to load the raw coefficients and make a manual prediction.
The complete example is listed below.
Can you write Python code for it?

# load a coefficients and from file and make a manual prediction
import numpy

def predict(coef, history):
	yhat = coef[0]
	for i in range(1, len(coef)):
		yhat += coef[i] * history[-i]
	return yhat

# load model
coef = numpy.load('man_model.npy')
lag = numpy.load('man_data.npy')
last_ob = numpy.load('man_obs.npy')
# make prediction
prediction = predict(coef, lag)
# transform prediction
yhat = prediction + last_ob[0]
print('Prediction: %f' % yhat)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
lets predict the service charge
Can you write Python code for it?

dist = 11
print ('The service charge should be: Rs %.2f' % model.predict([[dist]]))

dist = 2
print ('The service charge should be: Rs %.2f' % model.predict([[dist]]))
dist = 9
print ('The service charge should be: Rs %.2f' % model.predict([[dist]]))


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
training data set
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/1.1 Simple Linear Regression.ipynb".
The first task is:
we can also verify that as the distance is increasing the price is also increasing.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
model = LinearRegression()
model.fit(D, C)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
1.5.2 Data Preprocessing

Create dummy variables, that is variables with only two values, zero and one.

In logistic regression models, encoding all of the independent variables as dummy variables allows easy interpretation and calculation of the odds ratios, and increases the stability and significance of the coefficients.
Can you write Python code for it?

data2 = pd.get_dummies(data, columns =['job', 'marital', 'default', 'housing', 'loan', 'poutcome'])


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
1.6.2 Predicting the test set results and creating confusion matrix
The confusion_matrix() function will calculate a confusion matrix and return the result as an array.
Can you write Python code for it?

y_pred = classifier.predict(X_test)
from sklearn.metrics import confusion_matrix
confusion_matrix = confusion_matrix(y_test, y_pred)
print(confusion_matrix)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
1.6.4 Compute precision, recall, F-measure and support
To quote from Scikit Learn:
The precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier to not label a sample as positive if it is negative.
The recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.
The F-beta score can be interpreted as a weighted harmonic mean of the precision and recall, where an F-beta score reaches its best value at 1 and worst score at 0.
The F-beta score weights the recall more than the precision by a factor of beta. beta = 1.0 means recall and precision are equally important.
The support is the number of occurrences of each class in y_test.
Can you write Python code for it?

from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
As you can see, the PCA has reduced the accuracy of our Logistic Regression model. This is because we use PCA to reduce the amount of the dimension, so we have removed information from our data.
Can you write Python code for it?

## Example 2: Predicting affair

# Import modules
import numpy as np
import pandas as pd
import statsmodels.api as sm
import matplotlib.pyplot as plt
from patsy import dmatrices
from sklearn.linear_model import LogisticRegression
from sklearn.cross_validation import train_test_split
from sklearn import metrics
from sklearn.cross_validation import cross_val_score


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Data Pre-Processing
Can you write Python code for it?

# load dataset
dta = sm.datasets.fair.load_pandas().data

# add "affair" column: 1 represents having affairs, 0 represents not
dta['affair'] = (dta.affairs > 0).astype(int)

dta.groupby('affair').mean()

dta.groupby('rate_marriage').mean()


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Data Visualization
Can you write Python code for it?

# show plots in the notebook
%matplotlib inline

# histogram of education
dta.educ.hist()
plt.title('Histogram of Education')
plt.xlabel('Education Level')
plt.ylabel('Frequency')

# histogram of marriage rating
dta.rate_marriage.hist()
plt.title('Histogram of Marriage Rating')
plt.xlabel('Marriage Rating')
plt.ylabel('Frequency')

# barplot of marriage rating grouped by affair (True or False)
pd.crosstab(dta.rate_marriage, dta.affair.astype(bool)).plot(kind='bar')
plt.title('Marriage Rating Distribution by Affair Status')
plt.xlabel('Marriage Rating')
plt.ylabel('Frequency')


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Drop the unknown columns
Can you write Python code for it?

data2.drop(data2.columns[[12, 16, 18, 21, 24]], axis=1, inplace=True)
data2.columns


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Interpretation: Of the entire test set, 88% of the promoted term deposit were the term deposit that the customers liked. Of the entire test set, 90% of the customer’s preferred term deposits that were promoted.
1.6.5 Classifier visualization playground
Can you write Python code for it?

from sklearn.decomposition import PCA
X = data2.iloc[:,1:]
y = data2.iloc[:,0]
pca = PCA(n_components=2).fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(pca, y, random_state=0)

plt.figure(dpi=120)
plt.scatter(pca[y.values==0,0], pca[y.values==0,1], alpha=0.5, label='YES', s=2, color='navy')
plt.scatter(pca[y.values==1,0], pca[y.values==1,1], alpha=0.5, label='NO', s=2, color='red')
plt.legend()
plt.title('Bank Marketing Data Set\nFirst Two Principal Components')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.gca().set_aspect('equal')
plt.show()

def plot_bank(X, y, fitted_model):
    plt.figure(figsize=(9.8,5), dpi=100)
    for i, plot_type in enumerate(['Decision Boundary', 'Decision Probabilities']):
        plt.subplot(1,2,i+1)
        mesh_step_size = 0.01  # step size in the mesh
        x_min, x_max = X[:, 0].min() - .1, X[:, 0].max() + .1
        y_min, y_max = X[:, 1].min() - .1, X[:, 1].max() + .1
        xx, yy = np.meshgrid(np.arange(x_min, x_max, mesh_step_size), np.arange(y_min, y_max, mesh_step_size))
        if i == 0:
            Z = fitted_model.predict(np.c_[xx.ravel(), yy.ravel()])
        else:
            try:
                Z = fitted_model.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:,1]
            except:
                plt.text(0.4, 0.5, 'Probabilities Unavailable', horizontalalignment='center',
                     verticalalignment='center', transform = plt.gca().transAxes, fontsize=12)
                plt.axis('off')
                break
        Z = Z.reshape(xx.shape)
        plt.scatter(X[y.values==0,0], X[y.values==0,1], alpha=0.8, label='YES', s=5, color='navy')
        plt.scatter(X[y.values==1,0], X[y.values==1,1], alpha=0.8, label='NO', s=5, color='red')
        plt.imshow(Z, interpolation='nearest', cmap='RdYlBu_r', alpha=0.15,
                   extent=(x_min, x_max, y_min, y_max), origin='lower')
        plt.title(plot_type + '\n' +
                  str(fitted_model).split('(')[0]+ ' Test Accuracy: ' + str(np.round(fitted_model.score(X, y), 5)))
   ...


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Let's use a stacked barplot to look at the percentage of women having affairs by number of years of marriage.
Can you write Python code for it?

affair_yrs_married = pd.crosstab(dta.yrs_married, dta.affair.astype(bool))
affair_yrs_married.div(affair_yrs_married.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Affair Percentage by Years Married')
plt.xlabel('Years Married')
plt.ylabel('Percentage')


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Logistic Regression
Can you write Python code for it?

# instantiate a logistic regression model, and fit with X and y
model = LogisticRegression()
model = model.fit(X, y)

# check the accuracy on the training set
model.score(X, y)

# what percentage had affairs?
y.mean()


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Only 32% of the women had affairs, which means that you could obtain 68% accuracy by always predicting "no". So we're doing better than the null error rate, but not by much.
Model Evaluation Using a Validation Set
Can you write Python code for it?

# evaluate the model by splitting into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
model2 = LogisticRegression()
model2.fit(X_train, y_train)

# predict class labels for the test set
predicted = model2.predict(X_test)
print (predicted)

# generate class probabilities
probs = model2.predict_proba(X_test)
print (probs)

# generate evaluation metrics
print (metrics.accuracy_score(y_test, predicted))
print (metrics.roc_auc_score(y_test, probs[:, 1]))

print (metrics.confusion_matrix(y_test, predicted))
print (metrics.classification_report(y_test, predicted))


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Prepare Data for Logistic Regression
To prepare the data, I want to add an intercept column as well as dummy variables for occupation and occupation_husb, since I'm treating them as categorial variables. The dmatrices function from the patsy module can do that using formula language.
Can you write Python code for it?

# create dataframes with an intercept column and dummy variables for
# occupation and occupation_husb
y, X = dmatrices('affair ~ rate_marriage + age + yrs_married + children + \
                  religious + educ + C(occupation) + C(occupation_husb)',
                  dta, return_type="dataframe")
print(X.columns)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
Split the data into training and test sets
Can you write Python code for it?

X = data2.iloc[:,1:]
y = data2.iloc[:,0]
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
The column names for the dummy variables are ugly, so let's rename those.
Can you write Python code for it?

# fix column names of X
X = X.rename(columns = {'C(occupation)[T.2.0]':'occ_2',
                        'C(occupation)[T.3.0]':'occ_3',
                        'C(occupation)[T.4.0]':'occ_4',
                        'C(occupation)[T.5.0]':'occ_5',
                        'C(occupation)[T.6.0]':'occ_6',
                        'C(occupation_husb)[T.2.0]':'occ_husb_2',
                        'C(occupation_husb)[T.3.0]':'occ_husb_3',
                        'C(occupation_husb)[T.4.0]':'occ_husb_4',
                        'C(occupation_husb)[T.5.0]':'occ_husb_5',
                        'C(occupation_husb)[T.6.0]':'occ_husb_6'})


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/2. Logistic Regression.ipynb".
The first task is:
We also need to flatten y into a 1-D array, so that scikit-learn will properly understand it as the response variable.
Can you write Python code for it?

# flatten y into a 1-D array
y = np.ravel(y)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/3. Ridge Regression.ipynb".
The first task is:
Here we can clearly observe that as the value of alpha increases, the model complexity reduces. Though higher values of alpha reduce overfitting, significantly high values can cause underfitting as well (eg. alpha = 5). Thus alpha should be chosen wisely. A widely accept technique is cross-validation, i.e. the value of alpha is iterated over a range of values and the one giving higher cross-validation score is chosen.
Can you write Python code for it?

#Set the display format to be scientific for ease of analysis
pd.options.display.float_format = '{:,.2g}'.format
coef_matrix_ridge


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/3. Ridge Regression.ipynb".
The first task is:
It is clearly evident that the size of coefficients increase exponentially with increase in model complexity. I hope this gives some intuition into why putting a constraint on the magnitude of coefficients can be a good idea to reduce model complexity.
Ridge Regression
As mentioned before, ridge regression performs ‘L2 regularization‘, i.e. it adds a factor of sum of squares of coefficients in the optimization objective. Thus, ridge regression optimizes the following:
Objective = RSS + α * (sum of square of coefficients)
Here, α (alpha) is the parameter which balances the amount of emphasis given to minimizing RSS vs minimizing sum of square of coefficients. α can take various values:

α = 0:
The objective becomes same as simple linear regression.
We’ll get the same coefficients as simple linear regression.

α = ∞:
The coefficients will be zero. Why? Because of infinite weightage on square of coefficients, anything less than zero will make the objective infinite.

0 < α < ∞:
The magnitude of α will decide the weightage given to different parts of objective.
The coefficients will be somewhere between 0 and ones for simple linear regression.

OR:
It works like linear regression, except that it prevents the polynomial's coefficients to explode. By adding a regularization term in the loss function, ridge regression imposes some structure on the underlying model.
The ridge regression model has a meta-parameter which represents the weight of the regularization term. We could try different values with trials and errors, using the Ridge class. However, scikit-learn includes another model called RidgeCV which includes a parameter search with cross-validation. In practice, it means that we don't have to tweak this parameter by hand: scikit-learn does it for us. Since the models of scikit-learn always follow the fit-predict API, all we have to do is replace lm.LinearRegression by lm.RidgeCV in the code above. We will give more details in the next section.
Can you write Python code for it?

import numpy as np
import scipy.stats as st
import sklearn.linear_model as lm
import matplotlib.pyplot as plt
%matplotlib inline

f = lambda x: np.exp(3 * x)
x_tr = np.linspace(0., 2, 200)
y_tr = f(x_tr)

x = np.array([0, .1, .2, .5, .8, .9, 1])
y = f(x) + np.random.randn(len(x))
plt.figure(figsize=(6,3));
plt.plot(x_tr[:100], y_tr[:100], '--k');
plt.plot(x, y, 'ok', ms=10);

# We create the model.
lr = lm.LinearRegression()
# We train the model on our training dataset.
lr.fit(x[:, np.newaxis], y);
# Now, we predict points with our trained model.
y_lr = lr.predict(x_tr[:, np.newaxis])
plt.figure(figsize=(6,3));
plt.plot(x_tr, y_tr, '--k');
plt.plot(x_tr, y_lr, 'g');
plt.plot(x, y, 'ok', ms=10);
plt.xlim(0, 1);
plt.ylim(y.min()-1, y.max()+1);
plt.title("Linear regression");

lrp = lm.LinearRegression()
plt.figure(figsize=(6,3));
plt.plot(x_tr, y_tr, '--k');

for deg, s in zip([2, 5], ['-', '.']):
    lrp.fit(np.vander(x, deg + 1), y);
    y_lrp = lrp.predict(np.vander(x_tr, deg + 1))
    plt.plot(x_tr, y_lrp, s, label='degree ' + str(deg));
    plt.legend(loc=2);
    plt.xlim(0, 1.4);
    plt.ylim(-10, 40);
    # Print the model's coefficients.
    print(' '.join(['%.2f' % c for c in lrp.coef_]))
plt.plot(x, y, 'ok', ms=10);
plt.title("Linear regression");

ridge = lm.RidgeCV()
plt.figure(figsize=(6,3));
plt.plot(x_tr, y_tr, '--k');

for deg, s in zip([2, 5], ['-', '.']):
    ridge.fit(np.vander(x, deg + 1), y);
    y_ridge = ridge.predict(np.vander(x_tr, deg + 1))
    plt.plot(x_tr, y_ridge, s, label='degree ' + str(deg));
    plt.legend(loc=2);
    plt.xlim(0, 1.5);
    plt.ylim(-5, 80);
    # Print the model's coefficients.
    print(' '.join(['%.2f' % c for c in ridge.coef_]))

plt.plot(x, y, 'ok', ms=10);
plt.title("Ridge regression");

from sklearn.linear_model import Ridge

def ridge_regression(data, predictors, alpha, models_to_plot={}):
    #Fit the model
    ridgereg = Ridge(alpha=alpha,normalize=True)
    ridgereg.fit(data[predictors],data['y'])
    y_pred = ridgereg.predict(data[predictors])

...


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/3. Ridge Regression.ipynb".
The first task is:
Lasso Regression
LASSO stands for Least Absolute Shrinkage and Selection Operator. There are 2 key words of interest ‘absolute‘ and ‘selection‘.
Lets consider the former first and worry about the latter later.
Lasso regression performs L1 regularization, i.e. it adds a factor of sum of absolute value of coefficients in the optimization objective. Thus, lasso regression optimizes the following:
Objective = RSS + α * (sum of absolute value of coefficients)
Here, α (alpha) works similar to that of ridge and provides a trade-off between balancing RSS and magnitude of coefficients. Like that of ridge, α can take various values. Lets iterate it here briefly:

α = 0: Same coefficients as simple linear regression
α = ∞: All coefficients zero (same logic as before)
0 < α < ∞: coefficients between 0 and that of simple linear regression
Can you write Python code for it?

from sklearn.linear_model import Lasso
def lasso_regression(data, predictors, alpha, models_to_plot={}):
    #Fit the model
    lassoreg = Lasso(alpha=alpha,normalize=True, max_iter=1e5)
    lassoreg.fit(data[predictors],data['y'])
    y_pred = lassoreg.predict(data[predictors])

    #Check if a plot is to be made for the entered alpha
    if alpha in models_to_plot:
        plt.subplot(models_to_plot[alpha])
        plt.tight_layout()
        plt.plot(data['x'],y_pred)
        plt.plot(data['x'],data['y'],'.')
        plt.title('Plot for alpha: %.3g'%alpha)

    #Return the result in pre-defined format
    rss = sum((y_pred-data['y'])**2)
    ret = [rss]
    ret.extend([lassoreg.intercept_])
    ret.extend(lassoreg.coef_)
    return ret

#Initialize predictors to all 15 powers of x
predictors=['x']
predictors.extend(['x_%d'%i for i in range(2,16)])

#Define the alpha values to test
alpha_lasso = [1e-15, 1e-10, 1e-8, 1e-5,1e-4, 1e-3,1e-2, 1, 5, 10]

#Initialize the dataframe to store coefficients
col = ['rss','intercept'] + ['coef_x_%d'%i for i in range(1,16)]
ind = ['alpha_%.2g'%alpha_lasso[i] for i in range(0,10)]
coef_matrix_lasso = pd.DataFrame(index=ind, columns=col)

#Define the models to plot
models_to_plot = {1e-10:231, 1e-5:232,1e-4:233, 1e-3:234, 1e-2:235, 1:236}

#Iterate over the 10 alpha values:
for i in range(10):
    coef_matrix_lasso.iloc[i,] = lasso_regression(data, predictors, alpha_lasso[i], models_to_plot)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/3. Ridge Regression.ipynb".
The first task is:
Note that this function will not plot the model fit for all the powers but will return the residual sum of squares (RSS) and coefficients for all the models. I’ll skip the details of the code for now to maintain brevity. I’ll be happy to discuss the same through comments below if required.
Now, we can make all 15 models and compare the results. For ease of analysis, we’ll store all the results in a Pandas dataframe and plot 6 models to get an idea of the trend. Consider the following code:
Can you write Python code for it?

#Initialize a dataframe to store the results:
col = ['rss','intercept'] + ['coef_x_%d'%i for i in range(1,16)]
ind = ['model_pow_%d'%i for i in range(1,16)]
coef_matrix_simple = pd.DataFrame(index=ind, columns=col)

#Define the powers for which a plot is required:
models_to_plot = {1:231,3:232,6:233,9:234,12:235,15:236}

#Iterate through all powers and assimilate results
for i in range(1,16):
    coef_matrix_simple.iloc[i-1,0:i+2] = linear_regression(data, power=i, models_to_plot=models_to_plot)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/3. Ridge Regression.ipynb".
The first task is:
Note the ‘Ridge’ function used here. It takes ‘alpha’ as a parameter on initialization. Also, keep in mind that normalizing the inputs is generally a good idea in every type of regression and should be used in case of ridge regression as well.
Now, lets analyze the result of Ridge regression for 10 different values of α ranging from 1e-15 to 20. These values have been chosen so that we can easily analyze the trend with change in values of α. These would however differ from case to case.
Note that each of these 10 models will contain all the 15 variables and only the value of alpha would differ. This is different from the simple linear regression case where each model had a subset of features.
Can you write Python code for it?

#Initialize predictors to be set of 15 powers of x
predictors=['x']
predictors.extend(['x_%d'%i for i in range(2,16)])

#Set the different values of alpha to be tested
alpha_ridge = [1e-15, 1e-10, 1e-8, 1e-4, 1e-3,1e-2, 1, 5, 10, 20]

#Initialize the dataframe for storing coefficients.
col = ['rss','intercept'] + ['coef_x_%d'%i for i in range(1,16)]
ind = ['alpha_%.2g'%alpha_ridge[i] for i in range(0,10)]
coef_matrix_ridge = pd.DataFrame(index=ind, columns=col)

models_to_plot = {1e-15:231, 1e-10:232, 1e-4:233, 1e-3:234, 1e-2:235, 5:236}
for i in range(10):
    coef_matrix_ridge.iloc[i,] = ridge_regression(data, predictors, alpha_ridge[i], models_to_plot)


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/3. Ridge Regression.ipynb".
The first task is:
Now that we have all the 15 powers, lets make 15 different linear regression models with each model containing variables with powers of x from 1 to the particular model number. For example, the feature set of model 8 will be – {x, x_2, x_3, … ,x_8}.
First, we’ll define a generic function which takes in the required maximum power of x as an input and returns a list containing – [ model RSS, intercept, coef_x, coef_x2, … upto entered power ]. Here RSS refers to ‘Residual Sum of Squares’ which is nothing but the sum of square of errors between the predicted and actual values in the training data set. The python code defining the function is:
Can you write Python code for it?

#Import Linear Regression model from scikit-learn.
from sklearn.linear_model import LinearRegression
def linear_regression(data, power, models_to_plot):
    #initialize predictors:
    predictors=['x']
    if power>=2:
        predictors.extend(['x_%d'%i for i in range(2,power+1)])

    #Fit the model
    linreg = LinearRegression(normalize=True)
    linreg.fit(data[predictors],data['y'])
    y_pred = linreg.predict(data[predictors])

    #Check if a plot is to be made for the entered power
    if power in models_to_plot:
        plt.subplot(models_to_plot[power])
        plt.tight_layout()
        plt.plot(data['x'],y_pred)
        plt.plot(data['x'],data['y'],'.')
        plt.title('Plot for power: %d'%power)

    #Return the result in pre-defined format
    rss = sum((y_pred-data['y'])**2)
    ret = [rss]
    ret.extend([linreg.intercept_])
    ret.extend(linreg.coef_)
    return ret


I am working on the file "Section 3 - Machine Learning/Supervised Learning Algorithm/Regression Analysis/3. Ridge Regression.ipynb".
The first task is:
This straight away gives us the following inferences:

The RSS increases with increase in alpha, this model complexity reduces
An alpha as small as 1e-15 gives us significant reduction in magnitude of coefficients. How? Compare the coefficients in the first row of this table to the last row of simple linear regression table.
High alpha values can lead to significant underfitting. Note the rapid increase in RSS for values of alpha greater than 1
Though the coefficients are very very small, they are NOT zero.
The first 3 are very intuitive. But #4 is also a crucial observation. Let’s reconfirm the same by determining the number of zeros in each row of the coefficients data set:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/03_getting_started_with_iris.ipynb".
The first task is:
Each value we are predicting is the response (also known as: target, outcome, label, dependent variable)
Classification is supervised learning in which the response is categorical
Regression is supervised learning in which the response is ordered and continuous

Requirements for working with data in scikit-learn

Features and response are separate objects
Features and response should be numeric
Features and response should be NumPy arrays
Features and response should have specific shapes
Can you write Python code for it?

# check the types of the features and response
print(type(iris.data))
print(type(iris.target))

# check the shape of the features (first dimension = number of observations, second dimensions = number of features)
print(iris.data.shape)

# check the shape of the response (single dimension matching the number of observations)
print(iris.target.shape)

# store feature matrix in "X"
X = iris.data

# store response vector in "y"
y = iris.target


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/03_getting_started_with_iris.ipynb".
The first task is:
Machine learning terminology

Each row is an observation (also known as: sample, example, instance, record)
Each column is a feature (also known as: predictor, attribute, independent variable, input, regressor, covariate)
Can you write Python code for it?

# print the names of the four features
print(iris.feature_names)

# print integers representing the species of each observation
print(iris.target)

# print the encoding scheme for species: 0 = setosa, 1 = versicolor, 2 = virginica
print(iris.target_names)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
150 observations
4 features (sepal length, sepal width, petal length, petal width)
Response variable is the iris species
Classification problem since response is categorical
More information in the UCI Machine Learning Repository

K-nearest neighbors (KNN) classification

Pick a value for K.
Search for the K observations in the training data that are "nearest" to the measurements of the unknown iris.
Use the most popular response value from the K nearest neighbors as the predicted response value for the unknown iris.

Example training data

KNN classification map (K=1)

KNN classification map (K=5)

Image Credits: Data3classes, Map1NN, Map5NN by Agor153. Licensed under CC BY-SA 3.0
Loading the data
Can you write Python code for it?

# import load_iris function from datasets module
from sklearn.datasets import load_iris

# save "bunch" object containing iris dataset and its attributes
iris = load_iris()

# store feature matrix in "X"
X = iris.data

# store response vector in "y"
y = iris.target

# print the shapes of X and y
print(X.shape)
print(y.shape)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
Resources

Nearest Neighbors (user guide), KNeighborsClassifier (class documentation)
Logistic Regression (user guide), LogisticRegression (class documentation)
Videos from An Introduction to Statistical Learning
Classification Problems and K-Nearest Neighbors (Chapter 2)
Introduction to Classification (Chapter 4)
Logistic Regression and Maximum Likelihood (Chapter 4)

Comments or Questions?

Email: kevin@dataschool.io
Website: http://dataschool.io
Twitter: @justmarkham
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
Returns a NumPy array
Can predict for multiple observations at once
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
Step 2: "Instantiate" the "estimator"

"Estimator" is scikit-learn's term for model
"Instantiate" means "make an instance of"
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
Step 3: Fit the model with data (aka "model training")

Model is learning the relationship between X and y
Occurs in-place
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
Step 4: Predict the response for a new observation

New observations are called "out-of-sample" data
Uses the information it learned during the model training process
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
Using a different classification model
Can you write Python code for it?

# import the class
from sklearn.linear_model import LogisticRegression

# instantiate the model (using the default parameters)
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X, y)

# predict the response for new observations
logreg.predict(X_new)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
Using a different value for K
Can you write Python code for it?

# instantiate the model (using the value K=5)
knn = KNeighborsClassifier(n_neighbors=5)

# fit the model with data
knn.fit(X, y)

# predict the response for new observations
knn.predict(X_new)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/04_model_training.ipynb".
The first task is:
scikit-learn 4-step modeling pattern
Step 1: Import the class you plan to use
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
Downsides of train/test split?

Provides a high-variance estimate of out-of-sample accuracy
K-fold cross-validation overcomes this limitation
But, train/test split is still useful because of its flexibility and speed

Resources

Quora: What is an intuitive explanation of overfitting?
Video: Estimating prediction error (12 minutes, starting at 2:34) by Hastie and Tibshirani
Understanding the Bias-Variance Tradeoff
Guiding questions when reading this article

Video: Visualizing bias and variance (15 minutes) by Abu-Mostafa

Comments or Questions?

Email: kevin@dataschool.io
Website: http://dataschool.io
Twitter: @justmarkham
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
KNN (K=1)
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X, y)
y_pred = knn.predict(X)
print(metrics.accuracy_score(y, y_pred))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
Known as training accuracy when you train and test the model on the same data

KNN (K=5)
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X, y)
y_pred = knn.predict(X)
print(metrics.accuracy_score(y, y_pred))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
Logistic regression
Can you write Python code for it?

# import the class
from sklearn.linear_model import LogisticRegression

# instantiate the model (using the default parameters)
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X, y)

# predict the response values for the observations in X
logreg.predict(X)

# store the predicted response values
y_pred = logreg.predict(X)

# check how many predictions were generated
len(y_pred)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
Problems with training and testing on the same data

Goal is to estimate likely performance of a model on out-of-sample data
But, maximizing training accuracy rewards overly complex models that won't necessarily generalize
Unnecessarily complex models overfit the training data

Image Credit: Overfitting by Chabacano. Licensed under GFDL via Wikimedia Commons.
Evaluation procedure #2: Train/test split

Split the dataset into two pieces: a training set and a testing set.
Train the model on the training set.
Test the model on the testing set, and evaluate how well we did.
Can you write Python code for it?

# print the shapes of X and y
print(X.shape)
print(y.shape)

# STEP 1: split X and y into training and testing sets
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
Repeat for KNN with K=1:
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print(metrics.accuracy_score(y_test, y_pred))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
Repeat for KNN with K=5:
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print(metrics.accuracy_score(y_test, y_pred))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
Training accuracy rises as model complexity increases
Testing accuracy penalizes models that are too complex or not complex enough
For KNN models, complexity is determined by the value of K (lower value = more complex)

Making predictions on out-of-sample data
Can you write Python code for it?

# instantiate the model with the best known parameters
knn = KNeighborsClassifier(n_neighbors=11)

# train the model with X and y (not X_train and y_train)
knn.fit(X, y)

# make a prediction for an out-of-sample observation
knn.predict([[3, 5, 4, 2]])


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/05_model_evaluation.ipynb".
The first task is:
What did this accomplish?

Model can be trained and tested on different data
Response values are known for the testing set, and thus predictions can be evaluated
Testing accuracy is a better estimate than training accuracy of out-of-sample performance
Can you write Python code for it?

# print the shapes of the new X objects
print(X_train.shape)
print(X_test.shape)

# print the shapes of the new y objects
print(y_train.shape)
print(y_test.shape)

# STEP 2: train the model on the training set
logreg = LogisticRegression()
logreg.fit(X_train, y_train)

# STEP 3: make predictions on the testing set
y_pred = logreg.predict(X_test)

# compare actual response values (y_test) with predicted response values (y_pred)
print(metrics.accuracy_score(y_test, y_pred))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
$$y = 2.88 + 0.0466 \times TV + 0.179 \times Radio + 0.00345 \times Newspaper$$
How do we interpret the TV coefficient (0.0466)?

For a given amount of Radio and Newspaper ad spending, a "unit" increase in TV ad spending is associated with a 0.0466 "unit" increase in Sales.
Or more clearly: For a given amount of Radio and Newspaper ad spending, an additional $1,000 spent on TV ads is associated with an increase in sales of 46.6 items.

Important notes:

This is a statement of association, not causation.
If an increase in TV ad spending was associated with a decrease in sales, $\beta_1$ would be negative.

Making predictions
Can you write Python code for it?

# make predictions on the testing set
y_pred = linreg.predict(X_test)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
Comparing these metrics:

MAE is the easiest to understand, because it's the average error.
MSE is more popular than MAE, because MSE "punishes" larger errors.
RMSE is even more popular than MSE, because RMSE is interpretable in the "y" units.

Computing the RMSE for our Sales predictions
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
Interpreting model coefficients
Can you write Python code for it?

# print the intercept and coefficients
print(linreg.intercept_)
print(linreg.coef_)

# pair the feature names with the coefficients
list(zip(feature_cols, linreg.coef_))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
Linear regression in scikit-learn
Can you write Python code for it?

# import model
from sklearn.linear_model import LinearRegression

# instantiate
linreg = LinearRegression()

# fit the model to the training data (learn the coefficients)
linreg.fit(X_train, y_train)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
Mean Absolute Error (MAE) is the mean of the absolute value of the errors:
$$\frac 1n\sum_{i=1}^n|y_i-\hat{y}_i|$$
Can you write Python code for it?

# calculate MAE by hand
print((10 + 0 + 20 + 10)/4.)

# calculate MAE using scikit-learn
from sklearn import metrics
print(metrics.mean_absolute_error(true, pred))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
Mean Squared Error (MSE) is the mean of the squared errors:
$$\frac 1n\sum_{i=1}^n(y_i-\hat{y}_i)^2$$
Can you write Python code for it?

# calculate MSE by hand
print((10**2 + 0**2 + 20**2 + 10**2)/4.)

# calculate MSE using scikit-learn
print(metrics.mean_squared_error(true, pred))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
Primary object types:

DataFrame: rows and columns (like a spreadsheet)
Series: a single column
Can you write Python code for it?

# display the last 5 rows
data.tail()

# check the shape of the DataFrame (rows, columns)
data.shape


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
Splitting X and y into training and testing sets
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

# default split is 75% for training and 25% for testing
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
The RMSE decreased when we removed Newspaper from the model. (Error is something we want to minimize, so a lower number for RMSE is better.) Thus, it is unlikely that this feature is useful for predicting Sales, and should be removed from the model.
Resources
Linear regression:

Longer notebook on linear regression by me
Chapter 3 of An Introduction to Statistical Learning and related videos by Hastie and Tibshirani (Stanford)
Quick reference guide to applying and interpreting linear regression by me
Introduction to linear regression by Robert Nau (Duke)

Pandas:

Three-part pandas tutorial by Greg Reda
read_csv and read_table documentation

Seaborn:

Official seaborn tutorial
Example gallery

Comments or Questions?

Email: kevin@dataschool.io
Website: http://dataschool.io
Twitter: @justmarkham
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/06_linear_regression.ipynb".
The first task is:
We need an evaluation metric in order to compare our predictions with the actual values!
Model evaluation metrics for regression
Evaluation metrics for classification problems, such as accuracy, are not useful for regression problems. Instead, we need evaluation metrics designed for comparing continuous values.
Let's create some example numeric predictions, and calculate three common evaluation metrics for regression problems:
Can you write Python code for it?

# define true and predicted response values
true = [100, 50, 30, 20]
pred = [90, 50, 50, 30]


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/07_cross_validation.ipynb".
The first task is:
Cross-validation example: model selection
Goal: Compare the best KNN model with logistic regression on the iris dataset
Can you write Python code for it?

# 10-fold cross-validation with the best KNN model
knn = KNeighborsClassifier(n_neighbors=20)
print(cross_val_score(knn, X, y, cv=10, scoring='accuracy').mean())

# 10-fold cross-validation with logistic regression
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
print(cross_val_score(logreg, X, y, cv=10, scoring='accuracy').mean())


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/07_cross_validation.ipynb".
The first task is:
Dataset contains 25 observations (numbered 0 through 24)
5-fold cross-validation, thus it runs for 5 iterations
For each iteration, every observation is either in the training set or the testing set, but not both
Every observation is in the testing set exactly once

Comparing cross-validation to train/test split
Advantages of cross-validation:

More accurate estimate of out-of-sample accuracy
More "efficient" use of data (every observation is used for both training and testing)

Advantages of train/test split:

Runs K times faster than K-fold cross-validation
Simpler to examine the detailed results of the testing process

Cross-validation recommendations

K can be any number, but K=10 is generally recommended
For classification problems, stratified sampling is recommended for creating the folds
Each response class should be represented with equal proportions in each of the K folds
scikit-learn's cross_val_score function does this by default

Cross-validation example: parameter tuning
Goal: Select the best tuning parameters (aka "hyperparameters") for KNN on the iris dataset
Can you write Python code for it?

from sklearn.cross_validation import cross_val_score

# 10-fold cross-validation with K=5 for KNN (the n_neighbors parameter)
knn = KNeighborsClassifier(n_neighbors=5)
scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy')
print(scores)

# use average accuracy as an estimate of out-of-sample accuracy
print(scores.mean())

# search for an optimal value of K for KNN
k_range = list(range(1, 31))
k_scores = []
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    scores = cross_val_score(knn, X, y, cv=10, scoring='accuracy')
    k_scores.append(scores.mean())
print(k_scores)

import matplotlib.pyplot as plt
%matplotlib inline

# plot the value of K for KNN (x-axis) versus the cross-validated accuracy (y-axis)
plt.plot(k_range, k_scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Cross-Validated Accuracy')


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/07_cross_validation.ipynb".
The first task is:
Improvements to cross-validation
Repeated cross-validation

Repeat cross-validation multiple times (with different random splits of the data) and average the results
More reliable estimate of out-of-sample performance by reducing the variance associated with a single trial of cross-validation

Creating a hold-out set

"Hold out" a portion of the data before beginning the model building process
Locate the best model using cross-validation on the remaining data, and test it using the hold-out set
More reliable estimate of out-of-sample performance since hold-out set is truly out-of-sample

Feature engineering and selection within cross-validation iterations

Normally, feature engineering and selection occurs before cross-validation
Instead, perform all feature engineering and selection within each cross-validation iteration
More reliable estimate of out-of-sample performance since it better mimics the application of the model to out-of-sample data

Resources

scikit-learn documentation: Cross-validation, Model evaluation
scikit-learn issue on GitHub: MSE is negative when returned by cross_val_score
Section 5.1 of An Introduction to Statistical Learning (11 pages) and related videos: K-fold and leave-one-out cross-validation (14 minutes), Cross-validation the right and wrong ways (10 minutes)
Scott Fortmann-Roe: Accurately Measuring Model Prediction Error
Machine Learning Mastery: An Introduction to Feature Selection
Harvard CS109: Cross-Validation: The Right and Wrong Way
Journal of Cheminformatics: Cross-validation pitfalls when selecting and assessing regression and classification models

Comments or Questions?

Email: kevin@dataschool.io
Website: http://dataschool.io
Twitter: @justmarkham
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/08_grid_search.ipynb".
The first task is:
Important: Specify a continuous distribution (rather than a list of values) for any continous parameters
Can you write Python code for it?

# n_iter controls the number of searches
rand = RandomizedSearchCV(knn, param_dist, cv=10, scoring='accuracy', n_iter=10, random_state=5)
rand.fit(X, y)
rand.grid_scores_

# examine the best model
print(rand.best_score_)
print(rand.best_params_)

# run RandomizedSearchCV 20 times (with n_iter=10) and record the best score
best_scores = []
for _ in range(20):
    rand = RandomizedSearchCV(knn, param_dist, cv=10, scoring='accuracy', n_iter=10)
    rand.fit(X, y)
    best_scores.append(round(rand.best_score_, 3))
print(best_scores)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/08_grid_search.ipynb".
The first task is:
Resources

scikit-learn documentation: Grid search, GridSearchCV, RandomizedSearchCV
Timed example: Comparing randomized search and grid search
scikit-learn workshop by Andreas Mueller: Video segment on randomized search (3 minutes), related notebook
Paper by Yoshua Bengio: Random Search for Hyper-Parameter Optimization

Comments or Questions?

Email: kevin@dataschool.io
Website: http://dataschool.io
Twitter: @justmarkham
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/08_grid_search.ipynb".
The first task is:
Using the best parameters to make predictions
Can you write Python code for it?

# train your model using all data and the best known parameters
knn = KNeighborsClassifier(n_neighbors=13, weights='uniform')
knn.fit(X, y)

# make a prediction on out-of-sample data
knn.predict([[3, 5, 4, 2]])

# shortcut: GridSearchCV automatically refits the best model using all of the data
grid.predict([[3, 5, 4, 2]])


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/08_grid_search.ipynb".
The first task is:
You can set n_jobs = -1 to run computations in parallel (if supported by your computer and OS)
Can you write Python code for it?

# fit the grid with data
grid.fit(X, y)

# view the complete results (list of named tuples)
grid.grid_scores_

# examine the first tuple
print(grid.grid_scores_[0].parameters)
print(grid.grid_scores_[0].cv_validation_scores)
print(grid.grid_scores_[0].mean_validation_score)

# create a list of the mean scores only
grid_mean_scores = [result.mean_validation_score for result in grid.grid_scores_]
print(grid_mean_scores)

# plot the results
plt.plot(k_range, grid_mean_scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Cross-Validated Accuracy')

# examine the best model
print(grid.best_score_)
print(grid.best_params_)
print(grid.best_estimator_)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
AUC is the percentage of the ROC plot that is underneath the curve:
Can you write Python code for it?

# IMPORTANT: first argument is true values, second argument is predicted probabilities
print(metrics.roc_auc_score(y_test, y_pred_prob))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
AUC is useful as a single number summary of classifier performance.
If you randomly chose one positive and one negative observation, AUC represents the likelihood that your classifier will assign a higher predicted probability to the positive observation.
AUC is useful even when there is high class imbalance (unlike classification accuracy).
Can you write Python code for it?

# calculate cross-validated AUC
from sklearn.cross_validation import cross_val_score
cross_val_score(logreg, X, y, cv=10, scoring='roc_auc').mean()


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Classification Error: Overall, how often is the classifier incorrect?

Also known as "Misclassification Rate"
Can you write Python code for it?

print((FP + FN) / float(TP + TN + FP + FN))
print(1 - metrics.accuracy_score(y_test, y_pred_class))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Classification accuracy: percentage of correct predictions
Can you write Python code for it?

# calculate accuracy
from sklearn import metrics
print(metrics.accuracy_score(y_test, y_pred_class))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Comparing the true and predicted response values
Can you write Python code for it?

# print the first 25 true and predicted responses
from __future__ import print_function
print('True:', y_test.values[0:25])
print('Pred:', y_pred_class[0:25])


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Conclusion:

Classification accuracy is the easiest classification metric to understand
But, it does not tell you the underlying distribution of response values
And, it does not tell you what "types" of errors your classifier is making

Confusion matrix
Table that describes the performance of a classification model
Can you write Python code for it?

# IMPORTANT: first argument is true values, second argument is predicted values
print(metrics.confusion_matrix(y_test, y_pred_class))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Conclusion:

Threshold of 0.5 is used by default (for binary problems) to convert predicted probabilities into class predictions
Threshold can be adjusted to increase sensitivity or specificity
Sensitivity and specificity have an inverse relationship

ROC Curves and Area Under the Curve (AUC)
Question: Wouldn't it be nice if we could see how sensitivity and specificity are affected by various thresholds, without actually changing the threshold?
Answer: Plot the ROC curve!
Can you write Python code for it?

# IMPORTANT: first argument is true values, second argument is predicted probabilities
fpr, tpr, thresholds = metrics.roc_curve(y_test, y_pred_prob)
plt.plot(fpr, tpr)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.0])
plt.title('ROC curve for diabetes classifier')
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.grid(True)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Confusion matrix advantages:

Allows you to calculate a variety of metrics
Useful for multi-class problems (more than two response classes)

ROC/AUC advantages:

Does not require you to set a classification threshold
Still useful when there is high class imbalance

Confusion Matrix Resources

Blog post: Simple guide to confusion matrix terminology by me
Videos: Intuitive sensitivity and specificity (9 minutes) and The tradeoff between sensitivity and specificity (13 minutes) by Rahul Patwari
Notebook: How to calculate "expected value" from a confusion matrix by treating it as a cost-benefit matrix (by Ed Podojil)
Graphic: How classification threshold affects different evaluation metrics (from a blog post about Amazon Machine Learning)

ROC and AUC Resources

Lesson notes: ROC Curves (from the University of Georgia)
Video: ROC Curves and Area Under the Curve (14 minutes) by me, including transcript and screenshots and a visualization
Video: ROC Curves (12 minutes) by Rahul Patwari
Paper: An introduction to ROC analysis by Tom Fawcett
Usage examples: Comparing different feature sets for detecting fraudulent Skype users, and comparing different classifiers on a number of popular datasets

Other Resources

scikit-learn documentation: Model evaluation
Guide: Comparing model evaluation procedures and metrics by me
Video: Counterfactual evaluation of machine learning models (45 minutes) about how Stripe evaluates its fraud detection model, including slides

Comments or Questions?

Email: kevin@dataschool.io
Website: http://dataschool.io
Twitter: @justmarkham
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Every observation in the testing set is represented in exactly one box
It's a 2x2 matrix because there are 2 response classes
The format shown here is not universal

Basic terminology

True Positives (TP): we correctly predicted that they do have diabetes
True Negatives (TN): we correctly predicted that they don't have diabetes
False Positives (FP): we incorrectly predicted that they do have diabetes (a "Type I error")
False Negatives (FN): we incorrectly predicted that they don't have diabetes (a "Type II error")
Can you write Python code for it?

# print the first 25 true and predicted responses
print('True:', y_test.values[0:25])
print('Pred:', y_pred_class[0:25])

# save confusion matrix and slice into four pieces
confusion = metrics.confusion_matrix(y_test, y_pred_class)
TP = confusion[1, 1]
TN = confusion[0, 0]
FP = confusion[0, 1]
FN = confusion[1, 0]


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
False Positive Rate: When the actual value is negative, how often is the prediction incorrect?
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Many other metrics can be computed: F1 score, Matthews correlation coefficient, etc.
Conclusion:

Confusion matrix gives you a more complete picture of how your classifier is performing
Also allows you to compute various classification metrics, and these metrics can guide your model selection

Which metrics should you focus on?

Choice of metric depends on your business objective
Spam filter (positive class is "spam"): Optimize for precision or specificity because false negatives (spam goes to the inbox) are more acceptable than false positives (non-spam is caught by the spam filter)
Fraudulent transaction detector (positive class is "fraud"): Optimize for sensitivity because false positives (normal transactions that are flagged as possible fraud) are more acceptable than false negatives (fraudulent transactions that are not detected)

Adjusting the classification threshold
Can you write Python code for it?

# print the first 10 predicted responses
logreg.predict(X_test)[0:10]

# print the first 10 predicted probabilities of class membership
logreg.predict_proba(X_test)[0:10, :]

# print the first 10 predicted probabilities for class 1
logreg.predict_proba(X_test)[0:10, 1]

# store the predicted probabilities for class 1
y_pred_prob = logreg.predict_proba(X_test)[:, 1]

# allow plots to appear in the notebook
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['font.size'] = 14

# histogram of predicted probabilities
plt.hist(y_pred_prob, bins=8)
plt.xlim(0, 1)
plt.title('Histogram of predicted probabilities')
plt.xlabel('Predicted probability of diabetes')
plt.ylabel('Frequency')