I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
Legend loc codes:
0   best                6   center left
1   upper right         7   center right
2   upper left          8   lower center
3   lower left          9   upper center
4   lower right         10  center
Subplots

subplot(rows,columns)
Access each subplot like a matrix. [x,y]
For example: subplot(2,2) makes four panels with the coordinates:
Can you write Python code for it?

fig, ax = plt.subplots(2,2)                                               # 2 rows 2 columns
fig.set_size_inches(11,8.5)                                               # width, height

fig.tight_layout()                                                        # Make better use of space on plot

ax[0,0].plot(t, s, color='b', marker='None', linestyle='--')              # Plot at [0,0]

ax[0,1].hist(h, bins=20, facecolor='MediumOrchid')                        # Plot at [0,1]

ax[1,0].hist(j,bins=20, facecolor='HotPink', histtype='stepfilled')       # Plot at [1,0]
ax[1,0].vlines(2.0, 0.0, 50.0, color='xkcd:seafoam green', linewidth=3)

ax[1,1].set_xscale('log')                                                 # Plot at [1,1] - x-axis set to log
ax[1,1].plot(t, s, color='r', marker='None', linestyle='--');


I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
Plotting Extras
Adding Legends
Can you write Python code for it?

theta = np.linspace(0.0,2*np.pi,300)

y1 = abs(np.sin(theta))
y2 = -abs(np.cos(theta))

plt.plot(theta, y1, color="b", label="|Sin|")
plt.plot(theta, y2, color="r", label="-|Cos|")

plt.legend(loc=0,shadow=True);


I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
Plotting from multiple external data files
Can you write Python code for it?

data_list = glob.glob('./MyData/12_data*.csv')

data_list

fig,ax = plt.subplots(1,1)                    # One window
fig.set_size_inches(11,8.5)                   # (width,height) - letter paper landscape

fig.tight_layout()                          # Make better use of space on plot

ax.set_xlim(0.0,80.0)
ax.set_ylim(15.0,100.0)

ax.set_xlabel("This is X")
ax.set_ylabel("This is Y")

for file in data_list:

    data = QTable.read(file, format='ascii.csv')
    ax.plot(data['x'], data['y'],marker="o",linestyle="None",markersize=7,label=file)

ax.legend(loc=0,shadow=True);


I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
Polar Plots
Can you write Python code for it?

theta = np.linspace(0,2*np.pi,1000)

fig = plt.figure()
ax = fig.add_subplot(111,projection='polar')

fig.set_size_inches(6,6)                   # (width,height) - letter paper landscape

fig.tight_layout()                          # Make better use of space on plot

ax.plot(theta,theta/5.0,label="spiral")
ax.plot(theta,np.cos(4*theta),label="flower")

ax.legend(loc=2, frameon=False);


I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
Pseudo 3D plots
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D

ax = plt.axes(projection='3d')
xx = np.cos(3*theta)
yy = np.sin(2*theta)

ax.plot(theta,xx,yy);
ax.view_init(azim = -140,elev = 15)


I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
Simple Histograms
Can you write Python code for it?

#Histogram of "h" with 20 bins

np.random.seed(42)
h = np.random.randn(500)

plt.hist(h, bins=20, facecolor='MediumOrchid');

mask2 = np.where(h>0.0)

np.random.seed(42)
j = np.random.normal(2.0,1.0,300)  # normal dist, ave = 2.0, std = 1.0

plt.hist(h[mask2], bins=20, facecolor='#b20010', histtype='stepfilled')
plt.hist(j,        bins=20, facecolor='#0200b0', histtype='stepfilled', alpha = 0.30);


I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
Simple plotting - with style
The default style of matplotlib is a bit lacking in style. Some would term it ugly. The new version of matplotlib has added some new styles that you can use in place of the default. Changing the style will effect all of the rest of the plots on the notebook.
Can you write Python code for it?

plt.style.available

plt.style.use('ggplot')

plt.plot(t,s)

plt.xlabel('time (s)')
plt.ylabel('voltage (mV)')
plt.title('This is a title')

plt.ylim(-1.5,1.5)

plt.plot(t, s, color='b', marker='None', linestyle='--');   # adding the ';' at then suppresses the Out[] line

mask1 = np.where((s>-0.4) & (s<0))

plt.plot(t, s, color='b', marker='None', linestyle='--')

plt.plot(t[mask1],s[mask1],color="g",marker="o",linestyle="None",markersize=8);


I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
Simple plotting - with style
The default style of matplotlib is a bit lacking in style. Some would term it ugly. The new version of matplotlib has added some new styles that you can use in place of the default. Changing the style will effect all of the rest of the plots on the notebook.
Examples of the various styles can be found here
Can you write Python code for it?


I am working on the file "06_PlottingWithPython.ipynb".
The first task is:
You have better control of the plot with the object oriented interface.
While most plt functions translate directly to ax methods (such as plt.plot() → ax.plot(), plt.legend() → ax.legend(), etc.), this is not the case for all commands.
In particular, functions to set limits, labels, and titles are slightly modified.
For transitioning between matlab-style functions and object-oriented methods, make the following changes:

plt.xlabel()  → ax.set_xlabel()
plt.ylabel() → ax.set_ylabel()
plt.xlim()  → ax.set_xlim()
plt.ylim() → ax.set_ylim()
plt.title() → ax.set_title()
Can you write Python code for it?

fig,ax = plt.subplots(1,1)                    # One window
fig.set_size_inches(11,8.5)                   # (width,height) - letter paper landscape

fig.tight_layout()                          # Make better use of space on plot

ax.set_xlim(0.0,1.5)

ax.spines['bottom'].set_position('zero')    # Move the bottom axis line to x = 0

ax.set_xlabel("This is X")
ax.set_ylabel("This is Y")

ax.plot(t, s, color='b', marker='None', linestyle='--')

ax.text(0.8, 0.6, 'Bad Wolf', color='green', fontsize=36)            # You can place text on the plot

ax.vlines(0.4, -0.4, 0.8, color='m', linewidth=3)                    # vlines(x, ymin, ymax)
ax.hlines(0.8,  0.2, 0.6, color='y', linewidth=5)                    # hlines(y, xmin, xmax)

fig.savefig('fig1.png', bbox_inches='tight')

import glob

glob.glob('*.png')


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+-Copy1.ipynb".
The first task is:
10.Schreibe nun ein Programm, das alle Ziffern 4 mit dem Buchstaben A ersetzte, alle Ziffern 5 mit dem Buchtaben B.
Can you write Python code for it?

newnewlist = []
for elem in newlst:
    if '4' in elem:
        elem = elem.replace('4', 'A')
    if '5' in elem:
        elem = elem.replace('5', 'B')
    newnewlist.append(elem)

newnewlist


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+-Copy1.ipynb".
The first task is:
2.Drucke alle die Zahlen von 0 bis 4 aus:
Can you write Python code for it?

for x in range(5):
    print(x)

for x in range(3, 6):
    print(x)


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+-Copy1.ipynb".
The first task is:
5.Addiere alle Zahlen in der Liste
Can you write Python code for it?


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+-Copy1.ipynb".
The first task is:
6.Addiere nur die Zahlen, die gerade sind
Can you write Python code for it?

evennumber = []
for elem in numbers:
    if elem % 2 == 0:
        evennumber.append(elem)
sum(evennumber)


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+-Copy1.ipynb".
The first task is:
7.Drucke mit einem For Loop 5 Mal hintereinander Hello World aus
Can you write Python code for it?

Satz = ['Hello World', 'Hello World','Hello World','Hello World','Hello World']
for elem in Satz:
    print(elem)

#Lösung


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+-Copy1.ipynb".
The first task is:
9.Schreibe einen For Loop, der die Nummern in der folgenden Liste von int in str verwandelt.
Can you write Python code for it?

lst = range(45,99)

newlst = []
for i in lst:
    i = str(i)
    newlst.append(i)
print(newlst)


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+.ipynb".
The first task is:
10.Schreibe nun ein Programm, das alle Ziffern 4 mit dem Buchstaben A ersetzte, alle Ziffern 5 mit dem Buchtaben B.
Can you write Python code for it?

newnewlist = [] #Replace ist ein wichtiger Befehl, da man so grosse Datenmengen bereinigt. Man ersetzt dann einfach durch ''
for elem in newlst:
    if '4' in elem:
        elem = elem.replace('4', 'A')
    if '5' in elem:
        elem = elem.replace('5', 'B')
    newnewlist.append(elem)

newnewlist


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+.ipynb".
The first task is:
4.Baue einen For-Loop, indem Du alle geraden Zahlen ausdruckst, die tiefer sind als 237.
Can you write Python code for it?


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+.ipynb".
The first task is:
5.Addiere alle Zahlen in der Liste
Can you write Python code for it?

sum(numbers)

count = 0
for x in numbers:
    count = count + x
print(count)

#Lösung:


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+.ipynb".
The first task is:
8.Entwickle ein Programm, das alle Nummern zwischen 2000 und 3200 findet, die durch 7, aber nicht durch 5 teilbar sind. Das Ergebnis sollte auf einer Zeile ausgedruckt werden. Tipp: Schaue Dir hier die Vergleichsoperanden von Python an.
Can you write Python code for it?

l=[]
for i in range(2000, 3201):
    if (i % 7==0) and (i % 5!=0):#!= entspricht nichtgleich
        l.append(str(i))

print(','.join(l))#join verbindet alle elemente einer liste, hier setze ich noch ein komma dazwischen


I am working on the file "06_Python_Rückblick/01+Rückblick+02+For-Loop-Übungen+.ipynb".
The first task is:
9.Schreibe einen For Loop, der die Nummern in der folgenden Liste von int in str verwandelt.
Can you write Python code for it?

lst = range(45,99)

newlst = []
for i in lst:
    i = str(i)
    newlst.append(i)
print(newlst)#return kommt immer in Funktionen!


I am working on the file "06_Python_Rückblick/03 Python Functions, 10 Übungen.ipynb".
The first task is:
1.Schreibe eine Funktion, die aus einer Liste, die grösste Zahl herauszieht. Es ist verboten mit "max" zu arbeiten. :-)
Can you write Python code for it?

lst = [3,7,14,222,6]
lst.reverse()
print(lst)

def maxi(element):
    element.sort()
    element.reverse()
    return element[0]# ich könnte auch element.reverse weglassen und einfach return element[-1] - gibt mir das letzte Element.

maxi(lst)


I am working on the file "06_Python_Rückblick/03 Python Functions, 10 Übungen.ipynb".
The first task is:
3.Schreibe eine Funktion, die alle Elemente einer Liste multipliziert.
Can you write Python code for it?

def multi(Menge):
    multi_elem = 1
    for elem in Menge:
        multi_elem = multi_elem * elem
    return multi_elem

multi(lst)


I am working on the file "06_Python_Rückblick/03 Python Functions, 10 Übungen.ipynb".
The first task is:
4.Schreibe eine Funktion, die einen String nimmt, und spiegelt. Also "hallo" zu "ollah".
Can you write Python code for it?

elem = input('Bitte geben Sie ein Wort ein ')
wort = list(elem)
print(wort)

def umkehr(wort):
    elem = input('Bitte geben Sie ein Wort ein ')
    wort = list(elem)
    wort.reverse()
    print(wort)

umkehr(wort)

def mirror(mylist):
    for elem in mylist:
        return mylist[::-1]


I am working on the file "06_Python_Rückblick/03 Python Functions, 10 Übungen.ipynb".
The first task is:
5.Schreibe eine Funktion, die prüft, ob eine Zahl in einer bestimmten Zahlenfolge zu finden ist.
Can you write Python code for it?

liste = [45, 34, 64,45]

def such(zahlen):
    zahl = int(input('Bitte geben Sie eine Zahl ein:'))
    if zahl in zahlen:
        return 'Treffer'
    else:
        return 'Kein Treffer'

such(liste)

def isinlist(zahl, liste):
    for elem in lst:
        if elem == number:
            return true
    return false

istinlist(34, liste)

isinlist(34, liste)

zahleliste = []


I am working on the file "06_Python_Rückblick/03 Python Functions, 10 Übungen.ipynb".
The first task is:
6.Lösche die mehrfach genannten Elemente aus der folgenden Liste.
Can you write Python code for it?

liste = [5,5,5,5,3,2,11,5]

def lösch(mehrfach):
    new_mehrfach = []
    for elem in mehrfach:
        if elem in new_mehrfach:
            new_mehrfach.append(elem)
        else:
            continue
    print(new_mehrfach)

lösch(liste)


I am working on the file "06_Python_Rückblick/03 Python Functions, 10 Übungen.ipynb".
The first task is:
7.Drucke die geraden Zahlen aus der folgenden Liste aus:
Can you write Python code for it?

lst = [34,23,22,443,45,78,23,89,23]

def gerade(summe):
    for elem in summe:
        if elem % 2 == 0:
            print(elem)
        else:
            continue

gerade(lst)


I am working on the file "06_Stack_Queue.ipynb".
The first task is:
Min Stack
Implement a stack with min() function, which will return the smallest number in the stack.
It should support push, pop and min operation all in O(1) cost.
http://www.lintcode.com/en/problem/min-stack/
https://leetcode.com/problems/min-stack/#/description
Can you write Python code for it?

class MinStack(object):

    def __init__(self):
        # do some intialize if necessary
        self.stack = []

    def push(self, number):
        # write yout code here
        curMin = self.min()
        if curMin is None or number < curMin:
            curMin = number
        self.stack.append([number, curMin])

    def pop(self):
        # pop and return the top item in stack
        pop = self.stack.pop()
        return pop[0]

    def min(self):
        # return the minimum number in stack
        if len(self.stack) == 0:
            return None
        else:
            return self.stack[-1][1]


I am working on the file "06_WebGIS/Notebooks/00-Importing-Local-Files.ipynb".
The first task is:
CSV Module
In the csv module we:

Open the datafile in read ('r') mode
Create a reader object with that file, specifying the delimiter (the default is a comma, but it is explicitly specified here for clarity).
Create an empty list to store the data
Loop over the rows in the reader appending each row to the list

The types of the resulting items in the list of list will always be strings, so if you need something else it will have to be converted later.
Can you write Python code for it?

import csv
datafile = open('./data/examp_data.txt', 'r')
datareader = csv.reader(datafile, delimiter=',')
data = []
for row in datareader:
    data.append(row)
data


I am working on the file "06_WebGIS/Notebooks/00-Importing-Local-Files.ipynb".
The first task is:
Pandas
Pandas is a powerful data management library that produces data structures and associated tools that are ideal for scientific computing tasks related to data. In particular, it produces a dataframe object that is much like R's dataframe and is designed to hold data with the standard structure of one row per record and one column per type of data (or field).
In Pandas we just use the read_csv() function to import text files. It has a lot of options, but will do most things automatically including detecting delimiters and detecting data types.
Can you write Python code for it?

import pandas as pd
data = pd.read_csv('./data/examp_data.txt')
data


I am working on the file "06_WebGIS/Notebooks/00-Importing-Local-Files.ipynb".
The first task is:
Pandas dataframes do behave a bit differently than a lot of list based structures in Python, but we'll learn how to work with them soon. If you just want to pull the core data out of a dataframe you can do this using the values member (a member is just a variable associated with an object).
Can you write Python code for it?


I am working on the file "06_WebGIS/Notebooks/01-Getting-data-with-Pandas.ipynb".
The first task is:
Another way around this is to invoke the skip rows option when reading the CSV. If you look at the file we are importing, you see that the first 49 rows are comments, then comes our header row, and then the field type row that we don't want. So we want to skip rows 1 thru 49 and also line 51. If we create a list of these row numbers, we can pass that to the skip_rows parameter...
Can you write Python code for it?

#Create a list of numbers 0 thru 49, recalling Python lists are zero-indexed...
rowsToSkip = range(49)
#Append '51' to the list
rowsToSkip.append(50)

#Use the read_csv function as before, but skip the rows we want to skip
dfNWIS = pd.read_csv(theURL, delimiter='\t', skiprows=rowsToSkip)
dfNWIS.head()


I am working on the file "06_WebGIS/Notebooks/01-Getting-data-with-Pandas.ipynb".
The first task is:
Now that we have this as a pandas data frame, we can analyze it here, or we can simply save a copy to our local machine. For the latter, pandas' to_csv() function works quite easily.
Can you write Python code for it?


I am working on the file "06_WebGIS/Notebooks/01-Getting-data-with-Pandas.ipynb".
The first task is:
So there's one catch: the second line of the dataframe is not the data we want, but rather a listing of the field type and width. Pandas offers two ways around this. First, we could just drop the first row...
Can you write Python code for it?

#Drop the first row, and again show the first 5 rows of data...
dfNWIS.drop(0,axis='rows',inplace=True)
dfNWIS.head()


I am working on the file "06_WebGIS/Notebooks/02-Fetching-Data-with-urllib.ipynb".
The first task is:
So now, let's look at how the zipfile module can unzip it
Can you write Python code for it?

#First open the local zip file as a zipFile object
zipObject = zipfile.ZipFile(localFile)

#Create a folder to hold the file

#Name the folder we'll create the same as the file, without the extension
outFolderName = localFile[:-4]

#Well us the os module to do check if the folder exists, and create if not
import os
if not os.path.exists(outFolderName):
    outFolder = os.mkdir(localFile[:-4])

zipObject.extractall(path=outFolderName)
zipObject.close()


I am working on the file "06_WebGIS/Notebooks/02-Fetching-Data-with-urllib.ipynb".
The first task is:
This works for photos too...
Can you write Python code for it?

imgURL = 'https://imgs.xkcd.com/comics/state_borders.png'
urllib.urlretrieve(url=imgURL,filename="map.jpg",);

#Display the file in our notebook
from IPython.display import Image
Image("map.jpg")


I am working on the file "06_WebGIS/Notebooks/03-Fetching-files-with-ftplib.ipynb".
The first task is:
Now that we are logged in, we can send ftp commands to get information if we want. For example, the welcome command returns a message the server admin wants you to see when you log into the server:
Can you write Python code for it?


I am working on the file "06_WebGIS/Notebooks/03-Fetching-files-with-ftplib.ipynb".
The first task is:
Now that we've defined the state and created some folder to hold the data, we'll finally dive into the ftplib module to access the server and grab our data
The first step is to log into the server. Most ftp servers are "anonymous ftp" servers meaning, they allow anyone to log in. However, proper etiquette dictates you login as "anonymous" and use your email so they have a record of who is accessing their data...
Below, we log into the EPA ftp server, creating an 'ftp' object which is our programmatic link to the server. Once we link to it, we log in. You should change "user@duke.edu" to your email address.
Can you write Python code for it?

ftp = ftplib.FTP(ftpURL)
ftp.login("anonymous","user@duke.edu")


I am working on the file "06_WebGIS/Notebooks/03-Fetching-files-with-ftplib.ipynb".
The first task is:
Now, let's get a list of the files in this file, done with the nlst command.
Can you write Python code for it?

files = ftp.nlst()
len(files) #Over 2000 files here!

#Create a list of just Rhode Island files, that is one's that end in "_RI.zip"
RI_files = ftp.nlst("*_RI.zip")
len(RI_files) #49 files; that's better...


I am working on the file "06_WebGIS/Notebooks/03-Fetching-files-with-ftplib.ipynb".
The first task is:
This is where the ftp module is much less intuitive. The ftp.retbinary command initiates the download of a binary file, but it requires a "callback function", which is an action telling the ftp module what to do with this stream of downloaded information. For our callback function, we'll be writing the binary contents to a file.
Don't worry too much about this syntax; just don't modify it (other than perhaps the output filename) and you'll be fine.
Can you write Python code for it?

#First we need to create a filename to which the remote contents will be written
outFilename = stateFolder + os.sep + f
print (outFilename)

#Ok, here's the command that actually grabs the file and writes it locally
ftp.retrbinary("RETR " + f,open(outFilename,'wb').write)

#Did it work?
os.listdir(stateFolder)

#Close the ftp connection
ftp.close()


I am working on the file "06_WebGIS/Notebooks/03-Fetching-files-with-ftplib.ipynb".
The first task is:
What we need to do is navigate to the folder holding the files we want, which we found by navigating the ftp server in a web browser - and stored in the ftpDirectory variable.
We get there using the ftp cwd (change working directory) command:
Can you write Python code for it?


I am working on the file "06_WebGIS/Notebooks/05-Scraping-Data-With-Requests-and-BeautifulSoup.ipynb".
The first task is:
BeautifulSoup is designed to intelligently read raw HTML code, i.e., what is stored in the response variable generated above. The command below reads in the raw HTML and parses it into logical components that we can command.
The lxml in the command specifies a particular parser for deconstructing the HTML...
Can you write Python code for it?

# BeautifulSoup
soup = BeautifulSoup(response.text, 'lxml')
type(soup)


I am working on the file "06_WebGIS/Notebooks/05-Scraping-Data-With-Requests-and-BeautifulSoup.ipynb".
The first task is:
Here we search the text of the web page's body for any instances of https://....png, that is any link to a PNG image embedded in the page. This is done using re and implementing regular expressions (see https://developers.google.com/edu/python/regular-expressions for more info on this useful module...)
The match object returned by search() holds information about the nature of the match, including the original input string, the regular expression used, and the location within the original string where the pattern occurs. The group property of the match is the full string that's returned
Can you write Python code for it?

#Search the page for emebedded links to PNG files
match = re.search('https://.*\.png', soup.body.text)

#What was found in the search
print(match.group())

#And here is some Juptyer code to display the picture resulting from it
from IPython.display import Image
Image(url=match.group())


I am working on the file "06_WebGIS/Notebooks/12-Exploring-the-BISON-API.ipynb".
The first task is:
Or we can witness Pandas cleverness again! Here, we convert the collection of observations into a data frame
Can you write Python code for it?

import pandas as pd
df = pd.DataFrame(data['data'])
df.head()


I am working on the file "06_WebGIS/Notebooks/12-Exploring-the-BISON-API.ipynb".
The first task is:
Yikes, that's much less readable than the NWIS output!
Well, that's because the response from the BISON server is in JSON format. We can use Python's build in json module to convert this raw JSON text to a handy dictionary (of dictionaries) that we can [somewhat] easily manipulate...
Can you write Python code for it?

#Import the module
import json

#Convert the response
data = json.loads(response.content)
type(data)

#Ok, if it's a dictionary, what are it's keys?
data.keys()

#What are the values of the 'data' key
data['data']

#Oh, it's a list of occurrences! Let's examine the first one
data['data'][0]
#We see it's a dictionary too

#We can get the latitude of the record from it's `decimalLatitude` key
data['data'][0]['decimalLatitude']


I am working on the file "06_WebGIS/Notebooks/Bird-Demo-Reuben.ipynb".
The first task is:
All the entries with the tag <em> are the subspecies entries.
Can you write Python code for it?

#Find all lines in the section with the tag 'em'
subSpecies = section.find_all('em')


I am working on the file "06_WebGIS/Notebooks/Bird-Demo-Reuben.ipynb".
The first task is:
We can loop through each subspecies found and print its name
Can you write Python code for it?

#Extract to a variable
for subSpp in subSpecies:
    print (subSpp.get_text())


I am working on the file "06_WebGIS/Notebooks/GeocodingWithOSM.ipynb".
The first task is:
An API request consists two components: the service endpoint and a set of parameters associated with the service.
When using the requests module to create and send our request, we supply the service endpoint is a string containing the server address (as a URL) and the service name (here, it's search). And the parameters are supplied in the form of a Python dictionary. Here, the two paramters we'll pass are the format and address parameters.
Can you write Python code for it?

#Form the request
osmURL = 'http://nominatim.openstreetmap.org/search'
params = {'format':'json','q':address}


I am working on the file "06_WebGIS/Notebooks/GeocodingWithOSM.ipynb".
The first task is:
Now let's inform the user of the result of the whole process...
Can you write Python code for it?

print "The address {0} is located at\n{1}° Lat, {2}° Lon".format(address,lat,lng)


I am working on the file "06_WebGIS/Notebooks/GeocodingWithOSM.ipynb".
The first task is:
Now, we can use requests to send our command off to the OSM server. The server's response is saved as the response variable.
Can you write Python code for it?

#Send the request
response = requests.get(osmURL, params)


I am working on the file "06_WebGIS/Notebooks/GeocodingWithOSM.ipynb".
The first task is:
Our response contains only one item in the JSON list. We'll extract to a dictionary and print it's items.
Can you write Python code for it?

dataDict = jsonObj[0]
print dataDict.keys()


I am working on the file "06_WebGIS/Notebooks/GeocodingWithOSM.ipynb".
The first task is:
What we really want from the response, however, is the data returned by the service. The json function of the response object converts the response to an object in JavaScript Object Notation, or JSON. JSON is esentially a list of dictionaries that we can easily manipulate in Python.
Can you write Python code for it?

#Read in the response as a JSON encoded object
jsonObj = response.json()


I am working on the file "06_WebGIS/Notebooks/GeocodingWithOSM.ipynb".
The first task is:
pprint or "pretty print" allows us to display JSON objects in a readable format. Let's make a pretty print of our JSON repsonse.
Can you write Python code for it?

from pprint import pprint
pprint(jsonObj)


I am working on the file "06_conclusiones.ipynb".
The first task is:
Análisis Descriptivo de los Tweets en Español
A partir de los tweets almacenados se presentan algunas estadísticas básicas que describen algunas características interesantes de los datos como la cantidad de usuarios por país y la movilidad de los usuarios.
Cantidad de usuarios por país
La siguiente figura muestra los usuarios por país, se puede observar que Estados Unidos es el que cuenta con un mayor número de usuarios, seguido por Argentina y en tercer lugar México. Ademas sorpresivamente Brasil es el cuarto y en quinto lugar se encuentra España.
Can you write Python code for it?

%matplotlib inline
from glob import glob
from multiprocessing import Pool
from tqdm import tqdm
from collections import Counter
def number_users(fname):
    return fname, len(read_data(fname))

fnames = [i for i in glob('spanish/users_by_country/*.json.gz') if len(i.split('.')[0].split('/')[1]) == 2]
p = Pool(8)
res = [x for x in p.imap_unordered(number_users, fnames)]
p.close()

country_code = Counter()
for name, value in res:
    code = name.split('.')[0].split('/')[1]
    country_code[code] = value
mc = country_code.most_common()
size = 19
first = mc[:size]
extra = ('REST', sum([x[1] for x in mc[size:]]))
first.append(extra)
plt.figure(figsize=(10, 10))
_ = plt.pie([x[1] for x in first], labels=[x[0] for x in first])


I am working on the file "06_conclusiones.ipynb".
The first task is:
Análisis de polaridad en Estados Unidos, Argentina, México y España

Estados Unidos
México
Argentina
España.
19 de marzo
19 de junio es importante para todas las naciones excepto España
20 de julio que se celebra el día del amigo en Argentina.
Can you write Python code for it?

pos = [read_data('spanish/polarity_by_country/%s.json.gz' % x) for x in ['US', 'AR', 'ES']]
us_pos, ar_pos, es_pos = pos
plt.figure(figsize=(20, 10))
for code, D, k in zip(['US', 'MX', 'AR', 'ES'], [us_pos, mx_pos, ar_pos, es_pos],
                     range(4)):
    D = remove_median(D)
    ticks = [str(x[0])[2:] for x in D]
    mu = [x[1] for x in D]
    plt.subplot(4, 1, k+1)
    plt.plot(mu)
    plt.title(code)
    index = np.argsort(mu)
    index = np.concatenate((index[:1], index[-10:]))
    _ = plt.xticks(index, [ticks[x] for x in index], rotation=45)
    plt.grid()
    plt.ylim(-0.20, 0.20)


I am working on the file "06_conclusiones.ipynb".
The first task is:
Efecto oscilatorio cuyo periodo corresponde a los días de la semana.
TGIF (Thanks god, it's friday).
Remover este fenómeno
Quitar la mediana por día
Can you write Python code for it?

def remove_median(pos):
    median = np.array(mx_pos[: -int((len(pos) % 7))])[:, 1]
    median.shape = (int(median.shape[0] / 7), 7)
    median = np.median(median, axis=0)
    median = np.concatenate((np.concatenate([median for x in range(int(len(pos) / median.shape[0]))], axis=0),
                            median[:int(len(mx_pos) % 7)]), axis=0)
    return [(x[0], x[1] - y) for x, y in zip(pos, median)]

plt.figure(figsize=(20, 10))
nmx_pos = remove_median(mx_pos)
mu = np.array([x[1] for x in nmx_pos])
plt.plot(mu)
index = np.argsort(mu)
index = np.concatenate((index[:1], index[-10:]))
_ = plt.xticks(index, [ticks[x] for x in index], rotation=90)
plt.grid()

plt.figure(figsize=(20, 20))
for k, D in enumerate([mx_pos, remove_median(mx_pos)]):
    ticks = [str(x[0])[2:] for x in D]
    mu = [x[1] for x in D]
    plt.subplot(4, 1, k+1)
    plt.plot(mu)
    index = np.argsort(mu)
    index = np.concatenate((index[:1], index[-10:]))
    _ = plt.xticks(index, [ticks[x] for x in index], rotation=45)
    plt.grid()


I am working on the file "06_conclusiones.ipynb".
The first task is:
Movilidad de los Tuiteros (de habla hispana)
La siguiente figura muestra cuáles son los países que visitan más frecuentemente usuarios de un país en particular. Por ejemplo, la mayoría de los usuarios de Estados Unidos que viajan a otro país viajan a México, en segundo lugar a Puerto Rico y así sucesivamente; los usuarios de Argentina viajan a Brasil en primer lugar;
los de México viajan a Estados Unidos; y los de España también viajan a Estados Unidos.
Can you write Python code for it?

def migration(country_code='MX'):
    fname = 'spanish/users_by_country/%s.json.gz' % country_code
    d = read_data(fname)
    other = Counter()
    for x in d.values():
        if len(x) == 1:
            continue
        c = Counter(x)
        for xx in c.most_common()[1:]:
            if xx[0] == country_code:
                continue
            other[xx[0]] += 1
    return other

plt.figure(figsize=(10, 10))
for k, c in enumerate(['US', 'AR', 'MX', 'ES']):
    other = migration(c)
    mc = other.most_common()
    first = mc[:size]
    extra = ('REST', sum([x[1] for x in mc[size:]]))
    first.append(extra)
    plt.subplot(2, 2, k+1)
    _ = plt.pie([x[1] for x in first], labels=[x[0] for x in first])
    plt.title(c)


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
<h3> Exploration using BigQuery </h3>
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
import google.cloud.bigquery as bigquery

bq = bigquery.Client()

sql = """
SELECT DISTANCE, DEP_DELAY
FROM dsongcp.flights_tzcorr
WHERE RAND() < 0.001 AND dep_delay > -20 AND dep_delay < 30 AND distance < 2000
"""
df = bq.query(sql).to_dataframe()

sns.set_style("whitegrid")
g = sns.jointplot(x=df['DISTANCE'], y=df['DEP_DELAY'], kind="hex", height=10, joint_kws={'gridsize':20})


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
<h3> Hexbin plot </h3>

Create a hexbin plot using Spark (repeat of what we did in BigQuery, except that we are now restricting to train days only).
Can you write Python code for it?

df = flights[(flights['distance'] < 2000) & (flights['dep_delay'] > -20) & (flights['dep_delay'] < 30)]
df.describe().show()

df.head(5)


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
Create a table definition (this is done lazily; the files won't be read until we issue a query):
Can you write Python code for it?

inputs = 'gs://{}/flights/tzcorr/all_flights-00000-*'.format(BUCKET) # 1/30th
#inputs = 'gs://{}/flights/tzcorr/all_flights-*'.format(BUCKET)  # FULL
flights = spark.read.json(inputs)

# this view can now be queried ...
flights.createOrReplaceTempView('flights')

# Note that distance is STRING
print(flights.schema)


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
Create dataframe of traindays.
Can you write Python code for it?

from pyspark.sql.types import StructType, StructField, StringType, BooleanType
schema = StructType([
    StructField('FL_DATE', StringType(), True),
    StructField('is_train_day', BooleanType(), True)
])
traindays = spark.read \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .csv('gs://{}/flights/trainday.csv'.format(BUCKET))

traindays.createOrReplaceTempView('traindays')

results = spark.sql('SELECT * FROM traindays')
results.head(5)

statement = """
SELECT
  f.FL_DATE AS date,
  CAST(distance AS FLOAT) AS distance,
  dep_delay,
  IF(arr_delay < 15, 1, 0) AS ontime
FROM flights f
JOIN traindays t
ON f.FL_DATE == t.FL_DATE
WHERE
  t.is_train_day AND
  f.dep_delay IS NOT NULL
ORDER BY
  f.dep_delay DESC
"""
flights = spark.sql(statement)


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
Do Bayes in each bin
Can you write Python code for it?

import pyspark.sql.functions as F
import pandas as pd

df = pd.DataFrame(columns=['dist_thresh', 'delay_thresh', 'frac_ontime'])
for m in range(0, 2): # len(distthresh)-1):
    for n in range(0, len(delaythresh)-1):
        bdf = flights[(flights['distance'] >= distthresh[m])
             & (flights['distance'] < distthresh[m+1])
             & (flights['dep_delay'] >= delaythresh[n])
             & (flights['dep_delay'] < delaythresh[n+1])]
        ontime_frac = bdf.agg(F.sum('ontime')).collect()[0][0] / bdf.agg(F.count('ontime')).collect()[0][0]
        print (m, n, ontime_frac)
        df = df.append({
            'dist_thresh': distthresh[m],
            'delay_thresh': delaythresh[n],
            'frac_ontime': ontime_frac
        }, ignore_index=True)


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
Evaluate model
Use the decision threshold to see how well model does on held-out flights in the first bucket
Can you write Python code for it?

m = 1
statement = """
SELECT
  '{0:.0f}-{1:.0f} miles' AS bin,
  ROUND(SUM(IF(dep_delay < {2:f} AND arr_delay < 15, 1, 0))/COUNT(*), 2) AS correct_nocancel,
  ROUND(SUM(IF(dep_delay >= {2:f} AND arr_delay < 15, 1, 0))/COUNT(*), 2) AS false_positive,
  ROUND(SUM(IF(dep_delay < {2:f} AND arr_delay >= 15, 1, 0))/COUNT(*), 2) AS false_negative,
  ROUND(SUM(IF(dep_delay >= {2:f} AND arr_delay >= 15, 1, 0))/COUNT(*), 2) AS correct_cancel,
  COUNT(*) AS total_flights
FROM flights f
JOIN traindays t
ON f.FL_DATE == t.FL_DATE
WHERE
  t.is_train_day == 'False' AND
  f.distance >= {0:f} AND f.distance < {1:f}
""".format( distthresh[m], distthresh[m+1], bayes[ bayes['dist_thresh'] == distthresh[m] ]['delay_thresh'].values[0] )
print(statement)
eval_flights = spark.sql(statement)
eval_flights.show()

distthresh[-1] = 100000 # bug in Spark SQL, which treats 'inf' as a column name
for m in range(0, len(distthresh)-1):
    statement = """
SELECT
  '{0:.0f}-{1:.0f} miles' AS bin,
  ROUND(SUM(IF(dep_delay < {2:f} AND arr_delay < 15, 1, 0))/COUNT(*), 2) AS correct_nocancel,
  ROUND(SUM(IF(dep_delay >= {2:f} AND arr_delay < 15, 1, 0))/COUNT(*), 2) AS false_positive,
  ROUND(SUM(IF(dep_delay < {2:f} AND arr_delay >= 15, 1, 0))/COUNT(*), 2) AS false_negative,
  ROUND(SUM(IF(dep_delay >= {2:f} AND arr_delay >= 15, 1, 0))/COUNT(*), 2) AS correct_cancel,
  COUNT(*) AS total_flights
FROM flights f
JOIN traindays t
ON f.FL_DATE == t.FL_DATE
WHERE
  t.is_train_day == 'False' AND
  f.distance >= {0:f} AND f.distance < {1:f}
""".format( distthresh[m], distthresh[m+1], bayes[ bayes['dist_thresh'] == distthresh[m] ]['delay_thresh'].values[0] )
    eval_flights = spark.sql(statement)
    eval_flights.show()


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
Example query over the view:
Can you write Python code for it?

results = spark.sql('SELECT COUNT(*) FROM flights WHERE dep_delay > -20 AND CAST(distance AS FLOAT) < 2000')
results.show()


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
Sample the dataframe so that it fits into memory (not a problem in development, but will be on full dataset); then plot it.
Can you write Python code for it?

pdf = df.sample(False, 0.02, 20).toPandas()  # to 100,000 rows approx on complete dataset
g = sns.jointplot(x=pdf['distance'], y=pdf['dep_delay'], kind="hex", height=10, joint_kws={'gridsize':20})


I am working on the file "06_dataproc/quantization.ipynb".
The first task is:
We get close to the 70% threshold only on the last bin.
Let's fine tune the delay thresh around the decision boundary.
Which we know is on the order of 15 minutes.
Can you write Python code for it?

delaythresh = range(10, 20)

df = pd.DataFrame(columns=['dist_thresh', 'delay_thresh', 'frac_ontime'])
for m in range(0, len(distthresh)-1):
    for n in range(0, len(delaythresh)-1):
        bdf = flights[(flights['distance'] >= distthresh[m])
             & (flights['distance'] < distthresh[m+1])
             & (flights['dep_delay'] >= delaythresh[n])
             & (flights['dep_delay'] < delaythresh[n+1])]
        ontime_frac = bdf.agg(F.sum('ontime')).collect()[0][0] / bdf.agg(F.count('ontime')).collect()[0][0]
        print (m, n, ontime_frac)
        df = df.append({
            'dist_thresh': distthresh[m],
            'delay_thresh': delaythresh[n],
            'frac_ontime': ontime_frac
        }, ignore_index=True)

df['score'] = abs(df['frac_ontime'] - 0.7)
bayes = df.sort_values(['score']).groupby('dist_thresh').head(1).sort_values('dist_thresh')
bayes

bayes.to_csv('gs://${BUCKET}/flights/bayes.csv'.format(BUCKET), index=False)
!gsutil cat gs://{BUCKET}/flights/bayes.csv

bayes[ bayes['dist_thresh'] == distthresh[1] ]['delay_thresh'].values[0]


I am working on the file "06_decision_trees.ipynb".
The first task is:
8.
Exercise: Grow a forest.
a. Continuing the previous exercise, generate 1,000 subsets of the training set, each containing 100 instances selected randomly. Hint: you can use Scikit-Learn's ShuffleSplit class for this.
Can you write Python code for it?

from sklearn.model_selection import ShuffleSplit

n_trees = 1000
n_instances = 100

mini_sets = []

rs = ShuffleSplit(n_splits=n_trees, test_size=len(X_train) - n_instances, random_state=42)
for mini_train_index, mini_test_index in rs.split(X_train):
    X_mini_train = X_train[mini_train_index]
    y_mini_train = y_train[mini_train_index]
    mini_sets.append((X_mini_train, y_mini_train))


I am working on the file "06_decision_trees.ipynb".
The first task is:
Exercise solutions
1. to 6.
See appendix A.
7.
Exercise: train and fine-tune a Decision Tree for the moons dataset.
a. Generate a moons dataset using make_moons(n_samples=10000, noise=0.4).
Adding random_state=42 to make this notebook's output constant:
Can you write Python code for it?

from sklearn.datasets import make_moons

X, y = make_moons(n_samples=10000, noise=0.4, random_state=42)


I am working on the file "06_decision_trees.ipynb".
The first task is:
Predicting classes and class probabilities
Can you write Python code for it?

tree_clf.predict_proba([[5, 1.5]])

tree_clf.predict([[5, 1.5]])


I am working on the file "06_decision_trees.ipynb".
The first task is:
Regression trees
Can you write Python code for it?

# Quadratic training set + noise
np.random.seed(42)
m = 200
X = np.random.rand(m, 1)
y = 4 * (X - 0.5) ** 2
y = y + np.random.randn(m, 1) / 10

from sklearn.tree import DecisionTreeRegressor

tree_reg = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg.fit(X, y)

from sklearn.tree import DecisionTreeRegressor

tree_reg1 = DecisionTreeRegressor(random_state=42, max_depth=2)
tree_reg2 = DecisionTreeRegressor(random_state=42, max_depth=3)
tree_reg1.fit(X, y)
tree_reg2.fit(X, y)

def plot_regression_predictions(tree_reg, X, y, axes=[0, 1, -0.2, 1], ylabel="$y$"):
    x1 = np.linspace(axes[0], axes[1], 500).reshape(-1, 1)
    y_pred = tree_reg.predict(x1)
    plt.axis(axes)
    plt.xlabel("$x_1$", fontsize=18)
    if ylabel:
        plt.ylabel(ylabel, fontsize=18, rotation=0)
    plt.plot(X, y, "b.")
    plt.plot(x1, y_pred, "r.-", linewidth=2, label=r"$\hat{y}$")

plt.figure(figsize=(11, 4))
plt.subplot(121)
plot_regression_predictions(tree_reg1, X, y)
for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
    plt.plot([split, split], [-0.2, 1], style, linewidth=2)
plt.text(0.21, 0.65, "Depth=0", fontsize=15)
plt.text(0.01, 0.2, "Depth=1", fontsize=13)
plt.text(0.65, 0.8, "Depth=1", fontsize=13)
plt.legend(loc="upper center", fontsize=18)
plt.title("max_depth=2", fontsize=14)

plt.subplot(122)
plot_regression_predictions(tree_reg2, X, y, ylabel=None)
for split, style in ((0.1973, "k-"), (0.0917, "k--"), (0.7718, "k--")):
    plt.plot([split, split], [-0.2, 1], style, linewidth=2)
for split in (0.0458, 0.1298, 0.2873, 0.9040):
    plt.plot([split, split], [-0.2, 1], "k:", linewidth=1)
plt.text(0.3, 0.5, "Depth=2", fontsize=13)
plt.title("max_depth=3", fontsize=14)

save_fig("tree_regression_plot")
plt.show()

export_graphviz(
        tree_reg1,
        out_file=image_path("regression_tree.dot"),
        feature_names=["x1"],
        rounded=True,
        filled=True
    )

tree_reg1 = DecisionTreeRegressor(random_state=42)
tree_reg2 = DecisionTreeRegressor(random_state=42,...


I am working on the file "06_decision_trees.ipynb".
The first task is:
Sensitivity to training set details
Can you write Python code for it?

X[(X[:, 1]==X[:, 1][y==1].max()) & (y==1)] # widest Iris-Versicolor flower

not_widest_versicolor = (X[:, 1]!=1.8) | (y==2)
X_tweaked = X[not_widest_versicolor]
y_tweaked = y[not_widest_versicolor]

tree_clf_tweaked = DecisionTreeClassifier(max_depth=2, random_state=40)
tree_clf_tweaked.fit(X_tweaked, y_tweaked)

plt.figure(figsize=(8, 4))
plot_decision_boundary(tree_clf_tweaked, X_tweaked, y_tweaked, legend=False)
plt.plot([0, 7.5], [0.8, 0.8], "k-", linewidth=2)
plt.plot([0, 7.5], [1.75, 1.75], "k--", linewidth=2)
plt.text(1.0, 0.9, "Depth=0", fontsize=15)
plt.text(1.0, 1.80, "Depth=1", fontsize=13)

save_fig("decision_tree_instability_plot")
plt.show()

from sklearn.datasets import make_moons
Xm, ym = make_moons(n_samples=100, noise=0.25, random_state=53)

deep_tree_clf1 = DecisionTreeClassifier(random_state=42)
deep_tree_clf2 = DecisionTreeClassifier(min_samples_leaf=4, random_state=42)
deep_tree_clf1.fit(Xm, ym)
deep_tree_clf2.fit(Xm, ym)

plt.figure(figsize=(11, 4))
plt.subplot(121)
plot_decision_boundary(deep_tree_clf1, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
plt.title("No restrictions", fontsize=16)
plt.subplot(122)
plot_decision_boundary(deep_tree_clf2, Xm, ym, axes=[-1.5, 2.5, -1, 1.5], iris=False)
plt.title("min_samples_leaf = {}".format(deep_tree_clf2.min_samples_leaf), fontsize=14)

save_fig("min_samples_leaf_plot")
plt.show()

angle = np.pi / 180 * 20
rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])
Xr = X.dot(rotation_matrix)

tree_clf_r = DecisionTreeClassifier(random_state=42)
tree_clf_r.fit(Xr, y)

plt.figure(figsize=(8, 3))
plot_decision_boundary(tree_clf_r, Xr, y, axes=[0.5, 7.5, -1.0, 1], iris=False)

plt.show()

np.random.seed(6)
Xs = np.random.rand(100, 2) - 0.5
ys = (Xs[:, 0] > 0).astype(np.float32) * 2

angle = np.pi / 4
rotation_matrix = np.array([[np.cos(angle), -np.sin(angle)], [np.sin(angle), np.cos(angle)]])
Xsr = Xs.dot(rotation_matrix)

tree_clf_s = DecisionTreeClassifier(random_state=42)
tree_clf_s.fit(Xs, ys)
tree_clf_sr =...


I am working on the file "06_decision_trees.ipynb".
The first task is:
b. Split it into a training set and a test set using train_test_split().
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)


I am working on the file "06_decision_trees.ipynb".
The first task is:
b. Train one Decision Tree on each subset, using the best hyperparameter values found above. Evaluate these 1,000 Decision Trees on the test set. Since they were trained on smaller sets, these Decision Trees will likely perform worse than the first Decision Tree, achieving only about 80% accuracy.
Can you write Python code for it?

from sklearn.base import clone

forest = [clone(grid_search_cv.best_estimator_) for _ in range(n_trees)]

accuracy_scores = []

for tree, (X_mini_train, y_mini_train) in zip(forest, mini_sets):
    tree.fit(X_mini_train, y_mini_train)

    y_pred = tree.predict(X_test)
    accuracy_scores.append(accuracy_score(y_test, y_pred))

np.mean(accuracy_scores)


I am working on the file "06_decision_trees.ipynb".
The first task is:
c. Now comes the magic. For each test set instance, generate the predictions of the 1,000 Decision Trees, and keep only the most frequent prediction (you can use SciPy's mode() function for this). This gives you majority-vote predictions over the test set.
Can you write Python code for it?

Y_pred = np.empty([n_trees, len(X_test)], dtype=np.uint8)

for tree_index, tree in enumerate(forest):
    Y_pred[tree_index] = tree.predict(X_test)

from scipy.stats import mode

y_pred_majority_votes, n_votes = mode(Y_pred, axis=0)


I am working on the file "06_decision_trees.ipynb".
The first task is:
c. Use grid search with cross-validation (with the help of the GridSearchCV class) to find good hyperparameter values for a DecisionTreeClassifier. Hint: try various values for max_leaf_nodes.
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

params = {'max_leaf_nodes': list(range(2, 100)), 'min_samples_split': [2, 3, 4]}
grid_search_cv = GridSearchCV(DecisionTreeClassifier(random_state=42), params, n_jobs=-1, verbose=1)

grid_search_cv.fit(X_train, y_train)

grid_search_cv.best_estimator_


I am working on the file "06_decision_trees.ipynb".
The first task is:
d. Evaluate these predictions on the test set: you should obtain a slightly higher accuracy than your first model (about 0.5 to 1.5% higher). Congratulations, you have trained a Random Forest classifier!
Can you write Python code for it?


I am working on the file "06_decision_trees.ipynb".
The first task is:
d. Train it on the full training set using these hyperparameters, and measure your model's performance on the test set. You should get roughly 85% to 87% accuracy.
By default, GridSearchCV trains the best model found on the whole training set (you can change this by setting refit=False), so we don't need to do it again. We can simply evaluate the model's accuracy:
Can you write Python code for it?

from sklearn.metrics import accuracy_score

y_pred = grid_search_cv.predict(X_test)
accuracy_score(y_test, y_pred)


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
... and the results visualized.
Can you write Python code for it?

ress['strike'] = strikes
ress.set_index('strike')['value'].plot(figsize=(10, 6))
plt.ylabel('option value estimates')


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
Again, the results visualized (and compared to the sequential results).
Can you write Python code for it?

plt.figure(figsize=(10, 6))
plt.plot(strikes, resp['value'].values, 'r.', label='parallel')
plt.plot(strikes, ress['value'].values, 'b', label='sequential')
plt.legend(loc=0)
plt.ylabel('option value estimates')


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
American Put Option
We also model only a single derivative instrument.
Can you write Python code for it?

gbm = geometric_brownian_motion('gbm_obj', me_gbm)

me_put = market_environment('put', dt.datetime(2015, 1, 1))
me_put.add_constant('maturity', dt.datetime(2015, 12, 31))
me_put.add_constant('strike', 40.)
me_put.add_constant('currency', 'EUR')
me_put.add_environment(val_env)

am_put = valuation_mcs_american_single(
                'am_put', mar_env=me_put, underlying=gbm,
                payoff_func='np.maximum(strike - instrument_values, 0)')


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
Large Portfolio
However, the derivatives_portfolio object we compose consists of 100 derivatives positions. Each option differes with respect to the strike.
Can you write Python code for it?

positions = {}
strikes = np.linspace(80, 120, 100)
for i, strike in enumerate(strikes):
    positions[i] = derivatives_position(
                        name='am_put_pos_%s' % strike,
                        quantity=1,
                        underlyings=['gbm'],
                        mar_env=me_put,
                        otype='American single',
        payoff_func='np.maximum(%5.3f - instrument_values, 0)' % strike)


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
Parallel Valuation
Second, the derivatives portfolio with parallel valuation.
Can you write Python code for it?

port_para = derivatives_portfolio(
                'portfolio',
                positions,
                val_env,
                risk_factors,
                correlations=None,
                parallel=True)  # parallel valuation


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
Single Risk Factor
The example is based on a single risk factor, a geometric_brownian_motion object.
Can you write Python code for it?

# constant short rate
r = constant_short_rate('r', 0.02)

# market environments
me_gbm = market_environment('gbm', dt.datetime(2015, 1, 1))

# geometric Brownian motion
me_gbm.add_constant('initial_value', 100.)
me_gbm.add_constant('volatility', 0.2)
me_gbm.add_constant('currency', 'EUR')
me_gbm.add_constant('model', 'gbm')

# valuation environment
val_env = market_environment('val_env', dt.datetime(2015, 1, 1))
val_env.add_constant('paths', 25000)
val_env.add_constant('frequency', 'M')
val_env.add_curve('discount_curve', r)
val_env.add_constant('starting_date', dt.datetime(2015, 1, 1))
val_env.add_constant('final_date', dt.datetime(2015, 12, 31))

# add valuation environment to market environments
me_gbm.add_environment(val_env)

risk_factors = {'gbm' : me_gbm}


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
Speed.up
The realized speed-up is of course dependend on the hardware used, and in particular the number of cores (threads) available.
Can you write Python code for it?

ts / tp
  # speed-up factor
  # of course harware-dependent

wi = 0.4
plt.figure(figsize=(10, 6))
plt.bar((1.5 - wi/2, 2.5 - wi/2), (ts/ts, tp/ts), width=wi)
plt.xticks((1.5, 2.5), ('sequential', 'parallel'))
plt.ylim(0, 1.1), plt.xlim(0.75, 3.25)
plt.ylabel('relative performance (lower = better)')
plt.title('DX Analytics Portfolio Valuation')


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
The call of the get_values method for the parall valuation case.
Can you write Python code for it?

t0 = time.time()
resp = port_para.get_values()
  # parallel valuation with as many cores as available
tp = time.time() - t0
print('Time in sec %.2f' % tp)


I am working on the file "06_dx_portfolio_parallel.ipynb".
The first task is:
The call of the get_values method to value all instruments ...
Can you write Python code for it?

t0 = time.time()
ress = port_sequ.get_values()
ts = time.time() - t0
print('Time in sec %.2f' % ts)


I am working on the file "06_finite_elements/fe_elastic_1d.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?

# Initialization of setup
# ---------------------------------------------------------------
# Basic parameters
nt    = 2000    # Number of time steps
vs    = 3000    # Wave velocity [m/s]
ro0   = 2500    # Density [kg/m^3]
nx    = 1000    # Number of grid points
isx   = 500     # Source location [m]
xmax  = 10000.  # Maximum length
eps   = 0.5     # Stability limit
iplot = 20      # Snapshot frequency

dx = xmax/(nx-1)           # calculate space increment
x  = np.arange(0, nx)*dx   # initialize space coordinates
x  = np.transpose(x)

h = np.diff(x)  # Element sizes [m]

# parameters
ro = x*0 + ro0
mu = x*0 + ro*vs**2

# time step from stabiity criterion
dt = 0.5*eps*dx/np.max(np.sqrt(mu/ro))
# initialize time axis
t   = np.arange(1, nt+1)*dt

# ---------------------------------------------------------------
# Initialize fields
# ---------------------------------------------------------------
u    = np.zeros(nx)
uold = np.zeros(nx)
unew = np.zeros(nx)

p    = np.zeros(nx)
pold = np.zeros(nx)
pnew = np.zeros(nx)


I am working on the file "06_finite_elements/fe_elastic_1d.ipynb".
The first task is:
4. The Stiffness matrix
On the other hand, the general form of the stiffness matrix is
\begin{equation}
K_{ij} = \int_{D} \mu \partial_x\varphi_i \partial_x\varphi_j dx = \int_{D_{\xi}} \mu \partial_\xi\varphi_i \partial_\xi\varphi_j d\xi
\end{equation}
at this point, the defined basis are introduced. Again, with the help of some algebraic treatment, we arrive at the explicit form of the stiffness matrix
Exercise 3
Implement the stiffness matrix
\begin{equation}
K_{ij} = \frac{\mu}{h}
 \begin{pmatrix}
   \ddots  &    &    &    & 0\
  -1 & 2 & -1 &    &  \
     &-1 &  2 & -1 &  \
     &   & -1 &  2 & -1\
   0 &   &    &    &  \ddots
 \end{pmatrix}
\end{equation}
Display the stiffness matrix to visually inspect how it looks like
Can you write Python code for it?


I am working on the file "06_finite_elements/fe_elastic_1d.ipynb".
The first task is:
5. Finite differences matrices
We implement a finite difference scheme in order to compare with the finite elements solution.
Exercise 4
Implement the finite differences matrices $M$ and $D$. Where $M$ is a diagonal mass matrix containing the inverse densities, and differentiation matrix
\begin{equation}
D_{ij} = \frac{\mu}{dt^2}
 \begin{pmatrix}
   -2 & 1  &    &    &  \
    1 & -2 & 1  &    &  \
      &  & \ddots  &  & \
      &   & 1 &  -2 & 1\
      &   &    &  1 & -2
 \end{pmatrix}
\end{equation}
Display both matrices to visually inspect how they look like
Can you write Python code for it?


I am working on the file "06_finite_elements/fe_elastic_1d.ipynb".
The first task is:
6. Finite element solution
Finally we implement the finite element solution using the computed mass $M$ and stiffness $K$ matrices together with a finite differences extrapolation scheme
\begin{equation}
\mathbf{u}(t + dt) = dt^2 (\mathbf{M}^T)^{-1}[\mathbf{f} - \mathbf{K}^T\mathbf{u}] + 2\mathbf{u} - \mathbf{u}(t-dt).
\end{equation}
Can you write Python code for it?


I am working on the file "06_finite_elements/fe_elastic_1d_solution.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?


I am working on the file "06_finite_elements/fe_elastic_1d_solution.ipynb".
The first task is:
2. Source time function
In 1D the propagating signal is an integral of the source time function. As we look for a Gaussian waveform, we initialize the source time function $f(t)$ using the first derivative of a Gaussian function.
\begin{equation}
f(t) = -\dfrac{2}{\sigma^2}(t - t_0)e^{-\dfrac{(t - t_0)^2}{\sigma^2}}
\end{equation}
Exercise 1
Initialize a source time function called 'src'.  Use $\sigma = 20 dt$ as Gaussian width, and time shift $t_0 = 3\sigma$. Then, visualize the source in a given plot.
Can you write Python code for it?


I am working on the file "06_finite_elements/fe_elastic_1d_solution.ipynb".
The first task is:
5. Finite differences matrices
We implement a finite difference scheme in order to compare with the finite elements solution.
Exercise 4
Implement the finite differences matrices $M$ and $D$. Where $M$ is a diagonal mass matrix containing the inverse densities, and differentiation matrix
\begin{equation}
D_{ij} = \frac{\mu}{dt^2}
 \begin{pmatrix}
   -2 & 1  &    &    &  \
    1 & -2 & 1  &    &  \
      &  & \ddots  &  & \
      &   & 1 &  -2 & 1\
      &   &    &  1 & -2
 \end{pmatrix}
\end{equation}
Display both matrices to visually inspect how they look like
Can you write Python code for it?


I am working on the file "06_finite_elements/fe_static_elasticity.ipynb".
The first task is:
Finite - Difference solution
Can you write Python code for it?

# Poisson's equation with relaxation method
# ---------------------------------------------------------------
nt = 500     # Number of time steps
iplot = 20   # Snapshot frequency

# non-zero boundary conditions
u  = np.zeros(nx)   # set u to zero
du = np.zeros(nx)   # du/dx
f  = np.zeros(nx)   # forcing

f[int(3*nx/4)] = 1./h

xfd = np.arange(0, nx)*h

# ---------------------------------------------------------------
# Initialize animated plot
# ---------------------------------------------------------------
plt.figure(figsize=(8,6))

line1 = plt.plot(x, xfe, color='r', lw=2, label='FE')
line2 = plt.plot(xfd, u, color='k', ls='-.', label='FD relaxation')
plt.title('Static Elasticity with relaxation method', size=16)
plt.ylabel('Displacement, $u$', size=16)
plt.xlabel('Position, $x$', size=16)
plt.legend(loc=4)
plt.grid(True)

plt.ion()   # set interective mode
plt.show()
# ---------------------------------------------------------------
for it in range(nt):
    # Calculate the average of u (omit boundaries)
    for i in range(1, nx-1):
        du[i] =u [i+1] + u[i-1]
    u = 0.5*( f*h**2/mu + du )
    u[0] = 0.15    # Boundary condition at x=0
    u[nx-1] = 0.05 # Boundary condition at x=1
    fd = u

    # --------------------------------------
    # Animation plot. Display both solutions
    if not it % iplot:
        for l in line2:
            l.remove()
            del l
        line1 = plt.plot(x, xfe, color='r', lw=2)
        line2 = plt.plot(xfd, fd, color='k', ls='-.')
        plt.gcf().canvas.draw()


I am working on the file "06_finite_elements/fe_static_elasticity.ipynb".
The first task is:
Finite - Element solution
Can you write Python code for it?


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
"What are all possible labels?"

0 airplane
1 automobile
2 bird
3 cat
4 deer
5 dog
6 frog
7 horse
8 ship
9 truck

"What is the range of values for the image data?"

32 x 32 x 3 x {0..255}

"Are the labels in order or random?"

random
Implement Preprocess Functions
Normalize
In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive.  The return object should be the same shape as x.
Can you write Python code for it?

def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    return np.array(x) / 255

tests.test_normalize(normalize)


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Check Point
This is your first checkpoint.  If you ever decide to come back to this notebook or have to restart the notebook, you can start from here.  The preprocessed data has been saved to disk.
Can you write Python code for it?

import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))

valid_labels


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Checkpoint
The model has been saved to disk.
Test Model
Test your model against the test dataset.  This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')

        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0

        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})
       ...


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Convolution and Max Pooling Layer
Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool to apply convolution then max pooling:
* Create the weight and bias using conv_ksize, conv_num_outputs and the shape of x_tensor.
* Apply a convolution to x_tensor using weight and conv_strides.
 * We recommend you use same padding, but you're welcome to use any padding.
* Add bias
* Add a nonlinear activation to the convolution.
* Apply Max Pooling using pool_ksize and pool_strides.
 * We recommend you use same padding, but you're welcome to use any padding.
Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer, but you can still use TensorFlow's Neural Network package. You may still use the shortcut option for all the other layers.
Can you write Python code for it?

def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    num_units = (int(x_tensor.shape[2]) / conv_strides[0] / pool_strides[0])**2 * conv_num_outputs
    deviation = 1/np.sqrt(num_units)

    W = tf.Variable(tf.truncated_normal([conv_ksize[0], conv_ksize[1],
                                        int(x_tensor.shape[3]), conv_num_outputs],
                                        mean=0.0, stddev=deviation))
    b = tf.Variable(tf.truncated_normal([conv_num_outputs],
                                        mean=0.0, stddev=deviation))

    x_conved = tf.nn.conv2d(x_tensor, W,
                            strides=[1, conv_strides[0], conv_strides[0], 1],
                            padding='SAME')
    x_biased = tf.nn.bias_add(x_conved, b)
    x_rected = tf.nn.relu(x_biased)
    return tf.nn.max_pool(x_rected,
                          ksize=[1, pool_ksize[0], pool_ksize[1], 1],
                          strides=[1, pool_strides[0], pool_strides[1], 1],
                          padding='SAME')

tests.test_con_pool(conv2d_maxpool)


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Create Convolutional Model
Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits.  Use the layers you created above to create this model:

Apply 1, 2, or 3 Convolution and Max Pool layers
Apply a Flatten Layer
Apply 1, 2, or 3 Fully Connected Layers
Apply an Output Layer
Return the output
Apply TensorFlow's Dropout to one or more layers in the model using keep_prob.
Can you write Python code for it?

def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    # Play around with different number of outputs, kernel size and stride

    # 32x32x3
    conv1 = conv2d_maxpool(x, 16,
                           [5, 5], [1, 1],
                           [2, 2], [2, 2])
    # 16x16x16
    conv2 = conv2d_maxpool(conv1, 32,
                           [3, 3], [1, 1],
                           [2, 2], [2, 2])
    # 8x8x32
    conv2f = flatten(conv2)

    # 2048
    fc1 = fully_conn(conv2f, 256)
    fc1d = tf.nn.dropout(fc1, keep_prob)

    # 128
    logits = output(fc1d, 10)

    #10
    return logits

##############################
## Build the Neural Network ##
##############################

# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()

# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()

# Model
logits = conv_net(x, keep_prob)

# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')

# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer().minimize(cost)

# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')

tests.test_conv_net(conv_net)


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Fully-Connected Layer
Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Can you write Python code for it?

def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    height = int(x_tensor.shape[1])
    deviation = 1/np.sqrt(num_outputs)

    W = tf.Variable(tf.truncated_normal([height, num_outputs],
                                        mean=0.0, stddev=deviation))
    b = tf.Variable(tf.truncated_normal([num_outputs],
                                        mean=0.0, stddev=deviation))
    fc1 = tf.add(tf.matmul(x_tensor, W), b)
    return tf.nn.relu(fc1)

tests.test_fully_conn(fully_conn)


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
One-hot encode
Just like the previous code cell, you'll be implementing a function for preprocessing.  This time, you'll implement the one_hot_encode function. The input, x, are a list of labels.  Implement the function to return the list of labels as One-Hot encoded Numpy array.  The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode.  Make sure to save the map of encodings outside the function.
Can you write Python code for it?

def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    from sklearn.preprocessing import LabelBinarizer
    encoder = LabelBinarizer()
    encoder.fit([i for i in range(10)])
    return encoder.transform(x)

tests.test_one_hot_encode(one_hot_encode)


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Output Layer
Implement the output function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Note: Activation, softmax, or cross entropy should not be applied to this.
Can you write Python code for it?

def output(x_tensor, num_outputs):
    """
    Apply a output layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    height = int(x_tensor.shape[1])
    deviation = 1/np.sqrt(num_outputs)

    W = tf.Variable(tf.truncated_normal([height, num_outputs],
                                        mean=0.0, stddev=deviation))
    b = tf.Variable(tf.truncated_normal([num_outputs],
                                        mean=0.0, stddev=deviation))
    return tf.add(tf.matmul(x_tensor, W), b)

tests.test_output(output)


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Randomize Data
As you saw from exploring the data above, the order of the samples are randomized.  It doesn't hurt to randomize it again, but you don't need to for this dataset.
Preprocess all the data and save it
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
Can you write Python code for it?

# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)


I am working on the file "06_image-classification/dlnd_image_classification.ipynb".
The first task is:
Train the Neural Network
Single Optimization
Implement the function train_neural_network to do a single optimization.  The optimization should use optimizer to optimize in session with a feed_dict of the following:
* x for image input
* y for labels
* keep_prob for keep probability for dropout
This function will be called for each batch, so tf.global_variables_initializer() has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
Can you write Python code for it?

def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    session.run(optimizer, feed_dict={x: feature_batch,
                                      y: label_batch,
                                      keep_prob: keep_probability})

tests.test_train_nn(train_neural_network)


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
7.-  concatenate((a1, a2, ..., an)) nos permite concatenar o juntar arrays de una misma dimensión.
Can you write Python code for it?

Erste = array([[2, 2], [4, 2]])
Zweite = array([[1, 1]])
concatenate((Erste, Zweite))


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
8.- logspace(n, m, k) permite producir un array con un número determinado de términos logarítmicamente espaciados donde n representa la potencia primera potencia $10^n$, m > n, la segunda potencia $10^m$ y k el número de términos y al igual que linspace, logspace también incluye los extremos del intervalo.
Can you write Python code for it?

Beispiel = logspace(1, 10, 6)
print(Beispiel)


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
A continuación se enuncian algunas otras funciones útiles de considerar para la manipulación de arrays con NumPy:
* ndim nos entrega la dimensión.
* size nos entrega el número de elementos.
* transpose nos permite transponer el arreglo de forma análoga como ocurre con una matriz invirtiendo todas las dimensiones.
1.2. Operaciones con arrays
NumPy nos permite realizar operaciones clásicas con los arreglos tanto elemento a elemento como globales. Las más comunes dentro de la clase elemento a elemento son:

Suma (+)
Resta (-)
Multiplicación (*)
División (/)
Potenciación (**)

Note que naturalmente se requiere que los arreglos posean las mismas dimensiones. En el caso de las funciones globales dentro de las más útiles es importante destacar:

max retorna el máximo elemento contenido en el array.
min retorna el mínimo elemento contenido en el array.
sum retorna la suma de los elementos.
prod retorna el producto de los elementos.
mean retorna el promedio por fila de los elementos del array.
std retorna la desviación estándar de los elementos contenidos en el array.
cross realiza el producto vectorial bajo dimensiones adecuadas.
outer retorna el producto externo de los vectores contenidos en un array.

1.3. Vectorización de funciones.
Considere la siguiente motivación: NumPy nos entrega la posibilidad de definir funciones que utilicen como datos de entrada arrays. Por ejemplo:
Can you write Python code for it?


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
Además de lo anterior, el interés de vectorizar el código se encuentra en que internamente al realizar lo anterior se eliminan índices que deben ser iterados para la operación a través de bucles sin embargo, esto produce más gasto de memoria y a veces, requiere inclusive replantear y adaptar el código. Veamos a continuación un breve ejemplo de lo anterior:
Ejemplo:
Dada una matriz $\mathcal{Q}_{m \times n}$ con $m > n$ se desea encontrar un vector ortogonal al espacio generado por las columnas de $\mathcal{Q}$.
Can you write Python code for it?

def ort_escalar(Q, v):
    m, n = Q.shape
    for j in range(n):
        v -= dot(Q[:,j], v)*Q[:,j]
    return v

def ort_vectorizado(Q, v):
    proyeccion = dot(Q.transpose(), v)
    v -= dot(Q, proyeccion)
    return v

m, n = 10000, 100
A = 10 * random.random((m, n))
Q, R = linalg.qr(A, mode = 'reduced')
del R

v = random.random(m)
v1 = v.copy()
v2 = v.copy()

%timeit ort_escalar(Q, v1)

%timeit ort_vectorizado(Q, v2)


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
De forma similar, para realizar la operación inversa, es decir, transformar arrays a listas, se puede utilizar la función  tolist .  a través de la sintaxis array.tolist().
Algunas funciones útiles para operar con arrays en NumPy son las siguientes:
1.- arange, el equivalente a range en Python, pero tiene como salida un array en vez de una lista y permite utilizar espaciados de tipo float
Ejemplo:
Can you write Python code for it?

Primero = arange(3)
print(Primero)

Segundo = arange(1, 10, 0.5)
print(Segundo)


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
En la práctica, se puede considerar tanto a x como a y como un objeto en la memoria que mira las características del array. Así, para copiar un array, podemos ocupar el comando copy para crear otro objeto que mire al array. Es importante destacar que copy entrega los valores del array en el momento en que se copia, por lo cual si se modifica el original no lo hará copy.
Ejemplo:
Can you write Python code for it?

x = array([4, 10, 3])
y = copy(x)
x[0] = 20
print(x)
print(y)


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
Lo anterior se debe a que se requiere primeramente vectorizar la función, es decir, realizar el proceso de transformación una implementación escalar a una implementación vectorial traduciéndose en el número de operaciones necesarias para operar con los elementos del array. Lo anterior se puede realizar a través de un decorador (una función que se invoca antes que nuestra función principal y la transforma). Así, usando @vectorize
Can you write Python code for it?

@vectorize

def Comparador(A):
    return A if A > 10 else -A

Comparador(A)


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
Sin embargo, en ocasiones dichas funciones pueden arrojarnos errores puesto que no es posible invocarlas directamente con esta clase de objetos.
Can you write Python code for it?

def Comparador(A):
    return A if A > 10 else -A

Beispiel = array([[2, 2, 10], [10, 4, 10]])
Comparador(A)


I am working on the file "06_libreria_numpy/Lección 3 Numerical Python.ipynb".
The first task is:
tanto en el caso de ones como zeros, la función ones_like y zeroz_like permiten crear un arreglo de igual tamaño a otro arreglo.
5.- eye genera la matriz identidad por lo que se debe indicar tanto número de filas como de columnas por lo cual permite incluso la generación de matrices no cuadradas. Es posible personalizar la diagonal que se desea llenar a través del parámetro k que indica la posición de la diagonal que se desea operar teniendo en mente que la posición por defecto, 0, corresponde a la diagonal principal.
Can you write Python code for it?

Erste = eye(4,3)
Zweite = eye(2,2)
Dritte = eye(4,3, k = -1)
Vierte = eye(4,3, k = 1)
print(Erste)
print(Zweite)
print(Dritte)
print(Vierte)


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
2. Librería Numpy
2.1 Array vs Matrix
Por defecto, la gran mayoria de las funciones de numpy y de scipy asumen que se les pasará un objeto de tipo array.
Veremos las diferencias entre los objetos array y matrix, pero recuerden utilizar array mientras sea posible.
Matrix
Una matrix de numpy se comporta exactamente como esperaríamos de una matriz:
Pros:

Multiplicación utiliza el signo * como es esperable.
Resulta natural si lo único que haremos es algebra lineal.

Contras:

Todas las matrices deben estar completamente alineadas para poder operar correctamente.
Operaciones elementwise son mas dificiles de definir/acceder.
Están exclusivamente definidas en 2D: un vector fila o un vector columna siguen siendo 2D.
Can you write Python code for it?

# Operaciones con np.matrix
A = np.matrix([[1,2],[3,4]])
B = np.matrix([[1, 1],[0,1]], dtype=float)
x = np.matrix([[1],[2]])
print "A =\n", A
print "B =\n", B
print "x =\n", x

print "A+B =\n", A+B
print "A*B =\n", A*B
print "A*x =\n", A*x
print "A*A = A^2 =\n", A**2
print "x.T*A =\n", x.T * A


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
2. Librería Numpy
2.5 Inputs y Outputs
Numpy permite guardar datos de manera sencilla con la función savetxt: siempre debemos dar el nombre del archivo y el array a guardar.
Existen variados argumentos opcionales, pero los mas importantes son:
* header: Línea a escribir como encabezado de los datos
* fmt: Formato con el cual se guardan los datos (%d para enteros, %.5f para flotantes con 5 decimales, %.3E para notación científica con 3 decimales, etc.).
Can you write Python code for it?

# Guardando el archivo con un header en español
encabezado = "Diametro Altura Volumen (Valores truncados a numeros enteros)"
np.savetxt("data/cherry_int.txt", data_int, fmt="%d", header=encabezado)


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
2.6 Índices
Observe que es posible repetir índices, por lo que el array obtenido puede tener más elementos que el array original.
En un arreglo 2d, es necesario pasar 2 arrays, el primero para las filas y el segundo para las columnas.
Can you write Python code for it?

x = np.linspace(10,20,11)
print "x = ", x
print "x.shape = ", x.shape

print "\n"
ind_x_1 = np.array([1,2,3,5,7])
print "ind_x_1 = ", ind_x_1
print "x[ind_x_1] = ", x[ind_x_1]
print "x[ind_x_1].shape = ", x[ind_x_1].shape

print "\n"
ind_x_2 = np.array([0,0,1,2,3,4,5,6,7,-3,-2,-1,-1])
print "ind_x_2 = ", ind_x_2
print "x[ind_x_2] = ", x[ind_x_2]
print "x[ind_x_2].shape = ", x[ind_x_2].shape

A = np.linspace(-90,90,10).reshape(2,5)
print "A = ", A
print "A.shape = ", A.shape

print "\n"
ind_row_A_1 = np.array([0,0,0,1,1])
ind_col_A_1 = np.array([0,2,4,1,3])
print "ind_row_A_1 = ", ind_row_A_1
print "ind_col_A_1 = ", ind_col_A_1
print "A[ind_row_A_1,ind_col_A_1] = ", A[ind_row_A_1,ind_col_A_1]
print "A[ind_row_A_1,ind_col_A_1].shape = ", A[ind_row_A_1,ind_col_A_1].shape

print "\n"
ind_row_A_2 = 1
ind_col_A_2 = np.array([0,1,3])
print "ind_row_A_2 = ", ind_row_A_2
print "ind_col_A_2 = ", ind_col_A_2
print "A[ind_row_A_2,ind_col_A_2] = ", A[ind_row_A_2,ind_col_A_2]
print "A[ind_row_A_2,ind_col_A_2].shape = ", A[ind_row_A_2,ind_col_A_2].shape


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
Contenido

Overview de Numpy y Scipy
Librería Numpy
Arreglos vs Matrices
Axis
Funciones basicas.
Input y Output
Tips

1. Overview de numpy y scipy
¿Cual es la diferencia entre numpy y scipy?
In an ideal world, NumPy would contain nothing but the array data type and the most basic operations: indexing, sorting, reshaping, basic elementwise functions, et cetera. All numerical code would reside in SciPy. However, one of NumPy’s important goals is compatibility, so NumPy tries to retain all features supported by either of its predecessors. Thus NumPy contains some linear algebra functions, even though these more properly belong in SciPy. In any case, SciPy contains more fully-featured versions of the linear algebra modules, as well as many other numerical algorithms. If you are doing scientific computing with python, you should probably install both NumPy and SciPy. Most new features belong in SciPy rather than NumPy.
Link stackoverflow
Por ser python un lenguaje open-source, existen miles de paquetes disponibles creados por individuos o comunidades.  Éstos pueden estar disponibles en un repositorio como github o bitbucket, o bien estar disponibles en el repositorio oficial de python: pypi. En un inicio, cuando no existía una librerías de cálculo científico oficial, varios candidatos proponían soluciones:
* numpy: tenía una excelente representación de vectores, matrices y arreglos, implementados en C y llamados fácilmente desde python
* scipy: proponía linkear a librerías ya elaboradas de calculo científico de alto rendimiento en C o fortran, permitiendo ejecutar rápidamente desde python.
Ambos projectos fueron creciendo en complejidad y alcance, y en vez de competir, decidieron dividir tareas y unificar fuerzas para proponer una plataforma de cálculo científico que reemplazara completamente otros programas.

numpy: Corresponde a lo relacionado con la estructura de los datos (arrays densos y sparse, matrices, constructores especiales, lectura de datos regulares, etc.), pero no las operaciones en sí. Por razones...
Can you write Python code for it?

import numpy as np
print np.version.version # Si alguna vez tienen problemas, verifiquen su version de numpy


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
Desafío 2: Derivación numérica
Implemente el cálculo de la segunda derivada, que puede obtenerse por diferencias finitas centradas mediante
$$ \frac{d f(x_i)}{dx} = \frac{1}{\Delta x^2} \Big( f(x_{i+1}) -2 f(x_{i}) + f(x_{i-1}) \Big)$$
Can you write Python code for it?

def g(x):
    return 1 + x**2 + np.sin(x)

x = np.linspace(0,1,10)
y = g(x)
d2ydx2 = 0 * x # FIX ME
x_aux = 0*d2ydx2 # FIX ME
# To plot
fig = plt.figure(figsize=(12,8))
plt.plot(x, y, label="f")
plt.plot(x_aux, d2ydx2, label="d2f/dx2")
plt.legend(loc="upper left")
plt.show()


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
Desafío 3
Complete el siguiente código:
* Se le provee un array A cuadrado
* Calcule un array B como la multiplicación element-wise de A por sí misma.
* Calcule un array C como la multiplicación matricial de A y B.
* Imprima la matriz C resultante.
* Calcule la suma, promedio y desviación estándar de los valores en la diagonal de C.
* Imprima los valores anteriormente calculados.
Can you write Python code for it?

A = np.outer(np.arange(3),np.arange(3))
print A
# FIX ME
# FIX ME
# FIX ME
# FIX ME
# FIX ME


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
Desafío 5

Lea el archivo data/cherry.txt
Escale la matriz para tener todas las unidades en metros o metros cubicos.
Guarde la matriz en un nuevo archivo data/cherry_mks.txt, con un encabezado apropiado y 2 decimales de precisión para el flotante (pero no en notación científica).
Can you write Python code for it?

# Leer datos
#FIX_ME#

# Convertir a mks
#FIX_ME#

# Guardar en nuevo archivo
#FIX_ME#


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
Desafío 6
La potencia de un aerogenerador, para $k$ una constante relacionada con la geometría y la eficiencia, $\rho$ la densidad del are, $r$ el radio del aerogenerador en metros y $v$ la velocidad el viento en metros por segundo, está dada por:
$$ P = \begin{cases} k \ \rho \ r^2 \ v^3, 3 \leq v \leq 25\ 0,\ eoc\end{cases}$$
Típicamente se considera una valor de $k=0.8$ y una densidad del aire de $\rho = 1.2$ [$kg/m^3$].
Calcule el número de aerogeneradores activos, la potencia promedio y la potencia total generada por los 11 generadores del parque Eólico Canela 1.
Los valores de radio del aerogenerador (en metros) y la velocidad del viento (en kilometros por hora) se indican a continuación en arreglos en el código numérico.
Can you write Python code for it?

import numpy as np
k = 0.8
rho = 1.2 #
r_m = np.array([ 25.,  25.,  25.,  25.,  25.,  25.,  20.,  20.,  20.,  20.,  20.])
v_kmh = np.array([10.4, 12.6,  9.7,  7.2, 12.3, 10.8, 12.9, 13.0,  8.6, 12.6, 11.2]) # En kilometros por hora
P = 0
n_activos = 0
P_mean = 0.0
P_total =  0.0
print "Existen %d aerogeneradores activos del total de %d" %(n_activos, r.shape[0])
print "La potencia promedio de los aeorgeneradores es {0:.2f} ".format(P_mean)
print "La potencia promedio de los aeorgeneradores es " + str(P_total)


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
Importante
Ipython notebook es interactivo y permite la utilización de tabulación para ofrecer sugerencias o enseñar ayuda (no solo para numpy, sino que para cualquier código en python).
Pruebe los siguientes ejemplos:
Can you write Python code for it?

# Presionar tabulacción con el cursor despues de np.arr
np.arr

# Presionar Ctr-Enter para obtener la documentacion de la funcion np.array usando "?"
np.array?

# Presionar Ctr-Enter
%who

x = 10
%who


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
Observación

Cabe destacar que al tomar slices (subsecciones) de un arreglo obtenemos siempre un arreglo con menores dimensiones que el original.
Esta notación es extremadamente conveniente, puesto que nos permite manipular el array sin necesitar conocer el tamaño del array y escribir de manera compacta las fórmulas numéricas.

Por ejemplo, implementar una derivada numérica es tan simple como sigue.
Can you write Python code for it?

def f(x):
    return 1 + x**2

x = np.array([0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]) # O utilizar np.linspace!
y = f(x) # Tan facil como llamar f sobre x
dydx = ( y[1:] - y[:-1] ) / ( x[1:] - x[:-1] )
x_aux = 0.5*(x[1:] + x[:-1])
# To plot
fig = plt.figure(figsize=(12,8))
plt.plot(x, y, '-s', label="f")
plt.plot(x_aux, dydx, '-s', label="df/dx")
plt.legend(loc="upper left")
plt.show()


I am working on the file "06_libreria_numpy/numpy.ipynb".
The first task is:
Revisemos si el archivo quedó bien escrito. Cambiaremos de python a bash para utilizar los comandos del terminal:
Can you write Python code for it?


I am working on the file "06_numpy_internals.ipynb".
The first task is:
If we want to avoid this behavior, so that when we get a new completely independent object B copied from A, then we need to do a so-called deep copy using the function np.copy:
Can you write Python code for it?

B = np.copy(A)

# now, if we modify B, A is not affected
B[0,0] = -5

B

A

aid(A) == aid(B)


I am working on the file "06_page_rank/page_rank.ipynb".
The first task is:
135318 из 123771 уникальных страниц. Предполагаю, что причина превосходства числа найденных страниц связанно с наличием редиректов, которые не предлагалось обрабатывать.
Посчитаем PageRank и найдём 20 топовых страниц.
Can you write Python code for it?

%%time
pagerank = nx.pagerank(graph)

top20 = [0.] * 20
for url, rank in pagerank.iteritems():
    heappushpop(top20, (rank, url))

while top20:
    rank, url = heappop(top20)
    print rank, url


I am working on the file "06_page_rank/page_rank.ipynb".
The first task is:
Как видно, у небольшого числа урлов оказалась проблема с кодировкой, дропнем их из графа. И посмотрим сколько страниц нашли.
Can you write Python code for it?


I am working on the file "06_preprocessing/06h_tftransform.ipynb".
The first task is:
Avoid training-serving skew using TensorFlow Transform
In this notebook, we show how to use tf.transform to carry out preprocessing efficiently,
but save the preprocessing operations so that they are automatically applied during inference.
Enable GPU and set up helper functions
This notebook and pretty much every other notebook in this repository
will run faster if you are using a GPU.
On Colab:
- Navigate to Edit→Notebook Settings
- Select GPU from the Hardware Accelerator drop-down
On Cloud AI Platform Notebooks:
- Navigate to https://console.cloud.google.com/ai-platform/notebooks
- Create an instance with a GPU or select your instance and add a GPU
Next, we'll confirm that we can connect to the GPU with tensorflow:
Can you write Python code for it?

import tensorflow as tf
print(tf.version.VERSION)
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))


I am working on the file "06_preprocessing/06h_tftransform.ipynb".
The first task is:
Display preprocessing data
Note that the files contain already preprocessed (scaled, resized images),
so we can simply read the data and display it.
Can you write Python code for it?

import matplotlib.pylab as plt
import numpy as np
import tensorflow as tf

IMG_HEIGHT = 448
IMG_WIDTH = 448
IMG_CHANNELS = 3
CLASS_NAMES = 'daisy dandelion roses sunflowers tulips'.split()

ds = tf.data.experimental.make_batched_features_dataset(
    './flower_tftransform/train-00000-of-00016.gz',
    batch_size=5,
    features = {
        'image': tf.io.FixedLenFeature([IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS], tf.float32),
        'label': tf.io.FixedLenFeature([], tf.string),
        'label_int': tf.io.FixedLenFeature([], tf.int64)
    },
    reader=lambda filenames: tf.data.TFRecordDataset(filenames, compression_type='GZIP')
)
for feats in ds.take(1):
    print(feats['image'].shape)

f, ax = plt.subplots(1, 5, figsize=(15,15))
for feats in ds.take(1):
    for idx in range(5): # batchsize
        ax[idx].imshow((feats['image'][idx].numpy()));
        ax[idx].set_title(feats['label'][idx].numpy())
        ax[idx].axis('off')


I am working on the file "06_preprocessing/06h_tftransform.ipynb".
The first task is:
Predictions
For serving, we will write a serving function that calls the transform model
followed by our actual model.
We will look at serving functions in Chapter 7. For now, we show how to explicitly
call the transform function followed by model.predict()
Can you write Python code for it?

!saved_model_cli show --all --dir ./flower_tftransform/tft/transform_fn

# get some files to do inference on.
filenames = [
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/98992760_53ed1d26a9.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9939430464_5f5861ebab.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9965757055_ff01b5ee6f_n.jpg'
]
img_bytes = [
    tf.io.read_file(filename) for filename in filenames
]
label = [
    'n/a' for filename in filenames
] # not used in inference
label_int = [
    -1 for filename in filenames
] # not used in inference

# by calling the preproc function, we get images of the right size & crop
preproc = tf.keras.models.load_model('./flower_tftransform/tft/transform_fn').signatures['transform_signature']
preprocessed = preproc(img_bytes=tf.convert_to_tensor(img_bytes),
                       label=tf.convert_to_tensor(label, dtype=tf.string),
                       label_int=tf.convert_to_tensor(label_int, dtype=tf.int64))

# then we call model.predict() and take the argmx of the result
pred_label_index = tf.math.argmax(model.predict(preprocessed)).numpy()
print(pred_label_index)


I am working on the file "06_profiling/06_actividad.ipynb".
The first task is:
Just an example of usage
Can you write Python code for it?

img = mpimg.imread('data/doge.jpg')
img = rgb2gray(img)

image_plot(img)

lbp_image = lbp_representation(img)
image_plot(lbp_image)

h = lbp_histogram(img)
histogram_plot(h)


I am working on the file "06_profiling/06_actividad.ipynb".
The first task is:
Local Binary Pattern Representation
Local Binary Pattern (LBP) is a simple yet very efficient texture operator which labels the pixels of an image by thresholding the neighborhood of each pixel and considers the result as a binary number.
Due to its discriminative power and computational simplicity, LBP texture operator has become a popular approach in various applications.
Perhaps the most important property of the LBP operator in real-world applications is its robustness to monotonic gray-scale changes caused, for example, by illumination variations. Another important property is its computational simplicity, which makes it possible to analyze images in challenging real-time settings.
Example of how LBP works:

Type of features LBP compute:

Then, each pixel in the image is encoded as a 8-bit binary number (or its corresponding decimal representation) which represents a unique feature. With this representation we can make a histogram of the distribution of such features!
Can you write Python code for it?

def lbp(region):
    """
    Region is a 3x3 ndarray
    """
    ret = (region > region[1,1]).astype(int)
    return (2**7)*ret[0,0] + (2**6)*ret[0,1] + (2**5)*ret[0,2] + (2**4)*ret[1,2] + (2**3)*ret[2,2] + \
           (2**2)*ret[2,1] + (2**1)*ret[2,0] + (2**0)*ret[1,0]

def lbp_representation(image):
    """
    Compute lbp representation of image
    """
    m,n = image.shape
    ret = np.empty((m-2,n-2))
    for i in range(1,m-1):
        for j in range(1,n-1):
            ret[i-1,j-1] = lbp(image[i-1:i+2, j-1:j+2])
    return ret

def lbp_histogram(image):
    """
    Compute lbp histogram representation of image
    """
    lbp_image = lbp_representation(image)
    m,n = lbp_image.shape
    hist = np.zeros(256)
    for i in range(m):
        for j in range(n):
            hist[lbp_image[i,j]] += 1
    return hist


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
<div id='Application' />
4.- Application: K-Means Clustering Algorithm
This is an algorithm that find structure over unlabeled data, i.e, it is an unsupervised learning algorithm. It is very simple, and works as follows:

initialize $k$ cluster centroids
Repeat the following:
2.1.- For each point, compute which centroid is nearest to it.
2.2.- For each centroid, move its location to the mean location of the points assigned to it.

Let's first generate a set of random 2D points:
Can you write Python code for it?

points = np.vstack(((np.random.randn(150, 2) * 0.75 + np.array([1, 0])),
                  (np.random.randn(50, 2) * 0.25 + np.array([-0.5, 0.5])),
                  (np.random.randn(50, 2) * 0.5 + np.array([-0.5, -0.5]))))

points.shape

plt.figure(figsize=(7,7))
plt.scatter(points[:, 0], points[:, 1])
plt.grid()
plt.show()

def initialize_centroids(points, k):
    """returns k centroids from the initial points"""
    centroids = points.copy()
    np.random.shuffle(centroids)
    return centroids[:k]


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
And lest visualize the choosen (initial) centroids:
Can you write Python code for it?

centroids = initialize_centroids(points, 3)

plt.figure(figsize=(7,7))
plt.scatter(points[:, 0], points[:, 1])
plt.scatter(centroids[:, 0], centroids[:, 1], c='r', s=100)
plt.grid()
plt.show()


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
And we can compare the performance improvement with the quotient between the best measured times:
Can you write Python code for it?