I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
Let us implement now the cosine similarity function between two items using the TF measurement.  It is defined as:
$$ cosim(a,b) = \frac{a \cdot b}{\|a\| \|b\|} = \frac{\sum a_i b_i}{\sqrt{\sum a_i^2} \sqrt{\sum b_i^2}} $$
EXERCISE: We observe in the formula that we need a dot product function (between two sparse dictionaries), and a norm function.  Complete the definition of the "dotproduct" and "norm" functions below.
The answer should be:

The dot product between tf_dict1 and tf_dict2 is 0.055556

The norm of tf_dict1 is 0.212762

The norm of tf_dict2 is 0.408248

The cosim of a vector with itself must be one: 1.000000

The cosim of a vector with itself must be one: 1.000000

The cosim between two vectors does not depend of the order: 0.639602 = 0.639602

Can you write Python code for it?

import math

def dotprod(tf_dict1, tf_dict2):
    <COMPLETAR>
    return dotProd

def norm(tf_dict1):
    <COMPLETAR>
    return norma

def cosim(tf_dict1, tf_dict2):
    <COMPLETAR>
    return cs

tf_dict1 = compute_tf(tokeniza(texto, stopwords))
tf_dict2 = compute_tf(tokeniza(texto[0:60], stopwords))

print "The dot product between tf_dict1 and tf_dict2 is %f\n" % dotprod(tf_dict1, tf_dict2)
print "The norm of tf_dict1 is %f\n" % norm(tf_dict1)
print "The norm of tf_dict2 is %f\n" % norm(tf_dict2)
print "The cosim of a vector with itself must be one: %f\n" % cosim(tf_dict1, tf_dict1)
print "The cosim of a vector with itself must be one: %f\n" % cosim(tf_dict2, tf_dict2)
print "The cosim between two vectors does not depend of the order: %f = %f\n" % (cosim(tf_dict1, tf_dict2) , cosim(tf_dict2, tf_dict1))


I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
Tokenizing the RDDS
We want to modify the structure of the RDDs such that every item has now the format (ID, [list of tokens])
EXERCISE:  Use the 'tokeniza' function to tokenize the amazonSmall RDD and count the number of unique tokens in that RDD. Repeat for the other datasets. It is better you define a 'tokenizeRDD' function that takes a RDD as argument (plus the stopwords) and returns a modified RDD.
The answer should be:

These are the first 5 elements of the processed amazonSmallRDD:

('b000jz4hqo', ['clickart', '950', '000', 'premier', 'image', 'pack', 'dvd', 'rom', 'broderbund'])

('b0006zf55o', ['ca', 'international', 'arcserve', 'lap', 'desktop', 'oem', '30pk', 'oem', 'arcserve', 'backup', 'v11', '1', 'win', '30u', 'laptops', 'desktops', 'computer', 'associates'])

('b00004tkvy', ['noah', 'ark', 'activity', 'center', 'jewel', 'case', 'ages', '3', '8', 'victory', 'multimedia'])

('b000g80lqo', ['peachtree', 'sage', 'premium', 'accounting', 'nonprofits', '2007', 'peachtree', 'premium', 'accounting', 'nonprofits', '2007', 'affordable', 'easy', 'use', 'accounting', 'solution', 'provides', 'donor', 'grantor', 'management', 're', 'like', 'nonprofit', 'organizations', 're', 'constantly', 'striving', 'maximize', 'every', 'dollar', 'annual', 'operating', 'budget', 'financial', 'reporting', 'programs', 'funds', 'advanced', 'operational', 'reporting', 'rock', 'solid', 'core', 'accounting', 'features', 'made', 'peachtree', 'choice', 'hundreds', 'thousands', 'small', 'businesses', 'result', 'accounting', 'solution', 'tailor', 'made', 'challenges', 'operating', 'nonprofit', 'organization', 'keep', 'audit', 'trail', 'record', 'report', 'changes', 'made', 'transactions', 'improve', 'data', 'integrity', 'prior', 'period', 'locking', 'archive', 'organization', 'data', 'snap', 'shots', 'data', 'closed', 'year', 'set', 'individual', 'user', 'profiles', 'password', 'protection', 'peachtree', 'restore', 'wizard', 'restores', 'backed', 'data', 'files', 'plus', 'web', 'transactions', 'customized', 'forms',...
Can you write Python code for it?

def tokenizeRDD(RDD, stopwords):
    tokenizedRDD = (RDD
                         .<COMPLETAR>
                         .cache()
                     )
    return tokenizedRDD

amazonSmallRDDtokenized = tokenizeRDD(amazonSmallRDD, stopwords)
print "Finished with amazonSmallRDD."
amazonRDDtokenized = tokenizeRDD(amazonRDD, stopwords)
print "Finished with amazonRDD."
googleSmallRDDtokenized = tokenizeRDD(googleSmallRDD, stopwords)
print "Finished with googleSmallRDD."
googleRDDtokenized = tokenizeRDD(googleRDD, stopwords)
print "Finished with googleRDD.\n"

print "These are the first 5 elements of the processed amazonSmallRDD:\n"
elements = amazonSmallRDDtokenized.take(5)
for l in elements:
    print l
    print " "


I am working on the file "0_Introduction/Intro_PySpark_3.ipynb".
The first task is:
We will use these functions to process the datasets and find the two most similar records between amazon and google.
EXERCISE: Generate a new "allPairsRDD" that contains all possible combinations between elements of googleSmallRDDtokenized and amazonSmallRDDtokenized. You may want to use the "cartesian" transformation. Transform "allPairsRDD" to obtain a new RDD with the format (googleID, amazonID, cosim). Finally, print the contents with the largest cosine similarity.
The answer should be:

The allPairsRDD has 40000 elements.

This is one of the elements in allPairsRDD:

[(('http://www.google.com/base/feeds/snippets/11448761432933644608', ['spanish', 'vocabulary', 'builder', 'expand', 'vocabulary', 'contains', 'fun', 'lessons', 'teach', 'entertain', 'll', 'quickly', 'find', 'mastering', 'new', 'terms', 'includes', 'games']), ('b000jz4hqo', ['clickart', '950', '000', 'premier', 'image', 'pack', 'dvd', 'rom', 'broderbund']))]

Can you write Python code for it?

allPairsRDD = (<COMPLETAR>
              .cartesian(<COMPLETAR>)
              .cache())

print "The allPairsRDD has %d elements.\n" % allPairsRDD.count()

print "This is one of the elements in allPairsRDD:\n"
print allPairsRDD.take(1)


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
2) Define a short name to use the library: import lib_name as lib
Can you write Python code for it?


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
2. Working with Python lists
Python lists are containers that hold a number of other objects, in a given order. To create a list, just put different comma-separated values between square brackets
Can you write Python code for it?

list1 = ['student', 'teacher', 1997, 2000]
print list1
list2 = [1, 2, 3, 4, 5 ]
print list2
list3 = ["a", "b", "c", "d"]
print list3


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
7. Modules import
Python lets you define modules which are files consisting of Python code. A module can define functions, classes and variables.
Most Python distributions already include the most popular modules with predefined libraries which make our programmer lifes easier. Some well-known libraries are: time, sys, os, numpy, ...
There are several ways to import a library:
1) Import all the contents of the library: import lib_name
Note: You have to call these methods as part of the library
Can you write Python code for it?

import time
import sys

print time.time()  # returns the current processor time in seconds
Nwait = 2

print "Waiting %d seconds..." % Nwait
sys.stdout.flush()  # Try to execute commenting this line...

time.sleep(2) # suspends execution for the given number of seconds
print time.time() # returns the current processor time in seconds again!!!
print "Done!"


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
As we said, you must define a function using def, then the name of the function and in brackets ( ) the list of arguments of the function. The function will not return anything unless you specify it with a return statement.
The expresion under the name of the function in triple quotes is a Documentation string or DOCSTRING and, as expected, it is used to document your code. For example, it is printed if you type the help command:
Can you write Python code for it?


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
Indentation
The above fragment, allows us also to discuss some important characteristics of the Python language syntaxis:

Unlike other languages, Python does not require to use the 'end' keyword to indicate that a given code fragment finishes. Instead, Python recurs to indentation.

Indentation in Python is mandatory. A block is composed by statatements indected at same level and if it constains a nested block it is simply indented further to the right.

As a convention each indentation consists of 4 spaces (for each level of indentation).

The condition lines conclude with ':', which are then followed by the indented blocks that will be executed only when the indicated conditions are satisfied.

The statement for lets you iterate over the items of any sequence (a list or a string), in the order that they appear in the sequence
Can you write Python code for it?

words = ['cat', 'window', 'open-course']
for w in words:
     print w, len(w)


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
Ok. This is what we expected, colors1 is keeping its own values.
Let's change one element of colors2 now:
Can you write Python code for it?

colors1 = ["red", "green"]
colors2 = colors1
colors2[1] = "blue"
print colors1


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
One tricky feature in python is how it evaluates the arguments that you pass to a function call. The most common evaluation strategies when passing arguments to a function have been call-by-value and call-by-reference. Python uses a mixture of these two, which is known as "Call-by-Object", sometimes also called "Call by Object Reference" or "Call by Sharing". Let's see it with an example:
Can you write Python code for it?

def add_square_to_list(x, my_list, dummy_list):
    x = x ** 2
    my_list.append(x)
    dummy_list = ["I", "am", "not", "a" , "dummy", "list"]

x = 5
my_list =[4, 9, 16]
dummy_list = ["I", "am", "a" , "dummy", "list"]

add_square_to_list(x, my_list, dummy_list)
print x
print my_list
print dummy_list


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
Ouch! That wasn't expected.
The explanation is that there has been no new assignment to colors2, colors2 still points to colors1. Only one of its elements, and consequently an element of colors1 has been changed.
It is possible to completely copy shallow list structures with the slice operator without having any of the side effects, which we have described above. But that will be a problem when having nested lists. In that case you should use import the module copy.
Can you write Python code for it?


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
The first two lines do not seem problematic. But when y is set to 50, what will happen to the value of x? C programmers will assume that x will be changed to 50 as well, because we said before that y "points" to the location of x. But this is not a C-pointer. Because x and y will not share the same value anymore, y gets his or her own memory location, containing 50 and x sticks to 42.
If you are not a C programmer, the observable results of the assignments answer our expectations. But it can be problematic, if we copy mutable objects like lists and dictionaries.
Python creates real copies only if it has to, i.e. if the user, the programmer, explicitly demands it.
Let's see a couple of examples:
Can you write Python code for it?

colors1 = ["red", "green"]
colors2 = colors1
colors2 = ["rouge", "vert"]
print colors1


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
To read the content of a file, use the function f.read():
Can you write Python code for it?

f2 = open('workfile', 'r')
text=f2.read()
f2.close()
print text


I am working on the file "0_Introduction/Intro_Python_1.ipynb".
The first task is:
You can also read line by line from the file identifier
Can you write Python code for it?

f2 = open('workfile', 'r')
for line in f2:
    print line

f2.close()


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
1.b Elementwise operations
One of the main advantages of numpy arrays is that operations are propagated to the individual elements of the array
Can you write Python code for it?


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
1.e Matrix multiplications
Finally numpy provides all the basic matrix operations: multiplications, dot products, ...
You can find information about them in the Numpy manual
Can you write Python code for it?

x=np.array([1,2,3])
y=np.array([1,2,3])
print x*y  #Element-wise
print np.multiply(x,y) #Element-wise
print sum(x*y) # dot product
print  #Fast matrix product


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
4.3 Train/Test splitting
Now we are about to start doing machine learning. But, first of all, we have to separate our data between train and test.
The train data will be used to adjust the parameters of our model  (train).
The test data will be used to evaluate our model.
EXERCISE: Use sklearn.cross_validation.train_test_split to split the data in train (60%) and test (40%). Save the results in variables named X_train, X_test, y_train, y_test.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
Random_state = 131

X_train, X_test, y_train, y_test = train_test_split(<COMPLETAR>, <COMPLETAR>, test_size=<COMPLETAR>, random_state=Random_state)

print X_train.shape
print X_test.shape
print y_train.shape
print y_test.shape


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
4.7 Saving the results
Finally we will save all our prediction for the model with K=1 in a csv file. To do so you can use the following code Snippet, where y_pred are the predicted output values for test.
Can you write Python code for it?

y_pred = y_predict_1.squeeze()
csv_file_object = csv.writer(open('output.csv', 'wb'))
for index, y_aux in enumerate(<COMPLETAR>):      # Run through each row in the csv file,
    csv_file_object.writerow([index,y_aux])


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
According to this plot, which classes seem more difficult to distinguish?
4. Regression example
Now that we know how to load some data and visualize it we will try to solve a simple regression task.
Our objective in this example is to predict the crime rates in different areas of the US using some socio-demographic data.
This dataset has 127 socioeconomic variables, of different nature: categorical, integer, real, and for some of them there are also missing data (check wikipedia). This is usually a problem when training machine learning models, but we will ignore that problem and take only a small number of variables that we think can be useful for regression and which have no missing values.

population: population for community
householdsize: mean people per household
medIncome: median household income

The objective in the regresion problem is another real value that contains the total number of violent crimes per 100K population.
4.1 Loading the data
First of all, load the data from file communities.csv in a new array. This array should have 1994 rows (instances) and 128 columns.
Can you write Python code for it?

csv_file_object = csv.reader(open('communities.csv', 'rb'))
datalist = []
for row in csv_file_object:
    datalist.append(row)
data = np.array(datalist)
print np.shape(data)


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
Check again the shape and dimension of my_array3
Can you write Python code for it?

print my_array3.shape
print my_array3.ndim


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
Check the functions np.linspace, np.logspace and np.meshgrid which let you create more sophisticated ranges
You can create numpy arrays in several ways. For example numpy provides a number of functions to create special types of matrices.
Create 3 arrays usings ones, zeros and eye. If you have any doubt about the parameters of the functions have a look at the help with the function help( ).
Can you write Python code for it?

A1 = np.zeros((3,4))
print A1
A2 = np.ones((2,6))
print A2
A3 = np.eye(5)
print A3


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
Compare this with operations over python lists:
Can you write Python code for it?


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
Define a new 3x2 array named my_array2 with  [1, 2, 3] in the first row and [4,5,6] in the second.
Check the dimensions of the array.
Can you write Python code for it?

my_array2 = np.array([[1, 2, 3],[4, 5, 6]])
print my_array2
print np.shape(my_array2)


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
EXERCISE: Plot each variable in X_com versus y_com to have a first (partial) view of the data.
Can you write Python code for it?

plt.figure()
plt.plot(<COMPLETAR>, 'bo')
plt.xlabel('X_com[0]')
plt.ylabel('y_com')

plt.figure()
plt.plot(<COMPLETAR>, 'ro')
plt.xlabel('X_com[1]')
plt.ylabel('y_com')

plt.figure()
plt.plot(<COMPLETAR>, 'go')
plt.xlabel('X_com[2]')
plt.ylabel('y_com')


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
EXERCISE: Try to compute the dot product with python arrays:
The answer should be:

14

Can you write Python code for it?

x=[1,2,3]
dot_product_x = <COMPLETAR>
print dot_product_x


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
If you have correctly computed it you will see that my_array3 is one dimensional. Sometimes this can be a problem when you are working with 2D matrixes (and vectors can be considered as 2D matrixes with one of the sizes equal to 1). To solve this, numpy provides the newaxis constant.
Can you write Python code for it?


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
In the previous code we have saved the features in matrix X and the class labels in the vector labels. Both are 2D numpy arrays.
We are also printing the shapes of each variable (see that we can also use array_name.shape to get the shape, apart from function shape( )). This shape checking is good to see if we are not making mistakes.
3.2 Visualizing the data
Extract the first two features of the data (sepal length and width) and plot the first versus the second in a figure, use a different color for the data corresponding to different classes.
First of all, you probably want to split the data according to each class label.
Can you write Python code for it?

x = X[:,0:2]
#print len(set(list(label)))
list_label = [l[0] for l in label]
labels = list(set(list_label))
colors = ['bo', 'ro', 'go']
#print list_label
plt.figure()
for i, l in enumerate(labels):
    pos = np.where(np.array(list_label) == l)
    plt.plot(x[pos,0], x[pos,1], colors[i])
    plt.xlabel('Sepal length')
    plt.ylabel('Sepal width')


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
Numpy also includes the functions hstack() and vstack() to concatenate by columns or rows, respectively.
EXERCISE: Use these functions to concatenate my_array and my_array2 by rows and columns.
The answer should be:

[[ 1 12]
 [ 3 14]]

Can you write Python code for it?


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
One important thing to consider when you do slicing are the dimensions of the output array. Run the following cell and check the shape of my_array3. Check also its dimension with ndim function:
Can you write Python code for it?

my_array = np.array([[1, 2],[3, 4]])
my_array3 = my_array[:,1]
print my_array3
print my_array[1,0:2]

print my_array3.shape
print my_array3.ndim


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
Take the columns (5,6,17) of the data and save them in a matrix X_com. This will be our input data. Convert this array into a float array. The shape should be (1994,3)
EXERCISE: Get the last column of the data and save it in an array called y_com. Convert this matrix into a float array.
Check that the shape is (1994,1) .
Can you write Python code for it?

X_com = <COMPLETAR>
Nrow = np.shape(data)[0]
Ncol = np.shape(data)[1]
print X_com.shape

y_com = <COMPLETAR>
print y_com.shape


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
Until now, we have created arrays defining their elements. But you can also create it defining the range
Can you write Python code for it?

my_new_array = np.arange(3,11,2)
print my_new_array


I am working on the file "0_Introduction/Intro_Python_2.ipynb".
The first task is:
When you try to index different rows and columns of a matrix you have to define it element by element. For example, consider that we want to select elements of rows [0, 3] and columns [0, 2], we have to define the row 0 index for each column to be selected....
Can you write Python code for it?

x = np.array([[ 0,  1,  2], [ 3,  4,  5], [ 6,  7,  8], [ 9, 10, 11]])

# We want to select elements of rows [0, 3] and columns [0, 2]
rows = np.array([[0, 0],[3, 3]], dtype=np.intp)
columns = np.array([[0, 2],[0, 2]], dtype=np.intp)

print x[rows, columns]


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
1.- Introduction to Python dictionaries
In this practice we will extensively use Python Dictionaries, in this preliminary section we will learn to build, modify and use them.
Python dictionaries are hash tables (also known as associative memories, associative arrays), a data structure indexed by keys, which can be any immutable (hashable) type. It is best to think of a dictionary as an unordered set of (key, value) pairs, with the requirement that the keys are unique (within one dictionary). Let's see a simple example of dictionary creation and manipulation.
Can you write Python code for it?

inventary_dict = {'milk': 23, 'coockies': 12, 'chocolate': 26, 'yogourt': 5}

print "This is the original dictionary:"
print inventary_dict
print " "

print "This is the value associated to 'milk':"
print inventary_dict['milk']
print " "

print "We add a new element to the dictionary:"
inventary_dict.update({'sugar': 103})
print inventary_dict
print " "

print "We increment the value of one of the elements:"
inventary_dict['coockies'] += 10
print inventary_dict
print " "


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
2.- Downloading a webpage
The urllib library allows to download any web content. Other tools for building crawling bots for massive downloading are also available (Scrapy).
We will use the urllib library to download a webpage.
EXERCISE: Complete the code below to compute the number of downloaded characters and print the first 1000.
Can you write Python code for it?

agent = {'User-Agent':"Mozilla/4.0"}
url1 = "https://www.u-tad.com/conocenos/conoce-u-tad/"
request = urllib2.Request(url1, headers=agent)
page = urllib2.urlopen(request).read()

n_caracteres_descargados = <COMPLETAR>
print "La página descargada tiene %d caracteres." % n_caracteres_descargados

print "Estos son los primeros 1000 caracteres:"
print "=" * 100
print <COMPLETAR>
print "=" * 100


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
3.- The Google Translate webpage syntax
Let's explore the behaviour of the Google Translate webpage.  Execute the following code, and open the resulting URL in a web browser (click on it, or copy-paste it to a web browser, if it does not automatically open). You should be able to read the translation text in that webpage. Try to identify the syntax of the query, and manually modify the destiny language, as well as the text to translate.
Can you write Python code for it?

url = "https://translate.google.com/m?hl=de&sl=auto&q=adiós+amigos"
print url


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
4.- Downloading the html code with the translation
We will construct a query link and download the page containing the desired translation.
 EXERCISE : Write a function 'get_html' that takes as input the destiny language and the text to be translated and returns the html code of the page.
Can you write Python code for it?

def get_html(lang, text):
    agent = {'User-Agent':"Mozilla/4.0"}
    url = <COMPLETAR>
    request = urllib2.Request(url, headers=agent)
    html = urllib2.urlopen(request).read()
    return html

html = get_html(destiny_language, my_text)

n_caracteres_descargados = <COMPLETAR>
print "La página descargada tiene %d caracteres." % n_caracteres_descargados
print "=" * 100
print html
print "=" * 100


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
5.- Postprocessing the downloaded webpage
We will analyze the html content to obtain the desired translation.
 EXERCISE : Complete the function 'translate' that takes as input the destiny language and the text to be translated and returns the translation.
Can you write Python code for it?

def translate(lang, text):
    html = <COMPLETAR>
    bs = BeautifulSoup(html, "html5lib")
    translation =bs.findAll('div')[2].text
    return translation

key = 'en'
print u"Traducción al " + unicode(languages_dict[key],'utf-8') + ":"
print translate(key, my_text)


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
7.- Translation to all available languages
We will use the goslate library to get a full dictionary of languages.
 EXERCISE : Add the code to print the full list of codes and languages names.
The answer should be:

gu: Gujarati
zh-TW: Chinese (Traditional)
gd: Scots Gaelic
ga: Irish
gl: Galician
lb: Luxembourgish
la: Latin
lo: Lao
(...)

Can you write Python code for it?

gs = goslate.Goslate()
all_languages_dict = gs.get_languages()

view_codes(<COMPLETAR>)


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
BeautifulSoup is a powerfull library to postprocess the HTML code, let's see one example, we will extract the text and remove the HTML markup.
Can you write Python code for it?

bs = BeautifulSoup(page, "html5lib")
for script in bs(["script", "style"]):
    script.extract()
text_utad = bs.get_text()
text_utad = ' '.join(text_utad.split())

print text_utad


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
EXERCISE :  Define  the following 'languages_dict' dictionary, that stores the languages our translator will work on. Note that both keys and values are strings:

en: Inglés
zh: Chino
de: Alemán
it: Italiano
es: Español

Can you write Python code for it?

languages_dict = <COMPLETAR>

print "Vamos a traducir de %s a %s." % (languages_dict['es'], languages_dict['it'])


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
EXERCISE : Translate the text to all languages in the dict.
Can you write Python code for it?


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
EXERCISE : Use the function in the previous section to translate the text to all languages in the dict.
Can you write Python code for it?

for <COMPLETAR>:
    print u"Traducción al " + unicode(languages_dict[key],'utf-8') + ":"
    print <COMPLETAR>
    print " "


I am working on the file "0_Introduction/Intro_Python_3.ipynb".
The first task is:
EXERCISE: Define a function that takes as argument the destiny language and the text to be translated and returns the url. Check the result by clicking on the printed link.
Can you write Python code for it?

destiny_language = 'it'
my_text = "Hola a todos mis amigos"

def url_translate(destiny_language, text):
    url = <COMPLETAR> % (destiny_language, "auto", text.replace(<COMPLETAR>))
    return url

url = url_translate(destiny_language, my_text)
print url


I am working on the file "0_intro_numerical_methods.ipynb".
The first task is:
Boat race
Given a river (say a sinusoid) find the total length actually rowed over a given interval
$$f(x) = A \sin x$$
Can you write Python code for it?

x = numpy.linspace(0, 4 * numpy.pi)
plt.plot(x, 2.0 * numpy.sin(x))
plt.title("River Sine")
plt.xlabel("x")
plt.ylabel("y")
plt.axis([0, 4*numpy.pi, -2, 2])
plt.show()


I am working on the file "0_intro_numerical_methods.ipynb".
The first task is:
We need to calculate the function $f(x)$'s arc-length from $[0, 4 \pi]$
$$L = \int_0^{4 \pi} \sqrt{1 + |f'(x)|^2} dx$$
In general need numerical quadrature.
Non-Linear population growth
Lotka-Volterra Predator-Prey model
$$\frac{d R}{dt} = R \cdot (a - b \cdot F)$$
$$\frac{d F}{dt} = F \cdot (c \cdot R + d)$$

Where are the steady states?
How do we solve the initial value problem?
How do we understand the non-linear dynamics?
How do we evaluate whether this is a good model?

Interpolation and Data Fitting
Finding trends in real data represented without a closed form (analytical form).
Sunspot counts
Can you write Python code for it?

data = numpy.loadtxt("./data/sunspot.dat")
data.shape
plt.plot(data[:, 0], data[:, 1])
plt.xlabel("Year")
plt.ylabel("Number")
plt.title("Number of Sunspots")
plt.show()


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
All of the 10 people are politicians, but about half of them have rather tenuous connections with Obama, other than the fact that they are politicians.

Francisco Barrio is a Mexican politician, and a former governor of Chihuahua.
Walter Mondale and Don Bonker are Democrats who made their career in late 1970s.
Wynn Normington Hugh-Jones is a former British diplomat and Liberal Party official.
Andy Anstett is a former politician in Manitoba, Canada.

Nearest neighbors with raw word counts got some things right, showing all politicians in the query result, but missed finer and important details.
For instance, let's find out why Francisco Barrio was considered a close neighbor of Obama.  To do this, let's look at the most frequently used words in each of Barack Obama and Francisco Barrio's pages:
Can you write Python code for it?

wiki[wiki['name'] == 'Barack Obama'][['word_count']].stack('word_count', new_column_name=['word','count']).sort('count',ascending=False)

def top_words(name):
    """
    Get a table of the most frequent words in the given person's wikipedia page.
    """
    row = wiki[wiki['name'] == name]
    word_count_table = row[['word_count']].stack('word_count', new_column_name=['word','count'])
    return word_count_table.sort('count', ascending=False)

obama_words = top_words('Barack Obama')
obama_words

barrio_words = top_words('Francisco Barrio')
barrio_words


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
But one may wonder, is Biden's article that different from Obama's, more so than, say, Schiliro's? It turns out that, when we compute nearest neighbors using the Euclidean distances, we unwittingly favor short articles over long ones. Let us compute the length of each Wikipedia document, and examine the document lengths for the 100 nearest neighbors to Obama's page.
Can you write Python code for it?

def compute_length(row):
    return len(row['text'])

wiki['length'] = wiki.apply(compute_length)

nearest_neighbors_euclidean = model_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=100)
nearest_neighbors_euclidean = nearest_neighbors_euclidean.join(wiki[['name', 'length']], on={'reference_label':'name'})

nearest_neighbors_euclidean.sort('rank')


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Extract word count vectors
As we have seen in Course 1, we can extract word count vectors using a GraphLab utility function.  We add this as a column in wiki.
Can you write Python code for it?

wiki['word_count'] = graphlab.text_analytics.count_words(wiki['text'])

wiki


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Find nearest neighbors
Let's start by finding the nearest neighbors of the Barack Obama page using the word count vectors to represent the articles and Euclidean distance to measure distance.  For this, again will we use a GraphLab Create implementation of nearest neighbor search.
Can you write Python code for it?

model = graphlab.nearest_neighbors.create(wiki, label='name', features=['word_count'],
                                          method='brute_force', distance='euclidean')


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
From a glance at the above table, things look better.  For example, we now see Joe Biden as Barack Obama's nearest neighbor!  We also see Hillary Clinton on the list.  This list looks even more plausible as nearest neighbors of Barack Obama.
Let's make a plot to better visualize the effect of having used cosine distance in place of Euclidean on our TF-IDF vectors.
Can you write Python code for it?

plt.figure(figsize=(10.5,4.5))
plt.figure(figsize=(10.5,4.5))
plt.hist(wiki['length'], 50, color='k', edgecolor='None', histtype='stepfilled', normed=True,
         label='Entire Wikipedia', zorder=3, alpha=0.8)
plt.hist(nearest_neighbors_euclidean['length'], 50, color='r', edgecolor='None', histtype='stepfilled', normed=True,
         label='100 NNs of Obama (Euclidean)', zorder=10, alpha=0.8)
plt.hist(nearest_neighbors_cosine['length'], 50, color='b', edgecolor='None', histtype='stepfilled', normed=True,
         label='100 NNs of Obama (cosine)', zorder=11, alpha=0.8)
plt.axvline(x=wiki['length'][wiki['name'] == 'Barack Obama'][0], color='k', linestyle='--', linewidth=4,
           label='Length of Barack Obama', zorder=2)
plt.axvline(x=wiki['length'][wiki['name'] == 'Joe Biden'][0], color='g', linestyle='--', linewidth=4,
           label='Length of Joe Biden', zorder=1)
plt.axis([1000, 5500, 0, 0.004])
plt.legend(loc='best', prop={'size':15})
plt.title('Distribution of document length')
plt.xlabel('# of words')
plt.ylabel('Percentage')
plt.rcParams.update({'font.size': 16})
plt.tight_layout()


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Indeed, the 100 nearest neighbors using cosine distance provide a sampling across the range of document lengths, rather than just short articles like Euclidean distance provided.
Moral of the story: In deciding the features and distance measures, check if they produce results that make sense for your particular application.
Problem with cosine distances: tweets vs. long articles
Happily ever after? Not so fast. Cosine distances ignore all document lengths, which may be great in certain situations but not in others. For instance, consider the following (admittedly contrived) example.
+--------------------------------------------------------+
|                                             +--------+ |
|  One that shall not be named                | Follow | |
|  @username                                  +--------+ |
|                                                        |
|  Democratic governments control law in response to     |
|  popular act.                                          |
|                                                        |
|  8:05 AM - 16 May 2016                                 |
|                                                        |
|  Reply   Retweet (1,332)   Like (300)                  |
|                                                        |
+--------------------------------------------------------+
How similar is this tweet to Barack Obama's Wikipedia article? Let's transform the tweet into TF-IDF features, using an encoder fit to the Wikipedia dataset.  (That is, let's treat this tweet as an article in our Wikipedia dataset and see what happens.)
Can you write Python code for it?

sf = graphlab.SFrame({'text': ['democratic governments control law in response to popular act']})
sf['word_count'] = graphlab.text_analytics.count_words(sf['text'])

encoder = graphlab.feature_engineering.TFIDF(features=['word_count'], output_column_prefix='tf_idf')
encoder.fit(wiki)
sf = encoder.transform(sf)
sf


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Let's look at the TF-IDF vectors for this tweet and for Barack Obama's Wikipedia entry, just to visually see their differences.
Can you write Python code for it?

tweet_tf_idf = sf[0]['tf_idf.word_count']
tweet_tf_idf

obama = wiki[wiki['name'] == 'Barack Obama']
obama


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Let's look at the top 10 nearest neighbors by performing the following query:
Can you write Python code for it?

model.query(wiki[wiki['name']=='Barack Obama'], label='name', k=10)


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Load Wikipedia dataset
We will be using the same dataset of Wikipedia pages that we used in the Machine Learning Foundations course (Course 1). Each element of the dataset consists of a link to the wikipedia article, the name of the person, and the text of the article (in lowercase).
Can you write Python code for it?

wiki = graphlab.SFrame('people_wiki.gl')

wiki

wiki['URI'][1]


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Note. Even though common words are swamping out important subtle differences, commonalities in rarer political words still matter on the margin. This is why politicians are being listed in the query result instead of musicians, for example. In the next subsection, we will introduce a different metric that will place greater emphasis on those rarer words.
TF-IDF to the rescue
Much of the perceived commonalities between Obama and Barrio were due to occurrences of extremely frequent words, such as "the", "and", and "his". So nearest neighbors is recommending plausible results sometimes for the wrong reasons.
To retrieve articles that are more relevant, we should focus more on rare words that don't happen in every article. TF-IDF (term frequency–inverse document frequency) is a feature representation that penalizes words that are too common.  Let's use GraphLab Create's implementation of TF-IDF and repeat the search for the 10 nearest neighbors of Barack Obama:
Can you write Python code for it?

wiki['tf_idf'] = graphlab.text_analytics.tf_idf(wiki['word_count'])

model_tf_idf = graphlab.nearest_neighbors.create(wiki, label='name', features=['tf_idf'],
                                                 method='brute_force', distance='euclidean')

model_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=10)


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Note. The join operation does not enforce any particular ordering on the shared column. So to obtain, say, the five common words that appear most often in Obama's article, sort the combined table by the Obama column. Don't forget ascending=False to display largest counts first.
Can you write Python code for it?


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Now, compute the cosine distance between the Barack Obama article and this tweet:
Can you write Python code for it?

obama_tf_idf = obama[0]['tf_idf']
graphlab.toolkits.distances.cosine(obama_tf_idf, tweet_tf_idf)


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Quiz Question. Among the words that appear in both Barack Obama and Francisco Barrio, take the 5 that appear most frequently in Obama. How many of the articles in the Wikipedia dataset contain all of those 5 words?
Answer.answer is 56066
Hint:
* Refer to the previous paragraph for finding the words that appear in both articles. Sort the common words by their frequencies in Obama's article and take the largest five.
* Each word count vector is a Python dictionary. For each word count vector in SFrame, you'd have to check if the set of the 5 common words is a subset of the keys of the word count vector. Complete the function has_top_words to accomplish the task.
  - Convert the list of top 5 words into set using the syntax
set(common_words)
    where common_words is a Python list. See this link if you're curious about Python sets.
  - Extract the list of keys of the word count dictionary by calling the keys() method.
  - Convert the list of keys into a set as well.
  - Use issubset() method to check if all 5 words are among the keys.
* Now apply the has_top_words function on every row of the SFrame.
* Compute the sum of the result column to obtain the number of articles containing all the 5 top words.
Can you write Python code for it?

obama_words = top_words('Barack Obama')

common_words = list(obama_words[:5]['word'])
type(common_words)
#mmon_words
set(common_words)

common_words = list(top_words('Barack Obama')[:5]['word'])  # Barack Obama 5 largest words
print common_words
def has_top_words(word_count_vector):
    # extract the keys of word_count_vector and convert it to a set
    unique_words = set(word_count_vector.keys())  #using keys() method and using set() method convert list to set
    # return True if common_words is a subset of unique_words
    # return False otherwise
    return set(common_words).issubset(unique_words)  # YOUR CODE HERE

wiki['has_top_words'] = wiki['word_count'].apply(has_top_words)

# use has_top_words column to answer the quiz question
print wiki['has_top_words']
sum(wiki['has_top_words'])


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Quiz Question. Measure the pairwise distance between the Wikipedia pages of Barack Obama, George W. Bush, and Joe Biden. Which of the three pairs has the smallest distance?
Answer. Biden and Bush
Hint: To compute the Euclidean distance between two dictionaries, use graphlab.toolkits.distances.euclidean. Refer to this link for usage.
Can you write Python code for it?

a = graphlab.SFrame(wiki[wiki['name']=='Barack Obama']['word_count'])[0]['X1']
b = graphlab.SFrame(wiki[wiki['name']=='George W. Bush']['word_count'])[0]['X1']
c = graphlab.SFrame(wiki[wiki['name']=='Joe Biden']['word_count'])[0]['X1']

graphlab.toolkits.distances.euclidean(a,b) # Obama and Bush

graphlab.toolkits.distances.euclidean(a,c) # Obama and Joe

graphlab.toolkits.distances.euclidean(b,c) # Bush and Joe+++++++++++++


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Relative to the rest of Wikipedia, nearest neighbors of Obama are overwhemingly short, most of them being shorter than 2000 words. The bias towards short articles is not appropriate in this application as there is really no reason to  favor short articles over long articles (they are all Wikipedia articles, after all). Many Wikipedia articles are 2500 words or more, and both Obama and Biden are over 2500 words long.
Note: Both word-count features and TF-IDF are proportional to word frequencies. While TF-IDF penalizes very common words, longer articles tend to have longer TF-IDF vectors simply because they have more words in them.
To remove this bias, we turn to cosine distances:
$$
d(\mathbf{x},\mathbf{y}) = 1 - \frac{\mathbf{x}^T\mathbf{y}}{\|\mathbf{x}\| \|\mathbf{y}\|}
$$
Cosine distances let us compare word distributions of two articles of varying lengths.
Let us train a new nearest neighbor model, this time with cosine distances.  We then repeat the search for Obama's 100 nearest neighbors.
Can you write Python code for it?

model2_tf_idf = graphlab.nearest_neighbors.create(wiki, label='name', features=['tf_idf'],
                                                  method='brute_force', distance='cosine')

nearest_neighbors_cosine = model2_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=100)
nearest_neighbors_cosine = nearest_neighbors_cosine.join(wiki[['name', 'length']], on={'reference_label':'name'})

nearest_neighbors_cosine.sort('rank')


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Since both tables contained the column named count, SFrame automatically renamed one of them to prevent confusion. Let's rename the columns to tell which one is for which. By inspection, we see that the first column (count) is for Obama and the second (count.1) for Barrio.
Can you write Python code for it?

combined_words = combined_words.rename({'count':'Obama', 'count.1':'Barrio'})
combined_words


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
The first 10 words should say: Obama, law, democratic, Senate, presidential, president, policy, states, office, 2011.
Quiz Question. Among the words that appear in both Barack Obama and Phil Schiliro, take the 5 that have largest weights in Obama. How many of the articles in the Wikipedia dataset contain all of those 5 words?
Answer.14
Can you write Python code for it?

common_words = set(list(combination2_words[:5]['word']))
common_words

# common_words = common_words

def has_top_words(word_count_vector):
    # extract the keys of word_count_vector and convert it to a set
    unique_words = set(word_count_vector.keys())
    # return True if common_words is a subset of unique_words
    # return False otherwise
    return common_words.issubset(unique_words)  # YOUR CODE HERE

wiki['has_top_words'] = wiki['word_count'].apply(has_top_words)

# use has_top_words column to answer the quiz question
print wiki['has_top_words']  # YOUR CODE HERE
sum(wiki['has_top_words'])


I am working on the file "0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Using the join operation we learned earlier, try your hands at computing the common words shared by Obama's and Schiliro's articles. Sort the common words by their TF-IDF weights in Obama's document.
Can you write Python code for it?

combination2_words = obama_tf_idf.join(schiliro_tf_idf,on='word').sort('weight',ascending=False)
combination2_words

combination2_words = combination2_words.rename({'weight':'Obama', 'weight.1':'Schiliro'})
combination2_words

combination2_words = combination2_words.sort('Obama', ascending=False)
combination2_words


I am working on the file "0_obtain_comic_data_j.ipynb".
The first task is:
Web APIでは，パラメータidで「ユニークID」「雑誌巻号ID」あるいは「雑誌コード」を，pageで検索結果の取得ページ番号（1ページあたり100件，デフォルトは1）を指定することができます．ここで，週刊少年ジャンプは「雑誌巻号ID」にJUMPrglを含むため，id=JUMPrglを指定します．また，週刊少年ジャンプの検索結果は合計24ページ（2320件）あるため，pageに1から24を順次指定する必要があります．
詳細はWebAPI仕様をご参照ください．
なお，2017年3月31日より文化庁メディア芸術データベースのURLが変更されたため，WebAPI仕様に記載のリクエストURL（https://mediaarts-db.jp/mg/api/v1/results_magazines）ではなく，新しいURL（https://mediaarts-db.bunka.go.jp/mg/api/v1/results_magazines）を使用する必要があることにご注意ください．
Can you write Python code for it?

magazines = search_magazine()

len(magazines)

magazines[0]

magazines[-1]


I am working on the file "0_obtain_comic_data_j.ipynb".
The first task is:
雑誌巻号情報の取得
以下の関数extract_data()で必要な目次情報を抽出し，save_data()で目次情報を保存します．
Can you write Python code for it?

def extract_data(content):
    """
    contentに含まれる目次情報を取得する関数です．
    - year: 発行年
    - no: 号数
    - title: 作品名
    - author: 著者
    - color: カラーか否か
    - pages: 掲載ページ数
    - start_page: 作品のスタートページ
    - best: 巻頭から数えた掲載順
    - worst: 巻末から数えた掲載順
    """

    # マンガ作品のみ抽出します．
    comics = [comic for comic in content['contents']
             if comic['category']=='マンガ作品']
    data = []
    year = int(content['basics']['date_indication'][:4])

    # 号数が記載されていない場合があるので，例外処理が必要です．
    try:
        no = int(content['basics']['number_indication'])
    except ValueError:
        no = content['basics']['number_indication']

    for comic in comics:
        title= comic['work']
        if not title:
            continue

        # ページ数が記載されていない作品があるので，例外処理が必要です．
        # 特に理由はないですが，無記載の作品は10ページとして処理を進めます．
        try:
            pages = int(comic['work_pages'])
        except ValueError:
            pages = 10

        # 「いぬまるだしっ」等，1週に複数話掲載されている作品に対応するため
        # data中にすでにtitleが含まれる場合は，新規datumとして登録せずに，
        # 既存のdatumのページ数のみ加算します．
        if len(data) > 0 and title in [datum['title'] for datum in data]:
            data[[datum['title'] for datum in
                  data].index(title)]['pages'] += pages
        else:
            data.append({
                'year': year,
                'no': no,
                'title': comic['work'],
                'author': comic['author'],
                'subtitle': comic['subtitle'],
                'color': int('カラー' in comic['note']),
                'pages': int(comic['work_pages']),
                'start_pages': int(comic['start_page'])
            })

    # 企画物のミニマンガを除外するため，合計5ページ以下のdatumはリストから除外します．
    filterd_data = [datum for datum in data if datum['pages'] > 5]
    for n, datum in enumerate(filterd_data):
        datum['best'] = n + 1
        datum['worst'] = len(filterd_data) - n

    return filterd_data


I am working on the file "0_obtain_comic_data_j.ipynb".
The first task is:
雑誌巻号検索結果の取得
以下の関数search_magazine()を使って，週刊少年ジャンプの雑誌巻号情報を検索し，結果を変数magazinesに保存します．ここで取得したユニークIDは，次節の「雑誌巻号情報の取得」に必要になります．
Can you write Python code for it?

def search_magazine(key='JUMPrgl', n_pages=25):
    """
    「ユニークID」「雑誌巻号ID」あるいは「雑誌コード」にkey含む雑誌を，
    n_pages分取得する関数です．
    """

    url = 'https://mediaarts-db.bunka.go.jp/mg/api/v1/results_magazines?id=' + \
        key + '&page='
    magazines = []

    for i in range(1, n_pages):
        response = urllib.request.urlopen(url + str(i))
        content = json.loads(response.read().decode('utf8'))
        magazines.extend(content['results'])
    return magazines


I am working on the file "0_quickstart/0_quickstart.ipynb".
The first task is:
3. 可視化
数値計算が完了したものの，数字の羅列では運動の様子が直感的には理解できない．そこで，振り子の運動の様子を表すアニメーション動画や，振子の角度の時間変化を表すグラフを作る．
以下に示すanimfunc はアニメーション動画を生成する関数である．基本的な原理はパラパラ漫画であり，動画フレームごとに振り子の絵を描いては保存するの繰り返しを行っている．フレームごとの更新はupdate_figure関数が行っている．
Can you write Python code for it?

import matplotlib.pyplot as plt
import matplotlib.animation as animation
from math import cos

def animfunc(s, t):
    ''' Create mp4 movie file of a pendulum '''
    plt.close()
    fig = plt.figure()
    plt.axis('scaled')
    plt.xlim(-1, 1)
    plt.ylim(-1.5, .5)
    plt.grid('on')
    draw_ceiling, = plt.plot([-2, 2], [0, 0], c='k', lw=2)
    draw_pendulum, = plt.plot([], [], lw=4, c='b')
    draw_mass, = plt.plot([], [], lw=2, marker='o', ms=20, mew=4, mec='b', mfc='c')
    indicate_time = plt.text(-0.3, 0.25, [], fontsize=12)

    def update_figure(i):
        ''' Set data of each movie frame '''
        mass_x = l*sin(s[i, 0])
        mass_y = - l*cos(s[i, 0])
        pendlum_x = [0, mass_x]
        pendlum_y = [0, mass_y]
        draw_pendulum.set_data(pendlum_x, pendlum_y)
        draw_mass.set_data(mass_x, mass_y)
        indicate_time.set_text('t = {0:4.2f} [s]'.format(t[i]))

    ''' Create a movie file '''
    line_ani = animation.FuncAnimation(fig, update_figure, frames=len(t))
    line_ani.save('./pendulum.mp4', fps=t_fps)
    print('pendulum.mp4 created')


I am working on the file "0_quickstart/0_quickstart.ipynb".
The first task is:
<video width="432" height="288" controls>
  <source src="pendulum.mp4" type="video/mp4">
</video>

一方，振子の動きをグラフに示したい場合は，例えば次のようにしてプロットを行えば良い．次のプログラムを実行することで，pendulum_graph.pngのようなイメージファイルが保存される．
Can you write Python code for it?

plt.figure()
plt.plot(t, s[:, 0])
plt.xlabel('t [s]')
plt.ylabel('theta [rad]')
plt.savefig('pendulum_graph.png')


I am working on the file "1 - Analysing InceptionV3 results.ipynb".
The first task is:
It seems almost suspicious that our bad image detection accuracy is so much lower than the other categories. Let's break down this accuracy data further into a confusion matrix:
Can you write Python code for it?

conf_matrix_df = pd.DataFrame(all_projects_solution.confusion_matrix, index=class_names, columns=class_names)
display(HTML(conf_matrix_df.to_html()))


I am working on the file "1 - Analysing InceptionV3 results.ipynb".
The first task is:
So, it's not quite as open-and-shut as the previous set of examples, but it still helps build confidence in the model, and support the hypothesis that the MapSwipe data is far from accurate.
Individual Project Accuracy
Everything we've done so far has considered one giant dataset, composed of a large number of projects (where each project corresponds to relatively small geographic area). It's interesting to see if the model's accuracy varies between the individual projects. To do this, I generated individual datasets for each project (using a similar workflow to that described previously), and then used the same model as before to grade each individual project's test dataset.
Can you write Python code for it?

import json
from os.path import isdir, join
import os
import urllib.request

from bokeh.plotting import figure, ColumnDataSource
from bokeh.models import HoverTool
from bokeh.io import output_notebook, show

with urllib.request.urlopen("http://api.mapswipe.org/projects.json") as url:
    projects = json.loads(url.read().decode())

individual_projects_dir = '../individual_projects/'
project_dirs = [d for d in os.listdir(individual_projects_dir) if isdir(join(individual_projects_dir, d))]
project_dirs.sort(key=int)

project_ids = []
accuracies = []
names = []
tile_counts = []

for project_id in project_dirs:
    solutions_csv = join(individual_projects_dir, project_id, 'test', 'solutions.csv')
    if (os.path.getsize(solutions_csv) > 0):
        solution = Solution(
            ground_truth_solutions_file_to_map(solutions_csv),
            predictions_file_to_map(join(individual_projects_dir, project_id, 'initial_inception_v3_all_layers.out'))
        )
        project_ids.append(project_id)
        accuracies.append(solution.accuracy * 100)
        names.append(projects[project_id]['name'])
        tile_counts.append(solution.tile_count)

output_notebook()

source = ColumnDataSource(data=dict(
    x=project_ids,
    y=accuracies,
    names=names,
    tile_counts=tile_counts
))

hover = HoverTool(tooltips=[
    ("Project ID", "@x"),
    ("Accuracy", "@y%"),
    ("Name", "@names"),
    ("Tile count", "@tile_counts")
])

p = figure(plot_width=800, plot_height=600, tools=[hover],
           title="Test accuracy for each MapSwipe project")

p.circle('x', 'y', size=10, source=source)

show(p)


I am working on the file "1 - Analysing InceptionV3 results.ipynb".
The first task is:
So, we're about 64% accurate. This means that 64% of the time, we select the right class for the tile (bad imagery, built, or empty). If we guessed at random, we'd expect to be 33% accurate (there are three classes, so we have a one in three chance of being correct). Let's break down that accuracy in to a per-category accuracy:
Can you write Python code for it?

category_accuracies_df = pd.DataFrame(all_projects_solution.category_accuracies, index=class_names, columns=['Test dataset'])
display(HTML(category_accuracies_df.transpose().to_html()))


I am working on the file "1 - Analysing InceptionV3 results.ipynb".
The first task is:
The rows correspond to what our model predicted, and the columns correspond to the official solution. If our model was perfect, we'd expect to have non-zero entries on the main diagonal (top left to bottom right), and zeroes everywhere else. The biggest non-zero entry corresponds to examples that officially (according to the MapSwipe data) are bad imagery, but our model has classified as empty. Let's take a look at the examples where we were most confident that the imagery was empty, but was actually bad (according to the official solution).
Can you write Python code for it?

quadkeys = [x[0] for x in all_projects_solution.classified_as(predicted_class='empty', solution_class='bad_imagery')[0:9]]
tableau(quadkeys, all_projects_solution)


I am working on the file "1 - Authorize Earth Engine.ipynb".
The first task is:
Finally, check if the notebook server is authorized to access the Earth Engine backend servers.
Can you write Python code for it?

# Code to check if authorized to access Earth Engine.
import cStringIO
import os
import urllib

def isAuthorized():
  try:
    ee.Initialize()
    return True
  except:
    return False

form_item_layout = ipywidgets.Layout(width="100%", align_items='center')

if isAuthorized():

  def revoke_credentials(sender):
    credentials = ee.oauth.get_credentials_path()
    if os.path.exists(credentials):
      os.remove(credentials)
    print('Credentials have been revoked.')

  # Define widgets that may be displayed.
  auth_status_button = ipywidgets.Button(
    layout=form_item_layout,
    disabled = True,
    description = 'The server is authorized to access Earth Engine',
    button_style = 'success',
    icon = 'check'
  )

  instructions = ipywidgets.Button(
    layout=form_item_layout,
    description = 'Click here to revoke authorization',
    button_style = 'danger',
    disabled = False,
  )
  instructions.on_click(revoke_credentials)

else:

  def save_credentials(sender):
    try:
      token = ee.oauth.request_token(get_auth_textbox.value.strip())
    except Exception as e:
      print(e)
      return
    ee.oauth.write_token(token)
    get_auth_textbox.value = ''  # Clear the textbox.
    print('Successfully saved authorization token.')

  # Define widgets that may be displayed.
  get_auth_textbox = ipywidgets.Text(
    placeholder='Paste authorization code here',
    description='Authentication Code:'
  )
  get_auth_textbox.on_submit(save_credentials)

  auth_status_button = ipywidgets.Button(
    layout=form_item_layout,
    button_style = 'danger',
    description = 'The server is not authorized to access Earth Engine',
    disabled = True
  )

  instructions = ipywidgets.VBox(
    [
      ipywidgets.HTML(
        'Click on the link below to start the authentication and authorization process. '
        'Once you have received an authorization code, paste it in the box below and press return.'
      ),
      ipywidgets.HTML(
        '<a href="{url}" target="auth">Open Authentication...


I am working on the file "1 - Authorize Earth Engine.ipynb".
The first task is:
Next, check if the Earth Engine API is available on the server.
Can you write Python code for it?

# Code to check the Earth Engine API library.
try:
  import ee
  print('The Earth Engine Python API (version {0}) is available on this server.'.format(
      ee.__version__
    ))
except ImportError:
  print('The Earth Engine Python API library is not available on this server.\n'
        'Please see https://developers.google.com/earth-engine/python_install '
        'for information on installing the library.')
  raise


I am working on the file "1 - Authorize Earth Engine.ipynb".
The first task is:
Once the server is authorized, you can retrieve data from Earth Engine and use it in the notebook.
Can you write Python code for it?

# Code to display an Earth Engine generated image.
from IPython.display import Image

url = ee.Image("CGIAR/SRTM90_V4").getThumbUrl({'min':0, 'max':3000})
Image(url=url)


I am working on the file "1 - Introduction.ipynb".
The first task is:
At run time, functions must be defined by the execution of a "def" before they are called.
Can you write Python code for it?


I am working on the file "1 - Introduction.ipynb".
The first task is:
In Jupyter notebook you can also find help on a function by pressing shift-TAB after a function's first parenthesis.
Can you write Python code for it?


I am working on the file "1 - Introduction.ipynb".
The first task is:
Notice also how the lines that make up the function or if-statement are grouped by all having the same level of indentation. We also presented 2 different ways to repeat strings, using the + operator which is more user-friendly, but * also works because it's Python's "repeat" operator, meaning that '-' * 10 gives '----------', a neat way to create an onscreen "line." In the code comment, we hinted that * works faster than +, the reason being that * calculates the size of the resulting object once whereas with +, that calculation is made each time + is called. Both + and * are called "overloaded" operators because they mean different things for numbers vs. for strings (and other data types).
The def keyword defines the function with its parameters within parentheses and its code indented. The first line of a function can be a documentation string ("docstring") that describes what the function does. The docstring can be a single line, or a multi-line description as in the example above. (Yes, those are "triple quotes," a feature unique to Python!) Variables defined in the function are local to that function, so the "result" in the above function is separate from a "result" variable in another function. The return statement can take an argument, in which case that is the value returned to the caller.
Here is code that calls the above repeat() function, printing what it returns:
Can you write Python code for it?

def happy(name):
    print repeat('  Yay ' + name, False)
    print repeat('Woo Hoo', True)


I am working on the file "1 - Introduction.ipynb".
The first task is:
Pressing shift-TAB twice in succession makes the documentation box bigger (showing more of the docs).
Recall our string "a".
Can you write Python code for it?


I am working on the file "1 - Introduction.ipynb".
The first task is:
The if-statement contains an obvious error, where the repeat() function is accidentally typed in as repeet(). The funny thing in Python ... this code compiles and runs fine so long as the name at runtime is not 'Tobi'. Only when a run actually tries to execute the repeet() will it notice that there is no such function and raise an error. This just means that when you first run a Python program, some of the first errors you see will be simple typos like this. This is one area where languages with a more verbose type system, like Java, have an advantage ... they can catch such errors at compile time (but of course you have to maintain all that type information ... it's a tradeoff).
Can you write Python code for it?


I am working on the file "1 - Introduction.ipynb".
The first task is:
There is another import form that looks like this: "from sys import argv, exit". That makes argv and exit() available by their short names; however, we recommend the original form with the fully-qualified names because it's a lot easier to determine where a function or attribute came from.
There are many modules and packages which are bundled with a standard installation of the Python interpreter, so you don't have do anything extra to use them. These are collectively known as the "Python Standard Library." Commonly used modules/packages include:

sys — access to exit(), argv, stdin, stdout, ...
re — regular expressions
os — operating system interface, file system

You can find the documentation of all the Standard Library modules and packages at http://docs.python.org/library.
help(), and dir()
Inside the Python interpreter, the help() function pulls up documentation strings for various modules, functions, and methods. These doc strings are similar to Java's javadoc. The dir() function tells you what the attributes of an object are. Below are some ways to call help() and dir() from the interpreter:
Can you write Python code for it?


I am working on the file "1 Introduction to Python/Exercise.ipynb".
The first task is:
3. Calculate percentage
Exercise: Caculate the percentage of people who are employed.
Present the results in this format: 11.4 %
Can you write Python code for it?

population = 10000000
employed = 4568600

# Round to 1 decimal.
round(3.3333333333, 1)


I am working on the file "1 Introduction to Python/Exercise.ipynb".
The first task is:
4. Manipulate text
Exercise: Replace Johns last name with your own.
Can you write Python code for it?

Name = "John Doe"
text = "The quick brown fox jumps over the lazy dog"

# Show the first, and fifth character.
print(text[0]) # Note, it starts with 0!
print(text[4])

# Show the first 3 characters.
print(text[0:3])

# Replace text.
print(text.replace("fox", "journalist"))

# Make text lower case, UPPER CASE or Title Case.
print(text.lower())
print(text.upper())
print(text.title())

# Character length.
len(text)

# Find the word "quick". Returns position of first character.
print(text.find("quick"))

# What happens when we don't find the word? It returns -1.
print(text.find("journalism"))


I am working on the file "1 Introduction to Python/Exercise.ipynb".
The first task is:
5. If-statements
Exercise: If John is a male and married, print his age. But if John is male and unmarried, print his height.
Use if and elif.
Can you write Python code for it?

Name = "John Doe"              # String.
Age = 40                       # Integer.
Height = 180.3                 # Float.
Married = True                 # Boolean (True/False).
Children = ["Emma", "Thomas"]  # List.
Female = False                 # Boolean (True/False).
Interests = []                 # Empty list.

if Age == 40:
    print(Name + " is 40 years old.")
else:
    print(Name + " is not 40 years old.")


I am working on the file "1 Introduction to Python/Exercise.ipynb".
The first task is:
6. For loops
Exercise: Print names of all children that does not have the name "Emma".
Can you write Python code for it?

# Print a range of numbers.
for i in range(5):
    print(i)

# Print each name in the list.
for name in Children:
    print(name)


I am working on the file "1 Introduction to Python/Introduction.ipynb".
The first task is:
For loops

range(to)
range(from, to, step)
can also use for with list
Can you write Python code for it?

# Loop 5 times.
for i in range(5):
    print(i)

# Loop from 0 to 100, by ever 25.
for i in range(0, 100, 25):
    print(i)

# We can use for loops on lists.
Children = ["Emma", "Thomas", "Nicole"]    # Make a list with 3 text strings.

for child in Children:
    print(child)

# We can use for loops on numbers.
YearsOld = [14, 5, 4]    # Make a list with 3 numbers.

for age in YearsOld:
    print(age)


I am working on the file "1 Introduction to Python/Introduction.ipynb".
The first task is:
Functions
Group things together with functions.
Can you write Python code for it?

# A function that we name "calc" that multiples two numbers together.
# It takes to variables as input (x and y).
# The function then returns the results of the multiplication.
def calc(x, y):
    return(x * y)

# Now we just use the name of the function.
calc(10, 5)


I am working on the file "1 Introduction to Python/Introduction.ipynb".
The first task is:
Python as a calculator
Can you write Python code for it?


I am working on the file "1 Mathematics and Python/Homework/1 linear algebra and texts similarity and function approximation/Task 1.ipynb".
The first task is:
2. Произвести токенизацию (разбиение текстов на слова). Удалить пустые слова.
Can you write Python code for it?

import re

dataWords = []
for line in dataSentencesLower:
    dataWords.append(re.split('[^a-z]', line))

for line in dataWords:
    print line

dataWordsCleared = [[] for i in xrange(len(dataWords))]

i = 0
for line in dataWords:
    for word in line:
        if word != '':
            dataWordsCleared[i].append(word)
    i = i + 1

for line in dataWordsCleared:
    print line


I am working on the file "1 Mathematics and Python/Homework/1 linear algebra and texts similarity and function approximation/Task 1.ipynb".
The first task is:
7. Запись ответа в файл.
Can you write Python code for it?

with open('answer.txt', 'w') as fileAnswer:
    for i in xrange(len(twoClosestValues)):
        fileAnswer.write(str(twoClosestValues[i][0]) + ' ')


I am working on the file "1 Mathematics and Python/Homework/1 linear algebra and texts similarity and function approximation/Task 2.ipynb".
The first task is:
1. Сформировать СЛАУ для многочлена первой степени, который должен совпадать с функцией в точках 1 и 15.
Can you write Python code for it?

#многочлен первой степени
arrCoord1 = np.array([1, 15])
N = 2
arrA1 = np.empty((0, N))
for i in xrange(N):
    arrA1Line = list()
    for j in xrange(N):
        arrA1Line.append(arrCoord1[i] ** j)
    arrA1 = np.append(arrA1, np.array([arrA1Line]), axis = 0)

arrB1 = np.array([func(coordinate) for coordinate in arrCoord1])

print arrCoord1
print arrA1
print arrB1

arrX1 = linalg.solve(arrA1, arrB1)
print arrX1

def func1(x): return arrX1[0] + arrX1[1] * x
arrFunc1 = np.array([func1(coordinate) for coordinate in arrCoordinates])

%matplotlib inline
import matplotlib.pylab as plt

plt.plot(arrCoordinates, arrFunction, arrCoordinates, arrFunc1)
plt.show()


I am working on the file "1 Mathematics and Python/Homework/1 linear algebra and texts similarity and function approximation/Task 2.ipynb".
The first task is:
3. Многочлен третьей степени в точка 1, 4, 10, 15.
Can you write Python code for it?

#многочлен третьей степени
arrCoord3 = np.array([1, 4, 10, 15])
N = 4
arrA3 = np.empty((0, N))
for i in xrange(N):
    arrA3Line = list()
    for j in xrange(N):
        arrA3Line.append(arrCoord3[i] ** j)
    arrA3 = np.append(arrA3, np.array([arrA3Line]), axis = 0)

arrB3 = np.array([func(coordinate) for coordinate in arrCoord3])

print arrCoord3
print arrA3
print arrB3

arrX3 = linalg.solve(arrA3, arrB3)
print arrX3

def func3(x): return arrX3[0] + arrX3[1] * x + arrX3[2] * (x ** 2) + arrX3[3] * (x ** 3)
arrFunc3 = np.array([func3(coordinate) for coordinate in arrCoordinates])

plt.plot(arrCoordinates, arrFunction, arrCoordinates, arrFunc1, arrCoordinates, arrFunc2, arrCoordinates, arrFunc3)
plt.show()

with open('answer2.txt', 'w') as fileAnswer:
    for item in arrX3:
        fileAnswer.write(str(item) + ' ')


I am working on the file "1 Mathematics and Python/Homework/2 optimization in python/Task 1.ipynb".
The first task is:
4. Теперь попробуйте найти минимум h(x) на отрезке [1, 30] с помощью дифференциальной эволюции. Значение функции h(x) в точке минимума – это ваш второй ответ в этом задании. Запишите его через пробел после предыдущего.
5. Обратите внимание на то, что полученные ответы различаются. Это ожидаемый результат, ведь BFGS использует градиент (в одномерном случае – производную) и явно не пригоден для минимизации рассмотренной нами разрывной функции. Попробуйте понять, почему минимум, найденный BFGS, именно такой (возможно в этом вам поможет выбор разных начальных приближений).
6. Выполнив это задание, вы увидели на практике, чем поиск минимума функции отличается от глобальной оптимизации, и когда может быть полезно применить вместо градиентного метода оптимизации метод, не использующий градиент. Кроме того, вы попрактиковались в использовании библиотеки SciPy для решения оптимизационных задач, и теперь знаете, насколько это просто и удобно.
Can you write Python code for it?

minFunc2Value2 = differential_evolution(func2, bounds)
print "Minimized f(x) (BFGS method): ", round(minFunc2Value2.fun, 3), "for x = ", round(minFunc2Value2.x, 3)
print "Number of iterations: ", minFunc2Value2.nit

minValuesAnswer2[1] = round(minFunc2Value2.fun, 2)
print minValuesAnswer2

with open("AnswerTask3.txt", "w") as fAnswer:
    for item in minValuesAnswer2:
        fAnswer.write(str(item) + ' ')


I am working on the file "1 Mathematics and Python/Homework/2 optimization in python/Task 1.ipynb".
The first task is:
6. Укажите в scipy.optimize.minimize в качестве метода BFGS (один из самых точных в большинстве случаев градиентных методов оптимизации), запустите из начального приближения x=2. Градиент функции при этом указывать не нужно – он будет оценен численно. Полученное значение функции в точке минимума - ваш первый ответ по заданию 1, его надо записать с точностью до 2 знака после запятой.
Can you write Python code for it?

minFunc1Value2 = minimize(func1, 2, method = 'BFGS')
print "Minimized f(x) (BFGS method): ", round(minFunc1Value2.fun, 3), "for x = ", round(minFunc1Value2.x, 3)
print "Number of iterations: ", minFunc1Value2.nit

minValuesAnswer1 = np.zeros( (2) )
minValuesAnswer1[0] = round(minFunc1Value2.fun, 2)
print minValuesAnswer1


I am working on the file "1 Mathematics and Python/Homework/2 optimization in python/Task 1.ipynb".
The first task is:
Задача 2. Глобальная оптимизация
1.Теперь попробуем применить к той же функции f(x) метод глобальной оптимизации — дифференциальную эволюцию.
2. Изучите документацию и примеры использования функции scipy.optimize.differential_evolution.
3. Обратите внимание, что границы значений аргументов функции представляют собой список кортежей (list, в который помещены объекты типа tuple). Даже если у вас функция одного аргумента, возьмите границы его значений в квадратные скобки, чтобы передавать в этом параметре список из одного кортежа, т.к. в реализации scipy.optimize.differential_evolution длина этого списка используется чтобы определить количество аргументов функции.
4. Запустите поиск минимума функции f(x) с помощью дифференциальной эволюции на промежутке [1, 30]. Полученное значение функции в точке минимума - ответ в задаче 2. Запишите его с точностью до второго знака после запятой. В этой задаче ответ - только одно число.
5. Заметьте, дифференциальная эволюция справилась с задачей поиска глобального минимума на отрезке, т.к. по своему устройству она предполагает борьбу с попаданием в локальные минимумы.
6. Сравните количество итераций, потребовавшихся BFGS для нахождения минимума при хорошем начальном приближении, с количеством итераций, потребовавшихся дифференциальной эволюции. При повторных запусках дифференциальной эволюции количество итераций будет меняться, но в этом примере, скорее всего, оно всегда будет сравнимым с количеством итераций BFGS. Однако в дифференциальной эволюции за одну итерацию требуется выполнить гораздо больше действий, чем в BFGS. Например, можно обратить внимание на количество вычислений значения функции (nfev) и увидеть, что у BFGS оно значительно меньше. Кроме того, время работы дифференциальной эволюции очень быстро растет с увеличением числа аргументов функции.
Can you write Python code for it?

from scipy.optimize import differential_evolution

bounds = [(1, 30)]
minFunc1Value4 = differential_evolution(func1, bounds)
print "Minimized f(x) (differential evolution): ", round(minFunc1Value4.fun, 3), "for x = ", round(minFunc1Value4.x, 3)
print "Number of iterations: ", minFunc1Value4.nit

with open("AnswerTask2.txt", "w") as fAnswer:
    fAnswer.write(str(round(minFunc1Value4.fun, 2)))


I am working on the file "1 Mathematics and Python/Homework/2 optimization in python/Task 1.ipynb".
The first task is:
Задача 3. Минимизация негладкой функции
1. Теперь рассмотрим функцию h(x) = int(f(x)) на том же отрезке [1, 30], т.е. теперь каждое значение f(x) приводится к типу int и функция принимает только целые значения.
2. Такая функция будет негладкой и даже разрывной, а ее график будет иметь ступенчатый вид. Убедитесь в этом, построив график h(x) с помощью matplotlib.
Can you write Python code for it?

def func2(x): return int(func1(x))

xArr2 = np.arange(1., 31., 0.01)
print xArr2
print "Shape x:", xArr2.shape
yArr2 = np.array([func2(x) for x in xArr2])
print yArr2
print "Shape y:", yArr2.shape

plt.plot(xArr2, yArr2)
plt.grid(True)
plt.axis([0, 30, -15, 5])
plt.show()


I am working on the file "1 Mathematics and Python/Homework/3 central limit theorem/Central limit theorem.ipynb".
The first task is:
Вычисление параметров нормальных распределений, которыми, согласно центральной предельной теореме, приближается распределение выборочных средних
Can you write Python code for it?

#Математическое ожидание и дисперсия

distributionSampleMean = k * xm / (k - 1)
distributionSampleVariance = np.zeros( len(sampleVolume) )
for i in xrange( len(sampleVolume) ):
    distributionSampleVariance[i] = xm**2 * k / ( (k - 1)**2 * (k - 2) ) / sampleVolume[i]

print 'Математическое ожидание: ' + str(distributionSampleMean)
for i in xrange( len(sampleVolume) ):
    print 'Дисперсия для n = ' + str(sampleVolume[i]) + ': ' + str(distributionSampleVariance[i])


I am working on the file "1 Mathematics and Python/Homework/3 central limit theorem/Central limit theorem.ipynb".
The first task is:
Генерация выборки объёма 1000
Can you write Python code for it?

# зададим случайную величину по закону двойного распределения Вейбулла

#параметры распределения
k = 4.0
xm = 3.0

#frozen distribution
distributionRv = sts.pareto(k, loc=0, scale=xm)

# сгенерируем 1000 значений
distributionSample = distributionRv.rvs(1000)

#проверка задания распределения
print 'Theoretical mean: ', distributionRv.mean()
print 'Theoretical standard deviation: ', distributionRv.std()


I am working on the file "1 Mathematics and Python/Lectures notebooks/1 introduction to ipython/introduction_to_ipython.ipynb".
The first task is:
Ниже аналоги команд для пользователей Windows:
Can you write Python code for it?

%%cmd
mkdir test_directory
cd test_directory
dir


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
1. Создание матриц
Приведем несколько способов создания матриц в NumPy.
Самый простой способ — с помощью функции numpy.array(list, dtype=None, ...).
В качестве первого аргумента ей надо передать итерируемый объект, элементами которого являются другие итерируемые объекты одинаковой длины и содержащие данные одинакового типа.
Второй аргумент является опциональным и определяет тип данных матрицы. Его можно не задавать, тогда тип данных будет определен из типа элементов первого аргумента. При задании этого параметра будет произведена попытка приведения типов.
Например, матрицу из списка списков целых чисел можно создать следующим образом:
Can you write Python code for it?


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
10. Собственные числа и собственные вектора матрицы
Напоминание теории. Для квадратных матриц определены понятия собственного вектора и собственного числа.
Пусть $A$ — квадратная матрица и $A \in \mathbb{R}^{n \times n}$. Собственным вектором матрицы $A$ называется такой ненулевой вектор $x \in \mathbb{R}^{n}$, что для некоторого $\lambda \in \mathbb{R}$ выполняется равенство $Ax = \lambda x$. При этом $\lambda$ называется собственным числом матрицы $A$. Собственные числа и собственные векторы матрицы играют важную роль в теории линейной алгебры и ее практических приложениях.
В NumPy собственные числа и собственные векторы матрицы вычисляются с помощью функции numpy.linalg.eig(a), где a — исходная матрица. В качестве результата эта функция выдает одномерный массив w собственных чисел и двумерный массив v, в котором по столбцам записаны собственные вектора, так что вектор v[:, i] соотвествует собственному числу w[i].
Can you write Python code for it?

a = np.array([[-1, -6], [2, 6]])
w, v = np.linalg.eig(a)

print "Матрица A:\n", a
print "Собственные числа:\n", w
print "Собственные векторы:\n", v


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
4. Умножение матриц и столбцов
Напоминание теории. Операция умножения определена для двух матриц, таких что число столбцов первой равно числу строк второй.
Пусть матрицы $A$ и $B$ таковы, что $A \in \mathbb{R}^{n \times k}$ и $B \in \mathbb{R}^{k \times m}$. Произведением матриц $A$ и $B$ называется матрица $C$, такая что $c_{ij} = \sum_{r=1}^{k} a_{ir}b_{rj}$, где $c_{ij}$ — элемент матрицы $C$, стоящий на пересечении строки с номером $i$ и столбца с номером $j$.
В NumPy произведение матриц вычисляется с помощью функции numpy.dot(a, b, ...) или с помощью метода array1.dot(array2), где array1 и array2 — перемножаемые матрицы.
Can you write Python code for it?

a = np.array([[1, 0], [0, 1]])
b = np.array([[4, 1], [2, 2]])
r1 = np.dot(a, b)
r2 = a.dot(b)

print "Матрица A:\n", a
print "Матрица B:\n", b
print "Результат умножения функцией:\n", r1
print "Результат умножения методом:\n", r2


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
9. Обращение матриц
Напоминание теории.  Для квадратных невырожденных матриц определено понятие обратной матрицы.
Пусть $A$ — квадратная невырожденная матрица. Матрица $A^{-1}$ называется обратной матрицей к $A$, если
$$AA^{-1} = A^{-1}A = I,
$$
где $I$ — единичная матрица.
В NumPy обратные матрицы вычисляются с помощью функции numpy.linalg.inv(a), где a — исходная матрица.
Can you write Python code for it?

a = np.array([[1, 2, 1], [1, 1, 4], [2, 3, 6]], dtype=np.float32)
b = np.linalg.inv(a)

print "Матрица A:\n", a
print "Обратная матрица к A:\n", b
print "Произведение A на обратную должна быть единичной:\n", a.dot(b)


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Более подробно о матричном умножении в NumPy
см. документацию.
5. Транспонирование матриц
Напоминание теории. Транспонированной матрицей $A^{T}$ называется матрица, полученная из исходной матрицы $A$ заменой строк на столбцы. Формально: элементы матрицы $A^{T}$ определяются как $a^{T}{ij} = a{ji}$, где $a^{T}_{ij}$ — элемент матрицы $A^{T}$, стоящий на пересечении строки с номером $i$ и столбца с номером $j$.
В NumPy транспонированная матрица вычисляется с помощью функции numpy.transpose() или с помощью метода array.T, где array — нужный двумерный массив.
Can you write Python code for it?

a = np.array([[1, 2], [3, 4]])
b = np.transpose(a)
c = a.T

print "Матрица:\n", a
print "Транспонирование функцией:\n", b
print "Транспонирование методом:\n",  c


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Более подробно о различных способах индексирования в массивах
см. документацию.
3. Векторы, вектор-строки и вектор-столбцы
Следующие два способа задания массива кажутся одинаковыми:
Can you write Python code for it?


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Более подробно о том, как создавать массивы в NumPy,
см. документацию.
2. Индексирование
Для получения элементов матрицы можно использовать несколько способов. Рассмотрим самые простые из них.
Для удобства напомним, как выглядит матрица d:
Can you write Python code for it?


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Еще один способ получения элементов — с помощью выражения array[list1, list2], где list1, list2 — некоторые списки целых чисел. При такой адресации одновременно просматриваются оба списка и возвращаются элементы матрицы с соответствующими координатами. Следующий пример более понятно объясняет механизм работы такого индексирования:
Can you write Python code for it?

print "Элементы матрицы d с координатами (1, 2) и (0, 3):\n", d[[1, 0], [2, 3]]


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Из матрицы можно получать целые строки или столбцы с помощью выражений array[i, :] или array[:, j] соответственно:
Can you write Python code for it?

print "Вторая строка матрицы d:\n", d[1, :]
print "Четвертый столбец матрицы d:\n", d[:, 3]


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Матрицы в NumPy можно умножать и на векторы:
Can you write Python code for it?

c = np.array([1, 2])
r3 = b.dot(c)

print "Матрица:\n", b
print "Вектор:\n", c
print "Результат умножения:\n", r3


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Обратите внимание: вектор (одномерный массив) и вектор-столбец или вектор-строка (двумерные массивы) являются различными объектами в NumPy, хотя математически задают один и тот же объект. В случае одномерного массива кортеж shape состоит из одного числа и имеет вид (n,), где n — длина вектора. В случае двумерных векторов в shape присутствует еще одна размерность, равная единице.
В большинстве случаев неважно, какое представление использовать, потому что часто срабатывает приведение типов. Но некоторые операции не работают для одномерных массивов. Например, транспонирование (о нем пойдет речь ниже):
Can you write Python code for it?

a = a.T
b = b.T

print "Вектор не изменился:\n", a
print "Его размерность также не изменилась:\n", a.shape
print "Транспонированный двумерный массив:\n", b
print "Его размерность изменилась:\n", b.shape


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Обратите внимание: размерность массива задается не двумя аргументами функции, а одним — кортежем!
Вот так —  np.ones(7, 5) — создать массив не получится, так как функции в качестве параметра shape передается 7, а не кортеж (7, 5).
И, наконец, третий способ — с помощью функции numpy.arange([start, ]stop, [step, ], ...), которая создает одномерный массив последовательных чисел из промежутка [start, stop) с заданным шагом step, и метода array.reshape(shape).
Параметр shape, как и в предыдущем примере, задает размерность матрицы (кортеж чисел). Логика работы метода ясна из следующего примера:
Can you write Python code for it?

v = np.arange(0, 24, 2)
print "Вектор-столбец:\n", v

d = v.reshape((3, 4))
print "Матрица:\n", d


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Обратите внимание: у вещественной матрицы собственные значения или собственные векторы могут быть комплексными.
11. Комплексные числа в питоне
Внимание: данный материал является дополнительным — его изучение не является необходимым для выполнения тестов.
Напоминание теории. Комплексными числами называются числа вида $x + iy$, где $x$ и $y$ — вещественные числа, а $i$ — мнимая единица (величина, для которой выполняется равенство $i^{2} = -1$). Множество всех комплексных чисел обозначается буквой $\mathbb{C}$ (подробнее про комплексные числа см. википедию).
В питоне комплескные числа можно задать следующим образом (j обозначает мнимую единицу):
Can you write Python code for it?

a = 3 + 2j
b = 1j

print "Комплексное число a:\n", a
print "Комплексное число b:\n", b


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Однако, на самом деле, это задание одномерного массива (то есть вектора) и двумерного массива:
Can you write Python code for it?

print "Вектор:\n", a
print "Его размерность:\n", a.shape
print "Двумерный массив:\n", b
print "Его размерность:\n", b.shape


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
С комплексными числами в питоне можно производить базовые арифметические операции так же, как и с вещественными числами:
Can you write Python code for it?

c = a * a
d = a / (4 - 5j)

print "Комплексное число c:\n", c
print "Комплексное число d:\n", d


I am working on the file "1 Mathematics and Python/Lectures notebooks/10 numpy arrays and operations with them/matrix_operations-.ipynb".
The first task is:
Элемент на пересечении строки i и столбца j можно получить с помощью выражения array[i, j].
Обратите внимание: строки и столбцы нумеруются с нуля!
Can you write Python code for it?

print "Второй элемент третьей строки матрицы:", d[2, 1]


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
А так можно вычислить значение функции плотности вероятности нормального распределения в заданной точке:
Can you write Python code for it?


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
А так — вычислять значения функций распределения и плотностей:
Can you write Python code for it?

x = np.linspace(0,5,100)
cdf = uniform_rv.cdf(x)
plt.plot(x, cdf)

plt.ylabel('$F(x)$')
plt.xlabel('$x$')

x = np.linspace(0,5,1000)
pdf = uniform_rv.pdf(x)
plt.plot(x, pdf)

plt.ylabel('$f(x)$')
plt.xlabel('$x$')


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
Другие распределения
Существует большое количество других стандартных семейств распределений, многие из которых также можно генерировать в Питоне.
Например, распределение хи-квадрат $\chi^2_k$, имеющее наутральный параметр $k$, который называется числом степеней свободы:
Can you write Python code for it?

x = np.linspace(0,30,100)
for k in [1, 2, 3, 4, 6, 9]:
    rv = sts.chi2(k)
    cdf = rv.cdf(x)
    plt.plot(x, cdf, label="$k=%s$" % k)
plt.legend()
plt.title("CDF ($\chi^2_k$)")

x = np.linspace(0,30,100)
for k in [1, 2, 3, 4, 6, 9]:
    rv = sts.chi2(k)
    pdf = rv.pdf(x)
    plt.plot(x, pdf, label="$k=%s$" % k)
plt.legend()
plt.title("PDF ($\chi^2_k$)")


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
Нормальное распределение
Вот так можно сгенерировать выборку из нормально распределённой случайной величины с параметрами $\mu=2.0$ и $\sigma=0.5$:
Can you write Python code for it?

mu = 2.0
sigma = 0.5

# зададим нормально распределенную случайную величину
norm_rv = sts.norm(loc=mu, scale=sigma)

# сгенерируем 10 значений
norm_rv.rvs(size=10)


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
Первый аргумент функции binom — значение параметра $n$, второй — параметра $p$.
Функция распределения:
Can you write Python code for it?

x = np.linspace(0,20,21)
cdf = binomial_rv.cdf(x)
plt.step(x, cdf)

plt.ylabel('$F(x)$')
plt.xlabel('$x$')


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
Посмотрим, как ведут себя биномиально распределенные величины при разных значениях параметров:
Can you write Python code for it?

x = np.linspace(0,45,46)
for N in [20, 30]:
    for p in [0.2, 0.7]:
        rv = sts.binom(N, p)
        cdf = rv.cdf(x)
        plt.step(x, cdf, label="$N=%s, p=%s$" % (N,p))
plt.legend()
plt.title("CDF (binomial)")

plt.ylabel('$F(X)$')
plt.xlabel('$x$')

x = np.linspace(0,45,46)
symbols = iter(['o', 's', '^', '+'])
for N in [20, 30]:
    for p in [0.2, 0.8]:
        rv = sts.binom(N, p)
        pmf = rv.pmf(x)
        plt.plot(x, pmf, next(symbols), label="$N=%s, p=%s$" % (N,p))
plt.legend()
plt.title("PMF (binomial)")

plt.ylabel('$P(X=x)$')
plt.xlabel('$x$')


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
Равномерное распределение на отрезке
Вот так можно сгенерировать выборку из случайной величины, имеющей равномерное распределение на отрезке $[a,b]$:
Can you write Python code for it?

a = 1
b = 4

# обратите внимание, что в этой функции задается левая граница и масштаб, а не левая и правая границы:
uniform_rv = sts.uniform(a, b-a)

uniform_rv.rvs(10)


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
Распределение Бернулли
Генерация выборок из распределения Бернулли с заданным параметром $p$:
Can you write Python code for it?

bernoulli_rv = sts.bernoulli(0.7)

bernoulli_rv.rvs(10)


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
Распределение Пуассона
Генерация выборок из распределения Пуассона с параметром $\lambda$:
Can you write Python code for it?

poisson_rv = sts.poisson(5)
poisson_rv.rvs(10)

x = np.linspace(0,30,31)
for l in [1, 5, 10, 15]:
    rv = sts.poisson(l)
    cdf = rv.cdf(x)
    plt.step(x, cdf, label="$\lambda=%s$" % l)
plt.legend()
plt.title("CDF (poisson)")

plt.ylabel('$F(x)$')
plt.xlabel('$x$')

x = np.linspace(0,30,31)

symbols = iter(['o', 's', '^', '+'])
for l in [1, 5, 10, 15]:
    rv = sts.poisson(l)
    pmf = rv.pmf(x)
    plt.plot(x, pmf, next(symbols), label="$\lambda=%s$" % l)
plt.legend()
plt.title("PMF (poisson)")

plt.ylabel('$P(X=x)$')
plt.xlabel('$x$')


I am working on the file "1 Mathematics and Python/Lectures notebooks/11 working with random numbers/stochastic_variables.ipynb".
The first task is:
Функция вероятности pmf для дискретных случайных величин заменяет функцию плотности pdf:
Can you write Python code for it?

x = np.linspace(0,20,21)
pmf = binomial_rv.pmf(x)
plt.plot(x, pmf, 'o')

plt.ylabel('$P(X=x)$')
plt.xlabel('$x$')


I am working on the file "1 Mathematics and Python/Lectures notebooks/12 sample distribution evaluation/sample_distribution_evaluation.ipynb".
The first task is:
Попробуем задавать число карманов гистограммы вручную:
Can you write Python code for it?

plt.hist(sample, bins=3, normed=True)
plt.ylabel('fraction of samples')
plt.xlabel('$x$')

plt.hist(sample, bins=40, normed=True)
plt.ylabel('fraction of samples')
plt.xlabel('$x$')


I am working on the file "1 Mathematics and Python/Lectures notebooks/12 sample distribution evaluation/sample_distribution_evaluation.ipynb".
The first task is:
Представим теперь, что эта выборка была получена не искусственно, а путём подбрасывания симметричного шестигранного кубика 100 раз. Оценим вероятности выпадения каждой из сторон с помощью частот:
Can you write Python code for it?

# посчитаем число выпадений каждой из сторон:
from collections import Counter

c = Counter(sample)

print("Число выпадений каждой из сторон:")
print(c)

# теперь поделим на общее число подбрасываний и получим вероятности:
print("Вероятности выпадений каждой из сторон:")
print({k: v/100.0 for k, v in c.items()})


I am working on the file "1 Mathematics and Python/Lectures notebooks/12 sample distribution evaluation/sample_distribution_evaluation.ipynb".
The first task is:
Это и есть оценка функции вероятности дискретного распределения.
Непрерывное распределение
Сгенерируем выборку объёма 100 из стандартного нормального распределения (с $\mu=0$ и $\sigma^2=1$):
Can you write Python code for it?

norm_rv = sts.norm(0, 1)
sample = norm_rv.rvs(100)


I am working on the file "1 Mathematics and Python/Lectures notebooks/9 vector operations/vector_operations.ipynb".
The first task is:
$\ell_{2}$ норма
$\ell_{2}$ норма (также известная как евклидова норма)
для вектора $x = (x_{1}, \dots, x_{n}) \in \mathbb{R}^{n}$ вычисляется по формуле:
$$
 \left\Vert x \right\Vert_{2} = \sqrt{\sum_{i=1}^n \left( x_{i} \right)^2}.
$$
Ей в функции numpy.linalg.norm(x, ord=None, ...) соответствует параметр ord=2.
Can you write Python code for it?

a = np.array([1, 2, -3])
print 'Вектор a:', a

print 'L2 норма вектора a:\n', norm(a, ord=2)


I am working on the file "1 Mathematics and Python/Lectures notebooks/9 vector operations/vector_operations.ipynb".
The first task is:
1. Создание векторов
Самый простой способ создать вектор в NumPy — задать его явно с помощью numpy.array(list, dtype=None, ...).
Параметр list задает итерируемый объект, из которого можно создать вектор. Например, в качестве этого параметра можно задать список чисел. Параметр dtype задает тип значений вектора, например, float — для вещественных значений и int — для целочисленных. Если этот параметр не задан, то тип данных будет определен из типа элементов первого аргумента.
Can you write Python code for it?

a = np.array([1, 2, 3, 4])
print 'Вектор:\n', a

b = np.array([1, 2, 3, 4, 5], dtype=float)
print 'Вещественный вектор:\n', b

c = np.array([True, False, True], dtype=bool)
print 'Булевский вектор:\n', c


I am working on the file "1 Mathematics and Python/Lectures notebooks/9 vector operations/vector_operations.ipynb".
The first task is:
3. Нормы векторов
Вспомним некоторые нормы, которые можно ввести в пространстве $\mathbb{R}^{n}$, и рассмотрим, с помощью каких библиотек и функций их можно вычислять в NumPy.
p-норма
p-норма (норма Гёльдера) для вектора $x = (x_{1}, \dots, x_{n}) \in \mathbb{R}^{n}$ вычисляется по формуле:
$$
\left\Vert x \right\Vert_{p} = \left( \sum_{i=1}^n \left| x_{i} \right|^{p} \right)^{1 / p},~p \geq 1.
$$
В частных случаях при:
* $p = 1$ получаем $\ell_{1}$ норму
* $p = 2$ получаем $\ell_{2}$ норму
Далее нам понабится модуль numpy.linalg, реализующий некоторые приложения линейной алгебры. Для вычисления различных норм мы используем функцию numpy.linalg.norm(x, ord=None, ...), где x — исходный вектор, ord — параметр, определяющий норму (мы рассмотрим два варианта его значений — 1 и 2). Импортируем эту функцию:
Can you write Python code for it?


I am working on the file "1 Mathematics and Python/Lectures notebooks/9 vector operations/vector_operations.ipynb".
The first task is:
scipy.spatial.distance.cdist(...) требует, чтобы размерность XA и XB была как минимум двумерная. По этой причине для использования этой функции необходимо преобразовать векторы, которые мы рассматриваем в этом ноутбуке, к вектор-строкам с помощью способов, которые мы рассмотрим ниже.
Параметры XA, XB — исходные вектор-строки, а metric и p задают метрику расстояния
(более подробно о том, какие метрики можно использовать, см. документацию).
Первый способ из вектора сделать вектор-строку (вектор-столбец) — это использовать метод array.reshape(shape), где параметр shape задает размерность вектора (кортеж чисел).
Can you write Python code for it?

a = np.array([6, 3, -5])
b = np.array([-1, 0, 7])
print 'Вектор a:', a
print 'Его размерность:', a.shape
print 'Вектор b:', b
print 'Его размерность:', b.shape

a = a.reshape((1, 3))
b = b.reshape((1, 3))
print 'После применения метода reshape:\n'
print 'Вектор-строка a:', a
print 'Его размерность:', a.shape
print 'Вектор-строка b:', b
print 'Его размерность:', b.shape

print 'Манхэттенское расстояние между a и b (через cdist):', cdist(a, b, metric='cityblock')