I am working on the file "01--intro-py.ipynb".
The first task is:
Given a list L, you can extract particular list values by an integer offset, or index.
The first element starts at index 0, and subsequence elements at positive integers. But you can also use negative indices to give offsets relative to one element past the last. Lastly, you can extract arbitrary subsequences, which are themselves (sub)lists, by a variety of different slicing operations. The following examples summarize these cases:

First element: L[0]
Last element: L[len (L)-1] == L[-1]
Second-to-last element: L[-2]
Slice, [a:b]: All elements starting at a and ending at b-1
Strided slice, [a:b:s]: All elements starting at a, ending at b-1, and including every s elements. That is, the indices a, a+s, a+2*s, ..., a+k*s <= b-1 < a+(k+1)*s for some largest integer k

Q: Use slicing to create an expression that is a sublist of E above containing only the values 8. (You may exploit the exact definition of E.)

A handy function is range (a, b), which returns a list of integers from $a$ to $b-1$, inclusive.
Can you write Python code for it?


I am working on the file "01--intro-py.ipynb".
The first task is:
Some things to observe:

Each statement within the body of square is indented. The Python language requires that you do this, to indicate both the beginning and the end of the function's definition. (The end occurs at the next line that is not indented.)
Following the function definition, there is an assignment statement whose right-hand side consists of two uses of the function, or two function calls. The form suggests that a function call is in fact a certain kind of expression.
Procedurally, the function call transfers control of the program to the function definition, with the arguments at the definition replaced by the values given at the call site, which is the location of the call.
Since the function call is an expression, it must take some value when the call is complete. The return statement within the function's definition tells you what that value is.

An important note about functions is that any identifiers assigned within its body are local to the body. That is, when a function is called (executed), it cannot normally replace the value of an identifier that was assigned outside the function.

Q: Based on the preceding claim, what is the value of 'z' in this example?
Can you write Python code for it?

z = 7

def tryToOverwrite (k):
    z = k
    y = z + k
    return y

tryToOverwrite (10)
z


I am working on the file "01--intro-py.ipynb".
The first task is:
You can of course name a list.
Can you write Python code for it?


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
2. Parsing the JSON response

We can read the content of the server’s response using .text
Can you write Python code for it?

# inspect the content of the response, parsing the result as text
response_text = r.text
print(response_text[:1000])


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
3. Looping through result pages

We're making progress, but we only have 10 items. The original response said we had 93 hits! Which means we have to make 93 /10, or 10 requests to get them all. Sounds like a job for a loop!
Can you write Python code for it?

# get number of hits total (in any page we request)
hits = data['response']['meta']['hits']
print("number of hits: ", str(hits))

# get number of pages
pages = hits // 10 + 1

# make an empty list where we'll hold all of our docs for every page
all_docs = []

# now we're ready to loop through the pages
for i in range(pages):

    print("collecting page", str(i))

    # set the page parameter
    search_params['page'] = i

    # make request
    r = requests.get(base_url + response_format, params=search_params)

    # get text and convert to a dictionary
    data = json.loads(r.text)

    # get just the docs
    docs = data['response']['docs']

    # add those docs to the big list
    all_docs = all_docs + docs

    # IMPORTANT pause between calls
    time.sleep(5)

print(len(all_docs))


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
Click on that link to see it returns! Notice that all Python is doing here for us is helping us construct a complicated URL built with & and = signs. You just noticed we could just as well copy and paste this URL to a browser and then save the response, but Python's requests library is much easier and scalable when making multiple queries in succession.
Challenge 1:  Adding a date range
What if we only want to search within a particular date range? The NYT Article API allows us to specify start and end dates.
Alter the search_params code above so that the request only searches for articles in the year 2015.
You're going to need to look at the documentation to see how to do this.
Can you write Python code for it?

# set date parameters here

# # uncomment to test
# r = requests.get(base_url + response_format, params=search_params)
# print(r.url)


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
It starts with a square bracket, so it looks like a list, and from a glance it looks like the list of articles we're interested in.
Can you write Python code for it?


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
Let's just save this list to a new variable. Often when using web APIs, you'll spend the majority of your time restructuring the response data to how you want it.
Can you write Python code for it?

docs = data['response']['docs']

print(docs[0])


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
Notice we assign each variable as a string. While the requests library will convert integers, it's better to be consistent and use strings for all parameters of a GET request. We choose JSON as the response format, as it is easy to parse quickly with Python, though XML is often an viable frequently offered alternative. JSON stands for "Javascript object notation." It has a very similar structure to a python dictionary -- both are built on key/value pairs.
You often want to send some sort of data in the URL’s query string. This data tells the API what information you want. In our case, we're going to look for articles about Duke Ellington. Requests allows you to provide these arguments as a dictionary, using the params keyword argument. In addition to the search term q, we have to put in the api-key term. We know these key names from the NYT API documentation.
Can you write Python code for it?

# set search parameters
search_params = {"q": "Duke Ellington",
                 "api-key": key}


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
Now, we have a response object called response. We can get all the information we need from this object. For instance, we can see that the URL has been correctly encoded by printing the URL. Click on the link to see what happens.
Can you write Python code for it?


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
That looks intimidating! But it's really just a big dictionary. The most time-consuming part of using APIs is traversing the various key-value trees to see where the information you want resides. Let's see what keys we got in there.
Can you write Python code for it?

print(data.keys())

# this is boring
print(data['status'])

# so is this
print(data['copyright'])

# this is what we want!
print(data['response'])

print(data['response'].keys())

print(data['response']['meta'].keys())

print(data['response']['meta']['hits'])


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
We can write our sifted information to a CSV now:
Can you write Python code for it?

header = final_docs[1].keys()

with open('all-docs.csv', 'w') as output_file:
    dict_writer = csv.DictWriter(output_file, header)
    dict_writer.writeheader()
    dict_writer.writerows(final_docs)

pandas.read_csv('all-docs.csv')


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
What you see here is JSON text, encoded as unicode text. As mentioned, JSON is bascially a Python dictionary, and we can convert this string text to a Python dictionary by using the loads to load from a string.
Can you write Python code for it?

# convert JSON response to a dictionary
data = json.loads(response_text)
print(data)


I am working on the file "01-APIs/01-API_workbook.ipynb".
The first task is:
Wow! That's a lot of information about just one article! But wait...
Can you write Python code for it?


I am working on the file "01-APIs/solutions/01-API_solutions.ipynb".
The first task is:
2. Parsing the JSON response

We can read the content of the server’s response using .text
Can you write Python code for it?


I am working on the file "01-APIs/solutions/01-API_solutions.ipynb".
The first task is:
3. Looping through result pages

We're making progress, but we only have 10 items. The original response said we had 93 hits! Which means we have to make 93 /10, or 10 requests to get them all. Sounds like a job for a loop!
Can you write Python code for it?


I am working on the file "01-APIs/solutions/01-API_solutions.ipynb".
The first task is:
It starts with a square bracket, so it looks like a list, and from a glance it looks like the list of articles we're interested in.
Can you write Python code for it?


I am working on the file "01-APIs/solutions/01-API_solutions.ipynb".
The first task is:
Looks like there were 93 hits total for our query. Let's take a look:
Can you write Python code for it?


I am working on the file "01-APIs/solutions/01-API_solutions.ipynb".
The first task is:
Now we're ready to make the request. We use the .get method from the requests library to make an HTTP GET Request.
Can you write Python code for it?

# make request
response = requests.get(base_url + response_format, params=search_params)


I am working on the file "01-APIs/solutions/01-API_solutions.ipynb".
The first task is:
That looks intimidating! But it's really just a big dictionary. The most time-consuming part of using APIs is traversing the various key-value trees to see where the information you want resides. Let's see what keys we got in there.
Can you write Python code for it?


I am working on the file "01-APIs/solutions/01-API_solutions.ipynb".
The first task is:
We can write our sifted information to a CSV now:
Can you write Python code for it?


I am working on the file "01-Container.ipynb".
The first task is:
2. list
list1
Can you write Python code for it?

list1 = [16,2,53,24,5,36,67,80]
list1

# 索引 indexing
list1[5]

# 分片 slicing
list1[:5]

list1[3:]

list1[-3:]

list1[1:6]

list1[-4:-1]

list1[0:6:2]

list1[6:0:-2]

list1[::-3]

# more functions
len(list1)

max(list1)

min(list1)

# 改变元素
list1[1]=30
list1

# 增加元素
list1.append(4)
list1

list1.sort()
list1


I am working on the file "01-Container.ipynb".
The first task is:
5. transform

转 tuple：tuple 函数 ( list / array / dictionary )
转 list：list 函数 ( tuple / array / dictionary )
转 array：np.array 函数 ( tuple / list )
Can you write Python code for it?


I am working on the file "01-Container.ipynb".
The first task is:
np.eye( N )
Can you write Python code for it?


I am working on the file "01-Container.ipynb".
The first task is:
np.linspace ( a,b,num=50 )
从a开始取num个值，直到b之前
Can you write Python code for it?


I am working on the file "01-Container.ipynb".
The first task is:
np.ones( shape )
Can you write Python code for it?


I am working on the file "01-Container.ipynb".
The first task is:
np.zeros( shape )
Can you write Python code for it?


I am working on the file "01-Container.ipynb".
The first task is:
check values in dictionary
Can you write Python code for it?


I am working on the file "01-Container.ipynb".
The first task is:
copy a dictionary
Can you write Python code for it?


I am working on the file "01-Container.ipynb".
The first task is:
list2
Can you write Python code for it?

# 算数
list2 = ['alpha','beta','gamma','gamma','alpha','alpha']
list2.count('alpha')

list2 = [2,56,8,2,9,9,47,2,2]
list2.count(2)


I am working on the file "01-Container.ipynb".
The first task is:
list3
Can you write Python code for it?

# 联合list
list3 = [56,8,7,91]
list3_add = [15,32,5]
list3.extend(list3_add)
list3


I am working on the file "01-Container.ipynb".
The first task is:
list4
Can you write Python code for it?

list4 = ['apple','banana','orange','peach']
list4.index('orange')            # 找出该元素位置

list4.insert(2,'cherry')         # 在2号位插入新元素
list4

list4.pop(1)         # 删除1号位元素（并显示被删除的元素）

list4

list4.remove('orange')        # 删除该元素（不显示被删除元素）
list4


I am working on the file "01-Container.ipynb".
The first task is:
modify a dictionary
Can you write Python code for it?

user = {'name':'Bosco','age':40,'sex':'male'}
user

user['age'] = 32      # 修改一个value
user

user['city'] = 'New York'      # 添加一个项目
user

del user['sex']      # 删除一个项目
user


I am working on the file "01-Container.ipynb".
The first task is:
some functions about array
Can you write Python code for it?

a.ndim      # rank of an array

a.shape       # dimension or shape of an array (result is a tuple)

a.size       # total number of items

a.dtype       # type of item (rq. array里面所有item都是同样的类型)


I am working on the file "01-Container.ipynb".
The first task is:
string
Can you write Python code for it?

list('hey')

sorted('hey')

sentence = "Take it away"
sentence.find('it')       # i:5

sen1 = ['hello','world','welcome','so','good']
sen2 = 'miao'
sen2.join(sen1)       # join函数（str）：在sen1的每个元素之间插入sen2

sentence = "I toOK a Photo"
sentence.lower()          # 大写字母转小写

sentence.upper()          # 小写转大写

sentence = "it was my bro"
sentence.replace('bro','sis')         # 替换字符string


I am working on the file "01-Container.ipynb".
The first task is:
键不可变，可以是数字、字符串、元组。
Can you write Python code for it?


I am working on the file "01-ElasticBarLinearFEM.ipynb".
The first task is:
Create RHS vector $\boldsymbol{b}$
We create the global RHS vector $\boldsymbol{b}$ by computing the cell RHS $\boldsymbol{b}_{e}$ cell-by-cell, and adding this to the global RHS vector. We allow the distributed load $f$ to vary with position, which is why we cannot compute it just once. For simplicity we will integrate the local RHS using the midpoint rule. This is exact if $f$ is constant, and is otherwise approximate.
We first create an empty global RHS vector:
Can you write Python code for it?


I am working on the file "01-ElasticBarLinearFEM.ipynb".
The first task is:
Create a mesh
We will create a mesh of length $L$ with a prescribed number of cells (elements) $n_{\text{cells}}$. For linear elements, the number of nodes $n_{\text{nodes}}$ is equal to $n_{\text{cells}} + 1$.
Can you write Python code for it?

L = 10.0
n_cells = 30
n_nodes = n_cells + 1


I am working on the file "01-ElasticBarLinearFEM.ipynb".
The first task is:
Distributed load
We now define the distributed load $f$. We will use a function that takes the coordinate $x$ as an argument so we possibly define loading terms that vary with position.
Can you write Python code for it?


I am working on the file "01-ElasticBarLinearFEM.ipynb".
The first task is:
Matplotlib can be used to visualise the mesh:
Can you write Python code for it?

plt.xlabel('$x$')
plt.title('finite element mesh')
plt.plot(mesh, [0]*len(mesh), 'o-');


I am working on the file "01-ElasticBarLinearFEM.ipynb".
The first task is:
To create a mesh from 0 to $L$ with equal size cells (elements) we will use the NumPy function linspace to generate an array of equally spaced points on the interval $[0, L]$.
Can you write Python code for it?


I am working on the file "01-ElasticBarLinearFEM.ipynb".
The first task is:
We now loop over each cell and compute $\int_{x_{i}}^{x_{i+1}} N_{1} f dx$ and $\int_{x_{i}}^{x_{i+1}} N_{2} f dx$ for each cell, and add the contribution to the global RHS vector:
Can you write Python code for it?

for element in range(n_cells):

    # Get cell length and midpoint
    l     = mesh[element + 1] - mesh[element]
    x_mid = (mesh[element + 1] + mesh[element])/2.0

    # Evaluate loading term
    f = distributed_load(x_mid)

    # Compute and add RHS contributions
    b[element:element + 2] += 0.5*l*f


I am working on the file "01-ElasticBarLinearFEM.ipynb".
The first task is:
We will also use ipywidgets (interactive widgets), so you will to make sure that are installed (ipywidgets is installed on the Azure notebook service).
A first solver
Elastic parameters
For our elastic rod, we define the Young's modulus $E$ and the cross-sectional area $A$. Both are assumed constant.
Can you write Python code for it?


I am working on the file "01-Fundamentals.ipynb".
The first task is:
Condition number
The condition number of a matrix is important when working with matrices numerically because is tells us something about how stable algorithms will be with respect to round-off errors, and how fast some iterative techniques will converge. Recall that the condition number $\kappa$ of a matrix $\boldsymbol{A}$ is defined as:
$$
\kappa(\boldsymbol{A}) = \| \boldsymbol{A} \| \|\boldsymbol{A}^{-1}\|
$$
If we use the 2-norm, it was shown that:
$$
\kappa_{2}(\boldsymbol{A}) = \frac{\sqrt{\lambda_{\max}(\boldsymbol{A}^{T}\boldsymbol{A})}}{\sqrt{\lambda_{\min}(\boldsymbol{A}^{T}\boldsymbol{A})}}
$$
Effect of poor conditioning on errors
It was shown in lectures that when solving $\boldsymbol{A} \boldsymbol{x} = \boldsymbol{b}$, if the condition number of $\boldsymbol{A}$ is large then small errors in $\boldsymbol{b}$ can manifest themselves as large errors in the solution, $\boldsymbol{x}$. We explore this now for the notoriously ill-conditioned Hilbert matrix. Entries of the Hilbert matrix $\boldsymbol{H}$ are given by
$$
H_{ij} = \frac{1}{i + j - 1}.
$$
We can use a SciPy function to create a $n \times n$ Hilbert matrix:
Can you write Python code for it?

import scipy.linalg as la

H = la.hilbert(10)
print(H[:5,:5])
print("Condition number is: {}".format(np.linalg.cond(H, 2)))


I am working on the file "01-Fundamentals.ipynb".
The first task is:
Condition number versus determinant
It was discussed in lectures that the condition number of a matrix and its determinant are not necessarily related. Some small examples were presented. Here we consider some larger problems.
We consider an $n \times n$ upper triangular matrix filled with two, and one on the diagonal:
$$
\boldsymbol{A} = \begin{bmatrix}
1 & 2 & \ldots & 2
\
  & 1 & \ddots & \vdots
\
  &   & \ddots & 2
\
   &  &        & 1
\end{bmatrix}
$$
This matrix has a determinant of one, and a condition number that grows with $n$. We can explore this with NumPy for increasing $n$.
Can you write Python code for it?

def test_matrix(n):
    A = np.zeros((n, n))
    A[np.triu_indices(n)] = 2.0
    np.fill_diagonal(A, 1.0)
    return A

for n in (2, 10, 100, 500):
    A = test_matrix(n)
    print("- Matrix size: {} x {}".format(n, n))
    print("   * l_2 condition number is: {}".format(np.linalg.cond(A, 2)))
    print("   * determinant is: {}".format(np.linalg.det(A)))


I am working on the file "01-Fundamentals.ipynb".
The first task is:
Even for this small Hilbert matrix, the condition number is large.
We now experiment with solving $\boldsymbol{A} (\boldsymbol{x}+ \delta \boldsymbol{x}) = \boldsymbol{b} + \delta \boldsymbol{b}$, and compare the error $\|\delta{\boldsymbol{x}}\| / \|\boldsymbol{x}\|$ to $\|\delta{\boldsymbol{b}}\| / \|\boldsymbol{b}\|$. We will presume that the NumPy linear solvers can cope with the exact system $\boldsymbol{A}\boldsymbol{x} =\boldsymbol{b}$ (in practice this will be an issue).
We first construct $\boldsymbol{b}$, $\delta\boldsymbol{b}$ and $\boldsymbol{b} + \delta\boldsymbol{b}$:
Can you write Python code for it?

b = np.ones(H.shape[0])
b_delta = 1.0e-6*np.random.rand(H.shape[0])

# Perturbed RHS
b1  = b + b_delta


I am working on the file "01-Fundamentals.ipynb".
The first task is:
Matrix norms
Norms of matrices can also be computed. The more interesting (and abstract) norms are operator norms. These are also known as induced norms.
Operator norms
For an $n \times n$ matrix $\boldsymbol{A}$, the norm of the matrix is a measure of the 'maximum change' in relative length it can induce when applied to a vector. If we consider:
$$
\| \boldsymbol{A}  \boldsymbol{x} \| \le C \| \boldsymbol{x}\| \quad \forall \boldsymbol{x} \in \mathbb{C}^{d},
$$
then the smallest possible $C$ is the norm of $\boldsymbol{A}$. The norm of $\boldsymbol{A}$ is denoted by $\|\boldsymbol{A}\|$:
$$
\| \boldsymbol{A}  \boldsymbol{x} \| \le \| \boldsymbol{A}\| \| \boldsymbol{x}\| \quad \forall \boldsymbol{x} \in \mathbb{C}^{d},
$$
This can be rearranged to provide the usual definition of a matrix norm:
$$
\| \boldsymbol{A} \| = \max_{\boldsymbol{x} \in \mathbb{C}^{n} \backslash \boldsymbol{0}}
    \frac{\| \boldsymbol{A} \boldsymbol{x}\|}{\|\boldsymbol{x}\| }
$$
To compute actual norms of a matrix, we need to choose how we measure the length of a vector, i.e. which norm to use. If we choose the $l_{2}$-norm, then:
$$
\| \boldsymbol{A} \|{2} = \max{\boldsymbol{x} \in \mathbb{C}^{n} \backslash \boldsymbol{0}}
    \frac{\| \boldsymbol{A} \boldsymbol{x}\|{2}}{\|\boldsymbol{x}\|{2} }
$$
As discussed in the lectures, some norms are relatively inexpensive to compute for large matrices, and others are expensive. We can again use NumPy to compute some matrix norms. We first create a matrix filled with random numbers:
Can you write Python code for it?

A = np.random.rand(5, 5) + 1j*np.random.rand(5, 5)
print(A)


I am working on the file "01-Fundamentals.ipynb".
The first task is:
Vector-like norms
It sometimes convenient to work with matrix norms that are similar to vector norms. A commonly used matrix norm is the Frobenius norm. It is analogous to the $l_{2}$ norm of a vector, and is defined by:
$$
\|\boldsymbol{A} \|{F} = \left( \sum{i}\sum_{i} a_{ij}^{2} \right)^{1/2}.
$$
To compute the Frobenius norm:
Can you write Python code for it?

A_frobenius = np.linalg.norm(A, 'fro')
print("The Frobenius norm of A is: {}".format(A_frobenius))


I am working on the file "01-Fundamentals.ipynb".
The first task is:
We can now compute a number of $l_{p}$ norms of $\boldsymbol{x}$:
Can you write Python code for it?

for p in range(1, 5):
    x_norm = np.linalg.norm(x, p)
    print("The l_{} norm of x is: {}".format(p, x_norm))


I am working on the file "01-Fundamentals.ipynb".
The first task is:
We now compare $\|\delta{\boldsymbol{x}}\| / \|\boldsymbol{x}\|$ and $\|\delta{\boldsymbol{b}}\| / \|\boldsymbol{b}\|$ using the $l_{2}$-norm:
Can you write Python code for it?

error_x = np.linalg.norm(x - x1, 2)/np.linalg.norm(x, 2)
error_b = np.linalg.norm(b_delta, 2)/np.linalg.norm(b, 2)
print("Relative error in x and b: {}, {}".format(error_x, error_b))


I am working on the file "01-Fundamentals.ipynb".
The first task is:
We now solve for $\boldsymbol{A} \boldsymbol{x}=  \boldsymbol{b}$ and $\boldsymbol{A} (\boldsymbol{x}+ \delta \boldsymbol{x}) = \boldsymbol{b} + \delta \boldsymbol{b}$:
Can you write Python code for it?

x  = np.linalg.solve(H, b)
x1 = np.linalg.solve(H, b1)


I am working on the file "01-Fundamentals.ipynb".
The first task is:
and then compute some norms:
Can you write Python code for it?

print("The 1-norm of A is: {}".format(np.linalg.norm(A, 1)))
print("The 2-norm of A is: {}".format(np.linalg.norm(A, 2)))
print("The max-norm of A is: {}".format(np.linalg.norm(A, np.inf)))


I am working on the file "01-GettingStarted.ipynb".
The first task is:
<img align="center" src="https://img.memesuper.com/a203efb762aefa0a25da2c367fc9f599_taxi-driver-going-to-uber-wtf-meme-png_420-215.jpeg" width="300" alt text="wtf meme" />
Keep calm! Soon you'll be flying too!
Can you write Python code for it?


I am working on the file "01-GettingStarted.ipynb".
The first task is:
Blowing your mind!
Can you write Python code for it?

42  # Just a number

type(42)  # What's the type of 42?

print(type(42))  # Let's print it!


I am working on the file "01-GettingStarted.ipynb".
The first task is:
Multiple Assignments
Can you write Python code for it?

# Varibles can be assigned with the same value...

x = y = z = 5

y += 10

print(x)
print(y)
print(z)

# ...or with multiple values of any type

x, y, z = 42, 3.14, 1+5j

b, s = True, "Hello!"

print(b, '\t\t', 'bool')
print(x, '\t\t', 'int')
print(y, '\t\t', 'float')
print(z, '\t\t', 'complex')
print(s, '\t\t', 'str')


I am working on the file "01-GettingStarted.ipynb".
The first task is:
Naming your variables can be fun - Python 3 only :-)
Can you write Python code for it?

import math

π = math.pi
r = 5
area = π * r**2

print(area)

m = 2
M = 8

µ = (M * m) / (M + m)

print(µ)

h = 6.62607004e-34  # Planck constant: m^2 kg s^-1

ħ = h / (2 * π)

print(ħ)


I am working on the file "01-GettingStarted.ipynb".
The first task is:
Python code looks almost like English
Can you write Python code for it?

love = this

this is love

love is True

love is False

love is not True or False

love is love


I am working on the file "01-GettingStarted.ipynb".
The first task is:
The Zen of Python
Can you write Python code for it?


I am working on the file "01-GettingStarted.ipynb".
The first task is:
Wait a minute!
Is int a class?
Is 42 an object?
Whaaat?! :o
Yep! In Python everything is an object!
Can you write Python code for it?


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
An example at https://github.com/NelisW/ComputationalRadiometry/blob/master/10-ImageUtilities.ipynb shows how to use interactive widgets when segmenting an image.
https://stackoverflow.com/questions/47102564/how-to-set-interact-arguments-programmatically
Can you write Python code for it?

n_weights = 10
weight_sliders = [widgets.FloatSlider(value=0,min=-2,max=2,step=0.1,description=f's{i}',
                                      disabled=False,continuous_update=False,orientation='horizontal',
                                     readout=True,readout_format='.2f') for i in range(n_weights)]

def PlotSuper(**kwargs):
    def f(x):
        y=0
        for i,weight in enumerate(kwargs.values()):
            if i==0:
                y+=weight
            else:
                y+=weight*np.sin(x*i)
        return y
    vf = np.vectorize(f)
    xx= np.arange(0,6,0.1)
    plt.plot(xx,vf(xx))
    plt.gca().set_ylim(-5,5)

kwargs = {f's{i}':slider for i,slider in enumerate(weight_sliders)}
interact(PlotSuper,**kwargs)


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
By just importing seaborn, the Matplotlib graphs is given a different style.  If seaborn is not installed do conda install seaborn.
Can you write Python code for it?

import seaborn as sns

t = np.arange(0.0, 2.0, 0.01)
s = np.sin(2*np.pi*t)
pl.plot(t, s)
pl.xlabel('time (s)')
pl.ylabel('voltage (mV)')
pl.title('About as simple as it gets, folks')
pl.grid(True)
# savefig("test.png")
# show()


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Class Descriptors
https://twitter.com/jakevdp/status/1121873857973870592/photo/1

The instance method (aka plain function) remains unbound when retrieved from the class so C.method.__get__ just returns itself. By contrast, the class method gets bound to the class in that case, so it returns a new bound method object.  In the first case both calls point to the same thing. In the second case since it is a classmethod, every call call creates a new instance of the method. I think this kind of explains the whole idea behind classmethods.

In the first cell, C.method evaluates to C.__dict__["method"] ; in second cell it evaluates to C.__dict__["method"].__get__(C, None) which turns new objects each time

In the first case, C.method points to the method() function.
In the second case, because the method() function is decorated with a @classmethod decorator, C.method returns an instance of bound method object.  Every time you write C.method a new object is created.

To add to your confusion, id(Example.clsmethod) == id(Example.clsmethod), because method objects use a free list. After the first id() call, the object gets deallocated and the second call can reuse the same object.
Can you write Python code for it?

class C:
    def method(self):
        pass

C.method is C.method

class C:
    @classmethod
    def method(cls):
        pass

print(C.method is C.method)

print(id(C.method)==id(C.method))

a = C.method
b = C.method
print(id(a)==id(b))

C.__dict__

print(type(C.method))
print(type(C.__dict__['method']))


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Class and Instance Attributes
https://twitter.com/jakevdp/status/1120898594519650304?s=09
Tricky Python bug I just hit due to an incorrect mental model of class \& instance attributes
Solution is to use self.__class__._num_instances += 1.
Can you write Python code for it?

class Foo:
    _num_instances = 0
    def __init__(self):
        self._num_instances += 1
#         self.__class__._num_instances += 1

f1 = Foo()
f2 = Foo()
print(Foo._num_instances)


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Clearing the IPython memory
If for some reason you need to clear out all memory in your notebook this can be done by selecting the Kernel Restart menu option.  This will clear the kernel memory, including the memory of all open notebooks. So use this avenue with care.  After restarting the kernel, all notebooks must be re-run from start to build the information again.
You can use the Cell | All Output | Clear menu option to remove all output from a notebook. It will be much smaller, but also empty of any output or embedded media. To see the full notebook with all calculation results, you would have to run all cells again.
Markdown (MD) language syntax
Markdown is a simplified syntax for simple text layout.  It allows you to embed simple formatting commands in a regular text file.  In an ASCII text editor you will see the markup but not formatted.  You can see the formatted version in a local or online markdown editor.

https://en.wikipedia.org/wiki/Markdown
https://daringfireball.net/projects/markdown/
https://help.github.com/articles/markdown-basics/

You can write markdown in any text editor or in a dedicated markdown editor:

https://notepad-plus-plus.org/
https://atom.io/
http://www.sitepoint.com/best-markdown-editors-windows/
http://www.sublimetext.com/  (paid software)
http://markdownpad.com/  (paid software)

IPython uses the markdown syntax for text in its non-code cells.  When IPython creates a new cell it is a code cell by default, you must remember to change it to a markdown cell on the menu.
You can also edit markdown online here:

https://github.com/benweet/stackedit
http://dillinger.io/
http://markdownlivepreview.com/

One confusing matter is the fact there are different variants of markdown. For more details on the syntax see
https://stackoverflow.com/editing-help
http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/notebooks/Part%204%20-%20Markdown%20Cells.ipynb
http://johnmacfarlane.net/pandoc/demo/example9/pandocs-markdown.html...
Can you write Python code for it?

%matplotlib inline

import pylab as pl
import numpy as np

t = np.arange(0.0, 2.0, 0.01)
s = np.sin(2*np.pi*t)
pl.plot(t, s)
pl.xlabel('time (s)')
pl.ylabel('voltage (mV)')
pl.title('About as simple as it gets, folks')
pl.grid(True)
# savefig("test.png")
# show()


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Command-line conversion
On the command line IPython has a convert subcommand to convert from the IPython format to other  formats. In the command window, where the file is located type:
ipython nbconvert filename.ipynb
This will create a fully self-contained HTML file that you can print or mail as a single file.  The only problem is that HTML does not print very well, especially with large figures.
The notebook can be converted to LaTeX and then compiled to a PDF format with the following command:
ipython nbconvert --to latex --post filename.ipynb
The LaTeX document style can be specified as follows:
ipython nbconvert --to latex filename.ipynb --template=article
once the tex document is ready run pdflatex:
pdflatex filename.tex
http://ipython.org/ipython-doc/rel-1.0.0/interactive/nbconvert.html
http://blog.fperez.org/2012/09/blogging-with-ipython-notebook.html
http://nbviewer.ipython.org/url/www.damian.oquanta.info/posts/blogging-with-nikola-and-ipython.ipynb   - blogging with Nikola
http://www.slideviper.oquanta.info/tutorial/slideshow_tutorial_slides.html?transition=none#/   - slides
http://www.damian.oquanta.info/   - a couple of links
http://nbviewer.ipython.org/github/Carreau/posts/blob/master/06-NBconvert-Doc-Draft.ipynb - using the nbconvert API
Tips to use IPython for publication ready work:  http://blog.juliusschulz.de/blog/ultimate-ipython-notebook
LaTeX docs with template control
The standard LaTeX converter does not provide good control over the style of the resultant document.  If you require better control of the document style look at my ipnb2tex script.  The script support floating figures and tables and citations.
See this PDF for an example of the output from this converter.  Using this converter you can fully control the LaTeX template to achieve the document format you require.
Making publication ready Python Notebooks provides very useful information on preparing notebooks such that it can be used for final...
Can you write Python code for it?

import os
# test to see if this is Linux or Windows
if os.path == '/':
    !ls *.ipynb
else:
    !dir *.ipynb


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Displaying tables in HTML
http://calebmadrigal.com/display-list-as-table-in-ipython-notebook/
Can you write Python code for it?

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["
"]
        for row in self:
            html.append("
")

            for col in row:
                html.append("{0}	".format(col))

            html.append("
")
        html.append("
")
        return ''.join(html)

import random
table = ListTable()
table.append(['x', 'y', 'x-y', '(x-y)**2'])
for i in range(7):
    x = random.uniform(0, 10)
    y = random.uniform(0, 10)
    table.append([x, y, x-y, (x-y)**2])

table


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Embed a video from YouTube.
Can you write Python code for it?

from IPython.display import YouTubeVideo
# a talk about IPython at Sage Days at U. Washington, Seattle.
# Video credit: William Stein.
if False:
    YouTubeVideo('1j_HxD4iLn8')


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Fine-tuning IPython typographic output appearance
Changing the fonts, colours and layout to suit your own style.
http://slendrmeans.wordpress.com/2012/12/05/better-typography-for-ipython-notebooks/
http://zulko.wordpress.com/2013/04/14/customize-your-ipython-notebook-with-css/
Adding IPython display output to existing objects
http://nbviewer.ipython.org/url/github.com/ipython/ipython/raw/master/examples/notebooks/Custom%20Display%20Logic.ipynb
For example, define a function that pretty-prints a polynomial as a LaTeX string:
Can you write Python code for it?

def poly2latex(p):
    terms = ['%.2g' % p.coef[0]]
    if len(p) > 1:
        term = 'x'
        c = p.coef[1]
        if c!=1:
            term = ('%.2g ' % c) + term
        terms.append(term)
    if len(p) > 2:
        for i in range(2, len(p)):
            term = 'x^%d' % i
            c = p.coef[i]
            if c!=1:
                term = ('%.2g ' % c) + term
            terms.append(term)
    px = '$P(x)=%s$' % '+'.join(terms)
    dom = r', domain: $[%.2g,\ %.2g]$' % tuple(p.domain)
    return px+dom

import numpy as np

p = np.polynomial.Polynomial([1,2,3], [-10, 10])
from IPython.display import Latex
Latex(poly2latex(p))


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
HTML formatting in dynamic output display
Use HTML to format the output of your code
http://python.6.x6.nabble.com/Printing-HTML-within-IPython-Notebook-IPython-specific-prettyprint-td5016624.html
Can you write Python code for it?

from IPython.display import display, HTML
for x in range(3):
    display(HTML("
Length " + str(x) + ""))


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
IPython works by starting a web server on your PC and then serving pages from this server.  This web server has an IP address of 127.0.0.1 (localhost), on port 8888, i.e., browser address 127.0.0.1:8888 or localhost:8888.  You can at any time open a new browser tab and enter this address and the server portal page should appear. This is sometimes convenient when you close all IPython tabs in the browser, but the server is still running.
Can you write Python code for it?


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
If the 127.0.0.1 loopback IP address does not appear in the list, add it as shown above.
Now close any ipython notebooks that you have, also the page at 127.0.0.1:8888. Close the command window and re-open a new one.  Open a new tab and enter the local address 127.0.0.1:8888. This should display the notebooks in the current directory and once opened, they should work.
Starting a new notebook
To create a new page click on the New Notebook button on the IPython notebook portal.
You can also make a copy of an existing notebook by selecting the appropriate menu option under the File menu.
Can you write Python code for it?


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Images can also be included as markdown by using the following format
!['replacement test'](images/ipythonhelp.png)

Images can also be included as markdown by using the following format

Embedded vs non-embedded images
As of IPython 0.13, images are embedded by default for compatibility with QtConsole, and the ability to still be displayed offline.
http://www.slideviper.oquanta.info/nbcreveal/sky_test.html?theme=sky#/7
Can you write Python code for it?

# by default Image data are embedded
picUrl = 'https://raw.githubusercontent.com/NelisW/pyradi/master/pyradi/doc/_images/pyradi.png'
Embed  = Image(picUrl)
display(Embed)

# if kwarg `url` is given, the embedding is assumed to be false
# SoftLinked = Image(url=picUrl)

# In each case, embed can be specified explicitly with the `embed` kwarg
# ForceEmbed = Image(url=picUrl, embed=True)


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Interactive widgets
IPython includes an architecture for interactive widgets that tie together Python code running in the kernel and JavaScript/HTML/CSS running in the browser. These widgets enable users to explore their code and data interactively. For details see
http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Interactive%20Widgets/Index.ipynb
https://github.com/ipython/ipython/tree/master/examples/Interactive%20Widgets
https://www.youtube.com/watch?v=VaV10VNZCLA
https://www.youtube.com/watch?v=vE_CJTen15M
https://www.youtube.com/watch?v=o7Tb7YhJZR0
https://www.youtube.com/watch?v=wxVx54ax47s
https://github.com/ipython/ipython/wiki/Widgets
The following examples are taken from
https://github.com/ipython/ipython-in-depth/tree/master/examples/Interactive%20Widgets
Upgrading widgets from IPython 2 to 3:
http://ipython.org/ipython-doc/3/whatsnew/version3_widget_migration.html
IPython widgets in Jupyter/IPython 4
https://keminglabs.com/print-the-docs-pdfs/1518024.pdf
https://github.com/ipython/ipywidgets/blob/master/examples/notebooks/Widget%20Basics.ipynb
The widget ecosystem changes frequently, the following code may not work....
Can you write Python code for it?

# import IPython.html.widgets as widgets
from IPython.display import display
import ipywidgets
from ipywidgets import widgets

[n for n in dir(ipywidgets) if n[0] == n[0].upper() and not n[0] == '_']

xx = widgets.FloatSlider(
    value=7.5,
    min=5.0,
    max=10.0,
    step=0.1,
    description='Test:',
)

y = ipywidgets.Checkbox(
    description='Check me',
    value=True,
)

w = ipywidgets.Dropdown(
    options = [ 'test 1', 'option 2', 'selection 3',],
    value='option 2',
    description='Number:',
)

#use ordered dic to get required sorting sequence
from collections import OrderedDict
foclens = [ 0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1.,   1.1,  1.2,  1.3,  1.4,  1.5]
m = ipywidgets.Dropdown(
    options = OrderedDict([(str(x), str(x)) for x in foclens]) ,
    value='0.4',
    description='Focal length:',
)

from IPython.display import display

display(xx)
display(y)
display(w)
display(m)

# http://stackoverflow.com/questions/28529157/dynamically-changing-dropdowns-in-ipython-notebook-widgets-and-spyre
# from IPython.html import widgets
from IPython.display import display

geo={'USA':['CHI','NYC'],'Russia':['MOW','LED']}

def print_city(city):
    print(city)

def select_city(country):
    cityW.options = geo[country]

scW = ipywidgets.Select(options=geo.keys())
init = scW.value
cityW = ipywidgets.Select(options=geo[init])
j = ipywidgets.interactive(print_city, city=cityW)
i = ipywidgets.interactive(select_city, country=scW)
display(i)
display(j)


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Interdependent widgets
The softmax function is used in neural networks.
Suppose we have a network with four neurons, and four corresponding weighted inputs, which we'll denote $z_{1}^{L}, z_{2}^{L}, z_{3}^{L}$, and $z_{4}^{L}$.
According to this function, the activation $a^L_j$ of the $j$ output neuron is
\begin{equation}
a_{j}^{L}=\frac{e^{z_{j}^{L}}}{\sum_{k} e^{z_{k}^{L}}}
\label{eq:c03-78}
\end{equation}
where in the denominator we sum over all the inputs $z^L_j$.
As you increase any one component, its output will increase
Shown below are adjustable sliders showing possible values for the weighted inputs, and a graph of the corresponding output activations. A good place to start exploration is by using the bottom slider to increase $z_{4}^{L}$.
As you increase $z_{4}^{L}$, you'll see an increase in the corresponding output activation, $a_{4}^{L}$, and a decrease in the other output activations. Similarly, if you decrease $z_{4}^{L}$ then $a_{4}^{L}$ will decrease, and all the other output activations will increase. In fact, if you look closely, you'll see that in both cases the total change in the other activations exactly compensates for the change in $a_{4}^{L}$. The reason is that the output activations are guaranteed to always sum up to 1.
In the code below there is a direct match between each slider and progress bar next to it.
The sliders and progress bars are created in a dict comprehension, using the same keys for sliders and progress bars.
These widgets have global scope and are available inside the softmax function.
The widgets are displayed manually (not automatically in interact) with the idea that these will be updated later.
ipywidgets.interact automatically displays the widgets when invoked, but we already displayed the widgets. Hence, the interactive function is called rather than interact, because interactive does not display/show the widgets.
Can you write Python code for it?

import numpy as np
from ipywidgets import HBox,VBox,Button,FloatSlider,FloatProgress,interactive

# set up the widgets with precalculated values
# these sliders and prog bars are visible and are updated below in the softmax function
sliders = {'1':[2.5,0.31], '2':[-1.,0.009], '3':[3.2,0.633], '4':[0.5,0.043]}
sld = {key:FloatSlider(min=-5.0, max=+5.0, value=f'{sliders[key][0]}', step=0.05,description=f'$z^L_{key}$') for key in sliders}
prb = {key:FloatProgress(value=f'{sliders[key][1]}',min=0,max=1.0,step=0.01,description=f'$a^L_{key}$',bar_style='info',orientation='horizontal') for key in sliders}

# build and display the widget grid in pairs of sliders and prog bars
lstD = [HBox([sld[key], prb[key]]) for key in sld]
display(VBox(lstD))

# function is invoked if any of the sliders change
# and the result is used to change the progress bar
def softmax(**lstZ):
    sum = 0
    for key in lstZ:
        sum += np.exp(lstZ[key])
    for key in lstZ:
        prb[key].value = np.exp(lstZ[key])/sum

#  `interactive` does not display/show the widgets, already done above.
w = interactive(softmax, **sld )


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Lorena Barba's tutorial, see this notebook for more detail on the Jupyter notebook.
Why use the IPython notebook?

The IPython notebook is an effective means to capture technical story lines or flow-of-thought;
being initially conceived as a lab book for science and technology investigations.
It is now also used for slides and as lecturing medium.

The IPython notebook can never replace formal documentation, at least in its present form. The notebook should also not be used as a primary software development environment. The notebook lives alongside other forms of documentation and coding. Having said that, there is a definite place for the IPython notebook in several environments such as engineering research and development, teaching, and scientific research and experimentation (which it was initially developed for).
The IPython notebook is a very effective means to capture your work as you progress through an investigation comprising research, coding, and record keeping.  It is also an excellent way to develop slides or teaching material where one wants to combine code, text and results in a story-line. It is used widely in Python conferences and lectures. The notebook is also a convenient means to do homework assignments.
http://nbviewer.ipython.org/urls/raw.github.com/ellisonbg/talk-strata2013/master/StrataIPythonSlides.ipynb - why use notebooks?
http://nbviewer.ipython.org/ - gallery of notebooks
https://github.com/jupyter/jupyter/wiki/A-gallery-of-interesting-Jupyter-Notebooks - a gallery of notebooks
Getting started in IPython
During early 2015 the IPython 2.x tool was upgraded to IPython 3.0. When installing, use the latest version you can find.
If you installed the Anaconda Python distribution (http://docs.continuum.io/anaconda/install.html), it already has IPython, no need to download it.
If you are not using Anaconda, follow the steps outlined on these sites:
http://ipython.org/ - start here, download the latest version from here....
Can you write Python code for it?


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Matplotlib in qt window
Use the ipython magic command pylab to control the graphing backend, switch between inline and qt as required.

http://stackoverflow.com/questions/14261903/how-can-i-open-the-interactive-matplotlib-window-in-ipython-notebook/14277370#14277370
Seaborn distribution plotting
The seaborn package provides special means to plot distributions
http://nbviewer.ipython.org/github/mwaskom/seaborn/blob/master/examples/plotting_distributions.ipynb
Including images into the notebook
There are (at least) two ways to include images in the notebook.
display(Image(filename='images/portalpage.png'))
HTML('')

The first form includes the image in its natural size, but the size can be adjusted by width and height function parameters.  The second form injects HTML code and, likewise, allows you to set the image size.
Can you write Python code for it?

HTML('<img src="images/ipythonhelp.png" width=400 height=200/>')

display(Image(filename='images/ipythonhelp.png', width=250, height=250))


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Once in the Sophos control panel, select the following menu option
<Configure><Anti-virus><Authorization...>.
The following window should appear:
Can you write Python code for it?


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Running notebook servers
This document describes how you can secure a notebook server and how to run it on a public interface:
http://ipython.org/ipython-doc/rel-1.1.0/interactive/public_server.html
Markdown formatting in dynamic output display
http://catherinedevlin.blogspot.com/2013/06/easy-html-output-in-ipython-notebook.html
For this to work, you first have to install the Python markdown package (assuming you have the pip python package installer):
pip install markdown
Use the following function to render a Python string in markdown syntax to display in IPython:
Can you write Python code for it?

import markdown
class MD(str):
    def _repr_html_(self):
        return markdown.markdown(self)

import math
a = 2
MD("""
Dynamic demonstration
--------------
This is a mixture of markdown **and** html:

The square root of {0} used to be somewhere near {1}""".format(a,math.sqrt(a)))


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
Tab-completion help is also available, just press TAB after the period
Can you write Python code for it?


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
The %edit magic is supposed to start up an editor from within IPython - I never got that to work under windows.
https://ipython.org/ipython-doc/dev/interactive/magics.html#magic-edit
http://ipython.org/ipython-doc/1/config/editors.html
http://stackoverflow.com/questions/15681153/external-editor-for-ipython-notebook
http://stackoverflow.com/questions/3438531/ipython-workflow-edit-run
This works ok if you have a fast editor:
http://stackoverflow.com/questions/28309430/edit-ipython-cell-in-an-external-editor
Just replace gvim with your editor's exe and make sure it is on the path.
You can manage environment variables of your notebook without restarting the jupyter server process, %env is the most convenient way.  Running %env without any arguments lists all environment variables.
Can you write Python code for it?

# The line below sets the environment variable OMP_NUM_THREADS
%env OMP_NUM_THREADS=4


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
The %who command without any arguments will list all variables that exist in the global scope. Passing a parameter like str will list only variables of that type.
Can you write Python code for it?

one = "for the money"
two = "for the show"
three = "to get ready now go cat go"
%who str


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
The following information is somewhat esoteric, you need not go into this
Simple progress bar
Note that clear_output wipes the entire cell output, including previous output
https://mikulskibartosz.name/how-to-display-a-progress-bar-in-jupyter-notebook-47bd4c2944bf
Can you write Python code for it?

def update_progress(progress, bar_length=20):

    from IPython.display import clear_output

    if isinstance(progress, int):
        progress = float(progress)
    if not isinstance(progress, float):
        progress = 0
    if progress < 0:
        progress = 0
    if progress >= 1:
        progress = 1
    block = int(round(bar_length * progress))
    clear_output(wait = True)
    text = "Progress: [{0}] {1:.1f}%".format( "#" * block + "-" * (bar_length - block), progress * 100)
    print(text)


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
The following is an example by Ketcheson, Ahmadia and Granger taken from
http://www.nature.com/news/ipython-interactive-demo-7.21492?article=1.16261
It demonstrates aliasing during sampling of a signal.  To see the effects of aliasing:

Run the next cell, then set the grid_points slider to 13.
Move the frequency slider to values above 10.
As the frequency increases, the measured signal (blue) has a lower frequency than the real one (red).
Can you write Python code for it?

# Import matplotlib (plotting) and numpy (numerical arrays).
# This enables their use in the Notebook.
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

# Import IPython's interact function which is used below to
# build the interactive widgets
from ipywidgets import interact#, interactive, fixed, interact_manual
# import ipywidgets as widgets

def plot_sine(frequency=4.0, grid_points=12, plot_original=True):
    """
    Plot discrete samples of a sine wave on the interval ``[0, 1]``.
    """
    x = np.linspace(0, 1, grid_points + 2)
    y = np.sin(2 * frequency * np.pi * x)

    xf = np.linspace(0, 1, 1000)
    yf = np.sin(2 * frequency * np.pi * xf)

    fig, ax = plt.subplots(figsize=(8, 6))
    ax.set_xlabel('x')
    ax.set_ylabel('signal')
    ax.set_title('Aliasing in discretely sampled periodic signal')

    if plot_original:
        ax.plot(xf, yf, color='red', linestyle='solid', linewidth=2)

    ax.plot(x,  y,  marker='o', linewidth=2)

# The interact function automatically builds a user interface for exploring the
# plot_sine function.
interact(plot_sine, frequency=(1.0, 22.0, 0.5), grid_points=(10, 60, 1), plot_original=True);


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
The following two cells illustrate how a slider is used in the widgets.interactive function to test the value of the slider and then do something with the value.  The example below shows how to pass 'fixed' or non-widget parameters to the function.  Any number of such widgets may be passed, but they must all be named.
For more examples see the links shown above.  An example of interactive image segmentation is shown in notebook '10-ImageUtilities' in this series.
Can you write Python code for it?

def doSomething(scale, thx):
    print('scale={} thx={} product={}'.format(scale, thx, scale * thx))
    return (scale, thx)

scale = 5.0
v = ipywidgets.interactive(doSomething, scale=ipywidgets.fixed(scale),
                        thx=ipywidgets.FloatSlider(value=128, min=0.0, max=255.0, step=1))
display(v)

form = widgets.VBox()
first = widgets.Text(description="First Name:")
last = widgets.Text(description="Last Name:")

students = widgets.VBox(visible=True, children=[
    widgets.Checkbox(description="Student1:", value=False),
    widgets.Checkbox(description="Student2:", value=False),
    ])

student = widgets.Checkbox(description="Student:", value=False)

school_info = widgets.VBox(visible=False, children=[
    widgets.Text(description="School:"),
    widgets.IntText(description="Grade:", min=0, max=12)
    ])

pet = widgets.Text(description="Pet's Name:")

form.children = [first, last, student, students, school_info, pet]

display(form)

def on_student_toggle(name, value):
    if value:
        school_info.visible = True
    else:
        school_info.visible = False

student.observe (on_student_toggle, 'value')
students.children[0].observe(on_student_toggle, 'value')
students.children[1].observe(on_student_toggle, 'value')

form = widgets.VBox()
first = widgets.Text(description="First Name:")
last = widgets.Text(description="Last Name:")

student = widgets.Checkbox(description="Student:", value=False)

school_info = widgets.VBox(visible=False, children=[
    widgets.Text(description="School:"),
    widgets.IntText(description="Grade:", min=0, max=12)
    ])

pet = widgets.Text(description="Pet's Name:")
form.children = [first, last, student, school_info, pet]
display(form)

def on_student_toggle(name, value):
    if value:
        school_info.visible = True
    else:
        school_info.visible = False
student.observe(on_student_toggle, 'value')

from IPython.display import display

float_range = widgets.FloatSlider()
string = widgets.Text(value='hi')
container =...


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
The%run magic runs a scipt along the stated path and prints the results to the cell output. Useful to run external scripts not coded in the notebook itself.  Just be sure to copy the script with the notebook.  The next cell writes a script to the current directory and then the following cell executes it.
Can you write Python code for it?

%%file helloipython.py
print('Hello IPython!')

%run helloipython.py


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
To learn more about a magic execute ? followed by the magic as in
?%pastebin

it will open a new panel where the docstring is displayed.
Can you write Python code for it?


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
UK English spell checker
 I have not yet had the time to figure out how to do this for Jupyter 4
Marco Pinto maintains a UK English hunspell-style list here.  To implement the UK dictionary in an Anaconda Jupyter installation  on Windows:

Download the two files en-GB.aff and en-GB.dic into the (new) folder C:\Anaconda\share\jupyter\nbextensions\typo\dictionaries\en_GB\.

Rename the two files to use underscore instead of dash/hyphen (look at the en_US equivalent).  Change en-GB.aff to en_GB.aff and change en-GB.dic to en_GB.dic.

Edit the file C:\Anaconda\share\jupyter\nbextensions\calico-spell-check.js to replace var lang = "en_US"; with var lang = "en_GB";.

It should be a simple matter to find the equivalent directories in Linux.
So now at least we have a UK dictionary, the remaining work is to add new words. Marco Pinto's GUI-based tool at http://marcoagpinto.cidadevirtual.pt/proofingtoolgui.html is exceptionally suitable tool for this purpose.  Doug Blank also pointed out: "Also, one can add words to the calico-spell-check extension by making a JSON object in a file named words.json and putting it next to calico-spell-check.js."
Multicursor editor support
Jupyter supports mutiple cursors, similar to Sublime Text. Simply click and drag your mouse while holding down Alt.
Optimising with IPython
https://robotwhale.wordpress.com/2014/08/17/optimization-with-ipython/
Local files
If you have local files in your Notebook directory, you can refer to these files in Markdown cells via relative URLs that are prefixed with files/ as in:
files/[subdirectory/]filename
Help
Introspection help is available  by typing the object's name followed by a question mark, then execute the cell.
It will print details about the object, including docstrings, function  call argument) and class constructor details.
Double click on the divider to close the help console.
Can you write Python code for it?

from collections import  defaultdict
# defaultdict?
display(Image(filename='images/introspection.png'))


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
You can also access the built in Python help system, by typing help(objectname), and then execute the cell.
Can you write Python code for it?


I am working on the file "01-IPythonHintsAndTips.ipynb".
The first task is:
https://github.com/jupyter-widgets/ipywidgets/blob/master/docs/source/examples/Widget%20Styling.ipynb
shows how to style the widgets using css.
Can you write Python code for it?

from ipywidgets import Button, Layout

b = Button(description='(50% width, 80px height) button',
           layout=Layout(width='50%', height='80px'))
b

c = Button(description='Another button with the same layout', layout=b.layout)

c

from ipywidgets import Button, HBox, VBox

words = ['correct', 'horse', 'battery', 'staple']
items = [Button(description=w) for w in words]
left_box = VBox([items[0], items[1]])
right_box = VBox([items[2], items[3]])
HBox([left_box, right_box])

from ipywidgets import IntSlider, Label
IntSlider(description=r'\(\int_0^t f\)')

from ipywidgets import Layout, Button, Box

items_layout = Layout( width='auto')     # override the default width of the button to 'auto' to let the button grow

box_layout = Layout(display='flex',
                    flex_flow='column',
                    align_items='stretch',
                    border='solid',
                    width='50%')

words = ['correct', 'horse', 'battery', 'staple']
items = [Button(description=word, layout=items_layout, button_style='danger') for word in words]
box = Box(children=items, layout=box_layout)
box

from ipywidgets import Layout, Button, Box, VBox

# Items flex proportionally to the weight and the left over space around the text
items_auto = [
    Button(description='weight=1; auto', layout=Layout(flex='1 1 auto', width='auto'), button_style='danger'),
    Button(description='weight=3; auto', layout=Layout(flex='3 1 auto', width='auto'), button_style='danger'),
    Button(description='weight=1; auto', layout=Layout(flex='1 1 auto', width='auto'), button_style='danger'),
 ]

# Items flex proportionally to the weight
items_0 = [
    Button(description='weight=1; 0%', layout=Layout(flex='1 1 0%', width='auto'), button_style='danger'),
    Button(description='weight=3; 0%', layout=Layout(flex='3 1 0%', width='auto'), button_style='danger'),
    Button(description='weight=1; 0%', layout=Layout(flex='1 1 0%', width='auto'), button_style='danger'),
 ]
box_layout = Layout(display='flex',
                   ...


I am working on the file "01-Index.ipynb".
The first task is:
IPython Project
Can you write Python code for it?


I am working on the file "01-InitNN/first-neural-network.ipynb".
The first task is:
Checking out the data
This dataset has the number of riders for each hour of each day from January 1 2011 to December 31 2012. The number of riders is split between casual and registered, summed up in the cnt column. You can see the first few rows of the data above.
Below is a plot showing the number of bike riders over the first 10 days in the data set. You can see the hourly rentals here. This data is pretty complicated! The weekends have lower over all ridership and there are spikes when people are biking to and from work during the week. Looking at the data above, we also have information about temperature, humidity, and windspeed, all of these likely affecting the number of riders. You'll be trying to capture all this with your model.
Can you write Python code for it?


I am working on the file "01-InitNN/first-neural-network.ipynb".
The first task is:
For checking if there is a relation between model failing and actual consumption, let us look at the correlation between mean squared error and original values. Visually, we see that this is unlikely to be true. It also has a small and safe to ignore Pearson correlation coefficient.
I don't completely understand why the model fails where it fails, but I suspect having a way to factor in seasonality from previous years would improve this.
Unit tests
Run these unit tests to check the correctness of your network implementation. These tests must all be successful to pass the project.
Can you write Python code for it?

import unittest

inputs = [0.5, -0.2, 0.1]
targets = [0.4]
test_w_i_h = np.array([[0.1, 0.4, -0.3],
                       [-0.2, 0.5, 0.2]])
test_w_h_o = np.array([[0.3, -0.1]])

class TestMethods(unittest.TestCase):

    ##########
    # Unit tests for data loading
    ##########
    def runTest(self):
        pass

    def test_data_path(self):
        # Test that file path to dataset has been unaltered
        self.assertTrue(data_path.lower() == 'bike-sharing-dataset/hour.csv')

    def test_data_loaded(self):
        # Test that data frame loaded
        self.assertTrue(isinstance(rides, pd.DataFrame))

    ##########
    # Unit tests for network functionality
    ##########

    def test_activation(self):
        network = NeuralNetwork(3, 2, 1, 0.5)
        # Test that the activation function is a sigmoid
        self.assertTrue(np.all(network.activation_function(0.5) == 1/(1+np.exp(-0.5))))

    def test_train(self):
        # Test that weights are updated correctly on training
        network = NeuralNetwork(3, 2, 1, 0.5)
        network.weights_input_to_hidden = test_w_i_h.copy()
        network.weights_hidden_to_output = test_w_h_o.copy()

        network.train(inputs, targets)
        self.assertTrue(np.allclose(network.weights_hidden_to_output,
                                    np.array([[ 0.37275328, -0.03172939]])))
        self.assertTrue(np.allclose(network.weights_input_to_hidden,
                                    np.array([[ 0.10562014,  0.39775194, -0.29887597],
                                              [-0.20185996,  0.50074398,  0.19962801]])))

    def test_run(self):
        # Test correctness of run method
        network = NeuralNetwork(3, 2, 1, 0.5)
        network.weights_input_to_hidden = test_w_i_h.copy()
        network.weights_hidden_to_output = test_w_h_o.copy()

        self.assertTrue(np.allclose(network.run(inputs), 0.09998924))

suite = unittest.TestLoader().loadTestsFromModule(TestMethods())
unittest.TextTestRunner().run(suite)


I am working on the file "01-InitNN/first-neural-network.ipynb".
The first task is:
How well does the model predict the data?
The model predicts the data quite well with 2 hidden nodes with a MSE Validation Loss ~0.395 and
Test loss of ~0.24 for 2 hidden nodes.
I infer this to mean that model is generalizable to completely unseen data in this case.
Where does it fail?
Can you write Python code for it?

def squared_error(y, Y):
    return (y - Y)**2

SE_Test = squared_error(network.run(test_features)[0], test_targets['cnt'].values)

fig, ax = plt.subplots(figsize=(8,4))

mean, std = scaled_features['cnt']
ax.plot(SE_Test, label='Squared Errors (on Test Data)')
# ax.plot((test_targets['cnt']*std + mean).values, label='Data')
ax.set_xlim(right=len(SE_Test))
ax.legend()
ax.set_xticks(np.arange(len(dates))[12::24])
_ = ax.set_xticklabels(dates[12::24], rotation=90)


I am working on the file "01-InitNN/first-neural-network.ipynb".
The first task is:
Load and prepare the data
A critical step in working with neural networks is preparing the data correctly. Variables on different scales make it difficult for the network to efficiently learn the correct weights. Below, we've written the code to load and prepare the data. You'll learn more about this soon!
Can you write Python code for it?

data_path = 'Bike-Sharing-Dataset/hour.csv'

rides = pd.read_csv(data_path)

rides.head()


I am working on the file "01-InitNN/first-neural-network.ipynb".
The first task is:
Thinking about your results
Answer these questions about your results. How well does the model predict the data? Where does it fail? Why does it fail where it does?

Note: You can edit the text in this cell by double clicking on it. When you want to render the text, press control + enter

Your answer below
How well does the model predict the data?
The model predicts the data quite well with 2 hidden nodes with a MSE Validation Loss ~0.395 and Test loss of ~0.24 for 2 hidden nodes. I infer this to mean that model is generalizable to completely unseen data in this case.
Can you write Python code for it?

print("Test Loss:", MSE(network.run(test_features)[0], test_targets['cnt'].values))


I am working on the file "01-InitNN/first-neural-network.ipynb".
The first task is:
Time to build the network
Below you'll build your network. We've built out the structure and the backwards pass. You'll implement the forward pass through the network. You'll also set the hyperparameters: the learning rate, the number of hidden units, and the number of training passes.
The network has two layers, a hidden layer and an output layer. The hidden layer will use the sigmoid function for activations. The output layer has only one node and is used for the regression, the output of the node is the same as the input of the node. That is, the activation function is $f(x)=x$. A function that takes the input signal and generates an output signal, but takes into account the threshold, is called an activation function. We work through each layer of our network calculating the outputs for each neuron. All of the outputs from one layer become inputs to the neurons on the next layer. This process is called forward propagation.
We use the weights to propagate signals forward from the input to the output layers in a neural network. We use the weights to also propagate error backwards from the output back into the network to update our weights. This is called backpropagation.

Hint: You'll need the derivative of the output activation function ($f(x) = x$) for the backpropagation implementation. If you aren't familiar with calculus, this function is equivalent to the equation $y = x$. What is the slope of that equation? That is the derivative of $f(x)$.

Below, you have these tasks:
1. Implement the sigmoid function to use as the activation function. Set self.activation_function in __init__ to your sigmoid function.
2. Implement the forward pass in the train method.
3. Implement the backpropagation algorithm in the train method, including calculating the output error.
4. Implement the forward pass in the run method.
Can you write Python code for it?

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

class NeuralNetwork(object):
    def __init__(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
        '''Set number of nodes in input, hidden and output layers.'''
        self.input_nodes = input_nodes
        self.hidden_nodes = hidden_nodes
        self.output_nodes = output_nodes

        # Initialize weights
        self.weights_input_to_hidden = np.random.normal(0.0, self.hidden_nodes**-0.5,
                                       (self.hidden_nodes, self.input_nodes))

#         print("input_hidden.shape",self.weights_input_to_hidden.shape)

        self.weights_hidden_to_output = np.random.normal(0.0, self.output_nodes**-0.5,
                                       (self.output_nodes, self.hidden_nodes))
        self.lr = learning_rate
        self.learning_rate = learning_rate

        #### Set this to your implemented sigmoid function ####
        # Activation function is the sigmoid function
        self.activation_function = sigmoid

        self.del_w_hidden_output = np.zeros(self.weights_input_to_hidden.shape)
        self.del_w_input_hidden = np.zeros(self.weights_hidden_to_output.shape)

    def train(self, inputs_list, targets_list):
        # Convert inputs list to 2d array
        inputs = np.array(inputs_list, ndmin=2).T
        targets = np.array(targets_list, ndmin=2).T

        #### Implement the forward pass here ####
        ### Forward pass ###
        # TODO: Hidden layer
#         print("weights_input_hidden.shape", self.weights_input_to_hidden.shape, "inputs.shape",inputs.shape)
        hidden_inputs = np.dot(self.weights_input_to_hidden, inputs)
#         print("hidden_inputs.shape:", hidden_inputs.shape)

        # signals into hidden layer
        hidden_outputs = self.activation_function(hidden_inputs)
#         print("hidden_outputs.shape:", hidden_outputs.shape)
        # signals from hidden layer

        # TODO: Output layer
#        ...


I am working on the file "01-InitNN/first-neural-network.ipynb".
The first task is:
We'll split the data into two sets, one for training and one for validating as the network is being trained. Since this is time series data, we'll train on historical data, then try to predict on future data (the validation set).
Can you write Python code for it?

# Hold out the last 60 days of the remaining data as a validation set
train_features, train_targets = features[:-60*24], targets[:-60*24]
val_features, val_targets = features[-60*24:], targets[-60*24:]


I am working on the file "01-Instalación.ipynb".
The first task is:
Aplicamos k-means
Can you write Python code for it?

# scikit-learn
kmeans = sklearn.cluster.KMeans(k, max_iter=300)
kmeans.fit(data)
means = kmeans.cluster_centers_

plot.scatter(data[:, 0], data[:, 1], c=labels)
plot.scatter(means[:, 0], means[:, 1], linewidths=2, color='r')
plot.show()


I am working on the file "01-Instalación.ipynb".
The first task is:
Crear un entorno
Anaconda nos permite tener distintos entornos, cada uno con distintas librerías (y versiones de librerías).
De esa manera evitamos conflictos si queremos tener aplicaciones que requieran librerías incompatibles entre sí.
Vamos a crear el entorno para Inteligencia de Negocio con la versión 3 de Python, y la librería scikit-learn se crea de la siguiente forma:

conda create -n IN

Y para activarlo se ejecuta

source activate IN

Si todo va bien,  en Linux, deberá de de verse "(IN)" en la línea de comandos.
Para desactivar se puede hacer:

source deactivate

Por qué Python
Python es un lenguaje muy usado en Machine-Learning y Ciencias de Datos en General. Tiene muchas ventajas:

Es Software Libre, así que no hay problemas de licencias.
Es un lenguaje muy fácil de aprender.
Librerías científicas muy buenas.
Fácil de integrar con otras librerías.

Librerías que usaremos
Las librerías que vamos a usar ya vienen instaladas por defecto. Son:

numpy, librería matemática, muy potente, y consigue la eficiencia de trabajar en C.
pandas, librería que permite trabajar con tablas de datos (como Excel) y leerlos y escribirlos de ficheros .csv y Excel.
scikit-learn, librería de 'Marchine Learning'.

Entorno de Desarrollo
Existen muchos entornos de desarrollo, como PyCharm, Spyder o PyDev (entorno Eclipse).
Nosotros vamos a usar Jupyter, que nos ofrece un entorno web para escribir código Python y ver el resultado de forma bastante interactiva.
Con conda se instala sólo. Para ejecutarlo primer vamos al directorio que queremos ejecutar y hacemos:

jupyter notebook

y luego desde el navegador abrimos la dirección, y podemos empezar a trabajar.
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo("qb7FT68tcA8")


I am working on the file "01-Instalación.ipynb".
The first task is:
Haciendo uso de Machine Learning
Python posee la estupenda librería scikit-learn para trabajar con Machine Learning, que implementa muchos interesantes métodos. Vamos a mostrarlo aplicando clustering usando el algoritmo K-means.
Primero cargamos las librerías
Can you write Python code for it?

import sklearn.datasets
import sklearn.cluster
import matplotlib.pyplot as plot

# Creamos los puntos
n = 1000
k = 4

# Generate fake data
data, labels = sklearn.datasets.make_blobs(
    n_samples=n, n_features=2, centers=k)


I am working on the file "01-Instalación.ipynb".
The first task is:
Notebooks
Los notebooks de Python son fichero terminados en ".pynb", que pueden ser abiertos desde el navegador usando jupyter. Estos notebooks se dividen en celdas que pueden contener texto y código Python que se puede ejecutar, mostrando la salida de su ejecución.
Github entiende el formato, y permite visualizar un notebook, pero no se puede ejecutar el código, para eso es necesario descargar el fichero y editarlo localmente.
Recursos
Existe una gran cantidad de Notebooks muy útiles, en la propia wiki de Jupyter se encuentra una extensa galería de interesantes notebooks
Ejemplo del lenguaje Python
Vamos a empezar con el Hello, World. Mientras que en C sería

include<stdio.h>
int main(void) {
   printf("Hello World\n");
   return 0;
}

El código en Python es mucho más sencillo.
Can you write Python code for it?


I am working on the file "01-Instalación.ipynb".
The first task is:
Primero pintamos los puntos
Can you write Python code for it?


I am working on the file "01-Instalación.ipynb".
The first task is:
Visualizando datos
Vamos a visualizar unos pocos datos
Can you write Python code for it?

%pylab inline

import matplotlib
import matplotlib.pyplot as plt
import numpy as np

x = np.arange(30)
plt.plot(x,  x**2)


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
5. Demonstration: Guess the Novel
To illustrate this process on a slightly larger scale, we will do the exactly what we did above, but will do so on two unknown novels. Your challenge: guess the novels from the most frequent words.
We will do this in one chunk of code, so another challenge for you during breaks or the next few weeks is to see how much of the following code you can follow (or, in computer science terms, how much of the code you can parse). If the answer is none, not to worry! Tomorrow we will take a step back and work on the nitty gritty of programming.
Can you write Python code for it?

# Read the two text files from your hard drive
# Assign first mystery text to variable 'text1' and second to 'text2'

text1 = open('text1.txt').read()
text2 = open('text2.txt').read()

# Tokenize both texts

text1_tokens = nltk.word_tokenize(text1)
text2_tokens = nltk.word_tokenize(text2)

# Set to lower case

text1_tokens_lc = [word.lower() for word in text1_tokens]
text2_tokens_lc = [word.lower() for word in text2_tokens]

# Remove stopwords

text1_tokens_nostops = [word for word in text1_tokens_lc if word not in stopwords.words('english')]
text2_tokens_nostops = [word for word in text2_tokens_lc if word not in stopwords.words('english')]

# Remove punctuation using the list of punctuation from the string pacage

text1_tokens_clean = [word for word in text1_tokens_nostops if word not in string.punctuation]
text2_tokens_clean = [word for word in text2_tokens_nostops if word not in string.punctuation]

# Frequency distribution

text1_word_frequency = collections.Counter(text1_tokens_clean)
text2_word_frequency = collections.Counter(text2_tokens_clean)

# Guess the novel!

text1_word_frequency.most_common(20)

# Guess the novel!

text2_word_frequency.most_common(20)


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
Better! The ten most frequent words now give us a pretty good sense of the substance of this sentence. But we still have problems. For example, the token "'s" sneaked in there. One solution is to keep adding stop words to our list, but this could go on forever and is not a good solution when processing lots of text.
There's another way of identifying content words, and it involves identifying the part of speech of each word.
4. Part-of-Speech Tagging
You may have noticed that stop words are typically short function words, like conjunctions and prepositions. Intuitively, if we could identify the part of speech of a word, we would have another way of identifying which contribute to the text's subject matter. NLTK can do that too!
NLTK has a <i>POS Tagger</i>, which identifies and labels the part-of-speech (POS) for every token in a text. The particular labels that NLTK uses come from the Penn Treebank corpus, a major resource from corpus linguistics.
You can find a list of all Penn POS tags here: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
Note that, from this point on, the code is going to get a little more complex. Don't worry about the particularities of each line. For now, we will focus on the NLP tasks themselves and the textual patterns they identify.
Can you write Python code for it?

# Let's revisit our original list of tokens

print(sentence_tokens)

# Use the NLTK POS tagger

nltk.pos_tag(sentence_tokens)

# Assign POS-tagged list to a variable

tagged_tokens = nltk.pos_tag(sentence_tokens)


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
Computational Text Analysis is not simply the processing of texts through computers, but involves reflection on the part of human interpreters. How were you able to tell what each novel was? Do you notice any differences between each novel's list of frequent words?
The patterns that we notice in our computational model often enrich and extend our research questions -- sometimes in surprising ways! What next steps would you take to investigate these novels?
6. Concordances and Similar Words using NLTK
Tallying word frequencies gives us a bird's-eye-view of our text but we lose one important aspect: context. As the dictum goes: "You shall know a word by the company it keeps."
Concordances show us every occurrence of a given word in a text, inside a window of context words that appear before and after it. This is helpful for close reading to get at a word's meaning by seeing how it is used. We can also use the logic of shared context in order to identify which words have similar meanings. To illustrate this, we can compare the way the word "monstrous" is used in our two novels.
Concordance
Can you write Python code for it?

# Transform our raw token lists in NLTK Text-objects
text1_nltk = nltk.Text(text1_tokens)
text2_nltk = nltk.Text(text2_tokens)

# Really they're no differnt from the raw text, but they have additional useful functions
print(text1_nltk)
print(text2_nltk)

# Like a concordancer!

text1_nltk.concordance("monstrous")

text2_nltk.concordance("monstrous")


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
Contextual Similarity
Can you write Python code for it?

# Get words that appear in a similar context to "monstrous"

text1_nltk.similar("monstrous")

text2_nltk.similar("monstrous")


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
Note on Term Frequency
Some of the most frequent words appear to summarize the sentence: in particular the words "humanistic", "digital", and "media". However, most of the these terms seem to add noise in the summary: "the", "it", "to", ".", etc.
There are many strategies for identifying the most important words in a text, and we will cover the most popular ones in the next week. Today, we will look at two of them. In the first, we will simply remove the noisey tokens. In the second, we will identify important words using their parts of speech.
3. Pre-Processing: Lower Case, Remove Stop Words and Punctuation
Typically, a text goes through a number of pre-processing steps before beginning to the actual analysis. We have already seen the tokenization step. Typically, pre-processing includes transforming tokens to lower case and removing stop words and punctuation marks.
Again, pre-processing is a non-trivial process that can have large impacts on the analysis that follows. For instance, what will be the most common token in our example sentence, once we set all tokens to lower case?
Lower Case
Can you write Python code for it?

# Let's revisit our original sentence

sentence

# And now transform it to lower case, all at once

sentence.lower()

# Okay, let's set our list of tokens to lower case, one at a time

# The syntax of the line below is tricky. Don't worry about it for now.
# We'll spend plenty of time on it tomorrow!

lower_case_tokens = [ word.lower()  for word in sentence_tokens ]

# Inspect

print(lower_case_tokens)


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
Now it's getting interesting
The "IN" tag refers to prepositions, so it's no surprise that it should be the most common. However, we can see at a glance now that the sentence contains a lot of adjectives, "JJ". This feels like it tells us something about the rhetorical style or structure of the sentence: certain qualifiers seem to be important to the meaning of the sentence.
Let's dig in to see what those adjectives are.
Can you write Python code for it?

# Let's filter our list, so it only keeps adjectives

adjectives = [word for word,pos in tagged_tokens if pos == 'JJ' or pos=='JJR' or pos=='JJS']

# Inspect

print( adjectives )

# Tally the frequency of each adjective

adj_frequency = collections.Counter(adjectives)

# Most frequent adjectives

adj_frequency.most_common(5)

# Let's do the same for nouns.

nouns = [word for word,pos in tagged_tokens if pos=='NN' or pos=='NNS']

# Inspect

print(nouns)

# Tally the frequency of the nouns

noun_frequency = collections.Counter(nouns)

# Most Frequent Nouns

print(noun_frequency.most_common(5))


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
Punctuation
Can you write Python code for it?

# Import a list of punctuation marks

import string

# Inspect

string.punctuation

# Remove punctuation marks from token list

tokens_clean = [word for word in tokens_nostops if word not in string.punctuation]

# See what's left

print(tokens_clean)


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
Re-count the Most Frequent Words
Can you write Python code for it?

# Count the new token list

word_frequency_clean = collections.Counter(tokens_clean)

# Most common words

word_frequency_clean.most_common(10)


I am working on the file "01-Intro to NLP/.ipynb_checkpoints/Intro to NLP-checkpoint.ipynb".
The first task is:
Stop Words
Can you write Python code for it?

# Import the stopwords list
from nltk.corpus import stopwords

# Take a look at what stop words are included

print(stopwords.words('english'))

# Try another language

print(stopwords.words('spanish'))

# Create a new variable that contains the sentence tokens but NOT the stopwords

tokens_nostops = [ word  for word in lower_case_tokens  if word not in stopwords.words('english') ]

# Inspect

print(tokens_nostops)


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
2. Tokenizing Text and Counting Words
The above output is how a human would read that sentence. Next we look the main way in which a computer "reads", or parses, that sentence.
The first step is typically to tokenize it, or to change it into a series of tokens. Each token roughly corresponds to either a word or punctuation mark. These smaller units are more straight-forward for the computer to handle for tasks like counting.
Can you write Python code for it?

# Import the NLTK (Natural Language Tool Kit) package

import nltk

# Download NLTK language models

nltk_data = ["punkt", "words", "stopwords", "averaged_perceptron_tagger", "maxent_ne_chunker", 'wordnet']
nltk.download(nltk_data)

# Tokenize our sentence!

nltk.word_tokenize(sentence)

# Create new variable that contains our tokenized sentence

sentence_tokens = nltk.word_tokenize(sentence)

# Inspect our new variable
# Note the square braces at the beginning and end that indicate we are looking at a list-type object

print(sentence_tokens)


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
5. Demonstration: Guess the Novel
To illustrate this process on a slightly larger scale, we will do the exactly what we did above, but will do so on two unknown novels. Your challenge: guess the novels from the most frequent words.
We will do this in one chunk of code, so another challenge for you during breaks or the next few weeks is to see how much of the following code you can follow (or, in computer science terms, how much of the code you can parse). If the answer is none, not to worry! Tomorrow we will take a step back and work on the nitty gritty of programming.
Can you write Python code for it?


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
And now verbs.
Can you write Python code for it?

# And we'll do the verbs in one fell swoop

verbs = [word for word,pos in tagged_tokens if pos == 'VB' or pos=='VBD' or pos=='VBG' or pos=='VBN' or pos=='VBP' or pos=='VBZ']
verb_frequency = collections.Counter(verbs)
print(verb_frequency.most_common(5))

# If we bring all of this together we get a pretty good summary of the sentence

print(adj_frequency.most_common(3))
print(noun_frequency.most_common(3))
print(verb_frequency.most_common(3))


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
Better! The ten most frequent words now give us a pretty good sense of the substance of this sentence. But we still have problems. For example, the token "'s" sneaked in there. One solution is to keep adding stop words to our list, but this could go on forever and is not a good solution when processing lots of text.
There's another way of identifying content words, and it involves identifying the part of speech of each word.
4. Part-of-Speech Tagging
You may have noticed that stop words are typically short function words, like conjunctions and prepositions. Intuitively, if we could identify the part of speech of a word, we would have another way of identifying which contribute to the text's subject matter. NLTK can do that too!
NLTK has a <i>POS Tagger</i>, which identifies and labels the part-of-speech (POS) for every token in a text. The particular labels that NLTK uses come from the Penn Treebank corpus, a major resource from corpus linguistics.
You can find a list of all Penn POS tags here: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
Note that, from this point on, the code is going to get a little more complex. Don't worry about the particularities of each line. For now, we will focus on the NLP tasks themselves and the textual patterns they identify.
Can you write Python code for it?


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
Contextual Similarity
Can you write Python code for it?


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
Note on Term Frequency
Some of the most frequent words appear to summarize the sentence: in particular the words "humanistic", "digital", and "media". However, most of the these terms seem to add noise in the summary: "the", "it", "to", ".", etc.
There are many strategies for identifying the most important words in a text, and we will cover the most popular ones in the next week. Today, we will look at two of them. In the first, we will simply remove the noisey tokens. In the second, we will identify important words using their parts of speech.
3. Pre-Processing: Lower Case, Remove Stop Words and Punctuation
Typically, a text goes through a number of pre-processing steps before beginning to the actual analysis. We have already seen the tokenization step. Typically, pre-processing includes transforming tokens to lower case and removing stop words and punctuation marks.
Again, pre-processing is a non-trivial process that can have large impacts on the analysis that follows. For instance, what will be the most common token in our example sentence, once we set all tokens to lower case?
Lower Case
Can you write Python code for it?


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
Note on Tokenization
While seemingly simple, tokenization is a non-trivial task.
For example, notice how the tokenizer has handled contractions: a contracted word is divided into two separate tokens! What do you think is the motivation for this? How else might you tokenize them?
Also notice each token is either a word or punctuation mark. In practice, it is sometimes useful to remove punctuation marks and at other times to include them, depending on the situation.
In the coming days, we will see other tokenizers and have opportunities to explore their reasoning. For now, we will look at a few examples of NLP tasks that tokenization enables.
Can you write Python code for it?

# How many tokens are in our list?

len(sentence_tokens)

# How often does each token appear in our list?

import collections

collections.Counter(sentence_tokens)

# Assign those token counts to a variable

token_frequency = collections.Counter(sentence_tokens)

# Get an ordered list of the most frequent tokens

token_frequency.most_common(10)


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
Now it's getting interesting
The "IN" tag refers to prepositions, so it's no surprise that it should be the most common. However, we can see at a glance now that the sentence contains a lot of adjectives, "JJ". This feels like it tells us something about the rhetorical style or structure of the sentence: certain qualifiers seem to be important to the meaning of the sentence.
Let's dig in to see what those adjectives are.
Can you write Python code for it?


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
Punctuation
Can you write Python code for it?


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
Re-count the Most Frequent Words
Can you write Python code for it?


I am working on the file "01-Intro to NLP/Intro to NLP.ipynb".
The first task is:
Stop Words
Can you write Python code for it?


I am working on the file "01-Intro-UNIX/Introduction_orig.ipynb".
The first task is:
News Groups Dataset
http://qwone.com/~jason/20Newsgroups/
he 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder: Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering.

DO NOT DOWNLOAD THESE FILES ONTO THE CLUSTER!

Instead, use the shared data directory /home/data/20_newsgroup/
Can you write Python code for it?

%%sh
DATADIR=/home/data/20_newsgroup/
ls $DATADIR | while read TOPIC; do
echo -n "Topic $TOPIC: number of documents "
ls $DATADIR/$TOPIC | wc -l
done | cat -b


I am working on the file "01-Intro-UNIX/Introduction_orig.ipynb".
The first task is:
Not bad. But this could be improved...
What are the most frequent words?
Can you write Python code for it?

%%sh
grep -v -f data/legalnotice.txt data/pg100.txt \
| tr -d '.,:?"' \
| tr 'A-Z' 'a-z' \
| tr ' ' '\n' \
| tr -s '\n' \
| grep -v -e '^[[:space:]]*$' \
| sort \
| uniq -c \
| sort -rn \
| head -30


I am working on the file "01-Intro-UNIX/Introduction_orig.ipynb".
The first task is:
UNIX Commands/Tools for Data Manipulation and Analysis

grep
sort
uniq
tr
wc
cut

Advanced

awk
sed

The Works  by Shakespeare
Let's see what we can do with Shakespeare's collected body of work.
We can download everything from http://www.gutenberg.org/ebooks/100 in plain text format
Can you write Python code for it?

%%sh
mkdir -p data
cd data
wget http://www.gutenberg.org/cache/epub/100/pg100.txt 2> /dev/null
cd ..
ls -l data


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
"Slice" a list using [ : ]

Just as with strings, we can get multiple items from a list using splicing
Note that the first index is included, while the second is excluded
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Challenge 1: Making and Printing Variables
Make 3 variables:
- name (with your full name)
- city (where you were born) and
- year (when you were born.)
Print these three variables so that it prints [your name] was born in [city] in [year].

Data Types
Every value has a type.

Every value in a program has a specific type.
Integer (int): counting numbers like 3 or -512.
Floating point number (float): fractional numbers like 3.14159 or -2.5.
Integers are used to count, floats are used to measure.

Character string (usually just called "string", str): text.
Written in either single quotes or double quotes (as long as they match).
The quotation marks aren't printed when the string is displayed.

Use the built-in function type to find the type of a value.

Use the built-in function type to find out what type a value has.
Works on variables as well.
But remember: the value has the type --- the variable is just a label.
Can you write Python code for it?

print(type(52))

pi = 3.14159
print(type(pi))

fitness = 'average'
print(type(fitness))


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Challenge 5: Exploring Lists
What does the following program print?
Can you write Python code for it?

hashtags = ['#March4Trump',
 '#Fascism',
 '#TwitterIsFascist',
 '#majority',
 '#CouldntEvenStopDeVos',
 '#IsTrumpCompromised',
 '#Berkeley',
 '#NotMyPresident',
 '#mondaymotivation',
 '#BlueLivesMatter',
 '#Action4Trump',
 '#impeachtrump'
 '#Periscope',
 '#march',
 '#TrumpRussia',
 '#obamagate',
 '#Resist',
 '#sedition',
 '#NeverTrump',
 '#maga']

print(hashtags[::2])
print()
print(hashtags[::-1])


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
For what its worth, you don't need to have a variable to use the upper() method, you could use it on the string itself.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
If you want to append items to the end of a list, use the append method.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Leave an index blank to get everything from the beginning / end
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Let's look at the upper method. What does it do? Lets take a look at the documentation. IPython lets us do this with a question mark ('?') before or after an object (again, a defined variable).
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Lists can be indexed from the back using a negative index.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Lists have Methods

Just like strings have methods, lists do too.
Remember that a method is like a function, but tied to a particular object.
Use object_name.method_name to call methods.
IPython lets us do tab completion after a dot ('.') to see what an object has to offer.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Lists may contain values of different types.

A single list may contain numbers, strings, and anything else.
Can you write Python code for it?

complex_list = ['life', 42, 'the universe', [1,2,3]]
print(complex_list)


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Lists’ values can be replaced by assigning to specific indices.
Can you write Python code for it?

country_list[0] = "Iran"
print('Country List is now:', country_list)


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Multiplying a character string by an integer replicates it.
Since multiplication is just repeated addition.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Must convert numbers to strings or vice versa when operating on them.

Cannot add numbers and strings.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Notice that we put a list inside of a list, which can itself be indexed. The same could be done for a string.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Strings are made up of sub-strings

You can think of strings as a sequence of smaller strings or characters.
We can access a piece of that sequence using [].
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Strings have a length (but numbers don't).

The built-in function len counts the number of characters in a string.
Can you write Python code for it?