I am working on the file "03-BuiltinDataStructures.ipynb".
The first task is:
The min(), max(), sum() functions.
Can you write Python code for it?

print(min(a))  # Only for list of numbers!
print(max(a))  # Only for list of numbers!
print(sum(a))  # Only for list of numbers!

b = ['hi', 'hello', 'abracadabra']

print(min(b, key=len))  # Only for list of strings!
print(max(b, key=len))  # Only for list of strings!


I am working on the file "03-BuiltinDataStructures.ipynb".
The first task is:
Warning! A shallow copy is only effective for non nested lists. If you have a nested list you'll need to do a deep copy!
Can you write Python code for it?

a = [[1, 2, 3], [4, 5, 6]]
b = a.copy()   # A shallow copy doesn't work for nested objects!

b[0][1] = 20
b[1][1] = 50

print(a)
print(b)

import copy

a = [[1, 2, 3], [4, 5, 6]]
b = copy.deepcopy(a)   # A deep copy will work for any kind object!
                       # If you are not sure what to do, just use
                       # copy.deepcopy() to be safe!

b[0][1] = 20
b[1][1] = 50

print(a)
print(b)


I am working on the file "03-BuiltinDataStructures.ipynb".
The first task is:
We can also declare a list of lists (a.k.a. nested lists), and use indexing to retrieve its items.
Can you write Python code for it?


I am working on the file "03-Close-Reading-I/01-Strings.ipynb".
The first task is:
Challenge 3: Working with strings
Below is a string of Edgar Allen Poe's "A Dream Within a Dream":
Can you write Python code for it?

poem = '''Take this kiss upon the brow!
And, in parting from you now,
Thus much let me avow —
You are not wrong, who deem
That my days have been a dream;
Yet if hope has flown away
In a night, or in a day,
In a vision, or in none,
Is it therefore the less gone?
All that we see or seem
Is but a dream within a dream.

I stand amid the roar
Of a surf-tormented shore,
And I hold within my hand
Grains of the golden sand —
How few! yet how they creep
Through my fingers to the deep,
While I weep — while I weep!
O God! Can I not grasp
Them with a tighter clasp?
O God! can I not save
One from the pitiless wave?
Is all that we see or seem
But a dream within a dream?'''


I am working on the file "03-Close-Reading-I/01-Strings.ipynb".
The first task is:
Strings are made up of sub-strings
You can think of strings as a sequence of smaller strings or characters. We can access a piece of that sequence using square brackets [].
Can you write Python code for it?


I am working on the file "03-Close-Reading-I/01-Strings.ipynb".
The first task is:
Strings have methods

There are other operations defined on string data. These are called string methods.
The Jupyter Notebooks lets you do tab-completion after a dot ('.') to see what methods an object (i.e., a defined variable) has to offer. Try it now!
Can you write Python code for it?


I am working on the file "03-Close-Reading-I/01-Strings.ipynb".
The first task is:
What do you think should happen when you take upper of an int?  What about a string representation of an int?
Can you write Python code for it?


I am working on the file "03-Close-Reading-I/01-Strings.ipynb".
The first task is:
What is the difference between poem.strip("?") and poem.replace("?", "") ?
At what index does the word "and" first appear? Where does it last appear?
How can you answer the above accounting for upper- and lowercase?
Challenge 4: Counting Text
Below is a string of Robert Frost's "The Road Not Taken":
Can you write Python code for it?

poem = '''Two roads diverged in a yellow wood,
And sorry I could not travel both
And be one traveler, long I stood
And looked down one as far as I could
To where it bent in the undergrowth;

Then took the other, as just as fair,
And having perhaps the better claim,
Because it was grassy and wanted wear;
Though as for that the passing there
Had worn them really about the same,

And both that morning equally lay
In leaves no step had trodden black.
Oh, I kept the first for another day!
Yet knowing how way leads on to way,
I doubted if I should ever come back.

I shall be telling this with a sigh
Somewhere ages and ages hence:
Two roads diverged in a wood, and I—
I took the one less traveled by,
And that has made all the difference.'''


I am working on the file "03-Close-Reading-I/01-Strings.ipynb".
The first task is:
You can see some of the logic for this when we consider implicit indices.
Can you write Python code for it?


I am working on the file "03-Close-Reading-I/01-Strings.ipynb".
The first task is:
You can slice strings using  [ : ]
If you want a range (or "slice") of a sequence, you get everything before the second index, i.e,. Python slicing is exclusive:
Can you write Python code for it?


I am working on the file "03-Close-Reading-I/01-Strings.ipynb".
The first task is:
You have to use the parenthesis at the end because upper is a method of the string class.

Don't forget, simply calling the method does not change the original variable, you must *reassign* the variable:

Can you write Python code for it?

print(full_name)

full_name = full_name.upper()
print(full_name)


I am working on the file "03-Eigenvalue_problem_solving.ipynb".
The first task is:
$QR$ factorisation
The full spectrum of a matrix
Our aim is to compute the full spectrum of the square $n \times n$
matrix $A$; that is, we want to find all its eigenvalues.
Method: transform to a simpler problem that is straightforwardly solved.
E.g. transform $A$ to $B$ with same spectrum, but $B$ triangular:
eigenvalues of a triangular matrix are the diagonal coefficients.
Schur’s theorem shows that every matrix $A$ has a similar triangular
matrix $B$, but is not useful for finding the matrix in practice.
A decomposition method for a general (not necessarily square)
matrix is the orthogonal factorisation. $A$ is written as a product of matrices,
some of which are orthogonal (i.e. real – $Q^{\dagger} =
  Q^T$ – and unitary, $Q^{-1} = Q^{\dagger} = Q^T$).
Simple example: Gram-Schmidt $QR$-factorisation
$$A = Q R,$$
with $Q$ an orthogonal matrix (i.e. $Q^TQ=I$), $R$ upper triangular. $A$ and $R$ may be $m
  \times n$ matrices, but $Q$ is always square ($m \times m$).
If $A = QR$ with $Q$ unitary then
$$B = R Q = \left( Q^{\dagger} Q \right) R Q = Q^{\dagger} A Q$$
which is similar to $A$.
Gram-Schmidt’s factorisation
Consider the Gram-Schmidt procedure, with the vectors to be considered in the process as columns of the matrix $A$. That is,
$$A = \left[ {\mathbf{a}}_1 \; | \; {\mathbf{a}}_2 \; | \; \cdots \; | \; {\mathbf{a}}_n \right].$$
Then,
$$\mathbf{u}_1 = \mathbf{a}_1, \qquad \mathbf{e}_1 = \frac{\mathbf{u}_1}{\|\mathbf{u}_1\|}$$
$$\mathbf{u}_2 = \mathbf{a}_2 - (\mathbf{a}_2.\mathbf{e}_1)\mathbf{e}_1 , \qquad \mathbf{e}_2 = \frac{\mathbf{u}_2}{\|\mathbf{u}_2\|}$$
$$\mathbf{u}{k+1} = \mathbf{a}{k+1} - (\mathbf{a}{k+1}.\mathbf{e}_1)\mathbf{e}_1 - \cdots -  (\mathbf{a}{k+1}.\mathbf{e}k)\mathbf{e}_k, \qquad \mathbf{e}{k+1} = \frac{\mathbf{u}{k+1}}{\|\mathbf{u}{k+1}\|}$$
Note that $\|\bullet\|$ is the $L_2$ norm.
The resulting QR factorisation is
$$A=\left[ {\mathbf{a}}_1 \; | \; {\mathbf{a}}_2 \; | \; \cdots \; | \; {\mathbf{a}}_n \right]=\left[ {\mathbf{e}}_1 \; | \; {\mathbf{e}}_2 \; | \; \cdots \; | \;...
Can you write Python code for it?

def gs_m(A):
    m,n= A.shape
    A= A.copy()
    Q= np.zeros((m,n))
    R= np.zeros((n,n))
    for k in range(n):
        uk = A[:,k]
        for j in range(0,k):
            uk = uk - np.dot(A[:,k],Q[:,j])*Q[:,j]
            R[j,k] = np.dot(A[:,k],Q[:,j])
        Q[:,k] = uk/np.linalg.norm(uk,2)
        R[k,k] = np.dot(A[:,k],Q[:,k])
    return Q, R


#This algorithm computes eigenvectors for symmetric matrices only
#For non Hermitian matrices, we need to use backward substitution with Schur vetors which is too compliated here...
def qr_algo(A, niterations_max=100, tol=1e-15):
    B = np.copy(A)
    Uk = np.identity(len(A))
    evals_approx = np.zeros((len(A),niterations_max))
    evals_approx[:,0] = np.diag(A)
    err = 1.0
    n = 0
    while err > tol and n < niterations_max-1:
        Q, R = np.linalg.qr(B)
        #Q, R  = gs_m(B)
        B = np.dot(R, Q)
        Uk = np.dot(Uk,Q)
        err = np.linalg.norm(np.tril(B, -1))
        n += 1
        evals_approx[:,n]=np.diag(B)
    return np.diag(B), Uk, evals_approx

A = np.array([[1.0,2.0,3.0],[4.0,5.0,6.0],[7.0,8.0,0.0]])
lamda, v = np.linalg.eig(A)
lamda_qr, vqr, lamda_seq = qr_algo(A)
print("QR gives eigenvalues of {}. Exact are {}.".format(lamda_qr, lamda))
print("QR gives eigenvectors of {}. Exact are {})".format(vqr, v))
#
TT = np.diag(lamda_qr)
print("\n")
print(np.dot(A,vqr))
print("\n")
print(np.dot(vqr,TT))
print("\n")
print(np.dot(np.transpose(A),A))
print(np.dot(A,np.transpose(A)))


I am working on the file "03-Eigenvalue_problem_solving.ipynb".
The first task is:
Generalized eigenvalue problem
$
\newcommand{\bx}{\boldsymbol{x}}
$
Many times in structural engineering, eigenvalue problems appear in their generalized form:
$$A\boldsymbol{u} = \lambda B\boldsymbol{u}.$$
The simplest way to solve a generalized eigenvalue problem is to recast it in the classic form:
$$C\boldsymbol{u} = \lambda \boldsymbol{u}$$
with $C=B^{-1}A$. Like in the inverse power method, do not use $B^{-1}$ directly, but solve linear system instead.
Can you write Python code for it?

A = np.array([[12, 3, 4], [3, 167, 6], [4, 6, -41]])
B = np.array([[6, 2, 4], [3, 3, 5], [6, 32, -6]])
C = np.linalg.solve(B,A)
#C is not a normal matrix so we won't be able to compute all its eigenvectors
# either with classic QR-algorithm and
print(np.dot(np.transpose(C),C))
print(np.dot(C,np.transpose(C)))
print("\n")
lamda, vl, vr = linalg.eig(A,B,left=True,right=True)
print("The eigenvalues of the generalized eigenvalue problem are {}.".format(lamda))
print("\n")

lamda_power1, lamda_seq1, vpm1 = full_power_method(C)

D = np.matrix(C)-lamda_power1*np.transpose(np.matrix(vpm1))*np.matrix(vpm1)
lamda_power2, lamda_seq2, vpm2 = full_power_method(D)

E = np.matrix(D)-lamda_power2*np.transpose(np.matrix(vpm2))*np.matrix(vpm2)
lamda_power3, lamda_seq3, vpm3 = full_power_method(E)

print("The eigenvalues of the generalized eigenvalue problem computed with the deflated power method are {},{},{}.".format(lamda_power1,lamda_power2,lamda_power3))
print("\n")


I am working on the file "03-Eigenvalue_problem_solving.ipynb".
The first task is:
Hotelling's deflation method
Suppose we have found the largest eigenvalue of the matrix $A$ with the power iiteration method, how do we find the second largest eigenvalue? One solution is, after finding the largest eigenvalue $\lambda_1$, to make it into the smallest by deflation and then go on to find the new largest one, let say $\lambda_2$.
Consider:
$$\left(A-\lambda_1\mathbf{u}_1\mathbf{u}_1^T\right)\mathbf{u}_j=A\mathbf{u}_j-\lambda_1\mathbf{u}_1\mathbf{u}_1^T\mathbf{u}_j=\lambda_j\mathbf{u}_j-\lambda_1\mathbf{u}_1\left(\mathbf{u}_1^T\mathbf{u}_j\right)$$
$$\text{If $j=1$ then:} \qquad \left(A-\lambda_1\mathbf{u}_1\mathbf{u}_1^T\right)\mathbf{u}_j=\lambda_1\mathbf{u}_1-\lambda_1\mathbf{u}_1\left(\mathbf{u}_1^T\mathbf{u}_1\right) = 0\mathbf{u}_1$$
$$\text{If $j\neq1$ then:} \qquad \left(A-\lambda_1\mathbf{u}_1\mathbf{u}_1^T\right)\mathbf{u}_j=\lambda_j\mathbf{u}_j-\lambda_1\mathbf{u}_1\left(0\right) = \lambda_j\mathbf{u}_j$$
thus, $\left(A-\lambda_1\mathbf{u}_1\mathbf{u}_1^T\right)$ has the same eigenvectors as $A$ and the same eigenvalues as $A$ except that the largest one has been replaced by $0$. Thus we can use the power method to find the next biggest one $\lambda_2$ and so on...
Can you write Python code for it?

A = np.array([[1.0,2.0,3.0],[4.0,5.0,6.0],[7.0,8.0,0.0]])
lamda_power, lamda_seq, vpm = full_power_method(A)
B = np.matrix(A)-lamda_power*np.transpose(np.matrix(vpm))*np.matrix(vpm)
lamda, v = np.linalg.eig(B)
print("The new eigenvalues of the deflated matrix B are {}.".format(lamda))
lamda_power2, lamda_seq2, vpm2 = full_power_method(B)
print("The next eigenvalue with the highest magnitude computed with the power method is {}.".format(lamda_power2))


I am working on the file "03-Eigenvalue_problem_solving.ipynb".
The first task is:
Let's apply this algorithm on a matrix A:
Can you write Python code for it?

A = np.array([[1.0,2.0,3.0],[4.0,5.0,6.0],[7.0,8.0,0.0]])
print(A)

lamda, v = np.linalg.eig(A) # This is the built-in eigenvalue problem solver of Numpy
lamda_power, lamda_seq, vpm = power_method(A)
print("The maximum eigenvalue from the power method is {} (exact is {}, error is {})".format(lamda_power, np.max(lamda),
                                                                                abs(lamda_power - np.max(lamda))))
print("The associated eigenvector from the power method is {} (exact is {})".format(vpm, v[:,0]))


I am working on the file "03-Eigenvalue_problem_solving.ipynb".
The first task is:
Revision
Eigenvalues
$
\newcommand{\bx}{\boldsymbol{x}}
$
Eigenvalues, $\lambda$ and their corresponding eigenvectors
$\boldsymbol{u}$ are non-zero solutions to the linear system
$$A\boldsymbol{u} = \lambda \boldsymbol{u}.$$
Matrix eigenvalues and eigenvectors are of significant importance in physics. Notably in numerical engineering where they give informations about critical buckling loads or natural frequencies of a structure.
Eigenvalues and polynomials
Standard definition of eigenvalues: the $n$ roots of the characteristic
polynomial
$$\det ( A - \lambda I) = 0.$$
Could compute roots e.g. by nonlinear root finding.
There are two essential problems with this:

Polynomials may be badly conditioned: a small change in the
    coefficients leads to a large change in the roots.

Frequently do not need all eigenvalues, but only the largest one(s).
    Computing all, and then sorting, excessively expensive.

A $1\%$ change in the last coefficient leads to massive changes for
$$p(z) = -120 + 274 z - 225 z^2 + 85 z^3 -15 z^4 + z^5;$$
the roots $(4, 5)$ become $(4.580 \pm 0.966 \sqrt{-1})$.
In the following, we will see to type of methods called iterative methods:

The power method is a classic method to find the largest, or smallest, or closest eigenvalue of a matrix

The QR algorithm introduced by Francis is an iterative method that computes all the eigenvalues and eigenvectors of a matrix.

The power method
The idea is to compute the largest eigenvalue without relying on characteristic polynomial. Look at what happen if we successively apply a matrix $A$ on a vector ${\boldsymbol{x}}$ representing a unit circle:
Can you write Python code for it?

theta = np.linspace(0.0, 2.0*np.pi)
X = np.zeros((5,2,len(theta)))
X[0,0,:] = np.cos(theta)
X[0,1,:] = np.sin(theta)
A = np.random.rand(2,2)
for n in range(4):
    for i in range(len(theta)):
        X[n+1,:,i] = np.dot(A, X[n,:,i])

fig = plt.figure(figsize=(10,6))
ax = fig.add_subplot(111, projection='3d')
ax.plot(X[0,0,:], X[0,1,:], label='Unit circle')
ax.plot(X[1,0,:], X[1,1,:], label=r"Circle acted on by $A$")
ax.plot(X[2,0,:], X[2,1,:], label=r"Circle acted on by $A^2$")
ax.plot(X[3,0,:], X[3,1,:], label=r"Circle acted on by $A^3$")
ax.legend()
fig.tight_layout()
plt.show()


I am working on the file "03-Eigenvalue_problem_solving.ipynb".
The first task is:
Shifted power method
Another minor variant allows us to find the eigenvalue closest to a
given complex number $\sigma$. We just have to make use of:
$$\lambda_i \text{ are eigenvalues of } A \Rightarrow
    \lambda_i - \sigma \text{ are eigenvalues of } A - \sigma
    \text{Id}$$
Therefore the smallest eigenvalue of $A - \sigma \text{Id}$ is the one
closest to $\sigma$; this is just an application of the inverse power
method.
Shifted power method example
The matrix
$$A =
        \begin{pmatrix}
          1 & 2 & 3 \
          4 & 5 & 6 \
          7 & 8 & 0
        \end{pmatrix}$$
has eigenvalues
$$\left{
          \begin{array}{c}
            12.1229\ -5.7345\ -0.3884
          \end{array}\right. .$$
The shifted power method shows linear convergence to $\lambda =
      -5.7345$ for the eigenvalue closest to $-5$.
Can you write Python code for it?

def shifted_power_method(A, sigma, niterations_max=50, tol=1e-15):
    Ashift = A - sigma * np.eye(len(A))
    xn = np.zeros((len(A), niterations_max+1))
    xn[:, 0] = np.ones((len(A),)) + 1e-7*np.random.rand(len(A))
    rn = np.ones((niterations_max+1,))
    for k in range(niterations_max):
        xn[:,k] = xn[:,k] / np.linalg.norm(xn[:,k])
        xn[:,k+1] = np.linalg.solve(Ashift, xn[:,k])
        rn[k+1] = np.sum(xn[:,k+1])/np.sum(xn[:,k])
        if (abs(rn[k+1]-rn[k]) < tol):
            break
    if k < niterations_max:
        rn[k+2:] = rn[k+1] # This ensures the later values are set to something sensible.
    return (1.0/rn[k+1] + sigma, 1.0/rn + sigma, xn[:,k+1]/ np.linalg.norm(xn[:,k+1]))

lamda_shift, lamda_seq, vpm = shifted_power_method(A, -5.0)

print("The eigenvalue closest to -5.0 from the shifted power method is {} (exact is {}, error is {})".format(lamda_shift, lamda[1],
                                                                                abs(lamda_shift - lamda[1])))
print("The associated eigenvector from the power method is {} (exact is {})".format(vpm, v[:,2]))


I am working on the file "03-Eigenvalue_problem_solving.ipynb".
The first task is:
We can plot the convergence of the power method algorithm:
Can you write Python code for it?

errors = np.abs(lamda_seq - np.max(lamda))
iterations = range(len(errors))

fig = plt.figure(figsize=(10,6))
ax = fig.add_subplot(111)
ax.semilogy(iterations, errors, 'kx')
ax.set_xlabel('Iteration')
ax.set_ylabel(r"$\|$ Error $\|$")
ax.set_title(r"Convergence of the power method, $n=3$")
fig.tight_layout()
plt.show()


I am working on the file "03-Generator-Expressions.ipynb".
The first task is:
But they can be iterated over just as easily
They're a really quick way to create a iterator, without the overhead of a list.
Can you write Python code for it?

sum_1 = 0
for x in list_comp:
    sum_1 += x

sum_2 = 0
for y in gen_exp:
    sum_2 += y

print(sum_1, sum_2)


I am working on the file "03-Generator-Expressions.ipynb".
The first task is:
They're one use only though
Note that once you've iterated through a generator expression once, it will just keep raising StopIteration. You have to be careful that you don't try to iterate through them multiple times, or you could create subtle bugs.
Can you write Python code for it?

for y in gen_exp:
    print(y)

next(gen_exp)


I am working on the file "03-IO/19-database_access.ipynb".
The first task is:
Exercise 1:
Create a list with the jobs of the third column of resultado. Tip: a list comprehension can be useful!
Can you write Python code for it?

all_jobs = [x[2] for x in resultado]

all_jobs = []
for i in resultado:
    all_jobs.append(i[2])
all_jobs


I am working on the file "03-IO/19-database_access.ipynb".
The first task is:
Exercise 2:
Compute a set of different jobs in resultado (this set will store some may items a different jobs are found in resultado). Clue: sets cannot store dupplicate elements!
Can you write Python code for it?


I am working on the file "03-IO/19-database_access.ipynb".
The first task is:
Exercise 3:
Compute the number of occurrences of each job in resultado. This can be accomplished by generating a dictionary with a structure: {job:number_of_occurrences}. Tip: use the result of the previous exercise (a set of different jobs) to create the required dictionary.
Can you write Python code for it?

jobs_count = {}
for i in diff_jobs:
    jobs_count[i] = 0
jobs_count

for i in all_jobs:
    jobs_count[i] += 1
jobs_count

jobs_count = {}
for i in diff_jobs:
    jobs_count[i] = all_jobs.count(i)
jobs_count

jobs_count['Analista']


I am working on the file "03-IO/19-database_access.ipynb".
The first task is:
Exercise 4:
Python dictionaries cannot be sorted (by definition). For this reason, create a list of tuples with a structure (number_of_ocurrences, job), using as input the result of the previous exercise.
Can you write Python code for it?

count_jobs = []
for i in jobs_count:
    count_jobs.append((jobs_count[i], i))
count_jobs


I am working on the file "03-IO/disk.ipynb".
The first task is:
Persistence of objects (serialization) ... in disk
Can you write Python code for it?

import pickle
list = ['red', 'green', 'blue']
pickle.dump(list, open('list.dat','wb'))

list2 = pickle.load(open('list.dat', 'rb'))
list2


I am working on the file "03-IO/disk.ipynb".
The first task is:
Reading the file:
Can you write Python code for it?

# The classic alternative:
file = io.open('/tmp/some_integers_1.txt', 'r')
while True:
    line = file.readline()
    if not line:
        break
#for i in range(3):
    print(int(line))
file.close()

# The pythonic alternarive:
with io.open('/tmp/some_integers_2.txt', 'r') as file:
    for line in file:
        print(int(line))

file.closed


I am working on the file "03-IO/disk.ipynb".
The first task is:
Write some integers:
Can you write Python code for it?

# The legacy way:
file = open('/tmp/some_integers_1.txt', 'w')
file.write('{}\n'.format(1))
file.write('{}\n'.format(2))
file.write('{}\n'.format(3))
file.close()

!cat /tmp/some_integers_1.txt

# The modern (pythonic) alternative:
with io.open('/tmp/some_integers_2.txt', 'w') as file:
    file.write('{}\n'.format(1))
    file.write('{}\n'.format(2))
    file.write('{}\n'.format(3))

file.closed

!cat /tmp/some_integers_2.txt


I am working on the file "03-IO/keyboard.ipynb".
The first task is:

2. Input from the OS prompt
Can you write Python code for it?

!cat argparse_example.py

!python argparse_example.py -h

!python argparse_example.py -i abc


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Batches
Implement get_batches to create batches of input and targets using int_text.  The batches should be a Numpy array with the shape (number of batches, 2, batch size, sequence length). Each batch contains two elements:
- The first element is a single batch of input with the shape [batch size, sequence length]
- The second element is a single batch of targets with the shape [batch size, sequence length]
If you can't fill the last batch with enough data, drop the last batch.
For exmple, get_batches([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 2, 3) would return a Numpy array of the following:
```
[
  # First Batch
  [
    # Batch of Input
    [[ 1  2  3], [ 7  8  9]],
    # Batch of targets
    [[ 2  3  4], [ 8  9 10]]
  ],
# Second Batch
  [
    # Batch of Input
    [[ 4  5  6], [10 11 12]],
    # Batch of targets
    [[ 5  6  7], [11 12 13]]
  ]
]
```
Can you write Python code for it?

def get_batches(int_text, batch_size, seq_length):
    """
    Return batches of input and target
    :param int_text: Text with the words replaced by their ids
    :param batch_size: The size of batch
    :param seq_length: The length of sequence
    :return: Batches as a Numpy array
    """
#     num_batches = int(len(int_text) / (batch_size * seq_length))
    num_batches = len(int_text) // (batch_size * seq_length)  #more pythonic, enforces integer result instead of float
    num_words = num_batches * batch_size * seq_length
    input_data = np.array(int_text[:num_words])
    target_data = np.array(int_text[1:num_words+1])

    input_data = input_data.reshape(batch_size, -1)
    target_data = target_data.reshape(batch_size, -1)

    input_batches = np.split(input_data, num_batches, 1)
    target_batches = np.split(target_data, num_batches, 1)

    return np.array(list(zip(input_batches, target_batches)))

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_batches(get_batches)


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Build the Graph
Build the graph using the neural network you implemented.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from tensorflow.contrib import seq2seq

train_graph = tf.Graph()
with train_graph.as_default():
    vocab_size = len(int_to_vocab)
    input_text, targets, lr = get_inputs()
    input_data_shape = tf.shape(input_text)
    cell, initial_state = get_init_cell(input_data_shape[0], rnn_size)
    logits, final_state = build_nn(cell, rnn_size, input_text, vocab_size, embed_dim)

    # Probabilities for generating words
    probs = tf.nn.softmax(logits, name='probs')

    # Loss function
    cost = seq2seq.sequence_loss(
        logits,
        targets,
        tf.ones([input_data_shape[0], input_data_shape[1]]))

    # Optimizer
    optimizer = tf.train.AdamOptimizer(lr)

    # Gradient Clipping
    gradients = optimizer.compute_gradients(cost)
    capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
    train_op = optimizer.apply_gradients(capped_gradients)


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Build the Neural Network
Apply the functions you implemented above to:
- Apply embedding to input_data using your get_embed(input_data, vocab_size, embed_dim) function.
- Build RNN using cell and your build_rnn(cell, inputs) function.
- Apply a fully connected layer with a linear activation and vocab_size as the number of outputs.
Return the logits and final state in the following tuple (Logits, FinalState)
Can you write Python code for it?

def build_nn(cell, rnn_size, input_data, vocab_size, embed_dim):
    """
    Build part of the neural network
    :param cell: RNN cell
    :param rnn_size: Size of rnns
    :param input_data: Input data
    :param vocab_size: Vocabulary size
    :param embed_dim: Number of embedding dimensions
    :return: Tuple (Logits, FinalState)
    """

    embedding = get_embed(input_data, vocab_size, embed_dim)
    outputs, final_state = build_rnn(cell, embedding)

    weights = tf.truncated_normal_initializer(dtype=tf.float32, stddev=0.1)0
    biases = tf.zeros_initializer()

    logits = tf.contrib.layers.fully_connected(outputs, vocab_size, None, None, weights_initializer=weights, biases_initializer=biases)
    return logits, final_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_nn(build_nn)


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Build the Neural Network
You'll build the components necessary to build a RNN by implementing the following functions below:
- get_inputs
- get_init_cell
- get_embed
- build_rnn
- build_nn
- get_batches
Check the Version of TensorFlow and Access to GPU
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Check Point
This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import numpy as np
import problem_unittests as tests

int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Checkpoint
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()
seq_length, load_dir = helper.load_params()


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Choose Word
Implement the pick_word() function to select the next word using probabilities.
Can you write Python code for it?

def pick_word(probabilities, int_to_vocab):
    """
    Pick the next word in the generated text
    :param probabilities: Probabilites of the next word
    :param int_to_vocab: Dictionary of word ids as the keys and words as the values
    :return: String of the predicted word
    """
    # strategy one: random choice from all
    idx = np.random.choice(len(probabilities), 30, p=probabilities)[0]
    picked_word = int_to_vocab[idx]

#     # strategy two: max probability every time
#     idx_argmax = np.argmax(probabilities)
#     picked_word = int_to_vocab[idx_argmax]
    return picked_word

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_pick_word(pick_word)


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Explore the Data
Play around with view_sentence_range to view different parts of the data.
Can you write Python code for it?

view_sentence_range = (0, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in text.split()})))
scenes = text.split('\n\n')
print('Number of scenes: {}'.format(len(scenes)))
sentence_count_scene = [scene.count('\n') for scene in scenes]
print('Average number of sentences in each scene: {}'.format(np.average(sentence_count_scene)))

sentences = [sentence for scene in scenes for sentence in scene.split('\n')]
print('Number of lines: {}'.format(len(sentences)))
word_count_sentence = [len(sentence.split()) for sentence in sentences]
print('Average number of words in each line: {}'.format(np.average(word_count_sentence)))

print()
print('The sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Generate TV Script
This will generate the TV script for you.  Set gen_length to the length of TV script you want to generate.
Can you write Python code for it?

gen_length = 250
# homer_simpson, moe_szyslak, or Barney_Gumble
prime_word = 'homer_simpson'

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_dir + '.meta')
    loader.restore(sess, load_dir)

    # Get Tensors from loaded model
    input_text, initial_state, final_state, probs = get_tensors(loaded_graph)

    # Sentences generation setup
    gen_sentences = [prime_word + ':']
    prev_state = sess.run(initial_state, {input_text: np.array([[1]])})

    # Generate sentences
    for n in range(gen_length):
        # Dynamic Input
        dyn_input = [[vocab_to_int[word] for word in gen_sentences[-seq_length:]]]
        dyn_seq_length = len(dyn_input[0])

        # Get Prediction
        probabilities, prev_state = sess.run(
            [probs, final_state],
            {input_text: dyn_input, initial_state: prev_state})

        pred_word = pick_word(probabilities[dyn_seq_length-1], int_to_vocab)

        gen_sentences.append(pred_word)

    # Remove tokens
    tv_script = ' '.join(gen_sentences)
    for key, token in token_dict.items():
        ending = ' ' if key in ['\n', '(', '"'] else ''
        tv_script = tv_script.replace(' ' + token.lower(), key)
    tv_script = tv_script.replace('\n ', '\n')
    tv_script = tv_script.replace('( ', '(')

    print(tv_script)


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Implement Generate Functions
Get Tensors
Get tensors from loaded_graph using the function get_tensor_by_name().  Get the tensors using the following names:
- "input:0"
- "initial_state:0"
- "final_state:0"
- "probs:0"
Return the tensors in the following tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
Can you write Python code for it?

def get_tensors(loaded_graph):
    """
    Get input, initial state, final state, and probabilities tensor from <loaded_graph>
    :param loaded_graph: TensorFlow graph loaded from file
    :return: Tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
    """
    input_tensor = loaded_graph.get_tensor_by_name('input:0')
    initial_state_tensor = loaded_graph.get_tensor_by_name('initial_state:0')
    final_state_tensor = loaded_graph.get_tensor_by_name('final_state:0')
    probs_tensor = loaded_graph.get_tensor_by_name('probs:0')
    return input_tensor, initial_state_tensor, final_state_tensor, probs_tensor

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_tensors(get_tensors)


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Implement Preprocessing Functions
The first thing to do to any dataset is preprocessing.  Implement the following preprocessing functions below:
- Lookup Table
- Tokenize Punctuation
Lookup Table
To create a word embedding, you first need to transform the words to ids.  In this function, create two dictionaries:
- Dictionary to go from the words to an id, we'll call vocab_to_int
- Dictionary to go from the id to word, we'll call int_to_vocab
Return these dictionaries in the following tuple (vocab_to_int, int_to_vocab)
Can you write Python code for it?

import numpy as np
import problem_unittests as tests

def create_lookup_tables(text):
    """
    Create lookup tables for vocabulary
    :param text: The text of tv scripts split into words
    :return: A tuple of dicts (vocab_to_int, int_to_vocab)
    """
    vocab = set(text)
    vocab_to_int = {c: i for i, c in enumerate(vocab)}
    int_to_vocab = dict(enumerate(vocab))
    chars = np.array([vocab_to_int[c] for c in text], dtype=np.int32)
#     print(int_to_vocab)
    return vocab_to_int, int_to_vocab

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_create_lookup_tables(create_lookup_tables)


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Preprocess all the data and save it
Running the code cell below will preprocess all the data and save it to file.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables)


I am working on the file "03-InitRNN/dlnd_tv_script_generation.ipynb".
The first task is:
Word Embedding
Apply embedding to input_data using TensorFlow.  Return the embedded sequence.
Can you write Python code for it?

# Question 1. What difference does it make if I use truncated_normal v/s random_uniform?
# Question 2. What difference does it make if I use the embed_sequence from tf.contrib.layers vs above approach?
# I could not observe difference with respect to the model and hence the questions

def get_embed(input_data, vocab_size, embed_dim):
    """
    Create embedding for <input_data>.
    :param input_data: TF placeholder for text input.
    :param vocab_size: Number of words in vocabulary.
    :param embed_dim: Number of embedding dimensions
    :return: Embedded input.
    """
#     embedding = tf.Variable(tf.truncated_normal((vocab_size, embed_dim), -1, 1))
#     embedding = tf.nn.embedding_lookup(embedding, input_data)
    embedding = tf.contrib.layers.embed_sequence(ids=input_data, vocab_size=vocab_size, embed_dim=embed_dim)
    return embedding

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_embed(get_embed)


I am working on the file "03-Introduction-to-Radiometry.ipynb".
The first task is:
Electromagnetic radiation can be modeled as a number of different phenomena: rays, electromagnetic waves, wavefronts, or particles. All of these models are mathematically related. The appropriate model to use depends on the task at hand. Either the electromagnetic wave model(developed by Maxwell) or the particle model (developed by Einstein) are used when most appropriate. The part of the electromagnetic spectrum normally considered in optical radiometry is as follows:
Can you write Python code for it?


I am working on the file "03-Introduction-to-Radiometry.ipynb".
The first task is:
Extend the above flux-transfer equation for multi-spectral calculations by noting that over a spectral width $d\lambda$ the radiance is given by $L = L_\lambda d\lambda$:
$$
d^3 \Phi_\lambda=
\frac{L_{01\lambda}\,dA_0\;\cos\theta_0\,dA_1\;\cos\theta_1
\;\tau_{01}\,d\lambda}{R_{01}^2},
$$
where $d^3\Phi_\lambda$ is the total flux in [W] or [q/s] flowing in a spectral width $d\lambda$ at wavelength $\lambda$, from a radiator with radiance $L_{0\lambda}$ with units [W/(m$^2$ $\cdot$sr$\cdot$ $\mu$m)] and projected surface area $dA_0\cos\theta_0$, through a receiver with projected surface area $dA_1\cos\theta_1$ at a distance $R_{01}$, with a transmittance of $\tau_{01}$ between the two surfaces. The transmittance $\tau_{01}$ now includes all of the spectral variables in the path between the source and the receiver.
To determine the total flux flowing from elemental area $dA_0$ through $dA_1$ over a wide spectral width, divide the wide spectral band into a large number $N$ of narrow widths $\Delta\lambda$ at wavelengths $\lambda_n$ and add the flux for all of these narrow bandwidths together as follows:
$$
d^2 \Phi=
\sum_{n=0}^{N}
\left(
\frac{L_{01\lambda_n}
\,dA_{0}\,\cos\theta_0\,
\,dA_{1}\,\cos\theta_1\,
\tau_{01\lambda_n}
\Delta\lambda}{R_{01}^2}
\right).
$$
By the Riemann--Stieltjes theorem in reverse, if now $\Delta\lambda\rightarrow 0$ and $N\rightarrow\infty$, the summation becomes the integral
$$
d^2 \Phi=
\int_{\lambda_1}^{\lambda_2}
\frac{L_{01\lambda}
\,dA_{0}\,\cos\theta_0\,
\,dA_{1}\,\cos\theta_1 \,\tau_{01\lambda}d\lambda}{R_{01}^2}\ .
$$
This equation describes the total flux at all wavelengths in the spectral range $\lambda_1$ to $\lambda_2$ passing
through the system. This equation is developed further in my book.
Conclusion
The flux transfer between any two arbitrary surfaces, over any spectral band can be calculated by
$$
\Phi=
\int_{A_0}
\int_{A_1}
\int_{\lambda_1}^{\lambda_2}
\frac{L_{01\lambda}
\,dA_{0}\,\cos\theta_0\,
\,dA_{1}\,\cos\theta_1 \,\tau_{01\lambda}d\lambda}{R_{01}^2}\ .
$$
In...
Can you write Python code for it?

try:
    import pyradi.ryutils as ryutils
    print(ryutils.VersionInformation('matplotlib,numpy,pyradi,scipy,pandas'))
except:
    print("pyradi.ryutils not found")


I am working on the file "03-Introduction-to-Radiometry.ipynb".
The first task is:
Lambertian radiators
A Lambertian source is, by definition, one whose radiance is completely independent of viewing angle.  Many (but not all) rough and natural surfaces produce radiation whose radiance is approximately independent of the angle of observation. These surfaces generally have a rough texture at microscopic scales. Planck-law blackbody radiators are also Lambertian sources (see my book). Any Lambertian radiator is completely described by its scalar radiance magnitude only, with no angular dependence in radiance.
The relationship between the exitance and radiance for such a Lambertian surface can be easily derived. If the  flux radiated from a Lambertian surface $\Phi$ [W] is known, it is a simple matter to calculate the exitance $M=\Phi/A$ [W/m$^2$], where $A$ is the radiating surface area.  The  exitance of a Lambertian radiator is related to radiance by the projected solid angle of $\pi$ sr,  not  the geometric solid angle of $2\pi$ sr as one might expect. The details are given in my book.
Conservation of radiance
Radiance is conserved for flux from a Lambertian surface propagation through a lossless optical
medium. Consider the construction below: two elemental areas $dA_0$ and $dA_1$ are separated by a distance $R_{01}$, with the angles between the normal vector of each surface and the line of sight given by $\theta_0$ and $\theta_1$. A total flux of $d^2\Phi$ is flowing through both the surfaces.  It can be shown (see my book) that for a Lambertian radiator the radiance in an arbitrary $dA_n$ is the same as the radiance in $dA_1$.
As light propagates through mediums with different refractive indices $n$  such as air, water, glass, etc., the entity called basic radiance, defined by $L/n^2$, is invariant. It can be shown that for light propagating from a medium with refractive index $n_1$ to a medium with refractive index $n_2$, the basic radiance is conserved:
$$
\frac{L_1}{n_1^2}=\frac{L_2}{n_2^2}.
$$
Can you write Python code for it?


I am working on the file "03-Introduction-to-Radiometry.ipynb".
The first task is:
Multi-spectral flux transfer
The optical power leaving a source undergoes a succession of  scaling or 'spectral filtering' processes as the flux propagates through the system, as shown below. This filtering varies with wavelength.
Examples of such filters are source emissivity, atmospheric transmittance, optical filter transmittance, and detector responsivity.  The multi-spectral filter approach described here is conceptually simple but fundamental to the calculation of radiometric flux.
Can you write Python code for it?


I am working on the file "03-Introduction-to-Radiometry.ipynb".
The first task is:
Spectral quantities
See notebook 4 in this series, Introduction to computational radiometry with pyradi, for a detailed description of spectral quantities.
Three  spectral domains are commonly used: wavelength $\lambda$ in [m], frequency $\nu$ in [Hz], and  wavenumber $\tilde{\nu}$ in [cm$^{-1}$] (the number of waves that will fit into a 1-cm length).
Spectral quantities indicate an amount of the quantity within a small spectral width $d\lambda$ around the value of $\lambda$: it is a spectral density. Spectral density quantity symbols are subscripted with a $\lambda$ or $\nu$, i.e., $L_\lambda$ or $L_\nu$. The dimensional units of a spectral  density quantity are indicated as [$\mu$m$^{-1}$] or [(cm$^{-1})^{-1}$], i.e., [W/(m$^2$ $\cdot$sr$\cdot$ $\mu$m)].
The relationship between the wavelength and wavenumber spectral domains is $\tilde{\nu}=10^4/\lambda$ , where $\lambda$ is in units of $\mu$m. The  conversion of a spectral density quantity such as [W/(m$^2$ $\cdot$sr$\cdot$cm$^{-1}$)] requires the derivative, %$d{\tilde{\nu}}=-\frac{10^4}{\lambda^2}d\lambda=-\frac{{\tilde{\nu}}^2}{10^4}d\lambda$.
$d{\tilde{\nu}}=-10^4d\lambda /\lambda^2=-\tilde{\nu}^2d\lambda/10^4$.
The derivative relationship converts between the spectral widths, and hence the spectral densities, in the two respective domains.
The conversion from a wavelength spectral density quantity to a wavenumber spectral density quantity is
$d{}L_{\tilde{\nu}}=d{}L_\lambda \lambda^2/10^4=d{} L_\lambda 10^4/\tilde{\nu}^2$.
Spectral quantities denote the amount in a small spectral width $d\lambda$ around a wavelength $\lambda$. It follows that the total  quantity over a spectral range can be determined by  integration (summation) over the spectral range of interest:
$$
L=\int_{\lambda_1}^{\lambda_2}L_\lambda d\lambda.
$$
The above integral satisfies the requirements of dimensional analysis (see my book) because the units of $L_\lambda$ are [W/(m$^2$ $\cdot$sr$\cdot$ $\mu$m)], whereas $d\lambda$ has the units of [$\mu$m], and $L$ has units of...
Can you write Python code for it?


I am working on the file "03-Introduction-to-Radiometry.ipynb".
The first task is:
The photon is a massless elementary particle and acts as the energy carrier for the electromagnetic wave.
Photon particles have discrete energy quanta proportional to the frequency of the electromagnetic energy, $Q = h\nu = hc/\lambda$, where $h$ is Planck's constant.
Definitions
The following figure (expanded from Pinson) and table defines the key radiometry units.  The difference operator '$d$' is used to denote 'a small quantity of ...'. This 'small quantity' of one variable is almost always related to a 'small quantity' of another variable in some physical dependency. For example, irradiance is defined as $E=d\Phi/dA$, which means that a small amount of flux $d\Phi$ impinges on a small area $dA$, resulting in an irradiance of $E$.  'Small' is defined as the extent or domain over which the quantity, or any of its dependent quantities, does not vary significantly. Because any finite-sized quantity varies over a finite-sized domain,  the $d$ operation is only valid over an infinitely small domain $dA=\lim_{\Delta A \to 0}\Delta A$.  The difference operator, written in the form of a differential such as $E=d\Phi/dA$, is not primarily meant to mean differentiation in the mathematical sense. Rather, it is used to indicate something that can be integrated (or summed).
In practice, it is impossible to consider infinitely many, infinitely small domains. Following the reductionist approach, any real system can, however, be assembled as the sum of a set of these small domains, by integration over the physical domain as in $A=\int dA$. Hence, the 'small-quantity' approach proves very useful to describe and understand the problem, whereas the real-world solution can be obtained as the sum of a set of such small quantities. In almost all of the cases in this notebook, it is implied that such 'small-quantity' domains will be integrated (or summed) over the (larger) problem domain.
Photon rates are measured in quanta per second.
The 'second' is an SI unit, whereas quanta is a unitless count: the number of photons. Photon...
Can you write Python code for it?


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
Below we solve the $50 \times 50$ stiffness matrix problem with the conjugate gradient method. The iterations terminate once the residual drops below a specified threshold. The solver prints at each iteration the relative error in the solution in the $\boldsymbol{A}$-norm and the relative residual in the $l_{2}$ norm.
Can you write Python code for it?


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
Calling the function for a $50 \times 50$ stiffness matrix:
Can you write Python code for it?


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
Compute eigenvalues directly:
Can you write Python code for it?

evals = np.linalg.eigvals(A)
print("Exact evals: {}".format(evals))


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
Computing the largest and second largest eigenvalues for $5 \times 5$ and $20 \times 20$ matrices, we see that they get closer as the matrix size increases:
Can you write Python code for it?

# Check largest and second eigenvalues of 5x5 stiffness matrix
print("Stiffness matrix 5 x 5:")
max_eigenpair(create_stiffness_matrix(5), print_values=True);

# Check largest and second eigenvalues of 10x10 stiffness matrix
print("Stiffness matrix 20 x 20:")
max_eigenpair(create_stiffness_matrix(20), print_values=True);


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
Matrix generators
We will test some methods with two common matrices - the stiffness matrix and the mass matrix. Both are symmetric positive definite (SPD), but otherwise have very different properties. One matrix will be easy to solve using iterative methods, and one is much more challenging. To work with these matrices, we implement functions to help generate matrices of a given size.
Stiffness matrix
The stiffness matrix (this matrix comes up in the module 3D7) tends to arise in elliptic differential equations. We will start with a function to build a $n \times n$ stiffness matrix:
Can you write Python code for it?

def create_stiffness_matrix(n):
    "Create a stiffness matrix of size n x n"

    # Create a zero matrix
    A = np.zeros((n + 1, n + 1))

    # Add components of small matrix k to A
    k = np.array([[1.0, -1.0], [-1.0, 1.0]])
    for i in range(len(A) - 1):
        A[i:i+2, i:i+2] += k

    # We will remove the last row and column of the matrix. This will
    # make the matrix invetible (SPD)
    A = np.delete(A, -1, 0)
    A = np.delete(A, -1, 1)
    return A


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
Now make the matrix Hermitian and perform power iterations:
Can you write Python code for it?

A = 0.5*(A + A.T.conjugate())
print(A)

evals = np.linalg.eigvals(A)
print("Exact evals: {}".format(evals))
x0 = np.ones(3) + 1j*np.ones(3)
for i in range(10):
    x0 = A.dot(x0)
    x0/np.linalg.norm(x0, 1)

    eig_est = np.vdot(x0, A.dot(x0)) / np.vdot(x0, x0)
    print("Eigen estimate: {}".format(eig_est))
    print("  Error: {}".format(np.abs(eig_est - evals[0])))


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
Perform power iterartions for the non-symmetric case:
Can you write Python code for it?

x0 = np.ones(3) + 1j*np.ones(3)
print(x0)
for i in range(10):
    x0 = A.dot(x0)
    x0/np.linalg.norm(x0, 1)

    eig_est = np.vdot(x0, A.dot(x0)) / np.vdot(x0, x0)
    print("Eigen estimate: {}".format(eig_est))
    print("  Error: {}".format(np.abs(eig_est - evals[0])))


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
Power iteration with the Rayleigh quotient
Define a complex matrix:
Can you write Python code for it?


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
The largest eigenvalue is less that one (only just!), so we can expect the Jacobi method to converge. However, it is close to one so we expect the convergence to be slow.
Experiment: compare the spectral radius for different size matrices.
Let's try solving $\boldsymbol{A} \boldsymbol{x} = \boldsymbol{b}$ with Jacobi's method using 15 iterations and with $\boldsymbol{b} = \boldsymbol{1}$:
Can you write Python code for it?

I = np.identity(A.shape[0])
b = np.ones(A.shape[1])
x = np.zeros(A.shape[1])
r0_norm = np.linalg.norm(b - A.dot(x), 2)
for k in range(15):
    x = Dinv.dot(b) + (I - Dinv.dot(A)).dot(x)
    r = b - A.dot(x)
    print("Step: {}, relative norm of residual (l2): {}".format(k + 1, np.linalg.norm(r, 2)/r0_norm))


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
The spectral radius is considerably small than one and much smaller than it was for the stiffness matrix, which is then evident in the much faster convergence.
Gauss-Seidel
For the Gauss-Seidel, method, $\boldsymbol{N}$ is the lower triangular part of $\boldsymbol{A}$ and $\boldsymbol{P}$ is the strictly upper triangular part of $\boldsymbol{A}$ ($\boldsymbol{P}$ is zero on the diagonal). Solving  $\boldsymbol{N} \boldsymbol{y} = \boldsymbol{f}$ is then analogous to the forward substitution step in a LU solver.
From the general expression for a stationary method, we multiply both sides by $\boldsymbol{N}$:
$$
\boldsymbol{N} \boldsymbol{x}{k+1} = \boldsymbol{b} + \boldsymbol{P}\boldsymbol{x}{k}
$$
We will solve our problem using this formulation. We create a $50 \times 50$ stiffness matrix, and then form the
$\boldsymbol{N}$ and $\boldsymbol{P}$ matrices:
Can you write Python code for it?

# Create etiffness matrix
A = create_stiffness_matrix(50)

# Form lower-triangular part of A
N = np.tril(A)

# Form P
P = N - A


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
The spectral radius is only half that for the Jacobi method, and this is manifest in the faster observed convergence.
Applications of stationary methods
Stationary methods are of limited use of their own as they tend to converge very slowly. For matrices that they do work well for, e.g. mass matrix, there are other iterative methods that are faster.
Stationary methods are however very useful in combination with other methods, e.g. as preconditioners in more sophisticated iterative methods and as 'smoothers' in multigrid methods.
Conjugate gradient method for $\boldsymbol{A} \boldsymbol{x} = \boldsymbol{b}$
We now look at one of the most important algorithms for solving $\boldsymbol{A} \boldsymbol{x} = \boldsymbol{b}$, the conjugate gradient (CG) method. The CG method is applicable to symmetric positive-definite (SPD) matrices.
The CG method is technically a direct method since in exact arithmetic is solves an $n \times n$ system in $k$ iterations, where $k$ is the number of distinct eigenvalues of $\boldsymbol{A}$. In the worst case (all eigenvalues are distinct) it requires $n$ iterations.
In practice, the CG method is used as an iterative method to find approximate solutions. If solving an $n \times n$ system required $n$ steps it would generally not be competitive with other methods, and round-off error can spoil the party.
The error in the CG method decreases monotonically in the energy norm, i.e. if $\boldsymbol{x}$ is the exact solution then:
$$
(\boldsymbol{x} - \boldsymbol{x}{k+1})^{T} \boldsymbol{A} (\boldsymbol{x} - \boldsymbol{x}{k+1}) <
(\boldsymbol{x} - \boldsymbol{x}{k})^{T} \boldsymbol{A} (\boldsymbol{x} - \boldsymbol{x}{k})
$$
This implies that if we stop iterating before the residual is (almost) zero, we will have a better solution than we started with.
Below we create below a function for the conjugate gradient (CG) method. It will also solve the system directly so we can compare the error at each iteration (this would of course not be done in practice, but we do it here to illustrate...
Can you write Python code for it?

def cg(A):
    "Conjugate gradient solver"

    # RHS vector
    b = np.ones(A.shape[1])

    # Initial guess (zero vector)
    x0 = np.zeros(A.shape[1])

    # Compute exact solution (to use in computing error at each iterate)
    x_exact = np.linalg.solve(A, b)
    e = x_exact - x0
    e0_norm = np.sqrt(e.dot(A.dot(e)))
    print("Initial error (A-norm): {}".format(e.dot(A.dot(e))))

    # Convergence tolerance to exit solver
    tolerance = 1.0e-9

    # Create starting vectors
    r0 = b - A.dot(x0)
    r0_norm = np.linalg.norm(r0)
    p0 = r0.copy()

    # Start iterations
    for k in range(200):
        print("Step: {}".format(k))

        alpha = r0.dot(r0)/(p0.dot(A.dot(p0)))
        x1 = x0 + alpha*p0
        r1 = r0 - alpha*A.dot(p0)

        # Compute error in x (this is for studying the algorithm, and of
        # course would not be done in practice)
        e = x_exact - x1
        e_norm = np.sqrt(e.dot(A.dot(e)))
        print("  Relative error in x (A-norm): {}".format(e_norm/e0_norm))

        # Compute norm of residual and check for converge
        r_norm = np.linalg.norm(r1)
        print("  Relative residual (l2-norm):  {}".format(r_norm/r0_norm))
        if r_norm < tolerance:
            break

        beta = r1.dot(r1)/r0.dot(r0)
        p1 = r1 + beta*p0

        # Update for next step
        p0, r0, x0 = p1, r1, x1


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
The spectral radius is very slightly smaller that for the Jacobi case, but still very close to one so we cannot expect good convergence.
Now the solver. We write a function to perform 15 Gauss-Seidel iterations, using SciPy for the forward substitution step involving $\boldsymbol{N}$:
Can you write Python code for it?

import scipy.linalg as LA

def gauss_seidel_solver(A):
    "Compute spectral radius and perform 15 Gauss-Seidel iterations"
    N = np.tril(A)
    P = N - A

    evals = np.linalg.eigvals(np.linalg.inv(N).dot(P))
    print("Largest eigenvalue (rho) is: {}".format(np.max(abs(evals))))

    b = np.ones(A.shape[1])
    x = np.zeros(A.shape[1])
    r0_norm = np.linalg.norm(b - A.dot(x), 2)
    for k in range(15):
        c = b + P.dot(x)
        x = LA.solve_triangular(N, c, lower=True)

        # Compute residual to monitor convergence
        r = b - A.dot(x)
        print("Step: {}, relative norm of residual (l2): {}".format(k + 1, np.linalg.norm(r, 2)/r0_norm))


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
We can see from the decrease in the residual that the Gauss-Seidel method converges somewhat faster than the Jacobi method, but it is still far too slow to be of any practical use on its own.
As for the Jacobi method, we will collect the above steps in sude a function and test for the $50\times 50$ mass matrix.
Can you write Python code for it?

def gauss_seidel_solver(A):
    "Compute spectral radius and perform 15 Gauss-Seidel iterations"
    N = np.tril(A)
    P = N - A

    evals = np.linalg.eigvals(np.linalg.inv(N).dot(P))
    print("Largest eigenvalue (rho) is: {}".format(np.max(abs(evals))))

    b = np.ones(A.shape[1])
    x = np.zeros(A.shape[1])
    r0_norm = np.linalg.norm(b - A.dot(x), 2)
    for k in range(15):
        c = b + P.dot(x)
        x = LA.solve_triangular(N, c, lower=True)

        # Compute residual to monitor convergence
        r = b - A.dot(x)
        print("  Step: {}, relative norm of residual (l2): {}".format(k + 1, np.linalg.norm(r, 2)/r0_norm))

gauss_seidel_solver(create_mass_matrix(50))


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
We can see that the residual is decreasing, but extremely slowly. This is to be expected because $\rho(\boldsymbol{N}^{-1} \boldsymbol{P})$ is very close to one.
We now try with the mass matrix. For this we collect the above steps in a function, and then call the function for a $50 \times 50$ mass matrix.
Can you write Python code for it?

def jacobi_solver(A):
    N = np.diag(np.diag(A))
    Dinv = np.diag(1.0/np.diag(A))
    P = N - A
    evals = np.linalg.eigvals(Dinv.dot(P))
    print("Spectral radius (rho) is: {}".format(np.max(abs(evals))))

    I = np.identity(A.shape[0])

    b = np.ones(A.shape[1])
    x = np.zeros(A.shape[1])
    r0_norm = np.linalg.norm(b - A.dot(x), 2)
    for k in range(15):
        x = Dinv.dot(b) + (I - Dinv.dot(A)).dot(x)
        r = b - A.dot(x)
        print("  Step: {}, relative norm of residual (l2): {}".format(k + 1, np.linalg.norm(r, 2)/r0_norm))

jacobi_solver(create_mass_matrix(50) )


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
We now check the spectral radius for stiffness matrix and the Gauss-Seidel method:
Can you write Python code for it?

evals = np.linalg.eigvals(np.linalg.inv(N).dot(P))
print("Largest eigenvalue (rho) is: {}".format(np.max(abs(evals))))


I am working on the file "03-IterativeMethods.ipynb".
The first task is:
We see that, as expected, the CG method for an $n \times n$ matrix solves the problem in $n$ iterations. However if, if we were satisfied with a less accurate solution we could have terminated the iterations sooner.
Note that the reduction of the error in $\boldsymbol{A}$-norm ($\sqrt{\boldsymbol{e}^{T}\boldsymbol{A}\boldsymbol{e}}$) decreases monotonically. The $l_{2}$-norm of the residual $\boldsymbol{r}{k} = \boldsymbol{b} - \boldsymbol{A}\boldsymbol{x}{k}$ increases in the first step (the relative norm of the residual is greater than one), and then starts to decrease. The monotonic convergence in the $\boldsymbol{A}$-norm is what we expect from analysis of the CG method.
We now test for the $50 \times 50$ mass matrix:
Can you write Python code for it?


I am working on the file "03-JupyterandIPython.ipynb".
The first task is:
IPython
Can you write Python code for it?


I am working on the file "03-ML/01-ML.ipynb".
The first task is:
1.1.1 Preparing the Features
Before moving onto Machine Learning properly, let us visualise the amount of votes into the 3 possible categories (1 : yes, 2 : no , 3 : absention).
Can you write Python code for it?


I am working on the file "03-ML/01-ML.ipynb".
The first task is:
1.1.2 Classification of our data
The classifier we will  use is the Random Forest. In order to evaluate the performance of our results, we will use several tools, which will help us to understand better the results we obtain.

The cross_validation module of scikit-learn allows us to test the performance of our classification. The cross_val_score method returns the percentage of accuracy of our classification (average of the testing error of each iteration of the cross-validation. The cv field allows us to chose the number of folds of cross-validation we want to perform (e.g. cv=5 -> 5-fold cross-validation).
The F1 score takes into account the false positives and the false negatives in the process of outputting the score. Hence, a model with a high prediction accuracy can get very poor results in the F1-metric if for instance it outputs everything to an output which is dominant in the population (cf. the examples "Everybody has cancer" in the lecture 07 of the course).
The confusion matrix plots the detail of the classification, and allows us to visualise the false positives, false negatives. We can compute the F1-score from this matrix.
The feature importances allows us to see which of the features are the most significant for the classification.

If we turn now to the coding, we have several functions implemented in the ML-helpers.py file, which are as follows :
- The cross_validation function does perform the cv_param-fold cross-validation, and outputs the cross-validation result, along with F1-score and the confusion matrix, in order for us to understand the shape of our results. We perform by default a 20-fold cross validation, as we want a result as stable as possible.
- The plot_feature_importances function does rank each feature of X accordingly to the role it plays into the classification of the data we give it. This allows us to see whether a key subset of our features would turn out to be outstandingly better than the rest at determining the outcome of the vote of the deputee
- The function...
Can you write Python code for it?

estimator = RandomForestClassifier()

title = "Learning Curves (Random Forest Classifier with 2 classes)"
plot_learning_curve(estimator,X,Y,title,20)


I am working on the file "03-ML/01-ML.ipynb".
The first task is:
1.1.3 Results
Having found the depth at wich our tree does not overfit, we want to focus on understanding the results we get. We will see, given the features we have, whether our algorithm is able to classify correctly.
N.B. The less deep the tree, the dumber the classifier.
Can you write Python code for it?

Y_predicted = cross_validation(X, Y, cv_param=20, max_depth=10)
features_2 = plot_feature_importances(X,Y,pred_field, max_depth=10)


I am working on the file "03-ML/01-ML.ipynb".
The first task is:
Importing the quite heavy DataFrame with the voting fields and the results. We drop a useless column and create a Name field, which will contain both the first and the last name of a person, so we can then create a model for each unique deputee at the parliament.
Can you write Python code for it?

path = '../datas/nlp_results/'
voting_df = pd.read_csv(path+'voting_with_topics_sentiment.csv')
print('Entries in the DataFrame',voting_df.shape)

#Dropping the useless column
#voting_df = voting_df.drop('Unnamed: 0',1)

#Putting numerical values into the columns that should have numerical values

num_cols = ['BillTitle', 'BusinessTitle','text','text_eng','FirstName','LastName']
voting = voting_df.drop(num_cols,axis=1).apply(pd.to_numeric)
voting['text'] = voting_df.text
#Inserting the full name at the second position
voting.insert(1,'Name', voting_df['FirstName'] + ' ' + voting_df['LastName'])

voting.head(10)


I am working on the file "03-ML/01-ML.ipynb".
The first task is:
The first reduction of our DataFrame is to suppress all the entries of the Decision field, which contain either a 4, a 5, a 6 or a 7. It basically means that the person did not take part to the voting, and that is hence not useful to our purpose.
Can you write Python code for it?

voting_df = voting[((voting.Decision != 4) & (voting.Decision != 5) & (voting.Decision != 6) & (voting.Decision != 7))]
print(voting_df.shape)
#print('Top number of entries in the df :\n', voting_df.Name.value_counts())


I am working on the file "03-ML/01-ML.ipynb".
The first task is:
The plot describes the classification into binary output. The default RandomForestClassifier method clearly overfits our data, this is notably due to the fact of having an unfixed depth for the depth of the tree. This is why we will iterate over different depth and fix it to a value that yields a good result, and the fact of having a max depth will mitigate overfitting.
We now compute the result of the cross validation given a certain depth, varying from 1 to 20 and plot the result
Can you write Python code for it?

cv_score = np.zeros(20)
tr_score = np.zeros(20)

cv_param = 20
for i in range(1,21):
    forest = RandomForestClassifier(max_depth = i)
    cv_score[i-1] = cross_val_score(forest,X,Y,cv = cv_param).mean()
    forest.fit(X,Y)
    tr_score[i-1] = forest.score(X,Y)

plot_fig(cv_score,tr_score,
         "Cross validation score against the depth of the random forest",
         "Max depth of the random forest","Cross validation score")


I am working on the file "03-ML/01-ML.ipynb".
The first task is:
We format the output DataFrame in a useful way, storing the name of the deputee who voted, their actual Decision, our Predicted Decision and the fields that were used in the learning.
Can you write Python code for it?

df_out = df_deputee[np.r_[['Decision'],pred_field]]
df_out.insert(1,'Predicted Decision',Y_predicted + 1)

#Optional line to round the probabilities to 3 decimals (will not sum to one, but simpler to look at it.)
df_out = df_out.apply(np.around,decimals=3)

if not os.path.exists("../datas/treated_data/Voting_prediction"):
    os.makedirs("../datas/treated_data/Voting_prediction")
df_out.to_csv('../datas/treated_data/Voting_prediction/'+'prediction_'+
              df_deputee['Name'].unique()[0].lower().replace(' ','_')+'.csv',index=False)

df_out.head(100)


I am working on the file "03-NLP/POS.ipynb".
The first task is:
'A' and 'The' are articles or determiners (DT), 'is' a verb 3rd person (VBZ), 'always' an adverb (RB) and 'faster' is a comparative (RBR). Still everything correct
Can you write Python code for it?

print(getSentencePOS("I eat pizza and pasta but I do not eat meat"))


I am working on the file "03-NLP/POS.ipynb".
The first task is:
As you can see, the training_corpus is a list with all words extracted from English articles, together with their POS tag.
Almost one million of them!
Can you write Python code for it?

# load in the test corpus
with open("../datasets/WSJ_24.pos", 'r') as f:
    testing_corpus = f.readlines()  # list

print("A sample of the testing corpus")
print(testing_corpus[0:10])

len(testing_corpus)


I am working on the file "03-NLP/POS.ipynb".
The first task is:
Ok, back to dogs.
Here you can see that 'I' is identified as a Pronoun (PRP), 'have' as a Verb conjugated (VBP) and 'big' is an adjective (JJ). Everything correct.
Can you write Python code for it?

print(getSentencePOS("A lion is always faster than the turtle"))


I am working on the file "03-NLP/POS.ipynb".
The first task is:
Parts-of-speech tagging
We will start with the simplest possible parts-of-speech tagger and we will build up to the state of the art.
Training
In this section, we will find the words that are not ambiguous.
- For example, the word is is a verb and it is not ambiguous.
- In the WSJ corpus, $86$% of the token are unambiguous (meaning they have only one tag)
- About $14\%$ are ambiguous (meaning that they have more than one tag)
Before we start predicting the tags of each word, we will need to compute a few dictionaries that will help to generate the tables.
Emission counts
The first dictionary to  compute (and the one that will be used initially) is the emissionCounts dictionary. This dictionary will be used to compute:
$$P(w_i|t_i)\tag{1}$$
In other words, we will use it to compute the probability of a word given its tag.
In order  to compute this probability, we will create an emissionCounts dictionary where
- The keys are (tag, word)
- The values are the number of times that pair showed up in the training set.
Transition counts
The second dictionary is the transitionCounts dictionary which computes the number of times each tag happened next to another tag.
This dictionary will be used to compute:
$$P(t_i |t_{i-1}) \tag{2}$$
This is the probability of a tag at position $i$ given the tag at position $i-1$.
In order to compute it the  transitionCounts dictionary has:
- the keys are (prev_tag, tag)
- the values are the number of times those two tags appeared in that order.
To calculate the transition probabilities, we actually only use the parts of speech tags from the
training corpus. So to calculate the probability of a speech tag transitioning to another  one, we first have to count the occurrences of that tag combination in the corpus, which is N. The number of all tag pairs starting with the first tag is M, for this corpus at least. So N out of M tag sequences in the training corpus starts with the first parts of speech tag. In other words, that is the transition probability for the second tag...
Can you write Python code for it?

import string

# Punctuation characters
punct = set(string.punctuation)

# Morphology rules used to assign unknown word tokens
noun_suffix = ["action", "age", "ance", "cy", "dom", "ee", "ence", "er", "hood", "ion", "ism", "ist", "ity", "ling", "ment", "ness", "or", "ry", "scape", "ship", "ty"]
verb_suffix = ["ate", "ify", "ise", "ize"]
adj_suffix = ["able", "ese", "ful", "i", "ian", "ible", "ic", "ish", "ive", "less", "ly", "ous"]
adv_suffix = ["ward", "wards", "wise"]

def assign_unk(tok):
    """
    Assign unknown word tokens
    """
    # Digits
    if any(char.isdigit() for char in tok):
        return "--unk_digit--"

    # Punctuation
    elif any(char in punct for char in tok):
        return "--unk_punct--"

    # Upper-case
    elif any(char.isupper() for char in tok):
        return "--unk_upper--"

    # Nouns
    elif any(tok.endswith(suffix) for suffix in noun_suffix):
        return "--unk_noun--"

    # Verbs
    elif any(tok.endswith(suffix) for suffix in verb_suffix):
        return "--unk_verb--"

    # Adjectives
    elif any(tok.endswith(suffix) for suffix in adj_suffix):
        return "--unk_adj--"

    # Adverbs
    elif any(tok.endswith(suffix) for suffix in adv_suffix):
        return "--unk_adv--"

    return "--unk--"

# Helper: substitues word not in the vocabulary with "unknown"
def get_word_tag(line, vocab):
    if not line.split():
        word = "--n--"
        tag = "--s--"
        return word, tag
    else:
        word, tag = line.split()
        if word not in vocab:
            # Handle unknown words
            word = assign_unk(word)
        return word, tag
    return None

from collections import defaultdict

def create_dictionaries(corpus, vocab):
    """
    Input:
        corpus: a corpus where each line has a word followed by its tag.
        vocab: a dictionary where keys are words in vocabulary and value is an index
    Output:
        emission_counts: a dictionary where the keys are (tag, word) and the values are the counts
        transition_counts: a...


I am working on the file "03-NLP/POS.ipynb".
The first task is:
Prediction
Let's compute the accuracy of the model:

assign a part of speech to a word (the most frequent POS for that word in the training set).
then evaluate how well this approach works.  Each time we predict based on the most frequent POS for the given word, we check whether the actual POS of that word is the same.  If so, the prediction was correct!
calculate the accuracy as the number of correct predictions divided by the total number of words for which we  predicted the POS tag.
Can you write Python code for it?

# Initialize the number of correct predictions to zero
num_correct = 0

# Get the (tag, word) tuples, stored as a set
all_words = set(emissionCounts.keys())

# Get the number of (word, POS) tuples in the corpus 'y'
total = len(testing_corpus)

for word, y_tup in zip(y_prepr, testing_corpus):
    # Split the (word, POS) string into a list of two items
    y_tup_l = y_tup.split()

    # Verify that y_tup contain both word and POS
    if len(y_tup_l) == 2:

        # Set the true POS label for this word
        true_label = y_tup_l[1]
    else:
        # If the y_tup didn't contain word and POS, go to next word
        continue

    count_final = 0
    pos_final = ''

    if word in vocabulary:
        for pos in tags:

            # define the key as the tuple containing the POS and word
            key = (pos,word)

            # check if the (pos, word) key exists in the emissionCounts dictionary
            if key in emissionCounts:
                # get the emission count of the (pos,word) tuple
                count = emissionCounts[key]

                # keep track of the POS with the largest count
                if count > count_final:
                    # update the final count (largest count)
                    count_final = count

                    # update the final POS
                    pos_final = pos

        # If the final POS (with the largest count) matches the true POS:
        if pos_final == true_label:
            # Update the number of correct predictions
            num_correct += 1

accuracy = num_correct / total

print(f"Accuracy of prediction using predict_pos is {accuracy:.4f}")


I am working on the file "03-NLP/POS.ipynb".
The first task is:
So dog is a singular noun (NN) and dogs is a plural noun (NNS).
So far so good.
Let's parse an entire sentence and we define another helper function for it:
Can you write Python code for it?

def getSentencePOS(sentence):
    """
    Input:
        sentence: the sentence to be tagged. String.
    Output:
        a list of tuples: each word with its most probable tag
    """
    words = sentence.split(" ") # split into words (a list)
    sentencePOS = [] # second list for the tags

    for word in words:
        wordPOS = getWordPOS(word) # find the word's POS
        sentencePOS.append(wordPOS)

            # return a list of tuples: (word, POS)
    return list(zip(words, sentencePOS)) # combine the two lists

print(getSentencePOS("I have a black cat"))


I am working on the file "03-NLP/POS.ipynb".
The first task is:
The transition dictionary shows how often we go from a tag to another, for example from DT (a determiner, an article such as 'the', or 'a') to a NNP (proper noun) is 9044 times
Can you write Python code for it?

print("emission examples: ")
for ex in list(emissionCounts.items())[200:203]:
    print (ex)
print()

print("ambiguous word example: ")
for tup,cnt in emissionCounts.items():
    if tup[1] == 'back': print (tup, cnt)


I am working on the file "03-NLP/POS.ipynb".
The first task is:
We got mapped a total of 46 POS tags which is great: every tag mentioned in the Penn Databank is here.
The 'tags' are the Parts-of-speech designations found in the training data.
- "NN" is noun, singular,
- 'NNS' is noun, plural.
- In addition, there are helpful tags like '--s--' which indicate a start of a sentence.
- You can get a more complete description at Penn Treebank II tag set.
Can you write Python code for it?

print("transition examples: ")
for ex in list(transitionCounts.items())[:3]:
    print(ex)
print()


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
An alternative way (without lambda functions) to sort a collection:
Can you write Python code for it?

from operator import itemgetter
sorted(totalWords.items(), key=itemgetter(1), reverse=True)


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
Another useful symbol is \w which match ANY alphanumeric character or \W which matches any NON alphanumeric character.
So, an alternative way could be:
Can you write Python code for it?

processedText = re.sub(r'[^\s\w]', '', strippedText) # remove punctuation
processedText

processedText4 = re.sub(r'^\s+', r'', processedText) # remove spaces at the beginning
processedText4

textTokens = processedText.split()
totalWords = Counter(textTokens);
totalWords


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
As you can see the most common words are not really meaningful but we can remove them
Stopwords (https://en.wikipedia.org/wiki/Stop_words) are common (English) words that do not contribute much to the content or meaning of a document (e.g., "the", "a", "is", "to", etc.).
Can you write Python code for it?

f = open("stopwords.txt")

stopWordsText = f.read().splitlines()  # splitlines is used to remove newlines

f.close()

stopWords = set(stopWordsText)

betterTokens = [token for token in textTokens if token not in stopWords]

betterTokens[:10]

getMostCommonWords(betterTokens)


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
As you can see, tokens are words but also symbols like *.
This is because we have split the string simply by using blank spaces.
But you can pass other separators to the function split() such as commas.
Tokens frequency
I have the number of tokens for this sample text. Let's say now that I want to count the frequency for each token.
This can be done quickly by using the Python package Counter from collections.
Can you write Python code for it?

from collections import Counter

totalWords = Counter(textTokens);
totalWords


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
Generate a words cloud from a text
Small example: we generate a words cloud from a text.
To make it more interesting we take as text a novel directly from the web (Project Gutenberg).
Can you write Python code for it?

from urllib.request import urlopen

def getWordCloud(filename):
    textTokens = [] # tokens will be added here
    lines = 0
    path = "http://www.gutenberg.org/files/"

    url = path + filename + "/" + filename + "-0.txt"
    f = urlopen(url)

    for line in f:
               # every line gets processed
        lineTokens = preprocessText(line.decode('utf-8'))
                # append the tokens to my list
        textTokens.extend(lineTokens)

        lines += 1  # finally move to the next line

    fs = open("stopwords.txt")
    stopWordsText = fs.read().splitlines()
    stopWords = set(stopWordsText)
    fs.close()
    betterTokens = [token for token in textTokens if token not in stopWords]

    wordsCount = Counter(betterTokens) # count the occurences

          # put each token and its occurrence in a file
    with open("wordcloud_"+filename+".txt", 'a') as fw:
        for line in wordsCount.most_common():
    #        freq = (line[1] + 10 // 2) // 10
     #       if freq > 2:
            fw.write(str(line[1]) + ' ' + line[0] + '\n')

getWordCloud("1342") # Jane Austen Pride and Prejudice


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
Let's put these results into functions we can re-use
Can you write Python code for it?

def tokenise(text):
    return text.split() # split by space; return a list

def removePunctuation(text):
    processedText = re.sub(r'([^\s\w_]|_)+', '', text.strip()) # remove punctuation

    return processedText

def preprocessText(text, lowercase=True):
    if lowercase:
        processedText = removePunctuation(text.lower())
    else:
        processedText = removePunctuation(text)

    return tokenise(processedText)

def getMostCommonWords(tokens, n=10):
    wordsCount = Counter(tokens) # count the occurrences

    return wordsCount.most_common()[:n]


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
Now the token you is also counted correctly 3 times.
A collection can be sorted easily:
Can you write Python code for it?

print (totalWords.items())

sorted(totalWords.items(), key=lambda x:x[1],reverse=True)


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
There are a number of problems:
* some word (like The/THE or You/you) is in two different tokens because of capital vs. small letter
* some token contains punctuation marks which makes the same word counted twice
* same token consists of symbols (like *)
Remove capital letters
This is very easy to do in Python by using the lower() function.
Can you write Python code for it?

loweredText = sampleText.lower()

loweredText

textTokens = loweredText.split()
totalWords = Counter(textTokens);
totalWords


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
This is an easy way to read the entire text but:
1. It's quite memory inefficient
2. It's slower than processing data as it is read, because it defers any processing done on read data until after all data has been read into memory, rather than processing as data is read.
A better way is to read the file line by line.
We can do this with a simple loop which will go through each line.
Note that the block keyword with will automatically close the file at the end of the block.
Can you write Python code for it?

textTokens = [] # tokens will be added here
lines = 0

with open(filePath) as f:
    for line in f:
               # I know that the very first line is the book title
        if lines == 0:
            print ("Title: {}".format(line))

               # every line gets processed
        lineTokens = preprocessText(line)
                # append the tokens to my list
        textTokens.extend(lineTokens)

        lines += 1  # finally move to the next line

print ("The text has {} lines".format (lines))
print ("The text has {} tokens".format (len(textTokens)))

textTokens[:10]


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
To remove punctuaction we can use regular expressions.
Regular expression are very powerful for match patterns in a sequence of characters.
The way in RE to match specific characters is to list them inside square brackets. For example [abc] will match only a single a,b or c letter an nothing else. A shorthand for matching sequential characters is to use the dash, for example [a-z] will match any single lowercase letter and [0-9] any single digit character.
To exclude characters from the matching we use the ^ (hat) symbol, for example [^abc] will match any single character except the letters a, b or c.
Finally there is also a special symbol for the whitespaces as they are so ubiquitous in text. Whitespaces are blank spaces, tabs, newlines, carriage returns. The symbol \s will match any of them.
The function re.sub() takes as input a starting string, a string to substitute and a pattern to match and returns the string with the matches replaced with the given substring using the given pattern.
for example re.sub(r'[s]', 'z', "whatsapp") will return "whatzapp".
So one way to remove the punctuation is to replace any character that is NOT a letter, a number or a whitespace with an empty substring:
Can you write Python code for it?

import re

processedText = re.sub(r'[^a-z0-9\s]', '', strippedText) # keep only numbers and letters
processedText


I am working on the file "03-NLP/helloworld-nlp.ipynb".
The first task is:
Unique tokens
Another useful data structure in Python is the set which is an unordered collection of distinct objects
Arranging the tokens in a set means that they will put only once,  and could be a smaller data collection useful to see how many distinct tokens are in a text or to see if a specific token is in the text or not.
Can you write Python code for it?

uniqueTokens = set(textTokens)
print ("The text has {} unique tokens".format (len(uniqueTokens)))
print (" -> lexical diversity: each token in average is repeated {} times".format(len(textTokens) / len(uniqueTokens)))

sorted(uniqueTokens)[200:205]

'accuse' in uniqueTokens

'phone' in uniqueTokens


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
And now we apply one of the NLTK stemmer, the Porter stemmer:
Can you write Python code for it?

porter = nltk.PorterStemmer()
[porter.stem(t) for t in words1]


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
And we can apply it to an entire book, "The Prince" by Machiavelli that we used last time:
Can you write Python code for it?

# If you would like to work with the raw text you can use 'bookRaw'
with open('../datasets/ThePrince.txt', 'r') as f:
    bookRaw = f.read()

bookTokens = nltk.word_tokenize(bookRaw)
bookText = nltk.Text(bookTokens) # special format
nBookTokens= len(bookTokens) # or alternatively len(bookText)

print ("*** Analysing book ***")
print ("The book is {} chars long".format (len(bookRaw)))
print ("The book has {} tokens".format (nBookTokens))


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
As mentioned above, the NTLK tokeniser works in a more sophisticated way than just splitting by spaces, therefore we got this time more tokens.
Sentences
NTLK has a function to tokenise a text not in words but in sentences.
Can you write Python code for it?

text1 = "This is the first sentence. A liter of milk in the U.S. costs $0.99. Is this the third sentence? Yes, it is!"
sentences = nltk.sent_tokenize(text1)
len(sentences)

sentences


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
As you see, all 5 different words have been reduced to the same stem and would be now the same lexical token.
Can you write Python code for it?

stemmedWords = [porter.stem(w) for w in betterWords]
topBook = get_top_words(stemmedWords)
topBook[:20]  # Output top 20 words


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
As you see, it is not splitting just after each full stop but check if it's part of an acronym (U.S.) or a number (0.99).
It also splits correctly sentences after question or exclamation marks but not after commas.
Can you write Python code for it?

sentences = nltk.sent_tokenize(bookRaw) # extract sentences
nSent = len(sentences)
print ("The book has {} sentences".format (nSent))
print ("and each sentence has in average {} tokens".format (nBookTokens / nSent))


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
It works with different adjectives, it doesn't look only at prefixes and suffixes.
You would wonder why stemmers are used, instead of always using lemmatisers: stemmers are much simpler, smaller and faster and for many applications good enough.
Now we lemmatise the book:
Can you write Python code for it?

lemmatisedWords = [lemmatizer.lemmatize(w, 'n') for w in betterWords]
topBook = get_top_words(lemmatisedWords)
topBook[:20]  # Output top 20 words


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
It's not nouns (NN) but interections (IN) such as preposition or conjunction.
Extra note: Parsing the grammar structure
Words can be ambiguous and sometimes is not easy to understand which kind of POS is a word, for example in the sentence "visiting aunts can be a nuisance", is visiting a verb or an adjective?
Tagging a PoS depends on the context, which can be ambiguous.
Making sense of a sentence is easier if it follows a well-defined grammatical structure, such as : subject + verb + object
NLTK allows to define a formal grammar which can then be used to parse a text.  The NLTK ChartParser is a procedure for finding one or more trees (sentences have internal organisation that can be represented using a tree) corresponding to a grammatically well-formed sentence.
Can you write Python code for it?

# Parsing sentence structure
text2 = nltk.word_tokenize("Alice loves Bob")
grammar = nltk.CFG.fromstring("""
S -> NP VP
VP -> V NP
NP -> 'Alice' | 'Bob'
V -> 'loves'
""")

parser = nltk.ChartParser(grammar)
trees = parser.parse_all(text2)
for tree in trees:
    print(tree)


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
Most common tokens
What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?
The NTLK FreqDist class is used to encode “frequency distributions”, which count the number of times that something occurs, for example a token.
Its most_common() method then returns a list of tuples where each tuple is of the form (token, frequency). The list is sorted in descending order of frequency.
Can you write Python code for it?

def get_top_words(tokens):
        # Calculate frequency distribution
    fdist = nltk.FreqDist(tokens)
    return fdist.most_common()

topBook = get_top_words(bookTokens)
  # Output top 20 words
topBook[:20]


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
Now the word princ is counted 281 times, exactly like the sum of prince and princes.
A note here: Stemming usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes.
Prince and princes become princ.
A different flavour is the lemmatisation that we will see in one second, but first a note about stemming in other languages than English.
Stemming in other languages
Snowball is an improvement created by Porter: a language to create stemmers and have rules for many more languages than English.
For example Italian:
Can you write Python code for it?

from nltk.stem.snowball import SnowballStemmer
stemmerIT = SnowballStemmer("italian")

inputIT = "Io ho tre mele gialle, tu hai una mela gialla e due pere verdi"
wordsIT = inputIT.split(' ')

[stemmerIT.stem(w) for w in wordsIT]


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
Now we removed the punctuation and the capital letters but the most common token is "the", not a significative word ...
As we have seen last time, these are so-called stop words that are very common and are normally stripped from a text when doing these kind of analysis.
Meaningful most common tokens
A simple approach could be to filter the tokens that have a length greater than 5 and frequency of more than 150.
Can you write Python code for it?

meaningfulWords = [word for (word,freq) in topBook if len(word) > 5 and freq > 80]
sorted(meaningfulWords)


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
Stemming
Above, in the list of words we have both prince and princes which are respectively the singular and plural version of the same word (the stem). The same would happen with verb conjugation (love and loving are considered different words but are actually inflections of the same verb).
Stemmer is the tool that reduces such inflectional forms into their stem, base or root form and NLTK has several of them (each with a different heuristic algorithm).
Can you write Python code for it?

input1 = "List listed lists listing listings"
words1 = input1.lower().split(' ')
words1


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
This would work but would leave out also tokens such as I and you which are actually significative.
The better approach - that we have seen earlier how - is to remove stopwords using external files containing the stop words.
NLTK has a corpus of stop words in several languages:
Can you write Python code for it?

from nltk.corpus import stopwords

stopwordsEN = set(stopwords.words('english')) # english language

betterWords = [w for w in bookWords if w not in stopwordsEN]

topBook = get_top_words(betterWords)
# Output top 20 words
topBook[:20]


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
We tell the lemmatise that the words are nouns. In this case it considers the same lemma words such as list (singular noun) and lists (plural noun) but leave as they are the other words.
Can you write Python code for it?

[lemmatizer.lemmatize(w, 'v') for w in words1] # v = verbs


I am working on the file "03-NLP/introNLTK.ipynb".
The first task is:
Yes, the lemma now is prince.
But note that we consider all words in the book as nouns, while actually a proper way would be to apply the correct type to each single word.
Part of speech (PoS)
In traditional grammar, a part of speech (abbreviated form: PoS or POS) is a category of words which have similar grammatical properties.
For example, an adjective (red, big, quiet, ...) describe properties while a verb (throw, walk, have) describe actions or states.
Commonly listed parts of speech are noun, verb, adjective, adverb, pronoun, preposition, conjunction, interjection.
Can you write Python code for it?

text1 = "Children shouldn't drink a sugary drink before bed."
tokensT1 = nltk.word_tokenize(text1)
nltk.pos_tag(tokensT1)


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
And this time it predicted correctly that Lisbon is the capital city of Portugal!
Model Accuracy
Now we will test the new function on the dataset and check the accuracy of the model:
$$\text{Accuracy}=\frac{\text{Correct # of predictions}}{\text{Total # of predictions}}$$
Can you write Python code for it?

def get_accuracy(word_embeddings, data):
    '''
    Input:
        word_embeddings: a dictionary where the key is a word and the value is its embedding
        data: a pandas dataframe containing all the country and capital city pairs

    Output:
        accuracy: the accuracy of the model
    '''

    num_correct = 0

    # loop through the rows of the dataframe
    for i, row in data.iterrows():

        city1 = row['city1']
        country1 = row['country1']

        city2 =  row['city2']
        country2 = row['country2']

        # use get_country to find the predicted country2
        predicted_country2, _ = get_country(city1, country1, city2, word_embeddings)

        # if the predicted country2 is the same as the actual country2...
        if predicted_country2 == country2:
            # increment the number of correct predictions by 1
            num_correct += 1

    m = len(data)

    # calculate the accuracy by dividing the number of correct predictions by
    # the number of rows in the data dataframe (length of dataframe)
    accuracy = num_correct / m

    return accuracy

accuracy = get_accuracy(word_embeddings, data)
print(f"Accuracy is {accuracy:.2f}")


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
Each of the word embedding is a 300-dimensional vector.
Can you write Python code for it?

countryVector = word_embeddings['country'] # Get the vector representation for the word 'country'
print(type(countryVector)) # Print the type of the vector. Note it is a numpy array
print(countryVector[0:10]) # Print the first ten values of the vector.


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
However, it does not always work:
Can you write Python code for it?


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
Now that the model is loaded, we can take a look at the word representations. First, note that the word_embeddings is a dictionary. Each word is the key to the entry, and the value is its corresponding vector presentation. Remember that square brackets allow access to any entry if the key exists.
Can you write Python code for it?


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
Now we will load the word embeddings as a Python dictionary.
As stated, these have already been obtained through a machine learning algorithm.
It's a pre-trained subset of the huge Google News dataset (see Introduction).
Can you write Python code for it?

import pickle  # object serializer: convert byte stream into a Python object

word_embeddings = pickle.load(open("../datasets/word_embeddings_subset.p", "rb"))


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
Plotting the vectors using PCA
Now we will explore plotting the vectors and the distance between word vectors after reducing their dimension, using the principal component analysis (PCA) technique that we have already seen. As we noticed, we are working in a 300-dimensional space in this case. Although from a computational perspective we will be able to perform a good job, it is impossible to visualise results in such high dimensional spaces.
PCA is a method that projects our vectors in a space of reduced dimension, while keeping the maximum information about the original vectors in their reduced counterparts. In this case, by maximum infomation we mean that the Euclidean distance between the original vectors and their projected siblings is minimal. Hence vectors that were originally close in the embeddings dictionary, will produce lower dimensional vectors that are still close to each other.
When you map out the words, similar words will be clustered next to each other. For example, the words 'sad', 'happy', 'joyful' all describe emotion and are supposed to be near each other when plotted. The words: 'oil', 'gas', and 'petroleum' all describe natural resources. Words like 'city', 'village', 'town' could be seen as synonyms and describe a similar thing.
Before plotting the words, we need to first reduce each word vector with PCA into 2 dimensions and compute the eigenvectors and the eigenvalues of the covariance matrix.
We start by defining two helper functions:
Can you write Python code for it?

def computePCA(X, n_components=2):
    """
    Input:
        X: of dimension (m,n) where each row corresponds to a word vector
        n_components: Number of components you want to keep (reduce to n_components)
    Output:
        X_reduced: data transformed in n dims/columns + regenerated original data
                   new matrix should be of dimension `m, n_components`
    """

    # First mean center the data
    X_demeaned = X - np.mean(X,axis=0)

    # calculate the covariance matrix
    covariance_matrix = np.cov(X_demeaned, rowvar=False)

    # calculate eigenvectors & eigenvalues of the covariance matrix
    eigen_vals, eigen_vecs = np.linalg.eigh(covariance_matrix, UPLO='L')

    # sort eigenvalue in increasing order (get the indices from the sort)
    idx_sorted = np.argsort(eigen_vals)

    # reverse the order so that it's from highest to lowest.
    idx_sorted_decreasing = idx_sorted[::-1]

    # sort the eigen values by idx_sorted_decreasing
    eigen_vals_sorted = eigen_vals[idx_sorted_decreasing]

    # sort eigenvectors using the idx_sorted_decreasing indices
    eigen_vecs_sorted = eigen_vecs[:,idx_sorted_decreasing]

    # select the first n eigenvectors (n is desired dimension
    # of rescaled data array, or dims_rescaled_data)
    eigen_vecs_subset = eigen_vecs_sorted[:,0:n_components]

    # transform the data by multiplying the transpose of the eigenvectors
    # with the transpose of the de-meaned data
    # Then take the transpose of that product.
    X_reduced = np.dot(eigen_vecs_subset.transpose(),X_demeaned.transpose()).transpose()

    return X_reduced

def getVectors(embeddings, words):
    """
    Input:
        embeddings: the words vectors
        words: a list of words
    Output:
        X: a matrix where the rows are the embeddings corresponding to the rows on the list

    """
    m = len(words)
    X = np.zeros((1, 300))

    for word in words:
        english = word
        eng_emb = embeddings[english]
        X = np.row_stack((X,...