I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
(5 pts) Implement the power method for a given matrix $A$, an initial guess $x_0$ and a number of iterations num_iter. It should be organized as a function power_method(A, x0, num_iter) that outputs approximation to eigenvector $x$, eigenvalue $\lambda$ and history of residuals ${\|Ax_k - \lambda_k x_k\|_2}$. Make sure that the method conveges to the correct solution on a matrix $\begin{bmatrix} 2 & -1 \ -1 & 2 \end{bmatrix}$ which is known to have the largest eigenvalue equal to $3$.
Can you write Python code for it?


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
Calculate top 10 nearest neighbours with the corresponding cosine similarities for the words {numerical, linear, algebra} and insert them in the correspoding functions in pset2.py.

Problem 4 (eigenvalues)  45 pts
1. Theoretical tasks

(5 pts) Prove that normal matrix is Hermitian iff its eigenvalues are real. Prove that normal matrix is unitary iff its eigenvalues satisfy $|\lambda| = 1$.

a)

normal matrix is Hermitian then eigenvalues are real
$$
A^{}A = AA^{}, A = A^{} \
Ax = \lambda x, x^{} A^{} = \overline{\lambda} x^ \
x^Ax = x^ \lambda x = \lambda \|x\|^2_2 \
(x^ A x)^ = (x^ \lambda x)^,\, x^ A^ x = \overline{\lambda} \|x\|^2_2 = x^ A x \
\lambda \|x\|^2_2 = \overline{\lambda} \|x\|^2_2 \
\lambda = \overline{\lambda} \to \lambda_i \in \mathbb{R} \
$$
eigenvalues are real then normal matrix is Hermitian
$$
x^Ax = \lambda \|x\|^2_2 \
x^ A^ x = \overline{\lambda} \|x\|^2_2 = \lambda \|x\|^2_2 \
x^(A-A^)x = \lambda \|x\|^2_2 - \lambda\|x\|^2_2 = 0 \
x \neq 0 \to A = A^ \
$$ for each non-zero eigenvector we will get $A=A^$ then $A$ is Hermitian

b)

normal matrix is unitary then eigenvalues satisfy $|\lambda| = 1$
$$
(Ax, Ax) = (\lambda x, \lambda x) = \lambda^2 \|x\|^2_2 \to \
x^ A^ A x = \lambda^2\|x\|^2_2 \to x^ x = \lambda^2 \|x\|^2_2 \to \lambda^2 = 1 \to |\lambda| = 1
$$
eigenvalues satisfy $|\lambda| = 1$ then normal matrix is unitary
$$
x^ A^ A x = \lambda^2 x^ I x \to x^(A^ A - I)x = 0
$$
for every non-zero eigen-vector $x: A^ A = I = A A^$ hence normal matrix is unitary q.e.d

(5 pts) The following problem illustrates instability of the Jordan form. Find theoretically the eigenvalues of the perturbed Jordan block:
$$
    J(\varepsilon) =
    \begin{bmatrix}
     \lambda & 1 & & & 0 \
     & \lambda & 1 & & \
     &  & \ddots & \ddots & \
     & & & \lambda & 1 \
     \varepsilon & & & & \lambda  \
    \end{bmatrix}{n\times n}
$$ Comment how eigenvalues of $J(0)$ are perturbed for large $n$.
$$
det(J(\varepsilon) - \alpha I) = 0 \
J(\varepsilon) =
    \begin{bmatrix}
   ...
Can you write Python code for it?

# implement the functions in the pset2.py file
from pset2 import pagerank_matrix, power_method, pagerank_matvec

G = np.matrix([[0, 0, 1, 0, 0],
               [1, 0, 1, 0, 0],
               [0, 1, 0, 0, 0],
               [0, 0, 0, 0, 1],
               [0, 0, 0, 1, 0]])
A = pagerank_matrix(G)
print("Matrix A:")
print(A)

ev = np.linalg.eig(A)
print(ev[0])
print("Largest eigenvalue is 1 with multiplicity 2")


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
Construct the word vocabulary from the obtained sentences which enumerates words which occur more than $r=200$ times in the corpus.
Can you write Python code for it?

def create_vocabulary(sentences, r=200):
    vocabulary = {}
    # Your code is here
    return vocabulary

vocab = create_vocabulary(sentences)


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
Here it can be seen that after regularization $\lambda_2' = d \lambda_2 $, where $d$ is damping factor.
The convergence rate is $q = d \lambda_2 $
There(p. 46) it is proven that:
If the spectrum of the stochastic matrix $S$ is ${1, λ_2, λ_3,...,λ_n}$, then
the spectrum of the Google matrix $G$ is ${1, αλ_2, αλ_3, . . . , αλ_n}$

(2 pts) Implement fast matrix-vector product for $A_d$ as a function pagerank_matvec(A, d, x), which takes a PageRank matrix $A$ (in sparse format, e.g., csr_matrix), damping factor $d$ and a vector $x$ as an input and returns $A_dx$ as an output. <br>
Done in pset2.py

(1 pts) Generate a random adjacency matrix of size $10000 \times 10000$ with only 100 non-zero elements and compare pagerank_matvec performance with direct evaluation of $A_dx$.
Can you write Python code for it?

import random
import time

N = 10000
G = np.zeros((N, N))
subG = (np.random.random((10, 10)) - 0.5).clip(0,1)
subG[subG.nonzero()] = 1
p1, p3 = random.randint(1, N - 10), random.randint(1, N - 10)
p2, p4 = p1 + 10, p3 + 10
G[p1:p2,p3:p4] = subG
x = np.random.rand(N, 1)
G = sp.sparse.csr_matrix(G)

A = pagerank_matrix(G)
d = 0.97

%%time
y = pagerank_matvec(A, d, x)

%%time
y = np.dot((d * A + (1 - d)/A.shape[0] * np.ones(A.shape)), x)


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
In out algorithm we know that the matrix is banded and apply much faster algorithm which is linear of size of matrix and quadratic of band size.
Numpy algorithm is not aware of this and thus does much worse.
2. Completing the proof of existence of LU
Some details in lecture proofs about $LU$ were omitted. Let us complete them here.
- (5 pts) Prove that if $LU$ decomposition exists, then matrix is strictly regular.

Matrix is sctrictly regular if all leading principal submatrices of A are non-singular.

Let's represent matrix $A = LU$ in the following way:
$$
\begin{bmatrix} A_{11} & A_{12} \ A_{21} & A_{22} \end{bmatrix} =
\begin{bmatrix} L_{11} & 0 \ L_{21} & L_{22} \end{bmatrix}
\begin{bmatrix} U_{11} & U_{12} \ 0 & U_{22} \end{bmatrix} =
\begin{bmatrix} L_{11}U_{11} & L_{11}U_{12} \ L_{21}U_{11} & L_{21}U_{12} + L_{22}U_{22} \end{bmatrix}
$$
$A_{11}$ consists of $k$ leading principal submatrices
$$
det(A_{11}) = det(L_{11}U_{11}) = det(L_{11})det(U_{11}) = det(U_{11}) \neq 0
$$
since $U$ is triangular and hence $A_{11}$ is non-singular

(5 pts) Prove that if $A$ is a strictly regular matrix, then $A_1 = D - \frac 1a b c^T$ (see lectures for notations) is also strictly regular.
$$ A =
\begin{bmatrix} 1 & 0 \ z & I \end{bmatrix}
\begin{bmatrix} a & c^T \ 0 & A_{1} \end{bmatrix} = KM\
A_1 = D - \frac{1}{a}bc^T
$$
All leading submatrices of $A$ are non-singular and the matrix $K$ is elementary matrix
$$ M = K^{-1} A $$
$K^{-1}$ is elementary as well then $K^{-1}$ does not zero determinant of all submatrices in $A$ then
all leading submatrices of $M$ are non-singular.

As $a \neq 0$ then all leading submatrices of $A_1$ are non-singular, q.e.d.

3. Stability of LU
Let
$A = \begin{pmatrix}
\varepsilon & 1 & 0\
1 & 1 & 1 \
0 & 1 & 1
\end{pmatrix}.$
* (5 pts) Find analytically LU decomposition with and without pivoting for the matrix $A$.

w/o pivoting
$$
A = \begin{pmatrix}
1 & \frac{1}{\varepsilon} & 0\
0 & 1 - \frac{1}{\varepsilon} & 1 \
0 & 1 & 1
\end{pmatrix} \sim
\begin{pmatrix}
1 &...
Can you write Python code for it?

# Implement the functions in the ```pset2.py``` file
from pset2 import gram_schmidt_qr
from pset2 import modified_gram_schmidt_qr

n = 20
x = np.linspace(0, 1, n)
V = np.vander(x)
Q, _ = gram_schmidt_qr(V)
print("Error (first version):", np.linalg.norm(Q.T @ Q - np.identity(n), 2))
Q, _ = modified_gram_schmidt_qr(V)
print("Error (second version):", np.linalg.norm(Q.T @ Q - np.identity(n), 2))


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
Much faster!
DBLP: computer science bibliography
Download the dataset from here, unzip it and put dblp_authors.npz  and dblp_graph.npz in the same folder with this notebook. Each value (author name) from dblp_authors.npz corresponds to the row/column of the matrix from dblp_graph.npz. Value at row i and column j of the matrix from dblp_graph.npz corresponds to the number of times author i cited papers of the author j. Let us now find the most significant scientists according to PageRank model over DBLP data.

(4 pts) Load the weighted adjacency matrix and the authors list into Python using load_dblp(...) function. Print its density (fraction of nonzero elements). Find top-10 most cited authors from the weighted adjacency matrix. Now, make all the weights of the adjacency matrix equal to 1 for simplicity (consider only existence of connection between authors, not its weight). Obtain the PageRank matrix $A$ from the adjacency matrix and verify that it is stochastic.
Can you write Python code for it?

from scipy.sparse import load_npz
import numpy as np
def load_dblp(path_auth, path_graph):
    G = load_npz(path_graph).astype(float)
    with np.load(path_auth) as data: authors = data['authors']
    return G, authors

G, authors = load_dblp('dblp_authors.npz', 'dblp_graph.npz')
density = np.count_nonzero(G.data)/(G.shape[0]) ** 2
print("Density: ", density)
diags = np.squeeze(np.asarray(np.sum(G, axis=1)))
top10 = sorted(range(len(diags)), reverse=True, key=lambda k: diags[k])[:10]
for i,val in enumerate(top10):
    print(str(i + 1) + ". ", authors[val])
G[G.nonzero()]  = 1

A = pagerank_matrix(G)

columns_sum = np.squeeze(np.asarray(np.sum(A, axis=0)))

epsilon = 10e-13
passed = 1
for col in range(A.shape[0]):
    if (abs(columns_sum[col]) > epsilon) and (abs(columns_sum[col] - 1) > epsilon):
        passed = 0
        print(columns_sum[col])
        break

if (passed):
    print("Matrix is stochastic")
else:
    print("Matrix is not stochastic")


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
Roundoff errors leads to loss of orthogonality, the second function makes less amount of errors.
2. Householder QR (10 pts)

(7 pts) Implement algorithm for computing QR decomposition based on Householder reflections as a function householder_qr(A) that takes a rectangular matrix A and outputs Q,R.

(2 pts) Apply it to the Vandermonde matrix $V$ created above. Print out the error $\|Q^{\top}Q-I\|_2$, where $Q$ is given by householder_qr(A). Compare it to the corresponding results of Gram-Schmidt and modified Gram-Schmidt algorithms and comment on it.

(3 pts) For values of $n = {2,25,100,250,500}$, create a $B\in\mathbb{R}^{n\times n}$ and an upper triangular matrix $R\in\mathbb{R}^{n\times n}$ both filled with standard normal entries. Use numpy (or scipy) built-in QR decomposition function to obtain a random orthogonal matrix $Q$ from the decomposition of $B$. Then compute $A = QR$ and apply your Gram-Schmidt and Householder algorithms to find the $Q$ and $R$ factors of $A$ – denoted as $\hat{Q}$ and $\hat{R}$.
Calculate relative errors
$$\frac{\|R-\hat{R}\|_2}{\|R\|_2}, \frac{\|Q-\hat{Q}\|_2}{\|Q\|_2}, \frac{\|A-\hat{Q}\hat{R}\|_2}{\|A\|_2}$$
for each value of $n$ and for both algorithms.
Note: scale (multiply corresponding rows/columns by -1) $Q, R,\hat{Q},\hat{R}$ such that diagonal elements of $R$ and $\hat{R}$ be positive.

Comment on the relative errors in $Q$ and $R$ (forward error) compared to ones in $QR$ (backward error).
Comment on the backward error obtained for Gram-Schmidt compared to Householder.
Can you write Python code for it?


I am working on the file "NLAhw/hw2/kaziakhmedov_edgar_2.ipynb".
The first task is:
To find good word embeddings, Levy and Goldberg, 2015 proposed to find rank-$d$ SVD of Shifted Positive Pointwise Mutual Information (SPPMI) matrix

$$
U \Sigma V^\top \approx \text{SPPMI},
$$
where $\text{SPPMI}(w, c) = \max\left(\text{SPMI}(w, c), 0 \right)$ and $\text{SPMI}(w, c)$ is the element of the matrix $\text{SPPMI}$ at position $(w, c)$.
Then use $W=U\sqrt{\Sigma}$ as word embedding matrix. Your task is to reproduce their results. Write function constructs $\text{SPPMI}$ matrix, computes its SVD and produces word-vectors matrix $W$. Pay attention that $\text{SPPMI}$ matrix is sparse!
Can you write Python code for it?

def compute_embeddings(D, k, d=200):
    # Your code is here
    return embedding_matrix

k = 5 # negative sampling parameter
W = compute_embeddings(D, k)


I am working on the file "NLAhw/hw3/Edgar_Kaziakhmedov_PS3.ipynb".
The first task is:
(2 pts) Start the method with a random initial guess x0, set num_iter_fix=0 and comment why the method can converge to a wrong eigenvalue.
Can you write Python code for it?

x0 = np.random.random((A.shape[0],)).astype(np.float32)

eigs_builtin = np.sort(np.linalg.eigh(nx.laplacian_matrix(G).todense())[0])
x, eigs = partition(A, 1e-3, 0, 5, x0, 1e-5)
print("Are second smallest the same?", np.allclose(eigs_builtin[1], eigs[-2]))
print(eigs_builtin[1], eigs[-1])


I am working on the file "NLAhw/hw3/Edgar_Kaziakhmedov_PS3.ipynb".
The first task is:
5. Reconstruction (15pts)
Can you write Python code for it?

from scipy.sparse.linalg import gmres
maxiter = 200

def normalize(mask): #proper normalization to binary mask
    mask = np.clip(mask, a_min=0, a_max=1)
    mask = np.round(mask)
    mask = np.asarray(mask, dtype=int)
    return mask

errs=[]
def callback(err): #callback function to store the history of convergence
    global errs
    errs.append(err)
    return

mask, _ = gmres(L_Gx, radiointel.reshape(N**2,1), maxiter=maxiter, callback = callback)

plt.figure(figsize=(7,7))
plt.imshow(normalize(mask.real).reshape((N, N)), cmap='binary')
plt.title('Restored building plan', size=16)
plt.colorbar()

plt.figure(figsize=(14,7))
plt.semilogy(errs)
plt.grid()
plt.title('Convergence', size=16)


I am working on the file "NLAhw/hw3/Edgar_Kaziakhmedov_PS3.ipynb".
The first task is:
Algorithm must halt before num_iter_fix + num_iter_adapt iterations if the following condition is satisfied $$ \boxed{\|\lambda_k - \lambda_{k-1}\|_2 / \|\lambda_k\|_2 \leq \varepsilon} \text{ at some step } k.$$
Do not forget to use the orthogonal projection from above in the iterative process to get the correct eigenvector.
It is also a good idea to use shift=0 before the adaptive stragy is used. This, however, is not possible since the matrix $L$ is singular, and sparse decompositions in scipy does not work in this case. Therefore, we first use a very small shift instead.

(3 pts) Generate a random lollipop_graph using networkx library and find its partition. Draw this graph with vertices colored according to the partition.
Can you write Python code for it?

m, n = (10, 20)
G = nx.lollipop_graph(m, n)
A = nx.adjacency_matrix(G)
x0 = np.random.random((A.shape[0],)).astype(np.float64)

eigs_builtin = np.sort(np.linalg.eigh(nx.laplacian_matrix(G).todense())[0])
x, eigs = partition(A, 0.01, 10, 100, x0, 0.00001)
print("Are second smallest the same?", np.allclose(eigs_builtin[1], eigs[-1]))

pos = nx.spring_layout(G)
plt.figure(figsize=(14,7))
nx.draw_networkx(G, pos, node_color=np.sign(x))


I am working on the file "NLAhw/hw3/Edgar_Kaziakhmedov_PS3.ipynb".
The first task is:
In this problem you asked to use using FFT-matvec and make the convolution operator for the picture of the size $N\times N$, where $N=300$ with the following kernel (2D Helmholtz scattering):
$$
T_{\overline{i_1 j_1}, \overline{i_2 j_2} } \equiv eG_{i_1-j_1,i_2-j_2}  = \frac{-1j}{4} H^{(2)}0 \left( k_0 \cdot \Delta r{ \overline{i_1 j_1}, \overline{i_2 j_2} } \right), \quad i_1,j_1, i_2, j_2 = 0,\dots, N-1 $$
except when both $i_1=i_2$ and $j_1 = j_2$.
In that case set $$T_{i_1=i_2, j_1=j_2} = 0$$.
Here
$1j$ is the imaginary unit, $H^{(2)}_0(x)$ - (complex-valued) Hankel function of the second kind of the order 0. See 'scipy.special.hankel2'.
$$ \Delta r_{ \overline{i_1 j_1}, \overline{i_2 j_2} } = h \sqrt{  (i_1-i_2)^2 + (j_1-j_2)^2  } $$
$$ h = \frac{1}{N-1}$$
$$k_0 = 50.0$$
Tasks:

(5 pts) Create the complex-valued kernel $eG$ ($2N-1 \times 2N-1$)-sized matrix according with the instructions above. Note that at the point where $\Delta r=0$ value of $eG$ should be manually zet to zero. Store in the variable eG. Plot the eG.real of it with plt.imshow

(5 pts) Write function Gx that calculates matvec of $T$ by a given vector $x$. Make sure all calculations and arrays are in dtype=np.complex64. Hint: matvex with a delta function in pl

(3 pts) What is the complexity of one matvec?

(2 pts) Use scipy.sparse.linalg.LinearOperator  to create an object that has attribute .dot() (this object will be further used in the iterative process). Note that .dot() input and output must be 1D vectors, so do not forget to use reshape.
(15 pts) Write a function that takes an appropriate Krylov method(s) and solves linear system $Gx=b$ to deconvolve radiointel. The result should be binary mask array (real, integer, of 0s and 1s) of the plane of the building. Make sure it converged sufficiently and you did the post-processing properly. Plot the result as an image.

Note: You can use standart fft and ifft from e.g. numpy.fft
1. Kernel (5 pts)
Can you write Python code for it?

k0 = 50.0
N = 300

def make_eG(k0, N):
    # INPUT:
    # k0 #dtype = float
    # N #dtype = int
    # OUTPUT:
    # np.array, shape = (2N-1, 2N-1), dtype = np.complex64
    # eG = np.zeros((2*N - 1, 2*N - 1), dtype=np.complex64)
    row = np.square(np.linspace(-N + 1, N - 1, 2*N - 1))
    column = np.square(np.linspace(-N + 1, N - 1, 2*N - 1))
    row_m, col_m = np.meshgrid(row, column, indexing = 'ij')
    eG = -1j * hankel2(0, k0 / (N - 1) * np.sqrt(row_m + col_m)) / 4
    eG[N - 1, N - 1] = 0
    return np.roll(eG.astype(np.complex64), shift=N-1, axis=(0,1))

eG = make_eG(k0=k0, N=N)
plt.imshow(eG.real)


I am working on the file "NLAhw/hw3/Edgar_Kaziakhmedov_PS3.ipynb".
The first task is:
Problem 3 (30 pts)
Say hi to the drone
You received a radar-made air scan data of a terrorist hideout made from a heavy-class surveillance drone. Unfortunately, it was made with an old-fashioned radar, so the picture is convolved with the diffractive pattern. You need to deconvolve the picture to recover the building plan.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
from scipy.special import hankel2
radiointel = np.load('radiointel.npy')
plt.subplot(1,2,1)
plt.imshow( np.abs(radiointel) )
plt.title('Intensity')
plt.subplot(1,2,2)
plt.imshow( np.angle(radiointel), cmap='bwr' )
plt.title('Phase')
plt.show()


I am working on the file "NLOUnitarityBounds.ipynb".
The first task is:
Beta Functions
Can you write Python code for it?

def betafunctions(l1, l3):
    # returns a list with the beta functions for the quartic couplings l1 and l3
    return np.array([14*l1**2 + 2*l3**2, 6*l1**2 + 4*l1*l3 + 6*l3**2])/(16*np.pi**2)


I am working on the file "NLOUnitarityBounds.ipynb".
The first task is:
Beta function contributions to the partial wave matrices (w/ factor of -3/2 included)
Can you write Python code for it?

def bMy1s1(l1, l3):
    return -(3/2)*My1s1(*betafunctions(l1, l3))
def bMy1s0(l1, l3):
    return -(3/2)*My1s0(*betafunctions(l1, l3))
def bMy0s1(l1, l3):
    return -(3/2)*My0s1(*betafunctions(l1, l3))
def bMy0s0(l1, l3):
    return -(3/2)*My0s0(*betafunctions(l1, l3))


I am working on the file "NLOUnitarityBounds.ipynb".
The first task is:
Plot
compare against FIG. 1 (a) of arXiv:1502.08511, which was made entirely using Mathematica.
Can you write Python code for it?

import matplotlib

%matplotlib inline

matplotlib.rc('text', usetex = True)
matplotlib.rc('font', family = 'serif')
matplotlib.rcParams['xtick.direction'] = 'in'
matplotlib.rcParams['ytick.direction'] = 'in'

delta = 0.3
x = np.arange(-15.0, 15.3, delta)
y = np.arange(-15.0, 15.3, delta)
X, Y = np.meshgrid(x, y)
Z1 = (-(4*X + Y)/(16*np.pi))**2
Z2 = ((X - 2*Y)/(16*np.pi))**2
Z3 = (-(2*X - Y)/(16*np.pi))**2

plt.figure(figsize=(6,6))

plt.contourf(X, Y, Z3, [0.25, 100], colors='green', alpha=0.3)
plt.contourf(X, Y, Z2, [0.25, 100], colors='orange', alpha=0.3)
plt.contourf(X, Y, Z1, [0.25, 100], colors='blue', alpha=0.3)
plt.contour(X, Y, Z3, [0.25], colors='green')
plt.contour(X, Y, Z2, [0.25], colors='orange')
plt.contour(X, Y, Z1, [0.25], colors='blue')

plt.title(r"FIG. 1(a): $\displaystyle\frac{1}{4} \geq \left(a_0^{(0)}\right)^2$", fontsize=14)
plt.xlabel(r"$\displaystyle\lambda_1(s)$", fontsize=16)
plt.ylabel(r"$\displaystyle\lambda_3(s)$", fontsize=16)


I am working on the file "NLP Overview.ipynb".
The first task is:
Chunks
Chunking is the process of collecting patterns of Part of Speech together, representing some meaning.  Analysis of a sentence which identifies the constituents (noun groups - "[The red tree] grows near the river", verbs, verb groups, etc.)
Our goal is detect into digital text, thing like "where are different entities located," or "which person is employed by what organization". Its the way in which we extract structured data (entities and relations) from unstructured text.
<img width=580px height=150px src="http://www.nltk.org/images/chunk-treerep.png" >
Can you write Python code for it?

from nltk import word_tokenize,pos_tag
from nltk.chunk import RegexpParser

chunker = RegexpParser(r'''
NP:
{<DT><NN.*><.*>*<NN.*>}
}<VB.*>{
''')

print tags
print chunker.parse(tags)


I am working on the file "NLP Overview.ipynb".
The first task is:
Entity Recognition - Chunking
Part of chunking. We look for chunks of Part Of Speech.  The goal is to detect entities: Person, Location, Time, etc.  I will use here the library ne_chunker. But we can train our own models, or provide grammar rules as before.
Can you write Python code for it?

from nltk.chunk import ne_chunk

sentence = "Daryl A. is the head of the coworking place Commoncode Corp. from where many people work in Melbourne, Australia."
pos_tags = pos_tag(word_tokenize(sentence))
pos_tags

from IPython.display import display
display(pos_tags)


I am working on the file "NLP Overview.ipynb".
The first task is:
Normalization
Another type of textual noise is about the multiple representations exhibited by single word.
For example – “play”, “player”, “played”, “plays” and “playing” are the different variations of the word – “play”, Though they mean different but contextually all are similar. The step converts all the disparities of a word into their normalized form (also known as lemma). Normalization is a pivotal step for feature engineering with text as it converts the high dimensional features (N different features) to the low dimensional space (1 feature), which is an ideal ask for any ML model.
The most common lexicon normalization practices are :
Stemming:  Stemming is a rudimentary rule-based process of stripping the suffixes (“ing”, “ly”, “es”, “s” etc) from a word.
Lemmatization: Lemmatization, on the other hand, is an organized & step by step procedure of obtaining the root form of the word, it makes use of vocabulary (dictionary importance of words) and morphological analysis (word structure and grammar relations).

Below is the sample code that performs lemmatization and stemming using python’s popular library – NLTK.
Can you write Python code for it?

from nltk.stem.wordnet import WordNetLemmatizer
lem = WordNetLemmatizer()

from nltk.stem.porter import PorterStemmer
stem = PorterStemmer()

word = "multiplying"
lem.lemmatize(word, "v")
stem.stem(word)


I am working on the file "NLP Overview.ipynb".
The first task is:
Part of Speech tagging is used for many important purposes in NLP:
Word Sense Disambiguation
Some language words have multiple meanings according to their usage. For example, in the two sentences below:
I. “Please book my flight for Delhi”
II. “I am going to read this book in the flight”
“Book” is used with different context, however the part of speech tag for both of the cases are different. In sentence I, the word “book” is used as v erb, while in II it is used as no un. (Lesk Algorithm is also us ed for similar purposes)
Improving word-based features
A learning model could learn different contexts of a word when used word as the features, however if the part of speech tag is linked with them, the context is preserved, thus making strong features. For example:
Sentence -“book my flight, I will read this book”
Tokens – (“book”, 2), (“my”, 1), (“flight”, 1), (“I”, 1), (“will”, 1), (“read”, 1), (“this”, 1)
Tokens with POS – (“book_VB”, 1), (“my_PRP$”, 1), (“flight_NN”, 1), (“I_PRP”, 1), (“will_MD”, 1), (“read_VB”, 1), (“this_DT”, 1), (“book_NN”, 1)
Normalization and Lemmatization
POS tags are the basis of lemmatization process for converting a word to its base form (lemma).
Efficient Stopword Removal
POS tags are also useful in efficient removal of stopwords.
For example, there are some tags which always define the low frequency / less important words of a language. For example: (IN – “within”, “upon”, “except”), (CD – “one”,”two”, “hundred”), (MD – “may”, “must” etc)
Word Senses
In linguistics, a word sense is one of the meanings of a word.  Until now, we worked with tokens and POS. So, for instance in "the man sit down on the bench near the river.", the token [bench] could be bench as a constructed object by humans where people sit, or the natural side where the river meets the land.

WordNet: A semantic graph for words. NLTK provides a interface to the API

<img src="https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcSFZ2l8g_3316qek21ZEIkTS0WIYs8-lfTvXtO3YGWHEGpdDiMG">
Lets see some functions to handle...
Can you write Python code for it?

from nltk.corpus import wordnet as wn #loading wordnet module

wn.synsets('human')

wn.synsets('human')[0].definition

wn.synsets('human')[1].definition

human = wn.synsets('Human',pos=wn.NOUN)[0]
human

human.hypernyms()

human.hyponyms()

bike = wn.synsets('bicycle')[0]
bike

girl = wn.synsets('girl')[1]
girl

bike.wup_similarity(human)

girl.wup_similarity(human)


I am working on the file "NLP Overview.ipynb".
The first task is:
Part of speech tagging
Apart from the grammar relations, every word in a sentence is also associated with a part of speech (pos) tag (nouns, verbs, adjectives, adverbs etc). The pos tags defines the usage and function of a word in the sentence. Here is a list of all possible pos-tags defined by Pennsylvania university. Following code using NLTK performs pos tagging annotation on input text. (it provides several implementations, the default one is perceptron tagger)
Can you write Python code for it?

from nltk import pos_tag
#this is a Classifier, given a token assign a class
#pos_tag Already defined in the library. We can train our own.

tags = pos_tag(tokens)

text = "I am using Data Science Experience at Florida Blue for Natural Language Processing"
tokens = word_tokenize(text)
print pos_tag(tokens)

# Let's apply this to our sample text from our website.
tags


I am working on the file "NLP Overview.ipynb".
The first task is:
Regex Filtering
Another approach is to use the regular expressions while dealing with special patterns of noise. We have explained regular expressions in detail in one of our previous article. Following python code removes a regex pattern from the input text:
Can you write Python code for it?

import re

def _remove_regex(input_text, regex_pattern):
    urls = re.finditer(regex_pattern, input_text)
    for i in urls:
        input_text = re.sub(i.group().strip(), '', input_text)
    return input_text

regex_pattern = "#[\w]*"

_remove_regex("remove this #FloridaBlue from tweet text", regex_pattern)


I am working on the file "NLP Overview.ipynb".
The first task is:
Sentiment Analyisis
Is a phrase expresing a positive opinion? a negative opinion? how can we measure that? We will decompose sentences into their smaller units: tokens, and we will measure how probable they distribute on positive/negative sentences along the text.
<h2> Lets Classify Text! </h2>
Can you write Python code for it?

#We will use movie reviews, already separated as positive and negative. Specially interest are those bigrams,
#that are in positive or negative sentences (pair of words (word1, word2) that appear consecutives.)
from nltk.corpus import movie_reviews

import nltk.classify.util
from nltk.classify import NaiveBayesClassifier
from nltk.corpus import movie_reviews

#This is the function that given a word, return a dict {word:True}. This will be our feature in the classifier.
def word_feats(words):
    return dict([(word, True) for word in words])

#neg_ids, pos_ids keep all the files for negative reviews, and positive reviews respectively.
neg_ids = movie_reviews.fileids('neg')
pos_ids = movie_reviews.fileids('pos')

#So, lets take the positive/negative words, create the feature for such word, and store it in a negative/positive features list.
neg_feats = [(word_feats(movie_reviews.words(fileids=[f])), 'neg') for f in neg_ids]
pos_feats = [(word_feats(movie_reviews.words(fileids=[f])), 'pos') for f in pos_ids]

#Separating 3/4 of this featured words for training, 1/4 for test.
neg_len_train = len(neg_feats)*3/4
pos_len_train = len(pos_feats)*3/4

train_feats = neg_feats[:neg_len_train] + pos_feats[:pos_len_train]
test_feats = neg_feats[neg_len_train:] + pos_feats[pos_len_train:]

#training a NaiveBayes Classifier with our training featured words.
classifier = NaiveBayesClassifier.train(train_feats)

#Lts check accuracy.
print 'accuracy: ', nltk.classify.util.accuracy(classifier, test_feats)

#Lets see which words fit best in each class.
classifier.show_most_informative_features()

#SO WE TRAINED A CLASSIFIER FOR MOVIE REVIEWS. IT MEANS, FOR EVERY WORD THAT WE TRAINED,
#IT NOWS THAT THIS WORD IS PROBABLE IN NEGATIVES WITH PROB P(W/POS) AND POSITIVE P(W/POS) (BAYES THEOREME).

sentence = "Florida Blue, movie is incredible!"
tokens = [word for word in word_tokenize(sentence)]

pos_tags = [pos for pos in pos_tag(tokens)]

pos_tags

feats = word_feats( [word for (word,_) in pos_tags]...


I am working on the file "NLP Overview.ipynb".
The first task is:
Standardization
Text data often contains words or phrases which are not present in any standard lexical dictionaries. These pieces are not recognized by search engines and models.
Some of the examples are – acronyms, hashtags with attached words, and colloquial slangs. With the help of regular expressions and manually prepared data dictionaries, this type of noise can be fixed, the code below uses a dictionary lookup method to replace social media slangs from a text.
Can you write Python code for it?

translation_dict = {'rt':'Retweet', 'dm':'direct message', "awsm" : "awesome", "luv" :"love"}


I am working on the file "NLP Primer.ipynb".
The first task is:
Create a Bag-of-Words (BOW) and display the words that occur most often
Can you write Python code for it?

#max_keys = []

#keys = []
#values = []
#for k,v in counts.items():
#    keys.append(k)
#    values.append(int(v))

#bag = pd.DataFrame({"words" : keys, "count" : values}).set_index("words")
#bag = bag.sort_values(by="count",ascending=False)[0:40]
#bag

import scipy

X = scipy.sparse.csr_matrix(scounts.values.T)
X.shape

from sklearn.feature_extraction.text import TfidfTransformer

tfidf = TfidfTransformer()
tfidf.fit(X)
scores = tfidf.transform(X)

id_bad,  = np.where(scounts.index == "bad")[0]
id_nice, = np.where(scounts.index == "nice")[0]

print(id_bad, id_nice)

print(scores.shape)

for k in range(1):
    print(scores[k,:])

import seaborn as sns
sns.set_style("white")

def plot_embedding(word1, word2):
    id_bad,  = np.where(scounts.index == word1)[0]
    id_nice, = np.where(scounts.index == word2)[0]
    y = np.zeros((18, 2))

    for k in range(18):

        vec = scores[k,:].toarray()[0]

        y[k,0] = vec[id_bad]
        y[k,1] = vec[id_nice]

    y = y / (y**2).sum(axis=-1, keepdims=True)**0.5

    figure(figsize=(5,5))
    scatter(y[:,0], y[:,1], c=range(18), cmap="Set1")
    for i, txt in enumerate(corpus.keys()):
        if "shake" in txt or True:
            annotate(txt, (y[i,0],y[i,1]))

    plot(np.linspace(0,1/2**0.5,10), np.linspace(0,1/2**0.5,10))

    xlim([0, 1])
    ylim([0, 1])
    xlabel(word1)
    ylabel(word2)

def query(query):
    pass

plot_embedding("love", "lie")
show()

# macbeth —> macht, orakel, schicksaal, könig
# hamlet —>  macht, könig, rache, liebe

"""
among, fear, brought, ground, turn, water, wish, arm, dead, sound, lost, none, lay, saying, bed,
run, meet, doubt, fall, need, truth, ready, free, close, making, sit, wood, please, secret, fast,
red, past, dare, age, met, lie, laugh, noise, tongue, rise, breath, besides, wise, fly, angry,
write, shake, [, ], forgot, gently, lettered, care., caring, self-same, 7:57, tumble, flexion,
43:25, interweaving, 119:36, 43:3, no, stomach's, japonica, money-making, elihoenai, erst, vibration,
feebleness,...


I am working on the file "NLP Primer.ipynb".
The first task is:
Preprocessing: Stemming, Tokenization, Bag-of-Words
Use the WordNet stemmer.
After preprocessing, a Bag of Words is generated excluding stop words for all documents in the corpus, returned as a matrix.
Can you write Python code for it?

stemmer = nltk.WordNetLemmatizer()
tokens = {k : word_tokenize(corpus[k]) for k in corpus.keys()}

stemmed_stopwords = [stemmer.lemmatize(t.lower()) for t in stopwords.words('english')]
stemmed_tokens = {k : [stemmer.lemmatize(t.lower()) for t in tokens[k]] for k in corpus.keys()}

index = set()
for k in corpus.keys():
    for token in stemmed_tokens[k]:
        index.add(token)

counts = pd.DataFrame(index=index, columns=corpus.keys(), data=0)

for key in corpus.keys():
    print(key)
    for t in stemmed_tokens[key]:
        if t in stemmed_stopwords:
            continue
        counts[key][t] += 1
counts

# store as sparse dataframe
scounts = counts.to_sparse(fill_value=0)
print("Density", scounts.density)

scounts.to_pickle("corpus_wordcounts.npy")

scounts = pd.read_pickle("corpus_wordcounts.npy")
cols = list(scounts.keys())
scounts.sort_values(by="shakespeare-hamlet", ascending=False)
keywords = (scounts/scounts.sum(axis=0)).prod(axis=1).sort_values(ascending=False)

print(", ".join(list(keywords.index[100:300])))


I am working on the file "NLP Workshop (complete).ipynb".
The first task is:
For each word $ w $ in a given document $ D $, we can multiply the term frequency
$$\frac { { D }{ w } }{ { W }{ D } } $$
where $ { D }_{ w } $ is the number of occurrences of word $ w $ in document $ D $
and $ { W }_{ D } $ is the total number of words in document $ D $
with the word's IDF that we just calculated to get TF-IDF scores, the highest ones being words that likely to be good representatives of that document.
Can you write Python code for it?

comment = data[80][-1]
blob = TextBlob(comment.lower())
num_words_in_comment = len(blob.words)
word_count = blob.word_counts

tf_scores = {}
for word, count in word_count.items():
    if word not in stopwords and len(word) > 2:
        tf_scores[word] = float(count)/num_words_in_comment

tf_idf = {}
for word, tf in tf_scores.items():
    tf_idf[word] = tf*word_idf[word]

sorted(tf_idf.iteritems(), key=lambda k: k[1], reverse=True)[:5]


I am working on the file "NLP Workshop (complete).ipynb".
The first task is:
Introducing TextBlob
Like a supercharged string, with lots of NLP niceties
Can you write Python code for it?

blob = TextBlob(data[80][-1])
blob

blob.sentences

blob.words

blob.tokens

blob.noun_phrases


I am working on the file "NLP Workshop (complete).ipynb".
The first task is:
Note that TF-IDF can be tweaked in lots of other ways if you aren't getting good results.
It can also be done with "n-grams"— phrases that are n words long to capture multi word phrases like "gay rights" or "hillary clinton"
Additional demonstrations
Boiling down words: stemming
Can you write Python code for it?

from nltk.stem.porter import PorterStemmer

stemmer = PorterStemmer()
print stemmer.stem('political')
print stemmer.stem('politics')
print stemmer.stem('politician')


I am working on the file "NLP Workshop (complete).ipynb".
The first task is:
Sentiment Analysis
Can you write Python code for it?

blob = TextBlob(data[41][-1])
blob

blob.sentiment

blob.sentences[1].sentiment


I am working on the file "NLP Workshop (complete).ipynb".
The first task is:
We could continue to add on stopwords as we try to make these keywords better. But it's kind of like playing whack-a-mole
An additional solution to The Problem: add a new term to our "representative-ness" measure that accounts for the overall rarity of the word
$$\frac { { n }_{ w } }{ N }  $$
where ${ n }_{ w }$ is the number of documents containing word $ w $, and $ N $ is the total number of documents.
But we want a potential keyword to have a lower score if it is common in the corpus and a higher score if it is rarer, so we flip it:
$$\frac { N }{ { n }_{ w } }  $$
It's also common to take the log of this to reduce the amount of disparity between extremely common and extremely uncommon terms.
$$ \log \frac { N }{ { n }_{ w } }  $$
This is called IDF, or Inverse Document Frequency. Let's calculate it for all the words in our comment dataset!
Can you write Python code for it?

N_documents = float(len(data))
word_document_counts = collections.Counter()
word_idf = {}

for row in data[1:]:
    blob = TextBlob(row[-1].lower())
    words = blob.word_counts.keys()
    word_document_counts.update(words)

for key, val in word_document_counts.iteritems():
    word_idf[key] = math.log(N_documents/val)


I am working on the file "NLP Workshop.ipynb".
The first task is:
Basic python string manipulation
Can you write Python code for it?

comment_text

# strings are like lists of characters

# use a colon for start:end indexes

# they can be stuck together easily

# and split apart
comment_text

split_on_questions =

# it's easy to strip whitespace off of them
for string in split_on_questions:

# and cast them to one case
cleaned =
cleaned

# join them back together

# and look for substring inside them


I am working on the file "NLP Workshop.ipynb".
The first task is:
Challenge: get overall word counts for all comments combined
potential approaches:
* glue together all comments into one big blob
* get word counts for each comment individually and use Counter's update function
The Problem: words we use frequently don't make good unique identifiers.
One solution: use a list of words we don't want to include
"Stop Words"
Can you write Python code for it?

stopwords = nltk.corpus.stopwords.words('english')

nltk.download()

for key in word_count.keys():


I am working on the file "NLP Workshop.ipynb".
The first task is:
Introducing TextBlob
Like a supercharged string, with lots of NLP niceties
Can you write Python code for it?

blob = TextBlob(data[80][-1])
blob

# we can get lists of sentences

# lists of words

# lists of "tokens" (punctuation included)

# even parts of speech and noun phrases


I am working on the file "NLP Workshop.ipynb".
The first task is:
Note that TF-IDF can be tweaked in lots of other ways if you aren't getting good results.
It can also be done with "n-grams"— phrases that are n words long to capture multi word phrases like "gay rights" or "hillary clinton"
Additional demonstrations
Boiling down words: stemming
Can you write Python code for it?


I am working on the file "NLP Workshop.ipynb".
The first task is:
Seeing words in context: concordance
Can you write Python code for it?

from nltk.text import Text
tokens = TextBlob(data[80][-1]).tokens
text_object = Text(tokens)
text_object.concordance('Hilary')


I am working on the file "NLP Workshop.ipynb".
The first task is:
Sentiment Analysis
Can you write Python code for it?


I am working on the file "NLP Workshop.ipynb".
The first task is:
Summarizing/keywording text
How might we find representative words or phrases of a document?
A place to start: which words appear at the highest frequency in this document?
Can you write Python code for it?

word_count = collections.Counter(

word_count


I am working on the file "NLP Workshop.ipynb".
The first task is:
We could continue to add on stopwords as we try to make these keywords better. But it's kind of like playing whack-a-mole
An additional solution to The Problem: add a new term to our "representative-ness" measure that accounts for the overall rarity of the word
$$\frac { { n }_{ w } }{ N }  $$
where ${ n }_{ w }$ is the number of documents containing word $ w $, and $ N $ is the total number of documents.
But we want a potential keyword to have a lower score if it is common in the corpus and a higher score if it is rarer, so we flip it:
$$\frac { N }{ { n }_{ w } }  $$
It's also common to take the log of this to reduce the amount of disparity between extremely common and extremely uncommon terms.
$$ \log \frac { N }{ { n }_{ w } }  $$
This is called IDF, or Inverse Document Frequency. Let's calculate it for all the words in our comment dataset!
Can you write Python code for it?

N_documents = float(len(data))
word_document_counts =
word_idf = {}

for row in data[1:]:
    blob = TextBlob(row[-1].lower())

# calculate IDFs


I am working on the file "NLP checks.ipynb".
The first task is:
Named Entity Recognition
Can you write Python code for it?

nltk.download(["maxent_ne_chunker", "words", "stopwords"])

text = nltk.word_tokenize("US president Barack Obama signed a new treaty with the Indian prime minister Narendra Modi, in New Delhi.")
pos_tags = nltk.pos_tag(text)
print pos_tags

try:
    chunk_tags = nltk.ne_chunk(pos_tags, binary=False)
except:
    print "Done"
print chunk_tags


I am working on the file "NLP checks.ipynb".
The first task is:
POS Tagging
Can you write Python code for it?

nltk.download("averaged_perceptron_tagger")

text = nltk.word_tokenize("And now for something completely different")
nltk.pos_tag(text)


I am working on the file "NLP checks.ipynb".
The first task is:
Sentiwordnet
More details at:
* http://www.nltk.org/howto/sentiwordnet.html
* http://sentiwordnet.isti.cnr.it/
Can you write Python code for it?

nltk.download("sentiwordnet")

from nltk.corpus import sentiwordnet as swn

breakdown = swn.senti_synset('breakdown.n.03')
print breakdown


I am working on the file "NLP with NLTK.ipynb".
The first task is:
All of these taggers work pretty well - but you can (and should train them on your own corpora).
Stemming and Lemmatization
We have an immense number of word forms as you can see from our various counts in the FreqDist above - it is helpful for many applications to normalize these word forms (especially applications like search) into some canonical word for further exploration. In English (and many other languages) - mophological context indicate gender, tense, quantity, etc. but these sublties might not be necessary:
Stemming = chop off affixes to get the root stem of the word:
running --> run
flowers --> flower
geese   --> geese

Lemmatization = look up word form in a lexicon to get canonical lemma
women   --> woman
foxes   --> fox
sheep   --> sheep

There are several stemmers available:
- Lancaster (English, newer and aggressive)
- Porter (English, original stemmer)
- Snowball (Many langauges, newest)

The Lemmatizer uses the WordNet lexicon
Can you write Python code for it?

from nltk.stem.snowball import SnowballStemmer
from nltk.stem.lancaster import LancasterStemmer
from nltk.stem.porter import PorterStemmer

text = list(nltk.word_tokenize("The women running in the fog passed bunnies working as computer scientists."))

snowball = SnowballStemmer('english')
lancaster = LancasterStemmer()
porter = PorterStemmer()

for stemmer in (snowball, lancaster, porter):
    stemmed_text = [stemmer.stem(t) for t in text]
    print stemmed_text

from nltk.stem.wordnet import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(t) for t in text]
print lemmas


I am working on the file "NLP with NLTK.ipynb".
The first task is:
As you can see, this takes a bit of time to build the index in memory, one of the reasons it's not suggested to use this class in production code. Now that we can do searching and similarity, find the common contexts of a set of words:
Can you write Python code for it?


I am working on the file "NLP with NLTK.ipynb".
The first task is:
Given some context surrounding a word, we can discover similar words, e.g. words that that occur frequently in the same context and with a similar distribution: Distributional similarity:
Can you write Python code for it?

moby.similar("monstrous")
austen = nltk.text.Text(nltk.corpus.gutenberg.words('austen-sense.txt'))
print
austen.similar("monstrous")


I am working on the file "NLP with NLTK.ipynb".
The first task is:
Introduction to NLP with NLTK
Natural Language Processing (NLP) is often taught at the academic level from the perspective of computational linguists. However, as data scientists, we have a richer view of the natural language world - unstructured data that by its very nature has latent information that is important to humans. NLP practioners have benefited from machine learning techniques to unlock meaning from large corpora, and in this class we’ll explore how to do that particularly with Python and with the Natural Language Toolkit (NLTK).
NLTK is an excellent library for machine-learning based NLP, written in Python by experts from both academia and industry. Python allows you to create rich data applications rapidly, iterating on hypotheses. The combination of Python + NLTK means that you can easily add language-aware data products to your larger analytical workflows and applications.
Quick Overview of NLTK
NLTK stands for the Natural Language Toolkit and is written by two eminent computational linguists, Steven Bird (Senior Research Associate of the LDC and professor at the University of Melbourne) and Ewan Klein (Professor of Linguistics at Edinburgh University). NTLK provides a combination of natural language corpora, lexical resources, and example grammars with language processing algorithms, methodologies and demonstrations for a very pythonic "batteries included" view of Natural Language Processing.
As such, NLTK is perfect for researh driven (hypothesis driven) workflows for agile data science. Its suite of libraries includes:

tokenization, stemming, and tagging
chunking and parsing
language modeling
classification and clustering
logical semantics

NLTK is a useful pedagogical resource for learning NLP with Python and serves as a starting place for producing production grade code that requires natural language analysis. It is also important to understand what NLTK is not:

Production ready out of the box
Lightweight
Generally applicable
Magic

NLTK provides a variety of tools that can be used...
Can you write Python code for it?


I am working on the file "NLP with NLTK.ipynb".
The first task is:
Named Entity Recognition
NLTK has an excellent MaxEnt backed Named Entity Recognizer that is trained on the Penn Treebank. You can also retrain the chunker if you'd like - the code is very readable to extend it with a Gazette or otherwise.
Can you write Python code for it?

print nltk.ne_chunk(nltk.pos_tag(nltk.word_tokenize("John Smith is from the United States of America and works at Microsoft Research Labs")))


I am working on the file "NLP with NLTK.ipynb".
The first task is:
Note that the lemmatizer has to load the WordNet corpus which takes a bit.
Typical normalization of text for use as features in machine learning models looks something like this:
Can you write Python code for it?

import string

## Module constants
lemmatizer  = WordNetLemmatizer()
stopwords   = set(nltk.corpus.stopwords.words('english'))
punctuation = string.punctuation

def normalize(text):
    for token in nltk.word_tokenize(text):
        token = token.lower()
        token = lemmatizer.lemmatize(token)
        if token not in stopwords and token not in punctuation:
            yield token

print list(normalize("The eagle ... flies at midnight."))


I am working on the file "NLP with NLTK.ipynb".
The first task is:
The nltk.text.Text class is a wrapper around a sequence of simple (string) tokens - intended only for the initial exploration of text usually via the Python REPL. It has the following methods:

common_contexts
concordance
collocations
count
plot
findall
index

You shouldn't use this class in production level systems, but it is useful to explore (small) snippets of text in a meaningful fashion.
The corcordance function performs a search for the given token and then also provides the surrounding context:
Can you write Python code for it?


I am working on the file "NLP with NLTK.ipynb".
The first task is:
To explore much of the built in corpus, use the following methods:
Can you write Python code for it?

# Lists the various corpora and CorpusReader classes in the nltk.corpus module
for name in dir(nltk.corpus):
    if name.islower() and not name.startswith('_'): print name

# For a specific corpus, list the fileids that are available:
print nltk.corpus.shakespeare.fileids()

print nltk.corpus.gutenberg.fileids()

print nltk.corpus.stopwords.fileids()


I am working on the file "NLP with NLTK.ipynb".
The first task is:
your turn, go ahead and explore similar words and contexts - what does the common context mean?
NLTK also uses matplotlib and pylab to display graphs and charts that can show dispersions and frequency. This is especially interesting for the corpus of innagural addresses given by U.S. presidents.
Can you write Python code for it?

inaugural = nltk.text.Text(nltk.corpus.inaugural.words())
inaugural.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Find the top 5 topic words for each component

 Using the components dataframe create a dictionary with components as keys, and top words as values:

 Make an empty dictionary and loop through each row of NMF components

 Add to the dictionary where the key is the NMF component and the value is the topic words for that component (the column names with the largest component values)

Can you write Python code for it?

# create dictionary with the key = component, value = top 5 words
topic_dict = {}
for i in range(0,160):
    component = components_df.iloc[i, :]
    topic_dict[i] = component.nlargest()

# look at a few of the component topics
print(topic_dict[124].index)
print(topic_dict[10].index)


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Frequency plots for most used words in the federal data and twitter data
Can you write Python code for it?

# frequency plot for the most used Federal Data
df = pd.DataFrame(top_30_fed, columns=['Federal Data', 'frequency'])
ax = df.plot(kind='bar', x='Federal Data',legend=None, figsize=(14,10))
# change tick markes
plt.tick_params(top='off', bottom = 'on', left = 'on', right = 'off', labelleft = 'on', labelbottom = 'on')
plt.ylabel('Frequency', fontsize=20)
plt.xlabel('Top Federal Data Words', fontsize=20)
plt.tight_layout()
plt.show()

# frequency plot for the most used words in the twitter data
df = pd.DataFrame(top_30_tweet, columns=['Twitter Data', 'frequency'])
ax = df.plot(kind='bar', x='Twitter Data',legend=None, figsize=(14,10))
plt.ylabel('Frequency')
# change tick markes
plt.tick_params(top='off', bottom = 'on', left = 'on', right = 'off', labelleft = 'on', labelbottom = 'on')
plt.ylabel('Frequency', fontsize=20)
plt.xlabel('Top Twitter Words', fontsize=20)
plt.tight_layout()
plt.show()

# find the unique words in each dataset
joint_words = list((set(all_document_words)).intersection(all_twitter_words))
# make array of zeros
values = np.zeros(len(joint_words))
# create dictionary
joint_words_dict = dict(zip(joint_words, values))
# create a dictionary with a word as key, and a value = number of documents that contain the word for Twitter
twitter_document_freq = joint_words_dict.copy()
for word in joint_words:
    for lst in twitter_data.text_tokenized:
        if word in lst:
            twitter_document_freq[word]= twitter_document_freq[word] + 1

# create a dictionary with a word as key, and a value = number of documents that contain the word for Fed Data
fed_document_freq = joint_words_dict.copy()
for word in joint_words:
    for lst in fed_data.token_text:
        if word in lst:
            fed_document_freq[word]= fed_document_freq[word] + 1
df = pd.DataFrame([fed_document_freq, twitter_document_freq]).T
df.columns = ['Fed', 'Tweet']
df['% Fed'] = (df.Fed/len(df.Fed))*100
df['% Tweet'] = (df.Tweet/len(df.Tweet))*100
top_joint_fed = df[['% Fed','% Tweet']].sort_values(by='% Fed',...


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Group Again
Many of the unique groups are only different by one document. Those groups are similar to each other and should be combined.
Can you write Python code for it?

# make a list of groups that are similar
similar_groups = []
for num1 in range(len(unique_groups)):
    for num2 in range(len(unique_groups)):
        crossover_count = len((set(unique_groups[num1]) & set(unique_groups[num2])))
        if crossover_count == 4:
            lst = [num1, num2]
            lst.sort(key=int)
            if lst not in similar_groups:
                similar_groups.append(lst)

# create list of document ID's of similar documents
similar_docs = []
for group1, group2 in similar_groups:
    combo = list(set(unique_groups[group1]) & set(unique_groups[group2]))
    if combo not in similar_docs:
        similar_docs.append(combo)

similar_docs


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Identify nonsense words for both twitter and federal data
Can you write Python code for it?

# nonsense words, and standard words like proclimation and dates
more_stop = set(['presidential', 'documents', 'therfore','i','donald', 'j', 'trump', 'president', 'order',
                 'authority', 'vested', 'articles','january','february','march','april','may','june','july','august','september','october',
                 'november','december','jan','feb','mar','apr','jun','jul','aug','sep','oct','nov','dec',
                 '2017','2018','act','agencies','agency','wh','rtlwanjjiq','pmgil08opp','blkgzkqemw','qcdljff3wn','erycjgj23r ','fzep1e9mo7','m0hmpbuz6c','rdo6jt2pip','kyv866prde','aql4jlvndh',
             'tx5snacaas','t0eigo6lp8','jntoth0mol','8b8aya7v1s', 'x25t9tqani','q7air0bum2','ypfvhtq8te','ejxevz3a1r','1zo6zc2pxt',
             'strciewuws','lhos4naagl','djlzvlq6tj', 'theplumlinegs', '3eyf3nir4b','cbewjsq1a3','lvmjz9ax0u',
             'dw0zkytyft','sybl47cszn','6sdcyiw4kt','¼ï','yqf6exhm7x','cored8rfl2','6xjxeg1gss','dbvwkddesd',
             'ncmsf4fqpr','twunktgbnb','ur0eetseno','ghqbca7yii','cbqrst4ln4','c3zikdtowc','6snvq0dzxn','ekfrktnvuy',
             'k2jakipfji','œthe ','p1fh8jmmfa','vhmv7qoutk','mkuhbegzqs','ajic3flnki','mvjbs44atr',
             'wakqmkdpxa','e0bup1k83z','ðÿ','ºðÿ','µðÿ','eqmwv1xbim','hlz48rlkif','td0rycwn8c','vs4mnwxtei','75wozgjqop',
             'e1q36nkt8g','u8inojtf6d','rmq1a5bdon','5cvnmhnmuh','pdg7vqqv6m','s0s6xqrjsc','5cvnmhnmuh','wlxkoisstg',
             'tmndnpbj3m','dnzrzikxhd','4qckkpbtcr','x8psdeb2ur','fejgjt4xp9','evxfqavnfs','aty8r3kns2','pdg7vqqv6m','nqhi7xopmw',
             'lhos4naagl','32tfova4ov','zkyoioor62','np7kyhglsv','km0zoaulyh','kwvmqvelri','pirhr7layt',
             'v3aoj9ruh4','https','cg4dzhhbrv','qojom54gy8','75wozgjqop','aty8r3kns2','nxrwer1gez','rvxcpafi2a','vb0ao3s18d',
             'qggwewuvek','ddi1ywi7yz','r5nxc9ooa4','6lt9mlaj86','1jb53segv4','vhmv7qoutk','i7h4ryin3h',
             'aql4jlvndh','yfv0wijgby','nonhjywp4j','zomixteljq','iqum1rfqso','2nl6slwnmh','qejlzzgjdk',
            ...


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Most used @tags
Can you write Python code for it?

# frequency plot for the most used @ tags
df = pd.DataFrame(at_tags_top20, columns=['@ Tag', 'frequency'])
ax = df.plot(kind='bar', x='@ Tag',legend=None, figsize=(14,10))
# change tick markes
plt.tick_params(top='off', bottom = 'on', left = 'on', right = 'off', labelleft = 'on', labelbottom = 'on')
#plt.xticks(rotation=45)
plt.ylabel('Frequency', fontsize=25)
plt.xlabel('@Tag', fontsize=25)
plt.tight_layout()
plt.show()


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Part 1: Data Wrangle
Load and transform the data for analysis

    The first step in this analysis is to obtain and clean the data. The twitter data was obtained using the [Trump Twitter Archive](http://www.trumptwitterarchive.com/archive), the data is from 01/20/2017 - 03/02/2018 2:38 PM MST. I used the Federal Register's website to obtain all of the actions published by the Executive Office for the same time frame.

 The data included in this repo has already been wrangled using the code in this section. The webscraping and pdf conversion takes a bit of time, so unless you need a different time frame I would suggest just using the included pickeled datasets

Collect and Clean Twitter Data
Convert json data to pandas dataframe
Can you write Python code for it?

# load json twitter data
twitter_json = r'data/twitter_01_20_17_to_3-2-18.json'
# Convert to pandas dataframe
tweet_data = pd.read_json(twitter_json)


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Part 2 Conclusions:
<p>
    The informal and non-regular grammar used in tweets makes a direct comparison with documents published by the Executive Office, which uses formal vocabulary and grammar, difficult. Thus, I will use two different methods to analyze the data, topic determination with non-zero matrix factorization (NMF) and document similarity using cosine similarity. The cosine similarity metric compares the distance between document vectors, instead of direct word comparison. Higher cosine similarities between two documents indicate greater topic similarity.
</p>

Part 3: Analyze Twitter and Federal Data using NMF and Cosine Similarity

Part 3A: Transform data for analysis
Can you write Python code for it?

# load federal document data from pickle file
fed_reg_data = r'data/fed_reg_data.pickle'
fed_data = pd.read_pickle(fed_reg_data)
# load twitter data from csv
twitter_file_path = r'data\twitter_01_20_17_to_3-2-18.pickle'
twitter_data = pd.read_pickle(twitter_file_path)
# Change the index (date), to a column
fed_data['date'] = fed_data.index
twitter_data['date'] = twitter_data.index
# keep text strings and rename columns
fed = fed_data[['str_text', 'date']].rename({'str_text': 'texts'}, axis = 'columns')
tweet = twitter_data[['text', 'date']].rename({'text': 'texts'}, axis = 'columns')

# Add a label for the type of document (Tweet = 0, Fed = 1)
tweet['label'] = 0
fed['label'] = 1

# concatinate the dataframes
comb_text = pd.concat([fed,tweet])

# Re_index so that each doc has a unique id_number
comb_text = comb_text.reset_index()
comb_text['ID'] = range(0,len(comb_text))

# Look at the dataframe to make sure it works
comb_text = comb_text[['texts','date','label', 'ID']]
comb_text.head(3)


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Part 3B: Create a matrix representation of the text

 Computers cannot understand a text like humans, so in order to analyze text data, I first need to make every word a feature (column) in an array, where each document (row) is represented by a weighted* frequency of each word (column) they contain. An example text and array are shown below.

 Using Scikit Learn to create a word-frequency array:

 Define list of stop words (nonsense or non-meaninful words, such as 'the', 'a', 'of', 'q34fqwer3').

 Instantiate a tf-idf object (term frequency-inverse document frequency reweighting), that removes the stop words, and filters any word that appears in 99% of the documents

 Create a matrix representation of the documents

 Create list of the words each feature(column) represents

*Weighting the word frequencies lowers the importance of very frequently used, domain-specific, words
Can you write Python code for it?

# Instantiate TfidfVectorizer to remove common english words, and any word used in 99% of the documents
tfidf = TfidfVectorizer(stop_words = my_stop , max_df = 0.99)
# create matrix representation of all documents
text_mat = tfidf.fit_transform(comb_text.texts)
# make a list of feature words
words = tfidf.get_feature_names()


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Part 4: Use the cosine similarity results to explore how (or if) President Trump's tweets and official actions correlate

Part 4A: Find Twitter documents that have at least one federal document in its top 5 cosine similarity scores (and vice versa)

  Using the results of part 3C, find the types of documents are the most similar, then sum the labels (0=twitter, 1= federal document). If similar documents are a mix of tweets and federal documents, then the sum of their value will be either 1,2,3 or 4.

 Create a dataframe with the document ID number as the index and the document type label (tweet = 0, fed_doc = 1)

 Loop through each document in the dataframe and use the similarity dictionary to find the list of most similar document ID numbers and the sum of the similarity scores

 For each list of similar documents, sum the value of the document type labels. If the sum value is 1, 2, 3, or 4, that means there are both tweets and federal documents in the group

Can you write Python code for it?

# dataframe with document ID and labels
doc_label_df = comb_text[['label', 'ID']].copy().set_index('ID')
# inialize list for the sum of all similar documents label
label_sums =[]
similarity_score_sum = []
# loop through all of the documents
for doc_num in doc_label_df.index:
    # sum the similarity scores
    similarity_sum = similarity_dict[doc_num][1].sum()
    similarity_score_sum.append(similarity_sum)

    #find the list of similar documents
    similar_doc_ID_list = list(similarity_dict[doc_num][0])
    # loop through labels
    s_label = 0
    for ID_num in similar_doc_ID_list:
        # sum the label values for each similar document
        s_label = s_label + doc_label_df.loc[ID_num].label

    # append the sum of the labels for ONE document
    label_sums.append(s_label)
# add the similarity score sum to dataframe as separate column
doc_label_df['similarity_score_sum'] = similarity_score_sum

# add the similar document's summed label value to the dataframe as a separate column
doc_label_df['sum_of_labels'] = label_sums


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Part 4b: Group tweets that have similar federal documents (and vice versa)

  Isolate documents with mixed types of similar documents and high similarity scores

 Filter dataframe to include only top_similar_label_sums with a value of 1, 2, 3, or 4

 Filter again to only include groups with high combinded similarity scores

 Remove any duplicate groups

Can you write Python code for it?

# Filter dataframe for federal documents with similar tweets, and vice versa
df_filtered = doc_label_df[doc_label_df['sum_of_labels'] != 0][doc_label_df['sum_of_labels'] != 5].copy().reset_index()

# Look at the ones that have all top 5 documents with a cosine similarity score of 0.9 or above.
#The sum of scores need to be 4.6 or higher
similar_score_min = 4.6
highly_similar = df_filtered[df_filtered.similarity_score_sum >= similar_score_min]

# create a list of all the group lists
doc_groups = []
for doc_id in highly_similar.ID:
    doc_groups.append(sorted(list(similarity_dict[doc_id][0])))

# make the interior lists tuples, then make a set of them
unique_groups = list(set([tuple(x) for x in doc_groups]))


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Save clean twitter data as pickle
Can you write Python code for it?

# pickle data
tweet_pickle_path = r'data/twitter_01_20_17_to_3-2-18.pickle'
tweet_data.to_pickle(tweet_pickle_path)


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Scrape and format Data from Federal Register
Can you write Python code for it?

# Define the 2017 url that contains all of the Executive Office of the President's published documents
executive_office_url_2017 = r'https://www.federalregister.gov/index/2017/executive-office-of-the-president'
executive_office_url_2018 = r'https://www.federalregister.gov/index/2018/executive-office-of-the-president'
# scrape all urls for pdf documents published in 2017 and 2018 by the U.S.A. Executive Office
pdf_urls= []
for url in [executive_office_url_2017,executive_office_url_2018]:
    response = requests.get(url)
    pattern = re.compile(r'https:.*\.pdf')
    pdfs = re.findall(pattern, response.text)
    pdf_urls.append(pdfs)

# writes all of the pdfs to the data folder
start = 'data/'
end = '.pdf'
num = 0
for i in range(0,(len(pdf_urls))):
    for url in pdf_urls[i]:
        ver = str(num)
        pdf_path = start + ver + end
        r = requests.get(url)
        file = open(pdf_path, 'wb')
        file.write(r.content)
        file.close()
        num = num + 1


I am working on the file "NLP-comparing-tweets-fed_docs.ipynb".
The first task is:
Topic analysis using NMF

 Deconstruct the text matrix with non-zero matrix factorization (NMF) into discreet components:

 Instantiate NMF model with 200 components and initialized with Nonnegative Double Singular Value Decomposition (NNDSVD, better for sparseness)

 Fit(learn the NMF model for the tf-idf matrix) model

 Transform the model, which applies the fit to the matirix

 Make a dataframe with the NMF components for each word

Can you write Python code for it?

# instantiate model
NMF_model = NMF(n_components=200 , init = 'nndsvd')
# fit the model
NMF_model.fit(text_mat)
# transform the text frequecy matrix using the fitted NMF model
nmf_features = NMF_model.transform(text_mat)
# create a dataframe with words as a columns, NMF components as rows
components_df = pd.DataFrame(NMF_model.components_, columns = words)


I am working on the file "NLP/00-Multivariate_LSTM/notebooks/02_Predictive_Maintenance_using_multivariate_LSTM.ipynb".
The first task is:
Binary classification
Predict if an asset will fail within certain time frame (e.g. cycles)
Can you write Python code for it?

import keras
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import os

# Setting seed for reproducibility
np.random.seed(1234)
PYTHONHASHSEED = 0

from sklearn import preprocessing
from sklearn.metrics import confusion_matrix, recall_score, precision_score
from keras.models import Sequential,load_model
from keras.layers import Dense, Dropout, LSTM

# define path to save model
model_path = 'binary_model.h5'


I am working on the file "NLP/00-Multivariate_LSTM/notebooks/02_Predictive_Maintenance_using_multivariate_LSTM.ipynb".
The first task is:
Data Ingestion
Can you write Python code for it?

# read training data - It is the aircraft engine run-to-failure data.
train_df = pd.read_csv('PM_train.txt', sep=" ", header=None)
train_df.drop(train_df.columns[[26, 27]], axis=1, inplace=True)
train_df.columns = ['id', 'cycle', 'setting1', 'setting2', 'setting3', 's1', 's2', 's3',
                     's4', 's5', 's6', 's7', 's8', 's9', 's10', 's11', 's12', 's13', 's14',
                     's15', 's16', 's17', 's18', 's19', 's20', 's21']

train_df = train_df.sort_values(['id','cycle'])

# read test data - It is the aircraft engine operating data without failure events recorded.
test_df = pd.read_csv('PM_test.txt', sep=" ", header=None)
test_df.drop(test_df.columns[[26, 27]], axis=1, inplace=True)
test_df.columns = ['id', 'cycle', 'setting1', 'setting2', 'setting3', 's1', 's2', 's3',
                     's4', 's5', 's6', 's7', 's8', 's9', 's10', 's11', 's12', 's13', 's14',
                     's15', 's16', 's17', 's18', 's19', 's20', 's21']

# read ground truth data - It contains the information of true remaining cycles for each engine in the testing data.
truth_df = pd.read_csv('PM_truth.txt', sep=" ", header=None)
truth_df.drop(truth_df.columns[[1]], axis=1, inplace=True)


I am working on the file "NLP/00-Multivariate_LSTM/notebooks/02_Predictive_Maintenance_using_multivariate_LSTM.ipynb".
The first task is:
Data Preprocessing
Can you write Python code for it?

#######
# TRAIN
#######
# Data Labeling - generate column RUL(Remaining Usefull Life or Time to Failure)
rul = pd.DataFrame(train_df.groupby('id')['cycle'].max()).reset_index()
rul.columns = ['id', 'max']
train_df = train_df.merge(rul, on=['id'], how='left')
train_df['RUL'] = train_df['max'] - train_df['cycle']
train_df.drop('max', axis=1, inplace=True)
# generate label columns for training data
# we will only make use of "label1" for binary classification,
# while trying to answer the question: is a specific engine going to fail within w1 cycles?
w1 = 30
w0 = 15
train_df['label1'] = np.where(train_df['RUL'] <= w1, 1, 0 )
train_df['label2'] = train_df['label1']
train_df.loc[train_df['RUL'] <= w0, 'label2'] = 2

# MinMax normalization (from 0 to 1)
train_df['cycle_norm'] = train_df['cycle']
cols_normalize = train_df.columns.difference(['id','cycle','RUL','label1','label2'])
min_max_scaler = preprocessing.MinMaxScaler()
norm_train_df = pd.DataFrame(min_max_scaler.fit_transform(train_df[cols_normalize]),
                             columns=cols_normalize,
                             index=train_df.index)
join_df = train_df[train_df.columns.difference(cols_normalize)].join(norm_train_df)
train_df = join_df.reindex(columns = train_df.columns)

######
# TEST
######
# MinMax normalization (from 0 to 1)
test_df['cycle_norm'] = test_df['cycle']
norm_test_df = pd.DataFrame(min_max_scaler.transform(test_df[cols_normalize]),
                            columns=cols_normalize,
                            index=test_df.index)
test_join_df = test_df[test_df.columns.difference(cols_normalize)].join(norm_test_df)
test_df = test_join_df.reindex(columns = test_df.columns)
test_df = test_df.reset_index(drop=True)

# We use the ground truth dataset to generate labels for the test data.
# generate column max for test data
rul = pd.DataFrame(test_df.groupby('id')['cycle'].max()).reset_index()
rul.columns = ['id', 'max']
truth_df.columns = ['more']
truth_df['id'] = truth_df.index + 1
truth_df['max'] = rul['max'] +...


##################################
# Data Preprocessing
##################################

#######
# TRAIN
#######
# Data Labeling - generate column RUL(Remaining Usefull Life or Time to Failure)
rul = pd.DataFrame(train_df.groupby('id')['cycle'].max()).reset_index()
rul.columns = ['id', 'max']
train_df = train_df.merge(rul, on=['id'], how='left')
train_df['RUL'] = train_df['max'] - train_df['cycle']
train_df.drop('max', axis=1, inplace=True)

# generate label columns for training data
# we will only make use of "label1" for binary classification,
# while trying to answer the question: is a specific engine going to fail within w1 cycles?
w1 = 30
w0 = 15
train_df['label1'] = np.where(train_df['RUL'] <= w1, 1, 0 )
train_df['label2'] = train_df['label1']
train_df.loc[train_df['RUL'] <= w0, 'label2'] = 2

# MinMax normalization (from 0 to 1)
train_df['cycle_norm'] = train_df['cycle']
cols_normalize = train_df.columns.difference(['id','cycle','RUL','label1','label2'])
min_max_scaler = preprocessing.MinMaxScaler()
norm_train_df = pd.DataFrame(min_max_scaler.fit_transform(train_df[cols_normalize]),
                             columns=cols_normalize,
                             index=train_df.index)
join_df = train_df[train_df.columns.difference(cols_normalize)].join(norm_train_df)
train_df = join_df.reindex(columns = train_df.columns)

#train_df.to_csv('PredictiveManteinanceEngineTraining.csv', encoding='utf-8',index = None)

######
# TEST
######
# MinMax normalization (from 0 to 1)
test_df['cycle_norm'] = test_df['cycle']
norm_test_df = pd.DataFrame(min_max_scaler.transform(test_df[cols_normalize]),
                            columns=cols_normalize,
                            index=test_df.index)
test_join_df = test_df[test_df.columns.difference(cols_normalize)].join(norm_test_df)
test_df = test_join_df.reindex(columns = test_df.columns)
test_df = test_df.reset_index(drop=True)
print(test_df.head())

# We use the ground truth dataset to generate labels for the test data.
# generate column max for test...


I am working on the file "NLP/00-Multivariate_LSTM/notebooks/02_Predictive_Maintenance_using_multivariate_LSTM.ipynb".
The first task is:
Evaluate on Validation set
Can you write Python code for it?

# We pick the last sequence for each id in the test data
seq_array_test_last = [test_df[test_df['id']==id][sequence_cols].values[-sequence_length:]
                       for id in test_df['id'].unique() if len(test_df[test_df['id']==id]) >= sequence_length]

seq_array_test_last = np.asarray(seq_array_test_last).astype(np.float32)
#print("seq_array_test_last")
#print(seq_array_test_last)
#print(seq_array_test_last.shape)

# Similarly, we pick the labels
#print("y_mask")
y_mask = [len(test_df[test_df['id']==id]) >= sequence_length for id in test_df['id'].unique()]
label_array_test_last = test_df.groupby('id')['RUL'].nth(-1)[y_mask].values
label_array_test_last = label_array_test_last.reshape(label_array_test_last.shape[0],1).astype(np.float32)
#print(label_array_test_last.shape)
#print("label_array_test_last")
#print(label_array_test_last)

# if best iteration's model was saved then load and use it
if os.path.isfile(model_path):
    estimator = load_model(model_path,custom_objects={'r2_keras': r2_keras})

    # test metrics
    scores_test = estimator.evaluate(seq_array_test_last, label_array_test_last, verbose=2)
    print('\nMAE: {}'.format(scores_test[1]))
    print('\nR^2: {}'.format(scores_test[2]))

    y_pred_test = estimator.predict(seq_array_test_last)
    y_true_test = label_array_test_last

    test_set = pd.DataFrame(y_pred_test)
    test_set.to_csv('submit_test.csv', index = None)

    # Plot in blue color the predicted data and in green color the
    # actual data to verify visually the accuracy of the model.
    fig_verify = plt.figure(figsize=(10, 5))
    plt.plot(y_pred_test, color="blue")
    plt.plot(y_true_test, color="green")
    plt.title('prediction')
    plt.ylabel('value')
    plt.xlabel('row')
    plt.legend(['predicted', 'actual data'], loc='upper left')
    plt.show()
    fig_verify.savefig("model_regression_verify.png")


I am working on the file "NLP/00-Multivariate_LSTM/notebooks/02_Predictive_Maintenance_using_multivariate_LSTM.ipynb".
The first task is:
Model Evaluation on Test set
Can you write Python code for it?

# summarize history for R^2
fig_acc = plt.figure(figsize=(10, 10))
plt.plot(history.history['r2_keras'])
plt.plot(history.history['val_r2_keras'])
plt.title('model r^2')
plt.ylabel('R^2')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
fig_acc.savefig("model_r2.png")

# summarize history for MAE
fig_acc = plt.figure(figsize=(10, 10))
plt.plot(history.history['mean_absolute_error'])
plt.plot(history.history['val_mean_absolute_error'])
plt.title('model MAE')
plt.ylabel('MAE')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
fig_acc.savefig("model_mae.png")

# summarize history for Loss
fig_acc = plt.figure(figsize=(10, 10))
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
fig_acc.savefig("model_regression_loss.png")

# training metrics
scores = model.evaluate(seq_array, label_array, verbose=1, batch_size=200)
print('\nMAE: {}'.format(scores[1]))
print('\nR^2: {}'.format(scores[2]))

y_pred = model.predict(seq_array,verbose=1, batch_size=200)
y_true = label_array

test_set = pd.DataFrame(y_pred)
test_set.to_csv('submit_train.csv', index = None)


I am working on the file "NLP/01-One_hot_encoding/notebooks/One_hot_encoding.ipynb".
The first task is:
Binary encoding:
Can you write Python code for it?

onehot_encoder = OneHotEncoder(sparse=False)
label_encoded = label_encoded.reshape(len(label_encoded), 1)
onehot_encoded = onehot_encoder.fit_transform(label_encoded)
print(onehot_encoded)


I am working on the file "NLP/01-One_hot_encoding/notebooks/One_hot_encoding.ipynb".
The first task is:
Define example :
Can you write Python code for it?

data = ['cold',
        'cold',
        'warm',
        'cold',
        'hot',
        'hot',
        'warm',
        'cold',
        'warm',
        'hot']
values = array(data)
print(values)


I am working on the file "NLP/01-One_hot_encoding/notebooks/One_hot_encoding.ipynb".
The first task is:
Define universe of possible input values:
Can you write Python code for it?

alphabet = 'abcdefghijklmnopqrstuvwxyz '


I am working on the file "NLP/01-One_hot_encoding/notebooks/One_hot_encoding.ipynb".
The first task is:
Integer encoding:
Can you write Python code for it?

label_encoder = LabelEncoder()
label_encoded = label_encoder.fit_transform(values)
print(label_encoded)


I am working on the file "NLP/01-One_hot_encoding/notebooks/One_hot_encoding.ipynb".
The first task is:
Invert first example:
Can you write Python code for it?

inverted = label_encoder.inverse_transform([argmax(onehot_encoded[0, :])])


I am working on the file "NLP/01-One_hot_encoding/notebooks/One_hot_encoding.ipynb".
The first task is:
Part 02b -- One-hot encode using keras:
Importing libraries:
Can you write Python code for it?

from numpy import array
from numpy import argmax
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder
from keras.utils import to_categorical


I am working on the file "NLP/01-One_hot_encoding/notebooks/One_hot_encoding.ipynb".
The first task is:
Part 02c -- One-hot encode using keras for numerical categories:
Can you write Python code for it?

from numpy import array
from numpy import argmax
from keras.utils import to_categorical
# define example
data = [1, 3, 2, 0, 3, 2, 2, 1, 0, 1]
data = array(data)
print(data)
# one hot encode
encoded = to_categorical(data)
print(encoded)
# invert encoding
inverted = argmax(encoded[0])

print(inverted)


I am working on the file "NLP/03-Natural_language_generation/src/03_Alice_NLP_generator.ipynb".
The first task is:
Building the LSTM model:
Adding empty model,
Adding first layer --> an LSTM unit with 128 nodes and shape of 40 by length of characters/punctuations in the document,
Adding another layer --> a regular densely-connected NN layer with the number of nodes being the number of unique characters/punctuation in the document,
Adding another layer --> an activation function to the output, with a softmax activation.
Can you write Python code for it?

# build the model: a single LSTM
print('Build model...')
model = Sequential()
model.add(LSTM(128, input_shape=(maxlen, len(chars))))
model.add(Dense(len(chars)))
model.add(Activation('softmax'))


I am working on the file "NLP/03-Natural_language_generation/src/03_Alice_NLP_generator.ipynb".
The first task is:
Cutting the document into semi-redundant sentences, where each element in the sentences list contain 40 sentences that overlap with the previous element's sentences. Also, storing character in each next_chars array's elements, where the current element is the 40th character after the previous character.
Can you write Python code for it?

# cut the text in semi-redundant sequences of maxlen characters
maxlen = 40
step = 3
sentences = []
next_chars = []
for i in range(0, len(text) - maxlen, step):
    sentences.append(text[i: i + maxlen])
    next_chars.append(text[i + maxlen])
print('nb sequences:', len(sentences))


I am working on the file "NLP/03-Natural_language_generation/src/03_Alice_NLP_generator.ipynb".
The first task is:
Defining sample function, which is a helper function to sample an index from a probability array.
Can you write Python code for it?

def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)


I am working on the file "NLP/03-Natural_language_generation/src/03_Alice_NLP_generator.ipynb".
The first task is:
Instantiating a RMSProp optimizer and compiling our LSTM model with the optimizer. This optimizer is usually a good choice for RNNs. 'lr' is the learning rate.
model.compile means to start the learning process with the optimizer.
Can you write Python code for it?

optimizer = RMSprop(lr=0.01)
model.compile(loss='categorical_crossentropy', optimizer=optimizer)


I am working on the file "NLP/03-Natural_language_generation/src/03_Alice_NLP_generator.ipynb".
The first task is:
Loading and reading Alice.txt corpus, saving characters (unique alphabet and punctuation characters in corpus) in array, and making dictionary associating each character with it's position in the character array (making two dictionaries where the key and position are either the key or value)
Can you write Python code for it?

path = get_file('Alice.txt', origin='https://github.com/rahulremanan/python_tutorial/blob/master/Alice.txt')
text = open(path).read().lower()
print('corpus length:', len(text))

chars = sorted(list(set(text)))
print('total chars:', len(chars))
char_indices = dict((c, i) for i, c in enumerate(chars))
indices_char = dict((i, c) for i, c in enumerate(chars))


I am working on the file "NLP/03-Natural_language_generation/src/03_Alice_NLP_generator.ipynb".
The first task is:
Making X boolean (false) array with a shape of the length of the sentences by the step (40) by the length of the unique characters/punctuation in the document.
Making y boolean (false) array with a shape of the length of the sentences by the length of the unique characters/punctuation in the document.
Then, going through each sentence and character in the sentence, storing a 1 (converting false to true) in the respective sentence and characters in X and y.
Can you write Python code for it?

print('Vectorization...')
X = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool)
y = np.zeros((len(sentences), len(chars)), dtype=np.bool)
for i, sentence in enumerate(sentences):
    for t, char in enumerate(sentence):
        X[i, t, char_indices[char]] = 1
    y[i, char_indices[next_chars[i]]] = 1


I am working on the file "NLP/04-Character_embedding/src/04_Char_embedding.ipynb".
The first task is:
Defining helper functions.
Can you write Python code for it?

# test code to sample on 10% for functional model testing

def random_subset(X, y, p=0.1):

    idx = np.random.randint(X.shape[0], size=int(X.shape[0] * p))
    X = X[idx, :]
    y = y[idx]
    return (X, y)

# https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html
def generate_embedding_matrix(embeddings_path):
    print('Processing pretrained character embeds...')
    embedding_vectors = {}
    with open(embeddings_path, 'r') as f:
        for line in f:
            line_split = line.strip().split(" ")
            vec = np.array(line_split[1:], dtype=float)
            char = line_split[0]
            embedding_vectors[char] = vec

    embedding_matrix = np.zeros((len(chars), 300))
    #embedding_matrix = np.random.uniform(-1, 1, (len(chars), 300))
    for char, i in char_indices.items():
        #print ("{}, {}".format(char, i))
        embedding_vector = embedding_vectors.get(char)
        if embedding_vector is not None:
            embedding_matrix[i] = embedding_vector

    # Use PCA from sklearn to reduce 300D -> 50D
    if use_pca:
        pca = PCA(n_components=embedding_dim)
        pca.fit(embedding_matrix)
        embedding_matrix_pca = np.array(pca.transform(embedding_matrix))
        embedding_matrix_result = embedding_matrix_pca
        print (embedding_matrix_pca)
        print (embedding_matrix_pca.shape)
    else:
        embedding_matrix_result = embedding_matrix
    return embedding_matrix_result

def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds + 1e-6) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)


I am working on the file "NLP/04-Character_embedding/src/04_Char_embedding.ipynb".
The first task is:
Hello World! Python Workshops @ Think Coffee
3-5pm, 7/30/17
Day 3, Alice NLP generator
@python script author (original content): Rahul
@jupyter notebook converted tutorial author: Nick Giangreco
Ntbk of python script in same directory. Building an RNN based on Lewis Carrol's Alice in Wonderland text.
Importing modules
Can you write Python code for it?

from __future__ import print_function
from keras.models import Model
from keras.layers import Dense, Activation, Embedding
from keras.layers import LSTM, Input
from keras.layers.merge import concatenate
from keras.optimizers import RMSprop, Adam
from keras.utils.data_utils import get_file
from keras.layers.normalization import BatchNormalization
from keras.callbacks import Callback, ModelCheckpoint
from sklearn.decomposition import PCA
from keras.utils import plot_model
import numpy as np
import random
import sys
import csv
import os
import h5py
import time


I am working on the file "NLP/04-Character_embedding/src/04_Char_embedding.ipynb".
The first task is:
Making X boolean (false) array with a shape of the length of the sentences by the step (40) by the length of the unique characters/punctuation in the document.
Making y boolean (false) array with a shape of the length of the sentences by the length of the unique characters/punctuation in the document.
Then, going through each sentence and character in the sentence, storing a 1 (converting false to true) in the respective sentence and characters in X and y.
Can you write Python code for it?

print('Vectorization...')
X = np.zeros((len(sentences), maxlen), dtype=np.int)
y = np.zeros((len(sentences), len(chars)), dtype=np.bool)
for i, sentence in enumerate(sentences):
    for t, char in enumerate(sentence):
        X[i, t] = char_indices[char]
    y[i, char_indices[next_chars[i]]] = 1


I am working on the file "NLP/04-Character_embedding/src/04_Char_embedding.ipynb".
The first task is:
Model training. Use one epoch instead of ten.
Can you write Python code for it?

ep = 1

start_time = time.time()
for iteration in range(1, 20):
    print()
    print('-' * 50)
    print('Iteration', iteration)

    logger = BatchLossLogger()
    # X_train, y_train = random_subset(X, y)
    # history = model.fit(X_train, [y_train, y_train], batch_size=batch_size,
    #                     epochs=1, callbacks=[logger, checkpointer])
    history = model.fit(X, [y, y], batch_size=batch_size,
                        epochs=ep, callbacks=[logger, checkpointer])
    loss = str(history.history['main_out_loss'][-1]).replace(".", "_")

    f2 = open('./output/iter-{:02}-{:.6}.txt'.format(iteration, loss), 'w')

    start_index = random.randint(0, len(text) - maxlen - 1)

    for diversity in [0.2, 0.5, 1.0, 1.2]:
        print()
        print('----- diversity:', diversity)
        f2.write('----- diversity:' + ' ' + str(diversity) + '\n')

        generated = ''
        sentence = text[start_index: start_index + maxlen]
        generated += sentence
        print('----- Generating with seed: "' + sentence + '"')
        f2.write('----- Generating with seed: "' + sentence + '"' + '\n---\n')
        sys.stdout.write(generated)

        for i in range(1200):
            x = np.zeros((1, maxlen), dtype=np.int)
            for t, char in enumerate(sentence):
                x[0, t] = char_indices[char]

            preds = model.predict(x, verbose=0)[0][0]
            next_index = sample(preds, diversity)
            next_char = indices_char[next_index]

            generated += next_char
            sentence = sentence[1:] + next_char

            sys.stdout.write(next_char)
            sys.stdout.flush()
        f2.write(generated + '\n')
        print()
    f2.close()

    # Write embeddings for current characters to file
    # The second layer has the embeddings.

    embedding_weights = model.layers[1].get_weights()[0]
    f3 = open('./output/char-embeddings.txt', 'w')
    for char in char_indices:
        if ord(char) < 128:
            embed_vector = embedding_weights[char_indices[char], :]
    ...


I am working on the file "NLP/04-Character_embedding/src/04_Char_embedding.ipynb".
The first task is:
Setting params for model setup and build.
Can you write Python code for it?

embeddings_path = "./glove.840B.300d-char.txt" # http://nlp.stanford.edu/data/glove.840B.300d.zip
embedding_dim = 300
batch_size = 32
use_pca = False
lr = 0.001
lr_decay = 1e-4
maxlen = 300
consume_less = 2   # 0 for cpu, 2 for gpu


I am working on the file "NLP/1-Basic skills/jieba中文处理.ipynb".
The first task is:
ChineseAnalyzer for Whoosh 搜索引擎
from jieba.analyse import ChineseAnalyzer
待验证
Can you write Python code for it?

命令行分词
使用示例：python -m jieba news.txt > cut_result.txt
命令行选项（翻译）：
使用: python -m jieba [options] filename

结巴命令行界面。

固定参数:
  filename              输入文件

可选参数:
  -h, --help            显示此帮助信息并退出
  -d [DELIM], --delimiter [DELIM]
                        使用 DELIM 分隔词语，而不是用默认的' / '。
                        若不指定 DELIM，则使用一个空格分隔。
  -p [DELIM], --pos [DELIM]
                        启用词性标注；如果指定 DELIM，词语和词性之间
                        用它分隔，否则用 _ 分隔
  -D DICT, --dict DICT  使用 DICT 代替默认词典
  -u USER_DICT, --user-dict USER_DICT
                        使用 USER_DICT 作为附加词典，与默认词典或自定义词典配合使用
  -a, --cut-all         全模式分词（不支持词性标注）
  -n, --no-hmm          不使用隐含马尔可夫模型
  -q, --quiet           不输出载入信息到 STDERR
  -V, --version         显示版本信息并退出

如果没有指定文件名，则使用标准输入。

--help 选项输出：
$> python -m jieba --help
Jieba command line interface.

positional arguments:
  filename              input file

optional arguments:
  -h, --help            show this help message and exit
  -d [DELIM], --delimiter [DELIM]
                        use DELIM instead of ' / ' for word delimiter; or a
                        space if it is used without DELIM
  -p [DELIM], --pos [DELIM]
                        enable POS tagging; if DELIM is specified, use DELIM
                        instead of '_' for POS delimiter
  -D DICT, --dict DICT  use DICT as dictionary
  -u USER_DICT, --user-dict USER_DICT
                        use USER_DICT together with the default dictionary or
                        DICT (if specified)
  -a, --cut-all         full pattern cutting (ignored with POS tagging)
  -n, --no-hmm          don't use the Hidden Markov Model
  -q, --quiet           don't print loading messages to stderr
  -V, --version         show program's version number and exit

If no filename specified, use STDIN instead.


I am working on the file "NLP/1-Basic skills/jieba中文处理.ipynb".
The first task is:
Tokenize：返回词语在原文的起止位置
注意，输入参数只接受 unicode
Can you write Python code for it?

print("这是默认模式的tokenize")
result = jieba.tokenize(u'自然语言处理非常有用')
for tk in result:
    print("%s\t\t start: %d \t\t end:%d" % (tk[0],tk[1],tk[2]))

print(u"\n-----------我是神奇的分割线------------\n")

print("这是搜索模式的tokenize")
result = jieba.tokenize(u'自然语言处理非常有用', mode='search')
for tk in result:
    print("%s\t\t start: %d \t\t end:%d" % (tk[0],tk[1],tk[2]))


I am working on the file "NLP/1-Basic skills/jieba中文处理.ipynb".
The first task is:
jieba.lcut以及jieba.lcut_for_search直接返回 list
Can you write Python code for it?

result_lcut = jieba.lcut("小明硕士毕业于中国科学院计算所，后在哈佛大学深造")
print(result_lcut)
print(" ".join(result_lcut))
print(" ".join(jieba.lcut_for_search("小明硕士毕业于中国科学院计算所，后在哈佛大学深造")))


I am working on the file "NLP/1-Basic skills/jieba中文处理.ipynb".
The first task is:
关于TF-IDF 算法的关键词抽取补充

关键词提取所使用逆向文件频率（IDF）文本语料库可以切换成自定义语料库的路径
用法： jieba.analyse.set_idf_path(file_name) # file_name为自定义语料库的路径
关键词提取所使用停止词（Stop Words）文本语料库可以切换成自定义语料库的路径
用法： jieba.analyse.set_stop_words(file_name) # file_name为自定义语料库的路径

基于 TextRank 算法的关键词抽取¶

jieba.analyse.textrank(sentence, topK=20, withWeight=False, allowPOS=('ns', 'n', 'vn', 'v')) 直接使用，接口相同，注意默认过滤词性。
jieba.analyse.TextRank() 新建自定义 TextRank 实例

基本思想:

将待抽取关键词的文本进行分词
以固定窗口大小(默认为5，通过span属性调整)，词之间的共现关系，构建图
计算图中节点的PageRank，注意是无向带权图
Can you write Python code for it?

import jieba.analyse as analyse
lines = open('人工智能火了-人类有些尴尬了.txt').read()
print("  ".join(analyse.textrank(lines, topK=20, withWeight=False, allowPOS=('ns', 'n', 'vn', 'v'))))
print("---------------------我是分割线----------------")
print("  ".join(analyse.textrank(lines, topK=20, withWeight=False, allowPOS=('ns', 'n'))))


I am working on the file "NLP/1-Basic skills/jieba中文处理.ipynb".
The first task is:
关键词提取¶
基于 TF-IDF 算法的关键词抽取
import jieba.analyse
 - jieba.analyse.extract_tags(sentence, topK=20, withWeight=False, allowPOS=())
   - sentence 为待提取的文本
   - topK 为返回几个 TF/IDF 权重最大的关键词，默认值为 20
   - withWeight 为是否一并返回关键词权重值，默认值为 False
   - allowPOS 仅包括指定词性的词，默认值为空，即不筛选
Can you write Python code for it?

import jieba.analyse as analyse
lines = open('人工智能火了-人类有些尴尬了.txt').read()
print(type(lines))
seg_list = analyse.extract_tags(lines,topK=15,withWeight=False,allowPOS=())
#print(seg_list)
text = " ".join(seg_list)
print(text)


I am working on the file "NLP/1-Basic skills/jieba中文处理.ipynb".
The first task is:
添加用户自定义词典
很多时候我们需要针对自己的场景进行分词，会有一些领域内的专有词汇。
- 1.可以用jieba.load_userdict(file_name)加载用户字典
- 2.少量的词汇可以自己用下面方法手动添加：
  - 用 add_word(word, freq=None, tag=None) 和 del_word(word) 在程序中动态修改词典
  - 用 suggest_freq(segment, tune=True) 可调节单个词语的词频，使其能（或不能）被分出来。
Can you write Python code for it?

print('/'.join(jieba.cut('如果放到旧字典中将出错。', HMM=False)))

jieba.suggest_freq(('中', '将'), True)

print('/'.join(jieba.cut('如果放到旧字典中将出错。', HMM=False)))


I am working on the file "NLP/1-Basic skills/jieba中文处理.ipynb".
The first task is:
词性标注

jieba.posseg.POSTokenizer(tokenizer=None) 新建自定义分词器，tokenizer 参数可指定内部使用的 jieba.Tokenizer 分词器。jieba.posseg.dt 为默认词性标注分词器。
标注句子分词后每个词的词性，采用和 ictclas 兼容的标记法。
具体的词性对照表参见计算所汉语词性标记集
Can you write Python code for it?

import jieba.posseg as pseg
words = pseg.cut("我爱人工智能")
print(words)
for word,flag in words:
    print("%s %s" % (word,flag))


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
Bidirectional GRU
Create a recurrent neural network model
Can you write Python code for it?

def model_BiDir_GRU():
  model = Sequential()
  model.add(Embedding(max_features, 100, input_length=max_words))
  model.add(SpatialDropout1D(0.25))
  model.add(Bidirectional(GRU(128)))
  model.add(Dropout(0.5))
  model.add(Dense(5, activation='softmax'))

  return model


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
Build and compile the CNN +GRU model
Can you write Python code for it?

model3 = model_CNN_GRU()

model3.compile(loss='categorical_crossentropy',optimizer=Adam(lr=0.001),metrics=['accuracy'])
model3.summary()


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
Build and compile the CNN model
Can you write Python code for it?

model2 = model_CNN()

model2.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
model2.summary()


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
CNN +GRUModel
Create a recurrent neural network model
Can you write Python code for it?

def model_CNN_GRU():
  model= Sequential()
  model.add(Embedding(max_features,100,input_length=max_words))
  model.add(Conv1D(64,kernel_size=3,padding='same',activation='relu'))
  model.add(MaxPooling1D(pool_size=2))
  model.add(Dropout(0.25))
  model.add(GRU(128,return_sequences=True))
  model.add(Dropout(0.3))
  model.add(Flatten())
  model.add(Dense(128,activation='relu'))
  model.add(Dropout(0.5))
  model.add(Dense(5,activation='softmax'))
  return model


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
CNN Model
Create a recurrent neural network model
Can you write Python code for it?

def model_CNN():
  model= Sequential()
  model.add(Embedding(max_features,100,input_length=max_words))
  model.add(Dropout(0.2))
  model.add(Conv1D(64,kernel_size=3,padding='same',activation='relu',strides=1))
  model.add(GlobalMaxPooling1D())
  model.add(Dense(128,activation='relu'))
  model.add(Dropout(0.2))
  model.add(Dense(num_classes,activation='softmax'))
  return model


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
Convert labels to categorical variables
Can you write Python code for it?


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
Create a recurrent neural network model
Can you write Python code for it?

def model_Glove():
  model = Sequential()
  model.add(Embedding(max_features, embed_dim, input_length=X_train.shape[1],weights=[embedding_matrix],trainable=True))
  model.add(SpatialDropout1D(0.25))
  model.add(Bidirectional(GRU(128,return_sequences=True)))
  model.add(Bidirectional(GRU(64,return_sequences=False)))
  model.add(Dropout(0.5))
  model.add(Dense(num_classes, activation='softmax'))

  return model


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
Create train-validation split for training the model
Can you write Python code for it?

X_train_text,X_val_text,y_train,y_val=train_test_split(train_text,y,test_size=0.2,stratify=y,random_state=123)
print(X_train_text.shape,y_train.shape)
print(X_val_text.shape,y_val.shape)

all_words=' '.join(X_train_text)
all_words=word_tokenize(all_words)
dist=FreqDist(all_words)
num_unique_word=len(dist)
num_unique_word


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
Download NLTK datasets
Specify the NLTK corpus as 'punkt' or 'all'
Can you write Python code for it?

if setup:
  nltk.download()

def clean_review(review_col):
    review_corpus=[]
    for i in range(0,len(review_col)):
        review=str(review_col[i])
        review=re.sub('[^a-zA-Z]',' ',review)
        #review=[stemmer.stem(w) for w in word_tokenize(str(review).lower())]
        review=[lemma.lemmatize(w) for w in word_tokenize(str(review).lower())]
        review=' '.join(review)
        review_corpus.append(review)
    return review_corpus

df['clean_review']=clean_review(df.Phrase.values)
df.head()

df_train=df[df.Sentiment!=-999]
print (df_train.shape)
df_train.head()

df_test=df[df.Sentiment==-999]
df_test.drop('Sentiment',axis=1,inplace=True)
print(df_test.shape)
df_test.head()

del df
gc.collect()

train_text=df_train.clean_review.values
test_text=df_test.clean_review.values
target=df_train.Sentiment.values


I am working on the file "NLP/10-Sentiment_analysis/notebook/Rotten_Tomatoes_movie_reviews_classifier.ipynb".
The first task is:
Download the Rotten Tomatoes movie reviews dataset
Can you write Python code for it?

! kaggle competitions download -c movie-review-sentiment-analysis-kernels-only

! kaggle datasets download -d terenceliu4444/glove6b100dtxt

! rm /root/.kaggle/kaggle.json

if setup:
  ! unzip -q /content/train.tsv.zip
  ! unzip -q /content/test.tsv.zip

if setup:
  ! unzip -q /content/glove6b100dtxt.zip