I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMIntroExtended.ipynb".
The first task is:
For what follows I'm going to use the george library. The main reason for this is that the matrix inversion in numpy is not very stable for large (> (10 x 10)) matrices.
Can you write Python code for it?

import george
from george import kernels

import emcee

# we need to define three functions:
# a log likelihood, a log prior & a log posterior.

# set the loglikelihood:
def lnlike2(p, x, y):

    # update kernel parameters:
    gp.kernel[:] = p

    # compute covariance matrix:
    gp.compute(x)

    # calculate the likelihood:
    ll = gp.lnlikelihood(y, quiet=True)

    # return
    return ll if np.isfinite(ll) else 1e25

# set the logprior
def lnprior2(p):

    # note that "p" contains the ln()
    # of the parameter values - set your
    # prior ranges appropriately!

    lnh,lnlam,lnsig = p

    # really crappy prior:
    if (-2.<lnh<5.) and (-1.<lnlam<10.) and (-30.<lnsig<0.):
        return 0.0

    return -np.inf

# set the logposterior:
def lnprob2(p, x, y):

    lp = lnprior2(p)

    return lp + lnlike2(p, x, y) if np.isfinite(lp) else -np.inf

# initiate george with the exponential squared kernel:
kernel = 1.0*kernels.ExpSquaredKernel(30.0)+kernels.WhiteKernel(1e-5)
gp = george.GP(kernel, mean=0.0)

# put all the data into a single array:
data = (x_train,y_train)

# set your initial guess parameters
# remember george keeps these in ln() form!
initial = gp.kernel[:]
print "Initial guesses: ",np.exp(initial)

# set the dimension of the prior volume
# (i.e. how many parameters do you have?)
ndim = len(initial)
print "Number of parameters: ",ndim

# The number of walkers needs to be more than twice
# the dimension of your parameter space... unless you're crazy!
nwalkers = 10

# perturb your inital guess parameters very slightly
# to get your starting values:
p0 = [np.array(initial) + 1e-4 * np.random.randn(ndim)
      for i in xrange(nwalkers)]

# initalise the sampler:
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob2, args=data)

# run a few samples as a burn-in:
print("Running burn-in")
p0, lnp, _ = sampler.run_mcmc(p0, 1000)
sampler.reset()

# take the highest likelihood point from the burn-in as a
# starting point and now begin your production...


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMIntroExtended.ipynb".
The first task is:
Let's plot this up:
Can you write Python code for it?

# mu and sig are currently lists - turn them into numpy arrays:
mu=np.array(mu);sig=np.array(sig)

# make some plots:

# left-hand plot
ax = pl.subplot(121)
pl.scatter(x_train,y_train)  # plot the training points
pl.plot(x1,y1,ls=':')        # plot the original data they were drawn from
pl.title("Input")

# right-hand plot
ax = pl.subplot(122)
pl.plot(x_test,mu,ls='-')     # plot the predicted values
pl.plot(x_test,y_test,ls=':') # plot the original values

# shade in the area inside a one standard deviation bound:
ax.fill_between(x_test,mu-sig,mu+sig,facecolor='lightgrey', lw=0, interpolate=True)
pl.title("Predicted")

pl.scatter(x_train,y_train)  # plot the training points

# display the plot:
pl.show()


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMIntroExtended.ipynb".
The first task is:
Make the covariance kernel a squared-exponential,
$k(x_1,x_2) = h^2 \exp{ \left( \frac{-(x_1 - x_2)^2}{\lambda^2} \right)}$,
just like Eq. 3.11 in Roberts et al. (2012).
http://www.robots.ox.ac.uk/~sjrob/Pubs/Phil.%20Trans.%20R.%20Soc.%20A-2013-Roberts-.pdf
Can you write Python code for it?

def cov_kernel(x1,x2,h,lam):
    """
    Squared-Exponential covariance kernel
    """
    k12 = h**2*np.exp(-1.*(x1 - x2)**2/lam**2)

    return k12


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMIntroExtended.ipynb".
The first task is:
Then let's import all the libraries we need...
Can you write Python code for it?

import numpy as np
import pylab as pl
from scipy import linalg as sl


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMIntroExtended.ipynb".
The first task is:
Using this kernel we can then recreate Fig. 5 from Roberts et al. (2012).
Can you write Python code for it?

# make an array of 200 evenly spaced positions between 0 and 20:
x1 = np.arange(0, 20.,0.01)

for i in range(0,3):

    h = 1.0

    if (i==0): lam = 0.1
    if (i==1): lam = 1.0
    if (i==2): lam = 5.0

    # make a covariance matrix:
    K = make_K(x1,h,lam)

    # five realisations:
    for j in range(0,5):

        # draw samples from a co-variate Gaussian distribution, N(0,K):
        y1 = np.random.multivariate_normal(np.zeros(len(x1)),K)

        tmp2 = '23'+str(i+3+1)
        pl.subplot(int(tmp2))
        pl.plot(x1,y1)

    tmp1 = '23'+str(i+1)
    pl.subplot(int(tmp1))
    pl.imshow(K)
    pl.title(r"$\lambda = $"+str(lam))

pl.show()


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMIntroExtended.ipynb".
The first task is:
We  can also extract the expectation value for each parameter and the individual confidence intervals:
Can you write Python code for it?


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMIntroExtended.ipynb".
The first task is:
We can use this kernel to calculate  the value of each element in our covariance matrix:
$\mathbf{K(x,x)} = \left(
\begin{array}{cccc}
k(x_1,x_1) & k(x_1,x_2) & ... & k(x_1,x_n) \
k(x_2,x_1) & k(x_2,x_2) & ... & k(x_2,x_n) \
\vdots & \vdots & \vdots & \vdots \
k(x_n,x_1) & k(x_n,x_2) & ... & k(x_n,x_n)
\end{array}
\right).$
We can then populate a covariance matrix, $K(\mathbf{x},\mathbf{x})$, for our data:
Can you write Python code for it?

def make_K(x, h, lam):

    """
    Make covariance matrix from covariance kernel
    """

    # for a data array of length x, make a covariance matrix x*x:
    K = np.zeros((len(x),len(x)))

    for i in range(0,len(x)):
        for j in range(0,len(x)):

            if (i==j):
                noise = 1e-5
            else:
                noise = 0.0

            # calculate value of K for each separation:
            K[i,j] = cov_kernel(x[i],x[j],h,lam) + noise**2

    return K


I am working on the file "TIARA/GaussianProcessModelling/NOTEBOOKS/GPMIntroExtended.ipynb".
The first task is:
[Note: Depending on  the selection of training points you might want to specify some axis ranges for these plots]
At this point we've come to the end of the material presented in the lecture.
--END--
Extra material
What if we don't know the values of the hyper-parameters in our covariance kernel? We can fit them directly from the data!
Can you write Python code for it?

# set number of training points
frac = 0.05
nx_training = int(len(x1)*frac)
print "Using ",nx_training," points."

# randomly select the training points:
tmp = np.random.uniform(low=0.0, high=2000.0, size=nx_training)
tmp = tmp.astype(int)

condition = np.zeros_like(x1)
for i in tmp: condition[i] = 1.0

y_train = y1[np.where(condition==1.0)]
x_train = x1[np.where(condition==1.0)]
y_test = y1[np.where(condition==0.0)]
x_test = x1[np.where(condition==0.0)]

pl.scatter(x_train,y_train)  # plot the training points
pl.plot(x1,y1,ls=':')        # plot the original data they were drawn from
pl.title("Training Data")
pl.show()


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
...and let's take a look at it:
Can you write Python code for it?


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
Let's see how that looks:
Can you write Python code for it?

pl.subplot(121),pl.imshow(cat, cmap = 'gray')
pl.title('Cat'), pl.xticks([]), pl.yticks([])
pl.subplot(122),pl.imshow(cat_spectrum, cmap = 'gray')
pl.title('Fourier Cat'), pl.xticks([]), pl.yticks([])
pl.show()


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
Next we'll dive straight in and Fourier Transform our cat. Note that there are a couple of steps to this, the first is the fft itself and the second is an fft "shift". This is necessary because of the way that the frequency space is ordered in the Fourier Transform.
Can you write Python code for it?

cat_squiggle = np.fft.fft2(cat)
cat_squiggle_shifted = np.fft.fftshift(cat_squiggle)
cat_spectrum = 20*np.log(np.abs(cat_squiggle_shifted))


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
Now let's filter out the large Fourier frequencies:
Can you write Python code for it?

cat_squiggle_lpf = np.copy(cat_squiggle_shifted)
cat_squiggle_lpf[np.where(filter_fnc==0.)] = 0.+0.*1j


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
Now we can set things up for filtering our Fourier Cat. We need to know (1) the dimenions of the image and (2) where the centre is.
Can you write Python code for it?

rows, cols = cat.shape
crow,ccol = rows/2 , cols/2


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
Open up the input image:
Can you write Python code for it?


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
This looks like:
Can you write Python code for it?

#pl.subplot(121),pl.imshow(cat, cmap = 'gray')
#pl.title('Cat'), pl.xticks([]), pl.yticks([])
pl.subplot(121),pl.imshow(20*np.log(np.abs(cat_squiggle_lpf)), cmap = 'gray')
pl.title('Filtered Fourier Cat'), pl.xticks([]), pl.yticks([])
pl.subplot(122),pl.imshow(cat_filtered_lpf)
pl.title('LPF Cat'), pl.xticks([]), pl.yticks([])
pl.show()


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
To start with, let's make a filter function that separates the inner most 40 x 40 pixels from everything else.
Can you write Python code for it?

filter_fnc = np.zeros(cat_squiggle_shifted.shape)
filter_fnc[crow-20:crow+20, ccol-20:ccol+20] = 1.0
pl.imshow(filter_fnc)
pl.show()


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
We can then Fourier transform this back into image space.
Can you write Python code for it?

cat_filtered = np.fft.ifftshift(cat_squiggle_hpf)
cat_filtered_hpf = np.fft.ifft2(cat_filtered)
cat_filtered_hpf = np.abs(cat_filtered_hpf)


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
We can then use this filter to, firstly, mask out the inner most 40 x 40 pixels in Fourier space. This removes our small Fourier frequencies, i.e. the large scale information in our image.
Can you write Python code for it?

cat_squiggle_hpf = np.copy(cat_squiggle_shifted)
cat_squiggle_hpf[np.where(filter_fnc==1.)] = 0.0+0*1j


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
We need to import some libraries:
Can you write Python code for it?

import numpy as np   # for array manipulation and the fft
import pylab as pl   # for plotting
import cv2           # for image file handling


I am working on the file "TIARA/RadioImaging/FourierCat.ipynb".
The first task is:
and Fourier transform that back into image space:
Can you write Python code for it?

cat_filtered = np.fft.ifftshift(cat_squiggle_lpf)
cat_filtered_lpf = np.fft.ifft2(cat_filtered)
cat_filtered_lpf = np.abs(cat_filtered_lpf)


I am working on the file "TIARA/Tutorial/KeplerLightCurve.ipynb".
The first task is:
I'm going to set bounds on the available parameters space, i.e. our prior volume, using the ranges taken from Table 4 of https://arxiv.org/pdf/1706.05459.pdf
Can you write Python code for it?

import scipy.optimize as op

# extract our initial guess at parameters
# from the george kernel and put it in a
# vector:
p0 = gp.kernel.vector

# set prior ranges
# Note that these are in *logarithmic* space
bnds = ((-20.,0.),(2.,8.),(0.,3.),(np.log(0.5),np.log(100.)),(-20.,0))

# run optimization:
results = op.minimize(nll, p0, method='L-BFGS-B', jac=grad_nll, bounds=bnds)

# print the value of the optimised parameters:
print np.exp(results.x)


I am working on the file "TIARA/Tutorial/KeplerLightCurve.ipynb".
The first task is:
In the paper there are two suggested kernels for modelling the covariance of the Kepler data (Eqs. 55 & 56). In the paper the authors fit Eq 56 - here we are going to fit Eq. 55.
We can do this using a combination of kernels from the george library.
Exponential Squared Kernel:
$$
k_1(x_i,x_j)=h_1 \exp(−\frac{(x_i-x_j)^2}{2 \sigma^2})
$$
Exp-Sine-Squared Kernel:
$$
k_2(x_i,x_j)=h_2 \exp \left( \Gamma \sin^2 \left[ \frac{\pi}{P}|x_i - x_j|\right] \right)
$$
Our combined kernel is therefore:
$$
k(x_i,x_j)=h \exp(−\frac{(x_i-x_j)^2}{2 \sigma^2})\exp \left( \Gamma \sin^2 \left[ \frac{\pi}{P}|x_i - x_j|\right] \right)
$$
where $h = h_1 h_2$.
However, following https://arxiv.org/pdf/1706.05459.pdf, we are also going to add a white noise kernel in:
$$
k_3(x_i,x_j)=c \delta_{ij}
$$
So our final kernel will be:
$$
k = (k_1k_2) + k_3
$$
Can you write Python code for it?

# h =1.0; sigma = 1.0; Gamma = 2.0/1.0^2; P = 3.8
k1 = 1.0**2 * kernels.ExpSquaredKernel(1.0**2) \
                        * kernels.ExpSine2Kernel(2.0 / 1.0**2, 3.8)

k2 = kernels.WhiteKernel(0.01)

kernel = k1+k2

# first we feed our combined kernel to the George library:
gp = george.GP(kernel, mean=0.0)

# then we compute the covariance matrix:
gp.compute(time)

t = np.arange(np.min(time),np.max(time),0.1)

# calculate expectation and variance at each point:
mu, cov = gp.predict(value, t)
std = np.sqrt(np.diag(cov))

ax = pl.subplot(111)
pl.plot(t,mu)
ax.fill_between(t,mu-std,mu+std,facecolor='lightblue', lw=0, interpolate=True)
pl.scatter(time,value,s=2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()


I am working on the file "TIARA/Tutorial/KeplerLightCurve.ipynb".
The first task is:
Read the Kepler data from the file:
Can you write Python code for it?

time=[];value=[]
while True:
    line = datafile.readline()
    if not line: break

    items=line.split()
    if (items[0][0]!='|'):
        time.append(float(items[1]))
        value.append(float(items[2]))

time=np.array(time)
value=np.array(value)


I am working on the file "TIARA/Tutorial/KeplerLightCurve.ipynb".
The first task is:
Specify the datafile containing Kepler data for the object KIC 1430163:
Can you write Python code for it?

filename="KIC1430163.tbl"
datafile = open(filename,'r')


I am working on the file "TIARA/Tutorial/KeplerLightCurve.ipynb".
The first task is:
The key parameter here is the period, which is the fourth number along. We expect this to be about 3.9 and... it is already coming out with that kind of value!
From the paper:
This star has a published rotation period of 3.88 ± 0.58 days, measured using traditional periodogram and autocorrelation function approaches applied to Kepler data from Quarters 0–16 (Mathur et al. 2014), covering about four years.
Let's now pass these optimised parameters to george and recompute our prediction:
Can you write Python code for it?

# pass the parameters to the george kernel:
gp.kernel[:] = results.x

t = np.arange(np.min(time),np.max(time),0.1)

# calculate expectation and variance at each point:
mu, cov = gp.predict(value, t)
std = np.sqrt(np.diag(cov))

ax = pl.subplot(111)
pl.plot(t,mu)
ax.fill_between(t,mu-std,mu+std,facecolor='lightblue', lw=0, interpolate=True)
pl.scatter(time,value,s=2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:

Can you write Python code for it?

import emcee

# we need to define three functions:
# a log likelihood, a log prior & a log posterior.


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
And the time has been made relative to the first measurement:
Can you write Python code for it?


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
First we need to define a log(likelihood). We'll use the log(likelihood) implemented in the george library, which implements:
$$
\ln L = -\frac{1}{2}(y - \mu)^{\rm T} C^{-1}(y - \mu) - \frac{1}{2}\ln |C\,| + \frac{N}{2}\ln 2\pi
$$
(see Eq. 5 in https://arxiv.org/pdf/1706.05459.pdf).
Can you write Python code for it?

# set the loglikelihood:
def lnlike(p, x, y):

    lnB = np.log(p[0])
    lnC = p[1]
    lnL = np.log(p[2])
    lnP = np.log(p[3])

    p0 = np.array([lnB,lnC,lnL,lnP])

    # update kernel parameters:
    gp.set_parameter_vector(p0)

    # calculate the likelihood:
    ll = gp.log_likelihood(y)

    # return
    return ll if np.isfinite(ll) else 1e25


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
I'm going to set bounds on the available parameters space, i.e. our prior volume, using the ranges taken from Table 4 of https://arxiv.org/pdf/1706.05459.pdf
Can you write Python code for it?

import scipy.optimize as op

# extract our initial guess at parameters
# from the celerite kernel and put it in a
# vector:
p0 = gp.get_parameter_vector()

# set prior ranges
# Note that these are in *logarithmic* space
bnds = ((-10.,0.),(-5.,5.),(-5.,-1.5),(-3.,5.))

# run optimization:
results = op.minimize(nll, p0, method='L-BFGS-B', jac=grad_nll, bounds=bnds, args=(value, gp))

# print the value of the optimised parameters:
print np.exp(results.x)
print("Final log-likelihood: {0}".format(-results.fun))


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
Import some libraries:
Can you write Python code for it?


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
In the paper there are two suggested kernels for modelling the covariance of the Kepler data (Eqs. 55 & 56). In the paper the authors fit Eq 56 - here we are going to fit Eq. 56.
$$
k(\tau) = \frac{B}{1+C}\exp^{-\tau/L} \left[ \cos{\left( \frac{2\pi\tau}{P} \right)} + (1+C) \right]
$$
This is the same as the CustomTerm described in the celerite documentation here: http://celerite.readthedocs.io/en/stable/python/kernel/
There is one small difference though - the exponent is expressed differently. This doesn't mean we need to change anything... except for our prior bounds because we're going to apply those as logarithmic bounds so we will need to put a minus sign in front of them since $\log(1/x) = -\log(x)$.
Can you write Python code for it?

import autograd.numpy as np

class CustomTerm(terms.Term):
    parameter_names = ("log_a", "log_b", "log_c", "log_P")

    def get_real_coefficients(self, params):
        log_a, log_b, log_c, log_P = params
        b = np.exp(log_b)
        return (
            np.exp(log_a) * (1.0 + b) / (2.0 + b), np.exp(log_c),
        )

    def get_complex_coefficients(self, params):
        log_a, log_b, log_c, log_P = params
        b = np.exp(log_b)
        return (
            np.exp(log_a) / (2.0 + b), 0.0,
            np.exp(log_c), 2*np.pi*np.exp(-log_P),
        )


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
Make a plot like the one in Figure 7:
Can you write Python code for it?

pl.subplot(111)
pl.scatter(time,value,s=0.2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
Now let's run the production MCMC:
Can you write Python code for it?

# take the highest likelihood point from the burn-in as a
# starting point and now begin your production run:
print("Running production")
p = p0[np.argmax(lnp)]
p0 = [p + 1e-5 * np.random.randn(ndim) for i in xrange(nwalkers)]
p0, _, _ = sampler.run_mcmc(p0, 5000)

print "Finished"

import acor

# calculate the convergence time of our
# MCMC chains:
samples = sampler.flatchain
s2 = np.ndarray.transpose(samples)
tau, mean, sigma = acor.acor(s2)
print "Convergence time from acor: ", tau
print "Number of independent samples:", 5000.-(20.*tau)

# get rid of the samples that were taken
# before convergence:
delta = int(20*tau)
samples = sampler.flatchain[delta:,:]

samples[:, 2] = np.exp(samples[:, 2])
b_mcmc, c_mcmc, l_mcmc, p_mcmc = map(lambda v: (v[1], v[2]-v[1], v[1]-v[0]),
                             zip(*np.percentile(samples, [16, 50, 84],
                                                axis=0)))

# specify prediction points:
t = np.arange(np.min(time),np.max(time),0.1)

# update the kernel hyper-parameters:
hp = np.array([b_mcmc[0], c_mcmc[0], l_mcmc[0], p_mcmc[0]])

lnB = np.log(p[0])
lnC = p[1]
lnL = np.log(p[2])
lnP = np.log(p[3])

p0 = np.array([lnB,lnC,lnL,lnP])
gp.set_parameter_vector(p0)

print hp
# calculate expectation and variance at each point:
mu, cov = gp.predict(value, t)

ax = pl.subplot(111)
pl.plot(t,mu)
ax.fill_between(t,mu-std,mu+std,facecolor='lightblue', lw=0, interpolate=True)
pl.scatter(time,value,s=2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()

import corner

# Plot it.
figure = corner.corner(samples, labels=[r"$B$", r"$lnC$", r"$L$", r"$P$"],
                         quantiles=[0.16,0.5,0.84],
                         #levels=[0.39,0.86,0.99],
                         levels=[0.68,0.95,0.99],
                         title="KIC 1430163",
                         show_titles=True, title_args={"fontsize": 12})


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
Specify the datafile containing Kepler data for the object KIC 1430163:
Can you write Python code for it?


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
The paper says:
We run 500 steps of burn-in, followed by 5000 steps of MCMC using emcee.
First let's run the burn-in:
Can you write Python code for it?

# run a few samples as a burn-in:
print("Running burn-in")
p0, lnp, _ = sampler.run_mcmc(p0, 500)
sampler.reset()


I am working on the file "TIARA/Tutorial/KeplerLightCurveCelerite.ipynb".
The first task is:
We can then use these inputs to initiate our sampler:
Can you write Python code for it?

# initalise the sampler:
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=data)


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
2. Corpus Processing
Topic modelling algorithms process vectorized data. In order to apply them, we need to transform the raw text input data into a vector representation. To do so, we will remove irrelevant information from the text data and preserve as much relevant information as possible to capture the semantic content in the document collection.
Thus, we will proceed with the following steps:

Tokenization
Homogeneization
Cleaning
Vectorization

2.1. Tokenization
For the first steps, we will use some of the powerfull methods available from the Natural Language Toolkit. In order to use the word_tokenize method from nltk, you might need to get the appropriate libraries using nltk.download(). You must select option "d) Download", and identifier "punkt"
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "punkt"
nltk.download('punkt')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
2.2.2. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
Task: Apply the .stem() method, from the stemmer object created in the first line, to corpus_filtered.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')
corpus_stemmed = []

for n, token_list in enumerate(corpus_filtered):
    print "\rStemming article {0} out of {1}".format(n + 1, n_art),

    # Convert all tokens in token_list to lowercase, remove non alfanumeric tokens and stem.
    # Store the result in a new token list, clean_tokens.
    # scode: stemmed_tokens = <FILL IN>
    stemmed_tokens = [stemmer.stem(token) for token in token_list]

    # Add art to the stemmed corpus
    # scode: <FILL IN>
    corpus_stemmed.append(stemmed_tokens)

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemmed[0][0:30]

Test.assertTrue((len([c for c in corpus_stemmed[0] if c!=stemmer.stem(c)]) < 0.1*len(corpus_stemmed[0])),
                'It seems that stemming has not been applied properly')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
2.4. Vectorization
Up to this point, we have transformed the raw text collection of articles in a list of articles, where each article is a collection of the word roots that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix). To do so, we will start using the tools provided by the gensim library.
As a first step, we create a dictionary containing all tokens in our text corpus, and assigning an integer identifier to each one of them.
Can you write Python code for it?

# Create dictionary of tokens
D = gensim.corpora.Dictionary(corpus_clean)
n_tokens = len(D)

print "The dictionary contains {0} tokens".format(n_tokens)
print "First tokens in the dictionary: "
for n in range(10):
    print str(n) + ": " + D[n]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Exercise (All in one): Note that, for pedagogical reasons, we have used a different for loop for each text processing step creating a new corpus_xxx variable after each step. For very large corpus, this could cause memory problems.
As a summary exercise, repeat the whole text processing, starting from corpus_text up to computing the bow, with the following modifications:

Use a single for loop, avoiding the creation of any intermediate corpus variables.
Use lemmatization instead of stemming.
Remove all tokens appearing in only one document and less than 2 times.
Save the result in a new variable corpus_bow1.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Exercise (Visualizing categories): Repeat the previous exercise with a second wikipedia category. For instance, you can take "communication".

Save the result in variable corpus_bow2.
Determine the most frequent terms in corpus_bow1 (term1) and corpus_bow2 (term2).
Transform each article in corpus_bow1 and corpus_bow2 into a 2 dimensional vector, where the first component is the frecuency of term1 and the second component is the frequency of term2
Draw a dispersion plot of all 2 dimensional points, using a different marker for each corpus. Could you differentiate both corpora using the selected terms only? What if the 2nd most frequent term is used?
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Exercise: Count the number of tokens appearing only in a single article.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Exercise: Represent graphically those 20 tokens that appear in the highest number of articles. Note that you can use the code above (headed by # SORTED TOKEN FREQUENCIES) with a very minor modification.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>

# SORTED TOKEN FREQUENCIES (I):
# Count the number of occurrences of each token.
token_count2 = np.zeros(n_tokens)
for x in corpus_bow_flat:
    token_count2[x[0]] += (x[1]>0)

# Sort by decreasing number of occurences
ids_sorted2 = np.argsort(- token_count2)
tf_sorted2 = token_count2[ids_sorted2]

# SORTED TOKEN FREQUENCIES (II):
# Example data
n_bins = 25
hot_tokens2 = [D[i] for i in ids_sorted2[n_bins-1::-1]]
y_pos2 = np.arange(len(hot_tokens2))
z2 = tf_sorted2[n_bins-1::-1]/n_art

plt.figure()
plt.barh(y_pos2, z2, align='center', alpha=0.4)
plt.yticks(y_pos2, hot_tokens2)
plt.xlabel('Number of articles')
plt.title('Token distribution')
plt.show()
display()


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Exercise: There are usually many tokens that appear with very low frequency in the corpus. Count the number of tokens appearing only once, and what is the proportion of them in the token list.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>
# Example data
cold_tokens = [D[i] for i in ids_sorted if tf_sorted[i]==1]

print "There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100)


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

# Example data
n_bins = 25
hot_tokens = [D[i] for i in ids_sorted[n_bins-1::-1]]
y_pos = np.arange(len(hot_tokens))
z = tf_sorted[n_bins-1::-1]/n_art

plt.barh(y_pos, z, align='center', alpha=0.4)
plt.yticks(y_pos, hot_tokens)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()
display()

# SORTED TOKEN FREQUENCIES:

# Example data
plt.semilogy(tf_sorted)
plt.ylabel('Total number of occurrences')
plt.xlabel('Token rank')
plt.title('Token occurrences')
plt.show()
display()


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
In the second step, let us create a numerical version of our corpus using the doc2bow method. In general, D.doc2bow(token_list) transform any list of tokens into a list of tuples (token_id, n), one per each token in token_list, where token_id is the token identifier (according to dictionary D) and n is the number of occurrences of such token in token_list.
 Task: Apply the doc2bow method from gensim dictionary D, to all tokens in every article in corpus_clean. The result must be a new list named corpus_bow where each element is a list of tuples (token_id, number_of_occurrences).
Can you write Python code for it?

# Transform token lists into sparse vectors on the D-space
# scode: corpus_bow = <FILL IN>
corpus_bow = [D.doc2bow(doc) for doc in corpus_clean]

Test.assertTrue(len(corpus_bow)==n_art, 'corpus_bow has not the appropriate size')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Note that we can interpret each element of corpus_bow as a sparse_vector. For example, a list of tuples
[(0, 1), (3, 3), (5,2)]

for a dictionary of 10 elements can be represented as a vector, where any tuple (id, n) states that position id must take value n. The rest of positions must be zero.
[1, 0, 0, 3, 0, 2, 0, 0, 0, 0]

These sparse vectors will be the inputs to the topic modeling algorithms.
Note that, at this point, we have built a Dictionary containing
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Now, we have stored the whole text collection in two lists:

corpus_titles, which contains the titles of the selected articles
corpus_text, with the text content of the selected wikipedia articles

You can browse the content of the wikipedia articles to get some intuition about the kind of documents that will be processed.
Can you write Python code for it?

n = 5
print corpus_titles[n]
print corpus_text[n]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
One of the advantages of the lemmatizer method is that the result of lemmmatization is still a true word, which is more advisable for the presentation of text processing results and lemmatization.
However, without using contextual information, lemmatize() does not remove grammatical differences. This is the reason why "is" or "are" are preserved and not replaced by infinitive "be".
As an alternative, we can apply .lemmatize(word, pos), where 'pos' is a string code specifying the part-of-speech (pos), i.e. the grammatical role of the words in its sentence. For instance, you can check the difference between wnl.lemmatize('is') and wnl.lemmatize('is, pos='v').
2.3. Cleaning
The third step consists of removing those words that are very common in language and do not carry out usefull semantic content (articles, pronouns, etc).
Once again, we might need to load the stopword files using the download tools from nltk
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "stopwords"
nltk.download("stopwords")


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Task: Apply the .lemmatize() method, from the WordNetLemmatizer object created in the first line, to corpus_filtered.
Can you write Python code for it?

wnl = WordNetLemmatizer()

# Select stemmer.
corpus_lemmat = []

for n, token_list in enumerate(corpus_filtered):
    print "\rLemmatizing article {0} out of {1}".format(n + 1, n_art),

    # scode: lemmat_tokens = <FILL IN>
    lemmat_tokens = [wnl.lemmatize(token) for token in token_list]

    # Add art to the stemmed corpus
    # scode: <FILL IN>
    corpus_lemmat.append(lemmat_tokens)

print "\nLet's check the first tokens from document 0 after lemmatization:"
print corpus_lemmat[0][0:30]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Task: In the second line below we read a list of common english stopwords. Clean corpus_stemmed by removing all tokens in the stopword list.
Can you write Python code for it?

corpus_clean = []
stopwords_en = stopwords.words('english')
n = 0
for token_list in corpus_stemmed:
    n += 1
    print "\rRemoving stopwords from article {0} out of {1}".format(n, n_art),

    # Remove all tokens in the stopwords list and append the result to corpus_clean
    # scode: clean_tokens = <FILL IN>
    clean_tokens = [token for token in token_list if token not in stopwords_en]

    # scode: <FILL IN>
    corpus_clean.append(clean_tokens)

print "\n Let's check tokens after cleaning:"
print corpus_clean[0][0:30]

Test.assertTrue(len(corpus_clean) == n_art, 'List corpus_clean does not contain the expected number of articles')
Test.assertTrue(len([c for c in corpus_clean[0] if c in stopwords_en])==0, 'Stopwords have not been removed')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
Task: Insert the appropriate call to word_tokenize in the code below, in order to get the tokens list corresponding to each Wikipedia article:
Can you write Python code for it?

corpus_tokens = []

for n, art in enumerate(corpus_text):
    print "\rTokenizing article {0} out of {1}".format(n + 1, n_art),
    # This is to make sure that all characters have the appropriate encoding.
    art = art.decode('utf-8')

    # Tokenize each text entry.
    # scode: tokens = <FILL IN>
    tokens = word_tokenize(art)

    # Add the new token list as a new element to corpus_tokens (that will be a list of lists)
    # scode: <FILL IN>
    corpus_tokens.append(tokens)

print "\n The corpus has been tokenized. Let's check some portion of the first article:"
print corpus_tokens[0][0:30]

Test.assertEquals(len(corpus_tokens), n_art, "The number of articles has changed unexpectedly")
Test.assertTrue(len(corpus_tokens) >= 100,
                "Your corpus_tokens has less than 100 articles. Consider using a larger dataset")


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
You can try with any other categories, but take into account that some categories may contain very few articles. Select a category with at least 100 articles. You can browse the wikipedia category tree here, https://en.wikipedia.org/wiki/Category:Contents, for instance, and select the appropriate one.
We start downloading the text collection.
Can you write Python code for it?

# Loading category data. This may take a while
print "Loading category data. This may take a while..."
cat_data = category.Category(site, cat)

corpus_titles = []
corpus_text = []

for n, page in enumerate(cat_data.getAllMembersGen()):
    print "\rLoading article {0}".format(n + 1),
    corpus_titles.append(page.title)
    corpus_text.append(page.getWikiText())

n_art = len(corpus_titles)
print "\nLoaded " + str(n_art) + " articles from category " + cat


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
and a bow representation of a corpus with
Can you write Python code for it?

print "{0} Wikipedia articles".format(len(corpus_bow))


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db.ipynb".
The first task is:
which appears
Can you write Python code for it?

print "{0} times in the whole corpus".format(tf_sorted[0])


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
1. Corpus acquisition.
In these notebooks we will explore some tools for text analysis and two topic modeling algorithms available from Python toolboxes.
To do so, we will explore and analyze collections of Wikipedia articles from a given category, using wikitools, that makes the capture of content from wikimedia sites very easy.
(As a side note, there are many other available text collections to work with. In particular, the NLTK library has many examples, that you can explore using the nltk.download() tool.
import nltk
nltk.download()

for instance, you can take the gutemberg dataset
Mycorpus = nltk.corpus.gutenberg
text_name = Mycorpus.fileids()[0]
raw = Mycorpus.raw(text_name)
Words = Mycorpus.words(text_name)

Also, tools like Gensim or Sci-kit learn include text databases to work with).
In order to use Wikipedia data, we will select a single category of articles:
Can you write Python code for it?

site = wiki.Wiki("https://en.wikipedia.org/w/api.php")
# Select a category with a reasonable number of articles (>100)
# cat = "Economics"
cat = "Pseudoscience"
print cat


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
2. Corpus Processing
Topic modelling algorithms process vectorized data. In order to apply them, we need to transform the raw text input data into a vector representation. To do so, we will remove irrelevant information from the text data and preserve as much relevant information as possible to capture the semantic content in the document collection.
Thus, we will proceed with the following steps:

Tokenization
Homogeneization
Cleaning
Vectorization

2.1. Tokenization
For the first steps, we will use some of the powerfull methods available from the Natural Language Toolkit. In order to use the word_tokenize method from nltk, you might need to get the appropriate libraries using nltk.download(). You must select option "d) Download", and identifier "punkt"
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "punkt"
# nltk.download("punkt")


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
2.4. Vectorization
Up to this point, we have transformed the raw text collection of articles in a list of articles, where each article is a collection of the word roots that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix). To do so, we will start using the tools provided by the gensim library.
As a first step, we create a dictionary containing all tokens in our text corpus, and assigning an integer identifier to each one of them.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
Exercise  (bigrams): nltk provides an utility to compute n-grams from a list of tokens, in nltk.util.ngrams. Join all tokens in corpus_clean in a single list and compute the bigrams. Plot the 20 most frequent bigrams in the corpus.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>
# Check the code below to see how ngrams works, and adapt it to solve the exercise.
# from nltk.util import ngrams
# sentence = 'this is a foo bar sentences and i want to ngramize it'
# sixgrams = ngrams(sentence.split(), 2)
# for grams in sixgrams:
#     print grams


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
Exercise (All in one): Note that, for pedagogical reasons, we have used a different for loop for each text processing step creating a new corpus_xxx variable after each step. For very large corpus, this could cause memory problems.
As a summary exercise, repeat the whole text processing, starting from corpus_text up to computing the bow, with the following modifications:

Use a single for loop, avoiding the creation of any intermediate corpus variables.
Use lemmatization instead of stemming.
Remove all tokens appearing in only one document and less than 2 times.
Save the result in a new variable corpus_bow1.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
Exercise (Visualizing categories): Repeat the previous exercise with a second wikipedia category. For instance, you can take "communication".

Save the result in variable corpus_bow2.
Determine the most frequent terms in corpus_bow1 (term1) and corpus_bow2 (term2).
Transform each article in corpus_bow1 and corpus_bow2 into a 2 dimensional vector, where the first component is the frecuency of term1 and the second component is the frequency of term2
Draw a dispersion plot of all 2 dimensional points, using a different marker for each corpus. Could you differentiate both corpora using the selected terms only? What if the 2nd most frequent term is used?
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
Exercise: Represent graphically those 20 tokens that appear in the highest number of articles. Note that you can use the code above (headed by # SORTED TOKEN FREQUENCIES) with a very minor modification.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
Exercise: There are usually many tokens that appear with very low frequency in the corpus. Count the number of tokens appearing only once, and what is the proportion of them in the token list.
Can you write Python code for it?

# scode: cold_tokens = <FILL IN>

print "There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100)


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
Task: Apply the .lemmatize() method, from the WordNetLemmatizer object created in the first line, to corpus_filtered.
Can you write Python code for it?

wnl = WordNetLemmatizer()

# Select stemmer.
corpus_lemmat = []

for n, token_list in enumerate(corpus_filtered):
    print "\rLemmatizing article {0} out of {1}".format(n + 1, n_art),

    # scode: lemmat_tokens = <FILL IN>

    # Add art to the stemmed corpus
    # scode: <FILL IN>

print "\nLet's check the first tokens from document 0 after lemmatization:"
print corpus_lemmat[0][0:30]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
Task: In the second line below we read a list of common english stopwords. Clean corpus_stemmed by removing all tokens in the stopword list.
Can you write Python code for it?

corpus_clean = []
stopwords_en = stopwords.words('english')
n = 0
for token_list in corpus_stemmed:
    n += 1
    print "\rRemoving stopwords from article {0} out of {1}".format(n, n_art),

    # Remove all tokens in the stopwords list and append the result to corpus_clean
    # scode: clean_tokens = <FILL IN>

    # scode: <FILL IN>

print "\n Let's check tokens after cleaning:"
print corpus_clean[0][0:30]

Test.assertTrue(len(corpus_clean) == n_art, 'List corpus_clean does not contain the expected number of articles')
Test.assertTrue(len([c for c in corpus_clean[0] if c in stopwords_en])==0, 'Stopwords have not been removed')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
Task: Insert the appropriate call to word_tokenize in the code below, in order to get the tokens list corresponding to each Wikipedia article:
Can you write Python code for it?

corpus_tokens = []

for n, art in enumerate(corpus_text):
    print "\rTokenizing article {0} out of {1}".format(n + 1, n_art),
    # This is to make sure that all characters have the appropriate encoding.
    art = art.decode('utf-8')

    # Tokenize each text entry.
    # scode: tokens = <FILL IN>

    # Add the new token list as a new element to corpus_tokens (that will be a list of lists)
    # scode: <FILL IN>

print "\n The corpus has been tokenized. Let's check some portion of the first article:"
print corpus_tokens[0][0:30]

Test.assertEquals(len(corpus_tokens), n_art, "The number of articles has changed unexpectedly")
Test.assertTrue(len(corpus_tokens) >= 100,
                "Your corpus_tokens has less than 100 articles. Consider using a larger dataset")


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
You can try with any other categories, but take into account that some categories may contain very few articles. Select a category with at least 100 articles. You can browse the wikipedia category tree here, https://en.wikipedia.org/wiki/Category:Contents, for instance, and select the appropriate one.
We start downloading the text collection.
Can you write Python code for it?

# Loading category data. This may take a while
print "Loading category data. This may take a while..."
cat_data = category.Category(site, cat)

corpus_titles = []
corpus_text = []

for n, page in enumerate(cat_data.getAllMembersGen()):
    print "\r Loading article {0}".format(n + 1),
    corpus_titles.append(page.title)
    corpus_text.append(page.getWikiText())

n_art = len(corpus_titles)
print "\nLoaded " + str(n_art) + " articles from category " + cat


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/databricks/TM1_NLP_db_student.ipynb".
The first task is:
which appears
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
1. Corpus acquisition.
In this notebook we will explore some tools for text analysis available from Python toolboxes.
To do so, we will explore and analyze collections of Wikipedia articles from a given category, using wikitools, that makes the capture of content from wikimedia sites very easy.
(As a side note, there are many other available text collections to work with. In particular, the NLTK library has many examples, that you can explore using the nltk.download() tool.
import nltk
nltk.download()

for instance, you can take the gutemberg dataset
Mycorpus = nltk.corpus.gutenberg
text_name = Mycorpus.fileids()[0]
raw = Mycorpus.raw(text_name)
Words = Mycorpus.words(text_name)

Also, tools like Gensim or Sci-kit learn include text databases to work with).
In order to use Wikipedia data, we will select a single category of articles:
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
2.2. Homogeneization
By looking at the tokenized corpus you may verify that there are many tokens that correspond to punktuation signs and other symbols that are not relevant to analyze the semantic content. They can be removed using the stemming tool from nltk.
The homogeneization process will consist of:

Removing capitalization: capital alphabetic characters will be transformed to their corresponding lowercase characters.
Removing non alphanumeric tokens (e.g. punktuation signs)
Stemming/Lemmatization: removing word terminations to preserve the root of the words and ignore grammatical information.

2.2.1. Filtering
Let us proceed with the filtering steps 1 and 2 (removing capitalization and non-alphanumeric tokens).
Task: Convert all tokens in corpus_tokens to lowercase (using .lower() method) and remove non alphanumeric tokens (that you can detect with .isalnum() method). You can do it in a single line of code...
Can you write Python code for it?

corpus_filtered = []

for n, token_list in enumerate(corpus_tokens):
    print "\rFiltering article {0} out of {1}".format(n + 1, n_art),

    # Convert all tokens in token_list to lowercase, remove non alfanumeric tokens and stem.
    # Store the result in a new token list, clean_tokens.
    # scode: filtered_tokens = <FILL IN>
    filtered_tokens = [token.lower() for token in token_list if token.isalnum()]

    # Add art to corpus_filtered
    # scode: <FILL IN>
    corpus_filtered.append(filtered_tokens)

print "\nLet's check the first tokens from document 0 after filtering:"
print corpus_filtered[0][0:30]

Test.assertTrue(all([c==c.lower() for c in corpus_filtered[23]]), 'Capital letters have not been removed')
Test.assertTrue(all([c.isalnum() for c in corpus_filtered[13]]), 'Non alphanumeric characters have not been removed')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
2.2.2. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
Task: Apply the .stem() method, from the stemmer object created in the first line, to corpus_filtered.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')
corpus_stemmed = []

for n, token_list in enumerate(corpus_filtered):
    print "\rStemming article {0} out of {1}".format(n + 1, n_art),

    # Convert all tokens in token_list to lowercase, remove non alfanumeric tokens and stem.
    # Store the result in a new token list, clean_tokens.
    # scode: stemmed_tokens = <FILL IN>
    stemmed_tokens = [stemmer.stem(token) for token in token_list]

    # Add art to the stemmed corpus
    # scode: <FILL IN>
    corpus_stemmed.append(stemmed_tokens)

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemmed[0][0:30]

Test.assertTrue((len([c for c in corpus_stemmed[0] if c!=stemmer.stem(c)]) < 0.1*len(corpus_stemmed[0])),
                'It seems that stemming has not been applied properly')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Alternatively, we can apply lemmatization. For english texts, we can use the lemmatizer from NLTK, which is based on WordNet. If you have not used wordnet before, you will likely need to download it from nltk
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "wordnet"
# nltk.download()


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Before starting with the semantic analyisis, it is interesting to observe the token distribution for the given corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (I):
# Create a "flat" corpus with all tuples in a single list
corpus_bow_flat = [item for sublist in corpus_bow for item in sublist]

# Initialize a numpy array that we will use to count tokens.
# token_count[n] should store the number of ocurrences of the n-th token, D[n]
token_count = np.zeros(n_tokens)

# Count the number of occurrences of each token.
for x in corpus_bow_flat:
    # Update the proper element in token_count
    # scode: <FILL IN>
    token_count[x[0]] += x[1]

# Sort by decreasing number of occurences
ids_sorted = np.argsort(- token_count)
tf_sorted = token_count[ids_sorted]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Exercise (bigrams): nltk provides an utility to compute n-grams from a list of tokens, in nltk.util.ngrams. Join all tokens in corpus_clean in a single list and compute the bigrams. Plot the 20 most frequent bigrams in the corpus.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Exercise: Count the number of tokens appearing only in a single article.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Exercise: Represent graphically those 20 tokens that appear in the highest number of articles. Note that you can use the code above (headed by # SORTED TOKEN FREQUENCIES) with a very minor modification.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>

# SORTED TOKEN FREQUENCIES (I):
# Count the number of occurrences of each token.
token_count2 = np.zeros(n_tokens)
for x in corpus_bow_flat:
    token_count2[x[0]] += (x[1]>0)

# Sort by decreasing number of occurences
ids_sorted2 = np.argsort(- token_count2)
tf_sorted2 = token_count2[ids_sorted2]

# SORTED TOKEN FREQUENCIES (II):
# Example data
n_bins = 25
hot_tokens2 = [D[i] for i in ids_sorted2[n_bins-1::-1]]
y_pos2 = np.arange(len(hot_tokens2))
z2 = tf_sorted2[n_bins-1::-1]

plt.barh(y_pos2, z2, align='center', alpha=0.4)
plt.yticks(y_pos2, hot_tokens2)
plt.xlabel('Number of articles')
plt.title('Token distribution')
plt.show()


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

# Example data
n_bins = 25
hot_tokens = [D[i] for i in ids_sorted[n_bins-1::-1]]
y_pos = np.arange(len(hot_tokens))
z = tf_sorted[n_bins-1::-1]/n_art

plt.figure()
plt.barh(y_pos, z, align='center', alpha=0.4)
plt.yticks(y_pos, hot_tokens)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()

# SORTED TOKEN FREQUENCIES:

# Example data
plt.figure()
plt.semilogy(tf_sorted)
plt.ylabel('Total number of occurrences')
plt.xlabel('Token rank')
plt.title('Token occurrences')
plt.show()


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
In the second step, let us create a numerical version of our corpus using the doc2bow method. In general, D.doc2bow(token_list) transform any list of tokens into a list of tuples (token_id, n), one per each token in token_list, where token_id is the token identifier (according to dictionary D) and n is the number of occurrences of such token in token_list.
 Task: Apply the doc2bow method from gensim dictionary D, to all tokens in every article in corpus_clean. The result must be a new list named corpus_bow where each element is a list of tuples (token_id, number_of_occurrences).
Can you write Python code for it?

# Transform token lists into sparse vectors on the D-space
# scode: corpus_bow = <FILL IN>
corpus_bow = [D.doc2bow(doc) for doc in corpus_clean]

Test.assertTrue(len(corpus_bow)==n_art, 'corpus_bow has not the appropriate size')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Note that we can interpret each element of corpus_bow as a sparse_vector. For example, a list of tuples
[(0, 1), (3, 3), (5,2)]

for a dictionary of 10 elements can be represented as a vector, where any tuple (id, n) states that position id must take value n. The rest of positions must be zero.
[1, 0, 0, 3, 0, 2, 0, 0, 0, 0]

These sparse vectors will be the inputs to the topic modeling algorithms.
Note that, at this point, we have built a Dictionary containing
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Now, we have stored the whole text collection in two lists:

corpus_titles, which contains the titles of the selected articles
corpus_text, with the text content of the selected wikipedia articles

You can browse the content of the wikipedia articles to get some intuition about the kind of documents that will be processed.
Can you write Python code for it?

n = 5
print corpus_titles[n]
print corpus_text[n+1]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Task: In the second line below we read a list of common english stopwords. Clean corpus_stemmed by removing all tokens in the stopword list.
Can you write Python code for it?

corpus_clean = []
stopwords_en = stopwords.words('english')
n = 0
for token_list in corpus_stemmed:
    n += 1
    print "\rRemoving stopwords from article {0} out of {1}".format(n, n_art),

    # Remove all tokens in the stopwords list and append the result to corpus_clean
    # scode: clean_tokens = <FILL IN>
    clean_tokens = [token for token in token_list if token not in stopwords_en]

    # scode: <FILL IN>
    corpus_clean.append(clean_tokens)

print "\n Let's check tokens after cleaning:"
print corpus_clean[0][0:30]

Test.assertTrue(len(corpus_clean) == n_art, 'List corpus_clean does not contain the expected number of articles')
Test.assertTrue(len([c for c in corpus_clean[0] if c in stopwords_en])==0, 'Stopwords have not been removed')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
Task: Insert the appropriate call to word_tokenize in the code below, in order to get the tokens list corresponding to each Wikipedia article:
Can you write Python code for it?

corpus_tokens = []

for n, art in enumerate(corpus_text):
    print "\rTokenizing article {0} out of {1}".format(n + 1, n_art),
    # This is to make sure that all characters have the appropriate encoding.
    art = art.decode('utf-8')

    # Tokenize each text entry.
    # scode: tokens = <FILL IN>
    tokens = word_tokenize(art)

    # Add the new token list as a new element to corpus_tokens (that will be a list of lists)
    # scode: <FILL IN>
    corpus_tokens.append(tokens)

print "\n The corpus has been tokenized. Let's check some portion of the first article:"
print corpus_tokens[0][0:30]

Test.assertEquals(len(corpus_tokens), n_art, "The number of articles has changed unexpectedly")
Test.assertTrue(len(corpus_tokens) >= 100,
                "Your corpus_tokens has less than 100 articles. Consider using a larger dataset")


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
ids_sorted is a list of all token ids, sorted by decreasing number of occurrences in the whole corpus. For instance, the most frequent term is
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP.ipynb".
The first task is:
which appears
Can you write Python code for it?

print "{0} times in the whole corpus".format(tf_sorted[0])


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
2. Corpus Processing
Topic modelling algorithms process vectorized data. In order to apply them, we need to transform the raw text input data into a vector representation. To do so, we will remove irrelevant information from the text data and preserve as much relevant information as possible to capture the semantic content in the document collection.
Thus, we will proceed with the following steps:

Tokenization
Homogeneization
Cleaning
Vectorization

2.1. Tokenization
For the first steps, we will use some of the powerfull methods available from the Natural Language Toolkit. In order to use the word_tokenize method from nltk, you might need to get the appropriate libraries using nltk.download(). You must select option "d) Download", and identifier "punkt"
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "punkt"
# nltk.download()


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
2.2.2. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
Task: Apply the .stem() method, from the stemmer object created in the first line, to corpus_filtered.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')
corpus_stemmed = []

for n, token_list in enumerate(corpus_filtered):
    print "\rStemming article {0} out of {1}".format(n + 1, n_art),

    # Apply stemming to all tokens in token_list and save them in stemmed_tokens
    # scode: stemmed_tokens = <FILL IN>

    # Add stemmed_tokens to the stemmed corpus
    # scode: <FILL IN>

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemmed[0][0:30]

Test.assertTrue((len([c for c in corpus_stemmed[0] if c!=stemmer.stem(c)]) < 0.1*len(corpus_stemmed[0])),
                'It seems that stemming has not been applied properly')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
2.4. Saving results
The dictionary D and the Bag of Words in corpus_bow are the key inputs to the topic model algorithms analyzed in the following notebook. Save them to be ready to use them during the next session.
Can you write Python code for it?

import pickle
data = {}
data['D'] = D
data['corpus_bow'] = corpus_bow
pickle.dump(data, open("wikiresults.p", "wb"))


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
2.4. Vectorization
Up to this point, we have transformed the raw text collection of articles in a list of articles, where each article is a collection of the word roots that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix). To do so, we will start using the tools provided by the gensim library.
As a first step, we create a dictionary containing all tokens in our text corpus, and assigning an integer identifier to each one of them.
Can you write Python code for it?

# Create dictionary of tokens
D = gensim.corpora.Dictionary(corpus_clean)
n_tokens = len(D)

print "The dictionary contains {0} tokens".format(n_tokens)
print "First tokens in the dictionary: "
for n in range(10):
    print str(n) + ": " + D[n]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
At this point, it is good to make sure to understand what has happened. In corpus_clean we had a list of token lists. With it, we have constructed a Dictionary, D, which assign an integer identifier to each token in the corpus.
After that, we have transformed each article (in corpus_clean) in a list tuples (id, n).
Can you write Python code for it?

print "Original article (after cleaning): "
print corpus_clean[0][0:30]
print "Sparse vector representation (first 30 components):"
print corpus_bow[0][0:30]
print "The first component, {0} from document 0, states that token 0 ({1}) appears {2} times".format(
    corpus_bow[0][0], D[0], corpus_bow[0][0][1])


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Before starting with the semantic analyisis, it is interesting to observe the token distribution for the given corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (I):
# Create a "flat" corpus with all tuples in a single list
corpus_bow_flat = [item for sublist in corpus_bow for item in sublist]

# Initialize a numpy array that we will use to cont tokens.
# token_count[n] should store the number of ocurrences of the n-th token, D[n]
token_count = np.zeros(n_tokens)

# Count the number of occurrences of each token.
for x in corpus_bow_flat:
    # Update the proper element in token_count
    # scode: <FILL IN>

# Sort by decreasing number of occurences
ids_sorted = np.argsort(- token_count)
tf_sorted = token_count[ids_sorted]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Exercise (All in one): Note that, for pedagogical reasons, we have used a different for loop for each text processing step creating a new corpus_xxx variable after each step. For very large corpus, this could cause memory problems.
As a summary exercise, repeat the whole text processing, starting from corpus_text up to computing the bow, with the following modifications:

Use a single for loop, avoiding the creation of any intermediate corpus variables.
Use lemmatization instead of stemming.
Remove all tokens appearing in only one document and less than 2 times.
Save the result in a new variable corpus_bow1.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Exercise (Visualizing categories): Repeat the previous exercise with a second wikipedia category. For instance, you can take "communication".

Save the result in variable corpus_bow2.
Determine the most frequent terms in corpus_bow1 (term1) and corpus_bow2 (term2).
Transform each article in corpus_bow1 and corpus_bow2 into a 2 dimensional vector, where the first component is the frecuency of term1 and the second component is the frequency of term2
Draw a dispersion plot of all 2 dimensional points, using a different marker for each corpus. Could you differentiate both corpora using the selected terms only? What if the 2nd most frequent term is used?
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Exercise: Count the number of tokens appearing only in a single article.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Exercise: Represent graphically those 20 tokens that appear in the highest number of articles. Note that you can use the code above (headed by # SORTED TOKEN FREQUENCIES) with a very minor modification.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Exercise: There are usually many tokens that appear with very low frequency in the corpus. Count the number of tokens appearing only once, and what is the proportion of them in the token list.
Can you write Python code for it?

# scode: cold_tokens = <FILL IN>

print "There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100)


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Note that we can interpret each element of corpus_bow as a sparse_vector. For example, a list of tuples
[(0, 1), (3, 3), (5,2)]

for a dictionary of 10 elements can be represented as a vector, where any tuple (id, n) states that position id must take value n. The rest of positions must be zero.
[1, 0, 0, 3, 0, 2, 0, 0, 0, 0]

These sparse vectors will be the inputs to the topic modeling algorithms.
Note that, at this point, we have built a Dictionary containing
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Task: Apply the .lemmatize() method, from the WordNetLemmatizer object created in the first line, to corpus_filtered.
Can you write Python code for it?

wnl = WordNetLemmatizer()

# Select stemmer.
corpus_lemmat = []

for n, token_list in enumerate(corpus_filtered):
    print "\rLemmatizing article {0} out of {1}".format(n + 1, n_art),

    # scode: lemmat_tokens = <FILL IN>

    # Add art to the stemmed corpus
    # scode: <FILL IN>

print "\nLet's check the first tokens from document 0 after lemmatization:"
print corpus_lemmat[0][0:30]


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Task: In the second line below we read a list of common english stopwords. Clean corpus_stemmed by removing all tokens in the stopword list.
Can you write Python code for it?

corpus_clean = []
stopwords_en = stopwords.words('english')
n = 0
for token_list in corpus_stemmed:
    n += 1
    print "\rRemoving stopwords from article {0} out of {1}".format(n, n_art),

    # Remove all tokens in the stopwords list and append the result to corpus_clean
    # scode: clean_tokens = <FILL IN>

    # scode: <FILL IN>

print "\n Let's check tokens after cleaning:"
print corpus_clean[0][0:30]

Test.assertTrue(len(corpus_clean) == n_art, 'List corpus_clean does not contain the expected number of articles')
Test.assertTrue(len([c for c in corpus_clean[0] if c in stopwords_en])==0, 'Stopwords have not been removed')


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
Task: Insert the appropriate call to word_tokenize in the code below, in order to get the tokens list corresponding to each Wikipedia article:
Can you write Python code for it?

corpus_tokens = []

for n, art in enumerate(corpus_text):
    print "\rTokenizing article {0} out of {1}".format(n + 1, n_art),
    # This is to make sure that all characters have the appropriate encoding.
    art = art.decode('utf-8')

    # Tokenize each text entry.
    # scode: tokens = <FILL IN>

    # Add the new token list as a new element to corpus_tokens (that will be a list of lists)
    # scode: <FILL IN>

print "\n The corpus has been tokenized. Let's check some portion of the first article:"
print corpus_tokens[0][0:30]

Test.assertEquals(len(corpus_tokens), n_art, "The number of articles has changed unexpectedly")
Test.assertTrue(len(corpus_tokens) >= 100,
                "Your corpus_tokens has less than 100 articles. Consider using a larger dataset")


I am working on the file "TM1.IntrodNLP/NLP_py2_wikitools/notebooks/TM1_NLP_student.ipynb".
The first task is:
and a bow representation of a corpus with
Can you write Python code for it?

print "{0} Wikipedia articles".format(len(corpus_bow))


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
1. Corpus acquisition
In this block we will work with collections of text documents. The objectives will be:

Find the most important topics in the collection and assign documents to topics
Analyze the structure of the collection by means of graph analysis

We will work with a collection of research projects funded by the US National Science Foundation, that you can find under the ./data directory. These files are publicly available from the NSF website.
(As a side note, there are many other available text collections to work with. In particular, the NLTK library has many examples, that you can explore using the nltk.download() tool.
import nltk
nltk.download()

for instance, you can take the gutemberg dataset
Mycorpus = nltk.corpus.gutenberg
text_name = Mycorpus.fileids()[0]
raw = Mycorpus.raw(text_name)
Words = Mycorpus.words(text_name)

Also, tools like Gensim or Sci-kit learn include text databases to work with).
1.1. Exploring file structure
NSF project information is provided in XML files. Projects are yearly grouped in .zip files, and each project is saved in a different XML file. To explore the structure of such files, we will use the file 160057.xml. Parsing XML files in python is rather easy using the ElementTree module.
To introduce some common functions to work with XML files we will follow this tutorial.
1.1.1. File format
To start with, you can have a look at the contents of the example file. We are interested on the following information of each project:

Project identifier
Project Title
Project Abstract
Budget
Starting Year (we will ignore project duration)
Institution (name, zipcode, and state)
Can you write Python code for it?

xmlfile = '../data/1600057.xml'

with open(xmlfile,'r') as fin:
    print(fin.read())


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
2. Corpus Processing
Topic modelling algorithms process vectorized data. In order to apply them, we need to transform the raw text input data into a vector representation. To do so, we will remove irrelevant information from the text data and preserve as much relevant information as possible to capture the semantic content in the document collection.
Thus, we will proceed with the following steps:

Tokenization
Homogeneization
Cleaning
Vectorization

2.1. Tokenization
For the first steps, we will use some of the powerful methods available from the Natural Language Toolkit. In order to use the word_tokenize method from nltk, you might need to get the appropriate libraries using nltk.download(). You must select option "d) Download", and identifier "punkt"
Can you write Python code for it?

from nltk import download

# You should comment this code fragment if the package is already available.
# download('punkt')


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
2.2. Homogeneization
By looking at the tokenized corpus you may verify that there are many tokens that correspond to punktuation signs and other symbols that are not relevant to analyze the semantic content. They can be removed using the stemming or lemmatization tools from nltk.
The homogeneization process will consist of:

Removing capitalization: capital alphabetic characters will be transformed to their corresponding lowercase characters.
Removing non alphanumeric tokens (e.g. punktuation signs)
Stemming/Lemmatization: removing word terminations to preserve the root of the words and ignore grammatical information.

Exercise 5: Convert all tokens in corpus_tokens to lowercase (using the .lower() method) and remove non alphanumeric tokens (that you can detect with .isalnum() method). You can complete the following code fragment with a single line of code ...
Can you write Python code for it?

corpus_filtered = []

for n, tokens in enumerate(corpus_tokens):
    if not n%1000:
        print('\rFiltering document', n, 'out of', ndocs, end='', flush=True)

    #<SOL>
    corpus_filtered.append([el.lower() for el in tokens if el.isalnum()])
    #</SOL>

print('\n',corpus_filtered[0])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
2.2.1. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
The lemmatizer from NLTK is based on WordNet. If you have not used wordnet before, you will likely need to download it from nltk (use the nltk.download() command)
Can you write Python code for it?

from nltk.stem import SnowballStemmer
from nltk.stem import WordNetLemmatizer

stemmer = SnowballStemmer('english')
wnl = WordNetLemmatizer()

print('Result for the first document in the corpus applying stemming')
print([stemmer.stem(el) for el in corpus_filtered[0]])

print('Result for the first document in the corpus applying lemmatization')
print([wnl.lemmatize(el) for el in corpus_filtered[0]])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
2.3. Cleaning
The third step consists of removing those words that are very common in language and do not carry out usefull semantic content (articles, pronouns, etc).
Once again, we might need to load the stopword files using the download tools from nltk
Exercise 7: In the second line below we read a list of common english stopwords. Clean lemmatized_abstracts by removing all tokens in the stopword list.
Can you write Python code for it?

from nltk.corpus import stopwords
stopwords_en = stopwords.words('english')

corpus_clean = []

for n, doc in enumerate(corpus_lemmatized):
    if not n%100:
        print('\rCleaning document', n, 'out of', ndocs, end='', flush=True)

    # Remove all tokens in the stopwords list and append the result to corpus_clean
    # <SOL>
    clean_tokens = [token for token in doc if token not in stopwords_en]
    # </SOL>
    corpus_clean.append(clean_tokens)

print('\n Let us check tokens after cleaning:')
print(corpus_clean[0])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
At this point, it is good to make sure to understand what has happened. In clean_abstracts we had a list of token lists. With it, we have constructed a Dictionary, D, which assigns an integer identifier to each token in the corpus.
After that, we have transformed each article (in clean_abstracts) in a list tuples (id, n).
Can you write Python code for it?

print('Original document (after cleaning):')
print(corpus_clean[0])
print('Sparse vector representation (first 10 components):')
print(corpus_bow[0][:10])
print('Word counts for the first project (first 10 components):')
print(list(map(lambda x: (D[x[0]], x[1]), corpus_bow[0][:10])))


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
Children are nested, and we can access specific child nodes by index. We can also access the text of specified elements. For instance:
Can you write Python code for it?

for child in root[0]:
    print(child.tag, child.attrib, child.text)


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

# Example data
n_art = len(NSF_data)
n_bins = 25
hot_tokens = [D[i] for i in ids_sorted[n_bins-1::-1]]
y_pos = np.arange(len(hot_tokens))
z = tf_sorted[n_bins-1::-1]/n_art

plt.figure()
plt.barh(y_pos, z, align='center', alpha=0.4)
plt.yticks(y_pos, hot_tokens)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()

# SORTED TOKEN FREQUENCIES:

# Example data
plt.figure()
plt.semilogy(tf_sorted)
plt.ylabel('Total number of occurrences')
plt.xlabel('Token rank')
plt.title('Token occurrences')
plt.show()


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
It also has children nodes over which we can iterate:
Can you write Python code for it?

for child in root:
    print(child.tag, child.attrib)


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
One of the advantages of the lemmatizer method is that the result of lemmmatization is still a true word, which is more advisable for the presentation of text processing results and lemmatization.
However, without using contextual information, lemmatize() does not remove grammatical differences. This is the reason why "is" or "are" are preserved and not replaced by infinitive "be".
As an alternative, we can apply .lemmatize(word, pos), where 'pos' is a string code specifying the part-of-speech (pos), i.e. the grammatical role of the words in its sentence. For instance, you can check the difference between wnl.lemmatize('is') and wnl.lemmatize('is, pos='v').
Exercise 6: Complete the following code fragment to lemmatize all documents in the corpus
Can you write Python code for it?

corpus_lemmatized = []

for n, doc in enumerate(corpus_filtered):
    if not n%200:
        print('\rLemmatizing document', n, 'out of', ndocs, end='', flush=True)

    #<SOL>
    corpus_lemmatized.append([wnl.lemmatize(el) for el in doc])
    #</SOL>

print('\nResult for the first document in the dataset applying lemmatization')
print('\n',corpus_lemmatized[0])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
We will extract some characteristics of the constructed dataset:
Can you write Python code for it?

print('Number of projects in dataset:', len(NSF_data))

####
budget_data = list(map(lambda x: x['budget'], NSF_data))
print('Average budget of projects in dataset:', np.mean(budget_data))

####
insti_data = list(map(lambda x: x['institution'], NSF_data))
print('Number of unique institutions in dataset:', len(set(insti_data)))

####
counts = dict()
for project in NSF_data:
    counts[project['year']] = counts.get(project['year'],0) + 1

print('Breakdown of projects by starting year:')
for el in counts:
    print(el, ':', counts[el])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
fromstring() parses XML from a string directly into an Element, which is the root element of the parsed tree. Other parsing functions may create an ElementTree, but we will not cover them here.
As an Element, root has a tag and a dictionary of attributes:
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
ids_sorted is a list of all token ids, sorted by decreasing number of occurrences in the whole corpus. For instance, the most frequent term is
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
or directly reading a string:
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_professor.ipynb".
The first task is:
which appears
Can you write Python code for it?

print("{0} times in the whole corpus".format(tf_sorted[0]))


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
1. Corpus acquisition
In this block we will work with collections of text documents. The objectives will be:

Find the most important topics in the collection and assign documents to topics
Analyze the structure of the collection by means of graph analysis

We will work with a collection of research projects funded by the US National Science Foundation, that you can find under the ./data directory. These files are publicly available from the NSF website.
(As a side note, there are many other available text collections to work with. In particular, the NLTK library has many examples, that you can explore using the nltk.download() tool.
import nltk
nltk.download()

for instance, you can take the gutemberg dataset
Mycorpus = nltk.corpus.gutenberg
text_name = Mycorpus.fileids()[0]
raw = Mycorpus.raw(text_name)
Words = Mycorpus.words(text_name)

Also, tools like Gensim or Sci-kit learn include text databases to work with).
1.1. Exploring file structure
NSF project information is provided in XML files. Projects are yearly grouped in .zip files, and each project is saved in a different XML file. To explore the structure of such files, we will use the file 160057.xml. Parsing XML files in python is rather easy using the ElementTree module.
To introduce some common functions to work with XML files we will follow this tutorial.
1.1.1. File format
To start with, you can have a look at the contents of the example file. We are interested on the following information of each project:

Project identifier
Project Title
Project Abstract
Budget
Starting Year (we will ignore project duration)
Institution (name, zipcode, and state)
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
1.1.2. Parsing XML
XML is an inherently hierarchical data format, and the most natural way to represent it is with a tree. The ElementTree module has two classes for this purpose:

ElementTree represents the whole XML document as a tree
Element represents a single node in this tree

We can import XML data by reading an XML file:
Can you write Python code for it?

import xml.etree.ElementTree as ET
tree = ET.parse(xmlfile)
root = tree.getroot


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
1.2. Building the dataset
Now, we will use the function you just implemented, to create a database that we will use throughout this module.
For simplicity, and given that the dataset is not too large, we will keep all projects in the RAM. The dataset will consist of a list containing the dictionaries associated to each of the considered projects in a time interval.
Can you write Python code for it?

# Construct an iterator (or a list) for the years you want to work with
years = [2015, 2016]
datafiles_path = '../data/'
NSF_data = []

for year in years:

    zpobj = zp.ZipFile(join(datafiles_path, str(year)+'.zip'))
    for fileinzip in zpobj.namelist():
        if fileinzip.endswith('xml'):

            #Some files seem to be incorrectly parsed
            try:
                project_dictio = parse_xmlproject(zpobj.read(fileinzip))
                if project_dictio['abstract']:
                    NSF_data.append(project_dictio)
            except:
                pass


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
2.2.1. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
The lemmatizer from NLTK is based on WordNet. If you have not used wordnet before, you will likely need to download it from nltk (use the nltk.download() command)
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
2.3. Cleaning
The third step consists of removing those words that are very common in language and do not carry out usefull semantic content (articles, pronouns, etc).
Once again, we might need to load the stopword files using the download tools from nltk
Exercise 7: In the second line below we read a list of common english stopwords. Clean lemmatized_abstracts by removing all tokens in the stopword list.
Can you write Python code for it?

from nltk.corpus import stopwords
stopwords_en = stopwords.words('english')

corpus_clean = []

for n, doc in enumerate(corpus_lemmatized):
    if not n%100:
        print('\rCleaning document', n, 'out of', ndocs, end='', flush=True)

    # Remove all tokens in the stopwords list and append the result to corpus_clean
    # <SOL>
    # </SOL>
    corpus_clean.append(clean_tokens)

print('\n Let us check tokens after cleaning:')
print(corpus_clean[0])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
2.4. Vectorization
Up to this point, we have transformed the raw text collection in a list of documents, where each documen is a collection of the words that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix). To do so, we will start using the tools provided by the gensim library.
As a first step, we create a dictionary containing all tokens in our text corpus, and assigning an integer identifier to each one of them.
Can you write Python code for it?

# Create dictionary of tokens
D = gensim.corpora.Dictionary(corpus_clean)
n_tokens = len(D)

print('The dictionary contains', n_tokens, 'terms')
print('First terms in the dictionary:')
for n in range(10):
    print(str(n), ':', D[n])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
Children are nested, and we can access specific child nodes by index. We can also access the text of specified elements. For instance:
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
Exercise 10: There are usually many tokens that appear with very low frequency in the corpus. Count the number of tokens appearing only once, and what is the proportion of them in the token list.
Can you write Python code for it?

# cold_tokens = <FILL IN>

print("There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100))


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
Exercise 12:  Count the number of tokens appearing only in a single article.
Exercise (bigrams): nltk provides an utility to compute n-grams from a list of tokens, in nltk.util.ngrams. Join all tokens in corpus_clean in a single list and compute the bigrams. Plot the 20 most frequent bigrams in the corpus.
2.5. Dictionary properties
As a final comment, note that gensim dictionaries contain a method dfs to compute the word counts automatically. In the code below we build a list all_counts that contains tuples (terms, document_counts).
Can you write Python code for it?

all_counts = [(D[el], D.dfs[el]) for el in D.dfs]
all_counts = sorted(all_counts, key=lambda x: x[1])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
For the rest of this notebook, we will work with the abstracts only. The list of all abstract will be the corpus w will work with.
Exercise 2: Generate a list of abstracts from NSF_data
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
In the second step, let us create a numerical version of our corpus using the doc2bow method. In general, D.doc2bow(token_list) transforms any list of tokens into a list of tuples (token_id, n), one per each token in token_list, where token_id is the token identifier (according to dictionary D) and n is the number of occurrences of such token in token_list.
Exercise 9: Apply the doc2bow method from gensim dictionary D, to all tokens in every document in clean_abstracts. The result must be a new list named corpus_bow where each element is a list of tuples (token_id, number_of_occurrences).
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
It also has children nodes over which we can iterate:
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
Note that we can interpret each element of corpus_bow as a sparse_vector. For example, a list of tuples
[(0, 1), (3, 3), (5,2)]

for a dictionary of 10 elements can be represented as a vector, where any tuple (id, n) states that position id must take value n. The rest of positions must be zero.
[1, 0, 0, 3, 0, 2, 0, 0, 0, 0]

These sparse vectors will be the inputs to the topic modeling algorithms.
As a summary, the following variables will be relevant for the next chapters:

D: A gensim dictionary. Term strings can be accessed using the numeric identifiers. For instance, D[0] contains the string corresponding to the first position in the BoW representation.
corpus_bow: BoW corpus. A list containing an entry per project in the dataset, and consisting of the (sparse) BoW representation for the abstract of that project.
NSF_data: A list containing an entry per project in the dataset, and consisting of metadata for the projects in the dataset

The way we have constructed the corpus_bow variable guarantees that the order is preserved, so that the projects are listed in the same order in the lists corpus_bow and NSF_data.
Before starting with the semantic analyisis, it is interesting to observe the token distribution for the given corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (I):
# Create a "flat" corpus with all tuples in a single list
corpus_bow_flat = [item for sublist in corpus_bow for item in sublist]

# Initialize a numpy array that we will use to count tokens.
# token_count[n] should store the number of ocurrences of the n-th token, D[n]
token_count = np.zeros(n_tokens)

# Count the number of occurrences of each token.
for x in corpus_bow_flat:
    # Update the proper element in token_count
    # scode: <FILL IN>

# Sort by decreasing number of occurences
ids_sorted = np.argsort(- token_count)
tf_sorted = token_count[ids_sorted]


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
One of the advantages of the lemmatizer method is that the result of lemmmatization is still a true word, which is more advisable for the presentation of text processing results and lemmatization.
However, without using contextual information, lemmatize() does not remove grammatical differences. This is the reason why "is" or "are" are preserved and not replaced by infinitive "be".
As an alternative, we can apply .lemmatize(word, pos), where 'pos' is a string code specifying the part-of-speech (pos), i.e. the grammatical role of the words in its sentence. For instance, you can check the difference between wnl.lemmatize('is') and wnl.lemmatize('is, pos='v').
Exercise 6: Complete the following code fragment to lemmatize all documents in the corpus
Can you write Python code for it?

corpus_lemmatized = []

for n, doc in enumerate(corpus_filtered):
    if not n%200:
        print('\rLemmatizing document', n, 'out of', ndocs, end='', flush=True)

    #<SOL>
    #</SOL>

print('\nResult for the first document in the dataset applying lemmatization')
print('\n',corpus_lemmatized[0])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
We can also filter out terms that appear in too few or too many of the documents in the dataset:
Exercise 8: Use method filter_extremes over dictionary D to remove words appearing in less than 5 documents and, also, those appearing in more than 75% of documents
Can you write Python code for it?

# <SOL>
# </SOL>

print('The dictionary contains', n_tokens, 'terms')
print('First terms in the dictionary:')
for n in range(10):
    print(str(n), ':', D[n])


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
fromstring() parses XML from a string directly into an Element, which is the root element of the parsed tree. Other parsing functions may create an ElementTree, but we will not cover them here.
As an Element, root has a tag and a dictionary of attributes:
Can you write Python code for it?


I am working on the file "TM1.IntrodNLP/NLP_py3_NSF/notebook/NLPintro_student.ipynb".
The first task is:
ids_sorted is a list of all token ids, sorted by decreasing number of occurrences in the whole corpus. For instance, the most frequent term is
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
1. Corpus acquisition.
In this notebook we will explore some tools for text processing and analysis and two topic modeling algorithms available from Python toolboxes.
To do so, we will explore and analyze collections of Wikipedia articles from a given category, using wikitools, that makes easy the capture of content from wikimedia sites.
(As a side note, there are many other available text collections to test topic modelling algorithm. In particular, the NLTK library has many examples, that can explore them using the nltk.download() tool.
import nltk
nltk.download()

for instance, you can take the gutemberg dataset
Mycorpus = nltk.corpus.gutenberg
text_name = Mycorpus.fileids()[0]
raw = Mycorpus.raw(text_name)
Words = Mycorpus.words(text_name)

Also, tools like Gensim or Sci-kit learn include text databases to work with).
In order to use Wikipedia data, we will select a single category of articles:
Can you write Python code for it?

site = wiki.Wiki("https://en.wikipedia.org/w/api.php")
# Select a category with a reasonable number of articles (>100)
# cat = "Economics"
cat = "Pseudoscience"
print cat


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
3. Semantic Analysis
The dictionary D and the Bag of Words in corpus_bow are the key inputs to the topic model algorithms. In this section we will explore two algorithms:

Latent Semantic Indexing (LSI)
Latent Dirichlet Allocation (LDA)

The topic model algorithms in gensim assume that input documents are parameterized using the tf-idf model. This can be done using
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
3.1. Latent Semantic Indexing (LSI)
Now we are ready to apply a topic modeling algorithm. Latent Semantic Indexing is provided by LsiModel.
Task: Generate a LSI model with 5 topics for corpus_tfidf and dictionary D. You can check de sintaxis for gensim.models.LsiModel.
Can you write Python code for it?

# Initialize an LSI transformation
n_topics = 5

# scode: lsi = <FILL IN>
lsi = gensim.models.LsiModel(corpus_tfidf, id2word=D, num_topics=n_topics)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
3.2. Latent Dirichlet Allocation (LDA)
There are several implementations of the LDA topic model in python:

Python library lda.
Gensim module: gensim.models.ldamodel.LdaModel
Sci-kit Learn module: sklearn.decomposition

3.2.1. LDA using Gensim
The use of the LDA module in gensim is similar to LSI. Furthermore, it assumes that a tf-idf parametrization is used as an input, which is not in complete agreement with the theoretical model, which assumes documents represented as vectors of token-counts.
To use LDA in gensim, we must first create a lda model object.
Can you write Python code for it?

ldag = gensim.models.ldamodel.LdaModel(
    corpus=corpus_tfidf, id2word=D, num_topics=10, update_every=1, passes=10)

ldag.print_topics()