I am working on the file "DifferentCode/Jupiter_notebooks/Logistic_Regression/Preprocessing_LogisticRegression_Binary.ipynb".
The first task is:
Описание датасета
Задача: по 38 признакам, связанных с заявкой на грант (область исследований учёных, информация по их академическому бэкграунду, размер гранта, область, в которой он выдаётся) предсказать, будет ли заявка принята. Датасет включает в себя информацию по 6000 заявкам на гранты, которые были поданы в университете Мельбурна в период с 2004 по 2008 год.
Полную версию данных с большим количеством признаков можно найти на https://www.kaggle.com/c/unimelb.
Can you write Python code for it?

data = pd.read_csv('data.csv')
data.shape

X = data.drop('Grant.Status', 1)
y = data['Grant.Status']


I am working on the file "DifferentCode/Jupiter_notebooks/Logistic_Regression/Preprocessing_LogisticRegression_Binary.ipynb".
The first task is:
Предобработка данных
Базовым этапом в предобработке любого датасета для логистической регрессии будет кодирование категориальных признаков, а так же удаление или интерпретация пропущенных значений (при наличии того или другого).
Can you write Python code for it?


I am working on the file "DifferentCode/Jupiter_notebooks/Logistic_Regression/Preprocessing_LogisticRegression_Binary.ipynb".
The first task is:
Сравнение признаковых пространств.
Can you write Python code for it?

data_numeric_scaled = pd.DataFrame(X_train_real_scaled, columns=numeric_cols)
list_cols = ['Number.of.Successful.Grant.1', 'SEO.Percentage.2', 'Year.of.Birth.1']
scatter_matrix(data_numeric_scaled[list_cols], alpha=0.5, figsize=(10, 10))
plt.show()


I am working on the file "DifferentCode/Jupiter_notebooks/Logistic_Regression/Preprocessing_LogisticRegression_Binary.ipynb".
The first task is:
Также в нём присутствуют пропущенные значения. Очевидны решением будет исключение всех данных, у которых пропущено хотя бы одно значение. Сделаем это:
Can you write Python code for it?


I am working on the file "DifferentialEquations.ipynb".
The first task is:
Evidently the forward Euler method is stable if $h\lambda$ is in the unit circle centered at $z=-1$, but not stable otherwise.
Implicit methods
Recall that forward Euler is the step
$$ \tilde y(h) = y(0) + h f(0, y(0)) . $$
This can be evaluated explicitly; all the terms on the right hand side are known so the approximation $\tilde y(h)$ is computed merely by evaluating the right hand side.
Let's consider an alternative, backward Euler (or "implicit Euler"),
$$ \tilde y(h) = y(0) + h f(h, \tilde y(h)) . $$
This is a (generally) nonlinear equation for $\tilde y(h)$.
For the test equation $y' = \lambda y$, the backward Euler method is
$$ \tilde y(h) = y(0) + h \lambda \tilde y(h) $$
or
$$ \tilde y(h) = \frac{y(0)}{1 - h \lambda} . $$
Evidently the stability function is
$$ R(z) = \frac{1}{1-z} .$$
Can you write Python code for it?

pyplot.figure()
C = pyplot.contour(xx, yy, numpy.abs(1/(1-zz)), [.5, 1, 1.5, 2])
pyplot.clabel(C)


I am working on the file "DifferentialEquations.ipynb".
The first task is:
Linear Stability Analysis
Why did Euler diverge (even if slowly) while RK4 solved this problem accurately?
And why do both methods diverge if the step size is too large?
We can understand the convergence of methods by analyzing the test problem
$$ y' = \lambda y $$
for different values of $\lambda$ in the complex plane.
One step of the Euler method with step size $h$ maps
$$ y \to y + h \lambda y = (1 + h \lambda) y =: R(h \lambda) y $$
where we have introduced the complex-valued function $R(z)$.

When is this map stable?
Can you write Python code for it?

pyplot.figure()
x = numpy.linspace(-2,2)
xx, yy = numpy.meshgrid(x, x)
zz = xx + 1j*yy
R = 1 + zz
C = pyplot.contour(xx, yy, numpy.abs(R))
pyplot.clabel(C)


I am working on the file "DifferentialEquations.ipynb".
The first task is:
Observations

We need to solve a system of equations on each time step.
The cost to solve the linear system is $O(n^3)$ for a system of $n$ equations.
The Jacobian matrix is $n\times n$, so requires $n^2$ storage when everything else requires $O(n)$.
I used finite differencing to compute the Jacobian, but many problems have efficient ways to compute an exact Jacobian (it just takes some programming).
Backward Euler is stable (as predicted by theory), but not very accurate.
What happens when you make the time step smaller or larger?

Midpoint Method
What if instead of evaluating the function at the end of the time step, we evaluated in the middle of the time step using the average of the endpoint values.  After all, something similar improved accuracy for our numerical integration...
$$ \tilde y(h) = y(0) + h f\left(\frac h 2, \frac{\tilde y(h) + y(0)}{2} \right) $$
Can you write Python code for it?

def ode_midpoint(f, y0, tfinal=1, h=0.1):
    y = y0.copy()
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        h = min(h, tfinal - t)
        # Solve x = y + h f(x)
        def residual(x):
            return x - h * f(t+h/2, (x + y)/2) - y
        y = solve_newtonfd(residual, y)
        t += h
        thist.append(t)
        yhist.append(y.copy())
    return numpy.array(thist), numpy.array(yhist)

thist, yhist = ode_midpoint(test.f, y0, h=.2, tfinal=50)
pyplot.figure()
pyplot.plot(thist, yhist, '*')
pyplot.plot(thist, test.y(thist, y0))
pyplot.title('Midpoint')

y0 = numpy.array([.2])
pyplot.figure()
for test in tests:
    thist, yhist = ode_midpoint(test.f, y0, h=1, tfinal=20)
    pyplot.plot(thist, yhist, '*', label=repr(test)+' Midpoint')
    pyplot.plot(thist, test.y(thist, y0), label=repr(test)+' exact')
pyplot.legend(loc='upper right')
pyplot.title('Cosine')


I am working on the file "DifferentialEquations.ipynb".
The first task is:
Questions

Does shrinking $h$ make this more accurate?
What if tfinal is extended?
What are the eigenvalues of $A$?

Runge-Kutta 4
Let's try a different method.  (We'll look back at where this comes from later.)
Can you write Python code for it?

def ode_rk4(f, y0, tfinal=1, h=0.1):
    y = numpy.array(y0)
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        h = min(h, tfinal - t)
        k1 = f(t, y)
        k2 = f(t+h/2, y + k1*h/2)
        k3 = f(t+h/2, y + k2*h/2)
        k4 = f(t+h, y + k3*h)
        y += h/6 * (k1 + 2*k2 + 2*k3 + k4)
        t += h
        thist.append(t)
        yhist.append(y.copy())
    return numpy.array(thist), numpy.array(yhist)

thist, yhist = ode_rk4(test.f, y0, h=.1, tfinal=5)
pyplot.figure()
pyplot.plot(thist, yhist, '*', label=repr(test)+' RK4')
pyplot.plot(thist, test.y(thist, y0), label=repr(test)+' exact')


# sample function
def f(x):
    return x**2

# derivative function
def f_p(x):
    return (f(x+dx)-f(x)) / dx

# second derivative function
def f_pp(x):
    return ((f_p(x+dx))-f_p(x)) / dx

# graph data arrays
x_data = np.linspace(-10,10,201)
y = f(x_data)

# our x value
x = float(input('x value from -10 to 10\n'))
# delta x, smallest value possible, python cant handle nummbers too small
dx = 0.0000001

# find derivative
M = f_p(x)
# create tangent line to plot
tan = M * (x_data-x) + f(x)

# find second derivative
M2 = f_pp(x)
# create tangent line to plot
tan2 = M2 * (x_data-x) + f(x)

# set up plot
fig, ax = plt.subplots()
# plot graph
ax.plot(x_data,y,x,f(x),'om',x_data,tan,x_data,tan2)
ax.grid(True)
#ax.axis([-10, 10, -100, 100])
plt.show()

print "f' = ",M
print "f'' = ",M2


I am working on the file "Differentiation/Differentiation.ipynb".
The first task is:
to find the gradient for $x^n$:
$f(x)=x^2$
  $\frac{d}{dx}f(x) = 2x$
$f(x)=x^3$
  $\frac{d}{dx}f(x) = 3x^2$
$f(x)=x^4$
  $\frac{d}{dx}f(x) = 4x^3$
$f(x)=5x^3 + 3x$
  $\frac{d}{dx}f(x) = 15x^2 + 3$
the formula is:
$f(x) = ax^n$
$\frac{d}{dx}f(x) = anx^{n-1}$
With y in this format:
$y = \frac{a}{x^n}$
We can use this equation:
$\frac{dy}{dx} = -\frac{an}{x^{n+1}}$
Example 1:
$y =  \frac{2}{x^3}$
$\frac{dy}{dx} = -\frac{6}{x^4}$
Example 2:
$y = \frac{1}{x}-\frac{2}{x^2}+\frac{3}{x^3}$
$\frac{dy}{dx} = -\frac{1}{x^2}+\frac{4}{x^3}-\frac{9}{x^4}$
Quick Rules:
Sum/Difference Rule:
(sum for addition and difference for subtraction)
$$\frac{d}{dx}[f(x)+g(x)] = \frac{d}{dx}f(x) + \frac{d}{dx}g(x)$$
Constant Multiple Rule:
where c is a constant and f is a function
$$(cf(x))′=c(f(x))'$$
Chain Rule
The Chain Rule is used to find the derivative of an equation with a power too large to expand using the binomial theorem
$$\frac{d}{dx}f(g(x)) = f'(g(x))\cdot g'(x)$$
Example 1:
$y = (3x^2 + 5x)^{10}$
let $u = 3x^2 + 5x$
so now the derivative of the function being applied to u:
$y = u^{10}$
$\frac{dy}{du} = 10u^9$
also differentiating u itself:
$\frac{du}{dx} = 6x + 5$
and to combine them:
$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$
the $du$'s cancel each other out
so the answer is:
(for an exam we can skip the middel steps and go to here directly)
$\frac{dy}{dx} = 10(3x^2 + 5x)^9(6x + 5)$
$\frac{dy}{dx} = (60x + 50)(3x^2 + 5x)^9$
Can you write Python code for it?

# Python example of previous example
# since we are using a computer, we dont need the chain rule but this is to prove it works

# function from previous example
def f(x):
    return (3*x**2 + 5*x)**10

# create line data
x_data = np.linspace(0,0.4,4001)
y = f(x_data)

# set params
x = 0.3
d = 0.000001

# use equation from example
M_chain = (60*x + 50)*(3*x**2 + 5*x)**9
# create tangent
tan_chain = M_chain * (x_data-x) + f(x)

# use equation from first princeple
M = (f(x+d)-f(x)) /d
# create tangent
tan = M * (x_data-x) + f(x)

# plot
fig, ax = plt.subplots()
fig, ax2 = plt.subplots()
ax.grid(True)
ax2.grid(True)
ax.plot(x_data,y,x,f(x),'om',x_data,tan,'g:')
ax2.plot(x_data,y,x,f(x),'om',x_data,tan_chain,'r--')
plt.show()
print 'dydx = ',M
print 'using chain rule:\ndydx =', M_chain


I am working on the file "Diffusion Monte Carlo.ipynb".
The first task is:
During each step of the production phase, the data is accumulated to evaluate the energy, energy variance and wave function $\psi$.
Can you write Python code for it?

class Accumulator(object):
    def __init__(self):
        self.E_sum = 0
        self.E_squared_sum = 0
        self.r_max = 4.0
        self.N_psi = 100
        self.psi = np.zeros((self.N_psi,))

    def reset(self):
        self.E_sum = 0
        self.E_squared_sum = 0
        self.psi = np.zeros((self.N_psi,))

    def handle_data(self, E_T, r):
        self.E_sum += E_T
        self.E_squared_sum += E_T**2

        for j in range(r.shape[0]):
            r_squared = r[j,:].dot(r[j,:])
            i_bin = int(np.sqrt(r_squared) / self.r_max * self.N_psi)
            if i_bin < self.N_psi:
                self.psi[i_bin] += 1

# Initialize walkers
r = np.zeros((N_T, 3))
N = N_T
E_T = 0 # initial guess for the ground state energy

thermal_steps = int(0.2*T_steps)

acc = Accumulator()

# Time step
for i in range(thermal_steps+T_steps):
    # Diffusion step
    r = diffusion(r, dt)

    # Branching step
    r, N = branching(r, E_T, dt)

    # Adjustment
    E_T += adjust(N_T, N)

    # Accumulation
    if i == thermal_steps:
        acc.reset()
    acc.handle_data(E_T, r)

E_avg = acc.E_sum/T_steps
E_var = acc.E_squared_sum/T_steps - E_avg**2

print E_avg
print E_var
print acc.psi

plt.plot(acc.psi)
plt.show()


I am working on the file "Diffusion Monte Carlo.ipynb".
The first task is:
In the branching step, the number of walkers is modified by the parameter $E_T$, in the physical picture due to the potential energy term. The branching is implemented by computing the branching factor $q$ via
$$q=e^{-\Delta\tau (V(x) - E_T)}$$
which then determines if a walker is cloned, survives or dies. If $q<1$, the walker dies with a probability of $1-q$. If $q > 1$, the walker is copied.
Can you write Python code for it?

def branching(r, E_T, dt):

    r_new = np.zeros((0,3))
    q = np.zeros((r.shape[0],))

    for j in range(r.shape[0]):

        # branching factor
        q[j] = np.exp(-dt * (pot(r[j,:]) - E_T))

        # branching
        if q[j] - int(q[j]) > random.uniform():
            count_new = int(q[j]) + 1
        else:
            count_new = int(q[j])

        # generate new walker array
        for c in range(count_new):
            r_new = np.append(r_new,r[j:j+1,:],axis=0)

    return r_new, r_new.shape[0]


I am working on the file "Diffusion Monte Carlo.ipynb".
The first task is:
Potential of the harmonic oscillator in three dimensions
$$V(\mathbf x) = \frac{1}{2}\mathbf x ^2$$
Can you write Python code for it?


I am working on the file "Diffusion Monte Carlo.ipynb".
The first task is:
Set the time step size, the target number of walkers as well as the target number of time steps
Can you write Python code for it?


I am working on the file "Diffusion Monte Carlo.ipynb".
The first task is:
The first 20% of the time steps are used as thermalization steps, after that, the measured values are reset and the real simulation continues
In a timestep
1. Compute one DMC step on each walker
2. Remove dead walkers
3. Adjust $E_T$ to drive $N$ towards $N_T$
4. Accumulate data to measure $\left<E\right>$, its variance and the ground state wave function
In the diffusion step, each walker is moved randomly in the search space, in the physical picture due to the kinetic energy term. The random step is choosen from a Gaussian distribution with variance $\Delta t$.
Can you write Python code for it?

def diffusion(r, dt):
    return r + random.normal(size=(r.shape[0],3)) * np.sqrt(dt)


I am working on the file "Diffusion of FAKE news.ipynb".
The first task is:
Calculate and plot standart deviation
Can you write Python code for it?

avgdiff_std = [0 for _ in range(max_days)] # standard deviation for every day for all collections
number_tweets = [[] for _ in range(max_days)] # number of tweets for every day for every collection

for d in range(max_days):
    for c in collections:
        # if there is an entry for this day for this collection
        if d < len(day[c]):
            # add number of tweets for this day for this colletion to the number_tweets for this day
            number_tweets[d].append(day[c][d])

    # calculate standard deviation for this day
    avgdiff_std[d] = np.std(number_tweets[d])

plt.ylabel('Standart deviation for average number of tweets per day')
plt.xlabel('Day')
plt.step(range(1,len(avgdiff_std)+1),avgdiff_std, 'r')
plt.title('Standard deviation for fake news average')
plt.show()


I am working on the file "Diffusion of FAKE news.ipynb".
The first task is:
Calculate and share of values inside one standard deviation for every day
Can you write Python code for it?

inside_std = [0 for _ in range(max_days)] # number of values inside one standard deviation for every day
inside_std_share = [0 for _ in range(max_days)] # share of values inside one standard deviation for every day

for d in range(max_days):
    for c in collections:
        # set borders of mean plusminus one std
        lowest = averagediff[d] - avgdiff_std[d]
        highest = averagediff[d] + avgdiff_std[d]

        # if there is entray for this day for this collection and its value is inside the borderes
        if d < len(day[c]) and (day[c][d] >= lowest and day[c][d] <= highest):

            # increment number of values inside one std for this day
            inside_std[d] += 1

    # calculate the share of values inside one std for this day
    inside_std_share[d] = inside_std[d] / float(len(number_tweets[d]))

plt.ylabel('Percent of values in 1 std from average')
plt.xlabel('Day')
plt.scatter(range(1,len(inside_std_share)+1),inside_std_share, c='r')
plt.title('Percentage of values inside the range\n of one standard deviation from mean for fake news')
plt.show()


I am working on the file "Diffusion of FAKE news.ipynb".
The first task is:
Calculate average diffusion duration (number of days until difussion is dead)
Can you write Python code for it?

diffDurationAvg = 0; # average duration of diffusion

durations  = [len(day[col]) for col in collections] # all durations

diffDurationAvg = np.mean(durations) # mean duration

diffDurationAvg_std = np.std(durations) # standard deviation for the mean

print "Average diffusion duration: %.2f days" % diffDurationAvg
print "Standard deviation: %.2f days" % diffDurationAvg_std


I am working on the file "Diffusion of FAKE news.ipynb".
The first task is:
In logarithmic scale
Can you write Python code for it?

plt.ylabel('Average number of tweets')
plt.xlabel('Day')
plt.yscale('log')
plt.step(range(1,len(averagediff)+1),averagediff, 'r', label="fake news")
plt.step(range(1,len(averagediff_real)+1),averagediff_real, 'g', label="real news")
plt.legend()
plt.title('Average diffusion for both types of news in logarithmic scale')
plt.show()


I am working on the file "Diffusion of FAKE news.ipynb".
The first task is:
Init config
Select appropriate:
- database server (line 1): give pymongo.MongoClient() an appropriate parameter, else it is localhost
- database (line 2): either client.databasename or client.['databasename']
Can you write Python code for it?

client = pymongo.MongoClient("46.101.236.181")
db = client.allfake

# get collection names
collections = sorted([collection for collection in db.collection_names()])


I am working on the file "Diffusion of FAKE news.ipynb".
The first task is:
Plot cummulative diffusion of all news together
Can you write Python code for it?

summ_of_diffusions_cumulative = [0] * max_days
summ_of_diffusions_cumulative[0] = summ_of_diffusions[0]
for d in range(1, max_days):
    summ_of_diffusions_cumulative[d] += summ_of_diffusions_cumulative[d-1] + summ_of_diffusions[d]

plt.step(range(len(summ_of_diffusions_cumulative)),summ_of_diffusions_cumulative, 'g')
plt.xlabel('Day')
plt.ylabel('Cummulative number of tweets')
plt.title('Cummulative diffusion of all fake news together')
plt.show()


I am working on the file "Diffusion of FAKE news.ipynb".
The first task is:
Store average diffusion data on hard drive to use by another jupyter notebook
Can you write Python code for it?

averagediff_fake = averagediff
%store averagediff_fake


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
Average diffusion per day for all news
Can you write Python code for it?

averagediff = [0 for _ in range(max_days)] # average diffusion for every day

for collection in collections:
    for i,d in enumerate(day[collection]):
        averagediff[i] += d / len(collections)

plt.xlabel('Day')
plt.ylabel('Average number of tweets')
plt.step(range(1,len(averagediff)+1),averagediff, 'g')
plt.title('Average diffusion of real news')
plt.show()


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
Calculate and plot share of values inside one standard deviation for every day
Can you write Python code for it?

inside_std = [0 for _ in range(max_days)] # number of values inside one standard deviation for every day
inside_std_share = [0 for _ in range(max_days)] # share of values inside one standard deviation for every day

for d in range(max_days):
    for c in collections:
        # set borders of mean plusminus one std
        lowest = averagediff[d] - avgdiff_std[d]
        highest = averagediff[d] + avgdiff_std[d]

        # if there is entray for this day for this collection and its value is inside the borderes
        if d < len(day[c]) and (day[c][d] >= lowest and day[c][d] <= highest):

            # increment number of values inside one std for this day
            inside_std[d] += 1

    # calculate the share of values inside one std for this day
    inside_std_share[d] = inside_std[d] / float(len(number_tweets[d]))

plt.ylabel('Percent of values in 1 std from average')
plt.xlabel('Day')
plt.scatter(range(1,len(inside_std_share)+1),inside_std_share, c='g')
plt.title('Percentage of values inside the range\n of one standard deviation from mean for real news')
plt.show()


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
Calculate and plot standart deviation
Can you write Python code for it?

avgdiff_std = [0 for _ in range(max_days)] # standard deviation for every day for all collections
number_tweets = [[] for _ in range(max_days)] # number of tweets for every day for every collection

for d in range(max_days):
    for c in collections:
        # if there is an entry for this day for this collection
        if d < len(day[c]):
            # add number of tweets for this day for this colletion to the number_tweets for this day
            number_tweets[d].append(day[c][d])

    # calculate standard deviation for this day
    avgdiff_std[d] = np.std(number_tweets[d])

plt.ylabel('Standart deviation for average number of tweets per day')
plt.xlabel('Day')
plt.step(range(1,len(avgdiff_std)+1),avgdiff_std, 'g')
plt.title('Standard deviation for real news average')
plt.show()


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
In logarithmic scale
Can you write Python code for it?

plt.ylabel('Average number of tweets')
plt.xlabel('Day')
plt.yscale('log')
plt.step(range(1,len(averagediff_fake)+1),averagediff_fake, 'r', range(1,len(averagediff)+1),averagediff, 'g')
plt.show()


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
Init config
Select appropriate:
- database server (line 1): give pymongo.MongoClient() an appropriate parameter, else it is localhost
- database (line 2): either client.databasename or client.['databasename']
Can you write Python code for it?


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
Plot average diffusion for both real and fake news on one graph
Can you write Python code for it?

# from hard drive, load data for average diffusion of fake news
%store -r averagediff_fake

plt.xlabel('Day')
plt.ylabel('Average number of tweets')
plt.step(range(1,len(averagediff)+1),averagediff, 'g', label="real news")
plt.step(range(1,len(averagediff_fake)+1),averagediff_fake, 'r', label="fake news")
plt.legend()
plt.title('Average diffusion for both types of news')
plt.show()


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
Plot cummulative diffusion of all news together
Can you write Python code for it?

summ_of_diffusions_cumulative = [0] * max_days #
summ_of_diffusions_cumulative[0] = summ_of_diffusions[0]
for d in range(1, max_days):
    summ_of_diffusions_cumulative[d] += summ_of_diffusions_cumulative[d-1] + summ_of_diffusions[d]

plt.step(range(len(summ_of_diffusions_cumulative)),summ_of_diffusions_cumulative, 'g')
plt.xlabel('Day')
plt.ylabel('Cummulative number of tweets')
plt.title('Cummulative diffusion of all real news together')
plt.show()


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
Plot diffusion for every day for all news together
Can you write Python code for it?

# the longest duration of diffusion among all news headlines
max_days = max([len(day[coll]) for coll in \
                         [days_col for days_col in day] ])

summ_of_diffusions = [0] * max_days # summary diffusion for every day

# calculate summary diffusion for every day
for d in range(max_days):
    for c in collections:
        # if there is an entry for this day for this collection, add its number of tweets to the number of this day
        if d < len(day[c]):
            summ_of_diffusions[d] += day[c][d]

plt.step(range(len(summ_of_diffusions)),summ_of_diffusions, 'g')
plt.xlabel('Day')
plt.ylabel('Number of tweets')
plt.title('Diffusion of all real news together')
plt.show()


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
Store average diffusion data on hard drive to use by another jupyter notebook
Can you write Python code for it?

averagediff_real = averagediff
%store averagediff_real


I am working on the file "Diffusion of REAL news.ipynb".
The first task is:
The same graph but in logarithmic scale
Can you write Python code for it?

plt.ylabel('Average number of tweets')
plt.xlabel('Day')
plt.yscale('log')
plt.step(range(1,len(averagediff)+1),averagediff, 'g')
plt.show()


I am working on the file "Diffusion/DMC_propagator.ipynb".
The first task is:
Diffusion Monte Carlo propagators
Most of the equations taken from Chapter 24 ("Projector quantum Monte Carlo") in "Interacting Electrons" (2016) by R.M. Martin, L. Reining, and D.M. Ceperley.
Trotter breakup
Can you write Python code for it?

T_op = Symbol('That')   # Kinetic energy operator
V_op = Symbol('Vhat')   # Potential energy operator
tau = Symbol('tau')     # Projection time
n = Symbol('n',isinteger=True)  # Number of timestep divisions
dt = Symbol(r'\Delta\tau')  # Time for individual timestep

# Eq. 24.7
Eq(exp(-tau *(T_op + V_op)),
  Limit(exp(-dt*T_op) * exp(-dt*V_op),n,oo))


I am working on the file "Diffusion/DMC_propagator.ipynb".
The first task is:
Sampling the drift-diffusion term
Can you write Python code for it?

chi = Symbol('chi')   # gaussian random sample with zero mean and variance delta tau
r = Symbol('r')
rp = Symbol("r'")

# Sample new positions with this formula  (Eq 23.13)
# Question - how to detemine sampling formula from evolution equation/distribution above?
sample_drift_diffusion = Eq(rp, r + dt * F + chi)
sample_drift_diffusion


I am working on the file "Diffusion/DMC_propagator.ipynb".
The first task is:
Scaling the drift
In QMCPACK, the drift term is scaled.  (From C. J. Umrigar, M. P. Nightingale, K. J. Runge "A diffusion Monte Carlo algorithm with very small time-step errors"  JCP 99, 2865 (1993) doi: 10.1063/1.465195 )
Can you write Python code for it?

Fmag = Symbol('Fmag^2')
epsilon = Symbol('epsilon')

drift_scale = Piecewise( (tau,Fmag < epsilon ),
         ((sqrt(1 + 2*Fmag*tau)-1)/Fmag, True))
drift_scale

scaled_drift = F*drift_scale
scaled_drift


I am working on the file "Diffusion/DMC_propagator.ipynb".
The first task is:
Values for Testing
Can you write Python code for it?

class SymPrinter(printing.lambdarepr.NumPyPrinter):
    def _print_Symbol(self, expr):
        if expr.name == r'\Delta\tau':
            return 'dt'
        return expr.name

# RNG corresponding to src/ParticleBase/RandomSeqGenerator.h
def gaussian_rng_list(n):
    input_rng = [0.5]*(n+1)
    slightly_less_than_one = 1.0 - sys.float_info.epsilon
    vals = []
    for i in range(0,n,2):
        temp1 = math.sqrt(-2.0 * math.log(1.0- slightly_less_than_one*input_rng[i]))
        temp2 = 2*math.pi*input_rng[i+1]
        vals.append(temp1*math.cos(temp2))
        vals.append(temp2*math.sin(temp2))
    if n%2 == 1:
        temp1 = math.sqrt(-2.0 * math.log(1.0- slightly_less_than_one*input_rng[n-1]))
        temp2 = 2*math.pi*input_rng[n]
        vals.append(temp1*math.cos(temp2))
    return vals

chi_vals = np.array(gaussian_rng_list(6)).reshape((2,3))
chi_vals

r_vals = np.array( [ [1.0, 0.0, 0.0],
                     [0.0, 0.0, 1.0]])
tau_val = 0.1
scaled_chi_vals = chi_vals * math.sqrt(tau_val)

drift_diffuse_func = lambdify((r, F, chi, dt),sample_drift_diffusion.rhs, printer=SymPrinter)
scaled_drift_func = lambdify((tau, Fmag, F), scaled_drift.subs(epsilon, sys.float_info.epsilon) )
# For a constant wavefunction, gradient is zero
for r_val, chi_val in zip(r_vals, scaled_chi_vals):
    rp_val = np.zeros(3)
    rp_val = drift_diffuse_func(r_val, np.zeros(3), chi_val, tau_val)
    print rp_val

#  For a linear wavefunction, gradient is constant
grad_coeff = np.array([ 1.0, 2.0, 3.0])
for r_val, chi_val in zip(r_vals, scaled_chi_vals):
    rp_val = np.zeros(3)
    # Scaled drift is already multiplied by dt, accomodate by setting dt param to 1.0
    rp_val = drift_diffuse_func(r_val, scaled_drift_func(tau_val, np.dot(grad_coeff, grad_coeff),grad_coeff), chi_val, 1.0)
    print ['%.15g'%v for v in rp_val]

# Compute scaled drift
drift_scale.subs({epsilon:sys.float_info.epsilon, tau:tau_val, Fmag:np.dot(grad_coeff, grad_coeff)})


I am working on the file "DifraccionFraunhofer/Difraccion_Ejercicio.ipynb".
The first task is:
Como vemos en la  ecuación (4) hay una relación lineal entre la posición $x_m$ en la que se localiza el mínimo de difracción de orden m y el orden
de difracción m. Por ejemplo la distancia entre el mínimo de orden -2 y el de orden 3 está dada por:
$$x_3 - x_2 = \frac{\lambda D}{b}(3-(-2)) = \frac{5\lambda D}{b}$$ (5)
Esta relación la aprovecharemos para estimar el valor de la anchura de la rendija problema si todos los demás parámetros son conocidos.
Ejercicio 1

Obtener la dimension de la rendija $b$ a partir de la diferencia en las posiciones de dos minimos, las cuales pueden medirse en el perfil de la irradiancia mostrado en la figura de la derecha. Para ello utilizar los parametros de longitud de onda y distancia de trabajo del codigo mostrado anteriormente.

2.2 Difracción por una abertura circular
Al incidir un haz de luz sobre una abertura circular, el diagrama de difracción que se observa consiste en una serie de franjas circulares
concéntricas. El fenómeno es similar al de una rendija, de hecho podríamos considerar la abertura circular como una colecci´on de rendijas de
longitud variable que rellenan toda la abertura. En un punto dado de la pantalla la irradiancia resultante se obtiene mediante la suma coherente de
las contribuciones de cada una de las rendijas elementales. Esto da lugar a que la expresi´on de la irradiancia sea ligeramente distinta de la
expresión (1) y viene dada por:
$$I(\theta) = I(0) \left(\frac{2 J_1\left(k a \sin\left(\theta\right)\right)}{k a \sin(\theta)}\right)^2$$ (6)
donde a es el radio de la abertura circular, $\theta$ la posición angular del punto de observación en la pantalla respecto al centro de la abertura y
$k = 2\pi/\lambda$ es el número de ondas. En la expresión (6) $J_1$ representa la función de Bessel de primera clase de orden 1. La expresión (6) recibe el nombre de figura de Airy.
(A) Diagrama de difracción de una abertura circular
(B) Representación de la intensidad a lo largo la línea AB de la figura de Airy, normalizada a su valor máximo
Can you write Python code for it?

R0 = 1e-4 #Radio de la abertura
r = np.sqrt(x**2 + y**2)
circular = r<R0

#Transformada de fourier
tfcirc = fftshift(fft2(circular))
irradiancia = np.abs(tfcirc).T**2

fig2,ax = plt.subplots(1,2,figsize=(15,5))
ax[0].pcolormesh(xnew*100,ynew*100,irradiancia,cmap='hot') # para que salga en cm
ax[0].set_xlim(-4,4)
ax[0].set_ylim(-4,4)
ax[0].set_title("Irradiancia en la pantalla")
ax[0].set_xlabel('X (cm)')
ax[0].set_ylabel('Y (cm)')

# Perfil de irradiancia a lo largo del eje X
corte = irradiancia[100,:]
ax[1].plot(xnew*100,corte)
ax[1].set_title("Perfil de Irradiancia")
ax[0].set_xlabel('X (cm)')
ax[0].set_ylabel('Irradiancia (u.a.)')


I am working on the file "DifraccionFraunhofer/Difraccion_Ejercicio.ipynb".
The first task is:
En la figura 2 se muestra la distribución de irradiancia producida por una abertura circular junto con la gráfica de la ecuación (6). El primer mínimo
de irradiancia (primer anillo oscuro) se obtiene cuando:
$$k a \sin(\theta) = 3.83$$ (7)
Si la distancia desde la abertura a la pantalla de observación es considerablemente mayor que el radio del primer anillo oscuro, podemos aproximar en
la ecuación (7) el seno por el ángulo. De esta manera el ángulo subtendido por el primer anillo oscuro está dado por:
$$\theta ≈ \frac{3.83}{\pi}\frac{\lambda}{2a} = 1.22\frac{\lambda}{D_a}$$ (8)
donde $D_a$ = 2a es el diámetro de la abertura circular. El radio del primer anillo oscuro será:
$$r_1 = 1.22\frac{\lambda D}{D_a}$$ (9)
Ejercicio 2

Obtener el diametro de la abertura circular $D_a$ utilizando las figuras anteriores y la ecuacion deducida para determinar la posicion del primer minimo del patron de difraccion

2.3 Doble Rendija
Can you write Python code for it?

width_rendija = 0.5e-4
distance_rendijas = 5e-4
doblerendija = np.abs(x-distance_rendijas)<width_rendija/2   # Definición de la función rendija
doblerendija += np.abs(x+distance_rendijas)<width_rendija/2

#Transformada de fourier
tfdoble = fftshift(fft2(doblerendija))
irradiancia = np.abs(tfdoble).T**2

fig3,ax = plt.subplots(1,3,figsize=(18,5))

ax[0].pcolormesh(x*1000,y*1000,doblerendija,cmap='hot') # para que salga en cm
ax[0].set_xlim(-1,1)
ax[0].set_ylim(-0,0.1)
ax[0].set_title("Doble rendija")
ax[0].set_xlabel('X (cm)')

ax[1].pcolormesh(xnew*100,ynew*100,irradiancia,cmap='hot') # para que salga en cm
ax[1].set_xlim(-4,4)
ax[1].set_ylim(-0,0.1)
ax[1].set_title("Irradiancia en la pantalla")
ax[1].set_xlabel('X (cm)')
ax[1].set_ylabel('Y (cm)')

# Perfil de irradiancia a lo largo del eje X
corte = irradiancia[100,:]
ax[2].plot(xnew*100,corte)
ax[2].set_title("Perfil de Irradiancia")
ax[2].set_xlabel('X (cm)')
ax[2].set_ylabel('Irradiancia (u.a.)')


I am working on the file "Digit Classification using SciKit-Learn.ipynb".
The first task is:
Extract 90% of our available data as training data for the models.
Can you write Python code for it?

X_train = X_digits[:int(.9 * n_samples)]
X_train

y_train = y_digits[:int(.9 * n_samples)]
y_train


I am working on the file "Digit Classification using SciKit-Learn.ipynb".
The first task is:
Use the Logistic Regression algorithm
Can you write Python code for it?

logistic = linear_model.LogisticRegression()
logistic

log_regression_fitting = logistic.fit(X_train, y_train)
log_regression_fitting

log_regression_score = log_regression_fitting.score(X_test, y_test)
print('LogisticRegression score: %f' % log_regression_score)


I am working on the file "Digit Classification using SciKit-Learn.ipynb".
The first task is:
We can see from the outcomes KNN was better at predicting the target result, with ~96% accuracy.
Can you write Python code for it?

print('KNN score: %f' % knn_score)
print('LGR score: %f' % log_regression_score)


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Accuracy (98.61%) of Convolution Neural Network (CNN) model has improved as compared to the accuracy (97.95%) of Multi-layer Perceptron (MLP) model.
The accuracy of CNN model can be further increased by:
- increasing the epoch number while fitting the model
- adding more convolution and pooling layers to the model
Plot correctly and incorrectly predicted images
Let's plot some images which are correctly predicted and some images which are incorrectly predicted on our test dataset.
Can you write Python code for it?

# get predicted values
predicted_classes = model.predict_classes(X_val)

# get index list of all correctly predicted values
correct_indices = np.nonzero(np.equal(predicted_classes, y_val))[0]

# get index list of all incorrectly predicted values
incorrect_indices = np.nonzero(np.not_equal(predicted_classes, y_val))[0]

print ('Correctly predicted: %i' % np.size(correct_indices))
print ('Incorrectly predicted: %i' % np.size(incorrect_indices))

plt.figure(figsize=[20,8])
for i, correct in enumerate(correct_indices[:6]):
    plt.subplot(1,6,i+1)
    plt.imshow(X_val[correct].reshape(28,28), cmap='gray', interpolation='none')
    plt.title("Predicted {}, Class {}".format(predicted_classes[correct], y_val[correct]))

plt.figure(figsize=[20,8])
for i, incorrect in enumerate(incorrect_indices[:6]):
    plt.subplot(1,6,i+1)
    plt.imshow(X_val[incorrect].reshape(28,28), cmap='gray', interpolation='none')
    plt.title("Predicted {}, Class {}".format(predicted_classes[incorrect], y_val[incorrect]))


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Confusion Matrix
Can you write Python code for it?

# we have digit labels from 0 to 9
# we can either manually create a class variable with those labels
# class_names = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

# or, we can take unique values from train dataset's labels
class_names = np.unique(y_train)

# Compute confusion matrix
cnf_matrix = confusion_matrix(y_val, predicted_classes)
np.set_printoptions(precision=2)

print ('Confusion Matrix in Numbers')
print (cnf_matrix)
print ('')

cnf_matrix_percent = cnf_matrix.astype('float') / cnf_matrix.sum(axis=1)[:, np.newaxis]

print ('Confusion Matrix in Percentage')
print (cnf_matrix_percent)
print ('')

true_class_names = class_names
predicted_class_names = class_names

df_cnf_matrix = pd.DataFrame(cnf_matrix,
                             index = true_class_names,
                             columns = predicted_class_names)

df_cnf_matrix_percent = pd.DataFrame(cnf_matrix_percent,
                                     index = true_class_names,
                                     columns = predicted_class_names)

plt.figure(figsize = (8,6))

#plt.subplot(121)
ax = sns.heatmap(df_cnf_matrix, annot=True, fmt='d')
ax.set_ylabel('True values')
ax.set_xlabel('Predicted values')
ax.set_title('Confusion Matrix in Numbers')

'''
plt.subplot(122)
ax = sns.heatmap(df_cnf_matrix_percent, annot=True)
ax.set_ylabel('True values')
ax.set_xlabel('Predicted values')
'''


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Converting target variable values into one-hot format
The output/target variable is in the format 0 to 9. As this is a multi-class classification problem, we convert the output class values into one-hot format which is simply a binary matrix, i.e.

value 0 will be converted to one-hot format as [1, 0, 0, 0, 0, 0, 0, 0, 0]
value 1 will be converted to one-hot format as [0, 1, 0, 0, 0, 0, 0, 0, 0]
value 2 will be converted to one-hot format as [0, 0, 1, 0, 0, 0, 0, 0, 0]
and so on...
Can you write Python code for it?

print (y_train.shape)
print (y_train[0])

# one hot encode outputs
# note that we have new variables with capital Y
# Y_train is different than y_train
Y_train = np_utils.to_categorical(y_train)
num_classes = Y_train.shape[1]

print (y_train.shape, Y_train.shape)
print (y_train[0], Y_train[0])


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Define Convolutional Neural Network (CNN) Model
Convolution Layer
- We define 32 feature maps with the size of 5x5 matrix
- We use ReLU (Rectified Linear Units) as the activation function
- This layer expects input image size of 1x28x28 ([pixels][height][weight])
Max Pooling Layer
- It has a pool size of 2x2
Dropout Layer
- Configured to randomly exclude 20% of neurons in the layer to reduce overfitting
Flatten
- Flattens the image into a single dimensional vector which is required as input by the fully connected layer
Fully connected Layer
- Contains 128 neurons
- relu is used as an activation function
- Output layer has num_classes=10 neurons for the 10 classes
- softmax activation function is used in the output layer
- adam gradient descent algorithm is used as optimizer to learn and update weights
Can you write Python code for it?

# baseline model for CNN
def baseline_model():
    # create model
    model = Sequential()
    model.add(Conv2D(32, (5, 5), input_shape=(1, 28, 28), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.2))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dense(num_classes, activation='softmax'))
    # compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Fit and Evaluate Model
The model is fit over 5 epochs/iteration. It takes a batch of 200 images in each iteration. Validation dataset is used for validation. The epochs may be increased to improve accuracy.
Finally, validation dataset is used to evaluate the model by calculating the model's classification accuracy.
Can you write Python code for it?

model = baseline_model()
model.fit(X_train, Y_train, validation_data=(X_val, Y_val), epochs=5, batch_size=200, verbose=1)

model.summary()

scores = model.evaluate(X_val, Y_val, verbose=0)
print (scores)
print ('Score: {}'.format(scores[0]))
print ('Accuracy: {}'.format(scores[1]))


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Get values of data
Can you write Python code for it?

X_train = X_train.values.astype('float32') # pixel values of all images in train set
y_train = y_train.values.astype('int32') # labels of all images
X_test = test.values.astype('float32') # pixel values of all images in test set


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Normalizing input values
As we can see above, the pixel values for each image are gray scaled between 0 and 255. We now, normalize those values from 0-255 to 0-1.
Can you write Python code for it?

# pixel values are gray scale between 0 and 255
# normalize inputs from 0-255 to 0-1
X_train = X_train / 255
X_test = X_test / 255
print (X_train[1])


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Plot correctly and incorrectly predicted images
Let's plot some images which are correctly predicted and some images which are incorrectly predicted on our validation dataset.
Can you write Python code for it?


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Plotting images and their class values
Can you write Python code for it?

plt.figure(figsize=[20,8])
for i in range(6):
    plt.subplot(1,6,i+1)
    # Here, we reshape the 784 pixels vector values into 28x28 pixels image
    plt.imshow(X_train[i].reshape(28, 28), cmap='gray', interpolation='none')
    plt.title("Class {}".format(y_train[i]))

# fix random seed for reproducibility
random_seed = 7
np.random.seed(random_seed)


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Reshaping images
The image dimension expected by Keras for 2D (two-dimensional) convolution is in the format of [pixels][width][height].
For RGB color image, the first dimension (pixel) value would be 3 for the red, green and blue components. It's like having 3 image inputs for every single color image. In our case (for MNIST handwritten images), we have gray scale images. Hence, the pixel dimension is set as 1.
Can you write Python code for it?

# reshape to be [samples][pixels][width][height]
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28).astype('float32')
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28).astype('float32')
X_val = X_val.reshape(X_val.shape[0], 1, 28, 28).astype('float32')

print (num_pixels, X_train.shape, X_test.shape, X_val.shape)

print (X_train[1])


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Splitting train dataset into training and validation set
We split the train dataset into two parts in 9:1 ratio. 90% will be the actual training set and the remaining 10% will be the validation/testing set. We train our model using the training set and test the accuracy of the model using the validation set.
Can you write Python code for it?

# Split the entire training set into two separate sets: Training set and Validation set
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.10, random_state=random_seed)

print (X_train.shape, Y_train.shape, X_val.shape, Y_val.shape)
num_pixels = X_train.shape[1]

print (Y_val)
# converting one-hot format of digits to normal values/labels
y_val = np.argmax(Y_val, 1) # reverse of to_categorical
print (y_val)
# Note that: capital Y_val contains values in one-hot format and small y_val contains normal digit values


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-Kaggle.ipynb".
The first task is:
Viewing shape and content of data
Can you write Python code for it?

print (X_train.shape)
print (y_train.shape)

print (y_train[0])
print (X_train[0])


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Confusion Matrix
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
import seaborn as sns
sns.set() # setting seaborn default for plots

class_names = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, predicted_classes)
np.set_printoptions(precision=2)

print ('Confusion Matrix in Numbers')
print (cnf_matrix)
print ('')

cnf_matrix_percent = cnf_matrix.astype('float') / cnf_matrix.sum(axis=1)[:, np.newaxis]

print ('Confusion Matrix in Percentage')
print (cnf_matrix_percent)
print ('')

true_class_names = class_names
predicted_class_names = class_names

df_cnf_matrix = pd.DataFrame(cnf_matrix,
                             index = true_class_names,
                             columns = predicted_class_names)

df_cnf_matrix_percent = pd.DataFrame(cnf_matrix_percent,
                                     index = true_class_names,
                                     columns = predicted_class_names)

plt.figure(figsize = (8,6))

#plt.subplot(121)
ax = sns.heatmap(df_cnf_matrix, annot=True, fmt='d')
ax.set_ylabel('True values')
ax.set_xlabel('Predicted values')
ax.set_title('Confusion Matrix in Numbers')

'''
plt.subplot(122)
ax = sns.heatmap(df_cnf_matrix_percent, annot=True)
ax.set_ylabel('True values')
ax.set_xlabel('Predicted values')
'''


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Converting target variable values into one-hot format
The output/target variable is in the format 0 to 9. As this is a multi-class classification problem, we convert the output class values into one-hot format which is simply a binary matrix, i.e.

value 0 will be converted to one-hot format as [1, 0, 0, 0, 0, 0, 0, 0, 0]
value 1 will be converted to one-hot format as [0, 1, 0, 0, 0, 0, 0, 0, 0]
value 2 will be converted to one-hot format as [0, 0, 1, 0, 0, 0, 0, 0, 0]
and so on...
Can you write Python code for it?

print (y_train.shape)
print (y_train[0])

# one hot encode outputs
# note that we have new variables with capital Y
# Y_train is different than y_train
Y_train = np_utils.to_categorical(y_train)
Y_test = np_utils.to_categorical(y_test)
num_classes = Y_test.shape[1]

print (y_train.shape, Y_train.shape)
print (y_train[0], Y_train[0])


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Define Convolutional Neural Network (CNN) Model
Convolution Layer
- We define 32 feature maps with the size of 5x5 matrix
- We use ReLU (Rectified Linear Units) as the activation function
- This layer expects input image size of 1x28x28 ([pixels][height][weight])
Max Pooling Layer
- It has a pool size of 2x2
Dropout Layer
- Configured to randomly exclude 20% of neurons in the layer to reduce overfitting
Flatten
- Flattens the image into a single dimensional vector which is required as input by the fully connected layer
Fully connected Layer
- Contains 128 neurons
- relu is used as an activation function
- Output layer has num_classes=10 neurons for the 10 classes
- softmax activation function is used in the output layer
- adam gradient descent algorithm is used as optimizer to learn and update weights
Can you write Python code for it?


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Define Simple Perceptron Model
Generally, neural networks have the following properties:
- an input layer as a single vector
- zero or multiple hidden layers after input layer
- an output layer after hidden layers which represents class scores in classification problem
- each neuron in a hidden layer is fully connected to all neurons in the previous layer
- neurons in a single layer function independently and do not have any connection with other neurons of the same layer
A single-layer perceptron model is the simplest kind of neural network where there are only two layers: input layer and output layer. The inputs are directly fed into the outputs via a series of weights. It's a feed-forward network where the information moves in only one direction, i.e. forward direction from input nodes to output nodes.
A multi-layer perceptron model is the other kind of neural network where there are one or more hidden layers in between input and output layers. The information flows from input layer to hidden layers and then to output layers. These models can be of feed-forward type or they can also use back-propagation method. In back-propagation, the error is calculated in the output layer by computing the difference of actual output and predicted output. The error is then distributed back to the network layers. Based on this error, the algorithm will adjust the weights of each connection in order to reduce the error value. This type of learning is also referred as deep learning.
We create a simple neural network model with one hidden layer with 784 neurons. Our input layer will also have 784 neurons as we have flattened out training dataset into a single 784 dimensional vector.
softmax activation is used in the output layer.
adam gradient descent optimizer is used to learn weights.
Can you write Python code for it?

def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
    model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
    # compile model
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Fit and Evaluate Model
The model is fit over 5 epochs/iteration. It takes a batch of 200 images in each iteration. Test data is used as validation set. The epochs may be increased to improve accuracy.
Finally, test data is used to evaluate the model by calculating the model's classification accuracy.
Can you write Python code for it?

model = baseline_model()
model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=5, batch_size=200, verbose=1)

model.summary()

scores = model.evaluate(X_test, Y_test, verbose=0)
print (scores)
print ('Score: {}'.format(scores[0]))
print ('Accuracy: {}'.format(scores[1]))


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Normalizing input values
As we can see above, the pixel values for each image are gray scaled between 0 and 255. We now, normalize those values from 0-255 to 0-1.
Can you write Python code for it?


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Plot correctly and incorrectly predicted images
Let's plot some images which are correctly predicted and some images which are incorrectly predicted on our test dataset.
Can you write Python code for it?

# get predicted values
predicted_classes = model.predict_classes(X_test)

# get index list of all correctly predicted values
correct_indices = np.nonzero(np.equal(predicted_classes, y_test))[0]

# get index list of all incorrectly predicted values
incorrect_indices = np.nonzero(np.not_equal(predicted_classes, y_test))[0]

print ('Correctly predicted: %i' % np.size(correct_indices))
print ('Incorrectly predicted: %i' % np.size(incorrect_indices))

plt.figure(figsize=[20,8])
for i, correct in enumerate(correct_indices[:6]):
    plt.subplot(1,6,i+1)
    plt.imshow(X_test[correct].reshape(28,28), cmap='gray', interpolation='none')
    plt.title("Predicted {}, Class {}".format(predicted_classes[correct], y_test[correct]))

plt.figure(figsize=[20,8])
for i, incorrect in enumerate(incorrect_indices[:6]):
    plt.subplot(1,6,i+1)
    plt.imshow(X_test[incorrect].reshape(28,28), cmap='gray', interpolation='none')
    plt.title("Predicted {}, Class {}".format(predicted_classes[incorrect], y_test[incorrect]))


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Plotting images and their class values
Can you write Python code for it?

plt.figure(figsize=[20,8])
for i in range(6):
    plt.subplot(1,6,i+1)
    #plt.imshow(X_train[i])
    plt.imshow(X_train[i], cmap='gray', interpolation='none')
    plt.title("Class {}".format(y_train[i]))

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Reshaping images
The image dimension expected by Keras for 2D (two-dimensional) convolution is in the format of [pixels][width][height].
For RGB color image, the first dimension (pixel) value would be 3 for the red, green and blue components. It's like having 3 image inputs for every single color image. In our case (for MNIST handwritten images), we have gray scale images. Hence, the pixel dimension is set as 1.
Can you write Python code for it?

# reshape to be [samples][pixels][width][height]
X_train = X_train.reshape(X_train.shape[0], 1, 28, 28).astype('float32')
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28).astype('float32')

print (num_pixels, X_train.shape, X_test.shape)

print (X_train[1])


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
The above confusion matrix heatmap shows that:
- Most of value 5 was predicted as 3. 21 images of digit 5 were predicted as 3.
- The second most incorrect prediction was of number 8. 17 images of digit 8 were predicted as 3.
- The third highest wrong prediction was of number 9. 12 images of digit 9 were predicted as 4.
Improve Accuracy using Convolution Neural Network (CNN) Model
Convolutional Neural Networks (CNN) are similar to Multi-layer Perceptron Neural Networks. They are also made up of neurons that have learnable weights and biases. CNNs have been successfully applied to analyzing visual imagery. They are mostly being applied in image and video recognition, recommender systems and natural language processing.
A CNN consists of multiple hidden layers. The hidden layers are either convolutional, pooling or fully connected.
Convolution layer: Feature extraction is done in this layer. This layer applies convolution operation to the input and pass the result to the next layer. In the image classification problem, a weight matrix is defined in the convolution layer. A dot product is computed between the weight matrix and a small part (as the size of the weight matrix) of the input image. The weight runs across the image such that all the pixels are covered at least once, to give a convolved output.

The weight matrix behaves like a filter in an image extracting particular information from the original image matrix.
A weight combination might be extracting edges, while another one might a particular color, while another one might just blur the unwanted noise.
The weights are learnt such that the loss function is minimized similar to a Multi-layer Perceptron.
Therefore weights are learnt to extract features from the original image which help the network in correct prediction.
When we have multiple convolutional layers, the initial layer extract more generic features, while as the network gets deeper, the features extracted by the weight matrices are more and more complex and more suited to the problem at...
Can you write Python code for it?

# load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()


I am working on the file "Digit Recognizer/Handwritten-Digits-Recognizer-MNIST.ipynb".
The first task is:
Viewing shape and content of data
Can you write Python code for it?

print (X_train.shape)
print (y_train.shape)

# print first train image values
# it contains a matrix of 28 rows and 28 cols
print (X_train[0])


I am working on the file "Digit Recognizer/notebook/Digit Recognizer.ipynb".
The first task is:
Evaluating model using splitted data set
Can you write Python code for it?

from sklearn.metrics import accuracy_score
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans

#pre-processing split train
my_pca=PCA(n_components=0.9)
pca_split_train_features=my_pca.fit_transform(split_train_features)

#pre-processing split test features
pca_split_test_features=my_pca.transform(split_test_features)

#fit and predict using split data
model=KMeans(init='k-means++')
model.fit(pca_split_train_features,split_train_target)
split_prediction=model.predict(pca_split_test_features)
score=accuracy_score(split_test_target, split_prediction)
print (score(split_test_target, split_prediction))


I am working on the file "Digit Recognizer/notebook/Digit Recognizer.ipynb".
The first task is:
First let's try to plot digit from first 5 data point
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

def plot_img(sample):
    fig=plt.figure(figsize=(10,10))
    fig.subplots_adjust(left=0, right=1, bottom=0, top=1,hspace=0.05, wspace=0.05)
    for i in  range(sample.shape[0]):
        img=np.reshape(sample[i],(28,28))
        p=fig.add_subplot(sample.shape[0],sample.shape[0],i+1, xticks=[],
yticks=[])
        p.imshow(img,cmap=plt.cm.bone)

plot_img(train_features[0:5])


I am working on the file "Digit Recognizer/notebook/Digit Recognizer.ipynb".
The first task is:
Number of features has reduced from 784 to 87
Can you write Python code for it?

from sklearn.cluster import KMeans
model=KMeans(init='k-means++')
model.fit(pca_train_features,train_target)


I am working on the file "Digit Recognizer/notebook/Digit Recognizer.ipynb".
The first task is:
PCA transformation
Now, we shall try to reduce dimensionality of all features from 'pixel0' to 'pixel783' by using Principal component analysis (PCA), PCA is an orthogonal linear transformation that turns a set of possibly correlated variables into a new set of variables that are as uncorrelated as possible.
there are several class to implement different kind of PCA in sklearn but we will work with PCA class.
Can you write Python code for it?

from sklearn.decomposition import PCA
my_pca=PCA(n_components=0.9)
pca_train_features=my_pca.fit_transform(train_features)
pca_train_features.shape


I am working on the file "Digit Recognizer/notebook/Digit Recognizer.ipynb".
The first task is:
Predicting with Kmeans
Can you write Python code for it?

#features from test data
test_data=pd.read_csv('/test.csv')
test_features=test_data.values[:,;]
pca_test_features=my_pca.transform(test_features)

#predicting original test set.
prediction=model.predict(pca_test_features)


I am working on the file "Digit recognition/FFT Digit recognition.ipynb".
The first task is:
Apply learning on the first half of the digits
Can you write Python code for it?

N = n_samples//2
N = 10000
classifier.fit(mnist.data[:N], mnist.target[:N])


I am working on the file "Digit recognition/FFT Digit recognition.ipynb".
The first task is:
Creating a support vector classifier
Can you write Python code for it?

# gamma = 0.001, which was in the example for 8x8 images, seems to
# heavily overfit 28x28 MNIST data with N=10000 samples, predicts only 1
#classifier = svm.SVC(gamma=0.001)
# TODO: There must be a bug!
classifier = svm.SVC()


I am working on the file "Digit recognition/FFT Digit recognition.ipynb".
The first task is:
It's 70000 examples of handwritten digits of size 28x28 pixels, labeled from 0 to 9.
The data is ordered and needs to be shuffled
Can you write Python code for it?

mnist.data, mnist.target = utils.shuffle(mnist.data, mnist.target)


I am working on the file "Digit recognition/FFT Digit recognition.ipynb".
The first task is:
Pick the first 15 images for visualization
Can you write Python code for it?

n_samples = len(mnist.data)

fig = plt.figure()
for i in range(15):
    img = mnist.data[i].reshape(28, 28)

    ax = fig.add_subplot(3, 5, i+1)
    ax.axis('off')
    ax.imshow(img, cmap=plt.cm.gray, interpolation='nearest')
    ax.set_title('# {}'.format(i))


I am working on the file "Digit recognition/FFT Digit recognition.ipynb".
The first task is:
Predictions
Can you write Python code for it?

expected = mnist.target[N:2*N]
predicted = classifier.predict(mnist.data[N:2*N])


I am working on the file "Digit recognition/FFT Digit recognition.ipynb".
The first task is:
Scikit-learn SVM doesn't seem to be of the fastest kind, need to check parameters.
* Doesn't appear to be memory problem that it stalls at 35000 samples in this case. 10000 samples was in a minute or few.
* May try more optimized learning approach.
Can you write Python code for it?

print("Classification report for classifier %s:\n%s\n"
      % (classifier, metrics.classification_report(expected, predicted)))
print("Confusion matrix:\n%s" % metrics.confusion_matrix(expected, predicted))

expected, predicted

np.unique(predicted)

expected = mnist.target[:N]
predicted = classifier.predict(mnist.data[:N])

mnist.data[N:2*N].shape


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
After creating this DataFrame, it becomes very easy to retrieve specific information from our corpus. What is the frequency, for instance, of the word 'the' in each text?
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
After loading these documents, we can now represent them using a bag of words model. To this end, we will use a library called scikit-learn, or sklearn in shorthand, which is increasingly popular in text analysis nowadays. As you will see below, we import its CountVectorizer object below, and we apply it to our corpus, specifying that we would like to extract a maximum of 10 features from the texts. The means that we will only consider the frequencies of 10 words (to keep our model small enough to be workable for now).
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(max_features=10)
BOW = vec.fit_transform(texts).toarray()
print(BOW.shape)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
And then execute the following blocks -- and try to understand the intuition behind them:
Can you write Python code for it?

# first, we extract the counts:
nb_words = 8
wrds, cnts = zip(*bow.most_common(nb_words))
print(wrds)
print(cnts)

# now the plotting part:
import matplotlib.pyplot as plt
import numpy as np
fig, ax = plt.subplots()
bar_width = 0.5
idxs = np.arange(nb_words)
#print(idxs)
ax.bar(idxs, cnts, bar_width, color='blue', align='center')
plt.xticks(idxs, wrds)
plt.show()


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
As is clear from the shape info, we have obtained a 9 by 9 matrix, which holds the distance between each pair of texts. Note that the distance from a text to itself is of course zero (cf. diagonal cells):
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
As we can see, Jane Austen's novels form a tight and distinctive cloud; an author like Thackeray is apparantly more difficult to tell apart. The actual distance between nodes is hinted at on the horizontal length of the branches (i.e. the values on the x-axis in this plot). Note that in this code block too we can easily switch to, for instance, the Euclidean distance. Does the code block below produce better results?
Can you write Python code for it?

dm = squareform(pdist(BOW, 'euclidean'))
linkage_object = linkage(dm, method='ward')
d = dendrogram(Z=linkage_object, labels=authors, orientation='right')


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
As you can see, the little squares representing texts by the same author already show a tendency to invite lower distance scores. But how are these distances exactly calculated?
Each text in our distance matrix is represented as a row, consisting of 100 numbers. Such a list of numbers is also called a document vector, which is why the document modeling process described above is sometimes also called vectorization (cf. CountVectorizer). In digital text analysis, documents are compared by applying standard metrics from geometry to these documents vectors containing word frequencies. Let us have a closer look at one popular, but intuitively simple distance metric, the Manhattan city block distance. The formula behind this metric is very simple (don't be afraid of the mathematical notation; it won't bite):
$$manhattan(x, y) = \sum_{i=1}^{n} \left| x_i - y_i \right|$$
What this formula expresses, is that to calculate the distance between two documents, we loop over each word column in both texts and we calculate the absolute difference between the value for that item in each text. Afterwards we simply sum all the absolute differences.

DIY
Consider the following two dummy vectors:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
As you can see, the max_features argument which we used above restricts the model to the n words which are most frequent throughout our texts. These are typically smallish function words. Funnily, sklearn uses its own tokenizer, and this default tokenizer ignores certain words that are surprisingly enough absent in the vocabulary list we just inspected. Can you figure which words? Why are they absent?
Luckily, sklearn is flexible enough to allow us to use our own tokenizer. To use the nltk tokenizer for instance, we can simply pass it as an argument when we create the CountVectorizer. (Note that, depending on the speed of your machine, the following code block might actually take a while to execute, because the tokenizer now has to process entire novels, instead of a single sentence. Sit tight!)
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(max_features=10, tokenizer=nltk.word_tokenize)
BOW = vec.fit_transform(texts).toarray()
print(vec.get_feature_names())


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
As you can see, we obtain another sort of document matrix, where the number of columns corresponds to the number of topics we extracted. Let us now find out whether this representation yields anything useful. It is difficult to visualize 3,000+ documents all at once, so in the code block below, I select a smaller subset of 30 documents (and the corresponding filenames), using the random module.
Can you write Python code for it?

comb = list(zip(names, topic_repr))
import random
random.seed(10000)
random.shuffle(comb)
comb = comb[:30]
subset_names, subset_topic_repr = zip(*comb)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
By using 'indexing' (with square brackets), we can now access individual words in our word list. Check out the following print statements:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Can you make sense of these topics? Which are the main thematic categories that you can discern?

DIY
Try to run the algorithm with more topics and allow more iterations (but don't exaggerate!): do the results get more interpretable?

Now that we have build a topic model, we can use it to represent our corpus. Instead of representing each document as a vector containing word frequencies, we represent it as a vector containing topic scores. To achieve this, we can simply call the fit() function to the bag-of-words representation of our document:
Can you write Python code for it?

topic_repr = lda.transform(BOW)
print(topic_repr.shape)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Clustering algorithms are based on essentially based on the distances between texts: clustering algorithms typically start by calculating the distance between each pair of texts in a corpus, so that they know for each text how (dis)similar it is from any other text. Only after these pairwise-distances have been calculated, we can have the clustering algorithm start building a tree representation, in which the similar texts are joined together and merged into new nodes. To create a distance matrix, we use a number of functions from scipy (Scientific Python), a commonly used package for scientific applications.
Can you write Python code for it?

from scipy.spatial.distance import pdist, squareform


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Counting words
Once we have come up with a good way to split texts into individual tokens, it is time to start thinking about how we can represent texts via these tokens. One popular approach to this problem is called the bag-of-words model (BOW): this is a very old (and slightly naive) strategy for text representation, but is still surprisingly popular. Many spam filters, for instance, will still rely on a bag-of-words model when deciding which email messages will show up in your Junk folder.
The intuition behind this model is very simple: to represent a document, we consider it a 'bag', containing tokens in no particular order. We then characterize a particular text by counting how often each term occurs in it. Counting how often each word occurs in a list of tokens example from above is child's play in Python. For this purpose, we copy some of the code from the previous section and apply it too a larger paragraph:
Can you write Python code for it?

text = """It is a truth universally acknowledged, that a single man
in possession of a good fortune, must be in want of a wife. However
little known the feelings or views of such a man may be on his first
entering a neighbourhood, this truth is so well fixed in the minds
of the surrounding families, that he is considered the rightful
property of some one or other of their daughters. "My dear Mr. Bennet,"
said his lady to him one day, "have you heard that Netherfield Park is
let at last?" Mr. Bennet replied that he had not. "But it is," returned
she; "for Mrs. Long has just been here, and she told me all about it."
Mr. Bennet made no answer. "Do you not want to know who has taken it?"
cried his wife impatiently. "_You_ want to tell me, and I have no
objection to hearing it." This was invitation enough."""
lower_str = text.lower()
lower_tokens = nltk.word_tokenize(lower_str)
clean_tokens = [w for w in lower_tokens if w.isalpha()]
print('Word count:', len(clean_tokens))


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
DIY
Can you try to increase the number of words plotted? And change the color used for the bars to blue? And the width of the bars plotted?

The Bag of Words Model
We are almost there: we now know how to split documents into tokens and how we we count (and even visualize!) the frequencies of these items. Now it is only a small step towards a 'real' bag of words model. If we present a collection of texts under a bag of words model, what we really would like to end up with is a frequence table, which has a row for each document, and a column for all the tokens that occur in the collections, which is also called the vocabulary of the corpus. Each cell is then filled with the frequency of each vocabulary item, so that the final matrix will look like like a standard, two dimensional table which you all know from spreadsheet applications.
While creating such a matrix youself isn't too difficult in Python, here we will rely on an external package, which makes it really simple to efficiently create such matrices. The zipped folder which you downloaded for this coarse, contains a small corpus, containing novels by a number of famous Victorian novelists. Under data/victorian_small, for instance, you will find a number of files; the filenames indicate the author and (abbreviated) title of the novel contained in that file (e.g. Austen_Pride.txt). In the block below, I prepared some code to load these files from your hard drive into Python, which can execute now:
Can you write Python code for it?

# we import some modules which we need
import glob
import os

# we create three emptylist
authors, titles, texts = [], [], []

# we loop over the filenames under the directory:
for filename in sorted(glob.glob('data/victorian_small/*.txt')):

    # we open a file and read the contents from it:
    with open(filename, 'r') as f:
        text = f.read()

    # we derive the title and author from the filename
    author, title = os.path.basename(filename).replace('.txt', '').split('_')

    # we add to the lists:
    authors.append(author)
    titles.append(title)
    texts.append(text)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
DIY
Try out the indexes [0] and [-1] for the example with words. Which words are being printed when you use these indices? Do this makes sense? What is peculiar about the way Python 'counts'?

Note that words is a so-called list variable in technical terms, but that the individual elements of words are still plain strings:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Exercise
The code repository also contains a larger folder of novels, called victorian_large. Use the code block below to copy and paste code snippets from above, which you can slightly adapt to do the following things:
1. Read in the texts, producing 3 lists of texts, authors and titles. How many texts did you load (use the len() function)?
2. Select the text for David (Copperfield) by Charles Dickens and find out how often the word "and" is used in this text. Hint: use the nltk tokenizer and the Counter object from the collections module. Make sure no punctuation is included in your counts.
3. Vectorize the text using the CountVectorizer using the 250 most frequent words.
4. Normalize the resulting document matrix and draw a heatmap using blue colors.
5. Draw a cluster diagram and experiment with the distance metrics: which distance metric produces the 'best' result from the point of view of authorship clustering?
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Finally, let us visually inspect the BOW model which we have converted. To this end, we make use of pandas, a Python-package that is nowadays often used to work with all sorts of data tables in Python. In the code block below, we create a new table or 'DataFrame' and add the correct row and column names:
Can you write Python code for it?

import pandas as pd # conventional shorthand!
df = pd.DataFrame(BOW, columns=vec.get_feature_names(), index=titles)
print(df)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Functions like isalpha() return something that is called a 'boolean' value, a kind of variable that can only take two values, i.e. True or False. Such values are useful because you can use them to test whether some condition is true or not. For example, if isalpha() evaluates to False for a word, we can have Python ignore such a word.

DIY
Using some more complicated Python syntax (a so-called 'list generator'), it is very easy to filter out non-alphabetic strings. In the example below, I inserted a logical thinking error on purpose: can you adapt the line below and make it output the correct result? You will note that Python is really a super-intuitive programming language, because it almost reads like plain English.
Can you write Python code for it?

clean_tokens = [w for w in lower_tokens if not w.isalpha()]
print(clean_tokens)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Here we define a string of text by enclosing it with quotations marks and assigning it to a variable or container called text. The fact that Python still sees this piece of text as a continguous string of characters becomes evident when we ask Python to print out the length of text, using the len() function:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
If we inspect our new frequency table, we can see that the values are now neatly in the 0-1 range:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Let us have a look at the three most frequent items in the text:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Likewise, they evidently differ in length:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Many applications will not be very interested in punctuation marks, so can we can try to remove these as well. The isalpha() method allows you to determine whether a string only contains alphabetic characters:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Many people find it more intuitive to consider texts as a strings of words, rather than plain characters, because words correspond to more concrete entities. In Python, we can easily turn our original 'string' into a list of words:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Note how the function word_tokenize() neatly splits off punctuation! Many improvements nevertheless remain. To collapse the difference between uppercase and lowercase variables, for instance, we could first lowercase the original input string:
Can you write Python code for it?

lower_str = text.lower()
lower_tokens = nltk.word_tokenize(lower_str)
print(lower_tokens)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Now, we can efficiently 'scale' or normalize the matrix using these sums:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Obviously, the most common items in such a frequency list are typically small, grammatical words that are very frequent throughout all the texts in a language. Let us add a small visualisation of this information. We can use a barplot to show the top-frequency items in a more pleasing manner. In the following block, we use the matplotlib package for this purpose, which is a popular graphics package in Python. To make sure that it shows up neatly in our notebook, first execute this cell:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
One could say that characters are the 'atoms' or smallest meaningful units in computational text processing. Just as computer images use pixels as their fundamental building blocks, all digital text processing applications start from raw characters and it are these characters that are physically stored on your machines in bits and bytes.

DIY
Define a string containing your own name in the code block below and print its length. Insert a number of whitespaces at the end of your name  (i.e. tab the space bar a couple of times): does this change the length of the string?
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Or the frequency of 'and' in 'Emma'?
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Text analysis
Distance metrics
Now that we have converted our small dummy corpus to a bag-of-words matrix, we can finally start actually analyzing it! One very common technique to visualize texts is to render a cluster diagram or dendrogram. Such a tree-like visualization (an example will follow shortly) can be used to obtain a rough-and-ready first idea of the (dis)similarities between the texts in our corpus. Texts that cluster together under a similar branch in the resulting diagram, can be argued to be stylistically closer to each other, than texts which occupy completely different places in the tree. Texts by the same authors, for instance, will often form thight clades in the tree, because they are written in a similar style.
However, when comparing texts, we should be aware of the fact that documents can strongly vary in length. The bag-of-words model which we created above does not take into account that some texts might be longer than others, because it simply uses absolute frequencies, which will be much higher in the case of longer documents. Before comparing texts on the basis of word frequencies, it therefore makes sense to apply some sort of normalization. One very common type of normalization is to use relative, instead of absolute word frequencies: that means that we have to divide the original, absolute frequencies in a document, by the total number of word in that document. Remember that we are dealing with a 9x10 matrix at this stage:
Each of the 9 document rows which we obtain should now be normalized by dividing each word count by the total number of words which we recorded for that document. First, we therefore need to calculate the row-wise sum in our table.
Can you write Python code for it?

totals = BOW.sum(axis=1, keepdims=True)
print(totals)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
That looks great. Let us now build a model with a more serious vocabulary size (=300) for the actual cluster analysis:
Can you write Python code for it?

vec = CountVectorizer(max_features=300, tokenizer=nltk.word_tokenize)
BOW = vec.fit_transform(texts).toarray()
BOW = BOW / BOW.sum(axis=1, keepdims=True)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
The function pdist() ('pairwise distances') is a function which we can use to calculate the distance between each pair of texts in our corpus. Using the squareform() function, we will eventually obtain a 9x9 matrix, the structure of which is conceptually easy to understand: this square distance matrix (named dm) will hold for each of our 9 texts the distance to each other text in the corpus. Naturally, the diagonal in this matrix are all-zeroes (since the distance from a text to itself will be zero). We create this distance matrix as follows:
Can you write Python code for it?

dm = squareform(pdist(BOW))
print(dm.shape)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
This is an example of one popular distance metric which is currently used a lot in digital text analysis. Alternatives (which might ring a bell from math classes in high school) include the Euclidean distance or cosine distance. Our dm distance matrix from above can be created with any of these option, by specifying the correct metric when calling pdist(). Try out some of them!
Can you write Python code for it?

dm = squareform(pdist(BOW), 'cosine') # or 'euclidean', or 'cosine' etc.
fig, ax = plt.subplots()
heatmap = ax.pcolor(dm, cmap=plt.cm.Reds)
ax.set_xticks(np.arange(dm.shape[0])+0.5, minor=False)
ax.set_yticks(np.arange(dm.shape[1])+0.5, minor=False)
ax.set_xticklabels(titles, minor=False, rotation=90)
ax.set_yticklabels(authors, minor=False)
plt.show()


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
To have a peak at the content of this novel, we can now 'stack' indices as follows. Using the first square brackets ([2]) we select the third novel in the list, i.e. Sense and Sensibility by Jane Austen. Then, using a second index ([:300]), we print the first 300 characters in that novel.
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Tokenization
In the previous paragraph, we have adopted an extremely crude definition of a 'word', namely as a string of characters that doesn't contain any whitespace. There are of course many problems that arise if we use such a naive definition. Can you think of some?
In computer science, and computational linguistics in particular, people have come up with much smarter ways to divide texts into words. This process is called tokenization, which refers to the fact that this process divides strings of characters into a list of more meaningful tokens. One interesting package which we can use for this, is nltk (the Natural Language Toolkit), a package which has been specifically designed to deal with language problems. First, we have to import it, since it isn't part of the standard library of Python:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
Using the split() method, we split our original sentence into a word list along instances of whitespace characters. Note that, in technical terms, the variable type of text is different from that of the newly created variable words:
Can you write Python code for it?


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
We are now ready to draw the actual dendrogram, which we do in the following code block. Note that we annotate the outer leaf nodes in our tree (i.e. the actual texts) using the labels argument. With the orientation argument, we make sure that our dendrogram can be easily read from left to right:
Can you write Python code for it?

from scipy.cluster.hierarchy import dendrogram
d = dendrogram(Z=linkage_object, labels=titles, orientation='right')


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
We are now ready to start modelling the topics in this text collection. For this we make use of a popular technique called Latent Dirichlet Allocation or LDA, which is also included in the sklearn library. In the code block below, you can safely ignore most of the settings which we use when we initialize the model, but you should pay attention to the n_topics and max_iter parameter. The former controls how many topics we will extract from the document collection (this is one of few parameters which the model, sadly, does not learn itself). We start with a fairly small number of topics, but if you want a more finegrained analysis of your corpus you can always increase this parameter. The max_iter setting, finally, controls how long we let the model 'think': the more interations we allow, the better the model will get, but because LDA is as you will see fairly computationally intensive, it makes sense start with a relatively low number in this respect. You can now execute the following code block -- you will see that this code might take several minutes to complete.
Can you write Python code for it?

from sklearn.decomposition import LatentDirichletAllocation
lda = LatentDirichletAllocation(n_topics=50,
                                max_iter=10,
                                learning_method='online',
                                learning_offset=50.,
                                random_state=0)
lda.fit(BOW)


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
We can now use our clustering algorithm from above in an exactly parallel way. Go on and try it (because of the random aspect of the previous code block, it possible that you obtain a different random selection).
Can you write Python code for it?

dm = squareform(pdist(subset_topic_repr), 'cosine') # or 'euclidean', or 'cosine' etc.
linkage_object = linkage(dm, method='ward')
fig_size = plt.rcParams["figure.figsize"]
plt.rcParams["figure.figsize"] = [15, 9]
d = dendrogram(Z=linkage_object, labels=subset_names, orientation='right')


I am working on the file "Digital Text Analysis.ipynb".
The first task is:
You might already get a sense of the kind of topics that are being discussed. Also, you will notice that these are rather noisy data, which is challenging for humans to process manually. In the last part of this tutorial we will use a technique called topic modelling. This technique will automatically determine a number of topics or semantic word clusters that seem to be important in a document collection. The nice thing about topic modelling is that is a largely unsupervised technique, meaning that it does not need prior information about the document collection or the language it uses. It will simply inspect which words often co-occur in documents and are therefore more likely to be semantically related.
After fitting a topic model to a document collection, we can use it to inspect which topics have been detected. Additionally, we can use the model to infer to which extent these topics are present in new documents. Interestingly, the model does not assume that texts are always about a single topic; rather, it assumes that documents contain a mixture of different topics. A text about the transfer of a football player, for instance, might contain 80% of a 'sports' topic, 15% of a 'finance'-related topic, and 5% of a topic about 'Spanish lifestyle' etc. For topic modelling too, we first need to convert our corpus to a numerical format (i.e. 'vectorize' it as we did above). Luckily, we already know how to do that:
Can you write Python code for it?

vec = CountVectorizer(max_df=0.95, min_df=5, max_features=2000, stop_words='english')
BOW = vec.fit_transform(documents)
print(BOW.shape)


I am working on the file "DimensionReduction.ipynb".
The first task is:
An example (and comparison to PCA) is given below.
Can you write Python code for it?

# Execute the next 2 cells
# Example from Figure 7.4
# Author: Jake VanderPlas
# License: BSD
%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import NMF
from sklearn.decomposition import RandomizedPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA, and NMF components
def compute_PCA_NMF(n_components=5):
    spec_mean = spectra.mean(0)

    # PCA: use randomized PCA for speed
    pca = RandomizedPCA(n_components - 1)
    pca.fit(spectra)
    pca_comp = np.vstack([spec_mean,
                          pca.components_])

    # NMF requires all elements of the input to be greater than zero
    spectra[spectra < 0] = 0
    nmf = NMF(n_components)
    nmf.fit(spectra)
    nmf_comp = nmf.components_

    return pca_comp, nmf_comp

n_components = 5
decompositions = compute_PCA_NMF(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 10))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = ['PCA components', 'NMF components']

for i, comp in enumerate(decompositions):
    for j in range(n_components):
        ax = fig.add_subplot(n_components, 3, 3 * j + 1 + i)

        ax.yaxis.set_major_formatter(plt.NullFormatter())
        ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
        if j < n_components - 1:
            ax.xaxis.set_major_formatter(plt.NullFormatter())
        else:
            ax.set_xlabel('wavelength (Angstroms)')

        ax.plot(wavelengths, comp[j], '-k', lw=1)

        # plot zero line
       ...


I am working on the file "DimensionReduction.ipynb".
The first task is:
As with PCA and NMF, we can similarly do a reconstruction:
Can you write Python code for it?

# Execute this cell
#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
evecs = data['evecs']
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 2, 4, 8]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        #text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        #text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ## GTR: had to comment this out for some reason
    ## ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()


I am working on the file "DimensionReduction.ipynb".
The first task is:
Caveats I
PCA is a linear process, whereas the variations in the data may not be.  So it may not always be appropriate to use and/or may require a relatively large number of components to fully describe any non-linearity.
Note also that PCA can be very impractical for large data sets which exceed the memory per core as the computational requirement goes as $\mathscr{O}(D^3$) and the memory requirement goes as $\mathscr{O}(2D^2)$.
Missing Data
We have assumed so far that there is no missing data (e.g., bad pixels in the spectrum, etc.).  But often the data set is incomplete.  Since PCA encodes the flux correlation with wavelength (or whatever parameters are in your data set), we can actually use it to determine missing values.
An example is shown below.  Here, black are the observed spectra.  Gray are the regions where we have no data.  Blue is the PCA reconstruction, including the regions where there are no data.  Awesome, isn't it?
Can you write Python code for it?

# Execute this cell
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import ticker

from astroML.datasets import fetch_sdss_corrected_spectra
from astroML.datasets import sdss_corrected_spectra

#------------------------------------------------------------
# Get spectra and eigenvectors used to reconstruct them
data = fetch_sdss_corrected_spectra()
spec = sdss_corrected_spectra.reconstruct_spectra(data)
lam = sdss_corrected_spectra.compute_wavelengths(data)
evecs = data['evecs']
mu = data['mu']
norms = data['norms']
mask = data['mask']

#------------------------------------------------------------
# plot the results
i_plot = ((lam > 5750) & (lam < 6350))
lam = lam[i_plot]

specnums = [20, 8, 9]
subplots = [311, 312, 313]

fig = plt.figure(figsize=(8, 10))
fig.subplots_adjust(hspace=0)

for subplot, i in zip(subplots, specnums):
    ax = fig.add_subplot(subplot)

    # compute eigen-coefficients
    spec_i_centered = spec[i] / norms[i] - mu
    coeffs = np.dot(spec_i_centered, evecs.T)

    # blank out masked regions
    spec_i = spec[i]
    mask_i = mask[i]
    spec_i[mask_i] = np.nan

    # plot the raw masked spectrum
    ax.plot(lam, spec_i[i_plot], '-', color='k', lw=2,
            label='True spectrum')

    # plot two levels of reconstruction
    for nev in [10]:
        if nev == 0:
            label = 'mean'
        else:
            label = 'N EV=%i' % nev
        spec_i_recons = norms[i] * (mu + np.dot(coeffs[:nev], evecs[:nev]))
        ax.plot(lam, spec_i_recons[i_plot], label=label)

    # plot shaded background in masked region
    ylim = ax.get_ylim()
    mask_shade = ylim[0] + mask[i][i_plot].astype(float) * ylim[1]
    plt.fill(np.concatenate([lam[:1], lam, lam[-1:]]),
             np.concatenate([[ylim[0]], mask_shade, [ylim[0]]]),
             lw=0, fc='k', alpha=0.2)

    ax.set_xlim(lam[0], lam[-1])
    ax.set_ylim(ylim)
    ax.yaxis.set_major_formatter(ticker.NullFormatter())

    if subplot == 311:
        ax.legend(loc=1, prop=dict(size=14))

 ...


I am working on the file "DimensionReduction.ipynb".
The first task is:
Do the PCA transform, projecting to 2 dimensions and plot the results.
Can you write Python code for it?

# PCA
from sklearn.decomposition import PCA
pca = # Complete
# Complete
X_reduced = # Complete

plt.scatter(# Complete, c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()


I am working on the file "DimensionReduction.ipynb".
The first task is:
Execute the next 2 cells to produce a plot showing the ICA components.
Can you write Python code for it?

%matplotlib inline
#Example from Andy Connolly
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import FastICA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA
np.random.seed(500)
nrows = 500
n_components = 5
ind = np.random.randint(spectra.shape[0], size=nrows)

spec_mean = spectra[ind].mean(0)
# spec_mean = spectra[:50].mean(0)

ica = FastICA(n_components - 1)
ica.fit(spectra[ind])
ica_comp = np.vstack([spec_mean,ica.components_]) #Add the mean to the components

#Make plots
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'ICA components'

for j in range(n_components):

    # plot the components
    ax = fig.add_subplot(n_components, 2, 2*j+2)
    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax.plot(wavelengths, ica_comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    # plot the first j spectra
    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
 ...


I am working on the file "DimensionReduction.ipynb".
The first task is:
How much of the variance is explained by the first two components?  How about all of the components?
Can you write Python code for it?

print("The first component explains {:.3f} of the variance in the data.".format(# Complete
print("The second component explains {:.3f} of the variance in the data.".format(# Complete
print("All components explain {:.3f} of the variance in the data.".format(# Complete