I am working on the file "CQD/.ipynb_checkpoints/Project-checkpoint.ipynb".
The first task is:
(b)
We know that in our simulation we work with a system with low ionation probability, this suggests that the wave function does not move much from the initial position. All of this would indicate that the kinetic momentum is low, so the optimal gauge is the length gauge, as the velocity gauge would shift the momentum by $A(t)$.
I will first  do (d) and then (c) and (e) together.
(d)
We can approximate our system to an electron in a sinusoidal electric field in the classical regime. Newton's equation then is :
\begin{equation}
    m\ddot{x} = E_0 \sin(\omega t) sin^2(\frac{\omega t}{2 n})
\end{equation}
by integrating this equation we obtain:
\begin{equation}
    m\dot{x} =  \frac{E_0}{\omega} \frac{\cos(\omega t) (n^2 \cos(\frac{\omega t}{n}) - n^2 + 1) + n \sin(\omega t) \sin(\frac{\omega t}{n}) -1}{2 (n^2 - 1)}
\end{equation}
looking at the possible maxima of this function one can put an upper bound on the momentum in the case of $n \leq 1$:
\begin{equation}
    \tilde{p} \leq (n^2 E_0) / ((n^2 -1) \omega)
\end{equation}
if a general bound is needed one can use $((n^2 + 1) E_0)$ in place of the numerator. Obviously this is just an upper bound on the momentum and not a maximum, one can use a better value depending on the parameters, by calculating the maximum numerically.
Having an approximation for the maximum momentum we estimate the spacing of the spacial grid and temporal step with:
\begin{equation}
        \Delta x \leq \frac{\hbar\pi}{\tilde{p}} \quad \text{and} \quad \Delta t \leq \frac{\hbar\pi}{\tilde{E}} = \frac{2\pi \hbar}{\tilde{p}^2}
\end{equation}
For the simulation we need also a maximum and minimum for $x$. Since we do not expect  our wave function to move dramatically we can look and the ground state of the soft-core potential and look when the function is sufficiently close to 0. In the simulation we opted for -15 a.u., 15 a.u. as boundery.
Lastly we will choose the mean of the vanishing potential $\tilde{V}$ based around the size of the damping region $d$ and the average momentum $p$,...
Can you write Python code for it?

omega = .02
n = 3 / 2
E0 = .05
Z = 1
x0, x1 = -15, 15
xl, xr = -10, 10
d = x1 - xr
t_temp = np.linspace(0, 2 * np.pi * (n + .5) / omega, 1000)
A_max = max(A(t_temp, E0, omega, n)) # the maximum momentum is equal to the
                                     # maximum value of the magnetic potential
p_tilde = n**2 * E0 /(n**2 - 1) / omega

print('dx using the approximation                                 ',\
      "{:1.4f}".format(pi / p_tilde), 'a.u.')
print('dx using the maximum of the momentum calculated numerically',\
      "{:1.4f}".format(pi / A_max), 'a.u.')
print('dt using the approximation                                 ',\
      "{:1.4f}".format(2 * pi / p_tilde ** 2), 'a.u.')
print('dt using the maximum of the momentum calculated numerically',\
      "{:1.4f}".format(2 * pi / A_max ** 2), 'a.u.')

print("{:1.4f}".format(p_tilde / (8 * d)), 'a.u. < tilde_V <',\
      "{:1.4f}".format(p_tilde ** 3 / 2 ** 4), 'a.u.')

V_tilde = 5.
dx = pi / p_tilde
x = arange(x0, x1, dx)
N = len(x)
p = fftfreq(N, d=dx / (2 * pi))
dt = 2 * pi / p_tilde ** 2
ts = np.arange(0, 2 * np.pi * (n + .5) / omega, dt)

H = diag(ones(N - 1), -1) - 2 * diag(ones(N)) + diag(ones(N - 1), 1)
H *= -1 / (2 * dx ** 2)
H += diag(V(x, Z))
U_2 = exp(-1j * 0.5 * p ** 2 * dt)

_, Psi_tot = eigh(H)
Psi = Psi_tot[:, 0].astype('complex')

Psi /= np.sqrt(sum(abs(Psi) ** 2 * dx))
psi0 = deepcopy(Psi)

norm = np.zeros(len(ts))
overlap = np.zeros(len(ts))

for k, t in enumerate(ts):
    U_1 = exp(-0.5 * 1j * (V(x, 1) - 1j *
              vanish(V_tilde, x, xl, xr) - x * E(t, E0, omega, n)) * dt)

    Psi *= U_1
    Psi = fft(Psi)
    Psi *= U_2
    Psi = ifft(Psi)  # go to real space
    Psi *= U_1

    norm[k] = sum(abs(Psi) ** 2 * dx)
    overlap[k] = abs(vdot(Psi, psi0)) * dx


I am working on the file "CQD/.ipynb_checkpoints/Project-checkpoint.ipynb".
The first task is:
(f)
I run the previous program for different values of $E_0$.
Can you write Python code for it?

N_e = 20
ionizs = np.zeros(N_e)
norms = np.zeros((N_e, len(ts)))
for j, E0 in enumerate(np.linspace(0, .05, N_e)):
    Psi = deepcopy(psi0)
    for k, t in enumerate(ts):
        U_1 = exp(-0.5 * 1j * (V(x, 1) - 1j * vanish(V_tilde, x, -10, 10) - x * E(t, E0, omega, n)) * dt)

        Psi *= U_1
        Psi = fft(Psi)
        Psi *= U_2
        Psi = ifft(Psi)  # go to real space
        Psi *= U_1
        norms[j, k] = sum(abs(Psi) ** 2 * dx)
    ionizs[j] = 1 - sum(abs(Psi) ** 2 * dx)


I am working on the file "CQD/.ipynb_checkpoints/Project-checkpoint.ipynb".
The first task is:
2. The one-dimensional soft-core potential.
We can discretize the hamiltonian with the finite difference scheme and compute the eigenvalues and eigenvector. I plotted the eigenfunctions for the 3 lowest energy eigenstates and in the second plot you can see the different bound energies compered to the potential.
Can you write Python code for it?

def V(x, Z=1):
    return -Z / sqrt(2 / Z ** 2 + x ** 2)

N = 2 ** 10
x0, x1 = -25, 25
x = linspace(x0, x1, N)
dx = (x1 - x0) / (N - 1)

H = diag(ones(N - 1), -1) - 2 * diag(ones(N)) + diag(ones(N - 1), 1)
H *= -1 / (2 * dx**2)
H += diag(V(x))

E, Psi_tot = eigh(H)
E_bound=E[E<0]

for k, E_ in enumerate(sorted(E_bound)[:3]):
    print('E_{' + str(k) + '} = ' + "{:1.4f}".format(E_))
plot(x, Psi_tot[:, 0] / sqrt(dx), label=r'$\Psi_0(x)$')
plot(x, Psi_tot[:, 1] / sqrt(dx), label=r'$\Psi_1(x)$')
plot(x, Psi_tot[:, 2] / sqrt(dx), label=r'$\Psi_2(x)$')
legend(loc=1)
xlabel('x')
ylabel('$\Psi(t)$')

figure()
plot(x, V(x))
plot(x, E_bound * ones_like(x)[:, newaxis])
legend([r'$V(x)$', r'$E_0$', r'$E_1$', r'$E_2$'])
xlabel('x')
ylabel('Energy')


I am working on the file "CQD/.ipynb_checkpoints/Project-checkpoint.ipynb".
The first task is:
3. Ionization from a one-dimensional soft-core potential.
(a)
Can you write Python code for it?

def E(t, E0, omega, n):
    t_ = maximum(omega * t, 0)
    t_ = minimum(t_, 2 * np.pi * n)

    return E0 * sin(t_) * sin(t_ / (2 * n))

def A(t, E0, omega, n):
    pref = -E0 / omega
    t_ = maximum(omega * t, 0.)
    t_ = minimum(t_, 2 * np.pi * n)
    return pref * (cos(t_) * (n * n * cos(t_ / n) - n * n + 1) + n * sin(t_) *
                   sin(t_ / n) - 1) / (2 * (n * n - 1))

def vanish(V0, x, x0, x1):
    V0 *= 2
    xs, xe = x[0], x[-1]
    potential = np.maximum(0, (V0 * (x - x0) / (xs - x0)))
    return np.maximum(potential, (V0 * (x - x1) / (xe - x1)))

omega = .02
n = 3 / 2
E0 = .05
Z = 1
x0, x1 = -15, 15
dx = .1
x = arange(x0, x1, dx)
N = len(x)
p = fftfreq(N, d=dx / (2 * pi))
dt = 0.5
ts = np.arange(- pi / omega, 2 * np.pi * (n + .5) / omega, dt)

plot(ts, E(ts, E0, omega, n))
title('Electric field for n = 3/2')
xlabel('t')
ylabel('E(t)')
figure()
t_star = np.arange(-pi / omega, 2 * np.pi * (5 + .5) / omega, 0.01)
plot(t_star, E(t_star, E0, omega, 5))
xlabel('t')
ylabel('E(t)')
title('Electric field for n = 5')
figure()
plot(ts, A(ts, E0, omega, n))
xlabel('t')
ylabel('A(t)')
title('Magnetic potential field for n = 3/2')
figure()
plot(t_star, A(t_star, E0, omega, 5))
xlabel('t')
ylabel('A(t)')
title('Magnetic potential field for n = 5')


I am working on the file "CQD/.ipynb_checkpoints/Project-checkpoint.ipynb".
The first task is:
We see as expected that the lower the amplitude of the electric field the lower the final ionization probability will be.
Can you write Python code for it?

title('Ionization probabilies in time')
plot(ts, 1 - norms.T[:, ::-1])
legend([r'$E_0 = 0.05$ a.u.'])
xlabel(r't')
ylabel(r'$1 - |<\Psi(t)|\Psi(t)>|^2$')
figure()
title(r'ionization probabilies at $t_{end}$')
ylabel(r'$1 - |<\Psi(t_{end})|\Psi(t_{end})>|^2$')
xlabel(r'$E_0$')
plot(np.linspace(0, .05, N_e), ionizs)


I am working on the file "CQD/.ipynb_checkpoints/exercise1-checkpoint.ipynb".
The first task is:
(d) The SciPy library has two functions to calculate the eigen values of a dense matrix, one
that works for arbitrary quadratic matrices, one that is limited to Hermitian matrices.
Give three reasons why one should prefer the specialized function over the more general
in the case of Hermitian matrices.
Solution
My reasons are:
* A function specific for hermitian matrices is probably optimized to work faster than in the general case.
* It returns the eignvalues as real numbers and not complex with the immaginary part close to 0.
    * For this it returns the values sorted
* It helps to keep the code clean.

3.  Eigen functions of the quantum harmonic oscillator.
The normalized eigen functions of the quantum harmonic oscillator are given by
$$ \psi_n(x)\equiv \left\langle x \mid n \right\rangle = {1 \over \sqrt{2^n n! l}}~ \pi^{-1/4} e^{-x^2 / (2 l^2)} H_n(x/l), $$
where $H_n(x)$ denotes the Hermite polynomials, where the first few are given by
$H_n(x) = 1$,
$H_n(x) = 2x$,
$H_n(x) = 4x^2 - 2$
(a) Plot the first five eigen functions $\Psi_n(x)$ into a single diagram for ℓ = 1. Annotate your
plot appropriately.
Can you write Python code for it?

def quantum_eignfunctions(x, n, l=1):
    h = scipy.special.hermite(n)

    coef = 1 / np.sqrt(2 ** n *  factorial(n) * l) * np.pi ** -.25
    return coef * np.exp(-x**2 / (2 * l * l)) * h(x / l)

x = np.linspace(-5, 5, 1000)
for n in range(0, 5):
    plt.plot(x, quantum_eignfunctions(x, n), label=r'$n$ = ' + str(n))
plt.legend()
plt.xlabel(r'$x$')
plt.ylabel(r'$\Psi_n(x)$')
plt.grid(linestyle=':')


I am working on the file "CQD/Project.ipynb".
The first task is:
(b)
One can plot the given functions over the computation for $|\Psi_{1, 2}(t)|^2$ for values of $V = 1$ and $E_2  = 20$. For being able to distinguish the approximation from $|\Psi_{1, 2}(t)|^2$ we plot and enlargment for the the first coordinate.
Can you write Python code for it?


I am working on the file "CQD/Project.ipynb".
The first task is:
(b)
We know that in our simulation we work with a system with low ionation probability, this suggests that the wave function does not move much from the initial position. All of this would indicate that the kinetic momentum is low, so the optimal gauge is the length gauge, as the velocity gauge would shift the momentum by $A(t)$.
I will first  do (d) and then (c) and (e) together.
(d)
We can approximate our system to an electron in a sinusoidal electric field in the classical regime. Newton's equation then is :
\begin{equation}
    m\ddot{x} = E_0 \sin(\omega t) sin^2(\frac{\omega t}{2 n})
\end{equation}
by integrating this equation we obtain:
\begin{equation}
    m\dot{x} =  \frac{E_0}{\omega} \frac{\cos(\omega t) (n^2 \cos(\frac{\omega t}{n}) - n^2 + 1) + n \sin(\omega t) \sin(\frac{\omega t}{n}) -1}{2 (n^2 - 1)}
\end{equation}
looking at the possible maxima of this function one can put an upper bound on the momentum in the case of $n \leq 1$:
\begin{equation}
    \tilde{p} \leq (n^2 E_0) / ((n^2 -1) \omega)
\end{equation}
if a general bound is needed one can use $((n^2 + 1) E_0)$ in place of the numerator. Obviously this is just an upper bound on the momentum and not a maximum, one can use a better value depending on the parameters, by calculating the maximum numerically.
Having an approximation for the maximum momentum we estimate the spacing of the spacial grid and temporal step with:
\begin{equation}
        \Delta x \leq \frac{\hbar\pi}{\tilde{p}} \quad \text{and} \quad \Delta t \leq \frac{\hbar\pi}{\tilde{E}} = \frac{2\pi \hbar}{\tilde{p}^2}
\end{equation}
For the simulation we need also a maximum and minimum for $x$. Since we do not expect  our wave function to move dramatically we can look and the ground state of the soft-core potential and look when the function is sufficiently close to 0. In the simulation we opted for -15 a.u., 15 a.u. as boundery.
Lastly we will choose the mean of the vanishing potential $\tilde{V}$ based around the size of the damping region $d$ and the average momentum $p$,...
Can you write Python code for it?


I am working on the file "CQD/Project.ipynb".
The first task is:
(f)
I run the previous program for different values of $E_0$.
Can you write Python code for it?

N_e = 20
ionizs = np.zeros(N_e)
norms = np.zeros((N_e, len(ts)))
for j, E0 in enumerate(np.linspace(0, .05, N_e)):
    Psi = deepcopy(psi0)
    for k, t in enumerate(ts):
        U_1 = exp(-0.5 * 1j * (V(x, 1) - 1j *
                               vanish(V_tilde, x, -10, 10)
                               - x * E(t, E0, omega, n)) * dt)
        Psi *= U_1
        Psi = fft(Psi)
        Psi *= U_2
        Psi = ifft(Psi)  # go to real space
        Psi *= U_1
        norms[j, k] = sum(abs(Psi) ** 2 * dx)
    ionizs[j] = 1 - sum(abs(Psi) ** 2 * dx)


I am working on the file "CQD/Project.ipynb".
The first task is:
1. Superexchange in a three-level system.
(a)
For calculating the occupation probability we can diagonalize the hamiltonian and compute $\Psi(t) as$:
\begin{equation}
\Psi(t) = D^\dagger e^{-i\Lambda t /\hbar} D \Psi(0)
\end{equation}
Where $e^{-i\Lambda t /\hbar}$ is a diagonal matrix with on the diagonal $e^{-i\lambda_i t /\hbar}$ for each eigenvalue.
Can you write Python code for it?


I am working on the file "CQD/Project.ipynb".
The first task is:
2. The one-dimensional soft-core potential.
We can discretize the hamiltonian with the finite difference scheme and compute the eigenvalues and eigenvector. I plotted the eigenfunctions for the 3 lowest energy eigenstates and in the second plot you can see the different bound energies compered to the potential.
Can you write Python code for it?


I am working on the file "CRUCK v1.ipynb".
The first task is:

- CLEANDATA -

Can you write Python code for it?

modelRatings = df.pivot_table(index=['EQP_LOCAL_EQP'],columns=['EQP_MODEL_EQP'],values='EQP_FAIL_CNT_EQP').iloc[:, 1:10]
modelRatings.head()


I am working on the file "CRUCK v1.ipynb".
The first task is:

- PAIRWISE -

Can you write Python code for it?

similarModels = modelRatings.corrwith(MCARD9060Ratings)
similarModels = similarModels.dropna()
df = pd.DataFrame(similarModels)
df.head(20)


I am working on the file "CS/Part_1_Complexity_RunTimeAnalysis.ipynb".
The first task is:
3. Can we do better?
A polynomial algorithm for $fib$
Let’s try to understand why $fib$ is so slow. fib.call_count shows the count of recursive invocations triggered by a single call to $fib(5)$, which is 15. If you sketched it out, you will notice that many computations are repeated!
A more sensible scheme would store the intermediate results—the values $F_0 , F_1 , . . . , F_{n−1}$ as soon as they become known.
Lets do exactly that through memoization. Note that you can also do this by writing a polynomial algorithm.
Memoization
Tree-recursive computational processes can often be made more efficient through memoization, a powerful technique for increasing the efficiency of recursive functions that repeat computation. A memoized function will store the return value for any arguments it has previously received. A second call to fib(30) would not re-compute the return value recursively, but instead return the existing one that has already been constructed.
Memoization can be expressed naturally as a higher-order function, which can also be used as a decorator. The definition below creates a cache of previously computed results, indexed by the arguments from which they were computed. The use of a dictionary requires that the argument to the memoized function be immutable.
Can you write Python code for it?

def memo(f):
    cache = {}
    def memoized(n):
        if n not in cache:
            cache[n] = f(n) # Make a mapping between the key "n" and the return value of f(n)
        return cache[n]
    return memoized

fib = memo(fib)

t0 = time()

n = 400
fib(n)

print ('This memoized implementation of fib(', n, ') took', round(time() - t0, 4), 'secs')

t0 = time()

n = 300
fib(n)

print ('This memoized implementation of fib(', n, ') took', round(time() - t0, 4), 'secs')

# Here is the polynomial algorithm for fibonacci sequence
def fib2(n):
    if n == 0:
        return 0

    f = [0] * (n+1) # create an array f[0 . . . n]
    f[0], f[1] = 0, 1

    for i in range(2, n+1):
        f[i] = f[i-1] + f[i-2]

    return f[n]

fib2 = count(fib2)

t0 = time()

n = 3000
fib2(n)

print ('This polynomial implementation of fib2(', n, ') took', round(time() - t0, 4), 'secs')

fib2.call_count


I am working on the file "CS/Part_1_Complexity_RunTimeAnalysis.ipynb".
The first task is:
Instead of reporting that an algorithm takes, say, $ 5n^3 + 4n + 3$ steps on an input of size $n$, it is much simpler to leave out lower-order terms such as $4n$ and $3$ (which become insignificant as $n$ grows), and even the detail of the coefficient $5$ in the leading term (computers will be five times faster in a few years anyway), and just say that the algorithm takes time $O(n^3)$ (pronounced “big oh of $n^3$”).
It is time to define this notation precisely. In what follows, think of $f(n)$ and $g(n)$ as the running times of two algorithms on inputs of size $n$.

Let $f(n)$ and $g(n)$ be functions from positive integers to positive reals. We say $f = O(g)$ (which means that “$f$ grows no faster than $g$”) if there is a constant $c > 0$ such that
${f(n) ≤ c · g(n)}$.

Saying $f = O(g)$ is a very loose analog of “$f ≤ g$.” It differs from the usual notion of ≤ because of the constant c, so that for instance $10n = O(n)$. This constant also allows us to disregard what happens for small values of $n$.
Example:
For example, suppose we are choosing between two algorithms for a particular computational task. One takes $f_1(n) = n^2$ steps, while the other takes $f_2(n) = 2n + 20$ steps. Which is better? Well, this depends on the value of $n$. For $n ≤ 5$, $f_1(n)$ is smaller; thereafter, $f_2$ is the clear winner. In this case, $f_2$ scales much better as $n$ grows, and therefore it is superior.
Can you write Python code for it?

t = arange(0, 15, 1)
f1 = t * t
f2 = 2*t + 20

pyplt.title('Exponential time vs Linear time')
plot(t, f1, t, f2)
pyplt.annotate('$n^2$', xy=(8, 1), xytext=(10, 108))
pyplt.annotate('$2n + 20$', xy=(5, 1), xytext=(10, 45))
pyplt.xlabel('n')
pyplt.ylabel('Run time')
pyplt.grid(True)


I am working on the file "CS/Part_1_Complexity_RunTimeAnalysis.ipynb".
The first task is:
Whenever we have an algorithm, there are three questions we always ask about it:

Is it correct?
How much time does it take, as a function of n?
And can we do better?

1. Correctness
For this question, the answer is yes because it is almost a line by line implementation of the definition of the Fibonacci sequence.
2. Time complexity as a function of n
Let $T(n)$ be the number of computer steps needed to compute $fib(n)$; what can we say about this function? For starters, if $n$ is less than 2, the procedure halts almost immediately, after just a couple of steps. Therefore,
$$ T(n)≤2 \, \mbox{for} \, n≤1. $$
For larger values of $n$, there are two recursive invocations of $fib$, taking time $T (n − 1)$ and $T(n−2)$, respectively, plus three computer steps (checks on the value of $n$ and a final addition).
Therefore,
$$ T(n) = T(n−1) + T(n−2)+3\, \mbox{for} \,n>1. $$
Compare this to the recurrence relation for $F_n$, we immediately see that $T(n) ≥ F_n$.
This is very bad news: the running time of the algorithm grows as fast as the Fibonacci numbers! $T(n)$ is exponential in $n$, which implies that the algorithm is impractically slow except for very small values of $n$.
Let’s be a little more concrete about just how bad exponential time is. To compute $F_{200}$,
the $fib$ algorithm executes $T (200) ≥ F_{200} ≥ 2^{138}$ elementary computer steps. How long this actually takes depends, of course, on the computer used. At this time, the fastest computer in the world is the NEC Earth Simulator, which clocks 40 trillion steps per second. Even on this machine, $fib(200)$ would take at least $2^{92}$ seconds. This means that, if we start the computation today, it would still be going long after the sun turns into a red giant star.
Can you write Python code for it?

# This function provides a way to track function calls

def count(f):
    def counted(n):
        counted.call_count += 1
        return f(n)
    counted.call_count = 0
    return counted

fib = count(fib)

t0 = time()

n = 5
fib(n)

print ('This recursive implementation of fib(', n, ') took', round(time() - t0, 4), 'secs')
print ('And {0} calls to the function'.format(fib.call_count))

t0 = time()

n = 30
fib(n)

print ('This recursive implementation of fib(', n, ') took', round(time() - t0, 4), 'secs')
print ('And {0} calls to the function'.format(fib.call_count))


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(1c) Apply makePlural to the base RDD
Now pass each item in the base RDD into a map() transformation that applies the makePlural() function to each element. And then call the collect() action to see the transformed RDD.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
pluralRDD = wordsRDD.map(makePlural)
print pluralRDD.collect()

# TEST Apply makePlural to the base RDD(1c)
Test.assertEquals(pluralRDD.collect(), ['cats', 'elephants', 'rats', 'rats', 'cats'],
                  'incorrect values for pluralRDD')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(1d) Pass a lambda function to map
Let's create the same RDD using a lambda function.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
pluralLambdaRDD = wordsRDD.map(lambda s: s + "s")
print pluralLambdaRDD.collect()

# TEST Pass a lambda function to map (1d)
Test.assertEquals(pluralLambdaRDD.collect(), ['cats', 'elephants', 'rats', 'rats', 'cats'],
                  'incorrect values for pluralLambdaRDD (1d)')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(1e) Length of each word
Now use map() and a lambda function to return the number of characters in each word.  We'll collect this result directly into a variable.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
pluralLengths = (pluralRDD
                 .map(len)
                 .collect())
print pluralLengths

# TEST Length of each word (1e)
Test.assertEquals(pluralLengths, [4, 9, 4, 4, 4],
                  'incorrect values for pluralLengths')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(1f) Pair RDDs
The next step in writing our word counting program is to create a new type of RDD, called a pair RDD. A pair RDD is an RDD where each element is a pair tuple (k, v) where k is the key and v is the value. In this example, we will create a pair consisting of ('<word>', 1) for each word element in the RDD.
We can create the pair RDD using the map() transformation with a lambda() function to create a new RDD.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
wordPairs = wordsRDD.map(lambda a: (a, 1))
print wordPairs.collect()

# TEST Pair RDDs (1f)
Test.assertEquals(wordPairs.collect(),
                  [('cat', 1), ('elephant', 1), ('rat', 1), ('rat', 1), ('cat', 1)],
                  'incorrect value for wordPairs')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(2d) All together
The expert version of the code performs the map() to pair RDD, reduceByKey() transformation, and collect in one statement.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# secode method:
#dd.map(lambda x: (x, 1)).reduceByKey(lambda k, v: v + v).collect()
wordCountsCollected = (wordsRDD
                       .map(lambda a: (a, 1))
                       .reduceByKey(lambda a, b: a + b)
                       .collect())
print wordCountsCollected

# TEST All together (2d)
Test.assertEquals(sorted(wordCountsCollected), [('cat', 2), ('elephant', 1), ('rat', 2)],
                  'incorrect value for wordCountsCollected')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(3b) Mean using reduce
Find the mean number of words per unique word in wordCounts.
Use a reduce() action to sum the counts in wordCounts and then divide by the number of unique words.  First map() the pair RDD wordCounts, which consists of (key, value) pairs, to an RDD of values.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
from operator import add
print wordCounts.collect()
totalCount = wordCounts.map(lambda (k, v): v).reduce(lambda x, y: x + y)
print 'totalCount:', totalCount
average = totalCount / float(uniqueWords)
print totalCount
print round(average, 2)

# TEST Mean using reduce (3b)
Test.assertEquals(round(average, 2), 1.67, 'incorrect value of average')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(4c) Load a text file
For the next part of this lab, we will use the Complete Works of William Shakespeare from Project Gutenberg. To convert a text file into an RDD, we use the SparkContext.textFile() method. We also apply the recently defined removePunctuation() function using a map() transformation to strip out the punctuation and change all text to lowercase.  Since the file is large we use take(15), so that we only print 15 lines.
Can you write Python code for it?

# Just run this code
import os.path
baseDir = os.path.join('data')
inputPath = os.path.join('cs100', 'lab1', 'shakespeare.txt')
fileName = os.path.join(baseDir, inputPath)

shakespeareRDD = (sc
                  .textFile(fileName, 8)
                  .map(removePunctuation))
#print shakespeareRDD.collect()
print '\n'.join(shakespeareRDD
                .zipWithIndex()  # to (line, lineNum)
                .map(lambda (l, num): '{0}: {1}'.format(num, l))  # to 'lineNum: line'
                .take(15))


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(4d) Words from lines
Before we can use the wordcount() function, we have to address two issues with the format of the RDD:

The first issue is that  that we need to split each line by its spaces.

The second issue is we need to filter out empty lines.

Apply a transformation that will split each element of the RDD by its spaces. For each element of the RDD, you should apply Python's string split() function. You might think that a map() transformation is the way to do this, but think about what the result of the split() function will be.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
shakespeareWordsRDD = shakespeareRDD.flatMap(lambda s: s.split(' '))
shakespeareWordCount = shakespeareWordsRDD.count()
print shakespeareWordsRDD.top(5)
print shakespeareWordCount

# TEST Words from lines (4d)
# This test allows for leading spaces to be removed either before or after
# punctuation is removed.
Test.assertTrue(shakespeareWordCount == 927631 or shakespeareWordCount == 928908,
                'incorrect value for shakespeareWordCount')
Test.assertEquals(shakespeareWordsRDD.top(5),
                  [u'zwaggerd', u'zounds', u'zounds', u'zounds', u'zounds'],
                  'incorrect value for shakespeareWordsRDD')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(4e) Remove empty elements
The next step is to filter out the empty elements.  Remove all entries where the word is ''.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
shakeWordsRDD = shakespeareWordsRDD.filter(lambda s: s != '')
shakeWordCount = shakeWordsRDD.count()
print shakeWordCount

# TEST Remove empty elements (4e)
Test.assertEquals(shakeWordCount, 882996, 'incorrect value for shakeWordCount')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
(4f) Count the words
We now have an RDD that is only words.  Next, let's apply the wordCount() function to produce a list of word counts. We can view the top 15 words by using the takeOrdered() action; however, since the elements of the RDD are pairs, we need a custom sort function that sorts using the value part of the pair.
You'll notice that many of the words are common English words. These are called stopwords. In a later lab, we will see how to eliminate them from the results.
Use the wordCount() function and takeOrdered() to obtain the fifteen most common words and their counts.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
#print shakeWordsRDD.collect()
# method 1 (success)
#from operator import add
#top15WordsAndCounts = shakeWordsRDD.map(lambda w: (w, 1)).reduceByKey(add)
# method 2
top15WordsAndCounts = shakeWordsRDD.map(lambda w: (w, 1)).reduceByKey(lambda x, y: x + y).top(15, key = lambda (k, v): v)
print top15WordsAndCounts
print '\n'.join(map(lambda (w, c): '{0}: {1}'.format(w, c), top15WordsAndCounts))

# TEST Count the words (4f)
Test.assertEquals(top15WordsAndCounts,
                  [(u'the', 27361), (u'and', 26028), (u'i', 20681), (u'to', 19150), (u'of', 17463),
                   (u'a', 14593), (u'you', 13615), (u'my', 12481), (u'in', 10956), (u'that', 10890),
                   (u'is', 9134), (u'not', 8497), (u'with', 7771), (u'me', 7769), (u'it', 7678)],
                  'incorrect value for top15WordsAndCounts')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab1_word_count_student.ipynb".
The first task is:
Part 3: Finding unique words and a mean value
 (3a) Unique words
Calculate the number of unique words in wordsRDD.  You can use other RDDs that you have already created to make this easier.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
uniqueWords = len(set(wordsRDD.collect()))
print uniqueWords

# TEST Unique words (3a)
Test.assertEquals(uniqueWords, 3, 'incorrect count of uniqueWords')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(1b) Configuration and Initial RDD Creation
We are ready to specify the input log file and create an RDD containing the parsed log file data. The log file has already been downloaded for you.
To create the primary RDD that we'll use in the rest of this assignment, we first load the text file using sc.textfile(logFile) to convert each line of the file into an element in an RDD.
Next, we use map(parseApacheLogLine) to apply the parse function to each element (that is, a line from the log file) in the RDD and turn each line into a pair Row object.
Finally, we cache the RDD in memory since we'll use it throughout this notebook.
Can you write Python code for it?

import sys
import os
from test_helper import Test

baseDir = os.path.join('data')
inputPath = os.path.join('cs100', 'lab2', 'apache.access.log.PROJECT')
logFile = os.path.join(baseDir, inputPath)
print baseDir
print inputPath
print logFile

import sys
import os
from test_helper import Test

baseDir = os.path.join('data')
inputPath = os.path.join('cs100', 'lab2', 'apache.access.log.PROJECT')
logFile = os.path.join(baseDir, inputPath)

def parseLogs():
    """ Read and parse log file """
    parsed_logs = (sc
                   .textFile(logFile)
                   .map(parseApacheLogLine)
                   .cache())

    access_logs = (parsed_logs
                   .filter(lambda s: s[1] == 1)
                   .map(lambda s: s[0])
                   .cache())

    failed_logs = (parsed_logs
                   .filter(lambda s: s[1] == 0)
                   .map(lambda s: s[0]))
    failed_logs_count = failed_logs.count()
    if failed_logs_count > 0:
        print 'Number of invalid logline: %d' % failed_logs.count()
        for line in failed_logs.take(20):
            print 'Invalid logline: %s' % line

    print 'Read %d lines, successfully parsed %d lines, failed to parse %d lines' % (parsed_logs.count(), access_logs.count(), failed_logs.count())
    return parsed_logs, access_logs, failed_logs

parsed_logs, access_logs, failed_logs = parseLogs()


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(2f) Example: Top Endpoints
For the final example, we'll look at the top endpoints (URIs) in the log. To determine them, we first create a new RDD by using a lambda function to extract the endpoint field from the access_logs RDD using a pair tuple consisting of the endpoint and 1 which will let us count how many records were created by a particular host's request. Using the new RDD, we perform a reduceByKey function with a lambda function that adds the two values. We then extract the top ten endpoints by performing a takeOrdered with a value of 10 and a lambda function that multiplies the count (the second element of each pair) by -1 to create a sorted list with the top endpoints at the bottom.
Can you write Python code for it?

# Top Endpoints
endpointCounts = (access_logs
                  .map(lambda log: (log.endpoint, 1))
                  .reduceByKey(lambda a, b : a + b))

topEndpoints = endpointCounts.takeOrdered(10, lambda s: -1 * s[1])

print 'Top Ten Endpoints: %s' % topEndpoints
assert topEndpoints == [(u'/images/NASA-logosmall.gif', 59737), (u'/images/KSC-logosmall.gif', 50452), (u'/images/MOSAIC-logosmall.gif', 43890), (u'/images/USA-logosmall.gif', 43664), (u'/images/WORLD-logosmall.gif', 43277), (u'/images/ksclogo-medium.gif', 41336), (u'/ksc.html', 28582), (u'/history/apollo/images/apollo-logo1.gif', 26778), (u'/images/launch-logo.gif', 24755), (u'/', 20292)], 'incorrect Top Ten Endpoints'


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(3f) Exercise: Visualizing the Average Daily Requests per Unique Host
Using the result avgDailyReqPerHost from the previous exercise, use matplotlib to plot a "Line" graph of the average daily requests per unique host by day.
daysWithAvg should be a list of days and avgs should be a list of average daily requests per unique hosts for each corresponding day.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
print 'Average number of daily requests per Hosts is %s' % avgDailyReqPerHostList
daysWithAvg = avgDailyReqPerHost.map(lambda a : a[0]).collect()
avgs = avgDailyReqPerHost.map(lambda a : a[1]).collect()

# TEST Average Daily Requests per Unique Host (3f)
Test.assertEquals(daysWithAvg, [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], 'incorrect days')
Test.assertEquals(avgs, [13, 12, 14, 12, 12, 13, 13, 14, 13, 14, 13, 13, 13, 13, 13, 13, 13, 12, 12, 13, 12], 'incorrect avgs')

fig = plt.figure(figsize=(8,4.2), facecolor='white', edgecolor='white')
plt.axis([0, max(daysWithAvg), 0, max(avgs)+2])
plt.grid(b=True, which='major', axis='y')
plt.xlabel('Day')
plt.ylabel('Average')
plt.plot(daysWithAvg, avgs)
pass


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(4b) Exercise: Listing 404 Response Code Records
Using the RDD containing only log records with a 404 response code that you cached in part (4a), print out a list up to 40 distinct endpoints that generate 404 errors -  no endpoint should appear more than once in your list.
Can you write Python code for it?

print badRecords.take(1)

# TODO: Replace <FILL IN> with appropriate code

badEndpoints = badRecords.map(lambda log : (log.endpoint, 1))

badUniqueEndpoints = badEndpoints.reduceByKey(lambda a , b : a + b).map(lambda a : a[0])
print "badUniqueEndpoints.count():", badUniqueEndpoints.count()
print

badUniqueEndpointsPick40 = badUniqueEndpoints.take(40)
print '404 URLS: %s' % badUniqueEndpointsPick40

# TEST Listing 404 records (4b)

badUniqueEndpointsSet40 = set(badUniqueEndpointsPick40)
Test.assertEquals(len(badUniqueEndpointsSet40), 40, 'badUniqueEndpointsPick40 not distinct')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(4c) Exercise: Listing the Top Twenty 404 Response Code Endpoints
Using the RDD containing only log records with a 404 response code that you cached in part (4a), print out a list of the top twenty endpoints that generate the most 404 errors.
Remember, top endpoints should be in sorted order
Can you write Python code for it?

print badRecords.take(1)

# TODO: Replace <FILL IN> with appropriate code

badEndpointsCountPairTuple = badRecords.map(lambda log : (log.endpoint, 1))

badEndpointsSum = badEndpointsCountPairTuple.reduceByKey(lambda a , b: a + b).cache()
print "badEndP=pointsSum.count():", badEndpointsSum.count()
print

badEndpointsTop20 = badEndpointsSum.takeOrdered(20, lambda (a, b): -b)

print 'Top Twenty 404 URLs: %s' % badEndpointsTop20

# TEST Top twenty 404 URLs (4c)
Test.assertEquals(badEndpointsTop20, [(u'/pub/winvn/readme.txt', 633), (u'/pub/winvn/release.txt', 494), (u'/shuttle/missions/STS-69/mission-STS-69.html', 431), (u'/images/nasa-logo.gif', 319), (u'/elv/DELTA/uncons.htm', 178), (u'/shuttle/missions/sts-68/ksc-upclose.gif', 156), (u'/history/apollo/sa-1/sa-1-patch-small.gif', 146), (u'/images/crawlerway-logo.gif', 120), (u'/://spacelink.msfc.nasa.gov', 117), (u'/history/apollo/pad-abort-test-1/pad-abort-test-1-patch-small.gif', 100), (u'/history/apollo/a-001/a-001-patch-small.gif', 97), (u'/images/Nasa-logo.gif', 85), (u'/shuttle/resources/orbiters/atlantis.gif', 64), (u'/history/apollo/images/little-joe.jpg', 62), (u'/images/lf-logo.gif', 59), (u'/shuttle/resources/orbiters/discovery.gif', 56), (u'/shuttle/resources/orbiters/challenger.gif', 54), (u'/robots.txt', 53), (u'/elv/new01.gif>', 43), (u'/history/apollo/pad-abort-test-2/pad-abort-test-2-patch-small.gif', 38)], 'incorrect badEndpointsTop20')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(4e) Exercise: Listing 404 Response Codes per Day
Let's explore the 404 records temporally. Break down the 404 requests by day (cache() the RDD errDateSorted) and get the daily counts sorted by day as a list.
Since the log only covers a single month, you can ignore the month in your checks.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code

errDateCountPairTuple = badRecords.map(lambda log : (log.date_time.date().day, 1))

errDateSum = errDateCountPairTuple.reduceByKey(lambda a , b: a + b).cache()

errDateSorted = errDateSum.sortByKey().cache()

errByDate = errDateSorted.take(30)
print '404 Errors by day: %s' % errByDate

# TEST 404 response codes per day (4e)
Test.assertEquals(errByDate, [(1, 243), (3, 303), (4, 346), (5, 234), (6, 372), (7, 532), (8, 381), (9, 279), (10, 314), (11, 263), (12, 195), (13, 216), (14, 287), (15, 326), (16, 258), (17, 269), (18, 255), (19, 207), (20, 312), (21, 305), (22, 288)], 'incorrect errByDate')
Test.assertTrue(errDateSorted.is_cached, 'incorrect errDateSorted.is_cached')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(4f) Exercise: Visualizing the 404 Response Codes by Day
Using the results from the previous exercise, use matplotlib to plot a "Line" or "Bar" graph of the 404 response codes by day.
Can you write Python code for it?

print "errDateSorted.take(1):", errDateSorted.take(1)
print
print "errDateSorted.count():", errDateSorted.count()
print
print "errDateSorted.collect():", errDateSorted.collect()

# TODO: Replace <FILL IN> with appropriate code

daysWithErrors404 = errDateSorted.map(lambda a : a[0]).collect()
print "daysWithErrors404:", daysWithErrors404
print

errors404ByDay = errDateSorted.map(lambda a : a[1]).collect()
print "errors404ByDay:", errors404ByDay

# TEST Visualizing the 404 Response Codes by Day (4f)
Test.assertEquals(daysWithErrors404, [1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22], 'incorrect daysWithErrors404')
Test.assertEquals(errors404ByDay, [243, 303, 346, 234, 372, 532, 381, 279, 314, 263, 195, 216, 287, 326, 258, 269, 255, 207, 312, 305, 288], 'incorrect errors404ByDay')

fig = plt.figure(figsize=(8,4.2), facecolor='white', edgecolor='white')
plt.axis([0, max(daysWithErrors404), 0, max(errors404ByDay)])
plt.grid(b=True, which='major', axis='y')
plt.xlabel('Day')
plt.ylabel('404 Errors')
plt.plot(daysWithErrors404, errors404ByDay)
pass


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(4g) Exercise: Top Five Days for 404 Response Codes
Using the RDD errDateSorted you cached in the part (4e), what are the top five days for 404 response codes and the corresponding counts of 404 response codes?
Can you write Python code for it?

print errDateSorted.collect()

# TODO: Replace <FILL IN> with appropriate code

topErrDate = errDateSorted.takeOrdered(5 , lambda (a, b): -b)
print 'Top Five dates for 404 requests: %s' % topErrDate

# TEST Five dates for 404 requests (4g)
Test.assertEquals(topErrDate, [(7, 532), (8, 381), (6, 372), (4, 346), (15, 326)], 'incorrect topErrDate')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
(4h) Exercise: Hourly 404 Response Codes
Using the RDD badRecords you cached in the part (4a) and by hour of the day and in decreasing order, create an RDD containing how many requests had a 404 return code for each hour of the day. Cache the resulting RDD hourRecordsSorted and print that as a list.
Can you write Python code for it?

print "badRecords.count():", badRecords.count()
print
print "badRecords.take(1):", badRecords.take(1)

# TODO: Replace <FILL IN> with appropriate code

hourCountPairTuple = badRecords.map(lambda a : (a.date_time.hour , 1))

hourRecordsSum = hourCountPairTuple.reduceByKey(lambda a , b: a + b)

hourRecordsSorted = hourRecordsSum.sortByKey().cache()

errHourList = hourRecordsSorted.take(24)
print 'Top hours for 404 requests: %s' % errHourList

# TEST Hourly 404 response codes (4h)
Test.assertEquals(errHourList, [(0, 175), (1, 171), (2, 422), (3, 272), (4, 102), (5, 95), (6, 93), (7, 122), (8, 199), (9, 185), (10, 329), (11, 263), (12, 438), (13, 397), (14, 318), (15, 347), (16, 373), (17, 330), (18, 268), (19, 269), (20, 270), (21, 241), (22, 234), (23, 272)], 'incorrect errHourList')
Test.assertTrue(hourRecordsSorted.is_cached, 'incorrect hourRecordsSorted.is_cached')


I am working on the file "CS100.1x Introduction to Big Data with Apache Spark/lab2_apache_log_student.ipynb".
The first task is:
Part 3: Analyzing Web Server Log File
Now it is your turn to perform analyses on web server log files.
(3a) Exercise: Top Ten Error Endpoints
What are the top ten endpoints which did not have return code 200? Create a sorted list containing top ten endpoints and the number of times that they were accessed with non-200 return code.
Think about the steps that you need to perform to determine which endpoints did not have a 200 return code, how you will uniquely count those endpoints, and sort the list.
You might want to refer back to the previous Lab (Lab 1 Word Count) for insights.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# HINT: Each of these <FILL IN> below could be completed with a single transformation or action.
# You are welcome to structure your solution in a different way, so long as
# you ensure the variables used in the next Test section are defined (ie. endpointSum, topTenErrURLs).

not200 = access_logs.filter(lambda log : log.response_code != 200)

endpointCountPairTuple = not200.map(lambda log: (log.endpoint, 1))

endpointSum = endpointCountPairTuple.reduceByKey(lambda a, b : a + b).cache()

topTenErrURLs = endpointSum.takeOrdered(10, lambda s: -1 * s[1])
print 'Top Ten failed URLs: %s' % topTenErrURLs

# TEST Top ten error endpoints (3a)
Test.assertEquals(endpointSum.count(), 7689, 'incorrect count for endpointSum')
Test.assertEquals(topTenErrURLs, [(u'/images/NASA-logosmall.gif', 8761), (u'/images/KSC-logosmall.gif', 7236), (u'/images/MOSAIC-logosmall.gif', 5197), (u'/images/USA-logosmall.gif', 5157), (u'/images/WORLD-logosmall.gif', 5020), (u'/images/ksclogo-medium.gif', 4728), (u'/history/apollo/images/apollo-logo1.gif', 2907), (u'/images/launch-logo.gif', 2811), (u'/', 2199), (u'/images/ksclogosmall.gif', 1622)], 'incorrect Top Ten failed URLs (topTenErrURLs)')


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/features-checkpoint.ipynb".
The first task is:
Extract Features
For each image we will compute a Histogram of Oriented
Gradients (HOG) as well as a color histogram using the hue channel in HSV
color space. We form our final feature vector for each image by concatenating
the HOG and color histogram feature vectors.
Roughly speaking, HOG should capture the texture of the image while ignoring
color information, and the color histogram represents the color of the input
image while ignoring texture. As a result, we expect that using both together
ought to work better than using either alone. Verifying this assumption would
be a good thing to try for the bonus section.
The hog_feature and color_histogram_hsv functions both operate on a single
image and return a feature vector for that image. The extract_features
function takes a set of images and a list of feature functions and evaluates
each feature function on each image, storing the results in a matrix where
each column is the concatenation of all feature vectors for a single image.
Can you write Python code for it?

from cs231n.features import *

num_color_bins = 10 # Number of bins in the color histogram
feature_fns = [hog_feature, lambda img: color_histogram_hsv(img, nbin=num_color_bins)]
X_train_feats = extract_features(X_train, feature_fns, verbose=True)
X_val_feats = extract_features(X_val, feature_fns)
X_test_feats = extract_features(X_test, feature_fns)

# Preprocessing: Subtract the mean feature
mean_feat = np.mean(X_train_feats, axis=0, keepdims=True)
X_train_feats -= mean_feat
X_val_feats -= mean_feat
X_test_feats -= mean_feat

# Preprocessing: Divide by standard deviation. This ensures that each feature
# has roughly the same scale.
std_feat = np.std(X_train_feats, axis=0, keepdims=True)
X_train_feats /= std_feat
X_val_feats /= std_feat
X_test_feats /= std_feat

# Preprocessing: Add a bias dimension
X_train_feats = np.hstack([X_train_feats, np.ones((X_train_feats.shape[0], 1))])
X_val_feats = np.hstack([X_val_feats, np.ones((X_val_feats.shape[0], 1))])
X_test_feats = np.hstack([X_test_feats, np.ones((X_test_feats.shape[0], 1))])


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/features-checkpoint.ipynb".
The first task is:
Inline question 1:
Describe the misclassification results that you see. Do they make sense?
Neural Network on image features
Earlier in this assigment we saw that training a two-layer neural network on raw pixels achieved better classification performance than linear classifiers on raw pixels. In this notebook we have seen that linear classifiers on image features outperform linear classifiers on raw pixels.
For completeness, we should also try training a neural network on image features. This approach should outperform all previous approaches: you should easily be able to achieve over 55% classification accuracy on the test set; our best model achieves about 60% classification accuracy.
Can you write Python code for it?

print(X_train_feats.shape)

from cs231n.classifiers.neural_net import TwoLayerNet

input_dim = X_train_feats.shape[1]
hidden_dim = 500
num_classes = 10

net = TwoLayerNet(input_dim, hidden_dim, num_classes)
best_net = None

################################################################################
# TODO: Train a two-layer neural network on image features. You may want to    #
# cross-validate various parameters as in previous sections. Store your best   #
# model in the best_net variable.                                              #
################################################################################
pass
################################################################################
#                              END OF YOUR CODE                                #
################################################################################

# Run your neural net classifier on the test set. You should be able to
# get more than 55% accuracy.

test_acc = (net.predict(X_test_feats) == y_test).mean()
print(test_acc)


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/features-checkpoint.ipynb".
The first task is:
Load data
Similar to previous exercises, we will load CIFAR-10 data from disk.
Can you write Python code for it?

from cs231n.features import color_histogram_hsv, hog_feature

def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):
    # Load the raw CIFAR-10 data
    cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

    # Subsample the data
    mask = list(range(num_training, num_training + num_validation))
    X_val = X_train[mask]
    y_val = y_train[mask]
    mask = list(range(num_training))
    X_train = X_train[mask]
    y_train = y_train[mask]
    mask = list(range(num_test))
    X_test = X_test[mask]
    y_test = y_test[mask]

    return X_train, y_train, X_val, y_val, X_test, y_test

X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/features-checkpoint.ipynb".
The first task is:
Train SVM on features
Using the multiclass SVM code developed earlier in the assignment, train SVMs on top of the features extracted above; this should achieve better results than training SVMs directly on top of raw pixels.
Can you write Python code for it?

# Use the validation set to tune the learning rate and regularization strength

from cs231n.classifiers.linear_classifier import LinearSVM

learning_rates = [1e-9, 1e-8, 1e-7]
regularization_strengths = [5e4, 5e5, 5e6]

results = {}
best_val = -1
best_svm = None

pass
################################################################################
# TODO:                                                                        #
# Use the validation set to set the learning rate and regularization strength. #
# This should be identical to the validation that you did for the SVM; save    #
# the best trained classifer in best_svm. You might also want to play          #
# with different numbers of bins in the color histogram. If you are careful    #
# you should be able to get accuracy of near 0.44 on the validation set.       #
################################################################################
pass
################################################################################
#                              END OF YOUR CODE                                #
################################################################################

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print('lr %e reg %e train accuracy: %f val accuracy: %f' % (
                lr, reg, train_accuracy, val_accuracy))

print('best validation accuracy achieved during cross-validation: %f' % best_val)

# Evaluate your trained SVM on the test set
y_test_pred = best_svm.predict(X_test_feats)
test_accuracy = np.mean(y_test == y_test_pred)
print(test_accuracy)

# An important way to gain intuition about how an algorithm works is to
# visualize the mistakes that it makes. In this visualization, we show examples
# of images that are misclassified by our current system. The first column
# shows images that our system labeled as "plane" but whose true label is
# something other than "plane".

examples_per_class = 8
classes = ['plane', 'car', 'bird', 'cat',...


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
Backward pass
Implement the rest of the function. This will compute the gradient of the loss with respect to the variables W1, b1, W2, and b2. Now that you (hopefully!) have a correctly implemented forward pass, you can debug your backward pass using a numeric gradient check:
Can you write Python code for it?

from cs231n.gradient_check import eval_numerical_gradient

# Use numeric gradient checking to check your implementation of the backward pass.
# If your implementation is correct, the difference between the numeric and
# analytic gradients should be less than 1e-8 for each of W1, W2, b1, and b2.

loss, grads = net.loss(X, y, reg=0.05)

# these should all be less than 1e-8 or so
for param_name in grads:
    f = lambda W: net.loss(X, y, reg=0.05)[0]
    param_grad_num = eval_numerical_gradient(f, net.params[param_name], verbose=False)
    print('%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name])))


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
Debug the training
With the default parameters we provided above, you should get a validation accuracy of about 0.29 on the validation set. This isn't very good.
One strategy for getting insight into what's wrong is to plot the loss function and the accuracies on the training and validation sets during optimization.
Another strategy is to visualize the weights that were learned in the first layer of the network. In most neural networks trained on visual data, the first layer weights typically show some visible structure when visualized.
Can you write Python code for it?

# Plot the loss function and train / validation accuracies
plt.subplot(2, 1, 1)
plt.plot(stats['loss_history'])
plt.tight_layout()
plt.title('Loss history')
plt.xlabel('Iteration')
plt.ylabel('Loss')

plt.subplot(2, 1, 2)
plt.plot(stats['train_acc_history'], label='train')
plt.plot(stats['val_acc_history'], label='val')
plt.tight_layout()
plt.title('Classification accuracy history')
plt.legend()
plt.xlabel('Epoch')
plt.ylabel('Clasification accuracy')
plt.show()

from cs231n.vis_utils import visualize_grid

# Visualize the weights of the network

def show_net_weights(net):
    W1 = net.params['W1']
    W1 = W1.reshape(32, 32, 3, -1).transpose(3, 0, 1, 2)
    plt.imshow(visualize_grid(W1, padding=3).astype('uint8'))
    plt.gca().axis('off')
    plt.show()

show_net_weights(net)


# Plot the loss function and train / validation accuracies
plt.subplot(2, 1, 1)
plt.plot(stats['loss_history'])
plt.title('Loss history')
plt.xlabel('Iteration')
plt.ylabel('Loss')

plt.subplot(2, 1, 2)
plt.plot(stats['train_acc_history'], label='train')
plt.plot(stats['val_acc_history'], label='val')
plt.title('Classification accuracy history')
plt.xlabel('Epoch')
plt.ylabel('Clasification accuracy')
plt.show()

from cs231n.vis_utils import visualize_grid

# Visualize the weights of the network

def show_net_weights(net):
  W1 = net.params['W1']
  W1 = W1.reshape(32, 32, 3, -1).transpose(3, 0, 1, 2)
  plt.imshow(visualize_grid(W1, padding=3).astype('uint8'))
  plt.gca().axis('off')
  plt.show()

show_net_weights(net)


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
Forward pass: compute scores
Open the file cs231n/classifiers/neural_net.py and look at the method TwoLayerNet.loss. This function is very similar to the loss functions you have written for the SVM and Softmax exercises: It takes the data and weights and computes the class scores, the loss, and the gradients on the parameters.
Implement the first part of the forward pass which uses the weights and biases to compute the scores for all inputs.
Can you write Python code for it?


scores = net.loss(X)
print 'Your scores:'
print scores
print
print 'correct scores:'
correct_scores = np.asarray([
  [-0.81233741, -1.27654624, -0.70335995],
  [-0.17129677, -1.18803311, -0.47310444],
  [-0.51590475, -1.01354314, -0.8504215 ],
  [-0.15419291, -0.48629638, -0.52901952],
  [-0.00618733, -0.12435261, -0.15226949]])
print correct_scores
print

# The difference should be very small. We get < 1e-7
print 'Difference between your scores and correct scores:'
print np.sum(np.abs(scores - correct_scores))


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
Load the data
Now that you have implemented a two-layer network that passes gradient checks and works on toy data, it's time to load up our favorite CIFAR-10 data so we can use it to train a classifier on a real dataset.
Can you write Python code for it?

from cs231n.data_utils import load_CIFAR10

def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):
    """
    Load the CIFAR-10 dataset from disk and perform preprocessing to prepare
    it for the two-layer neural net classifier. These are the same steps as
    we used for the SVM, but condensed to a single function.
    """
    # Load the raw CIFAR-10 data
    cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

    # Subsample the data
    mask = list(range(num_training, num_training + num_validation))
    X_val = X_train[mask]
    y_val = y_train[mask]
    mask = list(range(num_training))
    X_train = X_train[mask]
    y_train = y_train[mask]
    mask = list(range(num_test))
    X_test = X_test[mask]
    y_test = y_test[mask]

    # Normalize the data: subtract the mean image
    mean_image = np.mean(X_train, axis=0)
    X_train -= mean_image
    X_val -= mean_image
    X_test -= mean_image

    # Reshape data to rows
    X_train = X_train.reshape(num_training, -1)
    X_val = X_val.reshape(num_validation, -1)
    X_test = X_test.reshape(num_test, -1)

    return X_train, y_train, X_val, y_val, X_test, y_test

# Invoke the above function to get our data.
X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()
print('Train data shape: ', X_train.shape)
print('Train labels shape: ', y_train.shape)
print('Validation data shape: ', X_val.shape)
print('Validation labels shape: ', y_val.shape)
print('Test data shape: ', X_test.shape)
print('Test labels shape: ', y_test.shape)


from cs231n.data_utils import load_CIFAR10

def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):
    """
    Load the CIFAR-10 dataset from disk and perform preprocessing to prepare
    it for the two-layer neural net classifier. These are the same steps as
    we used for the SVM, but condensed to a single function.
    """
    # Load the raw CIFAR-10 data
    cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

    # Subsample the data
    mask = range(num_training, num_training + num_validation)
    X_val = X_train[mask]
    y_val = y_train[mask]
    mask = range(num_training)
    X_train = X_train[mask]
    y_train = y_train[mask]
    mask = range(num_test)
    X_test = X_test[mask]
    y_test = y_test[mask]

    # Normalize the data: subtract the mean image
    mean_image = np.mean(X_train, axis=0)
    X_train -= mean_image
    X_val -= mean_image
    X_test -= mean_image

    # Reshape data to rows
    X_train = X_train.reshape(num_training, -1)
    X_val = X_val.reshape(num_validation, -1)
    X_test = X_test.reshape(num_test, -1)

    return X_train, y_train, X_val, y_val, X_test, y_test

# Invoke the above function to get our data.
X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()
print 'Train data shape: ', X_train.shape
print 'Train labels shape: ', y_train.shape
print 'Validation data shape: ', X_val.shape
print 'Validation labels shape: ', y_val.shape
print 'Test data shape: ', X_test.shape
print 'Test labels shape: ', y_test.shape


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
Run on the test set
When you are done experimenting, you should evaluate your final trained network on the test set; you should get above 48%.
We will give you extra bonus point for every 1% of accuracy above 52%.
Can you write Python code for it?

test_acc = (best_net.predict(X_test) == y_test).mean()
print('Test accuracy: ', test_acc)


test_acc = (best_net.predict(X_test) == y_test).mean()
print 'Test accuracy: ', test_acc


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
Train a network
To train our network we will use SGD with momentum. In addition, we will adjust the learning rate with an exponential learning rate schedule as optimization proceeds; after each epoch, we will reduce the learning rate by multiplying it by a decay rate.
Can you write Python code for it?

input_size = 32 * 32 * 3
hidden_size = 50
num_classes = 10
net = TwoLayerNet(input_size, hidden_size, num_classes)

# Train the network
stats = net.train(X_train, y_train, X_val, y_val,
            num_iters=1000, batch_size=200,
            learning_rate=1e-4, learning_rate_decay=0.95,
            reg=0.5, verbose=True)

# Predict on the validation set
val_acc = (net.predict(X_val) == y_val).mean()
print 'Validation accuracy: ', val_acc


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
Train the network
To train the network we will use stochastic gradient descent (SGD), similar to the SVM and Softmax classifiers. Look at the function TwoLayerNet.train and fill in the missing sections to implement the training procedure. This should be very similar to the training procedure you used for the SVM and Softmax classifiers. You will also have to implement TwoLayerNet.predict, as the training process periodically performs prediction to keep track of accuracy over time while the network trains.
Once you have implemented the method, run the code below to train a two-layer network on toy data. You should achieve a training loss less than 0.2.
Can you write Python code for it?

net = init_toy_model()
stats = net.train(X, y, X, y,
            learning_rate=1e-1, reg=1e-5,
            num_iters=100, verbose=False)

print 'Final training loss: ', stats['loss_history'][-1]

# plot the loss history
#plt.plot(stats['loss_history'])
#plt.xlabel('iteration')
#plt.ylabel('training loss')
#plt.title('Training Loss history')
#plt.show()


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
Tune your hyperparameters
What's wrong?. Looking at the visualizations above, we see that the loss is decreasing more or less linearly, which seems to suggest that the learning rate may be too low. Moreover, there is no gap between the training and validation accuracy, suggesting that the model we used has low capacity, and that we should increase its size. On the other hand, with a very large model we would expect to see more overfitting, which would manifest itself as a very large gap between the training and validation accuracy.
Tuning. Tuning the hyperparameters and developing intuition for how they affect the final performance is a large part of using Neural Networks, so we want you to get a lot of practice. Below, you should experiment with different values of the various hyperparameters, including hidden layer size, learning rate, numer of training epochs, and regularization strength. You might also consider tuning the learning rate decay, but you should be able to get good performance using the default value.
Approximate results. You should be aim to achieve a classification accuracy of greater than 48% on the validation set. Our best network gets over 52% on the validation set.
Experiment: You goal in this exercise is to get as good of a result on CIFAR-10 as you can, with a fully-connected Neural Network. For every 1% above 52% on the Test set we will award you with one extra bonus point. Feel free implement your own techniques (e.g. PCA to reduce dimensionality, or adding dropout, or adding features to the solver, etc.).
Can you write Python code for it?

best_net = None # store the best model into this

#################################################################################
# TODO: Tune hyperparameters using the validation set. Store your best trained  #
# model in best_net.                                                            #
#                                                                               #
# To help debug your network, it may help to use visualizations similar to the  #
# ones we used above; these visualizations will have significant qualitative    #
# differences from the ones we saw above for the poorly tuned network.          #
#                                                                               #
# Tweaking hyperparameters by hand can be fun, but you might find it useful to  #
# write code to sweep through possible combinations of hyperparameters          #
# automatically like we did on the previous exercises.                          #
#################################################################################
pass
#################################################################################
#                               END OF YOUR CODE                                #
#################################################################################

# visualize the weights of the best network
show_net_weights(best_net)


best_net = None # store the best model into this
best_acc = -1
best_stats = None

from cs231n.pca import PCA

#################################################################################
# TODO: Tune hyperparameters using the validation set. Store your best trained  #
# model in best_net.                                                            #
#                                                                               #
# To help debug your network, it may help to use visualizations similar to the  #
# ones we used above; these visualizations will have significant qualitative    #
# differences from the ones we saw above for the poorly tuned network.          #
#                                                                               #
# Tweaking hyperparameters by hand can be fun, but you might find it useful to  #
# write code to sweep through possible combinations of hyperparameters          #
# automatically like we did on the previous exercises.                          #
#################################################################################
hidden_sizes = [150]
learning_rates = [1e-3]
batch_sizes = [200]#[150]
regularizations = [0.25]#[0.5]
input_sizes = [32 * 32 * 3]

for in_size in input_sizes:
    X_train_pca = PCA(X_train, in_size)
    X_val_pca = PCA(X_val, in_size)

    for h_size in hidden_sizes:
        for lr in learning_rates:
            for batch in batch_sizes:
                for reg in regularizations:
                    print('>>>>> input_size=%d, hidden_size=%d, lr=%.5f, batch_size=%3d, reg=%.2f'
                         % (in_size, h_size, lr, batch, reg))
                    net = TwoLayerNet(in_size, h_size, num_classes)
                    stats = net.train(X_train_pca, y_train, X_val_pca, y_val,
                                num_iters=5000, batch_size=batch,
                                learning_rate=lr, learning_rate_decay=0.95,
                                reg=reg, verbose=False， dropout=True)

                    # Predict on the...


I am working on the file "CS231n/assignment1/.ipynb_checkpoints/two_layer_net-checkpoint.ipynb".
The first task is:
We will use the class TwoLayerNet in the file cs231n/classifiers/neural_net.py to represent instances of our network. The network parameters are stored in the instance variable self.params where keys are string parameter names and values are numpy arrays. Below, we initialize toy data and a toy model that we will use to develop your implementation.
Can you write Python code for it?

# Create a small net and some toy data to check your implementations.
# Note that we set the random seed for repeatable experiments.

input_size = 4
hidden_size = 10
num_classes = 3
num_inputs = 5

def init_toy_model():
    np.random.seed(0)
    return TwoLayerNet(input_size, hidden_size, num_classes, std=1e-1)

def init_toy_data():
    np.random.seed(1)
    X = 10 * np.random.randn(num_inputs, input_size)
    y = np.array([0, 1, 2, 2, 1])
    return X, y

net = init_toy_model()
X, y = init_toy_data()


I am working on the file "CS231n/assignment1/features.ipynb".
The first task is:
Extract Features
For each image we will compute a Histogram of Oriented
Gradients (HOG) as well as a color histogram using the hue channel in HSV
color space. We form our final feature vector for each image by concatenating
the HOG and color histogram feature vectors.
Roughly speaking, HOG should capture the texture of the image while ignoring
color information, and the color histogram represents the color of the input
image while ignoring texture. As a result, we expect that using both together
ought to work better than using either alone. Verifying this assumption would
be a good thing to try for the bonus section.
The hog_feature and color_histogram_hsv functions both operate on a single
image and return a feature vector for that image. The extract_features
function takes a set of images and a list of feature functions and evaluates
each feature function on each image, storing the results in a matrix where
each column is the concatenation of all feature vectors for a single image.
Can you write Python code for it?

from cs231n.features import *

num_color_bins = 10 # Number of bins in the color histogram
feature_fns = [hog_feature, lambda img: color_histogram_hsv(img, nbin=num_color_bins)]
X_train_feats = extract_features(X_train, feature_fns, verbose=True)
X_val_feats = extract_features(X_val, feature_fns)
X_test_feats = extract_features(X_test, feature_fns)

# Preprocessing: Subtract the mean feature
mean_feat = np.mean(X_train_feats, axis=0, keepdims=True)
X_train_feats -= mean_feat
X_val_feats -= mean_feat
X_test_feats -= mean_feat

# Preprocessing: Divide by standard deviation. This ensures that each feature
# has roughly the same scale.
std_feat = np.std(X_train_feats, axis=0, keepdims=True)
X_train_feats /= std_feat
X_val_feats /= std_feat
X_test_feats /= std_feat

# Preprocessing: Add a bias dimension
X_train_feats = np.hstack([X_train_feats, np.ones((X_train_feats.shape[0], 1))])
X_val_feats = np.hstack([X_val_feats, np.ones((X_val_feats.shape[0], 1))])
X_test_feats = np.hstack([X_test_feats, np.ones((X_test_feats.shape[0], 1))])


I am working on the file "CS231n/assignment1/features.ipynb".
The first task is:
Inline question 1:
Describe the misclassification results that you see. Do they make sense?
It seems that picture that misclassified is similar to labeled class by color space.
Neural Network on image features
Earlier in this assigment we saw that training a two-layer neural network on raw pixels achieved better classification performance than linear classifiers on raw pixels. In this notebook we have seen that linear classifiers on image features outperform linear classifiers on raw pixels.
For completeness, we should also try training a neural network on image features. This approach should outperform all previous approaches: you should easily be able to achieve over 55% classification accuracy on the test set; our best model achieves about 60% classification accuracy.
Can you write Python code for it?

print(X_train_feats.shape)

from cs231n.classifiers.neural_net import TwoLayerNet

input_dim = X_train_feats.shape[1]
hidden_dim = 1536
num_classes = 10

learning_rates = [5e-1]
regularization_strengths = [1e-3]
best_net = None
best_val = -1

################################################################################
# TODO: Train a two-layer neural network on image features. You may want to    #
# cross-validate various parameters as in previous sections. Store your best   #
# model in the best_net variable.                                              #
################################################################################

for lr in learning_rates:
    for rg in regularization_strengths:
        nn = TwoLayerNet(input_dim, hidden_dim, num_classes)
        nn.train(X_train_feats, y_train, X_val_feats, y_val,
                 num_iters=5000, batch_size=512,
                 learning_rate=lr, learning_rate_decay=0.9,
                 reg=rg, verbose=False)

        y_train_pred = nn.predict(X_train_feats)
        train_acc = np.mean(y_train == y_train_pred)

        y_val_pred = nn.predict(X_val_feats)
        val_acc = np.mean(y_val == y_val_pred)

        if val_acc > best_val:
             best_val = val_acc
             best_net = nn
        results[(lr, rg)] = (train_acc, val_acc)

        print('lr %e reg %e train accuracy: %f val accuracy: %f' % (
              lr, rg, train_acc, val_acc))

################################################################################
#                              END OF YOUR CODE                                #
################################################################################

# Run your neural net classifier on the test set. You should be able to
# get more than 55% accuracy.

test_acc = (best_net.predict(X_test_feats) == y_test).mean()
print(test_acc)


I am working on the file "CS231n/assignment1/features.ipynb".
The first task is:
Load data
Similar to previous exercises, we will load CIFAR-10 data from disk.
Can you write Python code for it?

from cs231n.features import color_histogram_hsv, hog_feature

def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):
    # Load the raw CIFAR-10 data
    cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

    # Subsample the data
    mask = list(range(num_training, num_training + num_validation))
    X_val = X_train[mask]
    y_val = y_train[mask]
    mask = list(range(num_training))
    X_train = X_train[mask]
    y_train = y_train[mask]
    mask = list(range(num_test))
    X_test = X_test[mask]
    y_test = y_test[mask]

    return X_train, y_train, X_val, y_val, X_test, y_test

X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()


I am working on the file "CS231n/assignment1/features.ipynb".
The first task is:
Train SVM on features
Using the multiclass SVM code developed earlier in the assignment, train SVMs on top of the features extracted above; this should achieve better results than training SVMs directly on top of raw pixels.
Can you write Python code for it?

# Use the validation set to tune the learning rate and regularization strength

from cs231n.classifiers.linear_classifier import LinearSVM

learning_rates = [1e-9, 1e-8, 1e-7]
regularization_strengths = [5e4, 5e5, 5e6]

results = {}
best_val = -1
best_svm = None

################################################################################
# TODO:                                                                        #
# Use the validation set to set the learning rate and regularization strength. #
# This should be identical to the validation that you did for the SVM; save    #
# the best trained classifer in best_svm. You might also want to play          #
# with different numbers of bins in the color histogram. If you are careful    #
# you should be able to get accuracy of near 0.44 on the validation set.       #
################################################################################

for lr in learning_rates:
    for rg in regularization_strengths:
        svm = LinearSVM()
        svm.train(X_train_feats, y_train, learning_rate=lr, reg=rg,
                  num_iters=3000, verbose=False)

        y_train_pred = svm.predict(X_train_feats)
        train_acc = np.mean(y_train == y_train_pred)

        y_val_pred = svm.predict(X_val_feats)
        val_acc = np.mean(y_val == y_val_pred)

        if val_acc > best_val:
             best_val = val_acc
             best_svm = svm
        results[(lr, rg)] = (train_acc, val_acc)

################################################################################
#                              END OF YOUR CODE                                #
################################################################################

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print('lr %e reg %e train accuracy: %f val accuracy: %f' % (
                lr, reg, train_accuracy, val_accuracy))

print('best validation accuracy achieved during cross-validation: %f' %...


# Use the validation set to tune the learning rate and regularization strength

from cs231n.classifiers.linear_classifier import LinearSVM

learning_rates = [1e-9, 1e-8, 1e-7]
regularization_strengths = [5e4, 5e5, 5e6]

results = {}
best_val = -1
best_svm = None

################################################################################
# TODO:                                                                        #
# Use the validation set to set the learning rate and regularization strength. #
# This should be identical to the validation that you did for the SVM; save    #
# the best trained classifer in best_svm. You might also want to play          #
# with different numbers of bins in the color histogram. If you are careful    #
# you should be able to get accuracy of near 0.44 on the validation set.       #
################################################################################
pass
################################################################################
#                              END OF YOUR CODE                                #
################################################################################

# Print out results.
for lr, reg in sorted(results):
    train_accuracy, val_accuracy = results[(lr, reg)]
    print('lr %e reg %e train accuracy: %f val accuracy: %f' % (
                lr, reg, train_accuracy, val_accuracy))

print('best validation accuracy achieved during cross-validation: %f' % best_val)

# Evaluate your trained SVM on the test set
y_test_pred = best_svm.predict(X_test_feats)
test_accuracy = np.mean(y_test == y_test_pred)
print(test_accuracy)

# An important way to gain intuition about how an algorithm works is to
# visualize the mistakes that it makes. In this visualization, we show examples
# of images that are misclassified by our current system. The first column
# shows images that our system labeled as "plane" but whose true label is
# something other than "plane".

examples_per_class = 8
classes = ['plane', 'car', 'bird', 'cat', 'deer',...


I am working on the file "CS231n/assignment1/knn.ipynb".
The first task is:
Cross-validation
We have implemented the k-Nearest Neighbor classifier but we set the value k = 5 arbitrarily. We will now determine the best value of this hyperparameter with cross-validation.
Can you write Python code for it?

num_folds = 5
k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]

X_train_folds = []
y_train_folds = []
################################################################################
# TODO:                                                                        #
# Split up the training data into folds. After splitting, X_train_folds and    #
# y_train_folds should each be lists of length num_folds, where                #
# y_train_folds[i] is the label vector for the points in X_train_folds[i].     #
# Hint: Look up the numpy array_split function.                                #
################################################################################

X_train_folds = np.array_split(X_train, num_folds)
y_train_folds = np.array_split(y_train, num_folds)

################################################################################
#                                 END OF YOUR CODE                             #
################################################################################

# A dictionary holding the accuracies for different values of k that we find
# when running cross-validation. After running cross-validation,
# k_to_accuracies[k] should be a list of length num_folds giving the different
# accuracy values that we found when using that value of k.
k_to_accuracies = {}

################################################################################
# TODO:                                                                        #
# Perform k-fold cross validation to find the best value of k. For each        #
# possible value of k, run the k-nearest-neighbor algorithm num_folds times,   #
# where in each case you use all but one of the folds as training data and the #
# last fold as a validation set. Store the accuracies for all fold and all     #
# values of k in the k_to_accuracies dictionary.                               #
################################################################################

for k in k_choices:
    for test_fold_index in...


I am working on the file "CS231n/assignment1/knn.ipynb".
The first task is:
Inline Question #1: Notice the structured patterns in the distance matrix, where some rows or columns are visible brighter. (Note that with the default color scheme black indicates low distances while white indicates high distances.)

What in the data is the cause behind the distinctly bright rows?
What causes the columns?

Your Answer: 1) There are no samples in the train data that close to test sample.
2) There are no samples in the test data that close to train sample.
Can you write Python code for it?

# Now implement the function predict_labels and run the code below:
# We use k = 1 (which is Nearest Neighbor).
y_test_pred = classifier.predict_labels(dists, k=1)

# Compute and print the fraction of correctly predicted examples
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))


I am working on the file "CS231n/assignment1/knn.ipynb".
The first task is:
Inline Question #1: Notice the structured patterns in the distance matrix, where some rows or columns are visible brighter. (Note that with the default color scheme black indicates low distances while white indicates high distances.)

What in the data is the cause behind the distinctly bright rows?
What causes the columns?

Your Answer: fill this in.
Can you write Python code for it?


I am working on the file "CS231n/assignment1/knn.ipynb".
The first task is:
We would now like to classify the test data with the kNN classifier. Recall that we can break down this process into two steps:

First we must compute the distances between all test examples and all train examples.
Given these distances, for each test example we find the k nearest examples and have them vote for the label

Lets begin with computing the distance matrix between all training and test examples. For example, if there are Ntr training examples and Nte test examples, this stage should result in a Nte x Ntr matrix where each element (i,j) is the distance between the i-th test and j-th train example.
First, open cs231n/classifiers/k_nearest_neighbor.py and implement the function compute_distances_two_loops that uses a (very inefficient) double loop over all pairs of (test, train) examples and computes the distance matrix one element at a time.
Can you write Python code for it?

# Open cs231n/classifiers/k_nearest_neighbor.py and implement
# compute_distances_two_loops.

# Test your implementation:
dists = classifier.compute_distances_two_loops(X_test)
print(dists.shape)

# We can visualize the distance matrix: each row is a single test example and
# its distances to training examples
plt.imshow(dists, interpolation='none')
plt.show()


I am working on the file "CS231n/assignment1/knn.ipynb".
The first task is:
You should expect to see a slightly better performance than with k = 1.
Can you write Python code for it?

# Now lets speed up distance matrix computation by using partial vectorization
# with one loop. Implement the function compute_distances_one_loop and run the
# code below:
dists_one = classifier.compute_distances_one_loop(X_test)

# To ensure that our vectorized implementation is correct, we make sure that it
# agrees with the naive implementation. There are many ways to decide whether
# two matrices are similar; one of the simplest is the Frobenius norm. In case
# you haven't seen it before, the Frobenius norm of two matrices is the square
# root of the squared sum of differences of all elements; in other words, reshape
# the matrices into vectors and compute the Euclidean distance between them.
difference = np.linalg.norm(dists - dists_one, ord='fro')
print('Difference was: %f' % (difference, ))
if difference < 0.001:
    print('Good! The distance matrices are the same')
else:
    print('Uh-oh! The distance matrices are different')

# Now implement the fully vectorized version inside compute_distances_no_loops
# and run the code
dists_two = classifier.compute_distances_no_loops(X_test)

# check that the distance matrix agrees with the one we computed before:
difference = np.linalg.norm(dists - dists_two, ord='fro')
print('Difference was: %f' % (difference, ))
if difference < 0.001:
    print('Good! The distance matrices are the same')
else:
    print('Uh-oh! The distance matrices are different')

# Let's compare how fast the implementations are
def time_function(f, *args):
    """
    Call a function f with args and return the time (in seconds) that it took to execute.
    """
    import time
    tic = time.time()
    f(*args)
    toc = time.time()
    return toc - tic

two_loop_time = time_function(classifier.compute_distances_two_loops, X_test)
print('Two loop version took %f seconds' % two_loop_time)

one_loop_time = time_function(classifier.compute_distances_one_loop, X_test)
print('One loop version took %f seconds' % one_loop_time)

no_loop_time = time_function(classifier.compute_distances_no_loops,...


I am working on the file "CS231n/assignment1/knn.ipynb".
The first task is:
You should expect to see approximately 27% accuracy. Now lets try out a larger k, say k = 5:
Can you write Python code for it?

y_test_pred = classifier.predict_labels(dists, k=5)
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print('Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy))


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
Backward pass
Implement the rest of the function. This will compute the gradient of the loss with respect to the variables W1, b1, W2, and b2. Now that you (hopefully!) have a correctly implemented forward pass, you can debug your backward pass using a numeric gradient check:
Can you write Python code for it?

from cs231n.gradient_check import eval_numerical_gradient

# Use numeric gradient checking to check your implementation of the backward pass.
# If your implementation is correct, the difference between the numeric and
# analytic gradients should be less than 1e-8 for each of W1, W2, b1, and b2.

loss, grads = net.loss(X, y, reg=0.1)

# these should all be less than 1e-8 or so
for param_name in grads:
  f = lambda W: net.loss(X, y, reg=0.1)[0]
  param_grad_num = eval_numerical_gradient(f, net.params[param_name], verbose=True)
  print '%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name]))


from cs231n.gradient_check import eval_numerical_gradient

# Use numeric gradient checking to check your implementation of the backward pass.
# If your implementation is correct, the difference between the numeric and
# analytic gradients should be less than 1e-8 for each of W1, W2, b1, and b2.

loss, grads = net.loss(X, y, reg=0.05)

# these should all be less than 1e-8 or so
for param_name in grads:
    f = lambda W: net.loss(X, y, reg=0.05)[0]
    param_grad_num = eval_numerical_gradient(f, net.params[param_name], verbose=False)
    print('%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name])))


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
Forward pass: compute loss
In the same function, implement the second part that computes the data and regularizaion loss.
Can you write Python code for it?

loss, _ = net.loss(X, y, reg=0.1)
correct_loss = 1.30378789133

# should be very small, we get < 1e-12
print 'Difference between your loss and correct loss:'
print np.sum(np.abs(loss - correct_loss))


loss, _ = net.loss(X, y, reg=0.05)
correct_loss = 1.30378789133

# should be very small, we get < 1e-12
print('Difference between your loss and correct loss:')
print(np.sum(np.abs(loss - correct_loss)))


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
Forward pass: compute scores
Open the file cs231n/classifiers/neural_net.py and look at the method TwoLayerNet.loss. This function is very similar to the loss functions you have written for the SVM and Softmax exercises: It takes the data and weights and computes the class scores, the loss, and the gradients on the parameters.
Implement the first part of the forward pass which uses the weights and biases to compute the scores for all inputs.
Can you write Python code for it?


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
Load the data
Now that you have implemented a two-layer network that passes gradient checks and works on toy data, it's time to load up our favorite CIFAR-10 data so we can use it to train a classifier on a real dataset.
Can you write Python code for it?

from cs231n.data_utils import load_CIFAR10

def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):
    """
    Load the CIFAR-10 dataset from disk and perform preprocessing to prepare
    it for the two-layer neural net classifier. These are the same steps as
    we used for the SVM, but condensed to a single function.
    """
    # Load the raw CIFAR-10 data
    cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

    # Subsample the data
    mask = list(range(num_training, num_training + num_validation))
    X_val = X_train[mask]
    y_val = y_train[mask]
    mask = list(range(num_training))
    X_train = X_train[mask]
    y_train = y_train[mask]
    mask = list(range(num_test))
    X_test = X_test[mask]
    y_test = y_test[mask]

    # Normalize the data: subtract the mean image
    mean_image = np.mean(X_train, axis=0)
    X_train -= mean_image
    X_val -= mean_image
    X_test -= mean_image

    # Reshape data to rows
    X_train = X_train.reshape(num_training, -1)
    X_val = X_val.reshape(num_validation, -1)
    X_test = X_test.reshape(num_test, -1)

    return X_train, y_train, X_val, y_val, X_test, y_test

# Invoke the above function to get our data.
X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()
print('Train data shape: ', X_train.shape)
print('Train labels shape: ', y_train.shape)
print('Validation data shape: ', X_val.shape)
print('Validation labels shape: ', y_val.shape)
print('Test data shape: ', X_test.shape)
print('Test labels shape: ', y_test.shape)


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
Run on the test set
When you are done experimenting, you should evaluate your final trained network on the test set; you should get above 48%.
We will give you extra bonus point for every 1% of accuracy above 52%.
Can you write Python code for it?

test_acc = (best_net.predict(X_test) == y_test).mean()
print 'Test accuracy: ', test_acc


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
Train a network
To train our network we will use SGD with momentum. In addition, we will adjust the learning rate with an exponential learning rate schedule as optimization proceeds; after each epoch, we will reduce the learning rate by multiplying it by a decay rate.
Can you write Python code for it?

input_size = 32 * 32 * 3
hidden_size = 50
num_classes = 10
net = TwoLayerNet(input_size, hidden_size, num_classes)

# Train the network
stats = net.train(X_train, y_train, X_val, y_val,
            num_iters=1000, batch_size=100,
            learning_rate=1e-4, learning_rate_decay=0.95,
            reg=0.25, verbose=True)

# Predict on the validation set
val_acc = (net.predict(X_val) == y_val).mean()
print('Validation accuracy: ', val_acc)


input_size = 32 * 32 * 3
hidden_size = 50
num_classes = 10
net = TwoLayerNet(input_size, hidden_size, num_classes)

# Train the network
stats = net.train(X_train, y_train, X_val, y_val,
            num_iters=1000, batch_size=200,
            learning_rate=1e-4, learning_rate_decay=0.95,
            reg=0.5, verbose=True)

# Predict on the validation set
val_acc = (net.predict(X_val) == y_val).mean()
print 'Validation accuracy: ', val_acc


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
Train the network
To train the network we will use stochastic gradient descent (SGD), similar to the SVM and Softmax classifiers. Look at the function TwoLayerNet.train and fill in the missing sections to implement the training procedure. This should be very similar to the training procedure you used for the SVM and Softmax classifiers. You will also have to implement TwoLayerNet.predict, as the training process periodically performs prediction to keep track of accuracy over time while the network trains.
Once you have implemented the method, run the code below to train a two-layer network on toy data. You should achieve a training loss less than 0.2.
Can you write Python code for it?

net = init_toy_model()
stats = net.train(X, y, X, y,
            learning_rate=1e-1, reg=1e-5,
            num_iters=300, verbose=True)

print 'Final training loss: ', stats['loss_history'][-1]

# plot the loss history
#plt.plot(stats['loss_history'])
#plt.xlabel('iteration')
#plt.ylabel('training loss')
#plt.title('Training Loss history')
#plt.show()


net = init_toy_model()
stats = net.train(X, y, X, y,
            learning_rate=1e-1, reg=5e-6,
            num_iters=100, verbose=False)

print('Final training loss: ', stats['loss_history'][-1])

# plot the loss history
plt.plot(stats['loss_history'])
plt.xlabel('iteration')
plt.ylabel('training loss')
plt.title('Training Loss history')
plt.show()


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
Tune your hyperparameters
What's wrong?. Looking at the visualizations above, we see that the loss is decreasing more or less linearly, which seems to suggest that the learning rate may be too low. Moreover, there is no gap between the training and validation accuracy, suggesting that the model we used has low capacity, and that we should increase its size. On the other hand, with a very large model we would expect to see more overfitting, which would manifest itself as a very large gap between the training and validation accuracy.
Tuning. Tuning the hyperparameters and developing intuition for how they affect the final performance is a large part of using Neural Networks, so we want you to get a lot of practice. Below, you should experiment with different values of the various hyperparameters, including hidden layer size, learning rate, numer of training epochs, and regularization strength. You might also consider tuning the learning rate decay, but you should be able to get good performance using the default value.
Approximate results. You should be aim to achieve a classification accuracy of greater than 48% on the validation set. Our best network gets over 52% on the validation set.
Experiment: You goal in this exercise is to get as good of a result on CIFAR-10 as you can, with a fully-connected Neural Network. For every 1% above 52% on the Test set we will award you with one extra bonus point. Feel free implement your own techniques (e.g. PCA to reduce dimensionality, or adding dropout, or adding features to the solver, etc.).
Can you write Python code for it?

best_net = None # store the best model into this

#################################################################################
# TODO: Tune hyperparameters using the validation set. Store your best trained  #
# model in best_net.                                                            #
#                                                                               #
# To help debug your network, it may help to use visualizations similar to the  #
# ones we used above; these visualizations will have significant qualitative    #
# differences from the ones we saw above for the poorly tuned network.          #
#                                                                               #
# Tweaking hyperparameters by hand can be fun, but you might find it useful to  #
# write code to sweep through possible combinations of hyperparameters          #
# automatically like we did on the previous exercises.                          #
#################################################################################
pass
#################################################################################
#                               END OF YOUR CODE                                #
#################################################################################

# visualize the weights of the best network
show_net_weights(best_net)


I am working on the file "CS231n/assignment1/two_layer_net.ipynb".
The first task is:
We will use the class TwoLayerNet in the file cs231n/classifiers/neural_net.py to represent instances of our network. The network parameters are stored in the instance variable self.params where keys are string parameter names and values are numpy arrays. Below, we initialize toy data and a toy model that we will use to develop your implementation.
Can you write Python code for it?

# Create a small net and some toy data to check your implementations.
# Note that we set the random seed for repeatable experiments.
import numpy as np
from cs231n.classifiers.neural_net import TwoLayerNet

input_size = 4
hidden_size = 10
num_classes = 3
num_inputs = 5

def init_toy_model():
  np.random.seed(0)
  return TwoLayerNet(input_size, hidden_size, num_classes, std=1e-1)

def init_toy_data():
  np.random.seed(1)
  X = 10 * np.random.randn(num_inputs, input_size)
  y = np.array([0, 1, 2, 2, 1])
  print X
  print y
  return X, y

net = init_toy_model()
X, y = init_toy_data()


# Create a small net and some toy data to check your implementations.
# Note that we set the random seed for repeatable experiments.

input_size = 4
hidden_size = 10
num_classes = 3
num_inputs = 5

def init_toy_model():
    np.random.seed(0)
    return TwoLayerNet(input_size, hidden_size, num_classes, std=1e-1)

def init_toy_data():
    np.random.seed(1)
    X = 10 * np.random.randn(num_inputs, input_size)
    y = np.array([0, 1, 2, 2, 1])
    return X, y

net = init_toy_model()
X, y = init_toy_data()