I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Checkpoint: Let's put these steps together.  First, initialize three centroids with the first 3 rows of tf_idf. Then, compute distances from each of the centroids to all data points in tf_idf. Finally, use these distance calculations to compute cluster assignments and assign them to cluster_assignment.
Can you write Python code for it?

# Students should write code here
centroids = tf_idf[0:3]
distances = pairwise_distances(tf_idf, centroids, metric='euclidean')
cluster_assignment = np.argmin(distances, axis=1)

if len(cluster_assignment)==59071 and \
   np.array_equal(np.bincount(cluster_assignment), np.array([23061, 10086, 25924])):
    print('Pass') # count number of data points for each cluster
else:
    print('Check your code again.')


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Checkpoint: Next, given the pairwise distances, we take the minimum of the distances for each data point. Fittingly, NumPy provides an argmin function. See this documentation for details.
Read the documentation and write code to produce a 1D array whose i-th entry indicates the centroid that is the closest to the i-th data point. Use the list of distances from the previous checkpoint and save them as distances. The value 0 indicates closeness to the first centroid, 1 indicates closeness to the second centroid, and so forth. Save this array as closest_cluster.
Hint: the resulting array should be as long as the number of data points.
Can you write Python code for it?

# Students should write code here

closest_cluster = np.argmin(distances, axis=1)

'''Test cell'''
reference = [list(row).index(min(row)) for row in distances]
if np.allclose(closest_cluster, reference):
    print('Pass')
else:
    print('Check your code again')


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Combining into a single function
Once the two k-means steps have been implemented, as well as our heterogeneity metric we wish to monitor, it is only a matter of putting these functions together to write a k-means algorithm that

Repeatedly performs Steps 1 and 2
Tracks convergence metrics
Stops if either no assignment changed or we reach a certain number of iterations.
Can you write Python code for it?

# Fill in the blanks
def kmeans(data, k, initial_centroids, maxiter, record_heterogeneity=None, verbose=False):
    '''This function runs k-means on given data and initial set of centroids.
       maxiter: maximum number of iterations to run.
       record_heterogeneity: (optional) a list, to store the history of heterogeneity as function of iterations
                             if None, do not store the history.
       verbose: if True, print how many data points changed their cluster labels in each iteration'''
    centroids = initial_centroids[:]
    prev_cluster_assignment = None

    for itr in xrange(maxiter):
        if verbose:
            print(itr)

        # 1. Make cluster assignments using nearest centroids
        # YOUR CODE HERE
        cluster_assignment = assign_clusters(data, centroids)

        # 2. Compute a new centroid for each of the k clusters, averaging all data points assigned to that cluster.
        # YOUR CODE HERE
        centroids = revise_centroids(data, k, cluster_assignment)

        # Check for convergence: if none of the assignments changed, stop
        if prev_cluster_assignment is not None and \
          (prev_cluster_assignment==cluster_assignment).all():
            break

        # Print number of new assignments
        if prev_cluster_assignment is not None:
            num_changed = np.sum(prev_cluster_assignment!=cluster_assignment)
            if verbose:
                print('    {0:5d} elements changed their cluster assignment.'.format(num_changed))

        # Record heterogeneity convergence metric
        if record_heterogeneity is not None:
            # YOUR CODE HERE
            score = compute_heterogeneity(data, k, centroids, cluster_assignment)
            record_heterogeneity.append(score)

        prev_cluster_assignment = cluster_assignment[:]

    return centroids, cluster_assignment


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
For the remainder of the assignment, we will use sparse matrices. Sparse matrices are matrices that have a small number of nonzero entries. A good data structure for sparse matrices would only store the nonzero entries to save space and speed up computation. SciPy provides a highly-optimized library for sparse matrices. Many matrix operations available for NumPy arrays are also available for SciPy sparse matrices.
We first convert the TF-IDF column (in dictionary format) into the SciPy sparse matrix format. We included plenty of comments for the curious; if you'd like, you may skip the next block and treat the function as a black box.
Can you write Python code for it?

def sframe_to_scipy(x, column_name):
    '''
    Convert a dictionary column of an SFrame into a sparse matrix format where
    each (row_id, column_id, value) triple corresponds to the value of
    x[row_id][column_id], where column_id is a key in the dictionary.

    Example
    >>> sparse_matrix, map_key_to_index = sframe_to_scipy(sframe, column_name)
    '''
    assert x[column_name].dtype() == dict, \
        'The chosen column must be dict type, representing sparse data.'

    # Create triples of (row_id, feature_id, count).
    # 1. Add a row number.
    x = x.add_row_number()
    # 2. Stack will transform x to have a row for each unique (row, key) pair.
    x = x.stack(column_name, ['feature', 'value'])

    # Map words into integers using a OneHotEncoder feature transformation.
    f = graphlab.feature_engineering.OneHotEncoder(features=['feature'])
    # 1. Fit the transformer using the above data.
    f.fit(x)
    # 2. The transform takes 'feature' column and adds a new column 'feature_encoding'.
    x = f.transform(x)
    # 3. Get the feature mapping.
    mapping = f['feature_encoding']
    # 4. Get the feature id to use for each key.
    x['feature_id'] = x['encoded_features'].dict_keys().apply(lambda x: x[0])

    # Create numpy arrays that contain the data for the sparse matrix.
    i = np.array(x['id'])
    j = np.array(x['feature_id'])
    v = np.array(x['value'])
    width = x['id'].max() + 1
    height = x['feature_id'].max() + 1

    # Create a sparse matrix.
    mat = csr_matrix((v, (i, j)), shape=(width, height))

    return mat, mapping

# The conversion will take about a minute or two.
tf_idf, map_index_to_word = sframe_to_scipy(wiki, 'tf_idf')

tf_idf


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
How to choose K
Since we are measuring the tightness of the clusters, a higher value of K reduces the possible heterogeneity metric by definition.  For example, if we have N data points and set K=N clusters, then we could have 0 cluster heterogeneity by setting the N centroids equal to the values of the N data points. (Note: Not all runs for larger K will result in lower heterogeneity than a single run with smaller K due to local optima.)  Let's see explore this general trend for ourselves by performing the following analysis.
Use the kmeans_multiple_runs function to run k-means with five different values of K.  For each K, use k-means++ and multiple runs to pick the best solution.  In what follows, we consider K=2,10,25,50,100 and 7 restarts for each setting.
IMPORTANT: The code block below will take about one hour to finish. We highly suggest that you use the arrays that we have computed for you.
Side note: In practice, a good implementation of k-means would utilize parallelism to run multiple runs of k-means at once. For an example, see scikit-learn's KMeans.
Can you write Python code for it?

#def plot_k_vs_heterogeneity(k_values, heterogeneity_values):
#    plt.figure(figsize=(7,4))
#    plt.plot(k_values, heterogeneity_values, linewidth=4)
#    plt.xlabel('K')
#    plt.ylabel('Heterogeneity')
#    plt.title('K vs. Heterogeneity')
#    plt.rcParams.update({'font.size': 16})
#    plt.tight_layout()

#start = time.time()
#centroids = {}
#cluster_assignment = {}
#heterogeneity_values = []
#k_list = [2, 10, 25, 50, 100]
#seed_list = [0, 20000, 40000, 60000, 80000, 100000, 120000]

#for k in k_list:
#    heterogeneity = []
#    centroids[k], cluster_assignment[k] = kmeans_multiple_runs(tf_idf, k, maxiter=400,
#                                                               num_runs=len(seed_list),
#                                                               seed_list=seed_list,
#                                                               verbose=True)
#    score = compute_heterogeneity(tf_idf, k, centroids[k], cluster_assignment[k])
#    heterogeneity_values.append(score)

#plot_k_vs_heterogeneity(k_list, heterogeneity_values)

#end = time.time()
#print(end-start)


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Implement k-means
Let us implement the k-means algorithm. First, we choose an initial set of centroids. A common practice is to choose randomly from the data points.
Note: We specify a seed here, so that everyone gets the same answer. In practice, we highly recommend to use different seeds every time (for instance, by using the current timestamp).
Can you write Python code for it?

def get_initial_centroids(data, k, seed=None):
    '''Randomly choose k data points as initial centroids'''
    if seed is not None: # useful for obtaining consistent results
        np.random.seed(seed)
    n = data.shape[0] # number of data points

    # Pick K indices from range [0, N).
    rand_indices = np.random.randint(0, n, k)

    # Keep centroids as dense format, as many entries will be nonzero due to averaging.
    # As long as at least one document in a cluster contains a word,
    # it will carry a nonzero weight in the TF-IDF vector of the centroid.
    centroids = data[rand_indices,:].toarray()

    return centroids


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
In lieu of indices, we can put in the list of Booleans to pick and choose rows. Only the rows that correspond to a True entry will be retained.
First, let's look at the data points (i.e., their values) assigned to cluster 1:
Can you write Python code for it?


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
In the above plot we show that heterogeneity goes down as we increase the number of clusters. Does this mean we should always favor a higher K? Not at all! As we will see in the following section, setting K too high may end up separating data points that are actually pretty alike. At the extreme, we can set individual data points to be their own clusters (K=N) and achieve zero heterogeneity, but separating each data point into its own cluster is hardly a desirable outcome. In the following section, we will learn how to detect a K set "too large".
Visualize clusters of documents
Let's start visualizing some clustering results to see if we think the clustering makes sense.  We can use such visualizations to help us assess whether we have set K too large or too small for a given application.  Following the theme of this course, we will judge whether the clustering makes sense in the context of document analysis.
What are we looking for in a good clustering of documents?
* Documents in the same cluster should be similar.
* Documents from different clusters should be less similar.
So a bad clustering exhibits either of two symptoms:
* Documents in a cluster have mixed content.
* Documents with similar content are divided up and put into different clusters.
To help visualize the clustering, we do the following:
* Fetch nearest neighbors of each centroid from the set of documents assigned to that cluster. We will consider these documents as being representative of the cluster.
* Print titles and first sentences of those nearest neighbors.
* Print top 5 words that have highest tf-idf weights in each centroid.
Can you write Python code for it?

def visualize_document_clusters(wiki, tf_idf, centroids, cluster_assignment, k, map_index_to_word, display_content=True):
    '''wiki: original dataframe
       tf_idf: data matrix, sparse matrix format
       map_index_to_word: SFrame specifying the mapping betweeen words and column indices
       display_content: if True, display 8 nearest neighbors of each centroid'''

    print('==========================================================')

    # Visualize each cluster c
    for c in xrange(k):
        # Cluster heading
        print('Cluster {0:d}    '.format(c)),
        # Print top 5 words with largest TF-IDF weights in the cluster
        idx = centroids[c].argsort()[::-1]
        for i in xrange(5): # Print each word along with the TF-IDF weight
            print('{0:s}:{1:.3f}'.format(map_index_to_word['category'][idx[i]], centroids[c,idx[i]])),
        print('')

        if display_content:
            # Compute distances from the centroid to all data points in the cluster,
            # and compute nearest neighbors of the centroids within the cluster.
            distances = pairwise_distances(tf_idf, centroids[c].reshape(1, -1), metric='euclidean').flatten()
            distances[cluster_assignment!=c] = float('inf') # remove non-members from consideration
            nearest_neighbors = distances.argsort()
            # For 8 nearest neighbors, print the title as well as first 180 characters of text.
            # Wrap the text at 80-character mark.
            for i in xrange(8):
                text = ' '.join(wiki[nearest_neighbors[i]]['text'].split(None, 25)[0:25])
                print('\n* {0:50s} {1:.5f}\n  {2:s}\n  {3:s}'.format(wiki[nearest_neighbors[i]]['name'],
                    distances[nearest_neighbors[i]], text[:90], text[90:180] if len(text) > 90 else ''))
        print('==========================================================')


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Let's assign these data points to the closest centroid.
Can you write Python code for it?

cluster_assignment = assign_clusters(data, centroids)
print cluster_assignment


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Let's compare the set of cluster heterogeneities we got from our 7 restarts of k-means using random initialization compared to the 7 restarts of k-means using k-means++ as a smart initialization.
The following code produces a box plot for each of these methods, indicating the spread of values produced by each method.
Can you write Python code for it?

plt.figure(figsize=(8,5))
plt.boxplot([heterogeneity.values(), heterogeneity_smart.values()], vert=False)
plt.yticks([1, 2], ['k-means', 'k-means++'])
plt.rcParams.update({'font.size': 16})
plt.tight_layout()


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Let's compute the cluster heterogeneity for the 2-cluster example we've been considering based on our current cluster assignments and centroids.
Can you write Python code for it?

compute_heterogeneity(data, 2, centroids, cluster_assignment)


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Likewise for cluster 0:
Can you write Python code for it?


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
More formally, dist[i,j] is assigned the distance between the ith row of X (i.e., X[i,:]) and the jth row of Y (i.e., Y[j,:]).
Checkpoint: For a moment, suppose that we initialize three centroids with the first 3 rows of tf_idf. Write code to compute distances from each of the centroids to all data points in tf_idf. Then find the distance between row 430 of tf_idf and the second centroid and save it to dist.
Can you write Python code for it?

# Students should write code here
centroids = tf_idf[0:2]
distances = pairwise_distances(tf_idf, centroids, metric='euclidean')
dist = distances[430, 1] # the 430 distance, 1 spot aka 2nd centroid
print dist
print pairwise_distances(tf_idf[430,:], tf_idf[1,:])

'''Test cell'''
if np.allclose(dist, pairwise_distances(tf_idf[430,:], tf_idf[1,:])):
    print('Pass')
else:
    print('Check your code again')


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Plotting convergence metric
We can use the above function to plot the convergence metric across iterations.
Can you write Python code for it?

def plot_heterogeneity(heterogeneity, k):
    plt.figure(figsize=(7,4))
    plt.plot(heterogeneity, linewidth=4)
    plt.xlabel('# Iterations')
    plt.ylabel('Heterogeneity')
    plt.title('Heterogeneity of clustering over time, K={0:d}'.format(k))
    plt.rcParams.update({'font.size': 16})
    plt.tight_layout()


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
Quiz Question. Which of the 10 clusters above contains the greatest number of articles?

Cluster 0: artists, poets, writers, environmentalists
Cluster 4: track and field athletes
Cluster 5: composers, songwriters, singers, music producers
Cluster 7: baseball players
Cluster 9: lawyers, judges, legal scholars

Quiz Question. Which of the 10 clusters contains the least number of articles?

Cluster 1: film directors
Cluster 3: politicians
Cluster 6: soccer (football) players
Cluster 7: baseball players
Cluster 9: lawyers, judges, legal scholars

There appears to be at least some connection between the topical consistency of a cluster and the number of its member data points.
Let us visualize the case for K=25. For the sake of brevity, we do not print the content of documents. It turns out that the top words with highest TF-IDF weights in each cluster are representative of the cluster.
Can you write Python code for it?

visualize_document_clusters(wiki, tf_idf, centroids[25](), cluster_assignment[25](), 25,
                            map_index_to_word, display_content=False) # turn off text for brevity


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
This makes sense since [0 0 0] is closer to [0 -0.5 0] than to [0.5 0.5 0].
Now let's look at the data points assigned to cluster 0:
Can you write Python code for it?


I am working on the file "2_kmeans-with-text-data_blank.ipynb".
The first task is:
To use the pre-computed NumPy arrays, first download kmeans-arrays.npz as mentioned in the reading for this assignment and load them with the following code.  Make sure the downloaded file is in the same directory as this notebook.
Can you write Python code for it?

def plot_k_vs_heterogeneity(k_values, heterogeneity_values):
    plt.figure(figsize=(7,4))
    plt.plot(k_values, heterogeneity_values, linewidth=4)
    plt.xlabel('K')
    plt.ylabel('Heterogeneity')
    plt.title('K vs. Heterogeneity')
    plt.rcParams.update({'font.size': 16})
    plt.tight_layout()

filename = 'kmeans-arrays.npz'

heterogeneity_values = []
k_list = [2, 10, 25, 50, 100]

if os.path.exists(filename):
    arrays = np.load(filename)
    centroids = {}
    cluster_assignment = {}
    for k in k_list:
        print k
        sys.stdout.flush()
        '''To save memory space, do not load the arrays from the file right away. We use
           a technique known as lazy evaluation, where some expressions are not evaluated
           until later. Any expression appearing inside a lambda function doesn't get
           evaluated until the function is called.
           Lazy evaluation is extremely important in memory-constrained setting, such as
           an Amazon EC2 t2.micro instance.'''
        centroids[k] = lambda k=k: arrays['centroids_{0:d}'.format(k)]
        cluster_assignment[k] = lambda k=k: arrays['cluster_assignment_{0:d}'.format(k)]
        score = compute_heterogeneity(tf_idf, k, centroids[k](), cluster_assignment[k]())
        heterogeneity_values.append(score)

    plot_k_vs_heterogeneity(k_list, heterogeneity_values)

else:
    print('File not found. Skipping.')


I am working on the file "2_numerical_solver/2_numerical_solver.ipynb".
The first task is:
$\theta$自身とその一階微分である$\dot{\theta}$を縦に並べた列ベクトルをつくり，これを$s$とする．（GiuHub上で行列が上手く表示されないのは仕方のないことなのかな？）
\begin{align}
  s =
  \begin{bmatrix}
    \theta \ \dot{\theta}
  \end{bmatrix}
\end{align}
常微分方程式$\dot{s} = f(s)$を解くために，初期値 $s_\mathrm{init}$ を与える．
Can you write Python code for it?

''' initial value '''
theta_init = 0 # initial value of theta [rad]
dtheta_init = 1 # initial value of dot theta [rad/s]
s_init = np.r_[theta_init, dtheta_init]
np.r_[s_init, s_init]


I am working on the file "2_numerical_solver/2_numerical_solver.ipynb".
The first task is:
math.sin と numpy.sin について
numpy ライブラリの sin 関数を使わず， math ライブラリから sin 関数をインポートしたのは，計算を高速化させるためである．mathライブラリで定義される正弦関数とnumpyライブラリで定義される正弦関数には違いがある．次の例で示すように，スカラー値の正弦の計算は numpy.sin よりもmath.sin の方が早い．
Can you write Python code for it?

import time
import math
import numpy

''' Sine function of math library '''
start = time.time()
for i in range(1000000):
    math.sin(i)
elapsed_time = time.time() - start
print ('Sine calculation by math: {0:f} [sec]'.format(elapsed_time))

''' Sine function of numpy library '''
start = time.time()
for i in range(1000000):
    numpy.sin(i)
elapsed_time = time.time() - start
print ('Sine calculation by numpy: {0:f} [sec]'.format(elapsed_time))


I am working on the file "2_numerical_solver/2_numerical_solver.ipynb".
The first task is:
そして，$s$の時間の一階微分を式で表す．ここで，$\ddot{\theta}$は先ほど求めた運動方程式によって，$\ddot{\theta}$を含まない形で表すことができる．
\begin{align}
  \frac{d}{dt} s =
  \begin{bmatrix}
    \dot{\theta} \ \ddot{\theta}
  \end{bmatrix} =
  \begin{bmatrix}
    \dot{\theta} \ -\frac{g}{l}\sin\theta
  \end{bmatrix}
\end{align}
こうして得られた常微分方程式は，数値計算に適した形となっている．具体的には，$\theta = $ s[0] ，$\dot{\theta} = $ s[1] とすることで，
def odefunc(s, t):
    theta = s[0]
    dtheta = s[1]
    ddtheta = -g/l*sin(theta) # <- This is the equation of motion
    return np.r_[dtheta, ddtheta]
のようにして常微分方程式をPythonの関数として記述したうえで
s = odeint(odefunc, s_init, t)
を実行することで常微分方程式を数値的に解くことができる．ただし odeint は SciPy ライブラリの integrate クラスで定義される，常微分方程式を解く関数である．
Can you write Python code for it?

def odefunc(s, t):
    theta = s[0]
    dtheta = s[1]
    ddtheta = -g/l*sin(theta) # <- Equation of motion
    return np.r_[dtheta, ddtheta]

s = odeint(odefunc, s_init, t)
print('ODE calculation finished.')


I am working on the file "2_tracking.ipynb".
The first task is:
2. Tracking in first and second order approximation without any collective effects
Remark
Because of the reasons mentioned above, we start the beam tracking from the first quadrupole after RF cavities.
Loading of beam distribution
In order to perform tracking we have to have beam distribution. We will load beam distribution from a ASTRA file ('beam_distrib.ast'). And we convert the Astra beam distribution to Ocelot format - ParticleArray. ParticleArray is designed for tracking.
In order to work with converters we have to import specific module from ocelot.adaptors

from ocelot.adaptors.astra2ocelot import *

After importing ocelot.adaptors.astra2ocelot we can use converter astraBeam2particleArray() to load and convert.
As you will see beam distribution consists of 200 000 particles (that is why loading can take a few second), charge 250 pC, initial energy is about 6.5 MeV.
ParticleArray is a class which includes several parameters and methods.
* ParticleArray.particles is a 1D numpy array with coordinates of particles in
$$ParticleArray.particles = [\vec{x_0}, \vec{x_1}, ..., \vec{x_n}], $$ where $$\vec{x_n} = (x_n, x_n', y_n, y_n', \tau_n, p_n)$$
* ParticleArray.s is the longitudinal coordinate of the reference particle in [m].
* ParticleArray.E is the energy of the reference particle in [GeV].
* ParticleArray.q_array - is a 1D numpy array of the charges each (macro) particles in [C]
Can you write Python code for it?

from ocelot.adaptors.astra2ocelot import *

#p_array_init = astraBeam2particleArray(filename='beam_130MeV.ast')
p_array_init = astraBeam2particleArray(filename='beam_130MeV_off_crest.ast')


I am working on the file "2_tracking.ipynb".
The first task is:
Beam distribution
Can you write Python code for it?

tau = np.array([p.tau for p in p_array])
dp = np.array([p.p for p in p_array])
x = np.array([p.x for p in p_array])
y = np.array([p.y for p in p_array])

ax1 = plt.subplot(311)
ax1.plot(-tau*1000, x*1000, 'r.')
plt.setp(ax1.get_xticklabels(), visible=False)
plt.ylabel("x, mm")
plt.grid(True)

ax2 = plt.subplot(312, sharex=ax1)
ax2.plot(-tau*1000, y*1000, 'r.')
plt.setp(ax2.get_xticklabels(), visible=False)
plt.ylabel("y, mm")
plt.grid(True)

ax3 = plt.subplot(313, sharex=ax1)
ax3.plot(-tau*1000, dp, 'r.')
plt.ylabel("dE/E")
plt.xlabel("s, mm")
plt.grid(True)
plt.show()


I am working on the file "2_tracking.ipynb".
The first task is:
Selection of the tracking order and lattice for the tracking.
MagneticLattice(sequence, start=None, stop=None, method=MethodTM()) have wollowing arguments:
* sequence - list of the elements,
* start - first element of the lattice. If None, then lattice starts from the first element of the sequence,
* stop - last element of the lattice. If None, then lattice stops by the last element of the sequence,
* method=MethodTM() - method of the tracking. MethodTM() class assigns transfer map to every element. By default all elements are assigned first order transfer map - TransferMap. One can create one's own map, but there are following predefined maps:
    - TransferMap - first order matrices.
    - SecondTM - 2nd order matrices.
    - KickTM - kick applyed.
    - RungeKuttaTM - Runge-Kutta integrator is applyed, but required 3D magnetic field function element.mag_field = lambda x, y, z: (Bx, By, Bz) (see example ocelot/demos/ebeam/tune_shift.py)
Can you write Python code for it?

# initialization of tracking method
method = MethodTM()

# for second order tracking we have to choose SecondTM
method.global_method = SecondTM

# for first order tracking uncomment next line
# method.global_method = TransferMap

# we will start simulation from the first quadrupole (QI.46.I1) after RF section.
# you can change stop element (and the start element, as well)
# START_73_I1 - marker before Dog leg
# START_96_I1 - marker before Bunch Compresion
lat_t = MagneticLattice(cell, start=QI_46_I1, stop=None, method=method)


I am working on the file "2_tracking.ipynb".
The first task is:
Tracking
for tracking we have to define following objects:
* Navigator defines step (dz) of tracking and which, if it exists, physical process will be applied at each step.
In order to add collective effects (Space charge, CSR or wake) method add_physics_proc() must be run.
- **Method:**
    * Navigator.add_physics_proc(physics_proc, elem1, elem2)
        - physics_proc - physics process, can be CSR, SpaceCharge or Wake,
        - elem1 and elem2 - first and last elements between which the physics process will be applied.

track(MagneticLatice, ParticleArray, Navigator) - the function performs tracking through the lattice [lat] of the particles [p_array]. This function also calculate twiss parameters of the beam distribution on each tracking step.
Can you write Python code for it?

navi = Navigator(lat_t)
p_array = deepcopy(p_array_init)
tws_track, p_array = track(lat_t, p_array, navi)

# you can change top_plot argument, for example top_plot=["alpha_x", "alpha_y"]
plot_opt_func(lat_t, tws_track, top_plot=["E"], fig_name=0, legend=False)
plt.show()


I am working on the file "2_traffic_signs/Traffic_Sign_Classifier.ipynb".
The first task is:
Question 6
Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It could be helpful to plot the images in the notebook.
Answer:
The images were taken from Google Streetview, manually centered cropped in the sign and reduced to 32x32 RGB png color. The rest of the preprocessing is of course the same as for the training/validation/test sets.
The images in general are sharp and not twisted/tilted (were taken when front facing the signs as much as possible)
Can you write Python code for it?

idx = np.random.choice(np.arange(len(imgs)), 5, replace=False)

test_imgs = [imgs[i] for i in idx]
test_labels = [int(labels[i]) for i in idx]

for i in range(len(test_imgs)):
    show_sample(test_imgs, test_labels, i)


I am working on the file "2_traffic_signs/Traffic_Sign_Classifier.ipynb".
The first task is:
Question 7
Is your model able to perform equally well on captured pictures when compared to testing on the dataset? The simplest way to do this check the accuracy of the predictions. For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate.
NOTE: You could check the accuracy manually by using signnames.csv (same directory). This file has a mapping from the class id (0-42) to the corresponding sign name. So, you could take the class id the model outputs, lookup the name in signnames.csv and see if it matches the sign from the image.
Answer:
Based on 22 images taken from the street signs in Google Street View, the accuracy is around 85% which is less than the ~90% from the test set.
For the 5 randomly picked images, it is 80%.
In order to see what to do with this differences, things like brightness, contrast and image distortion should be thought of by adding preprocessing or making the network more robust by adding layers. The last image (Right-of-way in next intersection) is considerably sharper and brighter than the samples. We could consider some image processing techniques like histogram normalization preprocessing for the training set.
Can you write Python code for it?

def test_accuracy(imgs, labels):
    X_new = np.array(imgs)
    y_new = np.array(labels)

    print ("Testing accuracy on {} images...".format(len(imgs)))
    with tf.Session() as sess:
        tf.train.Saver().restore(sess, tf.train.latest_checkpoint('.'))
        accuracy = evaluate(X_new, y_new, len(X_new))
        print("accuracy: {}".format(accuracy))
        print()

test_accuracy(imgs, labels)
test_accuracy(test_imgs, test_labels)


I am working on the file "2_traffic_signs/Traffic_Sign_Classifier.ipynb".
The first task is:
Question 8
Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)
tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.
Take this numpy array as an example:
```
(5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])
```
Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:
TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))
Looking just at the first row we get [ 0.34763842,  0.24879643,  0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.
Answer:
As it can be seen in the last block code below, for image #5, the label prediction is far off ('11' is 5th place)
This means the prediction is really innacurate for this image, and leaves room for further improvements discussed in Answer 7.

Note: Once you have completed all of the code implementations and successfully answered each...
Can you write Python code for it?

sm = tf.nn.softmax(logits)
top_k = tf.nn.top_k(sm, 5)

print ("Testing on {} images...".format(len(test_imgs)))
with tf.Session() as sess:
    tf.train.Saver().restore(sess, tf.train.latest_checkpoint('.'))
    res = sess.run(top_k, feed_dict={x: test_imgs})
    print("accuracy: {}".format(res))

print ("Test labels", test_labels)


I am working on the file "2_traffic_signs/Traffic_Sign_Classifier.ipynb".
The first task is:
Step 1: Dataset Summary & Exploration
The pickled data is a dictionary with 4 key/value pairs:

'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
'labels' is a 2D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
'sizes' is a list containing tuples, (width, height) representing the the original width and height the image.
'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below.
Can you write Python code for it?

def print_summary(feature_set, feature_set_name):
    print("Summary for feature set: {}".format(feature_set_name))
    print("-----------------------")
    print("features shape: {}".format(feature_set['features'].shape))
    print("labels shape: {}".format(feature_set['labels'].shape))
    print("sizes shape: {}".format(feature_set['sizes'].shape))
    print("coord shape: {}".format(feature_set['coords'].shape))
    print()

print_summary(train, 'train')
print_summary(test, 'test')


I am working on the file "2_traffic_signs/Traffic_Sign_Classifier.ipynb".
The first task is:
Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.
The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.
NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.
Can you write Python code for it?

import random
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl

mpl.rcParams.update({'figure.max_open_warning': 0})
%matplotlib inline

def load_label_descriptions(filename):
    ret = {}

    f = open(filename)
    for line in f:
        v = line.rstrip().split(',')
        ret[v[0]] = v[1]

    return ret

def show_sample(X, y, index):
    s = "Sample: {} with Label: {} ({})".format(index, y[index], label_descriptions[str(y[index])])

    image = X[index].squeeze()

    plt.figure(figsize=(1,1))
    plt.text(0, -10, s)
    plt.imshow(image, cmap="gray")

label_descriptions = load_label_descriptions('signnames.csv')
index = random.randint(0, len(X_train))

show_sample(X_train, y_train, index)


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Build the network
For the neural network, you'll build each layer into a function.  Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function.  This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.

Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section.  TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.
However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the conv2d class, tf.layers.conv2d, you would want to use the TF Neural Network version of conv2d, tf.nn.conv2d.

Let's begin!
Input
The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions
* Implement neural_net_image_input
 * Return a TF Placeholder
 * Set the shape using image_shape with batch size set to None.
 * Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_label_input
 * Return a TF Placeholder
 * Set the shape using n_classes with batch size set to None.
 * Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_keep_prob_input
 * Return a TF Placeholder for dropout keep probability.
 * Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in...
Can you write Python code for it?

import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a batch of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    return tf.placeholder(tf.float32, shape=(None, image_shape[0], image_shape[1], image_shape[2]), name="x")

def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    return tf.placeholder(tf.float32, shape=(None, n_classes), name="y")

def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    return tf.placeholder(tf.float32, name="keep_prob")

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Check Point
This is your first checkpoint.  If you ever decide to come back to this notebook or have to restart the notebook, you can start from here.  The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Checkpoint
The model has been saved to disk.
Test Model
Test your model against the test dataset.  This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_test.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')

        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0

        for test_feature_batch, test_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: test_feature_batch, loaded_y: test_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y:...


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Create Convolutional Model
Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits.  Use the layers you created above to create this model:

Apply 1, 2, or 3 Convolution and Max Pool layers
Apply a Flatten Layer
Apply 1, 2, or 3 Fully Connected Layers
Apply an Output Layer
Return the output
Apply TensorFlow's Dropout to one or more layers in the model using keep_prob.
Can you write Python code for it?

def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    # TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
    #    Play around with different number of outputs, kernel size and stride
    # Function Definition from Above:
    #conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    layer = conv2d_maxpool(x, 32, (4, 4), (2, 2), (4, 4), (2, 2))
    #layer = tf.nn.dropout(layer, keep_prob)
    layer = conv2d_maxpool(layer, 64, (3, 3), (2, 2), (3, 3), (2, 2))
    layer = tf.nn.dropout(layer, keep_prob)
    #layer = conv2d_maxpool(layer, 128, (4, 4), (2, 2), (2, 2), (2, 2))
    #layer = tf.nn.dropout(layer, keep_prob)

    # TODO: Apply a Flatten Layer
    # Function Definition from Above:
    layer = flatten(layer)

    # TODO: Apply 1, 2, or 3 Fully Connected Layers
    #    Play around with different number of outputs
    # Function Definition from Above:
    layer = fully_conn(layer, 40)
    layer = fully_conn(layer, 40)

    # TODO: Apply an Output Layer
    #    Set this to the number of classes
    # Function Definition from Above:
    #output(layer, num_outputs)

    # TODO: return output
    return output(layer, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

##############################
## Build the Neural Network ##
##############################

# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()

# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()

# Model
logits = conv_net(x, keep_prob)

# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')

# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer...


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Explore the Data
The dataset is broken into batches to prevent your machine from running out of memory.  The CIFAR-10 dataset consists of 5 batches, named data_batch_1, data_batch_2, etc.. Each batch contains the labels and images that are one of the following:
* airplane
* automobile
* bird
* cat
* deer
* dog
* frog
* horse
* ship
* truck
Understanding a dataset is part of making predictions on the data.  Play around with the code cell below by changing the batch_id and sample_id. The batch_id is the id for a batch (1-5). The sample_id is the id for a image and label pair in the batch.
Ask yourself "What are all possible labels?", "What is the range of values for the image data?", "Are the labels in order or random?".  Answers to questions like these will help you preprocess the data and end up with better predictions.
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import helper
import numpy as np

# Explore the dataset
batch_id = 3
sample_id = 1016
helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Fully Train the Model
Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'

print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            print_stats(sess, batch_features, batch_labels, cost, accuracy)

    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Fully-Connected Layer
Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Can you write Python code for it?

def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    weights = tf.Variable(tf.truncated_normal([int(x_tensor.shape[1]), num_outputs], stddev=0.1))
    bias = tf.Variable(tf.truncated_normal([num_outputs], stddev=0.1))

    return tf.nn.relu(tf.matmul(x_tensor, weights) + bias)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Hyperparameters
Tune the following parameters:
* Set epochs to the number of iterations until the network stops learning or start overfitting
* Set batch_size to the highest number that your machine has memory for.  Most people set them to common sizes of memory:
 * 64
 * 128
 * 256
 * ...
* Set keep_probability to the probability of keeping a node using dropout
Can you write Python code for it?

# TODO: Tune Parameters
epochs = 100
batch_size = 256
keep_probability = 0.8


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Implement Preprocess Functions
Normalize
In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive.  The return object should be the same shape as x.
Can you write Python code for it?

def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    # TODO: Implement Function
    return (x / np.max(x))

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Randomize Data
As you saw from exploring the data above, the order of the samples are randomized.  It doesn't hurt to randomize it again, but you don't need to for this dataset.
Preprocess all the data and save it
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Show Stats
Implement the function print_stats to print loss and validation accuracy.  Use the global variables valid_features and valid_labels to calculate validation accuracy.  Use a keep probability of 1.0 to calculate the loss and validation accuracy.
Can you write Python code for it?

def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    loss = session.run(cost, feed_dict = {x: feature_batch, y: label_batch, keep_prob: 1.0})
    accuracy = session.run(accuracy, feed_dict = {x: valid_features, y: valid_labels, keep_prob: 1.0})

    print("Loss: {}".format(loss))
    print("Validation Accuracy: {}".format(accuracy))


I am working on the file "3 - Convolutional Neural Net/Project/dlnd_image_classification.ipynb".
The first task is:
Train on a Single CIFAR-10 Batch
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy.  Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)


I am working on the file "3 - Convolutional Neural Net/Scripts/Simple_Autoencoder.ipynb".
The first task is:
Here I'll write a bit of code to train the network. I'm not too interested in validation here, so I'll just monitor the training loss.
Calling mnist.train.next_batch(batch_size) will return a tuple of (images, labels). We're not concerned with the labels here, we just need the images. Otherwise this is pretty straightfoward training with TensorFlow. We initialize the variables with sess.run(tf.global_variables_initializer()). Then, run the optimizer and get the loss with batch_cost, _ = sess.run([cost, opt], feed_dict=feed).
Can you write Python code for it?

epochs = 20
batch_size = 200
sess.run(tf.global_variables_initializer())
for e in range(epochs):
    for ii in range(mnist.train.num_examples//batch_size):
        batch = mnist.train.next_batch(batch_size)
        feed = {inputs_: batch[0], targets_: batch[0]}
        batch_cost, _ = sess.run([cost, opt], feed_dict=feed)

        print("Epoch: {}/{}...".format(e+1, epochs),
              "Training loss: {:.4f}".format(batch_cost))


I am working on the file "3 - Example Time Series - Landsat.ipynb".
The first task is:
$$
\begin{bmatrix}
t_1 & 1 \\
t_2 & 1 \\
\vdots & \vdots \\
t_n & 1
\end{bmatrix}
\begin{bmatrix}
x_0 \\
x_1
\end{bmatrix}=
\begin{bmatrix}
y_1 \\
y_2 \\
\vdots \\
y_n
\end{bmatrix}
$$
$$ \mathbf{A}\mathbf{x} = \mathbf{b} $$
$$ \mathbf{x} = \mathbf{A} \backslash \mathbf{b} $$
Can you write Python code for it?

A = array([ t, ones(len(t))]).transpose()
b = ndvi
x = linalg.lstsq(A,b)[0] # obtaining the parameters
x

b_hat = A.dot(x)

fig2 = figure(figsize=(12,4), dpi=80)
plot(time,b_hat.transpose(),'r-',t,b,'o')
fig2.fmt_xdata = mdates.DateFormatter('%Y-%m-%d')
fig2.autofmt_xdate()


I am working on the file "3 - Example Time Series - Landsat.ipynb".
The first task is:
Calculate NDVI
Can you write Python code for it?

band3 = yData[:,2]
band4 = yData[:,3]
ndvi = (band4 - band3) / (band4 + band3)


I am working on the file "3 - Example Time Series - Landsat.ipynb".
The first task is:
Next we contruct a filtered ImageCollection for the date range, and extract band information for a specified point location.
Can you write Python code for it?

collection = ee.ImageCollection('LE7_L1T').filterDate(startTime, endTime)
point = {'type':'Point', 'coordinates':[ -116.88629,36.56122]};  # death valley (should be stable)
info = collection.getRegion(point,500).getInfo()


I am working on the file "3 - Example Time Series - Landsat.ipynb".
The first task is:
Next we define the Landsat bands that we would like to plot, along with the starting and ending times.
Can you write Python code for it?

xBand = 'time'
yBandList = [
        'B1',
        u'B2',
        u'B3',
        u'B4',
        u'B5',
        u'B6_VCID_1',
        u'B6_VCID_2',
        u'B7',
        u'B8',
    ]
startTime = datetime.datetime(2000, 1, 1)
endTime = datetime.datetime(2004, 1, 1)


I am working on the file "3 - Example Time Series - Landsat.ipynb".
The first task is:
Next we extract time information and convert it to at Python datatime data type.
Can you write Python code for it?

# extract the time information
iTime = header.index('time')
# convert to Python datetime objects
time = [datetime.datetime.fromtimestamp(i/1000) for i in (data[0:,iTime].astype(int))]


I am working on the file "3 - Generate coupon probabilities - exponential.ipynb".
The first task is:
Generating Exponential Coupon Probabilities
Generating coupon probabilities can be accomplished in much the same manner as with the normal probabilities.  It is actually a little easier because the exponential distribution is defined by only one parameter ($\lambda$, the rate parameter).  The mean of the exponential distribution is $\frac{1}{\lambda}$, and we know that the mean coupon probability must be $\frac{1}{n}$, therefore we must set $\lambda=n$.  We can generate the probabilities in much the same way as before:
Can you write Python code for it?

n = 20 #number of coupons
scale = 1/n #scipy uses the scale parameter instead of lambda.  Scale and lambda are reciprocals of each other.
x = np.arange(n)+0.5 #arange goes from 0 to n-1, and I want it to go from 1 to n
p_x = stat.expon.ppf(x/n, loc=0, scale=scale)
print('unfilled probability: ', 1-np.sum(p_x))
p_x = p_x/np.sum(p_x)
print('mean prob = %1.16f' % np.mean(p_x), ', sum of all probs = %1.16f' % np.sum(p_x) )
plt.bar(x, p_x, align='center')
plt.axis([0,n,0,4/n])


I am working on the file "3 - Generate coupon probabilities - exponential.ipynb".
The first task is:
Inconvenient occurence
In generating the probabilities for the exponential distribution we had to re-scale the probabilities so they would sum to 1.  This is because of the "discretization" of the distribution leaves a little bit of the probabilities unfilled.  It's not a lot (1.7% missing when n=20), but it needs to be accounted for so we'll re-scale the distribution.  I'm not sure of another way to generate exponential probabilities without having to re-scale afterwards.
Limiting behavior
As the population size increases, we do not approach a uniform distribution, unlike the normal distribution.  The shape remains "exponential," low numbered coupons remain rare compared to high numbered coupons.
Can you write Python code for it?

n = 2000 #number of coupons
scale = 1/n #scipy uses the scale parameter instead of lambda.  Scale and lambda are reciprocals of each other.
x = np.arange(n)+0.5 #arange goes from 0 to n-1, and I want it to go from 1 to n
p_x = stat.expon.ppf(x/n, loc=0, scale=scale)
print('unfilled probability: ', 1-np.sum(p_x))
p_x = p_x/np.sum(p_x)
print('mean prob = %1.16f' % np.mean(p_x), ', sum of all probs = %1.16f' % np.sum(p_x) )
plt.bar(x, p_x, align='center')
plt.axis([0,n,0,4/n])


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
Convert those to sortable month names:
Can you write Python code for it?

# Series.dt gives us access to date-related methods
gig_months = gig_dates.dt.strftime('%m %b')
gig_months


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
DataFrame doesn't know what to do with a QuerySet; it wants something that looks more like a list.
We could use list(gig_values), but gig_values.iterator() is more efficient.
Can you write Python code for it?

gigs = pd.DataFrame(gig_values.iterator())
gigs.info()
gigs


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
How many gigs have we played each year?
The date Index also allows for fast aggregration:
Can you write Python code for it?

# resample('A') creates year-end groups like '2005-12-31'
# to_period() turns that into '2005'
years = gigs.resample('A').size().to_period()
years

years.plot.bar()


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
Introducing DataFrame
From the docs:

DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table. It is generally the most commonly used pandas object.

There are many ways to get a DataFrame, but we'll start with a list of dictionaries.
Can you write Python code for it?

df = pd.DataFrame([
    {'integer': 1, 'float': 1.0, 'string': 'one'},
    {'integer': 2, 'float': 2.0, 'string': 'two'},
    {'integer': 2, 'float': 2.0, 'string': 'two'},
    {'integer': 3, 'float': 3.0, 'string': 'three'},
])

# Print some details about the DataFrame
df.info()
df


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
Series and DataFrame support element-wise operations:
Can you write Python code for it?

df_index['float'] * df_index['integer']

df_index * df_index

number_format = 'Number {}'.format
df_index['integer'].apply(number_format)

df_index.applymap(number_format)


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
The Index plays a key role in slicing the DataFrame:
Can you write Python code for it?

# Slice by label
df_index.loc['two']

# Slice by position
df_index.iloc[-2:]


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
The Jupyter Notebook automatically renders DataFrame as HTML!
Note the first column; this is an Index, and is an essential component of DataFrame. Here, it was auto-generated, but we can also set it:
Can you write Python code for it?

df_index = df.set_index('string')
df_index


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
This is a good place to start, and we've already got the answer to "How many gigs have we played"?
However, there are a few ways we can make this easier to work with:

Shorter column names
Predictable column order
Indexed and sorted by date

For more control, we'll use a list of tuples to initialize the DataFrame.
Can you write Python code for it?

gig_values = Gig.objects.past().published().values_list('date', 'venue__name', 'venue__city')
gig_values[:5]

gigs = pd.DataFrame(gig_values.iterator(), columns=['date', 'venue', 'city'])

gigs['date'] = pd.to_datetime(gigs['date'])
gigs = gigs.set_index('date').sort_index()

gigs.info()
gigs.head()


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
Using DataFrame with Django
Django gives us a handy way to build a list of dictionaries:
Can you write Python code for it?

gig_values = Gig.objects.past().published().values('date', 'venue__name', 'venue__city')
gig_values[:5]


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
What cities have we played?
Can you write Python code for it?

gig_cities = gigs['city']
gig_cities

cities = gig_cities.value_counts()
cities

cities.describe()

# Adding the ; suppresses `<matplotlib.axes._subplots.AxesSubplot ...>`
cities[:10].sort_values().plot.barh();


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
What states have we played?
The Gig model doesn't have a state field, so we need to parse it out. In vanilla Python, we'd do:
Can you write Python code for it?


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
What venues have we played?
DataFrame has powerful grouping and aggregration functionality:
Can you write Python code for it?

venues = gigs.groupby(['venue', 'city']).size()
venues


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
When did we play in Pittsburgh?
We can use a Series of boolean values to slice our DataFrame:
Can you write Python code for it?

# Series.str gives us access to string methods
in_pgh = gig_cities.str.contains('Pittsburgh')
in_pgh

gigs[in_pgh]


I am working on the file "3 - Introducing Pandas.ipynb".
The first task is:
With pandas, we can do the same thing for every Series element:
Can you write Python code for it?

states = gig_cities.str.split(',').str.get(1).str.strip().value_counts()
states

states.describe()

# Don't rotate the x-axis labels
states[:5].plot.bar(rot=0);


I am working on the file "3 - Lists.ipynb".
The first task is:
Common error: note that the above methods do not return the modified list, they just modify the original list.
Can you write Python code for it?


I am working on the file "3 - Lists.ipynb".
The first task is:
So list.append() doesn't return a value. 'None' is a python value that means there is no value (roll with it). It's great for situations where in other languages you'd set variables to -1 or something.
List Build Up
One common pattern is to start a list a the empty list [], then use append() or extend() to add elements to it:
Can you write Python code for it?

list = []
list.append('a')
list.append('b')
list


I am working on the file "3 - Lists.ipynb".
The first task is:
The for/in constructs are very commonly used in Python code and work on data types other than list, so should just memorize their syntax. You may have habits from other languages where you start manually iterating over a collection, where in Python you should just use for/in.
You can also use for/in to work on a string. The string acts like a list of its chars, so for ch in s: print ch prints all the chars in a string.
Range
The range(n) function yields the numbers 0, 1, ... n-1, and range(a, b) returns a, a+1, ... b-1 -- up to but not including the last number. The combination of the for-loop and the range() function allow you to build a traditional numeric for loop:
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-2 NumPy Array Basics - Helpful Methods, Shortcuts, NaN Values, and Dtypes.ipynb".
The first task is:
First let’s checkout the type we’ve got, you can see it’s an ndarray which is the fundamental array in numpy. We can get the data type of the array. It’s worth elaborating a bit on dtypes or data types.
Unlike python, numpy has a variety of types. I’ll copy and paste the list here and speak to the different types.
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-2 NumPy Array Basics - Helpful Methods, Shortcuts, NaN Values, and Dtypes.ipynb".
The first task is:
If you have any worry about what kind of array you’re creating, just specify the type.
Can you write Python code for it?

np.array([True, 1, 2.0], dtype='bool_')

np.array([True, 1, 2.0], dtype='float_')

np.array([True, 1, 2.0], dtype='uint8')

npa


I am working on the file "3 - NumPy Basics/3-2 NumPy Array Basics - Helpful Methods, Shortcuts, NaN Values, and Dtypes.ipynb".
The first task is:
If you’ve got a nan value in your array. It will throw off all your calculations. You’ve got to explicitly check for it in numpy.
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-2 NumPy Array Basics - Helpful Methods, Shortcuts, NaN Values, and Dtypes.ipynb".
The first task is:
It’s worth starting that numpy arrays can be created in the fly simply by passing a list into the “array” function.
By default, numpy will try to guess what kinds of types you have in there but on creation you can also specify which type you would like. Now numpy arrays all have to have the same type, you can’t have a mix of floats and integers for example.
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-2 NumPy Array Basics - Helpful Methods, Shortcuts, NaN Values, and Dtypes.ipynb".
The first task is:
Now we’ve got our array and we’ve got a method or two that we can use to create them. Let’s learn a bit more about some of the functions that arrays give us.
Can you write Python code for it?

npa = np.arange(25)

npa

print(type(npa))


I am working on the file "3 - NumPy Basics/3-2 NumPy Array Basics - Helpful Methods, Shortcuts, NaN Values, and Dtypes.ipynb".
The first task is:
Numpy arrays also have some nifty functions that make our lives a lot easier. We can get the size, which is functionally equivalent to len in raw python. We can get the shape which gives us a hint as to where we’ll be going next, towards multidimensional arrays.
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-2 NumPy Array Basics - Helpful Methods, Shortcuts, NaN Values, and Dtypes.ipynb".
The first task is:
This allows for consistent memory allocation and vectorization.
Sometimes you can get strange types when you're not quite expecting them.
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-2 NumPy Array Basics - Helpful Methods, Shortcuts, NaN Values, and Dtypes.ipynb".
The first task is:
What about converting that to integers? We can use the as type method to convert it.
We can convert it to an integer
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-3 NumPy Array Basics - Vectorization.ipynb".
The first task is:
So who cares? Again it’s going to be efficiency thing just like boolean selection Let’s try something a bit more complex.
Define a function named new_func that cubes the value if it is less than 5 and squares it if it is greater or equal to 5.
Can you write Python code for it?

def new_func(numb):
    if numb < 10:
        return numb**3
    else:
        return numb**2

new_func(npa)


I am working on the file "3 - NumPy Basics/3-3 NumPy Array Basics - Vectorization.ipynb".
The first task is:
Speed comparisons with size.
Can you write Python code for it?

%timeit [new_func(x) for x in npa2]
%timeit vect_new_func(npa2)


I am working on the file "3 - NumPy Basics/3-3 NumPy Array Basics - Vectorization.ipynb".
The first task is:
We need to vectorize this operation and we do that with np.vectorize
We can then apply that to our entire array and it takes care of the complexity for us. We can think in terms of the data without having to think about each individual element.
Can you write Python code for it?

vect_new_func = np.vectorize(new_func)

type(vect_new_func)

vect_new_func(npa)

[new_func(x) for x in npa]


I am working on the file "3 - NumPy Basics/3-5 NumPy Array Basics - Querying, Slicing, Combining, and Splitting Arrays.ipynb".
The first task is:
First let’s generate some random arrays. We’ll generate some that are 1 dimension, 2 dimensional, and 3 dimensional.
Can you write Python code for it?

ar = np.arange(12)
ar

ar2 = np.random.random_integers(12, size=12)
ar2

ndim_ar = np.arange(12).reshape(3,4)
ndim_ar

ndim_ar2 = np.random.random_integers(12, size=(3,4))
ndim_ar2

ndim_ar3d = np.arange(8).reshape(2,2,2)
ndim_ar3d


I am working on the file "3 - NumPy Basics/3-5 NumPy Array Basics - Querying, Slicing, Combining, and Splitting Arrays.ipynb".
The first task is:
Now that we’ve got some experience querying, let’s go over combining different arrays.
Now let’s stack the first two arrays vertically, we’ll do that with vstack. We can do the same with multidimensional arrays.
Can you write Python code for it?

np.vstack((ar,ar2))

np.vstack((ndim_ar, ndim_ar2))


I am working on the file "3 - NumPy Basics/3-5 NumPy Array Basics - Querying, Slicing, Combining, and Splitting Arrays.ipynb".
The first task is:
Now you can probably guess what horizontal stacking is. That would be hstack.
Can you write Python code for it?

np.hstack((ndim_ar, ndim_ar2))

ar3 = np.hstack((ar,ar2))
ar3

ar


I am working on the file "3 - NumPy Basics/3-5 NumPy Array Basics - Querying, Slicing, Combining, and Splitting Arrays.ipynb".
The first task is:
Querying 1 dimensional arrays is easy, we just perform the lookup like we would a regular array.
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-5 NumPy Array Basics - Querying, Slicing, Combining, and Splitting Arrays.ipynb".
The first task is:
Querying 2 dimensional arrays is a bit more interesting, we use commas to separate the axis.
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-5 NumPy Array Basics - Querying, Slicing, Combining, and Splitting Arrays.ipynb".
The first task is:
Things get even more interesting with 3+ dimensions, obviously it’s a lot to keep track in your head. but We just go dimension by dimension.
We’ll get the first dimension, then all the items in the second dimension, then everything beyond the first item in the 3rd dimension.
Can you write Python code for it?


I am working on the file "3 - NumPy Basics/3-5 NumPy Array Basics - Querying, Slicing, Combining, and Splitting Arrays.ipynb".
The first task is:
We can do the same with hsplit and vsplit.
Can you write Python code for it?

np.hsplit(ar3,2)

ndim_ar3

np.vsplit(ndim_ar3,2)


I am working on the file "3 - NumPy Basics/3-5 NumPy Array Basics - Querying, Slicing, Combining, and Splitting Arrays.ipynb".
The first task is:
we can stack them dimensionally with dstack. Now we’ve got a 3 dimensional join of these two two dimensional arrays.
Can you write Python code for it?

ndim_ar3 = np.concatenate((ndim_ar,ndim_ar2), axis=0)
ndim_ar3

np.concatenate((ndim_ar,ndim_ar2), axis=1)

ndim_ar3d = np.dstack((ndim_ar,ndim_ar2))
ndim_ar3d


I am working on the file "3 Kernel, Bayes and Models.ipynb".
The first task is:
Classifer #5 Random Forest
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

# 1. Using entropy criterion

# Training
forest = RandomForestClassifier(criterion='entropy', n_estimators=10, random_state=1, n_jobs=2)
forest.fit(X_train, y_train)
# Testing
y_pred = forest.predict(X_test)
# Results
print('1. Using entropy criterion:')
print('   Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('   Accuracy: %.3f' % accuracy_score(y_test, y_pred))

# 2. Using Gini criterion

# Training
forest = RandomForestClassifier(criterion='gini', n_estimators=10, random_state=1, n_jobs=2)
forest.fit(X_train, y_train)
# Testing
y_pred = forest.predict(X_test)
# Results
print('2. Using Gini criterion:')
print('   Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('   Accuracy: %.3f' % accuracy_score(y_test, y_pred))


I am working on the file "3 Kernel, Bayes and Models.ipynb".
The first task is:
Classifier #1 Perceptron
Can you write Python code for it?

from sklearn.linear_model import Perceptron
from sklearn.metrics import accuracy_score

# Training
ppn = Perceptron(n_iter=800, eta0=0.1, random_state=0)
ppn.fit(X_train, y_train)
# Testing
y_pred = ppn.predict(X_test)
# Results
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))


I am working on the file "3 Kernel, Bayes and Models.ipynb".
The first task is:
Classifier #2 Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

# Training
lr = LogisticRegression(C=1.0, random_state=0) # we observe that changing C from 0.0001 to 1000 has ignorable effect
lr.fit(X_train, y_train)
# Testing
y_pred = lr.predict(X_test)
# Results
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))


I am working on the file "3 Kernel, Bayes and Models.ipynb".
The first task is:
Classifier #6 KNN
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

# Training
knn = KNeighborsClassifier(n_neighbors=5, p=2, metric='minkowski')
knn.fit(X_train, y_train)
# Testing
y_pred = knn.predict(X_test)
# Results
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))


I am working on the file "3 Kernel, Bayes and Models.ipynb".
The first task is:
Classifier #7 Naive Bayes
Can you write Python code for it?

from sklearn.naive_bayes import GaussianNB

# Training
gnb = GaussianNB()
gnb.fit(X_train, y_train)
# Testing
y_pred = gnb.predict(X_test)
# Results
print('Misclassified samples: %d out of %d' % ((y_test != y_pred).sum(), y_test.shape[0]))
print('Accuracy: %.3f' % accuracy_score(y_test, y_pred))


I am working on the file "3 News robot/Earthquake news robot.ipynb".
The first task is:
2. Check for errors
Don't assume that the data is always good. Create tests to make sure.
Can you write Python code for it?

# We start with the assumption that the data is news worthy.
IsNewsWorthy = True

# Are there errors in the data?
if data["Richter"] > 100:
    # A earthquake with magnitue of 100 seems unlikely, so we ignore it.
    IsNewsWorthy = False

# We are only interested in earthquakes in Sweden.
if data["Country"] != "Sweden":
    IsNewsWorthy = False


I am working on the file "3 News robot/Earthquake news robot.ipynb".
The first task is:
3. Create text
Basically, it's just a lot of if-statements.
How to make the code easier to read:

Don't nest if-statements inside each other too much. Use elif instead.
Use multiline strings, with """ around them.
Use .format on strings.

text = "My name is {0} and I am {1} years old"
text = text.format(Name, Age)
Which is the same as this:
text = "My name is " + Name + " and I am " + str(Age) + " years old"
Can you write Python code for it?

# If the earthquake is deemed news worthy, then create a journalistic text.
text = ""
if IsNewsWorthy:
    if (data["Richter"] > 6):
        # Text for large quake (6+).
        text = """BREAKING: Major earthquake in {0}

Today at {1} there was a severe earthquake in {2}, {3}, with a magnitude of {4} on the Richter scale.

"""
        text = text.format(data["City"], data["Datetime"], data["City"], data["Country"], data["Richter"])
    elif (data["Richter"] < 6 or data["Richter"] >= 3):
        # Text for medium quake (3-5).
        text = """Earthquake in {0}

Today at {1} there was a earthquake in {2}, {3}, with a magnitude of {4} on the Richter scale.

"""
        text = text.format(data["City"], data["Datetime"], data["City"], data["Country"], data["Richter"])

    # Add this at the end of all texts.
    text = text + "Published " + datetime.now().strftime("%Y-%m-%d %H:%M") + " by Ada the news robot"

# Look at the text
print(text)


I am working on the file "3 News robot/Scraping examples.ipynb".
The first task is:
2. Define scraping functions
Can you write Python code for it?

# Scrape all HTML from webpage.
def scrapewebpage(url):
	# Open URL and get HTML.
	web = urllib.request.urlopen(url)

	# Make sure there wasn't any errors opening the URL.
	if (web.getcode() == 200):
		html = web.read()
		return(html)
	else:
		print("Error %s reading %s" % str(web.getcode()), url)

# Helper function that scrape the webpage and turn it into soup.
def makesoup(url):
	html = scrapewebpage(url)
	return(BeautifulSoup(html, "lxml"))


I am working on the file "3 News robot/Scraping examples.ipynb".
The first task is:
3. BeautifulSoup examples
A) Find an id
Use find(id="name") to find the first HTML tag that has an id attribute like this:

Can you write Python code for it?

# Scrape Wikipedia main page.
wp_soup = makesoup("https://en.wikipedia.org/wiki/Main_Page")

# Match the <h2> tag with id the id mp-itn-h2
h2 = wp_soup.find(id="mp-itn-h2")

h2

# Only get the text inside <h2>.
h2.get_text()


I am working on the file "3 News robot/Scraping examples.ipynb".
The first task is:
B) Find a class
Use find("", "name") to find the first HTML tag that has a class attribute like this:

Can you write Python code for it?

# Scrape Wikipedia main page.
wp_soup = makesoup("https://en.wikipedia.org/wiki/Main_Page")

# Find the first HTML tag that has class mw-headline.
headline = wp_soup.find("", "mw-headline")

headline

# Only get the text inside the <span>.
headline.get_text()


I am working on the file "3 News robot/Scraping examples.ipynb".
The first task is:
C) Find everything with a class
Use find_all("", "name") to find all HTML tags that has a class attribute like this:

Can you write Python code for it?

# Scrape Wikipedia main page.
wp_soup = makesoup("https://en.wikipedia.org/wiki/Main_Page")

# Find all HTML tag that has class mw-headline.
all_headlines = wp_soup.find_all("", "mw-headline")

all_headlines

# Now we have a list that we can use a for loop.
for headline in all_headlines:
    headline = headline.get_text()
    print(headline)


I am working on the file "3 News robot/Scraping examples.ipynb".
The first task is:
D) Find all <h3>
Use find_all("h3") to get all <h3> (or something else).
Can you write Python code for it?

# Scrape Wikipedia main page.
wp_soup = makesoup("https://en.wikipedia.org/wiki/Main_Page")

# Find all HTML tag that has class mw-headline.
all_h3 = wp_soup.find_all("h3")

all_h3

# Now we have a list that we can use a for loop.
for h3 in all_h3:
    h3 = h3.get_text()
    print(h3)


I am working on the file "3 News robot/Scraping examples.ipynb".
The first task is:
E) Find all column values in a table
Use for to loop through a <table> and extract all column values.
Can you write Python code for it?

# Scrape a Wikipedia page with a table.
champ_soup = makesoup("https://en.wikipedia.org/wiki/European_Road_Championships")

# Find <table class="wikitable">.
table = champ_soup.find("table", "wikitable")

table

# Go through each row and take the text from 1st and 2nd column.
rows = table.find_all('tr')
for row in rows:
    cols = row.find_all('td')
    if len(cols) > 0:
        Year = cols[0].get_text()        # Get the text in the 1st column.
        Country = cols[1].get_text()     # Get the text in the 2nd column.

        print(Year + " " + Country)


I am working on the file "3 News robot/Scraping examples.ipynb".
The first task is:
F) Find a specific cell value in a table
Use this if you know the row and column of the <table> of the information you want to extract.
Can you write Python code for it?

table = champ_soup.find("table", "wikitable")

# Get cell value from row 5, column 1.
cell = table.find_all('tr')[5].find_all('td')[1].get_text()

cell


I am working on the file "3 News robot/Scraping examples.ipynb".
The first task is:
G) Find nested things
Use this if you want to find things that are nested inside each oher.
Can you write Python code for it?

# Scrape Wikipedia main page.
wp_soup = makesoup("https://en.wikipedia.org/wiki/Main_Page")

# Find <table class="mp-middle">.
middle_table = wp_soup.find("table", id="mp-middle")

# In the <table>, find <h2>.
h2 = middle_table.find("h2")

h2

# Only get the text inside the <h2>.
h2.get_text()


I am working on the file "3 News robot/Weather news robot.ipynb".
The first task is:
2. Define scraping functions
Can you write Python code for it?


I am working on the file "3 News robot/Weather news robot.ipynb".
The first task is:
3. Scrape weather
We'll scrape tomorrows weather from http://www.weatherpal.se/vader/Goteborg-2711537/
Can you write Python code for it?

weather_soup = makesoup("http://www.weatherpal.se/vader/Goteborg-2711537/")

# Get minimium and maximum temperatures for the next 6 days.
soup_min_temps = weather_soup.find_all("div", "min-temp")
soup_max_temps = weather_soup.find_all("div", "max-temp")

# Create empty lists to store temperatures.
min_temps = []
max_temps = []

# For each minimum temperature, put it in a list.
for temp in soup_min_temps:
    min_temps.append(temp.get_text())

# For each maxiumum temperature, put it in a list.
for temp in soup_max_temps:
    max_temps.append(temp.get_text())

# Lets check tomorrows weather.
print("Min celsius: " + min_temps[0])
print("Max celsius: " + max_temps[0])

# What date is it tomorrow?
tomorrow = datetime.today() + timedelta(days=1)

# Name of weekdays.
weekdays = ["Monday", "Tuesday", "Wednesday", "Thursday", "Saturday", "Sunday"]

# Use simpler names.
tomorrow_weekday = weekdays[tomorrow.weekday()]
tomorrow_mintemp = int(min_temps[0])
tomorrow_maxtemp = int(max_temps[0])


I am working on the file "3 News robot/Weather news robot.ipynb".
The first task is:
4. Check for errors
Don't assume that the data is always good. Create tests to make sure.
Can you write Python code for it?

# We start with the assumption that the data is news worthy.
IsNewsWorthy = True

# Are there errors in the data?
if tomorrow_mintemp < -40:
    IsNewsWorthy = False
elif tomorrow_maxtemp > 60:
    IsNewsWorthy = False

IsNewsWorthy


I am working on the file "3 News robot/Weather news robot.ipynb".
The first task is:
5. Create text
Basically, it's just a lot of if-statements.
How to make the code easier to read:

Don't nest if-statements inside each other too much. Use elif instead.
Use multiline strings, with """ around them.
Use .format on strings.

text = "My name is {0} and I am {1} years old"
text = text.format(Name, Age)
Which is the same as this:
text = "My name is " + Name + " and I am " + str(Age) + " years old"
Can you write Python code for it?

text = ""
if IsNewsWorthy:
    if (tomorrow_mintemp < 0):
        text = "At {0} it will be freezing. The temperature will be {1} celsius, {2} at max."
        text = text.format(tomorrow_weekday, tomorrow_mintemp, tomorrow_maxtemp)
    elif (tomorrow_mintemp == 0):
        text = "At {0} it will be just above freezing point. The temperature will be between {1} and {2} celsius."
        text = text.format(tomorrow_weekday, tomorrow_mintemp, tomorrow_maxtemp)
    elif (tomorrow_mintemp > 0 and tomorrow_mintemp < 7):
        text = "Tomorrow, {0}, we will not freeze our socks off. Fortunately, the temperature will go down to {1} celsius but hopefully rise up to the maximum of {2}."
        text = text.format(tomorrow_weekday, tomorrow_mintemp, tomorrow_maxtemp)
    elif (tomorrow_mintemp > 7):
        text = "Tomorrow, {0}, it will be quite warm. Fortunately, the temperature will be {1} celsius at minimum, {2} maximum."
        text = text.format(tomorrow_weekday, tomorrow_mintemp, tomorrow_maxtemp)

    # Add this at the end of all texts.
    text = text + "\n\n" + "Published " + datetime.now().strftime("%Y-%m-%d %H:%M") + " by Ada the news robot"

# Look at the text
print(text)


I am working on the file "3 News robot/Weather news robot.ipynb".
The first task is:
6. Save the results to a text file
Lets create a function that saves the text to a file.
Can you write Python code for it?

# Function to save the text to a file.
def savefile(filename, text):
    f = open(filename, mode="w")   # Open file for writing (w = writing, a = append, r = reading)
    f.write(text)
    f.close()

# Only save as text file if there is some text.
if text != "":
    savefile("newsrobot-weather.txt", text)
    print("Text published!")
else:
    print("Text is not published.")


I am working on the file "3 News robot/Weather news robot.ipynb".
The first task is:
7. Present the results (read from text file)
Lets create a function that reads the text from the text file.
Can you write Python code for it?

# Function that reads text from a file.
def readfile(filename):
    f = open(filename, mode="r")   # Open file for reading (w = writing, a = append, r = reading)
    lines = f.read()
    f.close()
    return(lines)

# Read the file created earlier by the news robot.
text = readfile("newsrobot-weather.txt")

# Look at the text from the file.
print(text)


I am working on the file "3 Unsupervised learning/Homework/1 banners placing/Размещение баннеров.ipynb".
The first task is:
Для сдачи задания выберите из получившихся 20 центров тот, который наименее удален от ближайшего к нему офиса. Ответ в этом задании — широта и долгота этого центра, записанные через пробел.
Can you write Python code for it?

print('Minimal distance: %f' % distances_from_nearest_office_sorted[0])
print('Coordinates of closest centroid: %s' % str(cluster_centers_more_sorted[0]))

#Writing the answer to the file
answer = str(cluster_centers_more_sorted[0, 0]) + ' ' + str(cluster_centers_more_sorted[0, 1])
with open('answer.txt', 'w') as fout:
    fout.write(answer)


I am working on the file "3 Unsupervised learning/Homework/1 banners placing/Размещение баннеров.ipynb".
The first task is:
Как мы помним, 20 баннеров надо разместить близ офисов компании. Найдем на Google Maps по запросу Carnival Cruise Line адреса всех офисов:
33.751277, -118.188740 (Los Angeles)
25.867736, -80.324116 (Miami)
51.503016, -0.075479 (London)
52.378894, 4.885084 (Amsterdam)
39.366487, 117.036146 (Beijing)
-33.868457, 151.205134 (Sydney)
Осталось определить 20 ближайших к ним центров кластеров. Т.е. посчитать дистанцию до ближайшего офиса для каждой точки и выбрать 20 с наименьшим значением.
Can you write Python code for it?

#Reading the Carnival Cruise Line offices file
offices = pd.read_csv('carnival_cruise_lines_officies.txt', skipinitialspace=True)
print(offices.info())
offices_for_dist = offices.loc[:, ['latitude', 'longitude']]
offices_for_dist

def distance(x, y):
    return np.sqrt( np.sum((x - y)**2) )

#Calculating distances from nearest offices
distances_from_nearest_office = np.empty( cluster_centers_more.shape[0] )
for i, item in enumerate(cluster_centers_more):
    min_dist = distance(item, offices_for_dist.loc[0])
    for j in range(len(offices)):
        dist = distance(item, offices_for_dist.loc[j])
        if dist < min_dist:
            min_dist = dist
    distances_from_nearest_office[i] = min_dist

#Sorting arrays on minimum distance
dist_sorted_indices = np.argsort(distances_from_nearest_office)
distances_from_nearest_office_sorted = distances_from_nearest_office[dist_sorted_indices]
cluster_centers_more_sorted = cluster_centers_more[dist_sorted_indices]


I am working on the file "3 Unsupervised learning/Homework/1 banners placing/Размещение баннеров.ipynb".
The first task is:
Теперь необходимо кластеризовать данные координаты, чтобы выявить центры скоплений туристов. Поскольку баннеры имеют сравнительно небольшую площадь действия, нам нужен алгоритм, позволяющий ограничить размер кластера и не зависящий от количества кластеров.
Эта задача — хороший повод познакомиться с алгоритмом MeanShift, который мы обошли стороной в основной части лекций. Его описание при желании можно посмотреть в sklearn user guide, а чуть позже появится дополнительное видео с обзором этого и некоторых других алгоритмов кластеризации. Используйте MeanShift, указав bandwidth=0.1, что в переводе из градусов в метры колеблется примерно от 5 до 10 км в средних широтах.
Примечание: на 396634 строках кластеризация будет работать долго. Быть очень терпеливым не возбраняется — результат от этого только улучшится. Но для того, чтобы сдать задание, понадобится сабсет из первых 100 тысяч строк. Это компромисс между качеством и затраченным временем. Обучение алгоритма на всём датасете занимает около часа, а на 100 тыс. строк — примерно 2 минуты, однако этого достаточно для получения корректных результатов.
Can you write Python code for it?

#Array for clustering (100000 entries and 'latitude' and 'longitude' columns)
checkins_cl = checkins.iloc[:100000, :].loc[:, ['latitude', 'longitude']]
print('Shape: ' + str(checkins_cl.shape))
checkins_cl.head()

#Clustering with MeanShift
ms = MeanShift(bandwidth=0.1)
ms.fit(checkins_cl)


I am working on the file "3 Unsupervised learning/Homework/2 principal component analysis (PCA)/PCA.ipynb".
The first task is:
Анализ основных недостатков метода главных компонент
Рассмотренные выше задачи являются, безусловно, модельными, потому что данные для них были сгенерированы в соответствии с предположениями метода главных компонент. На практике эти предположения, естественно, выполняются далеко не всегда. Рассмотрим типичные ошибки PCA, которые следует иметь в виду перед тем, как его применять.
Направления с максимальной дисперсией в данных неортогональны
Рассмотрим случай выборки, которая сгенерирована из двух вытянутых нормальных распределений:
Can you write Python code for it?

C1 = np.array([[10,0],[0,0.5]])
phi = np.pi/3
C2 = np.dot(C1, np.array([[np.cos(phi), np.sin(phi)],
                          [-np.sin(phi),np.cos(phi)]]))

data = np.vstack([np.random.multivariate_normal(mu, C1, size=50),
                  np.random.multivariate_normal(mu, C2, size=50)])
plt.scatter(data[:,0], data[:,1])
# построим истинные интересующие нас компоненты
plt.plot(data[:,0], np.zeros(data[:,0].size), color="g")
plt.plot(data[:,0], 3**0.5*data[:,0], color="g")
# обучим модель pca и построим главные компоненты
model = PCA(n_components=2)
model.fit(data)
plot_principal_components(data, model, scatter=False, legend=False)
c_patch = mpatches.Patch(color='c', label='Principal components')
plt.legend(handles=[g_patch, c_patch])
plt.draw()


I am working on the file "3 Unsupervised learning/Homework/2 principal component analysis (PCA)/PCA.ipynb".
The first task is:
В чём проблема, почему pca здесь работает плохо? Ответ прост: интересующие нас компоненты в данных коррелированны между собой (или неортогональны, в зависимости от того, какой терминологией пользоваться). Для поиска подобных преобразований требуются более сложные методы, которые уже выходят за рамки метода главных компонент.
Для интересующихся: то, что можно применить непосредственно к выходу метода главных компонент, для получения подобных неортогональных преобразований, называется методами ротации. Почитать о них можно в связи с другим методом уменьшения размерности, который называется Factor Analysis (FA), но ничего не мешает их применять и к главным компонентам.
Интересное направление в данных не совпадает с направлением максимальной дисперсии
Рассмотрим пример, когда дисперсии не отражают интересующих нас направлений в данных:
Can you write Python code for it?

C = np.array([[0.5,0],[0,10]])
mu1 = np.array([-2,0])
mu2 = np.array([2,0])

data = np.vstack([np.random.multivariate_normal(mu1, C, size=50),
                  np.random.multivariate_normal(mu2, C, size=50)])
plt.scatter(data[:,0], data[:,1])
# обучим модель pca и построим главные компоненты
model = PCA(n_components=2)
model.fit(data)
plot_principal_components(data, model)
plt.draw()


I am working on the file "3 Unsupervised learning/Homework/2 principal component analysis (PCA)/PCA.ipynb".
The first task is:
Вариационный взгляд на модель
Мы знаем, что каждой главной компоненте соответствует описываемая ей дисперсия данных (дисперсия данных при проекции на эту компоненту). Она численно равна значению диагональных элементов матрицы $\Lambda$, получаемой из спектрального разложения матрицы ковариации данных (смотри теорию выше).
Исходя из этого, мы можем отсортировать дисперсию данных вдоль этих компонент по убыванию, и уменьшить размерность данных, отбросив $q$ итоговых главных компонент, имеющих наименьшую дисперсию.
Делать это можно двумя разными способами. Например, если вы вдальнейшем обучаете на данных с уменьшенной размерностью модель классификации или регрессии, то можно запустить итерационный процесс: удалять компоненты с наименьшей дисперсией по одной, пока качество итоговой модели не станет значительно хуже.
Более общий способ отбора признаков заключается в том, что вы можете посмотреть на разности в дисперсиях в отсортированном ряде $\lambda_{(1)} > \lambda_{(2)} > \dots > \lambda_{(D)}$: $\lambda_{(1)}-\lambda_{(2)}, \dots, \lambda_{(D-1)} - \lambda_{(D)}$, и удалить те компоненты, на которых разность будет наибольшей. Именно этим методом вам и предлагается воспользоваться для тестового набора данных.
Задание 2. Ручное уменьшение размерности признаков посредством анализа дисперсии данных вдоль главных компонент
Рассмотрим ещё один набор данных размерности $D$, чья реальная размерность значительно меньше наблюдаемой (назовём её также $d$). От вас требуется:

Построить модель PCA с $D$ главными компонентами по этим данным.
Спроецировать данные на главные компоненты.
Оценить их дисперсию вдоль главных компонент.
Отсортировать дисперсии в порядке убывания и получить их попарные разности: $\lambda_{(i-1)} - \lambda_{(i)}$.
Найти разность с наибольшим значением и получить по ней оценку на эффективную размерность данных $\hat{d}$.
Построить график дисперсий и убедиться, что полученная оценка на $\hat{d}{opt}$ действительно имеет смысл, после этого внести полученное значение $\hat{d}{opt}$ в файл ответа.

Для...
Can you write Python code for it?

from sklearn.decomposition import PCA
from sklearn.cross_validation import cross_val_score as cv_score

def plot_variances(d_variances):
    n_components = np.arange(1,d_variances.size+1)
    plt.plot(n_components, d_variances, 'b', label='Component variances')
    plt.xlim(n_components[0], n_components[-1])
    plt.xlabel('n components')
    plt.ylabel('variance')
    plt.legend(loc='upper right')
    plt.show()

def write_answer_2(optimal_d):
    with open("pca_answer2.txt", "w") as fout:
        fout.write(str(optimal_d))

data = pd.read_csv('data_task2.csv')
print('Data shape: %s' % str(data.shape))

model_PCA = PCA(svd_solver='full')
model_PCA.fit_transform(data)

d_variances = model_PCA.explained_variance_
plot_variances(d_variances)

diff_d_variances = d_variances[:-1] - d_variances[1:]
eff_d_num = np.argmax(diff_d_variances) + 1
print('Effective number of dimensions: %d' % eff_d_num)

write_answer_2(eff_d_num)


I am working on the file "3 Unsupervised learning/Homework/2 principal component analysis (PCA)/PCA.ipynb".
The first task is:
Путём диагонализации истинной матрицы ковариаций $C$, мы можем найти преобразование исходного набора данных, компоненты которого наилучшим образом будут описывать дисперсию, с учётом их ортогональности друг другу:
Can you write Python code for it?

v, W_true = np.linalg.eig(C)

plt.scatter(data[:,0], data[:,1])
# построим истинные компоненты, вдоль которых максимальна дисперсия данных
plt.plot(data[:,0], (W_true[0,0]/W_true[0,1])*data[:,0], color="g")
plt.plot(data[:,0], (W_true[1,0]/W_true[1,1])*data[:,0], color="g")
g_patch = mpatches.Patch(color='g', label='True components')
plt.legend(handles=[g_patch])
plt.axis('equal')
limits = [np.minimum(np.amin(data[:,0]), np.amin(data[:,1])),
          np.maximum(np.amax(data[:,0]), np.amax(data[:,1]))]
plt.xlim(limits[0],limits[1])
plt.ylim(limits[0],limits[1])
plt.draw()


I am working on the file "3 Unsupervised learning/Homework/4 text themes/edit_CookingLDA_PA.ipynb".
The first task is:
Интерпретация построенной модели
Вы можете рассмотреть топы ингредиентов каждой темы. Большиснтво тем сами по себе похожи на рецепты; в некоторых собираются продукты одного вида, например, свежие фрукты или разные виды сыра.
Попробуем эмпирически соотнести наши темы с национальными кухнями (cuisine). Построим матрицу $A$ размера темы $x$ кухни, ее элементы $a_{tc}$ — суммы $p(t|d)$ по всем документам $d$, которые отнесены к кухне $c$. Нормируем матрицу на частоты рецептов по разным кухням, чтобы избежать дисбаланса между кухнями. Следующая функция получает на вход объект модели, объект корпуса и исходные данные и возвращает нормированную матрицу $A$. Ее удобно визуализировать с помощью seaborn.
Can you write Python code for it?

import pandas
import seaborn
from matplotlib import pyplot as plt
%matplotlib inline

def compute_topic_cuisine_matrix(model, corpus, recipes):
    # составляем вектор целевых признаков
    targets = list(set([recipe["cuisine"] for recipe in recipes]))
    # составляем матрицу
    tc_matrix = pandas.DataFrame(data=np.zeros((model.num_topics, len(targets))), columns=targets)
    for recipe, bow in zip(recipes, corpus):
        recipe_topic = model.get_document_topics(bow)
        for t, prob in recipe_topic:
            tc_matrix[recipe["cuisine"]][t] += prob
    # нормируем матрицу
    target_sums = pandas.DataFrame(data=np.zeros((1, len(targets))), columns=targets)
    for recipe in recipes:
        target_sums[recipe["cuisine"]] += 1
    return pandas.DataFrame(tc_matrix.values/target_sums.values, columns=tc_matrix.columns)

def plot_matrix(tc_matrix):
    plt.figure(figsize=(10, 10))
    seaborn.heatmap(tc_matrix, square=True)

# Визуализируйте матрицу


I am working on the file "3 Unsupervised learning/Homework/4 text themes/edit_CookingLDA_PA.ipynb".
The first task is:
Наша коллекция небольшая, и целиком помещается в оперативную память. Gensim может работать с такими данными и не требует их сохранения на диск в специальном формате. Для этого коллекция должна быть представлена в виде списка списков, каждый внутренний список соответствует отдельному документу и состоит из его слов. Пример коллекции из двух документов:
[["hello", "world"], ["programming", "in", "python"]]
Преобразуем наши данные в такой формат, а затем создадим объекты corpus и dictionary, с которыми будет работать модель.
Can you write Python code for it?

texts = [recipe["ingredients"] for recipe in recipes]
dictionary = corpora.Dictionary(texts)   # составляем словарь
corpus = [dictionary.doc2bow(text) for text in texts]  # составляем корпус документов

print texts[0]
print corpus[0]


I am working on the file "3 Unsupervised learning/Homework/4 text themes/edit_CookingLDA_PA.ipynb".
The first task is:
Составление корпуса
Can you write Python code for it?

from gensim import corpora, models
import numpy as np