I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
(2b) PCA on dataRandom
Next, use the PCA function we just developed to find the top two principal components of the spherical dataRandom we created in Visualization 1.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
randomData = sc.parallelize(dataRandom)

# Use pca on randomData
topComponentsRandom, randomDataScoresAuto, eigenvaluesRandom = pca(randomData)

print 'topComponentsRandom: \n{0}'.format(topComponentsRandom)
print ('\nrandomDataScoresAuto (first three): \n{0}'
       .format('\n'.join(map(str, randomDataScoresAuto.take(3)))))
print '\neigenvaluesRandom: \n{0}'.format(eigenvaluesRandom)

# TEST PCA on `dataRandom` (2b)
Test.assertTrue(checkBasis(topComponentsRandom.T,
                           [[-0.2522559 ,  0.96766056], [-0.96766056,  -0.2522559]]),
                'incorrect value for topComponentsRandom')
firstThreeRandom = [[36.61068572,  35.97314295,  35.59836628],
                    [61.3489929 ,  62.08813671,  60.61390415]]
Test.assertTrue(np.allclose(firstThreeRandom, np.vstack(np.abs(randomDataScoresAuto.take(3))).T),
                'incorrect value for randomDataScoresAuto')
Test.assertTrue(np.allclose(eigenvaluesRandom, [1.4204546, 0.99521397]),
                            'incorrect value for eigenvaluesRandom')


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
(2c) 3D to 2D
We will now use PCA to see if we can recover the 2-dimensional plane on which the data live. Parallelize the data, and use our PCA function from above, with $ \scriptsize k=2 $ components.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
threeDData = sc.parallelize(dataThreeD)
componentsThreeD, threeDScores, eigenvaluesThreeD = pca(threeDData)

print 'componentsThreeD: \n{0}'.format(componentsThreeD)
print ('\nthreeDScores (first three): \n{0}'
       .format('\n'.join(map(str, threeDScores.take(3)))))
print '\neigenvaluesThreeD: \n{0}'.format(eigenvaluesThreeD)

# TEST 3D to 2D (2c)
Test.assertEquals(componentsThreeD.shape, (3, 2), 'incorrect shape for componentsThreeD')
Test.assertTrue(np.allclose(np.sum(eigenvaluesThreeD), 969.796443367),
                'incorrect value for eigenvaluesThreeD')
Test.assertTrue(np.allclose(np.abs(np.sum(componentsThreeD)), 1.77238943258),
                'incorrect value for componentsThreeD')
Test.assertTrue(np.allclose(np.abs(np.sum(threeDScores.take(3))), 237.782834092),
                'incorrect value for threeDScores')


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
(2d) Variance explained
Finally, let's quantify how much of the variance is being captured by PCA in each of the three synthetic datasets we've analyzed.  To do this, we'll compute the fraction of retained variance by the top principal components.  Recall that the eigenvalue corresponding to each principal component captures the variance along this direction.  If our initial data is $\scriptsize d$-dimensional, then the total variance in our data equals: $ \scriptsize \sum_{i=1}^d \lambda_i $, where $\scriptsize \lambda_i$ is the eigenvalue corresponding to the $\scriptsize i$th principal component. Moreover, if we use PCA with some $\scriptsize k < d$, then we can compute the variance retained by these principal components by adding the top $\scriptsize k$ eigenvalues.  The fraction of retained variance equals the sum of the top $\scriptsize k$ eigenvalues divided by the sum of all of the eigenvalues.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def varianceExplained(data, k=1):
    """Calculate the fraction of variance explained by the top `k` eigenvectors.

    Args:
        data (RDD of np.ndarray): An RDD that contains NumPy arrays which store the
            features for an observation.
        k: The number of principal components to consider.

    Returns:
        float: A number between 0 and 1 representing the percentage of variance explained
            by the top `k` eigenvectors.
    """
    components, scores, eigenvalues = pca(data, k)
    <FILL IN>

varianceRandom1 = varianceExplained(randomData, 1)
varianceCorrelated1 = varianceExplained(correlatedData, 1)
varianceRandom2 = varianceExplained(randomData, 2)
varianceCorrelated2 = varianceExplained(correlatedData, 2)
varianceThreeD2 = varianceExplained(threeDData, 2)
print ('Percentage of variance explained by the first component of randomData: {0:.1f}%'
       .format(varianceRandom1 * 100))
print ('Percentage of variance explained by both components of randomData: {0:.1f}%'
       .format(varianceRandom2 * 100))
print ('\nPercentage of variance explained by the first component of correlatedData: {0:.1f}%'.
       format(varianceCorrelated1 * 100))
print ('Percentage of variance explained by both components of correlatedData: {0:.1f}%'
       .format(varianceCorrelated2 * 100))
print ('\nPercentage of variance explained by the first two components of threeDData: {0:.1f}%'
       .format(varianceThreeD2 * 100))

# TEST Variance explained (2d)
Test.assertTrue(np.allclose(varianceRandom1, 0.588017172066), 'incorrect value for varianceRandom1')
Test.assertTrue(np.allclose(varianceCorrelated1, 0.933608329586),
                'incorrect value for varianceCorrelated1')
Test.assertTrue(np.allclose(varianceRandom2, 1.0), 'incorrect value for varianceRandom2')
Test.assertTrue(np.allclose(varianceCorrelated2, 1.0), 'incorrect value for varianceCorrelated2')
Test.assertTrue(np.allclose(varianceThreeD2, 0.993967356912), 'incorrect value for varianceThreeD2')


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
(4b) Recreate with np.tile and np.eye
np.tile is useful for repeating arrays in one or more dimensions.  For example, np.tile(np.array([[1, 2], [3, 4]]), 2) produces np.array([[1, 2, 1, 2], [3, 4, 3, 4]])).
$$ np.tile( \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} , 2) \to \begin{bmatrix} 1 & 2 & 1& 2 \\ 3 & 4 & 3 & 4 \end{bmatrix} $$
Recall that np.eye can be used to create an identity array $ (\mathbf{I_n}) $.  For example, np.eye(3) produces np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]]).
$$ np.eye( 3 ) \to \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$
In this exercise, recreate sumEveryOther and sumEveryThird using np.tile and np.eye.
Can you write Python code for it?

# Reference for what to recreate
print 'sumEveryOther: \n{0}'.format(sumEveryOther)
print '\nsumEveryThird: \n{0}'.format(sumEveryThird)

# TODO: Replace <FILL IN> with appropriate code
# Use np.tile and np.eye to recreate the arrays
sumEveryOtherTile = np.tile(np.eye(2), 3)
sumEveryThirdTile = np.tile(np.eye(3), 2)

print sumEveryOtherTile
print 'sumEveryOtherTile.dot(vector): {0}'.format(sumEveryOtherTile.dot(vector))
print '\n', sumEveryThirdTile
print 'sumEveryThirdTile.dot(vector): {0}'.format(sumEveryThirdTile.dot(vector))

# TEST Recreate with `np.tile` and `np.eye` (4b)
Test.assertEquals(sumEveryOtherTile.shape, (2, 6), 'incorrect shape for sumEveryOtherTile')
Test.assertEquals(sumEveryThirdTile.shape, (3, 6), 'incorrect shape for sumEveryThirdTile')
Test.assertTrue(np.allclose(sumEveryOtherTile.dot(vector), [6, 9]),
                'incorrect value for sumEveryOtherTile')
Test.assertTrue(np.allclose(sumEveryThirdTile.dot(vector), [3, 5, 7]),
                'incorrect value for sumEveryThirdTile')


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
(4c) Recreate with np.kron
The Kronecker product is the generalization of outer products involving matrices, and we've included some examples below to illustrate the idea.  Please refer to the Wikipedia page for a detailed definition.  We can use np.kron to compute Kronecker products and recreate the sumBy arrays.  Note that $ \otimes $ indicates a Kronecker product.
$$ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \otimes \begin{bmatrix} 1 & 2 \end{bmatrix}  = \begin{bmatrix} 1 \cdot 1 & 1 \cdot 2 & 2 \cdot 1 & 2 \cdot 2 \\ 3 \cdot 1 & 3 \cdot 2 & 4 \cdot 1 & 4 \cdot 2 \end{bmatrix} = \begin{bmatrix} 1 & 2 & 2 & 4 \\ 3 & 6 & 4 & 8 \end{bmatrix}  $$
We can see how the Kronecker product continues to expand if we add another row to the second array.
$$ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \otimes \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} = \begin{bmatrix} 1 \cdot 1 & 1 \cdot 2 & 2 \cdot 1 & 2 \cdot 2 \\ 1 \cdot 3 & 1 \cdot 4 & 2 \cdot 3 & 2 \cdot 4 \\ 3 \cdot 1 & 3 \cdot 2 & 4 \cdot 1 & 4 \cdot 2 \\ 3 \cdot 3 & 3 \cdot 4 & 4 \cdot 3 & 4 \cdot 4 \end{bmatrix} = \begin{bmatrix} 1 & 2 & 2 & 4 \\ 3 & 4 & 6 & 8 \\ 3 & 6 & 4 & 8 \\ 9 & 12 & 12 & 16 \end{bmatrix} $$
For this exercise, you'll recreate the sumByThree and sumByTwo arrays using np.kron, np.eye, and np.ones.  Note that np.ones creates an array of all ones.
Can you write Python code for it?

# Reference for what to recreate
print 'sumByThree: \n{0}'.format(sumByThree)
print '\nsumByTwo: \n{0}'.format(sumByTwo)

# TODO: Replace <FILL IN> with appropriate code
# Use np.kron, np.eye, and np.ones to recreate the arrays
sumByThreeKron = np.kron(np.eye(2), np.ones(shape=(1,3)))
sumByTwoKron = np.kron(np.eye(3), np.ones(shape=(1,2)))

print sumByThreeKron
print 'sumByThreeKron.dot(vector): {0}'.format(sumByThreeKron.dot(vector))
print '\n', sumByTwoKron
print 'sumByTwoKron.dot(vector): {0}'.format(sumByTwoKron.dot(vector))

# TEST Recreate with `np.kron` (4c)
Test.assertEquals(sumByThreeKron.shape, (2, 6), 'incorrect shape for sumByThreeKron')
Test.assertEquals(sumByTwoKron.shape, (3, 6), 'incorrect shape for sumByTwoKron')
Test.assertTrue(np.allclose(sumByThreeKron.dot(vector),  [3, 12]),
                'incorrect value for sumByThreeKron')
Test.assertTrue(np.allclose(sumByTwoKron.dot(vector), [1, 5, 9]),
                'incorrect value for sumByTwoKron')


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
(4d) Aggregate by time As we discussed in Part (4a), we would like to incorporate knowledge of our experimental setup into our analysis. To do this, we'll first study the temporal aspects of neural response, by aggregating our features by time. In other words, we want to see how different pixels (and the underlying neurons captured in these pixels) react in each of the 20 seconds after a new visual pattern is displayed, regardless of what the pattern is.  Hence, instead of working with the 240 features individually, we'll aggregate the original features into 20 new features, where the first new feature captures the pixel response one second after a visual pattern appears, the second new feature is the response after two seconds, and so on.
We can perform this aggregation using a map operation. First, build a multi-dimensional array $ \scriptsize \mathbf{T} $ that, when dotted with a 240-dimensional vector, sums every 20-th component of this vector and returns a 20-dimensional vector. Note that this exercise is similar to (4b).  Once you have created your multi-dimensional array $ \scriptsize \mathbf{T} $, use a map operation with that array and each time series to generate a transformed dataset. We'll cache and count the output, as we'll be using it again.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# Create a multi-dimensional array to perform the aggregation
T = np.array(np.tile([[0]*19+[1]]*20, 12))

print T

# Transform scaledData using T.  Make sure to retain the keys.
timeData = scaledData.map(lambda (x, y) : (x, T.dot(y)))

timeData.cache()
print timeData.count()
print timeData.first()
timeDataFirst = timeData.values().first()
print timeDataFirst[:2]

# TEST Aggregate by time (4d)
Test.assertEquals(T.shape, (20, 240), 'incorrect shape for T')
timeDataFirst = timeData.values().first()
timeDataFifth = timeData.values().take(5)[4]
Test.assertEquals(timeData.count(), 46460, 'incorrect length of timeData')
Test.assertEquals(timeDataFirst.size, 20, 'incorrect value length of timeData')
Test.assertEquals(timeData.keys().first(), (0, 0), 'incorrect keys in timeData')
Test.assertTrue(np.allclose(timeDataFirst[:2], [0.00802155, 0.00607693]),
                'incorrect values in timeData')
Test.assertTrue(np.allclose(timeDataFifth[-2:],[-0.00636676, -0.0179427]),
                'incorrect values in timeData')


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
(4f) Aggregate by direction
Next, let's perform a second type of feature aggregation so that we can study the direction-specific aspects of neural response, by aggregating our features by direction. In other words, we want to see how different pixels (and the underlying neurons captured in these pixels) react when the zebrafish is presented with 12 direction-specific patterns, ignoring the temporal aspect of the reaction.  Hence, instead of working with the 240 features individually, we'll aggregate the original features into 12 new features, where the first new feature captures the average pixel response to the first direction-specific visual pattern, the second new feature is the response to the second direction-specific visual pattern, and so on.
As in Part (4c), we'll design a multi-dimensional array $ \scriptsize \mathbf{D} $ that, when multiplied by a 240-dimensional vector, sums the first 20 components, then the second 20 components, and so on. Note that this is similar to exercise (4c).  First create $ \scriptsize \mathbf{D} $, then use a map operation with that array and each time series to generate a transformed dataset. We'll cache and count the output, as we'll be using it again.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# Create a multi-dimensional array to perform the aggregation
D = np.kron(np.eye(12), np.ones(shape=(1,20)))

# Transform scaledData using D.  Make sure to retain the keys.
directionData = scaledData.map(lambda (x, y) : (x, D.dot(y)))

directionData.cache()
print directionData.count()
print directionData.first()

# TEST Aggregate by direction (4f)
Test.assertEquals(D.shape, (12, 240), 'incorrect shape for D')
directionDataFirst = directionData.values().first()
directionDataFifth = directionData.values().take(5)[4]
Test.assertEquals(directionData.count(), 46460, 'incorrect length of directionData')
Test.assertEquals(directionDataFirst.size, 12, 'incorrect value length of directionData')
Test.assertEquals(directionData.keys().first(), (0, 0), 'incorrect keys in directionData')
Test.assertTrue(np.allclose(directionDataFirst[:2], [ 0.03346365,  0.03638058]),
                'incorrect values in directionData')
Test.assertTrue(np.allclose(directionDataFifth[:2], [ 0.01479147, -0.02090099]),
                'incorrect values in directionData')


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Part 1: Work through the steps of PCA on a sample dataset
Visualization 1: Two-dimensional Gaussians
Principal Component Analysis, or PCA, is a strategy for dimensionality reduction. To better understand PCA, we'll work with synthetic data generated by sampling from the two-dimensional Gaussian distribution.  This distribution takes as input the mean and variance of each dimension, as well as the covariance between the two dimensions.
In our visualizations below, we will specify the mean of each dimension to be 50 and the variance along each dimension to be 1.  We will explore two different values for the covariance: 0 and 0.9. When the covariance is zero, the two dimensions are uncorrelated, and hence the data looks spherical.  In contrast, when the covariance is 0.9, the two dimensions are strongly (positively) correlated and thus the data is non-spherical.  As we'll see in Parts 1 and 2, the non-spherical data is amenable to dimensionality reduction via PCA, while the spherical data is not.
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

def preparePlot(xticks, yticks, figsize=(10.5, 6), hideLabels=False, gridColor='#999999',
                gridWidth=1.0):
    """Template for generating the plot layout."""
    plt.close()
    fig, ax = plt.subplots(figsize=figsize, facecolor='white', edgecolor='white')
    ax.axes.tick_params(labelcolor='#999999', labelsize='10')
    for axis, ticks in [(ax.get_xaxis(), xticks), (ax.get_yaxis(), yticks)]:
        axis.set_ticks_position('none')
        axis.set_ticks(ticks)
        axis.label.set_color('#999999')
        if hideLabels: axis.set_ticklabels([])
    plt.grid(color=gridColor, linewidth=gridWidth, linestyle='-')
    map(lambda position: ax.spines[position].set_visible(False), ['bottom', 'top', 'left', 'right'])
    return fig, ax

def create2DGaussian(mn, sigma, cov, n):
    """Randomly sample points from a two-dimensional Gaussian distribution"""
    np.random.seed(142)
    return np.random.multivariate_normal(np.array([mn, mn]), np.array([[sigma, cov], [cov, sigma]]), n)

dataRandom = create2DGaussian(mn=50, sigma=1, cov=0, n=100)

# generate layout and plot data
fig, ax = preparePlot(np.arange(46, 55, 2), np.arange(46, 55, 2))
ax.set_xlabel(r'Simulated $x_1$ values'), ax.set_ylabel(r'Simulated $x_2$ values')
ax.set_xlim(45, 54.5), ax.set_ylim(45, 54.5)
plt.scatter(dataRandom[:,0], dataRandom[:,1], s=14**2, c='#d6ebf2', edgecolors='#8cbfd0', alpha=0.75)
pass

dataCorrelated = create2DGaussian(mn=50, sigma=1, cov=.9, n=100)

# generate layout and plot data
fig, ax = preparePlot(np.arange(46, 55, 2), np.arange(46, 55, 2))
ax.set_xlabel(r'Simulated $x_1$ values'), ax.set_ylabel(r'Simulated $x_2$ values')
ax.set_xlim(45.5, 54.5), ax.set_ylim(45.5, 54.5)
plt.scatter(dataCorrelated[:,0], dataCorrelated[:,1], s=14**2, c='#d6ebf2',
            edgecolors='#8cbfd0', alpha=0.75)
pass


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Part 2: Write a PCA function and evaluate PCA on sample datasets
(2a) PCA function
We now have all the ingredients to write a general PCA function.  Instead of working with just the top principal component, our function will compute the top $\scriptsize k$ principal components and principal scores for a given dataset. Write this general function pca, and run it with correlatedData and $\scriptsize k = 2$. Hint: Use results from Part (1c), Part (1d), and Part (1e).
Note: As discussed in lecture, our implementation is a reasonable strategy when $\scriptsize d $ is small, though more efficient distributed algorithms exist when $\scriptsize d $ is large.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def pca(data, k=2):
    """Computes the top `k` principal components, corresponding scores, and all eigenvalues.

    Note:
        All eigenvalues should be returned in sorted order (largest to smallest). `eigh` returns
        each eigenvectors as a column.  This function should also return eigenvectors as columns.

    Args:
        data (RDD of np.ndarray): An `RDD` consisting of NumPy arrays.
        k (int): The number of principal components to return.

    Returns:
        tuple of (np.ndarray, RDD of np.ndarray, np.ndarray): A tuple of (eigenvectors, `RDD` of
            scores, eigenvalues).  Eigenvectors is a multi-dimensional array where the number of
            rows equals the length of the arrays in the input `RDD` and the number of columns equals
            `k`.  The `RDD` of scores has the same number of rows as `data` and consists of arrays
            of length `k`.  Eigenvalues is an array of length d (the number of features).
    """
    # Correlated Covariance matrix
    covMatrix = estimateCovariance(data)
    #print "covMatrix", covMatrix

    # Eigen decomposition
    eigVals, eigVecs = eigh(covMatrix)
    #print "eigVecs", eigVecs

    # Sort the eigVals
    inds = np.argsort(eigVals)

    #print "Inds", inds

    # Obtain the k-top
    k_top = eigVecs[:,[inds[i] for i in range(k-1, -1, -1)]]
    if k==1:
        k_top = eigVecs[:,inds[len(inds)-1]]
    #print "k_top", k_top

    # k-scores
    k_scores = data.map(lambda x : x.dot(k_top))

    # Return the `k` principal components, `k` scores, and all eigenvalues
    return k_top, k_scores, eigVals

# Run pca on correlatedData with k = 2
topComponentsCorrelated, correlatedDataScoresAuto, eigenvaluesCorrelated = pca(correlatedData)

# Note that the 1st principal component is in the first column
print 'topComponentsCorrelated: \n{0}'.format(topComponentsCorrelated)
print ('\ncorrelatedDataScoresAuto (first three): \n{0}'
       .format('\n'.join(map(str,...


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Part 3:  Parse, inspect, and preprocess neuroscience data then perform PCA
Data introduction
A central challenge in neuroscience is understanding the organization and function of neurons, the cells responsible for processing and representing information in the brain. New technologies make it possible to monitor the responses of large populations of neurons in awake animals. In general, neurons communicate through electrical impulses that must be recorded with electrodes, which is a challenging process. As an alternative, we can genetically engineer animals so that their neurons express special proteins that flouresce or light up when active, and then use microscopy to record neural activity as images. A recently developed method called light-sheet microscopy lets us do this in a special, transparent animal, the larval zebrafish, over nearly its entire brain. The resulting data are time-varying images containing the activity of hundreds of thousands of neurons. Given the raw data, which is enormous, we want to find compact spatial and temporal patterns: Which groups of neurons are active together? What is the time course of their activity? Are those patterns specific to particular events happening during the experiment (e.g. a stimulus that we might present). PCA is a powerful technique for finding spatial and temporal patterns in these kinds of data, and that's what we'll explore here!
(3a) Load neuroscience data
In the next sections we will use PCA to capture structure in neural datasets. Before doing the analysis, we will load and do some basic inspection of the data. The raw data are currently stored as a text file. Every line in the file contains the time series of image intensity for a single pixel in a time-varying image (i.e. a movie). The first two numbers in each line are the spatial coordinates of the pixel, and the remaining numbers are the time series. We'll use first() to inspect a single row, and print just the first 100 characters.
Can you write Python code for it?

import os
baseDir = os.path.join('data')
inputPath = os.path.join('cs190', 'neuro.txt')

inputFile = os.path.join(baseDir, inputPath)

lines = sc.textFile(inputFile)
print lines.first()[0:100]

# Check that everything loaded properly
assert len(lines.first()) == 1397
assert lines.count() == 46460


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Visualization 10: Top two components by direction
Again, let's view the scores from the first two PCs as a composite image.  When we preprocess by averaging across time (group by direction), and then perform PCA, we are only looking at variability related to stimulus direction. As a result, if neurons appear similar -- have similar colors -- in the image, it means that their responses vary similarly across directions, regardless of how they evolve over time. In the image below, we see a different pattern of similarity across regions of the brain.  Moreover, regions on either side of the midline are colored differently, which suggests that we are looking at a property, direction selectivity, that has a different representation across the two sides of the brain.
Can you write Python code for it?

scoresDirection = np.vstack(directionScores.collect())
imageOneDirection = scoresDirection[:,0].reshape(230, 202).T
imageTwoDirection = scoresDirection[:,1].reshape(230, 202).T
brainmap = polarTransform(2, [imageOneDirection, imageTwoDirection])
# with thunder: Colorize(cmap='polar', scale=2).transform([imageOneDirection, imageTwoDirection])

# generate layout and plot data
fig, ax = preparePlot(np.arange(0, 10, 1), np.arange(0, 10, 1), figsize=(9.0, 7.2), hideLabels=True)
ax.grid(False)
image = plt.imshow(brainmap, interpolation='nearest', aspect='auto')
pass


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Visualization 3: Three-dimensional data
So far we have worked with two-dimensional data. Now let's generate three-dimensional data with highly correlated features. As in Visualization 1, we'll create samples from a multivariate Gaussian distribution, which in three dimensions requires us to specify three means, three variances, and three covariances.
In the 3D graphs below, we have included the 2D plane that corresponds to the top two principal components, i.e. the plane with the smallest euclidean distance between the points and itself. Notice that the data points, despite living in three-dimensions, are found near a two-dimensional plane: the left graph shows how most points are close to the plane when it is viewed from its side, while the right graph shows that the plane covers most of the variance in the data.  Note that darker blues correspond to points with higher values for the third dimension.
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D

m = 100
mu = np.array([50, 50, 50])
r1_2 = 0.9
r1_3 = 0.7
r2_3 = 0.1
sigma1 = 5
sigma2 = 20
sigma3 = 20
c = np.array([[sigma1 ** 2, r1_2 * sigma1 * sigma2, r1_3 * sigma1 * sigma3],
             [r1_2 * sigma1 * sigma2, sigma2 ** 2, r2_3 * sigma2 * sigma3],
             [r1_3 * sigma1 * sigma3, r2_3 * sigma2 * sigma3, sigma3 ** 2]])
np.random.seed(142)
dataThreeD = np.random.multivariate_normal(mu, c, m)

from matplotlib.colors import ListedColormap, Normalize
from matplotlib.cm import get_cmap
norm = Normalize()
cmap = get_cmap("Blues")
clrs = cmap(np.array(norm(dataThreeD[:,2])))[:,0:3]

fig = plt.figure(figsize=(11, 6))
ax = fig.add_subplot(121, projection='3d')
ax.azim=-100
ax.scatter(dataThreeD[:,0], dataThreeD[:,1], dataThreeD[:,2], c=clrs, s=14**2)

xx, yy = np.meshgrid(np.arange(-15, 10, 1), np.arange(-50, 30, 1))
normal = np.array([0.96981815, -0.188338, -0.15485978])
z = (-normal[0] * xx - normal[1] * yy) * 1. / normal[2]
xx = xx + 50
yy = yy + 50
z = z + 50

ax.set_zlim((-20, 120)), ax.set_ylim((-20, 100)), ax.set_xlim((30, 75))
ax.plot_surface(xx, yy, z, alpha=.10)

ax = fig.add_subplot(122, projection='3d')
ax.azim=10
ax.elev=20
#ax.dist=8
ax.scatter(dataThreeD[:,0], dataThreeD[:,1], dataThreeD[:,2], c=clrs, s=14**2)

ax.set_zlim((-20, 120)), ax.set_ylim((-20, 100)), ax.set_xlim((30, 75))
ax.plot_surface(xx, yy, z, alpha=.1)
plt.tight_layout()
pass


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Visualization 4: 2D representation of 3D data
See the 2D version of the data that captures most of its original structure.  Note that darker blues correspond to points with higher values for the original data's third dimension.
Can you write Python code for it?

scoresThreeD = np.asarray(threeDScores.collect())

# generate layout and plot data
fig, ax = preparePlot(np.arange(20, 150, 20), np.arange(-40, 110, 20))
ax.set_xlabel(r'New $x_1$ values'), ax.set_ylabel(r'New $x_2$ values')
ax.set_xlim(5, 150), ax.set_ylim(-45, 50)
plt.scatter(scoresThreeD[:,0], scoresThreeD[:,1], s=14**2, c=clrs, edgecolors='#8cbfd0', alpha=0.75)
pass


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Visualization 5: Pixel intensity
Let's now see how a random pixel varies in value over the course of the time series.  We'll visualize a pixel that exhibits a standard deviation of over 100.
Can you write Python code for it?

example = rawData.filter(lambda (k, v): np.std(v) > 100).values().first()

# generate layout and plot data
fig, ax = preparePlot(np.arange(0, 300, 50), np.arange(300, 800, 100))
ax.set_xlabel(r'time'), ax.set_ylabel(r'flouresence')
ax.set_xlim(-20, 270), ax.set_ylim(270, 730)
plt.plot(range(len(example)), example, c='#8cbfd0', linewidth='3.0')
pass


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Visualization 6: Normalized data
Now that we've normalized our data, let's once again see how a random pixel varies in value over the course of the time series.  We'll visualize a pixel that exhibits a standard deviation of over 0.1.  Note the change in scale on the y-axis compared to the previous visualization.
Can you write Python code for it?

example = scaledData.filter(lambda (k, v): np.std(v) > 0.1).values().first()

# generate layout and plot data
fig, ax = preparePlot(np.arange(0, 300, 50), np.arange(-.1, .6, .1))
ax.set_xlabel(r'time'), ax.set_ylabel(r'flouresence')
ax.set_xlim(-20, 260), ax.set_ylim(-.12, .52)
plt.plot(range(len(example)), example, c='#8cbfd0', linewidth='3.0')
pass


I am working on the file "Lab05/ML_lab5_pca_student.ipynb".
The first task is:
Visualization 9: Top two components by time
Let's view the scores from the first two PCs as a composite image. When we preprocess by aggregating by time and then perform PCA, we are only looking at variability related to temporal dynamics. As a result, if neurons appear similar -- have similar colors -- in the resulting image, it means that their responses vary similarly over time, regardless of how they might be encoding direction. In the image below, we can define the midline as the horizontal line across the middle of the brain.  We see clear patterns of neural activity in different parts of the brain, and crucially note that the regions on either side of the midline are similar, which suggests that temporal dynamics do not differ across the two sides of the brain.
Can you write Python code for it?

scoresTime = np.vstack(timeScores.collect())
imageOneTime = scoresTime[:,0].reshape(230, 202).T
imageTwoTime = scoresTime[:,1].reshape(230, 202).T
brainmap = polarTransform(3, [imageOneTime, imageTwoTime])

# generate layout and plot data
fig, ax = preparePlot(np.arange(0, 10, 1), np.arange(0, 10, 1), figsize=(9.0, 7.2), hideLabels=True)
ax.grid(False)
image = plt.imshow(brainmap,interpolation='nearest', aspect='auto')
pass


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
Array vector for the first document
Can you write Python code for it?


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
Create an English stemmer
http://www.nltk.org/howto/stem.html for general intro.
http://www.nltk.org/api/nltk.stem.html for more details (including languages covered).
Can you write Python code for it?

s = nltk.stem.SnowballStemmer('english')
s.stem("cats"), s.stem("ran"), s.stem("jumped")


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
How many terms were extracted? use get_feature_names()
Can you write Python code for it?


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
If we modify build_analyzer() to apply the NLTK stemmer to the output of default build_analyzer(), we get a version that does stemming as well:
Can you write Python code for it?

import nltk.stem
english_stemmer = nltk.stem.SnowballStemmer('english')
class StemmedCountVectorizer(CountVectorizer):
    def build_analyzer(self):
        analyzer=super(StemmedCountVectorizer, self).build_analyzer()
        return lambda doc:(english_stemmer.stem(w) for w in analyzer(doc))


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
Integrating NLTK with SciKit's vectorizer
NLTK Stemmer
The stemmer can be used to stem documents before feeding into SciKit's vectorizer, thus obtaining a more compact index.
One way to do this is to define a new class StemmedCountVectorizer extending CountVectorizer by redifining the method build_analyzer() that handles preprocessing and tokenization.
http://scikitlearn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
build_analyzer() takes a string as input and outputs a list of tokens.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(stop_words="english")
analyze = vectorizer.build_analyzer()
analyze("John bought carrots and potatoes")


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
More preprocessing
For stemming and more advanced preprocessing, supplement SciKit Learn with another Python library, NLTK. Up next.
More advanced preprocessing with NLTK
NLTK is described in detail in a book by Bird, Klein and Loper available online:
http://www.nltk.org/book_1ed/ for version 2.7 of python
About NLTK

It is not the best
It is very easy to use

You should read the book linked above to get familiar with the package and with text preprocessing.
Can you write Python code for it?


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
NLTK for text analytics

NERs
Sentiment analysis
Extracting information from social media.
Can you write Python code for it?

from nltk.tokenize import word_tokenize
text = word_tokenize("And now for something completely different")

nltk.pos_tag(text)


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
Notes
You should always experiment and see if it is good to use stemming with your problem set. It might not be the best thing to do.
SOLR works for processing larger datasets, since Python and SciKit-Learn become less effective, and more industrial strength software is required. One example of such software is Apache SOLR, an open source indexing package available from:
http://lucene.apache.org/solr/
It produces Lucene-style indices that can be used by text analytics packages such as Mahout.
Elastic http://www.elastic.co/
Can you write Python code for it?

!ipython nbconvert --to script Lab1\ Text\ processing\ with\ python.ipynb


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
Number of times word "hard" occurs
Can you write Python code for it?


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
So now we can create an instance of this class:
Can you write Python code for it?

stem_vectorizer = StemmedCountVectorizer(min_df=1,
                                        stop_words='english')
stem_analyze = stem_vectorizer.build_analyzer()
Y = stem_analyze("John bought carrots and potatoes")

[tok for tok in Y]


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
Text processing with Scikit learn
We can use CountVectorizer to extract a bag of words representation from a collection of documents, using the SciKit-Learn method fit_transform. We will use a list of strings as documents.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer(min_df = 1)
content = ["How to format my hard disk", " Hard disk format problems "]

# fit_transform returns array of two rows, one per 'document'.
# each row has 7 elements, each element being the number of items
# a given feature occurred in that document.
X = vectorizer.fit_transform(content)

vectorizer.get_feature_names(), X.toarray()


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
Use this vectorizer to extract features
Compare this result to around 35,000 features we obtained using the unstemmed version.
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups
categories = ['alt.atheism', 'soc.religion.christian',
             'comp.graphics', 'sci.med']
twenty_train = fetch_20newsgroups(subset='train',
                                 categories=categories,
                                 shuffle=True, random_state=42)
train_counts = stem_vectorizer.fit_transform(twenty_train.data)

len(stem_vectorizer.get_feature_names())
print train_counts[:6]


I am working on the file "Lab1 Text processing with python.ipynb".
The first task is:
Using the 20 Newsgroups dataset
We are going to fetch just some categories so that it doesn't take that long to download the docs.
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups
categories = ['alt.atheism', 'soc.religion.christian',
              'comp.graphics', 'sci.med']
twenty_train = fetch_20newsgroups(subset='train',
                                 categories=categories, shuffle=True,
                                 random_state=42)


I am working on the file "Lab1-babypython.ipynb".
The first task is:
But perhaps we want to access Hamlet word by word and not line by line
Can you write Python code for it?

hamletfile=open("hamlet.txt")
hamlettext=hamletfile.read()
hamletfile.close()
hamlettokens=hamlettext.split()#split with no arguments splits on whitespace
len(hamlettokens)

print hamlettokens[:1000]


I am working on the file "Lab1-babypython.ipynb".
The first task is:
Dictionaries
These are the bread and butter. You will use them a lot. They even duck like lists. But be careful how.
Can you write Python code for it?

adict={'one':1, 'two': 2, 'three': 3}
print [i for i in adict], [(k,v) for k,v in adict.items()], adict.values()


I am working on the file "Lab1-babypython.ipynb".
The first task is:
Functions
Functions are even more the bread and butter. You'll see them as methods on objects, or standing alone by themselves.
Can you write Python code for it?

def square(x):
    return(x*x)
def cube(x):
    return x*x*x
square(5),cube(5)

print square, type(cube)


I am working on the file "Lab1-babypython.ipynb".
The first task is:
In Python, functions are "first-class". This is just a fancy way of saying, you can pass functions to other functions
Can you write Python code for it?

def sum_of_anything(x,y,f):
    print x,y,f
    return(f(x) + f(y))
sum_of_anything(3,4,square)


I am working on the file "Lab1-babypython.ipynb".
The first task is:
One can use the with syntax which cretaes a context. The file closing is then done automatically for us.
Can you write Python code for it?

with open("hamlet.txt") as hamletfile:
    hamlettext=hamletfile.read()
    hamlettokens=hamlettext.split()
    print len(hamlettokens)


I am working on the file "Lab1-babypython.ipynb".
The first task is:
Open files behave like lists too! Here we get each line in the file and find its length, using the comprehension syntax to put these lengths into a big list.
Can you write Python code for it?

linelengths=[len(line) for line in open("hamlet.txt")]#poor code as we dont close the file
print linelengths

sum(linelengths), np.mean(linelengths), np.median(linelengths), np.std(linelengths)


I am working on the file "Lab1-babypython.ipynb".
The first task is:
Python and Iteration (and files)
In working with python I always remember: a python is a duck.
What I mean is, python has a certain way of doing things. For example lets call one of these ways listiness. Listiness works on lists, dictionaries, files, and a general notion of something called an iterator.
But first, lets introduce the notion of a comprehension. Its a way of constructing a list
Can you write Python code for it?

alist=[1,2,3,4,5]
asquaredlist=[i*i for i in alist]
asquaredlist


I am working on the file "Lab1-babypython.ipynb".
The first task is:
Python has some nifty functions like enumerate and zip. The former gives a list of tuples with each tuple of the form (index, value), while the latter takes elements from each list and outs them together into a tuple, thus creating a list of tuples. The first is a duck, but the second isnt.
Can you write Python code for it?


I am working on the file "Lab1-babypython.ipynb".
The first task is:
Strings
Basically they behave like immutable lists
Can you write Python code for it?

lastword=hamlettokens[-1]
print(lastword)

lastword[-2]="k"#cant change a part of a string

lastword[-2]

You can join a list with a separator to make a string.

wierdstring=",".join(hamlettokens)
wierdstring[:1000]


I am working on the file "Lab1-babypython.ipynb".
The first task is:
The Python Language
Lets talk about using Python as a calculator...
Can you write Python code for it?


I am working on the file "Lab1-babypython.ipynb".
The first task is:
There are roughly 32,000 words in Hamlet.
The indexing of lists
Can you write Python code for it?

print hamlettext[:1000]#first 1000 characters from Hamlet.

print hamlettext[-1000:]#and last 1000 characters from Hamlet.


I am working on the file "Lab1-babypython.ipynb".
The first task is:
We find the 100 most used words...
Can you write Python code for it?

L=sorted(tokendict, key= lambda (k,v):v, reverse=True)[:100]
L


I am working on the file "Lab1-babypython.ipynb".
The first task is:
We then find a unique set of words using python's set data structure. We count how often those words occured usinf the count method on lists.
Can you write Python code for it?

uniquelctokens=set(hamletlctokens)
print uniquelctokens

tokendict={}
for ut in uniquelctokens:
    tokendict[ut]=hamletlctokens.count(ut)


I am working on the file "Lab1-babypython.ipynb".
The first task is:
While loops are also supported. continue continues to the next iteration of the loop skipping all the code below, while break breaks out of it.
Can you write Python code for it?

i=0
while i < 10:
    print i
    i=i+1
    if i < 5:
        continue
    else:
        break


I am working on the file "Lab1-babypython.ipynb".
The first task is:
You can construct then nicely using the function dict.
Can you write Python code for it?


I am working on the file "Lab1-babypython.ipynb".
The first task is:
and conversion to json
Can you write Python code for it?

import json

s=json.dumps(mydict)
print s

json.loads(s)


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
All together now
Lets see what hamlet gives us. We convert all words to lower-case
Can you write Python code for it?

hamletlctokens=[word.lower() for word in hamlettokens]
hamletlctokens.count("thou")


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Booleans and Control-flow
Lets test for belonging...
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Dictionaries
These are the bread and butter. You will use them a lot. They even duck like lists. But be careful how.
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Exceptions
This is the way to catch errors.
Can you write Python code for it?

try:
    f(1)#takes atleast 2 arguments
except:
    import sys
    print sys.exc_info()


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Functions
Functions are even more the bread and butter. You'll see them as methods on objects, or standing alone by themselves.
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Here is how we can print things. Something on the last line by itself is returned as the output value.
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
In Python, functions are "first-class". This is just a fancy way of saying, you can pass functions to other functions
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
One can use the with syntax which cretaes a context. The file closing is then done automatically for us.
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Open files behave like lists too! Here we get each line in the file and find its length, using the comprehension syntax to put these lengths into a big list.
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Python and Iteration (and files)
In working with python I always remember: a python is a duck.
What I mean is, python has a certain way of doing things. For example lets call one of these ways listiness. Listiness works on lists, dictionaries, files, and a general notion of something called an iterator.
But first, lets introduce the notion of a comprehension. Its a way of constructing a list
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Python functions can have positional arguments and keyword arguments. Positional arguments are stored in a tuple, and keyword arguments in a dictionary. Note the "starred" syntax
Can you write Python code for it?

def f(a,b,*posargs,**dictargs):
    print "got",a,b,posargs, dictargs
    return a
print f(1,3)
print f(1,3,4,d=1,c=2)


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Python supports if/elif/else clauses for multi-way conditionals
Can you write Python code for it?

def do_it(x):
    if x==1:
        print "One"
    elif x==2:
        print "Two"
    else:
        print x
do_it(1)

do_it(2), do_it(3)


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
Someone realized that design flaw and created izip.
Can you write Python code for it?

from itertools import izip
izip(alist, asquaredlist)

print enumerate(asquaredlist)

[k for k in enumerate(asquaredlist)]


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
The Python Language
Lets talk about using Python as a calculator...
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
The keys do not have to be strings. From python 2.7 you can use dictionary comprehensions as well
Can you write Python code for it?

mydict ={k:v for (k,v) in zip(alist, asquaredlist)}
mydict


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
There are roughly 32,000 words in Hamlet.
The indexing of lists
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
We can obtain the type of a variable, and use boolean comparisons tontest these types.
Can you write Python code for it?

a=5.0/6.0
print(a)
print type(a)

import types
type(a)==types.FloatType

type(a)==types.IntType


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
We then find a unique set of words using python's set data structure. We count how often those words occured usinf the count method on lists.
Can you write Python code for it?

uniquelctokens=set(hamletlctokens)

tokendict={}
for ut in uniquelctokens:
    tokendict[ut]=hamletlctokens.count(ut)


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
While loops are also supported. continue continues to the next iteration of the loop skipping all the code below, while break breaks out of it.
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
You can construct then nicely using the function dict.
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
and conversion to json
Can you write Python code for it?


I am working on the file "Lab1-babypython_original.ipynb".
The first task is:
range and xrange get the list of integers upto N. But xrange behaves like an iterator. The reason for this is that there is no point generaing all os a million integers. We can just add 1 to the previous one and save memory. So we trade off storage for computation.
Can you write Python code for it?

mylist=[]
for i in xrange(10):
    mylist.append(i)
mylist


I am working on the file "Lab1-git.ipynb".
The first task is:
Aha, now we are set my with both a remote and a local for mybranch1
Can you write Python code for it?

%%bash
cd /tmp/Testing
git branch -avv

%%bash
cd /tmp/Testing
git checkout master


I am working on the file "Lab1-git.ipynb".
The first task is:
Aha: one commit came from the course upstream, and one was a merge commit. In the case you had edited the README.md at the same time and comitted locally, you would have been asked to resolve the conflict in the merge (the second case above).
Lets push these changes to the origin now
Can you write Python code for it?

%%bash
cd /tmp/Testing
git push
git status


I am working on the file "Lab1-git.ipynb".
The first task is:
Changing only one file
The last git command I want to talk about, since it might come handy in the course, is the situation in which you dont want to merge an entire branch from the upstream, but just one file from it. There is a direct use case for it. Indeed, suppose I've made an error in this lab and want to correct it. So i fix it in the upstream. In the meanwhile you have edited some other files. You dont care to manually ignore my older copies of those files. So you want to fix just one file from this new branch. This is how you do it.
First you fetch from the remote (i ought to be doing this with the course remote but just want to illustrate it here)
Can you write Python code for it?

%%bash
cd /tmp/Testing
git fetch origin

%%bash
cd /tmp/Testing
git branch -avv

%%bash
cd /tmp/Testing
git checkout origin/mybranch1 -- world.md


I am working on the file "Lab1-git.ipynb".
The first task is:
Indeed, the way git works is by creating copies of remote branches locally. Then it just compares to these "copy" branches to see what changes have been made.
Sometimes we want to merge the changes in...precisely we want to merge the change from remotes/course/master in. In HW0 and later here, we'll show you a case where you want to simply create another branch.

merge

Having multiple branches is fine, but at some point, you'll want to combine the changes that you've made with those made by others. This is called merging.
There are two general cases when merging two branches: first, the two branches are different but the changes are in unrelated places; and second, the two branches are different and the changes are in the same locations in the same files.
The first scenario is easy. Git will simply apply both sets of changes to the appropriate places and put the resulting files into the staging area for you. Then you can commit the changes and push them back to the upstream repository. Your collaborator does the same, and everyone sees everything.
The second scenario is more complicated. Let's say the two changes set some variable to different values. Git can't know which is the correct value. One solution would be to simply use the more recent change, but this very easily leads to self-inconsistent programs. A more conservative solution, and the one git uses, is to simply leave the decision to the user. When git detects a conflict that it cannot resolve, merge fails, and git places a modified version of the offending file in your project directory. This is important: the file that git puts into your directory is not actually either of the originals. It is a new file that has special markings around the locations that conflicted. We shall not consider this case in this lab, deferring it to a time just before you work with other collaborators on your projects.
Lets merge in the changes from course/master: (The next 2-3 inputs only make sense if cs109/master is edited in-between)
Can you write Python code for it?

%%bash
cd /tmp/Testing
git merge course/master
git status


I am working on the file "Lab1-git.ipynb".
The first task is:
Jump onto the mybranch1 branch...
Can you write Python code for it?

%%bash
cd /tmp/Testing
git checkout mybranch1
git branch


I am working on the file "Lab1-git.ipynb".
The first task is:
Making changes
Ok! Enough poking around. Lets get down to business and add some files into our folder.
Now let's say that we want to add a new file to the project. The canonical sequence is "edit–add–commit–push".
Can you write Python code for it?

%%bash
cd /tmp/Testing
echo '# Hello world rahuldave' > hello.md
git status


I am working on the file "Lab1-git.ipynb".
The first task is:
Note that in git versions > 2.0, which you must use, git push will push the current branch to its appropriate remote, all of which can be seen through git branch -avv or by looking at the config file.
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
Pay close attention to the text above. It says we are on the master branch of our local repository, and that this branch is up-to-date with the master branch of the upstream  repository or remote named origin. We know this as clone brings down a copy of the remote branch:  "origin/master" represents the local copy of the branch that came from the upstream repository (nicknamed "origin" in this case). Branches are different, co-existing versions of your project. Here we have encountered two of them, but remember there is a third one in the repository we forked from, and perhaps many more, depending on who else made these forks (as well as my original clones of cs109/Testing).
This sounds like a recipe for disaster, but in a sense, branches are just like commits. They represent a snapshot of the project, by someone, at some particular point in time. In general you will only care about your own branches, and those of the "parent" remotes you forked/cloned from.
Configuration Information is stored in a special file config, in a hidden folder called .git in your working directory. (The index and the local repository are stored there as well...more on that in  a bit)
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
Poking around
We have a nice smelling fresh repository. Lets look around it.
log
Log tells you all the changes that have occured in this project as of now...
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
Remotes and fetching from them
If you're working with other people, then it's possible that they have made changes to the remote repository between the time you first cloned it and now. push will fail.
In our particular case of the Testing repository, this is not going to happen, since you just cloned it and presumably havent invited anyone to collaborate with you on it (you can do this from the settings link on the right side of the repository page).
However you can imagine that the original repository cs109/testing, which you are now divorced from, has changed, and that you somehow want to pull those changes in.
That's where the next two commands come in.

remote
We have seen so far that our repository has one "remote", or upstream repository, which has been identified with the word origin, as seen in .git/config. We now wish to add another remote, which we shall call course, which points to the original repository we forked from. We want to do this to pull in changes, in-case something changed there.
Can you write Python code for it?

%%bash
cd /tmp/Testing
git remote add course git@github.com:cs109/Testing.git
cat .git/config


I am working on the file "Lab1-git.ipynb".
The first task is:
The git commit -m... version is just a way to specify a commit message without opening a text editor (ipython notebook can't handle it). Otherwise you just say git commit or git commit -a (if you added a new file to the current branch on the repository)
Now we see that our branch, "master", has one more commit than the "origin/master" branch, the local copy of the branch that came from the upstream repository (nicknamed "origin" in this case). Let's push the changes.
push

The push command takes the changes you have made to your local repository and attempts to update a remote repository with them. If you're the only person working with both of these (which is how a solo GitHub project would work), then push should always succeed.
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
This tutorial is largely based on the repository:
git@github.com:rdadolf/git-and-github.git
which was created for IACS's ac297r course, By Robert Adolf. Most of the credit goes to him.
Version control is a way of tracking the change history of a project. Even if you have never used a version control tool, you've probably already done it manually: copying and renaming project folders ("paper-1.doc", "paper-2.doc", etc.) is a form of version control.
Git is a tool that automates and enhances a lot of the tasks that arise when dealing with larger, longer-living, and collaborative projects. It's also become the common underpinning to many popular online code repositories, GitHub being the most popular.
We'll go over the basics of git, but we should point out that a lot of talented people have given git tutorials, and we won't do any better than they have. In fact, if you're interested in learning git deeply and have some time on your hands, I suggest you stop reading this and instead read the Git Book. Scott Chacon and Ben Straub have done a tremendous job, and if you want to understand both the interfaces and the mechanisms behind git, this is the place to start.
In this document, we'll go over the concepts of git that are relevant to this course. You will be using git in a particular fashion, in this course, which is described in Homework 0. Homework 0 also contains details about how to install git; so if you havent done so, please do so and come back here.
Table of Contents

Lab 1: Version Control with Git
Why should you use version control?
Git Basics
Common Tasks in the version control of files.
Forking a repository
Cloning a repository
Poking around
Making changes
Remotes and fetching from them
Branching.
Recovering from a mistake
Changing only one file

The Homework git flow
The Lab git flow
Creating a new repository
Github Pages
Automatically Generated Project Pages
Static Project Pages

Git habits

At this point, we assume you have installed git as described in HW0 and have followed the instructions...
Can you write Python code for it?

%%bash
cd /tmp
rm -rf Testing #remove if it exists
git clone git@github.com:rahuldave/Testing.git

%%bash
ls /tmp/Testing


I am working on the file "Lab1-git.ipynb".
The first task is:
We seem to be ahead of our upstream-tracking repository by 2 commits..why?
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
We've added a file to the working directory, but it hasn't been staged yet.

add

When you've made a change to a set of files and are ready to create a commit, the first step is to add all of the changed files to the staging area. Add does that. Remember that what you see in the filesystem is your working directory, so the way to see what's in the staging area is with the status command. This also means that if you add something to the staging area and then edit it again, you'll also need to add the file to the staging area again if you want to remember the new changes.
Can you write Python code for it?

%%bash
cd /tmp/Testing
git add hello.md
git status


I am working on the file "Lab1-git.ipynb".
The first task is:
Why does the syntax have origin/mybranch1? Remember that multiple remotes may have the same branch...so you want to be specific. So we check out origin's mybranch1, and only want world.md.
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
You are always working on a given branch in a repository. Typically this is master. More on this later..You can know which branch you are on by typing git branch. The strred one is the one you are on.
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
You can combine a fetch and a merge together by simply doing a git pull. This will fail if you and your collaborator have worked on the same file (since you will have to merge by hand), but is a great shortcut when the files  worked on are different. I use it all the times on a personal level too, to shift work between two different machines, as long as I am not working on both at the same time. The usual use case is day work on a work computer, and then evening work at home on the laptop. Read the docs if you are interested.
Branching.
As you might have seen by now, everything in git is a branch. We have branches on remote (upstream) repositories, copies of remote branches in our local repository, and branches on local repositories which (so far) track remote branches (or more precisely local copies of remote repositories)
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
You could have created this branch using git checkout -b mybranch1. Lets see the status here.
Can you write Python code for it?


I am working on the file "Lab1-git.ipynb".
The first task is:
oops, that was not what we wanted. We undo it using git reset, to go back to the state at the last commit.
Can you write Python code for it?

%%bash
cd /tmp/Testing
git reset --hard origin/master

%%bash
cd /tmp/Testing
git status


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Aha, now we are set my with both a remote and a local for mybranch1
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
And all of these branches are nothing but commit-streams in disguise, as can be seen above. Its a very simple model which leads to a lot of interesting version control patterns.
Since branches are so light-weight, the recommended way of working on software using git is to create a new branch for each new feature you add, test it out, and if good, merge it into master. Then you deploy the software from master. But we have been using branches under the hood. Lets now lift the hood.

branch

Branches can also be created manually, and they are a useful way of organizing unfinished changes.
The branch command has two forms. The first:
git branch
simply lists all of the branches in your local repository. If you run it without having created any branches, it will list only one, called master. This is the default branch. You have seen the use of git branch -avv to show all branches.
The other form creates a branch with a given name:
It's important to note that the other branch is not active. If you make changes, they will still apply to the master branch, not my-new-branch. To change this, you need the next command.

checkout

Checkout switches the active branch. Since branches can have different changes, checkout may make the working directory look very different. For instance, if you have added new files to one branch, and then check another branch out, those files will no longer show up in the directory. They are still stored in the .git folder, but since they only exist in the other branch, they cannot be accessed until you check out the original branch.
You can combine creating a new branch and checking it out with the shortcut:
Ok so lets try this out on our repository....
Can you write Python code for it?

%%bash
cd /tmp/Testing
git branch mybranch1


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Each one of these "commits" is a SHA hash. It uniquely identifies all actions that have happened to this repository previously. We shall soon see how to add our own actions in. In the meanwhile, lets see the "status"of our working directory.
If you ever need help on a command, you can find the git man pages by hyphenating git and the command name:

status

Status is your window into the current state of your project. It can tell you which files you have changed and which files you currently have in your staging area.
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Indeed, the way git works is by creating copies of remote branches locally. Then it just compares to these "copy" branches to see what changes have been made.
Sometimes we want to merge the changes in...precisely we want to merge the change from remotes/course/master in. In HW0 and later here, we'll show you a case where you want to simply create another branch.

merge

Having multiple branches is fine, but at some point, you'll want to combine the changes that you've made with those made by others. This is called merging.
There are two general cases when merging two branches: first, the two branches are different but the changes are in unrelated places; and second, the two branches are different and the changes are in the same locations in the same files.
The first scenario is easy. Git will simply apply both sets of changes to the appropriate places and put the resulting files into the staging area for you. Then you can commit the changes and push them back to the upstream repository. Your collaborator does the same, and everyone sees everything.
The second scenario is more complicated. Let's say the two changes set some variable to different values. Git can't know which is the correct value. One solution would be to simply use the more recent change, but this very easily leads to self-inconsistent programs. A more conservative solution, and the one git uses, is to simply leave the decision to the user. When git detects a conflict that it cannot resolve, merge fails, and git places a modified version of the offending file in your project directory. This is important: the file that git puts into your directory is not actually either of the originals. It is a new file that has special markings around the locations that conflicted. We shall not consider this case in this lab, deferring it to a time just before you work with other collaborators on your projects.
Lets merge in the changes from course/master: (The next 2-3 inputs only make sense if cs109/master is edited in-between)
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Jump onto the mybranch1 branch...
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Making changes
Ok! Enough poking around. Lets get down to business and add some files into our folder.
Now let's say that we want to add a new file to the project. The canonical sequence is "edit–add–commit–push".
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Note that in git versions > 2.0, which you must use, git push will push the current branch to its appropriate remote, all of which can be seen through git branch -avv or by looking at the config file.
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Note that the file as automatically added to the index. We'll commit and push.
Can you write Python code for it?

%%bash
cd /tmp/Testing
git commit -m "want world in master"

%%bash
cd /tmp/Testing
git status

%%bash
cd /tmp/Testing
git push


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Notice that it is bootstrapped off the master branch and has the same files.
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Notice that the master branch only tracks the same branch on the origin remote. We havent set up any connection with the course remote as yet.
Now lets figure out how to get changes from an upstream repository, be it our origin upstream that a collaborator has pushed too, or another course remote to which your memory-added head TF has posted a change!

fetch

Let's say a collaborator has pushed changes to your shared upstream repository while you were editing. Their local repository and the upstream repository now both contain their changes, but your local repository does not. To update your local repository, you run fetch.
But what if you've committed changes in the meantime? Does your local repository contain your changes or theirs? The answer is that it contains a record of both, but they are kept separate. Remember that git repositories are not copies of your project files. They store all the contents of your files, along with a bunch of metadata, but in its own internal format. This is one of the reasons.
Let's say that you and your collaborator both edited the same line of the same file at the same time in different ways. On your respective machines, you both add and commit your different changes, and your collaborator pushes theirs to the upstream repository. When you run fetch, git adds a record of their changes to your local repository alongside your own. These are called branches, and they represent different, coexisting versions of your project. The fetch command adds your collaborator's branch to your local repository, but keeps yours as well.
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Now our file is in the staging area (Index), waiting to be committed.
I will sometimes simply use git add . in the top level of the repository. This adds all new files and changed files to the index, and is particularly useful if I have created multiple new files.

commit

When you're satisfied with the changes you've added to your staging area, you can commit those changes to your local repository with the commit command. Those changes will have a permanent record in the repository from now on.
Every commit has two features you should be aware of. The first is a hash. This is a unique identifier for all of the information about that commit, including the code changes, the timestamp, and the author. The second is a commit message. This is text that you can (and should) add to a commit to describe what the changes were.
Good commit messages are important.
Commit messages are a way of quickly telling your future self (and your collaborators) what a commit was about. For even a moderately sized project, digging through tens or hundreds of commits to find the change you're looking for is a nightmare without friendly summaries.
By convention, commit messages start with a single-line summary, then an empty line, then a more comprehensive description of the changes.
This is an okay commit message. The changes are small, and the summary is sufficient to describe what happened.
This is better. The summary captures the important information (major shift, direct vs. helper), and the full commit message describes what the high-level changes were.
This. Don't do this.
Can you write Python code for it?

%%bash
cd /tmp/Testing
git commit -m "Said hello to myself"
git status


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Ok we have committed. Lets try to push!
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Oops that failed. Why? git didnt know what to push to on origin, and didnt want to assume we wanted to call the branch mybranch1 on the remote. We need to tell that to git explicitly, as it tells us to.
Can you write Python code for it?

%%bash
cd /tmp/Testing
git push --set-upstream origin mybranch1


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Recovering from a mistake
Now suppose for a second that this mybranch1 was created by someone else. We wanted to get it down using fetch and play. But we called pull, which did an automatic merge for us.
Can you write Python code for it?

%%bash
cd /tmp/Testing
git pull origin mybranch1

%%bash
cd /tmp/Testing
git status


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Remotes and fetching from them
If you're working with other people, then it's possible that they have made changes to the remote repository between the time you first cloned it and now. push will fail.
In our particular case of the Testing repository, this is not going to happen, since you just cloned it and presumably havent invited anyone to collaborate with you on it (you can do this from the settings link on the right side of the repository page).
However you can imagine that the original repository cs109/testing, which you are now divorced from, has changed, and that you somehow want to pull those changes in.
That's where the next two commands come in.

remote
We have seen so far that our repository has one "remote", or upstream repository, which has been identified with the word origin, as seen in .git/config. We now wish to add another remote, which we shall call course, which points to the original repository we forked from. We want to do this to pull in changes, in-case something changed there.
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
See what branches we have created...
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
We add the file to the index, and then commit the files to the local repository on the mybranch branch.
Can you write Python code for it?

%%bash
cd /tmp/Testing
git add .
git status

%%bash
cd /tmp/Testing
git commit -m "Added another test file to demonstrate git features" -a
git status


I am working on the file "Lab1-git_original.ipynb".
The first task is:
Why does the syntax have origin/mybranch1? Remember that multiple remotes may have the same branch...so you want to be specific. So we check out origin's mybranch1, and only want world.md.
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
You are always working on a given branch in a repository. Typically this is master. More on this later..You can know which branch you are on by typing git branch. The strred one is the one you are on.
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
You can combine a fetch and a merge together by simply doing a git pull. This will fail if you and your collaborator have worked on the same file (since you will have to merge by hand), but is a great shortcut when the files  worked on are different. I use it all the times on a personal level too, to shift work between two different machines, as long as I am not working on both at the same time. The usual use case is day work on a work computer, and then evening work at home on the laptop. Read the docs if you are interested.
Branching.
As you might have seen by now, everything in git is a branch. We have branches on remote (upstream) repositories, copies of remote branches in our local repository, and branches on local repositories which (so far) track remote branches (or more precisely local copies of remote repositories)
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
You can see that a copy of a new remote branch has been made below, by providing the -avv argument to git branch.
Can you write Python code for it?


I am working on the file "Lab1-git_original.ipynb".
The first task is:
oops, that was not what we wanted. We undo it using git reset, to go back to the state at the last commit.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
6000 rows times 10 columns. A spredsheet is a table is a matrix. How can we access members of this tuple (brackets like so:() )
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Aha, we had some incomplete data. Lets get rid of it
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
And then we can again make a scatterplot
Can you write Python code for it?

plt.scatter(alist, asquaredlist);

print type(alist)


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
As the diagram above shows, pandas considers a table (dataframe) as a pasting of many "series" together, horizontally.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Cleaning
We first check the datatypes. Notice that review_count, rating_count are of type object (which means they are either strings or Pandas couldnt figure what they are), while year is a float.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Filtering
Here are two ways to get a filtered dataframe
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Here we create a mask and use it to "index" into the dataframe to get the rows we want.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Here we make a scatterplot in matplotlib of rating against year. By setting the alpha transparency low we can how the density of highly rated books on goodreads has changed.
Can you write Python code for it?

plt.scatter(df.year, df.rating, lw=2, alpha=.2)
plt.xlim([1900,2010])
plt.xlabel("Year")
plt.ylabel("Rating")


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
If you want to combine these conditions, use the second form and put '()' brackets around each condition. The query uses a boolean AND. Each condition ceates a mask of trues and falses.
Can you write Python code for it?

df[(df.year < 0) & (df.rating > 4)]#there were none greater than 4.5!


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
In other words, operations on numpy arrays, and by extension, Pandas Series, are vectorized. You can add two numpy lists by just using + whereas the result isnt what you might expect for regular python lists. To add regular python lists elementwise, you will need to use a loop:
Can you write Python code for it?

newlist=[]
for item in alist:
    newlist.append(item+item)
newlist


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
In other words, something is a duck if it quacks like a duck. A Pandas series quacks like a python list. They both support something called the iterator protocol, an notion of behaving in a "listy" way. And Python functions like plt.scatter will accept anything that behaves listy. Indeed here's one more example:
Can you write Python code for it?

plt.hist(df.rating_count.values, bins=100, alpha=0.5);

print type(df.rating_count), type(df.rating_count.values)


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Much cleaner now!
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Notice that you could just find the average since the Trues map to 1s.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
One can see the sparseness of review counts. This will be important when we learn about recommendations: we'll have to regularize our models to deal with it.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Oppos we got an error. Something is not right. Its trying to convert some python datatype: None into an int. This usually means data was missing. Was it?
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Or directly, in Pandas, which works since df.rating < 3 is a pandas Series.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Pythons and ducks
Notice that we used the series in the x-list and y-list slots in the scatter function in the plt module.
In working with python I always remember: a python is a duck.
What I mean is, python has a certain way of doing things. For example lets call one of these ways listiness. Listiness works on lists, dictionaries, files, and a general notion of something called an iterator.
A Pandas series plays like a python list:
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Querying
A spreadsheet is useless if you cant dice/sort/etc it. Here we look for all books with a rating less than 3.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Series and numpy lists behave similarly as well.
Vectorization
Numpy arrays are a bit different from regular python lists, and are the bread and butter of data science. Pandas Series are built atop them.
Can you write Python code for it?

alist + alist

np.array(alist)

np.array(alist)+np.array(alist)

np.array(alist)**2


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Suppose we try and fix this
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
The structure may be easier to see if we rescale the x-axis to be logarithmic.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
These are the column names.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
We can construct another list by using the syntax below, also called a list comprehension.
Can you write Python code for it?

asquaredlist=[i*i for i in alist]
asquaredlist


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
We can do this in more detail, plotting against a mean, with cutom binsize or number of bins. Note how to label axes and create legends.
Can you write Python code for it?

sns.set_context("poster")
meanrat=df.rating.mean()
#you can get means and medians in different ways
print meanrat, np.mean(df.rating), df.rating.median()
with sns.axes_style("ticks"):
    df.rating.hist(bins=30, alpha=0.9);
    plt.axvline(meanrat, 0.1, 1, color='r', label='Mean')
    plt.xlabel("average rating of book")
    plt.ylabel("Counts")
    plt.title("Ratings Histogram")
    plt.legend()
    #sns.despine()


I am working on the file "Lab1-pythonpandas.ipynb".
The first task is:
Why did that work?
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
6000 rows times 10 columns. A spredsheet is a table is a matrix. How can we access members of this tuple (brackets like so:() )
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Aha, we had some incomplete data. Lets get rid of it
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
But we get a 0? Why? In Python 2.x division is integer division by default. So one can fix by converting the df.shape[0] to a float
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Cleaning
We first check the datatypes. Notice that review_count, rating_count are of type object (which means they are either strings or Pandas couldnt figure what they are), while year is a float.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Filtering
Here are two ways to get a filtered dataframe
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Here we make a scatterplot in matplotlib of rating against year. By setting the alpha transparency low we can how the density of highly rated books on goodreads has changed.
Can you write Python code for it?

plt.scatter(df.year, df.rating, lw=0, alpha=.08)
plt.xlim([1900,2010])
plt.xlabel("Year")
plt.ylabel("Rating")


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
If you want to combine these conditions, use the second form and put '()' brackets around each condition. The query uses a boolean AND. Each condition ceates a mask of trues and falses.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
In other words, something is a duck if it quacks like a duck. A Pandas series quacks like a python list. They both support something called the iterator protocol, an notion of behaving in a "listy" way. And Python functions like plt.scatter will accept anything that behaves listy. Indeed here's one more example:
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Notice that you could just find the average since the Trues map to 1s.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Notice we have a table! A spreadsheet! And it indexed the rows. Pandas (borrowing from R) calls it a DataFrame. Lets see the types of the columns...
df, in python parlance, is an instance of the pd.DataFrame class, created by calling the pd.read_csv function, which cllas the DataFrame constructor inside of it. If you dont understand this sentence, dont worry, it will become clearer later. What you need to take away is that df is a dataframe object, and it has methods, or functions belonging to it, which allow it to do things. For example df.head() is a method that shows the first 5 rows of the dataframe.

The basics
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Or directly, in Pandas, which works since df.rating < 3 is a pandas Series.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Pythons and ducks
Notice that we used the series in the x-list and y-list slots in the scatter function in the plt module.
In working with python I always remember: a python is a duck.
What I mean is, python has a certain way of doing things. For example lets call one of these ways listiness. Listiness works on lists, dictionaries, files, and a general notion of something called an iterator.
A Pandas series plays like a python list:
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Querying
A spreadsheet is useless if you cant dice/sort/etc it. Here we look for all books with a rating less than 3.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Series and numpy lists behave similarly as well.
Vectorization
Numpy arrays are a bit different from regular python lists, and are the bread and butter of data science. Pandas Series are built atop them.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
So we ought to be able to do this
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Suppose we try and fix this
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
The shape of the object is:
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
The structure may be easier to see if we rescale the x-axis to be logarithmic.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
These are the column names.
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Vectorization is a powerful idiom, and we will use it a lot in this class. And, for almost all data intensive computing, we will use numpy arrays rather than python lists, as the python numerical stack is based on it.
You have seen this in idea in spreadsheets where you add an entire column to another one.
Two final examples
Can you write Python code for it?


I am working on the file "Lab1-pythonpandas_original.ipynb".
The first task is:
Why did that work?
Can you write Python code for it?


I am working on the file "Lab1.ipynb".
The first task is:
Intro to TensorFlow
What is a Computation Graph?
Everything in TensorFlow comes down to building a computation graph. What is a computation graph? Its just a series of math operations that occur in some order. Here is an example of a simple computation graph:
<img src="files/computation-graph.png">
This graph takes 2 inputs, (a, b) and computes an output (e). Each node in the graph is an operation that takes some input, does some computation, and passes its output to another node.
We could make this computation graph in TensorFlow in the following way:
Can you write Python code for it?

a = tf.placeholder(tf.float32)
b = tf.placeholder(tf.float32)
c = tf.add(a, b)
d = tf.subtract(b, 1)
e = tf.multiply(c, d)


I am working on the file "Lab1.ipynb".
The first task is:
Running this code, you’ll see the network train and output its performance as it learns. I was able to get it to 65.5% accuracy. This is just OK, considering random guessing gets you 33.3% accuracy. In the next tutorial, you'll learn some ways to improve upon this.
Concluding Thoughts
This was a brief introduction into TensorFlow. There is so, so much more to learn and explore, but hopefully this has given you some base knowledge to expand upon. As an additional exercise, you can see what you can do with this code to improve the performance. Ideas include: randomizing mini-batches, making the network deeper, using word embeddings (see below) rather than bag-of-words, trying different optimizers (like Adam), different weight initializations. We’ll explore some of these tomorrow.
More on Word Embeddings
In this lab we used Bag-of-Words to represent a tweet.  Word Embeddings are a more meaningful representation.  The basic idea is we represent a word with a vector $\phi$ by the context the word appears in. We do this by training a neural network to predict the context of words across a large training set. The weights of that neural network can then be thought of as a dense and useful representation that captures context. This is useful because now our representations of words captures actual semantic similarity.
Word Embeddings capture all kinds of useful semantic relationships. For example, one cool emergent property is $ \phi(king) - \phi(queen) = \phi(man) - \phi(woman)$. To learn more about the magic behind word embeddings we recommend Chris Olah's blog post. A common tool for generating Word Embeddings is word2vec.
Solutions
Can you write Python code for it?

a = tf.placeholder(tf.float32)
b = tf.placeholder(tf.float32)
c = tf.add(a, b)
d = tf.subtract(b, 1)
e = tf.multiply(c, d)

with tf.Session() as session:
    a_data, b_data = 3.0, 6.0
    feed_dict = {a: a_data, b: b_data}
    output = session.run([e], feed_dict=feed_dict)
    print(output) # 45.0

n_input_nodes = 2
n_output_nodes = 1
x = tf.placeholder(tf.float32, (None, n_input_nodes))
W = tf.Variable(tf.ones((n_input_nodes, n_output_nodes)), dtype=tf.float32)
b = tf.Variable(tf.zeros(n_output_nodes), dtype=tf.float32)

'''TODO: Define the operation for z (use tf.matmul to multiply W and x).'''
z = tf.matmul(x, W) + b

'''TODO: Define the operation for out (use tf.sigmoid).'''
out = tf.sigmoid(z)

test_input = [[0.5, 0.5]]
with tf.Session() as session:
    tf.global_variables_initializer().run(session=session)
    feed_dict = {x: test_input}
    output = session.run([out], feed_dict=feed_dict)
    print(output[0]) # This should output 0.73105. If not, double-check your code above

'''TODO: manually optimize weight_values and bias_value to classify points'''

# Modify weight_values, bias_value in the above code to adjust the decision boundary
# See if you can classify all the points correctly (all markers green)

weight_values = np.array([[-0.1], [0.2]]) # TODO change values and re-run
bias_value = np.array([[0.5]]) #TODO change values and re-run

# A pretty good boundary is made with:
weight_values = np.array([[0.03], [0.12]])
bias_value = np.array([[-0.5]])

x = tf.placeholder(tf.float32, (None, 2), name='x')
W = tf.Variable(weight_values, name='W', dtype=tf.float32)
b = tf.Variable(bias_value, name='b', dtype=tf.float32)
z = tf.matmul(x, W) + b
out = tf.sigmoid(z)

data = np.array([[2, 7], [1, 7], [3, 1], [3, 3], [4, 3], [4, 6], [6, 5], [7, 7], [7, 5], [2, 4], [2, 2]])
y = np.array([1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0])
with tf.Session() as session:
    tf.global_variables_initializer().run(session=session)
    utils.classify_and_plot(data, y, x, out, session)

# load data
X,...


I am working on the file "Lab1.ipynb".
The first task is:
Step 2: Making Placeholders
So we have our data loaded as numpy arrays. But remember, TensorFlow graphs begin with generic placeholder inputs, not actual data. We feed the actual data in later once the full graph has been defined. We define our placeholders like this:
Can you write Python code for it?

data_placeholder = tf.placeholder(tf.float32, shape=(None, vocab_size), name='data_placeholder')

'''TODO: Make the labels placeholder.''';
labels_placeholder = #todo


I am working on the file "Lab1.ipynb".
The first task is:
Step 4: Build Computation Graph
Now let’s define our computation graph.
Our first operation in our graph is a tf.matmul of our data input and our first set of weights. tf.matmul performs a matrix multiplication of two tensors. This is why it is so important that the dimensions of data_placeholder and h0_weights align (dimension 1 of data_placeholder must equal dimension 0 of h0_weights). We then just add the h0_bias variable and perform a nonlinearity transformation, in this case we use tf.nn.relu (ReLU). We do this again for the next hidden layer, and the final output logits.
Can you write Python code for it?

# Define Computation Graphs
hidden0 = tf.nn.relu(tf.matmul(data_placeholder, h0_weights) + h0_biases)

'''TODO: write the computation to get the output of the second hidden layer.''';
hidden1 = #todo

logits = tf.matmul(hidden1, out_weights) + out_biases


I am working on the file "Lab1.ipynb".
The first task is:
Step 5: Define a Loss Functions
We have defined our network, but we need a way to train it. Training a network comes down to optimizing our network to minimize a loss function, or a measure how good we're doing.  Then, we can take the gradient with respect to that performance and move in the right direction.
Since we are doing classification (pos vs neg), a common loss function to use is cross entropy:
$$L( \Theta ) = - \sum_i  y_i'\log{y_i}  $$
where $y$ is our predicted probability distribution and $y'$ is the true distribution. We can access this function in tensorflow with tf.nn.softmax_cross_entropy_with_logits.
Note that this loss is 0 if the prediction is correct.
Can you write Python code for it?

'''TODO: Define the loss.  Use tf.nn.softmax_cross_entropy_with_logits.'''
loss = #todo
learning_rate = 0.0002

# Define the optimizer operation.  This is what will take the derivate of the loss
# with respect to each of our parameters and try to minimize it.
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)
prediction = tf.nn.softmax(logits)

# Compute the accuracy
prediction_is_correct = tf.equal(tf.argmax(logits, 1), tf.argmax(labels_placeholder, 1))
accuracy = tf.reduce_mean(tf.cast(prediction_is_correct, tf.float32))


I am working on the file "Lab1.ipynb".
The first task is:
Tensorflow uses tf.placeholder to handle inputs to the model. This is like making a reservation at a restaurant. The restaurant reserves a spot for 5 people, but you are free to fill those seats with any set of friends you want to. tf.placeholder lets you specify that some input will be coming in, of some shape and some type. Only when you run the computation graph do you actually provide the values of this input data. You would run this simple computation graph like this:
Can you write Python code for it?

with tf.Session() as session:
    a_data, b_data = 3.0, 6.0
    feed_dict = {a: a_data, b: b_data}
    output = session.run([e], feed_dict=feed_dict)
    print(output) # 45.0


I am working on the file "Lab1.ipynb".
The first task is:
Tweet Sentiment Analysis
Let's move to a real-world task. We're going to be classifying tweets as positive, negative, or neutral. Check out the very negative tweet below:
<img src="files/tweet-model.jpg" style="width: 500px;">
Building the Model
Building an MLP
MLP or Multi-layer perceptron is a basic archetecture where where we multiply our representation with some matrix W and add some bias b and then apply some nonlinearity like tanh at each layer. Layers are fully connected to the next. As the network gets deeper, it's expressive power grows exponentially and so they can draw some pretty fancy decision boundaries. In this exercise, you'll build your own MLP, with 2 hidden layers (layers that aren't input or output).
To make training more stable and efficient, we'll actually evaluate 128 tweets at a time, and take gradients with respect to the loss on the 128. We call this idea training with mini-batches.
Step 1: Representing Words
In this model, we’ll be representing tweets as bag-of-words (BOW) representations. BOW representations are vectors where each element index represents a different word and its value represents the number of times this word appears in our sentence. This means that each sentence will be represented by a vector that is vocab_size long. Our output labels will be represented as a vector of size n_classes (3). We get this data with some utility functions:
Can you write Python code for it?

# load data
X, y, index_to_word, sentences = utils.load_sentiment_data_bow()
X_train, y_train, X_test, y_test = utils.split_data(X, y)
vocab_size = X.shape[1]
n_classes = y.shape[1]

print("Tweet:", sentences[5])
print("Label:", y[5])
print("Bag of Words Representation:", X_train[5])


I am working on the file "Lab1.ipynb".
The first task is:
We use feed_dict to pass in the actual input data into the graph. We use session.run to get the output from the c operation in the graph. Since e is at the end of the graph, this ends up running the entire graph and returning the number 45 - cool!
Neural Networks in Tensorflow
We can define neural networks in TensorFlow using computation graphs. Here is an example, very simple neural network (just 1 perceptron):
<img src="files/computation-graph-2.png">
This graph takes an input, (x) and computes an output (out). It does it with what we learned in class, out = sigmoid(W*x+b).
We could make this computation graph in TensorFlow in the following way:
Can you write Python code for it?

n_input_nodes = 2
n_output_nodes = 1
x = tf.placeholder(tf.float32, (None, n_input_nodes))
W = tf.Variable(tf.ones((n_input_nodes, n_output_nodes)), dtype=tf.float32)
b = tf.Variable(tf.zeros(n_output_nodes), dtype=tf.float32)

'''TODO: Define the operation for z (use tf.matmul to multiply W and x).'''
z = #todo

'''TODO: Define the operation for out (use tf.sigmoid).'''
out = #todo