I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
Remember that if the angle is small,
the cosine would be close to one and that means the two words are more similar. And as the angle approaches 90 degrees,
the cosine approaches zero and the words are less similar.
Finding the country of each capital
Now, we  will use the cosine distance to find the capital cities of countries, in a similar way as previously, putting all together in one handy function:
Can you write Python code for it?

def get_country(city1, country1, city2, embeddings):
    """
    Input:
        city1: a string (the capital city of country1)
        country1: a string (the country of capital1)
        city2: a string (another capital city of unknown country)
        embeddings: a dictionary where the keys are words and values are their embeddings
    Output:
        country: tuple, the most likely country and its similarity score
    """

    # store the city1, country 1, and city 2 in a set called group
    group = set((city1, country1, city2))

    # get embeddings of city 1
    city1Emb = word_embeddings[city1]

    # get embedding of country 1
    country1Emb =  word_embeddings[country1]

    # get embedding of city 2
    city2Emb = word_embeddings[city2]

    # get embedding of country 2 (it's a combination of the embeddings of country 1, city 1 and city 2)
    # Remember: King - Man + Woman = Queen
    connectingVector = country1Emb - city1Emb + city2Emb

    # Initialize the similarity to -1 (it will be replaced by a similarities that are closer to +1)
    bestSimilarity = -1

    # initialize country to an empty string
    country = ''

    # loop through all words in the embeddings dictionary
    for word in embeddings.keys():

        # first check that the word is not already in the 'group'
        if word not in group:

            # get the word embedding
            wordEmb = word_embeddings[word]

            # calculate cosine similarity between embedding of country 2 and the word in the embeddings dictionary
            currentSimilarity = cosine_similarity(connectingVector, wordEmb)

            if currentSimilarity > bestSimilarity:

                bestSimilarity = currentSimilarity

                # store the country as a tuple, which contains the word and the similarity
                country = (word, bestSimilarity)

    return country

# Testing the function, note to make it more robust you can return the 5 most similar words.
get_country('Athens', 'Greece', 'Cairo',...


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
The distance between the words village and town is plotted in blue.
Note that it has a similar length than the distance between happy and sad. This is a characteristic that can be useful for pattern recognition and predictions and we will  see how now.
Predict relationships among words
Now we will write a function that will use the word embeddings (specifically the distance) to predict relationships among words!
* The function will take as input three words.
* The first two are related to each other.
* It will predict a 4th word which is related to the third word in a similar manner as the two first words are related to each other.
* As an example, "Athens is to Greece as Bangkok is to ______"?
We will use the function to tell the capital of a country.
To do this, we will first compute the distance: the cosine similarity or the Euclidean distance.
Predicting capitals
Now, applying vector difference and addition, one can create a vector representation for a new word. For example, we can say that the vector difference between 'France' and 'Paris' represents the concept of Capital city.
One can move from the city of Madrid in the direction of the concept of Capital, and obtain something close to the corresponding country to which Madrid is the Capital.
How cool is that?
Can you write Python code for it?

capital = getVector('France') - getVector('Paris')
country = getVector('Madrid') + capital

print(country[0:5]) # Print the first 5 values of the vector


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
We can observe that the vector 'country' that we expected to be the same as the vector for Spain is not exactly it.
Can you write Python code for it?

diff = country - getVector('Spain')
print(diff[0:5])


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
When you perform a PCA reduction, you retain as much as possible information but something is always lost.  This is why you normally keep all dimensions for your model building and use PCA for visualisations or some reduction.
Here is an example of another characteristic of word vectors that was not immediately visible in the previous plots which used a reduced-dimension data set.
Can you write Python code for it?

  # a subset of words
words = ['happy', 'sad', 'city', 'town', 'village', 'country', 'continent', 'joyful']
embeddedWords = np.array([getVector(word) for word in words]) # Convert each word to its vector representation

fig, ax = plt.subplots(figsize = (10, 10)) # Create custom size image

col1 = 3 # NOTE: we select a specific column (dimension) for the x axis
col2 = 2 # NOTE: and another specific column (one of the 300 dimensions) for the y axis

# Print an arrow for each word
for word in embeddedWords:
    ax.arrow(0, 0, word[col1], word[col2], head_width=0.005, head_length=0.005, fc='r', ec='r', width = 1e-5)

ax.scatter(embeddedWords[:, col1], embeddedWords[:, col2]); # Plot a dot for each word

# Add the word label over each dot in the scatter plot
for i in range(0, len(words)):
    ax.annotate(words[i], (embeddedWords[i, col1], embeddedWords[i, col2]))

plt.show()


I am working on the file "03-NLP/word-vectors.ipynb".
The first task is:
Yes, as expected!
Now you have a simple process to get unknown relationships between words by the use of known relationships between others.
The only catch here is that you need a vector space where the representations capture the relative meaning of words.
Can you write Python code for it?

find_closest_word(getVector('Italy') - getVector('Rome') + getVector('Madrid'))


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Adding or subtracting two matrices
Consider two small $2 \times 2$ matrices, where $2 \times 2$ denotes the # of rows $\times$ the # of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \ a_{21} & a_{22} \end{smallmatrix} \bigr)$ and $B$=$\bigl( \begin{smallmatrix} b_{11} & b_{12} \ b_{21} & b_{22} \end{smallmatrix} \bigr)$.  To find the result of $A-B$, simply subtract each element of A with the corresponding element of B:
$$
\begin{equation}
    A -B =
    \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix} -
    \begin{bmatrix} b_{11} & b_{12} \
      b_{21} & b_{22}
    \end{bmatrix}
    =
    \begin{bmatrix}
      a_{11}-b_{11} & a_{12}-b_{12} \
      a_{21}-b_{21} & a_{22}-b_{22}
    \end{bmatrix}
\end{equation}
$$
Addition works exactly the same way:
$$
\begin{equation}
    A + B =
    \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix} +
    \begin{bmatrix} b_{11} & b_{12} \
      b_{21} & b_{22}
    \end{bmatrix}
    =
    \begin{bmatrix}
      a_{11}+b_{11} & a_{12}+b_{12} \
      a_{21}+b_{21} & a_{22}+b_{22}
    \end{bmatrix}
\end{equation}
$$
An important point to know about matrix addition and subtraction is that it is only defined when $A$ and $B$ are of the same size.  Here, both are $2 \times 2$.  Since operations are performed element by element, these two matrices must be conformable- and for addition and subtraction that means they must have the same numbers of rows and columns.  I like to be explicit about the dimensions of matrices for checking conformability as I write the equations, so write
$$
A_{2 \times 2} + B_{2 \times 2}= \begin{bmatrix}
  a_{11}+b_{11} & a_{12}+b_{12} \
  a_{21}+b_{21} & a_{22}+b_{22}
\end{bmatrix}_{2 \times 2}
$$
Notice that the result of a matrix addition or subtraction operation is always of the same dimension as the two operands.
Let's define another matrix, B, that is also $2 \times 2$ and add it to A:
Can you write Python code for it?

B = np.random.randn(2,2)
print B

# A = np.array([[1,0], [0,1]])
A = np.eye(2)
A

A+B


B = np.random.randn(2,2)
print B

A = np.array([[1,0], [0,1]])
A

A*B


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Closed-form Linear Regression
And compute the parametes $\beta_0$  and $\beta_1$ according to
$$ \beta = (X^\prime X)^{-1} X^\prime y $$

Note:
This not only looks elegant but can also be written in Julia code. However, matrix inversion $M^{-1}$ requires $O(d^3)$ iterations for a $d\times d$ matrix.<br />
https://www.coursera.org/learn/ml-regression/lecture/jOVX8/discussing-the-closed-form-solution
Can you write Python code for it?

beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)

yhat = X.dot(beta)
yhat.shape

plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=1, color="red")


beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)

yhat = X.dot(beta)
yhat.shape

plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=1, color="red")

beta


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Creating arrays, scalars, and matrices in Python
Scalars can be created easily like this:
Can you write Python code for it?


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Evaluation: Root-mean-square Deviation
The root-mean-square deviation (RMSD) or root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model or an estimator and the values actually observed. The RMSD represents the sample standard deviation of the differences between predicted values and observed values. These individual differences are called residuals when the calculations are performed over the data sample that was used for estimation, and are called prediction errors when computed out-of-sample. The RMSD serves to aggregate the magnitudes of the errors in predictions for various times into a single measure of predictive power. RMSD is a good measure of accuracy, but only to compare forecasting errors of different models for a particular variable and not between variables, as it is scale-dependent.$^1$

$^1$  Hyndman, Rob J. Koehler, Anne B.; Koehler (2006). "Another look at measures of forecast accuracy". International Journal of Forecasting. 22 (4): 679–688. doi:10.1016/j.ijforecast.2006.03.001.
Can you write Python code for it?

import math
RSMD = math.sqrt(np.square(yhat_m-y_m).sum()/n)
print RSMD


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Example: Linear Regression
Linear regression is an approach for modeling the relationship between a scalar dependent variable $y$ and one or more explanatory variables (or independent variables) denoted $X$. The case of one explanatory variable is called simple linear regression. For more than one explanatory variable, the process is called multiple linear regression.$^1$ (This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.$^2$
We assume that the equation
$y_i = \beta_0 + \beta_1 X_i + \epsilon_i$ where $\epsilon_i \approx N(0, \sigma^2)$

$^1$ David A. Freedman (2009). Statistical Models: Theory and Practice. Cambridge University Press. p. 26. A simple regression equation has on the right hand side an intercept and an explanatory variable with a slope coefficient. A multiple regression equation has two or more explanatory variables on the right hand side, each with its own slope coefficient
$^2$ Rencher, Alvin C.; Christensen, William F. (2012), "Chapter 10, Multivariate regression – Section 10.1, Introduction", Methods of Multivariate Analysis, Wiley Series in Probability and Statistics, 709 (3rd ed.), John Wiley & Sons, p. 19, ISBN 9781118391679.
Can you write Python code for it?

n = 100  # numeber of samples
Xr = np.random.rand(n)*99.0
y = -7.3 + 2.5*Xr + np.random.randn(n)*27.0
plt.plot(Xr, y, "o", alpha=0.5)


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Let's add the bias, i.e. a column of $1$s to the explanatory variables
Can you write Python code for it?

X = np.vstack((np.ones(n), Xr)).T
print X.shape
X[0:10,:]


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Matrix Addition and Subtraction
Adding or subtracting a scalar value to a matrix
To learn the basics, consider a small matrix of dimension $2 \times 2$, where $2 \times 2$ denotes the number of rows $\times$ the number of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Consider adding a scalar value (e.g. 3) to the A.
$$
\begin{equation}
    A+3=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix}+3
    =\begin{bmatrix}
      a_{11}+3 & a_{12}+3 \
      a_{21}+3 & a_{22}+3
    \end{bmatrix}
\end{equation}
$$
The same basic principle holds true for A-3:
$$
\begin{equation}
    A-3=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix}-3
    =\begin{bmatrix}
      a_{11}-3 & a_{12}-3 \
      a_{21}-3 & a_{22}-3
    \end{bmatrix}
\end{equation}
$$
Notice that we add (or subtract) the scalar value to each element in the matrix A.  A can be of any dimension.
This is trivial to implement, now that we have defined our matrix A:
Can you write Python code for it?

A = np.array(range(6)).reshape([2,3])
print(A)

result = A + 3
#or
result = 3 + A
print result


A

result = A + 3
#or
result = 3 + A
print result


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Matrix Division
The term matrix division is actually a misnomer.  To divide in a matrix algebra world we first need to invert the matrix.  It is useful to consider the analog case in a scalar work.  Suppose we want to divide the $f$ by $g$.  We could do this in two different ways:
$$
\begin{equation}
    \frac{f}{g}=f \times g^{-1}.
\end{equation}
$$
In a scalar seeting, these are equivalent ways of solving the division problem.  The second one requires two steps: first we invert g and then we multiply f times g.  In a matrix world, we need to think about this second approach.  First we have to invert the matrix g and then we will need to pre or post multiply depending on the exact situation we encounter (this is intended to be vague for now).
Inverting a Matrix
As before, consider the square $2 \times 2$ matrix $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \ a_{21} & a_{22}\end{smallmatrix} \bigr)$.  Let the inverse of matrix A (denoted as $A^{-1}$) be
$$
\begin{equation}
    A^{-1}=\begin{bmatrix}
             a_{11} & a_{12} \
             a_{21} & a_{22}
           \end{bmatrix}^{-1}=\frac{1}{a_{11}a_{22}-a_{12}a_{21}}   \begin{bmatrix}
                     a_{22} & -a_{12} \
                     -a_{21} & a_{11}
                   \end{bmatrix}
\end{equation}
$$
The inverted matrix $A^{-1}$ has a useful property:
$$
\begin{equation}
    A \times A^{-1}=A^{-1} \times A=I
\end{equation}
$$
where I, the identity matrix (the matrix equivalent of the scalar value 1), is
$$
\begin{equation}
    I_{2 \times 2}=\begin{bmatrix}
             1 & 0 \
             0 & 1
           \end{bmatrix}
\end{equation}
$$
furthermore, $A \times I = A$ and $I \times A = A$.
An important feature about matrix inversion is that it is undefined if (in the $2 \times 2$ case), $a_{11}a_{22}-a_{12}a_{21}=0$.  If this relationship is equal to zero the inverse of A does not exist.  If this term is very close to zero, an inverse may exist but $A^{-1}$ may be poorly conditioned meaning it is prone to rounding error and is likely...
Can you write Python code for it?

# note, we need a square matrix (# rows = # cols), use C:
C_inverse = np.linalg.inv(C)
print C_inverse


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Matrix Multiplication
Multiplying a scalar value times a matrix
As before, let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Suppose we want to multiply A times a scalar value (e.g. $3 \times A$)
$$
\begin{equation}
    3 \times A = 3 \times \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix}
    =
    \begin{bmatrix}
      3a_{11} & 3a_{12} \
      3a_{21} & 3a_{22}
    \end{bmatrix}
\end{equation}
$$
is of dimension (2,2).  Scalar multiplication is commutative, so that $3 \times A$=$A \times 3$.  Notice that the product is defined for a matrix A of any dimension.
Similar to scalar addition and subtration, the code is simple:
Can you write Python code for it?


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Mechanics
Indexing and Slicing
examples from https://www.tutorialspoint.com/numpy/numpy_indexing_and_slicing.htm
Can you write Python code for it?

a = np.arange(10)
s = slice(2,7,2)
print a
print a[s]

a = np.arange(10)
b = a[2:7:2]
print a
print b

a = np.arange(10)
b = a[5]
print a
print b

a = np.arange(10)
print a
print a[2:5]

print a[2:5]
print a[5:8]
print np.concatenate([a[2:5],a[5:8]])
print a[2:8]

a = np.arange(10)
print a
print a[-1]
print a[-2:-5:-1]

a = np.array([[1,2,3],[3,4,5],[4,5,6]])
print a

# slice items starting from index
print 'Now we will slice the array from the index a[1:]'
print a[1:]
print "slicing along 2 dimensions"
print a[1:,2]
print "aside: if a is a matrix, we keep the original shape of the slice..."
print np.matrix(a)[1:,2]

# array to begin with
a = np.array([[1,2,3],[3,4,5],[4,5,6]])

print 'Our array is:'
print a
print '\n'

# this returns array of items in the second column
print 'The items in the second column are:'
print a[...,1]
print '\n'

# Now we will slice all items from the second row
print 'The items in the second row are:'
print a[1,...]
print '\n'

# Now we will slice all items from column 1 onwards
print 'The items column 1 onwards are:'
print a[...,1:]

# in-class exercise:
a = np.matrix('[4 5 6 7; 4 1 0 1; 5 0 1 3; 9 8 3 2]')
print a
# how can you get the [[1,0],[0,1]] matrix in the middle of a?

# answer:
a[1:3,1:3]

# in-class exercise:
a = np.arange(18)
# what is the transpose of the 3x3 matrix formed from taking the even elements of a?

# answer:
a[0:18:2].reshape([3,3]).T


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Multiple Linear Regression
Can you write Python code for it?

n = 100  # numeber of samples
X1 = np.random.rand(n)*99.0
X2 = np.random.rand(n)*51.0 - 26.8
X3 = np.random.rand(n)*5.0 + 6.1
X4 = np.random.rand(n)*1.0 - 0.5
X5 = np.random.rand(n)*300.0

y_m = -7.3 + 2.5*X1 + -7.9*X2 + 1.5*X3 + 10.0*X4 + 0.13*X5 + np.random.randn(n)*7.0
plt.hist(y_m, bins=20)
;

X_m = np.vstack((np.ones(n), X1, X2, X3, X4, X5)).T
X_m.shape
X_m[:5]

beta_m = np.linalg.inv(X_m.T.dot(X_m)).dot(X_m.T).dot(y_m)
beta_m

yhat_m = X_m.dot(beta_m)
yhat_m.shape
plt.hist(yhat_m, bins=20);


n = 100  # numeber of samples
X1 = np.random.rand(n)*99.0
X2 = np.random.rand(n)*51.0 - 26.8
X3 = np.random.rand(n)*5.0 + 6.1
X4 = np.random.rand(n)*1.0 - 0.5
X5 = np.random.rand(n)*300.0

y_m = -7.3 + 2.5*X1 + -7.9*X2 + 1.5*X3 + 10.0*X4 + 0.13*X5 + np.random.randn(n)*7.0
plt.hist(y_m, bins=20)
;

X_m = np.vstack((np.ones(n), X1, X2, X3, X4, X5)).T
X_m.shape

beta_m = np.linalg.inv(X_m.T.dot(X_m)).dot(X_m.T).dot(y_m)
beta_m

yhat_m = X.dot(beta_m)
yhat_m.shape


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Multiplying two matricies
Now, consider the $2 \times 1$ vector $C=\bigl( \begin{smallmatrix} c_{11} \
  c_{21}
\end{smallmatrix} \bigr)$
Consider multiplying matrix $A_{2 \times 2}$ and the vector $C_{2 \times 1}$.  Unlike the addition and subtraction case, this product is defined.  Here, conformability depends not on the row and column dimensions, but rather on the column dimensions of the first operand and the row dimensions of the second operand.  We can write this operation as follows
$$
\begin{equation}
    A_{2 \times 2} \times C_{2 \times 1} =
    \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix}{2 \times 2}
    \times
    \begin{bmatrix}
    c{11} \
    c_{21}
    \end{bmatrix}{2 \times 1}
    =
    \begin{bmatrix}
      a{11}c_{11} + a_{12}c_{21} \
      a_{21}c_{11} + a_{22}c_{21}
    \end{bmatrix}_{2 \times 1}
\end{equation}
$$
Alternatively, consider a matrix C of dimension $2 \times 3$ and a matrix A of dimension $3 \times 2$
$$
\begin{equation}
    A_{3 \times 2}=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22} \
      a_{31} & a_{32}
    \end{bmatrix}{3 \times 2}
    ,
    C{2 \times 3} =
    \begin{bmatrix}
          c_{11} & c_{12} & c_{13} \
          c_{21} & c_{22} & c_{23} \
    \end{bmatrix}_{2 \times 3}
    \end{equation}
$$
Here, A $\times$ C is
$$
\begin{align}
    A_{3 \times 2} \times C_{2 \times 3}=&
    \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22} \
      a_{31} & a_{32}
    \end{bmatrix}{3 \times 2}
    \times
    \begin{bmatrix}
      c{11} & c_{12} & c_{13} \
      c_{21} & c_{22} & c_{23}
    \end{bmatrix}{2 \times 3} \
    =&
    \begin{bmatrix}
      a{11} c_{11}+a_{12} c_{21} & a_{11} c_{12}+a_{12} c_{22} & a_{11} c_{13}+a_{12} c_{23} \
      a_{21} c_{11}+a_{22} c_{21} & a_{21} c_{12}+a_{22} c_{22} & a_{21} c_{13}+a_{22} c_{23} \
      a_{31} c_{11}+a_{32} c_{21} & a_{31} c_{12}+a_{32} c_{22} & a_{31} c_{13}+a_{32} c_{23}
    \end{bmatrix}_{3 \times 3}
\end{align}
$$
So in general, $X_{r_x \times c_x}...
Can you write Python code for it?

# Let's redefine A and C to demonstrate matrix multiplication:
A = np.arange(6).reshape((3,2))
C = np.random.randn(2,2)

print A.shape
print C.shape


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Numpy load, save data files
Can you write Python code for it?

ls -l HW03/

%%sh
./HW03/preprocess_data.sh HW03/Camera.csv HW03/Camera_cleaned.csv
head HW03/Camera_cleaned.csv

DATA = np.genfromtxt('HW03/Camera_cleaned.csv', delimiter=';')

DATA.

np.max(DATA[1:,2])

np.nanargmin()


%ls -l HW03/

%%sh
./HW03/preprocess_data.sh HW03/Camera.csv HW03/Camera_cleaned.csv
head HW03/Camera.csv
head HW03/Camera_cleaned.csv

DATA = np.genfromtxt('HW03/Camera_cleaned.csv', delimiter=';', names=True, dtype='float')

DATA

DATA['Max_resolution'].max()

np.nanargmin(DATA['Max_resolution'])

np.savetxt('Cameras_TMP.csv', DATA, delimiter=',')

%%sh
ls -l
rm Cameras_TMP.csv

import pickle

with open('Cameras.pkl','wb') as f:
    pickle.dump(DATA, f) # pickle.dump(DATA, open('Cameras.pkl','wb'))

# Shortcut:
np.save('Cameras2.pkl', DATA, allow_pickle=True)

%ls -l

with open('Cameras.pkl','rb') as f:
    datapkl = pickle.load(f)

# Shortcut:
datapkl2 = np.load('Cameras2.pkl.npy')

datapkl

datapkl2

%rm Cameras.pkl
%rm Cameras2.pkl.npy

# For saving multiple variables in workspace: import shelve


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
One important property of transposing a matrix is the transpose of a product of two matrices.  Let matrix A be of dimension $N \times M$ and let B of of dimension $M \times P$.  Then
$$
\begin{equation}
    (AB)^{\prime}=B^{\prime}A^{\prime}
\end{equation}
$$
For more information, see this http://en.wikipedia.org/wiki/Matrix_transposition on matrix transposition.  This is also easy to implement:
Can you write Python code for it?

print B.T.dot(A.T)
print "Is identical to:"
print (A.dot(B)).T

B.shape

print B, '\n'
B[0, 3]

A = np.arange(12).reshape((3,4))
A

A[2,:].shape

A[:,1].reshape(1,3).shape


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Random Numbers
Can you write Python code for it?

v1 = np.random.rand(100)
v2 = np.random.randn(100)
plt.plot(range(v1.shape[0]), v1, '.')
plt.plot(range(v2.shape[0]), v2, '.')

plt.hist(v1)
;

v2 = np.random.randn(10000)
plt.hist(v2, bins=100)
;

v3 = np.random.beta(3,2, 1000)
plt.hist(v3, bins=100)
;


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Regularization, Ridge-Regression
Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, is a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting.
In general, a regularization term $R(f)$ is introduced to a general loss function:

for a loss function  $V$ that describes the cost of predicting $f(x)$ when the label is
$y$, such as the square loss or hinge loss, and for the term
$\lambda$  which controls the importance of the regularization term.
$R(f)$ is typically a penalty on the complexity of
$f$, such as restrictions for smoothness or bounds on the vector space norm.$^1$
A theoretical justification for regularization is that it attempts to impose Occam's razor on the solution, as depicted in the figure. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters.
Regularization can be used to learn simpler models, induce models to be sparse, introduce group structure into the learning problem, and more.
We're going to add the L2 term $\lambda||\beta||_2^2$ to the regression equation, which yields to$^2$
$$ \beta = (X^\prime X + \lambda I)^{-1} X^\prime y $$

$^1$ Bishop, Christopher M. (2007). Pattern recognition and machine learning (Corr. printing. ed.). New York: Springer. ISBN 978-0387310732.
$^2$ http://stats.stackexchange.com/questions/69205/how-to-derive-the-ridge-regression-solution
Can you write Python code for it?

p = X.shape[1]  ## get number of parameters
lam = 10.0
p, lam

beta2 = np.linalg.inv(X.T.dot(X) + lam*np.eye(p)).dot(X.T).dot(y)

yhat2 = X.dot(beta2)

RSMD2 = math.sqrt(np.square(yhat2-y).sum()/n)
print RSMD2

##n = float(X.shape[0])
print "      RMSE = ", math.sqrt(np.square(yhat-y).sum()/n)
print "Ridge RMSE = ", math.sqrt(np.square(yhat2-y).sum()/n)
plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=0.7, color="red")
plt.plot(X[:,1], yhat2, "-", alpha=0.7, color="green")


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Similarity: Euclidian vs Cosine
Example Nearest Neighbor search
Nearest Neighbor search is a common technique in Machine Learning
Can you write Python code for it?

### Pure iterative Python ###

points = [[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]]
qPoint = [4,5,3]

minIdx = -1
minDist = -1
for idx, point in enumerate(points):  # iterate over all points
        print "index is %d, point is %s" % (idx, point)
        dist = sum([(dp-dq)**2 for dp,dq in zip(point,qPoint)])**0.5  # compute the euclidean distance for each point to q
        if dist < minDist or minDist < 0:  # if necessary, update minimum distance and index of the corresponding point
            minDist = dist
            minIdx = idx

print 'Nearest point to q: ', points[minIdx]

zip(point, qPoint)

# # # Equivalent NumPy vectorization # # #
import numpy as np
points = np.array([[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]])
qPoint = np.array([4,5,3])
minIdx = np.argmin(np.linalg.norm(points-qPoint,axis=1))  # compute all euclidean distances at once and return the index of the smallest one
print 'Nearest point to q: ', points[minIdx]

print points.shape
print qPoint.shape

print points
print qPoint
print points-qPoint

from numpy.linalg import norm

norm(points-qPoint)

1.0-points[0,:].dot(qPoint)/(norm(points[0,:])*norm(qPoint))


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
This code sets up Ipython Notebook environments (lines beginning with %), and loads several libraries and functions.  The core scientific stack in python consists of a number of free libraries.  The ones I have loaded above include:

sympy: provides for symbolic computation (solving algebra problems)
numpy: provides for linear algebra computations
matplotlib.pyplot: provides for the ability to graph functions and draw figures
scipy: scientific python provides a plethora of capabilities
seaborn: makes matplotlib figures even pretties (another library like this is called bokeh).  This is entirely optional and is purely for eye candy.

Creating arrays, scalars, and matrices in Python
Scalars can be created easily like this:
Can you write Python code for it?


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
This code sets up Ipython Notebook environments (lines beginning with %), and loads several libraries and functions.  The core scientific stack in python consists of a number of free libraries.  The ones I have loaded above include:

sympy: provides for symbolic computation (solving algebra problems)
numpy: provides for linear algebra computations
matplotlib.pyplot: provides for the ability to graph functions and draw figures
scipy: scientific python provides a plethora of capabilities
seaborn: makes matplotlib figures even pretties (another library like this is called bokeh).  This is entirely optional and is purely for eye candy.
Can you write Python code for it?


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Transposing a Matrix
At times it is useful to pivot a matrix for conformability- that is in order to matrix divide or multiply, we need to switch the rows and column dimensions of matrices.  Consider the matrix
$$
\begin{equation}
    A_{3 \times 2}=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22} \
      a_{31} & a_{32}
    \end{bmatrix}{3 \times 2}
\end{equation}
$$
The transpose of A (denoted as $A^{\prime}$) is
$$
\begin{equation}
   A^{\prime}=\begin{bmatrix}
      a{11} & a_{21} & a_{31} \
      a_{12} & a_{22} & a_{32} \
    \end{bmatrix}_{2 \times 3}
\end{equation}
$$
Can you write Python code for it?

A = np.arange(6).reshape((6,1))
B = np.arange(6).reshape((1,6))

A.dot(B)

B.dot(A)

A = np.arange(6).reshape((3,2))
B = np.arange(8).reshape((2,4))
print "A is"
print A

print "The Transpose of A is"
print A.T


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
Vectors and Lists
The numpy library (we will reference it by np) is the workhorse library for linear algebra in python.  To creat a vector simply surround a python list ($[1,2,3]$) with the np.array function:
Can you write Python code for it?

x_list = [1,2,3]
print x_list
print type(x_list)

x_vector = np.array([1,2,3])
print x_vector
print type(x_vector)

x_list*2

x_vector*2


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
We could have done this by defining a python list and converting it to an array:
Can you write Python code for it?

c_list = [1,2]
print "The list:",c_list
print "Has length:", len(c_list)

c_vector = np.array(c_list)
print "The vector:", c_vector
print "Has shape:",c_vector.shape

z = [5,6]
print "This is a list, not an array:",z
print type(z)

A = np.array([[0, 1, 2], [5, 6, 7]])
print A.shape
print type(A)

v = np.array([1,2,3,4])
print v.shape
print len(v)


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_class.ipynb".
The first task is:
We will use the numpy dot operator to perform the these multiplications. You can use it two ways to yield the same result:
Can you write Python code for it?

print A.dot(C)
print np.dot(A,C)

# What would happen to
C.dot(A)


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Adding or subtracting two matrices
Consider two small $2 \times 2$ matrices, where $2 \times 2$ denotes the # of rows $\times$ the # of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \ a_{21} & a_{22} \end{smallmatrix} \bigr)$ and $B$=$\bigl( \begin{smallmatrix} b_{11} & b_{12} \ b_{21} & b_{22} \end{smallmatrix} \bigr)$.  To find the result of $A-B$, simply subtract each element of A with the corresponding element of B:
$$
\begin{equation}
    A -B =
    \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix} -
    \begin{bmatrix} b_{11} & b_{12} \
      b_{21} & b_{22}
    \end{bmatrix}
    =
    \begin{bmatrix}
      a_{11}-b_{11} & a_{12}-b_{12} \
      a_{21}-b_{21} & a_{22}-b_{22}
    \end{bmatrix}
\end{equation}
$$
Addition works exactly the same way:
$$
\begin{equation}
    A + B =
    \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix} +
    \begin{bmatrix} b_{11} & b_{12} \
      b_{21} & b_{22}
    \end{bmatrix}
    =
    \begin{bmatrix}
      a_{11}+b_{11} & a_{12}+b_{12} \
      a_{21}+b_{21} & a_{22}+b_{22}
    \end{bmatrix}
\end{equation}
$$
An important point to know about matrix addition and subtraction is that it is only defined when $A$ and $B$ are of the same size.  Here, both are $2 \times 2$.  Since operations are performed element by element, these two matrices must be conformable- and for addition and subtraction that means they must have the same numbers of rows and columns.  I like to be explicit about the dimensions of matrices for checking conformability as I write the equations, so write
$$
A_{2 \times 2} + B_{2 \times 2}= \begin{bmatrix}
  a_{11}+b_{11} & a_{12}+b_{12} \
  a_{21}+b_{21} & a_{22}+b_{22}
\end{bmatrix}_{2 \times 2}
$$
Notice that the result of a matrix addition or subtraction operation is always of the same dimension as the two operands.
Let's define another matrix, B, that is also $2 \times 2$ and add it to A:
Can you write Python code for it?


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Check that $C\times C^{-1} = I$:
Can you write Python code for it?

print C.dot(C_inverse)
print "Is identical to:"
print C_inverse.dot(C)


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Closed-form Linear Regression
And compute the parametes $\beta_0$  and $\beta_1$ according to
$$ \beta = (X^\prime X)^{-1} X^\prime y $$

Note:
This not only looks elegant but can also be written in Julia code. However, matrix inversion $M^{-1}$ requires $O(d^3)$ iterations for a $d\times d$ matrix.<br />
https://www.coursera.org/learn/ml-regression/lecture/jOVX8/discussing-the-closed-form-solution
Can you write Python code for it?

beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)

yhat = X.dot(beta)
yhat.shape

plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=1, color="red")


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Concatenate, Reshape
Numpy load, save data files
Similarity: Euclidian vs Cosine
Example Nearest Neighbor search
Nearest Neighbor search is a common technique in Machine Learning
Can you write Python code for it?

### Pure iterative Python ###
points = [[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]]
qPoint = [4,5,3]
minIdx = -1
minDist = -1
for idx, point in enumerate(points):  # iterate over all points
        dist = sum([(dp-dq)**2 for dp,dq in zip(point,qPoint)])**0.5  # compute the euclidean distance for each point to q
        if dist < minDist or minDist < 0:  # if necessary, update minimum distance and index of the corresponding point
            minDist = dist
            minIdx = idx

print 'Nearest point to q: ', points[minIdx]

# # # Equivalent NumPy vectorization # # #
import numpy as np
points = np.array([[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]])
qPoint = np.array([4,5,3])
minIdx = np.argmin(np.linalg.norm(points-qPoint,axis=1))  # compute all euclidean distances at once and return the index of the smallest one
print 'Nearest point to q: ', points[minIdx]


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Evaluation: Root-mean-square Deviation
The root-mean-square deviation (RMSD) or root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model or an estimator and the values actually observed. The RMSD represents the sample standard deviation of the differences between predicted values and observed values. These individual differences are called residuals when the calculations are performed over the data sample that was used for estimation, and are called prediction errors when computed out-of-sample. The RMSD serves to aggregate the magnitudes of the errors in predictions for various times into a single measure of predictive power. RMSD is a good measure of accuracy, but only to compare forecasting errors of different models for a particular variable and not between variables, as it is scale-dependent.$^1$

$^1$  Hyndman, Rob J. Koehler, Anne B.; Koehler (2006). "Another look at measures of forecast accuracy". International Journal of Forecasting. 22 (4): 679–688. doi:10.1016/j.ijforecast.2006.03.001.
Can you write Python code for it?

import math
RSMD = math.sqrt(np.square(yhat_m-y_m).sum()/n)
print RSMD


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Example: Linear Regression
Linear regression is an approach for modeling the relationship between a scalar dependent variable $y$ and one or more explanatory variables (or independent variables) denoted $X$. The case of one explanatory variable is called simple linear regression. For more than one explanatory variable, the process is called multiple linear regression.$^1$ (This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.$^2$
We assume that the equation
$y_i = \beta_0 + \beta_1 X_i + \epsilon_i$ where $\epsilon_i \approx N(0, \sigma^2)$

$^1$ David A. Freedman (2009). Statistical Models: Theory and Practice. Cambridge University Press. p. 26. A simple regression equation has on the right hand side an intercept and an explanatory variable with a slope coefficient. A multiple regression equation has two or more explanatory variables on the right hand side, each with its own slope coefficient
$^2$ Rencher, Alvin C.; Christensen, William F. (2012), "Chapter 10, Multivariate regression – Section 10.1, Introduction", Methods of Multivariate Analysis, Wiley Series in Probability and Statistics, 709 (3rd ed.), John Wiley & Sons, p. 19, ISBN 9781118391679.
Can you write Python code for it?

n = 100  # numeber of samples
Xr = np.random.rand(n)*99.0
y = -7.3 + 2.5*Xr + np.random.randn(n)*27.0
plt.plot(Xr, y, "o", alpha=0.5)


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Matrix Addition and Subtraction
Adding or subtracting a scalar value to a matrix
To learn the basics, consider a small matrix of dimension $2 \times 2$, where $2 \times 2$ denotes the number of rows $\times$ the number of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Consider adding a scalar value (e.g. 3) to the A.
$$
\begin{equation}
    A+3=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix}+3
    =\begin{bmatrix}
      a_{11}+3 & a_{12}+3 \
      a_{21}+3 & a_{22}+3
    \end{bmatrix}
\end{equation}
$$
The same basic principle holds true for A-3:
$$
\begin{equation}
    A-3=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix}-3
    =\begin{bmatrix}
      a_{11}-3 & a_{12}-3 \
      a_{21}-3 & a_{22}-3
    \end{bmatrix}
\end{equation}
$$
Notice that we add (or subtract) the scalar value to each element in the matrix A.  A can be of any dimension.
This is trivial to implement, now that we have defined our matrix A:
Can you write Python code for it?

result = A + 3
#or
result = 3 + A
print result


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Matrix Multiplication
Multiplying a scalar value times a matrix
As before, let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Suppose we want to multiply A times a scalar value (e.g. $3 \times A$)
$$
\begin{equation}
    3 \times A = 3 \times \begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22}
    \end{bmatrix}
    =
    \begin{bmatrix}
      3a_{11} & 3a_{12} \
      3a_{21} & 3a_{22}
    \end{bmatrix}
\end{equation}
$$
is of dimension (2,2).  Scalar multiplication is commutative, so that $3 \times A$=$A \times 3$.  Notice that the product is defined for a matrix A of any dimension.
Similar to scalar addition and subtration, the code is simple:
Can you write Python code for it?


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
One important property of transposing a matrix is the transpose of a product of two matrices.  Let matrix A be of dimension $N \times M$ and let B of of dimension $M \times P$.  Then
$$
\begin{equation}
    (AB)^{\prime}=B^{\prime}A^{\prime}
\end{equation}
$$
For more information, see this http://en.wikipedia.org/wiki/Matrix_transposition on matrix transposition.  This is also easy to implement:
Can you write Python code for it?

print B.T.dot(A.T)
print "Is identical to:"
print (A.dot(B)).T


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
This code sets up Ipython Notebook environments (lines beginning with %), and loads several libraries and functions.  The core scientific stack in python consists of a number of free libraries.  The ones I have loaded above include:

sympy: provides for symbolic computation (solving algebra problems)
numpy: provides for linear algebra computations
matplotlib.pyplot: provides for the ability to graph functions and draw figures
scipy: scientific python provides a plethora of capabilities
seaborn: makes matplotlib figures even pretties (another library like this is called bokeh).  This is entirely optional and is purely for eye candy.

Creating arrays, scalars, and matrices in Python
Scalars can be created easily like this:
Can you write Python code for it?


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Transposing a Matrix
At times it is useful to pivot a matrix for conformability- that is in order to matrix divide or multiply, we need to switch the rows and column dimensions of matrices.  Consider the matrix
$$
\begin{equation}
    A_{3 \times 2}=\begin{bmatrix}
      a_{11} & a_{12} \
      a_{21} & a_{22} \
      a_{31} & a_{32}
    \end{bmatrix}{3 \times 2}
\end{equation}
$$
The transpose of A (denoted as $A^{\prime}$) is
$$
\begin{equation}
   A^{\prime}=\begin{bmatrix}
      a{11} & a_{21} & a_{31} \
      a_{12} & a_{22} & a_{32} \
    \end{bmatrix}_{2 \times 3}
\end{equation}
$$
Can you write Python code for it?

A = np.arange(6).reshape((3,2))
B = np.arange(8).reshape((2,4))
print "A is"
print A

print "The Transpose of A is"
print A.T


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
Vectors and Lists
The numpy library (we will reference it by np) is the workhorse library for linear algebra in python.  To creat a vector simply surround a python list ($[1,2,3]$) with the np.array function:
Can you write Python code for it?

x_vector = np.array([1,2,3])
print x_vector


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
We could have done this by defining a python list and converting it to an array:
Can you write Python code for it?

c_list = [1,2]
print "The list:",c_list
print "Has length:", len(c_list)

c_vector = np.array(c_list)
print "The vector:", c_vector
print "Has shape:",c_vector.shape

z = [5,6]
print "This is a list, not an array:",z
print type(z)


I am working on the file "03-NumPy-and-Linear-Algebra/Introduction_orig.ipynb".
The first task is:
We will use the numpy dot operator to perform the these multiplications. You can use it two ways to yield the same result:
Can you write Python code for it?


I am working on the file "03-Operationalizing/Operationalizing.ipynb".
The first task is:
3. Arithmetic!
Can you write Python code for it?

# Our dataframe

spam_df

# Pandas will produce a few descriptive statistics for each column

spam_df.describe()

# Multiply entries of the dataframe by 10

spam_df*10

# Add 10 to each entry

spam_df+10

# Of course our dataframe hasn't changed

spam_df

# What if we just want to add the values in the column?

sum(spam_df['Bacon'])

# We can also perform operations among columns
# Pandas knows to match up individual entries in each column

spam_df['Bacon']/spam_df['Eggs']


I am working on the file "03-Operationalizing/Operationalizing.ipynb".
The first task is:
Extending our Methods
Beyond a simple list, we often find it useful to organize information into lists of lists. That is, a list in which each entry is itself a list of elements. For example, we may not want to treat a poem as a flat list of words but instead would like to group words into their constitutive lines.
Can you write Python code for it?

# There's a twist!

first_sonnet.split('\n')

# Assign the list of whole lines to a new variable
sonnet_lines = first_sonnet.split('\n')

# How long is this list?

len(sonnet_lines)

# Create a list of lists!

[line.split() for line in sonnet_lines]

# Assign this to a variable

tokens_by_line = [line.split() for line in sonnet_lines]

# Check its length

len(tokens_by_line)

# Pull out the second line

tokens_by_line[1]

# How long is that second line?

len(tokens_by_line[1])

# Pull up an individual word

tokens_by_line[1][3]

## EX. Retrieve the word 'thereby' from the list of 'tokens_by_line' by calling its indices.

## EX. Find the average number of words per line in 'tokens_by_line'.


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:

3. The apply function
We can also apply a function to each row. To get a word count of a text file we would take the length of the split string like this:
len(text_split)
If we want to do this on every row in our dataframe, we can use the apply() function.
Can you write Python code for it?

df['word_count'] = df['text_split'].apply(len)
df


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:

5. Extracting Text from a Dataframe
We may want to extract the text from our dataframe, to do further analyses on the text only. We can do this using the tolist() function and the join() function.
Can you write Python code for it?

novels = df['text'].tolist()
print(novels[:1])

#turn all of the novels into one long string using the join function
cat_novels = ''.join(n for n in novels)
print(cat_novels[:100])


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:

4. Applying NLTK Functions and the lambda function
If we want to apply nltk functions we can do so using .apply().  If we want to use list comprehension on the split text, we have to introduce one more Python trick: the lambda function. This simply allows us to write our own function to apply to each row in our dataframe. For example, we may want tokenize our text instead of splitting on the white space. To do this we can use the lambda function.
Note: If you want to explore lambda functions more, see the notebook titled A-Bonus_LambdaFunctions.ipynb in this folder.
Because of the length of the novels tokenizing the text takes a bit of time. We'll instead tokenize the title only.
Can you write Python code for it?

df['title_tokens'] = df['title'].apply(nltk.word_tokenize)
df['title_tokens']


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:

1. Descriptive Statistics and Visualization
The first thing we probably want to do is describe our data, to make sure everything is in order. We can use the describe function for the numerical data, and the value_counts function for categorical data.
Can you write Python code for it?

print(df.describe()) #get descriptive statistics for all numerical columns
print()
print(df['author gender'].value_counts()) #frequency counts for categorical data
print()
print(df['year'].value_counts()) #treat year as a categorical variable
print()
print(df['year'].mode()) #find the year in which the most novels were published


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:
Notice this is a typical dataframe, possibly with more columns as strings than numbers. The text in contained in the column 'text'.
Notice also there are missing texts. For now, we will drop these texts so we can move forward with text analysis. In your own work, you should justify dropping missing texts when possible.
Can you write Python code for it?

df = df.dropna(subset=["text"])
df

##Ex: Print the first text in the dataframe (starts with "A DOG WITH A BAD NAME").
###Hint: Remind yourself about the syntax for slicing a dataframe


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:
Oops! That doesn't look right! Python automatically converted the year to scientific notation. We can set that option to False.
Can you write Python code for it?

plt.ticklabel_format(useOffset=False) #forces Python to not convert numbers
grouped_year['text'].count().plot(kind = 'line')
plt.show()


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:
We can graph this via a line graph.
Can you write Python code for it?

grouped_year['text'].count().plot(kind = 'line')
plt.show()


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:
We haven't done any text analysis yet. Let's apply some of our text analysis techniques to the text, add columns with the output, and analyze/visualize the output.

2. The str attribute
Luckily for us, pandas has an attribute called 'str' which allows us to access Python's built-in string functions.
For example, we can make the text lowercase, and assign this to a new column.
Note: You may get a "SettingWithCopyWarning" highlighted with a pink background. This is not an error, it is Python telling you that while the code is valid, you might be doing something stupid. In this case, the warning is a false positive. In most cases you should read the warning carefully and try to fix your code.
Can you write Python code for it?

df['text_lc'] = df['text'].str.lower()
df

##Ex: create a new column, 'text_split', that contains the lower case text split into list.
####HINT: split on white space, don't tokenize it.


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:
What is the average length of each novel in our data? With pandas, this is easy!
Can you write Python code for it?


I am working on the file "03-Pandas_and_DTM/00-PandasAndTextAnalysis.ipynb".
The first task is:
With this tokenized list we might want to, for example, remove punctuation. Again, we can use the lambda function, with list comprehension.
Can you write Python code for it?

df['title_tokens_clean'] = df['title_tokens'].apply(lambda x: [word for word in x if word not in list(string.punctuation)])
df['title_tokens_clean']


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords.ipynb".
The first task is:
How do we know what word corresponds to each number? We can access the words themselves through the CountVectorizer function get_feature_names().
Can you write Python code for it?


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords.ipynb".
The first task is:
Let's look at the 20 words with highest tf-idf weights.
Can you write Python code for it?


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords.ipynb".
The first task is:
Now lets compare the words with the highest tf-idf weight for each genre.
Note: there are other ways to do this. Challenge: what is a different approach to identifying rows from a certain genre in our dtm?
Can you write Python code for it?

#pull out the reviews for three genres, Rap, Alternative/Indie Rock, and Jazz
dtm_rap = dtm_tfidf_df_genre[dtm_tfidf_df_genre['GENRE']=="Rap"]
dtm_indie = dtm_tfidf_df_genre[dtm_tfidf_df_genre['GENRE']=="Alternative/Indie Rock"]
dtm_jazz = dtm_tfidf_df_genre[dtm_tfidf_df_genre['GENRE']=="Jazz"]

#print the words with the highest tf-idf scores for each genre
print("Rap Words")
print(dtm_rap.max(numeric_only=True).sort_values(ascending=False)[0:20])
print()
print("Indie Words")
print(dtm_indie.max(numeric_only=True).sort_values(ascending=False)[0:20])
print()
print("Jazz Words")
print(dtm_jazz.max(numeric_only=True).sort_values(ascending=False)[0:20])


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords.ipynb".
The first task is:
Ok! We have successfully identified content words, without removing stop words and without part-of-speech tagging. What else do you notice about this list?

5. Identifying Distinctive Words
What can we do with this? These scores are best used when you want to identify distinctive words for individual documents, or groups of documents, compared to other groups or the corpus as a whole. To illustrate this, let's compare three genres and identify the most distinctive words by genre.
First we add the genre of the document into our dtm weighted by tf-idf scores, and then compare genres.
Can you write Python code for it?

#Copy our tfidf df to a new df to add genre
dtm_tfidf_df_genre = dtm_tfidf_df
#add a 'GENRE' column to our tfidf df
dtm_tfidf_df_genre['GENRE'] = df['genre']
#Question: Why is 'GENRE' in caps?

dtm_tfidf_df_genre


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords.ipynb".
The first task is:
Our next step is to turn the text into a document term matrix using the scikit-learn function called CountVectorizer. There are two ways to do this. We can turn it into a sparse matrix type, which can be used within scikit-learn for further analyses. We do this by "fitting" the text to our CountVectorizer object, which calculates the full vocabulary in our corpus, and then "transform" our text by counting the number of times each word occurs in each docuent. We combine these two steps by calling the fit_transform() function.
Let's first look at the documentation for CountVectorizer.
Can you write Python code for it?

#import the function CountVectorizer
from sklearn.feature_extraction.text import CountVectorizer

#Create our CountVectorizer object
countvec = CountVectorizer()

sklearn_dtm = countvec.fit_transform(df.body)
print(sklearn_dtm)


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords.ipynb".
The first task is:
Question: What does this tell us about our data?
What else does the DTM enable? Because it is in the format of a matrix, we can perform any matrix algebra or vector manipulation on it, which enables some pretty exciting things (think vector space and Euclidean geometry). But, what do we lose when we reprsent text in this format?
Today, we will use variations on the DTM to find distinctive words in this dataset.

4. Tf-idf scores
How to find content words in a corpus is a long-standing question in text analysis. We have seen a few ways of doing this: removing stop words and identifying and counting only nouns, verbs, and adjectives. Today, we'll learn one more simple approach to this: word scores. The idea behind words scores is to weight words not just by their frequency, but by their frequency in one document compared to their distribution across all documents. Words that are frequent, but are also used in every single document, will not be indicative of the content of that document. We want to instead identify frequent words that are unevenly distributed across the corpus.
One of the most popular ways to weight words (beyond frequency counts) is tf-idf scores. By offsetting the frequency of a word by its document frequency (the number of documents in which it appears) will in theory filter out common terms such as 'the', 'of', and 'and'; what we have been calling stop words.
More precisely, the inverse document frequency is calculated as such:
number_of_documents / number_documents_with_term
so:
tfidf_word1 = word1_frequency_document1 * (number_of_documents / number_document_with_word1)
You can, and often should, normalize the numerator:
tfidf_word1 = (word1_frequency_document1 / word_count_document1) * (number_of_documents / number_document_with_word1)
We can calculate this manually, but scikit-learn has a built-in function to do so. We'll use it, but a challenge for you: use Pandas to calculate this manually.
To do so, we simply do the same thing we did above with CountVectorizer, but...
Can you write Python code for it?

#import the function
from sklearn.feature_extraction.text import TfidfVectorizer

tfidfvec = TfidfVectorizer()

#create the dtm, but with cells weigthed by the tf-idf score.
dtm_tfidf_df = pandas.DataFrame(tfidfvec.fit_transform(df.body).toarray(), columns=tfidfvec.get_feature_names(), index = df.index)

#view results
dtm_tfidf_df


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords.ipynb".
The first task is:
This format is called Compressed Sparse Format. It save a lot of memory to store the dtm in this format, but it is difficult to look at for a human. To illustrate the techniques in this lesson we will first convert this matrix back to a Pandas dataframe, a format we're more familiar with.
Note: This is a case of do as I say, not as I do. As we continue we will rarely transform a DTM into a Pandas dataframe, because of memory issues. I'm doing it today so we can understand the intuition behind the DTM, word scores, and distinctive words.
Can you write Python code for it?

#we do the same as we did above, but covert it into a Pandas dataframe. Note this takes quite a bit more memory, so will not be good for bigger data.
#don't understand this code? we'll go through it, but don't worry about understanding it.
dtm_df = pandas.DataFrame(countvec.fit_transform(df.body).toarray(), columns=countvec.get_feature_names(), index = df.index)

#view the dtm dataframe
dtm_df


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords_ExerciseSolutions.ipynb".
The first task is:
1. Explore the Data using Pandas
You folks are experts at this now. Write Python code using pandas to do the following exploration of the data:

What different genres are in the data?
Who are the reviewers?
Who are the artists?
What is the average score given?
What is the average score by genre? What is the genre with the highest average score?
Can you write Python code for it?

print(df['genre'].value_counts())
print(df['critic'].value_counts())
print(df['artist'].value_counts())

##Ex: print the average number of times each word is used in a review
#Print this out sorted from highest to lowest.

print(dtm_df.mean().sort_values(ascending=False))


I am working on the file "03-Pandas_and_DTM/01-DTM_DistinctiveWords_ExerciseSolutions.ipynb".
The first task is:
Ex: Compare the distinctive words for two artists in the data
Note: the artists should have a number of reviews, so check your frequency counts to identify artists.
HINT: Copy and paste the above code and modify it as needed.
Can you write Python code for it?

print(df['artist'].value_counts())

#merge this into the dtm_tfidf_df
dtm_tfidf_df_artist = dtm_tfidf_df
dtm_tfidf_df['ARTIST'] = df['artist']
#pull out the reviews for three genres, Rap, Alternative/Indie Rock, and Jazz
dtm_rem_artist = dtm_tfidf_df_artist[dtm_tfidf_df_artist['ARTIST']=="R.E.M."]
dtm_af_artist = dtm_tfidf_df_artist[dtm_tfidf_df_artist['ARTIST']=="Arcade Fire"]

#print the words with the highest tf-idf scores for each genre
print("R.E.M. Words")
print(dtm_rem_artist.max(numeric_only=True).sort_values(ascending=False)[0:20])
print()
print("Arcade Fire Words")
print(dtm_af_artist.max(numeric_only=True).sort_values(ascending=False)[0:20])


I am working on the file "03-Pandas_and_DTM/01.1-Bonus_DifferenceOfProportions.ipynb".
The first task is:
Finally, we subtract one row from another, and add the output as a third row.
Can you write Python code for it?

novels_df.loc[2] = novels_df.loc[0] - novels_df.loc[1]
novels_df


I am working on the file "03-Pandas_and_DTM/01.1-Bonus_DifferenceOfProportions.ipynb".
The first task is:
Next we divide each frequency cell by the word count. This syntax gets a bit tricky, so let's walk through it.
Can you write Python code for it?

novels_df = novels_df.iloc[:,:].div(novels_df.word_count, axis=0)
novels_df


I am working on the file "03-Pandas_and_DTM/01.1-Bonus_DifferenceOfProportions.ipynb".
The first task is:
Notice the number of rows and columns.
Question: What does this mean?
Next, we need to get a word frequency count for each novel, which we can do by summing across the entire row. Note how the syntax is different here compared to when we summed one column across all rows.
Can you write Python code for it?

novels_df['word_count'] = novels_df.sum(axis=1)
novels_df


I am working on the file "03-Pandas_and_DTM/01.1-Bonus_DifferenceOfProportions.ipynb".
The first task is:
Stop words are still in there. Why?
We can, of course, manually remove stop words. This does successfully identify distinctive content words.
We can do this in the CountVectorizer step, by setting the correct option.
Can you write Python code for it?

#change stop_words option to 'english
countvec_sw = CountVectorizer(stop_words="english")

#same as code above
novels_df_sw = pandas.DataFrame(countvec_sw.fit_transform(text_list).toarray(), columns=countvec_sw.get_feature_names())
novels_df_sw['word_count'] = novels_df_sw.sum(axis=1)
novels_df_sw = novels_df_sw.iloc[:,0:].div(novels_df_sw.word_count, axis=0)
novels_df_sw.loc[2] = novels_df_sw.loc[0] - novels_df_sw.loc[1]
novels_df_sw.loc[2].sort_values(axis=0, ascending=False)


I am working on the file "03-Pandas_and_DTM/01.3-Bonus_LambdaFunctions.ipynb".
The first task is:
If we want to do something to the entire value of the column cells we can use the apply function.
For example, we want to take the length of the list we just created (the 'text_split' column), we can apply the len function.
Can you write Python code for it?

df['word_count'] = df['text_split'].apply(len)
df['word_count']


I am working on the file "03-Pandas_and_DTM/01.3-Bonus_LambdaFunctions.ipynb".
The first task is:
Sometimes we can't use the apply function alone, we must also use the lambda function. This is the case if the column contains a list, and we want to loop through the list. For example, if we want to remove punctuation from our title tokens, we can do this using list comprehension. If we try to do this using apply only we get an error.
Can you write Python code for it?

df['title_token_clean'] = df['title_token'].apply([word for word in df['title_token'] if word not in list(string.punctuation)])
df['title_token_clean']


I am working on the file "03-Pandas_and_DTM/01.3-Bonus_LambdaFunctions.ipynb".
The first task is:
The lambda function is like the apply function, but allows us to do more if needed. We can re-do what we did above using the lambda function.
Can you write Python code for it?

#apply the len function using .apply
df['word_count'] = df['text_split'].apply(len)
#apply the len function using lambda. This line does the same thing as line 2 above
df['word_count_lambda'] = df['text_split'].apply(lambda x: len(x))

#apply the nltk.word_tokenize function using .apply
df['title_token'] = df['title'].apply(nltk.word_tokenize)
#do the same using lambda. The next line does the same as line 7 above.
df['title_token_lambda'] = df['title'].apply(lambda x: nltk.word_tokenize(x))

df[['word_count', 'word_count_lambda','title_token', 'title_token_lambda']]


I am working on the file "03-Pandas_and_DTM/01.3-Bonus_LambdaFunctions.ipynb".
The first task is:
To make a list object callable and to define the variable to indicate each element in th list, we can write a lambda function.
Can you write Python code for it?

df['title_token_clean'] = df['title_token'].apply(lambda x: [word for word in x if word not in list(string.punctuation)])
df['title_token_clean']


I am working on the file "03-Project-McNulty/Cleaning_and_dummies.ipynb".
The first task is:
After going through the list, I have decided to drop 5 columns!
These will not be relevant to the task at hand. Although, I could use some natural language processig via NLTK to parse job descriptions and loan descriptions. I will leave this for another day.
It is also important to note that I will be dropping variables that hint (i.e. information leakage) at what the final result will be.
Can you write Python code for it?

df.drop(columns_to_drop, axis=1, inplace=True)

df.shape #just what we expected


I am working on the file "03-Project-McNulty/Cleaning_and_dummies.ipynb".
The first task is:
After reviewing the above, the following columns need to be changed to categorical datatypes from float64.
- policy_code
I will first make it an object datatype as later I will write a function that changed all object datatypes into categorical datatypes.
Can you write Python code for it?


I am working on the file "03-Project-McNulty/Cleaning_and_dummies.ipynb".
The first task is:
I will manipulate my dataset in order for it to be compatible with both GLM's and classification algorithms.
As such, I will create dummies and scale all of data. Scaling is incredibly important for KNN and will improve model performance for Logisitc Regression.
I am not particularly concerned with coeff interpretability as the purpose is the assign a class.
By manipulating my data in this way, I will be ready to be used by any ML model.
I will use 3 in particular:

Dummy Classifier. This will the global baseline I have to beat.

KNN (My most constrained model)

Logistic Regression

Random Forests

Note that Multi-collinearity does NOT matter for models like DT's and RF's - however, it will matter for Logistic regression. I will first throw all of my data at LGR, and remove variables (multi-collinear) accordingly (this creates a baselinne model for lgr).
My project will be concerned with classifying whether an individual will re-pay their loan on time. I will change the 'loan status' feature in this dataset into a binary form of "Fully Paid" or "Late"
Can you write Python code for it?

df['loan_status'].unique()

mask = df['loan_status'] != "Fully Paid"
rows_to_change = df[mask]
rows_to_change.loc[:, 'loan_status'] = 'Late'
df.update(rows_to_change)

df['loan_status'].unique() #sweet!

df.shape # no dimensionality lost

plot_corr_matrix(df)


I am working on the file "03-Project-McNulty/Cleaning_and_dummies.ipynb".
The first task is:
My categorical features (those of type Object) have np.nan values, I will change these to something more meaningful like "Missing Data".
I will then create dummies for all of my categorical features. This will lead to an explosion in the number of columns - this will be more computationally expensive, however, this is NOT an explosion in the 'feature space' as our dataframe contains the same amount of information.
Can you write Python code for it?

object_columns = df.select_dtypes(include=['object']).columns

for c in object_columns:
    df.loc[df[df[c].isnull()].index, c] = "missing"


I am working on the file "03-Project-McNulty/Cleaning_and_dummies.ipynb".
The first task is:
So, our dataset is comprised of features which are categorical and features that are numeric. We need to ensure that the object datatypes are converted to categorical datatypes.
Also, whether we use a GLM or classifier, we need to ensure that these datatypes stay consistent.
NOTE: changing columns to categorical datatypes will NOT change how a machine learning model interprets the data. i.e. The algorithm will still think that 5 > 4. As such, one hot encoding (i.e. making dummies) is the only way to ensure that a Machine Learning Model can detect the presence of a particular attribute.
I will be changing the object datatypes to categorical purely for data consistency within the dataframe.
Can you write Python code for it?

obj_df = df.select_dtypes(include=['object'])

obj_df_cols = obj_df.columns

for col in obj_df_cols:
    df[col] = df[col].astype("category")

df.dtypes.unique() #This is what we wanted!

df.shape

df.head()

unique_val_dict = {}
for col in df.columns:
    if col not in unique_val_dict:
        unique_val_dict[col] = df[col].unique()

unique_val_dict #will use this later when making flask app.

category_columns = df.select_dtypes(include=['category']).columns
df = pd.get_dummies(df, columns=category_columns, drop_first=True)

df.shape


I am working on the file "03-Project-McNulty/Random_Forest_Parameter_Tuning.ipynb".
The first task is:
Below I created a simple for loop to test out our random forest model with a range of different parameters.
A much cleaner implementation of this would be via gridsearchCV, however, due to some technical difficulties on AWS. I have had to implment it this way!
Later on in this notebook, I fun a loop with the best model I could find. Intermediate testing and analysis has been ommitted in this notebook for brevity.
Can you write Python code for it?

model_params = [{"criterion":'entropy', "min_samples_split": 10, "class_weight": 'balanced'}, {"criterion":'entropy', "min_samples_split":9, "class_weight": 'balanced'}, {"criterion":'entropy', "min_samples_split":8, "class_weight": 'balanced'}, {"criterion":'entropy', "min_samples_split":7, "class_weight": 'balanced'}, {"criterion":'entropy', "min_samples_split":6, "class_weight": 'balanced'}, {"criterion":'entropy', "min_samples_split":5, "class_weight": 'balanced'}, {"criterion":'gini', "min_samples_split":10, "class_weight": 'balanced'}, {"criterion":'gini', "min_samples_split":9, "class_weight": 'balanced'}, {"criterion":'gini', "min_samples_split":8, "class_weight": 'balanced'}, {"criterion":'gini', "min_samples_split":7, "class_weight": 'balanced'}, {"criterion":'gini', "min_samples_split":6, "class_weight": 'balanced'}, {"criterion":'gini', "min_samples_split":5, "class_weight": 'balanced'}]

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=222)

def best_features(feature_importance_list):
	all_features = []

	for index, importance_val in enumerate(feature_importance_list):
		all_features.append((index,importance_val))

	all_features = sorted(all_features, key=lambda x: x[1], reverse=True)

	best_features = all_features[:35]

	for i in best_features:
	    print(df.columns[i[0]])
	    try:
	        print(lookup_description(df.columns[i[0]]))
	    except:
	        print("DUMMY VARIABLE!")
	    print()

target_names = ["PAID", "CHARGED OFF"]

for param_dict in model_params:
	model = RandomForestClassifier(n_estimators=23, n_jobs=-1, criterion=param_dict['criterion'], min_samples_split=param_dict["min_samples_split"], class_weight=param_dict["class_weight"], random_state=222)
	print(param_dict)
	print()
	y_pred = cross_val_predict(model, X, y, cv=cv, n_jobs=-1)
	print()
	print(classification_report(y, y_pred,target_names=target_names))
	print()
	cnf_matrix = confusion_matrix(y, y_pred)
	print("Accuracy Score: ", np.median(accuracy_score(y, y_pred)))
	print()
	print("F1 score - Binary: ",...


I am working on the file "03-Project-McNulty/Random_Forest_Parameter_Tuning.ipynb".
The first task is:
From the above we can see that by having no weights, we obtain a significatnly worse F1 Binary score!
What if we increase the min_sample_split variable?
Upon doing some experimentation (ommitted in this notebook) I found that the following random forest performed the best wtih regards to F1 score and accuracy:

'min_samples_split': 40
'class_weight': 'balanced'
'criterion': 'entropy'
n_estimators = 50

I will now re-run this model will 100 estimators and 11 fold cross validation to produce our final model! I will also pickle this model for later use!
Can you write Python code for it?

final_cv = StratifiedKFold(n_splits=11, shuffle=True, random_state=222)

final_model = RandomForestClassifier(n_estimators=100, n_jobs=-1, criterion='entropy', min_samples_split=40, class_weight='balanced', random_state=223)

y_pred = cross_val_predict(final_model, X, y, cv=final_cv, n_jobs=-1)

print(classification_report(y, y_pred,target_names=target_names))

print()

cnf_matrix = confusion_matrix(y, y_pred)

print("Accuracy Score: ", np.median(accuracy_score(y, y_pred)))

print()

print("F1 score - Binary: ", np.median(f1_score(y, y_pred, average='binary')))
print()
print("F1 score - Micro: ", np.median(f1_score(y, y_pred, average='micro')))
print()
print("F1 score - Weighted: ", np.median(f1_score(y, y_pred, average='weighted')))
print()
show_confusion_matrix(cnf_matrix, class_labels=target_names)
print()
fit_object = final_model.fit(X,y)
print()

best_features(fit_object.feature_importances_)
print()
joblib.dump(fit_object, 'rf_model_final.pkl');

proba = cross_val_predict(final_model, X, y, cv=final_cv, method='predict_proba')

proba = unpickle_object("proba_rf_model.pkl")

fpr, tpr, thresholds = roc_curve(y, proba[:, 1])

fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(2):
    fpr[i], tpr[i], _ = roc_curve(y, proba[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])

lw = 2

trace1 = go.Scatter(x=fpr[1], y=tpr[1],
                    mode='lines',
                    line=dict(color='darkorange', width=lw),
                    name='ROC curve (area = %0.2f)' % roc_auc[1]
                   )

trace2 = go.Scatter(x=[0, 1], y=[0, 1],
                    mode='lines',
                    line=dict(color='navy', width=lw, dash='dash'),
                    showlegend=False)

layout = go.Layout(title='Receiver operating characteristic example',
                   xaxis=dict(title='False Positive Rate'),
                   yaxis=dict(title='True Positive Rate'))

fig = go.Figure(data=[trace1, trace2], layout=layout)

py.iplot(fig);

Image("newplot.png") #static image as pyplot isnt interactive when...


I am working on the file "03-Project-McNulty/feature_reduction_35.ipynb".
The first task is:
This notebook will select the top 35 features from out dataset.
I will rescale the resulting columns - while I am keenly aware this makes no difference to the Random Forest Model, I am just doing it for consistency.
I also pickle the scaler as we will be using this in our flask web app to transform the input data.
Can you write Python code for it?

df = unpickle_object("dummied_dataset.pkl")

df.shape

#this logic will be important for flask data entry.

float_columns = df.select_dtypes(include=['float64']).columns

for col in float_columns:
    if "mths" not in col:
        df[col].fillna(df[col].median(), inplace=True)
    else:
        if col == "inq_last_6mths":
            df[col].fillna(0, inplace=True)
        elif col == "mths_since_last_delinq":
            df[col].fillna(999, inplace=True)
        elif col == "mths_since_last_record":
            df[col].fillna(999, inplace=True)
        elif col == "collections_12_mths_ex_med":
            df[col].fillna(0, inplace=True)
        elif col == "mths_since_last_major_derog":
            df[col].fillna(999, inplace=True)
        elif col == "mths_since_rcnt_il":
            df[col].fillna(999, inplace=True)
        elif col == "acc_open_past_24mths":
            df[col].fillna(0, inplace=True)
        elif col == "chargeoff_within_12_mths":
            df[col].fillna(0, inplace=True)
        elif col == "mths_since_recent_bc":
            df[col].fillna(999, inplace=True)
        elif col == "mths_since_recent_bc_dlq":
            df[col].fillna(999, inplace=True)
        elif col == "mths_since_recent_inq":
            df[col].fillna(999, inplace=True)
        elif col == "mths_since_recent_revol_delinq":
            df[col].fillna(999, inplace=True)

top_35 = ["int_rate",
          "dti",
          "term_ 60 months",
          "bc_open_to_buy",
          "revol_util",
          "installment",
          "avg_cur_bal",
          "tot_hi_cred_lim",
          "revol_bal",
          "funded_amnt_inv",
          "bc_util",
          "tot_cur_bal",
          "total_bc_limit",
          "total_rev_hi_lim",
          "funded_amnt",
          "loan_amnt",
          "mo_sin_old_rev_tl_op",
          "total_bal_ex_mort",
          "issue_d_Dec-2016",
          "total_acc",
          "mo_sin_old_il_acct",
          "mths_since_recent_bc",
          "total_il_high_credit_limit",
          "inq_last_6mths",
...


I am working on the file "03-Semantics-Variables.ipynb".
The first task is:
Again, this makes perfect sense if you think of x and y as pointers, and the "=" operator as an operation that changes what the name points to.
You might wonder whether this pointer idea makes arithmetic operations in Python difficult to track, but Python is set up so that this is not an issue. Numbers, strings, and other simple types are immutable: you can't change their value – you can only change what values the variables point to.
So, for example, it's perfectly safe to do operations like the following:
Can you write Python code for it?

x = 10
y = x
x += 5  # add 5 to x's value, and assign it to x
print("x =", x)
print("y =", y)


I am working on the file "03-Semantics-Variables.ipynb".
The first task is:
This behavior might seem confusing if you're wrongly thinking of variables as buckets that contain data.
But if you're correctly thinking of variables as pointers to objects, then this behavior makes sense.
Note also that if we use "=" to assign another value to x, this will not affect the value of y – assignment is simply a change of what object the variable points to:
Can you write Python code for it?

x = 'something else'
print(y)  # y is unchanged


I am working on the file "03-Semantics-Variables.ipynb".
The first task is:
We've created two variables x and y which both point to the same object.
Because of this, if we modify the list via one of its names, we'll see that the "other" list will be modified as well:
Can you write Python code for it?

print(y)

x.append(4) # append 4 to the list pointed to by x
print(y) # y's list is modified as well!


I am working on the file "03-Semantics-Variables.ipynb".
The first task is:
When we call x += 5, we are not modifying the value of the 10 object pointed to by x; we are rather changing the variable x so that it points to a new integer object with value 15.
For this reason, the value of y is not affected by the operation.
Everything Is an Object
Python is an object-oriented programming language, and in Python everything is an object.
Let's flesh-out what this means. Earlier we saw that variables are simply pointers, and the variable names themselves have no attached type information.
This leads some to claim erroneously that Python is a type-free language. But this is not the case!
Consider the following:
Can you write Python code for it?

x = 4
type(x)

x = 'hello'
type(x)

x = 3.14159
type(x)


I am working on the file "03-Semantics-Variables.ipynb".
The first task is:
When we say that everything in Python is an object, we really mean that everything is an object – even the attributes and methods of objects are themselves objects with their own type information:
Can you write Python code for it?


I am working on the file "03-Semantics-Variables.ipynb".
The first task is:
While it might be expected for compound objects like lists to have attributes and methods, what is sometimes unexpected is that in Python even simple types have attached attributes and methods.
For example, numerical types have a real and imag attribute that returns the real and imaginary part of the value, if viewed as a complex number:
Can you write Python code for it?


I am working on the file "03-Semantics-Variables.ipynb".
The first task is:
While users of statically-typed languages might miss the type-safety that comes with declarations like those found in C,
C
int x = 4;
this dynamic typing is one of the pieces that makes Python so quick to write and easy to read.
There is a consequence of this "variable as pointer" approach that you need to be aware of.
If we have two variable names pointing to the same mutable object, then changing one will change the other as well!
For example, let's create and modify a list:
Can you write Python code for it?


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Apostrof u tekstu označenom apostrofima:
Can you write Python code for it?


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Bitno je primetiti da se u promenljivoj nalazi ono što smo joj poslednje dodelili. Ono što se pre toga u njoj nalazilo - nestaje.
Promenljivoj se mogu dodeljivati i rezultati operacija, na primer:
Can you write Python code for it?


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Interakcija s korisnikom (user input)
Funkcija input se koristi za interakciju s korisnikom. Ona omogućava korisniku našeg programa da unosi tekst s tastature. Njen jedini argument je tekst koji prikazujemo korisniku (kako bismo mu objasnili kakav ulaz s tastature očekujemo). Njen izlaz je string koji sadrži ono što je korisnik uneo preko tastature. Ukoliko očekujemo broj kao ulaz, možemo upotrebiti funkcije za konverziju (int i float), čiji je argument string, a izlaz broj.
Can you write Python code for it?

name = input("What is your name?\n")
print("Hello, %s!" % name)

# The radius of the circle
radius = float(input("Please enter the circle radius: "))

# The value of the number pi, used throughout the program.
# If we want to use another value for pi, we only need to change it here
pi = 3.14159

# The circumference of the circle
circumference = 2 * radius * pi

# The area of the circle
area = radius * radius * pi

# Print the input and the result
print("The circle radius is %f and pi is %f." % (radius, pi))
print("The circumference is %f and the area is %f." % (circumference, area) )


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Jedan od operanada u tim izrazima može biti i sama promenljiva čiju vrednost računamo u izrazu, pod uslovom da smo joj ranije dodelili neku vrednost. U tom slučaju, "stara" vrednost će biti iskorišćena u računanju izraza, zatim će ta "stara" vrednost nestati, a promenljiva će sadržati novu vrednost - rezultat operacije. To pokazuje sledeći primer.
Can you write Python code for it?

a = 10
# Print the "old" value of a
print("Old a = %d" % a)
# New a is equal old a (which is 10) plus 1
a = a + 1
# Print the new a
print("New a = %d" % a)


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Navodnici u tekstu označenom apostrofima:
Can you write Python code for it?

print('"Python" is a programming language.')


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Neki od tipova podataka u Pajtonu

stringovi
celi brojevi (integer)
razlomljeni brojevi (float)
logički tip (boolean)

Za razliku od programskih jezika kao što je C, koji koriste statičke tipove, Pajton koristi dinamičke tipove. To znači da promenljiva nije predodređena da sadrži samo jedan tip, već se u nju može smestiti objekat bilo kog tipa, jer Pajton informacije o tipu čuva uz objekat, a ne uz promenljivu (kao što to radi C).
Informaciju o tipu podatka koji nosi promenljiva možemo dobiti korišćenjem funkcije <i>type</i>.
Can you write Python code for it?

# A string
my_variable = "This is a string."
print(my_variable)
print(type(my_variable))
print("\n")

# An integer
my_variable = 3
print(my_variable)
the_type = type(my_variable)
print(the_type)
print("\n")

# A float
my_variable = 3.5
print(my_variable)
print(type(my_variable))
print("\n")

# A boolean
my_variable = 3 > 5
print(my_variable)
print(type(my_variable))


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Obratite pažnju na to kako je kod obojen. Videćete da je tekst prekinut ispred reči "small", a novi tekst je otpočet nakon reči "small". Sama reč "small" je van teksta i Pajton pokušava da je interpretira, što dovodi do sintaksne greške.
Silazna kosa crta ("\") kao karakter ima posebno značenje u Pajtonu, kada se nalazi "u okviru teksta" (između navodnika). Naziva se escape karakterom. Korišćenjem ovog karaktera možemo reći Pajtonu da je karakter koji sledi deo teksta, čak i ukoliko se radi o "specijalnom" karakteru - u ovom slučaju navodnici, karakter koji služi za označavanje početka i završetka tesksta. Na taj način između navodnika možemo imati navodnike, koji se interpretiraju kao deo teksta.
Can you write Python code for it?


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Operator za formatiranje stringa
Izmenićemo prethodni program tako da u dve linije prikazuje nešto poput sledećeg teksta:

The circle radius is 5 and pi is 3.141590.

The circumference is 31.415900 and the area is 78.539750.

Svaki poziv funkcije print prikazuje tekst u zasebnom redu. Ovo nam onemogućava da tekst i rezultat prikažemo u istom redu. Korišćenjem operatora za formatiranje stringa možemo prevazići ovo ograničenje.

Kad se koristi nad brojevima, znak % odgovara operatoru modulusa, ili ostatka pri deljenju. Kad se koristi nad tekstualnim podacima (stringovima), znak % označava operator formatiranja stringa, i služi za dodavanje brojeva, teksta ili drugih odgovarajućih podataka u već postojeći string (tekst).
Can you write Python code for it?

# The radius of the circle
radius = 5

# The value of the number pi, used throughout the program.
# If we want to use another value for pi, we only need to change it here
pi = 3.14159

# The circumference of the circle
circumference = 2 * radius * pi

# The area of the circle
area = radius * radius * pi

# Print the input and the result
print("The circle radius is %d and pi is %f." % (radius, pi))
print("The circumference is %f and the area is %f." % (circumference, area) )


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Pajton očekuje da string počet navodnicima završimo u istom redu i ukoliko napišemo naredbu gde string počinje u jednom redu (otvoreni navodnici) a završava se u sledećem redu (zatvoreni navodnici), izazvaćemo sintaksnu grešku, jer Pajton ne očekuje "specijalan karakter" za novi red u sred stringa (kao u naredbi iznad). Ukoliko ipak želimo da "razbijemo" string u više linija, svaki novi red, koji Pajton shvata kao specijalan karakter, možemo "izbeći" <i>escape</i> karakterom. Sledi primer.
Can you write Python code for it?

print("This is a one-line string but it's a little too long so I want to break\
it into two lines")


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Pokušajte da funkciji <i>print</i> prosledite tekst koji sadrži novi red.
Can you write Python code for it?


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Ponašanje znakova i operatora s različitim tipovima podataka
Različiti znakovi označavaju različite operatore, u zavisnosti od tipa podataka. Na primer, procenat (%) kod celih brojeva označava ostatak pri deljenju, a kod stringova operator formatiranja stringa.
Zbir dva cela broja je ceo broj.
Can you write Python code for it?

# Adding two integers
sum_of_integers = 3 + 5
print(sum_of_integers)
print(type(sum_of_integers))


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Silazna kosa crta (backslash) kao karakter produžetka linije
Silazna kosa crta (<i>backslash</i>) može se koristiti i za razbijanje jedne linije koda u više linija. Koristi se za preglednost koda i nema nikakvog uticaja na izvršavanje programa. Sledi nekoliko primera.
Can you write Python code for it?


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Tipovi podataka u Pajtonu
Do sada smo uglavnom koristili tip podataka koji se naziva string. Stringovi u Pajtonu služe za prikazivanje teksta, koji može sadržati slova, cifre, interpunkcijske znake itd. Uglavnom se koristi za komunikaciju sa korisnikom. Pajton ga prepoznaje zahvaljujući navodnicima, apostrofima, ili trostrukim apostrofima, kao što smo videli u prethodnim primerima.
Pored stringova, postoje i tipovi koji služe za prikazivanje brojeva. U prethodnim primerima smo koristili celobrojni tip (integer) i razlomljeni tip (float). Od engleskih naziva potiču i njihovi formati koji se koriste u kombinaciji s operatorom formatiranja stringa - %d za celobrojni tip (d od decimal) i %f (f od float).
Probajte da prikažete celobrojni tip koristeći format razlomljenog tipa i obrnuto. Pajton će automatski uraditi konverziju tipa podataka u odgovarajući tip. Konverzijom se može izgubiti deo informacije, na primer, konverzijom razlomljenog broja u celobrojni tip, dobićemo zaokružen broj kao rezultat.
Can you write Python code for it?

print("This is an integer: %f" % 3)

print("This is a float: %d" % 3.35)


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Trostruki navodnici
Trostruki navodnici označavaju tekst, baš kao i navodnici i apostrofi, uz pogodnost da se između njih mogu naći svi specijalni karakteri - apostrofi, navodnici, oznake novog reda... Jedino za šta bismo morali koristiti "escape" karakter jesu trostruki navodnici, jer oni označavaju kraj teksta otpočetog trostrukim navodnicima.
Can you write Python code for it?

print("""First line.
Second line.
All kinds of "special" characters here! Even backslashes! \ \\ \\\ """)


I am working on the file "03-Stringovi-promenljive-tipovi-podataka-interakcija.ipynb".
The first task is:
Znak "+" kod brojeva označava sabiranje, a kod stringova konkatenaciju (pripajanje drugog stringa prvom).
Can you write Python code for it?

# Add numbers
number1 = 10
number2 = 11
print(number1 + number2)
print("\n")

# Concatenate strings
str1 = "My name is "
str2 = "John Smith"
print(str1 + str2)


I am working on the file "03-adult-income-by-census.ipynb".
The first task is:
Above plot shows percentage of population with respect to education and income, and it seems people with Masters and PhD tend to earn to more (more number of people are in >50K bucket).
Intuition 2:
People earn more as they get more experience.
Can you write Python code for it?


I am working on the file "03-adult-income-by-census.ipynb".
The first task is:
Data exploration
Can you write Python code for it?

def read_data(path):
    return pd.read_csv(path,
                       index_col=False,
                       skipinitialspace=True,
                       names=['age', 'workclass', 'fnlwgt', 'education', 'education_num',
                           'marital_status', 'occupation', 'relationship', 'race', 'sex',
                           'capital_gain', 'capital_loss', 'hours_per_week', 'native_country',
                           'income']
                       )

train = read_data('./data/adult/adult.data')
test = read_data('./data/adult/adult.test')

train = train.append(test)
train.head()

train.hist(figsize=(12, 9))


I am working on the file "03-adult-income-by-census.ipynb".
The first task is:
First plot shows distribution of age with respect to income <= 50K. Age is used as an proxy to experience. Assumption here is people continue to work as they age and acquire more skills in the process. As per intuition, number of people making less than 50K decreases as per age.
Second plot shows income > 50K. More interestingly, data shows a peak around 45. This indicates either there aren't enough poeple of age 45+ earning more than 50K in the data or income decreases as people approach retirement.
Feature construction
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder, OneHotEncoder

lencoder = LabelEncoder()
oencoder = OneHotEncoder()

features = pd.DataFrame()
features['age'] = train['age']
features['education_num'] = train['education_num']
features['hours_per_week'] = train['hours_per_week']
features['fnlwgt'] = train['fnlwgt']
features['sex'] = lencoder.fit_transform(train.sex)
features['occupation'] = lencoder.fit_transform(train.occupation)

features.income = train.income
features.income = lencoder.fit_transform(features.income)
features.head()


I am working on the file "03-adult-income-by-census.ipynb".
The first task is:
Model fitting
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import train_test_split

x_train, x_test, y_train, y_test = train_test_split(features.drop('income'), features.income)

model = RandomForestClassifier()
model.fit(x_train, y_train)

y_hat = model.predict(x_test)


I am working on the file "03-ahorros.ipynb".
The first task is:
*Ejemplo.--* Realice el mismo ejemplo anterior, pero considerando que la tasa de interés es del 16% nominal a partir de 2016-06.
Can you write Python code for it?

## tasa de interés
nrate = cf.interest_rate([24] * 12, start='2016-1', freq='M', chgpts={5:16})

## depósitos
cflo = cf.cashflow(const_value=[100]*11 + [0], start='2016-1', freq='M')

## modelado
x = cf.savings(deposits =  cflo,  # deposito periodico
               initbal  =   100,  # balance inicial
               nrate    = nrate)  # tasa de interes mensual
x


I am working on the file "03-ahorros.ipynb".
The first task is:
*Ejemplo.--* Se tiene una cuenta con un saldo inicial de $ 100. Se hacen depósitos  al final de cada trimestre por $ 50 (se hará el primer depósito en 3 meses). La tasa nominal es del 36% con capitalización mensual y cambiará a 24% a partir del 13avo mes (incluido). ¿Cuál será el saldo al final de mes 24?.
Can you write Python code for it?

nrate = cf.interest_rate(const_value=[36]*24, start='2000-01', freq='M', chgpts={'2001-01':24})
nrate

cflo = cf.cashflow(const_value=0, periods=24, start='2000-01', freq='M')
cflo[[3*t-1 for t in range(1, 9)]] = 50
cflo

x = cf.savings(deposits =  cflo, # depósito períodico
               initbal  =   100, # balance inicial
               nrate    = nrate) # tasa de interés mensual
x


I am working on the file "03-classification-homework.ipynb".
The first task is:
Analysis of algorithms
Can you write Python code for it?

cls = OnlineSVMClassifier()

logistic_cls = OnlineLogisticRegressionClassifier()


I am working on the file "03-cloudlab-genilib.ipynb".
The first task is:
Important
This notebook is to be run inside Jupyter. If you see In [ ]: to the left of a cell, it means that this is an executable Jupyter cell.
To run a Jupyter cell, one of the followings can be done:
- Press the Run button in the tool bar
- Hit Shift-Enter
- Hit Ctrl-Enter
In an executable Jupyter cell, the ! denotes a Linux command (or a sequence of commands) that will be sent to execute in the CentOS VM. All Linux commands in shell will assume a starting directory that is the current directory of the notebook.
In an executable Jupyter cell, the %% at the first line of the cell denotes a cell magic (a single configuration option that directs how the cell is executed). %%writefile is a cell magic that instruct Jupyter to not execute the remainder of the cell, but to save them in to a file whose path is specified after the cell magic.
 Step 1. Set up emulab geni-lib package for CloudLab
Open a new terminal and run the following command:
$ sudo yum install -y unzip
$ conda create -n python2 python=2.7
$ source activate python2
$ conda install ipykernel
$ python -m ipykernel install --name python2 --user
$ conda install lxml

Restart your Jupyter Server
Reopen this notebook, go to Kernel, and change Kernel to Python 2
Can you write Python code for it?

!unzip codes/cloudlab/emulab-0.9.zip -d codes/cloudlab

!cd codes/cloudlab/emulab-geni-lib-1baf79cf12cb/;\
    source activate python2;\
    python setup.py install --user

!ls /home/lngo/.local/lib/python2.7/site-packages/

!rm -Rf codes/cloudlab/emulab-geni-lib-1baf79cf12cb/


I am working on the file "03-cloudlab-genilib.ipynb".
The first task is:
Step 2: Reload geni-lib for the first time
On the top bar of this notebook, select Kernel and then Restart
Can you write Python code for it?

%%writefile codes/cloudlab/xenvm.py

"""An example of constructing a profile with a single Xen VM from CloudLab.

Instructions:
Wait for the profile instance to start, and then log in to the VM via the
ssh port specified below.  (Note that in this case, you will need to access
the VM through a high port on the physical host, since we have not requested
a public IP address for the VM itself.)
"""

import geni.portal as portal
import geni.rspec.pg as rspec

# Create a Request object to start building the RSpec.
request = portal.context.makeRequestRSpec()
# Create a XenVM
node = request.XenVM("node")
node.disk_image = "urn:publicid:IDN+emulab.net+image+emulab-ops:CENTOS7-64-STD"
node.routable_control_ip = "true"

node.addService(rspec.Execute(shell="/bin/sh",
                              command="sudo yum update"))
node.addService(rspec.Execute(shell="/bin/sh",
                              command="sudo yum install -y httpd"))
node.addService(rspec.Execute(shell="/bin/sh",
                              command="sudo systemctl restart httpd.service"))
# Print the RSpec to the enclosing page.
portal.context.printRequestRSpec()


I am working on the file "03-cloudlab-genilib.ipynb".
The first task is:
Step 3: Test emulab geni-lib installation
Executing the cell below should produce an XML element with the following content:
<rspec xmlns:client="http://www.protogeni.net/resources/rspec/ext/client/1" xmlns:emulab="http://www.protogeni.net/resources/rspec/ext/emulab/1" xmlns:jacks="http://www.protogeni.net/resources/rspec/ext/jacks/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.geni.net/resources/rspec/3" xsi:schemaLocation="http://www.geni.net/resources/rspec/3 http://www.geni.net/resources/rspec/3/request.xsd" type="request">
  <rspec_tour xmlns="http://www.protogeni.net/resources/rspec/ext/apt-tour/1">
    <description type="markdown">An example of constructing a profile with a single Xen VM.</description>
    <instructions type="markdown">Wait for the profile instance to start, and then log in to the VM via the
ssh port specified below.  (Note that in this case, you will need to access
the VM through a high port on the physical host, since we have not requested
a public IP address for the VM itself.)
</instructions>
  </rspec_tour>
  <node client_id="node" exclusive="false">
    <sliver_type name="emulab-xen"/>
  </node>
</rspec>
Can you write Python code for it?

!source activate python2;\
  python codes/cloudlab/xenvm.py