I am working on the file "Session 3 - Classification.ipynb".
The first task is:
1.  Classes as Targets
Now that we've seen how to train and evaluate a linear model for real-valued responses, next we turn to classification.  At first glance, jumping from regression to classification seems trivial.  Say there are two classes, the first denoted by 0 and the second by 1.  We could just set each $y_{i}$ to 0 or 1 according to its class membership and fit a linear model just as before.
 Here's an example doing just that on some artificial data...
Can you write Python code for it?

### function for shuffling the data and labels
def shuffle_in_unison(a, b):
    rng_state = np.random.get_state()
    np.random.shuffle(a)
    np.random.set_state(rng_state)
    np.random.shuffle(b)

### calculate classification errors
# return a percentage: (number misclassified)/(total number of datapoints)
def calc_classification_error(predictions, class_labels):
    n = predictions.size
    num_of_errors = 0.
    for idx in xrange(n):
        if (predictions[idx] >= 0.5 and class_labels[idx]==0) or (predictions[idx] < 0.5 and class_labels[idx]==1):
            num_of_errors += 1
    return num_of_errors/n

# set the random number generator for reproducability
np.random.seed(182)

#### create artificial data
N = 400
D = 2
# Sample the features from a Multivariate Normal Dist.
mean1 = [13,5]
mean2 = [5,5]
covariance = [[13,5],[5,13]]
class1_features = np.random.multivariate_normal(mean1,covariance,N/2)
class2_features = np.random.multivariate_normal(mean2,covariance,N/2)
features = np.vstack((class1_features, class2_features))
# add column of ones for bias term
features = np.hstack((features,np.ones((N,1))))
# Set the class labels
class1_labels = [0]*(N/2)
class2_labels = [1]*(N/2)
class_labels = class1_labels+class2_labels
# shuffle the data
shuffle_in_unison(features, class_labels)
class_labels = np.array(class_labels)[np.newaxis].T

### fit the linear model --- OLS Solution
beta = np.dot(np.linalg.inv(np.dot(features.T, features)),np.dot(features.T,class_labels))

### compute error on training data
predictions = np.dot(features, beta)
print "Classification Error on Training Set: %.2f%%" %(calc_classification_error(predictions, class_labels) * 100)

### generate a plot
plot_classification_data(class1_features, class2_features, beta)


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
24% error is respectable, but there's still room for improvement.  In general, working with natural language is one of the hardest application domains in Machine Learning due to the fact that we often have to reduce the abstract, sometimes ambiguous semantic meaning to a superficial token.
STUDENT ACTIVITY
In the time remaining in the session, we'd like you to try an open-ended activity to get experience implementing the full prediction pipeline.  We've provided some suggestions below, but feel free to improvise.
Suggestion #1: Feature Engineering for 20 News Groups
Can you beat the baseline error rate on the 20 News Groups dataset?  One way to do this is to have better features--word counts are rather blunt.  Go read about TFIDF and then use SciKit-Learn's TFIDF Vectorizer to compute a new feature matrix for the 20 News Groups dataset.  You should be able to get an error rate of about 40% if not better.  The code is started for you below.
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer

#### YOUR CODE GOES HERE

print "Classification Error on the Test Set: %.2f%%" %(calc_classification_error(predictions, test_targets) * 100)

# compute the baseline error since the classes are imbalanced
print "Baseline Error: %.2f%%" %(100 - sum(test_targets)*100./len(test_targets))


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
3.  Logistic Regression: A Summary
Data
We observe pairs $(\mathbf{x}{i},y{i})$ where
\begin{eqnarray}
y_{i} \in { 0, 1} &:& \mbox{class label} \
\mathbf{x}{i} = (1, x{i,1}, \dots, x_{i,D}) &:& \mbox{set of $D$ explanatory variables (aka features) and a bias term }
\end{eqnarray}
 Parameters
\begin{eqnarray}
\mathbf{\beta}^{T} = (\beta_{0}, \dots, \beta_{D}) : \mbox{values encoding the relationship between the features and label}
\end{eqnarray}
 Transformation Function
\begin{equation}
f(z_{i}=\mathbf{x}{i} \mathbf{\beta} ) = (1+e^{-\mathbf{x}{i} \mathbf{\beta} })^{-1}
\end{equation}
Error Function
\begin{eqnarray}
\mathcal{L} = \sum_{i=1}^{N} -y_{i} \log f(\mathbf{x}{i} \mathbf{\beta} ) - (1-y{i}) \log (1-f(\mathbf{x}_{i} \mathbf{\beta} ))
\end{eqnarray}
Can you write Python code for it?

### compute the cross-entropy error
# labels: Numpy array containing the true class labels
# f: column vector of predictions (i.e. output of logistic function)
def cross_entropy(labels, f):
    return np.sum(-1*np.multiply(labels,np.log(f)) - np.multiply((np.ones(N)-labels),np.log(np.ones(N)-f)))


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
4.  Newton's Method
Choosing the step-size, $\alpha$, can be painful since there is no principled way to set it.  We have little intuition for what parameter space really looks like and therefore no sense of how to move most efficiently.  Knowing the curvature of the space will solve this problem (to some extent).  Therefore, we arrive at Newton's Method:
\begin{equation}
\beta_{t+1} = \beta_{t} - (\frac{\partial^{2} \mathcal{L}}{\partial \beta \partial \beta^{T}})^{-1} \nabla_{\beta} \mathcal{L}
\end{equation}
where $(\frac{\partial^{2} \mathcal{L}}{\partial \beta \partial \beta^{T}})^{-1}$ is the inverse of the matrix of second derivatives, also known as the Hessian Matrix.  For Logistic regression, the Hessian is
\begin{equation}
\frac{\partial^{2} \mathcal{L}}{\partial \beta \partial \beta^{T}} = \mathbf{X}^{T}\mathbf{A}\mathbf{X}
\end{equation}
where $\mathbf{A}= \mathrm{diag}(f(X_{i}\beta)(1-f(X_{i}\beta)))$, a matrix with $f''$ along its diagonal.
Our new parameter update is:
\begin{eqnarray}
\beta_{t+1} &=& \beta_{t} - (\mathbf{X}^{T}\mathbf{A}\mathbf{X})^{-1}\mathbf{X}^{T}[f(\mathbf{X}\beta) - \mathbf{y}]
\end{eqnarray}
As you can see, we no longer need to specify a step-size.  We've replaced $\alpha$ with $(\frac{\partial^{2} \mathcal{L}}{\partial \beta \partial \beta^{T}})^{-1}$ and everything else stays the same.
STUDENT ACTIVITY (10 MINS)
Write a function that computes the Hessian matrix ($\mathbf{X}^{T}\mathbf{A}\mathbf{X}$).
Can you write Python code for it?

def compute_Hessian(features, f):
    # X = feature matrix, size NxD),
    # f = predictions (logistic outputs), size Nx1
    # TO DO: return the Hessian matrix, size DxD
    n = len(features)
    A = np.multiply(f,np.ones(n)[np.newaxis].T-f)
    A = np.diag(A.T[0])
    return np.dot(features.T, np.dot(A ,features))

# a few tests to make sure your function is working
X = np.array([[1,2],[3,4],[5,6]])
f = np.array([.1,.3,.5])[np.newaxis].T
print "Should print [[  8.23  10.2 ];[ 10.2   12.72]]:"
print compute_Hessian(X,f)
print
X = np.array([[1],[4],[6]])
f = np.array([.01,.13,.55])[np.newaxis].T
print "Should print [[ 10.7295]]:"
print compute_Hessian(X,f)


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
Experiments
6.  Dataset #1: NBA Shot Outcomes
The first real dataset we'll tackle is one describing the location and outcome of shots taken in professional basketball games.  Let's use Pandas to load and examine the data.
Can you write Python code for it?

nba_shot_data = pd.read_csv('./data/nba_experiment/NBA_xy_features.csv')
nba_shot_data.head()

nba_shot_data.describe()


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
Great!  We've improved by a few percentage points.  Let's look at which features the model weighted.
Can you write Python code for it?

for idx, feature in enumerate(nba_all_features):
    if idx<11:
        print "%s: %.2f" %(feature, skl_LogReg.coef_[0][idx])


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
Interestingly, the classifier exploited the location features very little.  The position of the player was much more important, especially if he was a center.
## 7.  Dataset #2: 20 News Groups
For the second experiment, we'll work with the very popular '20 News Groups' dataset consisting of, well, 20 different categories of articles.  SciKit-Learn already has it ready for import.
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer

# use SciKit Learn's loading methods
categories = ['soc.religion.christian', 'alt.atheism']
train_20ng = fetch_20newsgroups(subset='train', remove=('headers', 'footers', 'quotes'), categories=categories)
test_20ng = fetch_20newsgroups(subset='test', remove=('headers', 'footers', 'quotes'), categories=categories)

# transform the text into word counts
vectorizer = CountVectorizer(stop_words='english', max_features=1000)
train_vectors = vectorizer.fit_transform(train_20ng.data)
test_vectors = vectorizer.transform(test_20ng.data) #use the transform fit to the training data

train_targets = train_20ng.target
test_targets = test_20ng.target

print "The training data size is "+str(train_vectors.shape)
print "The test data size is "+str(test_vectors.shape)
# print the first 500 words of an article
print "Example text:"
print train_20ng.data[0][:500]
print
print "Example count vector:"
#print train_vectors[0].todense()


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
Learning $\beta$
Like Linear Regression, learning a Logistic Regression model will entail minimizing the error function $\mathcal{L}$ above.  Can we solve for $\beta$ in closed form?  Let's look at the derivative of $\mathcal{L}$ with respect to $\beta$:
\begin{eqnarray}
\frac{\partial \mathcal{L}{i}}{\partial \mathbf{\beta}} &=& \frac{\partial \mathcal{L}{i}}{\partial f(z_{i})} \frac{\partial f(z_{i})}{\partial z_{i}} \frac{\partial z_{i}}{\partial \mathbf{\beta}}\
&=& \left[\frac{-y_{i}}{f(\mathbf{x}{i} \mathbf{\beta})} - \frac{y{i}-1}{1-f(\mathbf{x}{i} \mathbf{\beta})} \right] f(\mathbf{x}{i} \mathbf{\beta})(1-f(\mathbf{x}{i} \mathbf{\beta}))\mathbf{x}{i}\
&=& [-y_{i}(1-f(\mathbf{x}{i} \mathbf{\beta} )) - (y{i}-1)f(\mathbf{x}{i} \mathbf{\beta} )]\mathbf{x}{i}\
&=& [f(\mathbf{x}{i} \mathbf{\beta} ) - y{i}]\mathbf{x}_{i}
\end{eqnarray}
Can you write Python code for it?

### compute the gradient (derivative w.r.t. Beta)
# features: NxD feature matrix
# labels: Numpy array containing the true class labels
# f: column vector of predictions (i.e. output of logistic function)
def compute_Gradient(features, labels, f):
    return np.sum(np.multiply(f-labels,features),0)[np.newaxis].T

np.sum([1,2],0)

?np.sum


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
Let's look at the training progress to see how much more efficient Newton's method is.
Can you write Python code for it?

# plot difference between with vs without Newton
plt.figure()
# grad descent w/ step size
plt.plot(range(len(no_Newton_errors)), no_Newton_errors, 'k-', linewidth=4, label='Without Newton')
# newton's method
plt.plot(range(len(error)), error, 'g-', linewidth=4, label='With Newton')
plt.ylim([0,300000])
plt.xlim([0,150])
plt.legend()
plt.title("Newton's Method vs. Gradient Descent w/ Step Size")
plt.xlabel("Training Iteration")
plt.ylabel("Cross-Entropy Error")
plt.show()


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
Not bad.  We're beating the random baseline of 45% error.  However, visualizing the decision boundary exposes a systemic problem with using a linear model on this dataset: it is not powerful enough to adapt to the geometry of the court.  This is a domain-specific contraint that should be considered when selecting the model and features.  For instance, a Gaussian-based classifier works a bit better, achieving 39.02% error.  Its decision boundary is visualized below.
<img src="https://raw.githubusercontent.com/enalisnick/NBA_shot_analysis/master/results/spatial_features_results/Gaussian_Mixture_Model.png" alt="" style="width: 250px;"/>
Can we do better by adding more features?  For instance, if we knew the position (Guard vs. Forward vs. Center) of the player taking the shot, would that help?  Let's try.  First, load a new dataset.
Can you write Python code for it?

# first we need to extract the file from the zip
import zipfile
zip = zipfile.ZipFile('./data/nba_experiment/NBA_all_features.csv.zip')
zip.extractall('./data/nba_experiment/')

nba_all_features = pd.read_csv('./data/nba_experiment/NBA_all_features.csv')
nba_all_features.head()


I am working on the file "Session 3 - Classification.ipynb".
The first task is:
We see that the first derivative contains the term $f(X_{i}\beta)$, meaning the gradient depends on $\beta$ in some non-linear way.  We have no choice but to use the Gradient Descent algorithm:
- Randomly initialize $\beta$
- Until $\alpha || \nabla \mathcal{L} || < tol $:
    - $\mathbf{\beta}{t+1} = \mathbf{\beta}{t} - \alpha \nabla_{\mathbf{\beta}} \mathcal{L}$
 Putting it all together in a simple example...
Can you write Python code for it?

# set the random number generator for reproducability
np.random.seed(49)

# Randomly initialize the Beta vector
beta = np.random.multivariate_normal([0,0,0], [[1,0,0],[0,1,0],[0,0,1]], 1).T
# Initialize the step-size
alpha = 0.00001
# Initialize the gradient
grad = np.infty
# Set the tolerance
tol = 1e-6
# Initialize error
old_error = 0
error = [np.infty]

# Run Gradient Descent
start_time = time.time()
iter_idx = 1
# loop until gradient updates become small
while (alpha*np.linalg.norm(grad) > tol) and (iter_idx < 300):
    f = logistic(np.dot(features,beta))
    old_error = error[-1]
    # track the error
    error.append(cross_entropy(class_labels, f))
    grad = compute_Gradient(features, class_labels, f)
    # update parameters
    beta = beta - alpha*grad
    iter_idx += 1
end_time = time.time()
print "Training ended after %i iterations, taking a total of %.2f seconds." %(iter_idx, end_time-start_time)
print "Final Cross-Entropy Error: %.2f" %(error[-1])

# compute error on training data
predictions = logistic(np.dot(features, beta))
print "Classification Error on Training Set: %.2f%%" %(calc_classification_error(predictions, class_labels) * 100)

# generate the plot
plot_classification_data(class1_features, class2_features, beta, logistic_flag=True)

np.random.multivariate_normal([0,0,0], [[1,0,0],[0,1,0],[0,0,1]], 1).T.shape


I am working on the file "Session 3/ipython/.ipynb_checkpoints/Lesson 6 - Supervised learning-Copy1-checkpoint.ipynb".
The first task is:
1.2 Training the ML algorithm
NLTK module is built for working with language data.  NLTK supports classification, tokenization, stemming, tagging, parsing, and semantic reasoning functionalities. We will use the NLTK module and employ the naive Bayes method to classify words as being either positive or negative sentiment. You can also use other modules specifically meant for ML eg. sklearn module.
Step 1: Create the feature set
We will use the corpus of sentiments from the word_sentiment.csv file to create a feature dataset which we will use to train and test the ML model.
Can you write Python code for it?

import csv

def feature_extractor(word):
    """Extract the features for a given word and return a dictonary of the features"""
    start_letter = word[0]
    last_letter = word[-1]
    return {'start_letter' : start_letter,'last_letter' : last_letter}

def ML_train(sentiment_corpus):
    """Create feature set from the corpus given to to it."""
    feature_set = []
    with open(sentiment_corpus,'rt',encoding = 'utf-8') as sentobj:
        sentiment_handle = csv.reader(sentobj)

        for sentiment in sentiment_handle:
            new_row = []
            new_row.append(feature_extractor(sentiment[0])) #get the dictionary of features for a word
            if int(sentiment[1]) >= 0: # Club the sentiment values (-5 to + 5) to just positive or negative
                new_row.append('positive')
            else:
                new_row.append('negative')
            feature_set.append(new_row)
        print(feature_set)

def main():
    sentiment_csv = "C:/Users/kmpoo/Dropbox/HEC/Teaching/Python for PhD Mar 2018/python4phd/Session 3/Sent/word_sentiment.csv"
    ML_train(sentiment_csv)

main()


I am working on the file "Session 3/ipython/.ipynb_checkpoints/Lesson 6 - Supervised learning-Copy1-checkpoint.ipynb".
The first task is:
Excercise
Improve the feature extractor (by adding new features) so that the test accuracy can go up to atleast 70%.
Can you write Python code for it?


I am working on the file "Session 3/ipython/Lesson 6 - Supervised learning.ipynb".
The first task is:
Excercise
Improve the feature extractor (by adding new features) so that the test accuracy can go up to atleast 70%.
Can you write Python code for it?


I am working on the file "Session 3/ipython/Lesson 6 - Supervised learning.ipynb".
The first task is:
Step 4: Use ML method (naive Bayes) to create the classifier model
The NLTK module gives us several ML methods to create a classifier model using our training set and based on our selected features.
Can you write Python code for it?

import csv
import random
import nltk

def feature_extractor(word):
    """Extract the features for a given word and return a dictonary of the features"""
    start_letter = word[0]
    last_letter = word[-1]
    return {'start_letter' : start_letter,'last_letter' : last_letter}

def ML_train(sentiment_corpus):
    """Create feature set from the corpus given to to it. Split the  feature set into training and testing sets.
    Train the classifier using the naive Bayes model and return the classifier. """
    feature_set = []
    with open(sentiment_corpus,'rt',encoding = 'utf-8') as sentobj:
        sentiment_handle = csv.reader(sentobj)

        for sentiment in sentiment_handle:
            new_row = []
            new_row.append(feature_extractor(sentiment[0])) #get the dictionary of features for a word
            if int(sentiment[1]) >= 0: # Club the sentiment values (-5 to + 5) to just positive or negative
                new_row.append('positive')
            else:
                new_row.append('negative')
            feature_set.append(new_row)

        random.shuffle(feature_set)
        # We need to shuffle the features since the word_sentiment.csv had words arranged in alphabetical order
        train_set = feature_set[:1500] #the first 1500 words becomes our training set
        test_set = feature_set[1500:]
        classifier = nltk.NaiveBayesClassifier.train(train_set)
        # Note: to create the classifier we need to provide a dictonary of features and the label ONLY
        return classifier

def main():
    sentiment_csv = "C:/Users/kmpoo/Dropbox/HEC/Teaching/Python for PhD Mar 2018/python4phd/Session 3/Sent/word_sentiment.csv"
    classifier = ML_train(sentiment_csv)
    input_word = input('Enter a word ').lower()
    sentiment = classifier.classify(feature_extractor(input_word))
    print('Sentiment of word "', input_word,'" is : ',sentiment)

main()


I am working on the file "Session 3/ipython/Lesson 6 - Supervised learning.ipynb".
The first task is:
Step 5: Testing the model
Find how good the model is in identifying the labels. Ensure that the test set is distinct from the training corpus. If we simply re-used the training set as the test set, then a model that simply memorized its input, without learning how to generalize to new examples, would receive misleadingly high scores. The function nltk.classify.accuracy() will calculate the accuracy of a classifier model on a given test set.
Can you write Python code for it?

import csv
import random
import nltk

def feature_extractor(word):
    """Extract the features for a given word and return a dictonary of the features"""
    start_letter = word[0]
    last_letter = word[-1]
    return {'start_letter' : start_letter,'last_letter' : last_letter}

def ML_train(sentiment_corpus):
    """Create feature set from the corpus given to to it. Split the  feature set into training and testing sets.
    Train the classifier using the naive Bayes model and return the classifier. """
    feature_set = []
    with open(sentiment_corpus,'rt',encoding = 'utf-8') as sentobj:
        sentiment_handle = csv.reader(sentobj)

        for sentiment in sentiment_handle:
            new_row = []
            new_row.append(feature_extractor(sentiment[0])) #get the dictionary of features for a word
            if int(sentiment[1]) >= 0: # Club the sentiment values (-5 to + 5) to just positive or negative
                new_row.append('positive')
            else:
                new_row.append('negative')
            feature_set.append(new_row)

        random.shuffle(feature_set)
        # We need to shuffle the features since the word_sentiment.csv had words arranged in alphabetical order
        train_set = feature_set[:1500] #the first 1500 words becomes our training set
        test_set = feature_set[1500:]
        classifier = nltk.NaiveBayesClassifier.train(train_set)
        # Note: to create the classifier we need to provide a dictonary of features and the label ONLY
        print('Test accuracy of the classifier = ',nltk.classify.accuracy(classifier, test_set))
        print(classifier.show_most_informative_features())
        return classifier

def main():
    sentiment_csv = "C:/Users/kmpoo/Dropbox/HEC/Teaching/Python for PhD Mar 2018/python4phd/Session 3/Sent/word_sentiment.csv"
    classifier = ML_train(sentiment_csv)
    input_word = input('Enter a word ').lower()
    sentiment = classifier.classify(feature_extractor(input_word))
    print('Sentiment of word "', input_word,'"...


I am working on the file "Session 3/ipython/Lesson 7- Sklearn and Randomforest.ipynb".
The first task is:
Step 2: Manipulate the dataframe and add new features
Create a new feature "length of the sentence"
Can you write Python code for it?

dataframe = dataframe.assign(nWords = lambda x : x['review_text'].str.split().str.len() )
dataframe['bi_senti'] = [ "positive" if x >= 4 else "negative" for x in dataframe['sentiment']]
print(dataframe)
print(dataframe['bi_senti'].value_counts())


I am working on the file "Session 3/ipython/Lesson 7- Sklearn and Randomforest.ipynb".
The first task is:
Step 3: Split into train and test samples
Can you write Python code for it?

from sklearn.utils import shuffle #To shuffle the dataframe
from sklearn.model_selection import train_test_split

dataframe = shuffle(dataframe)
df_train, df_test = train_test_split(dataframe, test_size=0.2)
print("size of trainig data ", len(df_train))


I am working on the file "Session 3/ipython/Lesson 7- Sklearn and Randomforest.ipynb".
The first task is:
Step 4: Tokenization and veectorize the sentence
Text data requires special preparation before you can start using it for predictive modeling.
The text must be parsed to remove words, called tokenization. Then the words need to be encoded as integers or floating point values for use as input to a machine learning algorithm, called feature extraction (or vectorization).
Convert sentences to TF-IDF vectors
tf-idf ( term frequency–inverse document frequency), is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. The tf–idf value increases proportionally to the number of times a word appears in the document and is offset by the number of documents in the corpus that contain the word, which helps to adjust for the fact that some words appear more frequently in general.
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer

def tfidf_vectorizer(corpus):
  """This function converts the input sentence into a sparse matrix of tfidf vectors"""
  tokenizer = TfidfVectorizer( strip_accents = "unicode", analyzer="word", stop_words="english", ngram_range=(1,2), max_features=20000)
  tokenizer.fit(corpus)
  return tokenizer

vectorizer = tfidf_vectorizer(dataframe['review_text'])
train_x = vectorizer.transform(df_train['review_text'])
test_x = vectorizer.transform(df_test['review_text'])

print(train_x)


I am working on the file "Session 3/ipython/Lesson 7- Sklearn and Randomforest.ipynb".
The first task is:
Step 5:Generate the classifier
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

rfc = RandomForestClassifier(n_estimators=10)
classifier = rfc.fit(train_x, df_train['bi_senti'])
acc = classifier.score(test_x,df_test['bi_senti'])
print("accuracy of rfc is = ", acc)
rfc.predict_proba(test_x)[0:10]


I am working on the file "Session 3/ipython/Lesson 7- Sklearn and Randomforest.ipynb".
The first task is:
Step 6: Use the model to predict the sentiment of your sentence
Can you write Python code for it?

s = pd.Series("his movie was absolutely horrible. A boring, random, nonsensical mess from start to finish. The film is incompetently directed from a very poor script. It feels more like a superhero movie from the early 2000's such as Catwoman or Daredevil. Watching it makes it clear that the people involved had no idea what they were doing, and should never have been put in charge of a project this size to begin with. The story makes no sense, and the whole reason Batman wants to kill Superman is contrived. Batman and Superman hate each other because they both cause collateral damage and human death, and neither one ever sees fit to point out their similarities, or try and talk to each other about their different perspectives. Apparently that would have been too interesting, so of course Snyder didn't include it.")
x = vectorizer.transform(s)
print(rfc.predict_proba(x))


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
1.  Warm-up
Let's start with a warm-up exercise.  In the data directory you'll find a dataset of recent movies and their ratings according to several popular websites.  Let's load it with Pandas...
Can you write Python code for it?

# load a dataset of recent movies and their ratings across several websites
movie_data = pd.read_csv('../data/movie_ratings.csv')
# reduce it to just the ratings categories
movie_data = movie_data[['FILM','RottenTomatoes','RottenTomatoes_User','Metacritic','Metacritic_User','Fandango_Ratingvalue', 'IMDB']]
movie_data.head()

movie_data.describe()


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
2.  Feature Transformations
Good features are crucial for training well-performing classifiers: 'garbage in, garbage out.'  In this section we introduce several transformations that are commonly applied to data as a preprocessing step before training a classifier.
2.1 Normal Standardization
Recall the formula of the standard linear model: $$\hat Y = f(\beta^{T} \mathbf{X}) $$ where $\hat Y$ is the predictions, $f(\cdot)$ is the transformation function, $\beta$ is the weights (parameters), and $X$ is the $N \times D$ matrix of features.  For simplicity, assume there are just two features: $$ \beta^{T} \mathbf{x}{i} = \beta{1}x_{i,1} + \beta_{2}x_{i,2}.$$  Usually $x_{i,1}$ and $x_{i,2}$ will be measured in different units.  For instance, in the movie ratings data, the Rotten Tomatoes dimension is on a $0-100$ scale and the Fandango ratings are on $0-5$.  The difference in scale causes one dimension to dominate the inner product.  Linear models can learn to cope with this imbalance by changing the scales of the weights accordingly, but this makes optimization harder because gradient steps are unequal across dimensions.
One way to get rid of hetergeneous scales is to standardize the data so that the values in each dimension are distributed according to the standard Normal distribution.  In math, this means we'll transform the data like so: $$\mathbf{X}{std} = \frac{\mathbf{X} - \boldsymbol{\mu}{X}}{\boldsymbol{\sigma}_{X}}. $$  This is also called 'z-score scaling.'  Let's examine the affect of this transformation on training error.
Can you write Python code for it?

# perform z-score scaling
features_mu = np.mean(features, axis=0)
features_sigma = np.std(features, axis=0)
std_features = (features - features_mu)/features_sigma

# re-train model
lm = LogisticRegression()
lm.fit(std_features, labels)

### compute error on training data
predictions = lm.predict(std_features)
print "Classification error on training set: %.3f%%" %(calc_classification_error(predictions, labels)*100)
# compute the baseline error since the classes are imbalanced
print "Baseline Error: %.2f%%" %((sum(labels)*100.)/len(labels))


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
3.  Dimensionality Reduction
Sometimes the data calls for more aggressive transformations.  High-dimensional data is usually hard to model because classifiers are likely to overfit.  Regularization is one way to combat high dimensionality, but often it can not be enough.  This section will cover dimensionality reduction--a technique for reducing the number of features while still preserving curcial information.  This is a form of unsupervised learning since we use no class information.
3.1 Image Dataset: Bob Ross Paintings

In this section and throughout the next session, we'll use a dataset of Bob Ross' paintings.  Images are a type of data that notoriously have redundant features and whose dimensionality can be reduced significantly, without much loss of information.  We'll explore this phenomenom via 403 $400 \times 300$ full-color images of natural landscape paintings.
Before we load the data, let's take a minute to review how image data is stored on a computer.  Of course, all the computer sees are numbers ranging from 0 to 255.  Each pixel takes on one of these values.  Furthermore, there are three layers to color images--one for red, blue, and green values.  Therefore, the painting we'll examine are represented as $300 \times 400 \times 3$-dimensional tensors (multi-dimensional arrays).  This layering is depicted below.

While images need to be represented with three dimensions to be visualized, the learning algorithms we'll consider don't need any notion of color values so I've already flattened the images into vector form, i.e. to create a matrix of size $403 \times 360000$.  Let's load the dataset...
Can you write Python code for it?

# un-zip the paintings file
import zipfile
zipper = zipfile.ZipFile('../data/bob_ross/bob_ross_paintings.npy.zip')
zipper.extractall('../data/bob_ross/')

# load the 403 x 360,000 matrix
br_paintings = np.load(open('../data/bob_ross/bob_ross_paintings.npy','rb'))
print "Dataset size: %d x %d"%(br_paintings.shape)


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
3.2 Principal Component Analysis
As we've seen, the dataset has many, many, many more features (columns) than examples (rows).  Simple Lasso or Ridge regularization probably won't be enough to prevent overfitting so we have to do something more drastic.  In this section, we'll cover Principal Component Analysis, a popular technique for reducing the dimensionality of data.
Unsupervised Learning
PCA does not take into consideration labels, only the input features.  We can think of PCA as performing unsupervised 'inverse' prediction.  Our goal is: for a datapoint $\mathbf{x}{i}$, find a lower-dimensional representation $\mathbf{h}{i}$ such that $\mathbf{x}{i}$ can be 'predicted' from $\mathbf{h}{i}$ using a linear transformation.  In math, this statement can be written as $$\mathbf{\tilde x}{i} = \mathbf{h}{i} \mathbf{W}^{T} \text{ where } \mathbf{h}{i} = \mathbf{x}{i} \mathbf{W}. $$  $\mathbf{W}$ is a $D \times K$ matrix of parameters that need to be learned--much like the $\beta$ vector in regression models.  $D$ is the dimensionality of the original data, and $K$ is the dimensionality of the compressed representation $\mathbf{h}_{i}$.  The graphic below reiterates the above described PCA pipline:

Optimization
Having defined the PCA model, we look to write learning as an optimization process.  Recall that we wish to make a reconstruction of the data, denoted $\mathbf{\tilde x}{i}$, as close as possible to the original input: $$\mathcal{L} = \sum{i=1}^{N} (\mathbf{x}{i} - \mathbf{\tilde x}{i})^{2}.$$  We can make a substitution for $\mathbf{\tilde x}{i}$ from the equation above: $$ = \sum{i=1}^{N} (\mathbf{x}{i} - \mathbf{h}{i}\mathbf{W}^{T})^{2}.$$  And we can make another substitution for $\mathbf{h}{i}$, bringing us to the final form of the loss function: $$ = \sum{i=1}^{N} (\mathbf{x}{i} - \mathbf{x}{i}\mathbf{W}\mathbf{W}^{T})^{2}.$$
We could perform gradient descent on $\mathcal{L}$, just like we do for logistic regression models, but there exists a deterministic solution.  We won't show the...
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA(n_components=400)
start_time = time.time()
reduced_paintings = pca.fit_transform(br_paintings)
end_time = time.time()

print "Training took a total of %.2f seconds." %(end_time-start_time)
print "Preserved percentage of original variance: %.2f%%" %(pca.explained_variance_ratio_.sum() * 100)
print "Dataset is now of size: %d x %d"%(reduced_paintings.shape)


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
3.3 PCA for Visualization
PCA can also be done for visualization purposes.  Let's perform PCA on the movie ratings dataset and see if any semblence of the class structure can be seen.
Can you write Python code for it?

# get the movie features
movie_features = movie_data[['RottenTomatoes','RottenTomatoes_User','Metacritic','Metacritic_User','Fandango_Ratingvalue']].as_matrix()

# perform standard scaling again but via SciKit-Learn
from sklearn.preprocessing import StandardScaler
z_scaler = StandardScaler()
movie_features = z_scaler.fit_transform(movie_features)

pca = PCA(n_components=2)
start_time = time.time()
movie_2d_proj = pca.fit_transform(movie_features)
end_time = time.time()

print "Training took a total of %.4f seconds." %(end_time-start_time)
print "Preserved percentage of original variance: %.2f%%" %(pca.explained_variance_ratio_.sum() * 100)
print "Dataset is now of size: %d x %d"%(movie_2d_proj.shape)

labels = movie_data['FILM'].tolist()
classes = movie_data['IMDB'].tolist()

# color the points by IMDB ranking
labels_to_show = []
colors = []
for idx, c in enumerate(classes):
    if c > 7.25:
        colors.append('g')
        if c > 8.:
            labels_to_show.append(labels[idx])
    else:
        colors.append('r')
        if c < 4.75:
            labels_to_show.append(labels[idx])

# plot data
plt.scatter(movie_2d_proj[:, 0], movie_2d_proj[:, 1], marker = 'o', c = colors, s = 150, alpha = .6)

# add movie title annotations
for label, x, y in zip(labels, movie_2d_proj[:, 0].tolist(), movie_2d_proj[:, 1].tolist()):
    if label not in labels_to_show:
        continue
    if x < 0:
        text_x = -20
    else:
        text_x = 150
    plt.annotate(label.decode('utf-8'),xy = (x, y), xytext = (text_x, 40),
        textcoords = 'offset points', ha = 'right', va = 'bottom',
        arrowprops = dict(arrowstyle = '->', connectionstyle = 'arc3, rad=0'),
        bbox = dict(boxstyle = 'round,pad=0.5', fc = 'b', alpha = 0.2))

plt.title('PCA Projection of Movies')
plt.show()


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
STUDENT ACTIVITY (until end of session)
Your task is to reproduce the above PCA examples on a new dataset of images.  Let's load it...
Can you write Python code for it?

from sklearn.datasets import fetch_olivetti_faces

faces_dataset = fetch_olivetti_faces(shuffle=True)
faces = faces_dataset.data # 400 flattened 64x64 images
person_ids = faces_dataset.target # denotes the identity of person (40 total)

print "Dataset size: %d x %d" %(faces.shape)
print "And the images look like this..."
plt.imshow(np.reshape(faces[200,:], (64, 64)), cmap='Greys_r')
plt.show()


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
Subtask 2: Reconstruct an image
Can you write Python code for it?

### Your code goes here ###

# Use learned transformation matrix to project back to the original 4096-dimensional space
# Remember you need to use np.reshape()

###########################

img_idx = 70
reconstructed_img = pca.inverse_transform(faces_reduced[img_idx,:])
original_img = faces[70,:]

# subplot for original image
ax1 = plt.subplot(1,2,1)
ax1.imshow(np.reshape(original_img, (64, 64)), cmap='Greys_r')
ax1.set_title("Original Image")

# subplot for reconstruction
ax2 = plt.subplot(1,2,2)
ax2.imshow(np.reshape(reconstructed_img, (64, 64)), cmap='Greys_r')
ax2.set_title("Reconstruction")
plt.show()


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
I've created a function that we'll use later to create visualizations.  It is a bit messy and not essential to the material so don't worry about understanding it.  I'll be happy to explain it to anyone interested during a break or after the session.
Can you write Python code for it?

from matplotlib.colors import ListedColormap

# Another messy looking function to make pretty plots of basketball courts
def visualize_court(log_reg_model, coord_type='cart', court_image = '../data/nba/nba_court.jpg'):
    two_class_cmap = ListedColormap(['#FFAAAA', '#AAFFAA']) # light red for miss, light green for make
    x_min, x_max = 0, 50 #width (feet) of NBA court
    y_min, y_max = 0, 47 #length (feet) of NBA half-court
    grid_step_size = 0.2
    grid_x, grid_y = np.meshgrid(np.arange(x_min, x_max, grid_step_size), np.arange(y_min, y_max, grid_step_size))
    features = np.c_[grid_x.ravel(), grid_y.ravel()]
    # change coordinate system
    if coord_type == 'polar':
        features = np.c_[grid_x.ravel(), grid_y.ravel()]
        hoop_location = np.array([25., 0.])
        features -= hoop_location
        dists = np.sqrt(np.sum(features**2, axis=1))
        angles = np.arctan2(features[:,1], features[:,0])
        features = np.hstack([dists[np.newaxis].T, angles[np.newaxis].T])

    grid_predictions = log_reg_model.predict(features)
    grid_predictions = grid_predictions.reshape(grid_x.shape)
    fig, ax = plt.subplots()
    court_image = plt.imread(court_image)
    ax.imshow(court_image, interpolation='bilinear', origin='lower',extent=[x_min,x_max,y_min,y_max])
    ax.imshow(grid_predictions, cmap=two_class_cmap, interpolation = 'nearest',
              alpha = 0.60, origin='lower',extent=[x_min,x_max,y_min,y_max])
    plt.xlim(x_min, x_max)
    plt.ylim(y_min, y_max)
    plt.title( "Make / Miss Prediction Boundaries" )
    plt.show()


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
Let's visualize two of the paintings...
Can you write Python code for it?

img_idx = 70
reconstructed_img = pca.inverse_transform(reduced_paintings[img_idx,:])
original_img = br_paintings[70,:]

# subplot for original image
ax1 = plt.subplot(1,2,1)
ax1.imshow(np.reshape(original_img, (300, 400, 3)))
ax1.set_title("Original Painting")

# subplot for reconstruction
ax2 = plt.subplot(1,2,2)
ax2.imshow(np.reshape(reconstructed_img, (300, 400, 3)))
ax2.set_title("Reconstruction")
plt.show()


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
Now let's transform the Cartesian coordinates into polar coordinates: (x,y) --> (radius, angle)...
Can you write Python code for it?

### Transform coordinate system

# radius coordinate: calculate distance from point to hoop
hoop_location = np.array([25.5, 0.])
train_features -= hoop_location
test_features -= hoop_location
train_dists = np.sqrt(np.sum(train_features**2, axis=1))
test_dists = np.sqrt(np.sum(test_features**2, axis=1))

# angle coordinate: use arctan2 function
train_angles = np.arctan2(train_features[:,1], train_features[:,0])
test_angles = np.arctan2(test_features[:,1], test_features[:,0])

# combine vectors into polar coordinates
polar_train_features = np.hstack([train_dists[np.newaxis].T, train_angles[np.newaxis].T])
polar_test_features = np.hstack([test_dists[np.newaxis].T, test_angles[np.newaxis].T])

pd.DataFrame(polar_train_features, columns=["Radius","Angle"]).head()

#Train model
start_time = time.time()
lm.fit(polar_train_features, np.ravel(train_class_labels))
end_time = time.time()
print "Training ended after %.2f seconds." %(end_time-start_time)

# compute the classification error on test data
predictions = lm.predict(polar_test_features)
print "Classification Error on the Test Set: %.2f%%" %(calc_classification_error(predictions, np.array(test_class_labels)) * 100)

# compute the baseline error since the classes are imbalanced
print "Baseline Error: %.2f%%" %(np.sum(test_class_labels)/len(test_class_labels)*100)

# visualize the boundary on the basketball court
visualize_court(lm, coord_type='polar')


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
Standard Normal scaling is a common and usually default first step, especially when you know the data in measured in different units.
2.2 Domain-Specific Transformations
Sometimes the data calls for a specific transformation.  We'll demonstrate this on the NBA dataset used in our first workshop.  Let's load it...
Can you write Python code for it?

nba_shot_data = pd.read_csv('../data/nba/NBA_xy_features.csv')
nba_shot_data.head()


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
This dataset contains 400 64x64 pixel images of 40 people each exhibiting 10 facial expressions.  The images are in gray-scale, not color, and therefore flattened vectors contain 4096 dimensions.
Subtask 1: Run PCA
Can you write Python code for it?

?PCA

### Your code goes here ###

# train PCA model on 'faces'
from sklearn.decomposition import PCA

pca = PCA(n_components=100)

start_time = time.time()
faces_reduced = pca.fit_transform(faces)
end_time = time.time()

###########################

print "Training took a total of %.2f seconds." %(end_time-start_time)
print "Preserved percentage of original variance: %.2f%%" %(pca.explained_variance_ratio_.sum() * 100)
print "Dataset is now of size: %d x %d"%(faces_reduced.shape)


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
We also need functions for shuffling the data and calculating classification errrors.
Can you write Python code for it?

### function for shuffling the data and labels
def shuffle_in_unison(features, labels):
    rng_state = np.random.get_state()
    np.random.shuffle(features)
    np.random.set_state(rng_state)
    np.random.shuffle(labels)

### calculate classification errors
# return a percentage: (number misclassified)/(total number of datapoints)
def calc_classification_error(predictions, class_labels):
    n = predictions.size
    num_of_errors = 0.
    for idx in xrange(n):
        if (predictions[idx] >= 0.5 and class_labels[idx]==0) or (predictions[idx] < 0.5 and class_labels[idx]==1):
            num_of_errors += 1
    return num_of_errors/n


I am working on the file "Session 4 - Features_I_Transformations_DimensionalityReduction.ipynb".
The first task is:
and then visualize two of the images...
Can you write Python code for it?

# subplot containing first image
ax1 = plt.subplot(1,2,1)
br_painting = br_paintings[70,:]
ax1.imshow(np.reshape(br_painting, (300, 400, 3)))

# subplot containing second image
ax2 = plt.subplot(1,2,2)
br_painting = br_paintings[33,:]
ax2.imshow(np.reshape(br_painting, (300, 400, 3)))
plt.show()


I am working on the file "Session 4.ipynb".
The first task is:
Data base for library management system
Can you write Python code for it?

connn = sqlite3.connect("Library Management System.txt")

curs = connn.cursor()

sql = '''CREATE TABLE
books(book_id text,
isbn integer not null unique,
book_name text)'''
curs.execute(sql)

sql2 = '''CREATE TABLE
Student( roll number integer not null unique,
name text not null,
faculty text)'''
curs.execute(sql2)

sql3 = '''CREATE TABLE
Teacher(name text not null,
faculty text)
'''
curs.execute(sql3)


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
0.1  Load the dataset of handwritten digits
We are going to use the MNIST dataset throughout this session.  Let's load the data...
Can you write Python code for it?

mnist = pd.read_csv('../data/mnist_train_100.csv', header=None)

# load the 70,000 x 784 matrix
from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original').data

# reduce to 5k instances
np.random.shuffle(mnist)
#mnist = mnist[:5000,:]/255.
print "Dataset size: %d x %d"%(mnist.shape)

# subplot containing first image
ax1 = plt.subplot(1,2,1)
digit = mnist[1,:]
ax1.imshow(np.reshape(digit, (28, 28)), cmap='Greys_r')

# subplot containing second image
ax2 = plt.subplot(1,2,2)
digit = mnist[2,:]
ax2.imshow(np.reshape(digit, (28, 28)), cmap='Greys_r')
plt.show()


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
2.3  SciKit Learn Version
We can hack the Scikit-Learn Regression neural network into an Autoencoder by feeding it the data back as the labels...
Can you write Python code for it?

from sklearn.neural_network import MLPRegressor

# set the random number generator for reproducability
np.random.seed(39)

# define the dimensionality of the hidden rep.
n_components = 200

# define model
ae = MLPRegressor(hidden_layer_sizes=(n_components,), activation='logistic')

### train Autoencoder
start_time = time.time()
ae.fit(mnist, mnist)
end_time = time.time()

recon_error = np.mean(np.sum((mnist - ae.predict(mnist))**2, 1))
W = ae.coefs_[0]
b = ae.intercepts_[0]
reduced_mnist = logistic(np.dot(mnist, W) + b)

print
print "Training ended after a total of %.2f seconds." %(end_time-start_time)
print "Final Reconstruction Error: %.2f" %(recon_error)
print "Dataset is now of size: %d x %d"%(reduced_mnist.shape)

img_idx = 5
reconstructed_img = np.dot(reduced_mnist[img_idx,:], ae.coefs_[1]) + ae.intercepts_[1]
original_img = mnist[img_idx,:]

# subplot for original image
ax1 = plt.subplot(1,2,1)
ax1.imshow(np.reshape(original_img, (28, 28)), cmap='Greys_r')
ax1.set_title("Original Digit")

# subplot for reconstruction
ax2 = plt.subplot(1,2,2)
ax2.imshow(np.reshape(reconstructed_img, (28, 28)), cmap='Greys_r')
ax2.set_title("Reconstruction")
plt.show()

# two components to show
comp1 = 0
comp2 = 150

# subplot
ax1 = plt.subplot(1,2,1)
filter1 = W[:, comp1]
ax1.imshow(np.reshape(filter1, (28, 28)), cmap='Greys_r')

# subplot
ax2 = plt.subplot(1,2,2)
filter2 = W[:, comp2]
ax2.imshow(np.reshape(filter2, (28, 28)), cmap='Greys_r')
plt.show()


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
2.4  Denoising Autoencoder (DAE)
Lastly, we are going to examine an extension to the Autoencoder called a Denoising Autoencoder (DAE).  It has the following loss fuction: $$\mathcal{L}{\text{DAE}} = \sum{i=1}^{N} (\mathbf{x}{i} - f((\hat{\boldsymbol{\zeta}} \odot \mathbf{x}{i})\mathbf{W})\mathbf{W}^{T})^{2} \ \text{ where } \hat{\boldsymbol{\zeta}} \sim \text{Bernoulli}(p).$$  In words, what we're doing is drawning a Bernoulli (i.e. binary) matrix the same size as the input features, and feeding a corrupted version of $\mathbf{x}_{i}$.  The Autoencoder, then, must try to recreate the original data from a lossy representation.  This has the effect of forcing the Autoencoder to use features that better generalize.
Let's make the simple change that implements a DAE below...
Can you write Python code for it?

# set the random number generator for reproducability
np.random.seed(39)

# define the dimensionality of the hidden rep.
n_components = 200

# Randomly initialize the Beta vector
W = np.random.uniform(low=-4 * np.sqrt(6. / (n_components + mnist.shape[1])),\
                      high=4 * np.sqrt(6. / (n_components + mnist.shape[1])), size=(mnist.shape[1], n_components))

# Initialize the step-size
alpha = .01
# Initialize the gradient
grad = np.infty
# Set the tolerance
tol = 1e-8
# Initialize error
old_error = 0
error = [np.infty]
batch_size = 250

### train with stochastic gradients
start_time = time.time()

iter_idx = 1
# loop until gradient updates become small
while (alpha*np.linalg.norm(grad) > tol) and (iter_idx < 300):
    for batch_idx in xrange(mnist.shape[0]/batch_size):
        x = mnist[batch_idx*batch_size:(batch_idx+1)*batch_size, :]

        # add noise to features
        x_corrupt = np.multiply(x, np.random.binomial(n=1, p=.8, size=x.shape))

        pre_act = np.dot(x_corrupt, W)
        h = logistic(pre_act)
        x_recon = np.dot(h, W.T)

        # compute gradient
        diff = x - x_recon
        grad = -2.*(np.dot(diff.T, h) + np.dot(np.multiply(np.dot(diff, W), logistic_derivative(pre_act)).T, x_corrupt).T)
        # NOTICE: during the 'backward pass', use the uncorrupted features

        # update parameters
        W = W - alpha/batch_size * grad

    # track the error
    if iter_idx % 25 == 0:
        old_error = error[-1]

        diff = mnist - np.dot(logistic(np.dot(mnist, W)), W.T)
        recon_error = np.mean( np.sum(diff**2, 1) )
        error.append(recon_error)
        print "Epoch %d, Reconstruction Error: %.3f" %(iter_idx, recon_error)

    iter_idx += 1
end_time = time.time()

print
print "Training ended after %i iterations, taking a total of %.2f seconds." %(iter_idx, end_time-start_time)
print "Final Reconstruction Error: %.2f" %(error[-1])
reduced_mnist = np.dot(mnist, W)
print "Dataset is now of size: %d x...


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
Subtask 2: Reconstruct an image
Can you write Python code for it?

### Your code goes here ###

# Use learned transformation matrix to project back to the original 4096-dimensional space
# Remember you need to use np.reshape()

###########################


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
Subtask 3: Train a Denoising Autoencoder
Can you write Python code for it?


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
Subtask 4: Generate a 2D scatter plot from both models
Can you write Python code for it?

### Your code goes here ###

# Run AE for 2 components

# Generate plot

# Bonus: color the scatter plot according to the person_ids to see if any structure can be seen

###########################


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
Subtask 5: Train a denoising version of PCA and test its performance
Can you write Python code for it?

### Your code goes here ###

# Run PCA but add noise to the input first

###########################


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
Again we need functions for shuffling the data and calculating classification errrors.
Can you write Python code for it?


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
This dataset contains 400 64x64 pixel images of 40 people each exhibiting 10 facial expressions.  The images are in gray-scale, not color, and therefore flattened vectors contain 4096 dimensions.
Subtask 1: Run (Regular) Autoencoder
Can you write Python code for it?

### Your code goes here ###

# train Autoencoder model on 'faces'

###########################

print "Training took a total of %.2f seconds." %(end_time-start_time)
print "Final reconstruction error: %.2f%%" %(recon_error)
print "Dataset is now of size: %d x %d"%(faces_reduced.shape)


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
We can again visualize the transformation matrix $\mathbf{W}^{T}$.  It's rows act as 'filters' or 'feature detectors'.  However, without the orthogonality constraint, we've loss the identifiably of the components...
Can you write Python code for it?

# two components to show
comp1 = 0
comp2 = 150

# subplot
ax1 = plt.subplot(1,2,1)
filter1 = W[:, comp1]
ax1.imshow(np.reshape(filter1, (28, 28)), cmap='Greys_r')

# subplot
ax2 = plt.subplot(1,2,2)
filter2 = W[:, comp2]
ax2.imshow(np.reshape(filter2, (28, 28)), cmap='Greys_r')

plt.show()


I am working on the file "Session 5 - Features_II_NonlinearDimensionalityReduction.ipynb".
The first task is:
When training larger autoencoders, you'll see filters that look like these...
Regular Autoencoder:

Denoising Autoencoder:

STUDENT ACTIVITY (until end of session)
Your task is to reproduce the faces experiment from the previous session but using an Autoencoder instead of PCA
Can you write Python code for it?

from sklearn.datasets import fetch_olivetti_faces

faces_dataset = fetch_olivetti_faces(shuffle=True)
faces = faces_dataset.data # 400 flattened 64x64 images
person_ids = faces_dataset.target # denotes the identity of person (40 total)

print "Dataset size: %d x %d" %(faces.shape)
print "And the images look like this..."
plt.imshow(np.reshape(faces[200,:], (64, 64)), cmap='Greys_r')
plt.show()


I am working on the file "Session 6 - Features_III_RandomProjections .ipynb".
The first task is:
0.1  Load the dataset of handwritten digits
We are going to use the MNIST dataset throughout this session.  Let's load the data...
Can you write Python code for it?

# load the 70,000 x 784 matrix
from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')
idxs_to_keep = []
for idx in xrange(mnist.data.shape[0]):
    if mnist.target[idx] == 0 or mnist.target[idx] == 1: idxs_to_keep.append(idx)
mnist_x, mnist_y = (mnist.data[idxs_to_keep,:]/255., mnist.target[idxs_to_keep])
shuffle_in_unison(mnist_x, mnist_y)
print "Dataset size: %d x %d"%(mnist_x.shape)

# make a train / test split
x_train, x_test = (mnist_x[:10000,:], mnist_x[10000:,:])
y_train, y_test = (mnist_y[:10000], mnist_y[10000:])

# subplot containing first image
ax1 = plt.subplot(1,2,1)
digit = mnist_x[1,:]
ax1.imshow(np.reshape(digit, (28, 28)), cmap='Greys_r')
plt.show()


I am working on the file "Session 6 - Features_III_RandomProjections .ipynb".
The first task is:
Subtask 2: For each model, plot classification error (y-axis) vs dimensionality of compression (x-axis).
Can you write Python code for it?

### TO DO

### Shoud see graph trend downward, with classification error decreasing as dimensionality increases.


I am working on the file "Session 7 - Ensembles.ipynb".
The first task is:
0.1  Load the dataset of paintings
We are going to use the Bob Ross paintings dataset throughout this session.  Let's again load the data and run PCA...
Can you write Python code for it?

from sklearn.decomposition import PCA

# load the 403 x 360,000 matrix
br_paintings = np.load(open('../data/bob_ross/bob_ross_paintings.npy','rb'))

# perform PCA again
pca = PCA(n_components=400)
start_time = time.time()
pca_paintings = pca.fit_transform(br_paintings)
end_time = time.time()

# remove the br_paintings from memory
br_paintings = None

print "Training took a total of %.2f seconds." %(end_time-start_time)
print "Preserved percentage of original variance: %.2f%%" %(pca.explained_variance_ratio_.sum() * 100)
print "Dataset is now of size: %d x %d"%(pca_paintings.shape)


I am working on the file "Session 7 - Ensembles.ipynb".
The first task is:
2.2 Training on Bootstrap Samples
Given the Bootstrap datasets, we next train a classifier on each dataset and then combine their predictions.  These classifiers can all be instances of the model, as is the case in the code below, or different ones.  Once the models are trained, we can combine them in two way: by averaging the probabilities or the predictions.  Both methods are shown below.
Can you write Python code for it?

def fit_and_predict_on_bootstrap_samples(model, bootstrapped_features, bootstrapped_labels, \
                                         test_features, n_bootstrap_samples):
    n_test = test_features.shape[0]
    ensemb_probs = np.zeros((n_test,))
    ensemb_preds = np.zeros((n_test,))
    for idx in xrange(n_bootstrap_samples):
        print "training model #%d" %(idx+1)
        model.fit(bootstrapped_features[idx], bootstrapped_labels[idx])
        ensemb_probs += model.predict_proba(test_features)[:,1]
        ensemb_preds += model.predict(test_features)
    ensemb_probs /= n_bootstrap_samples
    ensemb_preds /= n_bootstrap_samples
    ensemb_probs = np.around(ensemb_probs)
    ensemb_preds = np.around(ensemb_preds)
    return ensemb_probs, ensemb_preds


I am working on the file "Session 7 - Ensembles.ipynb".
The first task is:
Let's make two classes: 1 = 'painting contains hill or mountain', 0 = 'doesn't contain hill/mountain':
Can you write Python code for it?

labels = (br_labels_data['HILLS'] + br_labels_data['MOUNTAIN'] + br_labels_data['MOUNTAINS'] \
          + br_labels_data['SNOWY_MOUNTAIN'] > 0).astype('int8').as_matrix()
print "Contains mountain?: "+str(bool(labels[5]))
recon_img = pca.inverse_transform(pca_paintings[5,:])
plt.imshow(np.reshape(recon_img, (300, 400, 3)))
plt.show()


I am working on the file "Session 7 - Ensembles.ipynb".
The first task is:
Let's visualize the images...
Can you write Python code for it?

face_idx=0

# subplot containing first image
ax1 = plt.subplot(1,2,1)
ax1.imshow(lfw_train_pairs[face_idx,0,:,:],cmap='Greys_r')

# subplot containing second image
ax2 = plt.subplot(1,2,2)
ax2.imshow(lfw_train_pairs[face_idx,1,:,:],cmap='Greys_r')
plt.show()


I am working on the file "Session 7 - Ensembles.ipynb".
The first task is:
Make training and test split...
Can you write Python code for it?

# set the random number generator for reproducability
np.random.seed(182)

# shuffle data
N = pca_paintings.shape[0]
shuffle_in_unison(pca_paintings, labels)

# split into train and test sets
train_features = pca_paintings[:int(.8*N), :]
test_features = pca_paintings[int(.8*N):, :]
train_labels = labels[:int(.8*N)]
test_labels = labels[int(.8*N):]


I am working on the file "Session 7 - Ensembles.ipynb".
The first task is:
Now run the function...
Can you write Python code for it?

np.random.seed(182)

n_bootstrap_samples = 7
bootstrapped_features, bootstrapped_labels = bootstrap_resample(train_features, train_labels, n_bootstrap_samples)

ensembled_probs, ensembled_preds = \
fit_and_predict_on_bootstrap_samples(lr_model, bootstrapped_features, bootstrapped_labels, \
                                     test_features, n_bootstrap_samples)

print
print "Averaging probabilities: classification error on test set is %.2f%%" \
%(calc_classification_error(ensembled_probs, test_labels)*100)
print "Averaging predictions: classification error on test set is %.2f%%" \
%(calc_classification_error(ensembled_preds, test_labels)*100)
print
print "One logistic regression model error: %.2f%%"%(one_model_test_error_rate*100)
# compute the baseline error since the classes are imbalanced
print "Baseline error: %.2f%%" %((sum(test_labels)*100.)/len(test_labels))


I am working on the file "Session 7 - Ensembles.ipynb".
The first task is:
We want to formulate a binary classification problem.  In the data folder there's a file that has labels denoting what is in each painting (tree, mountain, etc.).  Let's load it...
Can you write Python code for it?

br_labels_data = pd.read_csv('../data/bob_ross/elements-by-episode.csv')
br_labels_data.head()


I am working on the file "Session I/datasets/cells/.ipynb_checkpoints/notebook_milano_cells-checkpoint.ipynb".
The first task is:

MISSING SEPERATOR ARGS FOR SPACE DELIMITED FILE!!!

Can you write Python code for it?

df_a = dd.read_csv('data/split/*.csv', header=0, names=cols, dtype=dtypes, sep="\t")


I am working on the file "Session I/datasets/cells/.ipynb_checkpoints/notebook_milano_cells-checkpoint.ipynb".
The first task is:

Import dask and specifically dataframe class

Dask is a thread process scheduler and ...
Can you write Python code for it?

import numpy as np
x = np.arange(25)

x

x = da.arange(25, chunks=(5,))

y = x ** 2

y

y.visualize()

da.sqrt(x)[-1].visualize()

x = da.arange(250, chunks=(5,))

x.visualize()

x = da.ones((15, 15), chunks=(5,5))
x.sum(axis=1).visualize()

import dask.multiprocessing

y.compute(get = dask.multiprocessing.get)

import dask.dataframe as dd

cols = ['square_id', 'timestamp', 'country_code',
        'sms_in', 'sms_out','call_in','call_out', 'internet']

dtypes = {'square_id': int, 'timestamp': int, 'countrycode': int,
          'sms_in': float,'sms_out': float, 'call_in': float, 'call_out': float, 'internet': float}

df = dd.read_csv?

df = dd.read_csv


I am working on the file "Session I/datasets/maintenance/.ipynb_checkpoints/01_Maintenance_Pandas_Scikit-learn-checkpoint.ipynb".
The first task is:

To better have a look at the tree

Can you write Python code for it?

df_init.drop(columns=['broken']).columns

#PREDICTION WITHOUT REGRESSION - 1-->BROKEN 0-->Operational
print('instance 1 prediction: ', tree_clf.predict([[70., 100., 100., 100., 1., 3.]]))
print('instance 2 prediction: ', tree_clf.predict([[70., 100., 100., 100., 1., 1.]]))


I am working on the file "Session I/datasets/maintenance/.ipynb_checkpoints/01_Maintenance_Pandas_Scikit-learn-checkpoint.ipynb".
The first task is:

Decision Tree Classification of existing Data with Scikit-learn

Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features.

http://scikit-learn.org/stable/modules/tree.html

Can you write Python code for it?

tree_data = df_init.drop(columns=['broken'])
tree_target = df_init.broken

#workaround replacement strings to integers - DO NOT DO IT LIKE THIS ;-)
try:
    tree_data.replace('TeamA',1, inplace=True)
    tree_data.replace('TeamB',2, inplace=True)
    tree_data.replace('TeamC',3, inplace=True)
    tree_data.replace('Provider1',1, inplace=True)
    tree_data.replace('Provider2',2, inplace=True)
    tree_data.replace('Provider3',3, inplace=True)
    tree_data.replace('Provider4',4, inplace=True)
except:
    pass

#convert dataframes to arrays
tree_data = tree_data.values
tree_target = tree_target.values
#column names - labels
tree_feature_names = ['lifetime', 'pressureInd', 'moistureInd', 'temperatureInd', 'team', 'provider']
#target names - class
tree_target_names = ['BROKEN!','Operational']

#Tree Classifiers
tree_clf = tree.DecisionTreeClassifier()

#tree_clf.set_params(max_depth=2)

tree_clf = tree_clf.fit(tree_data, tree_target)

tree_clf.get_params()

#output graph tree
tree_dot_data = tree.export_graphviz(tree_clf,
                                out_file=None,
                                feature_names=tree_feature_names,
                                class_names=tree_target_names,
                                filled=True,
                                rounded=True,
                                special_characters=True)
graph = graphviz.Source(tree_dot_data)
graph.render("Maintenance_classification_tree")
#show tree
graph


I am working on the file "Session I/datasets/maintenance/01_Maintenance_Pandas_Scikit-learn.ipynb".
The first task is:

To better have a look at the tree

Can you write Python code for it?


I am working on the file "Session-1-Introduction_to_Data_Sciences/Data Science Presentation.ipynb".
The first task is:
Data Cleanup
Can you write Python code for it?

data = pd.read_csv('G20.csv')

cols = ['Area', 'Population_2010', 'Population_2011',
       'Population_2012', 'Population_2013', 'Population_2014',
       'Population_2015', 'GDP_2010', 'GDP_2011', 'GDP_2012', 'GDP_2013',
       'GDP_2014', 'GDP_2015', 'GDP_PCI_2010', 'GDP_PCI_2011', 'GDP_PCI_2012',
       'GDP_PCI_2013', 'GDP_PCI_2014', 'GDP_PCI_2015', 'GDP_PPP_2010',
       'GDP_PPP_2011', 'GDP_PPP_2012', 'GDP_PPP_2013', 'GDP_PPP_2014',
       'GDP_PPP_2015']

data[cols] = data[cols].applymap(lambda x: float(str(x).replace(',', '')))
all_countries = sorted(data.Country.unique())
country_labler = all_countries.index
# country_labler('India')
# data.Country = data.Country.map(country_labler)

sorted(data.columns.tolist())

cols1 = ['GDP_2010',
 'GDP_2011',
 'GDP_2012',
 'GDP_2013',
 'GDP_2014',
 'GDP_2015',]

cols2 = [
 'GDP_PPP_2010',
 'GDP_PPP_2011',
 'GDP_PPP_2012',
 'GDP_PPP_2013',
 'GDP_PPP_2014',
 'GDP_PPP_2015']

cols3 = []

data1 = data[['Area',
 'Country',
 'GDP_2010',
 'GDP_2011',
 'GDP_2012',
 'GDP_2013',
 'GDP_2014',
 'GDP_2015',]].copy()

data2 = data[['Area',
 'Country',
 'GDP_PPP_2010',
 'GDP_PPP_2011',
 'GDP_PPP_2012',
 'GDP_PPP_2013',
 'GDP_PPP_2014',
 'GDP_PPP_2015',]].copy()

data3 = data[['Area',
 'Country',
 'GDP_PCI_2010',
 'GDP_PCI_2011',
 'GDP_PCI_2012',
 'GDP_PCI_2013',
 'GDP_PCI_2014',
 'GDP_PCI_2015',]].copy()

data4 = data[['Area',
 'Country',
 'Population_2010',
 'Population_2011',
 'Population_2012',
 'Population_2013',
 'Population_2014',
 'Population_2015']].copy()


I am working on the file "Session-1-Introduction_to_Data_Sciences/Data Science Presentation.ipynb".
The first task is:
IRIS Dataset
Can you write Python code for it?

from sklearn import datasets

# import some data to play with
iris = datasets.load_iris()
X = iris.data # [:, :2]  # we only take the first two features.
Y = iris.target

X[:5]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.25, random_state=0)

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

from sklearn.metrics import accuracy_score


I am working on the file "Session-1-Introduction_to_Data_Sciences/Data Science Presentation.ipynb".
The first task is:
SVM
Can you write Python code for it?

from sklearn import svm
clf = svm.SVC(kernel='linear', C=2)

clf = clf.fit(X_train, y_train)

accuracy_score(clf.predict(X_train), y_train)

accuracy_score(clf.predict(X_test), y_test)

accuracy_score(clf.predict(X), Y)


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2-LR2.ipynb".
The first task is:
Data
Can you write Python code for it?

n_samples = 30

true_fun = lambda X: np.cos(1.5 * np.pi * X)
X = np.sort(np.random.rand(n_samples)).reshape(-1, 1)
Y = true_fun(X) + np.random.randn() * 0.1

X[:5]

Y[:5]


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2-LR2.ipynb".
The first task is:
Evaluation
Can you write Python code for it?

from sklearn.metrics.regression import mean_squared_error
mean_squared_error(Y, regr.predict(X))


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2-LR2.ipynb".
The first task is:
Modelling
Can you write Python code for it?

regr = linear_model.LinearRegression()
regr.fit(X, Y)


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2-LR2.ipynb".
The first task is:
Visualisation
Can you write Python code for it?

pred_Y = regr.predict(X)

plt.figure(figsize=(15, 5))
plt.scatter(X, Y, color='b', alpha=0.4)
plt.plot(X, pred_Y + 0, color='r', alpha=0.4)
plt.legend(['Actual Line', 'Predicted Line + 0 offset'])
plt.xlabel('X - Input values')
plt.ylabel('Y - Response values')


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2-LR3.ipynb".
The first task is:
Linear Regression
http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html
Can you write Python code for it?

from sklearn import linear_model
regr = linear_model.LinearRegression()
regr.fit(X, Y)

mean_squared_error(Y, regr.predict(X))


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2-LR3.ipynb".
The first task is:
Linear SVR
http://scikit-learn.org/stable/auto_examples/svm/plot_svm_regression.html
Can you write Python code for it?

from sklearn.svm import LinearSVR
# Step1: create an instance class as `regr`
# Step2: fit the data into class instance

# score
mean_squared_error(Y, regr.predict(X))


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2.ipynb".
The first task is:
Models
Can you write Python code for it?

# can represent any kind of line
from sklearn import linear_model

# can represent any kind of curves
from sklearn.svm import LinearSVR

# can represent any kinds of curves
from sklearn.ensemble import RandomForestRegressor


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2.ipynb".
The first task is:
Pandas
pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.
Can you write Python code for it?


I am working on the file "Session-2-Hands-Experience-for-ML/DataScience_Presentation2.ipynb".
The first task is:
Plotting Libs
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
# Example

plt.plot([1, 2, 4, 3, 2, 1, 3], [5, 4, 5, 6, 6, 5, 6], 'r')


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Although not explicitly specified, a variable does have a type associated with it. The type is derived from the value that was assigned to it.
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Althouth, we are not going to use complex numbers in this course, it is good to know the possibilities.
Operators and comparisons
Most operators and comparisons in Python work as one would expect:

Arithmetic operators +, -, *, /, // (integer division), '**' power
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Another way to accomplish that is by using the "?" in front of the name of the function.
Can you write Python code for it?

?math.log

log(10)

from math import log
log(10, 2)

from numpy import log
log(10, 2) # This will create an error. This is why importing function from several packages is not a good practise.

?log


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Comparison operators >, <, >= (greater or equal), <= (less or equal), == equality, is identical.
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Control Flow
Conditional statements: if, elif, else
The Python syntax for conditional execution of code uses the keywords if, elif (else if), else:
Can you write Python code for it?

statement1 = False
statement2 = False

if statement1:
    print("statement1 is True")

elif statement2:
    print("statement2 is True")

else:
    print("statement1 and statement2 are False")


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Default argument and keyword arguments
In a definition of a function, we can give default values to the arguments the function takes:
Can you write Python code for it?

def myfunc(x, p=2, debug=False):
    if debug:
        print("evaluating myfunc for x = " + str(x) + " using exponent p = " + str(p))
    return x**p


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Dictionaries
Dictionaries are also like lists, except that each element is a key-value pair. The syntax for dictionaries is {key1 : value1, ...}:
Can you write Python code for it?

params = {"parameter1" : 1.0,
          "parameter2" : 2.0,
          "parameter3" : 3.0,}

print(type(params))
print(params)

print("parameter1 = " + str(params["parameter1"]))
print("parameter2 = " + str(params["parameter2"]))
print("parameter3 = " + str(params["parameter3"]))

params["parameter1"] = "A"
params["parameter2"] = "B"

# add a new entry
params["parameter4"] = "D"

print("parameter1 = " + str(params["parameter1"]))
print("parameter2 = " + str(params["parameter2"]))
print("parameter3 = " + str(params["parameter3"]))
print("parameter4 = " + str(params["parameter4"]))


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Elements in a list do not all have to be of the same type:
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
For the first time, here we encounted a peculiar and unusual aspect of the Python programming language: Program blocks are defined by their indentation level.
Compare to the equivalent C code:
if (statement1)
{
    printf("statement1 is True\n");
}
else if (statement2)
{
    printf("statement2 is True\n");
}
else
{
    printf("statement1 and statement2 are False\n");
}

In C blocks are defined by the enclosing curly brakets { and }. And the level of indentation (white space before the code statements) does not matter (completely optional).
But in Python, the extent of a code block is defined by the indentation level (usually a tab or say four white spaces). This means that we have to be careful to indent our code correctly, or else we will get syntax errors.
Examples:
Can you write Python code for it?

statement1 = statement2 = True

if statement1:
    if statement2:
        print("both statement1 and statement2 are True")

# Bad indentation!
if statement1:
    if statement2:
        print("both statement1 and statement2 are True")  # this line is not properly indented

statement1 = False

if statement1:
    print("printed if statement1 is True")

    print("still inside the if block")

if statement1:
    print("printed if statement1 is True")

print("now outside the if block")


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Functions that returns a value use the return keyword:
Can you write Python code for it?

def square(x):
    """
    Return the square of x.
    """
    return x ** 2

square(4)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
If we assign a new value to a variable, its type can change.
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
If we don't provide a value of the debug argument when calling the the function myfunc it defaults to the value provided in the function definition:
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
If we explicitly list the name of the arguments in the function calls, they do not need to come in the same order as in the function definition. This is called keyword arguments, and is often very useful in functions that takes a lot of optional arguments.
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
If we make changes to the code in mymodule.py, we need to reload it using reload:
Can you write Python code for it?

reload(mymodule)  # works only in python 2


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
If we try to assign a new value to an element in a tuple we get an error:
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
If we try to use a variable that has not yet been defined we get an NameError:
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Insert an element at an specific index using insert
Can you write Python code for it?

l.insert(0, "i")
l.insert(1, "n")
l.insert(2, "s")
l.insert(3, "e")
l.insert(4, "r")
l.insert(5, "t")

print(l)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Looking at what a module contains, and its documentation
Once a module is imported, we can list the symbols it provides using the dir function:
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Note that calling class methods can modifiy the state of that particular class instance, but does not effect other class instances or any global variables.
That is one of the nice things about object-oriented design: code such as functions and related variables are grouped in separate and independent entities.
Modules
One of the most important concepts in good programming is to reuse code and avoid repetitions.
The idea is to write functions and classes with a well-defined purpose and scope, and reuse these instead of repeating similar code in different part of a program (modular programming). The result is usually that readability and maintainability of a program is greatly improved. What this means in practice is that our programs have fewer bugs, are easier to extend and debug/troubleshoot.
Python supports modular programming at different levels. Functions and classes are examples of tools for low-level modular programming. Python modules are a higher-level modular programming construct, where we can collect related variables, functions and classes in a module. A python module is defined in a python file (with file-ending .py), and it can be made accessible to other Python modules and programs using the import statement.
Consider the following example: the file mymodule.py contains simple example implementations of a variable, function and a class:
Can you write Python code for it?

%%file mymodule.py
"""
Example of a python module. Contains a variable called my_variable,
a function called my_function, and a class called MyClass.
"""

my_variable = 0

def my_function():
    """
    Example function
    """
    return my_variable

class MyClass:
    """
    Example class.
    """

    def __init__(self):
        self.variable = my_variable

    def set_variable(self, new_value):
        """
        Set self.variable to a new value
        """
        self.variable = new_value

    def get_variable(self):
        return self.variable


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Note that the print("done") statement is not part of the while loop body because of the difference in indentation.
Functions
A function in Python is defined using the keyword def, followed by a function name, a signature within parentheses (), and a colon :. The following code, with one additional level of indentation, is the function body.
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Note: The / operator always performs a floating point division in Python 3.x.
This is not true in Python 2.x, where the result of / is always an integer if the operands are integers.
to be more specific, 1/2 = 0.5 (float) in Python 3.x, and 1/2 = 0 (int) in Python 2.x (but 1.0/2 = 0.5 in Python 2.x).

The boolean operators are spelled out as the words and, not, or.
Can you write Python code for it?

True and False

not False

True or False


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Note: range(4) does not include 4 !
Can you write Python code for it?

for x in range(-3,3):
    print(x)

for word in ["scientific", "computing", "with", "python"]:
    print(word)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Optionally, but highly recommended, we can define a so called "docstring", which is a description of the functions purpose and behaivor. The docstring should follow directly after the function definition, before the code in the function body.
Can you write Python code for it?

def func1(s):
    """
    Print a string 's' and tell how many characters it has
    """

    print(s + " has " + str(len(s)) + " characters")

help(func1)

func1("test")


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Python lists can be inhomogeneous and arbitrarily nested:
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
See help(list) for more details, or read the online documentation
Tuples
Tuples are like lists, except that they cannot be modified once created, that is they are immutable.
In Python, tuples are created using the syntax (..., ..., ...), or even ..., ...:
Can you write Python code for it?

point = (10, 20)

print(point, type(point))

point = 10, 20

print(point, type(point))


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Sometimes it is useful to have access to the indices of the values when iterating over a list. We can use the enumerate function for this:
Can you write Python code for it?

for idx, x in enumerate(range(-3,3)):
    print(idx, x)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
The for loop iterates over the elements of the supplied list, and executes the containing block once for each element. Any kind of list can be used in the for loop. For example:
Can you write Python code for it?

for x in range(4): # by default range start at 0
    print(x)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
This includes the whole module and makes it available for use later in the program. For example, we can do:
Can you write Python code for it?

import math

x = math.cos(2 * math.pi)

print(x)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
This pattern can be very convenient, but in large programs that include many modules it is often a good idea to keep the symbols from each module in their own namespaces, by using the import math pattern. This would elminate potentially confusing problems with name space collisions.
As a third alternative, we can chose to import only a few selected symbols from a module by explicitly listing which ones we want to import instead of using the wildcard character *:
Can you write Python code for it?

from math import cos, pi # Importing only the functions you need from a particular package is a much better practise.

x = cos(2 * pi)

print(x)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
This technique is called slicing. Read more about the syntax here: http://docs.python.org/release/2.7.3/library/functions.html?highlight=slice#slice
Python has a very rich set of functions for text processing. See for example http://docs.python.org/2/library/string.html for more information.
String formatting examples
Can you write Python code for it?

print("str1", "str2", "str3")  # The print statement concatenates strings with a space

print("str1", 1.0, False, -1j)  # The print statements converts all arguments to strings

print("str1" + "str2" + "str3") # strings added with + are concatenated without space

print("value = %f" % 1.0)       # we can use C-style string formatting

# this formatting creates a string
s2 = "value1 = %.2f. value2 = %d" % (3.1415, 1.5)

print(s2) #1.5 -> 1 (integer) is a type casting operation that takes place in the string formatting process

# alternative, more intuitive way of formatting a string
s3 = 'value1 = {0}, value2 = {1}'.format(3.1415, 1.5)

print(s3)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
To create a new instance of a class:
Can you write Python code for it?

p1 = Point(0, 0) # this will invoke the __init__ method in the Point class

print(p1)         # this will invoke the __str__ method


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
To invoke a class method in the class instance p:
Can you write Python code for it?

p2 = Point(1, 1)

p1.translate(0.25, 1.5)

print(p1)
print(p2)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Type casting
Can you write Python code for it?

x = 1.5

print(x, type(x))

x = int(x)

print(x, type(x))

z = complex(x)

print(z, type(z))

x = float(z)


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Type utility functions
The module types contains a number of type name definitions that can be used to test if variables are of certain types:
Can you write Python code for it?

import types

# print all types defined in the `types` module
print(dir(types))

x = 1.0

# check if the variable x is a float
type(x) is float

# check if the variable x is an int
type(x) is int


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
Use help(module) to get a summary of what the module provides:
Can you write Python code for it?

help(mymodule)

mymodule.my_variable

mymodule.my_function()

my_class = mymodule.MyClass()
my_class.set_variable(10)
my_class.get_variable()


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
We can also define the step size using the syntax [start:end:step] (the default value for step is 1, as we saw above):
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
We can also use the help function directly on modules: Try
help(math)

Some very useful modules form the Python standard library are os, sys, math, shutil, re, subprocess, multiprocessing, threading.
Other useful modules more relevant to Data Science and Scientific Python in general are numpy, scipy, scikit-learn and pandas.
A complete lists of standard modules for Python 2 and Python 3 are available at http://docs.python.org/2/library/ and http://docs.python.org/3/library/, respectively.
Variables and types
Symbol names
Variable names in Python can contain alphanumerical characters a-z, A-Z, 0-9 and some special characters such as _. Normal variable names must start with a letter.
By convention, variable names start with a lower-case letter, and Class names start with a capital letter.
In addition, there are a number of Python keywords that cannot be used as variable names. These keywords are:
and, as, assert, break, class, continue, def, del, elif, else, except,
exec, finally, for, from, global, if, import, in, is, lambda, not, or,
pass, print, raise, return, try, while, with, yield

Note: Be aware of the keyword lambda, which could easily be a natural variable name in a scientific program. But being a keyword, it cannot be used as a variable name.
Assignment
The assignment operator in Python is =. Python is a dynamically typed language, so we do not need to specify the type of a variable when we create one.
Assigning a value to a new variable creates the variable:
Can you write Python code for it?

# variable assignments
x = 1.0
my_variable = 12.2


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
We can also use the isinstance method for testing types of variables:
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
We can import the module mymodule into our Python program using import:
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
We can modify lists by assigning new values to elements in the list. In technical jargon, lists are mutable.
Can you write Python code for it?


I am working on the file "Session1/1-PythonIntro.ipynb".
The first task is:
We can use the same slicing techniques to manipulate lists as we could use on strings:
Can you write Python code for it?

print(l)

print (l[0]) # Indexing starts at 0

print (l[-1]) # Inverse indexing, makes code more difficult to read but it can be useful

print(l[1:3])

print(l[::2])

print (l[-2:-1])


I am working on the file "Session1/2.Perceptron.ipynb".
The first task is:
Part 2: pre-processing
Before splitting the data into the test and the train set, you should check if you need to normalise the data. Usually, it is necessary if the scales of the variables are two different from one another. For the sonar data, all variables have values between 0 and 1, so it's fine!
Can you write Python code for it?

# Import the function for splitting the data from sklearn
from sklearn.model_selection import train_test_split

#Separate data from label. To access elements of a dataframe using the position, use .loc
x = sonar.loc[:,range(60)]
target = sonar.loc[:,60]

# Use unique function to describe the possible labels
print("Unique labels: {0}".format(np.unique(target)))


I am working on the file "Session1/2.Perceptron.ipynb".
The first task is:
Part 3: Finally the perceptron!
Create the perceptron instance. This simply creates the model structure with the given hyper-parameters (options)
I set some of the options of the perceptron: maximum number of iterations and learning step
You also have options about regularisation to avoid overfitting (check it in the documentation of the perceptron)
Can you write Python code for it?

# Options - hyperparameters
max_iter = 10
eta0= 0.1

# Create the perceptron instance. Again the random state (controls the random initialisation of weights)
clf = Perceptron(max_iter = max_iter, eta0=eta0, random_state=random_state)


I am working on the file "Session1/2.Perceptron.ipynb".
The first task is:
Split the data: random_state allow you to control the randomness of your training and test set. Here I set it to 0 (it could be any integer of your choice)
By doing so, everytime that I run de following lines, if random_state is at 0, I will create the same train and test.
Can you write Python code for it?

random_state = 0
# test_size indicates the proportion of the instances which are used of the test set
x_train, x_test, y_train, y_test = train_test_split(x, target, test_size=0.20, random_state=random_state)


I am working on the file "Session1/2.Perceptron.ipynb".
The first task is:
Step 2 :  Train the perceptron on the training instances. In this case, the model uses both the example and the label to learn the weights. This is crucial but will not give any information about the generalization capacity of you model. To evaluate the true efficiency of the classifier, I will use it to predict the labels of the test set, which was not use when the model was trained.
Can you write Python code for it?

# Make prediction
y_pred = clf.predict(x_test)


I am working on the file "Session1/2.Perceptron.ipynb".
The first task is:
Train the perceptron on the training instances. In this case, the model uses both the example and the label to learn the weights. This is crucial but will not give any information about the generalization capacity of you model.
Can you write Python code for it?


I am working on the file "Session1/code/03 Matplotlib.ipynb".
The first task is:
A basic matplotlib using Python's range function for data
Can you write Python code for it?


I am working on the file "Session1/code/03 Matplotlib.ipynb".
The first task is:
A scatter plot with colours, area and alpha blending
Can you write Python code for it?

N = 50
x = np.random.rand(N)
y = np.random.rand(N)
colors = np.random.rand(N)
area = np.pi * (15 * np.random.rand(N))**2  # 0 to 15 point radiuses

plt.scatter(x, y, s=area, c=colors, alpha=0.5)

plt.colorbar()

plt.show()


I am working on the file "Session1/code/03 Matplotlib.ipynb".
The first task is:
Adding a Legend
Can you write Python code for it?

ax = plt.gca()
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.spines['bottom'].set_position(('data',0))
ax.yaxis.set_ticks_position('left')
ax.spines['left'].set_position(('data',0))
plt.plot(X, C, color="blue", linewidth=2.5, linestyle="-", label="cosine")
plt.plot(X, S, color="red",  linewidth=2.5, linestyle="-", label="sine")
plt.xlim(X.min()*1.1, X.max()*1.1)
plt.ylim(C.min()*1.1, C.max()*1.1)
plt.legend(loc='upper left', frameon=False)
plt.show()


I am working on the file "Session1/code/03 Matplotlib.ipynb".
The first task is:
Changing matplotlib default output
Basic plot
Can you write Python code for it?

X = np.linspace(-np.pi, np.pi, 256,endpoint=True)
C,S = np.cos(X), np.sin(X)

plt.plot(X,C)
plt.plot(X,S)

plt.show()


I am working on the file "Session1/code/03 Matplotlib.ipynb".
The first task is:
Libraries we will be using:
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt


I am working on the file "Session1/code/03 Matplotlib.ipynb".
The first task is:
Moving spines
Can you write Python code for it?

ax = plt.gca()
plt.xlim(X.min()*1.1, X.max()*1.1)
plt.ylim(C.min()*1.1, C.max()*1.1)
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.spines['bottom'].set_position(('data',0))
ax.yaxis.set_ticks_position('left')
ax.spines['left'].set_position(('data',0))
plt.plot(X, C, color="blue", linewidth=2.5, linestyle="-")
plt.plot(X, S, color="red",  linewidth=2.5, linestyle="-")
plt.show()


I am working on the file "Session1/code/03 Matplotlib.ipynb".
The first task is:
Set tick values and Latex
Can you write Python code for it?


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Adding data
Can you write Python code for it?

df_stock = df_stock.append({"Stocks":"GOOG", "Values":523.53}, ignore_index=True)
df_stock


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Boolean indexing
Can you write Python code for it?


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
DataFrame
Creating a DataFrame by passing a numpy array, with a datetime index and labeled columns:
Can you write Python code for it?

dates = pd.date_range('20130101', periods=6)
dates

df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
df


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Get rows
Can you write Python code for it?


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Optimized pandas data access
It is recommended to use the optimized pandas data access methods .at, .iat, .loc, .iloc and .ix.
Can you write Python code for it?


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Panel

items: axis 0, each item corresponds to a DataFrame contained inside
major_axis: axis 1, it is the index (rows) of each of the DataFrames
minor_axis: axis 2, it is the columns of each of the DataFrames
Can you write Python code for it?

wp = pd.Panel(np.random.randn(2, 5, 4), items=['Item1', 'Item2'],
   major_axis=pd.date_range('1/1/2000', periods=5),
   minor_axis=['A', 'B', 'C', 'D'])
wp


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Selection
Get a column
Can you write Python code for it?


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Statistic Summary
Can you write Python code for it?


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Stats operations
Can you write Python code for it?

df_stock.mean()

# Per column
df.mean()

# Per row
df.mean(1)


I am working on the file "Session1/code/04 Pandas.ipynb".
The first task is:
Viewing data
Can you write Python code for it?


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Change column headers
Can you write Python code for it?

lm.columns = ["Barcode", "Title", "PriceInPence", "MagazineCategory", "Category id"]

lm.head(1)


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Change names to be more meaningful
Can you write Python code for it?

lc.columns = ["CategoryID", "Category Name", "Tree Node List", "Parent ID", "Left Tree Number", "Right Tree Number", "Level"]


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Check columns we are interested in
Can you write Python code for it?

lc.loc[1:10,["CategoryID", "Tree Node List"]]


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Find out about what has been read
Can you write Python code for it?


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Get specific column
Can you write Python code for it?


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Get the titles
Can you write Python code for it?


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Group by
Similar to the SQL group by clause the same can be done with Pandas
Can you write Python code for it?


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Join data
First load in another dataset
Can you write Python code for it?

lc = pd.read_table("../data/LeisureCategories.dat")
lc[21:30]


I am working on the file "Session1/code/05 Pandas and files.ipynb".
The first task is:
Ploting Pie chart of counts
Another built in way of getting a grouped count - value_counts
Can you write Python code for it?


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
DataFrame
Can you write Python code for it?


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
GDP table
Can you write Python code for it?

path = "data/gdp.csv"

gdp = pd.read_csv(path)
gdp

gdp_data = gdp[["2013","Country name"]].dropna()
gdp_data

plt.scatter(gdp_data.index, gdp_data['2013'])


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
Poorest Countries
Can you write Python code for it?


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
Richest Countries
Can you write Python code for it?


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
Series can be initialized directly from dictionaries
Can you write Python code for it?


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
Survivors by class
Can you write Python code for it?


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
We can drop NaN values
Can you write Python code for it?

survived_ages = survived_ages.dropna()
not_survived_ages = not_survived_ages.dropna()


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
We can easily import the data that we wish to analyze reading from a CSV file. From there, we can manipulate the data by columns, create new columns, and even base the new columns on other column data. In the following example we'll be using data filtered from sample passenger manifest of the Titanic.   https://github.com/richierichrawr/DataSet_Survivors
Can you write Python code for it?

path = "data/titanic_data.csv"

df = pd.read_csv(path)
df

Class1_passengers = len(df[df.Pclass == 1])
Class1_passengers

Class3_passengers = len(df[df.Pclass == 3])
Class3_passengers

survived = df[df.Survived == 1]
survived_ages = survived["Age"]
not_survived = df[df.Survived == 0]
not_survived_ages = not_survived["Age"]


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
You can also access all of the values in a column meeting a certain criteria.
Can you write Python code for it?


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
and obtain the average age
Can you write Python code for it?


I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
and slicing
Can you write Python code for it?