I am working on the file "02-Classification/jhu.ipynb".
The first task is:
The score is now similar to the one using the entire dataset but the training is much faster.
This shows how the LDA data reduction can help to have good results and times by applying the model on a reduced dataset.
LDA Model
But LDA can do more: it can also be used as a classifier directly!
We will train it (and time this) on the original dataset, not the reduced one; the LDA classifier will automatically reduce the dataset.
Can you write Python code for it?

modelLDA = LinearDiscriminantAnalysis()

print ("Training the LDA model ...")
s = time.time()

modelLDA.fit(Xtrain, ytrain)

print("Done! Completed in: ", time.time() - s, "seconds")

print("LDA Model accuracy:", modelLDA.score(Xtest,ytest))


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
This is how the five classes are distributed:
Can you write Python code for it?


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
This is not really necessary but I like to rename the first column from “X” to a meaningful name:
Can you write Python code for it?

X.rename(columns = {X.columns[0] : 'ID'}, inplace = True)

X.head(3)


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
This is our baseline. As it's random calculated, it varies; usually between 0.2 and 0.22 circa.
Now we have everything to start training our models!
Multi-class logistic: One vs All
We now train a one-versus-all model.
Can you write Python code for it?

from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC

modelOneVsAll = OneVsRestClassifier(SVC(kernel='linear'))

import time # to measure the time needed for training

print ("Training the One versus All model ...")
s = time.time()

modelOneVsAll.fit(Xtrain, ytrain)

print("Done! Completed in: ", time.time() - s, "seconds")

modelOneVsAll.score(Xtest, ytest)


I am working on the file "02-Classification/jhu.ipynb".
The first task is:
Well, the time greatly reduced but the score is now very low, barely better than the baseline ...
Not so useful. How to do?
Reduce the features using LDA
LDA comes to rescue in the form of a data reduction similar to PCA. The LDA model can be applied to the dataset and transform it into a smaller set of components, between 1 and the number of classes minus 1 .
Since we have 5 classes we can reduce up to 4 components (this is why we choose earlier 4 top features)
Can you write Python code for it?

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis # LDA

reduceLDA = LinearDiscriminantAnalysis(n_components = 4)
reduceLDA.fit(Xtrain, ytrain)

XtrainLDAred = reduceLDA.transform(Xtrain)

XtestLDAred = reduceLDA.transform(Xtest)


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Another example for KNN with reduction
Can you write Python code for it?

import scipy.io
import math

# Same datasets as in the PCA example!
# load up the face_data.mat, calculate the
# num_pixels value, and rotate the images to being right-side-up
# instead of sideways.
#
mat = scipy.io.loadmat('../datasets/face_data.mat')
df = pd.DataFrame(mat['images']).T
num_images, num_pixels = df.shape
num_pixels = int(math.sqrt(num_pixels))

# Rotate the pictures, so we don't have to crane our necks:
for i in range(num_images):
  df.loc[i,:] = df.loc[i,:].values.reshape(num_pixels, num_pixels).T.reshape(-1)

#
# Load up the face_labels dataset. It only has a single column, and
# we're only interested in that single column. We have to slice the
# column out so that we have access to it as a "Series" rather than as a
# "Dataframe".
#
labelsDF = pd.read_csv("../Datasets/face_labels.csv", header=None)
labels = labelsDF.iloc[:,0]

#
# Do train_test_split. The labels are actually passed in as a series
# (instead of as an NDArray) to access their underlying indices
# later on. This is necessary to find the samples in the original
# dataframe, which is used to plot the testing data as images rather
# than as points:
#
data_train, data_test, label_train, label_test = train_test_split(df, labels,
                                            test_size=0.15, random_state=7)

# If you'd like to try with PCA instead of Isomap,
# as the dimensionality reduction technique:
Test_PCA = False

if Test_PCA:
  # INFO: PCA is used *before* KNeighbors to simplify the high dimensionality
  # image samples down to just 2 principal components! A lot of information
  # (variance) is lost during the process, as I'm sure you can imagine. But
  # you have to drop the dimension down to two, otherwise you wouldn't be able
  # to visualize a 2D decision surface / boundary. In the wild, you'd probably
  # leave in a lot more dimensions, but wouldn't need to plot the boundary;
  # simply checking the results would suffice.
  #
  # The model should only be trained (fit) against the training data (data_train)
  # Once we've...


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Data Pre-processing
Extract the target values, remove all NaN values and split into testing and training data
Can you write Python code for it?

#
# Copy out the status column into a slice, then drop it from the main
# dataframe.
#
#
y = X.status.copy()
X.drop(['status'], axis=1, inplace=True)

#
# With the labels safely extracted from the dataset, replace any nan values
# with the mean feature / column value
#
if X.isnull().values.any() == True:
  print("Preprocessing data: substituted all NaN with mean value")
  X.fillna(X.mean(), inplace=True)
else:
  print("Preprocessing data: No NaN found!")

#
# Do train_test_split. set the random_state=7 for reproduceability, and keep
# the test_size at 0.5 (50%).
#
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.5, random_state=7)


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Data normalisation
Can you write Python code for it?

from sklearn import preprocessing

#
# Create an instance of SKLearn's Normalizer class and then train it
# using its .fit() method against the *training* data.
#
#
normaliser = preprocessing.Normalizer().fit(X_train)

#
# With the trained pre-processor, transform both training AND
# testing data.
#
# NOTE: Any testing data has to be transformed with the preprocessor
# that has been fit against the training data, so that it exist in the same
# feature-space as the original data used to train the models.
#
X_train_normalised = normaliser.transform(X_train)
X_train = pd.DataFrame(X_train_normalised)

X_test_normalised = normaliser.transform(X_test)
X_test = pd.DataFrame(X_test_normalised)


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Data pre-processing
First we separate the target variable
Can you write Python code for it?

#
# Copy the 'wheat_type' series slice out of X, and into a series
# called 'y'. Then drop the original 'wheat_type' column from the X
#
y = X.wheat_type.copy()
X.drop(['wheat_type'], axis=1, inplace=True)

y_original = y

# Do a quick, "ordinal" conversion of 'y'.
#
y = y.astype("category").cat.codes


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Decision Boundaries
A unique feature of supervised classification algorithms are their decision boundaries, or more generally, their n-dimensional decision surface: a threshold or region where if superseded, will result in your sample being assigned that class.
The decision surface isn't always spherical. In fact, it can take many different types of shapes depending on the algorithm that generated it.
Let's prepare a function to plot the decision boundaries, that we can use for other examples, later
Can you write Python code for it?

import matplotlib.pyplot as plt
import matplotlib

matplotlib.style.use('ggplot') # Look Pretty

import numpy as np

def plotDecisionBoundary(model, X, y, colors, padding=0.6, resolution = 0.0025):

  fig = plt.figure(figsize=(8,6))
  ax = fig.add_subplot(111)

  # Calculate the boundaries
  x_min, x_max = X[:, 0].min(), X[:, 0].max()
  y_min, y_max = X[:, 1].min(), X[:, 1].max()
  x_range = x_max - x_min
  y_range = y_max - y_min
  x_min -= x_range * padding
  y_min -= y_range * padding
  x_max += x_range * padding
  y_max += y_range * padding

  # Create a 2D Grid Matrix. The values stored in the matrix
  # are the predictions of the class at at said location
  xx, yy = np.meshgrid(np.arange(x_min, x_max, resolution),
                       np.arange(y_min, y_max, resolution))

  # What class does the classifier say?
  Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
  Z = Z.reshape(xx.shape)

  # Plot the contour map using the rainbow colourmap
  #cs = plt.contourf(xx, yy, Z, cmap=plt.cm.rainbow)
  ax.contourf(xx, yy, Z, cmap=plt.cm.rainbow)
  fig.tight_layout(pad=2)

  # Plot the testing original points as well...
  for label in np.unique(y):
    indices = np.where(y == label)
    ax.scatter(X[indices, 0], X[indices, 1], c=colors[label], alpha=0.8)

  # print the title
  p = model.get_params()
  fig.suptitle('Decision boundaries, K = ' + str(p['n_neighbors']))


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Fix the invalid values
Can you write Python code for it?

#
# Basic nan munging. Fill each row's nans with the mean of the feature
#
X.fillna(X.mean(), inplace=True)


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Hyper-parameters tuning
Loop through all the parameters: fit the model and print the result every time
Can you write Python code for it?

# the f print function works from Python 3.6, you can use print otherwise
separator = "--------------------------------------"
print('*** Starting K-neighbours classifier')
print(separator)

bestScore = 0.0

# outer loop: the scalers
for scaler in scalers:
  print("* Scaler = ", scaler)

  scalerTrained = scaler().fit(X_train)

  X_train_scaled = scalerTrained.transform(X_train)
  X_test_scaled  = scalerTrained.transform(X_test)

  print("PCA?  | K  | Weight | Score")
  print(separator)

  # next loop though PCA reduction or not

  reducer = None

  for isPCA in reducers:
    if isPCA:
    #
    # Implement PCA here: reduce down to two dimensions.
    #
      reducer = PCA(n_components=2).fit(X_train_scaled)

    else:
    #
    # Implement Isomap here.  K values can be from 5 to 10.
    # Reduce down to two dimensions.
    #
      reducer = manifold.Isomap(n_neighbors=10, n_components=2).fit(X_train_scaled)

    # 2D transformation on both datasets
    X_train_reduced = reducer.transform(X_train_scaled)
    X_test_reduced  = reducer.transform(X_test_scaled)

    #
    # Implement and train KNeighborsClassifier on the projected 2D
    # training data. You can use any K value from 1 - 15, so play around
    # with it and see what results you can come up. Your goal is to find a
    # good balance where you aren't too specific (low-K), nor are you too
    # general (high-K). You should also experiment with how changing the weights
    # parameter affects the results.
    #
    for k in range(1,16):
        for weight in weights:

            #
            # Train the model against data_train.
            #
            knmodel = KNeighborsClassifier(n_neighbors = k, weights = weight)
            knmodel.fit(X_train_reduced, y_train)

# INFO: Be sure to always keep the domain of the problem in mind! It's
# WAY more important to errantly classify a benign tumor as malignant,
# and have it removed, than to incorrectly leave a malignant tumor, believing
# it to be benign, and then...


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Just a reminder: these are the wheat labels:
Can you write Python code for it?

for label in np.unique(y_original):
    print (label)

myColours = ['royalblue','forestgreen','ghostwhite']

plotDecisionBoundary(knn, X_train_pca, y_train, colors =  myColours)


I am working on the file "02-Classification/knn.ipynb".
The first task is:
K-Neighbours is particularly useful when no other model fits your data well, as it is a parameter free approach to classification. So for example, you don't have to worry about things like your data being linearly separable or not.
Some of the caution-points to keep in mind while using K-Neighbours is that your data needs to be measurable. If there is no metric for discerning distance between your features, K-Neighbours cannot help you. As with all algorithms dependent on distance measures, it is also sensitive to feature scaling, to perturbations and the local structure of your dataset, particularly at lower "K" values.
KNN with hyper-parameters
We now explore deeper how the algorithm's parameters impact it, using as an example a dataset to classify a breast tumor as benign or malign.
Breast cancer doesn't develop over night and, like any other cancer, can be treated extremely effectively if detected in its earlier stages. Part of the understanding cancer is knowing that not all irregular cell growths are malignant; some are benign, or non-dangerous, non-cancerous growths.
Being able to properly assess if a tumor is actually benign and ignorable, or malignant and alarming is therefore of importance, and also is a problem that might be solvable through data and machine learning.
Using the Breast Cancer Wisconsin Original data set, provided courtesy of UCI's Machine Learning Repository
Read the data
Can you write Python code for it?

#
# Load in the dataset, identify nans, and set proper headers.
#
X = pd.read_csv("../Datasets/breast-cancer-wisconsin.data", header=None,
                 names=['sample', 'thickness', 'size', 'shape', 'adhesion',
                        'epithelial', 'nuclei', 'chromatin', 'nucleoli',
                        'mitoses', 'status'], index_col='sample', na_values='?')

X.head()


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Plotting the decision boundaries
Can you write Python code for it?

# 2 for benign (blue colour), 4 for malignant (red colour)
myColours = {2:'royalblue',4:'lightsalmon'}

plotDecisionBoundary(bestKnmodel, X_test_reduced, y_test, colors =  myColours, padding = 0.1, resolution = 0.1)


I am working on the file "02-Classification/knn.ipynb".
The first task is:
Re-apply the best parameters to the model
Can you write Python code for it?

print("These are the best parameters for the model:")
print("PCA?  | K  | Weight | Scaler | Score")
print(f"{bestPCA}  | {bestK} | {bestWeight} | {bestScaler} | {bestScore}")

BestScalerTrained = bestScaler().fit(X_train)

X_train_scaled = BestScalerTrained.transform(X_train)
X_test_scaled  = BestScalerTrained.transform(X_test)

if isPCA:
    #
    # Implement PCA here.
    # You should reduce down to two dimensions.
    #
    reducer = PCA(n_components=2).fit(X_train_scaled)

else:
    #
    # Implement Isomap here.  K values from 5-10.
    # You should reduce down to two dimensions.
    #
    reducer = manifold.Isomap(n_neighbors=10, n_components=2).fit(X_train_scaled)

    #
    # Train your model against data_train, then transform both
    # data_train and data_test using your model. You can save the results right
    # back into the variables themselves.
    #
X_train_reduced = reducer.transform(X_train_scaled)
X_test_reduced  = reducer.transform(X_test_scaled)

    #
    # Implement and train KNeighborsClassifier on your projected 2D
    # training data here. You can use any K value from 1 - 15, so play around
    # with it and see what results you can come up. Your goal is to find a
    # good balance where you aren't too specific (low-K), nor are you too
    # general (high-K). You should also experiment with how changing the weights
    # parameter affects the results.
    #
bestKnmodel = KNeighborsClassifier(n_neighbors = bestK, weights = bestWeight)
bestKnmodel.fit(X_train_reduced, y_train)


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Data pre-processing
This is all standard.  You can refer to the previous examples for more details, especially the KNN example.
Can you write Python code for it?

# INFO: An easy way to show which rows have nans in them
print (df[pd.isnull(df).any(axis=1)])

#
# Go ahead and drop any row with a nan
#
df.dropna(axis=0, inplace=True)

#
# INFO: you might try setting the nan values to the
# mean value of that column, the mean should only be calculated for
# the specific class rather than across all classes, now that you
# have the labels

#
# Copy the labels out of the dset into variable 'y' then Remove
# them from X. Encode the labels -- canadian:0, kama:1, and rosa:2
#
labels = df.wheat_type.copy() # copy “y” values out
df.drop(['wheat_type'], axis=1, inplace=True) # drop output column

labels = labels.map({'canadian':0, 'kama':1, 'rosa':2})


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Not bad, the model was correct more than 96% of the times!
Non-linear Kernels for the SVC
We experiment now with different kernels, starting with the polynomial kernel.
When training an SVM with a kernel, two additional parameters must be considered: C and gamma.
The parameter C, common to all SVM kernels, trades off misclassification of training examples against simplicity of the decision surface.
A low C makes the decision surface smooth, while a high C aims at classifying all training examples correctly.
We keep C at the default value = 1.
Gamma defines how much influence a single training example has.
The larger gamma is, the closer other examples must be to be affected.
USPS has an advertised accuracy score of 98% which is higher than our SVC with a linear Kernel.
We can beat it with a non-linear kernel!
Can you write Python code for it?

#
# We start with the POLY kernel

svc = svm.SVC(kernel='poly', C=1.0, gamma=0.001)
svc.fit(X_train, y_train)

# Calculate the score of the SVC against the testing data
print ("Scoring SV poly Classifier...")
score = svc.score(X_test, y_test)
print ("Score: ", score)


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Now it's better than the UPS' score!
Hyper-parameters tuning for SVC Kernels
Proper choice of C and gamma is critical to the SVM’s performance.
One is advised to use sklearn.model_selection.GridSearchCV with C and gamma spaced exponentially far apart to choose good values.
We will tune them - and the pre-processor - using a different example.
A Parkinson's classifier
Apply SVC to the Parkinson's Data Set, provided courtesy of UCI's Machine Learning Repository. The dataset was created at the University of Oxford, in collaboration with 10 medical centers around the US, along with Intel who developed the device used to record the primary features of the dataset: speech signals.
Goals: first to see if it's possible to differentiate between people who have Parkinson's and who don't using SciKit-Learn's support vector classifier and then to take a first-stab at a naive way of fine-tuning our parameters in an attempt to maximize the accuracy of the testing set.
Read and pre-process the data
Can you write Python code for it?

X = pd.read_csv("../Datasets/parkinsons.data")

X.drop(['name'], axis=1, inplace=True) # drop name column

y = X.status.copy() # copy “y” values out from status

X.drop(['status'], axis=1, inplace=True) # drop status column

# Perform a train/test split. 30% test group size, with a random_state equal to 7.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    random_state=7)


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Read the data
Can you write Python code for it?

#
# Load up the wheat dataset into dataframe 'X'
#
df = pd.read_csv("../Datasets/wheat.data", index_col='id')


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Same for the dimensionality reduction: feel free to experiment with PCA or Isomap
Can you write Python code for it?

from sklearn.decomposition import PCA
from sklearn import manifold

usePCA = False # change this to use PCA as dimensionality reducer

if usePCA:
    reducer = PCA(n_components=7).fit(X_train)
else:
    reducer = manifold.Isomap(n_neighbors=3, n_components=6).fit(X_train)

X_train = reducer.transform(X_train)
X_test  = reducer.transform(X_test)


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Split into training and testing data sets
Can you write Python code for it?

#
# Split data into test / train sets
#

X_train, X_test, y_train, y_test = train_test_split(df, labels, test_size=0.3,
                                                    random_state=7)


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Train the Knn classifier
Can you write Python code for it?

#
# Create an KNeighbors classifier
#
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5)


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Utility function: benchmark times
Can you write Python code for it?

import time

def benchmark(model, X_train, X_test, y_train, y_test, wintitle='Figure 1'):
  print ('\n\n' + wintitle + ' Results')

  # the only purpose of doing many iterations was to get a more accurate
  # count of the time it took for each classifier
  s = time.time()
  for i in range(iterations):
    #
    # train the classifier on the training data / labels:
    #
    model.fit(X_train, y_train)

  #print ("{0} Iterations Training Time: ".format(iterations), time.time() - s)
  print(f"{iterations} Iterations Training Time: {time.time() - s:.3f}")

  scoreBch = 0

  s = time.time()
  for i in range(iterations):
    #
    # score the classifier on the testing data / labels:
    #
    scoreBch = model.score(X_test, y_test)

  #print ("{0} Iterations Scoring Time: ".format(iterations), time.time() - s)
  print(f"{iterations} Iterations Scoring Time: {time.time() - s:.3f}")

  print ("High-Dimensionality Score: ", round((scoreBch*100), 3))


I am working on the file "02-Classification/svm.ipynb".
The first task is:
Utility function: draw the plots
This is a convenience function to break any higher-dimensional space down and view cross sections of it.
Can you write Python code for it?

import matplotlib as mpl
import matplotlib.pyplot as plt

def drawPlots(model, X_train, X_test, y_train, y_test, wintitle='Figure 1'):

  # If this line throws an error, use plt.style.use('ggplot') instead
  mpl.style.use('ggplot') # Look Pretty

  padding = 3
  resolution = 0.5
  max_2d_score = 0
  score = 0

  y_colors = ['#ff0000', '#00ff00', '#0000ff']
  my_cmap = mpl.colors.ListedColormap(['#ffaaaa', '#aaffaa', '#aaaaff'])
  colors = [y_colors[i] for i in y_train]
  num_columns = len(X_train.columns)

  fig = plt.figure()
  fig.canvas.set_window_title(wintitle)

  cnt = 0
  for col in range(num_columns):
    for row in range(num_columns):
      # Easy out
      if FAST_DRAW and col > row:
        cnt += 1
        continue

      ax = plt.subplot(num_columns, num_columns, cnt + 1)
      plt.xticks(())
      plt.yticks(())

      # Intersection:
      if col == row:
        plt.text(0.5, 0.5, X_train.columns[row], verticalalignment='center', horizontalalignment='center', fontsize=12)
        cnt += 1
        continue

      # Only select two features to display, then train the model
      X_train_bag = X_train.iloc[:, [row,col]]
      X_test_bag = X_test.iloc[:, [row,col]]
      model.fit(X_train_bag, y_train)

      # Create a mesh to plot in
      x_min, x_max = X_train_bag.iloc[:, 0].min() - padding, X_train_bag.iloc[:, 0].max() + padding
      y_min, y_max = X_train_bag.iloc[:, 1].min() - padding, X_train_bag.iloc[:, 1].max() + padding
      xx, yy = np.meshgrid(np.arange(x_min, x_max, resolution),
                           np.arange(y_min, y_max, resolution))

      # Plot Boundaries
      plt.xlim(xx.min(), xx.max())
      plt.ylim(yy.min(), yy.max())

      # Prepare the contour
      Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
      Z = Z.reshape(xx.shape)
      plt.contourf(xx, yy, Z, cmap=my_cmap, alpha=0.8)
      plt.scatter(X_train_bag.iloc[:, 0], X_train_bag.iloc[:, 1], c=colors, alpha=0.5)

      score = round(model.score(X_test_bag, y_test) * 100, 3)
      #plt.text(0.5, 0,...


I am working on the file "02-Classification/svm.ipynb".
The first task is:
visual confirmation of accuracy
Here we can print more digits with indication of what was the predicted label (in red if it was wrong):
Can you write Python code for it?

# Visual Confirmation of accuracy

fig = plt.figure()

  # Make some guesses
y_guess = svc.predict(X_test)

num_rows = 10
num_cols = 5

index = 0
for col in range(num_cols):
    for row in range(num_rows):
        plt.subplot(num_cols, num_rows, index + 1)

      # 8x8 is the size of the image, 64 pixels
        plt.imshow(X_test.iloc[index,:].values.reshape(8,8), cmap=plt.cm.gray_r, interpolation='nearest')

      # Green = Guessed right
      # Red = Fail!
        fontcolor = 'g' if y_test[index] == y_guess[index] else 'r'
        plt.title('Label: %i' % y_guess[index], fontsize=7, color=fontcolor)
        plt.axis('off')
        index += 1
fig.set_tight_layout(True)
plt.show()


I am working on the file "02-Classification/trees.ipynb".
The first task is:
Encode the dataset using dummies
Can you write Python code for it?

#
# Encode labels poisonous / edible
y = y.map({'p':0, 'e':1})

# Encode the rest
#
X = pd.get_dummies(X)


I am working on the file "02-Classification/trees.ipynb".
The first task is:
Pre-process the data
remove missing data
Can you write Python code for it?

#
# drop any row with a nan
#
X.dropna(axis=0, inplace=True)
print (X.shape)


I am working on the file "02-Classification/trees.ipynb".
The first task is:
Pre-processing the data
What we want to do is to predict the wheat type based on the wheat seed characteristics.
To simplify the boundary plot, we consider only two of the characteristics.
Can you write Python code for it?

# we keep groove and perimeter, plus of course the target: wheat type
wheatSimpler = df.drop(columns = ['compactness', 'width', 'length', 'area', 'asymmetry'])

wheatSimpler.info()

# An easy way to show which rows have nans in them

print (wheatSimpler[pd.isnull(wheatSimpler).any(axis=1)])

#
# Only a few rows. Go ahead and drop any row with a nan
#
wheatSimpler.dropna(axis=0, inplace=True)

#
# INFO: In the future, you might try setting the nan values to the
# mean value of that column, groove; the mean should only be calculated for
# the specific class rather than across all classes, in this case for canadian type

#
# Copy the labels out of the dataframe into variable 'labels' , then remove
# them from the dataframe. Encode the labels:
# canadian:0, kama:1, and rosa:2
#
labels = wheatSimpler.wheat_type.copy() # copy “y” values out
wheatSimpler.drop(['wheat_type'], axis=1, inplace=True) # drop output column

labels = labels.map({'canadian':0, 'kama':1, 'rosa':2}) # encoding


I am working on the file "02-Classification/trees.ipynb".
The first task is:
Train the model
Can you write Python code for it?

#
# Create a DT classifier. No need to set any parameters, let;s get the ful depth
#

model = tree.DecisionTreeClassifier()

#
# train the classifier on the training data / labels:
#
model.fit(X_train, y_train)

tree.plot_tree(model, feature_names=X_train.columns, filled=True)


I am working on the file "02-Classification/trees.ipynb".
The first task is:
Train the tree model
Can you write Python code for it?

#
# We use the SkLearn module tree
#
from sklearn import tree

#
# note that I force the tree depth to a maximum of two (for simplification)
#

model = tree.DecisionTreeClassifier(max_depth=2, random_state=2)
model.fit(X_train, y_train)


I am working on the file "02-Classification/trees.ipynb".
The first task is:
You can see that the predictor space has been segmented in a number of simple regions, four to be precise and a wheat type is predicted according to its position in one of the regions.
Another example: poisonous mushrooms
Use decision trees to peruse The Mushroom Data Set, drawn from the Audubon Society Field Guide to North American Mushrooms (1981). The data set details mushrooms described in terms of many physical characteristics, such as cap size  and stalk length, along with a classification of poisonous or edible.
You can find the dataset in the above UCI link or in the Dataset folder here in github.
As a standard disclaimer, if you eat a random mushroom you find, you are doing so at your own risk.
Read the data
Can you write Python code for it?

#
# Load up the mushroom dataset into dataframe 'X'
# Header information is on the dataset's website at the UCI ML Repo
#
colNames=['label', 'cap-shape','cap-surface','cap-color','bruises','odor',
          'gill-attachment','gill-spacing','gill-size','gill-color','stalk-shape',
          'stalk-root','stalk-surface-above-ring','stalk-surface-below-ring',
          'stalk-color-above-ring','stalk-color-below-ring','veil-type',
          'veil-color','ring-number','ring-type','spore-print-color','population',
          'habitat']
X = pd.read_csv("../datasets/agaricus-lepiota.data", header=None, na_values='?',
                names=colNames)

X.head()

X.describe()


I am working on the file "02-Classification/trees.ipynb".
The first task is:
separate label
Can you write Python code for it?

#
# : Copy the labels out of the dset into variable 'y' then Remove
# them from X.

y = X[X.columns[0]].copy()
X.drop(X.columns[0], axis=1,inplace=True)


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
2 means very positive, 0 is neutral and -2 is very negative
Can you write Python code for it?


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
Binary Classification
Now, this is a more generic case, where we have 5 classes as target.
The case you see more often is the binary one, with only two classes, which has some special characteristics and metrics.
Let's convert our target into a binary one:  a tweet can be either negative or not negative (i.e., positive or neutral).
First of all, we need to transform our original dataset to reduce the sentiment classes to only two classes.
Can you write Python code for it?

X.loc[X.Avg < 0] = -1 # negative sentiment
X.loc[X.Avg >= 0] = 1 # NON-negative sentiment


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
In a two-class problem, we are often looking to discriminate between observations with a specific outcome, from normal observations. Such as a disease state or no disease state or spam versus no-spam.
One being the positive event and the other the no-event, the negative event.
In our case, let's say the negative event is the negative tweet and the positive event is the NON-negative tweet.
These are basic terms used in binary classification:
“true positive” for correctly predicted event values (in our scenario the non-negative tweets: positive or neutral).
“true negative” for correctly predicted no-event values (in our scenario the negative tweets).
“false positive” for incorrectly predicted event values. In Hypothesis Testing it is also known as Type 1 error or the incorrect rejection of Null Hypothesis.
“false negative” for incorrectly predicted no-event values. It is also known as Type 2 error, which leads to the failure in rejection of Null Hypothesis.
Can you write Python code for it?

tn, fp, fn, tp = matBinary.ravel()

print("True Negatives: ",tn)
print("False Positives: ",fp)
print("False Negatives: ",fn)
print("True Positives: ",tp)


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
It's more clear if we visualise it as a heat map:
Can you write Python code for it?

import matplotlib.pyplot as plt

labels = ['strongly neg.', 'negative', 'neutral', 'positive', 'strongly pos.']
fig = plt.figure()
ax = fig.add_subplot(111)
cm = ax.matshow(mat)
# plot the title, use y to leave some space before the labels
plt.title("Confusion matrix - Tweets arranged by sentiment", y=1.2)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.setp(ax.get_xticklabels(), rotation=-30, ha="right",
             rotation_mode="anchor")

plt.xlabel("Predicted")
plt.ylabel("Actual")
# Loop over data dimensions and create text annotations.
for i in range(len(mat)):
    for j in range(len(mat)):
        text = ax.text(j, i, mat[i, j],
                       ha="center", va="center", color="w")
    # Create colorbar
fig.colorbar(cm);


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
Now we remove the stopwords. First we define which are the common words (stopwords) to be removed:
Can you write Python code for it?

import os
def readStopwords():
    '''
    returns stopwords as strings

    Assume that a file called "stopwords.txt"
    exists in the folder
    '''
    filename = "stopwords.txt"
    path = os.path.join("", filename)
    file = open(path, 'r')
    return file.read().splitlines()  # splitlines is used to remove newlines

stopWords = set(readStopwords())

"the" in stopWords  # quick test

stopWords.add("apple")
stopWords.add("appl")
stopWords.add("iphone")
stopWords.add("ipad")
stopWords.add("ipod")
stopWords.add("itunes")
stopWords.add("ios")
stopWords.add("http")

print ("apple" in stopWords)
print ("google" in stopWords)


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
Now we use the voctoriser to transform the corpus into a sparse matrix where each tweet has 1 if the feature is present in it or 0 if not.
Can you write Python code for it?

bagOfWords = cv.transform(corpus)

bagOfWords

sum_words = bagOfWords.toarray().sum(axis=0)

words_freq = [(word, sum_words[idx]) for word, idx in cv.vocabulary_.items()]

words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)

words_freq[:10]


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
ROC (Receiver Operating Characteristic) curve
Until now, we have seen classification problems where we predict the target class directly.
Sometimes it can be more insightful or flexible to predict the probabilities for each class instead. From one side you will get an idea of how confident is the classifier for each class, on the other side you can use them to calibrate the threshold for how to interpret the predicted probabilities.
For example, in a binary classifier the default is to use a threshold of 0.5, meaning that a probability less than 0.5 is a negative outcome and a probability equal or over 0.5 is a positive outcome.
But this threshold can be adjusted to tune the behavior of the model for the specific problem, e.g. to reduce more of one or another type of error, as we have seen above.
Think about a classifier that predict if an event is a nuclear attack or not. Clearly you want to have as less as possible false alarms!
A diagnostic tools helpin in choosing the right threshold is the ROC curve.
This is the plot of the ‘True Positive Rate’ (Sensitivity) on the y-axis against the ‘False Positive Rate’ (1 minus Specificity) on the x-axis, at different classification thresholds between 0 and 1.
It captures all the thresholds simultaneously and the area under the ROC curve measures how well a parameter can distinguish between two groups.
Threshold =0 is at the axis origin (0,0) while the threshold = 1 is at the top right end of the curve.
- high threshold: means high specificity and low sensitivity
- Low threshold: means low specificity and high sensitivity
Put another way, it plots the false alarm rate versus the hit rate.
Let's see an example using our binary classification above.
First, you need probabilities to create the ROC curve.
Can you write Python code for it?

probs = classifier.predict_proba(X_test) # get the probabilities

preds = probs[:,1]  ## keep probabilities for the positive outcome only
fpr, tpr, threshold = metrics.roc_curve(y_test, preds)  # calculate roc
roc_auc = metrics.auc(fpr, tpr)  # calculate AUC

plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.plot([0, 1], [0, 1],'r--')  # plot random guessing

plt.legend(loc = 'lower right')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
Sensitivity is a real number between 0 and 1. A sensitivity of 1 means that ALL the Negative cases have been correctly classified.
Specificity
Can you write Python code for it?

#Specificity
Specificity = tn/(tn+fp)
print("Specificity {:0.2f}".format(Specificity))


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
Similarly, you can calculate the Negative Predictive Power:
Can you write Python code for it?

# Negative Predictive Value
print("Negative predictive Power: {:0.2f}".format(tn / (tn+fn)))


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
So our Naive Bayesian model does better than the simple baseline.
Predict the sentiment of a new tweet
The classifier can be applied to new tweets, of course, to predict their sentiment:
Can you write Python code for it?

# for simplicity, it re-uses the vectorizer and the classifier without passing them
# as arguments. Industrialising it would mean to create a pipeline with
# vectoriser > classifier > label string
def predictSentiment(t):
    bow = cv.transform([t])
    prediction = classifier.predict(bow)
    if prediction == 0:
        return "Neutral"
    elif prediction > 0:
        return "Positive"
    else:
        return "Negative"

predictSentiment("I don't know what to think about apple!")


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
The F1 score is the harmonic mean of the Precision & Sensitivity, and is used to indicate a balance between them. It ranges from 0 to 1; F1 Score reaches its best value at 1 (perfect precision & sensitivity) and worst at 0.
Can you write Python code for it?

# F1 Score
f1 = (2 * Precision * Sensitivity) / (Precision + Sensitivity)
print("F1 Score {:0.2f}".format(f1))


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
The ROC curve is a useful tool for a few reasons:

The curves of different models can be compared directly in general or for different thresholds.
The area under the curve (AUC) can be used as a summary of the model skill.

A random guessing classifier (the red line above) has an Area Under the Curve (often referred as AUC) of 0.5, while AUC for a perfect classifier is equal to 1.
In general AUC of above 0.8 is considered "good".
Looking at the ROC curve you can choose a threshold that gives a desirable balance between the:
- cost of failing to detect positive
- cost of raising false alarms
Precision and F1 score metrics
Called also Positive Predictive Power, the Precision measures somehow how "exact" it is, i.e.  the ability of the model to return only relevant instances. If your use case/problem statement involves minimising the False Positives then Precision is something you need:
Can you write Python code for it?

# Precision
Precision = tp/(tp+fp)
print("Precision or Positive Predictive Power: {:0.2f}".format(Precision))


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
The classifier was correct 64% of times (not only if a tweet was negative but also if it was strongly negative or moderately negative).
A very useful metric is the confusion matrix that displays the predictions and the actual values in a matrix:
Can you write Python code for it?

mat = metrics.confusion_matrix(y_test, predictions)

mat


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
The numbers in the diagonal are all the times when the predicted sentiment for a tweet was the same as the actual sentiment.
Now we can define accuracy as the sum of all the values in the diagonal divided by the total of the values.
The best accuracy would be 1.0 when all values are on the diagonal (no errors!), whereas the worst is 0.0 (nothing correct)!
Can you write Python code for it?

correctPredictions = sum(mat[i][i] for i in range(len(mat)))
correctPredictions

print("Accuracy: {:.2}".format(correctPredictions / len(y_test)))


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
This tells us that in our test dataset we have 178 observation with neutral sentiment and 118 with positive or negative tweets.
So the accuracy of a baseline model that always predict non-negative tweets would be:
Can you write Python code for it?

print("Accuracy baseline: {:.2}".format(neutralTweets / len(y_test)))


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
To remove a word from the corpus if that word is contained in our stopwords set, we need first to tokenise the corpus (i.e., split it into words or tokens):
Can you write Python code for it?

# tokenise
corpusTokens = [s.split() for s in corpusNoPunct]

corpusTokens[0:3]

# Stem document
from nltk import PorterStemmer
porter = PorterStemmer()

corpus = []
for tweet in corpusTokens:
    cleanTokens = [token for token in tweet if token not in stopWords] # a list of tokens
    stemmedTokens = [porter.stem(token) for token in cleanTokens]
    cleanTweet = ' '.join(stemmedTokens)

    corpus.append(cleanTweet)

corpus[0:5]


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
Train and test the classifier
Now we train the Naive Bayes classifier
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB

classifier = MultinomialNB()

classifier.fit(X_train, y_train)

predictions = classifier.predict(X_test)

predictions[0:100]


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
We need to re-apply the classifier
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(df, X.Avg, test_size=0.25)

classifier = MultinomialNB()  # 0.77

classifier.fit(X_train, y_train)

predictionsTwo = classifier.predict(X_test)

predictionsTwo[0:100]


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
We put it into a data frame to use it in the classifier
Can you write Python code for it?

df = pd.DataFrame(bagOfWords.toarray())

df.shape

df.info()

df.head(1)


I am working on the file "02-Classification/tweets.ipynb".
The first task is:
What do we use the ROC for?
Can you write Python code for it?

classifierTuned = MultinomialNB(class_prior=[.4, 0.6]) # try to max specificity

classifierTuned.fit(X_train, y_train)
predictionsTuned = classifierTuned.predict(X_test)

matTuned = metrics.confusion_matrix(y_test, predictionsTuned)
matTuned

tn, fp, fn, tp = matTuned.ravel()

Accuracy = (tn+tp)/(tp+tn+fp+fn)
print("Accuracy: {:.2f}".format(Accuracy)) # it was 0.79

Sensitivity = tp/(tp+fn)
print("Sensitivity {:0.2f}".format(Sensitivity)) #it was 0.9

Specificity = tn/(tn+fp)
print("Specificity {:0.2f}".format(Specificity)) # it was 0.53


I am working on the file "02-Data-and-Twitter.ipynb".
The first task is:

APIs
What is an API?

API stands for Application Programming Interface

a set of rules and procedures that facilitate interactions between computers and their applications

Web APIs

allows users to query a remote database over the internet

take on a variety of formats

majority adhere to a particular style known as Reperesentational State Transfer or REST

"RESTful" APIs are conveinent because we can use them to query databases using URLs

RESTful Web APIs are All Around You...
Consider a simple Google search.
Go ahead and search something.
Ever wonder what all that extra stuff in the address bar was all about?

RESTful Web APIs are All Around You...
It looks like Google makes its query by taking the search terms, separating each of them with a "+", and appending them to the link:
https://www.google.com/#q=
So that we have
https://www.google.com/#q=search1+search2
So can change our Google search by adding some terms to the URL.

Some Basic Terminology: URL

Uniform Resource Location
a string of characters that, when interpreted via the Hypertext Transfer Protocol (HTTP)
points to a data resource, notably files written in Hypertext Markup Language (HTML) or a subset of a database.

Some Basic Terminology: HTTP Methods / Verbs

GET: requests a representation of a data resource corresponding to a particular URL.  The process of executing the GET method is often referred to as a "GET request" and is the main method used for querying RESTful databases.

HEAD, POST, PUT, DELETE: other common methods, though mostly never used for database querying.

How Do GET Requests Work?  A Web Browsing Example

Surfing the Web = Making a bunch of GET Requests

For instance, I open my web browser and type in http://www.wikipedia.org.  Once I hit return, I'd see a webpage.

Several different processes occured, however, between me hitting "return" and the page finally being rendered.

Step 1: The GET Request

web browser took the entered character string
used the command-line tool "Curl"...
Can you write Python code for it?

import requests

response = requests.get('https://data.cityofchicago.org/api/views/4ijn-s7e5/rows.json')

import json

data = json.loads(response.text)

data.keys()

data['meta']['view']


I am working on the file "02-Data-and-Twitter.ipynb".
The first task is:
Now we can use the Mapbox API:
Can you write Python code for it?

import plotly.plotly as py
from plotly.graph_objs import *

mapbox_access_token = ''

map_data = Data([
    Scattermapbox(
        lat=lats,
        lon=longs,
        mode='markers',
        marker=Marker(
            size=14
        ),
        text=tweet_texts,
    )
])

layout = Layout(
    autosize=True,
    hovermode='closest',
    mapbox=dict(
        accesstoken=mapbox_access_token,
        bearing=0,
        center=dict(
            lat=37.85,
            lon=-122.28
        ),
        pitch=0,
        zoom=7
    ),
)

fig = dict(data=map_data, layout=layout)
py.iplot(fig, filename='Berkeley Tweet Map', validate=False)


I am working on the file "02-Data-and-Twitter.ipynb".
The first task is:
OR
Can you write Python code for it?

# get tweets from specific handle
my_tweets_result = api.user_timeline(screen_name='UCBerkeley', count=10)

my_tweets = []

for t in my_tweets_result:
    my_tweets.append(t)


I am working on the file "02-Data-and-Twitter.ipynb".
The first task is:
Raw Word Counts
Can you write Python code for it?

from collections import Counter

tweet_words = []

for t in results_tweets:
    tweet_words.extend(t.text.split())

Counter(tweet_words).most_common()


I am working on the file "02-Data-and-Twitter.ipynb".
The first task is:
Reading a .csv is just as easy:
Can you write Python code for it?

with open("my_data.csv", "r") as f:
    csv_data = list(csv.reader(f))

print(csv_data)


I am working on the file "02-Data-and-Twitter.ipynb".
The first task is:
Save to CSV:
First create the rows:
Can you write Python code for it?

tweet_rows = []

for t in data:
    tweet_data = {}
    tweet_data['created'] = t.created_at
    tweet_data['user'] = t.user.screen_name
    tweet_data['text'] = t.text
    tweet_data['retweet_count'] = t.retweet_count
    tweet_data['hashtags'] = '; '.join([d['text'] for d in t.entities['hashtags']])
    tweet_data['user_mentions'] = '; '.join([d['screen_name'] for d in t.entities['user_mentions']])
    tweet_data['urls'] = '; '.join([d['url'] for d in t.entities['urls']])

    if t.geo:
        tweet_data['coordinates'] = t.geo['coordinates']
    else:
        tweet_data['coordinates'] = 'NA'

    tweet_rows.append(tweet_data)


I am working on the file "02-Data-and-Twitter.ipynb".
The first task is:
Sentiment Analysis
Can you write Python code for it?

import nltk

nltk.download('punkt')

from textblob import TextBlob

for t in results_tweets:

    blob = TextBlob(t.text)

    for sentence in blob.sentences:
        pol = sentence.sentiment.polarity
        print(pol, sentence)
    print()


I am working on the file "02-Data-and-Twitter.ipynb".
The first task is:
What could be useful here?
Can you write Python code for it?

print(results_tweets[0].created_at)
print()
print(results_tweets[0].text)
print()
print(results_tweets[0].retweet_count)
print()
print(results_tweets[0].entities['user_mentions'])
print()
print(results_tweets[0].entities['hashtags'])
print()
print(results_tweets[0].entities['urls'])
print()
print(results_tweets[0].user.profile_image_url_https)
print()
print(results_tweets[0].geo)


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Funkcija <i>print</i> može prikazati na ekranu, ili "štampati":
* tekst (koji joj se prosleđuje pod navodnicima) kao što smo videli u uvodnoj lekciji
* rezultate logičkih operacija - "True" i "False"
* brojeve
* ...
Primer sa brojevima:
Can you write Python code for it?

# Print a number
print(2)
# Print a number which is a result of a mathematical operation
print(2 + 3 - 5 + 8)


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Isprobajte operacije poređenja. Njih možete isprobati na zanimljiv način tako što ćete napraviti sledeći program (iskucate kod u fajl, sačuvate fajl i iz Vindovs konzole kažete Pajtonu da pokrene taj fajl - <i>python ime_fajla.py</i>.
Can you write Python code for it?

### This is a cool program for demonstrating Python's logical operations ###

# Demonstrate the operator "greater than"
print("Is it true that 5 is greater than 4?")
print(5 > 4)

# Demonstrate the operator "less than"
print("Is it true that 100 is less than 50?")
print(100 < 50)

# Demonstrate the operator "greater than or equal to"
print("Is it true that 3 is greater that or equal to 5?")
print(3 >= 5)

# Demonstrate the operator "less than or equal to"
print("Is it true that 6 is less than or equal to 6?")
print(6 >= 6)


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Isprobajte operaciju <i>moduo</i>, odnosno "ostatak pri deljenju sa". Na primer:
Can you write Python code for it?


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Isprobajte operaciju celobrojnog deljenja ili "deljenja sa zaokruživanjem na dole":
Can you write Python code for it?


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Označavanje sintakse bojama (syntax highlighting) u editoru Notepad++
Pokrenite <i>Notepad++</i>. Otkucajte u njemu kod programa "Zdravo, svete", kao u lekciji Uvod.
Can you write Python code for it?


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Pajton ne prepoznaje reči poput "ovo", "je", "zanimljiv" itd. Međutim, prepoznaje tarabu (#) i ona u Pajtonu označava komentare. U programskim jezicima, komentari su nešto što se ignoriše, kao da ne postoji. Pošto za samo izvršenje programa nemaju nikakvo značenje (jer se ignorišu), koriste se uglavnom za dve stvari:
* za opisivanje programa
* za "izbacivanje" određenih delova koda
Opisivanje programa je bitno, jer kod postaje čitljiviji za čoveka. Kada neko drugi pogleda vaš kod, ili treba da nastavi da ga razvija, lakše će mu biti ukoliko on sadrži komentare koji objašnjavaju zašto određeni delovi izgledaju baš tako kako izgledaju. I vama samima će komentari u sopstvenom kodu značiti mnogo kada svoj kod gledate posle dužeg vremena, jer ćete zaboraviti "šta je pisac hteo da kaže".

Prilikom otkrivanja grešaka u kodu (debagiranja, debagovanja), često ćete želeti da uklonite neki njegov deo. Umesto da te delove brišete, čuvate u nekom drugom fajlu, pa vraćate u svoj program, možete ih jednostavno pretvoriti u komentare (dodavanjem tarabe na početku linije). Većina editora ima prečicu na tastaturi za ove potrebe, koja je naročito korisna kada veliki broj susednih linija pretvarate u komentare. U editoru Notepad++, prečica CTRL + K pretvara sve označene linije koda u komentar, a prečica CTRL + SHIFT + K ih vraća.

Primeri zanimljivih i duhovitih komentara: http://stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered
Brojevi i račun
Pokrenite Pajton konzolu. U njoj isprobajte matematičke operacije sabiranja, oduzimanja, množenja i deljenja. Na primer:
Can you write Python code for it?


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Ponašanje ovog operatora je zanimljivo kada je s leve strane negativan broj. Naime, doći će do deljenja sa zaokruživanjem na dole. Na primer, u računanju -37 % 10, prvo se računa "količnik", koji je u ovom slučaju -4 (zbog zaokruživanja na dole). Zatim se ostatak računa kao razlika deljenika (-37) i proizvoda količnika (-4) i delioca (10) -37 - (-4 * 10).
Can you write Python code for it?


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Primetićete da Notepad++ označava različitim bojama funkciju print i tekst pod navodnicima, odnosno da "zna" da se radi o Pajton kodu. Notepad++ može da zaključi kako da boji tekst (syntax highlight) na osnovu ekstenzije fajla koji prikazuje, bez eksplicitnog naznačavanja programskog jezika koje nam je bilo neophodno za prethodni dokument.
Izvršavanje komandi koje se nalaze u fajlu
Otvorite konzolu i u njoj pređite u direktorijum u kome je sačuvan malopređašnji fajl:

cd "c:\programs".

Zatim dajte komandu Pajtonu da izvrši kod koji se nalazi u dokumentu koji smo maločas napravili. U vindovs konzolo otkucajte:

python program1.py

Na ekranu ćete videti sledeći izlaz (ovde se vidi i kod i izlaz, vi ćete videti samo izlaz):
Can you write Python code for it?

print("This is the first line.")
print("This is the second line.")
print("This is the third line.")


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
Redosled izvršavanja operacija i zagrade važe kao i u matematici. U tehničkoj dokumentaciji jezika Pajton, ovo je objašnjeno do tančina. Na primer:
Can you write Python code for it?


I am working on the file "02-Fajlovi-komentari-racun.ipynb".
The first task is:
U većini vežbi, sav Pajton kod ćemo kucati u editoru Notepad++, čuvati u fajlove s ekstenzijom "py", a izvršavati iz konzole koristeći komandu:

python ime_programa.py
Komentari u Pajtonu
Napravite fajl željenog naziva, u njega dodajte sledeći kod, i izvršite komandom koju smo koristili u ovoj lekciji (python ime_programa.py).
Can you write Python code for it?

# Ovo je moj program
print("Moj program")


I am working on the file "02-Generators-As-Iterators.ipynb".
The first task is:
Generators behave like iterators
Generators are used to produce values. We most often use for loops to consume those values. Just like you can iterate through a list or another object in python, you can iterate through a generator.
Can you write Python code for it?

countdown_gen = countdown(5)
for item in countdown_gen:
    print(item)


I am working on the file "02-Generators-As-Iterators.ipynb".
The first task is:
They implement the same interface as iterators
You can call iter on the object returned by calling a generator function, and you can call next on the object returned by iter. When the iteration is complete, it will raise StopIteration
They do so by suspending execution
Between each call to next, the generator still exists, and is in a state of suspended execution. That means that all sorts of other computation could occur between the different calls to next. We'll talk more about the implications of this later.
Can you write Python code for it?

count_3 = iter(countdown(3))
print(next(count_3))
print(next(count_3))
print(next(count_3))
print(next(count_3))

print(next(count_3))


I am working on the file "02-InitCNN/dlnd_image_classification.ipynb".
The first task is:
Build the network
For the neural network, you'll build each layer into a function.  Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function.  This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.

Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section.  TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.
However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the conv2d class, tf.layers.conv2d, you would want to use the TF Neural Network version of conv2d, tf.nn.conv2d.

Let's begin!
Input
The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions
* Implement neural_net_image_input
 * Return a TF Placeholder
 * Set the shape using image_shape with batch size set to None.
 * Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_label_input
 * Return a TF Placeholder
 * Set the shape using n_classes with batch size set to None.
 * Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_keep_prob_input
 * Return a TF Placeholder for dropout keep probability.
 * Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in...
Can you write Python code for it?

import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a batch of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    return tf.placeholder(tf.float32, shape=[None, *image_shape], name = 'x')

def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    return tf.placeholder(tf.float32, shape=[None, n_classes], name = 'y')

def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    return tf.placeholder(tf.float32, name='keep_prob')

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


I am working on the file "02-InitCNN/dlnd_image_classification.ipynb".
The first task is:
Fully Train the Model
Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'

print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            print_stats(sess, batch_features, batch_labels, cost, accuracy)

    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)


I am working on the file "02-InitCNN/dlnd_image_classification.ipynb".
The first task is:
Implement Preprocess Functions
Normalize
In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive.  The return object should be the same shape as x.
Can you write Python code for it?

def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    # TODO: Implement Function
    return np.array(x/255)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)


I am working on the file "02-InitCNN/dlnd_image_classification.ipynb".
The first task is:
One-hot encode
Just like the previous code cell, you'll be implementing a function for preprocessing.  This time, you'll implement the one_hot_encode function. The input, x, are a list of labels.  Implement the function to return the list of labels as One-Hot encoded Numpy array.  The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode.  Make sure to save the map of encodings outside the function.
Hint: Don't reinvent the wheel.
Can you write Python code for it?

from sklearn import preprocessing

encoder = preprocessing.LabelBinarizer()

def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    # TODO: Implement Function
    encoder.fit(x)
    encoder.classes_ = np.array(list(range(10)))
    return encoder.transform(x)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)


I am working on the file "02-InitCNN/dlnd_image_classification.ipynb".
The first task is:
Randomize Data
As you saw from exploring the data above, the order of the samples are randomized.  It doesn't hurt to randomize it again, but you don't need to for this dataset.
Preprocess all the data and save it
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)


I am working on the file "02-InitCNN/dlnd_image_classification.ipynb".
The first task is:
Show Stats
Implement the function print_stats to print loss and validation accuracy.  Use the global variables valid_features and valid_labels to calculate validation accuracy.  Use a keep probability of 1.0 to calculate the loss and validation accuracy.
Can you write Python code for it?

def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : x: feature_batch: Batch of Numpy image data
    : y: label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    : keep_prob: 1.0 # added by me
    """
    acc = session.run(accuracy, feed_dict={x: valid_features, y: valid_labels, keep_prob: 1.0})
    loss = session.run(cost, feed_dict={x: feature_batch, y: label_batch, keep_prob: 1.0})
    print('Acc: {} Loss: {}'.format(acc, loss))


I am working on the file "02-InitCNN/dlnd_image_classification.ipynb".
The first task is:
Train on a Single CIFAR-10 Batch
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy.  Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)


I am working on the file "02-InitCNN/dlnd_image_classification.ipynb".
The first task is:
Train the Neural Network
Single Optimization
Implement the function train_neural_network to do a single optimization.  The optimization should use optimizer to optimize in session with a feed_dict of the following:
* x for image input
* y for labels
* keep_prob for keep probability for dropout
This function will be called for each batch, so tf.global_variables_initializer() has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
Can you write Python code for it?

def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    session.run(optimizer, feed_dict={x: feature_batch, y: label_batch, keep_prob: keep_probability})
#      no return here, this is an execution function

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)


I am working on the file "02-Intro to Python/Intro to Python.ipynb".
The first task is:
2. A Few Tricks Up Our Sleeve
String Methods
The creators of Python recognize that human language has many important yet idiosyncratic features, so they have tried to make it easy for us to identify and manipulate them. For example, in the demonstration at the very beginning of the workshop, we referred to the idea of the suffix: the final letters of a word tell us something about its grammatical role and potentially the author's argument.
We can analyze or manipulate certain features of a string using its <i>methods</i>. These are basically internal functions that every string automatically possesses. Note that even though the method may transform the string at hand, they don't change it permanently!
Can you write Python code for it?

# Let's assign a variable to perform methods upon

greeting = "Hello, World!"

# We saw the 'endswith' method at the very beginning
# Note the type of output that gets printed

greeting.startswith('H'), greeting.endswith('d')

# We can check whether the string is a letter or a number

this_string = 'f'

this_string.isalpha()

# When there are multiple characters, it checks whether *all*
# of the characters belong to that category

greeting.isalpha(), greeting.isdigit()

# Similarly, we can check whether the string is lower or upper case

greeting.islower(), greeting.isupper(), greeting.istitle()

# Sometimes we want not just to check, but to change the string

greeting.lower(), greeting.upper()

# The case of the string hasn't changed!

greeting

# But if we want to permanently make it lower case we re-assign it

greeting = greeting.lower()

greeting

# Oh hey. And strings are kind of like lists, so we can slice them similarly

greeting[:3]

# Strings may be like lists of characters, but as humans we often treat them as
# lists of words. We can tell the computer to divide a string into word-like units.

greeting.split()

## EX. Return the second through eighth characters in 'greeting'

## EX. Split the string below into a list of words and assign this to a new variable
## Note: A slash at the end of a line allows a string to continue onto the next one unbroken

new_string = "It, is a truth universally acknowledged, that a single \
man in possession of a good fortune must be in want of a wife."


I am working on the file "02-Intro to Python/Intro to Python.ipynb".
The first task is:
Exploratory NLP Tasks
Now that we have some of Python's basics in our toolkit, we can immediately perform the kinds of tasks that are the digital humanist's bread and butter. When we first meet a text in the wild, we often wish to find out a little about it before digging in deeply, so we start with simple questions like "How many words are in this text?" or "How long is the average word?"
Can you write Python code for it?

##     Run the cell below to read in the text of "Pride and Prejudice."

##     How many words are in the novel?
##     How many words in the novel appear in title case?
##     Approximately how long is the average word in the novel?

austen_string = open('Austen - Pride and Prejudice.txt').read()


I am working on the file "02-Intro to Python/Intro to Python.ipynb".
The first task is:
Lists
The numbers and strings we have just looked at are the two basic data types that we will focus our attention on in this workshop. (In a few days, we will look at a third data type, boolean, which consists of  True/False values.) When we are working with just a few numbers or strings, it is easy to keep track of them, but as we collect more we will want a system to organize them.
One such organizational system is a list. This contains values (regardless of type) in order, and we can perform operations on it very similarly to the way we did with numbers.
Can you write Python code for it?

# A list in which each element is a string

['Call', 'me', 'Ishmael']

# Let's assign a couple lists to variables

list1 = ['Call', 'me', 'Ishmael']
list2 = ['In', 'the', 'beginning']

## Q. Predict what will happen when we perform the following operations

print(list1+list2)
print(list1*5)

# As with a string, we can find out the length of a list

len(list1)

# Sometimes we just want a single value from the list at a time

list1[0]

list1[1]

list1[2]

# Or maybe we want the first few

list1[0:2]

list1[:2]

# Of course, lists can contain numbers or even a mix of numbers and strings

list3 = [7,8,9]
list4 = [7,'ate',9]

# And python is smart with numbers, so we can add them easily!

sum(list3)

## EX. Concatenate 'list1' and 'list2' into a single list.
##     Retrieve the third element from the combined list.
##     Retrieve the fourth through sixth elements from the combined list.


I am working on the file "02-Intro to Python/Intro to Python.ipynb".
The first task is:
Strings
In Python, human language text gets represented as a <i>string</i>. These contain sequential sets of characters and they are offset by quotation marks, either double (") or single (').
We will explore different kinds of operations in Python that are specific to human language objects, but it is useful to start by trying to see them as the computer does, as numerical representations.
Can you write Python code for it?

# The iconic string

print("Hello, World!")

# Assign these strings to variables

a = "Hello"
b = 'World'

# Try out arithmetic operations.
# When we add strings we call it 'concatenation'

print(a+b)
print(a*5)

# Unlike a number that consists of a single value, a string is a
# sequence of characters. We can find out the length of that sequence.

len("Hello, World!")

## EX. How long is the string below?

this_string = "It was the best of times; it was the worst of times."


I am working on the file "02-Intro to Python/Intro to Python.ipynb".
The first task is:
Variable assignment
Assigning variables is something that we do all the time in programming. These aren't quite like the variables from high school algebra, where <i>x</i> represents an unknown to solve for. Instead these are like notes to ourselves that we want to save some value(s) for later use.
Note that the equals sign is directional, like an arrow, telling the computer to give a certain value to a certain label.
Can you write Python code for it?

# 'a' is being given the value 2; 'b' is given 5

a = 2
b = 5

# Let's perform an operation on the variables

a+b

# Variables can have many different kinds of names

this_number = 2
b/this_number


I am working on the file "02-IntroToNLP/00-IntroToNLP.ipynb".
The first task is:

Exericse
Can you write Python code for it?

####Ex: create a new variable that contains the sentence tokens without the stopwords
#Ex: Count the most frequent 10 words on this new variable and print out the results


I am working on the file "02-IntroToNLP/00-IntroToNLP.ipynb".
The first task is:

2. Most Frequent Words
We are often also interested in the most frequent words, which can help us quickly summarize a text. NLTK has a built-in function to do this!
Can you write Python code for it?

#apply the nltk function FreqDist to count the number of times each token occurs.
word_frequency = nltk.FreqDist(sentence_tokens)

#this creates an NLTK object
print(word_frequency)

#print out the 10 most frequent words using the function most_common
print(word_frequency.most_common(10))


I am working on the file "02-IntroToNLP/00-IntroToNLP.ipynb".
The first task is:
Notice each token is either a word or punctuation. Notice also this is not a trivial task, nor does this completely solve the issue.
For example, one of these tokens is ["'s"], which indicates "is" in the contraction "it is". How would we distinguish this from ["'s"] as a possessive? Should we rather tokenize it's to ['it', 'is']? But then we would erase the contraction, which may be important later. There are many questions like this. The problem with natural language is they are ambiguous, and never strictly follow a set of rules.
Assumming we're happy with the tokenization, why is it helpful?
We can now summarize the sentence/text in interesting and potentially helpful ways. For example, we can count the number of tokens in the sentence.
Can you write Python code for it?

#The number of tokens is the length of the list, or the number of elements in the list
print(len(sentence_tokens))


I am working on the file "02-IntroToNLP/00-IntroToNLP.ipynb".
The first task is:
Now we can remove punctuation using list comprehension.
Can you write Python code for it?

sentence_tokens_clean = [word for word in sentence_tokens if word not in punctuations]
print(sentence_tokens_clean)


I am working on the file "02-IntroToNLP/00-IntroToNLP.ipynb".
The first task is:
Question: What do you notice about this output? What datatype is it?
Can you write Python code for it?

##EX: Print the type-token ratio for this sentence


I am working on the file "02-IntroToNLP/00-IntroToNLP.ipynb".
The first task is:
The most frequent words do suggest what the sentence is about, in particular the words "humanistic", "digital", "media", and "traditional".
But there are many frequent words that are not helpful in summarizing the text, for example, "the", "and", "to", and "." So the most frequent words do not necessarily help us understand the content of a text.
How can we use a computer to identify important, interesting, or content words in a text? There are many ways to do this. Today, we'll learn one simple way to identify words that will help us summarize the content of a text. We'll explore a different method later.

3. Pre-Processing: Lower Case, Removing Stop Words
First, scholars typically go through a number of pre-processing steps before getting to the actual analysis. We have already removed punctuation, which is a common pre-processing step. Another common step is converting all words to lower-case, so that the word "Humanities" and "humanities" count as the same word. (For some tasks this is appropriate. Think of reasons why we might NOT want to do this.)
To convert to lower case we use the function lower()
Can you write Python code for it?

sentence_tokens_clean_lc = [word.lower() for word in sentence_tokens_clean]

#see the result
print(sentence_tokens_clean_lc)

#Ex: Calculate the type-token ratio for the sentence after you have changed everything to lowercas


I am working on the file "02-IntroToNLP/00-IntroToNLP.ipynb".
The first task is:
We can also calculate the type-token ratio (TTR). We know what a token is. But many tokens are repeated in a text. For example, in this sentence, the token "the" appears 5 times. "The" is a type. The 5 "the"s in the sentence are tokens. The TTR is simply the number of types divided by the number of tokens. A high TTR indicates a large amount of lexical variation or lexical diversity and a low TTR indicates relatively little lexical variation. The type-token ratio of speech, for example, is less than that of written language. What might we expect of the two novels we have been analyzing?
Can you write Python code for it?


I am working on the file "02-IntroToNLP/00-IntroToNLP_ExerciseSolution.ipynb".
The first task is:
1.  Type-Token Ratio
Divide the length of the set of the text (the unique elements) by the length of the full list
Can you write Python code for it?

print("TTR for Pride and Prejudice")
print(len(set(austen_list_clean))/len(austen_list_clean))
print("TTR for A Garland for Girls")
print(len(set(alcott_list_clean))/len(alcott_list_clean))


I am working on the file "02-IntroToNLP/00-IntroToNLP_ExerciseSolution.ipynb".
The first task is:
2. Frequent Words
Can you write Python code for it?

austen_word_frequency = nltk.FreqDist(austen_list_clean)
alcott_word_frequency = nltk.FreqDist(alcott_list_clean)

print("Frequent words in Pride and Prejudice")
print(austen_word_frequency.most_common(10))
print("Frequent words in A Garland for Girls")
print(alcott_word_frequency.most_common(10))


I am working on the file "02-IntroToNLP/00-IntroToNLP_ExerciseSolution.ipynb".
The first task is:
3. Bonus: Plotting word distribution
Can you write Python code for it?

austen_word_frequency.plot(50, cumulative=True)
austen_word_frequency.plot(50, cumulative=False)


I am working on the file "02-IntroToNLP/00-IntroToNLP_ExerciseSolution.ipynb".
The first task is:
5. Bonus: Concordances (if time)
The NLTK package has many built-in functions for natural language processing. I encourage you to explore the full range of techniques available. I'll go over two more here: concordance() and similar().
The concordance() function lists out every time the specified words appears in the text along with the surrounding context.
Can you write Python code for it?

marx_string = open('../Data/Marx_CommunistManifesto.txt', encoding='utf-8').read()
prince_string = open('../Data/Machiavelli_ThePrince.txt', encoding='utf-8').read()

marx_list = word_tokenize(marx_string)
prince_list = word_tokenize(prince_string)

marx_nltk = nltk.Text(marx_list)
prince_nltk = nltk.Text(prince_list)
print(prince_nltk)
marx_nltk

marx_nltk.concordance('people')
prince_nltk.concordance('people')


I am working on the file "02-IntroToNLP/00-IntroToNLP_ExerciseSolution.ipynb".
The first task is:
The most frequent words are, of course, the stop words. These words do not tell us much about the content of the text.
4. Count word frequencies after removing stop words
Can you write Python code for it?

austen_list_clean_sw = [word for word in austen_list_clean if word not in stopwords.words('english')]
alcott_list_clean_sw = [word for word in alcott_list_clean if word not in stopwords.words('english')]

austen_word_frequency_sw = nltk.FreqDist(austen_list_clean_sw)
alcott_word_frequency_sw = nltk.FreqDist(alcott_list_clean_sw)

print("Frequent words in Pride and Prejudice")
print(austen_word_frequency_sw.most_common(20))
print()
print("Frequent words in A Garland for Girls")
print(alcott_word_frequency_sw.most_common(20))


I am working on the file "02-IntroToNLP/00-IntroToNLP_ExerciseSolution.ipynb".
The first task is:
The text.similar() method takes a word w, finds all contexts w1 w w2, then finds all words w' that appear in the same context, i.e. w1 w' w2
Can you write Python code for it?

print("Marx")
marx_nltk.similar('people')
print()
print("Machiavelli")
prince_nltk.similar('people')


I am working on the file "02-IntroToNLP/01-POS_Tagging.ipynb".
The first task is:
And now verbs.
Can you write Python code for it?

#verbs = [word for (word,pos) in tagged_sentence_tokens if pos == 'VB' or pos=='VBD' or pos=='VBG' or pos=='VBN' or pos=='VBP' or pos=='VBZ']
verbs = [word for (word,pos) in tagged_sentence_tokens if pos in ['VB', 'VBD','VBG','VBN','VBP','VBZ']]

#print all of the verbs
print(verbs)

##Ex: Print the most frequent nouns, adjective, and verbs in the sentence
######What does this tell us?
######Compare this to what we did earlier with removing stop words.

##Ex: Compare the most frequent part-of-speech used in two of the texts in our data folder


I am working on the file "02-IntroToNLP/01-POS_Tagging.ipynb".
The first task is:
Now comes more complicated code. Stay with me. The above output is a list of tuples. A tuple is a sequence of Python objects. In this case, each of these tuples is a sequence of strings. To loop through tuples is intuitively the same as looping through a list, but slightly different syntax.
Note that this is not a list of lists, as we saw in our lesson on Pandas. This is a list of tuples.
Let's pull out the part-of-speech tag from each tuple above and save that to a list. Notice the order stays exactly the same.
Can you write Python code for it?

word_tags = [tag for (word, tag) in tagged_sentence_tokens]
print(word_tags)


I am working on the file "02-IntroToNLP/01-POS_Tagging.ipynb".
The first task is:
Question: What is the difference in syntax for the above code compared to our standard list comprehension code?

Counting words based on their part of speech
We can count the part-of-speech tags in a similar way we counted words, to output the most frequent types of words in our text. We can also count words based on their part of speech.
First, we count the frequency of each part-of-speech tag.
Can you write Python code for it?

tagged_frequency = nltk.FreqDist(word_tags)
tagged_frequency.most_common()


I am working on the file "02-IntroToNLP/01-POS_Tagging.ipynb".
The first task is:
This sentence contains a lot of adjectives. So let's first look at the adjectives. Notice the syntax here.
Can you write Python code for it?

adjectives = [word for (word,pos) in tagged_sentence_tokens if pos == 'JJ' or pos=='JJR' or pos=='JJS']

#print all of the adjectives
print(adjectives)


I am working on the file "02-LeastSquares.ipynb".
The first task is:
Clearly the condition number for the Legendre case is dramatically smaller.
Non-equispaced interpolaton points
We have seen the Runge effect where interpolating polynomials can exhibit large oscillations close to the ends of the domain, with the effect becoming more pronounced as the polynomial degree is increased. A well-known approach to reducing the oscillations is to use evaluation points that clustered towards the ends of the domain. In particular, the roots of orthogonal polynomials can be particularly good evaluation points.
To test, we will interpolate the Runge function at the roots of Legendre polynomials.
Can you write Python code for it?

@interact(N=(0, 30))
def plot(N):
    # Get the roots of the Legendre polynomials
    x_p = np.linspace(-1, 1, N)
    x_p = np.polynomial.legendre.leggauss(N)[0]
    print(f"Number of interpolation points: {len(x_p)} ")

    # Evaluate the Runge function
    y_p = runge(x_p)

    # Use NumPy function to compute the polynomial
    from numpy.polynomial import Legendre as P
    p = P.fit(x_p, y_p, N);

    x_e = np.linspace(-1, 1, N)

    # Evaluate the Runge function
    y_e = runge(x_e)

    # Use NumPy function to compute the polynomial
    from numpy.polynomial import Legendre as P
    p_e = P.fit(x_e, y_e, N);

    plt.xlabel('$x$')
    plt.ylabel('$y$')
    plt.plot(np.linspace(-1, 1, 100), runge( np.linspace(-1, 1, 100)),'--', label='Runge function');
    plt.plot(x_p, np.zeros(len(x_p)),'o', label="Legendre roots");
    plt.plot(*p.linspace(200), '-', label='interpolation (clustered)');
    plt.plot(*p_e.linspace(200), '-', label='interpolation (equispaced)');
    plt.plot(x_e, np.zeros(len(x_e)),'x', label="Equispaced points");

    plt.ylim(-0.2, 1.5)
    plt.legend();


I am working on the file "02-LeastSquares.ipynb".
The first task is:
However, we see that the oscillations near the ends of the internal become worse with a higher degree polynomial.
By wrapping this problem in a function we can make it interactive:
Can you write Python code for it?

from ipywidgets import widgets
from ipywidgets import interact

@interact(order=(0, 19))
def plot(order):
    x_p = np.linspace(-1, 1, order + 1)
    A = np.vander(x_p, order + 1)
    y_p = runge(x_p)
    c = np.linalg.solve(A, y_p)
    p = np.poly1d(c)
    y_fit = p(x_fit)

    # Plot the interpolating polynomial and the sample points
    plt.xlabel('$x$')
    plt.ylabel('$f$')
    plt.title('Points of the Runge function interpolated by a polynomial')
    plot = plt.plot(x_p, y_p, 'o', label='points')
    plot = plt.plot(x_fit, y_fit,'-', label='interpolate ' + str(order) )
    plot = plt.plot(x, runge(x), '--', label='exact')
    plt.legend()


I am working on the file "02-LeastSquares.ipynb".
The first task is:
Next, we will sample the Runge function at points and fit a polynomial to these evaluation points. If we sample the function at $n$ points we can fit a polynomial of degree $n - 1$:
$$
f = c_{n-1} x^{n-1} + c_{n-2} x^{n-2} + \ldots  + c_{1} x + c_{0}.
$$
We can find the polynomial coefficients $c_{i}$ by solving  $\boldsymbol{A} \boldsymbol{c} = \boldsymbol{y}p$, where $\boldsymbol{A}$ is the Vandermonde matrix:
$$
\boldsymbol{A} = \begin{bmatrix}
     x{1}^{n-1} & x_{1}^{n-2} & \ldots & x_{1}^{2}  & x_{1} & 1
    \
     x_{2}^{n-1} & x_{2}^{n-2} & \ldots & x_{2}^{2}  & x_{2} & 1
    \
    \vdots  & \vdots & \vdots    & \ldots & \vdots
    \
     x_{n}^{n-1} & x_{n}^{n-2} & \ldots & x_{n}^{2}  & x_{n} & 1
\end{bmatrix}
$$
and the vector $\boldsymbol{c}$ contains the unknown polynomial coefficients
$$
\boldsymbol{c} = \begin{bmatrix}
    c_{19} & c_{20} & \ldots & c_{0}
\end{bmatrix}^{T}
$$
and the vector $\boldsymbol{y}p$ contains the points $y(x{i})$ that we wish to fit.
Note: the ordering in each row of the Vandermonde matrix above is reversed with respect to what you will find in most books. We do this because the default NumPy function for generating the Vandermonde matrix uses the above ordering.
Using the NumPy built-in function to generate the Vandermonde matrix for $n_p$ points (polynomial degree $n_{p} -1)$:
Can you write Python code for it?

n_p = 5
x_p = np.linspace(-1, 1, n_p)
A = np.vander(x_p, n_p)
print(x_p)
print(A)


I am working on the file "02-LeastSquares.ipynb".
The first task is:
Note how the polynomial fitting function oscillates near the ends of the interval. We might think that a richer, higher-order degree polynomial would provide a better fit:
Can you write Python code for it?

n_p = 13
x_p = np.linspace(-1, 1, n_p)
A = np.vander(x_p, n_p)
y_p = runge(x_p)
c = np.linalg.solve(A, y_p)
p = np.poly1d(c)
y_fit = p(x_fit)

# Plot the interpolating polynomial and the sample points
plt.xlabel('$x$')
plt.ylabel('$f$')
plt.title('Points of the Runge function interpolated by a polynomial')
plot = plt.plot(x_p, y_p, 'o', label='points')
plot = plt.plot(x_fit, y_fit,'-', label='interpolate')
plot = plt.plot(x, runge(x), '--', label='exact')
plt.legend();


I am working on the file "02-LeastSquares.ipynb".
The first task is:
NumPy has a function poly1d to turn the coefficients into a polynomial object, and it can display a representation of the polynomial:
Can you write Python code for it?


I am working on the file "02-LeastSquares.ipynb".
The first task is:
Orthogonal polynomials
In the preceding, we worked with a monomial basis:
$$
1, \ x, \ x^{2},  \ldots , \ x^{n-1}
$$
and considered polynomials of the form
$$
f = c_{n-1} x^{n-1} + c_{n-2} x^{n-2} + \ldots  + c_{1} x + c_{0}.
$$
where we would pick (or solver for in the case of interpolation) the coefficients ${c_{i}}$.
There are, however, alternatives to the monomial basis with remarkably rich and fascinating properties. We will consider Legendre polynomials on the internal $[-1, 1]$. There are various expressions for computing Legendre polynomials. One expressions for computing the Legendre polynomial of degree $n$, $P_{n}$, is:
$$
(n+1) P_{n+1}(x) = (2n+1) x P_{n}(x) - n P_{n-1}(x)
$$
where $P_{0} = 1$ and $P_{0} = x$. The special feature of Legendre polynomial is that:
$$
\int_{-1}^{1} P_{m}(x) P_{n}(x) \, dx = 0 \quad {\rm if} \ m \ne n,
$$
i.e. Legendre polynomials of different degree are orthogonal to each other.
Plotting the Legendre polynomials up to $P_{9}$:
Can you write Python code for it?

for order in range(10):
   x, y = np.polynomial.legendre.Legendre.basis(order, [-1, 1]).linspace(100)
   plt.plot(x, y, label="P_{}".format(order))
plt.grid(True)
plt.legend();
plt.savefig("legendre.pdf")


I am working on the file "02-LeastSquares.ipynb".
The first task is:
Plotting the polynomial:
Can you write Python code for it?

# Evaluate polynomial at some points
y_ls = p_ls(x_fit)

# Plot
plt.xlabel('$x$')
plt.ylabel('$f$')
# plt.ylim(-1.1, 1.1)
plt.title('Least-squares fit of 20 points')
plt.plot(x_p, y_p, 'o', x_fit, y_ls,'-');


I am working on the file "02-LeastSquares.ipynb".
The first task is:
Solving for the coefficients:
Can you write Python code for it?

y_p = runge(x_p)
c = np.linalg.solve(A, y_p)


I am working on the file "02-LeastSquares.ipynb".
The first task is:
To plot the fitted polynomial, we evaluate it at at 200 points:
Can you write Python code for it?

# Create an array of 200 equally spaced points on [-1, 1]
x_fit = np.linspace(-1, 1, 200)

# Evaluate the polynomial at the points
y_fit = p(x_fit)

# Plot the interpolating polynomial and the sample points
plt.xlabel('$x$')
plt.ylabel('$f$')
plt.title('Points of the Runge function interpolated by a polynomial')
plot = plt.plot(x_p, y_p, 'o', label='points')
plot = plt.plot(x_fit, y_fit,'-', label='interpolate')
plot = plt.plot(x, runge(x), '--', label='exact')
plt.legend();


I am working on the file "02-LeastSquares.ipynb".
The first task is:
and then solve $$\boldsymbol{A}^{T}\boldsymbol{A} \boldsymbol{c} = \boldsymbol{A}^{T} \boldsymbol{y}$$ and create a NumPy polynomial from the coefficients:
Can you write Python code for it?

ATA = (A.T).dot(A)

y_p = runge(x_p)
c_ls = np.linalg.solve(ATA, (A.T).dot(y_p))
p_ls = np.poly1d(c_ls)
print(p_ls)


I am working on the file "02-LinearFrame.ipynb".
The first task is:
Finally we solve and represent the solution
Can you write Python code for it?

Usol = np.linalg.solve(Kbc,Fbc)
frame.set_displacement(Usol)
#frame.plot_with_label()
frame.plot_displaced()


I am working on the file "02-LinearFrame.ipynb".
The first task is:
Hence, we use the dedicated functions to assemble the linear system and impose bcs
Can you write Python code for it?

F = frame.assemble_F()
K = frame.assemble_K()
Kbc, Fbc = frame.bc_apply(K,F,blocked_dof,bc_values)


I am working on the file "02-LinearFrame.ipynb".
The first task is:
Straight beam
Let us first consider a straight beam, and show how we can use the LinearFrame class to solve a beam problem discretised into several elements, as in the 01-LinearBeam notebook.
We can define the frame as a set of nodes and elements, and start by plotting it
Can you write Python code for it?


I am working on the file "02-LinearFrame.ipynb".
The first task is:
We can note the use of the rotation from the local to global coordinate systems in the different steps inside the program (e.g. assembling).
The rotation matrix allowing to pass a vector (e.g. local displacement) from the local to the global coordinate system is
Can you write Python code for it?

alpha = sp.Symbol('alpha')
frame.rotation_matrix(alpha)


I am working on the file "02-NLP_Sentiment/01-nlp_Gensim.ipynb".
The first task is:
Note that in the algorithm below, we need to choose the number of topics, which is the number of clusters of data that we want to find. Note that the accuracy of our algorithm depends a lot on picking a good number of topics.
Can you write Python code for it?

%%time
ldamodel = gensim.models.ldamodel.LdaModel(corpus,num_topics=11,id2word = dictionary)#, passes=1)
#ldamodel = gensim.models.hdpmodel.HdpModel(corpus, id2word=dictionary, T=50)

import pyLDAvis.gensim as gensimvis
import pyLDAvis

vis_data = gensimvis.prepare(ldamodel, corpus, dictionary)
pyLDAvis.display(vis_data)

ldamodel.save('../datas/lda/ldamodelFR.model')


I am working on the file "02-NLP_Sentiment/02-MLOnVotation.ipynb".
The first task is:
2. Creation of the Voting DataFrame
We load the Voting DataFrame, take only the relevant fields for us and add the topic probability distribution before exporting it. It will be ready for our Machine Learning algorithm later on.
Can you write Python code for it?

dataset_tmp = []
path = '../datas/scrap/Voting'
allFiles = glob.glob(os.path.join(path, 'Session*.csv'))

for file_ in allFiles:
    print(file_)
    data_tmp = pd.read_csv(file_)
    dataset_tmp += [data_tmp]
data_frame = pd.concat(dataset_tmp)


I am working on the file "02-NLP_Sentiment/02-MLOnVotation.ipynb".
The first task is:
2.1 slection of interesting fields in data_frame
We take only the relevant fields to us, that is
- BillTitle : The name of the voted law
- BusinessTitle : The description of what is talked about at the parliament
- FirstName and LastName : The name of the persone voting
- Decision : The vote of the person
- text : a field which combine BillTitle and BusinessTitle, we will have a better NLP clustering using this field with our LDA model.
Can you write Python code for it?

parl = data_frame.ParlGroupCode.unique().tolist()
#for group in parl :
#    data_frame.loc[data_frame.ParlGroupCode==group,'ParlGroupCode']= parl.index(group)

data_frame.head()

votation_frame = data_frame#[['BillTitle','BusinessTitle','FirstName','LastName','Decision','ParlGroupCode','VoteEnd']]
votation_frame = votation_frame.fillna(value='')
votation_frame['text'] = votation_frame['BillTitle']+' '+votation_frame['BusinessTitle']


I am working on the file "02-NLP_Sentiment/02-MLOnVotation.ipynb".
The first task is:
Finally merging the topics with the original frame containing the name and decision of parlementeer ?
Can you write Python code for it?

(pd.merge(votation_frame,topics_frame)).to_csv('../datas/nlp_results/voting_with_topics.csv',index=False)

def insert_topic_unique(data_frame) :
    dict_ = {}
    dict_['text'] =data_frame['text'].values
    dict_['Topic'] = []
    with open ("../datas/lda/topics.txt", "r") as myfile:
        s=myfile.read()
    topics = s.split('\n')
    topics_dic = {}
    for topic in topics :
        if len(topic)>1 :
            name = topic.split(':')
        topics_dic[name[0]] = name[1]
    dictionary = gensim.corpora.Dictionary.load('../datas/lda/ldaDictionary')
    for index, text in zip(data_frame.index,data_frame['text'].values) :
        if index%1000 == 0 :
            print(index)
        max_ = 0
        for topic in getTopicForQuery(text,stop_words,dictionary,ldamodel) :
            max_old = max_
            max_ = max(max_,topic[1])
            if max_old != max_ :
                topic_id = topic[0]

        dict_['Topic'] += [topics_dic[str(topic_id)]]
    return dict_

topics_s = insert_topic_unique(topic_learning_frame)
topics_ss = pd.DataFrame(topics_s)
topics_ss.head()

topics_ss.to_csv("../datas/nlp_results/voting_single_topic.csv")

data_complete = (pd.merge(votation_frame,topics_ss))
data_complete.head(2)

data_complete.VoteEnd = [x[:10] for x in data_complete.VoteEnd]
data_complete.head(2)

not_used = ['Unnamed: 0','BusinessNumber','BusinessShortNumber','CantonID','ID','IdLegislativePeriod',
           'IdSession','IdVote','PersonNumber','RegistrationNumber','BillTitle','BusinessTitle','DecisionText',
           'Language','MeaningNo','MeaningYes','ParlGroupColour','ParlGroupCode','ParlGroupNameAbbreviation',
           'Subject','text']
data_used = data_complete.drop(not_used ,axis=1)
#data_used.head()
data_mean = data_used.set_index(['VoteEnd','Topic','FirstName'])

data_mean.head(1000)
data_used.to_csv("../datas/nlp_results/voting_single_topic_not_unique.csv")

for year in data_mean.index.get_level_values('VoteEnd').unique() :
    for Topic in...


I am working on the file "02-NLP_Sentiment/02-MLOnVotation.ipynb".
The first task is:
Now we load the lda model we use along with the stop words, in order to have them available for the time we will use them, avoiding to reload them every time we call getTopicForQuery. We also load our lda model for once.
Can you write Python code for it?

stop_words_de = get_stop_words('de')

with open ("../datas/stop_dictionaries/French_stop_words_changed.txt", "r") as myfile:
    stop_words=myfile.read()
stop_words = stop_words.split(',')

stop_words = stop_words_de+stop_words
ldamodel = gensim.models.LdaModel.load('../datas/lda/ldamodelFR.model', mmap='r')


I am working on the file "02-NLP_Sentiment/02-MLOnVotation.ipynb".
The first task is:
We create a smaller DataFrame which contains only the subjects that are voted, we do not repeat the text each time for each person who votes. Hence we will perform the NLP once on each unique entry.
Can you write Python code for it?

text_dict = {'text': votation_frame.text.unique()}
topic_learning_frame = pd.DataFrame(text_dict)
topic_learning_frame.head()


I am working on the file "02-NLP_Sentiment/02-MLOnVotation.ipynb".
The first task is:
We now create a frame using the topics found using insert_topic
Can you write Python code for it?

from yandex_translate import YandexTranslate
translate = YandexTranslate('trnsl.1.1.20161208T132730Z.fe490b34d7db4e4f.0a4c7781a0273d520073a1550b6a6624c1c3fd0a')
text_eng = []
for text in topic_learning_frame.text:
    s = translate.translate(text, 'fr-en')
    text_eng += [s['text'][0]]

topics_dict = insert_topic(topic_learning_frame)
topics_frame = pd.DataFrame(topics_dict)
#topics_frame['text_eng'] =  text_eng

topics_frame.head(5)
#topics_frame.to_csv('../datas/nlp_results/voting_with_topics_unique.csv',index=False)


I am working on the file "02-NumPy.ipynb".
The first task is:
5 amostras aleatórias tiradas da distribuição normal de média 0 e variância 1.
Can you write Python code for it?


I am working on the file "02-NumPy.ipynb".
The first task is:
5 amostras aleatórias tiradas da distribuição normal de média 10 e variância 3.
Can you write Python code for it?


I am working on the file "02-NumPy.ipynb".
The first task is:
A diferença entre list e array é que a arrays são homógenas!
Can you write Python code for it?

lst[-1] = 'Um string'
lst

arr[-1] = 'Um string'
arr

arr.dtype

arr[-1] = 1.234
arr


I am working on the file "02-NumPy.ipynb".
The first task is:
Dados com máscara (Masked arrays)
Can you write Python code for it?

plt.pcolormesh(data)

import numpy.ma as ma

data = ma.masked_invalid(data)

plt.pcolormesh(np.flipud(data.T))
plt.colorbar()

data.max(), data.min(), data.mean()

z = [1, 10, 100, 120, 13, 140, 180, 20, 300, 40,5, 500, 60, 80]

fig, ax = plt.subplots()
ax.plot(data[42, :], z, 'ko')
ax.invert_yaxis()