I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Null accuracy: accuracy that could be achieved by always predicting the most frequent class
Can you write Python code for it?

# examine the class distribution of the testing set (using a Pandas Series method)
y_test.value_counts()

# calculate the percentage of ones
y_test.mean()

# calculate the percentage of zeros
1 - y_test.mean()

# calculate null accuracy (for binary classification problems coded as 0/1)
max(y_test.mean(), 1 - y_test.mean())

# calculate null accuracy (for multi-class classification problems)
y_test.value_counts().head(1) / len(y_test)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Precision: When a positive value is predicted, how often is the prediction correct?

How "precise" is the classifier when predicting positive instances?
Can you write Python code for it?

print(TP / float(TP + FP))
print(metrics.precision_score(y_test, y_pred_class))


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Question: Can we predict the diabetes status of a patient given their health measurements?
Can you write Python code for it?

# define X and y
feature_cols = ['pregnant', 'insulin', 'bmi', 'age']
X = pima[feature_cols]
y = pima.label

# split X and y into training and testing sets
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# train a logistic regression model on the training set
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(X_train, y_train)

# make class predictions for the testing set
y_pred_class = logreg.predict(X_test)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
ROC curve can help you to choose a threshold that balances sensitivity and specificity in a way that makes sense for your particular context
You can't actually see the thresholds used to generate the curve on the ROC curve itself
Can you write Python code for it?

# define a function that accepts a threshold and prints sensitivity and specificity
def evaluate_threshold(threshold):
    print('Sensitivity:', tpr[thresholds > threshold][-1])
    print('Specificity:', 1 - fpr[thresholds > threshold][-1])

evaluate_threshold(0.5)

evaluate_threshold(0.3)


I am working on the file "Section 3 - Machine Learning/ThirdParty-scikit-learn-videos-master/09_classification_metrics.ipynb".
The first task is:
Sensitivity: When the actual value is positive, how often is the prediction correct?

How "sensitive" is the classifier to detecting positive instances?
Also known as "True Positive Rate" or "Recall"
Can you write Python code for it?

print(TP / float(TP + FN))
print(metrics.recall_score(y_test, y_pred_class))


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
Cluster quality metrics evaluated (see Clustering performance evaluation for definitions and discussions of the metrics):
| Shorthand     | full name                     |
|------------   |-----------------------------  |
| homo          | homogeneity score             |
| compl         | completeness score            |
| v-meas        | V measure                     |
| ARI           | adjusted Rand index           |
| AMI           | adjusted mutual information   |
| silhouette    | silhouette coefficient        |
Unsupervised Learning: Dimensionality Reduction and Visualization
Dimensionality Reduction: PCA
Dimensionality reduction derives a set of new artificial features smaller than the original feature set. Here we’ll use Principal Component Analysis (PCA), a dimensionality reduction that strives to retain most of the variance of the original data. We’ll use sklearn.decomposition.PCA on the iris dataset:
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()

X = iris.data
y = iris.target


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
Furthermore, the samples components do no longer carry any linear correlation:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
It seems that the ham emails are more often plain text, while spam has quite a lot of HTML. Moreover, quite a few ham emails are signed using PGP, while no spam is. In short, it seems that the email structure is a usual information to have.
Now let's take a look at the email headers:
Can you write Python code for it?

for header, value in spam_emails[0].items():
    print(header,":",value)

spam_emails[0]["Subject"]


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
Let us project the iris dataset along those first two dimensions::
Can you write Python code for it?

X_pca = pca.transform(X)
X_pca.shape

print(X_pca[2])


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
Let's throw in some stemming! For this to work, you need to install the Natural Language Toolkit (NLTK). It's as simple as running the following command (don't forget to activate your virtualenv first; if you don't have one, you will likely need administrator rights, or use the --user option):
Can you write Python code for it?

try:
    import nltk

    stemmer = nltk.PorterStemmer()
    for word in ("Computations", "Computation", "Computing", "Computed", "Compute", "Compulsive"):
        print(word, "=>", stemmer.stem(word))
except ImportError:
    print("Error: stemming requires the NLTK module.")
    stemmer = None

try:
    import urlextract # may require an Internet connection to download root domain names

    url_extractor = urlextract.URLExtract()
    print(url_extractor.find_urls("Will it detect github.com and https://youtu.be/7Pq-S557XQU?t=3m32s"))
except ImportError:
    print("Error: replacing URLs requires the urlextract module.")
    url_extractor = None


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
Let's try this transformer on a few emails:
Can you write Python code for it?

X_few = X_train[:3]
X_few_wordcounts = EmailToWordCounterTransformer().fit_transform(X_few)
X_few_wordcounts


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
Okay, before we learn too much about the data, let's not forget to split it into a training set and a test set:
Can you write Python code for it?

import numpy as np
from sklearn.model_selection import train_test_split

X = np.array(ham_emails + spam_emails)
y = np.array([0] * len(ham_emails) + [1] * len(spam_emails))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
Once fitted, PCA exposes the singular vectors in the components_ attribute:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
PCA normalizes and whitens the data, which means that the data is now centered on both components with unit variance:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
Spam classifier
Can you write Python code for it?

import os
import tarfile

files = ["20030228_easy_ham.tar.bz2", "20050311_spam_2.tar.bz2"]
SPAM_PATH = os.path.join("datasets", "spam")

def fetch_spam_data(spam_url=SPAM_URL, spam_path=SPAM_PATH):
    for filename in (files):
        path = os.path.join(SPAM_PATH, filename)
        tar_bz2_file = tarfile.open(path)
        tar_bz2_file.extractall(path=SPAM_PATH)
        tar_bz2_file.close()

fetch_spam_data()

HAM_DIR = os.path.join(SPAM_PATH, "easy_ham")
SPAM_DIR = os.path.join(SPAM_PATH, "spam_2")
ham_filenames = [name for name in sorted(os.listdir(HAM_DIR)) if len(name) > 20]
spam_filenames = [name for name in sorted(os.listdir(SPAM_DIR)) if len(name) > 20]

print(len(ham_filenames), len(spam_filenames))

import email
import email.policy

def load_email(is_spam, filename, spam_path=SPAM_PATH):
    directory = "spam_2" if is_spam else "easy_ham"
    with open(os.path.join(spam_path, directory, filename), "rb") as f:
        return email.parser.BytesParser(policy=email.policy.default).parse(f)

ham_emails = [load_email(is_spam=False, filename=name) for name in ham_filenames]
spam_emails = [load_email(is_spam=True, filename=name) for name in spam_filenames]

print(spam_emails[6].get_content().strip())

print(ham_emails[1].get_content().strip())

def get_email_structure(email):
    if isinstance(email, str):
        return email
    payload = email.get_payload()
    if isinstance(payload, list):
        return "multipart({})".format(", ".join([
            get_email_structure(sub_email)
            for sub_email in payload
        ]))
    else:
        return email.get_content_type()

from collections import Counter

def structures_counter(emails):
    structures = Counter()
    for email in emails:
        structure = get_email_structure(email)
        structures[structure] += 1
    return structures

structures_counter(ham_emails).most_common()

structures_counter(spam_emails).most_common()


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
The eigenfaces example: chaining PCA and SVMs
The goal of this example is to show how an unsupervised method and a supervised one can be chained for better prediction. It starts with a didactic but lengthy way of doing things, and finishes with the idiomatic approach to pipelining in scikit-learn.
Here we’ll take a look at a simple facial recognition example. Ideally, we would use a dataset consisting of a subset of the Labeled Faces in the Wild data that is available with sklearn.datasets.fetch_lfw_people(). However, this is a relatively large download (~200MB) so we will do the tutorial on a simpler, less rich dataset. Feel free to explore the LFW dataset.
Can you write Python code for it?

from sklearn import datasets
faces = datasets.fetch_olivetti_faces()
faces.data.shape

from matplotlib import pyplot as plt
fig = plt.figure(figsize=(8, 6))
# plot several images
for i in range(15):
    ax = fig.add_subplot(3, 5, i + 1, xticks=[], yticks=[])
    ax.imshow(faces.images[i], cmap=plt.cm.bone)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(faces.data,
        faces.target, random_state=0)

print(X_train.shape, X_test.shape)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
We are now ready to train our first spam classifier! Let's transform the whole dataset:
Can you write Python code for it?

from sklearn.pipeline import Pipeline

preprocess_pipeline = Pipeline([
    ("email_to_wordcount", EmailToWordCounterTransformer()),
    ("wordcount_to_vector", WordCounterToVectorTransformer()),
])

X_train_transformed = preprocess_pipeline.fit_transform(X_train)

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

log_clf = LogisticRegression()
score = cross_val_score(log_clf, X_train_transformed, y_train, cv=3, verbose=3)
score.mean()


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
We are ready to put all this together into a transformer that we will use to convert emails to word counters. Note that we split sentences into words using Python's split() method, which uses whitespaces for word boundaries. This works for many written languages, but not all. For example, Chinese and Japanese scripts generally don't use spaces between words, and Vietnamese often uses spaces even between syllables. It's okay in this exercise, because the dataset is (mostly) in English.
Can you write Python code for it?

from sklearn.base import BaseEstimator, TransformerMixin

class EmailToWordCounterTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, strip_headers=True, lower_case=True, remove_punctuation=True,
                 replace_urls=True, replace_numbers=True, stemming=True):
        self.strip_headers = strip_headers
        self.lower_case = lower_case
        self.remove_punctuation = remove_punctuation
        self.replace_urls = replace_urls
        self.replace_numbers = replace_numbers
        self.stemming = stemming
    def fit(self, X, y=None):
        return self
    def transform(self, X, y=None):
        X_transformed = []
        for email in X:
            text = email_to_text(email) or ""
            if self.lower_case:
                text = text.lower()
            if self.replace_urls and url_extractor is not None:
                urls = list(set(url_extractor.find_urls(text)))
                urls.sort(key=lambda url: len(url), reverse=True)
                for url in urls:
                    text = text.replace(url, " URL ")
            if self.replace_numbers:
                text = re.sub(r'\d+(?:\.\d*(?:[eE]\d+))?', 'NUMBER', text)
            if self.remove_punctuation:
                text = re.sub(r'\W+', ' ', text, flags=re.M)
            word_counts = Counter(text.split())
            if self.stemming and stemmer is not None:
                stemmed_word_counts = Counter()
                for word, count in word_counts.items():
                    stemmed_word = stemmer.stem(word)
                    stemmed_word_counts[stemmed_word] += count
                word_counts = stemmed_word_counts
            X_transformed.append(word_counts)
        return np.array(X_transformed)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/0. Introduction.ipynb".
The first task is:
What does this matrix mean? Well, the 65 in the third row, first column, means that the third email contains 65 words that are not part of the vocabulary. The 0 next to it means that the first word in the vocabulary is not present in this email. The 1 next to it means that the second word is present once, and so on. You can look at the vocabulary to know which words we are talking about. The first word is "the", the second word is "of", etc.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
Advantages

Random (uniform) label assignments have a AMI score close to 0.0 for any value of n_clusters and n_samples (which is not the case for raw Mutual Information or the V-measure for instance).
Bounded range [0, 1]: Values close to zero indicate two label assignments that are largely independent, while values close to one indicate significant agreement. Further, values of exactly 0 indicate purely independent label assignments and a AMI of exactly 1 indicates that the two label assignments are equal (with or without permutation).
No assumption is made on the cluster structure: can be used to compare clustering algorithms such as k-means which assumes isotropic blob shapes with results of spectral clustering algorithms which can find cluster with “folded” shapes.

Drawbacks

Contrary to inertia, MI-based measures require the knowledge of the ground truth classes while almost never available in practice or requires manual assignment by human annotators (as in the supervised learning setting).
However MI-based measures can also be useful in purely unsupervised setting as a building block for a Consensus Index that can be used for clustering model selection.

NMI and MI are not adjusted against chance.

$H(U) = - $$\sum_{i=1}^{|U|}P(i)\log(P(i))$

where P(i) = |U_i| / N is the probability that an object picked at random from U falls into class U_i.
Can you write Python code for it?

print(__doc__)

# Author: Olivier Grisel <olivier.grisel@ensta.org>
# License: BSD 3 clause

import numpy as np
import matplotlib.pyplot as plt
from time import time
from sklearn import metrics

def uniform_labelings_scores(score_func, n_samples, n_clusters_range,
                             fixed_n_classes=None, n_runs=5, seed=42):
    """Compute score for 2 random uniform cluster labelings.

    Both random labelings have the same number of clusters for each value
    possible value in ``n_clusters_range``.

    When fixed_n_classes is not None the first labeling is considered a ground
    truth class assignment with fixed number of classes.
    """
    random_labels = np.random.RandomState(seed).randint
    scores = np.zeros((len(n_clusters_range), n_runs))

    if fixed_n_classes is not None:
        labels_a = random_labels(low=0, high=fixed_n_classes, size=n_samples)

    for i, k in enumerate(n_clusters_range):
        for j in range(n_runs):
            if fixed_n_classes is None:
                labels_a = random_labels(low=0, high=k, size=n_samples)
            labels_b = random_labels(low=0, high=k, size=n_samples)
            scores[i, j] = score_func(labels_a, labels_b)
    return scores

score_funcs = [
    metrics.adjusted_rand_score,
    metrics.v_measure_score,
    metrics.adjusted_mutual_info_score,
    metrics.mutual_info_score,
]

# 2 independent random clusterings with equal cluster number

n_samples = 100
n_clusters_range = np.linspace(2, n_samples, 10).astype(np.int)

plt.figure(1)

plots = []
names = []
for score_func in score_funcs:
    print("Computing %s for %d values of n_clusters and n_samples=%d"
          % (score_func.__name__, len(n_clusters_range), n_samples))

    t0 = time()
    scores = uniform_labelings_scores(score_func, n_samples, n_clusters_range)
    print("done in %0.3fs" % (time() - t0))
    plots.append(plt.errorbar(
        n_clusters_range, np.median(scores, axis=1), scores.std(axis=1))[0])
    names.append(score_func.__name__)

plt.title("Clustering measures...


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
All, mutual_info_score, adjusted_mutual_info_score and normalized_mutual_info_score are symmetric: swapping the argument does not change the score. Thus they can be used as a consensus measure:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
Bad (e.g. independent labelings) have zero scores:
Can you write Python code for it?

labels_true = [0, 1, 2, 0, 3, 4, 5, 1]
labels_pred = [1, 1, 0, 0, 2, 2, 2, 2]
metrics.fowlkes_mallows_score(labels_true, labels_pred)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
Fowlkes-Mallows scores
The Fowlkes-Mallows index (sklearn.metrics.fowlkes_mallows_score) can be used when the ground truth class assignments of the samples is known. The Fowlkes-Mallows score FMI is defined as the geometric mean of the pairwise precision and recall:
$\text{FMI} = $$\frac{\text{TP}}{\sqrt{(\text{TP} + \text{FP}) (\text{TP} + \text{FN})}}$
Where TP is the number of True Positive (i.e. the number of pair of points that belong to the same clusters in both the true labels and the predicted labels), FP is the number of False Positive (i.e. the number of pair of points that belong to the same clusters in the true labels and not in the predicted labels) and FN is the number of False Negative (i.e the number of pair of points that belongs in the same clusters in the predicted labels and not in the true labels).
The score ranges from 0 to 1. A high value indicates a good similarity between two clusters.
Can you write Python code for it?

from sklearn import metrics
labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 1, 1, 2, 2]

metrics.fowlkes_mallows_score(labels_true, labels_pred)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
It is defined as the number of pairs of objects that are either in the same group or in different groups in both partitions divided by the total number of pairs of objects. The Rand index lies between 0 and 1. When two partitions agree perfectly, the Rand index achieves the maximum value 1. A problem with Rand index is that the expected value of the Rand index between two random partitions is not a constant. This problem is corrected by the adjusted Rand index that assumes the generalized hyper-geometric distribution as the model of randomness. The adjusted Rand index has the maximum value 1, and its expected value is 0 in the case of random clusters. A larger adjusted Rand index means a higher agreement between two partitions. The adjusted Rand index is recommended for measuring agreement even when the partitions compared have different numbers of clusters.
The Rand index or Rand measure (named after William M. Rand) in statistics, and in particular in data clustering, is a measure of the similarity between two data clusterings. A form of the Rand index may be defined that is adjusted for the chance grouping of elements, this is the adjusted Rand index. From a mathematical standpoint, Rand index is related to the accuracy, but is applicable even when class labels are not used.

a, the number of pairs of elements in  S that are in the same subset in  X and in the same subset in  Y
b, the number of pairs of elements in  S that are in different subsets in  X and in different subsets in  Y
c, the number of pairs of elements in  S that are in the same subset in  X and in different subsets in  Y
d, the number of pairs of elements in  S that are in different subsets in  X and in the same subset in  Y

$R = $$\frac{a+b}{a+b+c+d} = \frac{a+b}{{n \choose 2 }}$
$${\displaystyle \overbrace {ARI} ^{\text{Adjusted Index}}={\frac {\overbrace {\sum {ij}{\binom {n{ij}}{2}}} ^{\text{Index}}-\overbrace {[\sum {i}{\binom {a{i}}{2}}\sum {j}{\binom {b{j}}{2}}]/{\binom {n}{2}}} ^{\text{Expected Index}}}{\underbrace {{\frac...
Can you write Python code for it?

from sklearn import metrics

labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 1, 1, 1, 2]

metrics.adjusted_rand_score(labels_true, labels_pred)

labels_pred = [1, 1, 0, 0, 3, 3]
metrics.adjusted_rand_score(labels_true, labels_pred)

labels_pred = [1, 10, 1, 0, 10, 0]
metrics.adjusted_rand_score(labels_true, labels_pred)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
Mutual Information based scores
Given the knowledge of the ground truth class assignments labels_true and our clustering algorithm assignments of the same samples labels_pred, the Mutual Information is a function that measures the agreement of the two assignments, ignoring permutations. Two different normalized versions of this measure are available, Normalized Mutual Information(NMI) and Adjusted Mutual Information(AMI). NMI is often used in the literature while AMI was proposed more recently and is normalized against chance:
Can you write Python code for it?

from sklearn import metrics
labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 1, 1, 2, 2]

metrics.adjusted_mutual_info_score(labels_true, labels_pred)

from sklearn import metrics
labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 0, 1, 1, 1]

metrics.adjusted_mutual_info_score(labels_true, labels_pred)

from sklearn import metrics
labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 0, 1, 1, 2]
print(metrics.adjusted_rand_score(labels_true, labels_pred))
print(metrics.adjusted_mutual_info_score(labels_true, labels_pred) )

from sklearn import metrics
labels_true = [0, 0, 0, 1, 1, 1]
labels_pred = [0, 0, 0, 1, 5, 1]
print(metrics.adjusted_rand_score(labels_true, labels_pred))
metrics.adjusted_mutual_info_score(labels_true, labels_pred)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
Note v_measure_score is symmetric: it can be used to evaluate the agreement of two independent assignments on the same dataset.
This is not the case for completeness_score and homogeneity_score: both are bound by the relationship:
Can you write Python code for it?

metrics.homogeneity_score(labels_true, labels_pred) == metrics.completeness_score(labels_pred, labels_true)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
One can permute 0 and 1 in the predicted labels, rename 2 to 3 and get the same score
Can you write Python code for it?

labels_pred = [1, 1, 0, 0, 3, 3]
metrics.fowlkes_mallows_score(labels_true, labels_pred)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
Perfect labeling is scored 1.0:
Can you write Python code for it?

labels_pred = labels_true[:]
metrics.adjusted_mutual_info_score(labels_true, labels_pred)
metrics.normalized_mutual_info_score(labels_true, labels_pred)


I am working on the file "Section 3 - Machine Learning/UnSupervised Learning Algorithm/2. Clustering performance evaluation.ipynb".
The first task is:
The V-measure is actually equivalent to the mutual information (NMI) discussed above normalized by the sum of the label entropies.
Homogeneity, completeness and V-measure can be computed at once using homogeneity_completeness_v_measure as follows:
metrics.homogeneity_completeness_v_measure(labels_true, labels_pred)
The following clustering assignment is slightly better, since it is homogeneous but not complete:
Can you write Python code for it?

labels_pred = [0, 0, 0, 1, 2, 2]
metrics.homogeneity_completeness_v_measure(labels_true, labels_pred)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Adding a legend
Can you write Python code for it?

ax = plt.gca()  # gca stands for 'get current axis'
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.spines['bottom'].set_position(('data',0))
ax.yaxis.set_ticks_position('left')
ax.spines['left'].set_position(('data',0))
plt.plot(X, C, color="blue", linewidth=2.5, linestyle="-.", label="cosine")
plt.plot(X, S, color="red",  linewidth=2.5, linestyle="-", label="sine")

plt.legend(loc='upper left')


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Bar Plots
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

n = 12
X = np.arange(n)
Y1 = (1 - X / float(n)) * np.random.uniform(0.5, 0.7, n)
Y2 = (1 - X / float(n)) * np.random.uniform(0.5, 1.0, n)

plt.axes([0.025, 0.025, 0.95, 0.95])
plt.bar(X, +Y1, facecolor='#9999ff', edgecolor='white')
plt.bar(X, -Y2, facecolor='#ff9999', edgecolor='white')

for x, y in zip(X, Y1):
    plt.text(x + 0.4, y + 0.05, '%.2f' % y, ha='center', va= 'bottom')

for x, y in zip(X, Y2):
    plt.text(x + 0.4, -y - 0.05, '%.2f' % y, ha='center', va= 'top')

plt.xlim(-.5, n)
plt.xticks(())
plt.ylim(-1.25, 1.25)
plt.yticks(())

plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Contour Plots
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

def f(x,y):
    return (1 - x / 2 + x**5 + y**3) * np.exp(-x**2 -y**2)

n = 256
x = np.linspace(-3, 3, n)
y = np.linspace(-3, 3, n)
X,Y = np.meshgrid(x, y)

plt.axes([0.025, 0.025, 0.95, 0.95])

plt.contourf(X, Y, f(X, Y), 8, alpha=.75, cmap=plt.cm.hot)
C = plt.contour(X, Y, f(X, Y), 8, colors='black', linewidth=.5)
plt.clabel(C, inline=1, fontsize=10)

plt.xticks(())
plt.yticks(())
plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Instantiating defaults
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

# Create a figure of size 8x6 inches, 80 dots per inch
plt.figure(figsize=(8, 6), dpi=80)

# Create a new subplot from a grid of 1x1
plt.subplot(1, 1, 1)

X = np.linspace(-np.pi, np.pi, 256, endpoint=True)
C, S = np.cos(X), np.sin(X)

# Plot cosine with a blue continuous line of width 1 (pixels)
plt.plot(X, C, color="blue", linewidth=1.0, linestyle="-")

# Plot sine with a green continuous line of width 1 (pixels)
plt.plot(X, S, color="green", linewidth=1.0, linestyle="-")

# Set x limits
plt.xlim(-4.0, 4.0)

# Set x ticks
plt.xticks(np.linspace(-4, 4, 9, endpoint=True))

# Set y limits
plt.ylim(-1.0, 1.0)

# Set y ticks
plt.yticks(np.linspace(-1, 1, 5, endpoint=True))

# Save figure using 72 dots per inch
# plt.savefig("exercise_2.png", dpi=72)

# Show result on screen
plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
NOTE:
Use the below code for

IPython console:  %matplotlib

 Jupyter notebook: %matplotlib inline

Plotting
The most important function in matplotlib is plot, which allows you to plot 2D data. Here is a simple example:
Can you write Python code for it?

# Compute the x and y coordinates for points on a sine curve
x = np.arange(0, 3 * np.pi, 0.1)
y = np.sin(x)

# Plot the points using matplotlib
plt.plot(x, y)
plt.show()

a = np.array([1, 4, 5, 66, 77, 334], int)
plt.plot(a)
plt.show()

Lets add more details to the graphs.

y_cos = np.cos(x)
y_sin = np.sin(x)
# Plot the points using matplotlib
plt.plot(x, y_sin)
plt.plot(x, y_cos)
plt.xlabel('x axis label')
plt.ylabel('y axis label')
plt.title('Sine and Cosine Graph')
plt.legend(['Sine', 'Cosine'])
plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Plots with fill
Can you write Python code for it?

x = np.arange(0.0, 2, 0.01)
y1 = np.sin(2*np.pi*x)
y2 = 1.2*np.sin(4*np.pi*x)

fig, (ax1, ax2, ax3) = plt.subplots(3, 1, sharex=True)

ax1.fill_between(x, 0, y1)
ax1.set_ylabel('between y1 and 0')

ax2.fill_between(x, y1, 1)
ax2.set_ylabel('between y1 and 1')

ax3.fill_between(x, y1, y2)
ax3.set_ylabel('between y1 and y2')
ax3.set_xlabel('x')
plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Plotting with default settings
Matplotlib comes with a set of default settings that allow customizing all kinds of properties. You can control the defaults of almost every property in matplotlib: figure size and dpi, line width, color and style, axes, axis and grid properties, text and font properties and so on.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

X = np.linspace(-np.pi, np.pi, 256, endpoint=True)
C, S = np.cos(X), np.sin(X)

plt.plot(X, C)
plt.plot(X, S)

plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Scatter Plots
Can you write Python code for it?

n = 1024
X = np.random.normal(0,1,n)
Y = np.random.normal(0,1,n)

plt.scatter(X,Y)

# with colors
import numpy as np
import matplotlib.pyplot as plt

n = 1024
X = np.random.normal(0, 1, n)
Y = np.random.normal(0, 1, n)
T = np.arctan2(Y, X)
print(t)
plt.axes([0.025, 0.025, 0.95, 0.95])
plt.scatter(X, Y, s=75, c=T, alpha=.5)

plt.xlim(-1.5, 1.5)
plt.xticks(())
plt.ylim(-1.5, 1.5)
plt.yticks(())

plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Set up a subplot grid that has height 2 and width 1,
and set the first such subplot as active.
plt.subplot(1, 2, 1)
Make the first plot
plt.plot(x, y_sin)
plt.title('Sine')
Set the second subplot as active, and make the second plot.
plt.subplot(1, 2, 2)
plt.plot(x, y_cos)
plt.title('Cosine')
Show the figure.
plt.show()
Can you write Python code for it?

# Set up a subplot grid that has height 2 and width 1,
# and set the first such subplot as active.
plt.subplot(2, 2, 1)

# Make the first plot
plt.plot(x, y_sin)
plt.title('Sine')

# Set the second subplot as active, and make the second plot.
plt.subplot(2, 2, 2)
plt.plot(x, y_cos)
plt.title('Cosine')

a = np.linspace(-1, 1, num=100)
y_arcsin = np.arcsin(a)
y_arccos = np.arccos(a)

plt.subplot(2, 2, 3)
plt.plot(a, y_arcsin)
plt.title('arcsin')

plt.subplot(2, 2, 4)
plt.plot(a, y_arccos)
plt.title('arccos')

# Show the figure.
plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Setting limits
Current limits of the figure are a bit too tight and we want to make some space in order to clearly see all data points
Can you write Python code for it?

plt.xlim(X.min() * 1.1, X.max() * 1.1)
plt.ylim(C.min() * 1.1, C.max() * 1.1)

plt.plot(X, C, color="blue", linewidth=2.5, linestyle="-")
plt.plot(X, S, color="red",  linewidth=2.5, linestyle="-")


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Setting ticks
Current ticks are not ideal because they do not show the interesting values (+/-π,+/-π/2) for sine and cosine. We’ll change them such that they show only these values.
Can you write Python code for it?

plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi])
plt.yticks([-1, 0, +1])
plt.plot(X, C, color="blue", linewidth=2.5, linestyle="-")
plt.plot(X, S, color="red",  linewidth=2.5, linestyle="-")

t = 2 * np.pi / 3
plt.plot([t, t], [0, np.cos(t)], color='blue', linewidth=2.5, linestyle="--")
plt.scatter([t, ], [np.cos(t), ], 50, color='blue')

plt.annotate(r'$cos(\frac{2\pi}{3})=-\frac{1}{2}$',
             xy=(t, np.cos(t)), xycoords='data',
             xytext=(-90, -50), textcoords='offset points', fontsize=16,
             arrowprops=dict(arrowstyle="->", connectionstyle="arc3,rad=.2"))

plt.plot([t, t],[0, np.sin(t)], color='red', linewidth=2.5, linestyle="--")
plt.scatter([t, ],[np.sin(t), ], 50, color='red')

plt.annotate(r'$sin(\frac{2\pi}{3})=\frac{\sqrt{3}}{2}$',
             xy=(t, np.sin(t)), xycoords='data',
             xytext=(+10, +30), textcoords='offset points', fontsize=16,
             arrowprops=dict(arrowstyle="->", connectionstyle="arc3,rad=.2"))
plt.plot(X, C, color="blue", linewidth=2.5, linestyle="-")
plt.plot(X, S, color="red",  linewidth=2.5, linestyle="-")

ax = plt.gca()  # gca stands for 'get current axis'
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.spines['bottom'].set_position(('data',0))
ax.yaxis.set_ticks_position('left')
ax.spines['left'].set_position(('data',0))

plt.plot(X, C, color="blue", linewidth=2.5, linestyle="-.")
plt.plot(X, S, color="red",  linewidth=2.5, linestyle="-")


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Simple plot
Can you write Python code for it?

import numpy as np

X = np.linspace(-np.pi, np.pi, 256, endpoint=True)
C, S = np.cos(X), np.sin(X)

print(C)
print(S)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Subplots
With subplot you can arrange plots in a regular grid. You need to specify the number of rows and columns and the number of the plot. Note that the gridspec command is a more powerful alternative.
Can you write Python code for it?

x = np.arange(0, 3 * np.pi, 0.1)
y_sin = np.sin(x)
y_cos = np.cos(x)

# Set up a subplot grid that has height 2 and width 1,
# and set the first such subplot as active.
plt.subplot(2, 1, 1)

# Make the first plot
plt.plot(x, y_sin)
plt.title('Sine')

# Set the second subplot as active, and make the second plot.
plt.subplot(2, 1, 2)
plt.plot(x, y_cos)
plt.title('Cosine')

# Show the figure.
plt.show()

# Set up a subplot grid that has height 2 and width 1,
# and set the first such subplot as active.
plt.subplot(2, 1, 1)

# Make the first plot
plt.plot(x, y_sin)
plt.title('Sine')

# Set the second subplot as active, and make the second plot.
plt.subplot(2, 1, 2)
plt.plot(x, y_cos)
plt.title('Cosine')

# Show the figure.
plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
Subplots
You can plot different things in the same figure using the subplot function. Here is an example:
Can you write Python code for it?

# Compute the x and y coordinates for points on sine and cosine curves
x = np.arange(0, 3 * np.pi, 0.1)
y_sin = np.sin(x)
y_cos = np.cos(x)

# Set up a subplot grid that has height 2 and width 1,
# and set the first such subplot as active.
plt.subplot(2, 1, 1)

# Make the first plot
plt.plot(x, y_sin)
plt.title('Sine')

# Set the second subplot as active, and make the second plot.
plt.subplot(2, 1, 2)
plt.plot(x, y_cos)
plt.title('Cosine')

# Show the figure.
plt.show()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/matplotlib/Matplotlib.ipynb".
The first task is:
We can also change the attributes of the graph
Can you write Python code for it?

plt.figure(figsize=(10, 6), dpi=80)

plt.plot(X, C, color="blue", linewidth=2.5, linestyle="-.")
plt.plot(X, S, color="red",  linewidth=2.5, linestyle="-")


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:

Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
A data type object represent, fixed block of memory corresponding to an array, depending on the following aspects:
* Size of data
* Type of data (integer, float or Python object)
* Byte order (little-endian or big-endian)
* In case of custom type, the name & data type of each field and part of the memory block taken by each field
* If data type is a subarray, its shape and data type
Numpy supports following data types, some of them might not be present in python.
Data Types|Description|Range|
:-----:|:-----:|:-----:|
bool_|Boolean (True or False) stored as a byte| 0 1
int_|Default integer type (same as C long; normally either int64 or int32)|
intc| Identical to C int (normally int32 or int64)|
intp| Integer used for indexing (same as C ssize_t; normally either int32 or int64)|
int8| Byte | (-128 to 127)
int16| Integer | (-32768 to 32767)
int32| Integer | (-2147483648 to 2147483647)
int64| Integer |(-9223372036854775808 to 9223372036854775807)|
uint8| Unsigned integer |(0 to 255)|
uint16| Unsigned integer |(0 to 65535)|
uint32| Unsigned integer |(0 to 4294967295)|
uint64| Unsigned integer |(0 to 18446744073709551615)|
float_| Shorthand for float64||
float16| Half precision float: sign bit, 5 bits exponent, 10 bits mantissa||
float32| Single precision float: sign bit, 8 bits exponent, 23 bits mantissa| |
float64| Double precision float: sign bit, 11 bits exponent, 52 bits mantissa||
complex_|Shorthand for complex128||
complex64| Complex number, represented by two 32-bit floats (real and imaginary components)||
complex128| Complex number, represented by two 64-bit floats (real and imaginary components)||
built-in data type has a character code that uniquely identifies it.
| Char code     | Data Types                |
|---------- |------------------------   |
| 'b'       | boolean                   |
| 'i'       | (signed) integer          |
| 'u'       | unsigned integer          |
| 'f'       | floating-point            |
| 'c'       | complex-floating point    |
| 'm'       | timedelta                 |
| 'M'...
Can you write Python code for it?

dt = np.dtype('i1')
print(dt)
dt = np.dtype('i2')
print(dt)
dt = np.dtype('i4')
print(dt)
dt = np.dtype('i8')
print(dt)

dt = np.dtype([('we will be using this user later in our examples.','a128')])
print(dt)

user=np.dtype([('first_name','S30'), ('last_name','S30'), ('pay', 'i2'), ('designation_id', 'f2')])
print(user)

a4 = np.array([[1, 3, 2], [1, 4, 5]],  dtype=np.int8)
print(a4)

a4 = np.array([[1, 3, 2], [1, 4, 5]],  dtype=np.float)
print(a4)

# TODO How much memory space a variable is taking


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Apart from computing mathematical functions using arrays, we frequently need to reshape or otherwise manipulate data in arrays. The simplest example of this type of operation is transposing a matrix; to transpose a matrix, simply use the T attribute of an array object:
Can you write Python code for it?

x = np.array([[1,2], [3,4]])
print(x)
print(x.T)

# Note that taking the transpose of a rank 1 array does nothing:
v = np.array([1,2,3])
print(v)
print(v.T)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Array indexing
Numpy offers several ways to index into arrays.
Slicing
numpy arrays can also be sliced, similar to Python lists. Since arrays may be multidimensional, slice for each dimension of the array needs to be provided.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Array iteration
Just like normal array np array's can also be iterated
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Array math
In numpy standard mathematical operations when used with arrays, operate at element level, which makes it mandetory that the  arrays  should  be  the  same  size.
Can you write Python code for it?

import numpy as np

x = np.array([[1,2],[3,4]], dtype=np.float64)
y = np.array([[5,6],[7,8]], dtype=np.float64)
print(x)
print(y)

print(x + y)
print(np.add(x, y))

print(x - y)
print(np.subtract(x, y))

print(x * y)
print(np.multiply(x, y))

print(x / y)
print(np.divide(x, y))

print(np.sqrt(x))

am = np.array([[3,5,7,9], [2, 4, 5, 6], [12, 14, 15, 6]], float)
print(am)
print(am.shape)

am1 = np.array([[3,5,7,9], [2, 4, 5, 6]], float)
print(am1)
print(am1.shape)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Array shape manipulation
Flattening
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Arrays or N-Dimension Array
One of tools provided by NumPy is ndarray. It is high-performance multidimensional array object that is a powerful data structure for efficient computation of arrays and matrices. To work with these arrays, there’s a huge amount of high-level mathematical functions operate on these matrices and arrays.
In other words, numpy array is a n dimentional grid of same data type values and is indexed by a tuple of non-negative integers.

Rank of the array: The number of dimenstions
Shape of an array: Tuple of integers giving the size of the array along each dimension.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Basic reductions
Computing sums
Can you write Python code for it?

a = np.array(range(10))
print(a)

print(a.sum())


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Boolean array indexing:
Boolean array indexing lets you pick out arbitrary elements of an array. Frequently this type of indexing is used to select the elements of an array that satisfy some condition. Here is an example:
Can you write Python code for it?

import numpy as np

a = np.array([[1,2], [3, 4], [5, 6]])

bool_idx = (a > 2)   # Find the elements of a that are bigger than 2;
                     # this returns a numpy array of Booleans of the same
                     # shape as a, where each slot of bool_idx tells
                     # whether that element of a is > 2.

print(bool_idx)

# We use boolean array indexing to construct a rank 1 array
# consisting of the elements of a corresponding to the True values
# of bool_idx
print(a[bool_idx])

# We can do all of the above in a single concise statement:
print(a[a > 2])

am = np.array([[3,5,7,9], [2, 4, 5, 6.101]])
bool_idx2 = (am > 5)
print(bool_idx2)
print(am[bool_idx2])

am = np.array([[3,5,7,9], [2, 4, 5, 6]], int)
bool_idx2 = (am >= 5)
print(bool_idx2)
print(am[bool_idx2])

print(am[am >= 5])


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Broadcasting two arrays together follows these rules:

If the arrays do not have the same rank, prepend the shape of the lower rank array with 1s until both shapes have the same length.
The two arrays are said to be compatible in a dimension if they have the same size in the dimension, or if one of the arrays has size 1 in that dimension.
The arrays can be broadcast together if they are compatible in all dimensions.
After broadcasting, each array behaves as if it had shape equal to the elementwise maximum of shapes of the two input arrays.
In any dimension where one array had size 1 and the other array had size greater than 1, the first array behaves as if it were copied along that dimension

Functions that support broadcasting are known as universal functions. The list of all universal functions is listed at http://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs
Can you write Python code for it?

import numpy as np

# Compute outer product of vectors
v = np.array([1,2,3])  # v has shape (3,)
w = np.array([4,5])    # w has shape (2,)
# To compute an outer product, we first reshape v to be a column
# vector of shape (3, 1); we can then broadcast it against w to yield
# an output of shape (3, 2), which is the outer product of v and w:
# [[ 4  5]
#  [ 8 10]
#  [12 15]]
print(np.reshape(v, (3, 1)) * w)

# Add a vector to each row of a matrix
x = np.array([[1,2,3], [4,5,6]])
# x has shape (2, 3) and v has shape (3,) so they broadcast to (2, 3),
# giving the following matrix:
# [[2 4 6]
#  [5 7 9]]
print(x + v)

# Add a vector to each column of a matrix
# x has shape (2, 3) and w has shape (2,).
# If we transpose x then it has shape (3, 2) and can be broadcast
# against w to yield a result of shape (3, 2); transposing this result
# yields the final result of shape (2, 3) which is the matrix x with
# the vector w added to each column. Gives the following matrix:
# [[ 5  6  7]
#  [ 9 10 11]]
print((x.T + w).T)
# Another solution is to reshape w to be a column vector of shape (2, 1);
# we can then broadcast it directly against x to produce the same
# output.
print(x + np.reshape(w, (2, 1)))

# Multiply a matrix by a constant:
# x has shape (2, 3). Numpy treats scalars as arrays of shape ();
# these can be broadcast together to shape (2, 3), producing the
# following array:
# [[ 2  4  6]
#  [ 8 10 12]]
print(x * 2)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Dimension shuffling
Can you write Python code for it?

a = np.arange(4*3*2).reshape(4, 3, 2)
print(a)
print(a.shape)

b = a.transpose(1, 2, 0)
print(b)
print(b.shape)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Extrema (min, max, etc)
Can you write Python code for it?

import numpy as np
b = np.random.rand(3, 1, 2)*10
print(b)

b.min()

b.max()

b.argmin()  # index of minimum

b.argmax()  # index of max


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
In the above examples am and am1 are of different shapes, thus any maths operation performed will result in failure as shown in the below maths operations.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Initialize ndarray
numpy arrays can be initialize using nested Python lists, and elements can be accessed using square brackets, similar to python nested lists.
Can you write Python code for it?

import numpy as np

a = np.array(range(2,20,4))   # Create a rank 1 array
print(type(a))            # Prints "<class 'numpy.ndarray'>"
print(a.shape)

for i in a:
    print(i, end=", ")
print()

b = np.array([[2, 4, 6],
              [11, 13, 15]])    # Create a rank 2 array
print(b.shape)
print(b[0, 0], b[0, 1], b[1, 0])

c = np.array([[[2, 4], [6, 8]],
              [[11, 13], [15, 17]],
              [[111, 113], [115, 117]]])    # Create a rank 3 array
print(c.shape)

b = np.array([[2, [4], 6],
              [11, [13], 15]])


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Integer array indexing:
When you index into numpy arrays using slicing, the resulting array view will always be a subarray of the original array. In contrast, integer array indexing allows you to construct arbitrary arrays using the data from another array. Here is an example:
Can you write Python code for it?

a = np.array([[1, 2], [3, 4], [5, 6]])
print(a)
print("*"*20)
# print(a.shape)
# print("*"*20)
# An example of integer array indexing.
# The returned array will have shape (3,) and
print(a[[0, 1, 2], [0, 1, 0]])  # Prints "[1 4 5]"
print("*"*20)
print(a[0, 0])
print(a[1, 1])
print(a[2, 0])
print("*"*20)
# The above example of integer array indexing is equivalent to this:
print(np.array([a[0, 0], a[1, 1], a[2, 0]]))  # Prints "[1 4 5]"
print("*"*20)
# When using integer array indexing, you can reuse the same
# element from the source array:
print(a[[0, 0], [1, 1]])  # Prints "[2 2]"
print("*"*20)
# Equivalent to the previous integer array indexing example
print(np.array([a[0, 1], a[0, 1]]))  # Prints "[2 2]"


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Logical operations
Can you write Python code for it?

np.all([True, True, False])

np.all([False, False, False])

np.all([True, True, True])

np.any([True, True, False])

np.any([False, False, False])

a = np.zeros((100, 100))
print(a)

print(np.any(a != 0))
print(np.any(a == 0))

np.all(a == a)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Maths constants such as pi are also provided
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Now lets try to do get the sum for n-dimension array
sum by rows and by columns
Can you write Python code for it?

x = np.array([[2, 4], [6, 8]])
print(x)
print(x.sum())

print(x.sum(axis=0))

print(x.sum(axis=1))


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
NumPy Array Comparisons
NumPy makes it possible to test to see if rows match certain values using mathematical comparison operations like <, >, >=, <=, and ==.
Can you write Python code for it?

print(a)

a > 67

a = np.array([1, 3, 5, 7])
b = np.array([1, 3, 5, 7])
c = np.array([1, 3, 5, 2])

print(a == b)
print(c == b)
print(c == a)

print(a > b)
print(c > b)
print(c > a)

print(a < b)
print(c < b)
print(c < a)

print(a >= b)
print(c >= b)
print(c >= a)

print(a <= b)
print(c <= b)
print(c <= a)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Numpy also offers large  library  of  common  mathematical functions  that  can  be  applied elementwise to  arrays.
Some of the other maths related functions are as follows:
abs, sign, sqrt, exp, (log, log10), (sin,  cos,  tan,  arcsin,  arccos, arctan), (sinh, cosh, tanh, arcsinh, arccosh and arctanh).
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Numpy also provide functions to find the lower, upper, or nearest (rounded) integer through floor, ceil, and rint
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Reshaping
Can you write Python code for it?

print(a.shape)
b = a.ravel()
print(b)
b = b.reshape((2, 3))
print(b)
print( b.reshape((3,2)))
print( b.reshape((1,6)))
print( b.reshape((6, 1)))

print( b.reshape((2,2))) # should error out

# Provide the reason for the error

b[0, 0] = 9911
print(a)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Reshaping NumPy Arrays
We can change the shape of arrays while still preserving all of their elements. This often can make it easier to access array elements. The simplest reshaping is to flip the axes, so rows become columns, and vice versa. We can accomplish this with the numpy.transpose function:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Resizing
Can you write Python code for it?

a = np.arange(4)
print(a)

a.resize((8,))
print(a)

### NOTE: However, it must not be referred to somewhere else:
b = a
a.resize((6,))


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Sorting data
Can you write Python code for it?

a = np.array([[4, 3, 5], [1, 2, 1]])

b = np.sort(a, axis=1)
print(b)

# in-place sorting
# Note: Sorts each row separately!

a.sort(axis=1)
print(a)

a.sort(axis=0)
print(a)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Statistics
Can you write Python code for it?

x = np.array(np.random.rand(2, 2, 3)*10)
y = np.array(np.random.rand(2, 2, 3)*10)
print(x)
print(y)
print(x.mean())
print(y.mean())

print(np.median(x))
print(np.median(y))

print(np.median(y, axis=-1))

x.std()          # full population standard dev.


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Subsetting
One of the powerful things we can do with a Boolean array and a NumPy array is select only certain rows or columns in the NumPy array
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
Transposition
Can you write Python code for it?

a = np.triu(np.ones((4,4)), 1)
print(a)

print(a.T)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
above failed as all the elements were not of same data type
Other ways to create narray
Numpy also provides many other functions to create arrays
Can you write Python code for it?

a = np.zeros((2,4))   # Create an array of all zeros
print(a)

print("*"*30)
b = np.ones((1, 4 , 2))    # Create an array of all ones
print(b)

print("*"*30)
c = np.full((2,5), 101)  # Create a constant array
print(c)

print("*"*30)
d = np.eye(5)         # Create an identity matrix
print(d)

print("*"*30)
e = np.random.random((1, 4))  # Create an array filled with random values
print(e)

print("*"*30)
e = np.random.random((2, 4))  # Create an array filled with random values
print(e)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
and the below code will work without any issue as smaller array can be repeated on the larger array.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
broadcasting
Numpy broadcasting allows us to perform this computation without actually creating multiple copies of v. Consider this version, using broadcasting:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
but, following example will fail.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
reference  - https://docs.scipy.org/doc/numpy/reference/routines.array-manipulation.html
Broadcasting
Broadcasting is a powerful mechanism that allows numpy to work with arrays of different shapes when performing arithmetic operations. Frequently we have a smaller array and a larger array, and we want to use the smaller array multiple times to perform some operation on the larger array.
For example, suppose that we want to add a constant vector to each row of a matrix. We could do it like this:
Can you write Python code for it?

x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = np.array([1, 0, 1])
y = np.empty_like(x)   # Create an empty matrix with the same shape as x

# Add the vector v to each row of the matrix x with an explicit loop
for i in range(4):
    y[i, :] = x[i, :] + v

# Now y is the following
# [[ 2  2  4]
#  [ 5  5  7]
#  [ 8  8 10]
#  [11 11 13]]
print(y)

y2 = x + v
print(x)
print(v)
print(y2)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
the last example does not look that convinent, so lets try the following one
Can you write Python code for it?

print(b[1:2, 0])
print(b[1:2, :1])
print(b[1:2, ::2])
print(b[1:2, :])

ar = np.random.random((4, 2, 3))
print(ar)
print(ar.shape)

t1 = ar[1: 2]
print(t1)
print(t1.shape)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/Numpy ver - 0.0.2.ipynb".
The first task is:
the numpy.reshape function to reshape an array to a certain shape we specify. The below code will turn the second row of wines into a 2-dimensional array with 2 rows & 3 cols
Can you write Python code for it?

c = b.reshape((3,2))
print(c)
print(c.shape)

c = b.reshape((4,2))
print(c)
print(c.shape)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
A number of functions allow testing of the values in an array. The nonzero function gives a
tuple of indices of the nonzero values in an array. The number of items in the tuple equals the
number of axes of the array:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Array Functions
All the functions available are as follows
'all', 'any', 'argmax', 'argmin', 'argpartition', 'argsort', 'astype', 'base', 'byteswap', 'choose', 'clip', 'compress', 'conj', 'conjugate', 'copy', 'ctypes', 'cumprod', 'cumsum', 'data', 'diagonal', 'dot', 'dtype', 'dump', 'dumps', 'fill', 'flags', 'flat', 'flatten', 'getfield', 'imag', 'item', 'itemset', 'itemsize', 'max', 'mean', 'min', 'nbytes', 'ndim', 'newbyteorder', 'nonzero', 'partition', 'prod', 'ptp', 'put', 'ravel', 'real', 'repeat', 'reshape', 'resize', 'round', 'searchsorted', 'setfield', 'setflags', 'shape', 'size', 'sort', 'squeeze', 'std', 'strides', 'sum', 'swapaxes', 'take', 'tobytes', 'tofile', 'tolist', 'tostring', 'trace', 'transpose', 'var', 'view'
We are going to cover few of them in coming sections. I have tried to place them in logical sections for easy understanding.
Following are used to manage the arguments
- 'argmax',
- 'argmin',
- 'argpartition',
- 'argsort'
astype
Can you write Python code for it?

am.base

am.choose

am.clip(3)

am.clip(1, 4)

am.compress([2,2])

am.conjugate()


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Array item selection and manipulation
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Array iteration
Just like normal array np array's can also be iterated
Can you write Python code for it?

for i in a:
    print(i, "....")
#     print("---")

for i in am:
    print(i, "....")
#     print("---")


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Arrays can be of multiple dimensions, following examples are of type multiple dimension
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Basic array operations
The argmin and argmax functions return the array indices of the minimum and maximum values
Can you write Python code for it?

#### max
print(a.max())

#### argmax
print(a.argmax())

#### min
print(a.min())

#### argmax
print(a.argmin())

# unique, Unique elements can be extracted from an array
np.unique(a)

#### sorting , sorting can happed on only 1-D arrays

print(sorted(a))

ad1 = np.array([3,4,52,23,43,2], int)

print(sorted(ad1))


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
But, arrays that do not match in the number of dimensions will be broadcasted by Python to perform mathematical operations.  This often means that the smaller array will be repeated
as necessary to perform the operation indicated
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Compound Boolean expressions can also be used on the array elements using the following functions,
logical_and
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Data Type Objects (dtype)
A data type object represent, fixed block of memory corresponding to an array, depending on the following aspects:
* Size of data
* Type of data (integer, float or Python object)
* Byte order (little-endian or big-endian)
* In case of custom type, the name & data type of each field and part of the memory block taken by each field
* If data type is a subarray, its shape and data type
Numpy supports following data types, some of them might not be present in python.
Data Types|Description|Range|
:-----:|:-----:|:-----:|
bool_|Boolean (True or False) stored as a byte| 0 1
int_|Default integer type (same as C long; normally either int64 or int32)|
intc| Identical to C int (normally int32 or int64)|
intp| Integer used for indexing (same as C ssize_t; normally either int32 or int64)|
int8| Byte | (-128 to 127)
int16| Integer | (-32768 to 32767)
int32| Integer | (-2147483648 to 2147483647)
int64| Integer |(-9223372036854775808 to 9223372036854775807)|
uint8| Unsigned integer |(0 to 255)|
uint16| Unsigned integer |(0 to 65535)|
uint32| Unsigned integer |(0 to 4294967295)|
uint64| Unsigned integer |(0 to 18446744073709551615)|
float_| Shorthand for float64||
float16| Half precision float: sign bit, 5 bits exponent, 10 bits mantissa||
float32| Single precision float: sign bit, 8 bits exponent, 23 bits mantissa| |
float64| Double precision float: sign bit, 11 bits exponent, 52 bits mantissa||
complex_|Shorthand for complex128||
complex64| Complex number, represented by two 32-bit floats (real and imaginary components)||
complex128| Complex number, represented by two 64-bit floats (real and imaginary components)||
built-in data type has a character code that uniquely identifies it.
| Char code     | Data Types                |
|---------- |------------------------   |
| 'b'       | boolean                   |
| 'i'       | (signed) integer          |
| 'u'       | unsigned integer          |
| 'f'       | floating-point            |
| 'c'       | complex-floating point    |
| 'm'       |...
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
For two dimensional arrays, the diagonal can be extracted:
Can you write Python code for it?

ad2 = np.array([[1, 2], [3, 4]], int)
print(ad2.diagonal())


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
In the above examples am and am1 are of different shapes, thus any operation performed on the will result in failure as shown in the below maths operations,
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Maths Operations
In numpy standard mathematical operations when used with arrays, operate at element level, which makes it mandetory that the  arrays  should  be  the  same  size.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Numpy also offers large  library  of  common  mathematical functions  that  can  be  applied elementwise to  arrays.
Some of the other maths related functions are as follows:
abs, sign, sqrt, exp, (log, log10), (sin,  cos,  tan,  arcsin,  arccos, arctan), (sinh, cosh, tanh, arcsinh, arccosh and arctanh).
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Numpy also provide functions to find the lower, upper, or nearest (rounded) integer through floor, ceil, and rint
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
Other ways to create narray
NumPy's arange method is similar to range method and it returns numpy array.
Can you write Python code for it?

np.arange(6, dtype=complex)

np.arange(6, dtype=int)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
and the below code will work without any issue as smaller array can be repeated on the larger array.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
array takes two parameters, list and data type.
The list is converted to narray with the requested data type. lets update the above example for float and observe the changes in the array elements.
Can you write Python code for it?

af = np.array([2, 1, 4, 5, 77777777777], float)
print(af)

a2 = np.array([2, 1, 4, 5, 8])
print(a2)

class T:
    def __init__(self):
        self.x = 1

a3 = np.array([2, 1, 4, 5, 8], T)
print(a3)
print(a3.dtype)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
but, following example will fail.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
for multi-dimensional array, multiple assinations is also available
Can you write Python code for it?

for i, j , k, l in am:
    print(i, j, k, l)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
in order for the 'a' to work they should be something similar to below example
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
lets find all the methods available in array using introspection method dir
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
logical_not
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
ndarray.dtype
and data type of array can be obtained using dtype
Can you write Python code for it?

print(am.dtype)
print(am1.dtype)
print(am2.dtype)
print(a.dtype)
print(af.dtype)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
ndarray.shape
Array dimensions can be found by using function shape
Can you write Python code for it?

print(am.shape)
print(am1.shape)
print(am2.shape)
print(a.shape)
print(af.shape)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
now lets try the same with different length arrays and observe the error message
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
or using the character code as shown in the above tables
Can you write Python code for it?

#int8, int16, int32, int64, ... can be replaced by equivalent string 'i1', 'i2','i4', 'i8' and so on.

dt = np.dtype('i1')
print(dt)
dt = np.dtype('i2')
print(dt)
dt = np.dtype('i4')
print(dt)
dt = np.dtype('i8')
print(dt)

dt = np.dtype([('we will be using this user later in our examples.','a128')])
print(dt)

user=np.dtype([('first_name','S30'), ('last_name','S30'), ('pay', 'i2'), ('designation_id', 'f2')])
print(user)


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
similarly, zero's and ones's can also be used to create narray.
Can you write Python code for it?

np.ones((2,3), dtype='i8')

np.zeros((2,3), dtype=float)

print(type(af[1]))


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
we can use the ndarray as normal array.
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/numpy/numpy.ipynb".
The first task is:
we will be using this user later in our examples.
N-Dimension Array
One of tools provided by NumPy is ndarray. It is high-performance multidimensional array object that is a powerful data structure for efficient computation of arrays and matrices.
To work with these arrays, there’s a huge amount of high-level mathematical functions operate on these matrices and arrays.
In other words, numpy array is a n dimentional grid of same data type values and is indexed by a tuple of non-negative integers.
The number of dimenstions is the rank of the array and
numpy.array(object, dtype=None, copy=True, order=None, subok=False, ndmin=0)
The above constructor takes the following parameters:
| Name      | Description       |
|---    |---    |
| object    |  Any object exposing the array interface method returns an array, or any (nested) sequence    |
| dtype     | Desired data type of array, optional      |
| copy      | Optional. By default (true), the object is copied     |
| order     | C (row major) or F (column major) or A (any) (default)    |
| subok     | By default, returned array forced to be a base class array. If true, sub-classes passed through   |
| ndimin    | Specifies minimum dimensions of resultant array   |
An ndarray can be created from a list:
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/scipy/SciPy.ipynb".
The first task is:
SciPy Organization
| Subpackage    | Description                                               |
|-------------  |--------------------------------------------------------   |
| cluster       | Clustering algorithms                                     |
| constants     | Physical and mathematical constants                       |
| fftpack       | Fast Fourier Transform routines                           |
| integrate     | Integration and ordinary differential equation solvers    |
| interpolate   | Interpolation and smoothing splines                       |
| io            | Input and Output                                          |
| linalg        | Linear algebra                                            |
| ndimage       | N-dimensional image processing                            |
| odr           | Orthogonal distance regression                            |
| optimize      | Optimization and root-finding routines                    |
| signal        | Signal processing                                         |
| sparse        | Sparse matrices and associated routines                   |
| spatial       | Spatial data structures and algorithms                    |
| special       | Special functions                                         |
| stats         | Statistical distributions and functions                   |
Relationship with numpy
As Scipy builds on Numpy, it uses it for all basic array handling requirements. Although scipy also contains functions from numpy and numpy.lib.scimath, but it is better to use them directly from the numpy module.
Index Tricks
Can you write Python code for it?


I am working on the file "Section 3 - Machine Learning/libs/core_libs/scipy/SciPy.ipynb".
The first task is:
Shape manipulation
functions are routines for squeezing out length- one dimensions from N-dimensional arrays, ensuring that an array is at least 1-, 2-, or 3-dimensional, and stacking (concatenating) arrays by rows, columns, and “pages “(in the third dimension). Routines for splitting arrays (roughly the opposite of stacking arrays) are also available.
Polynomials
There are two (interchangeable) ways to deal with 1-d polynomials in SciPy. The first is to use the poly1d class from Numpy. This class accepts coefficients or polynomial roots to initialize a polynomial. The polynomial object can then be manipulated in algebraic expressions, integrated, differentiated, and evaluated. It even prints like a polynomial:
Can you write Python code for it?

from numpy import poly1d

p = poly1d([3,4,5])
print(p)

print(p*p)

print(p.integ(k=6))


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Area ID correction
For some reason most area IDs have an added 0 except for the overseas areas 971, 972, 973, 974, and 976. Corsica is split into the areas 2A and 2B, which are labelled 201 and 202 in this dataframe. We will keep 201 and 202 to keep the column filled with integers. So we will divide by 10 only the values that are multiples of 10.
Can you write Python code for it?


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
First look at the data structure
Can you write Python code for it?

caract_df = pd.read_csv("data/2014_France_caracteristiques.csv",sep=',', encoding='latin-1')

caract_df.head()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Half of the longitude, latitude and the gps variable are missing. Many of the adresses are also missing and we don't have any use for a very precise location so we can drop them. We still have the area code ("dep") for a rough location in France.
Can you write Python code for it?


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
It is more convenient to reclassify the types using categories from 1 to 24:
Can you write Python code for it?

newMap = {}
for i,I in enumerate(clean_df['catv'].value_counts().sort_index().index):
    newMap[I] = i+1
clean_df['catv'] = clean_df['catv'].map(newMap)
clean_df['catv'].value_counts().sort_index()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Let's correct those for clarity sake.
Can you write Python code for it?

def correct_area_id(sr):
    if ( (sr['area id'] %10) == 0):
        sr['area id'] = int(sr['area id']/10)
    return sr
clean_df = clean_df.apply(correct_area_id,axis=1)

clean_df['area id'].unique()

## This is done in the CSVtoSQLconverter.py script
# clean_df.to_sql(name='characteristics', con=sqlEngine, if_exists = 'replace', index=False)


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Locations dataframe
The second csv file contains the details of the accident's location.
Can you write Python code for it?

lieux_df = pd.read_csv("data/2014_France_lieux.csv",sep=',', encoding='latin-1')

lieux_df.head()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Remove columns
Most columns are really useful except for the traffic sense which has mostly zeroes meaning that the sense wasn't recorded. We can remove it.
Can you write Python code for it?

vehicules_df['senc'].value_counts().head()

clean_df = vehicules_df.drop('senc',axis=1)

clean_df.head()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Remove unreliable information
The precise location of the accident has not been reliably recorded so we won't be able for example to get a reliable map of the location of the accidents.
Can you write Python code for it?


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Remove useless information
Let's check which information we can drop.
Can you write Python code for it?


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Rename columns
Again work is needed to clarify the meaning of each column.
Can you write Python code for it?

clean_df.rename(columns={'Num_Acc':'accident id','catr':'road type','circ':'traffic mode',
                         'nbv':'nb lanes','vosp':'reserved lane','prof':'road profil',
                         'plan':'road alignment','lartpc':'central reservation',
                         'larrout':'road width','surf':'road surface','infra':'installations',
                         'situ':'location','env1':'school distance'},
                inplace=True)
clean_df.head()

## This is done in the CSVtoSQLconverter.py script
# clean_df.to_sql(name='locations', con=sqlEngine, if_exists = 'replace', index=False)


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Rename columns
And now the renaming of the columns.
Can you write Python code for it?

clean_df.rename(columns={'Num_Acc':'accident id', 'catv':'vehicle type','obs':'fixed obj hit',
                         'obsm':'moving obj hit', 'choc':'impact location','manv':'maneuver',
                         'occutc':'nb occupants public transit', 'num_veh':'vehicle id'},inplace=True)

clean_df.head()

## This is done in the CSVtoSQLconverter.py script
# clean_df.to_sql(name='vehicles', con=sqlEngine, if_exists = 'replace', index=False)


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Rename the columns
Can you write Python code for it?

clean_df.rename(columns={'Num_Acc':'accident id', 'num_veh':'vehicle id',
                         'place':'location in vehicle','catu':'user type',
                         'grav':'severity','sexe':'sex','trajet':'journey type',
                         'locp':'pedestrian location','actp':'pedestrian action',
                         'etatp':'pedestrian company'},inplace=True)

clean_df.head()

## This is done in the CSVtoSQLconverter.py script
# clean_df.to_sql(name='users', con=sqlEngine, if_exists = 'replace', index=False)


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Reorganize vehicle type categories
The "catv" column contains vehicle types with values between 1 and 99. However only 24 categories are used as we can see here:
Can you write Python code for it?


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Some of the most unreliable columns are fortunately also not very useful since they pertain to the address of the accident which we have already decided to drop.
Can you write Python code for it?

clean_df = lieux_df.drop(['voie','v1','v2','pr','pr1'],axis=1)
clean_df.head()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
The birth year is not that interesting by itself. The age of the users seems more interesting. We can replace the birth year by the age.
Can you write Python code for it?

clean_df['age'] = 2014 - clean_df['an_nais']

clean_df.drop('an_nais',inplace=True,axis=1)
clean_df.head()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
The date is saved in many columns which is not necessarily practical. Also the format for the time is not too handy.
The column labels are quite cryptic and not just because they are in French.
Convert time and date to datetime
Can you write Python code for it?


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
The time of the accident is stored in an integer in 24H format such that 5:20pm becomes 1720.
However some of the strings are "1" or "15" or "29".
One possible explanation is that the recorded number was "029" for 29 minutes past midnight but once converted to an integer it became "29".
We can verify this by plotting the two distributions of values for hrmn > 100 (the hour after midnight) and hrmn >= 2300 (the hour before midnight), and compare them.
Can you write Python code for it?

# The hour after midnight
caract_df[caract_df['hrmn'] < 100]['hrmn'].plot.hist(bins=range(1,60),figsize=(12,6))

# The hour before midnight
caract_df[caract_df['hrmn'] >= 2300]['hrmn'].plot.hist(bins=range(2300,2360),figsize=(12,6))


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
Vehicle dataframe
All the details of the vehicles involved in the accident are listed in this dataframe. This time the same accident ID can appear in more than one entry since each entry is one of the vehicles involved in the accident.
Can you write Python code for it?

vehicules_df = pd.read_csv("data/2014_France_vehicules.csv",sep=',', encoding='latin-1')

vehicules_df.head()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
We can now confidently convert integers < 100 to strings 4-character long and starting with 0.
Can you write Python code for it?

def columns_to_datetime(sr):
    stringall = "%d-%d-%d %04d" % (sr['an'], sr['mois'], sr['jour'], sr['hrmn'])
    sr = sr.append(pd.Series(dt.datetime.strptime(stringall, "%y-%m-%d %H%M"), index=["datetime"]))
    return sr

clean_df = caract_df.apply(columns_to_datetime,axis=1)

clean_df.head()

# Let's check that our conversion worked properly
clean_df[clean_df['hrmn'] == 29]

# We can remove the original time and date columns
clean_df.drop(['an','mois','jour','hrmn'],axis=1, inplace=True)
clean_df.head()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
We should always have the first digit since the value 9 represent 'others' for non-defined types.
Can you write Python code for it?

def safety_to_twocolumns(sr):
    if not np.isnan(sr['secu']):
        safetychar = "%2d" % (sr['secu'])
        if safetychar[0] != ' ':
            sr = sr.append(pd.Series(int(safetychar[0]), index=["safety gear type"]))
        sr = sr.append(pd.Series(int(safetychar[1]), index=["safety gear worn"]))
    return sr

clean_df = usagers_df.apply(safety_to_twocolumns,axis=1)

clean_df.head()

clean_df.drop('secu',inplace=True,axis=1)
clean_df.head()


I am working on the file "Section1 - Data Wrangling.ipynb".
The first task is:
We will keep all the columns of this dataframe since they all contain some interesting data.
Reformat columns
The 'secu' and 'an_nais' columns represent respectively the safety gear used (or not used) and the birth year of each user.
The 'secu' is not too practical to use as it is since it store the type of safety gear in the first digit, and if the safety gear was used or not in the second digit of an integer. It will be simpler to have 2 columns for these two items.
Can you write Python code for it?


I am working on the file "Section2a - Data Visualization.ipynb".
The first task is:
Evolution over the years
Can you write Python code for it?

PerYear = PerMonth.groupby(['year'],as_index=False).sum()
PerYear

g = sns.factorplot(x="year",y='accident count',
                   data=PerYear, kind='bar', size=5, aspect=2.0)


I am working on the file "Section2a - Data Visualization.ipynb".
The first task is:
The 5 years we have follow the same trends overall. We can safely average over them.
Can you write Python code for it?

PerMonthMean = PerMonth.groupby('month', as_index=False).mean()
PerMonthMean['per day'] = PerMonthMean['accident count'] / PerMonthMean['number days']
PerMonthMean

sns.factorplot(x='month', y='per day', data=PerMonthMean, size=5,aspect=2.0)


I am working on the file "Section2b - Data Visualization.ipynb".
The first task is:
Passengers involved in accidents
Can you write Python code for it?

men_women_dist(users_df[(users_df['user type'] == 2) & (users_vt_df['vehicle type'] == 7)] ,
               'All accidents, passengers of cars')


I am working on the file "Section2b - Data Visualization.ipynb".
The first task is:
The best is to have one function drawing the same plots for different input dataframes, i.e. for different conditions.
Can you write Python code for it?

def men_women_dist(in_df, title=''):
    """Plot the age distribution for men and women, and the ratio."""
    # Extract age for each gender
    women_sr = in_df[in_df['sex']==2]['age'].dropna()
    men_sr = in_df[in_df['sex']==1]['age'].dropna()

    # Bin the ages in bins with 2 years of width
    bin_range = range(0,102,2)
    women_bins = pd.cut(women_sr, bin_range,right=False).value_counts().sort_index()
    men_bins = pd.cut(men_sr, bin_range,right=False).value_counts().sort_index()

    # Calculate the ratio between men and women
    ratio_bins = men_bins/women_bins
    error_bins = ratio_bins * np.sqrt((men_bins+women_bins)/(men_bins*women_bins))

    # Plot the stacked distribution of age for men and women
    plt.subplots(figsize=(14,8))
    plt.subplot(211)
    plt.hist([women_sr,men_sr],bins=50,range=(0,100) , stacked=True, normed = False, histtype='bar',
            rwidth=1.0, color=[colors[1],colors[0]]);
    plt.title(title)
    plt.ylabel('counts')
    leg_men = mpatches.Patch(color=colors[0], label='Men')
    leg_women = mpatches.Patch(color=colors[1], label='Women')
    plt.legend(handles=[leg_men,leg_women],frameon=True)

    # Plot the ratio men/women
    plt.subplot(212)
    plt.errorbar(range(1,100,2),ratio_bins,xerr=1.0,yerr=error_bins, fmt='none')
    plt.xlim([0,100])
    plt.xlabel('age in years')
    plt.ylabel('ratio men/women')

men_women_dist(users_df, 'All accidents, all users')


I am working on the file "Section2b - Data Visualization.ipynb".
The first task is:
Who is involved ?
We will start by investigating the Users dataframe to see who is the most, or the least, involved in car accidents. In particular let's look at the age and gender distributions.
Can you write Python code for it?

users_df = pd.read_sql_query('''SELECT * FROM safer_roads.users
                            WHERE `accident id` > 2014E+8 AND `accident id` < 2015E+8''',
                            sqlEngine)
users_df.head()


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
A few different corrections are needed:
1. For "location in vehicle" we can replace the NaN with zeros as a consistent place holder to specify that this wasn't recorded.
1. The entries with NaN values in the "age" column can be safely removed since they represent very few entries. The zeroes in that column should be babies less than one year old.
1. The "journey type", "safety gear worn", and "safety gear type" have a category "unknown" or "other" which we can use to replace the NaN entries. We can also use this category for the entries with zero in journey type.
1. The "pedestrian action" and "pedestrian location" are mostly filled with zeroes indicating that they were not recorded therefore we won't use them as features.
Can you write Python code for it?

# 1. Replace NaN with zeros
users_df['location in vehicle'].fillna(0, inplace=True)
# 2. Remove rows with NaN in age column
users_df.dropna(subset=['age'], inplace=True)
# 3. Replace NaN with "other" or "unknown" categories
users_df['journey type'].fillna(9, inplace=True)
users_df['safety gear type'].fillna(9, inplace=True)
users_df['safety gear worn'].fillna(3, inplace=True)
users_df.replace(to_replace={'journey type':{0:9}}, inplace=True)

check_columns(users_df, ['location in vehicle', 'user type', 'age', 'sex',
                       'journey type', 'safety gear worn','safety gear type',
                       'severity'])

feat_target_df = users_df[['accident id','vehicle id','severity','location in vehicle', 'user type',
                        'age', 'sex', 'journey type', 'safety gear worn','safety gear type']]
feat_target_df.head()


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Create the testing sample
We must convert the categorical variables and rescale the continuous variables using the same scaling factors as for the training sample.
Can you write Python code for it?

testing_df = mldata_df.head(120000).tail(20000)

all_col = testing_df.columns
categ_col = all_col.drop(['severity','age','weight diff'])
testing_df = pd.get_dummies(testing_df,prefix=categ_col, columns=categ_col)
testing_df.head()

testing_df['age'] = testing_df['age'] / max_age
testing_df['weight diff'] = (testing_df['weight diff'] - min_weight_diff) / (max_weight_diff - min_weight_diff)
testing_df[['age','weight diff']].describe().loc[['min','max']]


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Create the training sample
Since we already shuffled the entries in the dataset prior to storing them in the SQL database, we can just use X entries to have a totally random sample for testing. However for the training sample we need to correct for the class imbalance first.
We will use part of the original dataset to create our training sample.
Can you write Python code for it?

raw_training_df = mldata_df.head(100000)
n_sev2 = raw_training_df[raw_training_df.severity==2].shape[0]
print("We have {} entries with severity 2. We need the same amount for the other classes.".format(n_sev2))

list_severities = []
for sev_id in range(1,5):
    list_severities.append(raw_training_df[raw_training_df.severity==sev_id].head(n_sev2))
training_df = pd.concat(list_severities)
plot_severity(training_df)


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Creation of the training and testing samples
This section will explain the code in CreateTrainAndTestSamples.py which will be used in the other parts of section3 to create the training and testing samples out of the dataset in the MySQL dataset.
Can you write Python code for it?

# mldata_df = pd.read_sql_query('SELECT * FROM ml_dataset',sqlEngine)
mldata_df = feat_target_df
mldata_df.head()


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Engineered features
From the existing features we can derive new features to improve the predicting power.
Weight differential
The relative size of the vehicles involved in an accident has a direct impact on the gravity of the accident. A complementary piece of information would be the speed to deduce the momentum however we do not have any data on that.
When going through the dataset user by user, the vehicle information associated to each driver or passenger ('user type' = 1 or 2) correspond to their own vehicle therefore we have no information on the other vehicles in the accident. If the entry is for a pedestrian ('user type' = 3 or 4) then the associated vehicle is the vehicle that hit the pedestrian.
We will create a new column for the weight differential taking into account the two cases:
- for passengers and drivers the difference will be between their vehicle and the heaviest vehicle involved in the accident
- for pedestrian will simply take the weight of the vehicle that hit them
The mapping from 'vehicle type' to a crude estimate of the average vehicle weight in kilograms is stored in the Mapper.py script.
Can you write Python code for it?

from Mapper import Vehicle_Weights
# Frist we map all the vehicle types to the average weight
feat_target_df['weight diff'] = feat_target_df['vehicle type'].map(Vehicle_Weights)
# Then calculate the differential for drivers and passengers
mask = feat_target_df['user type'].isin([1,2])
feat_target_df.ix[mask,'weight diff'] = feat_target_df.groupby('accident id')['weight diff']\
                                        .transform(lambda x: x - x.max())


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Features from the Vehicles dataframe
Can you write Python code for it?

vehicles_df = pd.read_sql_query('SELECT * FROM safer_roads.vehicles',
                            sqlEngine)
vehicles_df.head()

wanted_cols = ['vehicle type', 'fixed obj hit', 'moving obj hit',
              'impact location', 'maneuver']
check_columns(vehicles_df, wanted_cols)


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
For most columns we can again simply replace NaN by 0 which corresponds to an "unknown" category.
The columns "road width" and "installations" have respectively 35% and 90% of zeros so it is preferable to drop them.
Can you write Python code for it?

wanted_cols.remove('road width')
wanted_cols.remove('installations')

for col in wanted_cols[1:]:
    locations_df[col].fillna(0,inplace=True)

wanted_cols.extend(['accident id'])
feat_target_df = feat_target_df.merge(locations_df[wanted_cols],
                                on=['accident id'],how='inner')
feat_target_df.head()


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Imbalance of severity classes
The four classes of severity are very imbalance with the classe 2 (death) being significantly (and fortunately) smaller than the other classes:
Can you write Python code for it?

def plot_severity(df):
    severity_sr = df['severity'].map({1:'Indemn',2:'Dead',3:'Hospitalized injured',4:'Lightly injured'})
    sns.countplot(x='severity', data=DataFrame(severity_sr),
              order=['Indemn','Lightly injured','Hospitalized injured','Dead']);

plot_severity(mldata_df)


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Now our training sample is totally balanced.
Convert categorical variables
In order to process the dataset through machine learning algorithms we must first convert each category into a binary variable. Pretty much all the columns are categories except for the age and the weight differential so we'll remove them from the list of columns that needs to be converted.
Can you write Python code for it?

all_col = training_df.columns
categ_col = all_col.drop(['severity','age','weight diff'])
training_df = pd.get_dummies(training_df,prefix=categ_col, columns=categ_col)
training_df.head()


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Of all the columns containing floats, the age is the only one that can be stored as float.
Can you write Python code for it?

float_col = float_col.drop('age')
for col in float_col:
    feat_target_df[col] = feat_target_df[col].astype(int)


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Randomize entries
In order to facilitate the creation of test and training samples in the next section, we shuffle now the entries in the dataframe prior to storing them in the SQL database.
Can you write Python code for it?

feat_target_df.head()

feat_target_df = feat_target_df.reindex(np.random.permutation(feat_target_df.index))
feat_target_df.head()


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Remove irrelevant columns
We do not need the accident and vehicle IDs anymore since we have created the dataset and don't need to relate the rows with one another anymore, so we can remove them.
Can you write Python code for it?

feat_target_df.drop(['accident id','vehicle id'],axis=1,inplace=True)
feat_target_df.head()


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Rescale the continuous variables
Many machine learning algorithms will performed better if all the variables have the sample scale. At the moment the categorical variables take values 0 and 1. The weight differential range on the other hand is few orders of magnitude larger.
Can you write Python code for it?

training_df[['age','weight diff']].describe().loc[['min','max']]

max_age = training_df['age'].max()
training_df['age'] = training_df['age'] / max_age

max_weight_diff = training_df['weight diff'].max()
min_weight_diff = training_df['weight diff'].min()
training_df['weight diff'] = (training_df['weight diff'] - min_weight_diff) / (max_weight_diff - min_weight_diff)

training_df[['age','weight diff']].describe().loc[['min','max']]


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
Target and Features from the Users dataframe
We start with the Users dataframe because each entry in our features dataframe will correspond to one road user involved in an accident and have the target.
Can you write Python code for it?

users_df = pd.read_sql_query('SELECT * FROM safer_roads.users',
                            sqlEngine)
users_df.head()


I am working on the file "Section3a - Predicting the severity.ipynb".
The first task is:
We keep the original numbers of entries in this dataframe to know how much we lost overall during the preparation of the dataset.
Can you write Python code for it?

original_nb_entries = users_df.shape[0]

pd.options.display.float_format = '{:20,.2f}'.format

def check_columns(dataf,columns):
    zeroes_sr = (dataf[columns] == 0).astype(int).sum(axis=0)
    neg_sr = (dataf[columns] < 0).astype(int).sum(axis=0)
    nans_sr = dataf[columns].isnull().sum()

    nbent = dataf.shape[0]
    rows = []
    rows.append(DataFrame(zeroes_sr,columns=['nb zeroes']))
    rows.append(DataFrame((100*zeroes_sr/nbent),columns=['% zeroes']))
    rows.append(DataFrame(neg_sr,columns=['nb neg. vals']))
    rows.append(DataFrame((100*neg_sr/nbent),columns=['% neg. vals']))
    rows.append(DataFrame(nans_sr,columns=['nb nans']))
    rows.append(DataFrame((100*nans_sr/nbent),columns=['% nans']))

    checks_df = pd.concat(rows,axis=1)
    print(' - Total number of entries: {}\n'.format(dataf.shape[0]))
    print('List of NaN, zero and negative entries:\n{}\n\n'.format(checks_df))

check_columns(users_df, ['location in vehicle', 'user type', 'age', 'sex',
                       'journey type', 'safety gear worn','safety gear type',
                       'pedestrian action', 'pedestrian location','severity'])


I am working on the file "Section_0_Introduction.ipynb".
The first task is:
<hr>

Bokeh
http://bokeh.pydata.org/en/latest/
Bokeh is a Python interactive visualization library that targets modern web browsers for presentation.
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://bokeh.pydata.org/en/latest/', width='100%', height=350)