I am working on the file "Curves.ipynb".
The first task is:
Poligonul de control este succesiunea de segmente ce unesc cate doua puncte consecutive, numite puncte de control.
Bézier a definit curbele care-i poarta numele, definind curbe  polinomiale, adica curbe de forma:
$$\left{\begin{array}{lll}x(t)&=&a_0+a_1 t+\cdots+ a_n t^n\
y(t)&=&c_0+c_1t+\cdots+c_n t^n\end{array}\right.\quad t\in[a,b]$$
dar nu in acest fel, adica nu exprimand polinoamele in baza canonica $1,t, \ldots, t^n$, ci in baza Bernstein,
$B^n_0(t), B^n_1(t), \ldots, B_n^n(t)$, unde $B^n_k(t)=C_n^k t^k (1-t)^{n-k}$, $k=\overline{0,n}$.
O curba Bézier de puncte de control ${\bf b}_0, {\bf b}_1, \ldots, {\bf b}_n$, este o curba avand parametrizarea $b(t)=B^n_0(t) {\bf b}_0+B^n_1(t) {\bf b}_1+\cdots+B^n_n(t) {\bf b}_n$, $t\in[0,1]$.
Daca $b(t)=(x(t), y(t))$, atunci coordonatele parametrizarii sunt:
$$
\left{\begin{array}{lll}
x(t)&=&x({\bf b}_0)B^n_0(t)+x({\bf b}_1)B^n_1(t)+\cdots+x({\bf b}_n)B^n_n(t)\
y(t)&=&y({\bf b}_0)B^n_0(t)+y({\bf b}_1)B^n_1(t)+\cdots+y({\bf b}_n)B^n_n(t)\end{array}\right.$$
unde  $x({\bf b}_k), y({\bf b}_k)$ semnifica  coordonata $x$, respectiv $y$, a punctului ${\bf b}_k$, $k=\overline{0,n}$.
Curbele Bézier  folosite in grafica, in modelarea geometrica a diverse produse sau in designul fonturilor
sunt curbele Bézier de grad 3, adica curbele Bézier definite de 4 puncte de control
${\bf b}_0$, ${\bf b}_1$, ${\bf b}_2$, ${\bf b}_3$.
De exemplu curba Bézier definita
de punctele de control:
    $${\bf b}_0=(2,-1),{\bf b}_1 =(4,5), {\bf b}_2=(7, 6), {\bf b}_3 =(9,1)$$
are parametrizarea $$b(t)=(x(t), y(t)), \:\:\left{\begin{array}{lll}
x(t)&=& 2B^3_0(t)+4B^3_1(t)+7B^3_2(t)+9B^3_3(t)\
y(t)&=& -1B^3_0(t)+5B^3_1(t)+6B^3_2(t)+B^3_3(t)\end{array}\right.\quad t\in[0,1]$$
Spre deosebire de Bézier  care a definit curbele ce-i poarta numele printr-o parametrizare, de Casteljau a dat o definitie
procedurala a acestor curbe.
Folosind definitia lui Bézier, adica parametrizarea  $b(t)=\sum_{k=0}^n {\bf b}_{k} B^n_k(t)$ a curbei, pentru...
Can you write Python code for it?


I am working on the file "Curves.ipynb".
The first task is:
Proprietati ale curbelor Bezier

Parametrizarea in baza Bernstein a unei curbe Bézier,
$b(t)=B^n_0(t) {\bf b}_0+B^n_1(t) {\bf b}_1+\cdots+B^n_n(t) {\bf b}_n$, $t\in[0,1]$,
este o combinatie convexa a punctelor de control, deoarece pentru fiecare $t\in[0,1]$,
polinoamele Bernstein, $B^n_k(t)=C_n^k t^k(1-t)^{n-k}\geq 0$ si suma lor este egala cu 1.

Intradevar:
$$\sum_{k=0}^n B_k^n(t)=\underbrace{\sum_{k=0}^n C_n^k t^k(1-t)^{n-k}=(t+(1-t))^n}_{formula\:\: binomului\:\: lui\:\: Newton}=1$$
Deci o curba Bézier este inclusa in infasuratoarea convexa a punctelor sale de control.
Aceasta proprietate are importanta in grafica, pentru a sti unde anume in plan este plasata curba. In exemplul de mai sus am exploatat deja aceasta proprietate, indicand ca dreptunghi de desen, un dreptunghi care include toate punctele de control, deci si infasuratoarea lor convexa.
Can you write Python code for it?

b=np.array([[0,0], [1.5, 2.3], [3.7, -1], [6, 3], [2.6, 2]],float)
plt.subplot(3,1,1)
plt.title('Curba Bezier raportata relativ la poligonul de control')
DrawBezier(b)

plt.subplot(3,1,2)
plt.title('Infasuratoarea convexa a punctelor de control')
cmin, cmax=dreptunghiDesen(b)
plt.axis([cmin[0], cmax[0], cmin[1], cmax[1]])
Vx=[]
Vy=[]
for i in [0,1,3,2, 0]:
    Vx.append(b[i,0])
    Vy.append(b[i,1])
plt.fill(Vx,Vy, color='#99FFCC', edgecolor='g')
plt.plot(b[:, 0], b[:,1],  'bo')

plt.subplot(3,1,3)
plt.title('Curba Bezier, inclusa in infasuratoarea punctelor de control')
plt.fill(Vx,Vy, color='#99FFCC')
DrawBezier(b)

plt.tight_layout(0.75)# aceasta functie include 0.75* spatiu intre figurile din subplots


I am working on the file "Curves.ipynb".
The first task is:
Remarcam ca fara nici o comanda speciala, o data cu  apelul functiei plt.polar sunt generate cercuri
concentrice cu originea in pol si in lungul cercului exterior sunt marcate valorile in grade (nu radiani)
ale unghiului polar $\theta$. Sunt afisate si razele cercurilor concentrice.
Cercul cu centrul in origine si de raza $a$, $x^2+y^2=a^2$, are in coordonate polare ecuatia:
$$r=a,$$
iar semidreapta  ce porneste din origine si formeaza cu axa polara unghiul $\theta_0$ are ecuatia $$\theta=\theta_0$$
Cu alte cuvinte, cercul $r=a$ este locul geometric al punctelor din plan ce au aceeasi distanta polara la $O$, distanta egala cu $a$. Semidreapta $\theta=\theta_0$ este locul geometric al punctelor din plan care au aceeasi coordonata polara $\theta_0$.
Can you write Python code for it?

#cercul r=2

theta=np.arange(0, 2*np.pi, 0.01)
r=2*np.ones(theta.size)
plt.subplot(1,2,1, polar='true')
plt.plot(theta, r, 'r', lw=2)

#semidreapta theta= 2 pi/3
r=np.arange(0,2, 0.01)
theta=(2*np.pi/3)*np.ones(r.size)

plt.subplot(1,2,2, polar='true')
plt.plot(theta, r, 'r', lw=2)
plt.tight_layout(2)# aceasta functie include 2 spatii intre figurile din subplots


I am working on the file "Curves.ipynb".
The first task is:
Trifoiul cu 4 foi are  ecuatia $r=\sin(4\theta)$, $\theta\in[0,2\pi]$.
Can you write Python code for it?

theta=np.arange(0,2*np.pi, 0.01)
plt.polar(theta, np.sin(4*theta), 'r')
plt.axis('off')# efectul acestui apel este suspendarea cercurilor si semidreptelor reperului polar


I am working on the file "Customer Churn Modeling.ipynb".
The first task is:
Confusion Matrix
Can you write Python code for it?

cm = perf_stack_test.confusion_matrix()
cm = cm.table.as_data_frame()
cm
confusionMatrix = ff.create_table(cm)
confusionMatrix.layout.height=300
confusionMatrix.layout.width=800
confusionMatrix.layout.font.size=17
py.iplot(confusionMatrix)


I am working on the file "Customer Churn Modeling.ipynb".
The first task is:
Leaderboard
Can you write Python code for it?


I am working on the file "Customer Churn Modeling.ipynb".
The first task is:
Super Learner
The super learner is a prediction method designed to find the optimal combination of a collection of prediction algorithms. The super learner algorithm finds the combination of algorithms minimizing the cross-validated risk. The super learner framework is9 built in the theory of cross-validation and allows for a general class of prediction algorithms to be considered for the ensemble. http://biostats.bepress.com/ucbbiostat/paper266/ (Polley & Van der Laan, 2010)
Can you write Python code for it?

# Reset variables
# del allModels, gridGBM, gridRF, grids, dfGridGBM, dfGridRF, ensemble

# Reset variables
# del SuperModel, BestModel, Model3, Model4, Model5, Model6, Model7, Model8, Model9, Model10

%%time
# GBM hyperparameters
nfolds = 5
gbm_hyper_params = {"learn_rate":[0.075, 0.1], "nbins":[10,15,20],"ntrees": [20,30,40], "max_depth": [5,7,9], "sample_rate": [0.75, 0.8, 0.85, 0.9]}
search_criteria = {"strategy": "RandomDiscrete", "max_models": 6}

# Setup the GBM grid search
gridGBM = H2OGridSearch(model=H2OGradientBoostingEstimator(balance_classes=True, seed=123, nfolds=nfolds, fold_assignment="Modulo", keep_cross_validation_predictions=True),
                     hyper_params=gbm_hyper_params,
                     search_criteria=search_criteria,
                     grid_id="gbm_grid_binomial")

# Start the GBM training
gridGBM.train(x=x, y=y, training_frame=train)

# Random Forest hyperparameters
rf_hyper_params = {"mtries":[12,15,18],"nbins":[10,20,30], "ntrees": [25,50,75], "max_depth": [5,7], "sample_rate": [0.75, 0.8, 0.85, 0.9]}
gridRF = H2OGridSearch(model=H2ORandomForestEstimator(balance_classes=True, seed=123, nfolds=nfolds, fold_assignment="Modulo", keep_cross_validation_predictions=True),
                      hyper_params=rf_hyper_params,
                      search_criteria=search_criteria,
                      grid_id="rf_grid_binomial")

# Start the Random Forest training
gridRF.train(x=x, y=y, training_frame=train)

# List the GBMs and Random Forests that we wish to ensemble
grids = gridGBM.model_ids + gridRF.model_ids

# Train the super learner
ensemble = H2OStackedEnsembleEstimator(model_id="GBM-RF-ensemble", base_models=grids, training_frame=train, validation_frame=test)
ensemble.train(x=x, y=y, training_frame=train)

# Evaluate ensemble performance on the test data
perf_stack_test = ensemble.model_performance(test)

# Compare the super learner to the base learners. First, combine all the base models into a single list, sorted by...


I am working on the file "Customer Churn Modeling.ipynb".
The first task is:
Variable Importances
Below we plot variable importances as reported by the best performing algo in the ensemble.
Can you write Python code for it?

best = h2o.get_model(baselearner_best_name)

importances = best.varimp(use_pandas=True)
importances = importances.loc[:,['variable','relative_importance']].groupby('variable').mean()
importances.sort_values(by="relative_importance", ascending=False).iplot(kind='bar', colors='#5AC4F2', theme='white')


I am working on the file "Customer Churn with AutoML.ipynb".
The first task is:
Best Model vs the Base Learners
This plot shows the ROC curves for the Super Model, the Best Base Model, and 9 next best models in the ensemble.
Can you write Python code for it?

Model0 = np.array(h2o.get_model(leaders[0,0]).roc(xval=True))
Model1 = np.array(h2o.get_model(leaders[1,0]).roc(xval=True))
Model2 = np.array(h2o.get_model(leaders[2,0]).roc(xval=True))
Model3 = np.array(h2o.get_model(leaders[3,0]).roc(xval=True))
Model4 = np.array(h2o.get_model(leaders[4,0]).roc(xval=True))
Model5 = np.array(h2o.get_model(leaders[5,0]).roc(xval=True))
Model6 = np.array(h2o.get_model(leaders[6,0]).roc(xval=True))
Model7 = np.array(h2o.get_model(leaders[7,0]).roc(xval=True))
Model8 = np.array(h2o.get_model(leaders[8,0]).roc(xval=True))
Model9 = np.array(h2o.get_model(leaders[9,0]).roc(xval=True))

layout = go.Layout(autosize=False, width=725, height=575,  xaxis=dict(title='False Positive Rate', titlefont=dict(family='Arial, sans-serif', size=15, color='grey')),
                                                           yaxis=dict(title='True Positive Rate', titlefont=dict(family='Arial, sans-serif', size=15, color='grey')))

traceChanceLine = go.Scatter(x = [0,1], y = [0,1], mode = 'lines+markers', name = 'chance', line = dict(color = ('rgb(136, 140, 150)'), width = 4, dash = 'dash'))
Model0Trace = go.Scatter(x = Model0[0], y = Model0[1], mode = 'lines', name = 'Model 0', line = dict(color = ('rgb(26, 58, 126)'), width = 3))
Model1Trace = go.Scatter(x = Model1[0], y = Model1[1], mode = 'lines', name = 'Model 1', line = dict(color = ('rgb(156, 190, 241))'), width = 1))
Model2Trace = go.Scatter(x = Model2[0], y = Model2[1], mode = 'lines', name = 'Model 2', line = dict(color = ('rgb(156, 190, 241)'), width = 1))
Model3Trace = go.Scatter(x = Model3[0], y = Model3[1], mode = 'lines', name = 'Model 3', line = dict(color = ('rgb(156, 190, 241)'), width = 1))
Model4Trace = go.Scatter(x = Model4[0], y = Model4[1], mode = 'lines', name = 'Model 4', line = dict(color = ('rgb(156, 190, 241)'), width = 1))
Model5Trace = go.Scatter(x = Model5[0], y = Model5[1], mode = 'lines', name = 'Model 5', line = dict(color = ('rgb(156, 190, 241)'), width = 1))
Model6Trace = go.Scatter(x = Model6[0], y = Model6[1],...


I am working on the file "Customer Churn with AutoML.ipynb".
The first task is:
Leaderboard
Can you write Python code for it?


I am working on the file "Customer Churn with AutoML.ipynb".
The first task is:
Load The Trifacta Prepared Dataset
Can you write Python code for it?

accessKey = os.environ['BRETT_AWS_ACCESS_KEY']
s3file = 'https://trifactapro.s3.amazonaws.com/churn.csv?AWSAccessKeyId=' + accessKey + '&Expires=1522155539&Signature=Imj9nbjqsdarjbAGKkjeHB9PwWE%3D'

# Load some data
churnDF = pd.read_csv(s3file, delimiter=',')
churnDF["Churn"] = churnDF["Churn"].replace(to_replace=False, value='Retain')
churnDF["Churn"] = churnDF["Churn"].replace(to_replace=True, value='Churn')
churnDFs = churnDF.sample(frac=0.07) # Sample for speedy viz
churnDF.head(5)

%%capture
pandas_profiling.ProfileReport(churnDF)


I am working on the file "Customer Churn with AutoML.ipynb".
The first task is:
Scatterplot Matrix
Can you write Python code for it?

# separate the calls data for plotting

churnDFs = churnDFs[['Account Length','Day Calls','Eve Calls','CustServ Calls','Churn']]

# Create scatter plot matrix of call data
splom = ff.create_scatterplotmatrix(churnDFs, diag='histogram', index='Churn',
                                  colormap= dict(
                                      Churn = '#9CBEF1',
                                      Retain = '#04367F'
                                      ),
                                  colormap_type='cat',
                                  height=560, width=650,
                                  size=4, marker=dict(symbol='circle'))
py.iplot(splom)

%%capture

#h2o.connect(ip="35.225.239.147")
h2o.init(nthreads=1, max_mem_size="768m")

%%capture

# Split data into training and testing frames

from sklearn import cross_validation
from sklearn.model_selection import train_test_split

training, testing = train_test_split(churnDF, train_size=0.8, stratify=churnDF["Churn"], random_state=9)
train = h2o.H2OFrame(python_obj=training).drop("State")
test = h2o.H2OFrame(python_obj=testing).drop("State")

# Set predictor and response variables
y = "Churn"
x = train.columns
x.remove(y)


I am working on the file "Customer Churn with AutoML.ipynb".
The first task is:
Variable Importances
Below we plot variable importances as reported by the best performing algo in the ensemble.
Can you write Python code for it?

importances = h2o.get_model(leaders[2, 0]).varimp(use_pandas=True)
importances = importances.loc[:,['variable','relative_importance']].groupby('variable').mean()
importances.sort_values(by="relative_importance", ascending=False).iplot(kind='bar', colors='#5AC4F2', theme='white')

import matplotlib.pyplot as plt
plt.figure()
bestModel = h2o.get_model(leaders[2, 0])
plt = bestModel.partial_plot(data=test, cols=["Day Mins","CustServ Calls","Day Charge"])


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:

Exercise Set V

As we saw earlier, PCA has a lot of other uses. Since we wanted to visualize our data in 2 dimensions, restricted the number of dimensions to 2 in PCA. But what is the true optimal number of dimensions?

Exercise: Using a new PCA object shown in the next cell, plot the `explained_variance_` field and look for the elbow point, the point where the curve's rate of descent seems to slow sharply. This value is one possible value for the optimal number of dimensions. What is it?

Can you write Python code for it?

#your turn
# Initialize a new PCA model with a default number of components.
# import sklearn.decomposition
# pca = sklearn.decomposition.PCA()
# pca.fit(X)

# Do the rest on your own :)

pca = PCA()
pca.fit(x_cols)

print(pca.explained_variance_)

plt.plot(range(1, len(pca.explained_variance_)+1), pca.explained_variance_, 'o-')
plt.xlabel('Component')
plt.ylabel('Variance')
plt.title('Variance by Component')
plt.show()


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
Choosing K: The Silhouette Method
There exists another method that measures how well each datapoint $x_i$ "fits" its assigned cluster and also how poorly it fits into other clusters. This is a different way of looking at the same objective. Denote $a_{x_i}$ as the average distance from $x_i$ to all other points within its own cluster $k$. The lower the value, the better. On the other hand $b_{x_i}$ is the minimum average distance from $x_i$ to points in a different cluster, minimized over clusters. That is, compute separately for each cluster the average distance from $x_i$ to the points within that cluster, and then take the minimum. The silhouette $s(x_i)$ is defined as
$$s(x_i) = \frac{b_{x_i} - a_{x_i}}{\max{\left( a_{x_i}, b_{x_i}\right)}}$$
The silhouette score is computed on every datapoint in every cluster. The silhouette score ranges from -1 (a poor clustering) to +1 (a very dense clustering) with 0 denoting the situation where clusters overlap. Some criteria for the silhouette coefficient is provided in the table below.

| Range       | Interpretation                                |
|-------------|-----------------------------------------------|
| 0.71 - 1.0  | A strong structure has been found.            |
| 0.51 - 0.7  | A reasonable structure has been found.        |
| 0.26 - 0.5  | The structure is weak and could be artificial.|
| < 0.25      | No substantial structure has been found.      |

Source: http://www.stat.berkeley.edu/~spector/s133/Clus.html
Fortunately, scikit-learn provides a function to compute this for us (phew!) called sklearn.metrics.silhouette_score. Take a look at this article on picking $K$ in scikit-learn, as it will help you in the next exercise set.

Checkup Exercise Set III

Exercise: Using the documentation for the `silhouette_score` function above, construct a series of silhouette plots like the ones in the article linked above.

Exercise: Compute the average silhouette score for each...
Can you write Python code for it?

# Your turn.

from sklearn.metrics import silhouette_samples, silhouette_score
import matplotlib.cm as cm

silhouette_score_by_k = {}

for k in ks:
    fig, (ax1) = plt.subplots(1, 1)

    ax1.set_xlim([-0.2, 0.6])
    ax1.set_ylim([0, len(x_cols) + (k + 1) * 10])

    clusterer = KMeans(n_clusters=k, random_state=10)
    cluster_labels = clusterer.fit_predict(x_cols)

    silhouette_avg = silhouette_score(x_cols, cluster_labels)
    print("For n_clusters =", k,
          "The average silhouette_score is :", silhouette_avg)
    silhouette_score_by_k[k] = silhouette_avg

    sample_silhouette_values = silhouette_samples(x_cols, cluster_labels)

    y_lower = 10
    for i in range(k):
        ith_cluster_silhouette_values = \
            sample_silhouette_values[cluster_labels == i]

        ith_cluster_silhouette_values.sort()

        size_cluster_i = ith_cluster_silhouette_values.shape[0]
        y_upper = y_lower + size_cluster_i

        color = cm.spectral(float(i) / k)
        ax1.fill_betweenx(np.arange(y_lower, y_upper),
                          0, ith_cluster_silhouette_values,
                          facecolor=color, edgecolor=color, alpha=0.7)

        ax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))

        y_lower = y_upper + 10

    ax1.set_title("The silhouette plot for the various clusters.")
    ax1.set_xlabel("The silhouette coefficient values")
    ax1.set_ylabel("Cluster label")

    ax1.axvline(x=silhouette_avg, color="red", linestyle="--")

    ax1.set_yticks([])
    ax1.set_xticks([-0.2, -0.1, 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6])

    plt.suptitle(("Silhouette analysis for KMeans clustering on sample data "
                  "with n_clusters = %d" % k),
                 fontsize=14, fontweight='bold')

    plt.show()

for k, score in silhouette_score_by_k.items():
    print("K =", k, " => ", "silhouette score =", score)


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
Data
The dataset contains information on marketing newsletters/e-mail campaigns (e-mail offers sent to customers) and transaction level data from customers. The transactional data shows which offer customers responded to, and what the customer ended up buying. The data is presented as an Excel workbook containing two worksheets. Each worksheet contains a different dataset.
Can you write Python code for it?

df_offers = pd.read_excel("./WineKMC.xlsx", sheetname=0)
df_offers.columns = ["offer_id", "campaign", "varietal", "min_qty", "discount", "origin", "past_peak"]
df_offers.head()


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
Data wrangling
We're trying to learn more about how our customers behave, so we can use their behavior (whether or not they purchased something based on an offer) as a way to group similar minded customers together. We can then study those groups to look for patterns and trends which can help us formulate future offers.
The first thing we need is a way to compare customers. To do this, we're going to create a matrix that contains each customer and a 0/1 indicator for whether or not they responded to a given offer.

Checkup Exercise Set I

Exercise: Create a data frame where each row has the following columns (Use the pandas [`merge`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html) and [`pivot_table`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.pivot_table.html) functions for this purpose):

 customer_name

 One column for each offer, with a 1 if the customer responded to the offer

Make sure you also deal with any weird values such as `NaN`. Read the documentation to develop your solution.

Can you write Python code for it?

#your turn

df = df_transactions.merge(df_offers, on='offer_id')
df = df.pivot_table(index='customer_name', columns='offer_id', values='n', fill_value=0)
df.head()


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
K-Means Clustering
Recall that in K-Means Clustering we want to maximize the distance between centroids and minimize the distance between data points and the respective centroid for the cluster they are in. True evaluation for unsupervised learning would require labeled data; however, we can use a variety of intuitive metrics to try to pick the number of clusters K. We will introduce two methods: the Elbow method, the Silhouette method and the gap statistic.
Choosing K: The Elbow Sum-of-Squares Method
The first method looks at the sum-of-squares error in each cluster against $K$. We compute the distance from each data point to the center of the cluster (centroid) to which the data point was assigned.
$$SS = \sum_k \sum_{x_i \in C_k} \sum_{x_j \in C_k} \left( x_i - x_j \right)^2 = \sum_k \sum_{x_i \in C_k} \left( x_i - \mu_k \right)^2$$
where $x_i$ is a point, $C_k$ represents cluster $k$ and $\mu_k$ is the centroid for cluster $k$. We can plot SS vs. $K$ and choose the elbow point in the plot as the best value for $K$. The elbow point is the point at which the plot starts descending much more slowly.

Checkup Exercise Set II

Exercise:

 What values of $SS$ do you believe represent better clusterings? Why?

 Create a numpy matrix `x_cols` with only the columns representing the offers (i.e. the 0/1 colums)

 Write code that applies the [`KMeans`](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) clustering method from scikit-learn to this matrix.

 Construct a plot showing $SS$ for each $K$ and pick $K$ using this plot. For simplicity, test $2 \le K \le 10$.

 What challenges did you experience using the Elbow method to pick $K$?

 Make a bar chart showing the number of points in each cluster for k-means under the best $K$.

Can you write Python code for it?


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
The Sum of Squares (SS) measures the sum of the squared distances from a data point to the centroid of that cluster. You want to minimize the SS in a cluster to make the cluster more distinct. The best case scenario would be large distances between different clusters with relatively small SS for each cluster, but this is unlikely. There is a trade off with minimizing the SS and the number of clusters. You can decrease the SS by increasing the number of clusters but it is also ideal to keep the number of clusters as low as possible. Due to this trade off, the best K (number of clusters) can be chosen by using the elbow method.
Can you write Python code for it?

import numpy as np

print(df.columns)
x_cols = np.matrix(df[df.columns])
x_cols

from sklearn.cluster import KMeans

inertias = []
ks = range(2, 11)

for k in ks:
    kMeans_model = KMeans(n_clusters=k)
    kMeans_model.fit(x_cols)
    inertias.append(kMeans_model.inertia_)

plt.plot(ks, inertias)
plt.xlabel("K")
plt.ylabel("SS")
plt.title("K vs SS")
plt.show()


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
Ultimately, comparing the results of different clustering methods can be quite subjective since we do not have correct labels such as in supervised learning. We can still attempt to compare the results by using the Silhouette score.
Can you write Python code for it?

from sklearn.cluster import AffinityPropagation

best_score = 0
print('Affinity Propagation')
for d in np.arange(0.5, 1, 0.05):
    model = AffinityPropagation(damping=d)
    clusters = model.fit_predict(x_cols)

    score = silhouette_score(x_cols, clusters)

    if score > best_score:
        best_score = score

    print('damping =', d,': Silhouette score =', score)

print('\nBest Score =', best_score)

from sklearn.cluster import SpectralClustering

best_score = 0
print('Spectral Clustering')
for k in ks:
    model = SpectralClustering(n_clusters=k)
    clusters = model.fit_predict(x_cols)

    score = silhouette_score(x_cols, clusters)

    if score > best_score:
        best_score = score

    print('n_clusters =', k,': Silhouette score =', score)

print('\nBest Score =', best_score)

from sklearn.cluster import AgglomerativeClustering

best_score = 0
print('Agglomerative Clustering')
for k in ks:
    model = AgglomerativeClustering(n_clusters=k)
    clusters = model.fit_predict(x_cols)

    score = silhouette_score(x_cols, clusters)

    if score > best_score:
        best_score = score

    print('n_clusters =', k,': Silhouette score =', score)

print('\nBest Score =', best_score)

from sklearn.cluster import DBSCAN

best_score = 0
for e in np.arange(1, 2.6, 0.1):
    for sample in range(1,11):
        model = DBSCAN(eps= e, min_samples=sample)
        clusters = model.fit_predict(x_cols)

        if len(collections.Counter(clusters).keys()) > 1:
            score = silhouette_score(x_cols, clusters)

            if score > best_score:
                best_score = score

            print('eps=', e, ', min_samples=', sample, ': Silhouette score =', score)

print('\nBest Score =', best_score)


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
Using this plot, it is difficult to apply the elbow method and choose the best K. The plot is almost linear and there is no clear "elbow". We can arbitrarily choose 5 clusters for now, but a better method to choose K should be used.
Can you write Python code for it?

import collections

kMeans_model = KMeans(n_clusters=5, random_state=5)
clusters = kMeans_model.fit_predict(x_cols)

count_by_cluster = collections.OrderedDict(sorted(collections.Counter(clusters).items()))

plt.bar(list(count_by_cluster.keys()), list(count_by_cluster.values()))
plt.title("Data Points in Each Cluster")
plt.xlabel("Cluster")
plt.ylabel("Number of Data Points")

for k, v in count_by_cluster.items():
    print(k,":", v)


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
We see that the first dataset contains information about each offer such as the month it is in effect and several attributes about the wine that the offer refers to: the variety, minimum quantity, discount, country of origin and whether or not it is past peak. The second dataset in the second worksheet contains transactional data -- which offer each customer responded to.
Can you write Python code for it?

df_transactions = pd.read_excel("./WineKMC.xlsx", sheetname=1)
df_transactions.columns = ["customer_name", "offer_id"]
df_transactions['n'] = 1
df_transactions.head()


I am working on the file "Customer Segmentation Using Clustering/Mini_Project_Clustering.ipynb".
The first task is:
What we've done is we've taken those columns of 0/1 indicator variables, and we've transformed them into a 2-D dataset. We took one column and arbitrarily called it x and then called the other y. Now we can throw each point into a scatterplot. We color coded each point based on it's cluster so it's easier to see them.
Can you write Python code for it?

plt.scatter(x, y, c=clusters, cmap="Set1")
plt.show()


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Data Exploration
In this section, you will begin exploring the data through visualizations and code to understand how each feature is related to the others. You will observe a statistical description of the dataset, consider the relevance of each feature, and select a few sample data points from the dataset which you will track through the course of this project.
Run the code block below to observe a statistical description of the dataset. Note that the dataset is composed of six important product categories: 'Fresh', 'Milk', 'Grocery', 'Frozen', 'Detergents_Paper', and 'Delicatessen'. Consider what each category represents in terms of products you could purchase.
Can you write Python code for it?

# Display a description of the dataset
display(data.describe())


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Implementation: Dimensionality Reduction
When using principal component analysis, one of the main goals is to reduce the dimensionality of the data — in effect, reducing the complexity of the problem. Dimensionality reduction comes at a cost: Fewer dimensions used implies less of the total variance in the data is being explained. Because of this, the cumulative explained variance ratio is extremely important for knowing how many dimensions are necessary for the problem. Additionally, if a signifiant amount of variance is explained by only two or three dimensions, the reduced data can be visualized afterwards.
In the code block below, you will need to implement the following:
 - Assign the results of fitting PCA in two dimensions with good_data to pca.
 - Apply a PCA transformation of good_data using pca.transform, and assign the results to reduced_data.
 - Apply a PCA transformation of log_samples using pca.transform, and assign the results to pca_samples.
Can you write Python code for it?

# TODO: Apply PCA by fitting the good data with only two dimensions
pca = PCA(n_components=2).fit(good_data)

# TODO: Transform the good data using the PCA fit above
reduced_data = pca.transform(good_data)

# TODO: Transform log_samples using the PCA fit above
pca_samples = pca.transform(log_samples)

# Create a DataFrame for the reduced data
reduced_data = pd.DataFrame(reduced_data, columns = ['Dimension 1', 'Dimension 2'])


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Implementation: Outlier Detection
Detecting outliers in the data is extremely important in the data preprocessing step of any analysis. The presence of outliers can often skew results which take into consideration these data points. There are many "rules of thumb" for what constitutes an outlier in a dataset. Here, we will use Tukey's Method for identfying outliers: An outlier step is calculated as 1.5 times the interquartile range (IQR). A data point with a feature that is beyond an outlier step outside of the IQR for that feature is considered abnormal.
In the code block below, you will need to implement the following:
 - Assign the value of the 25th percentile for the given feature to Q1. Use np.percentile for this.
 - Assign the value of the 75th percentile for the given feature to Q3. Again, use np.percentile.
 - Assign the calculation of an outlier step for the given feature to step.
 - Optionally remove data points from the dataset by adding indices to the outliers list.
NOTE: If you choose to remove any outliers, ensure that the sample data does not contain any of these points!
Once you have performed this implementation, the dataset will be stored in the variable good_data.
Can you write Python code for it?

# For each feature find the data points with extreme high or low values
for feature in log_data.keys():

    # TODO: Calculate Q1 (25th percentile of the data) for the given feature
    Q1 = np.percentile(log_data[feature], 25)

    # TODO: Calculate Q3 (75th percentile of the data) for the given feature
    Q3 = np.percentile(log_data[feature],75)

    # TODO: Use the interquartile range to calculate an outlier step (1.5 times the interquartile range)
    step = (Q3-Q1)*1.5

    # Display the outliers
    print "Data points considered outliers for the feature '{}':".format(feature)
    display(log_data[~((log_data[feature] >= Q1 - step) & (log_data[feature] <= Q3 + step))])

# OPTIONAL: Select the indices for data points you wish to remove
outliers  = [65, 66, 75, 128, 154]

# Remove the outliers, if any were specified
good_data = log_data.drop(log_data.index[outliers]).reset_index(drop = True)


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Implementation: Selecting Samples
To get a better understanding of the customers and how their data will transform through the analysis, it would be best to select a few sample data points and explore them in more detail. In the code block below, add three indices of your choice to the indices list which will represent the customers to track. It is suggested to try different sets of samples until you obtain customers that vary significantly from one another.
Can you write Python code for it?

# TODO: Select three indices of your choice you wish to sample from the dataset
indices = [100,200,300]

# Create a DataFrame of the chosen samples
samples = pd.DataFrame(data.loc[indices], columns = data.keys()).reset_index(drop = True)
print "Chosen samples of wholesale customers dataset:"
display(samples)


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Observation
After applying a natural logarithm scaling to the data, the distribution of each feature should appear much more normal. For any pairs of features you may have identified earlier as being correlated, observe here whether that correlation is still present (and whether it is now stronger or weaker than before).
Run the code below to see how the sample data has changed after having the natural logarithm applied to it.
Can you write Python code for it?

# Display the log-transformed sample data
display(log_samples)


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Observation
Once we have the original feature projections (in red), it is easier to interpret the relative position of each data point in the scatterplot. For instance, a point the lower right corner of the figure will likely correspond to a customer that spends a lot on 'Milk', 'Grocery' and 'Detergents_Paper', but not so much on the other product categories.
From the biplot, which of the original features are most strongly correlated with the first component? What about those that are associated with the second component? Do these observations agree with the pca_results plot you obtained earlier?
Clustering
In this section, you will choose to use either a K-Means clustering algorithm or a Gaussian Mixture Model clustering algorithm to identify the various customer segments hidden in the data. You will then recover specific data points from the clusters to understand their significance by transforming them back into their original dimension and scale.
Question 6
What are the advantages to using a K-Means clustering algorithm? What are the advantages to using a Gaussian Mixture Model clustering algorithm? Given your observations about the wholesale customer data so far, which of the two algorithms will you use and why?
Answer: K-Means clustering uses a hard assignment of points where as the Gaussian Mixture Model uses soft assignment. Given that the customer data seems to not have clear and distinct boundaries I will use the Gaussian Mixture Model as it assigns points to clusters probablistically rather than using hard assignments.
Implementation: Creating Clusters
Depending on the problem, the number of clusters that you expect to be in the data may already be known. When the number of clusters is not known a priori, there is no guarantee that a given number of clusters best segments the data, since it is unclear what structure exists in the data — if any. However, we can quantify the "goodness" of a clustering by calculating each data point's silhouette coefficient. The silhouette coefficient for a data...
Can you write Python code for it?

from sklearn.mixture import GaussianMixture
from sklearn.metrics import silhouette_score
# TODO: Apply your clustering algorithm of choice to the reduced data
GM = GaussianMixture(n_components = 2)
clusterer = GM.fit(reduced_data)

# TODO: Predict the cluster for each data point
preds = clusterer.predict(reduced_data)

# TODO: Find the cluster centers
centers = clusterer.means_

# TODO: Predict the cluster for each transformed sample data point
sample_preds = clusterer.predict(pca_samples)

# TODO: Calculate the mean silhouette coefficient for the number of clusters chosen
score = silhouette_score(reduced_data, preds)
print score


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Observation
Run the code below to see how the log-transformed sample data has changed after having a PCA transformation applied to it using only two dimensions. Observe how the values for the first two dimensions remains unchanged when compared to a PCA transformation in six dimensions.
Can you write Python code for it?

# Display sample log-data after applying PCA transformation in two dimensions
display(pd.DataFrame(np.round(pca_samples, 4), columns = ['Dimension 1', 'Dimension 2']))


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Question 1
Consider the total purchase cost of each product category and the statistical description of the dataset above for your sample customers.
What kind of establishment (customer) could each of the three samples you've chosen represent?
Hint: Examples of establishments include places like markets, cafes, and retailers, among many others. Avoid using names for establishments, such as saying "McDonalds" when describing a sample customer as a restaurant.
Answer: Sample 0 has a values which are significantly higher than the mean in the categories of Grocery and Detergents_Paper but values which are much closer to the mean in the other categories. This sample could represent supermarket which sells freshly prepared food. Sample 1 has a much higher than mean value for Milk and Grocery and significantly below mean value for Fresh and Delicatessen. This sample could represent a supermarket - selling mostly Groceries, Milk. Sample 2 has very high values, compared to the mean, for Fresh and very low values for Frozen. This sample could represent a Cafe or Restaurant which sells a lot of fresh food but little frozen food.
Implementation: Feature Relevance
One interesting thought to consider is if one (or more) of the six product categories is actually relevant for understanding customer purchasing. That is to say, is it possible to determine whether customers purchasing some amount of one category of products will necessarily purchase some proportional amount of another category of products? We can make this determination quite easily by training a supervised regression learner on a subset of the data with one feature removed, and then score how well that model can predict the removed feature.
In the code block below, you will need to implement the following:
 - Assign new_data a copy of the data by removing a feature of your choice using the DataFrame.drop function.
 - Use sklearn.cross_validation.train_test_split to split the dataset into training and testing sets.
   - Use the removed feature as your target...
Can you write Python code for it?

from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import r2_score

# TODO: Make a copy of the DataFrame, using the 'drop' function to drop the given feature
new_data = data.drop(['Milk'], axis = 1)

# TODO: Split the data into training and testing sets using the given feature as the target
X_train, X_test, y_train, y_test = train_test_split(new_data, data['Milk'], test_size=0.25, random_state=23)

# TODO: Create a decision tree regressor and fit it to the training set
regressor = DecisionTreeRegressor()
regressor = regressor.fit(X_train, y_train)

# TODO: Report the score of the prediction using the testing set
y_pred = regressor.predict(X_test)
score = r2_score(y_test, y_pred)
print "R^2 score for prediction using testing set:"
print score


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Question 2
Which feature did you attempt to predict? What was the reported prediction score? Is this feature necessary for identifying customers' spending habits?
Hint: The coefficient of determination, R^2, is scored between 0 and 1, with 1 being a perfect fit. A negative R^2 implies the model fails to fit the data.
Answer: I tried to predict the Fresh feature from the data and the score was -1.56. This implies that the remaining data was unable to model the Fresh feature. This implies that the feature is necessary for identifying customer spending habits. By contrast the Milk feature was predicted with a score of 0.47. This implies that the feature can be modelled using the other features to a reasonable extent and might not be necessary to predict customer spending habits.
Visualize Feature Distributions
To get a better understanding of the dataset, we can construct a scatter matrix of each of the six product features present in the data. If you found that the feature you attempted to predict above is relevant for identifying a specific customer, then the scatter matrix below may not show any correlation between that feature and the others. Conversely, if you believe that feature is not relevant for identifying a specific customer, the scatter matrix might show a correlation between that feature and another feature in the data. Run the code block below to produce a scatter matrix.
Can you write Python code for it?

# Produce a scatter matrix for each pair of features in the data
pd.plotting.scatter_matrix(data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Question 3
Are there any pairs of features which exhibit some degree of correlation? Does this confirm or deny your suspicions about the relevance of the feature you attempted to predict? How is the data for those features distributed?
Hint: Is the data normally distributed? Where do most of the data points lie?
Answer: There seems to be a correlation between Grocery and Milk as well as between Detergents_Paper and Grocery. The correlation between Milk and Grocery seems to be weaker than between Detergents_Paper and Grocery. This confirms why the Milk feature was hard to predict. The data is not normally distributed, with a significant number of points lying at the lower end of the scale.
Data Preprocessing
In this section, you will preprocess the data to create a better representation of customers by performing a scaling on the data and detecting (and optionally removing) outliers. Preprocessing data is often times a critical step in assuring that results you obtain from your analysis are significant and meaningful.
Implementation: Feature Scaling
If data is not normally distributed, especially if the mean and median vary significantly (indicating a large skew), it is most often appropriate to apply a non-linear scaling — particularly for financial data. One way to achieve this scaling is by using a Box-Cox test, which calculates the best power transformation of the data that reduces skewness. A simpler approach which can work in most cases would be applying the natural logarithm.
In the code block below, you will need to implement the following:
 - Assign a copy of the data to log_data after applying logarithmic scaling. Use the np.log function for this.
 - Assign a copy of the sample data to log_samples after applying logarithmic scaling. Again, use np.log.
Can you write Python code for it?

# TODO: Scale the data using the natural logarithm
log_data = np.log(data)

# TODO: Scale the sample data using the natural logarithm
log_samples = np.log(samples)

# Produce a scatter matrix for each pair of newly-transformed features
pd.plotting.scatter_matrix(log_data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Question 5
How much variance in the data is explained in total by the first and second principal component? What about the first four principal components? Using the visualization provided above, discuss what the first four dimensions best represent in terms of customer spending.
Hint: A positive increase in a specific dimension corresponds with an increase of the positive-weighted features and a decrease of the negative-weighted features. The rate of increase or decrease is based on the individual feature weights.
Answer: The first two components account for approximately 71.6% of the variance. When this is extended to the first four components they account for approximately 93.4% of the variance. The first four dimensions probably represent different types of customer spending, i.e. shop types. The first dimension has large weights between Detergents_Paper, Milk, and Grocery, which are negatively correlated to Fresh and Frozen. This could indicate purchasing of household items. Dimension 2 has strong weights in Fresh, Frozen, and Delicatessen. This could indicate purchasing of food items which are generally bought together. Dimension 3 indicates there is a negative relationship between Fresh and Delicatessen, which could be a measure which shows people who buy a lot of Delicatessen items are unlikely to spend a lot on Fresh food. Dimension 4 shows strong negative correlations between Frozen and Delicatessen. Indicating that people who spend a large amount of Frozen spend less on Delicatessen, and vice versa.
Observation
Run the code below to see how the log-transformed sample data has changed after having a PCA transformation applied to it in six dimensions. Observe the numerical value for the first four dimensions of the sample points. Consider if this is consistent with your initial interpretation of the sample points.
Can you write Python code for it?

# Display sample log-data after having a PCA transformation applied
display(pd.DataFrame(np.round(pca_samples, 4), columns = pca_results.index.values))


I am working on the file "Customer Segments/customer_segments.ipynb".
The first task is:
Visualizing a Biplot
A biplot is a scatterplot where each data point is represented by its scores along the principal components. The axes are the principal components (in this case Dimension 1 and Dimension 2). In addition, the biplot shows the projection of the original features along the components. A biplot can help us interpret the reduced dimensions of the data, and discover relationships between the principal components and original features.
Run the code cell below to produce a biplot of the reduced-dimension data.
Can you write Python code for it?

# Create a biplot
vs.biplot(good_data, reduced_data, pca)


I am working on the file "Customer database query/Customer database query.ipynb".
The first task is:
The format of the customer.xml file records:
<T>
    <C_CUSTKEY>1</C_CUSTKEY>
    <C_NAME>Customer#000000001</C_NAME>
    <C_ADDRESS>IVhzIApeRb ot,c,E</C_ADDRESS>
    <C_NATIONKEY>15</C_NATIONKEY>
    <C_PHONE>25-989-741-2988</C_PHONE>
    <C_ACCTBAL>711.56</C_ACCTBAL>
    <C_MKTSEGMENT>BUILDING</C_MKTSEGMENT>
    <C_COMMENT>regular, regular platelets are fluffily according to the even attainments. blithely iron</C_COMMENT>
  </T>
Can you write Python code for it?

lot = read_xml_file(r'customer.xml',
                    ['int', 'string', 'string', 'int', 'string', 'float', 'string', 'string'])

with sqlite.connect(r'customer.db') as con:
    cur = con.cursor()
    # Write a SQL query to erase any existing "customer" table in the database
    cur.execute("drop table if exists customer;")

    # Write a SQL query to create a new "customer" table with the following columns:
    # C_CUSTKEY (integer)  C_NAME (text)  C_ADDRESS (text)  C_NATIONKEY (integer)  C_PHONE (text)  C_ACCTBAL (real)  C_MKTSEGMENT (text)  C_COMMENT (text)
    cur.execute("create table customer(C_CUSTKEY integer, C_NAME text, C_ADDRESS text, C_NATIONKEY integer, C_PHONE text, C_ACCTBAL real, C_MKTSEGMENT text, C_COMMENT text)")

    # Write a SQL query to insert the values in the list-of-tuples variable lot into the customer table
    cur.executemany("insert into customer values(?,?,?,?,?,?,?,?)", lot)
    con.commit()

    # Write a SQL query to compute the count of all records (rows) in the customer table
    cur.execute("select Count(*) from customer;")
    print "customer count: "
    check_result(str(cur.fetchone()[0]), "1500")

    # Write a SQL query to compute the average account balance (C_ACCTBAL) over all rows in the customer table.
    cur.execute("select avg(C_ACCTBAL) from customer;")
    print "customer average: "
    check_result(str(cur.fetchone()[0]), "4454.57706")

    # Write a SQL query to get the C_CUSTKEY, C_NAME, and C_ACCTBAL columns from the customer table that
    # have C_ACCTBAL >= 1000, and sort the results by C_CUSTKEY
    cur.execute("select C_CUSTKEY, C_NAME, C_ACCTBAL from customer where (C_ACCTBAL >= '1000') order by C_CUSTKEY; ")

    # fetchall() method gets all records. It returns a list of tuples.
    # Each of the tuples represent a row in the table.
    rows = cur.fetchall()

    f = open(r'desiredCustomers.txt', 'w')
    f.write('Customer Key' + '\t' + 'Name' + '\t' + 'Account Balance\n')
    for row in rows:
       ...


I am working on the file "CustomerChurnwithAutoML.ipynb".
The first task is:
Automatic Machine Learning
The Automatic Machine Learning (AutoML) function automates the supervised machine learning model training process. The current version of AutoML trains and cross-validates a Random Forest, an Extremely-Randomized Forest, a random grid of Gradient Boosting Machines (GBMs), a random grid of Deep Neural Nets, and a Stacked Ensemble of all the models.
Can you write Python code for it?

%%capture
# Run AutoML until 11 models are built
autoModel = H2OAutoML(max_models = 9)
autoModel.train(x = x, y = y,
          training_frame = train,
          validation_frame = test,
          leaderboard_frame = test)


I am working on the file "CustomerChurnwithAutoML.ipynb".
The first task is:
Best Model vs the Base Learners
This plot shows the ROC curves for the best models
Can you write Python code for it?

Model0 = np.array(h2o.get_model(leaders[0,0]).roc(xval=True))
Model1 = np.array(h2o.get_model(leaders[1,0]).roc(xval=True))
Model2 = np.array(h2o.get_model(leaders[2,0]).roc(xval=True))
Model3 = np.array(h2o.get_model(leaders[3,0]).roc(xval=True))
Model4 = np.array(h2o.get_model(leaders[4,0]).roc(xval=True))
Model5 = np.array(h2o.get_model(leaders[5,0]).roc(xval=True))
Model6 = np.array(h2o.get_model(leaders[6,0]).roc(xval=True))
Model7 = np.array(h2o.get_model(leaders[7,0]).roc(xval=True))
Model8 = np.array(h2o.get_model(leaders[8,0]).roc(xval=True))
Model9 = np.array(h2o.get_model(leaders[9,0]).roc(xval=True))

layout = go.Layout(autosize=False, width=725, height=575,  xaxis=dict(title='False Positive Rate', titlefont=dict(family='Arial, sans-serif', size=15, color='grey')),
                                                           yaxis=dict(title='True Positive Rate', titlefont=dict(family='Arial, sans-serif', size=15, color='grey')))

traceChanceLine = go.Scatter(x = [0,1], y = [0,1], mode = 'lines+markers', name = 'chance', line = dict(color = ('rgb(136, 140, 150)'), width = 4, dash = 'dash'))
Model0Trace = go.Scatter(x = Model0[0], y = Model0[1], mode = 'lines', name = 'Model 0', line = dict(color = ('rgb(26, 58, 126)'), width = 3))
Model1Trace = go.Scatter(x = Model1[0], y = Model1[1], mode = 'lines', name = 'Model 1', line = dict(color = ('rgb(156, 190, 241))'), width = 1))
Model2Trace = go.Scatter(x = Model2[0], y = Model2[1], mode = 'lines', name = 'Model 2', line = dict(color = ('rgb(156, 190, 241)'), width = 1))
Model3Trace = go.Scatter(x = Model3[0], y = Model3[1], mode = 'lines', name = 'Model 3', line = dict(color = ('rgb(156, 190, 241)'), width = 1))
Model4Trace = go.Scatter(x = Model4[0], y = Model4[1], mode = 'lines', name = 'Model 4', line = dict(color = ('rgb(156, 190, 241)'), width = 1))
Model5Trace = go.Scatter(x = Model5[0], y = Model5[1], mode = 'lines', name = 'Model 5', line = dict(color = ('rgb(156, 190, 241)'), width = 1))
Model6Trace = go.Scatter(x = Model6[0], y = Model6[1],...


I am working on the file "CustomerChurnwithAutoML.ipynb".
The first task is:
Business Impact Matrix
Weighting Predictions With a Dollar Value
-   Correctly predicting retain: +$5
-   Correctly predicting churn: +$75
-   Incorrectly predicting retain: -$150
-   Incorrectly predicting churn: -$1.5
Can you write Python code for it?

CorrectPredictChurn = cm.loc[1,'1']
CorrectPredictChurnImpact = 75
cm1 = CorrectPredictChurn*CorrectPredictChurnImpact

IncorrectPredictChurn = cm.loc[1,'0']
IncorrectPredictChurnImpact = -5
cm2 = IncorrectPredictChurn*IncorrectPredictChurnImpact

IncorrectPredictRetain = cm.loc[0,'1']
IncorrectPredictRetainImpact = -150
cm3 = IncorrectPredictRetain*IncorrectPredictRetainImpact

CorrectPredictRetain = cm.loc[0,'0']
CorrectPredictRetainImpact = 5
cm4 = IncorrectPredictRetain*CorrectPredictRetainImpact

data_matrix = [['Business Impact', '($) Predicted Churn', '($) Predicted Retain', '($) Total'],
               ['($) Actual Churn', cm1, cm3, '' ],
               ['($) Actual Retain', cm2, cm4, ''],
               ['($) Total', cm1+cm2, cm3+cm4, cm1+cm2+cm3+cm4]]

impactMatrix = ff.create_table(data_matrix, height_constant=20, hoverinfo='weight')
impactMatrix.layout.height=300
impactMatrix.layout.width=800
impactMatrix.layout.font.size=17
py.iplot(impactMatrix)

print("Total customers evaluated: 2132")

print("Total value created by the model: $" + str(cm1+cm2+cm3+cm4))

print("Total value per customer: $" +str(round(((cm1+cm2+cm3+cm4)/2132),3)))

#%%capture
# Save the best model

#path = h2o.save_model(model=h2o.get_model(leaders[0, 0]), force=True)
#os.rename(h2o.get_model(leaders[0, 0]).model_id, "AutoML-leader")

#%%capture
#LoadedEnsemble = h2o.load_model(path="AutoML-leader")
#print(LoadedEnsemble)


I am working on the file "CustomerChurnwithAutoML.ipynb".
The first task is:
Confusion Matrix
Can you write Python code for it?

cm = h2o.get_model(leaders[1, 0]).confusion_matrix(xval=True)
cm = cm.table.as_data_frame()
cm
confusionMatrix = ff.create_table(cm)
confusionMatrix.layout.height=300
confusionMatrix.layout.width=800
confusionMatrix.layout.font.size=17
py.iplot(confusionMatrix)


I am working on the file "CustomerChurnwithAutoML.ipynb".
The first task is:
Leaderboard
Can you write Python code for it?


I am working on the file "CustomerChurnwithAutoML.ipynb".
The first task is:
Load The Dataset
Can you write Python code for it?

# Load some data
churnDF = pd.read_csv('https://trifactapro.s3.amazonaws.com/churn.csv', delimiter=',')
churnDF.head(5)

#%%capture
#pandas_profiling.ProfileReport(churnDF)

churnDF["Churn"] = churnDF["Churn"].replace([True, False],[1,0])
churnDF["Int'l Plan"] = churnDF["Int'l Plan"].replace(["no","yes"],[0,1])
churnDF["VMail Plan"] = churnDF["VMail Plan"].replace(["no","yes"],[0,1])
churnDF.drop(["State", "Area Code", "Phone"], axis=1, inplace=True)


I am working on the file "CustomerChurnwithAutoML.ipynb".
The first task is:
Variable Importances
Below we plot variable importances as reported by the best performing algo in the ensemble.
Can you write Python code for it?

importances = h2o.get_model(leaders[2, 0]).varimp(use_pandas=True)
importances = importances.loc[:,['variable','relative_importance']].groupby('variable').mean()
importances.sort_values(by="relative_importance", ascending=False).iplot(kind='bar', colors='#5AC4F2', theme='white')

import matplotlib.pyplot as plt
plt.figure()
bestModel = h2o.get_model(leaders[2, 0])
plt = bestModel.partial_plot(data=test, cols=["Day Mins","CustServ Calls","Day Charge"])


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.1 - Numpy.ipynb".
The first task is:
Do tworzenia sekwencji liczbowych jako obiekty typu array należy wykorzystać funkcję arange.
Can you write Python code for it?

np.arange(10)

np.arange(5,10)

np.arange(5,10,0.5)


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.1 - Numpy.ipynb".
The first task is:
Możemy również zmieniać kształt (reshape) wyjścia funkcji arange.
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.1 - Numpy.ipynb".
The first task is:
Tablice mogą sie składać z różnych typów danych (ale tylko jednego typu danych równocześnie, stąd jednorodność).
Can you write Python code for it?

x = np.array([1,2,3])
print x.dtype
x = np.array([0.1,0.2,0.3])
print x
print x.dtype
x = np.array([1,2,3],dtype='float64')
print x.dtype


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.1 - Numpy.ipynb".
The first task is:
Tablice typu array mają wiele przydatnych wbudowanych metod.
Can you write Python code for it?

x.sum(axis=0)

x.sum(axis=1)

x.mean(axis=0)

x.mean(axis=1)


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.1 - Numpy.ipynb".
The first task is:
Tak jak w przypadku wszystkich funkcji pythonowych możemy uzyskać dodatkowe informacje o funkcjach NumPy za pomocą polecenia help(nazwa.funkcji):
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.2 - NumPy, kilka podstawowych funkcji.ipynb".
The first task is:
Inne funkcje
Indeksy i zakresy
tablice jednowymiarowe zachowują sie podobnie do zwykłych list pythonowych.
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.2 - NumPy, kilka podstawowych funkcji.ipynb".
The first task is:
Pętle i drukowanie
Can you write Python code for it?

for row in x:
    print row

for element in x.flat:
    print element


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.2 - NumPy, kilka podstawowych funkcji.ipynb".
The first task is:
Sklejanie tablic
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.2 - NumPy, kilka podstawowych funkcji.ipynb".
The first task is:
Typowe funkcje matematyczne
Can you write Python code for it?

x = np.arange(1,5)
np.sqrt(x) * np.pi

2**4

np.power(2,4)

np.log(np.e)

x = np.arange(5)
x.max() - x.min()


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.2 - NumPy, kilka podstawowych funkcji.ipynb".
The first task is:
Wielowmiarowy tablice mają po jednym indeksie na wymiar
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.2 - NumPy, kilka podstawowych funkcji.ipynb".
The first task is:
Zaskakujące może być działanie operatora *, który nie oblicza iloczynu macierzy. Odpowiada za to funkcja dot.
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.3 - NumPy, algebra liniowa.ipynb".
The first task is:
A macierz wierszowa w NumPy.
$$ X =
    \begin{pmatrix}
    3 & 4 & 5 & 6
    \end{pmatrix}$$
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.3 - NumPy, algebra liniowa.ipynb".
The first task is:
Macierz odwrotna
Can you write Python code for it?

A = np.array([[-4,-2],[5,5]])
A

invA = np.linalg.inv(A)
invA

np.round(np.dot(A,invA))


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.3 - NumPy, algebra liniowa.ipynb".
The first task is:
Obiekty typu matrix
Macierze ogólne omówiliśmy już w poprzednich dokumentach:
$$A_{m,n} =
    \begin{pmatrix}
     a_{1,1} & a_{1,2} & \cdots & a_{1,n} \
     a_{2,1} & a_{2,2} & \cdots & a_{2,n} \
     \vdots  & \vdots  & \ddots & \vdots  \
     a_{m,1} & a_{m,2} & \cdots & a_{m,n}
    \end{pmatrix}$$
Oprócz obiektów typu array istnieje wyspecjalizowany obiekt matrix, dla którego operacje * (mnożenie) oraz **-1 (odwracanie) są określone w sposób właściwy dla macierzy (w przeciwieństwu do operacji elementowych dla obietków array).
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.3 - NumPy, algebra liniowa.ipynb".
The first task is:
Operacje na macierzach
Wyznacznik
Can you write Python code for it?


I am working on the file "Cwiczenia/01/Uczenie Maszynowe - Ćwiczenia 1.3 - NumPy, algebra liniowa.ipynb".
The first task is:
Ponieważ $AA^{-1} = A^{-1}A = I$.
Wartości i wektory własne
Can you write Python code for it?

a = np.diag((1, 2, 3))
a

w,v = np.linalg.eig(a)
w

v


I am working on the file "Cwiczenia/02/Uczenie Maszynowe - Ćwiczenia 2.1 - Wykresy i krzywe.ipynb".
The first task is:
Powierzchnie 3D
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure(figsize=(15,10))
ax = fig.add_subplot(111, projection='3d')

X = np.arange(-5, 5, 0.25)
Y = np.arange(-5, 5, 0.25)
X, Y = np.meshgrid(X, Y)
R = np.sqrt(X**2 + Y**2)
Z = np.sin(R)

surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.jet,
        linewidth=0, antialiased=True)

ax.set_zlim(-1.01, 1.01)

plt.show()


I am working on the file "Cwiczenia/02/Uczenie Maszynowe - Ćwiczenia 2.2 - Wykresy kolumnowe, punktowe, histogramy.ipynb".
The first task is:
Wykresy punktowe
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure(figsize=(15,7))
ax1 = fig.add_subplot(121)

## the data
N=1000
x = np.random.randn(N)
y = np.random.randn(N)

## left panel
ax1.scatter(x,y,color='blue',s=5,edgecolor='none')
ax1.set_aspect(1./ax1.get_data_ratio()) # make axes square

## right panel
ax2 = fig.add_subplot(122)
props = dict(alpha=0.5, edgecolors='none' )

handles = []
colors = ['blue', 'green', 'magenta', 'cyan']
for color in colors:
    x = np.random.randn(N)
    y = np.random.randn(N)
    s = np.random.randint(50,200)
    handles.append(ax2.scatter(x, y, c=color, s=s, **props))

ax2.set_ylim([-5,11])
ax2.set_xlim([-5,11])

ax2.legend(handles, colors)
ax2.grid(True)
ax2.set_aspect(1./ax2.get_data_ratio())
plt.show()


I am working on the file "Cython Sandbox.ipynb".
The first task is:
basic : fibonacchi example (pure python and cython)
Can you write Python code for it?

def pyfib(n):
    a,b = 1,1
    for i in range(n):
        a,b = a+b, a
    return a

pyfib(5)

%%cython
def fib(int n):
    cdef int i,a,b
    a,b = 1,1
    for i in range(n):
        a,b = a+b, a
    return a

fib(10)

[fib(i) for i in range(10)]

pyfib(10)

%timeit fib(10)
%timeit pyfib(10)


I am working on the file "Cython Sandbox.ipynb".
The first task is:
difference between cdef and cpdef
Can you write Python code for it?

%%cython
cdef f1(int x):
    return x*x

cpdef f2(int x):
    return x*x

cpdef f3(int x):
    return f1(x)

#dir()

f2(3)


I am working on the file "Cython Sandbox.ipynb".
The first task is:
distance function (pure python and cython)
Can you write Python code for it?

%%cython
cpdef float distance(float[:] x, float[:] y):
    cdef int i
    cdef int n = x.shape[0]
    cdef float d = 0.0
    for i in range(n):
        d += (x[i]-y[i])**2
    return d

import numpy as np
x = np.array([1,2,3], np.float32)
y = np.array([7,6,5], np.float32)
dist = distance(x,y)
print dist

def pydistance(x, y, n):
    d = 0.0
    for i in range(n):
        d += (x[i]-y[i])**2
    return d

%timeit distance(x,y)
%timeit pydistance(x,y,3)


I am working on the file "CythonBootstrap.ipynb".
The first task is:
Another Example, with a polynomial this time ...

For now, lets begin with a polynomial function, and compare how to do this in python and cython! ....
Now consider a function such as
$f(x) = a_0x^n + a_1x^{(n-1)} + a_2x^{(n-2)} ..... a_nx^0$
where in the case below n is selected as 2, and
  - $a_0 = 0.1$,
  - $a_1=0.5$
  - $a_2=0.25$.
The cython function to do this called "cypoly" while the python version is called "pypoly". Each function is defined with a functional programming techinque of lambda and map, as shown below.
Can you write Python code for it?

%%file ./src/cyPoly.pyx
def cypoly(int n, int k):
    return map(lambda x:(1.0*x**2 + 0.5*x + 0.25*x), range(k))

!makecython ./src/cyPoly.pyx

def pypoly(n,k):
    return map(lambda x:.1*x**2 + .5*x + 0.25*x, range(k))


I am working on the file "CythonBootstrap.ipynb".
The first task is:
Introducing runcython !!

Is located on Github
Easy installation == pip install runcython
Russell91 on Github

https://github.com/Russell91/runcython
There is a runcython and makecython function calls . . . . .
Can you write Python code for it?

!makecython ./src/cyFib.pyx

def pyfib(n):
    a = 0
    b = 1
    index = 0
    while b < n:
        a, b = b, a+b
        index += 1
    return b

%timeit pyfib(1000)

import cyFib
%timeit cyFib.cyfib(1000)


I am working on the file "CythonBootstrap.ipynb".
The first task is:
NOW THAT IS A CONSIDERABLE SPEEDUP  ...

Fibonnaci function shows a factor of over 1500 % Improvement
Let's take a look at disassembly for some reasons for this ....
Can you write Python code for it?

import dis
dis.dis(pyfib)

import cProfile
cProfile.run('pyfib(1000)')


I am working on the file "CythonBootstrap.ipynb".
The first task is:
Now's lets do something graphically, like plot a trig function. Let's also use a float/double type.
Can you write Python code for it?

%%file ./src/sineWave.pyx
import cython
from libc.math cimport sin

def sinewave(double x):
    """ Calculate a sinewave for specified number of cycles, Ncycles, at a given frequency."""
    return sin(x)

!makecython ./src/sineWave.pyx

from src import sineWave
import math
angle90 = math.pi/2
sineWave.sinewave(angle90)


I am working on the file "CythonBootstrap.ipynb".
The first task is:
Summary & Conclusions
This talk has presented the basics of getting started with Cython and IPython/Jupyter Notebook. There were examples presented on how to compile Cython programs with a setup.py and distuils as well as a nice application, runcython. Basic programs and some programs with arrays were demonstrated.
Cython is flexible, and it's flexibility is matched by it's performance.
It's realitively easy to use, but it does have some details to watch out for when working with arrays, references, etc.
Overall

Cython enables Python code to be transformed easily
The transformed Python code is signficantly faster
Wide support and documentation exists for Cython
Language has evolved and grown over the past few years with widespread support
Usage in Ipython Notebook / Jupyter is now well supported
Can be used on a wide variety of programs, ranging from math to logic.

Transform your Python with Cython !!
Can you write Python code for it?

!python-config --cflags

!python-config --ldflags

!ls -a ./src

%%file ./src/exponent.pyx

"""
module:: This is a Cython file that uses decorators for arguments.

"""
import cython

@cython.locals(a = cython.int, x = cython.int, y = cython.int)
def power(a, x):
    """ funtion that uses cython.locals """
    y = a**x
    return y

!makecython ./src/exponent.pyx

from src import exponent

exponent?

import src

%%file ./src/setup.py

from distutils.core import setup, Extension
from Cython.Build import cythonize

#=========================================
# Setup the extensions
#=========================================
sources = [ "./src/cyMath.pyx", "./src/helloCython.pyx",
           "./src/cy_math.pyx", "./src/bits.pyx",
           "./src/printString.pyx", "./src/exponent.pyx"]

#for fileName in sources:
#    setup(ext_modules=cythonize(str(fileName)))

map(lambda fileName : setup(ext_modules=cythonize(str(fileName))), sources)

!python ./src/setup.py build_ext --inplace

from src import exponent

[exponent.power(x,3) for x in range(10)]

def quadPy(a,x):
    return a*(x**2)

%timeit quadPy(2.0, 5.0)


I am working on the file "Código/iterativos_matriciales.ipynb".
The first task is:
Observamos que la matriz $T$ con menor radio espectral es $T_{SOR}$, por lo que esperamos que de las iteraciones de los 3 métodos, este sea el que logre una convergencia más rápida, tal como se observa a continuación.
Solución exacta $\vec{x} = (3, 4, -5) $.
Can you write Python code for it?

# Aplicación de los métodos iterativos
x_J = Matrix([[0], [0], [0]])
x_J_list = [x_J]
x_G = Matrix([[0], [0], [0]])
x_G_list = [x_G]
x_SOR = Matrix([[0], [0], [0]])
x_SOR_list = [x_SOR]
for i in range(11):
    x_J = T_J*x_J + c_J
    x_J_list.append(x_J)
    x_G = T_G*x_G + c_G
    x_G_list.append(x_G)
    x_SOR = T_SOR*x_SOR + c_SOR
    x_SOR_list.append(x_SOR)

# Método de Jacobi
M = x_J
for i in range(1, len(x_J_list)):
    print(x_J_list[i].evalf())

# Método de Gauss Seidel
M = x_G
for i in range(1, len(x_G_list)):
    print(x_G_list[i].evalf())

# Método de SOR
M = x_SOR
for i in range(1, len(x_SOR_list)):
    print(x_SOR_list[i].evalf())


I am working on the file "Código/iterativos_matriciales.ipynb".
The first task is:
Para el caso del método de SOR, es necesario asignar un valor para el parámetro de relajación, en este caso $\omega =$ 1,25.
Can you write Python code for it?


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Creamos nuestro array de datos:
Can you write Python code for it?


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
El problema viene cuando se quiere vectorizar esta función y permitir que h pueda ser un array de alturas. Esto es muy conveniente cuando queremos pintar alguna propiedad con matplotlib, por ejemplo.
Se intuye que hay dos formas de hacer esto: utilizando funciones de NumPy o iterando por cada elemento del array. La primera solución se hace farragosa, y la segunda, gracias a la proverbial lentitud de Python, es extremadamente lenta. Mi amigo Álex y yo llevamos pensando sobre este problema años, y nunca hemos llegado a una solución satisfactoria (incluso encontramos algunos bugs en numpy.piecewise por el camino). Este año decidimos cerrar este asunto definitivamente así que con el equipo AeroPython exploramos varias implementaciones distintas. Hasta que por fin lo conseguimos: usamos numba para acelerar los bucles.

Como podéis leer en la discusión original, la función de la primera columna está escrita en C++. ¿Impresionado? ;)
Solución de Navier de una placa plana
Para mi proyecto fin de carrera me encontré con la necesidad de calcular la deflexión de una placa rectangular, simplemente apoyada en sus cuatro bordes (es decir, los bordes pueden girar: no están empotrados) sometida a una carga transversal. Este problema tiene solución analítica conocida desde hace tiempo, hallada por Navier:
$$w(x,y) = \sum_{m=1}^\infty \sum_{n=1}^\infty \frac{a_{mn}}{\pi^4 D}\,\left(\frac{m^2}{a^2}+\frac{n^2}{b^2}\right)^{-2}\,\sin\frac{m \pi x}{a}\sin\frac{n \pi y}{b}$$
siendo $a_{mn}$ los coeficientes de Fourier de la carga aplicada. Como veis, para cada punto $(x, y)$ hay que hacer una doble suma en serie; si encima queremos evaluar esto en un meshgrid, necesitamos un cuádruple bucle. Ya se anticipa que por muy hábiles que estemos, a Python le va a costar.
La clave estuvo, una vez más, en usar numba para optimizar los bucles. En GitHub tenéis el código completo, pero la parte importante es esta:
Can you write Python code for it?

@numba.jit(nopython=True)
def a_mn_point(P, a, b, xi, eta, mm, nn):
    """Navier series coefficient for concentrated load.

    """
    return 4 * P * sin(mm * pi * xi / a) * sin(nn * pi * eta / b) / (a * b)

@numba.jit(nopython=True)
def plate_displacement(xx, yy, ww, a, b, P, xi, eta, D, max_m, max_n):
    max_i, max_j = ww.shape
    for mm in range(1, max_m):
        for nn in range(1, max_n):
            for ii in range(max_i):
                for jj in range(max_j):
                    a_mn = a_mn_point(P, a, b, xi, eta, mm, nn)
                    ww[ii, jj] += (a_mn / (mm**2 / a**2 + nn**2 / b**2)**2
                                   * sin(mm * pi * xx[ii, jj] / a)
                                   * sin(nn * pi * yy[ii, jj] / b)
                                   / (pi**4 * D))


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
En este caso la traza es inservible y especificar los tipos de entrada no va a ayudar. Solo para verificar, vamos a ver qué pasa con el rendimiento si no forzamos el modo estricto:
Can you write Python code for it?

busca_min_jit_object = numba.jit()(busca_min)

%timeit busca_min_jit_object(data)


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Parece que los tiempos son más o menos parecidos, pero al menos ya no tengo dos objetos en memoria que van a crecer de manera aleatoria. Vamos a ver ahora cómo nos puede ayudar numba a acelerar este código.
Paso 2: aplicando numba.jit(nopython=True)
Como hemos dicho antes, vamos a forzar que numba funcione en modo nopython para garantizar que obtenemos una mejora en el rendimiento. Si intentamos compilar la función definida en primer lugar va a fallar, porque ya hemos dicho más arriba que una de las condiciones es que no se puede asignar memoria a objetos nuevos:
Can you write Python code for it?

busca_min_jit = numba.jit(nopython=True)(busca_min)
busca_min_jit(data)


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Paso 1: analizar el código
Lo primero que pensé cuando vi esta función es que no me gustaba nada hacer append a esas dos listas tantas veces. Pero a continuación me pregunté si realmente tendrían tantos elementos... averigüémoslo:
Can you write Python code for it?

mx, my = busca_min(data)
mx.size / data.size


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Pocas ganancias respecto a la función sin compilar. ¿Qué pasa si intentamos lo mismo con la segunda función?
Can you write Python code for it?

busca_min_np_jit = numba.jit(nopython=True)(busca_min_np)
busca_min_np_jit(data)


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Por tanto, vemos que la versión con numba es el doble de rápida que la versión con Cython. Sobre gustos no hay nada escrito: yo por ejemplo valoro no «salirme» de Python usando numba mientras que a otro puede no importarle incluir especificaciones de tipos como en Cython. Los números, eso sí, son los números :)
Más casos de éxito
La atmósfera estándar
El cálculo de propiedades termodinámicas de la atmósfera estándar es un problema clásico que todo aeronáutico ha afrontado alguna vez muy al principio de su carrera formativa. La teoría es simple: imponemos una ley de variación de la temperatura con la altura $T = T(h)$, la presión se obtiene por consideraciones hidrostáticas $p = p(T)$ y la densidad por la ecuación de los gases ideales $\rho = \rho(p, T)$. La particularidad de la atmósfera estándar es que imponemos que la variación de la temperatura con la altura es una función simplificada y definida a trozos, así que calcular temperatura, presión y densidad dada una altura se parece mucho a hacer esto:
Can you write Python code for it?


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Tenemos que más de un 10 % de los elementos de la matriz cumplen la condición de ser «mínimos locales», así que no es nada despreciable. Esto en nuestro ejemplo hace un total de más de 400 000 elementos:
Can you write Python code for it?


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Vamos a empezar por importar los paquetes necesarios e inicializar la semilla del generador de números aleatorios:
Can you write Python code for it?

import numpy as np
import numba

np.random.seed(0)


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Veamos qué ocurre ahora:
Can you write Python code for it?

busca_min_np_jit(data)

np.testing.assert_array_equal(busca_min(data)[0], busca_min_np_jit(data)[0])
np.testing.assert_array_equal(busca_min(data)[1], busca_min_np_jit(data)[1])

%timeit busca_min_np_jit(data)


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Y evalúo los rendimientos:
Can you write Python code for it?

%timeit busca_min(data)

%timeit busca_min_np(data)


I am working on the file "Cómo acelerar tu código Python con numba.ipynb".
The first task is:
Y voy a copiar descaradamente la función de Kiko:
Can you write Python code for it?

def busca_min(malla):
    minimosx = []
    minimosy = []
    for i in range(1, malla.shape[1]-1):
        for j in range(1, malla.shape[0]-1):
            if (malla[j, i] < malla[j-1, i-1] and
                malla[j, i] < malla[j-1, i] and
                malla[j, i] < malla[j-1, i+1] and
                malla[j, i] < malla[j, i-1] and
                malla[j, i] < malla[j, i+1] and
                malla[j, i] < malla[j+1, i-1] and
                malla[j, i] < malla[j+1, i] and
                malla[j, i] < malla[j+1, i+1]):
                minimosx.append(i)
                minimosy.append(j)

    return np.array(minimosx), np.array(minimosy)

busca_min(data)


I am working on the file "D3Notes/D3.js Workbook.ipynb".
The first task is:
Nice Venn diagram, nothing D3 specific though
Can you write Python code for it?

%%HTML
<style>

@import url(../style.css?aea6f0a);

circle {
  fill: none;
  fill-opacity: .2;
  stroke: black;
  stroke-width: 1.5px;
}

</style>

<svg width="720" height="240">
  <g transform="translate(0,128)">
    <g transform="translate(300)">
      <circle r="110" style="fill: rgb(49, 130, 189);"></circle>
      <text y="-120" dy=".35em" text-anchor="middle" style="font-weight: bold;">Data</text>
      <text x="-50" dy=".35em" text-anchor="middle">Enter</text>
    </g>
    <text x="360" dy=".35em" text-anchor="middle">Update</text>
    <g transform="translate(420)">
      <circle r="110" style="fill: rgb(230, 85, 13);"></circle>
      <text y="-120" dy=".35em" text-anchor="middle" style="font-weight: bold;">Elements</text>
      <text x="50" dy=".35em" text-anchor="middle">Exit</text>
    </g>
  </g>
</svg>


I am working on the file "D3Notes/D3.js Workbook.ipynb".
The first task is:
Tutorial #2 - Load External Data
tutorial
source code
Load data from and external '.tsv' file
from source
"To use this data in a web browser, we need to download the file from a web server and then parse it, which converts the text of the file into usable JavaScript objects. Fortunately, these two tasks can be performed by a single function, d3.tsv.
Loading data introduces a new complexity: downloads are asynchronous. When you call d3.tsv, it returns immediately while the file downloads in the background. At some point in the future when the download finishes, your callback function is invoked with the new data, or an error if the download failed. In effect your code is evaluated out of order:"
```javascript
// 1. Code here runs first, before the download starts.
d3.tsv("data.tsv", function(error, data) {
  // 3. Code here runs last, after the download finishes.
});
// 2. Code here runs second, while the file is downloading.
```
Can you write Python code for it?

%%writefile tutorial.bar2.html
<!DOCTYPE html>
<meta charset="utf-8">
<style>

.chart rect {
  fill: steelblue;
}

.chart text {
  fill: white;
  font: 10px sans-serif;
  text-anchor: end;
}

</style>
<svg class="chart"></svg>
<script src="//d3js.org/d3.v3.min.js"></script>
<script>

var width = 420,
    barHeight = 20;

var x = d3.scale.linear()
    .range([0, width]);

var chart = d3.select(".chart")
    .attr("width", width);

d3.tsv("data.tsv", type, function(error, data) {
  x.domain([0, d3.max(data, function(d) { return d.value; })]);

  chart.attr("height", barHeight * data.length);

  var bar = chart.selectAll("g")
      .data(data)
    .enter().append("g")
      .attr("transform", function(d, i) { return "translate(0," + i * barHeight + ")"; });

  bar.append("rect")
      .attr("width", function(d) { return x(d.value); })
      .attr("height", barHeight - 1);

  bar.append("text")
      .attr("x", function(d) { return x(d.value) - 3; })
      .attr("y", barHeight / 2)
      .attr("dy", ".35em")
      .text(function(d) { return d.value; });
});

function type(d) {
  d.value = +d.value; // coerce to number
  return d;
}

</script>

%%writefile data.tsv
name	value
Locke	4
Reyes	8
Ford	15
Jarrah	16
Shephard	23
Kwon	42

from IPython.display import IFrame
IFrame("tutorial.bar2.html", width=850, height=150)


I am working on the file "D3Notes/D3.js Workbook.ipynb".
The first task is:
Tutorial 3 - Rotating Columns
tutorial
source code
Important notes
Fixed width vs calculated width
from source
"We previously multiplied the var barHeight by the index of each data point (0, 1, 2, …) to produce fixed-height bars. The resulting chart’s height thus depended on the size of the dataset. But here the opposite behavior is desired: the chart width is fixed and the bar width variable. So rather than fix the barHeight, now we compute the barWidth by dividing the available chart width by the size of the dataset, data.length."
...
javascript
d3.tsv("data.tsv", type, function(error, data) {
  y.domain([0, d3.max(data, function(d) { return d.value; })]);
javascript
  var barWidth = width / data.length;
...
Loading Labels
...
javascript
chart.append("g")
    .attr("class", "y axis")
    .call(yAxis)
  .append("text")
    .attr("transform", "rotate(-90)")
    .attr("y", 6)
    .attr("dy", ".71em")
    .style("text-anchor", "end")
    .text("Frequency");
...
"Unit-appropriate number formatting also improves legibility by tailoring the display to your data. Since our chart displays relative frequency, percentages are more appropriate than the default behavior which shows a number between 0 and 1. A format string as the second argument to axis.ticks will customize the tick formatting, and the scale will automatically choose a precision appropriate to the tick interval."
...
javascript
var yAxis = d3.svg.axis()
    .scale(y)
    .orient("left")
    .ticks(10, "%");
...
Can you write Python code for it?

%%writefile tutorial.bar3.vertical.html
<!DOCTYPE html>
<meta charset="utf-8">
<style>

.bar {
  fill: steelblue;
}

.bar:hover {
  fill: brown;
}

.axis--x path {
  display: none;
}

</style>
<svg width="960" height="500"></svg>
<script src="https://d3js.org/d3.v4.min.js"></script>
<script>

var svg = d3.select("svg"),
    margin = {top: 20, right: 20, bottom: 30, left: 40},
    width = +svg.attr("width") - margin.left - margin.right,
    height = +svg.attr("height") - margin.top - margin.bottom;

var x = d3.scaleBand().rangeRound([0, width]).padding(0.1),
    y = d3.scaleLinear().rangeRound([height, 0]);

var g = svg.append("g")
    .attr("transform", "translate(" + margin.left + "," + margin.top + ")");

d3.tsv("data.tutorial3.tsv", function(d) {
  d.frequency = +d.frequency;
  return d;
}, function(error, data) {
  if (error) throw error;

  x.domain(data.map(function(d) { return d.letter; }));
  y.domain([0, d3.max(data, function(d) { return d.frequency; })]);

  g.append("g")
      .attr("class", "axis axis--x")
      .attr("transform", "translate(0," + height + ")")
      .call(d3.axisBottom(x));

  g.append("g")
      .attr("class", "axis axis--y")
      .call(d3.axisLeft(y).ticks(10, "%"))
    .append("text")
      .attr("transform", "rotate(-90)")
      .attr("y", 6)
      .attr("dy", "0.71em")
      .attr("text-anchor", "end")
      .text("Frequency");

  g.selectAll(".bar")
    .data(data)
    .enter().append("rect")
      .attr("class", "bar")
      .attr("x", function(d) { return x(d.letter); })
      .attr("y", function(d) { return y(d.frequency); })
      .attr("width", x.bandwidth())
      .attr("height", function(d) { return height - y(d.frequency); });
});

</script>

%%writefile data.tutorial3.tsv
letter	frequency
A	.08167
B	.01492
C	.02782
D	.04253
E	.12702
F	.02288
G	.02015
H	.06094
I	.06966
J	.00153
K	.00772
L	.04025
M	.02406
N	.06749
O	.07507
P	.01929
Q	.00095
R	.05987
S	.06327
T	.09056
U	.02758
V	.00978
W	.02360
X	.00150
Y	.01974
Z	.00074

from IPython.display import...


I am working on the file "D3Notes/D3.js Workbook.ipynb".
The first task is:
Tutorial 4 - Selecting Elements, Entering and Exiting
Selecting Elements
Entering Elements
To size, move, etc, simply enter the
from source
"...By appending to the enter selection, we can create new circles for any missing data."
```javascript
var svg = d3.select("svg");
var circle = svg.selectAll("circle")
    .data([32, 57, 112, 293]);
var circleEnter = circle.enter().append("circle");
```
Modify attributes
javascript
circleEnter.attr("cy", 60);
circleEnter.attr("cx", function(d, i) { return i * 100 + 30; });
circleEnter.attr("r", function(d) { return Math.sqrt(d); });
Exiting Elements
from source
"... you have too many existing elements, and you want to remove some of them. Again you can select nodes and remove them manually, but the exit selection computed by a data join is more powerful."
```javascript
var circle = svg.selectAll("circle")
    .data([32, 57]);
...
circle.exit().remove();
```
Can you write Python code for it?

# source: https://bost.ocks.org/mike/constancy/
from IPython.display import IFrame
IFrame("https://bost.ocks.org/mike/constancy/", width=970, height=530)

%pwd

from IPython.display import IFrame
IFrame("https://bost.ocks.org/mike/join/", width=970, height=530)


I am working on the file "DANDP3.ipynb".
The first task is:
As seen above, a query for 10 postcodes that do not fit the format now returns an empty list, showing the problem was fixed.
OBS: A similar process can be done for phone numbers, however it is an intricate process for brazilian phone numbers are there are many different formats. (At this point numbers can have 8 to 10 digits excluding area code, there is an optional designation of operator, and other complicating issues.) This will require a little more time consuming research.
3. Incorrect Postcode
Can you write Python code for it?

t = sp.find({'address.postcode' : {'$regex' : '^2'}})
pprint.pprint(list(t))


I am working on the file "DANDP3.ipynb".
The first task is:
How many street names start with X? And Z?
Can you write Python code for it?

x = sp.find({'data_prim':'way', 'name': {'$regex': street_starts_with('X')}}).count()
print 'Starting with X:', x
z = sp.find({'data_prim':'way', 'name': {'$regex': street_starts_with('Z')}}).count()
print 'Starting with Z:', z


I am working on the file "DANDP3.ipynb".
The first task is:
It seems there is a mix of incorrect format, such as '05025010' instead of '05025-010' and typos like an extra number or a missing one. My solution is in the first case, reformat, the second, discard. This was included in shape_element, and the data re-parsed and re-loaded into MongoDB.
Can you write Python code for it?


I am working on the file "DANDP3.ipynb".
The first task is:
The above output is totally unexpected, as we should see only "node", "relation" or "way". A little research was helpful in pinpointing the issue, mainly that relation data primitive were translating into nodes with incorrect 'type' assignment, because many of the relations themselves had a 'type' attribute or tag child. Full explanation on this reference. The forum post also contains the solution, which is to not use the word "type". I decided to call this key "data_prim" in short for "data primitive".
After fixing the Python code, making a new JSON file, clearig the old collection off the database and loading the new one, when we run the queries below the expected outputs are produced.
Can you write Python code for it?

types = sp.distinct('data_prim')
pprint.pprint(types)

cursor = sp.find({'data_prim' : 'way'})
a = len(list(cursor))
print a, 'ways'
cursor = sp.find({'data_prim' : 'node'})
b = len(list(cursor))
print b, 'nodes'
cursor = sp.find({'data_prim' : 'relation'})
c =  len(list(cursor))
print c, 'relations'
print 'discrepancy:', 2168319 - a - b - c


I am working on the file "DANDP3.ipynb".
The first task is:
The above postcode is incorrect, it is supposed to be 02545-000.
4. Missing Postcodes (CEPs)
Can you write Python code for it?

a = sp.find({'address' : {'$exists' : 1}}).count()
b = sp.find({'address' : {'$exists' : 1}, 'address.postcode' : {'$exists' : 1}}).count()
c = sp.find({'address' : {'$exists' : 1}, 'address.postcode' : {'$exists' : 0}}).count()
print 'number of addresses:', a
print 'number of addresses with CEP:', b
print 'number of addresses without CEP:', c
print 'percentage of addresses missing CEP:',int((float(c)/float(a))*100),'%'


I am working on the file "DANDP3.ipynb".
The first task is:
The original output of this query was 4. It seems odd that a metropolis so big would have 4 relations. Investigating further, I found there were 261671 ways and 1900291 nodes.
Can you write Python code for it?


I am working on the file "DANDP3.ipynb".
The first task is:
This shows this data is severely incomplete and does not lend itself for statistics, not on religion, anyway. There are lenty more than 2 Jewish places of worship in Sao Paulo.
 number of pizza places:
Can you write Python code for it?


I am working on the file "DANDP3.ipynb".
The first task is:
number of documents by data primitive:
Can you write Python code for it?

cursor = sp.find({'data_prim' : 'way'})
a = len(list(cursor))
print a, 'ways'
cursor = sp.find({'data_prim' : 'node'})
b = len(list(cursor))
print b, 'nodes'
cursor = sp.find({'data_prim' : 'relation'})
c =  len(list(cursor))
print c, 'relations'


I am working on the file "DANDP3.ipynb".
The first task is:
top 10 religions
Can you write Python code for it?

pipe = [{'$match' : {'amenity': 'place_of_worship'}},
        {'$group': {'_id': '$religion', 'count': {'$sum': 1}}},
        {'$sort' : {'count': -1}},
        {'$limit' : 10},
        {'$project' : {'religion' : 1, 'count': 1}}
        ]
c = sp.aggregate(pipe)
pprint.pprint(list(c))


I am working on the file "DANDP3.ipynb".
The first task is:
unique user ids:
Can you write Python code for it?


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab1_MNIST_DataLoader-checkpoint.ipynb".
The first task is:
Data download
We will download the data onto the local machine. The MNIST database is a standard set of handwritten digits that has been widely used for training and testing of machine learning algorithms. It has a training set of 60,000 images and a test set of 10,000 images with each image being 28 x 28 grayscale pixels. This set is easy to use visualize and train on any computer.
Can you write Python code for it?

# Functions to load MNIST images and unpack into train and test set.
# - loadData reads image data and formats into a 28x28 long array
# - loadLabels reads the corresponding labels data, 1 for each image
# - load packs the downloaded image and labels data into a combined format to be read later by
#   CNTK text reader

def loadData(src, cimg):
    print ('Downloading ' + src)
    gzfname, h = urlretrieve(src, './delete.me')
    print ('Done.')
    try:
        with gzip.open(gzfname) as gz:
            n = struct.unpack('I', gz.read(4))
            # Read magic number.
            if n[0] != 0x3080000:
                raise Exception('Invalid file: unexpected magic number.')
            # Read number of entries.
            n = struct.unpack('>I', gz.read(4))[0]
            if n != cimg:
                raise Exception('Invalid file: expected {0} entries.'.format(cimg))
            crow = struct.unpack('>I', gz.read(4))[0]
            ccol = struct.unpack('>I', gz.read(4))[0]
            if crow != 28 or ccol != 28:
                raise Exception('Invalid file: expected 28 rows/cols per image.')
            # Read data.
            res = np.fromstring(gz.read(cimg * crow * ccol), dtype = np.uint8)
    finally:
        os.remove(gzfname)
    return res.reshape((cimg, crow * ccol))

def loadLabels(src, cimg):
    print ('Downloading ' + src)
    gzfname, h = urlretrieve(src, './delete.me')
    print ('Done.')
    try:
        with gzip.open(gzfname) as gz:
            n = struct.unpack('I', gz.read(4))
            # Read magic number.
            if n[0] != 0x1080000:
                raise Exception('Invalid file: unexpected magic number.')
            # Read number of entries.
            n = struct.unpack('>I', gz.read(4))
            if n[0] != cimg:
                raise Exception('Invalid file: expected {0} rows.'.format(cimg))
            # Read labels.
            res = np.fromstring(gz.read(cimg), dtype = np.uint8)
    finally:
        os.remove(gzfname)
    return res.reshape((cimg, 1))

def...


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab1_MNIST_DataLoader-checkpoint.ipynb".
The first task is:
Save the images
Save the images in a local directory. While saving the data we flatten the images to a vector (28x28 image pixels becomes an array of length 784 data points).

The labels are encoded as 1-hot encoding (label of 3 with 10 digits becomes 0001000000, where the first index corresponds to digit 0 and the last one corresponds to digit 9.
Can you write Python code for it?

# Save the data files into a format compatible with CNTK text reader
def savetxt(filename, ndarray):
    dir = os.path.dirname(filename)

    if not os.path.exists(dir):
        os.makedirs(dir)

    if not os.path.isfile(filename):
        print("Saving", filename )
        with open(filename, 'w') as f:
            labels = list(map(' '.join, np.eye(10, dtype=np.uint).astype(str)))
            for row in ndarray:
                row_str = row.astype(str)
                label_str = labels[row[-1]]
                feature_str = ' '.join(row_str[:-1])
                f.write('|labels {} |features {}\n'.format(label_str, feature_str))
    else:
        print("File already exists", filename)

# Save the train and test files (prefer our default path for the data)
data_dir = os.path.join("..", "Examples", "Image", "DataSets", "MNIST")
if not os.path.exists(data_dir):
    data_dir = os.path.join("data", "MNIST")

print ('Writing train text file...')
savetxt(os.path.join(data_dir, "Train-28x28_cntk_text.txt"), train)

print ('Writing test text file...')
savetxt(os.path.join(data_dir, "Test-28x28_cntk_text.txt"), test)

print('Done')


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab1_MNIST_DataLoader-checkpoint.ipynb".
The first task is:
Visualize the data
Here, we use matplotlib to display one of the training images and it's associated label.
Can you write Python code for it?

# Plot a random image
sample_number = 5001
plt.imshow(train[sample_number,:-1].reshape(28,28), cmap="gray_r")
plt.axis('off')
print("Image Label: ", train[sample_number,-1])


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:

Model Creation
Our multi-layer perceptron will be relatively simple with 2 hidden layers (num_hidden_layers). The number of nodes in the hidden layer being a parameter specified by hidden_layers_dim. The figure below illustrates the entire model we will use in this tutorial in the context of MNIST data.

If you are not familiar with the terms hidden_layer and number of hidden layers, please review the module 3 course videos.
Each Dense layer (as illustrated below) shows the input dimensions, output dimensions and activation function that layer uses. Specifically, the layer below shows: input dimension = 784 (1 dimension for each input pixel), output dimension = 400 (number of hidden nodes, a parameter specified by the user) and activation function being relu.

In this model we have 2 dense layer called the hidden layers each with an activation function of relu.  These are followed by the dense output layer with no activation.
The output dimension (a.k.a. number of hidden nodes) in the 2 hidden layer is set to 400. The number of hidden layers is 2.
The final output layer emits a vector of 10 values. Since we will be using softmax to normalize the output of the model we do not use an activation function in this layer. The softmax operation comes bundled with the loss function we will be using later in this tutorial.
Can you write Python code for it?

num_hidden_layers = 2
hidden_layers_dim = 400
#hidden_layers_dim = 50


I am working on the file "DAT236x Deep Learning Explained/.ipynb_checkpoints/Lab3_MultiLayerPerceptron-checkpoint.ipynb".
The first task is:

Run the trainer
We are now ready to train our fully connected neural net. We want to decide what data we need to feed into the training engine.
In this example, each iteration of the optimizer will work on minibatch_size sized samples. We would like to train on all 60000 observations. Additionally we will make multiple passes through the data specified by the variable num_sweeps_to_train_with. With these parameters we can proceed with training our simple multi-layer perceptron network.
Can you write Python code for it?

# Create the reader to training data set
reader_train = create_reader(train_file, True, input_dim, num_output_classes)

# Map the data streams to the input and labels.
input_map = {
    label  : reader_train.streams.labels,
    input  : reader_train.streams.features
}

# Run the trainer on and perform model training
training_progress_output_freq = 500

plotdata = {"batchsize":[], "loss":[], "error":[]}

for i in range(0, int(num_minibatches_to_train)):

    # Read a mini batch from the training data file
    data = reader_train.next_minibatch(minibatch_size, input_map = input_map)

    trainer.train_minibatch(data)
    batchsize, loss, error = print_training_progress(trainer, i, training_progress_output_freq, verbose=1)

    if not (loss == "NA" or error =="NA"):
        plotdata["batchsize"].append(batchsize)
        plotdata["loss"].append(loss)
        plotdata["error"].append(error)