I am working on the file "Session1/code/Pandas.ipynb".
The first task is:
pandas allows access to its elements using labels:
Can you write Python code for it?


I am working on the file "Session2/Clustering.ipynb".
The first task is:
Gap Statistic
The Gap Statistic was developped by reasearchers from stanford, and relies on the variation of the within-cluster variation. To have more details, you
To compute it we will use the implementation of https://github.com/milesgranger/gap_statistic
TO DO
* Install the library gap statistic
* Run the optimalK function on df_micro with a maximum number of clusters set to 30
Can you write Python code for it?

def optimalK(data, nrefs=3, maxClusters=15):
    """
    Calculates KMeans optimal K using Gap Statistic from Tibshirani, Walther, Hastie
    Params:
        data: ndarry of shape (n_samples, n_features)
        nrefs: number of sample reference datasets to create
        maxClusters: Maximum number of clusters to test for
    Returns: (gaps, optimalK)
    """
    gaps = np.zeros((len(range(1, maxClusters)),))
    resultsdf = pd.DataFrame({'clusterCount':[], 'gap':[]})
    for gap_index, k in enumerate(range(1, maxClusters)):

        # Holder for reference dispersion results
        refDisps = np.zeros(nrefs)

        # For n references, generate random sample and perform kmeans getting resulting dispersion of each loop
        for i in range(nrefs):

            # Create new random reference set
            randomReference = np.random.random_sample(size=data.shape)

            # Fit to it
            km = KMeans(k)
            km.fit(randomReference)

            refDisp = km.inertia_
            refDisps[i] = refDisp

        # Fit cluster to original data and create dispersion
        km = KMeans(k)
        km.fit(data)

        origDisp = km.inertia_

        # Calculate gap statistic
        gap = np.log(np.mean(refDisps)) - np.log(origDisp)

        # Assign this loop's gap statistic to gaps
        gaps[gap_index] = gap

        resultsdf = resultsdf.append({'clusterCount':k, 'gap':gap}, ignore_index=True)

    return (gaps.argmax() + 1, resultsdf)  # Plus 1 because index of 0 means 1 cluster is optimal, index 2 = 3 clusters are optimal

k, gapdf = optimalK(df_micro, nrefs=5, maxClusters=30)
print ('Optimal k is: ', k)

plt.plot(gapdf.clusterCount, gapdf.gap, linewidth=3)
plt.scatter(gapdf[gapdf.clusterCount == k].clusterCount, gapdf[gapdf.clusterCount == k].gap, s=250, c='r')
plt.xlabel('Cluster Count')
plt.ylabel('Gap Value')
plt.title('Gap Values by Cluster Count')
plt.show()


I am working on the file "Session2/Clustering.ipynb".
The first task is:
How to set k ?
The Elbow method
The Elbow method is a method of interpretation and validation of consistency within cluster analysis designed to help finding the appropriate number of clusters in a dataset. This method looks at the percentage of variance explained as a function of the number of clusters. One should choose a number of clusters so that adding another cluster doesn't give much better modeling of the data.
If one plots the percentage of variance explained by the clusters against the number of clusters the first clusters will add much information (explain a lot of variance), but at some point the marginal gain in explained variance will drop, giving an angle in the graph. The number of clusters is chosen at this point, hence the "elbow criterion"
Can you write Python code for it?

cluster_range = range(1,50)
cluster_errors = []

for num_clusters in cluster_range:
  clust = KMeans(n_clusters=num_clusters, random_state=0, n_init=10)
  clust.fit(df_micro)
  cluster_errors.append(clust.inertia_)

clusters_df = pd.DataFrame( { "num_clusters":cluster_range, "cluster_errors": cluster_errors } )
clusters_df[0:10]
plt.figure(figsize=(12,6))
plt.plot( clusters_df.num_clusters, clusters_df.cluster_errors, marker = "o" )


I am working on the file "Session2/Clustering.ipynb".
The first task is:
Initialisation : be careful!
The initialisation step requires the to set the number of clusters K. To this end, one can either use a priori information and set it manually but there also exists several approach to determine it, including for instance, the Elbow method and the Gap Statistic.
In addition one need to initialise the centroid. Commonly used initialization methods are Forgy and Random Partition. The Forgy method randomly chooses $k$ observations from the data set and uses them as the initial means. The Random Partition method first randomly assigns a cluster to each observation and then proceeds to the update step, thus computing the initial mean to be the centroid of the cluster's randomly assigned points. The Forgy method tends to spread the initial means out, while Random Partition places all of them close to the center of the data set. For expectation maximization and standard k-means algorithms, the Forgy method of initialization is preferable.
TO DO :
* run Kmeans twice (with K=10 to speed up things) on the df_micro data with random intialisations then compare the obtained labels from both runs with the adjusted rand index from the metric library (available in sklearn).
 same than before but this time set the number of initialisation to 50.
 swith the initialisation method to Kmeans++ and run previous experiments once again
Can you write Python code for it?

from sklearn import metrics

# 1 random initisalition
kmeans = KMeans(n_clusters=10, init='random')
res = kmeans.fit(df_micro)
label_1 = res.labels_
centroids_1 = res.cluster_centers_

kmeans = KMeans(n_clusters=10, init='random')
res = kmeans.fit(df_micro)
label_2 = res.labels_
centroids_2 = res.cluster_centers_

metrics.adjusted_rand_score(label_1, label_2)

# 50 random initialisations
kmeans = KMeans(n_clusters=10, init='random', n_init=50)
res = kmeans.fit(df_micro)
label_1 = res.labels_
centroids_1 = res.cluster_centers_

kmeans = KMeans(n_clusters=10, init='random',n_init=50)
res = kmeans.fit(df_micro)
label_2 = res.labels_
centroids_2 = res.cluster_centers_

metrics.adjusted_rand_score(label_1, label_2)

# 50 initisalition and improve strategy
kmeans = KMeans(n_clusters=10, init='k-means++', n_init=50)
res = kmeans.fit(df_micro)
label_1 = res.labels_
centroids_1 = res.cluster_centers_

kmeans = KMeans(n_clusters=10, init='k-means++',n_init=50)
res = kmeans.fit(df_micro)
label_2 = res.labels_
centroids_2 = res.cluster_centers_

metrics.adjusted_rand_score(label_1, label_2)


I am working on the file "Session2/Clustering.ipynb".
The first task is:
Running K-means
We are now ready to run Kmeans! To this end, we will use the implementation given by scikit learn.
As for the previous session, we first initialise the algorithm then fit it to our data.
TO DO :
* run Kmeans and set the number of clusters to 10
* describe the obtained labels (distribution of objects among them)
* print the obtained centroids
* use the pca plot to visualise the labels obtained with Kmeans. To this end, you just need to change the parameters c in the previous scatter plot and to replace the current one with the obtained labels.
Can you write Python code for it?

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=10)
res = kmeans.fit(df_micro)
labels = res.labels_
res.cluster_centers_

plt.scatter(projected[:, 0], projected[:, 1],
            c=labels, edgecolor='none', alpha=0.5,
            cmap=plt.cm.get_cmap('nipy_spectral', 10))
plt.xlabel('component 1')
plt.ylabel('component 2')
plt.colorbar();


I am working on the file "Session2/Clustering.ipynb".
The first task is:
Visualisation with PCA
Wikipedia
Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. The number of distinct principal components is equal to the smaller of the number of original variables or the number of observations minus one. This transformation is defined in such a way that the first principal component has the largest possible variance (that is, accounts for as much of the variability in the data as possible), and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to the preceding components. The resulting vectors are an uncorrelated orthogonal basis set.
Basically we will only use PCA for a visualisation purpose. Our goal is to get a 2D visualisation of a 5717-dimensional dataset. Just keep in mind that PCA relies on the assumption that the data are linearly separable, but it's not always the case!
TO DO : execute the following code!
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
Y_sklearn = pca.fit(df_micro)
projected = pca.fit_transform(df_micro)

import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (10, 6)

plt.scatter(projected[:, 0], projected[:, 1],
            c=df.Pop.astype('category').cat.codes, edgecolor='none', alpha=0.5,
            cmap=plt.cm.get_cmap('nipy_spectral', 10))
plt.xlabel('component 1')
plt.ylabel('component 2')
plt.colorbar();


I am working on the file "Session2/Clustering.ipynb".
The first task is:
What about mixture model ?
In the probabilistic framework of mixture models, we assume that the data are generated according to a mixture of probability density functions, with cluster-specific parameters.
$$ p(x,\theta) = \sum_k \pi_k f(x,\theta_k)$$
where, $\pi_k$ can be interpreted as the proportion of each cluster and $\theta_k$ is the set of parameters. For instance, in the case of a Gaussian mixture we have $\theta_k = (\mu_k,\sigma_k)$.
Then, the goal is to estimate the set of parameters $\theta_k$ and to compute the partition of the objects which is assumed to a hidden variable of the model.
Can you write Python code for it?

# 1 run of the Gaussian mixture models
from sklearn import mixture
gmm = mixture.GaussianMixture(n_components=10).fit(df_micro)
labels_gmm = gmm.predict(df_micro)


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Comparison to a Possion distribution

why would you compare to a Poisson?
for Possion, $E[X] = Var[X]$
or equivalent: Fano-factor $F = \frac{Var[X]}{E[X]} = 1$
Can you write Python code for it?

mu = np.mean(trueCFUs_all)
sigma= np.std(trueCFUs_all)
print "Mean: %f Fano factor %f" % (mu,sigma**2/mu)

# put the poisson distribution as a comparision which has the same mean as the data
x_ECDF,y_ECDF = ecdf(trueCFUs_all)

x = np.arange(0,100)
y =stats.poisson.cdf(x,mu)

plt.subplot(1,2,1)
plt.plot(x_ECDF, y_ECDF, linewidth=2)
plt.plot(x,y, linewidth=2)
plt.xlim([0,300])
plt.xlabel('#CFUs'), plt.ylabel('CDF'), plt.legend(['Data','Poisson fit'])

# DO THE PDF as well, much easier to see
#plt.figure()
plt.subplot(1,2,2)
sns.distplot(trueCFUs_all,norm_hist=True, kde=False)
plt.plot(x,stats.poisson.pmf(x,mu), linewidth=2)
plt.xlabel('#CFUs'), plt.ylabel('CDF'), plt.legend(['Poisson fit', 'Data'])


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Converting to actual CFU numbers
due to loss from injection to spleen
colonies_per_spleen = f * trueCFUs
Can you write Python code for it?

#just divide each dict entry by 'fraction'
tmpFunc = lambda x: np.ceil(x/fraction)

# apply this function to each dictionary entry
trueCFUs = {k: tmpFunc(v) for k,v in colonies_per_spleen.items()}

print "#Colonies per spleen"
print colonies_per_spleen["exp1"]

print "Estimated #CFU per spleen"
print trueCFUs["exp1"]


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Create some trees/colonies, calculate  number of CFUs within
Can you write Python code for it?

def pretty_print_tree(tree):
    for gen in tree:
        print gen

p0 = 0.4
nGens = 5
aTree = simulateTree(p0,nGens)
bTree = simulateTree(p0,nGens)
print 'First tree'
pretty_print_tree(aTree)

print 'Second tree'
pretty_print_tree(bTree)

#CFUs per generation
print '\n#CFUs, first tree'
print map(lambda x: sum(x==0), aTree)

print '\n#CFUs, second tree'
print map(lambda x: sum(x==0), bTree)


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Defining properties of stem cells

multipotent and able to differentiate
visual inspection of spleen colonies
spleen colonies contain diff. cells

- are capable of **self-renewal**
    - secondary "transplants"
    - cells within the spleen colonies still have stem cell props

BASIC DEFINITION OF STEM CELLS EVEN TODAY

# Basic Questions
- how is this balance between self-renewal and differentation regulated
- Theories
     - **deterministic** : tighly controlled, each cell knows what to do
     - **stochastic**: no control, seemingly random

# The data
> The distribution of colony forming cells among spleen colonies, Siminovitch 1963

- Table 2,3 of Siminovitch
- Number of CFU per colony (i.e. #stem cells per colony)

Can you write Python code for it?

colonies_per_spleen = dict()
colonies_per_spleen["exp1"] = np.concatenate([np.repeat(0,24),
                                      np.repeat(1,6),
                                      np.array([2,6,19,21,23,36])
                                      ])

colonies_per_spleen["exp2"] = np.concatenate([np.repeat(0,3),
                                      np.repeat(1,3),
                                      np.repeat(2,2),
                                      np.array([3,3, 20,20, 32])
                                      ])

colonies_per_spleen["exp3"]= np.concatenate([np.repeat(0,12),
                                      np.repeat(1,8),
                                      np.repeat(2,5),
                                      np.repeat(3,2),
                                      np.array([4,5,5,7,8,8,11,13,20,23,29,46])
                                      ])


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Distribution of #CFUs over time
Can you write Python code for it?

nTrees = 1000
nGens = 20
p0 = 0.4

# assemble a matrix of tree vs #CFUs(t)  (one row, one tree; one col one timepoint)
cfus_over_time = np.zeros((nTrees, nGens+1))
for i in range(nTrees):
    tree = simulateTree(p0, nGens)
    nCFU = map(lambda x: np.sum(x==0), tree)
    cfus_over_time[i,:] = nCFU

print cfus_over_time[0:5,:].astype(int)

import seaborn as sns

plt.plot(log10(cfus_over_time.T))
plt.xlabel('Generation')
plt.ylabel("log10(#CFU)")

# violin plot
plt.figure()
sns.violinplot((log10(cfus_over_time+1)), inner='points')
plt.xlabel('Generation')
plt.ylabel("log10(#CFU+1)")


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Efficiency of the CFUs

only a fraction $f$ of stem cells makes it into the spleen
measure $f$:
inject known amount of CFUs into mouse
remove spleen 2h after
disolve spleen, reinject, count

Kind of circular argument (3. is again a function of f)
<img src="figures/spleenScetch.png" alt="Drawing" style="width: 600px;"/>
Can you write Python code for it?

fraction = 0.17 # the fraction of CFUs that actually go to the spleen


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Exercise: Fitting the Birth Death model to the data

no likelihood available (at least not so simple -> branching processes)
simulate data, compare to observations
comparing the means/variance
histogram distance
Can you write Python code for it?

"Simulate a bunch of trees and get the #CFUs over time"
def sim_cfu_over_time(p0, nTrees, nGens):
    cfus_over_time = np.zeros((nTrees, nGens+1))
    for i in range(nTrees):
        tree = simulateTree(p0, nGens)
        nCFU = map(lambda x: np.sum(x==0), tree)
        cfus_over_time[i,:] = nCFU
    return cfus_over_time

"calculate a KS test between observed and simulated dists"
def distance_function(obs, sim):
    K,pval = stats.ks_2samp(obs,sim)
    return K, pval
    #return np.abs(np.mean(obs)- np.mean(sim))

p0_grid = np.linspace(0, 1, 50)
distances, pvals = np.zeros(p0_grid.shape), np.zeros(p0_grid.shape)
observedData = trueCFUs_all # experimental data after 10 days/20divisions

for i, p0 in enumerate(p0_grid):
    simData = sim_cfu_over_time(p0, nTrees = 100, nGens= 20)
    simData_lastGen = simData[:,-1]
    distances[i], pvals[i] = distance_function(obs= observedData, sim=simData_lastGen)

def plot_fitted_stoch_model(p0_array, K_array, pval_array, observedData):
    bestP0 = p0_array[argmin(K_array)]
    plt.subplot(2,2,1)
    plt.plot(p0_array, K_array)
    plt.vlines(bestP0,0,1)
    xlabel('p0'), ylabel('KS distance')

    plt.subplot(2,2,2)
    plt.plot(p0_array, log10(pvals))
    xlabel('p0'), ylabel('KS pval')
    bestpval = pval_array[argmin(K_array)]
    title(bestpval)

    plt.subplot(2,2,3)
    simData = sim_cfu_over_time(p0=0.4, nTrees = 100, nGens= 20)

    x,y =ecdf(observedData)
    plt.plot(x, y)

    x,y =ecdf(simData[:,-1])
    plt.plot(x, y)
    xlabel('#CFU'), ylabel('CDF')
    legend(['observed','simulated'])

    binning = np.arange(0,300,step=5)
    plt.subplot(2,2,4)
    plt.hist(observedData, bins=binning, histtype="stepfilled", alpha=.5)
    plt.hist(simData[:,-1], bins=binning, histtype="stepfilled", alpha=.5)
    xlabel('#CFU'), ylabel('PDF')

plot_fitted_stoch_model(p0_grid, distances, pvals, observedData)


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Hematopoiesis

best studied stem cell system
easy access
clean (celltypes are known)

Current view

Orkin, S. H. & Zon, L. I. Hematopoiesis: an evolving paradigm for stem cell biology. Cell 132, 631–44 (2008).

Orkin, S. H. & Zon, L. I. Hematopoiesis: an evolving paradigm for stem cell biology. Cell 132, 631–44 (2008).
Hematopoiesis in 1960

About the paper

Till, J. E., McCulloch, E. a. & Siminovitch, L. A stochastic model of stem cell proliferation, based on the growth of spleen colony-forming cells . Proc. Natl. Acad. Sci. U. S. A. 51, 29–36 (1964).

First evidence for stem cells by Till/McCulloch around 1960

The Radiation Sensitivity of Normal Mouse Bone Marrow Cells, Determined by Quantitative Marrow Transplantation into Irradiated Mice

This one is a follow up, analysing data from

Siminovitch, L., McCulloch, E. & Till, J. The distribution of colony forming cells among spleen colonies'

Evidence for stem cells

Inject bone marrow cells into irradiated mice

Marrow cells form colonies in spleen (german: 'Milz', part of blood/lymph-system):

Colony forming cells/units (CFU)

Analysis of the colonies/CFUs

Rare celltype: 1/10000 cells
Colonies contain different kinds of blood cells
multipotent
Colonies contain CFUs again (secondary transplants)
self renewal
Capable of rapid proliferation: $10^6$ cell progeny in 10 days
What's the number of divisions and the division time?
Can you write Python code for it?

nCells = 1e6
nDays  = 10
nDivisions = np.log2(nCells)
time_per_division = nDays * 24 / nDivisions
print "#Generations/Divisions: %d\n\nAverage generation time %.2f (hours)" %(round(nDivisions), time_per_division)


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Instead

also works the other way around:
estimate the posterior of $N$ for observed $f$ and $k$:

<p>
$$P(N|k,f)=\frac{L(k,f|N) \cdot \pi(N)}{\int L(k,f|N) \cdot \pi(N)\;dN}$$
Can you write Python code for it?

n = np.arange(300)
data = unique(colonies_per_spleen["exp1"]) #just considering the first experiment
posteriorArray = np.zeros((len(data), len(n)))
for i,d in enumerate(data):
    likelihood_n = binom.pmf(d, n, fraction)
    prior_n = 1 #constant
    posterior_n = likelihood_n * prior_n / np.sum(likelihood_n*prior_n)
    posteriorArray[i,:] = posterior_n

plot(n,posteriorArray.T)
xlabel('N'), ylabel('Posterior(N|k,f)'), legend(data),  ylim([0, 0.1])


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Last generation
Can you write Python code for it?

lastGen_nCFUs = cfus_over_time[:,-1]

plt.subplot(1,2,1)
plt.hist(lastGen_nCFUs, bins=50)
xlabel('#CFUs')
ylabel('Frequency in last generation')

# CDF
plt.subplot(1,2,2)
x_ECDF_lastGen,y_ECDF_lastGen = ecdf(lastGen_nCFUs)
plt.plot(x_ECDF_lastGen, y_ECDF_lastGen, linewidth=2)
xlabel('#CFUs')
ylabel('CDF')


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
Trying to reproduce Figure 8
Monte Carlo simulation vs $\Gamma$-distribution
Can you write Python code for it?

nGens = 20
p0 = 0.4
nTrees_fig8 = 71
fig8_cfu_20 = np.zeros((nTrees_fig8))
for i in range(nTrees_fig8):
    tree = simulateTree(p0, nGens)
    nCFU = map(lambda x: np.sum(x==0), tree)
    fig8_cfu_20[i] = nCFU[-1]

xTmp, yTmp = ecdf(fig8_cfu_20)
plot(xTmp, yTmp*nTrees_fig8)

#fit by gamma
shape, loc, scale = stats.gamma.fit(fig8_cfu_20)
fittedGamma_synthData = stats.gamma(a=shape, loc=loc, scale=scale)
fittedCDF  =  fittedGamma_synthData.cdf(xTmp[:-1])
plot(xTmp[:-1], fittedCDF*nTrees_fig8)


I am working on the file "Session2/Til_paper.ipynb".
The first task is:
What's the catch here?
Statistical uncertainty
e.g. all the spleens with 0 colonies: hard to say if 0,1 or 6 stem cells were injected and all lost
Binomial sampling

$N$ stem cells injected: unknown
$k$ colonies observed
prob of colony formation: $f=0.17$

What range of $N$ is plausible for given $k$, $f$ ?
Usually
Binomial distribution
- $P(k|N,\theta) = \binom{N}{k} \cdot \theta^k (1-\theta)^{N-k}$
- usually, we estimate the posterior $P(\theta|k,N)$
- HOW?

Bayes' Law

$$P(\theta|k,N)=\frac{L(k,N|\theta) \cdot \pi(\theta)}{\int L(k,N|\theta) \cdot \pi(\theta)\;d\theta}$$
Can you write Python code for it?

#observations
N = 100
k = 10

#inference
tmpP = np.linspace(0,1,100)
like_theta = binom.pmf(k,n=N,p=tmpP)
prior_theta = 1 # simply constant
posterior_theta = like_theta * prior_theta/ np.sum(like_theta * prior_theta)

plot(tmpP,posterior_theta)
xlabel('theta'), ylabel('Posterior(theta|k)'), xlim([0,0.4])


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
.fit(..) to fit estimator to the data
.transform(..) to transform the count matrix to tf-idf
It is possible to merge the fit and transform stages using .fit_transform(..)
Calculate tfidf
Can you write Python code for it?

tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
X_train_tfidf.shape

print("In first document tf-idf for application: {0} subject: {1} to: {2}").format(
    X_train_tfidf[0, 5285], X_train_tfidf[0, 31077], X_train_tfidf[0, 32493])


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Back to '20 newsgroups dataset'
Can you write Python code for it?

print(metrics.classification_report(twenty_test.target, predicted_svm,
    target_names=twenty_test.target_names))


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Evaluation
Can you write Python code for it?

import numpy as np
twenty_test = fetch_20newsgroups(subset='test',
    categories=categories, shuffle=True, random_state=42)
docs_test = twenty_test.data
predicted_bayes = text_clf_bayes_fit.predict(docs_test)
np.mean(predicted_bayes == twenty_test.target)


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Here we can see that the predictions:
- found ant 3 times and should have found it twice hence precision of 0.67.
- never predicted ant when shouldn't have hence recall of 1.
- f1 source is the mean of precision and recall
- support of 2 meaning there were 2 in the true data set.
http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html
Confusion matix
Can you write Python code for it?

metrics.confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"])


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
In the confusion_matrix the labels give the order of the rows.

ant was correctly categorised twice and was never miss categorised
bird was correctly categorised once and was categorised as cat once
cat was correctly categorised twice and was categorised as an ant once
Can you write Python code for it?

metrics.accuracy_score(y_true, y_pred, normalize=True, sample_weight=None)


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Load in the training set of data
Can you write Python code for it?

from sklearn.datasets import fetch_20newsgroups
twenty_train = fetch_20newsgroups(subset='train',categories=categories, shuffle=True, random_state=42)

twenty_train.target_names


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Moving on from that lets see where the improvements where made.
Can you write Python code for it?

print(metrics.classification_report(twenty_test.target, predicted_svm,
    target_names=twenty_test.target_names))

metrics.confusion_matrix(twenty_test.target, predicted_svm)

print(metrics.classification_report(twenty_test.target, predicted_tuned,
    target_names=twenty_test.target_names))

metrics.confusion_matrix(twenty_test.target, predicted_tuned)


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Note target names not in same order as in the categories array
Count of documents
Can you write Python code for it?


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Path to file on your machine
Can you write Python code for it?


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Show the first 8 lines of text from one of the documents formated with line breaks
Can you write Python code for it?


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Show the the targets categories of first 10 documents. As a list and show there names.
Can you write Python code for it?

print(twenty_train.target[:10])
for t in twenty_train.target[:10]:
    print(twenty_train.target_names[t])


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
So we can see the naïve Bayes classifier got a lot more correct in some cases but also included a higher proportion in the last category.
Can you write Python code for it?


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Try a support vector machine instead
Can you write Python code for it?

from sklearn.linear_model import SGDClassifier
text_clf_svm = Pipeline([('vect', CountVectorizer()),
                         ('tfidf', TfidfTransformer()),
                         ('clf', SGDClassifier(loss='hinge', penalty='l2',
                                           alpha=1e-3, n_iter=5, random_state=42)),])
text_clf_svm_fit = text_clf_svm.fit(twenty_train.data, twenty_train.target)
predicted_svm = text_clf_svm_fit.predict(docs_test)
np.mean(predicted_svm == twenty_test.target)


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
Using a CountVectorizer method we can get the integer identifier of a word.
Can you write Python code for it?


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
We can see that atheism is miss categorised as Christian and science and medicine as computer graphics a high proportion of the time using the support vector machine.
Parameter tuning
Transformation and classifiers can have various parameters. Rather than manually tweaking each parameter in the pipeline it is possible to use grid search instead.
Here we try a couple of options for each stage. The more options the longer the grid search will take.
Can you write Python code for it?

from sklearn.grid_search import GridSearchCV
parameters = {'vect__ngram_range': [(1, 1), (1, 2)],
              'tfidf__use_idf': (True, False),
              'clf__alpha': (1e-3, 1e-4),
}

gs_clf = GridSearchCV(text_clf_svm_fit, parameters, n_jobs=-1)


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
We can see the support vector machine got a higher number than naïve Bayes. What does it mean? We move on to metrics.
Using metrics
Classification report & Confusion matix
Here we will use a simple example to show classification reports and confusion matrices.

y_true is the test data
y_pred is the prediction
Can you write Python code for it?

from sklearn import metrics

y_true = ["cat", "ant", "cat", "cat", "ant", "bird", "bird"]
y_pred = ["ant", "ant", "cat", "cat", "ant", "cat", "bird"]
print(metrics.classification_report(y_true, y_pred,
    target_names=["ant", "bird", "cat"]))


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
We can see where the 91% score came from.
Can you write Python code for it?

# We got the evaluation score this way before:
print(np.mean(predicted_svm == twenty_test.target))
# We get the same results using metrics.accuracy_score
print(metrics.accuracy_score(twenty_test.target, predicted_svm, normalize=True, sample_weight=None))


I am working on the file "Session2/code/01 Working with text.ipynb".
The first task is:
What are two problems with using a word count in a document?
From occurrences to frequencies
$\text{Term Frequencies tf} = \text{occurrences of each word} / \text{total number of words}$
tf-idf is "Term Frequencies times Inverse Document Frequency"
Calculating tfidf
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tfidf_2stage = tf_transformer.transform(X_train_counts)
X_train_tfidf_2stage.shape


I am working on the file "Session201811/code/09 Pandas - Part 1.ipynb".
The first task is:
Normalizing data
If we look at two of the features in the data we can see they are of different scales.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
# Histogram
bins=range(0,100,10)

plt.hist(df["Age"].values, bins, alpha=0.5, label='age')
plt.show()

plt.hist(df["BMI"].values, bins, alpha=0.5, label='BMI')
plt.show()

plt.hist(df["Age"].values, bins, alpha=0.5, label='age')
plt.hist(df["BMI"].values, bins, alpha=0.5, label='BMI')
plt.show()


I am working on the file "Session201811/code/09 Pandas - Part 1.ipynb".
The first task is:
We are now going to normalize the data so we give both data items the same weight.

for each column, we compute the mean and remove the standard deviation
Let's say we have points x1, x2,.. xn in column "AGE"
mean = $(1/n) * (x1+x2+...xn)$
std = $\sqrt{(1/n) * ( (x1-mean)^2 + (x2 -mean)^2 + ...)}$
Can you write Python code for it?

# Let's start with an example on the AGE feature
# I create a new array for easier manipulation
arr_age = df["Age"].values
arr_age[:10]


I am working on the file "Session201811/code/09 Pandas - Part 1.ipynb".
The first task is:
We can use standard deviation to normalize data.
Here we generate a random set of data that creates a dataset that follows a Standard deviation from the mean.
https://en.wikipedia.org/wiki/Standard_deviation
Can you write Python code for it?

from numpy.random import normal
gaussian_numbers = normal(size=5000)
plt.hist(gaussian_numbers, bins=np.linspace(-5.0, 5.0, num=20)) # Set bin bounds
plt.show()


I am working on the file "Session201811/code/09 Pandas - Part 2.ipynb".
The first task is:
Interpolation
As you can see from the printed values, there are a few NaN in our array.
It probably means that the data was not recorded at this date.
We have several ways of dealing with that :

We can ignore the relevant data line
Or we can infer the missing data with, say, an average of a few previous points.

We are going to infer the missing data through interpolation.
Say we have 1,NaN,2.
We infer the missing value with $(1+2)/2 = 1.5$
Can you write Python code for it?

# Let's see how to do that with pandas

# Get the data to interpolate
ozone = df_temp["Ozone"].values
solar = df_temp["Solar.R"].values
# Get a series of timestamps
timestamps = pd.to_datetime(df_temp["Date"].values)

# Create a new Series with the timestamp as index
#(or we could have set the index as timestamp in our df_temp dataframe)
s_ozone = pd.Series(ozone, index=timestamps)
s_solar = pd.Series(solar, index=timestamps)

oz_interp = s_ozone.interpolate(method = "time")
sol_interp = s_solar.interpolate(method = "time")

df_temp["Ozone_interp"] = oz_interp.values
df_temp["Solar.R_interp"] = sol_interp.values

df_temp.head(n=10) # use 10 rows to see how interpolation works

#Let's plot the data
import matplotlib.pylab as plt
from matplotlib.gridspec import GridSpec

%matplotlib inline
fig = plt.figure(figsize=(15, 15))
gs = GridSpec(2, 2, bottom=0.18, left=0.18, right=0.88)

axOz = fig.add_subplot(gs[0])
axSol = fig.add_subplot(gs[1])
axWind = fig.add_subplot(gs[2])
axTemp = fig.add_subplot(gs[3])

# Get time axis
x_date = df_temp["Date"].values

# Get the y values
y_oz = df_temp["Ozone_interp"].values
y_sol = df_temp["Solar.R_interp"].values
y_temp = df_temp["Temp"].values
y_wind = df_temp["Wind"].values

# Plot
axOz.plot(x_date, y_oz, label = "Ozone in ppm", linewidth=1.5)
axSol.plot(x_date, y_sol, label = "Solar Radiation", linewidth=1.5)
axWind.plot(x_date, y_temp, label = "Temperature in F", linewidth=1.5)
axTemp.plot(x_date, y_wind, label = "Wind", linewidth=1.5)

#####################
# Figure cosmetics
#####################

# Axis labels, legend and formatting
for ax in [axOz, axSol, axWind, axTemp]:
    ax.set_xlabel("time", fontsize=22)
    ax.legend(loc="best", fontsize=22)

# improve plot layout
gs.tight_layout(fig, h_pad=3)

#plt.savefig("visual.png") #uncomment to save plot
plt.show()


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:

Assessing feature importance with Random Forests
With scikit learn's implementation random Forests we can see feature importance.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

feat_labels = df_wine.columns[1:]

forest = RandomForestClassifier(n_estimators=500,
                                random_state=1)

forest.fit(X_train, y_train)
importances = forest.feature_importances_

indices = np.argsort(importances)[::-1]

for f in range(X_train.shape[1]):
    print("%2d) %-*s %f" % (f + 1, 30,
                            feat_labels[indices[f]],
                            importances[indices[f]]))

plt.title('Feature Importance')
plt.bar(range(X_train.shape[1]),
        importances[indices],
        align='center')

plt.xticks(range(X_train.shape[1]),
           feat_labels[indices], rotation=90)
plt.xlim([-1, X_train.shape[1]])
plt.tight_layout()
#plt.savefig('images/04_09.png', dpi=300)
plt.show()


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:

Bringing features onto the same scale
Most machine learning algorithms behave much better if features are on the same scale.
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

mms = MinMaxScaler()
X_train_norm = mms.fit_transform(X_train)
X_test_norm = mms.transform(X_test)

X[0,:]

X_train[0,:]

X_train_norm[0,:]

from sklearn.preprocessing import StandardScaler

stdsc = StandardScaler()
X_train_std = stdsc.fit_transform(X_train)
X_test_std = stdsc.transform(X_test)

X_train_std[0,:]


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:

Eliminating samples or features with missing values
Can you write Python code for it?

# remove rows that contain missing values

df.dropna(axis=0)

# remove columns that contain missing values

df.dropna(axis=1)

# remove columns that contain missing values

df.dropna(axis=1)

csv_data = \
'''A,B,C,D
1.0,2.0,3.0,4.0
,,,
4.0,,,
4.0,6.0,,
5.0,6.0,,8.0
10.0,11.0,12.0,'''

# If you are using Python 2.7, you need
# to convert the string to unicode:

if (sys.version_info < (3, 0)):
    csv_data = unicode(csv_data)

df2 = pd.read_csv(StringIO(csv_data))
df2

# only drop rows where all columns are NaN

df2.dropna(how='all')

# drop rows that have less than 3 real values

df2.dropna(thresh=3)

# only drop rows where NaN appear in specific columns (here: 'C')

df2.dropna(subset=['C'])


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:

Imputing missing values
Removing null data may be fine but it may reduce the volume of data too much.
Can you write Python code for it?

# again: our original array
df.values

# impute missing values via the column mean

from sklearn.preprocessing import Imputer

imr = Imputer(missing_values='NaN', strategy='mean', axis=0)
imr = imr.fit(df.values)
imputed_data = imr.transform(df.values)
imputed_data

# impute missing values via the row mean

imr = Imputer(missing_values='NaN', strategy='mean', axis=1)
imr = imr.fit(df.values)
imputed_data = imr.transform(df.values)
imputed_data


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:

Mapping ordinal features
here we can change value in place, within the data frame
Can you write Python code for it?

size_mapping = {'XL': 3,
                'L': 2,
                'M': 1}

df['size'] = df['size'].map(size_mapping)
df


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:

Partitioning a dataset into a seperate training and test set
Can you write Python code for it?

df_wine = pd.read_csv('https://archive.ics.uci.edu/'
                      'ml/machine-learning-databases/wine/wine.data',
                      header=None)

# if the Wine dataset is temporarily unavailable from the
# UCI machine learning repository, un-comment the following line
# of code to load the dataset from a local path:

# df_wine = pd.read_csv('wine.data', header=None)

df_wine.columns = ['Class label', 'Alcohol', 'Malic acid', 'Ash',
                   'Alcalinity of ash', 'Magnesium', 'Total phenols',
                   'Flavanoids', 'Nonflavanoid phenols', 'Proanthocyanins',
                   'Color intensity', 'Hue', 'OD280/OD315 of diluted wines',
                   'Proline']

print('Class labels', np.unique(df_wine['Class label']))
df_wine.head()

from sklearn.model_selection import train_test_split

X, y = df_wine.iloc[:, 1:].values, df_wine.iloc[:, 0].values

X_train, X_test, y_train, y_test =\
    train_test_split(X, y,
                     test_size=0.3,
                     random_state=0,
                     stratify=y)
# X data
# y class label that will be used to train
# Test size 0.3 = 30% test data, the rest training data


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:

Sequential feature selection algorithms
Can you write Python code for it?

from sklearn.base import clone
from itertools import combinations
import numpy as np
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

class SBS():
    def __init__(self, estimator, k_features, scoring=accuracy_score,
                 test_size=0.25, random_state=1):
        self.scoring = scoring
        self.estimator = clone(estimator)
        self.k_features = k_features
        self.test_size = test_size
        self.random_state = random_state

    def fit(self, X, y):

        X_train, X_test, y_train, y_test = \
            train_test_split(X, y, test_size=self.test_size,
                             random_state=self.random_state)

        dim = X_train.shape[1]
        self.indices_ = tuple(range(dim))
        self.subsets_ = [self.indices_]
        score = self._calc_score(X_train, y_train,
                                 X_test, y_test, self.indices_)
        self.scores_ = [score]

        while dim > self.k_features:
            scores = []
            subsets = []

            for p in combinations(self.indices_, r=dim - 1):
                score = self._calc_score(X_train, y_train,
                                         X_test, y_test, p)
                scores.append(score)
                subsets.append(p)

            best = np.argmax(scores)
            self.indices_ = subsets[best]
            self.subsets_.append(self.indices_)
            dim -= 1

            self.scores_.append(scores[best])
        self.k_score_ = self.scores_[-1]

        return self

    def transform(self, X):
        return X[:, self.indices_]

    def _calc_score(self, X_train, y_train, X_test, y_test, indices):
        self.estimator.fit(X_train[:, indices], y_train)
        y_pred = self.estimator.predict(X_test[:, indices])
        score = self.scoring(y_test, y_pred)
        return score

import matplotlib.pyplot as plt
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5)

# selecting features
sbs = SBS(knn,...


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
<br>
Can you write Python code for it?

for f in range(X_selected.shape[1]):
    print("%2d) %-*s %f" % (f + 1, 30,
                            feat_labels[indices[f]],
                            importances[indices[f]]))


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:

Documentation for sklearn.preprocessing.Imputer

Understanding the scikit-learn estimator API
Can you write Python code for it?


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
A visual example:
Can you write Python code for it?

ex = np.array([0, 1, 2, 3, 4, 5])

print('standardized:', (ex - ex.mean()) / ex.std())

# Please note that pandas uses ddof=1 (sample standard deviation)
# by default, whereas NumPy's std method and the StandardScaler
# uses ddof=0 (population standard deviation)

# normalize
print('normalized:', (ex - ex.min()) / (ex.max() - ex.min()))


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
Applied to the standardized Wine data ...
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

lr = LogisticRegression(penalty='l1', C=1.0)
lr.fit(X_train_std, y_train)
print('Training accuracy:', lr.score(X_train_std, y_train))
print('Test accuracy    :', lr.score(X_test_std, y_test))

lr.intercept_


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
Below we can see the effect on the training and test accuracy.
Can you write Python code for it?

lr = LogisticRegression(penalty='l1', C=0.01)
lr.fit(X_train_std, y_train)
print('Training accuracy:', lr.score(X_train_std, y_train))
print('Test accuracy    :', lr.score(X_test_std, y_test))

lr = LogisticRegression(penalty='l1', C=0.1)
lr.fit(X_train_std, y_train)
print('Training accuracy:', lr.score(X_train_std, y_train))
print('Test accuracy    :', lr.score(X_test_std, y_test))

lr = LogisticRegression(penalty='l1', C=1)
lr.fit(X_train_std, y_train)
print('Training accuracy:', lr.score(X_train_std, y_train))
print('Test accuracy    :', lr.score(X_test_std, y_test))

lr = LogisticRegression(penalty='l1', C=10)
lr.fit(X_train_std, y_train)
print('Training accuracy:', lr.score(X_train_std, y_train))
print('Test accuracy    :', lr.score(X_test_std, y_test))


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
Dealing with missing data
Identifying missing values in tabular data
Can you write Python code for it?

# Sample csv

import pandas as pd
from io import StringIO
import sys

csv_data = \
'''A,B,C,D
1.0,2.0,3.0,4.0
5.0,6.0,,8.0
10.0,11.0,12.0,'''

# If you are using Python 2.7, you need
# to convert the string to unicode:

if (sys.version_info < (3, 0)):
    csv_data = unicode(csv_data)

df = pd.read_csv(StringIO(csv_data))
df

# Give a count of null values for each column
df.isnull().sum()

# access the underlying NumPy array
# via the `values` attribute
df.values


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
For regularized models in scikit-learn that support L1 regularization, we can simply set the penalty parameter to 'l1' to obtain a sparse solution:
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
LogisticRegression(penalty='l1')


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
Here we are using fit and transform methods as a preprocessor, for example from the Min Max Scaler, to map data from it's original form to one better suited for Machine learning.
Later the model created is used for transforming initial test data and then target data when it comes along.
As we have seen it is possible to either remove null data or impute values.
One thing to bear in mind is the shape of the data you supply to the fit and transform methods must always be the same shape.
Can you write Python code for it?


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
Here we can see the all the weights for the three classes and the 13 dimensions in the wine dataset.
Can you write Python code for it?


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
Sparse solutions with L1-regularization
Can you write Python code for it?


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
The use of watermark is optional. You can install this IPython extension via "pip install watermark". For more information, please see: https://github.com/rasbt/watermark.

Overview

Dealing with missing data
Identifying missing values in tabular data
Eliminating samples or features with missing values
Imputing missing values
Understanding the scikit-learn estimator API
Handling categorical data
Nominal and ordinal features
Mapping ordinal features
Encoding class labels
Performing one-hot encoding on nominal features
Partitioning a dataset into a separate training and test set
Bringing features onto the same scale
Selecting meaningful features
L1 and L2 regularization as penalties against model complexity
A geometric interpretation of L2 regularization
Sparse solutions with L1 regularization
Sequential feature selection algorithms
Assessing feature importance with Random Forests
Summary

Can you write Python code for it?

# Use the IPython/jupyter feature to show images inline with the notebook
# output rather than have images popup.
from IPython.display import Image
%matplotlib inline


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
This shows the intercept of each of the three models being used.
Can you write Python code for it?

# A numpy function to set precision
np.set_printoptions(8)


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
This time the fit method from, for example from Logistic Regression, is used with training data and training labels to generate a model.
This model is then used to predict outcomes of the test data. The output is labels.

Handling categorical data
Nominal and ordinal features
Can you write Python code for it?

import pandas as pd

df = pd.DataFrame([['green', 'M', 10.1, 'class1'],
                   ['red', 'L', 13.5, 'class2'],
                   ['blue', 'XL', 15.3, 'class1']])

df.columns = ['color', 'size', 'price', 'classlabel']
df


I am working on the file "Session201811/code/10 ch04.ipynb".
The first task is:
What is the problem with this approach?
Can you write Python code for it?

from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(categorical_features=[0])
ohe.fit_transform(X).toarray()

# return dense array so that we can skip
# the toarray step

ohe = OneHotEncoder(categorical_features=[0], sparse=False)
ohe.fit_transform(X)

df

# one-hot encoding via pandas - just color as a nominal value

pd.get_dummies(df[['price', 'color', 'size']])

# one-hot encoding via pandas - both color and class label as nominal values

pd.get_dummies(df[['price', 'color', 'size','classlabel']])

# multicollinearity guard in get_dummies

pd.get_dummies(df[['price', 'color', 'size']], drop_first=True)

# multicollinearity guard in get_dummies
# - both color and class label as nominal values

pd.get_dummies(df[['price', 'color', 'size','classlabel']], drop_first=True)

X

# multicollinearity guard for the OneHotEncoder

ohe = OneHotEncoder(categorical_features=[0])
ohe.fit_transform(X).toarray()

ohe.fit_transform(X).toarray()[:, 1:]


I am working on the file "Session201811/code/11 Supervised Machine Learning - scikit learn.ipynb".
The first task is:
Here we can see that the predictions:
- precision = $2/3 = 0.67$ (2 ants in test data and matched but found an extra 1 in prediction).
- recall = $2/2 = 1$ (2 ants in test data and these matched in prediction).
- f1-score = $(0.67 + 1) / 2 = 0.8$ mean of precision and recall.
- support shows that there are 2 ants, 2 birds and 3 cats in the test data.
http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html
Can you write Python code for it?


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Calculate the freq distribution for this text collection of documents. Then let's see the most common words.
Can you write Python code for it?

# Write code to compute frequency distribution of text_collection
freq_dist = ...

# Let's see most common 10 words.
freq_dist.most_common(10)


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Finally, rerun with the bigger document set and replot the cumulative word frequencies.
Recall that we already have the documents loaded in the variable raw_docs
Can you write Python code for it?

# Write code to create a NLTK TextCollection with raw_to_text_2
text_collection = ...

# Write code to compute frequency distribution of text_collection
freq_dist = ...

# Write code to plot the frequencies again


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Let's automate: write a function
Using above steps, we will now write a function. We will call this function raw_to_text. This function will take a list of raw texts and will return a NLTK TextCollection objects, representing the list of input text.
Can you write Python code for it?

"""
Converts a list of raw text to a NLTK TextCollection object.
Applies lower-casing and punctuation removal.
Returns:
text_collection - a NLTK TextCollection object
"""
def raw_to_text(raw_list):
    text_list = []
    for raw in raw_list:
        # Write code for lower-casing and punctuation removal

        # Write code to tokenize and create NLTK Text object
        # Name the variable 'text' to store the Text object

        # storing the text in the list
        text_list.append(text)

    # Write code to create TextCollection from the list text_list
    text_collection = nltk.text.TextCollection(text_list) # TO DELETE

    # return text collection
    return text_collection


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Let's test the function with some sample data
Can you write Python code for it?

raw_list_sample = ["The dog sat on the mat.",
                   "The cat sat on the mat!",
                   "We have a mat in our house."]

# Call the above raw_to_text function for the sample text
text_collection_sample = ...


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Like before we can compute the frequency distribution for this collection.
Can you write Python code for it?

# Write code to compute the frequency 'mat' in the collection.
freq = ...
print("Frequency of word \'mat\' = ", freq)

# Write code to compute and display the frequency distribution of text_collection_sample


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
NLTK TextCollection object can be used to calculate basic statistics.
  1. count the number of occurances (or term frequency) of a word
  2. obtain a frequency distribution of all the words in the text
Note: The NLTK Text objects created in the intermediate steps can also be used to calculate similar statistics at document level.
Can you write Python code for it?

# Frequency of a word
freq = text_collection.count("grenoble")
print("Frequency of word \'grenoble\' = ", freq)

# Frequency distribution
freq_dist = nltk.FreqDist(text_collection)
freq_dist


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
NLTK already provides us with modules to easily tokenize the text. We will tokenize pieces of raw texts using word_tokenize function of NLTK package.
Can you write Python code for it?

import nltk

# Tokenization using NLTK
tokens1 = nltk.word_tokenize(raw1)
tokens2 = nltk.word_tokenize(raw2)

# print the tokens
print(tokens1)
print(tokens2)


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Observations:
   1. Just 30 most frequent tokens make up around 260,000 out of 709,460 ($\approx 36.5\%$)
   2. Most of these are very common words such as articles, pronouns etc.
Stop word filtering
Stop words are words which are filtered out before or after processing of natural language data (text). There is no universal stop-word list. Often, stop word lists include short function words, such as "the", "is", "at", "which", and "on". Removing  stop-words has been shown to increase the performance of different tasks like search.
A file of stop_words.txt is included. We will now:
   1. Load the contents of the file 'stop_words.txt' where each line is a stop word, and create a stop-word list.
   2. Modify the function raw_to_text to perform (a) stop-word removal (b) numeric words removal
Note: Each line of the file stop_words.txt is a stop word.
Can you write Python code for it?

# Write code to load stop-word list from file 'stop_words.txt'
# Hint: use the same strategy you used to load documents
stopwords = set(...)

"""
VERSION 2
Converts a list of raw text to a NLTK TextCollection object.
Applies lower-casing, punctuation removal and stop-word removal.
Returns:
text_collection: a NLTK TextCollection object
"""
# Write function "raw_to_text_2".
"""
Hint 1: consult the above function "raw_to_text",
Hint 2: add a new block in the function for removing stop words
Hint 3: to remove stop words from a of tokens -
   - create an ampty list to store clean tokens
   - for each token in the token list:
         if the token is not in stop word list
            store it in the clean token list
"""


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Preprocessing
Upper case, Punctuations
A computer does not require upper case letters and punctuations.
Note: Python already provides a list of punctuations. We simply need to import it.
Can you write Python code for it?

from string import punctuation

s = "Hello, World!!"

# Write code to lower case the string
s = ...

# Write code to remove punctuations
# HINT: for loop and for each punctuation use string replace() method
for ...
    s = ...

print(s)


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Something bigger
We will use DBPedia Ontology Classification Dataset. It includes first paragraphs of Wikipedia articles. Each paragraph is assigned one of 14 categories. Here is an example of an abstract under Written Work catgory:

The Regime: Evil Advances/Before They Were Left Behind is the second prequel novel in the Left Behind series written by Tim LaHaye and Jerry B. Jenkins. It was released on Tuesday November 15 2005. This book covers more events leading up to the first novel Left Behind. It takes place from 9 years to 14 months before the Rapture.

In this hands-on we will use 15,000 documents belonging to three categories, namely Album, Film and Written Work.
The file corpus.txt supplied here, contains 15,000 documents. Each line of the file is a document.
Now we will:
   1. Load the documents as a list
   2. Create a NLTK TextCollection
   3. Analyze different counts
Note: Each line of the file corpus.txt is a document
Can you write Python code for it?

# Write code to load documents as a list
"""
Hint 1: open the file using open()
Hint 2: use read() to load the content
Hint 3: use splitlines() to get separate documents
"""
raw_docs = ...

print("Loaded " + str(len(raw_docs)) + " documents.")

# Write code to create a NLTK TextCollection
# Hint: use raw_to_text function
text_collection = ...

# Print total number of words in these documents
print("Total number of words = ", len(text_collection))
print("Total number of unique words = ", len(set(text_collection)))


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Something does not seem right!! Can you point out what?
Let's try by visualizing it.
Can you write Python code for it?

# importing Python package for plotting
import matplotlib.pyplot as plt

# To plot
plt.subplots(figsize=(12,10))
freq_dist.plot(30, cumulative=True)


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
This is a hands on session, so this is time you write some of code. Let's try that.
Can you write Python code for it?

# Write code to print any string...

# Then run the code.


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Tokenization : NLTK
Natural Language Toolkit (NLTK) is a platform to work with human or natural language data using Python.
As usual, we will first convert everything to lowercase and remove punctuations.
Can you write Python code for it?

raw1 = "Grenoble is a city in southeastern France, at the foot of the French Alps, on the banks of Isère."
raw2 = "Grenoble is the capital of the department of Isère and is an important scientific centre in France."

# Write code here to convert everything in lower case and to remove punctuation.

print(raw1)
print(raw2)
# Again, SHIFT+ENTER to run the code.


I am working on the file "Session3/1.TextProcessing.ipynb".
The first task is:
Zipf law
Verify whether the dataset follows the Zipf law, by plotting the data on a log-log graph, with the axes being log (rank order) and log (frequency). You expect to obtain an alomost straight line.
Can you write Python code for it?

import numpy as np
import math

counts = np.array(list(freq_dist.values()))
tokens = np.array(list(freq_dist.keys()))
ranks = np.arange(1, len(freq_dist)+1)

# Obtaining indices that would sort the array in descending order
indices = np.argsort(-counts)
frequencies = counts[indices]

# Plotting the ranks vs frequencies
plt.subplots(figsize=(12,10))
plt.yscale('log')
plt.xscale('log')
plt.title("Zipf plot for our data")
plt.xlabel("Frequency rank of token")
plt.ylabel("Absolute frequency of token")
plt.grid()
plt.plot(ranks, frequencies, 'o', markersize=0.9)
for n in list(np.logspace(-0.5, math.log10(len(counts)-1), 17).astype(int)):
    dummy = plt.text(ranks[n], frequencies[n], " " + tokens[indices[n]],
                     verticalalignment="bottom", horizontalalignment="left")
plt.show()


I am working on the file "Session3/2.TextClassification.ipynb".
The first task is:
A Bigger Collection
We will use DBPedia Ontology Classification Dataset. It includes first paragraphs of Wikipedia articles. Each paragraph is assigned one of 14 categories. Here is an example of an abstract under Written Work catgory:

The Regime: Evil Advances/Before They Were Left Behind is the second prequel novel in the Left Behind series written by Tim LaHaye and Jerry B. Jenkins. It was released on Tuesday November 15 2005. This book covers more events leading up to the first novel Left Behind. It takes place from 9 years to 14 months before the Rapture.

In this hands-on we will use 15,000 documents belonging to three categories, namely Album, Film and Written Work.
The file corpus.txt supplied here, contains 15,000 documents. Each line of the file is a document.
Now we will:
   1. Load the documents as a list
   2. Create TF-IDF vectors
Note: Each line of the file corpus.txt is a document.
Can you write Python code for it?

# Write code to load documents as a list
# Hint: recall the strategy you used in the previous notebook
raw_docs = ...

print("Loaded " + str(len(raw_docs)) + " documents.")

# Write code to convert raw documents into TF-IDF matrix.
"""
Hint: - create a TfidfVectorizer, and do not forget to remove stopwords
      - use fit_transform to vectorize raw_docs
"""
tfidf = ...
X_tfidf = ...


I am working on the file "Session3/2.TextClassification.ipynb".
The first task is:
Other Classifiers
You have virtually an endless option to choose your classifier. Let's try some more.
Method is simple:
1. import relevant packages
2. create an instance of the classifier
3. fit with the training data and training labels
4. predict with the test data
5. evaluate by comparing  predicted labels and test labels
Perceptron
Can you write Python code for it?

# 1. Write code to import Perceptron
from ...

# 2. Write code to create Perceptron classifier
classifierPer = ...

# 3. Write code to "fit" the classifier with training data and labels
...

# 4. Write code to "predict" labels for the test set
predictionsPer = ...

# 5. Write code to calculate accuracy
accuracyPer = ...

print("Test accuracy: " + str(accuracyPer * 100) + "%")


I am working on the file "Session3/2.TextClassification.ipynb".
The first task is:
Sparse Vs Dense Matrices
Dense matrices store every entry in the matrix. Sparse matrices only store the nonzero entries. Sparse matrices don't have a lot of extra features, and some algorithms may not work for them. You use them when you need to work with matrices that would be too big for the computer to handle them, but they are mostly zero, so they compress easily. Be aware of issues that may arise at:
- dot product
- slicing (row, column)
In python these are taken care almost automatically, by using sparse dot product and implementations of csr and csc matrices (scipy.sparse.csr_matrix, scipy.sparse.csc_matrix, etc..).
Can you write Python code for it?

print("Count Matrix:")
print(X_sample.todense())
print("\nWords in vocabulary:")
print(vectorizer.get_feature_names())


I am working on the file "Session3/2.TextClassification.ipynb".
The first task is:
Text Classifier
Machine learning algorithms need a training set. In our text classification scenario, we need category or class labels for all 15000 documents in the collection.
In our collection we have documents from three categories: "Album" (category 12), "Film" (category 13) and "Written Work" (category 14). For each document we know the labels. The labels are stored in labels.txt file. Each line of corpus.txt file corresponds to the label in the same line of file labels.txt.
Lets load the labels.
Note: Each line of the file labels.txt is a document.
Verify that the number of loaded labels and number of loaded documents are same.
Can you write Python code for it?

# Write code to load labels list from file 'labels.txt'
# Hint: use the same strategy you used to load documents
labels = ...

print("Loaded " + str(len(labels)) + " labels.")


I am working on the file "Session3/2.TextClassification.ipynb".
The first task is:
Training the Classifier
First we will use Multinomial Naive Bayes classifier. Scikit-learn provides the module MultinomialNB
Can you write Python code for it?

# Write code to import MultinomialNB
from ...

# Write code to create a MultinomialNB classifier
classifierNB = ...

# Write code to train the classifier using "fit" function.
# Hint: you need to provide training data and labels for training


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
And again, map the function onto the filenames, and load the results into the table.
Can you write Python code for it?

# and then lets map the function onto all the data
result = lview.map(calculate_features, filenames)

# and get the result
all_feats = result.get()

# and now load it all into our table
for feats in all_feats:
    cur.execute("""insert into lcFeats(objId,
                                       gStd, rStd, iStd, zStd,
                                       gAmp,  rAmp,  iAmp,  zAmp,
                                       gMAD, rMAD, iMAD, zMAD,
                                       gBeyond, rBeyond, iBeyond, zBeyond,
                                       gSkew, rSkew, iSkew, zSkew,
                                       gMinusR, rMinusI, iMinusZ) values {}""".format(feats))


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
And now that you have results, load them in the database as before.
Can you write Python code for it?

for data in all_data:
    cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
As we have many light curve files (in principle as many as 37 billion...), we will define a light curve class to ease our handling of the data.
 Problem 1b
Fix the lc class definition below.
Hint - the only purpose of this problem is to make sure you actually read each line of code below, it is not intended to be difficult.
Can you write Python code for it?

class ANTARESlc():
    '''Light curve object for NOAO formatted data'''

    def __init__(self, filename):
        '''Read in light curve data'''
        DFlc = pd.read_csv(filename, delim_whitespace=True, comment = '#')
        self.DFlc = DFlc
        self.filename = filename

    def plot_multicolor_lc(self):
        '''Plot the 4 band light curve'''
        fig, ax = plt.subplots()
        g = ax.errorbar(self.DFlc['t'].ix[self.DFlc['pb'] == 'g'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'g'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'g'],
                     fmt = 'o', color = '#78A5A3', label = r"$g'$")
        r = ax.errorbar(self.DFlc['t'].ix[self.DFlc['pb'] == 'r'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'r'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'r'],
                     fmt = 'o', color = '#CE5A57', label = r"$r'$")
        i = ax.errorbar(self.DFlc['t'].ix[self.DFlc['pb'] == 'i'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'i'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'i'],
                     fmt = 'o', color = '#E1B16A', label = r"$i'$")
        z = ax.errorbar(self.DFlc['t'].ix[self.DFlc['pb'] == 'z'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'z'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'z'],
                     fmt = 'o', color = '#444C5C', label = r"$z'$")
        ax.legend(fancybox = True)
        ax.set_xlabel(r"$\mathrm{MJD}$")
        ax.set_ylabel(r"$\mathrm{flux}$")


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Following the sqlite3 import, we must first connect to the database. If we attempt a connection to a database that does not exist, then a new database is created. Here we will create a new database file, called miniANTARES.db.
Can you write Python code for it?


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
For now - don't worry about what the labels mean (though if you inspect the light curves you may be able to figure this out...)
Problem 4a
Query the database to select features and labels for the light curves in your training set. Store the results of these queries in numpy arrays, X and y, respectively, which are suitable for the various scikit-learn machine learning algorithms.
Hint - recall that databases do not store ordered results.
Hint 2 - recall that scikit-learn expects y to be a 1d array. You will likely need to convert a 2d array to 1d.
Can you write Python code for it?

cur.execute("""select label
               from lcLabels
               order by objId asc""")
y = np.array(cur.fetchall()).ravel()

cur.execute("""select gStd, rStd, iStd, zStd,
                      gAmp,  rAmp,  iAmp,  zAmp,
                      gMAD, rMAD, iMAD, zMAD,
                      gBeyond, rBeyond, iBeyond, zBeyond,
                      gSkew, rSkew, iSkew, zSkew,
                      gMinusR, rMinusI, iMinusZ
               from lcFeats
               order by objId asc""")
X = np.array(cur.fetchall())


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Let's unpack everything that happened in these two commands. First - if the table rawPhot already exists, we drop it to start over from scratch. (this is useful here, but should not be adopted as general practice)
Second - we create the new table rawPhot, which has 6 columns: id - a running index for every row in the table, objId - an ID to identify which source the row belongs to, t - the time of observation in MJD, pb - the passband of the observation, flux the observation flux, and dflux the uncertainty on the flux measurement. In addition to naming the columns, we also must declare their type. We have declared id as the primary key, which means this value will automatically be assigned and incremented for all data inserted into the database. We have also declared pb as a variable character of length 1, which is more useful and restrictive than simply declaring pb as text, which allows any freeform string.
Now we need to insert the raw flux measurements into the database. To do so, we will use the ANTARESlc class that we created earlier. As an initial example, we will insert the first 3 observations from the source FAKE010.
Can you write Python code for it?

filename = "training_set_for_LSST_DSFP/FAKE001.dat"
lc = ANTARESlc(filename)

objId = int(filename.split('FAKE')[1].split(".dat")[0])

cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[0])))
cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[1])))
cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[2])))


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Load all of the raw photometric observations into the rawPhot table in the database.
Normally, you'd load these files one by one in serial, looping over each file. This is what happens in miniAntares_serial.ipynb. Instead, we're going to use our cluster to parallelize this operation.
Problem 3b
First, build a list of filenames:
Hint - you can use glob to select all of the files being loaded.
Hint 2 - you have already loaded the data from FAKE001 into the table.
Can you write Python code for it?

# build your list of filenames here
import glob
filenames = glob.glob("training_set_for_LSST_DSFP/FAKE*.dat")

# execute this
@require(ANTARESlc, 'pandas as pd')
def load_data(filename):
    lc = ANTARESlc(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])
    data = [(objId,) + tuple(x) for x in lc.DFlc.values] # array of tuples
    return data


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Now that we have a cursor object, we can populate the database. As an example we will start by creating a table to hold all the raw photometry (though ultimately we will not use this table for analysis).
Note - there are many cursor methods capable of interacting with the database. The most common, execute, takes a single SQL command as its argument and executes that command. Other useful methods include executemany, which is useful for inserting data into the database, and executescript, which take an SQL script as its argument and executes the script.
In many cases, as below, it will be useful to use triple quotes in order to improve the legibility of your code.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if is already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Now that we have loaded the raw observations, we need to create a new table to store summary statistics for each object. This table will include everything we've added to the ANTARESlc class.
Can you write Python code for it?

cur.execute("""drop table if exists lcFeats""") # drop the table if it already exists
cur.execute("""create table lcFeats(
                id integer primary key,
                objId int,
                gStd float,
                rStd float,
                iStd float,
                zStd float,
                gAmp float,
                rAmp float,
                iAmp float,
                zAmp float,
                gMAD float,
                rMAD float,
                iMAD float,
                zMAD float,
                gBeyond float,
                rBeyond float,
                iBeyond float,
                zBeyond float,
                gSkew float,
                rSkew float,
                iSkew float,
                zSkew float,
                gMinusR float,
                rMinusI float,
                iMinusZ float,
                FOREIGN KEY(objId) REFERENCES rawPhot(objId)
                )
""")


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Now, figure out how to use the DirectView access to the cluster to map this function on to all the filenames, and get the results. Examine a few of them.
Can you write Python code for it?

result = lview.map(load_data, filenames[1:])
all_data = result.get()


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
OH YAY! That's much better a more robust to noisy ratty data, the likes of which you will get all the darn time in astronomy!
Except, is this feasible to run?? Can we practically do it on 10^4--5 alerts every 37 seconds for ANTARES??
Well you have the tools to examine this! Let's profile the Gaussian Processes vs all the features
Can you write Python code for it?

import cProfile
def dummy_func():

    all_fits = []
    all_feats = []
    for filename in ftest:
        result  = calculate_gpfit(filename)
        result2 = calculate_features(filename)
        all_feats.append(result2)
        all_fits.append(result)

! rm -f gpstats.cprof
cProfile.run('dummy_func()','gpstats.cprof')


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Problem 1c
Confirm the corrections made in 1b by plotting the multiband light curve for the source FAKE010.
Can you write Python code for it?

lc = ANTARESlc('training_set_for_LSST_DSFP/FAKE010.dat')

lc.plot_multicolor_lc()


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Problem 3e
Confirm that the data loaded correctly by counting the number of sources with gAmp > 2.
How many sources have gMinusR = -999?
Hint - you should find 9 and 2, respectively.
Can you write Python code for it?

cur.execute("""select count(*) from lcFeats where gAmp > 2""")

nAmp2 = cur.fetchone()[0]

cur.execute("""select count(*) from lcFeats where gMinusR = -999""")
nNoColor = cur.fetchone()[0]

print("There are {:d} sources with gAmp > 2".format(nAmp2))
print("There are {:d} sources with no measured i' - z' color".format(nNoColor))


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Problem 4e
Which model are you going to use in your miniANTARES? Justify your answer.
Write solution to 4e here
In this case we are going to adopt the SVM model as it is a factor of 20 times faster than RF, while providing nearly identical performance from an accuracy stand point.
Problem 5) Class Predictions for New Sources
Now that we have developed a basic infrastructure for dealing with streaming data, we may reap the rewards of our efforts. We will use our ANTARES-like software to classify newly observed sources.
Problem 5a
Load the light curves for the new observations (found in full_testset_for_LSST_DSP) into the a table in the database.
Hint - ultimately it doesn't matter much one way or another, but you may choose to keep new observations in a table separate from the training data. I'm putting it into a new testPhot database. Up to you.
Can you write Python code for it?

cur.execute("""drop table if exists testPhot""") # drop the table if is already exists
cur.execute("""create table testPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")
cur.execute("""drop table if exists testFeats""") # drop the table if it already exists
cur.execute("""create table testFeats(
                id integer primary key,
                objId int,
                gStd float,
                rStd float,
                iStd float,
                zStd float,
                gAmp float,
                rAmp float,
                iAmp float,
                zAmp float,
                gMAD float,
                rMAD float,
                iMAD float,
                zMAD float,
                gBeyond float,
                rBeyond float,
                iBeyond float,
                zBeyond float,
                gSkew float,
                rSkew float,
                iSkew float,
                zSkew float,
                gMinusR float,
                rMinusI float,
                iMinusZ float,
                FOREIGN KEY(objId) REFERENCES testPhot(objId)
                )
""")

new_obs_filenames = glob.glob("test_set_for_LSST_DSFP/FAKE*.dat")

result = lview.map(load_data, new_obs_filenames)
all_data = result.get()

for data in all_data:
    cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Problem 5b
Calculate features for the new observations and insert those features into a database table.
Hint - again, you may want to create a new table for this, up to you. I'm using the testFeats table.
Can you write Python code for it?

result = lview.map(calculate_features, new_obs_filenames)
all_feats = result.get()

for feats in all_feats:
    cur.execute("""insert into testFeats(objId,
                                       gStd, rStd, iStd, zStd,
                                       gAmp,  rAmp,  iAmp,  zAmp,
                                       gMAD, rMAD, iMAD, zMAD,
                                       gBeyond, rBeyond, iBeyond, zBeyond,
                                       gSkew, rSkew, iSkew, zSkew,
                                       gMinusR, rMinusI, iMinusZ) values {}""".format(feats))


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Problem 5c
Train the model that you adopted in 4e on the training set, and produce predictions for the newly observed sources.
What is the class distribution for the newly detected sources?
Hint - the training set was constructed to have a nearly uniform class distribution, that may not be the case for the actual observed distribution of sources.
Can you write Python code for it?

svm_clf = SVC(C=1.0, gamma = 0.1, kernel = 'rbf').fit(X, y)

cur.execute("""select gStd, rStd, iStd, zStd,
                      gAmp,  rAmp,  iAmp,  zAmp,
                      gMAD, rMAD, iMAD, zMAD,
                      gBeyond, rBeyond, iBeyond, zBeyond,
                      gSkew, rSkew, iSkew, zSkew,
                      gMinusR, rMinusI, iMinusZ
               from testFeats
               order by objId asc""")
X_new = np.array(cur.fetchall())

y_preds = svm_clf.predict(X_new)

print("""There are {:d}, {:d}, and {:d} sources
         in classes 1, 2, 3, respectively""".format(*list(np.bincount(y_preds)))) # be careful using bincount


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Problem 5d
This classification performance isn't great... Lets take a look at the actual class labels, since we sneakily left them in the file headers
Can you write Python code for it?

! grep label test_set_for_LSST_DSFP/*dat | sed -e 's/label:/ /' | awk '{print $2+1}' | sort | uniq -c


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
Putting what you've learnt together:
Building an mini-ANTARES (ish) Pipeline for Data Management and Discovery
========
Version 0.1

By AA Miller 2017 Apr 10
Edited G. Narayan 2017 Apr 26
As we just saw LSST will produce an unprecedented volume of time-domain information for the astronomical sky. $>37$ trillion individual photometric measurements will be recorded. While the vast, vast majority of these measurements will simply confirm the status quo, some will represent rarities that have never been seen before (e.g., LSST may be the first telescope to discover the electromagnetic counterpart to a LIGO graviational wave event), which the community will need to know about in ~real time.
Storing, filtering, and serving this data is going to be a huge <del>nightmare</del> challenge. ANTARES is one proposed solution to this challenge. In this exercise you will build a miniature version of ANTARES, which will require the application of several of the lessons from earlier this week.
Many of the difficult, and essential, steps necessary for ANTARES will be skipped here as they are too time consuming or beyond the scope of what we have previously covered. We will point out these challenges are we come across them.

0) There is a README.MD file along with this notebook.
Do it!

If everything worked OK up to this point, congratulations! You now have access to a (mini) parallel computing environment of your own.
Can you write Python code for it?

# first we need to construct a client that will interface with our cluster
from ipyparallel import Client, require
worker = Client()

# once we create a client, we can decide how to allocate tasks across the cluster
# we've got however many 'engines' you started in the cluster
# lets just use all of them

lview = worker[:]

# now if you want to import packages, you can import them across all the 'engines'
with lview.sync_imports():
    import numpy as np
    import scipy.stats as spstat
    import pandas as pd
    import os

# there's not much point in creating plots on the engines - we want to actually see them presumably
import matplotlib.pyplot as plt
%matplotlib notebook

# If everything has worked so far, you should see a list of worker IDs, and a corresponding list of process IDs.
# You can verify it with a `ps`
ar = lview.apply_async(os.getpid)
print("Engine ID:PID mapping: ", ar.get_dict())


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
The above procedure should look familiar to above, with one exception: the addition of the foreign key in the lcFeats table. The inclusion of the foreign key ensures a connected relationship between rawPhot and lcFeats. In brief, a row cannot be inserted into lcFeats unless a corresponding row, i.e. objId, exists in rawPhot. Additionally, rows in rawPhot cannot be deleted if there are dependent rows in lcFeats.
Problem 3d
Calculate features for every source in rawPhot and insert those features into the lcFeats table. You'll first need a function to calculate the features.
Can you write Python code for it?

# just like last time, we'll define a function to calculate the features for one lightcurve
# execute this

@require(ANTARESlc, 'pandas as pd', 'numpy as np', 'scipy.stats as spstat')
def calculate_features(filename):
    lc = ANTARESlc(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])
    lc.weighted_mean_flux()
    lc.normalized_flux_std()
    lc.normalized_amplitude()
    lc.normalized_MAD()
    lc.normalized_beyond_1std()
    lc.skew()
    lc.mean_colors()
    feats = (objId, lc.gStd, lc.rStd, lc.iStd, lc.zStd,
            lc.gAmp,  lc.rAmp,  lc.iAmp,  lc.zAmp,
            lc.gMAD, lc.rMAD, lc.iMAD, lc.zMAD,
            lc.gBeyond, lc.rBeyond, lc.iBeyond, lc.zBeyond,
            lc.gSkew, lc.rSkew, lc.iSkew, lc.zSkew,
            lc.gMinusR, lc.rMinusI, lc.iMinusZ)
    return feats


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
There are two things to highlight above: (1) we do not specify an id for the data as this is automatically generated, and (2) the data insertion happens via a tuple. In this case, we are taking advantage of the fact that a Python tuple is can be concatenated:
(objId,) + tuple(lc10.DFlc.ix[0]))

While the above example demonstrates the insertion of a single row to the database, it is far more efficient to bulk load the data. To do so we will delete, i.e. DROP, the rawPhot table and use some pandas manipulation to load the contents of an entire file at once via executemany.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if it already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")

# next 3 lines are already in name space; repeated for clarity
filename = "training_set_for_LSST_DSFP/FAKE001.dat"
lc = ANTARESlc(filename)
objId = int(filename.split('FAKE')[1].split(".dat")[0])

data = [(objId,) + tuple(x) for x in lc.DFlc.values] # array of tuples

cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
We now have a database connection object, conn. To interact with the database (create tables, load data, write queries) we need a cursor object.
Can you write Python code for it?


I am working on the file "Session3/Day4/ANTARES/miniAntaresSolutions_parallel.ipynb".
The first task is:
mini Challenge Problem
If there is less than 45 min to go, please skip this part.
Earlier it was claimed that bulk loading the data is faster than loading it line by line. For this problem - prove this assertion, use %%timeit to "profile" the two different options (bulk load with executemany and loading one photometric measurement at a time via for loop).
Compare both serial solutions, to load the data in parallel.
Hint - to avoid corruption of your current working database, miniANTARES.db, create a new temporary database for the pupose of running this test. Also be careful with the names of your connection and cursor variables.
Can you write Python code for it?

%%timeit
# bulk load solution

tmp_conn = sqlite3.connect("tmp1.db")
tmp_cur = tmp_conn.cursor()

tmp_cur.execute("""drop table if exists rawPhot""") # drop the table if it already exists
tmp_cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
                """)

for filename in filenames:
    lc = ANTARESlc(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    data = [(objId,) + tuple(x) for x in lc.DFlc.values] # array of tuples

    tmp_cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)

%%timeit
# bulk load solution

tmp_conn = sqlite3.connect("tmp1.db")
tmp_cur = tmp_conn.cursor()

tmp_cur.execute("""drop table if exists rawPhot""") # drop the table if it already exists
tmp_cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
                """)

for filename in filenames:
    lc = ANTARESlc(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    for obs in lc.DFlc.values:
        tmp_cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(obs)))


I am working on the file "Session3/code/02 Correlations - Air Quality.ipynb".
The first task is:
Working with NaN values
As you can see from the printed values, there are a few NaN in our array.
It probably means that the data was not recorded at this date.
We have several ways of dealing with that :

We can ignore the relevant data line
Or we can impute the missing data with, say, an average of a few previous points.

We are going to impute the missing data through interpolation.
Say we have 1,NaN,2.
We impute the missing value with (2+1)/2 - 1.5
Can you write Python code for it?

# Let's see how to do that with pandas

# Get the data to interpolate
ozone = df_temp["Ozone"].values
solar = df_temp["Solar.R"].values
# Get a series of timestamps
timestamps = pd.to_datetime(df_temp["Date"].values)

# Create a new Series with the timestamp as index
#(or we could have set the index as timestamp in our df_temp dataframe)
s_ozone = pd.Series(ozone, index=timestamps)
s_solar = pd.Series(solar, index=timestamps)

oz_interp = s_ozone.interpolate(method = "time")
sol_interp = s_solar.interpolate(method = "time")

df_temp["Ozone_interp"] = oz_interp.values
df_temp["Solar.R_interp"] = sol_interp.values

df_temp.head(n=10) # use 10 rows to see how interpolation works

#Let's plot the data
%matplotlib inline
fig = plt.figure(figsize=(15, 15))
gs = GridSpec(2, 2, bottom=0.18, left=0.18, right=0.88)

axOz = fig.add_subplot(gs[0])
axSol = fig.add_subplot(gs[1])
axWind = fig.add_subplot(gs[2])
axTemp = fig.add_subplot(gs[3])

# Get time axis
x_date = df_temp["Date"].values

# Get the y values
y_oz = df_temp["Ozone_interp"].values
y_sol = df_temp["Solar.R_interp"].values
y_temp = df_temp["Temp"].values
y_wind = df_temp["Wind"].values

# Plot
axOz.plot(x_date, y_oz, label = "Ozone in ppm", linewidth=1.5)
axSol.plot(x_date, y_sol, label = "Solar Radiation", linewidth=1.5)
axWind.plot(x_date, y_temp, label = "Temperature in F", linewidth=1.5)
axTemp.plot(x_date, y_wind, label = "Wind", linewidth=1.5)

#####################
# Figure cosmetics
#####################

# Axis labels, legend and formatting
for ax in [axOz, axSol, axWind, axTemp]:
    ax.set_xlabel("time", fontsize=22)
    ax.legend(loc="best", fontsize=22)

# improve plot layout
gs.tight_layout(fig, h_pad=3)

#plt.savefig("visual.png") #uncomment to save plot
plt.show()


I am working on the file "Session3/code/03 Supervised Learning - 00 Python basics and Logistic Regression.ipynb".
The first task is:
We use the <b> Iris dataset </b>
https://en.m.wikipedia.org/wiki/Iris_flower_data_set
The data set consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each sample: the length and the width of the sepals and petals, in centimetres. Based on the combination of these four features, Fisher developed a linear discriminant model to distinguish the species from each other.
logistic regression
While logistic regression gives each predictor (independent variable) a coefficient ‘b’ which measures its independent contribution to variations in the dependent variable, the dependent variable can only take on one of the two values: 0 or 1. What we want to predict from knowledge of relevant independent variables and coefficients is therefore not a numerical value of a dependent variable as in linear regression, but rather the probability (p) that it is 1 rather than 0 (belonging to one group rather than the other).
The outcome of the regression is not a prediction of a Y value, as in linear regression, but a probability of belonging to one of two conditions of Y, which can take on any value between 0 and 1 rather than just 0 and 1.
The crucial limitation of linear regression is that it cannot deal with dependent variable’s that are dichotomous and categorical. Many interesting variables are dichotomous: for example, consumers make a decision to buy or not buy, a product may pass or fail quality control, there are good or poor credit risks, an employee may be promoted or not. A range of regression techniques have been developed for analysing data with categorical dependent variables, including logistic regression and discriminant analysis.
Can you write Python code for it?

model = LogisticRegression()
model.fit(dataset.data, dataset.target)

expected = dataset.target
predicted = model.predict(dataset.data)

# classification metrics report builds a text report showing the main classification metrics
# In pattern recognition and information retrieval with binary classification,
# precision (also called positive predictive value) is the fraction of retrieved instances that are relevant,
# while recall (also known as sensitivity) is the fraction of relevant instances that are retrieved.
# Both precision and recall are therefore based on an understanding and measure of relevance.
# Suppose a computer program for recognizing dogs in scenes from a video identifies 7 dogs in a scene containing 9 dogs
# and some cats. If 4 of the identifications are correct, but 3 are actually cats, the program's precision is 4/7
# while its recall is 4/9.

# In statistical analysis of binary classification, the F1 score (also F-score or F-measure) is a measure of a test's accuracy.
# It considers both the precision p and the recall r of the test to compute the score:
# p is the number of correct positive results divided by the number of all positive results,
# and r is the number of correct positive results divided by the number of positive results that should have been returned.
# The F1 score can be interpreted as a weighted average of the precision and recall

print(metrics.classification_report(expected, predicted))

# Confusion matrix
# https://en.wikipedia.org/wiki/Confusion_matrix
# In the field of machine learning, a confusion matrix is a table layout that allows visualization of the performance
# of an algorithm, typically a supervised learning one.
# Each column of the matrix represents the instances in a predicted class
# while each row represents the instances in an actual class (or vice-versa)

# If a classification system has been trained to distinguish between cats, dogs and rabbits,
# a confusion matrix will summarize the results of testing the algorithm for further...


I am working on the file "Session3/code/04 Unsupervised and supervised Learning.ipynb".
The first task is:
It's a 50000 x 784 array :

There are 50000 handwritten digits
Each digit is stored in an array of dimension 784 = 28*28
This array stores the grayscale value of a 28*28 picture of the digit.

Let's visualise the first digit.
N.B. : Close the figure to continue
Can you write Python code for it?

%matplotlib inline
first_digit = train_data[0]
# reshape the digit to a 28*28 array
first_digit = np.reshape(first_digit, (28,28))
# Then plot the digit
fig = plt.figure()
im = plt.imshow(first_digit, cmap = mpl.cm.Greys)
im.set_interpolation("nearest")
plt.show()
# We recognize a handwritten 5.
# let's look at the actual class of this digit
first_digit_class = train_class[0]
print "First digit class :", first_digit_class
# it's indeed a 5 !


I am working on the file "Session3/code/04 Unsupervised and supervised Learning.ipynb".
The first task is:
Likewise, eval and test contain handwritten digits and their class.
We won't look at those for now.
Part 3 : pandas format
Now that we know the structure of the data, let's put it in a pandas dataframe. It will be easier to manipulate.
Can you write Python code for it?

# Let's define a list of feature names
# We have 784 pixels, let's index them from 0 to 783
lfeat = ["p" + str(i) for i in range(784)]

# Build a dataframe with all features
df_mnist = pd.DataFrame(train_data, columns = lfeat)
# Add the target = digit class
df_mnist["Class"] = train_class
# Let's have a look at the first few rows
df_mnist.iloc[:5,:]


I am working on the file "Session3/code/04 Unsupervised and supervised Learning.ipynb".
The first task is:
Part 2 : Data exploration
Let's have a word on the data.

train contains 2 arrays : a data and a target array
The data is stored in train[0]
The targets (= class of the digit) are stored in train[1]
Can you write Python code for it?

train_data = train[0]
train_class = train[1]
print train_data.shape


I am working on the file "Session3/code/04 Unsupervised and supervised Learning.ipynb".
The first task is:
Part 3 : First attempt at unsupervised classification
Let's see how far we can go with a simple unsupervised learning method
We will use the K-Means algorithm.
The KMeans algorithm clusters data by trying to separate samples in n groups
of equal variance, minimizing a criterion known as the inertia or within-cluster sum-of-squares.
This algorithm requires the number of clusters to be specified.

What KMeans finds are the location of centroids= the mean of each of the 10 groups of equal variances.
Of course, we hope the algorithm has found 10 well separated  groups of points
KMeans will classify using the following rule : a point will be associated to the nearest cluster (i.e. the group of points whose centroid is closest to this particular point).

N.B.
Let's call i the index of this cluster.
Of course, i is not equal to the class of the digit.
It could very well be that all the 8 digits belong to cluster #3
Can you write Python code for it?

# Initialise the kmeans method
# we use 10 clusters under the naive assumption of one cluster per class of digit
km = KMeans(n_clusters=10, n_jobs = -1, precompute_distances=True)
#n_jobs = -1 to speed up with max # of CPU
#precompute_distances = True to speed up algorithm as well

#We'll take a subset of data, otherwise, it takes too much time
data_subset = df_mnist[lfeat].values[:2000]
class_subset = df_mnist["Class"].values[:2000]
#Let's examine the statistics of our subset
for i in range(10):
  print "%s samples of class %s" % (len(np.where(class_subset == i)[0]),i)
#Now fit
pred_km = km.fit_predict(data_subset)


I am working on the file "Session3/code/04 Unsupervised and supervised Learning.ipynb".
The first task is:
Part 4 : Measuring the performance
Now we will evaluate the performance of the algorithm.
If we have the ground truth labels (i.e. we know to which class each training sample belongs), we can define the classification performance with metrics that measure the similarity between label assignments.
In our problem, this means that we compare the cluster assignment to the actual class of the digit, ignoring permutations. (cf. N.B. above, the cluster index may not be equal to the class index)
scikit-learn provides a range of such metrics.
We will report scores for three of them :
- Adjusted Rand Index
- Mutual information
- V-measure
All these scores span the [0,1] range, the higher, the better.
Can you write Python code for it?

print "Rand score:", metrics.adjusted_rand_score(class_subset, pred_km)
print "MI:", metrics.adjusted_mutual_info_score(class_subset, pred_km)
print "V:",metrics.v_measure_score(class_subset, pred_km)