I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Unsupervised learning
For some unsupervised learning tasks, the most relevant metrics are subjective measurements such as the apparent visual structure in a dimensional reduction approach for visualisation.
In other tasks, particularly clustering, objective metrics of the success of the unsupervised learning are important. There are a wide range of metrics that have been developed. Some of these are unsupervised metrics, such as the silhoutte coefficient, which measures how distinct clusters are. Others are supervised metrics and need a set of known cluster labels to evaluate performance.
Can you write Python code for it?

# this is the classic "Iris" dataset, that captures 4 attributes measured from various irises (of three distinct species)
iris = sklearn.datasets.load_iris()

def plot_clustering_performance(dataset, min_cluster, max_cluster):
    sil = []
    ami = []
    com = []
    hom = []
    vm = []
    kn = np.arange(min_cluster, max_cluster)
    for i in kn:
        # plot various metrics as a function of the number of clusters
        kmeans = sklearn.cluster.KMeans(n_clusters=i)
        clusters = kmeans.fit_predict(dataset.data)
        ami.append(sklearn.metrics.adjusted_mutual_info_score(dataset.target, clusters))
        com.append(sklearn.metrics.completeness_score(dataset.target, clusters))
        hom.append(sklearn.metrics.homogeneity_score(dataset.target, clusters))
        vm.append(sklearn.metrics.v_measure_score(dataset.target, clusters))

        sil.append(sklearn.metrics.silhouette_score(dataset.data, clusters))

    plt.plot(kn, sil)
    plt.plot(kn, ami)
    plt.plot(kn, com)
    plt.plot(kn, hom)
    plt.plot(kn, vm)
    plt.legend(["Silhouette (unsupervised)", "Adjust mutual info", "Completeness", "Homogeneity", "V-measure"])

plot_clustering_performance(iris, 2, 10)

# WARNING: can be slow!
digits = sklearn.datasets.load_digits()
plot_clustering_performance(digits, 2, 20)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
We can apply a more modern dimensional reduction method -- t-distributed stochastic neighbour embedding -- to the digits dataset we used in the PCA example.
Can you write Python code for it?

# This is very slow to run
tsne_digits = sklearn.manifold.TSNE(n_components=2).fit_transform(digit_data)
plt.scatter(tsne_digits[:,0], tsne_digits[:,1], c=digits.target, cmap='jet')


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
t-SNE separates out the digits very nicely into separate regions one for each digit, from the image data alone.
Clustering
Clustering tries to find well-seperated (in some sense) partitions of a data set. It is essentially a search for natural boundaries in the data. An example use of clustering might be gathering data from a large number of plants (e.g. measuring flower sizes, colouring, leaf symmetries, etc.) and then applying clustering to the data to try and separate out potential species groupings.
There are many clustering approaches. A simple one is k-means, which finds clusters via an iterative algortihm. The number of clusters must be chosen in advance. In general, it is hard to estimate the number of clusters, although there are algorithms for estimating this. k-means proceeds by choosing a set of $k$ random points as initial cluster seed points; classifiying each data point according to its nearest seed point; then moving the cluster point towards the mean position of all the data points that belong to it.
The k-means algorithm does not guarantee to find the best possible clustering -- it falls into local minima.
Can you write Python code for it?

# this is the classic "Iris" dataset, that captures 4 attributes measured from various irises (of three distinct species)
iris = sklearn.datasets.load_iris()

# we plot the dataset on two of the dimensions
plt.scatter(iris.data[:,2], iris.data[:,1], c=iris.target, cmap='jet')

plt.figure()
# now we cluster *without* any knowledge of the true class labels
kmeans = sklearn.cluster.KMeans(n_clusters=3)
kmeans_target = kmeans.fit_predict(iris.data)
plt.scatter(iris.data[:,2], iris.data[:,1], c=kmeans_target, cmap='jet')
# plot the cluster centres
plt.plot(kmeans.cluster_centers_[:,2], kmeans.cluster_centers_[:,1], 'r*', markersize=20)

## if we get the number of clusters wrong, the algorithm hallucinates clusters where there are none:
plt.figure()
# now we try and find 5 clusters
kmeans = sklearn.cluster.KMeans(n_clusters=5)
kmeans_target = kmeans.fit_predict(iris.data)
plt.scatter(iris.data[:,2], iris.data[:,1], c=kmeans_target, cmap='jet')
# plot the cluster centres
plt.plot(kmeans.cluster_centers_[:,2], kmeans.cluster_centers_[:,1], 'r*', markersize=20)


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
Approximations
Another way to approach the approximation of probabilistic models is simply to consider only the most likely outcome (and disregard the other alternatives). If we find random variable configurations that maximise the likelihood term, this is called maximum likelihood estimation (MLE). If we take a prior into account, this is known as maximum a posteriori estimation.
This works well when the distribution under consideration has a single strong mode, and is less representative of the true distribution when the distribution is diffuse or has multiple modes (spikes).
The mean of a set of values is the maximum likelihood estimator for the mean of the population distribution if the population is normally distributed; see above the difference between the MLE estimate and the full distribution of values.
Information theory
Entropy
A key property of a probability distribution is the entropy. Intuitively, this is a measure of the "suprise" an observer would have when observing draws from the distribution. A flat, uniform distribution is very "surprising" because the values are very hard to predict; a narrow, peaked distribution is unsurprising because the values are always very similar.
This is a precise quantification -- it gives the information in a distribution. The units of information are normally bits; where 1 bit of information tells you the answer to exactly one yes or no question. The entropy tells you exactly how many of bits are needed (at minimum) to communicate a value from a distribution to an observer who knows the distribution already.
The entropy of a (discrete) distribution can be computed as:
$$H(x) = \sum_{x} -p(x) \log_2(x)$$
Consider for example a Bernoulli process; i.e. a discrete random variable that can take on two states, like a coin toss. If we call our two possible states 0 and 1, we can characterise this with a single parameter $q$, where $p(x=0)=q$ and $p(x=1)=(1-q)$ (as $p(x=0)+p(x=1)$ must equal 1 -- the coin must land on one side or the other).
If this process is very...
Can you write Python code for it?

def binary_entropy(q):
    return -q * np.log(q)/np.log(2) - (1-q) * np.log(1-q)/np.log(2)

def plot_binary_entropy():
    x = np.linspace(0,1,100)
    plt.plot(x, binary_entropy(x))
    plt.xlabel('$q$')
    plt.ylabel('$H(X)$')

plot_binary_entropy()


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
Bigram model
We can build a very simple probabilistic language model using n-grams, where we predict the next occurence of a symbol given some previous sequence of $n$ symbols.
A very basic model is a character-level bigram model, where we model the probability of a character $c_n$ given a previously seen character $c_{n-1}$, $p(c_n|c_{n-1})$.
The probability distribution can be inferred from data simply by counting the occurence of each pair $(c_{n-1}, c_{n})$ and storing the result in a matrix which has dimensions $d \times d$ where $d$ is the number of distinct characters. If we then normalise by dividing each row by the total count of that character, we approximate the probability distribution of the language.
Can you write Python code for it?

def numerify(string):
    # remove all but letters and space (note that this is not a very efficient way to do this process)
    # and then convert to 0=space, 1-27 = a-z
    filtered_string = [max(1+ord(c.lower()) - ord('a'), 0) for c in string if c.isalpha() or c.isspace()]
    return filtered_string

def learn_bigram(string):
    # return a matrix with the bigram counts from string, including only letters and whitespace
    coded = numerify(string)
    joint = np.zeros((27,27))
    # iterate over sequential pairs
    for prev, this in zip(coded[:-1], coded[1:]):
        joint[prev, this] += 1
    # note that we add on an epsilon to avoid dividing by zero!
    bigram = joint.T / (np.sum(joint, axis=0)+1e-6)
    return bigram.T, joint.T

with open("macbeth.txt") as f:
    macbeth_bigram, macbeth_joint = learn_bigram(f.read())

with open("metamorphosis.txt") as f:
    metamorphosis_bigram, metamorphosis_joint = learn_bigram(f.read())

# The joint distribution
plt.imshow(macbeth_joint, interpolation='nearest')
plt.xticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.yticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.grid('off')

# The conditional distributions
plt.imshow(macbeth_bigram, interpolation='nearest')
plt.xticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.yticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.grid('off')

plt.figure()
plt.imshow(metamorphosis_bigram, interpolation='nearest')
plt.xticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.yticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.grid('off')


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
Conclusion
If you look at any 2D slice of a hypersphere, the points will appear to be concentrated at the centre as the dimension increases. But almost all of the volume is actually in a very thin shell at the outside of the sphere. This is quite counter-intuitive.
High-d spaces
The curse of dimensionality means that in high-dimensional spaces, data becomes extremely sparse because the volume is huge. This puts pressure on the learning algorithm which has to some how "fill in" the gaps between the data; the volume which must be inferred grows exponentially with dimension.
The non-intuitive geometrical properties of high-d spaces can lead to misunderstandings about the performance of learning algorithms.
Dealing with high-d spaces
One approach to mitigating high-d spaces in supervised learning problems is to apply pre-training, where unsupervised learning is used to infer a low-dimensional space, which is then classified. For many problems, the underlying space of variations is very limited and even if there is a huge vector of measurements, they all belong to a very small subspace.
Pre-training
For example, images of a rotating object might have thousands or millions of features but be characterised by a few variables. If manifold learning can extract this low-d latent space, then classification or regression becomes much simpler. Simple approaches like PCA can often find subspaces which contain much of the feature space needed for classifcation and ignore the irrelevant variables.
Can you write Python code for it?

## Using PCA in the sonar mines example
## We reduce the dimension to just 5 dimensions, then retrain

# proceed as in Lecture 1...
%cd datasets/uci
sonar_data = pd.read_csv("sonar.all-data")
sonar_features = np.array(sonar_data)[:,0:60].astype(np.float64)
sonar_labels = sklearn.preprocessing.label_binarize(np.array(sonar_data)[:,60], classes=['M', 'R']).ravel()

# split into a train and test section, holding out 30% (0.3) of the data for testing
sonar_train_features, sonar_test_features, sonar_train_labels, sonar_test_labels = sklearn.cross_validation.train_test_split(
    sonar_features, sonar_labels, test_size=0.3, random_state=0)

# fit an SVM as we did before
svm = sklearn.svm.SVC(C=12, gamma=0.1)
svm.fit(sonar_train_features, sonar_train_labels)
print "Score (original data): %f" % svm.score(sonar_test_features, sonar_test_labels)

## this is the new stuff...
# now apply PCA
n_components = 5
pca = sklearn.decomposition.PCA(n_components, whiten=True)
pca.fit(sonar_features)
low_d_train_features, low_d_test_features = pca.transform(sonar_train_features), pca.transform(sonar_test_features)
svm = sklearn.svm.SVC(C=12, gamma=0.1)
svm.fit(low_d_train_features, sonar_train_labels)
print "Score (PCA): %f" % svm.score(low_d_test_features, sonar_test_labels)

# note that an SVM doesn't generally benefit much from dimensionality reduction


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
In practice, this naive counting approach has serious problems as $n$ increases, both because storage requirements increase (an n-gram model stored like this needs $d^n$ entries) and because of the zero-count problem. As we increase $n$ there will be many sequences which never occur in the training data but have non-zero probability. Reducing the storage requirements requires pruning of the language model (removing very rare occurences which make up the long tail of the model); fixing the zero-count problem needs smoothing. There are a variety of algorithms used to smooth the distributions obtained via counting.
The zero count problem is very visible even in this simple bigram model with relatively large source texts (10s of thousands of characters), and can be seen as the black areas in the log probability plot. For example, the Macbeth model has $p(c_{n}=j|c_{n-1}=a)$ (the probability that $j$ follows $a$) as 0; but "aj" does not have zero probability of occuring in English text.
Sampling and computing likelihood
We can do interesting things with our simple probability model. For example, we can draw random samples from it. To draw samples, we use a Markov chain; a random walk. We fix some random initial start point and then work out what the distribution over characters is; draw a sample according to that distribution; append it to the context; and continue.
This is a random process that satisfies the Markov property: each new state depends only on the previous state (or on a fixed length set of previous states -- you can just extend what you mean by a state to include multiple previous steps).
Can you write Python code for it?

def sample_weighted(w):
    # given n probabilities, select an index according to those probabilities
    # we do this by generating a random number in [0,1], then summing up the probabilities
    # until we exceed this number. This is an unbiased way of sampling from the PMF
    r = np.random.uniform(0,1)
    cdf = 0
    i = 0
    while r>cdf:
        cdf += w[i]
        i += 1
    return i-1

def sample_chain(bigram, steps):
    # choose initial value
    v = sample_weighted(np.mean(bigram, axis=0))
    out = []
    for i in range(steps):
        out.append(" abcdefghijklmnopqrstuvwxyz"[v])
        # now select the PMF p(x|v)
        row = bigram[v,:]
        # and sample from it, updating v
        v = sample_weighted(row)
    # concatenate list into a single string
    return "".join(out)

print sample_chain(macbeth_bigram, 100)
print sample_chain(metamorphosis_bigram, 100)

# this is a uniform distribution over letters (i.e. p(c_n = x| c_{n-1} = y) is equal for all x and y)
print sample_chain(np.ones((27,27))/27.0, 100)


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
Non-real data: bag-of-words
A huge advantage of kernel based methods is that they do not need real valued data and they can work with models where computing the feature set would be very challenging.
All they need is a well behaved distance function. For example, we can perform dimensional reduction on text documents by defining a metric between two documents $d_i$ and $d_j$.
A very simple, but very effective metric is bag of words -- simply creating a binary vector of the words (1 if a word occurs, 0 if it doesn't) that occur in one document and taking the intersection with those that occur in the other. The really key thing here is we don't need to store these vectors or even compute the universe of all words -- we can just compute the documents as needed. Other binary vector metrics such as the cosine distance can be used in the same way.  We can also replace the binary values with the count of words in the document to weight words that occur often more heavily.
Support vector machines
Support vector machines use the kernel trick to project the data into a high dimensional space, and then use an efficient optimisation technique to find a linear classifier that maximises margins -- that is finds a (hyper)plane that separates the two classes with as large of an aggregate distance from any of the data points to the separating plane as possible.
This forces the plane to pass through areas where is less risk of confusion between the two classes and thus acts as a regulariser; the maximum margin criterion avoids "sneaky fits" where the separating plane precisely divides two classes in the dataset through a very narrow gap which is not robust to variations.
Can you write Python code for it?

np.random.seed(20)
# generate some sample data
x1 = np.dot(np.random.normal(0,1, (50,2)), np.array([[1,0.2],[-0.5, 0.6]])) + [3,5]
x2 = np.dot(np.random.normal(0,1, (50,2)), np.array([[1,0.2],[-0.5, 0.6]]))

# plot the data and a separating line
def plot_line(x1, x2, m, c):

    plt.scatter(x1[:,0], x1[:,1], c='g')
    plt.scatter(x2[:,0], x2[:,1], c='r')

    # plot "new data"
    nx1 = np.dot(np.random.normal(0,1, (50,2)), np.array([[1,0.2],[-0.5, 0.6]])) + [3,5]
    nx2 = np.dot(np.random.normal(0,1, (50,2)), np.array([[1,0.2],[-0.5, 0.6]]))

    plt.scatter(nx1[:,0], nx1[:,1], c='g', alpha=0.2)
    plt.scatter(nx2[:,0], nx2[:,1], c='r', alpha=0.2)

    xs = np.linspace(-8,8,50)
    plt.plot(xs, xs*m+c)
    plt.ylim(-8,8)
    plt.xlim(-8,8)

# plot a line that separates the data well
plot_line(x1, x2, -0.5, 3)
plt.title("Good margin")
plt.figure()

# plot a line that separates the data but is very sensitive to noise
plot_line(x1, x2, -1.5, 3.5)
plt.title("Bad margin")


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
Note that this is obviously a non-rigid transformation of the original dataset. We can compare standard PCA on the original features.
Can you write Python code for it?

pca = sklearn.decomposition.PCA(n_components=2)
pca.fit(iris.data)
pca_result = pca.transform(iris.data)
plt.scatter(pca_result[:,0], pca_result[:,1], c=iris.target, cmap='jet', s=40)


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
Overfitting and regularisation
Overfitting
We saw examples of overfitting in the previous lecture. Preventing overfitting is one of the major challenges of machine learning. It is easy to build models that can approximate any function; but approximating in a way that generalises to unseen data is much harder.
Enforcing structure to minimise overfitting is called regularisation. Techniques such as bagging or bootstrap methods that we saw in the previous lecture are examples of randomised approaches to regularisation.  By forcing learning algorithms to learn from impoverished datasets and then averaging the results, we can prevent the learning from conforming into the irrelvant "nooks and crannies" of the functions to be learned.
Penalisation
An alternative approach is to explicitly penalise functions for getting too complicated. This is called penalisation and is widely used in statistical modelling. An explicit penalty function is defined which applies a constraint to the optimisation of parameters to force the function being learned to stay "smooth".
Developing a penalty function is tricky, as it is not always clear how enforcing constraints of the parameters of a learning algorithm affects the generalisation properties of the result. However, there are many well-developed techniques for which good penalty function are known.
These functions often try to make the parameters small or to make as many of them as possible zero (maximising sparseness).
Lasso / ridge regression
One common approach is to apply penalties when fitting a linear model (i.e. $y=w^Tx$) that constrain the values of linear fit to be small. The specific norm used to measure the size of the coefficients gives rise to various regularisation approaches.
An approach which forces the coefficients to be small in the $L_2$ norm (i.e. the Euclidean distance of the coefficient vector to the origin) is called ridge regression; using the $L_1$ norm (the sum of coefficient values) is called lasso regression.
Can you write Python code for it?

# FIXME
n = 20

x = np.random.uniform(-1,1,(n,1))
y = 4*x[:,0] + np.random.normal(0,0.3, (n,))

plt.scatter(x,y)

# simple univariate regression

for n in range(20):
    ridge = sklearn.linear_model.Ridge(alpha=1*np.exp(-n), fit_intercept=True)
    ridge.fit(x,y)
    # predict output
    xs = np.linspace(-1,1,100)[:, None]
    ys = ridge.predict(xs)
    plt.plot(xs,ys,c=[n/20.0, n/20.0, n/20.0, 1])
    plt.ylim(-5,5)


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
Population and samples
Inferential statistics is concerned with estimating the properties of an unobserved "global" population of values from a limited set of observed samples.
This assumes that there is some underlying distribution from which samples are being drawn. If we assume a particular form of the distribution (e.g. assume it is normal), we can form estimators of parameters (such as the mean and variance) of this population distribution.
In other words, we might want to know the "true" mean height of adult men -- the distribution which is "generating" heights. But we can only observe a limited sample of heights by measuring people.
It turns out that if the population distribution is normal, the arithmetic mean of the observed samples $\hat{\mu}$ is an unbiased estimator of the true population mean $\mu$. Specific bounds can be put on this estimate; the standard error gives a measure of how close we expect that the arithmetic mean of samples is to the population mean.
Can you write Python code for it?

# population and sample example
def population_and_sample(mean, std):
    # we plot the estimated mean as the number of samples increases
    # along with the standard error on that estimate
    x = []
    n = 200
    sample_means = []
    sample_std_errs = []
    for i in range(n):
        x.append(np.random.normal(mean, std))
        sample_means.append(np.mean(x))
        sample_std_errs.append(stats.sem(x))

    # list to array
    sample_means = np.array(sample_means)
    sample_std_errs = np.array(sample_std_errs)

    plt.plot(sample_means)
    # sample error of one sample is undefined; replace NaN with 0
    sample_std_errs[0] = 0

    # ~1.96 * std. err. is the 95% confidence interval for the mean estimate
    plt.fill_between(np.arange(n), sample_means - 1.96 * sample_std_errs, sample_means + 1.96 * sample_std_errs, alpha=0.1)
    # line at population mean
    plt.axhline(mean)
    plt.xlim(-1, n)
    plt.xlabel("N samples")
    plt.ylabel("Value")
    plt.legend(['Sample mean', 'Population mean'])

population_and_sample(0, 1)


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
The standard error for the mean is:
    $$\text{SE} = \frac{s}{\sqrt{n}},$$
where $s$ is the sample standard deviation.
This tells us this estimate gets better relatively slowly as $n$ increases. The confidence interval of an estimate tells us how likely the parameter (here, the mean) is to have been within a range if we sampled from the distribution many times.
For a normal distribution, we'd expect the sample mean to be no more than ~1.96 $\text{SE}$ away from the population mean 95% of the time.
This does not mean that we'd expect 95% of the samples to be within 1.96$\text{SE}$ of the sample or population mean -- that range is called is the prediction interval.
Probability theory and Bayesian inference
Probability as a calculus of belief
Bayesians treat probability as a calculus of belief; in this model of thought, probabilities are measures of degrees of belief. $P(X)=0$ means a belief that $X$ cannot be true and $P(X)=1$ is a belief that $X$ is certainly 1.
There is an alternative school of thought that considers probabilities to only be the long-term behaviour of repeated events (e.g. the probability of a coin coming up heads in 0.5 because over the long term this will be the average proportion of times this occurs). A frequentist does not accept phrases like "what is the probability it is sunny just now?" as there is no long term behaviour for this.
Probability as the optimal way of representing uncertainty
Other representations of uncertainty are strictly inferior to probabilistic methods in the sense that a person, agent, computer or whatever is making decisions using probabilistic models will be at least as good as any other representation of uncertainty in a gambling setting. In other words, probabilistic decisions have the best possible "return" out of all decision systems when there is uncertainty.
Bayesians allow for belief in states to be combined and manipulated via the rules of probability. The key process in Bayesian logic is updating of beliefs. Given some prior belief (it's Glasgow,...
Can you write Python code for it?

### Fitting a normal with pymc
### We use Monte Carlo sampling to estimate the distribution of a normal distributions parameters given
### some observed data.
### Based on: http://matpalm.com/blog/2012/12/27/dead_simple_pymc/

def show_dag(model):
    dag = pymc.graph.dag(model)
    dag.write("graph.png",format="png")
    from IPython.display import Image
    i = Image(filename='graph.png')
    i

# generate data with a known distribution
data = np.random.normal(5, 10, (200,))

# specify the distribution of the unknown variables
mean = pymc.Uniform('mean', lower=min(data), upper=max(data))
precision = pymc.Uniform('precision', lower=0.0001, upper=1.0)

# specify the observed output variable
process =  pymc.Normal('process', mu=mean, tau=precision, value=data, observed=True)

model = pymc.Model([mean, precision, process])

# display the graphical model
show_dag(model)

# sample from the distribution
mcmc = pymc.MCMC(model)
mcmc.sample(iter=1000)
plt.figure()
plt.hist(mcmc.trace("mean")[:], normed=True)
plt.title("Mean of distribution")
plt.figure()
plt.hist(mcmc.trace("precision")[:], normed=True)
plt.title("Precision of distribution")

means = mcmc.trace("mean")[:]
precisions = mcmc.trace("precision")[:]
stds = np.sqrt(1.0/np.array(precisions))

plt.hist(data, bins=20, normed=True)
# plot each of the fits
x = np.linspace(-40, 40, 100)

for mean, std in zip(means, stds):
    y = stats.norm.pdf(x, mean, std)

    plt.plot(x,y,'b', alpha=0.01)
plt.ylim(0, 0.15)

# compute maximum likelihood estimate
mle_std = np.std(data)
mle_mean = np.mean(data)

y = stats.norm.pdf(x, mle_mean, mle_std)
plt.plot(x, y, 'r')


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
These aren't much like English, but they are very different from a uniform random selection of letters (the "monkey on the typewriter model") and the two models are also different from each other.
We can also take a string and compute how likely it is to have been the output of this Markov chain. We can compute the probability:
$$ p(\text{text}|\text{model}) = p(c2|c1) p(c3|c2) \dots = \prod_{i=1}^{n} p(c_{i+1}|c_i, \text{model})$$
but this will be very small -- instead, we can compute the log probability:
$$ \log p(\text{text}|\text{model}) = \sum_{i=1}^{n} \log p(c_{i+1}|c_i, \text{model})$$.
Can you write Python code for it?

def log_likelihood_bigram(string, bigram):
    symbols = numerify(string)
    llik = 0
    # we sum the log probabilities to avoid numerical underflow
    for prev, this in zip(symbols[:-1], symbols[1:]):
        llik += np.log(bigram[prev, this]+1e-8)
    return llik

# thane appears in Macbeth, but not in Metamorphosis
print log_likelihood_bigram("thane", macbeth_bigram)
print log_likelihood_bigram("thane", metamorphosis_bigram)

print "Romeo and Juliet"
with open("romeo_juliet.txt") as f:
    romeo_juliet = f.read()
    mb_llik = log_likelihood_bigram(romeo_juliet, macbeth_bigram)
    mm_llik = log_likelihood_bigram(romeo_juliet, metamorphosis_bigram)
    print "    ", "Macbeth", mb_llik, "Metamorphosis",  mm_llik, "Difference",mm_llik - mb_llik

print "The Trial"
with open("the_trial.txt") as f:
    the_trial = f.read()
    mb_llik = log_likelihood_bigram(the_trial, macbeth_bigram)
    mm_llik = log_likelihood_bigram(the_trial, metamorphosis_bigram)
    print "    ", "Macbeth", mb_llik, "Metamorphosis", mm_llik, "Difference", mm_llik - mb_llik


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
This suggests that we get about 0.65 bits of information about the successive character from the previous one for the Macbeth dataset, and about 0.79 bits for the Metamorphosis dataset; Kafka is marginally more predictable than Shakespeare.
High-dimensional spaces
Many problems in machine learning involve using many measurements of a simple; for example, an image is an array of luminance measurements. The dimension $d$ of the features has a massive impact on the performance and behaviour of learning algorithms, and many of the problems in machine learning are concerned with dealing with high-dimensional spaces.
High-dimensional can mean any $d>3$; a 20-dimensional feature set might be called medium-dimensional; a 1000-dimensional might be called high-dimensional; a 1M-dimensional dataset might be called extremely high-dimensional.
The geometric properties of high-d spaces are very counter-intuitive. The volume of space increases exponentially with $d$. There is a lot of empty space in high-dimensions, and where data is sparse it can be difficult to generalise in high-dimensional spaces. Some research areas, such as genetic analysis often have $n<<d$; i.e. many fewer samples than measurement features.
We can see some of the properties of high-d spaces visually.
The volume of a n-D sphere with radius $1/2$  is $$ V_n(R) = \frac{\pi^{n/2}}{\Gamma({n/2}+1)}\frac{1}{2}^n$$ (i.e. inscribed in a hypersphere). The volume of a unit cube is $$1^n=1$$
Can you write Python code for it?

def sphere_volume(n):
    return 0.5**n * np.pi**(n/2.0) / scipy.special.gamma(n/2.0+1)

x = np.arange(0,20)
plt.plot(x, [sphere_volume(xi) for xi in x])
plt.xlabel("Dimension")
plt.ylabel("Volume")
plt.figure()
plt.semilogy(x, [sphere_volume(xi) for xi in x])
plt.xlabel("Dimension")
plt.ylabel("Volume")


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
We can clearly see common features of English, such as the "qu" pair being dominant in the "q" row. We can see the structure more clearly if we look at the log probabilties:
Can you write Python code for it?

plt.imshow(np.log(macbeth_bigram+1e-8), cmap='gist_heat', interpolation='nearest')
plt.xticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.yticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.grid('off')
plt.figure()
plt.imshow(np.log(metamorphosis_bigram+1e-8), cmap='gist_heat', interpolation='nearest')
plt.xticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.yticks(range(27), ' abcdefghijklmnopqrstuvwxyz')
plt.grid('off')


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
We can define a function to plot this. We'll show the 1D distribution of radii, the distribution of radii as if they were on a circle and the true 2D projection of the hypersphere points.
Can you write Python code for it?

def plot_sphere_density(d):
    sphere_pts, line_pts, hyp_pts = sphere_points(2000,d)

    # plot the 2D sphere at the corresponding radius
    plt.scatter(sphere_pts[:,0], sphere_pts[:,1], alpha=0.5, s=2)
    # plot 1D points at the corresponding radius
    plt.scatter(line_pts, np.zeros_like(line_pts), c='g', alpha=0.1, s=2)
    # plot the 2D projection of the hypersphere points
    plt.scatter(-line_pts, np.zeros_like(line_pts), c='g', alpha=0.1, s=2)
    if d>1:
        plt.scatter(hyp_pts[:,0], hyp_pts[:,1], c='r', alpha=0.5, s=2)
    plt.axis("equal")

plot_sphere_density(1)

plot_sphere_density(2)

plot_sphere_density(3)

plot_sphere_density(8)

plot_sphere_density(128)


I am working on the file "CrashCourse_Lecture2.ipynb".
The first task is:
We can generate points randomly in a hypersphere. It's hard to visualise the hypersphere, but we can show the radii of points on a 2D circle:
Can you write Python code for it?

def sphere_points(n, d):
    # generate points on the unit circle (uniformly)
    xn = np.random.normal(0,1,(n,2))
    r = np.sqrt(np.sum(xn**2, axis=1))
    surface_points = (xn.T/r).T

    # generate points on the unit d-dimensional hypershphere (uniformly)
    xv = np.random.normal(0,1,(n,d))
    r_d = np.sqrt(np.sum(xv**2, axis=1))
    d_surface_points = (xv.T/r_d).T

    # generate points on the unit line
    xt = np.random.normal(0,1,(n,1))

    # radii of points uniformly distributed in a n-d hypersphere
    # can be drawn by sampling using the formula below
    # [see: http://math.stackexchange.com/questions/87230/picking-random-points-in-the-volume-of-sphere-with-uniform-probability?rq=1 ]
    radius = np.random.uniform(0, 1, n) ** (1.0/d) * 0.5
    return (surface_points.T*radius).T, radius, (d_surface_points.T*radius).T


I am working on the file "Crawlers/Twitter/cve_twitter_extraction.ipynb".
The first task is:
2. Tweet Extraction
We use the twython library for the tweet extraction. Twython is an actively maintained, pure Python wrapper for the Twitter API. It supports both normal and streaming Twitter APIs.
The primary task is to obtain the keys and tokens required to access the API and then access the functions in the wrappers.
The scripts below have two configurable parameters:
1. The query_word variable needs to be initialized with the keywords we are looking for on the twitter feed
2. The max_tweets variable is the number of tweets we plan to extract for the keywords mentioned above
If query_word is initialized to multiple words, the code will retrieve set of tweets that consists of all the words.
2.1. Streaming API
The streaming API is used to access all the current tweets. It returns approximately 1% of the tweets i.e. 60 tweets per second assuming a maximum of 6000 users tweet every second. There is no rate limit to the Streaming API.
The hyperlink has details related to the Twitter Streaming API limit:
1. URL 1
2. URL 2
Can you write Python code for it?

#import libraries
from twython import TwythonStreamer
from twython import Twython, TwythonError
import time
import sys
import os

#Configurable parameters
query_word="access-role"
max_tweets=5

searched_tweets_strm=[]

class MyStreamer(TwythonStreamer):
    def on_success(self, data):
        if len(searched_tweets_strm)<max_tweets:
            if 'text' in data:
                #print data['text'].encode('utf-8')
                searched_tweets_strm.append(data['text'].encode('utf-8'))
            else:
                print ("No tweets found")
                self.disconnect()
        else:
            print ("Max tweets extracted")
            sys.exit()

    def on_error(self, status_code, data):
        print (status_code, data)
        print ("Exception raised, waiting 15 minutes")
        time.sleep(15*60)

# Requires Authentication as of Twitter API v1.1
stream = MyStreamer(APP_KEY, APP_SECRET,
                    OAUTH_TOKEN, OAUTH_TOKEN_SECRET)

stream.statuses.filter(track=query_word)

print (searched_tweets_strm)


I am working on the file "Crawlers/Twitter/cve_twitter_extraction.ipynb".
The first task is:
A variable save_path is created which contains the path to the folder where the tweet files in json format will be stored. The folder name is saved in the format "tweet_MM_DD_YYYY".
Can you write Python code for it?

import os
now=datetime.datetime.now()
day=int(now.day)
month=now.month
year=int(now.year)
current_dir=os.getcwd()
save_path=os.path.join(current_dir, r'tweet_%i_%i_%i' % (now.month, now.day, now.year))
if not os.path.exists(save_path):
    os.makedirs(save)


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
CDU
CDU/CSU Fraktion
Can you write Python code for it?

domain = 'http://www.presseportal.de'
keyword = '/pm/7846/'
site = 'http://www.presseportal.de/nr/7846'
pages = ['{}/{}'.format(site, i * 27) for i in range(1, 621)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href']])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ParagraphExtractor(min_len=150)).save(path.join('data', 'CDU_Fraktion.txt'))

domain = 'http://www.presseportal.de'
keyword = '/pm/6518/'
site = 'http://www.presseportal.de/nr/6518'
pages = ['{}/{}'.format(site, i * 27) for i in range(1, 38)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href']])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ParagraphExtractor(min_len=150)).save(path.join('data', 'CDU.txt'))

domain = 'http://www.cdu-csu-ep.de'
keyword = '/presse/pressemitteilungen/'
blackwords = set(['content'])
site = 'http://www.cdu-csu-ep.de/pressearchiv.html'
pages = ['{}?start={}'.format(site, i * 5) for i in range(0, 643)]

def get_data():
    for page in pages:
        try:
            req = requests.get(page,...


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
Crawling german political party sites
Die Linke
News from the website of the party: http://www.die-linke.de/nc/die-linke/nachrichten
Can you write Python code for it?

domain = 'http://www.die-linke.de'
keyword = 'artikel'

site = 'http://www.die-linke.de/nc/die-linke/nachrichten'
pages = ['{}/browse/{}'.format(site, i) for i in range(1, 99)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href']])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ReadabilityExtractor(min_len=100)).save(path.join('data', 'Corpus_DieLinke.txt'))


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
FDP
News from the website of the party: http://www.fdp.de/pressemitteilungen
Can you write Python code for it?

domain = 'http://www.fdp.de'
keyword = '/content/'
blackwords = set(['datenschutz', 'impressum'])
site = 'http://www.fdp.de/pressemitteilungen'
pages = ['{}?page={}'.format(site, i) for i in range(1, 97)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href'] and check(link['href'], blackwords)])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ParagraphExtractor(min_len=100)).save(path.join('data', 'Corpus_FDP.txt'))


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
Grüne
Press releases from the website of the faction: http://www.gruene-bundestag.de/presse_ID_2000127
Can you write Python code for it?

domain = 'http://www.spdfraktion.de'
keyword = 'presse/pressemitteilungen/'
blackword = 'feed'
site = 'http://www.spdfraktion.de/presse/pressemitteilungen'
pages = ['{}?page={}'.format(site, i) for i in range(1, 733)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href'] and blackword not in link['href']])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ReadabilityExtractor(min_len=100)).save(path.join('data', 'Corpus_SPD_Fraktion.txt'))

domain = 'http://www.gruene-bundestag.de'
keyword = 'presse/pressemitteilungen/'
blackword = 'feed'
site = 'http://www.gruene-bundestag.de/presse_ID_2000127'
pages = ['{}/pb_id/100/seite/{}'.format(site, i) for i in range(2, 1322)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href'] and blackword not in link['href']])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ReadabilityExtractor(min_len=100)).save(path.join('data', 'Corpus_Grüne_Fraktion.txt'))


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
NPD
Can you write Python code for it?

keyword = '/?p='
site = 'http://aktion-widerstand.de/?page_id=11042'
pages = ['{}&paged={}'.format(site, i) for i in range(2, 335)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href']])
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ParagraphExtractor(min_len=100)).save(path.join('data', 'Corpus_NPD_Jung.txt'))

domain = 'http://www.npd-fraktion-mv.de'
keyword = '&view=article&'
blackwords = set(['content'])
site = 'http://www.npd-fraktion-mv.de/index.php?com=news&view=archive'
pages = ['{}&b={}&mid=8'.format(site, i * 50) for i in range(0, 38)]

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href'] and check(link['href'], blackwords)])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ReadabilityExtractor(min_len=100)).save(path.join('data', 'Corpus_NPD_MV.txt'))

domain = 'http://www.npd-fraktion-sachsen.de'
blackwords = set(['meldungen', 'category', 'author'])
site = 'http://www.npd-fraktion-sachsen.de/category/meldungen'
pages = ['{}/page/{}'.format(site, i) for i in range(2, 194)]

def get_data():
    for page in pages:
        try:
            req = requests.get(page,...


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
PR from the faction: http://www.liberale.de/page/pressemitteilungen
Can you write Python code for it?

domain = 'http://www.liberale.de'
keyword = '/content/'
blackwords = set(['datenschutz', 'impressum'])
site = 'http://www.liberale.de/page/pressemitteilungen'
pages = ['{}?page=0%2C{}'.format(site, i) for i in range(1, 1063)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href'] and check(link['href'], blackwords)])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ParagraphExtractor(min_len=100)).save(path.join('data', 'Corpus_FDP_Fraktion.txt'))


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
Press releases from the faction: http://www.linksfraktion.de/pressemitteilungen
Can you write Python code for it?

domain = 'http://www.linksfraktion.de'
keyword = 'pressemitteilungen'
site = 'http://www.linksfraktion.de/pressemitteilungen'
pages = ['{}/?s={}'.format(site, i) for i in range(1, 1384)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if link.get('href') and keyword in link['href']])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ReadabilityExtractor(min_len=100)).save(path.join('data', 'Corpus_DieLinke_Fraktion.txt'))


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
Press releases from the website of the party: http://www.die-linke.de/nc/presse/presseerklaerungen/presseerklaerungen
Can you write Python code for it?

domain = 'http://www.die-linke.de'
keyword = 'artikel'
site = 'http://www.die-linke.de/nc/presse/presseerklaerungen/presseerklaerungen'
pages = ['{}/browse/{}'.format(site, i) for i in range(1, 272)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href']])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ReadabilityExtractor(min_len=100)).save(path.join('data', 'Corpus_DieLinke_PR.txt'))


I am working on the file "Crawling Political Party Sites.ipynb".
The first task is:
SPD
Press releases from the website of the faction: http://www.spdfraktion.de/presse/pressemitteilungen
Can you write Python code for it?

domain = 'http://www.spdfraktion.de'
keyword = 'presse/pressemitteilungen/'
blackword = 'feed'
site = 'http://www.spdfraktion.de/presse/pressemitteilungen'
pages = ['{}?page={}'.format(site, i) for i in range(1, 733)]
pages.append(site)

def get_data():
    for page in pages:
        try:
            req = requests.get(page, timeout=10)
            soup = BeautifulSoup(req.content)
            links = set([link['href'] for link in soup.findAll('a')
                     if keyword in link['href'] and blackword not in link['href']])
            links = map(lambda x: '{}/{}'.format(domain, x), links)
            for article in list(links):
                article_req = requests.get(article)
                yield article_req.content
        except RequestException as error:
            logging.error('Error: %s', error)

HTML2Corpus(get_data(), extractor=ReadabilityExtractor(min_len=100)).save(path.join('data', 'Corpus_SPD_Fraktion.txt'))


I am working on the file "Crawling_Basics.ipynb".
The first task is:
31.10.2016
the "urllib2" package seems buggy and will request pages which are out of date. Let's use the "requests" (which uses the "urllib3") package instead.
Can you write Python code for it?

skills_info(city = 'San Francisco', state = 'CA')

skills_info(city = 'New York', state = 'NY')


I am working on the file "Crawling_Basics.ipynb".
The first task is:
A tag has a name (say, "someTag"). It contains a set of attribute:value(s). The following is an example of the structure of a tag:
$\text{<someTag attr1="value" attr2="value1 value2">  A String  </someTag>}$
Can you write Python code for it?

print tag.name
print tag["class"]
print tag.attrs


I am working on the file "Crawling_Basics.ipynb".
The first task is:
Now, let's do something slightly more serious:
Remark: the following functions for web crawling were originally written by Dr. Steinweg-Woods (https://jessesw.com/Data-Science-Skills/). His code was not up to date and some issues exist. I have cured some problems.
Can you write Python code for it?

def skills_dict(doc_frequency):
    prog_lang_dict = Counter({'R':doc_frequency['r'], 'Python':doc_frequency['python'],
                        'Java':doc_frequency['java'], 'C++':doc_frequency['c++'],
                        'Ruby':doc_frequency['ruby'],
                        'Perl':doc_frequency['perl'], 'Matlab':doc_frequency['matlab'],
                        'JavaScript':doc_frequency['javascript'], 'Scala': doc_frequency['scala']})
    analysis_tool_dict = Counter({'Excel':doc_frequency['excel'],  'Tableau':doc_frequency['tableau'],
                            'D3.js':doc_frequency['d3.js'], 'SAS':doc_frequency['sas'],
                            'SPSS':doc_frequency['spss'], 'D3':doc_frequency['d3']})
    hadoop_dict = Counter({'Hadoop':doc_frequency['hadoop'], 'MapReduce':doc_frequency['mapreduce'],
                    'Spark':doc_frequency['spark'], 'Pig':doc_frequency['pig'],
                    'Hive':doc_frequency['hive'], 'Shark':doc_frequency['shark'],
                    'Oozie':doc_frequency['oozie'], 'ZooKeeper':doc_frequency['zookeeper'],
                    'Flume':doc_frequency['flume'], 'Mahout':doc_frequency['mahout']})
    database_dict = Counter({'SQL':doc_frequency['sql'], 'NoSQL':doc_frequency['nosql'],
                        'HBase':doc_frequency['hbase'], 'Cassandra':doc_frequency['cassandra'],
                        'MongoDB':doc_frequency['mongodb']})
    overall_total_skills = prog_lang_dict + analysis_tool_dict + hadoop_dict + database_dict # Combine our Counter objects
    return overall_total_skills

def text_cleaner(url):

    try:
        session = requests.Session()
        soup = BeautifulSoup(session.get(url, timeout=5).content, 'lxml')  # let our beautiful soup to parse the site
    except:
        print "connection error or something wrong. URL=",url
        return

    for script in soup(["script", "style"]):           # Remove these two unnecessary tags: "script" and "style"
        _=script.extract()

    stopwords =...


I am working on the file "Crawling_Basics.ipynb".
The first task is:
Now, let's see a slightly complex situation:
Can you write Python code for it?

html_doc = """
<title>The Dormouse's story</title>

The Dormouse's story

Once upon a time there were three little sisters; and their names were

Elsie,
Lacie and
Tillie;
and they lived at the bottom of a well.

...

"""
soup = BeautifulSoup(html_doc, 'html.parser')


I am working on the file "Crawling_Basics.ipynb".
The first task is:
Now, let's see if we can print out all the strings of the site:
Can you write Python code for it?

for string in soup.strings:
    print(repr(string))
    print((string))
    print(type(string))
    print"======================================"


I am working on the file "Crawling_Basics.ipynb".
The first task is:
So many spaces there. The spaces can be removed via stripping the strings:
Can you write Python code for it?

for string in soup.stripped_strings:
    print(repr(string.encode("ascii")))


I am working on the file "Crawling_Basics.ipynb".
The first task is:
Well, this I understand. The above is nothing but the concept of the iterator.

(https://wiki.python.org/moin/Generators)
Python provides generator functions as a convenient shortcut to building iterators. Lets us rewrite the above iterator as a generator function:
Can you write Python code for it?

def firstn(n):
    num = 0
    while num < n:
        yield num
        num += 1

a=firstn(3)
for a in firstn(3):
    print a


I am working on the file "Crawling_Basics.ipynb".
The first task is:
Well, we could also put all the stripped strings into a list:
Can you write Python code for it?

[repr(string.encode("ascii")) for string in soup.stripped_strings]

print soup.prettify()


I am working on the file "Crawling_Basics.ipynb".
The first task is:
apparently, it is a generator. So, we can do the following:
Can you write Python code for it?

for sibling in soup.a.next_siblings:
    print(repr(sibling))


I am working on the file "Crawling_Basics.ipynb".
The first task is:
back to the soup example. Let's find the siblings of the tag "a" in an iterating way.
Can you write Python code for it?


I am working on the file "Crawling_Basics.ipynb".
The first task is:
generator and iterator
I now have the problem about the type "generator". I'd like to understand both the types "generator" and "iterator" in Python better.
Can you write Python code for it?

a = [1, 2, 3, 4]
b=123
print type(a.__iter__)
print type(a.__init__)
print type(b.__init__)
print type(b.__iter__)


I am working on the file "Crawling_Basics.ipynb".
The first task is:
learn: the function reduce
Can you write Python code for it?

print reduce(lambda x, y: x+y, [1, 2, 3, 4, 5])
final_site_list = ['http://www.indeed.com/jobs?q=%22','data+scientist', '%22&l=', 'New+York']
print reduce(lambda x,y: x+y,final_site_list)
print "".join(final_site_list)


I am working on the file "Crawling_Basics.ipynb".
The first task is:
learn: the usage of Counter:
Can you write Python code for it?

a=Counter({"a":1,"b":3})
print a
print a["b"]

cnt=Counter()
cnt.update(['red', 'blue', 'red', 'green', 'blue', 'blue'])
print cnt

cnt = Counter()
for word in ['red', 'blue', 'red', 'green', 'blue', 'blue']:
    cnt[word] += 1
print cnt
cnt.update(['red', 'blue', 'red', 'green', 'blue', 'blue'])
print cnt

print cnt.items()
print type(cnt.items())


I am working on the file "Crawling_Basics.ipynb".
The first task is:
move from unicode to ascii:
Can you write Python code for it?

for string in soup.strings:
    print(repr(string))
    print repr(string.encode("ascii"))
    print


I am working on the file "Crawling_Basics.ipynb".
The first task is:
on the other hand, id is not a multi-valued attribute:
Can you write Python code for it?

id_soup = BeautifulSoup('

')
id_soup.p['id']
# 'my id'


I am working on the file "Crawling_Basics.ipynb".
The first task is:
or find its parent's parent(now we have the easiest situation, i.e. they have only one direct parent):
Can you write Python code for it?


I am working on the file "Crawling_Basics.ipynb".
The first task is:
or the stripped strings(unnecessary spaces are removed)
Can you write Python code for it?

for string in soup.stripped_strings:
    print(repr(string))


I am working on the file "Crawling_Basics.ipynb".
The first task is:
recall:lambda function (http://www.secnetix.de/olli/Python/lambda_functions.hawk)
Can you write Python code for it?

foo = [2, 18, 9, 22, 17, 24, 8, 12, 27]

print filter(lambda x: x % 3 == 0, foo)    # from python official doc:
                                           # filter(function, iterable)
                                           # is equivalent to [item for item in iterable if function(item)]
print map(lambda x: x * 2 + 10, foo)
print reduce(lambda x, y: x + y, foo)
print sum(foo)


I am working on the file "Crawling_Basics.ipynb".
The first task is:
the use of the built-in functions "split" and "join":
Can you write Python code for it?

a = "Free your mind."
b = "Welcome to the desert... of the real."
c = "What is real? How do you define real?"

print(a)
print(a.split())
print

print(b)
print(b.split("o"))
print

print(c)
print(c.split(" ", 4))
print

print '+'.join("abc")
print '+'.join(["a","b","c"])


I am working on the file "Crawling_Danawa_Lg .ipynb".
The first task is:
다나와 사이트 크롤링
Can you write Python code for it?

driver.get("http://prod.danawa.com/list/?cate=112758&searchOption=btnAllOptUse=true/searchMaker%5B%5D=702/innerSearchKeyword=")

import pandas as pd
import pickle

post= driver.find_elements_by_css_selector("#productListArea")[0].find_elements_by_css_selector("div.main_prodlist.main_prodlist_list")[0]

post_elements = post.find_elements_by_css_selector("div.prod_main_info")

a = post_elements[0].text.strip()
a

import re
pattern = re.compile("[SSDDRLmTAM23.]{3,5} / [0-9]{1,3}[G,T]{1}B")
pattern2 = re.compile("[0-9]?[.]?[0-9]{1,3}[ K]{0,2}?g")
pattern2.findall(a)

df = pd.DataFrame(columns=["CPU" , "CPU2", "CPU3", "Ram" , "Memory_type" ,"Memory_size", "Weight" ,"Size", "Price"])

import re
def crawling():
    for post_element in post_elements:
        spec = post_element.find_element_by_css_selector("div.spec_list").text.strip()
        price = post_element.find_element_by_css_selector("div.prod_pricelist").text.strip()
        cpu_pattern = re.compile("코어i[0-9]{1}")
        cpu2_pattern = re.compile("[0-9]{1}세대")
        cpu3_pattern = re.compile("[0-9 ]{4}[A-Z]{1,2}")
        ram_pattern = re.compile(" [0-9]{1,2}GB")
        memory_pattern = re.compile("[SSDDRLmTAM234.]{3,5} / [0-9]{1,3}[G,T]{1}B")
        weight_pattern = re.compile("[0-9]?[.]?[0-9]{1,3}[ K]{0,2}?g")
        size_pattern = re.compile("[]?{1,2}[0-9]{1,2}[.]?[0-9 ]{1,3}인치")
        price_pattern = re.compile("[0-9]?[,]?[0-9]{2,3}[,][0-9]{3}원")

        cpu = cpu_pattern.findall(spec)[0]
        cpu2 = cpu2_pattern.findall(spec)[0]
        cpu3 = cpu3_pattern.findall(spec)[0]
        ram = ram_pattern.findall(spec)[0]
        memory = memory_pattern.findall(spec)
        if memory:
            memory_type = memory_pattern.findall(spec)[0].split("/")[0]
        else:
            memory = "HDD"

        memory_size = memory_pattern.findall(spec)[0].split("/")[1]

        weight = weight_pattern.findall(spec)
        if weight:
            weight = weight_pattern.findall(spec)[0]
        else:
            weight = "None"
        size =...


I am working on the file "Create Labeled Dataset.ipynb".
The first task is:
Get Statistics per-Project
Can you write Python code for it?

print('\t\t\t Positive \t Negative')
for project in projects:
    data = examples.filter(method__project__id=project.id)
    print('{0} \t {1} \t\t {2}'.format(str(project),
                                    data.filter(target=COHERENT).count(),
                                    data.filter(target=NOT_COHERENT).count()))


I am working on the file "Create Labeled Dataset.ipynb".
The first task is:
Settings the environment
Can you write Python code for it?

judges_combinations = (('leonardo.nole', 'rossella.linsalata'),
                       ('leonardo.nole', 'antonio.petrone'),
                       ('leonardo.nole', 'antonio.petrone'),
                       ('leonardo.nole', 'rossella.linsalata'),)

from coherence_dataset.settings import NOT_COHERENT, COHERENT
CODES_Labels = (NOT_COHERENT, COHERENT)
from collections import defaultdict
stats_results = defaultdict(list)

from evaluations import Judge
from coherence_dataset.models import Example

for pno, project in enumerate(projects):

    print('Processing Evaluation for ', str(project), ' Project')

    # Get Methods
    code_methods = project.code_methods.all()
    method_ids_map = dict()
    for method in code_methods:
        method_ids_map[method.id] = method

    print('Gathered ', len(method_ids_map.keys()), ' Methods')

    j1_usrname, j2_usrname = judges_combinations[pno]
    j1 = Judge(j1_usrname, project.name, project.version)
    j2 = Judge(j2_usrname, project.name, project.version)

    #getting just NC and CO evaluations
    j1_evals = j1.two_codes_evaluations
    j2_evals = j2.two_codes_evaluations

    project_stats = list()
    for i, label in enumerate(CODES_Labels):
        j1_evals_code = j1_evals[i]
        j2_evals_code = j2_evals[i]

        method_ids = j1_evals_code.intersection(j2_evals_code)

        print('Gathered ', len(method_ids),
              ' for {0} examples'.format('Positive' if i == 0 else 'Negative'))

        saved_instances_counter = 0
        for mid in method_ids:
            method = method_ids_map[mid]
            try:
                _ = method.example
            except Example.DoesNotExist:
                example = Example()
                example.method = method
                example.target = label
                example.save()
                saved_instances_counter += 1

        print('Saved ', saved_instances_counter,
              ' for {0}...


I am working on the file "Create Labeled Dataset.ipynb".
The first task is:
Test
Actually verify that instances have been saved into the DB and that querying the dataset will work as expected.
Can you write Python code for it?

from coherence_dataset.models import Example

examples = Example.objects.all()
print("Total examples in Dataset: ", examples.count())

from coherence_dataset.settings import COHERENT, NOT_COHERENT

print("Positive examples: ", examples.filter(target=COHERENT).count())

print("Positive examples: ", examples.filter(target=NOT_COHERENT).count())


I am working on the file "Creating Graphs with Pandas & matplotlib.ipynb".
The first task is:
Get date for when the PyLadies group was started
Can you write Python code for it?

def pyladies_created(pyladies_group):
    pyladies_dir = os.path.join(DATA_DIR, pyladies_group)
    pyladies_file = os.path.join(pyladies_dir, pyladies_group + ".json")
    with open(pyladies_file, "r") as f:
        data = json.load(f)
    created_ms = data.get("created")  # in ms after epoch
    created_s = created_ms / 1000
    created = datetime.datetime.fromtimestamp(created_s)
    year = created.strftime('%Y')
    month = created.strftime('%m')
    day = created.strftime('%d')
    return year, month, day


I am working on the file "Creating Graphs with Pandas & matplotlib.ipynb".
The first task is:
Load Data for PyLadies and their local Python User Groups
Can you write Python code for it?

DATA_DIR = "meetup_data"
MEMBER_JSON = "pug_members.json"
GROUP_DIRS = [d for d in os.listdir(DATA_DIR)]
PYLADIES_GROUPS = []
for group in GROUP_DIRS:
    if os.path.isdir(os.path.join(DATA_DIR, group)):
        PYLADIES_GROUPS.append(group)

def load_group_data(pyladies_group):
    pyladies_dir = os.path.join(DATA_DIR, pyladies_group)
    members_json = os.path.join(pyladies_dir, MEMBER_JSON)
    with open(members_json, "r") as f:
        return json.load(f)


I am working on the file "Creating Graphs with Pandas & matplotlib.ipynb".
The first task is:
To put it all together now
I'm only going to do a handful, not all the groups.  But this should give you an idea.
Also - some shit breaks. Surely I'll fix it in my Copious Amount of Free Time™ :D
Can you write Python code for it?

%matplotlib inline

def create_pyladies_graph(pyladies):
    group_data = load_group_data(pyladies)
    created = pyladies_created(pyladies)
    dfs = collect_dataframes(group_data)
    df = aggregate_dataframes(dfs)
    create_graph(df, pyladies, created, group_data.keys())

create_pyladies_graph(PYLADIES_GROUPS[2])

create_pyladies_graph(PYLADIES_GROUPS[11])

create_pyladies_graph(PYLADIES_GROUPS[14])

create_pyladies_graph(PYLADIES_GROUPS[15])

create_pyladies_graph(PYLADIES_GROUPS[19])

create_pyladies_graph(PYLADIES_GROUPS[22])

create_pyladies_graph(PYLADIES_GROUPS[28])

create_pyladies_graph(PYLADIES_GROUPS[30])

create_pyladies_graph(PYLADIES_GROUPS[39])


I am working on the file "Creating a Scalable Recommender System with Spark & Elasticsearch.ipynb".
The first task is:
2. Run ALS
Can you write Python code for it?

from pyspark.ml.recommendation import ALS
als = ALS(userCol="userId", itemCol="movieId", ratingCol="rating", regParam=0.1, rank=20)
model = als.fit(ratings_df)
model.userFactors.show(5)
model.itemFactors.show(5)


I am working on the file "Creating a Scalable Recommender System with Spark & Elasticsearch.ipynb".
The first task is:
3. Write ALS user and item factors to Elasticsearch
Utility functions for converting factor vectors
Can you write Python code for it?

from pyspark.sql.types import *
from pyspark.sql.functions import udf, lit

def convert_vector(x):
    '''Convert a list or numpy array to delimited token filter format'''
    return " ".join(["%s|%s" % (i, v) for i, v in enumerate(x)])
def reverse_convert(s):
    '''Convert a delimited token filter format string back to list format'''
    return  [float(f.split("|")[1]) for f in s.split(" ")]
def vector_to_struct(x, version):
    '''Convert a vector to a SparkSQL Struct with string-format vector and version fields'''
    return (convert_vector(x), version)
vector_struct = udf(vector_to_struct, \
                    StructType([StructField("factor", StringType(), True), \
                                StructField("version", StringType(), True)]))

# test out the vector conversion function
test_vec = model.userFactors.select("features").first().features
print test_vec
print
print convert_vector(test_vec)


I am working on the file "Creating a Scalable Recommender System with Spark & Elasticsearch.ipynb".
The first task is:
4. Recommend using Elasticsearch!
Can you write Python code for it?

from IPython.display import Image, HTML, display

def fn_query(query_vec, q="*", cosine=False):
    return {
    "query": {
        "function_score": {
            "query" : {
                "query_string": {
                    "query": q
                }
            },
            "script_score": {
                "script": "payload_vector_score",
                "lang": "native",
                "params": {
                    "field": "@model.factor",
                    "vector": query_vec,
                    "cosine" : cosine
                }
            },
            "boost_mode": "replace"
        }
    }
}

def get_similar(the_id, q="*", num=10, index="demo", dt="movies"):
    response = es.get(index=index, doc_type=dt, id=the_id)
    src = response['_source']
    if '@model' in src and 'factor' in src['@model']:
        raw_vec = src['@model']['factor']
        # our script actually uses the list form for the query vector and handles conversion internally
        query_vec = reverse_convert(raw_vec)
        q = fn_query(query_vec, q=q, cosine=True)
        results = es.search(index, dt, body=q)
        hits = results['hits']['hits']
        return src, hits[1:num+1]

def display_similar(the_id, q="*", num=10, index="demo", dt="movies"):
    movie, recs = get_similar(the_id, q, num, index, dt)
    # display query
    q_im_url = movie['image_url']
    display(HTML("
Get similar movies for:
"))
    display(Image(q_im_url, width=200))
    display(HTML("
"))
    display(HTML("
Similar movies:
"))
    sim_html = "

"
    i = 0
    for rec in recs:
        r_im_url = rec['_source']['image_url']
        r_score = rec['_score']
        sim_html += "	%2.3f	" % (r_im_url, r_score)
        i += 1
        if i % 5 == 0:
            sim_html += "

"
    sim_html += "

"
    display(HTML(sim_html))

display_similar(122886, num=5)

display_similar(122886, num=5, q="title:(NOT trek)")

display_similar(6377, num=5,...


I am working on the file "Creating a Scalable Recommender System with Spark & Elasticsearch.ipynb".
The first task is:
Check the data  was written correctly
Can you write Python code for it?

es.search(index="demo", doc_type="movies", q="star wars force", size=1)


I am working on the file "Creating a Scalable Recommender System with Spark & Elasticsearch.ipynb".
The first task is:
Convert factor vectors to [factor, version] form and write to Elasticsearch
Can you write Python code for it?

ver = model.uid
movie_vectors = model.itemFactors.select("id", vector_struct("features", lit(ver)).alias("@model"))
movie_vectors.select("id", "@model.factor", "@model.version").show(5)
user_vectors = model.userFactors.select("id", vector_struct("features", lit(ver)).alias("@model"))
user_vectors.select("id", "@model.factor", "@model.version").show(5)

# write data to ES, use:
# - "id" as the column to map to ES movie id
# - "update" write mode for ES
# - "append" write mode for Spark
movie_vectors.write.format("es") \
    .option("es.mapping.id", "id") \
    .option("es.write.operation", "update") \
    .save("demo/movies", mode="append")

user_vectors.write.format("es") \
    .option("es.mapping.id", "id") \
    .option("es.write.operation", "update") \
    .save("demo/users", mode="append")


I am working on the file "Credit Card Fraud Detection.ipynb".
The first task is:
<h2>**Data Exploration**</h2>

Alright, now that we have our transcations loaded, let's take a look at the data.
Can you write Python code for it?


I am working on the file "Credit Card Fraud Detection.ipynb".
The first task is:
Awesome, no missing values. Wish real life was this clean.
Let's see how Amount varies by Fraud / Not Fraud
Can you write Python code for it?

print('Fraud')
print(transactions.Amount[transactions.Class==1].describe())
print()
print('Not Fraud')
print(transactions.Amount[transactions.Class==0].describe())
print()

f, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(12,4))

bins = 50

ax1.hist(transactions.Amount[transactions.Class == 1], bins=bins)
ax1.set_title('Fraud')
ax2.hist(transactions.Amount[transactions.Class == 0], bins=bins)
ax2.set_title('Not Fraud')

plt.xlabel('Amount ($)')
plt.ylabel('Number of Transactions')
plt.yscale('log')
plt.show()


I am working on the file "Credit Card Fraud Detection.ipynb".
The first task is:
Fraudulent transactions are larger, on average, than non fraudulent transactions, despit the much longer tail on non fraudulent transactions.
Okay, next let's see how cyclical fraudulent and non fraudulent transactions are, respecitively.
Can you write Python code for it?

print('Fraud')
print(transactions.Time[transactions.Class==1].describe())
print()
print('Not Fraud')
print(transactions.Time[transactions.Class==0].describe())
print()

f, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(12,4))

bins = 50

ax1.hist(transactions.Time[transactions.Class == 1], bins=bins)
ax1.set_title('Fraud')
ax2.hist(transactions.Time[transactions.Class == 0], bins=bins)
ax2.set_title('Not Fraud')

plt.xlabel('Time (seconds from first transaction)')
plt.ylabel('Number of Transactions')
plt.yscale('log')
plt.show()


I am working on the file "Credit Card Fraud Detection.ipynb".
The first task is:
Looks like the data is all transformed and renamed, probably to anonymize the fields. This will make our work much less interpretible. Oh well, onward!
Can you write Python code for it?


I am working on the file "Crime-Weather-Analysis/Crime_Weather_Report-Analysis.ipynb".
The first task is:
Most common types of crimes in the past 5 years (2012-2017)
Can you write Python code for it?

#in general, what are the most common types of crimes in the past 5 years
plt.figure(figsize=(10,10))
sns.barplot(y=C_count['Primary Type'].value_counts().index.tolist(),
            x=C_count['Primary Type'].value_counts().values.tolist(),data=C_count,color='royalblue',orient='h')


I am working on the file "Crime-Weather-Analysis/Crime_Weather_Report-Analysis.ipynb".
The first task is:
Trend of crime in the past 5 years
Can you write Python code for it?

#total number of crimes during the summer
crime_peak = C_count.groupby(['Year','Month'])['Primary Type'].count().reset_index() #crime peak in the summer
crime_peak[(crime_peak['Month']>5) & (crime_peak['Month']<9)].groupby('Year')['Primary Type'].sum()

#getting the number of crimes, irrespective of crime type, in each month for each year
crime_monthly_count = C_count.groupby([C_count['Date'].dt.year,C_count['Date'].dt.month])['Primary Type'].count()
crime_monthly_count.index.names=['Year','Month']
crime_monthly_count = crime_monthly_count.reset_index()
crime_monthly_count.head(3)

# the drop in 2017 is due to the incomplete data..
plt.figure()
fig, ax = plt.subplots(figsize=(10,10))
for y in crime_monthly_count['Year'].unique().tolist():
    dat = crime_monthly_count[crime_monthly_count['Year']==y]
    ax.plot(dat['Month'], dat['Primary Type'], label = y)
plt.xticks(crime_monthly_count[crime_monthly_count['Year']==2012]['Month'].unique().tolist())
plt.xlabel('Month')
plt.ylabel("Number of Crimes")
plt.legend()
plt.show()


I am working on the file "Crime-Weather-Analysis/Crime_Weather_Report-Analysis.ipynb".
The first task is:
Visualizing any linear relationship between temperature and frequency of crime
Can you write Python code for it?

#breaking down the total number of crimes per day by crime type.
#i.e the 1462 crimes in January 1st, 2012 is now broken down to 335 incidents of thefts, 263 incidents of battery,..etc.
crimeReg = C_count.groupby(['Year','Month','Day'])['Primary Type'].value_counts()
crimeReg = pd.DataFrame(crimeReg)
crimeReg.columns=['Count']
crimeReg = crimeReg.reset_index()
crimeReg.head(3)

crimeReg_Temperature = crimeReg.merge(temp,on=['Year','Month','Day'],how='inner')
crimeReg_Temperature.head(3)
#crimeReg.to_csv('temp_reg_analysis.csv')

sns.lmplot(x='Mean_Temp',y='Count',hue='Primary Type',col='Primary Type',col_wrap=3,data=crimeReg_Temperature,scatter=True,fit_reg=False)


I am working on the file "Crimes by beat.ipynb".
The first task is:
Last question (for now): Did any particular type of crime spike before or after the pivot?
Can you write Python code for it?

for ucr in frame['IUCR'].unique():
    print ucr
    ucr_frame = frame[frame['IUCR'] == ucr]
    print '%d crimes on or after %s' % (ucr_frame[ucr_frame['Date Only'] >= pivot].Date.count(), pivot)
    print '%d crimes before %s' % (ucr_frame[ucr_frame['Date Only'] < pivot].Date.count(), pivot)
    print '---'


I am working on the file "Crimes by beat.ipynb".
The first task is:
Now let's graph it.
Can you write Python code for it?

# Let's get nicer-looking plots. Can't use ggplot because my version of matplotlib is too old (I think).
pd.set_option('display.mpl_style', 'default')
pd.set_option('display.width', 10000)
pd.set_option('display.max_columns', 60)
# We need to specifically ask matplotlib to display plots inline
%matplotlib inline
import matplotlib.pyplot as plt
frame.groupby('Date Only').count().plot(legend=None)


I am working on the file "Crimes by beat.ipynb".
The first task is:
Our first question: were there more crimes of all types committed before or after the pivot?
Can you write Python code for it?

print '%d crimes on or after %s' % (frame[frame['Date Only'] >= pivot].Date.count(), pivot)

print '%d crimes before %s' % (frame[frame['Date Only'] < pivot].Date.count(), pivot)


I am working on the file "Crimes by beat.ipynb".
The first task is:
This was run on 1/26/2016, 434 days after November 18, 2014. But the Data Portal only has data up to 1/18/2016, so we want to go to 426 days before the pivot, which works out to 9/18/2013.
Now let's download the crime data for processing. We've already filtered it on the Data Portal side to make sure it only contains a) the date range we want, b) the UCRs we're interested in, and c) the beats we're interested in.
Can you write Python code for it?

import pandas as pd
url = 'https://data.cityofchicago.org/api/views/qa42-2iy9/rows.csv?accessType=DOWNLOAD'
frame = pd.read_csv(url, parse_dates=['Date'])
print frame.head(2)
print '%d crimes found' % len(frame)


I am working on the file "Crocus/checking_snowhow_forcing.ipynb".
The first task is:
Checking out FORCING.nc
Can you write Python code for it?

forc_obs_f = r"..\Test\Data\FORCING_obsgrid.nc"
forc_arome_f = r"..\Test\Data\FORCING_arome.nc"

forc_arome_nc = netCDF4.Dataset(forc_arome_f, 'r')
forc_obs_nc = netCDF4.Dataset(forc_obs_f, 'r')

forc_rainf_obs_v = forc_obs_nc.variables["Rainf"]

lat_obs = forc_obs_nc.variables["LAT"][:]
lon_obs = forc_obs_nc.variables["LON"][:]

forc_rainf_arome_v = forc_arome_nc.variables["Rainf"]

lat_arome = forc_arome_nc.variables["LAT"][:]
lon_arome = forc_arome_nc.variables["LON"][:]

# choosing a point
N_forc = 35600

forc_rainf_arome = forc_rainf_arome_v[:].squeeze() # kg/m2 = mm

forc_rainf_obs = forc_rainf_obs_v[:].squeeze() # kg/m2 = mm

forc_time_arome = netCDF4.num2date(forc_arome_nc.variables["time"][:], forc_arome_nc.variables["time"].units)
forc_time_obs = netCDF4.num2date(forc_obs_nc.variables["time"][:], forc_obs_nc.variables["time"].units)

forc_diff = forc_rainf_obs - forc_rainf_arome

#whos

plt.plot(forc_time_obs, forc_rainf_obs[:, N_forc], color='r', label="Rain fall (OBS)")
plt.hold(True)
plt.plot(forc_time_arome, forc_rainf_arome[:, N_forc], color='b', label="Rain fall (AROME)")
#plt.plot(forc_diff[:, N_forc], color='k', label="Diff. Forc. OBS-AROME")
plt.legend()

precip_f = r"..\Test\Data\snowhow_pilot\seNorge_v2_0_PREC1h_grid_2015011221_2015011221.nc"
precip_mf = r"..\Test\Data\snowhow_pilot\seNorge_v2_0_PREC1h_grid_*.nc"

temp_f = r"..\Test\Data\snowhow_pilot\seNorge_v2_0_TEMP1h_grid_2015011221.nc"

snowrain_f = r"..\Test\Data\snowhow_pilot\snowrain_only_2015011221.nc"

# Is not the forcing file - CHANGE!!!
arome_f = r"..\Test\Data\snowhow_pilot\har25_00_20150112.nc"

precip_nc = netCDF4.Dataset(precip_f, 'r')
temp_nc = netCDF4.Dataset(temp_f, 'r')
snowrain_nc = netCDF4.Dataset(snowrain_f, 'r')

arome_nc = netCDF4.Dataset(arome_f, 'r')
precip_v = precip_nc.variables['precipitation_amount'] ## mm
temp_v = temp_nc.variables['temperature'] ## Celsius
snowrain_v = snowrain_nc.variables['precipitation_amount']

# setting indicies
X = 45
Y = 1185
N_arome = 21

time_v =...


I am working on the file "Crocus/checking_snowhow_forcing.ipynb".
The first task is:
Comparing snow-rain splitting
Can you write Python code for it?

f, (ax1, ax2, ax3) = plt.subplots(1, 3)
ax1.imshow(rainf)
ax1.set_title("Precip (grid): {0}".format(t_p))
ax2.imshow(snowrain)
ax2.set_title("Precip (conv): {0}".format(t_p))
ax3.imshow(diff)
ax3.set_title("Difference: {0}".format(t_p))


I am working on the file "Crocus/checking_snowhow_forcing.ipynb".
The first task is:
Difference between precipitation from obs.grid and AROME
Can you write Python code for it?

f, ([ax_obs, ax_aro], [ax_cobs, ax_caro]) = plt.subplots(2, 2)
im_obs = ax_obs.imshow(precip)
ax_obs.set_title("Precip (obs.grid): {0}".format(t_p))
plt.colorbar(im_obs, cax=ax_cobs, orientation="horizontal")
im_aro = ax_aro.imshow(rainf_arome[21, :, :])
ax_aro.set_title("Precip (AROME): {0}".format(time_arome[21]))
plt.colorbar(im_aro, cax=ax_caro, orientation="horizontal")


I am working on the file "Crocus/checking_snowhow_forcing.ipynb".
The first task is:
Parameters "" and "" are equal for the timestamp used.
Can you write Python code for it?

plt.plot(time_arome[:], rainf_arome[:, 320, 182])
plt.title("Arome precip in one grid cell")

rainf_arome_int = np.diff(rainf_arome[:, 320, 182], axis=0)
# reduced by one element
plt.plot(time_arome[1:], rainf_arome_int)
plt.title("Arome precip in one grid cell, per hour intensity")


I am working on the file "Crocus/checking_snowhow_forcing.ipynb".
The first task is:
The precipitation in the obs.grid is per hour, while precipitation in the arome-data-file is accumulated over the modelling periode (66 hours).
Is that regarded in following conversion process.
Can you write Python code for it?

plt.figure()
plt.imshow(snowf)
plt.title("Snow fall: {0}".format(t_p))
plt.colorbar()


I am working on the file "Cross Validation Code.ipynb".
The first task is:
Balancing Data
This section defines the data balancing function by over-sampling using the SMOTE algorithm (see SMOTE: Synthetic Minority Over-sampling Technique).
It takes a dataframe where each row contains the label (in column label) and the feature vector corresponding to that label. It returns a new dataframe of the same format, but with added rows resulted from the SMOTE oversampling process.
Can you write Python code for it?

from imblearn.over_sampling import SMOTE
from collections import Counter

def balance_smote(df):
    X = df.drop('label', axis=1)
    Y = df.label
    print('Original data shapes:', X.shape, Y.shape)

    smoX, smoY = X, Y
    c = Counter(smoY)
    while (min(c.values()) < max(c.values())):  # check if all classes are balanced, if not balance the first minority class
        smote = SMOTE(ratio="auto", kind='regular')
        smoX, smoY = smote.fit_sample(smoX, smoY)
        c = Counter(smoY)

    print('Balanced data shapes:', smoX.shape, smoY.shape)
    df_balanced = pd.DataFrame(smoX, columns=X.columns)
    df_balanced['label'] = smoY
    return df_balanced


I am working on the file "Cross Validation Code.ipynb".
The first task is:
Cross Validation Methodology
The following cv_test function carries out the cross validation test over n_iterations times and returns the accuracy scores and importance scores (for each feature). The cross validation steps are as follow:
* Split the input dataset (X, Y) into a training set and a test set using Stratified K-fold method with k = 10
* Train the Decision Tree classifier clf using the training set
* Score the accuracy of the classifier clf on the test set
* (Repeat the above until having done the required number of iterations)
Can you write Python code for it?

def cv_test(X, Y, n_iterations=1000, test_id=""):
    accuracies = []
    importances = []
    while len(accuracies) < n_iterations:
        skf = model_selection.StratifiedKFold(n_splits=10, shuffle=True)
        for train, test in skf.split(X, Y):
            clf = tree.DecisionTreeClassifier()
            clf.fit(X.iloc[train], Y.iloc[train])
            accuracies.append(clf.score(X.iloc[test], Y.iloc[test]))
            importances.append(clf.feature_importances_)
    print("Accuracy: %.2f%% ±%.4f <-- %s" % (np.mean(accuracies) * 100, t_confidence_interval(accuracies)[1] * 100, test_id))
    return accuracies, importances


I am working on the file "Cross Validation Code.ipynb".
The first task is:
The t_confidence_interval method below calculate the 95% confidence interval for a given list of values.
Can you write Python code for it?

def t_confidence_interval(an_array, alpha=0.95):
    s = np.std(an_array)
    n = len(an_array)
    return stats.t.interval(alpha=alpha, df=(n - 1), scale=(s / np.sqrt(n)))


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
<b> The above step is using 10-fold cross validation method to asset the model and returns the average cv_score</b>
Create random number to observe Overfitting and Underfitting
Can you write Python code for it?

def foo_func(x, err=0.5):
    y = 10 - 1./(x+0.1)
    if err > 0:
        y = np.random.normal(y, err)
    return y

def make_data(N=40, error=1.0, random_seed=1):
    np.random.seed(1)
    X = np.random.random(N)[:,np.newaxis]
    y = foo_func(X.ravel(), error)

    return X, y

X, y = make_data(40, error=1)
plt.figure(figsize=(10,8))
plt.scatter(X.ravel(),y)

X_test = np.linspace(-0.1, 1.1, 500)[:,None]

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
model = LinearRegression()
model.fit(X, y)
y_test = model.predict(X_test)
plt.figure(figsize=(10,8))
plt.scatter(X.ravel(),y)
plt.plot(X_test.ravel(),y_test)
plt.title("mean squared error: {0:.3g}".format(mean_squared_error(model.predict(X), y)))


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
 This above Validation Curves shows that when the degree is small, two curves are close to each other
==> causes UNDERFITTING.(Because it does not have enough complexity to represent the data)

If two curves are far apart, the training score seems very good although the validation score does not
==> causes OVERFITTING.(Because it has so much complexity, it will represent the noise instead of the real data)

===>>> From the plot, the best degree to choose to prevent both pitfalls mentioned above is 4
Can you write Python code for it?

model = PolynomialRegression(4).fit(X,y)
plt.figure(figsize=(10,8))
plt.scatter(X,y)
plt.plot(X_test, model.predict(X_test))

from sklearn.learning_curve import learning_curve

def plot_learning_curve(degree=3):
    train_sizes = np.linspace(0.05, 1, 20)
    N_train, val_train, val_test = learning_curve(PolynomialRegression(degree),
                                                  X, y, train_sizes, cv=5,
                                                  scoring=rms_error)
    plot_with_err(N_train, val_train, label='training scores')
    plot_with_err(N_train, val_test, label='validation scores')
    plt.xlabel('Training Set Size'); plt.ylabel('rms error')
    plt.ylim(0, 3)
    plt.xlim(5, 80)
    plt.legend()


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
From the scatter plot above, the model seems that cannot fit the majority of data. We use Linear Regression, it's a high bias model. In this example, our data is not linear, that's why this model is underfit.
We try to use a non-linear model to fit our data
In this example, I will use Polynomial Features which associates with Linear Regression to create a curve that hopefully will fit our data.

We merge Polynomial Features and Linear Regression into new model called Polynomial Regression.
Can you write Python code for it?

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree),
                         LinearRegression(**kwargs))

model = PolynomialRegression(2)
model.fit(X,y)
y_test = model.predict(X_test)
plt.figure(figsize=(10,8))
plt.scatter(X.ravel(), y)
plt.plot(X_test.ravel(), y_test)
plt.title("mean squared error: {0:.3g}".format(mean_squared_error(model.predict(X), y)))


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
Obviously, by increasing the degree, our model seems very fit the data and it tries to fit as much data as it can, including the noise.

 ==> OVERFITTING MODEL
VALIDATION CURVES and LEARNING CURVES
When you face a situation that your estimator performs terrably, you may concern:

* Use simpler or more complex model?
* Add more features to each observed data point?
* Add more training samples?

 << TRUE FACTS >>

- Sometimes using a more complicated model will give worse results.
- Sometimes adding training data will not improve your results.

Validation Curves: are plot of validation score and training score as a funcftion of model complexity

This plot is to help us in order to figure out which degree is good to apply to our model so that we can avoid either overfitting or underfitting.
Can you write Python code for it?

X, y = make_data(120, error=1.0)
plt.figure(figsize=(10,8))
plt.scatter(X, y);

from sklearn.learning_curve import validation_curve

def rms_error(model, X, y):
    y_pred = model.predict(X)
    return np.sqrt(np.mean((y - y_pred)**2))

degree = np.arange(0, 18)
val_train, val_test = validation_curve(PolynomialRegression(), X, y,
                                       'polynomialfeatures__degree', degree, cv=7,
                                       scoring=rms_error)

def plot_with_err(x, data, **kwargs):
    mu, std = data.mean(1), data.std(1) # mean, std deviation
    lines = plt.plot(x, mu, '-', **kwargs)
    plt.fill_between(x, mu - std, mu + std, edgecolor='none',
                     facecolor=lines[0].get_color(), alpha=0.2)
plt.figure(figsize=(10,8))
plot_with_err(degree, val_train, label='training scores')
plot_with_err(degree, val_test, label='validation scores')
plt.xlabel('degree'); plt.ylabel('rms error')
plt.legend()


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
We randomly choose the degree for our curve is 2 and the result seems very good for our model, the Mean Square Error is reduced.

Let's try with higher degree to see if the outcome is better.
Can you write Python code for it?

model = PolynomialRegression(30)
model.fit(X, y)
y_test = model.predict(X_test)
plt.figure(figsize=(10,8))
plt.scatter(X.ravel(), y)
plt.plot(X_test.ravel(), y_test)
plt.title("mean squared error: {0:.3g}".format(mean_squared_error(model.predict(X), y)))
plt.ylim(-4, 14)


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
<b>When the curves are close (underfitting), adding more data WILL NOT basically enhance the estimator </b>
Can you write Python code for it?

plt.figure(figsize=(10,8))
plot_learning_curve(3)

plt.figure(figsize=(10,8))
plot_learning_curve(10)


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
LOOK AT THIS OUTPUT!!!
We can see that the model seems to be very fit the data, 100% accuracy. This is totally an incorrect way to evaludate a model. We test the data on the same set of data used for training, that why it returns a very great performance
==> OVERFITTING MODEL
Using hold-out sets (splitting input data into train/test sets)
By doing this, we prevent our model from learning the noise. Then we can have better evaluation.

Note: To validate the performance of a model, we may use cross_val_score (cross validation score) in SKLEARN metrics or we may use directly the score function in each model.
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y)

knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print("{0} / {1} correct".format(np.sum(y_test == y_pred), len(y_test)))

knn.score(X_test, y_test)


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
Learning curves: are a plot of validation score and learning score as a function of number of training samples

Can you write Python code for it?


I am working on the file "Cross-Validation/Cross-Validation.ipynb".
The first task is:
We will get an example of K-NN classifier with digits dataset
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors = 1)
knn.fit(X,y)

y_pred = knn.predict(X)

print("{0} / {1} correct".format(np.sum(y == y_pred), len(y)))


I am working on the file "Cross-validation.ipynb".
The first task is:
Cross-Validation
Can you write Python code for it?

from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data
y = iris.target

from sklearn.cross_validation import cross_val_score
from sklearn.svm import LinearSVC

cross_val_score(LinearSVC(), X, y, cv=5)

cross_val_score(LinearSVC(), X, y, cv=5, scoring="f1_macro")


I am working on the file "Cross-validation.ipynb".
The first task is:
Exercises
Use  KFold cross validation and StratifiedKFold cross validation (3 or 5 folds) for LinearSVC on the iris dataset.
Why are the results so different? How could you get more similar results?
Can you write Python code for it?


I am working on the file "Cross-validation.ipynb".
The first task is:
Let's go to a binary task for a moment
Can you write Python code for it?

y % 2

cross_val_score(LinearSVC(), X, y % 2)

cross_val_score(LinearSVC(), X, y % 2, scoring="average_precision")

cross_val_score(LinearSVC(), X, y % 2, scoring="roc_auc")

from sklearn.metrics.scorer import SCORERS
print(SCORERS.keys())


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
2. Create a CrossCorrelation Object from two Light curves created above
To create a CrossCorrelation Object from LightCurves, simply pass both Lightvurves created above into the CrossCorrelation.
Can you write Python code for it?


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
3. Plot Cross Correlation for Different lags
To visulaize correlation for different values of time lags, simply call plot function on cs.
Can you write Python code for it?

cr.plot(labels = ['Time Lags (seconds)','Correlation'])


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
AutoCorrelation also supports {full,same,valid} modes similar to CrossCorrelation
Can you write Python code for it?

ac = AutoCorrelation(lc, mode = 'full')

ac.corr

ac.time_lags

ac.time_shift

ac.plot()


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
Creating CrossCorrelation with full mode now using same data as above.
Can you write Python code for it?

cr1 = CrossCorrelation(lc1, lc2, mode = 'full')

cr1.plot()

cr1.mode


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
CrossCorrelation Example
1. Create two light curves
There are two ways to create a Lightcurve.

1) Using an array of time stamps and an array of counts.

2) From the Photon Arrival times.
In this example, Lightcurve is created using arrays of time stamps and counts.
Generate an array of relative timestamps that's 10 seconds long, with dt = 0.03125 s, and make two signals in units of counts. The signal is a sine wave with amplitude = 300 cts/s, frequency = 2 Hz, phase offset of pi/2 radians, and mean = 1000 cts/s. We then add Poisson noise to the light curve.
Can you write Python code for it?

dt = 0.03125  # seconds
exposure = 10.  # seconds
freq = 1   # Hz
times = np.arange(0, exposure, dt)  # seconds

signal_1 = 300 * np.sin(2.*np.pi*freq*times) + 1000  # counts/s
signal_2 = 300 * np.sin(2.*np.pi*freq*times + np.pi/2) + 1000  # counts/s
noisy_1 = np.random.poisson(signal_1*dt)  # counts
noisy_2 = np.random.poisson(signal_2*dt)  # counts


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
Modes of Correlation
You can also specify an optional argument on modes of cross-correlation. <br>
There are three modes : 1) same   2) valid   3) full
Visit following ink on more details on mode : https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.signal.correlate.html
Default mode is 'same' and it gives output equal to the size of larger lightcurve and is most common in astronomy. You can see mode of your CrossCorrelation by calling mode attribute on the object.
Can you write Python code for it?


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
Now let's turn noisy_1 and noisy_2 into Lightcurve objects. This way we have two Lightcurves to calculate CrossCorrelation.
Can you write Python code for it?

lc1 = Lightcurve(times, noisy_1)
lc2 = Lightcurve(times, noisy_2)

len(lc1)

fig, ax = plt.subplots(1,1,figsize=(10,6))
ax.plot(lc1.time, lc1.counts, lw=2, color='blue')
ax.plot(lc1.time, lc2.counts, lw=2, color='red')
ax.set_xlabel("Time (s)", fontproperties=font_prop)
ax.set_ylabel("Counts (cts)", fontproperties=font_prop)
ax.tick_params(axis='x', labelsize=16)
ax.tick_params(axis='y', labelsize=16)
ax.tick_params(which='major', width=1.5, length=7)
ax.tick_params(which='minor', width=1.5, length=4)
plt.show()


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
Now, Cross Correlation values are stored in attribute corr, which is called below.
Can you write Python code for it?

cr.corr[:10]

# Time Resolution for Cross Correlation is same as that of each of the Lightcurves
cr.dt


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
Now, creating CrossCorrelation Object by passing lc1 and lc2 into the constructor.
Can you write Python code for it?

cs = CrossCorrelation(lc1, lc2)
print('Done')

cs.corr[:50]

# Time Resolution for Cross Correlation is same as that of each of the Lightcurves
cs.dt

cs.plot( ['Time Lags (seconds)','Correlation'])

cs.time_shift #seconds


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
The number of data points in corr and largest lightcurve are same in this mode.
Can you write Python code for it?


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
To create AutoCorrelation object, simply pass lightcurve into AutoCorrelation Constructor.<br> Using same Lighrcurve created above to demonstrate AutoCorrelation.
Can you write Python code for it?

lc = lc1

ac = AutoCorrelation(lc)
ac.n

ac.corr[:10]

ac.time_lags


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
Yet another Example with longer Lingcurve
I will be using same lightcurves as in the example above but with much longer duration and shorter lags.<br>
Both Lightcurves are chosen to be more or less same with a certain phase shift to demonstrate Correlation in a better way.
Again Generating two signals this time without poission noise so that time lag can be demonstrated. For noisy lightcurves, accurate calculation requires interpolation.
Can you write Python code for it?

dt = 0.0001  # seconds
exposure = 50.  # seconds
freq = 1       # Hz
times = np.arange(0, exposure, dt)  # seconds

signal_1 = 300 * np.sin(2.*np.pi*freq*times) + 1000 * dt # counts/s
signal_2 = 200 * np.sin(2.*np.pi*freq*times + np.pi/2) + 900 * dt  # counts/s


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
time_Shift for AutoCorrelation is always zero. Since signals are maximally correlated at zero lag.
Can you write Python code for it?


I am working on the file "CrossCorrelation/cross_correlation_notebook.ipynb".
The first task is:
time_shift is very close to 0.25 sec, in this case.
AutoCorrelation
Stingray has also separate class for AutoCorrelation. AutoCorrealtion is similar to crosscorrelation but involves only One Lightcurve.i.e. Correlation of Lightcurve with itself.
AutoCorrelation is part of stingray.crosscorrelation module. Following line imports AutoCorrelation.
Can you write Python code for it?

from stingray.crosscorrelation import AutoCorrelation