I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
Now the action.

Update: Most of the problems that we encountered were the result of errors in an older version of slate. Make sure that you have the most recent version of slate by running the following on your command line:
bash
$ pip install -U slate
...
$ pip show slate
If that doesn't work, you can also install the most recent version of the package directly from GitHub:
bash
$ pip install git+git://github.com/timClicks/slate.git
The code below has been modified to correct a few common errors. This takes considerably longer to run, but should result in much cleaner extractions.
Can you write Python code for it?

# 9, 10, and 13 are \t, \n, and \r, respectively.
m = dict.fromkeys([c for c in range(32) if c not in [9, 10, 13]])
problems = []
basepath = '../data/PDFs/'   # Folder containing PDF files.
for filename in os.listdir(basepath):
    filepath = os.path.join(basepath, filename)

    try:
        # Open the file.
        with codecs.open(filepath, 'r') as f:
            # Extract the text.
            extracted_text = slate.PDF(f)
    except:
        problems.append((filename, "Slate couldn't open the PDF"))
        continue

    # Correct over-spaced text (i.e. individual characters are separated by spaces).
    for i, page in enumerate(extracted_text):
        if np.mean([len(c) for c in page.split()]) < 2.:
            extracted_text[i] = re.sub('[ \t]{2,}', '_', page).replace(' ', '').replace('_', ' ')
        extracted_text[i] = re.sub('[ \t]{2,}', ' ', re.sub('[\n]{2,}', ' ', extracted_text[i]))

    # Join the pages, and decode.
    joined_text = '\n\n\n'.join(extracted_text)
    # The ``.translate(m)`` bit removes control characters.
    joined_text = joined_text.decode('utf-8').translate(m)

    # Figure out where to put the new text file.
    #  Updated to clean the filename -- NLTK chokes on non-ASCII filenames.
    textname = unidecode.unidecode(filename.replace('pdf', 'txt').decode('utf-8').translate(m))
    textpath = os.path.join(text_basepath, textname)

    # Open (create) the new text file
    with codecs.open(textpath, 'w', encoding='utf-8') as f:
        # Write the string to the file.
        f.write(joined_text)


I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
Now we have a PDF object.
Can you write Python code for it?


I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
We can generate the full paths to these files using the os.join() method.
Can you write Python code for it?

base_path = '../data/PDFs/'
for filename in os.listdir(base_path):
    print os.path.join(base_path, filename)


I am working on the file "1.1 Working with a corpus/1.1.2. Extracting plain text from text-embedded PDFs.ipynb".
The first task is:
You should now be able to find and open the text file at the path above.
Converting a whole bunch of PDFs
Now let's scale up. We'll assume that we have a folder containing all of our PDFs. First we have to generate a list of all of the files in that folder. Then we have to iterate over all of those files, and extract the text (as above), and write the text into a new set of files that we can use in our computational workflow.
The os package has a handy method called listdir() that will give us a list of files in a directory.
Can you write Python code for it?


I am working on the file "1.1. predicting if a player is going to complete the game or not.ipynb".
The first task is:
profiling players that complete the game
Can you write Python code for it?

players_stats[ (players_stats["complete"] == 1) & (players_stats["adventure"] == True) ].mean()


I am working on the file "1.1. predicting if a player is going to complete the game or not.ipynb".
The first task is:
simple decision tree
Can you write Python code for it?

from sklearn import tree

players_stats.head(5)

print len(players_stats)
print len(players_stats.drop_duplicates())

players_stats = players_stats.drop_duplicates()

from sklearn import tree

X = players_stats[ [ c for c in players_stats.columns if c not in [ "complete" ] ] ].as_matrix()
X = X.astype(int)

Y = players_stats["complete"].as_matrix()

clf = tree.DecisionTreeClassifier()
clf = clf.fit(X, Y)

from IPython.display import Image
from StringIO import StringIO

import pydot

dot_data = StringIO()
tree.export_graphviz(clf, out_file=dot_data,
                         feature_names=[ c for c in players_stats.columns if c not in [ "complete" ] ],
                         class_names= [ str(c) for c in range(0,2) ],
                         filled=True, rounded=True,
                         special_characters=True)

graph = pydot.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())

X_uncompleted = players_stats[ players_stats["complete"] == 0 ]
X_uncompleted = X_uncompleted.drop_duplicates()

print len(X_uncompleted)

X_uncompleted = X_uncompleted[ [ c for c in players_stats.columns if c not in [ "complete" ] ] ].as_matrix()

p = clf.predict(X)

print sum(p)

players_coordinates = pd.DataFrame.from_csv("data/players_coordinates.csv")

l = [ i for i, y in enumerate(p) if y == 1 ]

print len(l)

df = players_coordinates.iloc[l,:]
df = df[ df["y"] < 1500 ]

print len(df.index.unique())

players_list = list(df.index.unique())
plt.figure()

f, axarr = plt.subplots(len(players_list)/5+1, 5, figsize=(20,10))

for i, p in enumerate(players_list):
  select = df.loc[p,:]

  # print i

  ax = axarr[i/5, i%5]

  x = select["x"]
  y = select["y"]

  ax.plot(x,y, "-o")
  ax.axis('equal')

  ax.set_xlim([-300, 700])
  ax.set_ylim([0, 1200])

plt.show();


I am working on the file "1.2 Change and difference/1.2.1 Linear model with OLS.ipynb".
The first task is:
$N_{embryo}$
Can you write Python code for it?

print 'N_e', word_counts['embryo']
print 'N', word_counts.N()


I am working on the file "1.2 Change and difference/1.2.1 Linear model with OLS.ipynb".
The first task is:
$p("embryo") ~= \hat{p}("embryo") = f("embryo") $
Can you write Python code for it?

print word_probs.prob('embryo')    # Probability of an observed token to be 'embryo'.


I am working on the file "1.2 Change and difference/1.2.1 Linear model with OLS.ipynb".
The first task is:
...and then we use NLTK's MLEProbDist (Maximum Likelihood Estimator) to obtain the probability distribution.
Can you write Python code for it?


I am working on the file "1.2 Change and difference/1.2.1 Linear model with OLS.ipynb".
The first task is:
1.2. Change over time
In computational humanities, we are often interested in whether and how phenomena change over time. In this notebook we will perform a simple time-series analysis of tokens in our corpus. This will get us moving toward analyizing more complex temporal trends.
Load corpus and metadata
Just as we did in the last notebook, we'll load our texts and metadata.
Can you write Python code for it?

text_root = '../data/EmbryoProjectTexts/files'
zotero_export_path = '../data/EmbryoProjectTexts'

documents = nltk.corpus.PlaintextCorpusReader(text_root, 'https.+')
metadata = zotero.read(zotero_export_path, index_by='link', follow_links=False)


I am working on the file "1.2 Change and difference/1.2.1 Linear model with OLS.ipynb".
The first task is:
Has the prevalence of a token increased or decreased over time?
In the last notebook, we looked at the distribution (frequency) of tokens over time. It is one thing to make a pretty figure; it is another to say with some degree of confidence that a token is becoming more or less prevalent over time.
The simplest approach to this problem is a linear regression model:
$Y_i = \beta_0 + \beta X_i + \epsilon_i$
where $Y$ is the response variable (frequency of a token), $X$ is the predictor variable (publication date), $\beta$ is the regression coefficient, and $\epsilon_i$ is the error for observation $i$.
Up to now, we have discussed token frequency in terms of raw token counts. Since the number of texts per year may not be fixed, however, what we really want to model is the probability of a token. We don't have direct access to the probability of a token, but for most practical purposes the Maximum Likelihood Estimator for that probability is just the frequency $f(t) = \frac{N_{token}}{N_{total}}$ of the token.
To get the Probability Distribution of a token, we first calculate the frequency distribution:
Can you write Python code for it?

word_counts = nltk.FreqDist([normalize_token(token)
                             for token in documents.words()
                             if filter_token(token)])


I am working on the file "1.2 Change and difference/1.2.1 Linear model with OLS.ipynb".
The first task is:
The SciPy package provides a Ordinary Least Squares linear regression function called linregress(). We can use that to estimate the model parameters from our data.
Can you write Python code for it?

from scipy.stats import linregress
Beta, Beta0, r, p, stde = linregress(chicken_data.Year, chicken_data.Probability)

print '^Beta:', Beta
print '^Beta_0:', Beta0
print 'r-squared:', r*r
print 'p:', p

plt.scatter(chicken_data.Year, chicken_data.Probability)
plt.plot(chicken_data.Year, Beta0 + Beta*chicken_data.Year)    # Array math!
plt.ylim(chicken_data.Probability.min(), chicken_data.Probability.max())
plt.xlim(chicken_data.Year.min(), chicken_data.Year.max())

plt.ylabel('$\\hat{p}(\'chicken\'|year)$')
plt.show()    # Render the figure.


I am working on the file "1.2 Change and difference/1.2.1 Linear model with OLS.ipynb".
The first task is:
We need to use an hypothesis test that does not assume normality, and can handle the small sample size. One such approach is a permutation test.
Our null hypothesis is that:
$H_0: \beta = 0$
That is, that there is no change in the probability of our token (chicken, in this case) over time.
We will shuffle our response variable, $Y$, a whole bunch of times. Each time we shuffle the data (a permutation), we will re-calculate the regression parameter $\beta_{\pi}$. We can reject $H_0$ with a confidence of $p = 0.05$ iff our observed $\hat{\beta}$ falls outside the inner 95% of the resampled distribution.
Can you write Python code for it?

import numpy as np

# We can use underscores `_` for values that we don't want to keep.
samples = pd.DataFrame(columns=['Beta_pi', 'Beta0_pi'])
for i in xrange(1000):
    shuffled_probability = np.random.permutation(chicken_data.Probability)
    # linregress() returns five parameters; we only care about the first two.
    Beta_pi, Beta0_pi, _, _, _ = linregress(chicken_data.Year, shuffled_probability)
    samples.loc[i] = [Beta_pi, Beta0_pi]

plt.figure(figsize=(10, 5))

plt.subplot(121)
plt.hist(samples.Beta_pi)    # Histogram of Beta values from permutations.
plt.plot([Beta, Beta], [0, 200],     # Beta from the observed data.
         lw=5, label='$\\hat{\\beta}$')

# Plot the upper and lower bounds of the inner 95% probability.
Beta_upper = np.percentile(samples.Beta_pi, 97.5)
Beta_lower = np.percentile(samples.Beta_pi, 2.5)
plt.plot([Beta_upper, Beta_upper], [0, 200], color='k', lw=2, label='$p = 0.05$')
plt.plot([Beta_lower, Beta_lower], [0, 200], color='k', lw=2)

plt.legend()
plt.xlabel('$\\beta_{\\pi}$', fontsize=24)

# Same procedure for Beta0.
plt.subplot(122)
plt.hist(samples.Beta0_pi)
plt.plot([Beta0, Beta0], [0, 200], lw=5, label='$\\hat{\\beta_0}$')

Beta0_upper = np.percentile(samples.Beta0_pi, 97.5)
Beta0_lower = np.percentile(samples.Beta0_pi, 2.5)
plt.plot([Beta0_upper, Beta0_upper], [0, 200], color='k', lw=2, label='$p = 0.05$')
plt.plot([Beta0_lower, Beta0_lower], [0, 200], color='k', lw=2)

plt.legend()
plt.xlabel('$\\beta_{0\\pi}$', fontsize=24)
plt.tight_layout()
plt.show()


I am working on the file "1.2 Change and difference/1.2.2 Linear model - Bayesian approach.ipynb".
The first task is:
1.2.2. A Bayesian approach: the linear model
In the last few notebooks, we used a corpus that was generated from a Zotero collection. We read in our metadata from a Zotero RDF/XML document. Sometimes we either don't have access to additional metadata, or simply don't need it for our analysis. A very simple way to encode metadata is to include it in the file name. This isn't a very robust strategy, but for certain use-cases it does the job.
Can you write Python code for it?

text_root = '../data/SystemsBiology'
documents = nltk.corpus.PlaintextCorpusReader(text_root, '.+.txt')


I am working on the file "1.2 Change and difference/1.2.2 Linear model - Bayesian approach.ipynb".
The first task is:
In this corpus, the publication year is encoded as the first four characters of the filename. For examples:
Can you write Python code for it?


I am working on the file "1.2 Change and difference/1.2.2 Linear model - Bayesian approach.ipynb".
The first task is:
Instead of raw frequencies, we'll use token counts. The Poisson assumes that we are counting events (tokens) within periods/areas of equal size. So we'll ignore document boundaries, and obtain counts over equally-sized chunks of each annual subcorpus.
Can you write Python code for it?

organism_data = pd.DataFrame(columns=['Year', 'Count'])
i = 0
for year, indices in document_index.groupby('Year').groups.items():
    fileids = document_index.ix[indices].File
    tokens = [normalize_token(token)    # The tokens in this year.
              for token in documents.words(fileids=fileids)
              if filter_token(token)]

    N = len(tokens)    # Number of tokens in this year.
    chunk_size = 200    # This shouldn't be too large.
    for x in xrange(0, N, chunk_size):
        current_chunk = tokens[x:x+chunk_size]
        count = nltk.FreqDist(current_chunk)['organism']

        # Store the counts for this chunk as an observation.
        organism_data.loc[i] = [year, count]
        i += 1    # Increment the index variable.


I am working on the file "1.2 Change and difference/1.2.2 Linear model - Bayesian approach.ipynb".
The first task is:
It's helpful to view the data as a single scatterplot. I've tuned down the opacity on points, so that it's easier to see relative density. The orange dots show the mean count for each year.
Can you write Python code for it?

organism_data.plot(kind='scatter', x='Year', y='Count',
                   alpha=0.4, label='Raw data')
means = organism_data.groupby('Year').mean()
plt.scatter(means.index, means.Count,
            color='orange', s=50, label='Mean count')
plt.legend(loc='best')
plt.show()


I am working on the file "1.2 Change and difference/1.2.2 Linear model - Bayesian approach.ipynb".
The first task is:
M (below) is our MCMC simulation.
Can you write Python code for it?

M = pymc.MCMC(model)

M.sample(iter=100000, burn=10000, thin=50)

pymc.Matplot.plot(M)

organism_data.plot(kind='scatter', x='Year', y='Count', alpha=0.4, label='Raw data')
means = organism_data.groupby('Year').mean()
plt.scatter(means.index, means.Count, color='orange', s=50, label='Mean count')

years = organism_data.Year.unique()
years_offset = years - years.min()
Beta0_mean = M.Beta0.trace()[:].mean()
Beta_mean = M.Beta.trace()[:].mean()
predicted_counts = Beta0_mean + Beta_mean*years_offset
plt.plot(years, predicted_counts, lw=2, label='Model')

plt.legend(loc='best')
plt.show()


I am working on the file "1.2 Change and difference/1.2.2 Linear model - Bayesian approach.ipynb".
The first task is:
The histograms below summarize the observed count data for each year.
Can you write Python code for it?

organism_data.hist(column='Count', by='Year', figsize=(7, 7))
plt.show()


I am working on the file "1.2 Change and difference/1.2.4 Comparing word use between corpora.ipynb".
The first task is:
1.2.4. Comparing word use between corpora
In previous notebooks we examined changes in word use over time using several different statistical approaches. In this notebook, we will examine differences in word use between two different corpora.
Web of Science dataset
In this notebook we will use data retrieved from the ISI Web of Science database. One corpus is from the journal Plant Journal over the period 1991-2013. The other corpus is from the journal Plant Journal, 19991-2013. Each corpus is comprised of several WoS field-tagged metadata files contained in a folder.
Tethne's WoS parser can load all of the data files in a single directory all at once. This may take a few minutes, since Tethne goes to a lot of trouble in indexing all of the records for easy access later on.
Can you write Python code for it?

from tethne.readers import wos

pj_corpus = wos.read('../data/Baldwin/PlantJournal/')
pp_corpus = wos.read('../data/Baldwin/PlantPhysiology/')


I am working on the file "1.2 Change and difference/1.2.4 Comparing word use between corpora.ipynb".
The first task is:
Conditional frequency distribution
This next step should look familiar. We will create a conditional frequency distribution for words in these two corpora. We have two conditions: the journal is Plant Physiology and the journal is Plant Journal.
Can you write Python code for it?

word_counts = nltk.ConditionalFreqDist([
    (paper.journal, normalize_token(token))
    for paper in chain(pj_corpus, pp_corpus)    # chain() strings the two corpora together.
    for token in nltk.word_tokenize(getattr(paper, 'abstract', ''))
    if filter_token(token)
])


I am working on the file "1.2 Change and difference/1.2.4 Comparing word use between corpora.ipynb".
The first task is:
Here's the PDF of $\chi^2$ with one degree of freedom.
Can you write Python code for it?

X = np.arange(1, 100, 0.1)
plt.plot(X, distribution.pdf(X), lw=2)
plt.ylabel('Probability')
plt.xlabel('Value of $\chi^2$')
plt.show()


I am working on the file "1.2 Change and difference/1.2.4 Comparing word use between corpora.ipynb".
The first task is:
Is there a difference?
As a first step, we may wish to establish whether or not there is a difference between the two corpora. In this simplistic example, we will compare the rate at which a specific word is used in the two journals. In practice, your comparisons will probably be more sophisticated -- but this is a starting point.
So: Is the term photosynthesis used disproportionately in Plant Physiology compared to Plant Journal?
$H_0:  P("photosynthesis" \Bigm|J = "Plant Journal") = P("photosynthesis" \Bigm| J="Plant Physiology")$
To test this hypothesis, we will use Dunning's log-likelihood ratio, which is a popular metric in text analysis. In a nutshell, we want to assess whether or not the relative use of the term "photosynthesis" is sufficiently skewed to reject the null hypothesis.
The log likelihood ratio is calculated from a contingency table, similar to the one above. For a single word, our table will show the number of tokens that are the word "photosynthesis", and the number of tokens that are not, for each journal.
[ show table here ]
$
\sum_i O_i ln \frac{O_i}{E_i}
$
where $O_i$ is the observed value in cell $i$, and $E_i$ is the expected value in cell $i$.
First we will calculate the observed contingency table.
Can you write Python code for it?

plant_jour_photosynthesis = word_counts['PLANT JOURNAL']['photosynthesis']
plant_jour_notphotosynthesis = word_counts['PLANT JOURNAL'].N() - plant_jour_photosynthesis

plant_phys_photosynthesis = word_counts['PLANT PHYSIOLOGY']['photosynthesis']
plant_phys_notphotosynthesis = word_counts['PLANT PHYSIOLOGY'].N() - plant_phys_photosynthesis

# Create a 2x2 array.
contingency_table = np.array([[plant_jour_photosynthesis, plant_jour_notphotosynthesis],
                              [plant_phys_photosynthesis, plant_phys_notphotosynthesis]],
                             dtype=int)

contingency_table


I am working on the file "1.2 Change and difference/1.2.4 Comparing word use between corpora.ipynb".
The first task is:
Money.
A Bayesian approach
We have shown that these two corpora differ significantly in their usage of the term "photosynthesis". In many cases, we may want to go one step further, and actually quantify that difference. We can use a similar approach to the one that we used when comparing word use between years: use an MCMC simulation to infer mean rates of use (and credibility intervals) for each corpus.
Rather than starting with a null hypothesis that there is no difference between corpora, we will begin with the belief that there is an independent rate of use for each corpus. We will then infer those rates, and sample from their posterior distributions to generate credible intervals.
Once again, we will model the rate of use with the Poisson distribution. So we must generate count data for evenly-sized chunks of each corpus. We'll put all of our count observations into a single dataframe.
Can you write Python code for it?

count_data = pd.DataFrame(columns=['Journal', 'Year', 'Count'])
chunk_size = 400    # This shouldn't be too large.
i = 0

# The slice() function automagically divides each corpus up into
#  sequential years. We can use chain() to combine the two iterators
#  so that we only have to write this code once.
for year, papers in chain(pj_corpus.slice(), pp_corpus.slice()):
    tokens = [normalize_token(token)
              for paper in papers    # getattr() lets us set a default.
              for token in nltk.word_tokenize(getattr(paper, 'abstract', ''))
              if filter_token(token)]

    N = len(tokens)    # Number of tokens in this year.
    for x in xrange(0, N, chunk_size):
        current_chunk = tokens[x:x+chunk_size]
        count = nltk.FreqDist(current_chunk)['photosynthesis']

        # Store the count for this chunk as an observation.
        count_data.loc[i] = [paper.journal, year, count]
        i += 1    # Increment the index variable.

PJ_mean = pymc.Gamma('PJ_mean', beta=1.)
PP_mean = pymc.Gamma('PP_mean', beta=1.)

PJ_counts = pymc.Poisson('PJ_counts',
                         mu=PJ_mean,
                         value=count_data[count_data.Journal == 'PLANT JOURNAL'].Count,
                         observed=True)

PP_counts = pymc.Poisson('PP_counts',
                         mu=PP_mean,
                         value=count_data[count_data.Journal == 'PLANT PHYSIOLOGY'].Count,
                         observed=True)

model = pymc.Model({
    'PJ_mean': PJ_mean,
    'PP_mean': PP_mean,
    'PJ_counts': PJ_counts,
    'PP_counts': PP_counts
})

M1 = pymc.MCMC(model)
M2 = pymc.MCMC(model)
M3 = pymc.MCMC(model)

M1.sample(iter=20000, burn=2000, thin=20)
M2.sample(iter=20000, burn=2000, thin=20)
M3.sample(iter=20000, burn=2000, thin=20)

pymc.Matplot.plot(M1)

PJ_mean_samples = M1.PJ_mean.trace()[:]
PJ_mean_samples = np.append(PJ_mean_samples, M2.PJ_mean.trace()[:])
PJ_mean_samples = np.append(PJ_mean_samples, M3.PJ_mean.trace()[:])
PP_mean_samples =...


I am working on the file "1.2 Change and difference/1.2.4 Comparing word use between corpora.ipynb".
The first task is:
Now we calculate the expected counts from those probabilities. The expected counts can be found by multiplying the probabilities of the word occuring and not occuring by the total number of tokens in each corpus.
Can you write Python code for it?

# We multiply each 2-element array by a square matrix containing ones, and then
#  transpose one of the resulting matrices so that the product gives the expected
#  counts.
expected_counts = np.floor((np.ones((2, 2))*expected_probabilities)*\
                           (np.ones((2, 2))*contingency_table.sum(axis=1)).T).astype(int)

expected_counts


I am working on the file "1.2 Change and difference/1.2.4 Comparing word use between corpora.ipynb".
The first task is:
Now we obtain the log likelihood using the equation above:
Can you write Python code for it?

loglikelihood = np.sum(1.*contingency_table*np.log(1.*contingency_table/expected_counts))

loglikelihood


I am working on the file "1.2 Change and difference/1.2.4 Comparing word use between corpora.ipynb".
The first task is:
So, do the two corpora differ in terms of their use of the word "photosynthesis"? In other words, can we reject the null hypothesis (that they do not)? Per Dunning (1993), under the null hypothesis the distribution of the test statistic (log likelihood) should follow a $\chi^2$ distribution. So we can obtain the probability of the calculated log-likelihood under the null hypothesis using the PDF of $\chi^2$ with one degree of freedom.
The Scientific Python (SciPy) package has a whole bunch of useful distributions, including $\chi^2$.
Can you write Python code for it?

distribution = stats.chi2(df=1)    # df: degrees of freedom.


I am working on the file "1.2 Plotting - Timeseries.ipynb".
The first task is:
Exercise: Add a crosshair and hover tool to the plot
Can you write Python code for it?

from bokeh.models import ColumnDataSource, HoverTool
from collections import OrderedDict

# List all the tools that you want in your plot separated by comas, all in one string.

# Add the tools to your figure

# The hover tools doesn't render datetime appropriately. We'll need a string.

# To reference variables in the hover box, we'll need to use bokeh.ColumnDataSource instead of a pd.DataFrame

# Change plotting.line to get values from ColumnDataSource

# Set hover tool

# Copy your style from the previous exercise

# Show plot


I am working on the file "1.2-The Basics.ipynb".
The first task is:
Above, Python has inferred the type of the variable pizza to be an integer. Since strings can only be concatenated with other strings, our print statement generates an error. There are two ways we can resolve the error:

Declare the pizzas variable as type string (str) on assignment or
Re-cast the pizzas variable as a string within the print statement.
Can you write Python code for it?

customer = 'Carol'
pizzas = str(2)
print(customer + ' ordered ' + pizzas + ' pizzas.')

customer = 'Carol'
pizzas = 2
print(customer + ' ordered ' + str(pizzas) + ' pizzas.')


I am working on the file "1.2-The Basics.ipynb".
The first task is:
Given the following variable assignments:
x = 12
y = str(14)
z = donuts
Predict the output of the following:

y + z
x + y
x + int(y)
str(x) + y

Check your answers in the interpreter.
Variable Naming Rules
Variable names are case senstive and:

Can only consist of one "word" (no spaces).
Must begin with a letter or underscore character ('_').
Can only use letters, numbers, and the underscore character.

We further recommend using variable names that are meaningful within the context of the script and the research.
Reading Files
We can accomplish a lot by assigning variables within our code as demonstrated above, but often we are interested in working with objects and data that exist in other files and directories on our system.
When we want to read data files into a script, we do so by assigning the content of the file to a variable. This stores the data in memory and lets us perform processes and analyses on the data without changing the content of the source file.
There are several ways to read files in Python - many libraries have methods for reading text, Excel and Word documents, PDFs, etc. This morning we're going to demonstrate using the read() and readlines() method in the standard library, and the Pandasread_csv() function.
Can you write Python code for it?

# Read unstructured text

# One way is to open the whole file as a block
file_path = "./beowulf" # We can save the path to the file as a variable
file_in = open(file_path, "r") # Options are 'r', 'w', and 'a' (read, write, append)
beowulf_a = file_in.read()
file_in.close()
print(beowulf_a)

# Another way is to read the file as a list of individual lines

with open(file_path, "r") as b:
    beowulf_b = b.readlines()

print(beowulf_b)

# In order to get a similar printout to the first method, we use a for loop
# to print line by line - more on for loops below!

for l in beowulf_b:
    print(l)

# We now have two variables with the content of our 'beowulf' file represented using two different data structures.
# Why do you think we get the different outputs from the next two statements?

# Beowulf text stored as one large string
print("As string:", beowulf_a[0])

# Beowulf text stored as a list of lines
print("As list of lines:", beowulf_b[0])

# We can confirm our expectations by checking on the types of our two beowulf variables
print(type(beowulf_a))
print(type(beowulf_b))

# Read CSV files using the Pandas read_csv method.
# Note: Pandas also includes methods for reading Excel.

# First we need to import the pandas library
import pandas as pd

# Create a variable to hold the path to the file
fpath = "aaj1945_DataS1_Egg_shape_by_species_v2.csv"
egg_data = pd.read_csv(fpath)

# We can get all kinds of info about the dataset

# info() provides an overview of the structure
print(egg_data.info())

# Look at the first five rows
egg_data.head()

# Names of columns
print(egg_data.columns.values)

# Dimensions (number of rows and columns)
print(egg_data.shape)

# And much more! But as a final example we can perform operations on the data.
# Descriptive statistics on the "Number of eggs" column
print(egg_data["Number of eggs"].describe())

# Or all of the columns in whole table with numeric data types:
print(egg_data.describe())


I am working on the file "1.2-The Basics.ipynb".
The first task is:
Info on additional list methods is available at https://docs.python.org/3/library/stdtypes.html?highlight=lists#mutable-sequence-types
Because lists are ordered, it is possible to access list items by referencing their positions. Note that the position of the first item in a list is 0 (zero), not 1!
Can you write Python code for it?

string_list = ['apples', 'oranges', 'pears', 'grapes', 'pineapples']

string_list[0]

# We can use positions to 'slice' or select sections of a list:

string_list[3:]           # start at index '3' and continue to the end

string_list[:3]           # start at index '0' and go up to, but don't include index '3'

string_list[1:4]          # start at index '1' and go up to and don't include index '4'

# If we don't know the position of a list item, we can use the 'index()' method to find out.
# Note that in the case of duplicate list items, this only returns the position of the first one:

string_list.index('pears')

string_list.append('oranges')

string_list

string_list.index('oranges')

# one more time with lists and dictionaries
list_ex1 = my_stuff[0] + my_stuff[1] + int(my_stuff[2])
print(list_ex1)

# we can use parentheses to split a continuous group of commands over multiple lines
list_ex2 = (
    str(my_stuff[0])
    + str(my_stuff[1])
    + my_stuff[2]
    + my_stuff[3][0]
)
print(list_ex2)

dict_ex1 = (
    more_stuff['item1']
    + more_stuff['item2']
    + int(more_stuff['item3'])
)
print(dict_ex1)

dict_ex2 = (
    str(more_stuff['item1'])
    + str(more_stuff['item2'])
    + more_stuff['item3']
)
print(dict_ex2)

# Now try it yourself ...
# print out the phrase "The answer: 42" using the following
# variables and one or more of your own and the 'print()' function
# (remember spaces are characters as well)

start = "The"
answer = 42


I am working on the file "1.2-The Basics.ipynb".
The first task is:
Lists
https://docs.python.org/3/library/stdtypes.html?highlight=lists#list
Lists are a type of collection in Python. Lists allow us to store sequences of items that are typically but not always similar. All of the following lists are legal in Python:
Can you write Python code for it?

# Separate list items with commas!

number_list = [1, 2, 3, 4, 5]
string_list = ['apples', 'oranges', 'pears', 'grapes', 'pineapples']
combined_list = [1, 2, 'oranges', 3.14, 'peaches', 'grapes', 99.19876]

# Nested lists - lists of lists - are allowed.

list_of_lists = [[1, 2, 3],
                 ['oranges', 'grapes', 8],
                 [['small list'],
                  ['bigger', 'list', 55],
                  ['url_1', 'url_2']
                 ]
                ]


I am working on the file "1.2-The Basics.ipynb".
The first task is:
The ability to reassign variable values becomes important when iterating through groups of objects for batch processing or other purposes. In the example below, the value of b is dynamically updated every time the while loop is executed:
Can you write Python code for it?

a = 5
b = 10
while b > a:
    print("b="+str(b))
    b = b-1


I am working on the file "1.2-The Basics.ipynb".
The first task is:
There are cases when we may want to declare the data type, for example to assign a different data type from the default that will be inferred. Concatenating strings provides a good example.
Can you write Python code for it?

customer = 'Carol'
pizzas = 2
print(customer + ' ordered ' + pizzas + ' pizzas.')


I am working on the file "1.2-The Basics.ipynb".
The first task is:
There are multiple ways to create a list:
Can you write Python code for it?

# Create an empty list

empty_list = []

# As we did above, by using square brackets around a comma-separated sequence of items

new_list = [1, 2, 3]

# Using the type constructor

constructed_list = list('purple')

# Using a list comprehension

result_list = [i for i in range(1, 20)]


I am working on the file "1.2-The Basics.ipynb".
The first task is:
Try It Yourself
Go to the section 4.4. Numeric Types in the Python 3 documentation at https://docs.python.org/3.4/library/stdtypes.html. The table in that section describes different operators - try some!
What is the difference between the different division operators (/, //, and %)?
Variables
Variables allow us to store values for later use.
Can you write Python code for it?


I am working on the file "1.2-The Basics.ipynb".
The first task is:
Variable data types can be inferred, so Python does not require us to declare the data type of a variable on assignment.
Can you write Python code for it?


I am working on the file "1.2-The Basics.ipynb".
The first task is:
Variables can be reassigned:
Can you write Python code for it?


I am working on the file "1.2-The Basics.ipynb".
The first task is:
We can inspect our lists:
Can you write Python code for it?

empty_list

new_list

result_list

constructed_list


I am working on the file "1.2-The Basics.ipynb".
The first task is:
is equivalent to
Can you write Python code for it?

a = int(5)
type(a)

c = 'dogs'
print(type(c))

c = str('dogs')
print(type(c))


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
Let’s try it out:
Can you write Python code for it?

is_prime_naive(2**19 - 1), is_prime_naive(1000000)


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
Now let's use NumPy to calculate the eigenvalues and eigenvectors of the Hermitian matrix M:
Can you write Python code for it?


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
Prime numbers
Let’s now look at a function that checks whether a number is prime.
Can you write Python code for it?

def is_prime_naive(n):
    for i in range(2, n):       # Note: range(n, m) -> n, n+1, ..., m-1
        if n % i == 0:          # % is the modulo operation.
           return False
    return True


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
Since the above calculation is taking a very long time, let’s interrupt the computation by using the pull-down menu “Kernel”.
To speed up the routine, the first thing we can do is avoid to divide by even numbers:
Can you write Python code for it?

def is_prime_slow(n):
    if n % 2 == 0:
        return False
    for i in range(3, n, 2):
        if n % i == 0:
           return False
    return True


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
The above code is not in the spirit of NumPy, since one of NumPy's main goals is to avoid loops by vectorizing code.  Vectorization can bring numerical Python up to C speed by moving loops from slow Python to fast machine code.
The below cell re-creates the same matrix M as the above cell.  Note that i is no longer a scalar, but a vector of 99 integers.
Can you write Python code for it?

n = 100

i = np.arange(n - 1)
M = np.zeros((n, n))
M[i, i + 1] = M[i + 1, i] = -1

print(M)


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
The matplotlib package is the standard way to plot data in Python.  Let’s use it to plot the third eigenvector.  Note how we use NumPy’s “fancy indexing” to extract the the third (= index 2) column of the ‘vecs‘ matrix.
(The trailing semicolon has no meaning in Python.  It makes the notebook suppress the output of the preceding command.)
Can you write Python code for it?

%run matplotlib_setup.ipy
from matplotlib import pyplot

pyplot.plot(vecs[:, 2]);


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
To better understand the built-in range function, place the cursor on it in the above cell and press “shift + Tab” first once, and then a second time.  This works for most functions.
Your turn!
The above speedup is not significant.  To improve the runtime cost from $O(n)$ to $O(\sqrt{n})$, observe the following:

The square root of a number x is given by math.sqrt(x).
This function is part of the math package that needs to be imported with the import math statement, preferably before the function definition.
‘range‘ only accepts integer arguments.  To truncate the real number x towards zero, use int(x).

Copy the cell of is_prime_slow by using the “Edit” menu.  Paste it below this cell, and rename the function to is_prime.  Now use the above hints to make it significantly faster.
Check it with the huge prime number below.  It should evaluate instantly.
Can you write Python code for it?


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
Your turn!
As an exercise, write a loop that computes the product of all the eigenvalues of M.  Verify that it is equal to the determinant given by np.linalg.det.
Here is a faster way to calculate the product of the entries of a NumPy array:
Can you write Python code for it?


I am working on the file "1.2.python-refresher.ipynb".
The first task is:
Your turn!
The next cell is a copy of the previous one.  Modify it to print, say, the first 9 cube numbers.
Can you write Python code for it?


I am working on the file "1.3. Feature selection/1.3.1 Feature selection - Keywords.ipynb".
The first task is:
Feature selection: keywords
A major problem-area in text mining is determining the thematic or topical content of texts. One of the most basic problems in this area is to identify the terms in a text -- "keywords" -- that most accurately represents its distinctive thematic characteristics.
In this notebook, we will use Dunning's log-likelihood statistic to identify keywords for individual documents in a collection of texts. It is fairly typical that methods used for statistical analysis are also used for information extraction and classification.
We'll use the Embryo Project corpus from earlier notebooks. Recall that the plain text documents are stored separately from their metadata -- this is the format that you would expect from a Zotero RDF export.
Can you write Python code for it?

documents = nltk.corpus.PlaintextCorpusReader('../data/EmbryoProjectTexts/files', 'https.+')
metadata = zotero.read('../data/EmbryoProjectTexts', index_by='link', follow_links=False)


I am working on the file "1.3. Feature selection/1.3.1 Feature selection - Keywords.ipynb".
The first task is:
Our first step is to count up all of the words in each of the documents. This conditional frequency distribution should look familiar by now.
Can you write Python code for it?

wordcounts_per_document = nltk.ConditionalFreqDist([
    (fileid, normalize_token(token))
     for fileid in documents.fileids()
     for token in documents.words(fileids=[fileid])
     if filter_token(token)
])

from scipy import sparse

# We pick a single "focal" document that we want to characterize.
focal_fileid = documents.fileids()[3]

# Since this procedure will involve numerical matrices, we
#  need to map documents and words onto row and column indices.
#  These "dictionaries" will help us to keep track of those
#  mappings.
document_index = {}    # Maps int -> fileid (str).
vocabulary = {}        # Maps int -> word (str).
lookup = {}            # Reverse map for vocabulary (word (str) -> int).

# Containers for sparse data.
I = []          # Document vector.
J = []          # Word vector.
data = []       # Word count vector.
labels = []     # Vector of labels; either the URI of interest, or "Other".

# Here we transform the ConditionalFrequencyDist into three vectors (I, J, data)
#  that sparsely describe the document-word count matrix.
for i, (fileid, counts) in enumerate(wordcounts_per_document.iteritems()):
    document_index[i] = fileid
    for token, count in counts.iteritems():
        # Removing low-frequency terms is optional, but speeds things up
        #  quite a bit for this demonstration.
        if count < 3:
            continue

        # get() lets us
        j = lookup.get(token, len(vocabulary))
        vocabulary[j] = token
        lookup[token] = j

        I.append(i)
        J.append(j)
        data.append(count)
    labels.append(fileid if fileid == focal_fileid else 'Other')
    print '\r', i,

sparse_matrix = sparse.coo_matrix((data, (I, J)))

sparse_matrix.shape

from sklearn.feature_selection import chi2
from sklearn.feature_extraction.text import CountVectorizer

keyness, _ = chi2(sparse_matrix, labels)

ranking = np.argsort(keyness)[::-1]

_, words = zip(*sorted(vocabulary.items(), key=lambda i: i[0]))

words = np.array(words)

keywords =...


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
1.3.2 Features in texts: N-grams
In earlier notebooks, we treated individual tokens as separate and independent features in our texts. But words are rarely independent. First of all, they are often part of more complex phrases that refer to abstract concepts. In the context of computational humanities, it is usually those abstract concepts and not individual words that we want to interrogate. A major component of our analytic workflow, therefore, will be deciding what kinds of abstract features we are interested in, and developing strategies to extract those features.
Phrases
Multi-word phrases are an easy place to start. Our objective is to identify sequences of tokens that belong together as a phrase. For example, in the sentence "The quick brown fox jumps over the lazy dog", we want to be able to extract phrases like "brown fox" and "lazy dog". We do this by looking for N-grams whose joint probability are high in comparison to the individual probabilities of their constituent tokens.
An N-gram is a sequence of N adjacent tokens. A bigram is a sequence of 2 adjacent tokens. For example, here are the first 7 tokens in our corpus:
Can you write Python code for it?


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
Depending on what kind of phrases you are interested in, it may be desirable to filter N-grams based on the part of speach (POS) of their constituent tokens. The pos_tag() function generates a list of tokens and tags from our corpus. We can use those tagged tokens to build the collocate model, and then filter based on tags.

Note this step takes a bit of time to finish.
Can you write Python code for it?

tagged_words = nltk.pos_tag(documents.words())

tagged_words[:10]

finder = nltk.collocations.BigramCollocationFinder.from_words(tagged_words)
finder.apply_freq_filter(3)

# Only keep nouns (noun POS tags all start with ``N``).
word_filter = lambda token: not (filter_token(token[0]) and token[1].startswith('N'))
finder.apply_word_filter(word_filter)

finder.nbest(bigram_measures.pmi, 10)


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
Our task is to decide which of these N-grams are actually phrases.
Pointwise Mutual Information
Mutual Information is a measure of the mutual interdependence of two variables. Mutual information captures the extent to which knowing one variable reduces our uncertainty about another variable. Pointwise Mutual Information applies the concept of mutual information to specific events. PMI captures the likelihood that a specific event will occur given that another specific event was observed. To apply this to tokens in our corpus: the PMI of two tokens ($x$ and $y$) measures how likely it is that one specific token (say, $y$) will occur in a specific context (e.g. a bigram) given that another specific token (say, $x$) is observed in that context.
PMI is defined as:
$pmi(x; y) = log \frac{p(x, y)}{p(x)p(y)} = log \frac{p(x\mid y)}{p(x)} = log \frac{p(y\mid x)}{p(y)}$
The denominator in the equation above, $p(x)p(y)$, is the probability that $x$ and $y$ will occur together by chance alone given their independent distributions. The numerator, $p(x, y)$, is the probability of coincidence given their joint distribution.
It makes a little more sense if you think about those probabilities in terms of frequency (recall that the MLE for $p(x)$ in our corpus is $f(x)$). If two tokens occur more often together than they do on their own, then chances are those tokens are related to each other.
NLTK provides some tools to find colocations using PMI.
Can you write Python code for it?

bigram_measures = nltk.collocations.BigramAssocMeasures()
finder = nltk.collocations.BigramCollocationFinder.from_words(documents.words())

finder.nbest(bigram_measures.pmi, 10)


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
Rather than specifying a number of N-grams to accept as phrases, we can set a threshold PMI value. Let's first examine the distribution of scores. Here are the bottom 20 scores (using score_ngrams()).
Can you write Python code for it?


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
The MWETokenizer combines the two tokens in each bigram using an underscore _. So when we operate on the tokens below, we must take that into account.
---------    --------
Note:    In the code-cell below I have reversed the samples and the conditions -- this isn't strictly correct, but it makes it easy to quickly generate a plot of raw count data.
---------    --------
Can you write Python code for it?

words_over_time = nltk.ConditionalFreqDist([
    (normalize_token(token).replace('_', ' '), metadata[fileid].date)
    for fileid in documents.fileids()
    for token in tokenizer.tokenize(documents.words(fileids=[fileid]))
    if filter_token(token) or '_' in token
])

words_over_time.plot(conditions=['hans spemann', 'embryonic development',
                                 'ernst haeckel', 'water'])


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
To accept only N-grams with a PMI over 2, for example, we could do (selecting only the last 20):
Can you write Python code for it?

selected_ngrams = [ngram for ngram in finder.above_score(bigram_measures.pmi, 2.0)]
selected_ngrams[-20:]

selected_expressions = []
for (token1, pos1), (token2, pos2) in finder.above_score(bigram_measures.pmi, 2.0):
    selected_expressions.append((token1, token2))

selected_expressions[:10]

tokenizer = nltk.MWETokenizer(selected_expressions)


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
We end up with a lot of junk collocations. This is often due to very infrequent N-grams; consider a bigram with tokens that occur only once in the whole corpus: they will have a maximal PMI, since they don't occur independently. We can filter out infrequent N-grams using the apply_freq_filter() function.
Can you write Python code for it?


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
and here are the first 6 bigrams:
Can you write Python code for it?

[gram for gram in nltk.bigrams(documents.words())][:6]


I am working on the file "1.3. Feature selection/1.3.2 Features in Texts - N-grams.ipynb".
The first task is:
and the first 5 trigrams:
Can you write Python code for it?

[gram for gram in nltk.trigrams(documents.words())][:5]


I am working on the file "1.6 Layout.ipynb".
The first task is:
Exercise: Layout your plots using hplot and vplot
Can you write Python code for it?

from bokeh.plotting import vplot, hplot

# Create your layout

# Show layout


I am working on the file "1.6 Layout.ipynb".
The first task is:
Exercise: Store your layout in an html page
Can you write Python code for it?


I am working on the file "1.7. Latent Dirichlet Allocation/1.7. Finding concepts in texts - Latent Dirichlet Allocation.ipynb".
The first task is:
1.7. Finding concepts in texts - Latent Dirichlet Allocation
Latent Semantic Analysis provided a powerful way to begin interrogating relationships among texts.
In this notebook we use the gensim implementation of Online LDA (Hoffman et al 2010), which is an alternate of the typical Gibbs-Sampling MCMC approach.
Can you write Python code for it?

import nltk
from tethne.readers import zotero
import matplotlib.pyplot as plt
from nltk.corpus import stopwords
import gensim
import networkx as nx
import pandas as pd

from collections import defaultdict, Counter

wordnet = nltk.WordNetLemmatizer()
stemmer = nltk.SnowballStemmer('english')
stoplist = stopwords.words('english')

text_root = '../data/EmbryoProjectTexts/files'
zotero_export_path = '../data/EmbryoProjectTexts'

corpus = nltk.corpus.PlaintextCorpusReader(text_root, 'https.+')
metadata = zotero.read(zotero_export_path, index_by='link', follow_links=False)

def normalize_token(token):
    """
    Convert token to lowercase, and stem using the Porter algorithm.

    Parameters
    ----------
    token : str

    Returns
    -------
    token : str
    """
    return wordnet.lemmatize(token.lower())

def filter_token(token):
    """
    Evaluate whether or not to retain ``token``.

    Parameters
    ----------
    token : str

    Returns
    -------
    keep : bool
    """
    token = token.lower()
    return token not in stoplist and token.isalpha() and len(token) > 2


I am working on the file "1.7. Latent Dirichlet Allocation/1.7. Finding concepts in texts - Latent Dirichlet Allocation.ipynb".
The first task is:
Here we filter the tokens in each document, preserving the shape of the corpus.
Can you write Python code for it?

filtered_documents = [[token for token in document
                      if wordcounts[token] < 2000
                      and 1 < documentcounts[token] < 350]
                     for document in documents]


I am working on the file "1.7. Latent Dirichlet Allocation/1.7. Finding concepts in texts - Latent Dirichlet Allocation.ipynb".
The first task is:
It's easier to compute over integers, so we use a Dictionary to create a mapping between words and their integer/id representation.
Can you write Python code for it?


I am working on the file "1.7. Latent Dirichlet Allocation/1.7. Finding concepts in texts - Latent Dirichlet Allocation.ipynb".
The first task is:
We will represent our documents as a list of lists. Each sub-list contains tokens in the document.
Can you write Python code for it?

documents=[[normalize_token(token)
            for token in corpus.words(fileids=[fileid])
            if filter_token(token)]
           for fileid in corpus.fileids()]

years = [metadata[fileid].date for fileid in corpus.fileids()]


I am working on the file "1.7. Latent Dirichlet Allocation/1.7.2 Finding concepts in texts - Hierarchical Dirichlet Process.ipynb".
The first task is:
1.7.2 Finding concepts in texts - Hierarchical Dirichlet Process.
Can you write Python code for it?

import nltk
from tethne.readers import zotero
import matplotlib.pyplot as plt
from nltk.corpus import stopwords
import gensim
import networkx as nx
import pandas as pd

wordnet = nltk.WordNetLemmatizer()
stemmer = nltk.SnowballStemmer('english')
stoplist = stopwords.words('english')

text_root = '../data/EmbryoProjectTexts/files'
zotero_export_path = '../data/EmbryoProjectTexts'

corpus = nltk.corpus.PlaintextCorpusReader(text_root, 'https.+')
metadata = zotero.read(zotero_export_path, index_by='link', follow_links=False)

def normalize_token(token):
    """
    Convert token to lowercase, and stem using the Porter algorithm.

    Parameters
    ----------
    token : str

    Returns
    -------
    token : str
    """
    return wordnet.lemmatize(token.lower())

def filter_token(token):
    """
    Evaluate whether or not to retain ``token``.

    Parameters
    ----------
    token : str

    Returns
    -------
    keep : bool
    """
    token = token.lower()
    return token not in stoplist and token.isalpha() and len(token) > 2


I am working on the file "1.7. Latent Dirichlet Allocation/1.7.2 Finding concepts in texts - Hierarchical Dirichlet Process.ipynb".
The first task is:
Further filtering
LDA in Python is a bit computationally expensive, so anything we can do to cut down on "noise" will help. Let's take a look at wordcounts and documentcounts to see whether we can narrow in on more useful terms.
Can you write Python code for it?

wordcounts = nltk.FreqDist([token for document in documents for token in document])

wordcounts.plot(20)

documentcounts = nltk.FreqDist([token for document in documents for token in set(document)])

documentcounts.plot(80)


I am working on the file "1.7. Latent Dirichlet Allocation/1.7.2 Finding concepts in texts - Hierarchical Dirichlet Process.ipynb".
The first task is:
It's easier to compute over integers, so we use a Dictionary to create a mapping between words and their integer/id representation.
Can you write Python code for it?


I am working on the file "1.7. Latent Dirichlet Allocation/1.7.2 Finding concepts in texts - Hierarchical Dirichlet Process.ipynb".
The first task is:
The doc2bow() converts a document (series of tokens) into a bag-of-words representation.
Can you write Python code for it?

documents_bow = [dictionary.doc2bow(document) for document in filtered_documents]


I am working on the file "1.7. Latent Dirichlet Allocation/1.7.2 Finding concepts in texts - Hierarchical Dirichlet Process.ipynb".
The first task is:
We will represent our documents as a list of lists. Each sub-list contains tokens in the document.
Can you write Python code for it?


I am working on the file "1.7. Latent Dirichlet Allocation/1.7.2 Finding concepts in texts - Hierarchical Dirichlet Process.ipynb".
The first task is:
We're ready to fit the model! We pass our BOW-transformed documents, our dictionary, and the number of topics. update_every=0 disables an "online" feature in the sampler (used for very very large corpora), and passes=20 tells the sampler to pass over the whole corpus 20 times.
Can you write Python code for it?

hmodel = gensim.models.HdpModel(documents_bow, id2word=dictionary)

for t in hmodel.show_topics(topics=200, topn=5, formatted=True):
    print t

hdp_documents = hmodel[documents_bow]

hdp_documents[5]


I am working on the file "1.8. Skipgram Model/1.8. Skipgram.ipynb".
The first task is:
1.8 CBOW/Skip-gram model
Literature: here and here.
Can you write Python code for it?

import nltk
from tethne.readers import zotero
import matplotlib.pyplot as plt
from helpers import normalize_token, filter_token
import gensim
from itertools import groupby, islice

text_root = '../data/EmbryoProjectTexts/files'
zotero_export_path = '../data/EmbryoProjectTexts'

corpus = nltk.corpus.PlaintextCorpusReader(text_root, 'https.+')
metadata = zotero.read(zotero_export_path, index_by='link', follow_links=False)

years = []
documents = []
for fileid in corpus.fileids():
    for sentence in corpus.sents(fileids=[fileid]):
        years.append(metadata[fileid].date)
        documents.append([normalize_token(token)
                          for token in sentence
                          if filter_token(token)])

len(documents)

model = gensim.models.Word2Vec(documents, size=200, window=5)


I am working on the file "1.8. Skipgram Model/1.8. Skipgram.ipynb".
The first task is:
PCA on lower-dimensional projection
Can you write Python code for it?

from sklearn import decomposition

focal_words = ['egg', 'ontogeny', 'ovum', 'fetus', 'ivf', 'uterus', 'brain',
               'neuron', 'cell', 'transplantation', 'culture', 'stem', 'nucleus']

focal_indices = [model.index2word.index(w) for w in focal_words]

pca = decomposition.PCA(n_components=2)

t = pca.fit_transform(model.syn0[focal_indices, :])

plt.scatter(t[:, 0], t[:, 1])
for i in xrange(len(focal_indices)):
    w = focal_words[i]
    plt.text(t[i, 0], t[i, 1], w)


I am working on the file "1.8. Skipgram Model/1.8. Skipgram.ipynb".
The first task is:
Similary is measured using cosine distance in the low-dimensional embedding/vector space.
Can you write Python code for it?

model.most_similar(positive=['cell'])

model.similarity('egg', 'ovum'), model.similarity('egg', 'embryo'), model.similarity('egg', 'brain')

model.syn0.shape     # Words x Dimensions.

len(model.index2word)


I am working on the file "1.BasicPython.ipynb".
The first task is:
Also worth knowing are \n and \t : the newline and tab characters, respectively.
Can you write Python code for it?

# Written on-the-fly, because I got mad skills
print("This is a haiku\n\tI'm awful at poetry\nWait, this really worked")


I am working on the file "1.BasicPython.ipynb".
The first task is:
Can we access the last element ? What about the last two ?
Can you write Python code for it?


I am working on the file "1.BasicPython.ipynb".
The first task is:
Dictionaries
Let's take a very brief look at dictionaries. These are unordered containers that you can use to pair elements in, similar to a std::map if you're a C++ coder.
Can you write Python code for it?

pythonPoints = { "Quentin" : 1./3, "Paul" : 42, "Matthew" : 1e3 }
print(pythonPoints)

# Dictionaries associate keys with values
print(pythonPoints.keys())
print(pythonPoints.values())

# You can access them through their keys
print(pythonPoints["Paul"] * 2)

if "Ruthie" in pythonPoints :     # for dicts, "in" checks the keys
    print("Ruthie's here too !")

else :
    pythonPoints["Ruthie"] = 0

print("Ruthie has {} mad skillz.".format(pythonPoints["Ruthie"]))


I am working on the file "1.BasicPython.ipynb".
The first task is:
Earlier, we saw that range() can take two arguments : range(start, finish). It can actually take a third : range(start, finish, stride).
Can you write Python code for it?


I am working on the file "1.BasicPython.ipynb".
The first task is:
Here, %d is a format code for integer. %f is for floating point numbers ( floats ), and %s is for strings.
Note how, to pass more than one thing in, we had to put it into round brackets. This is a tuple, we mentioned it briefly in the last notebook. It's basically just an immutable list. String formatting like this takes tuples.
Can you write Python code for it?

# I modified this one from Learn Python The Hard Way ( highly recommended ) :
formatstring = "Start {} {}"
print(formatstring.format(formatstring, formatstring))


I am working on the file "1.BasicPython.ipynb".
The first task is:
Here, we used range() with two arguments. In Python 2, it generates a list from the first argument to the second argument minus 1. In Python 3, it returns an immutable iterable, but you can cast it to a list by calling something like list(range(5)). Also, note that we can feed the print function several things to print, separated by a comma.
Interacting Between Different Variable Types
Beware of integer division with Python 2. Unlike R, Python 2 doesn't assume that everything is a float unless explicitly told; it recognises that 2 is an integer, and this can be good and bad. In Python 3, we don't need to worry about this; the following code was run under a Python 3 kernel, but test it under Python 2 to see the difference.
Can you write Python code for it?

myint = 2
myfloat = 3.14
print(type(myint), type(myfloat))

# Multiplying an int with a float gives a float : the int was promoted.
print(myint * myfloat)
print(type(myint * myfloat))

# A minor difference between Python 2 and Python 3 :
print(7 / 3)
# Py2 : 2
# Py3 : 2.3333

# In Python 2, operations between same type gives the same type :
print(type(7 / 3))
# Py2 : <type 'int'>
# Py3 : <class 'float'>

# Quick hack with ints to floats - there's no need to typecast, just give it a float
print(float(7) / 3)
print(7 / 3.0)

# In Python 3, this is handled "correctly"; you can use // as integer division
print(7 // 3)

# Quick note for Py2 users - see https://www.python.org/dev/peps/pep-0238/
from __future__ import division
print(7 / 3)


I am working on the file "1.BasicPython.ipynb".
The first task is:
If you wanted to loop by indexing the list, we can use range(), which, in its simplest ( single-argument ) form, returns a list from 0 to that element minus 1.
Can you write Python code for it?

for i in range(len(mylist)) :
    print(i, mylist[i])


I am working on the file "1.BasicPython.ipynb".
The first task is:
List arithmetic ?
Can you write Python code for it?


I am working on the file "1.BasicPython.ipynb".
The first task is:
Lists have several methods ( count, sort, reverse, pop, insert, remove, ... ). Here are a few.
Can you write Python code for it?

print("Length is {} long.\n".format(len(mylist)))

print("There are {} ones in this list.\n".format(mylist.count(1)))

mylist.reverse()
print("Reversed ! {}".format(mylist))


I am working on the file "1.BasicPython.ipynb".
The first task is:
Looks good. We've just defined a function that takes one argument, n, and returns something based on what n is. The Fibonacci function is quite particular because it calls itself ( recursion ), but it's a small, fun example, so why not.
Can you write Python code for it?

def printFib(i) :
    print("The {}th number of the Fibonnaci sequence is {}.".format(i, fib(i)))

printFib(20)


I am working on the file "1.BasicPython.ipynb".
The first task is:
Python accesses elements in lists from 0, not from 1 as in Matlab or R. This will be familiar to C and Java users.
.append is a method of the list class - it's a function that belongs to the list object, and so you can call it directly from the object itself. This particular function appends something to the end of the list.
Can you write Python code for it?

print(mylist, "\n")

print(mylist[0])
print(mylist[1])
print(mylist[2])


I am working on the file "1.BasicPython.ipynb".
The first task is:
Strings are delimited by ", but can also use '. This is useful because you can now use one set of quotes inside another, and it'll still be one big string.
Can you write Python code for it?

s = "This is a string."
print(s)
print(type(s))
print(len(s))

s = 42
print(s)
print(type(s))


I am working on the file "1.BasicPython.ipynb".
The first task is:
Strings can be concatenated using the + operator. The len() function returns the length of a string.
Lists
Can you write Python code for it?

mylist = [1, 2.41341]
mylist.append("We can mix types !")

print(mylist)
print(type(mylist))


I am working on the file "1.BasicPython.ipynb".
The first task is:
The keyword in can also be used to check whether something is in a container :
Can you write Python code for it?


I am working on the file "1.BasicPython.ipynb".
The first task is:
There are a couple of other built-in containers, like tuples and sets. I won't go into them here, plainly because I have to use them so rarely that it's not worth the time during the session. If you want to read up : http://docs.python.org/2/tutorial/datastructures.html
List Comprehension and Inlines
Can you write Python code for it?

# Let's build a list of elements 1^2, 2^2, ..., 5^2
y = [i**2 for i in range(6)]

print(y)

# Want to keep your index ? Use a dictionary.
squares = { x : x**2 for x in range(6) }

for key, val in squares.items() :
    print("{} squared is {}".format(key, val))

# Also useful : zip()
# for key, val in zip(squares.keys(), squares.values()) :
#     print("{} : {}".format(key, val))

# We can inline if statements too
print(42 if type(42) is int else 32)

# Note this is interpreted as
# print (something if a, else print something_else)
# and not
# (print something) if a, else (do something_else)


I am working on the file "1.BasicPython.ipynb".
The first task is:
This statement is equivalent to saying s = s + 2**3, it's just shorthand. Also works with
-=, *=, /=, **=
Can you write Python code for it?

print(s == 42)
print(s == 50)
print(s > 10)


I am working on the file "1.BasicPython.ipynb".
The first task is:
What about if statements ?
Can you write Python code for it?

x = 5

if x > 3 :
    print("x is greater than 3.")

elif x == 5 :
    print("We aren't going to see this. Why ?")

else :
    print("x is not greater than 3.")

print("We can see this, it's not in the if statement.")


I am working on the file "1.BasicPython.ipynb".
The first task is:
What are the middle three elements ? Let's use the : operator. Like range(), it creates a list of integers.
[1:4] will give us elements 1, 2, and 3, because we stop at n-1, like with range().
Can you write Python code for it?


I am working on the file "1.BasicPython.ipynb".
The first task is:
What if I don't want to explicitly remember the size of the list ?
Can you write Python code for it?

# This will simply go from start to finish with a stride of 2
print(A[::2])

# And this one, from the second element to finish, with a stride of 2
print(A[1::2])

# So, uh... Reverse ?
print(A[::-1])


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.1 Binary_Variables.ipynb".
The first task is:
1. 伯努利分布
考虑一个二元随机变量$x\in{0,1}$，$x=1$的概率被记作参数$\mu$，因此$p(x=1|\mu)=\mu$，$p(x=0|\mu)=1-\mu$，其中$\mu\leq1$。
x的概率分布可以写成$Bern(x|\mu)=\mu^x(1-\mu)^{1-x}$，称为伯努利分布（$Bernoulli$ $distribution$）。
伯努利分布的均值和方差分别为$\mathbb{E}[x]=\mu$,$var[x]=\mu(1-\mu)$。
假设有x的观测值数据集$\mathbb{D}={x_1,...,x_N}$。假设每次观测都是独立地从$p(x|\mu)$中抽取的，因此关于$\mu$的似然函数是$p(\mathbb{D}|\mu)=\prod\limits_{n=1}^N p(x_n|\mu)=\prod\limits_{n=1}^N \mu^{x_n}(1-\mu)^{1-x_n}$。
其最大似然的估计值是$\mu_{ML}=\frac{1}{N}\sum\limits_{n=1}^N x_n=\frac{m}{N}$，如果把数据集里$x=1$的观测的数量记为m。
因此，在最大似然的框架中，x=1的概率是数据集中x=1的观测所占的比例。假设扔一枚硬币5次，碰巧5次硬币都正面朝上，那么$\mu_{ML}=1$。最大似然的结果会预测所有未来的观测值都是正面向上。这是最大似然中过拟合现象的一个极端例子。我们通过引入$\mu$的先验分布，来得到更加合理的解释。
Can you write Python code for it?


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.1 Binary_Variables.ipynb".
The first task is:
3. Beta分布
伯努利分布的似然函数是某个因子与$\mu^x(1-\mu)^{1-x}$的乘积的形式，如果我们选择一个正比于$\mu$和$(1-\mu)$的幂指数的先验概率分布，那么后验概率分布（正比于先验和似然函数的乘积）就会有着与先验分布相同的函数形式。这种性质叫做共轭性（conjugacy）。
Beta分布定义如下：
$$Beta(\mu|a, b) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)}\mu^{a-1}(1-\mu)^{b-1}$$
$\Gamma(x)$是Gamma函数，保证Beta分布归一化。
Beta分布的均值和方差为$$\mathbb{E}[\mu]=\frac{a}{a+b}，var[\mu]=\frac{ab}{(a+b)^2(a+b+1)}$$
参数a和b被称为超参数（hyperparameter），因为它们控制了参数$\mu$的概率分布。
Can you write Python code for it?

from scipy.stats import beta

fig = plt.figure(figsize=(12,8))
ax = fig.add_subplot(221)
x = np.linspace(0, 1, 100)
pbeta = beta.pdf(x, 0.1, 0.1)
ax.plot(x, pbeta, label='a=0.1\nb=0.1',lw=2)
plt.ylim(0,3)
plt.xlabel('$\mu$')
plt.grid(True)
plt.legend(loc="best")

ax = fig.add_subplot(222)
x = np.linspace(0, 1, 100)
pbeta = beta.pdf(x, 1, 1)
ax.plot(x, pbeta, 'g', label='a=1\nb=1',lw=2)
plt.xlabel('$\mu$')
plt.ylim(0,3)
plt.grid(True)
plt.legend(loc="best")

ax = fig.add_subplot(223)
x = np.linspace(0, 1, 100)
pbeta = beta.pdf(x, 2, 3)
ax.plot(x, pbeta, 'r', label='a=2\nb=3',lw=2)
plt.xlabel('$\mu$')
plt.ylim(0,3)
plt.grid(True)
plt.legend(loc="best")

ax = fig.add_subplot(224)
x = np.linspace(0, 1, 100)
pbeta = beta.pdf(x, 8, 4)
ax.plot(x, pbeta, 'm', label='a=8\nb=4',lw=2)
plt.xlabel('$\mu$')
plt.ylim(0,3)
plt.grid(True)
plt.legend(loc="best")

plt.show()


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.1 Binary_Variables.ipynb".
The first task is:
4. 后验概率
将Beta先验和二项似然函数相乘，得到后验概率分布的形式为：
$$p(\mu|m,l,a,b)\propto\mu^{m+a-1}(1-\mu)^{l+b-1}$$
其中$l=N-m$对应硬币反面朝上的样本数量。
具体得到归一化系数，公式如下：
$$p(\mu|m,l,a,b)=\frac{\Gamma(m+a+l+b)}{\Gamma(m+a)\Gamma(l+b)}\mu^{m+a-1}(1-\mu)^{l+b-1}$$
我们可以发现，如果数据集里有m次观测为x=1，有l次观测为x=0，那么从先验概率到后验概率，a的值变大了m，b的值变大了l。
Can you write Python code for it?

from scipy.misc import comb

pbinom = lambda m, N, p: comb(N, m) * p**m * (1-p)**(N-m)


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.1 Binary_Variables.ipynb".
The first task is:
伯努利分布直方图，u=0.6
Can you write Python code for it?

u = 0.6
x = np.arange(2)
bern = bernoulli.pmf(x, u)

plt.bar(x, bern, facecolor='green', alpha=0.5)
plt.xlabel('x')
plt.xlim(-0.2, 2)
plt.title('Histogram of the nernoulli distribution')

plt.show()


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.1 Binary_Variables.ipynb".
The first task is:
关于m的函数的二项分布直方图，其中N=10且u=0.25
Can you write Python code for it?

plt.bar(m, binData, facecolor='green', alpha=0.5)
plt.xlabel('m')
plt.xlim(-0.2, 10.5)
plt.title('Histogram of the binomial distribution')

plt.show()


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.2 Multinomial_Variables.ipynb".
The first task is:
${\alpha_k}=0.1$的情形：
Can you write Python code for it?


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.2 Multinomial_Variables.ipynb".
The first task is:
${\alpha_k}=1$的情形：
Can you write Python code for it?


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.3 The_Gaussian_Distribution.ipynb".
The first task is:
2. 高斯分布的几何形式
高斯对于x的依赖体现在二次型$\Delta^2=(\textbf{x}-\textbf{$\mu$})^T\Sigma^{-1}(\textbf{x}-\textbf{$\mu$})$上。$\Delta$被称为$\textbf{$\mu$}$和$\textbf{x}$之间的马氏距离(Mahalanobis distance)。当$\Sigma$是单位矩阵时，就变成了欧式距离。对于x空间中这个二次型事常数的曲面，高斯分布也是常数。
现在考虑协方差矩阵的特征向量方程$$\Sigma\textbf{$\mu$}_i=\lambda_i\textbf{$\mu$}_i$$
其中$i=1,...,D$。
由于$\Sigma$是实对称矩阵，因此它的特征值也是实数，并且特征向量可以被选成是单位正交的。
协方差矩阵可以表示成特征向量的展开形式$$\Sigma=\sum_\limits{i=1}^D\lambda_i\textbf{u}i\textbf{u}_i^T$$
协方差矩阵的逆矩阵可以表示为$$\Sigma^{-1}=\sum\limits{i=1}^D\frac{1}{\lambda_i}\textbf{u}_i\textbf{u}_i^T$$
于是二次型就变成了$$\Delta^2=\sum_\limits{i=1}^D\frac{y_i^2}{\lambda_i}$$
其中定义$y_i=\textbf{u}_i^T(\textbf{x}-\textbf{$\mu$})$。
我们把${y_i}$表示成单位正交向量$\textbf{u}_i$关于原始的$x_i$坐标经过平移和旋转后形成的新的坐标系。
定义$\textbf{y}=(y_1,...,y_D)^T$，我们有$$\textbf{y}=\textbf{U}(\textbf{x}-\textbf{$\mu$})$$
其中$\textbf{U}$是一个矩阵，它的行是向量$\textbf{u}_i^T$。
如果所有的特征值$\lambda_i$都是正数，那么这些曲面表示椭球面，椭球中心位于$\textbf{$\mu$}$，椭球的轴的方向沿着$\textbf{u}_i$，沿着轴向的缩放因子为$\lambda_i^{\frac{1}{2}}$。如下图所示：

高斯分布的局限
高斯分布的局限主要体现在其自由参数的数量和单峰分布上。
对于一般的协方差矩阵，其参数的总数随着维度D的增长呈平方的方式增长，为了简化参数，可以将协方差矩阵约束成对角矩阵或者各向同性协方差矩阵（正比于单位矩阵），这样虽然限制了概率分布的自由度的数量，并且很容易求协方差矩阵的逆矩阵，但却大大限制了概率密度的形式，限制了描述模型中相关性的能力。
高斯分布本质上是单峰的（只有一个最大值），而不能很好近似多峰分布。后面，我们会引入潜在变量来解决这一问题。通过引入离散型潜在变量，相当多的多峰分布可以使用混合高斯分布来描述；通过引入连续型潜在变量可以产生出一种模型，该模型的自由参数可以被控制成与数据空间的维度D无关，同时仍然允许模型描述数据集里主要的相关性关系。
不同形式的协方差矩阵对应的概率密度曲线
Can you write Python code for it?

import matplotlib.mlab as mlab
from mpl_toolkits.mplot3d import Axes3D

def plot_2d_normal(mux, muy, sigmaxx, sigmayy, sigmaxy):
    fig = plt.figure(figsize=(8,6))
    ax = fig.add_subplot()
    x = np.arange(0, 5, 0.1)
    y = np.arange(0, 5, 0.1)
    x, y = np.meshgrid(x, y)
    z = mlab.bivariate_normal(x, y, sigmaxx, sigmayy, mux, muy, sigmaxy)
    ret = plt.contourf(x, y, z, cmap=plt.get_cmap('coolwarm'))
    fig.colorbar(ret, shrink=0.5, aspect=5)
    plt.show()


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.3 The_Gaussian_Distribution.ipynb".
The first task is:
下面协方差矩阵是对角矩阵的情况，椭圆的轮廓线与坐标轴对齐。概率密度模型中有总数为2D个独立参数。
Can you write Python code for it?


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.3 The_Gaussian_Distribution.ipynb".
The first task is:
下面协方差矩阵正比于单位矩阵，该协方差矩阵又被称为各向同性协方差矩阵，轮廓线是同心圆。这使得模型有D+1个独立的参数。
Can you write Python code for it?


I am working on the file "1.PROBABILITY_DISTRIBUTIONS/1.3 The_Gaussian_Distribution.ipynb".
The first task is:
下面是一般形式的协方差矩阵对应的密度轮廓线，协方差矩阵有D(D+1)/2个独立参数，参数的总数随着D以平方的方式增长，大矩阵计算和求逆困难。
Can you write Python code for it?


I am working on the file "1.Preprocessing.ipynb".
The first task is:
Data Mining
First thing to do is to find the structure of the file. We can find a description in the "dblp.dtd" file and on the website:

The children of the root represent the individual data records. There are two types of records: publication records and person records.
The xml files contains: article, inproceedings, proceedings, book, incollection, phdthesis, mastersthesis, www, person, data
A publication record can be:
article: An article from a journal or magazine.
inproceedings: A paper in a conference or workshop proceedings.
proceedings: The proceedings volume of a conference or workshop.
book: An authored monograph or an edited collection of articles.
incollection: A part or chapter in a monograph.
phdthesis: A PhD thesis.
mastersthesis: A Master's thesis.

Most of the publication contain author/editor and title fields, that the ones we are interested in

There are field used to represent the author
<www key="homepages/r/CJvanRijsbergen">
<author>C. J. van Rijsbergen</author>
<author>Cornelis Joost van Rijsbergen</author>
<author>Keith van Rijsbergen</author>
<title>Home Page</title>
<url>http://www.dcs.gla.ac.uk/~keith/</url>
</www>
Those field are called person record. An author can be used called by different name (John Doe, J.Doe, Doe) so those records group together the different representation of an author. We need to collect and build a mapping to map a name to the author id (key of the www field). Thanks to the website, we know that :
Person records always have the key-prefix "homepages/",
the record level tag is always "www", and they always
contain a title element with the text content "Home Page"
Also all the name are unique. If there are 2 homonyms, an id is appened to the name
We collect titles and authors of each publication (if 2 authors participate at the same publication, it gives one publication to each authors).
In the meantime we collect the person description (all name and id). At the end we will be...
Can you write Python code for it?

from importlib import reload
import xml_parser
reload(xml_parser)
from xml_parser import Xml_parser

#Xml_parser = Xml_parser().collect_data("../pmi_data")


I am working on the file "1.Preprocessing.ipynb".
The first task is:
Observation of the data
Load the author-> publication dataset and plot the distribution of publications
Can you write Python code for it?

authorID_to_titles = utils.load_pickle("../pmi_data/authorID_to_publications.p")

authorID_to_count = {k:len(v['titles']) for k,v in tqdm(authorID_to_titles.items())}

fig = plt.figure()
ax = fig.add_subplot(111)
data = list(authorID_to_count.values())
binwidth = int((max(data)-min(data))/20)
ax.hist(data,
       bins=range(min(data), max(data) + binwidth, binwidth))
plt.show()
print("Max {0}, min {1}".format(max(data), min(data)))


I am working on the file "1.Preprocessing.ipynb".
The first task is:
We can observe that most of the authors have between 1 and 50 publications. A few author have around 50 and 200 publications which seems reasanable. Howerwer, the maximum is mode than 1000 which is to much. Let's investigate the outliers
Here we can found some very important informations about names :
- In the case of homonymous names, dblp aims to use the name version without any suffix (number after the name) as a person disambiguation page, and not as the bibliography of an actual person. Any publication listed on such a page has not been assigned to an actual author (i.e., someone with a homonym suffix number) yet.
- Since in the early days of dblp author name versions without a homonym suffix number had been used as normal bibliography pages, too, there can still be found examples of homonymous names where the entity without any suffix is still considered to model the bibliography of an actual person. Removing the last remnants of these examples by giving explicit suffix numbers is currently a work in progress at dblp.
- Currently, you could use the following rule of thumb to distinguish most cases adequately::
    - If a person page has a homonymous name variant using the suffix "0001", and the page carries no additional person information like a home page link or an affiliation note => Drop it (disambiguation page)
    - Otherwise :
        - if the earliest homonym name starts with suffix "0002"
        - there are additional pieces of author information given)
        - The number of publication is between 0 and 50
    =>then the page is likely to be intended to be the profile of an actual person

After investigation the notation of the data is quite messy. They are not consistent with the notification of the id: homepages/c/LeiChen0018, homepages/c/LeiChen-17, homepages/c/LeiChen18
So let's just try to apply the rule: if infos about the author ("note" or "url") keep it
Even by applying those rule it doesn't seems to remove all the disambiguation pages.
To keep it simple we will use a...
Can you write Python code for it?

def get_author_with_more_than(data, max_):
    more_than = [k for k, v in data.items() if v >max_]
    print("Authors with more than {0}: {1} ({2}%)".format(max_, len(more_than), round(len(more_than)/len(data)*100,4)))
get_author_with_more_than(authorID_to_count, 1010)
get_author_with_more_than(authorID_to_count, 500)
get_author_with_more_than(authorID_to_count, 300)
get_author_with_more_than(authorID_to_count, 200)
get_author_with_more_than(authorID_to_count, 190)
get_author_with_more_than(authorID_to_count, 50)


I am working on the file "1.Preprocessing.ipynb".
The first task is:
We will use 190 as a threshold. We won't remove to much authors (0.1%) and above 190 publications it seems too much
Can you write Python code for it?

authors_to_titles_clean= {author:v['titles'] for author, v in tqdm(authorID_to_titles.items()) if len(v['titles'])<=190}

authorID_to_count = {k:len(titles) for k,titles in tqdm(authors_to_titles_clean.items())}
fig = plt.figure()
ax = fig.add_subplot(111)
data = list(authorID_to_count.values())
binwidth = int((max(data)-min(data))/20)
ax.hist(data,
       bins=range(min(data), max(data) + binwidth, binwidth))
plt.show()
print("Max {0}, min {1}".format(max(data), min(data)))

utils.pickle_data(authors_to_titles_clean, "../pmi_data/authorID_to_publications_clean.p")


I am working on the file "1.how_to_retrieve_discretization_attributes_using_PredicSis.ai_python_SDK/How to retrieve classes, using PredicSis.ai python SDK.ipynb".
The first task is:
Discretization attributes are only available on contributive feature.
If you try to retrieve discretization attributes on non contributive feature, then you raise an AssertionError:
Can you write Python code for it?


I am working on the file "1.how_to_retrieve_discretization_attributes_using_PredicSis.ai_python_SDK/How to retrieve classes, using PredicSis.ai python SDK.ipynb".
The first task is:
Retrieve the predictive model
Can you write Python code for it?

pj = PredicSis.project('Outbound Mail Campaign')
model = pj.schema('Outbound Mail Campaign 2017-03-17 15:02:22')


I am working on the file "1/1.ipynb".
The first task is:
$W1$ has 1600 x 500 parameters, $b1$ has 500 parameters, $W2$ has 500 x 10 parameters, and $b2$ has 10 parameters. All in all, this makes 805510 parameters.
Can you write Python code for it?

# Note: These are only the first 10 weight vectors, the layer 500 all in all.
fig, axes = plt.subplots(2, 5, figsize=(15, 6))
for i, ax in enumerate(axes.flatten()):
    ax.imshow(W1.eval().T[i].reshape(40, 40), cmap='Greys', interpolation='none')


I am working on the file "1/1.ipynb".
The first task is:
2. Data
Can you write Python code for it?

fig, axes = plt.subplots(2, 5, figsize=(15, 6))
for i, ax in enumerate(axes.flatten()):
    ax.imshow(mnist.train.images[i].reshape(28, 28), cmap='Greys', interpolation='none')

fig, axes = plt.subplots(2, 5, figsize=(15, 6))
for i, ax in enumerate(axes.flatten()):
    ax.imshow(W.eval().T[i].reshape(28, 28), cmap='Greys', interpolation='none')


I am working on the file "1/1.ipynb".
The first task is:
3. ConvNet
Can you write Python code for it?

def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                        strides=[1, 2, 2, 1], padding='SAME')

x = tf.placeholder(tf.float32, shape=[None, 1600])
x_image = tf.reshape(x, [-1,40,40,1])
y_ = tf.placeholder(tf.float32, shape=[None, 10])

W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])

h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
h_pool1 = max_pool_2x2(h_conv1)

W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])

h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
h_pool2 = max_pool_2x2(h_conv2)

W_fc1 = weight_variable([10 * 10 * 64, 1024])
b_fc1 = bias_variable([1024])

h_pool2_flat = tf.reshape(h_pool2, [-1, 10*10*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)

keep_prob = tf.placeholder(tf.float32)
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])

y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

for i in range(20000):
    batch = translated_mnist.train.next_batch(50)
    if i%100 == 0:
        train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_: batch[1], keep_prob: 1.0})
        print("step %d, training accuracy %g"%(i, train_accuracy))
    train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})

print("test accuracy %g"%accuracy.eval(feed_dict={
    x: translated_mnist.test.images, y_: translated_mnist.test.labels, keep_prob: 1.0}))


I am working on the file "1/1.ipynb".
The first task is:
4. Filters
Can you write Python code for it?

W_conv1.eval().T[0, 0].shape

fig, axes = plt.subplots(4, 8, figsize=(15, 8))
for i, ax in enumerate(axes.flatten()):
    plt.sca(ax)
    plt.imshow(W_conv1.eval().T[i, 0], cmap='Greys', interpolation='none')
    plt.axis('off')


I am working on the file "1/1.ipynb".
The first task is:
Part 1: Simple MNIST
1. MNIST Tutorial
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

x = tf.placeholder(tf.float32, [None, 784])

W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))

y = tf.nn.softmax(tf.matmul(x, W) + b)

y_ = tf.placeholder(tf.float32, [None, 10])

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

sess = tf.InteractiveSession()
tf.global_variables_initializer().run()

for _ in range(1000):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels}))


I am working on the file "1/1.ipynb".
The first task is:
The weight vectors do not as clearly resemble digits as in part 1. This is because the digits are spread across the images and not centered at the same location. Therefore, the weight vectors cannot spezialize on individual digits (this is also why the accuracy is way lower).
2. Hidden layer
Can you write Python code for it?

def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)

x = tf.placeholder(tf.float32, [None, 1600])

W1 = weight_variable([1600, 500])
b1 = bias_variable([500])
y1 = tf.nn.relu(tf.matmul(x, W1) + b1)

W2 = weight_variable([500, 10])
b2 = bias_variable([10])
y = tf.nn.softmax(tf.matmul(y1, W2) + b2)

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

sess = tf.InteractiveSession()
tf.global_variables_initializer().run()

for _ in range(10000):
    batch_xs, batch_ys = translated_mnist.train.next_batch(50)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})

correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

print(sess.run(accuracy, feed_dict={x: translated_mnist.test.images, y_: translated_mnist.test.labels}))


I am working on the file "1/2 - KNN.ipynb".
The first task is:
The dataset consists of inputs in digits.images, and outputs in digits.target
Can you write Python code for it?

plt.gray()
plt.matshow(digits.images[50])
plt.show()

print(digits.images[50])
print(digits.target[50])


I am working on the file "1/2 - KNN.ipynb".
The first task is:
as well as kNN classification:
Can you write Python code for it?

X = [[0], [1], [2], [3]]
y = [0, 0, 1, 1]
from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=3)
neigh.fit(X, y)
print(neigh.predict([[1.8]]))


I am working on the file "1/CaseStudy1.ipynb".
The first task is:
*------------------------
Problem 4: Explore the data
Run some additional experiments with your data to gain familiarity with the twitter data ant twitter API
The following code was used to collect all Twitter followers of the Patriots, Broncos, and Panthers. Once collected, the followers were saved to files.
Can you write Python code for it?

# ## PATRIOTS
# patriots_friends_ids, patriots_followers_ids = get_friends_followers_ids(twitter_api, screen_name = 'Patriots')
# save_json('json/Patriots_Followers',patriots_followers_ids)
# save_json('json/Patriots_Friends', patriots_friends_ids)

# ## BRONCOS
# broncos_friends_ids, broncos_followers_ids = get_friends_followers_ids(twitter_api, screen_name = 'Broncos')
# save_json('json/Broncos_Followers',broncos_followers_ids)
# save_json('json/Broncos_Friends', broncos_friends_ids)

# ## PANTHERS
# panthers_friends_ids, panthers_followers_ids = get_friends_followers_ids(twitter_api, screen_name = 'Panthers')
# save_json('json/Panthers_Followers',panthers_followers_ids)
# save_json('json/Panthers_Friends', panthers_friends_ids)


I am working on the file "1/CaseStudy1.ipynb".
The first task is:
2. Find the most popular tweets in your collection of tweets
Please plot a table of the top 10 tweets that are the most popular among your collection, i.e., the tweets with the largest number of retweet counts.
Can you write Python code for it?

from collections import Counter
from prettytable import PrettyTable

# Create a list of all tweets with a retweeted_status key, and index the originator of that tweet and the text.
retweets = [
            (tweet['retweet_count'],
            tweet['retweeted_status']['user']['screen_name'],
            tweet['text'])

            #Ensure that a retweet exists
            for tweet in results
                if tweet.has_key('retweeted_status')
            ]

pretty_table = PrettyTable(field_names = ['Count','Screen Name','Text'])

# Sort tweets by descending number of retweets and display the top 10 results in a table.
[pretty_table.add_row(row) for row in sorted(retweets, reverse = True)[:10]]
pretty_table.max_width['Text'] = 50
pretty_table.align = 'l'
print pretty_table


I am working on the file "1/CaseStudy1.ipynb".
The first task is:
Another measure of tweet "popularity" could be the number of times it is favorited.  The following calculates the top-10 tweets with the most "favorites"
Can you write Python code for it?

from prettytable import PrettyTable

# Determine the number of "favorites" for each tweet collected.

favorites = [
            (tweet['favorite_count'],
             tweet['text'])
            for tweet in results
            ]

pretty_table = PrettyTable(field_names = ['Count','Text'])

# Sort tweets by descending number of favorites and display the top 10 results in a table.
[pretty_table.add_row(row) for row in sorted(favorites, reverse = True)[:10]]
pretty_table.max_width['Text'] = 75
pretty_table.align = 'l'
print pretty_table


I am working on the file "1/CaseStudy1.ipynb".
The first task is:
Compute the mutual friends within the two groups, i.e., the users who are in both friend list and follower list, plot their ID numbers and screen names in a table
Can you write Python code for it?

# Given a list of friends_ids and followers_ids, this function use set intersection to find the number of mutual friends.
# It then plots a table of the first (max_ids) listed mutual friends.

def display_mutual_friends(screen_name, friends_ids, followers_ids ,max_ids = None):
    friends_ids_set, followers_ids_set = set(friends_ids),set(followers_ids)

    print
    print '{0} has {1} mutual friends.  Here are {2}:'.format(screen_name, len(friends_ids_set.intersection(followers_ids_set)),max_ids)
    print
    mutual_friends_ids = list(friends_ids_set.intersection(followers_ids_set))
    table_ids_screen_names(twitter_api, user_ids = mutual_friends_ids, max_ids = max_ids)

display_mutual_friends(screen_name = screen_name, friends_ids = gronk_friends_ids, followers_ids = gronk_followers_ids, max_ids = 20)


I am working on the file "1/CaseStudy1.ipynb".
The first task is:
Next we wanted to estimate popularity of the Broncos and Panthers (the two remaining Super Bowl teams) in the Boston area.  Our chosen metric of "popularity" is the speed at which tweets are generated.  The following periodically collects tweets (constrained to the Boston geo zone) filtered for "Broncos" and "Panthers."  It tracks the number of such tweets collected in each time window, allowing us to estimate a Tweets per Minute ratio.
Can you write Python code for it?

# COLLECT TWEETS FROM STREAM WITH BRONCOS AND PANTHERS IN TWEET TEXT FROM BOSTON GEO ZONE

# from datetime import timedelta, datetime
# from time import sleep
# from twitter import TwitterStream

# track = "Broncos, Panthers"  # Tweets for Broncos OR Panthers
# locations = '-73.313057,41.236511,-68.826305,44.933163'  # New England / Boston geo zone

# NUMBER_OF_COLLECTIONS = 5 # number of times to collect tweets from stream
# COLLECTION_TIME = 2.5  # length of each collection in minutes
# WAIT_TIME = 10  # sleep time in between collections in minutes

# date_format = '%m/%d/%Y %H:%M:%S' # i.e. 1/1/2016 13:00:00

# broncos, panthers, counts = [], [], []
# for counter in range(1, NUMBER_OF_COLLECTIONS + 1):
#     print '------------------------------------------'
#     print 'COLLECTION NUMBER %s out of %s' % (counter, NUMBER_OF_COLLECTIONS)
#     broncos_counter, panthers_counter = 0, 0 # set the internal counter for Broncos and Panthers to 0
#     count_dict = {'start_time': datetime.now().strftime(format=date_format)} # add collection start time

#     # Create a new stream instance every collection to avoid rate limits
#     auth = oauth_login(consumer_key=CONSUMER_KEY, consumer_secret=CONSUMER_SECRET, token=OAUTH_TOKEN, token_secret=OAUTH_TOKEN_SECRET)
#     twitter_stream = TwitterStream(auth=auth)
#     stream = twitter_stream.statuses.filter(track=track, locations=locations)

#     endTime = datetime.now() + timedelta(minutes=COLLECTION_TIME)
#     while datetime.now() <= endTime:  # collect tweets while current time is less than endTime
#         for tweet in stream:
#             if 'text' in tweet.keys(): # check to see if tweet contains text
#                 if datetime.now() > endTime:
#                     break # if the collection time is up, break out of the loop
#                 elif 'Broncos' in tweet['text'] and 'Panthers' in tweet['text']:
#                     broncos.append(tweet), panthers.append(tweet) # if a tweet contains both Broncos and Panthers, add the tweet to both arrays
#     ...


I am working on the file "1/CaseStudy1.ipynb".
The first task is:
Report some statistics about the tweets you collected

The topic of interest: Patriots

The total number of tweets collected:  2,500

Average number of words per tweet: 14.5536

Lexical Diversity of all words contained in the collection of tweets: 0.1906

*-----------------------
Problem 2: Analyzing Tweets and Tweet Entities with Frequency Analysis
1. Word Count:
* Use the tweets you collected in Problem 1, and compute the frequencies of the words being used in these tweets.
* Plot a table of the top 30 words with their counts
Can you write Python code for it?

from collections import Counter
from prettytable import PrettyTable
import nltk

tweet_texts = [ tweet['text']
                 for tweet in results ]
words = [ word
          for tweet in tweet_texts
              for word in tweet.split()
                 if word not in ['RT', '&'] # filter out RT and ampersand
        ]

# Use the natural language toolkit to eliminate stop words

# nltk.download('stopwords') # download stop words if you do not have it
stop_words = nltk.corpus.stopwords.words('english')
non_stop_words = [w for w in words if w.lower() not in stop_words]

# frequency of words
count = Counter(non_stop_words).most_common()

# table of the top 30 words with their counts
pretty_table = PrettyTable(field_names=['Word', 'Count'])
[ pretty_table.add_row(w) for w in count[:30] ]
pretty_table.align['Word'] = 'l'
pretty_table.align['Count'] = 'r'
print pretty_table


I am working on the file "1/CaseStudy1.ipynb".
The first task is:
This code is used to load the above followers, having already been collected.  It then makes a venn-diagram comparing the mutual followers between the three teams.
Can you write Python code for it?

patriots_followers_ids = load_json('json/Patriots_Followers')
broncos_followers_ids = load_json('json/Broncos_Followers')
panthers_followers_ids = load_json('json/Panthers_Followers')

%matplotlib inline
from matplotlib_venn import venn3

patriots_followers_set = set(patriots_followers_ids)
broncos_followers_set = set(broncos_followers_ids)
panthers_followers_set = set(panthers_followers_ids)

venn3([patriots_followers_set, broncos_followers_set, panthers_followers_set], ('Patriots Followers', 'Broncos Followers',
                                                                                'Panthers Followers'))


I am working on the file "1/CaseStudy1.ipynb".
The first task is:
Use the following code to retreive all friends and followers of @RobGronkowski, one of the Patriots players.
Can you write Python code for it?

# Retrieve the friends and followers of a user, and save to a json file.

# screen_name = 'RobGronkowski'

# gronk_friends_ids, gronk_followers_ids = get_friends_followers_ids(twitter_api, screen_name = screen_name)

# filename = "json/gronk_friends"
# save_json(filename, gronk_friends_ids)

# filename = "json/gronk_followers"
# save_json(filename, gronk_followers_ids)

# Use this code to load the already-retrieved friends and followers from a json file.
gronk_followers_ids = load_json('json/gronk_followers')
gronk_friends_ids = load_json('json/gronk_friends')

# The following function retrieves the screen names of Twitter users, given their user IDs.  If a certain number of screen
# names is desired (for example, 20) max_ids limits the number retreived.

def get_screen_names(twitter_api, user_ids = None, max_ids = None):

    response = []

    items = user_ids

    # Due to individual user security settings, not all user profiles can be obtained.  Iterate over all user IDs
    # to ensure at least (max_ids) screen names are obtained.

    while len(response) < max_ids:
        items_str = ','.join([str(item) for item in items[:100]])
        items = items[100:]

        responses = make_twitter_request(twitter_api.users.lookup, user_id = items_str)

        response += responses

    items_to_info = {}

    # The above loop has retrieved all user information.
    for user_info in response:
        items_to_info[user_info['id']] = user_info

    # Extract only the screen names obtained.  The keys of items_to_info are the user ID numbers.
    names = [items_to_info[number]['screen_name']
            for number in items_to_info.keys()
            ]

    numbers =[number for number in items_to_info.keys()]

    return names , numbers

from prettytable import PrettyTable

# Given a set of user ids, this function calls get_screen_names and plots a table of the first (max_ids) ID's and screen names.
def table_ids_screen_names(twitter_api, user_ids = None, max_ids =...


I am working on the file "1/dlnd-your-first-neural-network.ipynb".
The first task is:
Scaling target variables
To make training the network easier, we'll standardize each of the continuous variables. That is, we'll shift and scale the variables such that they have zero mean and a standard deviation of 1.
The scaling factors are saved so we can go backwards when we use the network for predictions.
Can you write Python code for it?

quant_features = ['casual', 'registered', 'cnt', 'temp', 'hum', 'windspeed']
# Store scalings in a dictionary so we can convert back later
scaled_features = {}
for each in quant_features:
    mean, std = data[each].mean(), data[each].std()
    scaled_features[each] = [mean, std]
    data.loc[:, each] = (data[each] - mean)/std


I am working on the file "1/dlnd-your-first-neural-network.ipynb".
The first task is:
Splitting the data into training, testing, and validation sets
We'll save the last 21 days of the data to use as a test set after we've trained the network. We'll use this set to make predictions and compare them with the actual number of riders.
Can you write Python code for it?

# Save the last 21 days
test_data = data[-21*24:]
data = data[:-21*24]

# Separate the data into features and targets
target_fields = ['cnt', 'casual', 'registered']
features, targets = data.drop(target_fields, axis=1), data[target_fields]
test_features, test_targets = test_data.drop(target_fields, axis=1), test_data[target_fields]