I am working on the file "6. Network Statistical Inference (Student).ipynb".
The first task is:
Introduction
In this notebook, we will walk through a hacker's approach to statistical thinking, as applied to network analysis.
Statistics in a Nutshell
All of statistics can be broken down into two activities:

Descriptively summarizing data. (a.k.a. "descriptive statistics")
Figuring out whether something happened by random chance. (a.k.a. "inferential statistics")

Descriptive Statistics

Centrality measures: mean, median, mode
Variance measures: inter-quartile range (IQR), variance and standard deviation

Inferential Statistics

Models of Randomness (see below)
Hypothesis Testing
Fitting Statistical Models

Load Data
Let's load a protein-protein interaction network dataset.

This undirected network contains protein interactions contained in yeast. Research showed that proteins with a high degree were more important for the surivial of the yeast than others. A node represents a protein and an edge represents a metabolic interaction between two proteins. The network contains loops.
Can you write Python code for it?


I am working on the file "6. Network Statistical Inference (Student).ipynb".
The first task is:
Visually, it should be quite evident that the protein-protein graph did not come from an E-R distribution. Statistically, we can also use the hypothesis test procedure to quantitatively test this, using our simulated E-R data.
Can you write Python code for it?

# Conduct the hypothesis test.
ppG_var > np.percentile(er_vars, 99)  # we can only use the 99th percentile, because there are only 100 data points.


I am working on the file "6. Random Networks and Statistics (Instructor).ipynb".
The first task is:
Computing the interval between the 2.5th to the 97.5th percentile effectively gives you a centered 95% mass of the distribution. Knowing this range can be quite useful in other data analysis cases.
Live Class Exercise
Together, we will implement a random model for the number of trips that are taken between two stations.
In this procedure, we will copy the nodes graph G into a new graph G_random. This can be done by:
G_random = nx.Graph()
G_random.add_nodes_from(G.nodes(data=True))

Following that, we will manually map-reduce this computation.
Can you write Python code for it?

# Grab all the nodes from G.
G_random = nx.Graph()
G_random.add_nodes_from(G.nodes(data=True))
G_random.nodes(data=True)


I am working on the file "6. Random Networks and Statistics (Instructor).ipynb".
The first task is:
Exercise
Can you figure out the 2.5th, 97.5th, and 100th percentile of this distribution?
Hint: You may wish to use numpy's percentile function.
Can you write Python code for it?

import numpy as np

np.percentile(num_trips, [2.5, 97.5, 100])


I am working on the file "6. Random Networks and Statistics (Instructor).ipynb".
The first task is:
Exercise
Can you plot a histogram of the number of trips taken on each route?
Can you write Python code for it?

from collections import Counter
import matplotlib.pyplot as plt
%matplotlib inline

num_trips = [d['count'] for _,_,d in G.edges(data=True)]

plt.bar(Counter(num_trips).keys(), Counter(num_trips).values())


I am working on the file "6. Random Networks and Statistics (Student).ipynb".
The first task is:
Computing the interval between the 2.5th to the 97.5th percentile effectively gives you a centered 95% mass of the distribution. Knowing this range can be quite useful in other data analysis cases.
Live Class Exercise
Together, we will implement a random model for the number of trips that are taken between two stations.
In this procedure, we will copy the nodes graph G into a new graph G_random. This can be done by:
G_random = nx.Graph()
G_random.add_nodes_from(G.nodes(data=True))

Following that, we will manually map-reduce this computation.
Can you write Python code for it?

# Grab all the nodes from G.
G_random = nx.Graph()
G_random.add_nodes_from(G.nodes(data=True))
G_random.nodes(data=True)

# Implement the procedure by brute force. You will find it takes a bit of time.......... but because we have so many people
# in the class, we can brute force parallelize this :-).
from random import sample, choice
import math

n = 40 # the total number of students with Python 3.4 installed.

for i in range(math.ceil(total_trips/n)):
    if i % 1000 == 0:
        print(i)
    n1 = choice(G_random.nodes())
    n2 = choice(G_random.nodes()) #note: n1 and n2 might well be the same nodes.
    if (n1, n2) not in G_random.edges():
        G_random.add_edge(n1, n2, count=0)
    else:
        G_random.edge[n1][n2]['count'] += 1

your_id = 1 # change this number to the ID given to you.
nx.write_gpickle(G_random, 'datasets/divvy_2013/divvy_random{0}.pkl'.format(your_id))

# Load the graphs with randomly re-distributed trip counts as a list of graphs.
import os

def is_pkl(filename):
    """
    Checks if a file is a pickled graph.
    """
    if filename.split('.')[-1] == 'pkl':
        return True
    else:
        return False

def is_divvy_random_graph(filename):
    """
    Checks if it's a Divvy graph.
    """
    if 'divvy_random' in filename:
        return True
    else:
        return False

# Load the data into a list of graphs.
divvy_random_graphs = []
for f in os.listdir('datasets/divvy_2013'):
    if is_pkl(f) and is_divvy_random_graph(f):
        g = nx.read_gpickle('datasets/divvy_2013/{0}'.format(f))
        graphs.append(g)

# Plot the distribution of counts in a re-distributed set of trips.
import matplotlib.pyplot as plt
%matplotlib inline

G_random_master = nx.Graph()

for g in graphs:
    for sc, sk, d in g.edges(data=True):
        if (sc, sk) not in G_random_master.edges():
            G_random_master.add_edge(sc, sk, count=d['count'])
        if (sc, sk) in G_random_master.edges():
            G_random_master.edge[sc][sk]['count'] += d['count']

# Remove edges that...


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:

Can you write Python code for it?


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
Applying a stoplist
In many cases you may wish to apply a stoplist (a list of features to exclude from analysis) to a featureset. In our DfR dataset, the most common words are prepositions and other terms that don't really have anything to do with the topical content of the corpus.
Can you write Python code for it?

corpus.features['wordcounts'].top(5)    # The top 5 words in the FeatureSet.


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
But since not all of our WoS records come from >= 1990, there are a handful for which there are no abstract terms.
Can you write Python code for it?

print 'Only {0} of {1} papers have abstracts, however.'.format(len(wosCorpus.features['abstract'].features), len(wosCorpus.papers))

filter = lambda f, v, c, dc: f not in stoplist and 2 < dc < 400
wosCorpus.features['abstract_filtered'] = wosCorpus.features['abstract'].transform(filter)

print 'There are {0} features in the abstract_filtered featureset.'.format(len(wosCorpus.features['abstract_filtered'].index))

type(wosCorpus.features['abstract_filtered'])


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
Generating networks from topic models
The features module in the tethne.networks subpackage contains some useful methods for visualizing topic models as networks. You can import it just like the authors or papers modules.
Can you write Python code for it?

from tethne.networks import topics

termGraph = topics.terms(model, threshold=0.01)

termGraph.name = ''


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
Introduction to Tethne: Words and Topic Modeling
In this workbook we'll start working with word-based FeatureSets, and use the model.corpus.mallet module to  fit a Latent Dirichlet Allocation (LDA) topic model.
Before you start

Download the practice dataset from here, and store it in a place where you can find it. You'll need the full path to your dataset.

Loading JSTOR DfR datasets
First, import the dfr module from tethne.readers.
Can you write Python code for it?


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
Next we'll fit the model. We need to tell MALLET how many topics to fit (the hyperparameter Z), and how many iterations (max_iter) to perform. This step may take a little while, depending on the size of your corpus.
Can you write Python code for it?


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
Since we're not working from OCR texts (that's where JSTOR DfR data comes from), there are far fewer "junk" words. We end up with a much smaller vocabulary.
Can you write Python code for it?

print 'There are {0} features in the abstract featureset.'.format(len(wosCorpus.features['abstract'].index))


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
So you can retrieve specific Papers from Corpus.papers using their DOIs:
Can you write Python code for it?


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
The 'abstract' featureset is similar to the 'unigrams' featureset from the JSTOR DfR dataset, so we can perform topic modeling with it, too.
Can you write Python code for it?

wosModel = mallet.LDAModel(wosCorpus, featureset_name='abstract_filtered')

wosModel.fit(Z=50, max_iter=500)

wosModel.print_topics(Nwords=5)


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
The abstract_to_features method converts all of the available abstracts in our Corpus to a unigram featureset. It takes no arguments. The abstracts will be diced up into their constituent words, punctuation and capitalization is removed, and a featureset called abstractTerms is generated. By default, abstract_to_features will apply the NLTK stoplist and Porter stemmer.
Can you write Python code for it?

from tethne import tokenize
wosCorpus.index_feature('abstract', tokenize=tokenize, structured=True)


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
The network visualization below was generated in Cytoscape. Edge width is a function of the 'weight' attribute. Edge color is based on the 'topics' attribute, to give some sense of how which clusters of terms belong to which topics. We can see right away that terms like plants, populations, species, and selection are all very central to the topics retrieved by this model.

WoS abstracts
JSTOR DfR is not the only source of wordcounts with which we perform topic modeling. For records more recent than about 1990, the Web of Science includes abstracts in their bibliographic records.
Let's first spin up our WoS dataset.
Can you write Python code for it?

from tethne.readers import wos
wosCorpus = wos.read('/Users/erickpeirson/Projects/tethne-notebooks/data/wos')


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
To check the number of features in a featureset (e.g. the size of a vocabulary), look at the size of the index attribute.
Can you write Python code for it?

print 'There are %i words in the unigrams featureset' % len(corpus.features['wordcounts'].index)


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
Topic modeling with featuresets
Latent Dirichlet Allocation is a popular approach to discovering latent "topics" in large corpora. Many digital humanists use a software package called MALLET to fit LDA to text data. Tethne uses MALLET to fit LDA topic models.
Start by importing the mallet module from the tethne.model.corpus subpackage.
Can you write Python code for it?


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
Unlike WoS datasets, DfR datasets can contain wordcounts, bigrams, trigrams, and quadgrams in addition to bibliographic data. read will automagically load those data as FeatureSets.
Can you write Python code for it?

corpus = dfr.read('/Users/erickpeirson/Projects/tethne-notebooks/data/dfr')

print 'There are %i papers in this corpus.' % len(corpus.papers)

print 'This corpus contains the following features: \n\t%s' % '\n\t'.join(corpus.features.keys())

p = corpus[15]

from tethne.networks.authors import coauthors

graph = coauthors(corpus)

from tethne.writers.graph import to_graphml

to_graphml(graph, '/Users/erickpeirson/Desktop/coauthors.graphml')


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
We can also look at the representation of a topic over time using the topic_over_time() method. In the example below we'll print the first five of the topics on the same plot.
Can you write Python code for it?

plt.figure(figsize=(15, 5))
for k in xrange(5):
    x, y = model.topic_over_time(k)
    plt.plot(x, y, label='topic {0}'.format(k), lw=2, alpha=0.7)
plt.legend(loc='best')
plt.show()


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
We then create a function that will evaluate whether or not a word is in our stoplist. The function should take three arguments:

f -- the feature itself (the word)
v -- the number of instances of that feature in a specific document
c -- the number of instances of that feature in the whole FeatureSet
dc -- the number of documents that contain that feature

This function will be applied to each word in each document. If it returns 0 or None, the word will be excluded. Otherwise, it should return a numeric value (in this case, the count for that document).
In addition to applying the stoplist, we'll also exclude any word that occurs in more than 50 of the documents and less than 3 documents.
Can you write Python code for it?

def apply_stoplist(f, v, c, dc):
    if f in stoplist or dc > 50 or dc < 3:
        return 0
    return v


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
Whereas Corpora generated from WoS datasets are indexed by wosid, Corpora generated from DfR datasets are indexed by doi.
Can you write Python code for it?

corpus.indexed_papers.keys()[0:10]    # The first 10 dois in the Paper index.


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
You can inspect the inferred topics using the model's print_topics() method. By default, this will print the top ten words for each topic.
Can you write Python code for it?


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
You may apply any stoplist you like to a featureset. In this example, we import the Natural Language ToolKit (NLTK) stopwords corpus.
Can you write Python code for it?

from nltk.corpus import stopwords
stoplist = stopwords.words()


I am working on the file "6. Words and topic modeling.ipynb".
The first task is:
topics.cotopics() creates a network of topics, linked by virtue of their co-occurrence in documents. Use the threshold parameter to tune the density of the graph.
Can you write Python code for it?

coTopicGraph = topics.cotopics(wosModel, threshold=0.15)

print '%i nodes and %i edges' % (coTopicGraph.order(), coTopicGraph.size())

graph.write_graphml(coTopicGraph, '/Users/erickpeirson/Projects/tethne-notebooks/output/lda_coTopics.graphml')


I am working on the file "6. Голодающие дети Африки 2 - Ансамблирование, стекинг, блэндинг.ipynb".
The first task is:

Задание

Применив ансамблирование, стекинг и блэндинг улучшить вчерашние результаты
Вчерашние результаты для устрашения
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

from lightgbm import LGBMClassifier

clf2 = LGBMClassifier(max_bin=460,learning_rate=0.11,n_estimators=140,num_leaves=130)

scores = cross_val_score(clf2, x_train, y)
np.mean(scores), 2*np.std(scores)

clf2 = clf2.fit(x_train, y)

y_te = clf2.predict(x_test)
y_te

ans_nn = pd.DataFrame({'id': df_test['id'], 'status_group': y_encoder.inverse_transform(y_te)})
ans_nn.head()

#ans_nn.to_csv('ans_lightgbm.csv', index=False)


I am working on the file "6. Голодающие дети Африки 2 - Ансамблирование, стекинг, блэндинг.ipynb".
The first task is:
Out: спустя 4 часа работы fit висит в памяти и ничего не выдает. При выполнении mean(scores)=0.76407, тут результат для 'hard' без весов.
Can you write Python code for it?

#eclf = eclf.fit(x_train, y)

#y_te = eclf.predict(x_test)
#y_te

#ans_nn = pd.DataFrame({'id': df_test['id'], 'status_group': y_encoder.inverse_transform(y_te)})
#ans_nn.head()

#ans_nn.to_csv('ans_ansambl.csv', index=False)


I am working on the file "6. Голодающие дети Африки 2 - Ансамблирование, стекинг, блэндинг.ipynb".
The first task is:
out: Windows BlueScreen with "unexpected store exception" спустя 2 часа работы + pyhon.exe not responding error
Can you write Python code for it?

#sclf = sclf.fit(x_train, y)

#y_te = sclf.predict(x_test)
#y_te

#ans_nn = pd.DataFrame({'id': df_test['id'], 'status_group': y_encoder.inverse_transform(y_te)})
#ans_nn.head()

#ans_nn.to_csv('ans_stack.csv', index=False)


I am working on the file "6. Голодающие дети Африки 2 - Ансамблирование, стекинг, блэндинг.ipynb".
The first task is:
Данные
Зайдите на https://www.drivendata.org/ и зарегистрируйтесь. Для сегодняшней домашки будем данные брать именно отсюда.
Нас интересует конкурс https://www.drivendata.org/competitions/7/pump-it-up-data-mining-the-water-table/page/23/ .
В нем представлены данные, собранные Taarifa и Танзанийским Министерством Воды и Ирригации.
Постановка задачи следующая:
На территории Танзании установлено множество водяных насосов, которые спасают местное население от жажды. В зависимости от того, кем и когда установлен насос, а также зная, как им распоряжаются, можно попытаться предположить, какие из них функционируют, какие нуждаются в ремонте и какие не работают вовсе.
Этим мы и займемся, а заодно и прокачаемся в подборе гиперпараметров алгоритмов.
Can you write Python code for it?

train_X, train_y = pd.read_csv( # путь к вашему файлу train.csv
    'data/WaterTable/train.csv'
), pd.read_csv( # путь к вашему файлу trainLabels.csv
    'data/WaterTable/trainLabels.csv'
)
df = pd.merge(train_X, train_y, how='left')

df_test = pd.read_csv( # путь к вашему файлу test.csv
    'data/WaterTable/test.csv'
)


I am working on the file "6. Голодающие дети Африки 2 - Ансамблирование, стекинг, блэндинг.ipynb".
The first task is:
Опробуем Блэндинг
Can you write Python code for it?

import logging
import numpy as np

from sklearn.model_selection import train_test_split
from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import mean_absolute_error
from sklearn.neighbors import KNeighborsRegressor
#from xgboost import XGBClassifier
from lightgbm import LGBMClassifier

from heamy.dataset import Dataset
from heamy.estimator import Regressor, Classifier
from heamy.pipeline import ModelsPipeline

np.set_printoptions(precision=6)
np.set_printoptions(suppress=True)

np.random.seed(1000)

# create dataset
dataset = Dataset(x_train, y, x_test)

# initialize RandomForest & LinearRegression
model_rf = Regressor(dataset=dataset, estimator=RandomForestRegressor, parameters={'n_estimators': 50},name='rf')
model_lr = Regressor(dataset=dataset, estimator=LinearRegression, parameters={'normalize': True},name='lr')

# Stack two models
# Returns new dataset with out-of-fold predictions
pipeline = ModelsPipeline(model_rf,model_lr)
stack_ds = pipeline.stack(k=10,seed=111)

# Train LinearRegression on stacked data (second stage)
stacker = Regressor(dataset=stack_ds, estimator=LinearRegression)
results = stacker.predict()

results

# Use randomly sampled 20% of the data as a holdout dataset
res = model_lr.validate(mean_absolute_error, test_size=0.20)


I am working on the file "6. Голодающие дети Африки 2 - Ансамблирование, стекинг, блэндинг.ipynb".
The first task is:
Опробуем ансамблирование
Can you write Python code for it?

from sklearn.pipeline import make_pipeline
from mlxtend.classifier import EnsembleVoteClassifier
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LogisticRegression
from lightgbm import LGBMClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score

clfs = [make_pipeline(PolynomialFeatures(degree=2), LogisticRegression()),
        LGBMClassifier(),
        DecisionTreeClassifier()]

eclf = EnsembleVoteClassifier(clfs=clfs, voting='soft', weights=[1,2,1])

scores = cross_val_score(eclf, x_train, y)
np.mean(scores), 2*np.std(scores)


I am working on the file "6. Голодающие дети Африки 2 - Ансамблирование, стекинг, блэндинг.ipynb".
The first task is:
Опробуем стекинг
Can you write Python code for it?

from mlxtend.classifier import StackingCVClassifier, StackingClassifier
from sklearn.ensemble import RandomForestClassifier

sclf = StackingCVClassifier(clfs, RandomForestClassifier())

scores = cross_val_score(sclf, x_train, y)
np.mean(scores), 2*np.std(scores)


I am working on the file "6.00.1x.Introduction-to-Computer-Science-and-Programming-Using-Python/Notebooks/Week 02.ipynb".
The first task is:
2. COUNTING BOBS
Assume 's' is a string of lower case characters.
Write a program that prints the number of times the string 'bob' occurs in s. For example, if s = 'azcbobobegghakl', then your program should print
                    Number of times bob occurs is: 2
Can you write Python code for it?

s = 'azcbobobegghakl'
pattern = 'bob'
count =0
for position in range(0,len(s)):
    if s[position:position+3]==pattern:
        count+=1
print count


I am working on the file "6.00.1x.Introduction-to-Computer-Science-and-Programming-Using-Python/Notebooks/Week 02.ipynb".
The first task is:
3. USING BISECTION SEARCH TO MAKE THE PROGRAM FASTER
Can you write Python code for it?

range(12)

def payOffDebtBisection(balance,annualInterestRate,months=12):

    balance_remaining = balance
    monthlyInterestRate = annualInterestRate/12.0

    lower_limit = balance/12.0
    upper_limit = balance * ((1+monthlyInterestRate)**12)/12.0

    tolerance =0.01

    while abs(balance_remaining) > tolerance:
        balance_remaining = balance
        payment = ( lower_limit + upper_limit )/2

        for month in range(12):
            balance_remaining = balance_remaining-payment
            balance_remaining = balance_remaining*(1+monthlyInterestRate)

        if balance_remaining<=0:
            upper_limit = payment

        else:
            lower_limit = payment
    print 'Lowest Payment: ' + str(round(payment, 2))

payOffDebtBisection(320000,0.2)
payOffDebtBisection(999999,0.18)


I am working on the file "6.16Notes.ipynb".
The first task is:
define your own character classes
inside your regular expression, write [aeiou]
Can you write Python code for it?

[line for line in subjects if re.search("[aeiou][aeiou][aeiou][aeiou]", line)]

[line for line in subjects if re.search("F[wW]:", line)]


I am working on the file "6.16Notes.ipynb".
The first task is:
metacharacters

^ beginning of string
$ end of string
\b word boundary
Can you write Python code for it?

[line for line in subjects if res.search("^[Nn]ew [Yy]ork", line)]

[line for line in subjects if re.search(r"\boil\b", line)]


I am working on the file "6.16Notes.ipynb".
The first task is:
metacharacters 3: quantifiers
        * match zero or more times
        {n} matches exactly n times
        {n,m} matches at least n times, but no more than m times
        {n,} matches at least n times, but maybe infinite times
        + match at least once ({1,})
        ? match one time or zero times

[line for line in subjects if re.search(r"^R
 string matches regular expression if at the first line, you encounter .......
Can you write Python code for it?

[line for line in subjects if re.search(r"\b(?:[Cc]at|[kK]itty|[kK]itten)\b", line)]


I am working on the file "6.1_kalman_general.ipynb".
The first task is:
Système linéaire gaussien en tems discret
$$
   X_{k+1} = A\,X_{k} + B\,W_k\,,\ 0\leq k < k_{max}
$$

$X_k\to\mathbb{R}^n$
bruit: $W_k\to\mathbb{R}^m$, $W_k\sim N(0,Q_W)$
$A\in\mathbb{R}^{n\times n}$, $B\in\mathbb{R}^{m\times n}$

Il s'agit d'un système gaussien: $X_{0:k_{max}}$ est un vecteur aléatoire gaussien. (Notation: $Z_{k':k}=(Z_{k'},Z_{k'+1},\dots ,Z_{k})$, $k'\leq k$).
La moyenne $\bar X_k = \mathbb{E}(X_k)$ et la covariance $Q^X_k = \mathrm{Var}(X_k)$ sont donnés par:
\begin{align}
   \bar X_{k+1} &= A\,\bar X_{k}
   \
   Q^X_{k+1} &= A\,Q^X_{k}\,A^+B\,Q_W\,B^
\end{align}
Can you write Python code for it?

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

kmax = 300

EX0, VX0 = 10, 5
A, B, QW = 0.9, 1, 0.1
sQW = np.sqrt(QW)
sVX0 = np.sqrt(VX0)

def sys_lin(EX0, sVX0, A, B, sQW):
    W = sQW*np.random.randn(kmax)
    X = np.ones(kmax+1)
    X[0] = EX0+sVX0*np.random.randn()
    for k in range(kmax):
        X[k+1] = A*X[k]+B*W[k]
    return X

def sys_lin_loi(EX0, sVX0, A, B, sQW):
    espX = np.ones(kmax+1)
    varX = np.ones(kmax+1)
    espX[0] = EX0
    for k in range(kmax):
        espX[k+1] = A*espX[k]
        varX[k+1] = A*A*varX[k]+B*B*QW
    return espX, varX

X = sys_lin(EX0, sVX0, A, B, sQW)
espX, varX = sys_lin_loi(EX0, sVX0, A, B, sQW)

plt.plot([0, kmax], [0, 0], color="g", linestyle=':')
plt.plot(espX,color='k')
plt.fill_between(range(kmax+1),espX+2*np.sqrt(varX),
                 espX-2*np.sqrt(varX), color = '0.75', alpha=0.4)
plt.plot(X)
plt.show()

from ipywidgets import interact, fixed

def plt_sys_lin(A, B, iseed):
    np.random.seed(iseed)
    X = sys_lin(10, 0, A, B, 1)
    plt.plot([0, kmax], [0, 0], color="g", linestyle=':')
    plt.plot(X)
    plt.ylim([-4,15])
    plt.show()

interact(plt_sys_lin, A=(0,1,0.01), B=(0.,6,0.1),
         iseed=(1,100,1))
plt.show()


I am working on the file "6.1_kalman_general.ipynb".
The first task is:
Un peu de vectorisation
Can you write Python code for it?

kmax = 300

mcmax = 300
EX0, VX0 = 10, 5
A, B, QW = 0.9, 1, 0.1
sQW = np.sqrt(QW)
sVX0 = np.sqrt(VX0)

def sys_lin_vec(mcmax,EX0, sVX0, A, B, sQW):
    W = sQW*np.random.randn(kmax,mcmax)
    X = np.ones((kmax+1,mcmax))
    X[0,] = EX0+sVX0*np.random.randn()
    for k in range(kmax):
        X[k+1,] = A*X[k,]+B*W[k,]
    return X

X = sys_lin_vec(mcmax, EX0, sVX0, A, B, sQW)
plt.plot(X,alpha=.04,color='b')
plt.plot(espX,color='w')
plt.plot(espX+2*np.sqrt(varX),color='k')
plt.plot(espX-2*np.sqrt(varX),color='k')
plt.show()

mcmax = 10000
X = sys_lin_vec(mcmax, EX0, sVX0, A, B, sQW)
num_bins = 30
n, bins, patches = plt.hist(X[-1,], num_bins, normed=1,
                            facecolor='green', alpha=0.5)
plt.show()


I am working on the file "6.5 - Baby names.ipynb".
The first task is:
Define the extract_names() function below and change baby_names()
to call it.
For writing regex, it's nice to include a copy of the target
text for inspiration.
Here's what the html looks like in the baby.html files:
Suggested milestones for incremental development:
 * Extract the year and print it
 * Extract the names and rank numbers and just print them
 * Get the names data into a dict and print it
 * Build the [year, 'name rank', ... ] list and print it
 * Fix baby_names() to use the extract_names list
Can you write Python code for it?

def extract_names(filename):
    """
    Given a file name for baby.html, returns a list starting with the year string
    followed by the name-rank strings in alphabetical order.
    ['2006', 'Aaliyah 91', Aaron 57', 'Abagail 895', ' ...]
    """
    # +++your code here+++
    return

def baby_names(file_list, summary=False):
# +++your code here+++
# For each filename, get the names, then either print the text output
# or write it to a summary file

baby_names(['data/babynames/baby1990.html'])

wordcount('topcount', 'data/wiki.txt')

baby_names(['data/babynames/baby1996.html'], summary=True)

baby_names(['data/babynames/baby2000.html', 'data/babynames/baby2002.html'])


I am working on the file "6.RF.ipynb".
The first task is:
Key input parameters (in addition to decision trees)

bootstrap: Whether bootstrap samples are used when building trees
max_features: The number of features to consider when looking for the best split (auto = sqrt)
n_estimators: The number of trees in the forest
oob_score: Whether to use out-of-bag samples to estimate the generalization accuracy

Key output parameters

Feature Importance: The higher, the more important the feature
Out-of-Bag Score: Validation score of the training dataset obtained using an out-of-bag estimate.

Feature Importance
The relative rank (i.e. depth) of a feature used as a decision node in a tree can be used to assess the relative importance of that feature with respect to the predictability of the target variable. Features used at the top of the tree contribute to the final prediction decision of a larger fraction of the input samples. The expected fraction of the samples they contribute to can thus be used as an estimate of the relative importance of the features.
By averaging those expected activity rates over random trees one can reduce the variance of such an estimate and use it for feature selection.
Can you write Python code for it?

importances = clf.feature_importances_

# Importance of the features in the forest
importances

#Calculate the standard deviation of variable importance
std = np.std([tree.feature_importances_ for tree in clf.estimators_], axis=0)

std

indices = np.argsort(importances)[::-1]
indices

length = train_13_data.shape[1]

labels = []

for i in range(length):
    labels.append(train_13_data.columns[indices[i]])

# Plot the feature importances of the forest
plt.figure(figsize=(16, 6))
plt.title("Feature importances")
plt.bar(range(length), importances[indices], yerr=std[indices], align="center")
plt.xticks(range(length), labels)
plt.xlim([-1, length])
plt.show()


I am working on the file "6.RF.ipynb".
The first task is:
Out-of-Bag Error
The out-of-bag (OOB) error is the average error for each training observation calculated using predictions from the trees that do not contain it in their respective bootstrap sample. This allows the RandomForest to be fit and validated whilst being trained.
Can you write Python code for it?

import warnings
warnings.filterwarnings('ignore')

clf2 = RandomForestClassifier(warm_start=True, oob_score=True, max_features=None)

clf2.fit(train_13_data, train_13_target)

clf2.oob_score_

min_estimators = 10
max_estimators = 50
error_rate = []

for i in range(min_estimators, max_estimators + 1):
    clf2.set_params(n_estimators=i)
    clf2.fit(train_13_data, train_13_target)
    oob_error = 1 - clf2.oob_score_
    error_rate.append(oob_error)

error_rate_indice = [x for x in range(min_estimators, max_estimators + 1)]

plt.figure()
plt.plot(error_rate_indice, error_rate)
plt.xlim(min_estimators, max_estimators)
plt.xlabel("n_estimators")
plt.ylabel("OOB error rate")
plt.show()


I am working on the file "6.Recommendation - TFIDF-content-similarity.ipynb".
The first task is:
使用 BeautifulSoup 擷取 HTML 內容，並套用 Jieba 斷詞
Can you write Python code for it?

## getContent: for input aritcle, get it own word set via jieba.cut()
def getContent(x):
    from bs4 import BeautifulSoup
    soup = BeautifulSoup(x)
    text = soup.getText().replace('\n','').replace('\r','').replace(' ','').replace('\t','')
    import jieba
    r = list()
    for term in jieba.cut(text):
        if len(term) > 1 and checkword(term): r.append(term)
    return r

def checkword(x):
    return all(u'\u4e00' <= c <= u'\u9fff' for c in x)


I am working on the file "6.Recommendation - TFIDF-content-similarity.ipynb".
The first task is:
列出20個每篇文章都有出現的詞
Can you write Python code for it?

IDFTokens = idfsTrain.filter(lambda token: token[1] == 1).take(20)  ##takeOrdered(10, lambda s: -s[1])
for token in IDFTokens:
    print(token[0] + " " + str(token[1]))


I am working on the file "6.Recommendation - TFIDF-content-similarity.ipynb".
The first task is:
實作 Dot Product 涵式，透過 cosine similarity 計算文章相似度的分數
dotprod
- 針對兩篇文章的tfidf() dictionary，針對所有共同 key 的值做相乘，並將結果加總
norm
- 計算 cosine similarity 的 square root
cossim
- 計算兩篇文章的 cosine similarity
Can you write Python code for it?

import math

def dotprod(a, b):
    dotsum = 0
    for tk in a:
        if tk in b:
            dotsum += a[tk]*b[tk]
    return dotsum

def norm(a):
    return math.sqrt(dotprod(a,a))

def cossim(a, b):
    return dotprod(a,b)/(norm(a) * norm(b))

def cosineSimilarity(string1, string2, idfsDictionary):
    w1 = tfidf(string1, idfsDictionary)
    w2 = tfidf(string2, idfsDictionary)
    return cossim(w1, w2)


I am working on the file "6.Recommendation - TFIDF-content-similarity.ipynb".
The first task is:
探索文章裡的關鍵字
Can you write Python code for it?

showTopWord(u'http://bowpisces.pixnet.net/blog/post/162504740')

similarArticle = getSimilar(u'http://chahabi77.pixnet.net/blog/post/354943724')
for index in range(1, 4):
    print(similarArticle[index])

showTopWord(u'http://chahabi77.pixnet.net/blog/post/235296791')


I am working on the file "6.Recommendation - TFIDF-content-similarity.ipynb".
The first task is:
輸入一篇文章，取得三篇最相似文章
Can you write Python code for it?

similarArticle = getSimilar(u'http://bowpisces.pixnet.net/blog/post/152118460')
for index in range(1, 4):
    print(similarArticle[index])


I am working on the file "6/Applications in Finance.ipynb".
The first task is:
But let's try understanding how this data might come to be. For a stock price, we might assume that there are random movements up and down, so maybe if we look at the difference in stock price between timestamps, it might be distributed as a 0-centered random variable. But what kind of random variable? Usually our go to in statistics is to start with a normal distribution, since we understand it's properties, but in this case we can empirically find that the distribution has heavier tails (i.e. more probability mass farther away from 0) than a normal distribution. So, maybe it can be generated via a Student's t-distribution. Let's try looking at what the data would look like in that case:
Can you write Python code for it?

# generate 500 stocks, each with 2000 sampled changes (let's assume they all start at 0)
rpaths = np.random.standard_t(5, size=(2000,500))  # generate paths

# the cumulative sums plotted (i.e. the stock over time)
plt.figure(figsize=(16,5))
plt.plot(rpaths.cumsum(axis=0),alpha=0.5)
plt.show()

# the actual values sampled from the t-distribution
plt.figure(figsize=(16,5))
plt.hist(rpaths.T[1],bins=50)


I am working on the file "6/Applications in Finance.ipynb".
The first task is:
Nice! Turns out Marchenko and Pastur were right. Note that the theorem doesn't assume anything about the distribution of the entries in $X$ other than that it has finite variance.
Plan of Attack
Ok, if we want to reduce the noise of our time series (to understand the signal), we can replace the eigenvalues in our correlation matrix (from our real, Yahoo data) with the mean of the mean from the Marchenko/Pastur distribution, and use that to create a new correlation matrix. Then we can search over all positive semidefinite symmetric matrices (well, technically those with 1s in the main diagonal since that's what a correlation matrix looks like) to find the closes one.
How do we measure closeness between two matrices? We can use a metric called the Frobenius norm, which is the square root of sum of the squares of the elementwise differences between the two matrices. In essence, if the matrices are [[1, 2], [3, 4]] and [[5, 6],[7, 8]], then the Frobenius norm is:
$$\sqrt{4^2 + 4^2 + 4^2 + 4^2}$$
How do we do this search? Luckily since we're working with Python someone has written the code to do it for us (which works essentially via gradient descent). Let's import it now.
Can you write Python code for it?

from nearest_correlation import nearcorr

def lamplus(n,r):
    '''
    Upper eigenvalue limit of the Marchenko Pastur density
    '''
    q = float(n)/float(r)
    return (1+1/q+2*np.sqrt(1/q))

# Read in closing price data if it is not already a local variable
if 'cell' not in locals():
    df = pd.read_pickle('close_price.pkl')
else:
    df = cell

dte1 = '2010-07-01'
dte2 = '2015-07-01'
tickers = sorted(list(set(df['Ticker'].values)))                   # sorted list of unique tickers
tkrlens = [len(df[df.Ticker==tkr][dte1:dte2]) for tkr in tickers]  # find lengths of times series for each ticker
tkrmode = mode(tkrlens)[0][0]                                      # find mode of time series lengths

# idenfity tickers whose lenghts equal the mode and call these good tickers
good_tickers = [tickers[i] for i,tkr in enumerate(tkrlens) if tkrlens[i]==tkrmode]

rtndf = pd.DataFrame()  # initialize a return dataframe

# Normalize all time series to have mean zero and variance one and compute their returns
for tkr in good_tickers:
    tmpdf = df[df.Ticker==tkr]['Adj Close'][dte1:dte2]
    tmprtndf = ((tmpdf-tmpdf.shift(1))/tmpdf).dropna()
    rsdf = (tmprtndf-tmprtndf.mean())/tmprtndf.std()
    rtndf = pd.concat([rtndf, rsdf],axis=1)

rtndf = rtndf.dropna()
rtndf.columns = good_tickers
t,m = rtndf.shape
cmat = rtndf.corr()                                           # compute correlation matrix
evls, evcs = la.eig(cmat)                                     # compute eigenvalue/vector decomposition of matrix
evallst = map(abs,evls)                                       # take abs of evals (very small imag parts)
filtvals = [val for val in evallst if val < lamplus(t,m)]     # filter eigenvalues
sevlist = [np.mean(filtvals)]*len(filtvals)
feval = evallst[:(len(evallst)-len(sevlist))] + sevlist       # build list of new eigenvalues
rcmat = abs(np.dot(np.dot(evcs,np.diag(feval)),la.inv(evcs))) # reconstruct candidate correlation matrix
rcmat = (rcmat + rcmat.T)/2                        ...


I am working on the file "6/Applications in Finance.ipynb".
The first task is:
One of the usual tricks with time series is to look at the matrix of correlations of the series, and in particular one way to look at matrices is to look at their spectral distribution, or their eigenvalues:
Can you write Python code for it?

evalplot = la.eig(pd.DataFrame(rpaths).corr())[0]
plt.figure(figsize=(16,5))
plt.hist(evalplot,bins=50)


I am working on the file "6/Applications in Finance.ipynb".
The first task is:
So the eigenvalues seem to have a distribution: 0 before 0.25, increasing sharply, and then tailing off to 0 around 2.25. In fact, there's a theorem that tells us what the distribution of eigenvalues will look like:
Theorem (Marchenko & Pastur)
Let $X$ be a random $n \times r$ matrix whose i.i.d. entries have finite variance. Then the eigenvalues of the correlation matrix of $X$ are distributed as:
$$f(\lambda) = \frac{q}{2\pi\lambda}\sqrt{(\lambda_+ - \lambda)(\lambda - \lambda_-)}$$
where
$$\lambda_{\pm} = 1 + \frac{1}{q} + 2\sqrt{\frac{1}{q}}$$ and $q = n/r$.
Let's plot this function:
Can you write Python code for it?

# density distribution
def eigden(lam,n,r):
    q = float(n)/r
    lplus = 1+1/q+2*np.sqrt(1/q)
    lminus = 1+1/q-2*np.sqrt(1/q)
    return q/(2*np.pi*lam)*np.sqrt((lplus-lam)*(lam-lminus))

lamvls = np.linspace(0.001,3,1000)
plt.figure(figsize=(14,5))
plt.hist(evalplot,bins=75,normed=True)
plt.plot(lamvls,[eigden(lam,2000,500) for lam in lamvls])
plt.show()


I am working on the file "608_HW4/IS608_HW4.ipynb".
The first task is:
Create lists & graphs of the best and worst places to swim in the dataset.
Can you write Python code for it?

# Create a column showing whether or not the water quality is acceptable in a given place
# unnaceptable:
# 110 Enterococcus/100mL OR
# five or more samples with a geometric mean (a weighted average) > 30 Enterococcus/100mL.

# The prompt is really vague in defining the geometric mean condition - I used gmean()

# When calculated, geometric mean by site of highest 5 samples and single sample >= 110
# shows nearly all sites would have water with unacceptable water quality at some point

raw_data['swim'] = np.where(
    (raw_data.groupby('Site').EnteroCount.transform(max) > 110)
    | (raw_data.groupby('Site').EnteroCount.transform(lambda group: gmean(group.nlargest(5))) > 30),
    'unacceptable', 'acceptable')
raw_data.head(10)

# As a result of above criteria not being very useful, I instead grouped the data by site and
# calculated the mean entero_count for each site

mean_entero = raw_data.groupby(['Site'])['EnteroCount'].mean().sort_values()

# Top ten best places to swim
print "Ten best places to swim"
print "+++++++++++++++++++++++"
mean_entero.head(10)

# Ten worst places to swim
print "Ten worst places to swim"
print "++++++++++++++++++++++++"
mean_entero.tail(10).sort_values(ascending=False)

# write the results to a csv and read in for more flexible use (now and later)
mean_entero.to_csv( 'mean_entero.csv' )

mean_entero1 = pd.read_csv('mean_entero.csv')
mean_entero1.columns = ['Site','Average_Entero_Count']
mean_entero1.head() #make sure it looks right

# Plot the results
%matplotlib inline
sns.set_style("whitegrid")
sns.set_style("ticks")
plt.figure(figsize=(10, 20))
ax = sns.barplot(x ="Average_Entero_Count", y="Site", data=mean_entero1, palette="Blues_d")
ax.set(xlabel='Average Entero Count', ylabel='Site')
ax.set_title('Waterways ranked by Average Entero Count')
sns.despine()

# Don't go swimming in the Gowanus Canal


I am working on the file "608_HW4/IS608_HW4.ipynb".
The first task is:
Enter some sites into the search boxes
Can you write Python code for it?

site1 = raw_input('enter your first site --> ') # Gowanus Canal <- worst

site2 = raw_input('enter your second site --> ') # Croton Point Beach <- best water

# Gowanus is dirtiest, Croton Point Beach is cleanest
x = site_rain_data.loc[site_rain_data['Site'] == site1]
y = site_rain_data.loc[site_rain_data['Site'] == site2]
frames = [x, y]
result = pd.concat(frames)
del result['Date']
del result['SampleCount']
result.head()

plt.figure(figsize=(10, 10))
sns.lmplot('FourDayRainTotal', 'EnteroCount', data=result, hue='Site', palette="Set2", fit_reg=False, scatter_kws={"s": 50}, legend=False)
plt.legend(loc='upper right')
sns.despine()
sns.plt.ylim(0)
sns.plt.xlim(0)


I am working on the file "608_HW4/IS608_HW4.ipynb".
The first task is:
Is there a relationship between the amount of rain and water quality? Show this relationship graphically. If you can, estimate the effect of rain on quality at different sites and create a visualization to compare them.
Can you write Python code for it?

# Cleanup so we're using just the entero count and rain totals
del raw_data['Site']
del raw_data['Date']
del raw_data['SampleCount']
raw_data.head()

%matplotlib inline
# Scatterplot to show relationship between rainfall and Entero Count
plt.figure(figsize=(7.5, 6.5))
sns.regplot('FourDayRainTotal', 'EnteroCount', data=raw_data, fit_reg=False, x_jitter=1)
sns.despine()
sns.plt.ylim(0)
sns.plt.xlim(0)

# There seems to be a relatively strong relationship between rainfall and water quality once
# you go beyond ~3 inches of rain. At that point there are much fewer high Entero Count readings

# Compare any two locations with the following

# re-read in/clean up
data_url = "https://raw.githubusercontent.com/jlaurito/CUNY_IS608/master/lecture4/data/riverkeeper_data_2013.csv"
site_rain_data = pd.read_csv(data_url)
site_rain_data["Site"] = site_rain_data["Site"].astype('category') # like facets
site_rain_data["Date"] = pd.to_datetime(site_rain_data["Date"]) # dates as dates
site_rain_data["EnteroCount"] = site_rain_data["EnteroCount"].str.lstrip('><')
site_rain_data["EnteroCount"] = site_rain_data["EnteroCount"].astype('int64')
site_rain_data.head()


I am working on the file "60_Registration_Introduction.ipynb".
The first task is:
If we are satisfied with the results, save them to file.
Can you write Python code for it?

sitk.WriteImage(moving_resampled, os.path.join(OUTPUT_DIR, 'RIRE_training_001_mr_T1_resampled.mha'))
sitk.WriteTransform(final_transform, os.path.join(OUTPUT_DIR, 'RIRE_training_001_CT_2_mr_T1.tfm'))


I am working on the file "60_Registration_Introduction.ipynb".
The first task is:
Initial Alignment
Use the CenteredTransformInitializer to align the centers of the two volumes and set the center of rotation to the center of the fixed image.
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(fixed_image,
                                                      moving_image,
                                                      sitk.Euler3DTransform(),
                                                      sitk.CenteredTransformInitializerFilter.GEOMETRY)

moving_resampled = sitk.Resample(moving_image, fixed_image, initial_transform, sitk.sitkLinear, 0.0, moving_image.GetPixelIDValue())

interact(display_images_with_alpha, image_z=(0,fixed_image.GetSize()[2]), alpha=(0.0,1.0,0.05), fixed = fixed(fixed_image), moving=fixed(moving_resampled));


I am working on the file "60_Registration_Introduction.ipynb".
The first task is:
Now visually inspect the results.
Can you write Python code for it?

moving_resampled = sitk.Resample(moving_image, fixed_image, final_transform, sitk.sitkLinear, 0.0, moving_image.GetPixelIDValue())

interact(display_images_with_alpha, image_z=(0,fixed_image.GetSize()[2]), alpha=(0.0,1.0,0.05), fixed = fixed(fixed_image), moving=fixed(moving_resampled));


I am working on the file "60_Registration_Introduction.ipynb".
The first task is:
Post registration analysis
Query the registration method to see the metric value and the reason the optimization terminated.
The metric value allows us to compare multiple registration runs as there is a probabilistic aspect to our registration, we are using random sampling to estimate the similarity metric.
Always remember to query why the optimizer terminated. This will help you understand whether termination is too early, either due to thresholds being too tight, early termination due to small number of iterations - numberOfIterations, or too loose, early termination due to large value for minimal change in similarity measure - convergenceMinimumValue)
Can you write Python code for it?

print('Final metric value: {0}'.format(registration_method.GetMetricValue()))
print('Optimizer\'s stopping condition, {0}'.format(registration_method.GetOptimizerStopConditionDescription()))


I am working on the file "60_Registration_Introduction.ipynb".
The first task is:
Read images
We first read the images, casting the pixel type to that required for registration (Float32 or Float64) and look at them.
Can you write Python code for it?

fixed_image =  sitk.ReadImage(fdata("training_001_ct.mha"), sitk.sitkFloat32)
moving_image = sitk.ReadImage(fdata("training_001_mr_T1.mha"), sitk.sitkFloat32)

interact(display_images, fixed_image_z=(0,fixed_image.GetSize()[2]-1), moving_image_z=(0,moving_image.GetSize()[2]-1), fixed_npa = fixed(sitk.GetArrayFromImage(fixed_image)), moving_npa=fixed(sitk.GetArrayFromImage(moving_image)));


I am working on the file "60_Registration_Introduction.ipynb".
The first task is:
Registration
The specific registration task at hand estimates a 3D rigid transformation between images of different modalities. There are multiple components from each group (optimizers, similarity metrics, interpolators) that are appropriate for the task. Note that each component selection requires setting some parameter values. We have made the following choices:

Similarity metric, mutual information (Mattes MI):

Number of histogram bins, 50.

Sampling strategy, random.

Sampling percentage, 1%.

Interpolator, sitkLinear.

Optimizer, gradient descent:

Learning rate, step size along traversal direction in parameter space, 1.0 .

Number of iterations, maximal number of iterations, 100.

Convergence minimum value, value used for convergence checking in conjunction with the energy profile of the similarity metric that is estimated in the given window size, 1e-6.

Convergence window size, number of values of the similarity metric which are used to estimate the energy profile of the similarity metric, 10.

Perform registration using the settings given above, and take advantage of the built in multi-resolution framework, use a three tier pyramid.
In this example we plot the similarity metric's value during regisration. Note that the change of scales in the multi-resolution framework is readily visible.
Can you write Python code for it?

registration_method = sitk.ImageRegistrationMethod()

# Similarity metric settings.
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)

registration_method.SetInterpolator(sitk.sitkLinear)

# Optimizer settings.
registration_method.SetOptimizerAsGradientDescent(learningRate=1.0, numberOfIterations=100, convergenceMinimumValue=1e-6, convergenceWindowSize=10)
registration_method.SetOptimizerScalesFromPhysicalShift()

# Setup for the multi-resolution framework.
registration_method.SetShrinkFactorsPerLevel(shrinkFactors = [4,2,1])
registration_method.SetSmoothingSigmasPerLevel(smoothingSigmas=[2,1,0])
registration_method.SmoothingSigmasAreSpecifiedInPhysicalUnitsOn()

# Don't optimize in-place, we would possibly like to run this cell multiple times.
registration_method.SetInitialTransform(initial_transform, inPlace=False)

# Connect all of the observers so that we can perform plotting during registration.
registration_method.AddCommand(sitk.sitkStartEvent, start_plot)
registration_method.AddCommand(sitk.sitkEndEvent, end_plot)
registration_method.AddCommand(sitk.sitkMultiResolutionIterationEvent, update_multires_iterations)
registration_method.AddCommand(sitk.sitkIterationEvent, lambda: plot_values(registration_method))

final_transform = registration_method.Execute(sitk.Cast(fixed_image, sitk.sitkFloat32),
                                              sitk.Cast(moving_image, sitk.sitkFloat32))


I am working on the file "61_Registration_Introduction_Continued.ipynb".
The first task is:
Final registration
Version 1

 Single scale (not using image pyramid).

 Initial transformation is not modified in place.

Illustrate the need for scaling the step size differently for each parameter:

 SetOptimizerScalesFromIndexShift - estimated from maximum shift of voxel indexes (only use if data is isotropic).

 SetOptimizerScalesFromPhysicalShift - estimated from maximum shift of physical locations of voxels.

 SetOptimizerScalesFromJacobian - estimated from the averaged squared norm of the Jacobian w.r.t. parameters.

Look at the optimizer's stopping condition to ensure we have not terminated prematurely.

Can you write Python code for it?

registration_method = sitk.ImageRegistrationMethod()

registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)

registration_method.SetInterpolator(sitk.sitkLinear)

registration_method.SetOptimizerAsGradientDescent(learningRate=1.0, numberOfIterations=100)
# Scale the step size differently for each parameter, this is critical!!!
registration_method.SetOptimizerScalesFromPhysicalShift()

registration_method.SetInitialTransform(initial_transform, inPlace=False)

registration_method.AddCommand(sitk.sitkStartEvent, registration_callbacks.metric_start_plot)
registration_method.AddCommand(sitk.sitkEndEvent, registration_callbacks.metric_end_plot)
registration_method.AddCommand(sitk.sitkIterationEvent,
                               lambda: registration_callbacks.metric_plot_values(registration_method))

final_transform_v1 = registration_method.Execute(sitk.Cast(fixed_image, sitk.sitkFloat32),
                                                 sitk.Cast(moving_image, sitk.sitkFloat32))

print('Optimizer\'s stopping condition, {0}'.format(registration_method.GetOptimizerStopConditionDescription()))
print('Final metric value: {0}'.format(registration_method.GetMetricValue()))

# Save moving image after registration and view overlap using external viewer.
save_transform_and_image(final_transform_v1, fixed_image, moving_image, os.path.join(OUTPUT_DIR, "finalAlignment-v1"))


I am working on the file "61_Registration_Introduction_Continued.ipynb".
The first task is:
Initial Alignment
A reasonable guesstimate for the initial translational alignment can be obtained by using
the CenteredTransformInitializer (functional interface to the CenteredTransformInitializerFilter).
The resulting transformation is centered with respect to the fixed image and the
translation aligns the centers of the two images. There are two options for
defining the centers of the images, either the physical centers
of the two data sets (GEOMETRY), or the centers defined by the intensity
moments (MOMENTS).
Two things to note about this filter, it requires the fixed and moving image
have the same type even though it is not algorithmically required, and its
return type is the generic SimpleITK.Transform.
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(sitk.Cast(fixed_image,moving_image.GetPixelIDValue()),
                                                      moving_image,
                                                      sitk.Euler3DTransform(),
                                                      sitk.CenteredTransformInitializerFilter.GEOMETRY)

# Save moving image after initial transform and view overlap using external viewer.
save_transform_and_image(initial_transform, fixed_image, moving_image, os.path.join(OUTPUT_DIR, "initialAlignment"))


I am working on the file "61_Registration_Introduction_Continued.ipynb".
The first task is:
Loading Data
In this notebook we will work with CT and MR scans of the CIRS 057A multi-modality abdominal phantom. The scans are multi-slice DICOM images. The data is stored in a zip archive which is automatically retrieved and extracted when we request a file which is part of the archive.
Can you write Python code for it?

data_directory = os.path.dirname(fdata("CIRS057A_MR_CT_DICOM/readme.txt"))

# 'selected_series_moving/fixed' will be updated by the interact function.
selected_series_fixed = ''
selected_series_moving = ''

# Directory contains multiple DICOM studies/series, store the file names
# in dictionary with the key being the seriesID.
reader = sitk.ImageSeriesReader()
series_file_names = {}
series_IDs = reader.GetGDCMSeriesIDs(data_directory) #list of all series

if series_IDs: #check that we have at least one series
    for series in series_IDs:
        series_file_names[series] = reader.GetGDCMSeriesFileNames(data_directory, series)
    interact(DICOM_series_dropdown_callback, fixed_image=series_IDs, moving_image =series_IDs, series_dictionary=fixed(series_file_names));
else:
    print('This is surprising, data directory does not contain any DICOM series.')

# Actually read the data based on the user's selection.
reader.SetFileNames(series_file_names[selected_series_fixed])
fixed_image = reader.Execute()

reader.SetFileNames(series_file_names[selected_series_moving])
moving_image = reader.Execute()

# Save images to file and view overlap using external viewer.
sitk.WriteImage(fixed_image, os.path.join(OUTPUT_DIR, "fixedImage.mha"))
sitk.WriteImage(moving_image, os.path.join(OUTPUT_DIR, "preAlignment.mha"))


I am working on the file "61_Registration_Introduction_Continued.ipynb".
The first task is:
Look at the final transformation, what type is it?
Can you write Python code for it?


I am working on the file "61_Registration_Introduction_Continued.ipynb".
The first task is:
Look at the transformation, what type is it?
Can you write Python code for it?


I am working on the file "61_Registration_Introduction_Continued.ipynb".
The first task is:
Utility functions
A number of utility functions, saving a transform and corresponding resampled image, callback for selecting a
DICOM series from several series found in the same directory.
Can you write Python code for it?

def save_transform_and_image(transform, fixed_image, moving_image, outputfile_prefix):
    """
    Write the given transformation to file, resample the moving_image onto the fixed_images grid and save the
    result to file.

    Args:
        transform (SimpleITK Transform): transform that maps points from the fixed image coordinate system to the moving.
        fixed_image (SimpleITK Image): resample onto the spatial grid defined by this image.
        moving_image (SimpleITK Image): resample this image.
        outputfile_prefix (string): transform is written to outputfile_prefix.tfm and resampled image is written to
                                    outputfile_prefix.mha.
    """
    resample = sitk.ResampleImageFilter()
    resample.SetReferenceImage(fixed_image)

    # SimpleITK supports several interpolation options, we go with the simplest that gives reasonable results.
    resample.SetInterpolator(sitk.sitkLinear)
    resample.SetTransform(transform)
    sitk.WriteImage(resample.Execute(moving_image), outputfile_prefix+'.mha')
    sitk.WriteTransform(transform, outputfile_prefix+'.tfm')

def DICOM_series_dropdown_callback(fixed_image, moving_image, series_dictionary):
    """
    Callback from dropbox which selects the two series which will be used for registration.
    The callback prints out some information about each of the series from the meta-data dictionary.
    For a list of all meta-dictionary tags and their human readable names see DICOM standard part 6,
    Data Dictionary (http://medical.nema.org/medical/dicom/current/output/pdf/part06.pdf)
    """
    # The callback will update these global variables with the user selection.
    global selected_series_fixed
    global selected_series_moving

    img_fixed = sitk.ReadImage(series_dictionary[fixed_image][0])
    img_moving = sitk.ReadImage(series_dictionary[moving_image][0])

    # There are many interesting tags in the DICOM data dictionary, display a selected few.
   ...


I am working on the file "61_Registration_Introduction_Continued.ipynb".
The first task is:
Version 1.1
The previous example illustrated the use of the ITK v4 registration framework in an ITK v3 manner. We only referred to a single transformation which was what we optimized.
In ITK v4 the registration method accepts three transformations (if you look at the diagram above you will only see two transformations, Moving transform represents $T_{opt} \circ T_m$):

SetInitialTransform, $T_{opt}$ - composed with the moving initial transform, maps points from the virtual image domain to the moving image domain, modified during optimization.

SetFixedInitialTransform $T_f$- maps points from the virtual image domain to the fixed image domain, never modified.

SetMovingInitialTransform $T_m$- maps points from the virtual image domain to the moving image domain, never modified.

The transformation that maps points from the fixed to moving image domains is thus: $^M\mathbf{p}  = T_{opt}(T_m(T_f^{-1}(^F\mathbf{p})))$
We now modify the previous example to use $T_{opt}$ and $T_m$.
Can you write Python code for it?

registration_method = sitk.ImageRegistrationMethod()
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)
registration_method.SetInterpolator(sitk.sitkLinear)
registration_method.SetOptimizerAsGradientDescent(learningRate=1.0, numberOfIterations=100)
registration_method.SetOptimizerScalesFromPhysicalShift()

# Set the initial moving and optimized transforms.
optimized_transform = sitk.Euler3DTransform()
registration_method.SetMovingInitialTransform(initial_transform)
registration_method.SetInitialTransform(optimized_transform)

registration_method.AddCommand(sitk.sitkStartEvent, registration_callbacks.metric_start_plot)
registration_method.AddCommand(sitk.sitkEndEvent, registration_callbacks.metric_end_plot)
registration_method.AddCommand(sitk.sitkIterationEvent,
                               lambda: registration_callbacks.metric_plot_values(registration_method))

registration_method.Execute(sitk.Cast(fixed_image, sitk.sitkFloat32),
                            sitk.Cast(moving_image, sitk.sitkFloat32))

# Need to compose the transformations after registration.
final_transform_v11 = sitk.Transform(optimized_transform)
final_transform_v11.AddTransform(initial_transform)

print('Optimizer\'s stopping condition, {0}'.format(registration_method.GetOptimizerStopConditionDescription()))
print('Final metric value: {0}'.format(registration_method.GetMetricValue()))

# Save moving image after registration and view overlap using external viewer.
save_transform_and_image(final_transform_v11, fixed_image, moving_image, os.path.join(OUTPUT_DIR, "finalAlignment-v1.1"))


I am working on the file "620_HW8.ipynb".
The first task is:
Choose Corpus and Find Unique Words
I chose the Emma corpus from the nltk package. I am going to define unique words as the set of distinct alphabetic strings in the corpus, and I will remove common stop words such as 'a' and 'the'. This will also remove numbers and punctuation from the corpus. The length of the set is taken to survey the number of unique words.
Can you write Python code for it?

emma = nltk.corpus.gutenberg.words('austen-emma.txt')

# strip punctuation and numerics using isalpha() method
emma = [w for w in emma if w.isalpha()]
# strip out stop words
from nltk.corpus import stopwords
emma = [w for w in emma if w not in stopwords.words('english')]

# How many total unique words are in the corpus
emma_unique = set(emma)
len(emma_unique)


I am working on the file "620_HW8.ipynb".
The first task is:
Most Common Words and building a Frequency Distribution
Here we build a frequency distribution from the corpus and isolate the 200 most common words. This method returns the a sort list of tuples that is then loaded into a dataframe in order to calculate relative frequencies.
Can you write Python code for it?

# build the frequency distribution using FreqDist()
freq_emma = nltk.FreqDist(emma)

# make a dataframe to produce relative frequencies - top 200
emma_top200 = pd.DataFrame(freq_emma.most_common(200),columns=['word','count'])
emma_top200['rel_freq'] = emma_top['count']/float(len(emma))
emma_top200.head(10)


I am working on the file "620_HW8.ipynb".
The first task is:
The following barplot shows the relative frequencies of all 200 of the most frequent unique words.
Can you write Python code for it?

g = sns.barplot(x=emma_top200.word, y=emma_top200.rel_freq)


I am working on the file "620_HW8.ipynb".
The first task is:
We want to find out the number of most common unique words that make up approximately 50% of the dataset. By plotting the cumulative distribution we can see that approximately 250 words accounts for 50% of all words in the dataset. This is confirmed by summing the first 250 indexes of relative frequencies.
Can you write Python code for it?

# top 500
emma_top500 = pd.DataFrame(freq_emma.most_common(500),columns=['word','count'])
emma_top500['rel_freq'] = emma_top['count']/float(len(emma))

len(emma)/2.0   ## half of all words

sum(emma_top500[:250]['rel_freq']) # The first 250 words account for approximately half of all words

freq_emma.plot(250, cumulative=True)


I am working on the file "620_project2_101716.ipynb".
The first task is:
We now have two islands of the projected authors and journals. Examining the degree centrality will help reveal which nodes are the key to the networks.
Can you write Python code for it?

# degree centrality of both island clusters
a_degree = nx.degree_centrality(a_sg_island)
j_degree = nx.degree_centrality(j_sg_island)
pd.DataFrame.from_dict(a_degree,orient='index').sort_values(0,ascending=False).head()

pd.DataFrame.from_dict(j_degree,orient='index').sort_values(0,ascending=False).head()


I am working on the file "620_spam.ipynb".
The first task is:
Random forest
http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
Can you write Python code for it?

# Random forest - train
from sklearn import ensemble

rf = ensemble.RandomForestClassifier(criterion="entropy", random_state=88)
rf_fit = rf.fit(trainSetVars, trainSetClass)

rf_train = rf_fit.predict(trainSetVars)
model_summary(trainSetClass, rf_train)

# Random forest - test set
rf_test = rf_fit.predict(testSetVars)
model_summary(testSetClass, rf_test)

# Random forest feature importance
featImp(rf_fit, testSetVars)


I am working on the file "620_spam.ipynb".
The first task is:
Split data into two datasets: training & testing
Can you write Python code for it?

# Split into test, train, validate
percTrain = 0.7
percVal = 0.15
percTest = 0.15
N = len(spam)
trainNum = int(percTrain * N)
valNum = int(percVal * N)
testNum = N - trainNum - valNum

# Check targets
print "Training target: %d" %trainNum
print "Validation target: %d" %valNum
print "Testing target: %d" %testNum
print "Total: %d" %(trainNum + valNum + testNum)

# Split it up
from sklearn.cross_validation import train_test_split
trainSet, testSet = train_test_split(spam, test_size=testNum, random_state=8)
trainSet, valSet = train_test_split(trainSet, test_size=valNum, random_state=88)

# Check lengths
print "Training set: %d" %len(trainSet)
print "Validation set: %d" %len(valSet)
print "Testing set: %d" %len(testSet)
print "Total: %d" %(len(trainSet) + len(valSet) + len(testSet))


I am working on the file "62_Registration_Tuning.ipynb".
The first task is:
In some cases visual comparison of the registration errors using the same scale is not informative, as seen above [all points are grey/black]. We therefor set the color scale to the min-max error range found in the current data and not the range from the previous stage.
Can you write Python code for it?

final_errors_mean, final_errors_std, _, final_errors_max,_ = ru.registration_errors(final_transform_single_scale, fixed_points, moving_points, display_errors=True)


I am working on the file "62_Registration_Tuning.ipynb".
The first task is:
Initial Alignment
We use the CenteredTransformInitializer. Should we use the GEOMETRY based version or the MOMENTS based one?
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(sitk.Cast(fixed_image,moving_image.GetPixelIDValue()),
                                                      moving_image,
                                                      sitk.Euler3DTransform(),
                                                      sitk.CenteredTransformInitializerFilter.GEOMETRY)

initial_errors_mean, initial_errors_std, initial_errors_min, initial_errors_max, _ = ru.registration_errors(initial_transform, fixed_points, moving_points, min_err=pre_errors_min, max_err=pre_errors_max, display_errors=True)
print('After initialization, errors (TRE) in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(initial_errors_mean, initial_errors_std, initial_errors_max))


I am working on the file "62_Registration_Tuning.ipynb".
The first task is:
Read the RIRE data and generate a larger point set as a reference
Can you write Python code for it?

fixed_image =  sitk.ReadImage(fdata("training_001_ct.mha"), sitk.sitkFloat32)
moving_image = sitk.ReadImage(fdata("training_001_mr_T1.mha"), sitk.sitkFloat32)
fixed_fiducial_points, moving_fiducial_points = ru.load_RIRE_ground_truth(fdata("ct_T1.standard"))

# Estimate the reference_transform defined by the RIRE fiducials and check that the FRE makes sense (low)
R, t = ru.absolute_orientation_m(fixed_fiducial_points, moving_fiducial_points)
reference_transform = sitk.Euler3DTransform()
reference_transform.SetMatrix(R.flatten())
reference_transform.SetTranslation(t)
reference_errors_mean, reference_errors_std, _, reference_errors_max,_ = ru.registration_errors(reference_transform, fixed_fiducial_points, moving_fiducial_points)
print('Reference data errors (FRE) in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(reference_errors_mean, reference_errors_std, reference_errors_max))

# Generate a reference dataset from the reference transformation
# (corresponding points in the fixed and moving images).
fixed_points = ru.generate_random_pointset(image=fixed_image, num_points=100)
moving_points = [reference_transform.TransformPoint(p) for p in fixed_points]

# Compute the TRE prior to registration.
pre_errors_mean, pre_errors_std, pre_errors_min, pre_errors_max, _ = ru.registration_errors(sitk.Euler3DTransform(), fixed_points, moving_points, display_errors = True)
print('Before registration, errors (TRE) in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(pre_errors_mean, pre_errors_std, pre_errors_max))


I am working on the file "62_Registration_Tuning.ipynb".
The first task is:
Registration
Possible choices for simple rigid multi-modality registration framework (300 component combinations, in addition to parameter settings for each of the components):

Similarity metric, 2 options (Mattes MI, JointHistogram MI):

Number of histogram bins.

Sampling strategy, 3 options (NONE, REGULAR, RANDOM)

Sampling percentage.

Interpolator, 10 options (sitkNearestNeighbor, sitkLinear, sitkGaussian, sitkBSpline,...)

Optimizer, 5 options (GradientDescent, GradientDescentLineSearch, RegularStepGradientDescent...):

Number of iterations.

learning rate (step size along parameter space traversal direction).

In this example we will plot the similarity metric's value and more importantly the TREs for our reference data. A good choice for the former should be reflected by the later. That is, the TREs should go down as the similarity measure value goes down (not necessarily at the same rates).
Finally, we are also interested in timing our registration. Ipython allows us to do this with minimal effort using the timeit cell magic (Ipython has a set of predefined functions that use a command line syntax, and are referred to as magic functions).
Can you write Python code for it?

#%%timeit -r1 -n1
# to time this cell uncomment the line above
#the arguments to the timeit magic specify that this cell should only be run once. running it multiple
#times to get performance statistics is also possible, but takes time. if you want to analyze the accuracy
#results from multiple runs you will have to modify the code to save them instead of just printing them out.

registration_method = sitk.ImageRegistrationMethod()
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)
registration_method.SetInterpolator(sitk.sitkNearestNeighbor) #2. Replace with sitkLinear
registration_method.SetOptimizerAsGradientDescent(learningRate=1.0, numberOfIterations=100) #1. Increase to 1000
registration_method.SetOptimizerScalesFromPhysicalShift()

# Don't optimize in-place, we would like to run this cell multiple times
registration_method.SetInitialTransform(initial_transform, inPlace=False)

# Add callbacks which will display the similarity measure value and the reference data during the registration process
registration_method.AddCommand(sitk.sitkStartEvent, rc.metric_and_reference_start_plot)
registration_method.AddCommand(sitk.sitkEndEvent, rc.metric_and_reference_end_plot)
registration_method.AddCommand(sitk.sitkIterationEvent, lambda: rc.metric_and_reference_plot_values(registration_method, fixed_points, moving_points))

final_transform_single_scale = registration_method.Execute(sitk.Cast(fixed_image, sitk.sitkFloat32),
                                                           sitk.Cast(moving_image, sitk.sitkFloat32))

print('Final metric value: {0}'.format(registration_method.GetMetricValue()))
print('Optimizer\'s stopping condition, {0}'.format(registration_method.GetOptimizerStopConditionDescription()))
final_errors_mean, final_errors_std, _, final_errors_max,_ = ru.registration_errors(final_transform_single_scale, fixed_points, moving_points,...


I am working on the file "62_Registration_Tuning.ipynb".
The first task is:
Sufficient accuracy inside the ROI
Up to this point our accuracy evaluation has ignored the content of the image and is likely overly conservative. We have been looking at the registration errors inside the volume, but not necesserily in the smaller ROI.
To see the difference you will have to comment out the timeit magic in the code above, run it again, and then run the following cell.
Can you write Python code for it?

# Threshold the original fixed, CT, image at 0HU (water), resulting in a binary labeled [0,1] image.
roi = fixed_image> 0

# Our ROI consists of all voxels with a value of 1, now get the bounding box surrounding the head.
label_shape_analysis = sitk.LabelShapeStatisticsImageFilter()
label_shape_analysis.SetBackgroundValue(0)
label_shape_analysis.Execute(roi)
bounding_box = label_shape_analysis.GetBoundingBox(1)

# Bounding box in physical space.
sub_image_min = fixed_image.TransformIndexToPhysicalPoint((bounding_box[0],bounding_box[1], bounding_box[2]))
sub_image_max = fixed_image.TransformIndexToPhysicalPoint((bounding_box[0]+bounding_box[3]-1,
                                                           bounding_box[1]+bounding_box[4]-1,
                                                           bounding_box[2]+bounding_box[5]-1))
# Only look at the points inside our bounding box.
sub_fixed_points = []
sub_moving_points = []
for fixed_pnt, moving_pnt in zip(fixed_points, moving_points):
    if sub_image_min[0]<=fixed_pnt[0]<=sub_image_max[0] and \
       sub_image_min[1]<=fixed_pnt[1]<=sub_image_max[1] and \
       sub_image_min[2]<=fixed_pnt[2]<=sub_image_max[2] :
        sub_fixed_points.append(fixed_pnt)
        sub_moving_points.append(moving_pnt)

final_errors_mean, final_errors_std, _, final_errors_max,_ = ru.registration_errors(final_transform, sub_fixed_points, sub_moving_points, True)
print('After registration, errors in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(final_errors_mean, final_errors_std, final_errors_max))


I am working on the file "63_Registration_Initialization.ipynb".
The first task is:
Exhaustive optimizer
The exhaustive optimizer evaluates the similarity measure using a grid overlaid on the parameter space.
The grid is centered on the parameter values set by the SetInitialTransform, and the location of its vertices are determined by the numberOfSteps, stepLength and optimizer scales. To quote the documentation of this class: "a side of the region is stepLength(2numberOfSteps[d]+1)*scaling[d]."
Using this approach we have superfluous evaluations (15 evaluations corresponding to 3 values for rotations around the x axis and five for rotation around the z axis, as compared to the 8 evaluations using the MetricEvaluate method).
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(fixed_image,
                                                      modified_moving_image,
                                                      sitk.Euler3DTransform(),
                                                      sitk.CenteredTransformInitializerFilter.GEOMETRY)
registration_method = sitk.ImageRegistrationMethod()
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)
registration_method.SetInterpolator(sitk.sitkLinear)
# The order of parameters for the Euler3DTransform is [angle_x, angle_y, angle_z, t_x, t_y, t_z]. The parameter
# sampling grid is centered on the initial_transform parameter values, that are all zero for the rotations. Given
# the number of steps and their length and optimizer scales we have:
# angle_x = -pi, 0, pi
# angle_y = 0
# angle_z = -pi, -pi/2, 0, pi/2, pi
registration_method.SetOptimizerAsExhaustive(numberOfSteps=[1,0,2,0,0,0], stepLength = np.pi)
registration_method.SetOptimizerScales([1,1,0.5,1,1,1])

#Perform the registration in-place so that the initial_transform is modified.
registration_method.SetInitialTransform(initial_transform, inPlace=True)
registration_method.Execute(fixed_image, modified_moving_image)

final_transform = multires_registration(fixed_image, modified_moving_image, initial_transform)


I am working on the file "63_Registration_Initialization.ipynb".
The first task is:
Loading Data
Can you write Python code for it?

data_directory = os.path.dirname(fdata("CIRS057A_MR_CT_DICOM/readme.txt"))

fixed_series_ID = "1.2.840.113619.2.290.3.3233817346.783.1399004564.515"
moving_series_ID = "1.3.12.2.1107.5.2.18.41548.30000014030519285935000000933"

reader = sitk.ImageSeriesReader()
fixed_image = sitk.ReadImage(reader.GetGDCMSeriesFileNames(data_directory, fixed_series_ID), sitk.sitkFloat32)
moving_image = sitk.ReadImage(reader.GetGDCMSeriesFileNames(data_directory, moving_series_ID), sitk.sitkFloat32)

# To provide a reasonable display we need to window/level the images. By default we could have used the intensity
# ranges found in the images [SimpleITK's StatisticsImageFilter], but these are not the best values for viewing.
# Using an external viewer we identified the following settings.
fixed_intensity_range = (-1183,544)
moving_intensity_range = (0,355)

interact(lambda image1_z, image2_z, image1, image2,:ru.display_scalar_images(image1_z, image2_z, image1, image2,
                                                                             fixed_intensity_range,
                                                                             moving_intensity_range,
                                                                             'fixed image',
                                                                             'moving image'),
         image1_z=(0,fixed_image.GetSize()[2]-1),
         image2_z=(0,moving_image.GetSize()[2]-1),
         image1 = fixed(fixed_image),
         image2=fixed(moving_image));


I am working on the file "63_Registration_Initialization.ipynb".
The first task is:
Register using heuristic initialization approach (using multiple orientations)
As we want to account for significant orientation differences due to erroneous patient position (HFS...) we evaluate the similarity measure at locations corresponding to the various orientation differences. This can be done in two ways which will be illustrated below:

Use the ImageRegistrationMethod.MetricEvaluate() method.

Use the Exhaustive optimizer.

The former approach is more computationally intensive as it constructs and configures a metric object each time it is invoked. It is therefore more appropriate for use if the set of parameter values we want to evaluate are not on a rectilinear grid in the parameter space. The latter approach is appropriate if the set of parameter values are on a rectilinear grid, in which case the approach is more computationally efficient.
In both cases we use the CenteredTransformInitializer to obtain the initial translation.
MetricEvaluate
To use the MetricEvaluate method we create a ImageRegistrationMethod, set its metric and interpolator. We then iterate over all parameter settings, set the initial transform and evaluate the metric. The minimal similarity measure value corresponds to the best paramter settings.
Can you write Python code for it?

# Dictionary with all the orientations we will try. We omit the identity (x=0, y=0, z=0) as we always use it. This
# set of rotations is arbitrary. For a complete grid coverage we would have 64 entries (0,pi/2,pi,1.5pi for each angle).
all_orientations = {'x=0, y=0, z=90': (0.0,0.0,np.pi/2.0),
                    'x=0, y=0, z=-90': (0.0,0.0,-np.pi),
                    'x=0, y=0, z=180': (0.0,0.0,np.pi),
                    'x=180, y=0, z=0': (np.pi,0.0,0.0),
                    'x=180, y=0, z=90': (np.pi,0.0,np.pi/2.0),
                    'x=180, y=0, z=-90': (np.pi,0.0,-np.pi/2.0),
                    'x=180, y=0, z=180': (np.pi,0.0,np.pi)}

# Registration framework setup.
registration_method = sitk.ImageRegistrationMethod()
registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
registration_method.SetMetricSamplingPercentage(0.01)
registration_method.SetInterpolator(sitk.sitkLinear)

# Evaluate the similarity metric using the eight possible orientations, translation remains the same for all.
initial_transform = sitk.Euler3DTransform(sitk.CenteredTransformInitializer(fixed_image,
                                                                            modified_moving_image,
                                                                            sitk.Euler3DTransform(),
                                                                            sitk.CenteredTransformInitializerFilter.GEOMETRY))
registration_method.SetInitialTransform(initial_transform, inPlace=False)
best_orientation = (0.0,0.0,0.0)
best_similarity_value = registration_method.MetricEvaluate(fixed_image, modified_moving_image)

# Iterate over all other rotation parameter settings.
for key, orientation in all_orientations.items():
    initial_transform.SetRotation(*orientation)
    registration_method.SetInitialTransform(initial_transform)
    current_similarity_value = registration_method.MetricEvaluate(fixed_image,...


I am working on the file "63_Registration_Initialization.ipynb".
The first task is:
Visually evaluate our results:
Can you write Python code for it?

moving_resampled = sitk.Resample(modified_moving_image, fixed_image, final_transform, sitk.sitkLinear, 0.0, moving_image.GetPixelIDValue())

interact(ru.display_images_with_alpha, image_z=(0,fixed_image.GetSize()[2]), alpha=(0.0,1.0,0.05),
         image1 = fixed(sitk.IntensityWindowing(fixed_image, fixed_intensity_range[0], fixed_intensity_range[1])),
         image2=fixed(sitk.IntensityWindowing(moving_resampled, moving_intensity_range[0], moving_intensity_range[1])));


I am working on the file "64_Registration_Memory_Time_Tradeoff.ipynb".
The first task is:
Another option for resampling an image, without any transformation, is to use the ExpandImageFilter or
in its functional form SimpleITK::Expand. This filter accepts the interpolation method and an integral expansion factor. This is less flexible than the resample filter as we have less control over the resulting image's spacing.
On the other hand this requires less effort from the developer, a single line of code as compared to the cell above:
resampled_moving_image = sitk.Expand(moving_image,
                                     [int(original_s/new_s + 0.5) for original_s, new_s in zip(original_spacing, new_spacing)],
                                     sitk.sitkBSpline)
Registration
Initial Alignment
We will use the same initial alignment for both registrations.
Can you write Python code for it?

initial_transform = sitk.CenteredTransformInitializer(fixed_image,
                                                      moving_image,
                                                      sitk.Euler3DTransform(),
                                                      sitk.CenteredTransformInitializerFilter.GEOMETRY)


I am working on the file "64_Registration_Memory_Time_Tradeoff.ipynb".
The first task is:
Invest time and memory in exchange for future time savings
We now resample our moving image to a finer spatial resolution.
Can you write Python code for it?

# Isotropic voxels with 1mm spacing.
new_spacing = [1.0]*moving_image.GetDimension()

# Create resampled image using new spacing and size.
original_size = moving_image.GetSize()
original_spacing = moving_image.GetSpacing()
resampled_image_size = [int(spacing/new_s*size)
                        for spacing, size, new_s in zip(original_spacing, original_size, new_spacing)]
resampled_moving_image = sitk.Image(resampled_image_size, moving_image.GetPixelIDValue())
resampled_moving_image.SetSpacing(new_spacing)
resampled_moving_image.SetOrigin(moving_image.GetOrigin())
resampled_moving_image.SetDirection(moving_image.GetDirection())

# Resample original image using identity transform and the BSpline interpolator.
resample = sitk.ResampleImageFilter()
resample.SetReferenceImage(resampled_moving_image)
resample.SetInterpolator(sitk.sitkBSpline)
resample.SetTransform(sitk.Transform())
resampled_moving_image = resample.Execute(moving_image)

print('Original image size and spacing: {0} {1}'.format(original_size, original_spacing))
print('Resampled image size and spacing: {0} {1}'.format(resampled_moving_image.GetSize(),
                                                         resampled_moving_image.GetSpacing()))
print('Memory ratio: 1 : {0}'.format((np.array(resampled_image_size)/np.array(original_size).astype(float)).prod()))


I am working on the file "64_Registration_Memory_Time_Tradeoff.ipynb".
The first task is:
Load data
We use the the training data from the Retrospective Image Registration Evaluation (RIRE) project.
The RIRE reference, ground truth, data consists of a set of corresponding points in the fixed and moving coordinate systems. These points were obtained from fiducials embedded in the patient's skull and are thus sparse (eight points). We use these to compute the rigid transformation between the two coordinate systems, and then generate a dense reference. This generated reference data is more similar to the data you would use for registration evaluation.
Can you write Python code for it?

fixed_image =  sitk.ReadImage(fdata("training_001_ct.mha"), sitk.sitkFloat32)
moving_image = sitk.ReadImage(fdata("training_001_mr_T1.mha"), sitk.sitkFloat32)
fixed_fiducial_points, moving_fiducial_points = ru.load_RIRE_ground_truth(fdata("ct_T1.standard"))

R, t = ru.absolute_orientation_m(fixed_fiducial_points, moving_fiducial_points)
reference_transform = sitk.Euler3DTransform()
reference_transform.SetMatrix(R.flatten())
reference_transform.SetTranslation(t)

# Generate a reference dataset from the reference transformation (corresponding points in the fixed and moving images).
fixed_points = ru.generate_random_pointset(image=fixed_image, num_points=1000)
moving_points = [reference_transform.TransformPoint(p) for p in fixed_points]

interact(lambda image1_z, image2_z, image1, image2:ru.display_scalar_images(image1_z, image2_z, image1, image2),
         image1_z=(0,fixed_image.GetSize()[2]-1),
         image2_z=(0,moving_image.GetSize()[2]-1),
         image1 = fixed(fixed_image),
         image2=fixed(moving_image));


I am working on the file "64_Registration_Memory_Time_Tradeoff.ipynb".
The first task is:
Original Resolution
For this registration we use the original resolution and linear interpolation.
Can you write Python code for it?

%%timeit -r1 -n1
# The arguments to the timeit magic specify that this cell should only be run once.

# We define this variable as global so that it is accessible outside of the cell (timeit wraps the code in the cell
# making all variables local, unless explicitly declared global).
global original_resolution_errors

final_transform, optimizer_termination = register_images(fixed_image, moving_image, initial_transform, sitk.sitkLinear)
final_errors_mean, final_errors_std, _, final_errors_max, original_resolution_errors = ru.registration_errors(final_transform, fixed_points, moving_points)

print(optimizer_termination)
print('After registration, errors in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(final_errors_mean, final_errors_std, final_errors_max))


I am working on the file "65_Registration_FFD.ipynb".
The first task is:
Free Form Deformation
This function will align the fixed and moving images using a FFD. If given a mask, the similarity metric will be evaluated using points sampled inside the mask. If given fixed and moving points the similarity metric value and the target registration errors will be displayed during registration.
As this notebook performs intra-modal registration, we use the MeanSquares similarity metric (simple to compute and appropriate for the task).
Can you write Python code for it?

def bspline_intra_modal_registration(fixed_image, moving_image, fixed_image_mask=None, fixed_points=None, moving_points=None):

    registration_method = sitk.ImageRegistrationMethod()

    # Determine the number of Bspline control points using the physical spacing we want for the control grid.
    grid_physical_spacing = [50.0, 50.0, 50.0] # A control point every 50mm
    image_physical_size = [size*spacing for size,spacing in zip(fixed_image.GetSize(), fixed_image.GetSpacing())]
    mesh_size = [int(image_size/grid_spacing + 0.5) \
                 for image_size,grid_spacing in zip(image_physical_size,grid_physical_spacing)]

    initial_transform = sitk.BSplineTransformInitializer(image1 = fixed_image,
                                                         transformDomainMeshSize = mesh_size, order=3)
    registration_method.SetInitialTransform(initial_transform)

    registration_method.SetMetricAsMeanSquares()
    # Settings for metric sampling, usage of a mask is optional. When given a mask the sample points will be
    # generated inside that region. Also, this implicitly speeds things up as the mask is smaller than the
    # whole image.
    registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
    registration_method.SetMetricSamplingPercentage(0.01)
    if fixed_image_mask:
        registration_method.SetMetricFixedMask(fixed_image_mask)

    # Multi-resolution framework.
    registration_method.SetShrinkFactorsPerLevel(shrinkFactors = [4,2,1])
    registration_method.SetSmoothingSigmasPerLevel(smoothingSigmas=[2,1,0])
    registration_method.SmoothingSigmasAreSpecifiedInPhysicalUnitsOn()

    registration_method.SetInterpolator(sitk.sitkLinear)
    registration_method.SetOptimizerAsLBFGSB(gradientConvergenceTolerance=1e-5, numberOfIterations=100)

    # If corresponding points in the fixed and moving image are given then we display the similarity metric
    # and the TRE during the registration.
    if fixed_points and moving_points:
  ...


I am working on the file "65_Registration_FFD.ipynb".
The first task is:
Geting to know your data
While the POPI site states that image number 1 is end inspiration, and visual inspection seems to suggest this is correct, we should probably take a look at the lung volumes to ensure that what we expect is indeed what is happening.
Which image is end inspiration and which end expiration?
Can you write Python code for it?

label_shape_statistics_filter = sitk.LabelShapeStatisticsImageFilter()

for i, mask in enumerate(masks):
    label_shape_statistics_filter.Execute(mask)
    print('Lung volume in image {0} is {1} liters.'.format(i,0.000001*label_shape_statistics_filter.GetPhysicalSize(lung_label)))


I am working on the file "65_Registration_FFD.ipynb".
The first task is:
Perform Registration
The following cell allows you to select the images used for registration, runs the registration, and afterwards computes statstics comparing the target registration errors before and after registration and displays a histogram of the TREs.
To time the registration, uncomment the timeit magic.
<b>Note</b>: this creates a seperate scope for the cell. Variables set inside the cell, specifically tx, will become local variables and thus their value is not available in other cells.
Can you write Python code for it?

#%%timeit -r1 -n1

# Select the fixed and moving images, valid entries are in [0,9].
fixed_image_index = 0
moving_image_index = 7

tx = bspline_intra_modal_registration(fixed_image = images[fixed_image_index],
                                      moving_image = images[moving_image_index],
                                      fixed_image_mask = (masks[fixed_image_index] == lung_label),
                                      fixed_points = points[fixed_image_index],
                                      moving_points = points[moving_image_index]
                                     )
initial_errors_mean, initial_errors_std, _, initial_errors_max, initial_errors = ru.registration_errors(sitk.Euler3DTransform(), points[fixed_image_index], points[moving_image_index])
final_errors_mean, final_errors_std, _, final_errors_max, final_errors = ru.registration_errors(tx, points[fixed_image_index], points[moving_image_index])

plt.hist(initial_errors, bins=20, alpha=0.5, label='before registration', color='blue')
plt.hist(final_errors, bins=20, alpha=0.5, label='after registration', color='green')
plt.legend()
plt.title('TRE histogram');
print('Initial alignment errors in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(initial_errors_mean, initial_errors_std, initial_errors_max))
print('Final alignment errors in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(final_errors_mean, final_errors_std, final_errors_max))


I am working on the file "65_Registration_FFD.ipynb".
The first task is:
Utilities
Load utilities that are specific to the POPI data, functions for loading ground truth data, display and the labels for masks.
Can you write Python code for it?


I am working on the file "66_Registration_Demons.ipynb".
The first task is:
Loading Data
Load all of the images, masks and point data into corresponding lists. If the data is not available locally it will be downloaded from the original remote repository.
Take a look at the images. According to the documentation on the POPI site, volume number one corresponds to end inspiration (maximal air volume).
Can you write Python code for it?

images = []
masks = []
points = []
for i in range(0,10):
    image_file_name = 'POPI/meta/{0}0-P.mhd'.format(i)
    mask_file_name = 'POPI/masks/{0}0-air-body-lungs.mhd'.format(i)
    points_file_name = 'POPI/landmarks/{0}0-Landmarks.pts'.format(i)
    images.append(sitk.ReadImage(fdata(image_file_name), sitk.sitkFloat32)) #read and cast to format required for registration
    masks.append(sitk.ReadImage(fdata(mask_file_name)))
    points.append(read_POPI_points(fdata(points_file_name)))

interact(display_coronal_with_overlay, temporal_slice=(0,len(images)-1),
         coronal_slice = (0, images[0].GetSize()[1]-1),
         images = fixed(images), masks = fixed(masks),
         label=fixed(lung_label), window_min = fixed(-1024), window_max=fixed(976));


I am working on the file "66_Registration_Demons.ipynb".
The first task is:
Now we will use our newly minted multiscale framework to perform registration with the Demons filters. Some things you can easily try out by editing the code below:
1. Is there really a need for multiscale - just call the multiscale_demons method without the shrink_factors and smoothing_sigmas parameters.
2. Which Demons filter should you use - configure the other filters and see if our selection is the best choice (accuracy/time).
Can you write Python code for it?

# Define a simple callback which allows us to monitor the Demons filter's progress.
def iteration_callback(filter):
    print('\r{0}: {1:.2f}'.format(filter.GetElapsedIterations(), filter.GetMetric()), end='')

fixed_image_index = 0
moving_image_index = 7

# Select a Demons filter and configure it.
demons_filter =  sitk.FastSymmetricForcesDemonsRegistrationFilter()
demons_filter.SetNumberOfIterations(20)
# Regularization (update field - viscous, total field - elastic).
demons_filter.SetSmoothDisplacementField(True)
demons_filter.SetStandardDeviations(2.0)

# Add our simple callback to the registration filter.
demons_filter.AddCommand(sitk.sitkIterationEvent, lambda: iteration_callback(demons_filter))

# Run the registration.
tx = multiscale_demons(registration_algorithm=demons_filter,
                       fixed_image = images[fixed_image_index],
                       moving_image = images[moving_image_index],
                       shrink_factors = [4,2],
                       smoothing_sigmas = [8,4])

# Compare the initial and final TREs.
initial_errors_mean, initial_errors_std, _, initial_errors_max, initial_errors = ru.registration_errors(sitk.Euler3DTransform(), points[fixed_image_index], points[moving_image_index])
final_errors_mean, final_errors_std, _, final_errors_max, final_errors = ru.registration_errors(tx, points[fixed_image_index], points[moving_image_index])

plt.hist(initial_errors, bins=20, alpha=0.5, label='before registration', color='blue')
plt.hist(final_errors, bins=20, alpha=0.5, label='after registration', color='green')
plt.legend()
plt.title('TRE histogram');
print('\nInitial alignment errors in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(initial_errors_mean, initial_errors_std, initial_errors_max))
print('Final alignment errors in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(final_errors_mean, final_errors_std, final_errors_max))


I am working on the file "66_Registration_Demons.ipynb".
The first task is:
Running the Demons registration on this data will take a long time (run it before going home). If you are less interested in accuracy you can switch the optimizer from conjugate gradient to gradient, will run much faster but the results are worse.
Can you write Python code for it?

#%%timeit -r1 -n1
# Uncomment the line above if you want to time the running of this cell.

# Select the fixed and moving images, valid entries are in [0,9]
fixed_image_index = 0
moving_image_index = 7

tx = demons_registration(fixed_image = images[fixed_image_index],
                         moving_image = images[moving_image_index],
                         fixed_points = points[fixed_image_index],
                         moving_points = points[moving_image_index]
                         )
initial_errors_mean, initial_errors_std, _, initial_errors_max, initial_errors = ru.registration_errors(sitk.Euler3DTransform(), points[fixed_image_index], points[moving_image_index])
final_errors_mean, final_errors_std, _, final_errors_max, final_errors = ru.registration_errors(tx, points[fixed_image_index], points[moving_image_index])

plt.hist(initial_errors, bins=20, alpha=0.5, label='before registration', color='blue')
plt.hist(final_errors, bins=20, alpha=0.5, label='after registration', color='green')
plt.legend()
plt.title('TRE histogram');
print('Initial alignment errors in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(initial_errors_mean, initial_errors_std, initial_errors_max))
print('Final alignment errors in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(final_errors_mean, final_errors_std, final_errors_max))