I am working on the file "3 Unsupervised learning/Homework/4 text themes/edit_CookingLDA_PA.ipynb".
The first task is:
Считается, что когерентность хорошо соотносится с человеческими оценками интерпретируемости тем. Поэтому на больших текстовых коллекциях когерентность обычно повышается, если убрать фоновую лексику. Однако в нашем случае этого не произошло.
Изучение влияния гиперпараметра alpha
В этом разделе мы будем работать со второй моделью, то есть той, которая построена по сокращенному корпусу.
Пока что мы посмотрели только на матрицу темы-слова, теперь давайте посмотрим на матрицу темы-документы. Выведите темы для нулевого (или любого другого) документа из корпуса, воспользовавшись методом get_document_topics второй модели:
Can you write Python code for it?

print(ldamodel2.get_document_topics(corpus2[0]))
words_0 = [dictionary2[num] for num, _ in ldamodel2.get_document_topics(corpus2[0])]
words_0


I am working on the file "3 Unsupervised learning/Homework/4 text themes/edit_CookingLDA_PA.ipynb".
The first task is:
Также выведите содержимое переменной .alpha второй модели:
Can you write Python code for it?


I am working on the file "3 Unsupervised learning/Homework/4 text themes/edit_CookingLDA_PA.ipynb".
The first task is:
У вас должно получиться, что документ характеризуется небольшим числом тем. Попробуем поменять гиперпараметр alpha, задающий априорное распределение Дирихле для распределений тем в документах.
Задание 4. Обучите третью модель: используйте сокращенный корпус (corpus2 и dictionary2) и установите параметр alpha=1, passes=5. Не забудьте про фиксацию seed! Выведите темы новой модели для нулевого документа; должно получиться, что распределение над множеством тем практически равномерное. Чтобы убедиться в том, что во второй модели документы описываются гораздо более разреженными распределениями, чем в третьей, посчитайте суммарное количество элементов, превосходящих 0.01, в матрицах темы-документы обеих моделей. Другими словами, запросите темы  модели для каждого документа с параметром minimum_probability=0.01 и просуммируйте число элементов в получаемых массивах. Передайте две суммы (сначала для модели с alpha по умолчанию, затем для модели в alpha=1) в функцию save_answers4.
Can you write Python code for it?

np.random.seed(76543)
# здесь код для построения модели:
# обучение модели
%time ldamodel3 = models.LdaMulticore(corpus2, id2word=dictionary2, alpha=1, num_topics=40, passes=5, workers=1)

print(ldamodel3.get_document_topics(corpus2[0]))
words_0 = [dictionary2[num] for num, _ in ldamodel3.get_document_topics(corpus2[0])]
words_0

count_model2 = 0
count_model3 = 0
for doc in corpus2:
    count_model2 += len(ldamodel2.get_document_topics(doc, minimum_probability=0.01))
    count_model3 += len(ldamodel3.get_document_topics(doc, minimum_probability=0.01))
print('Number of elements with probability higher than 0.2')
print('Model 2 (alpha="symmetric"): %d' % count_model2)
print('Model 3 (alpha=1): %d' % count_model3)

def save_answers4(count_model2, count_model3):
    with open("cooking_LDA_pa_task4.txt", "w") as fout:
        fout.write(" ".join([str(el) for el in [count_model2, count_model3]]))

save_answers4(count_model2, count_model3)

np.random.seed(76543)
# здесь код для построения модели:
# обучение модели
%time ldamodel4 = models.LdaMulticore(corpus2, id2word=dictionary2, alpha=1, passes=5, workers=1)

count_model4 = 0
for doc in corpus2:
    count_model4 += len(ldamodel4.get_document_topics(doc, minimum_probability=0.01))
print('Number of elements with probability higher than 0.2')
print('Model 2 (alpha="symmetric"): %d' % count_model2)
print('Model 3 (alpha=1): %d' % count_model4)

save_answers4(count_model2, count_model4)


I am working on the file "3 Unsupervised learning/Homework/4 text themes/edit_CookingLDA_PA.ipynb".
The first task is:
Фильтрация словаря
В топах тем гораздо чаще встречаются первые три рассмотренных ингредиента, чем последние три. При этом наличие в рецепте курицы, яиц и грибов яснее дает понять, что мы будем готовить, чем наличие соли, сахара и воды. Таким образом, даже в рецептах есть слова, часто встречающиеся в текстах и не несущие смысловой нагрузки, и поэтому их не желательно видеть в темах. Наиболее простой прием борьбы с такими фоновыми элементами — фильтрация словаря по частоте. Обычно словарь фильтруют с двух сторон: убирают очень редкие слова (в целях экономии памяти) и очень частые слова (в целях повышения интерпретируемости тем). Мы уберем только частые слова.
Can you write Python code for it?

import copy
dictionary2 = copy.deepcopy(dictionary)


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Большинство авторов значимы в 1 теме, что логично.
Построение тематической карты авторов
По сути, в матрице $\Phi$, соответствующей модальности авторов, записаны тематические кластеры авторов. Для любого автора мы можем составить его тематический круг - авторов, разбирающихся в той же теме, что и данный. Интересующиеся слушатели могут попробовать выполнить эту процедуру для ученых, читающих лекции на Постнауке, которых они знают (например, на Постнауке есть лекции с К. В. Воронцовым - лектором текущего модуля :)
Составим карту близости авторов по тематике их исследований. Для этого применим метод понижения размерности MDS к тематическим профилям авторов.
Чтобы получить тематический профиль автора, распределение $p(t|a)$, нужно воспользоваться формулой Байеса:
$p(t|a) = \frac {p(a|t) p(t)} {\sum_t' p(a|t') p(t')}$. Все необходимые для этого величины у вас есть и записаны в переменных phi и pt.
Передайте матрицу тематических профилей авторов, записанных по строкам, в метод MDS с n_components=2. Используйте косинусную метрику (она хорошо подходит для поиска расстояний между векторами, имеющими фиксированную сумму компонент).
Can you write Python code for it?

from sklearn.manifold import MDS
from sklearn.metrics import pairwise_distances

prob_theme_author = np.empty(phi_a.shape)
for i in range(prob_theme_author.shape[0]):
    for j in range(prob_theme_author.shape[1]):
        prob_theme_author[i,j] = phi_a.iloc[i,j] * prob_theme.iloc[j,:] / np.sum(phi_a.iloc[i,:] * prob_theme.prob.values)

# Ваш код
similarities = pairwise_distances(prob_theme_author, metric='cosine')
mds = MDS(n_components=2, dissimilarity='precomputed', random_state=42)
pos = mds.fit_transform(similarities)


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
В последней теме "bcg" должны находиться общеупотребительные слова.
Важный шаг в работе с тематической моделью, когда речь идет о визуализации или создании тематического навигатора, это именование тем. Понять, о чем каждая тема, можно по списку ее топовых слов. Например, тему
частица взаимодействие физика кварк симметрия элементарный нейтрино стандартный материя протон бозон заряд масса ускоритель слабый

можно назвать "Физика элементарных частиц".
Дайте названия 29 предметным темам. Если вы не знаете, как назвать тему, назовите ее первым встретившимся в ней существительным, хотя при таком подходе навигатор будет менее информативным. Из названий тем составьте список из 29 строк и запишите го в переменную sbj_topic_labels. В переменной topic_labels будут храниться названия всех тем, включая фоновую.
Can you write Python code for it?

sbj_topic_labels = []   # запишите названия тем в список
for topic_name in model.topic_names[:29]:
    sbj_topic_labels.append(tokens[topic_name][0])

topic_labels = sbj_topic_labels + [u"Фоновая тема"]


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
В цикле для каждой темы выведите ее заголовок, в следующей строке - топ-10 слов темы, затем в виде списка ссылки на 10 наиболее релевантных (по $p(d|t)$) теме документов. Используйте html-разметку. Творчество приветствуется :)
Can you write Python code for it?

# Ваш код
for i, theme in enumerate(topic_labels):
    display(HTML("<h3>%s</h3>" % theme))
    for j in range(10):
        print(tokens[model.topic_names[i]][j]),
    print('')
    for k in range(10):
        print(theta.columns[prob_doc_theme_sorted_indices[i,k]])


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Вы должны увидеть, что фоновая тема имеет большую вероятность в почти каждом документе, и это логично. Кроме того, есть еще одна тема, которая чаще других встречается в документах. Судя по всему, это тема содержит много слов по науку в целом, а каждый документ (видео) в нашей коллекции связан с наукой. Можно (необязательно) дать этой теме название "Наука".
Помимо этих двух тем, фоновой и общенаучной, каждый документ характеризуется малым числом других тем.
Оценим $p(t)$ - долю каждой темы во всей коллекции. По формуле полной вероятности вычислять эти величины нужно как
$p(t) = \sum_d p(t|d) p(d)$. Согласно вероятностной модели, $p(d)$ пропорционально длине документа d. Поступим проще: будем полагать, что все документы равновероятны. Тогда оценить $p(t)$ можно, просуммировав $p(t|d)$ по всем документам, а затем разделив полученный вектор на его сумму.
Создайте переменную-датафрейм с T строками, индексированными названиями тем, и 1 столбцом, содержащим оценки $p(t)$. Выведите датафрейм на печать.
Can you write Python code for it?

# Ваш код
prob_theme_data = [np.sum(theta.iloc[i]) for i in range(theta.shape[0])]
prob_theme_data_normed = prob_theme_data / np.sum(prob_theme_data)
prob_theme = pd.DataFrame(data=prob_theme_data_normed, index=topic_labels, columns=['prob'])
prob_theme

prob_theme_max = prob_theme
prob_theme_min = prob_theme

print('Max 5 probabilities:')
for i in range(5):
    max_value = prob_theme_max.max()[0]
    print(prob_theme_max[prob_theme_max.values == max_value].index[0])
    prob_theme_max = prob_theme_max[prob_theme_max.values != max_value]

print('\nMin 3 probabilities:')
for i in range(3):
    min_value = prob_theme_min.min()[0]
    print(prob_theme_min[prob_theme_min.values == min_value].index[0])
    prob_theme_min = prob_theme_min[prob_theme_min.values != min_value]


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Выполните 30 итераций по коллекции (num_collection_passes), количество внутренних итераций установите равным 1. Используйте метод fit_offline модели.
Can you write Python code for it?

# Ваш код
model.num_document_passes = 1
model.fit_offline(batch_vectorizer=batch_vectorizer, num_collection_passes=30)


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Добавление score
Создайте два измерителя качества artm.TopTokensScore - по одному для каждой модальности; количество токенов 15. Названия для score придумайте самостоятельно.
Can you write Python code for it?

# Ваш код
model.scores.add(artm.TopTokensScore(name='top_tokens_score_mod1', class_id='text', num_tokens=15))
model.scores.add(artm.TopTokensScore(name='top_tokens_score_mod2', class_id='author', num_tokens=15))


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Добавьте разреживающий регуляризатор с коэффициентом tau=-1e5, указав название своего словаря, модальность текста в class_ids и все темы "sbjX" в topic_names.
Can you write Python code for it?

# Ваш код
topic_names_cleared = list(topic_names).remove('bcg')
model.regularizers.add(artm.SmoothSparsePhiRegularizer(tau=-1e5, class_ids='text', dictionary='dict',
                                                       topic_names=topic_names_cleared))


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Интерпретация тем
Используя созданные score, выведите топы слов и топы авторов в темах. Удобнее всего выводить топ слов каждой темы с новой строки, указывая название темы в начале строки, и аналогично с авторами.
Can you write Python code for it?

# Ваш код
tokens = model.score_tracker['top_tokens_score_mod1'].last_tokens
for topic_name in model.topic_names:
    print topic_name + ': ',
    for word in tokens[topic_name]:
        print word,
    print

# Ваш код
authors = model.score_tracker['top_tokens_score_mod2'].last_tokens
for topic_name in model.topic_names:
    print topic_name + ': ',
    for author in authors[topic_name]:
        print author,
    print


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Кроме того, подключив модуль ipython.core.display, можно использовать html-разметку в выводе. Например:
Can you write Python code for it?

from IPython.core.display import display, HTML
display(HTML(u"
Заголовок
"))   # также
,

display(HTML(u"

Пункт 1

Пункт 2

"))
display(HTML(u'Зеленый!'))
display(HTML(u'Еще один вариант вывода ссылки'))


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Найдите 5 самых распространенных и 3 наименее освещенных темы в коллекции (наибольшие и наименьшие $p(t)$ соответственно), не считая фоновую и общенаучную. Укажите названия, которые вы дали этим темам.
Визуализируйте матрицу $\Phi$ модальности авторов в виде изображения. Рекомендация: установите yticklabels=False в heatmap.
Can you write Python code for it?

# Ваш код
plt.figure(figsize=(20,10))
plt.title('Theta matrix for the first 100 documents')
sns.heatmap(phi_a.iloc[:100], cmap='YlGnBu', yticklabels=False)
plt.show();


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Построение модели
Мы будем строить модель в два этапа: сначала добавим сглаживающий регуляризатор фоновой темы и настроим параметры модели, затем - добавим разреживающий регуляризатор предметрых тем и выполним еще несколько итераций. Так мы сможем получить наиболее чистые от фоновых слов предметные темы. Сглаживающий и разреживающий регуляризаторы задаются одним и тем же классом artm.SmoothSparsePhiRegularizer: если коэффициент tau положительный, то регуляризатор будет сглаживающий, если отрицательный - разреживающий.
Если вы хотите подробнее разобраться, как выполняется регуляризация тематической модели в BigARTM, вы можете прочитать статью, раздел 4.
Добавьте сглаживающий регуляризатор с коэффициентом tau = 1e5, указав название своего словаря в dictionary, модальность текста в class_ids и тему "bcg" в topic_names.
Can you write Python code for it?

# Ваш код
model.regularizers.add(artm.SmoothSparsePhiRegularizer(tau=1e5, class_ids='text', dictionary='dict', topic_names='bcg'))


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Создавать навигатор мы будем прямо в jupiter notebook: это возможно благодаря тому факту, что при печати ссылки она автоматически превращается в гиперссылку.
Can you write Python code for it?

print "http://yandex.ru"   # получится кликабельная ссылка


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Создание простого тематического навигатора по Постнауке
Наш тематический навигатор будет для каждой темы показывать ее список слов, а также список релевантных теме документов.
Нам понадобятся распределения $p(d|t)$. По формуле Байеса $p(d|t) = \frac{p(t|d)p(d)}{\sum_{d'}p(t|d')p(d')}$, но поскольку мы считаем документы равновероятными, достаточно разделить каждую строку $\Theta$ на ее сумму, чтобы оценить распределение.
Отсортируйте матрицу $p(d|t)$ по убыванию $p(d|t)$ в каждой теме (то есть построчно). Нам понадобятся индексы наиболее вероятных документов в каждой теме, поэтому используйте функцию argmax.
Can you write Python code for it?

# Ваш код
prob_doc_theme = theta.values / np.array([np.sum(theme) for theme in theta.values])[:, np.newaxis]
prob_doc_theme_sorted_indices = prob_doc_theme.argsort(axis=1)[:,::-1]
prob_doc_theme_sorted_indices


I am working on the file "3 Unsupervised learning/Homework/5 text themes postnauka BigARTM/PostnaukaPeerReview.ipynb".
The first task is:
Считывание данных
Создайте объект класса artm.BatchVectorizer, который будет ссылаться на директорию с пакетами данных (батчами). Чтобы библиотека могла преобразовать текстовый файл в батчи, создайте пустую директорию и укажите ее название в параметре target_folder. Размер батча для небольших коллекций (как наша) не важен, вы можете указать любой.
Can you write Python code for it?

# Ваш код
batch_vectorizer = artm.BatchVectorizer(data_path='lectures.txt', data_format='vowpal_wabbit',
                                        target_folder='lectures_batches', batch_size=250)


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
В следующий раз данные можно считывать уже из батчей:
Can you write Python code for it?

batch_vectorizer = artm.BatchVectorizer(data_path="school_batches", data_format='batches')


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
В темах много общеупотребительных слов (так называемой, фоновой лексики). Чтобы этого избежать, будем использовать разреживающий регуляризатор для матрицы фи. Он будет подавлять слова, которые имеют большую частоту во всей коллекции.
Can you write Python code for it?

model_artm.regularizers.add(artm.SmoothSparsePhiRegularizer(name='SparsePhi', tau=-100, dictionary=dictionary))
#если вы хотите применять регуляризатор только к некоторым модальностям, указывайте это в параметре class_ids: class_ids=["text"]

model_artm.fit_offline(batch_vectorizer=batch_vectorizer, num_collection_passes=15)

for topic_name in model_artm.topic_names:
    print topic_name + ': ',
    tokens = model_artm.score_tracker["top_words"].last_tokens
    for word in tokens[topic_name]:
        print word,
    print

print model_artm.score_tracker["SparsityPhiScore"].last_value
print model_artm.score_tracker["SparsityThetaScore"].last_value


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Выведем топы слов:
Can you write Python code for it?

for topic_name in model_artm.topic_names:
    print topic_name + ': ',
    tokens = model_artm.score_tracker["top_words"].last_tokens
    for word in tokens[topic_name]:
        print word,
    print


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Давайте посмотрим также на разреженность матриц:
Can you write Python code for it?


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Матрица вероятностей тем в документах.
Can you write Python code for it?


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Можно попросить матрицы в чистом виде:
Can you write Python code for it?


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Необходимое число итераций можно отслеживать по графику перплексии. Когда она перестала меняться, модель сошлась.
Can you write Python code for it?


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Обратите внимание, что разреживать модель рекомендуется только после того, как она сошлась без регуляризации.
Сохранение и загрузка модели:
Can you write Python code for it?


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Попробуем менять коэффициент регуляризации:
Can you write Python code for it?

model_artm.regularizers['SparsePhi'].tau = -5*1e4

model_artm.fit_offline(batch_vectorizer=batch_vectorizer, num_collection_passes=15)

for topic_name in model_artm.topic_names:
    print topic_name + ': ',
    tokens = model_artm.score_tracker["top_words"].last_tokens
    for word in tokens[topic_name]:
        print word,
    print

# еще раз посмотрим на разреженность
print model_artm.score_tracker["SparsityPhiScore"].last_value
print model_artm.score_tracker["SparsityThetaScore"].last_value


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Создаем объект модели:
Can you write Python code for it?

T = 10   # количество тем
model_artm = artm.ARTM(num_topics=T, topic_names=["sbj"+str(i) for i in range(T)], class_ids={"text":1},
                       num_document_passes=1, reuse_theta=True, cache_theta=True, seed=-1)
# число после названия модальностей - это их веса


I am working on the file "3 Unsupervised learning/Lectures notebooks/4 BigARTM library/BigARTM.ipynb".
The first task is:
Создаем словарь и инициализируем модель с его помощью
Can you write Python code for it?

dictionary = artm.Dictionary('dictionary')
dictionary.gather(batch_vectorizer.data_path)

model_artm.scores.add(artm.PerplexityScore(name='PerplexityScore',
                                           use_unigram_document_model=False,
                                           dictionary='dictionary'))
model_artm.scores.add(artm.SparsityPhiScore(name='SparsityPhiScore', class_id="text"))
model_artm.scores.add(artm.SparsityThetaScore(name='SparsityThetaScore'))
model_artm.scores.add(artm.TopTokensScore(name="top_words", num_tokens=15, class_id="text"))

model_artm.initialize('dictionary')


I am working on the file "3--Earth_temperature_over_time.ipynb".
The first task is:
Better ah? Feel free to play around with the parameters and see how it changes.
Step 3: Apply regression
Let's now fit a straight line through the temperature-anomaly data, to see the trends. We need to perform a least-squares linear regression to find the slope and intercept of a line
$$y = mx+b$$
that fits our data. Thankfully, Python and NumPy are here to help! With polyfit(), we get the slope and y-intercept of the line that best fits the data. With poly1d(), we can build the linear function from its slope and y-intercept.
Can you write Python code for it?

m, b = numpy.polyfit(year, temp_anomaly, 1)

f_linear = numpy.poly1d((m, b))

print(f_linear)

pyplot.figure(figsize=(10, 5))

pyplot.plot(year, temp_anomaly, color='#2929a3', linestyle='-', linewidth=1, alpha=0.5)
pyplot.plot(year, f_linear(year), 'k--', linewidth=2, label='Linear regression')
pyplot.xlabel('Year')
pyplot.ylabel('Land temperature anomaly [°C]')
pyplot.legend(loc='best', fontsize=15)
pyplot.grid();


I am working on the file "3--Earth_temperature_over_time.ipynb".
The first task is:
Exercise Inspect the data by printing year and temp_anomaly
Step 2: Plot the data.
Let's first load our plotting library: the pyplot module of Matplotlib. To get the plots inside the notebook (rather than as popups), we use a special "magic" command, %matplotlib inline:
Can you write Python code for it?

from matplotlib import pyplot
%matplotlib inline


I am working on the file "3--Earth_temperature_over_time.ipynb".
The first task is:
How would we go about understanding the trends from the data on global temperature?
The first step in analyzing unknown data is to generate some simple plots using matplotlib. We are going to look at the temperature-anomaly history, contained in a file, and make our first plot to explore this data.
We are going to plot the data and then we'll fit a line to it to find a trend.
Let's get started!
Step 1: Read a data file.
We took the data from the NOAA (National Oceanic and Atmospheric Administration) webpage. Feel free to play around with the webpage and analyze data on your own, but for now, let's make sure we're working with the same dataset.
We have some sample data in the folder called data on the tutorial repository: Caminos.
The file contains the year on the first column, and averages of land temperature anomaly listed sequentially on the second column, from 1880 to 2016. We will read the file, then make an initial plot to see what it looks like.
Like in Lesson 2, we're assuming that you have all the files from this tutorial, or are working on the lesson after launching Binder. Executing the following cell will save the data in two arrays, one for each column.
Can you write Python code for it?

import numpy

year, temp_anomaly = numpy.loadtxt(fname='data/land_global_temperature_anomaly-1880-2016.csv', delimiter=',',
                              skiprows=5, unpack=True)


I am working on the file "3--Earth_temperature_over_time.ipynb".
The first task is:
Now we have a plot but if I give you this plot without any information you would not be able to figure out what kind of data it is! We need labels on the axis, a title and why not a better color, font and size of the ticks.
Publication quality plots should always be your standard for plotting.
How you present your data will allow others (and probably you in the future) to better understand your work.
Let's make the font of a specific size and type.
We don't want to write this out every time we create a plot.
Instead, the next few lines of code will apply for all the plots we create from now on.
Can you write Python code for it?

from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16


I am working on the file "3--Earth_temperature_over_time.ipynb".
The first task is:
Split regression
We have the linear function that best fits our data but this doesn't look like a good approximation. If you look at the plot you might have noticed that around 1970 the data temperature starts increasing faster.
What if we break the data in two (before and after 1970) and we do a liear regression in each segment? To do that, we need to find in which position of our year array is the year 1970 located. Thanksfully, numpy has a function called  numpy.where() that can help us. We need to pass a condition and numpy.where will tells us where in the array the condition is True.
Can you write Python code for it?


I am working on the file "3--Earth_temperature_over_time.ipynb".
The first task is:
To split the data, we can use a powerful instrument: the colon notation. A colon between two indices indicates a range of values from a start to an end. The rule is that [start:end] includes the element at index start but excludes the one at index end. For example, to grab the first 3 years in our year array, we do:
Can you write Python code for it?


I am working on the file "3--Earth_temperature_over_time.ipynb".
The first task is:
We are going to plot the same plot as before but now we will add a few things to make it prettier and publication quality.
Can you write Python code for it?

#You can set the size of the figure by doing:
pyplot.figure(figsize=(10,5))

#Plotting
pyplot.plot(year, temp_anomaly, color='#2929a3', linestyle='-', linewidth=1)
pyplot.title('Land global temperature anomalies. \n')
pyplot.xlabel('Year')
pyplot.ylabel('Land temperature anomaly [°C]')
pyplot.grid();


I am working on the file "3--Earth_temperature_over_time.ipynb".
The first task is:
You can add a semicolon at the end of the plotting command to avoid that stuff that appeared on top of the figure, that Out[x]: [< ...>] ugliness. Try it.
Can you write Python code for it?


I am working on the file "3-1_Pandas.ipynb".
The first task is:
<tt>loc</tt> does the same but with the explicit names (the last one is included)
Can you write Python code for it?

states.loc[:'New York']

states.loc[:,'Population':]


I am working on the file "3-1_Pandas.ipynb".
The first task is:
DataFrames have
* <tt>index</tt> that are the defined indices as in Series
* <tt>columns</tt> that are the columns names
* <tt>values</tt> that return a (2D) NumPy array with the contents
Can you write Python code for it?


I am working on the file "3-1_Pandas.ipynb".
The first task is:
Dataframe creation
To create DataFrames, the main methods are:
* from Series (as above)
Can you write Python code for it?

print(population,type(population))
states = pd.DataFrame({'Population': population, 'Area': area})
states


I am working on the file "3-1_Pandas.ipynb".
The first task is:
In Jupyter notebooks, DataFrames are displayed in a fancier way when the name of the dataframe is typed (instead of using <tt>print</tt>)
Can you write Python code for it?


I am working on the file "3-1_Pandas.ipynb".
The first task is:
Names and Values
Notice there can be missing values in DataFrames.
Can you write Python code for it?


I am working on the file "3-1_Pandas.ipynb".
The first task is:
Pandas DataFrames
DataFrames is a fundamental object of Pandas that mimicks what can be found in R for instance. Dataframes can be seen as an array of Series: to each index (corresponding to an individual for instance or a line in a table), a Dataframe maps multiples values; these values corresponds to the columns of the DataFrame which each have a name (as a string).
In the following example, we will construct a Dataframe from two Series with common indices.
Can you write Python code for it?

area = pd.Series( {'California': 423967, 'Texas': 695662, 'New York': 141297, 'Florida': 170312, 'Illinois': 149995})
population = pd.Series({'California': 38332521, 'Texas': 26448193, 'New York': 19651127, 'Florida': 19552860, 'Illinois': 12882135})

states = pd.DataFrame({'Population': population, 'Area': area})
print(states,type(states))


I am working on the file "3-1_Pandas.ipynb".
The first task is:
Series Indices
The main difference between NumPy arrays and Pandas Series is the presence of this <tt>index</tt> field. By default, it is set (as in NumPy arrays) as <tt>0,1,..,size_of_the_series</tt> but a Series index can be explicitly defined. The indices may be numbers but also strings. Then, the contents of the series have to be accessed using these defined indices.
Can you write Python code for it?


I am working on the file "3-1_Pandas.ipynb".
The first task is:
The type Series wraps both a sequence of values and a sequence of indices, which we can access with the <tt>values</tt> and <tt>index</tt> attributes.

values are the contents of the series as a NumPy array
Can you write Python code for it?


I am working on the file "3-1_Pandas.ipynb".
The first task is:
To ease the access, Pandas offers dedicated methods:
* <tt>iloc</tt> enables to access subparts of the dataframe as if it was a NumPy array.
Can you write Python code for it?


I am working on the file "3-1_Pandas.ipynb".
The first task is:
Warning:  When accessing a Dataframe, dataframe_name[column_name] return the corresponding column as a Series. dataframe_name[index_name] returns an error! We will see later how to access a specific index.
Can you write Python code for it?

print(states['Area'],type(states['Area']))

try:
    print(states['California'])
except KeyError as error:
    print("KeyError: ",error)


I am working on the file "3-1_Pandas.ipynb".
The first task is:
You may access columns directly with names, then you can access individuals with their index.
Can you write Python code for it?


I am working on the file "3-1_Pandas.ipynb".
The first task is:
from NumPy arrays (the columns and indices are taken as the array's ones)
Can you write Python code for it?

A = np.random.randn(5,3)
print(A,type(A))
dfA = pd.DataFrame(A)
dfA


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Aggregations [*]
Aggregations let you aggreagate several operations.
Can you write Python code for it?


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
And finally the ratings.
Can you write Python code for it?

pb_rows = ratings[ratings["rating"]<0]
pb_rows2 = ratings[ratings["rating"]>5]
tot_pb_rows = pb_rows.append(pb_rows2 )
tot_pb_rows

ratings.drop(tot_pb_rows.index, inplace=True)
ratings.reset_index(inplace=True,drop=True)


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Appending, Concatenating, and Merging
Thanks to naming, dataframes can be easily added, merged, etc. However, if some entries are missing (columns or indices), the operations may get complicated. Here the most standard situations are covered, take a look at the documentation (notably this one on merging, appending, and concatenating )

Appending is for adding the lines of one dataframe with another one with the same columns.
Can you write Python code for it?

A = pd.DataFrame(np.random.randint(0, 20, (2, 2)), columns=list('AB'))
A2 = pd.DataFrame(np.random.randint(0, 20, (3, 2)), columns=list('AB'))
print("A:\n",A,"\nA2:\n",A2)

A.append(A2) # this does not "append to A" but creates a new dataframe


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Filtering
Filtering is the action of deleting rows depending on a boolean function. For instance, the following removes the user with a rating of only one movie.
Can you write Python code for it?

ratings.groupby("userId")["rating"].count()

def filter_func(x):
    return x["rating"].count() >= 2

filtered = ratings.groupby("userId").filter(filter_func)
filtered

filtered.groupby("userId")["rating"].count()


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Improper values
Even without the missing values, some lines are problematic as they feature values outside of prescribed range (userId id of the user, integer greater than 1; movieId  id of the user, integer greater than 1; rating rating of the user to the movie, float between 0.0 and 5.0; imestamp timestamp, integer )
Can you write Python code for it?

ratings[ratings["userId"]<1]  # Identifying a problem


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Merging is for putting together two dataframes with hopefully common data

For a detailed view see  this doc on merging, appending, and concatenating
Can you write Python code for it?

df1 = pd.DataFrame({'employee': ['Bob', 'Jake', 'Lisa', 'Sue'],
                    'group': ['Accounting', 'Engineering', 'Engineering', 'HR']})
df1

df2 = pd.DataFrame({'employee': ['Lisa', 'Bob', 'Jake', 'Sue'],
                    'hire_date': [2004, 2008, 2012, 2014]})
df2

df3 = pd.merge(df1,df2)
df3

df4 = pd.DataFrame({'group': ['Accounting', 'Engineering', 'HR'],
                    'supervisor': ['Carly', 'Guido', 'Steve']})
df4

pd.merge(df3,df4)


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Missing values
Pandas provides functions that check if the values are missing:

isnull(): Generate a boolean mask indicating missing values
notnull(): Opposite of isnull()
Can you write Python code for it?


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Numpy operations
If we apply a NumPy function on a Pandas datframe, the result will be another Pandas dataframe with the indices preserved.
Can you write Python code for it?

df = pd.DataFrame(np.random.randint(0, 10, (3, 4)), columns=['A', 'B', 'C', 'D'])
df

np.cos(df * np.pi/2 ) - 1


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
So to get the mean of the ratings per user, the command is
Can you write Python code for it?


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Sometimes, indexes do not matter, they can be resetted using <tt>ignore_index=True</tt>.
Can you write Python code for it?


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
The most important settings of the <tt>concat</tt> function are <tt>pd.concat(objs, axis=0, join='outer',ignore_index=False)</tt> where

. objs is the list of dataframes to concatenate

. axis is the axis on which to concatenate 0 (default) for the lines and 1 for the columns

. join is to decide if we keep all columns/indices on the other axis ('outer' ,default), or the intersection ( 'inner')

. ignore_index is to decide is we keep the previous names (False, default) or give new ones (True)
For a detailed view see  this doc on merging, appending, and concatenating
Can you write Python code for it?

pd.concat([A,A2,A3],ignore_index=True)

pd.concat([A,A2,A3],axis=1)

pd.concat([A,A2,A3],axis=1,ignore_index=True,join='inner')


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
The pandas arithmetic functions also have an option to fill missing values by replacing the missing one in either of the dataframes by some value.
Can you write Python code for it?


I am working on the file "3-2_Dataframes.ipynb".
The first task is:
Transformations
Transforming is the actions of applying a transformation (sic).
For instance, let us normalize the ratings so that they have zero mean for each user.
Can you write Python code for it?

ratings.groupby("userId")["rating"].mean()

def center_ratings(x):
    x["rating"] = x["rating"] - x["rating"].mean()
    return x

centered = ratings.groupby("userId").apply(center_ratings)

centered.groupby("userId")["rating"].mean()


I am working on the file "3-3_Fancy_Visualization_with_Seaborn.ipynb".
The first task is:
<tt>displot</tt> is a mix of the two previous ones.
Can you write Python code for it?


I am working on the file "3-3_Fancy_Visualization_with_Seaborn.ipynb".
The first task is:
<tt>kdeplot</tt> provides density plots from an array or series (<tt>shade=True</tt> provide filled ones).
Can you write Python code for it?


I am working on the file "3-3_Fancy_Visualization_with_Seaborn.ipynb".
The first task is:
Melting dataframes
For displaying classification data, it is sometimes interesting to melt dataframes, that is separating
* id: the classes typically, things that are not numeric, that have to be kept in place (in our case with iris, the species)
* values: the columns corresponding to values (in our case with iris, the sepal_length, sepal_width, etc.)
The command <tt>pd.melt</tt> return a dataframe with as columns: the id, the variable (former column) name, and associated value.
Can you write Python code for it?

irisS = pd.melt(iris,id_vars="species",value_vars=["sepal_length","sepal_width","petal_length","petal_width"])
irisS.head()

sns.catplot( x=  "species" , y = "value" , col="variable"  , data=irisS , kind="box")


I am working on the file "3-3_Fancy_Visualization_with_Seaborn.ipynb".
The first task is:
Two-dimensional dataset may be represented by level sets with <tt>kdeplot</tt>.
Can you write Python code for it?

sns.kdeplot(data['x'],y = data['y'], shade=True, thresh=0.05, cmap="Reds", cbar=True)


I am working on the file "3-Scipy.ipynb".
The first task is:
Como el valor p es muy grande, no podemos descartar la hiopótesis que los dos conjuntos de datos aleatorios tienen medias diferentes.
Para testear si la media de una única muestra de datos tiene media 0.1 (la media verdadera es 0.0):
Can you write Python code for it?


I am working on the file "3-Scipy.ipynb".
The first task is:
Como era de esperar, vemos un peak en el espectro centrado alrededor de 1, que es la frecuencia que usamos para el oscilador.
Álgebra lineal
El módulo de álgebra lineal contiene muchas funciones relacionadas con matrices, incluyendo resolución de ecuaciones lineales, cálculo de valores propios, funciones de matrices (por ejemplo, para exponenciación matricial), varias decomposiciones diferentes (SVD, LU, cholesky), etc.
Una documentación detallada está disponible aquí: http://docs.scipy.org/doc/scipy/reference/linalg.html
Veremos cómo usar algunas de estas funciones:
Sistemas de ecuaciones lineales
Los sistemas de ecuaciones lineales de la forma
$A x = b$
donde $A$ es una matriz y $x,b$ son vectores, pueden ser resueltos del modo siguiente:
Can you write Python code for it?

A = np.array([[8,2,5], [1,5,2], [7,8,9]])
b = np.array([1,2,3])

x = sp.linalg.solve(A, b)

x

# verificamos la solución
(A @ x) - b


I am working on the file "3-Scipy.ipynb".
The first task is:
Como la señal es real, el espectro es simétrico. Por eso, sólo necesitamos graficar la parte que corresponde a las frecuencias positivas. Para extraer esa parte de w y F podemos usar algunos de los trucos con índices para arreglos NumPy que vimos en la clase 2:
Can you write Python code for it?

indices = np.where(w > 0) # selecciona sólo los índices de elementos que corresponden a frecuencias positivas
w_pos = w[indices]
F_pos = F[indices]

fig, ax = plt.subplots(figsize=(9,3))
ax.plot(w_pos, abs(F_pos))
ax.set_xlim(0, 5);


I am working on the file "3-Scipy.ipynb".
The first task is:
Como se muestra en este ejemplo, podemos usar 'Inf' y '-Inf' como límites de la integral.
Integrales de dimensión mayor se evalúan de forma similar:
Can you write Python code for it?

def integrando(x, y):
    return np.exp(-x**2-y**2)

x_inf = 0
x_sup = 10
y_inf = 0
y_sup = 10

val, errabs = dblquad(integrando, x_inf, x_sup, lambda x : y_inf, lambda x: y_sup)

print(val, errabs)


I am working on the file "3-Scipy.ipynb".
The first task is:
Conviertiendo entre distintos formatos de matriz dispersa:
Can you write Python code for it?

A

A = csr_matrix(A); A

A = csc_matrix(A); A


I am working on the file "3-Scipy.ipynb".
The first task is:
Ejemplo: Oscilador armónico amortiguado
Problemas de EDO son importantes en Física Computacional, de modo que veremos un ejemplo adicional: el oscilador armónico amortiguado. Este problema está bastante bien descrito en wikipedia (en inglés): http://en.wikipedia.org/wiki/Damping.
La ecuación de movimiento para el oscilador amortiguado es:
$\displaystyle \frac{\mathrm{d}^2x}{\mathrm{d}t^2} + 2\zeta\omega_0\frac{\mathrm{d}x}{\mathrm{d}t} + \omega^2_0 x = 0$
donde $x$ es la posición del oscilador, $\omega_0$ la frecuencia, y $\zeta$ es el factor de amortiguamiento. Para escribir esta EDO de segundo orden en la forma estándar, introducimos $p = \frac{\mathrm{d}x}{\mathrm{d}t}$:
$\displaystyle \frac{\mathrm{d}p}{\mathrm{d}t} = - 2\zeta\omega_0 p - \omega^2_0 x$
$\displaystyle \frac{\mathrm{d}x}{\mathrm{d}t} = p$
En la implementación de este ejemplo agregaremos algunos argumentos extras a la función del lado derecho de la EDO, en lugar de usar variables glovales como en el ejemplo anterior. Como consecuencia de los argumentos extra, necesitamos pasar un argumento clave args a la función odeint:
Can you write Python code for it?

def dy(y, t, zeta, w0):
    """
    El lado derecho de la EDO del oscilador amortiguado
    """
    x, p = y[0], y[1]

    dx = p
    dp = -2 * zeta * w0 * p - w0**2 * x

    return [dx, dp]

# condición inicial:
y0 = [1.0, 0.0]

# tiempos en los que se resolvera la EDO
t = np.linspace(0, 10, 1000)
w0 = 2*np.pi*1.0

# resuelve el sistema de EDOs para tres valores diferentes del factor de amortiguamiento

y1 = odeint(dy, y0, t, args=(0.0, w0)) # no amortiguado
y2 = odeint(dy, y0, t, args=(0.2, w0)) # subamortiguado
y3 = odeint(dy, y0, t, args=(1.0, w0)) # amortiguado crítico
y4 = odeint(dy, y0, t, args=(5.0, w0)) # sobreamortiguado

fig, ax = plt.subplots()
ax.plot(t, y1[:,0], 'k', label="no amortiguado", linewidth=0.25)
ax.plot(t, y2[:,0], 'r', label="subamortiguado")
ax.plot(t, y3[:,0], 'b', label=u"amortiguado crítico")
ax.plot(t, y4[:,0], 'g', label="sobreamortiguado")
ax.legend();


I am working on the file "3-Scipy.ipynb".
The first task is:
Encontrando las raíces de una función
Para encontrar las soluciones a una ecuación de la forma $f(x) = 0$ podemos usar la función fsolve. Ella requiere especificar un punto inicial:
Can you write Python code for it?

omega_c = 3.0
def f(omega):
    return np.tan(2*np.pi*omega) - omega_c/omega

fig, ax  = plt.subplots(figsize=(10,4))
x = np.linspace(0, 3, 1000)
y = f(x)
mask = np.where(abs(y) > 50)
x[mask] = y[mask] = np.NaN # elimina líneas verticales cuando la función cambia de signo
ax.plot(x, y)
ax.plot([0, 3], [0, 0], 'k')
ax.set_ylim(-5,5);

optimize.fsolve(f, 0.1)

optimize.fsolve(f, 0.6)

optimize.fsolve(f, 1.1)


I am working on the file "3-Scipy.ipynb".
The first task is:
Estadística:
Can you write Python code for it?

X.mean(), X.std(), X.var() # distribución de Poission

Y.mean(), Y.std(), Y.var() # distribucuón normal


I am working on the file "3-Scipy.ipynb".
The first task is:
Funciones Especiales
En muchos problemas de física computacional son importantes varias funciones matemáticas especiales. SciPy provee implementaciones de muchas de estas funciones especiales. Para más detalles, ver la lista de funciones en la documentación http://docs.scipy.org/doc/scipy/reference/special.html#module-scipy.special.
Para demostrar el uso típico de estas funciones especiales nos concentraremos en las funciones de Bessel:
Can you write Python code for it?

#
# El módulo scipy.special incluye muchas funciones de Bessel
# Aquí usaremos las funciones jn e yn, que son las funciones de Bessel
# de primera y segunda especie, y de orden real. Incluimos también las
# funciones jn_zeros e yn_zeros que entregan los ceros de las
# funciones jn e yn.
#
from scipy.special import jn, yn, jn_zeros, yn_zeros

n = 0    # orden de la función
x = 0.0

# Función de Bessel de primera especie
print("J_%d(%f) = %f" % (n, x, jn(n, x)))

x = 1.0
# Función de Bessel de segunda especie
print("Y_%d(%f) = %f" % (n, x, yn(n, x)))

x = np.linspace(0, 10, 100)

fig, ax = plt.subplots()
for n in range(4):
    ax.plot(x, jn(n, x), label=r"$J_%d(x)$" % n)
ax.legend()

# ceros de las funciones de Bessel
n = 0 # orden
m = 4 # número de raices a calcular
jn_zeros(n, m)


I am working on the file "3-Scipy.ipynb".
The first task is:
Integración
Integración numérica: cuadraturas
La evaluación numérica de una función, del tipo
$\displaystyle \int_a^b f(x) dx$
es llamada cuadratura numérica, o simplemente cuadratura. SciPy suministra funciones para diferentes tipos de cuadraturas, por ejemplo las funciones quad, dblquad y tplquad para calcular integrales simples, dobles o triples, respectivamente.
Can you write Python code for it?

from scipy.integrate import quad, dblquad, tplquad


I am working on the file "3-Scipy.ipynb".
The first task is:
Interpolación
La interpolación es simple y conveniente en Scipy: La función interp1d, cuando se le suministran arreglos describiendo datos X e Y, retorna un objeto que se comporta como una función que puede ser llamada para un valor de x arbitrary (en el rango cubierto por X), y que retorna el correspondiente valor interpolado de y:
Can you write Python code for it?

from scipy.interpolate import *

def f(x):
    return np.sin(x)

n = np.arange(0, 10)
x = np.linspace(0, 9, 100)

y_meas = f(n) + 0.1 * np.random.randn(len(n)) # simula medidas con error
y_real = f(x)

linear_interpolation = interp1d(n, y_meas)
y_interp1 = linear_interpolation(x)

cubic_interpolation = interp1d(n, y_meas, kind='cubic')
y_interp2 = cubic_interpolation(x)

fig, ax = plt.subplots(figsize=(10,4))
ax.plot(n, y_meas, 'bs', label='datos con ruido')
ax.plot(x, y_real, 'k', lw=2, label=u'función exacta')
ax.plot(x, y_interp1, 'r', label=u'interpolación lineal')
ax.plot(x, y_interp2, 'g', label=u'interpolación cúbica')
ax.legend(loc=3);


I am working on the file "3-Scipy.ipynb".
The first task is:
Introducción
El paquete SciPy agrega características a los algorítmos de bajo nivel de NumPy para arreglos multidimensionales, y provee un gran número de algorítmos de alto nivel de uso científico. Algunos de los tópicos que cubre SciPy son:

Funciones especiales (scipy.special)
Integración (scipy.integrate)
Optimización (scipy.optimize)
Interpolación (scipy.interpolate)
Transformada de Fourier (scipy.fftpack)
Procesamiento de señales (scipy.signal)
Álgebra lineal (scipy.linalg)
Problemas de Eigenvalores de matrices dispersas (scipy.sparse)
Estadística (scipy.stats)
Procesamiento de imágenes multi-dimensional (scipy.ndimage)
Entrada/Salida desde/hacia archivos (scipy.io)

Cada uno de estos submódulos provee un muchas funciones y clases que pueden ser usadas para resolver problemas en sus respectivos tópicos.
En esta clases veremos cómo usar algunos de estos subpaquetes.
Para acceder al paquete SciPy en un programa Python, comenzamos importando todo desde el módulo scipy.
Can you write Python code for it?


I am working on the file "3-Scipy.ipynb".
The first task is:
Las función quad acepta una gran cantidad de argumentos opcionales, que pueden ser usados para ajustar detalles del comportamiento de la función (ingrese help(quad) para más detalles).
El uso básico es el siguiente:
Can you write Python code for it?

# define una función simple para ser integrada
def f(x):
    return x

x_inf = 0 # el límite inferior de x
x_sup = 1 # el límite superior de x

val, errabs = quad(f, x_inf, x_sup)

print("valor de la integral =", val, ", error absoluto =", errabs)


I am working on the file "3-Scipy.ipynb".
The first task is:
Lectura adicional

http://www.scipy.org - La página oficial del proyecto SciPy.
http://docs.scipy.org/doc/scipy/reference/tutorial/index.html - Un tutorial sobre cómo comenzar a usar SciPy.
https://github.com/scipy/scipy/ - El códifo fuente de SciPy.
Can you write Python code for it?

# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = './css/aeropython.css'
HTML(open(css_file, "r").read())


I am working on the file "3-Scipy.ipynb".
The first task is:
Los vectores propios correspondientes al $n$-ésimo valor propio (guardado en evals[n]) es la $n$-ésima columna en evecs, es decir, evecs[:,n]. Para verificar esto, intentemos multiplicar los vectores propios con la matriz y comparar el resultado con el producto del vector propio y el valor propio:
Can you write Python code for it?

n = 1

A @ evecs[:,n] - evals[n] * evecs[:,n]


I am working on the file "3-Scipy.ipynb".
The first task is:
Matrices dispersas
Las matrices dispersas (sparse matrices) son a menudo útiles en simulaciones numéricas que involucran sistemas grandes, si es que el problema puede ser descrito en forma matricial donde las matrices o vectores contienen mayoritariamente ceros. Scipy tiene buen soporte para las matrices dispersas, con operaciones básicas de álgebra lineal (tales como resolución de ecuaciones, cálculos de valores propios, etc).
Existen muchas estrategias posibles para almacenar matrices dispersas de manera eficiente. Algunas de las más comunes son las así llamadas "formas coordenadas" (CCO), "forma de lista de listas" (LIL), y "compressed-sparse column" CSC (también  "compressed-sparse row", CSR). Cada formato tiene sus ventajas y desventajas. La mayoría de los algorítmos computacionales (resolución de ecuaciones, multiplicación de matrices, etc) pueden ser implementados eficientemente usando los formatos CSR o CSC, pero ellos no son tan intuitivos ni fáciles de inicializar. Por esto, a menudo una matriz dispersa es inicialmente creada en formato COO o LIL (donde podemos agregar elementos a la matriz dispersa eficientemente), y luego convertirlos a CSC o CSR antes de ser usadas en cálculos reales.
Para más información sobre los formatos para matrices dispersas, vea por ejemplo (en inglés): http://en.wikipedia.org/wiki/Sparse_matrix
<img src="./images/sparse.png" alt="" align="center"/>
Cuando creamos una matriz dispersa debemos elegir en qué formato la almacenaremos. Por ejemplo,
Can you write Python code for it?

from scipy.sparse import *

# matriz densa
M = np.array([[1,0,0,0], [0,3,0,0], [0,1,1,0], [1,0,0,1]])
M

# convierte de densa a dispersa
A = csr_matrix(M); A

# convierte de dispersa a densa
A.todense()


I am working on the file "3-Scipy.ipynb".
The first task is:
Optimización
La optimización (encontrar el máximo o el mínimo de una funciónn) constituye un campo amplio en matemáticas, y la optimización de funciones complicadas o de muchas variables puede ser complicada. Aquí sólo revisaremos algunos casos muy simples. Para una introducción detallada a la optimización con SciPy, ver (en inglés): http://scipy-lectures.github.com/advanced/mathematical_optimization/index.html
Para usar el módulo de optimización de Scipy hay que importar el módulo optimize:
Can you write Python code for it?


I am working on the file "3-Scipy.ipynb".
The first task is:
Para demostrar cómo calcular una transformada rápida de Fourier con SciPy, consideremos la FFT de la solución del oscilador armónico amortiguado del ejemplo anterior:
Can you write Python code for it?

N = len(t)
dt = t[1]-t[0]

# calcula la transformada rápida de Fourier
# y2 es la solución del oscilador subamortiguado del ejemplo anterior
F = fft(y2[:,0])

# calcula las frecuencias para las componentes en F
w = fftfreq(N, dt)

fig, ax = plt.subplots(figsize=(9,3))
ax.plot(w, abs(F));


I am working on the file "3-Scipy.ipynb".
The first task is:
Podemos calcular usando matrices dispersas como lo hacemos con matrices densas:
Can you write Python code for it?

A.todense()

(A * A).todense()

(A @ A).todense()

v = np.array([1,2,3,4])[:,np.newaxis]; v

# Multiplicación de matriz dispersa - vector denso
A * v

# el mismo resultado con matriz densa y vector denso
A.todense() * v


I am working on the file "3-Scipy.ipynb".
The first task is:
Podemos también usar las funciones brent o fminbound. Estas funciones tienen una sintaxis algo distinta y usan algoritmos diferentes.
Can you write Python code for it?

optimize.brent(f)

optimize.fminbound(f, -4, 2)  # busca el mínimo en el intervalo (-4,2)


I am working on the file "3-Scipy.ipynb".
The first task is:
Podemos usar la función fmin_bfgs para encontrar el mínimo de la función:
Can you write Python code for it?

x_min = optimize.fmin_bfgs(f, -2) # busca un mínimo local cerca -2
x_min

optimize.fmin_bfgs(f, 0.5)  # busca un mínimo local cerca 0.5


I am working on the file "3-Scipy.ipynb".
The first task is:
Test estadísticos
Test si dos conjuntos de datos aleatorios (independientes) vienen de la misma distribución:
Can you write Python code for it?

t_statistic, p_value = stats.ttest_ind(X.rvs(size=1000), X.rvs(size=1000))

print("t-statistic =", t_statistic)
print("valor p =", p_value)


I am working on the file "3-Scipy.ipynb".
The first task is:
Transformada de Fourier
Las transformadas de Fourier son unas de las herramientas universales de la Computación Científica,  que aparece una y otra vez en distintos contextos. SciPy suministra funciones para acceder ala clásica librería FFTPACK de NetLib, que es una librería eficiente y muy bien testeada para FFT, escrita en FORTRAN. La API de SciPy contiene algunas funciones adicionales, pero en general la API está íntimamente relacionada con la librería original en FORTRAN.
Para usar el módulo fftpack en un programa Python, debe incluir
Can you write Python code for it?

from scipy.fftpack import *
from numpy.fft import *


I am working on the file "3-Scipy.ipynb".
The first task is:
Un sistema de EDOs es usualmente formulado en forma estándar antes de ser resuelto numéricamente. La forma estánder es:
$y' = f(y, t)$
donde
$y = [y_1(t), y_2(t), ..., y_n(t)]$
y $f$ es una función que determina las derivadas de la función $y_i(t)$. Para resolver la EDO necesitamos conocer la función $f$ y una condición inicial, $y(0)$.
Note que EDOs de orden superior siempre pueden ser escritas en esta forma introduciendo nuevas variables para las derivadas intermedias.
Una vez definida la función f y el arreglo y_0, podemos usar la función odeint:
y_t = odeint(f, y_0, t)

donde t es un arreglo con las coordenadas temporales para las que se resolverá el sistema de EDOs. El resultado y_t es un arreglo con una linea para cada punto de tiempo t, y donde cada columna corresponde a una solución y_i(t) para ese tiempo.
Veremos cómo implementar f e y_0 en código Python en los siguientes ejemplos.
Ejemplo: péndulo doble
Consideremos un problema físico: El péndulo doble compuesto, descrito en más detalle aquí (en inglés): http://en.wikipedia.org/wiki/Double_pendulum.
Can you write Python code for it?


I am working on the file "3-TensorBoard.ipynb".
The first task is:
Ad ogni iterazione, eseguiamo sia l'operazione di training (train_step), che i nostri summary, tramite l'operazione merged. Usiamo quindi il FileWriter per scrivere i risultati di quest'ultima su disco:
Can you write Python code for it?

for i in range(300):
    summary, _ = sess.run([merged, train_step], feed_dict={X_tf: X, y_tf: y})
    train_writer.add_summary(summary, i)


I am working on the file "3-TensorBoard.ipynb".
The first task is:
Adesso possiamo procedere, come in precedenza, inizializzando un algoritmo di ottimizzazione:
Can you write Python code for it?

with tf.name_scope('train'):
    train_step = tf.train.AdamOptimizer().minimize(loss)


I am working on the file "3-TensorBoard.ipynb".
The first task is:
E, per finire, la funzione costo:
Can you write Python code for it?

loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=y_tf, logits=f_pre))


I am working on the file "3-TensorBoard.ipynb".
The first task is:
Inizializziamo una rete neurale con uno strato nascosto, allo stesso modo di quanto fatto nei tutorial precedenti, cominciando dai placeholder per input ed output desiderato:
Can you write Python code for it?

import tensorflow as tf
X_tf = tf.placeholder(tf.float32, [None, X.shape[1]])
y_tf = tf.placeholder(tf.float32, [None, 1])


I am working on the file "3-TensorBoard.ipynb".
The first task is:
Ricordiamo che definire le operazioni all'interno di dei name scope permette di visualizzarli in maniera più semplice sul grafo. Continuiamo con lo strato di output:
Can you write Python code for it?

with tf.name_scope('output_layer'):
    w2 = tf.Variable(np.random.randn(15, 1)*0.01, dtype=tf.float32)
    b2 = tf.Variable([0.01], dtype=tf.float32)
    f_pre = tf.matmul(h, w2) + b2
    f = tf.nn.sigmoid(f_pre)


I am working on the file "3-TensorBoard.ipynb".
The first task is:
Si noti la sintassi: possiamo eseguire più operazioni in una sola chiamata unendole in una lista. La chiamata restituisce quindi due valori, uno per ciascuna operazione. In questo caso, il risultato di merged viene passato al writer (insieme all'indice dell'iterazione), mentre il secondo valore viene ignorato.
Prima di passare alla TensorBoard, possiamo anche salvare il nostro grafo per visualizzarlo a sua volta interattivamente:
Can you write Python code for it?


I am working on the file "3-boston.ipynb".
The first task is:
3. Data Preparation

У нас есть пропущенные данные?
Нам нужно кодировать категориальные переменные?
Есть ли исключения?
Влезут ли наши данные в память или нам их нужно разбивать на части?
Can you write Python code for it?


I am working on the file "3-boston.ipynb".
The first task is:
4. Modeling
На этом этапе выбираются и применяются различные методы моделирования, и их параметры калибруются для оптимальных значений. Как правило, существует несколько методов для того же типа задачи. Некоторые методы имеют специфические требования к форме данных. Поэтому, часто требуется дополнительная подготовка данных.
Рассмотрим линейную регрессию:LinearRegression model to understand the concepts behind regression.

X - матрица известных переменных
y - переменная-цель (target variable)

Алгоритм
Большинство знакомы с уравнением линии:
$$y = mx + b$$
Например, у нас есть стенд с лимонадом. Мы могли бы сделать действительно простую модель, говоря, что доход от лимонада равен цене, продажам и аренде. Мы могли бы упростить это к линейному уравнению, который выглядит следующим образом
$$PredictedRevenue = (Sales * Price) - Rent$$
Но чтобы продавать лимонад, нужны не только продажи и цены. Мы могли бы учитывать другие переменные, такие как изменения цен, продаж, температуры и пешеходного движения. Мы могли бы создать более сложную модель, которая будет выглядеть примерно так.
$$PredictedRevenue = (Sales * Weight) + ( Price * Weight) + (Temperature * Weight) + (FootTraffic * Weight) - Rent$$
Весы в примере - это коэфициенты. Используя Multiple Linear Regression мы можем посчитать эти коэфициенты.
Используем 13 переменных для того что бы предсказать цену дома:
$$\hat{Y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + ... \beta_n N$$
$$PredictedPrice = Intercept + (Known CRIM * coefficient) + (Known ZN * coefficient) + ... (Known N * coefficient)$$
Can you write Python code for it?

X = df
boston = datasets.load_boston()
y = boston.target

from sklearn.linear_model import LinearRegression
lm = LinearRegression()

lm.fit(X, y)
print('Estimated intercept coefficent:', lm.intercept_)
print('Number of coefficients:', len(lm.coef_))

# def predict_house(crim, zn, indus, chas, nox, rm, age, dis, rad, tax, ptratio, b, lstat):
#     return lm.intercept_ + lm.coef_[0] * crim \
#             + zn * lm.coef_[1] \
#             + indus * lm.coef_[2] \
#             + chas * lm.coef_[3] \
#             + nox * lm.coef_[4] \
#             + rm * lm.coef_[5] \
#             + age * lm.coef_[6] \
#             + dist * lm.coef_[7] \
#             + rad * lm.coef_[8] \
#             + tax * lm.coef_[9] \
#             + ptratio * lm.coef_[10] \
#             + b * lm.coef_[11] \
#             + lstat * lm.coef_[12]

#print(X.iloc[[41]].values)
#print(y[41])
print("Predicted House price", lm.predict(X)[41] * 1000)
print("Equation house price", (lm.intercept_ + np.dot(lm.coef_, X.iloc[[41]].values[0])) * 1000)
print("Target price", y[41] * 1000)

print("Residual price", (lm.predict(X)[41] - y[41]) * 1000)

print("R^2 =", lm.score(X, y))

predicted = cross_val_predict(lm, boston.data, y, cv=10)

fig, ax = plt.subplots()
ax.scatter(y, predicted)
ax.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=4)
ax.set_xlabel('Measured')
ax.set_ylabel('Predicted')
plt.show()

errors = {}
for feature in boston.feature_names:
    lm = LinearRegression()
    lm.fit(X[[feature]], y)
    mean_squared_error = np.mean((y - lm.predict(X[[feature]])) ** 2)
    errors[feature] = mean_squared_error
errors = sorted(errors.items(), key=lambda x: x[1])

print("Features that Minimize Error")
print(30 * "-")
for e in errors:
    print(e[0], "\t", e[1])


I am working on the file "3-boston.ipynb".
The first task is:
See UCI Data page: https://archive.ics.uci.edu/ml/datasets/Housing
Data Columns:

CRIM: per capita crime rate by town
ZN: proportion of residential land zoned for lots over 25,000 sq.ft.
INDUS: proportion of non-retail business acres per town
CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
NOX: nitric oxides concentration (parts per 10 million)
RM: average number of rooms per dwelling
AGE: proportion of owner-occupied units built prior to 1940
DIS: weighted distances to five Boston employment centres
RAD: index of accessibility to radial highways
TAX: full-value property-tax rate per \$10,000
PTRATIO: pupil-teacher ratio by town
B: $1000(Bk - 0.63)^2$ where Bk is the proportion of blacks by town
LSTAT: % lower status of the population
MEDV: Median value of owner-occupied homes in \$1000's

Let's explore the data a bit, make some interactive plots and build a classifier for the data.
Can you write Python code for it?

sns.pairplot(df[["CRIM", "AGE", "RM", "MEDV"]])


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:

  In the following code snippet we will:

Create the network using the function you just made

Display a summary of the network

Can you write Python code for it?

# Shape of sample
sample_shape = x_train[0].shape

# Construct net
model = net_2(sample_shape, 10)
model.summary()


# Shape of sample
sample_shape = x_train[0].shape

# Construct net
model = net_1(sample_shape, 10)
model.summary()


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
CIFAR
The following explanantion of Cifar 10 comes from official cifar page:
The CIFAR-10 and CIFAR-100 are labeled subsets of the 80 million tiny images dataset. They were collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.
The CIFAR10 Dataset
The CIFAR-10 dataset consists of 60000 $32 \times 32$ colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.

Here are the classes in the dataset, as well as 10 random images from each:

        airplane

        <img src="https://www.cs.toronto.edu/~kriz/cifar-10-sample/airplane9.png"...
Can you write Python code for it?

from keras.datasets import cifar10

# The data, shuffled and split between train and test sets:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

target_2_class = {0:'airplane',
                  1:'automobile',
                  2:'bird',
                  3:'cat',
                  4:'deer',
                  5:'dog',
                  6:'frog',
                  7:'horse',
                  8:'ship',
                  9:'truck'}

# Code to plot the 5th training sample.
fig,ax1 = plt.subplots(1,1, figsize=(7,7))
ax1.imshow(x_train[5])
target = y_train[5][0]
title = 'Target is {} - Class {}'.format(target_2_class[target],target )
ax1.set_title(title)
ax1.grid(which='Major')
ax1.xaxis.set_major_locator(MaxNLocator(32))
ax1.yaxis.set_major_locator(MaxNLocator(32))
fig.canvas.draw()
time.sleep(0.1)

print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_train.shape))
print('Shape of y_test {}'.format(y_train.shape))


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Convolutional Neural Networks

In this notebook we will become familiar with a type of *layer* for artificial neural networks called convolutional layers. The data we will attempt to model using these types of networks will be images.

Images
For a computer an image is a matrix of data, where each pixel is represented by one or more values:
Matrix with one value per pixel  =  greyscale images
image_source
Matrix with three values per pixel  =  color images

image_source
The MNIST Dataset
We had a brief look at this dataset in the previous notebook, and here we will go through it again with much more detail. As before, the MNIST database (Modified National Institute of Standards and Technology database) is a multiclass classification problem where we are tasked with classifying a digit ($0-9$) based on a $28\times 28$ greyscale image:

The MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.
It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.

source
In the following example we will load data from MNIST.

input $\rightarrow$ 70000 samples of vectors
Each vector has 784 dimensions
Here presented as $28\times 28$ matrices $\rightarrow$ Greyscale images

target $\rightarrow$ 70000 integers indicating a digit from 0 to 9

In the following snippet of code we will:

Load the MNIST dataset

Plot the 5th sample of the training set

Can you write Python code for it?

# Plots will be displaying plots within the notebook
%matplotlib notebook
import matplotlib.pyplot as plt
from matplotlib.ticker import MaxNLocator

# NumPy is a package for manipulating N-dimensional array objects
import numpy as np

# Pandas is a data analysis package
import pandas as pd

#Library To test/verify some tasks
import problem_unittests as tests # Used to test ouw anwsers

# Mnist wrapper
from keras.datasets import mnist

# Code to load the data
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Print data shape
print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_train.shape))
print('Shape of y_test {}'.format(y_train.shape))

# Code to plot the 5th training sample.
fig,ax1 = plt.subplots(1,1, figsize=(7, 7))

ax1.imshow(x_train[5], cmap='gray')
title = 'Target = {}'.format(y_train[5])
ax1.set_title(title)
ax1.grid(which='Major')
ax1.xaxis.set_major_locator(MaxNLocator(28))
ax1.yaxis.set_major_locator(MaxNLocator(28))
fig.canvas.draw()
time.sleep(0.1)


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Data Pre-Processing
Before we start classifying digits we need to pre-process the data.
Your first task is to create a function that normalises 8-bit images from [0,255] to [0,1]:
Task I:  Implement an Image Normalisation Function

**Task**: Implement a function that normalises the images to the interval [0,1].

Inputs are integers in the interval [0,255]

Outputs should be floats in the interval [0,1]

Can you write Python code for it?

def normalise_images(images):
    """Normalise input images.
    """
    # Normalise image here

    return images

### Do *not* modify the following lines ###
tests.test_normalize_images(normalise_images)

# Normalize the data for future use
x_train = normalise_images(x_train)
x_test = normalise_images(x_test)


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Now that we have both added an extra dimension to the input data as well as one-hot encoded the target values, let's take a look at the shapes of the data matrices.
Can you write Python code for it?

print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_train.shape))
print('Shape of y_test {}'.format(y_train.shape))


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Should We Use Max-Pooling?
There is a recent discussion if max-pooling is a good solution for reducing the amount of data between layers of a network. Some recent approaches show that a similar, and sometimes better performance, can be achieved by using convolutions with strides larger than 1.

Getting rid of pooling. Many people dislike the pooling operation and think that we can get away without it. For example, Striving for Simplicity: The All Convolutional Net proposes to discard the pooling layer in favor of architecture that only consists of repeated CONV layers. To reduce the size of the representation they suggest using larger stride in CONV layer once in a while. Discarding pooling layers has also been found to be important in training good generative models, such as variational autoencoders (VAEs) or generative adversarial networks (GANs). It seems likely that future architectures will feature very few to no pooling layers.

source
Task VI: Implement a Convolutional Network Without Max-Pooling
Implement a convolutional neural network without pooling layers:

Task: Replicate the network we made before (`net_1()`), but this time:

Remove max pooling and add stride(s) of 2 to the second convolution block (see [Conv2D](https://keras.io/layers/convolutional/))

Can you write Python code for it?

def net_2(sample_shape, nb_classes):
    # Define the network input to have `sample_shape` shape
    input_x = None

    # Create network internals here
    x = None

    # Dense number_classes
    probabilities = Dense(nb_classes, activation='softmax')(x)

    # Define the output
    model = Model(inputs=input_x, outputs=probabilities)

    return model


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Target Pre-Processing
To classify our digits we need to use one-hot encoding to represent the target outputs. One-hot enconding is a robust yet simple solution to represent multi-categorical targets.
This enconding is an ideal representation to train a model using gradient descent algorithm with the softmax function we discussed in a previous notebook.
Example of one-hot encoding
Here's an example of how a one-hot encoding scheme looks like:

The core idea is that you transform multi-categorical data to a combination of several single class(es). By doing this we can, for each example, see whether it belongs to any class, where 1 indicates that it does and 0 otherwise.
Task III:  Implement a Function for One-Hot Encoding

**Task**: Implement a function that one-hot encodes a vector of numbers to a matrix of $K$ classes:

The first argument is vector with $N$ samples (dimensions)

The second argument is a number $K$ signifying the number of classes

For each sample of the vector you will create an array with $K$ dimensions

The one-hot encoded matrix should have zeros on all positions expect on the position indicated by the current sample in the input vector

Try to implement this function by yourself. If you have doubts ask for help
If you are running out of time use keras.utils.to_categorical(vector, number_classes) like we did in the previous notebook and come back to this task later
Can you write Python code for it?

def one_hot(vector, number_classes):
    """Return a one-hot encoded matrix given the argument vector.
    """
    # Where we will store our one-hots
    one_hot = []

    # One-hot encode `vector` here

    # Transform list to numpy array and return it
    return np.array(one_hot)

### Do *not* modify the following line ###
tests.test_one_hot(one_hot)

# One-hot encode the MNIST target values
y_train = one_hot (y_train, 10)
y_test = one_hot(y_test, 10)


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Task II:  Expand the Dimension of the Input
When we loaded the MNIST dataset each digit was represented by a matrix of size $(28, 28)$. However, the artificial neural network we will be building uses the concept of colour channels and feature maps even for greyscale images. This means that we have to transform $(28, 28)$ to $(28, 28, 1)$.

**Task**: Write a piece of code that add a new dimestion to `x_train` and `x_test`.

The shape of `x_train` should be $(60000, 28, 28, 1)$

The shape of `x_test` should be $(10000, 28, 28, 1)$

Take a look at [numpy.expand_dims()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.expand_dims.html) for how you might do this.

Can you write Python code for it?

# Write your code here
x_train = None
x_test = None

### Do *not* modify the following lines ###
print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_test.shape))
print('Shape of y_test {}'.format(y_test.shape))


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Task IX: Normalise the Images

Task: Use the `normalise_images()` function you created earlier to normalise the images in:

`x_test`

`x_train`

Can you write Python code for it?


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Task V: Define Hyperparameters and Train the Network
We need to define hyperparameters so our network can learn.

Task: Tune in the hyper-parameters until your `loss` and `val_loss` are both converging to low numbers:

Batch size

Number of training epochs

Keep in mind that training these kinds of networks will take longer than the ones we have looked at so far.
Can you write Python code for it?

# Define hyperparameters
batch_size = None
epochs = None

### Do *not* modify the following lines ###

# There is no learning rate because we are using the recommended
# values for the Adadelta optimiser more information here:
# https://keras.io/optimizers/

# We need to compile our model
model.compile(loss='categorical_crossentropy',
              optimizer='Adadelta',
              metrics=['accuracy'])

# Train
logs = model.fit(x_train, y_train,
                 batch_size=batch_size,
                 epochs=epochs,
                 verbose=2,
                 validation_split=0.1)

# Plot our losses and accuracy
fig, ax = plt.subplots(1,1)

pd.DataFrame(logs.history).plot(ax=ax)
ax.grid(linestyle='dotted')
ax.legend()

plt.show()

# Assess performance
print('='*80)
print('Assesing Test dataset...')
print('='*80)

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Task VII: Define Hyperparameters and Train the New Network
As before, we need to define some hyperparameters and train the network. Feel free to reuse the hyperparameters you found before.

Task: Tune in the hyper-parameters until your `loss` and `val_loss` are both converging to low numbers:

Batch size

Number of training epochs

Can you write Python code for it?

# Define hyperparameters
batch_size = None
epochs = None

### Do *not* modify the following lines ###

# As always we need to compile our model
model.compile(loss='categorical_crossentropy',
              optimizer='Adadelta',
              metrics=['accuracy'])

# Train
LOGS = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=2,
          validation_split = 0.1,)

# Plot our losses and accuracy
fig, ax = plt.subplots(1,1)

pd.DataFrame(logs.history).plot(ax=ax)
ax.grid(linestyle='dotted')
ax.legend()
fig.canvas.draw()

# Assess performance
print('='*80)
print('Assesing Test dataset...')
print('='*80)

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Task VIII: One-Hot Encode the Target Values

Task: Use the `one_hot()` function you created earlier to encode:

`y_test`

`y_train`

Can you write Python code for it?

y_train = None
y_test = None

### Do *not* modify the following line ###
# Print data sizes
print('Shape of x_train {}'.format(x_train.shape))
print('Shape of y_train {}'.format(y_train.shape))
print('Shape of x_test {}'.format(x_train.shape))
print('Shape of y_test {}'.format(y_train.shape))


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Task X: Create Your Convolutional Neural Network for CIFAR10

  Task: Create a neural network model to train on CIFAR using what we have learned so far.

Create a new network using either `net_1()` or `net_2()`

Display a summary of the network

Compile the model using eiter `Adadelta`, `Adagrad`, or `Adam` as your optimiser

Some of the code is filled in already so alter what you need.

Can you write Python code for it?

# Shape of samples
sample_shape = x_train[0].shape

# Construct net
model = None
model.summary()

# We need to compile our model network:
model.compile(loss='categorical_crossentropy',
              optimizer='Adam',
              metrics=['accuracy'])


I am working on the file "3-convnet/convolutional-network.ipynb".
The first task is:
Task XI: Train Your Model

  Task: : Train the model created in the previous cell on CIFAR10.

Train the network using `epochs = 30`

Train the network using a `batch_size = 128`

Use `validation_split = 0.2` when calling `fit()`

Plot the losses and accuracy

Assess the performance on the test set

We recommend that you do not copy-paste from the cells above, but rather re-write the code yourself. You can always take a look at earlier cells if you are unsure about what to do.

Can you write Python code for it?


I am working on the file "3-csvkit-intro.ipynb".
The first task is:
Exploring
csvkit's csvlook tool gives us a way to inspect the contents of CSV files without opening them up in a program like Excel (especially when the file is so large it might crash our program)
Can you write Python code for it?

%%bash
csvlook ../data/tate-collection/collection-1.2/artist_data.csv | head -n 5


I am working on the file "3-csvkit-intro.ipynb".
The first task is:
Imported successfully, now we can run any SQL query we want
Can you write Python code for it?

!sql2csv --db sqlite:///tate_artists.db --query "select * from artist_data where gender='Female'"

!sql2csv --db sqlite:///tate_artists.db --query "select name,dates from artist_data where gender='Female'" | \
    csvlook


I am working on the file "3-csvkit-intro.ipynb".
The first task is:
In our case, this still doesn't look that helpful because some of these columns (like name) are really wide
Summary Statistics
The csvkit tools do offer other ways of peeking at out data that can prove useful to us — for instance through providing summary statistics.
Can you write Python code for it?

!csvcut -c name,yearOfBirth,yearOfDeath,url ../data/tate-collection/collection-1.2/artist_data.csv | \
csvstat


I am working on the file "3-csvkit-intro.ipynb".
The first task is:
Let's sort by the year of birth to see which of our 2005 decedents were the oldest
Can you write Python code for it?

!csvcut -c name,yearOfBirth,yearOfDeath ../data/tate-collection/collection-1.2/artist_data.csv | \
csvgrep -c yearOfDeath -m 2005 | csvsort -c yearOfBirth | csvlook


I am working on the file "3-csvkit-intro.ipynb".
The first task is:
Power Usage
One of the really cool features of csvkit is that it allows us to move back and forth between CSVs and relational database structures quickly and easily — and even to treat our CSVs (for the purpose of rapid exploration) as though they were databases we could run SQL queries against
Can you write Python code for it?

!csvsql --query "select name from artist_data where yearOfBirth > 1700;" \
../data/tate-collection/collection-1.2/artist_data.csv | csvlook


I am working on the file "3-csvkit-intro.ipynb".
The first task is:
Super useful!
Searching
We can know use tools for searching (or "grepping") to find specific bits of information in our file — csvkit provides a csvgrep tool for doing just this
Can you write Python code for it?

!csvcut -c name,yearOfBirth,yearOfDeath ../data/tate-collection/collection-1.2/artist_data.csv | \
csvgrep -c yearOfBirth -m 1497 | csvlook

!csvcut -c name,yearOfBirth,yearOfDeath ../data/tate-collection/collection-1.2/artist_data.csv | \
csvgrep -c yearOfDeath -m 2005 | csvlook


I am working on the file "3-data-checks.ipynb".
The first task is:
A bit of basic pandas
Let's first start by reading in the CSV file as a pandas.DataFrame().
Can you write Python code for it?

import pandas as pd
df = pd.read_csv('data/boston_budget.csv')
df.head()


I am working on the file "3-data-checks.ipynb".
The first task is:
By having things YAML formatted, you preserve human-readability and computer-readability simultaneously.
Providing metadata should be something already done when doing analytics; YAML-format is a strong suggestion, but YAML schema will depend on use case.
Let's now switch roles, and pretend that we're on side of the "analyst" and are no longer the "data provider".
How would you check that the columns match the spec? Basically, check that every element in df.columns is present inside the metadata['columns'] list.
Exercise
Inside test_datafuncs.py, write a utility function, check_schema(df, meta_columns) that tests whether every column in a DataFrame is present in some metadata spec file. It should accept two arguments:

df: a pandas.DataFrame
meta_columns: A list of columns from the metadata spec.

```python
def check_schema(df, meta_columns):
    for col in df.columns:
        assert col in meta_columns, f'"{col}" not in metadata column spec'
```
In your test file, outside the function definition, write another test function, test_budget_schemas(), explicitly runs a test for just the budget data.
```python
def test_budget_schemas():
    columns = read_metadata('data/metadata_budget.yml')['columns']
    df = pd.read_csv('data/boston_budget.csv')
check_schema(df, columns)

```
Now, run the test. Do you get the following error? Can you spot the error?
```bash
    def check_schema(df, meta_columns):
        for col in df.columns:

      assert col in meta_columns, f'"{col}" not in metadata column spec'

E           AssertionError: " Amount" not in metadata column spec
E           assert ' Amount' in ['Fiscal Year', 'Service (Cabinet)', 'Department', 'Program #', 'Program', 'Expense Type', ...]

test_datafuncs_soln.py:63: AssertionError
=================================== 1 failed, 7 passed in 0.91 seconds ===================================
```
If there is even a slight mis-spelling, this kind of check will help you pinpoint where that is. Note how the "Amount" column is spelled with an extra space.
At this...
Can you write Python code for it?

import pandas as pd
import seaborn as sns
sns.set_style('white')
%matplotlib inline

df = pd.read_csv('data/boston_ei-corrupt.csv')
df.head()


I am working on the file "3-data-checks.ipynb".
The first task is:
Demo: Visual Diagnostics
We can use a package called missingno, which gives us a quick visual view of the completeness of the data. This is a good starting point for deciding whether you need to manually comb through the data or not.
Can you write Python code for it?

# First, we check for missing data.
import missingno as msno
msno.matrix(df)


I am working on the file "3-data-checks.ipynb".
The first task is:
Exercise
Did we just copy/paste the function?! It's time to stop doing this. Let's refactor the code into a function that can be called.
Categorical Data
For categorical-type data, we can plot the empirical distribution as well. (This example uses the smartphone_sanitization.csv dataset.)
Can you write Python code for it?

from collections import Counter

def empirical_catdist(data, ax, title=None):
    d = Counter(data)
    print(d)
    x = range(len(d.keys()))
    labels = list(d.keys())
    y = list(d.values())
    ax.bar(x, y)
    ax.set_xticks(x)
    ax.set_xticklabels(labels)

smartphone_df = pd.read_csv('data/smartphone_sanitization.csv')
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
empirical_catdist(smartphone_df['site'], ax=ax)


I am working on the file "3-data-checks.ipynb".
The first task is:
Immediately it's clear that there's a number of rows with empty values! Nothing beats a quick visual check like this one.
We can get a table version of this using another package called pandas_summary.
Can you write Python code for it?

# We can do the same using pandas-summary.
from pandas_summary import DataFrameSummary

dfs = DataFrameSummary(df)
dfs.summary()


I am working on the file "3-data-checks.ipynb".
The first task is:
Statistical Checks

Report on deviations from normality.

Normality?!

The Gaussian (Normal) distribution is commonly assumed in downstream statistical procedures, e.g. outlier detection.
We can test for normality by using a K-S test.

K-S test
From Wikipedia:

In statistics, the Kolmogorov–Smirnov test (K–S test or KS test) is a nonparametric test of the equality of continuous, one-dimensional probability distributions that can be used to compare a sample with a reference probability distribution (one-sample K–S test), or to compare two samples (two-sample K–S test). It is named after Andrey Kolmogorov and Nikolai Smirnov.
Can you write Python code for it?

from scipy.stats import ks_2samp
import numpy.random as npr

# Simulate a normal distribution with 10000 draws.
normal_rvs = npr.normal(size=10000)
result = ks_2samp(normal_rvs, df['labor_force_part_rate'].dropna())
result.pvalue < 0.05

fig = plt.figure()
ax = fig.add_subplot(111)
empirical_cumdist(normal_rvs, ax=ax)
empirical_cumdist(df['hotel_occup_rate'], ax=ax)


I am working on the file "3-data-checks.ipynb".
The first task is:
YAML Files
Describe data in a human-friendly & computer-readable format. The environment.yml file in your downloaded repository is also a YAML file, by the way!
Structure:
yaml
key1: value
key2:
- value1
- value2
- subkey1:
    - value3
Example YAML-formatted schema:
yaml
filename: boston_budget.csv
column_names:
- "Fiscal Year"
- "Service (cabinet)"
- "Department"
- "Program #"
...
- "Fund"
- "Amount"
YAML-formatted text can be read as dictionaries.
Can you write Python code for it?

spec = """
filename: boston_budget.csv
columns:
- "Fiscal Year"
- "Service (Cabinet)"
- "Department"
- "Program #"
- "Program"
- "Expense Type"
- "ACCT #"
- "Expense Category (Account)"
- "Fund"
- "Amount"
"""

import yaml
metadata = yaml.load(spec)
metadata


I am working on the file "3-data-checks.ipynb".
The first task is:
dfs.summary() returns a Pandas DataFrame; this means we can write tests for data completeness!
Exercise: Test for data completeness.
Write a test named check_data_completeness(df) that takes in a DataFrame and confirms that there's no missing data from the pandas-summary output. Then, write a corresponding test_boston_ei() that tests the schema for the Boston Economic Indicators dataframe.
```python
In test_datafuncs.py
from pandas_summary import DataFrameSummary
def check_data_completeness(df):
df_summary = DataFrameSummary(df).summary()
for col in df_summary.columns:
    assert df_summary.loc['missing', col] == 0, f'{col} has missing values'

def test_boston_ei():
    df = pd.read_csv('data/boston_ei.csv')
    check_data_completeness(df)
```
Exercise: Test for value correctness.
In the Economic Indicators dataset, there are four "rate" columns: ['labor_force_part_rate', 'hotel_occup_rate', 'hotel_avg_daily_rate', 'unemp_rate'], which must have values between 0 and 1.
Add a utility function to test_datafuncs.py, check_data_range(data, lower=0, upper=1), which checks the range of the data such that:
- data is a list-like object.
- data <= upper
- data >= lower
- upper and lower have default values of 1 and 0 respectively.
Then, add to the test_boston_ei() function tests for each of these four columns, using the check_data_range() function.
```python
In test_datafuncs.py
def check_data_range(data, lower=0, upper=1):
    assert min(data) >= lower, f"minimum value less than {lower}"
    assert max(data) <= upper, f"maximum value greater than {upper}"
def test_boston_ei():
    df = pd.read_csv('data/boston_ei.csv')
    check_data_completeness(df)
zero_one_cols = ['labor_force_part_rate', 'hotel_occup_rate',
                 'hotel_avg_daily_rate', 'unemp_rate']
for col in zero_one_cols:
    check_data_range(df['labor_force_part_rate'])

```
Distributions
Most of what is coming is going to be a demonstration of the kinds of tools that are potentially useful for you. Feel free to relax from coding, as these...
Can you write Python code for it?

import numpy as np
def compute_dimensions(length):
    """
    Given an integer, compute the "square-est" pair of dimensions for plotting.

    Examples:
    - length: 17 => rows: 4, cols: 5
    - length: 14 => rows: 4, cols: 4

    This is a utility function; can be tested separately.
    """
    sqrt = np.sqrt(length)
    floor = int(np.floor(sqrt))
    ceil = int(np.ceil(sqrt))

    if floor ** 2 >= length:
        return (floor, floor)
    elif floor * ceil >= length:
        return (floor, ceil)
    else:
        return (ceil, ceil)

compute_dimensions(length=17)

assert compute_dimensions(17) == (4, 5)
assert compute_dimensions(16) == (4, 4)
assert compute_dimensions(15) == (4, 4)
assert compute_dimensions(11) == (3, 4)

# Next, let's visualize the empirical CDF for each column of data.
import matplotlib.pyplot as plt

def empirical_cumdist(data, ax, title=None):
    """
    Plots the empirical cumulative distribution of values.
    """
    x, y = np.sort(data), np.arange(1, len(data)+1) / len(data)
    ax.scatter(x, y)
    ax.set_title(title)

data_cols = [i for i in df.columns if i not in ['Year', 'Month']]
n_rows, n_cols = compute_dimensions(len(data_cols))

fig = plt.figure(figsize=(n_cols*3, n_rows*3))
from matplotlib.gridspec import GridSpec
gs = GridSpec(n_rows, n_cols)
for i, col in enumerate(data_cols):
    ax = plt.subplot(gs[i])
    empirical_cumdist(df[col], ax, title=col)

plt.tight_layout()
plt.show()


I am working on the file "3-lists.ipynb".
The first task is:
Many ways to change the contents of lists besides adding values
Can you write Python code for it?

odds.append(11)
print('odds after adding a value: ', odds)

del odds[0]
print('odds after removing the first element:', odds)

odds.reverse()
print('odds after reversing:', odds)


I am working on the file "3-lists.ipynb".
The first task is:
While modifying in places - good to remember python treats list in a counterintuitive way
if we make a list and copy it, then modify in place, trouble ensues
Can you write Python code for it?

whos

odds = [1,3,5,7]
primes = odds
primes += [2]
print('primes:', primes)
print('odds:', odds)


I am working on the file "3-lists.ipynb".
The first task is:
mutable - data which can be changed in place
immutable - data which cannot be changed in place
strings and numbers are immutable, can only overwrite with new vals
lists and arrays are mutable: append, reorder, change elements
be careful - modifying data in place if 2 vars refer to same list and you modify list value, it will change both variables!
make copies to avoid
Can you write Python code for it?

#nested lists
x = [['pepper', 'zucchini', 'onion'],
     ['cabbage', 'lettuce', 'garlic'],
     ['apple', 'pear', 'banana']]


I am working on the file "3-lists.ipynb".
The first task is:
mutable and immutable objects

diff in strings and lists: we can change vales in a list, not string
cannot in char string
Can you write Python code for it?

names = ['Newton', 'Darwing', 'Turing'] #typo in Darwins name
print('names is orignally: ', names)
names[1] = 'Darwin'
print('final value of names:', names)


I am working on the file "3-lists.ipynb".
The first task is:
they point to same list! Why?
* python stores a list in memory and then can use multiple names to refer to the same list!
* if we want a copy instead we can use the list function so we don't do this
Can you write Python code for it?

odds = [1,3,5,7]
primes = list(odds)
primes += [2]
print('primes:', primes)
print('odds:', odds)


I am working on the file "3-lists.ipynb".
The first task is:
visual representation of indexing nested lists
Can you write Python code for it?


I am working on the file "3-lists.ipynb".
The first task is:
works but how about with a string
Can you write Python code for it?


I am working on the file "3-seq2seq-native-new.ipynb".
The first task is:
By this point implementations are quite long, so I put them in model_new.py, while notebook will illustrate the application.
Can you write Python code for it?

tf.reset_default_graph()
tf.set_random_seed(1)

with tf.Session() as session:

    # with bidirectional encoder, decoder state size should be
    # 2x encoder state size
    model = Seq2SeqModel(encoder_cell=LSTMCell(10),
                         decoder_cell=LSTMCell(20),
                         vocab_size=10,
                         embedding_size=10,
                         attention=True,
                         bidirectional=True,
                         debug=False)

    session.run(tf.global_variables_initializer())

    train_on_copy_task(session, model,
                       length_from=3, length_to=8,
                       vocab_lower=2, vocab_upper=10,
                       batch_size=100,
                       max_batches=3000,
                       batches_in_epoch=1000,
                       verbose=True)


I am working on the file "3-seq2seq-native-new.ipynb".
The first task is:
Fun exercise, compare performance of different seq2seq variants.
Comparison will be done using train loss tracks, since the task is algorithmic and data is generated directly from true distribution and out-of-sample testing doesn't really make sense.
Can you write Python code for it?

loss_tracks = dict()

def do_train(session, model):
    return train_on_copy_task(session, model,
                              length_from=3, length_to=8,
                              vocab_lower=2, vocab_upper=10,
                              batch_size=100,
                              max_batches=5000,
                              batches_in_epoch=1000,
                              verbose=False)

def make_model(**kwa):
    args = dict(cell_class=LSTMCell,
                num_units_encoder=10,
                vocab_size=10,
                embedding_size=10,
                attention=False,
                bidirectional=False,
                debug=False)
    args.update(kwa)

    cell_class = args.pop('cell_class')

    num_units_encoder = args.pop('num_units_encoder')
    num_units_decoder = num_units_encoder

    if args['bidirectional']:
        num_units_decoder *= 2

    args['encoder_cell'] = cell_class(num_units_encoder)
    args['decoder_cell'] = cell_class(num_units_decoder)

    return Seq2SeqModel(**args)


I am working on the file "3-seq2seq-native-new.ipynb".
The first task is:
Naturally, attention helps a lot when the task is simply copying from inputs to outputs.
Test GRU vs LSTM
Can you write Python code for it?

import time

tf.reset_default_graph()
tf.set_random_seed(1)
with tf.Session() as session:
    model = make_model(bidirectional=True, attention=True, cell_class=LSTMCell)
    session.run(tf.global_variables_initializer())
    t0 = time.time()
    lstm_track = do_train(session, model)
    lstm_took = time.time() - t0

tf.reset_default_graph()
tf.set_random_seed(1)
with tf.Session() as session:
    model = make_model(bidirectional=True, attention=True, cell_class=GRUCell)
    session.run(tf.global_variables_initializer())
    t0 = time.time()
    gru_track = do_train(session, model)
    gru_took = time.time() - t0

gru = pd.Series(gru_track, name='gru')
lstm = pd.Series(lstm_track, name='lstm')
tracks_batch = pd.DataFrame(dict(lstm=lstm, gru=gru))
tracks_batch.index.name = 'batch'

gru.index = gru.index / gru_took
lstm.index = lstm.index / lstm_took
tracks_time = pd.DataFrame(dict(lstm=lstm, gru=gru)).ffill()
tracks_time.index.name = 'time (seconds)'

tracks_batch.plot(figsize=(8, 5), title='GRU vs LSTM loss, batch-time')

tracks_time.plot(figsize=(8, 5), title='GRU vs LSTM loss, compute-time')


I am working on the file "3. Complexity/Tape_Integers.ipynb".
The first task is:
Designing Data function for testing
Can you write Python code for it?

def random_test_data(start,stop,size):
    import random

    return random.sample(xrange(start,stop), size)

small_simple_data = random_test_data(0,10,10)

print small_simple_data

print type(small_simple_data)

print tape_int_2_1(small_simple_data)