I am working on the file "2 Supervised learning/Homework/7 gradient boosting/grad_boosting.ipynb".
The first task is:
Задание 2
Заведите массив для объектов DecisionTreeRegressor (будем их использовать в качестве базовых алгоритмов) и для вещественных чисел (это будут коэффициенты перед базовыми алгоритмами).
В цикле обучите последовательно 50 решающих деревьев с параметрами max_depth=5 и random_state=42 (остальные параметры - по умолчанию). В бустинге зачастую используются сотни и тысячи деревьев, но мы ограничимся 50, чтобы алгоритм работал быстрее, и его было проще отлаживать (т.к. цель задания разобраться, как работает метод). Каждое дерево должно обучаться на одном и том же множестве объектов, но ответы, которые учится прогнозировать дерево, будут меняться в соответствие с полученным в задании 1 правилом.
Попробуйте для начала всегда брать коэффициент равным 0.9. Обычно оправдано выбирать коэффициент значительно меньшим - порядка 0.05 или 0.1, но т.к. в нашем учебном примере на стандартном датасете будет всего 50 деревьев, возьмем для начала шаг побольше.
В процессе реализации обучения вам потребуется функция, которая будет вычислять прогноз построенной на данный момент композиции деревьев на выборке X:
def gbm_predict(X):
    return [sum([coeff * algo.predict([x])[0] for algo, coeff in zip(base_algorithms_list, coefficients_list)]) for x in X]
(считаем, что base_algorithms_list - список с базовыми алгоритмами, coefficients_list - список с коэффициентами перед алгоритмами)
Эта же функция поможет вам получить прогноз на контрольной выборке и оценить качество работы вашего алгоритма с помощью mean_squared_error в sklearn.metrics.
Возведите результат в степень 0.5, чтобы получить RMSE. Полученное значение RMSE — ответ в пункте 2.
Can you write Python code for it?

def gbm_predict(X):
    return [sum([coeff * algo.predict([x])[0] for algo, coeff in zip(base_algorithms_list, coefficients_list)]) for x in X]

base_algorithms_list = []
coefficients_list = []

z = np.zeros( (y_train.shape) )

for _ in range(50):
    coefficients_list.append(0.9)
    dt_regressor = DecisionTreeRegressor(max_depth=5, random_state=42)
    dt_regressor.fit(X_train, L_derivative(y_train, z))
    base_algorithms_list.append(dt_regressor)
    z = gbm_predict(X_train)

alg_predict = gbm_predict(X_test)
alg_rmse = np.sqrt(mean_squared_error(y_test, alg_predict))
print(alg_rmse)

with open('answer2.txt', 'w') as fout:
    fout.write(str(alg_rmse))


I am working on the file "2 Supervised learning/Homework/7 gradient boosting/grad_boosting.ipynb".
The first task is:
Задание 3
Вас может также беспокоить, что двигаясь с постоянным шагом, вблизи минимума ошибки ответы на обучающей выборке меняются слишком резко, перескакивая через минимум.
Попробуйте уменьшать вес перед каждым алгоритмом с каждой следующей итерацией по формуле 0.9 / (1.0 + i), где i - номер итерации (от 0 до 49). Используйте качество работы алгоритма как ответ в пункте 3.
В реальности часто применяется следующая стратегия выбора шага: как только выбран алгоритм, подберем коэффициент перед ним численным методом оптимизации таким образом, чтобы отклонение от правильных ответов было минимальным. Мы не будем предлагать вам реализовать это для выполнения задания, но рекомендуем попробовать разобраться с такой стратегией и реализовать ее при случае для себя.
Can you write Python code for it?

base_algorithms_list = []
coefficients_list = []

z = np.zeros( (y_train.shape) )

for i in range(50):
    coeff = 0.9 / (1. + i)
    coefficients_list.append(coeff)
    dt_regressor = DecisionTreeRegressor(max_depth=5, random_state=42)
    dt_regressor.fit(X_train, L_derivative(y_train, z))
    base_algorithms_list.append(dt_regressor)
    z = gbm_predict(X_train)

alg_predict = gbm_predict(X_test)
alg_rmse = np.sqrt(mean_squared_error(y_test, alg_predict))
print(alg_rmse)

with open('answer3.txt', 'w') as fout:
    fout.write(str(alg_rmse))


I am working on the file "2 Supervised learning/Homework/7 gradient boosting/grad_boosting.ipynb".
The first task is:
Задание 4
Реализованный вами метод - градиентный бустинг над деревьями - очень популярен в машинном обучении. Он представлен как в самой библиотеке sklearn, так и в сторонней библиотеке XGBoost, которая имеет свой питоновский интерфейс. На практике XGBoost работает заметно лучше GradientBoostingRegressor из sklearn, но для этого задания вы можете использовать любую реализацию.
Исследуйте, переобучается ли градиентный бустинг с ростом числа итераций (и подумайте, почему), а также с ростом глубины деревьев. На основе наблюдений выпишите через пробел номера правильных из приведенных ниже утверждений в порядке возрастания номера (это будет ответ в п.4):
1. С увеличением числа деревьев, начиная с некоторого момента, качество работы градиентного бустинга не меняется существенно.

2. С увеличением числа деревьев, начиная с некоторого момента, градиентный бустинг начинает переобучаться.

3. С ростом глубины деревьев, начиная с некоторого момента, качество работы градиентного бустинга на тестовой выборке начинает ухудшаться.

4. С ростом глубины деревьев, начиная с некоторого момента, качество работы градиентного бустинга перестает существенно изменяться
Can you write Python code for it?

answer = str(2) + ' ' + str(3)
with open('answer4.txt', 'w') as fout:
    fout.write(answer)


I am working on the file "2 Supervised learning/Homework/8 neural networks/task_nn.ipynb".
The first task is:
В качестве альтернативного варианта, можно выполнить загрузку данных напрямую из репозитория UCI, воспользовавшись библиотекой urllib.
Can you write Python code for it?

import urllib
# URL for the Wine Quality Data Set (UCI Machine Learning Repository)
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv"
# загрузка файла
f = urllib.urlopen(url)
f.readline()  # пропуск заголовочной строки
data = np.loadtxt(f, delimiter=';')


I am working on the file "2 Supervised learning/Homework/8 neural networks/task_nn.ipynb".
The first task is:
Выделим из данных целевую переменную. Классы в задаче являются несбалинсированными: основной доле объектов поставлена оценка качества от 5 до 7. Приведем задачу к трехклассовой: объектам с оценкой качества меньше пяти поставим оценку 5, а объектам с оценкой качества больше семи поставим 7.
Can you write Python code for it?

TRAIN_SIZE = 0.7 # Разделение данных на обучающую и контрольную части в пропорции 70/30%
from sklearn.model_selection import train_test_split

y = data[:, -1]
np.place(y, y < 5, 5)
np.place(y, y > 7, 7)
y -= min(y)
X = data[:, :-1]
X = normalize(X)

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=TRAIN_SIZE, random_state=0)


I am working on the file "2 Supervised learning/Homework/8 neural networks/task_nn.ipynb".
The first task is:
Двуслойная нейронная сеть
Двуслойная нейронная сеть представляет собой функцию распознавания, которая може быть записана в виде следующей суперпозиции:
$f(x,W)=h^{(2)}\left(\sum\limits_{i=1}^D w_i^{(2)}h^{(1)}\left(\sum\limits_{j=1}^n w_{ji}^{(1)}x_j+b_i^{(1)}\right)+b^{(2)}\right)$, где
$x$ -- исходный объект (сорт вина, описанный 11 признаками), $x_j$ -- соответствующий признак,
$n$ --  количество нейронов во входном слое сети, совпадающее с количеством признаков,
$D$ --  количество нейронов в скрытом слое сети,
$w_i^{(2)}, w_{ji}^{(1)}, b_i^{(1)}, b^{(2)}$ --  параметры сети, соответствующие весам нейронов,
$h^{(1)}, h^{(2)}$ -- функции активации.
В качестве функции активации на скрытом слое сети используется линейная функция. На выходном слое сети используется функция активации softmax, являющаяся обобщением сигмоидной функции на многоклассовый случай:
$y_k=\text{softmax}k(a_1,...,a_k)=\frac{\exp(a_k)}{\sum{k=1}^K\exp(a_k)}.$
Настройка параметров сети
Оптимальные параметры сети $W_{opt}$ определяются путем минимизации функции ошибки:
$W_{opt}=\arg\min\limits_{W}L(W)+\lambda\|W\|^2$.
Здесь $L(W)$ является функцией ошибки многоклассовой классификации,
$L(W)=- \sum^N_{n=1}\sum^K_{k=1} t_{kn} log(y_{kn}),$
$t_{kn}$ -- бинарно закодированные метки классов, $K$ -- количество меток, $N$ -- количество объектов,
а $\lambda\|W\|^2$ является регуляризующим слагаемым, контролирующим суммарный вес параметров сети и предотвращающий эффект переобучения.
Оптимизация параметров выполняется методом обратного распространения ошибки (backpropagation).
Выполним загрузку основных модулей: ClassificationDataSet -- структура данных pybrain, buildNetwork -- инициализация нейронной сети, BackpropTrainer -- оптимизация параметров сети методом backpropagation, SoftmaxLayer -- функция softmax, соответствующая выходному слою сети, percentError -- функцию подсчета ошибки классификации (доля неправильных ответов).
Can you write Python code for it?

from pybrain.datasets import ClassificationDataSet # Структура данных pybrain
from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.structure.modules import SoftmaxLayer
from pybrain.utilities import percentError


I am working on the file "2 Supervised learning/Homework/8 neural networks/task_nn.ipynb".
The first task is:
Инициализируем двуслойную сеть и произведем оптимизацию ее параметров. Аргументами для инициализации являются:
ds.indim -- количество нейронов на входном слое сети, совпадает с количеством признаков (в нашем случае 11),
HIDDEN_NEURONS_NUM -- количество нейронов в скрытом слое сети,
ds.outdim -- количество нейронов на выходном слое сети, совпадает с количеством различных меток классов (в нашем случае 3),
SoftmaxLayer -- функция softmax, используемая на выходном слое для решения задачи многоклассовой классификации.
Can you write Python code for it?

np.random.seed(0) # Зафиксируем seed для получения воспроизводимого результата

# Построение сети прямого распространения (Feedforward network)
net = buildNetwork(ds_train.indim, HIDDEN_NEURONS_NUM, ds_train.outdim, outclass=SoftmaxLayer)
# ds.indim -- количество нейронов входного слоя, равное количеству признаков
# ds.outdim -- количество нейронов выходного слоя, равное количеству меток классов
# SoftmaxLayer -- функция активации, пригодная для решения задачи многоклассовой классификации

init_params = np.random.random((len(net.params))) # Инициализируем веса сети для получения воспроизводимого результата
net._setParameters(init_params)


I am working on the file "2 Supervised learning/Homework/8 neural networks/task_nn.ipynb".
The first task is:
Инициализируем основные параметры задачи: HIDDEN_NEURONS_NUM -- количество нейронов скрытого слоя, MAX_EPOCHS -- максимальное количество итераций алгоритма оптимизации
Can you write Python code for it?

# Определение основных констант
HIDDEN_NEURONS_NUM = 100 # Количество нейронов, содержащееся в скрытом слое сети
MAX_EPOCHS = 100 # Максимальное число итераций алгоритма оптимизации параметров сети


I am working on the file "2 Supervised learning/Homework/8 neural networks/task_nn.ipynb".
The first task is:
Рассчитаем значение доли неправильных ответов на обучающей и контрольной выборке.
Can you write Python code for it?

res_train = net.activateOnDataset(ds_train).argmax(axis=1) # Подсчет результата на обучающей выборке
print 'Error on train: ', percentError(res_train, ds_train['target'].argmax(axis=1)), '%' # Подсчет ошибки
res_test = net.activateOnDataset(ds_test).argmax(axis=1) # Подсчет результата на тестовой выборке
print 'Error on test: ', percentError(res_test, ds_test['target'].argmax(axis=1)), '%' # Подсчет ошибки


I am working on the file "2 Supervised learning/Lectures notebooks/1 sklearn datasets/sklearn.datasets.ipynb".
The first task is:
"Игрушечные" наборы данных
Наборы данных:
* load_iris
* load_boston
* load_diabetes
* load_digits
* load_linnerud
* etc
datasets.load_iris
Can you write Python code for it?

iris = datasets.load_iris()

iris

iris.keys()

print iris.DESCR

print "feature names: {}".format(iris.feature_names)
print "target names: {names}".format(names = iris.target_names)

iris.data[:10]

iris.target


I am working on the file "2 Supervised learning/Lectures notebooks/1 sklearn datasets/sklearn.datasets.ipynb".
The first task is:
datasets.make_classification
Can you write Python code for it?

simple_classification_problem = datasets.make_classification(n_features = 2, n_informative = 1,
                                                            n_redundant = 1, n_clusters_per_class = 1,
                                                            random_state = 1 )

plot_2d_dataset(simple_classification_problem, colors)

classification_problem = datasets.make_classification(n_features = 2, n_informative = 2, n_classes = 4,
                                                      n_redundant = 0, n_clusters_per_class = 1, random_state = 1)

colors = ListedColormap(['red', 'blue', 'green', 'yellow'])

plot_2d_dataset(classification_problem, colors)


I am working on the file "2 Supervised learning/Lectures notebooks/1 sklearn datasets/sklearn.datasets.ipynb".
The first task is:
Бонус: библиотека seaborn
Can you write Python code for it?

import seaborn as sns

sns.pairplot(iris_frame, hue = 'target')

?sns.set()

sns.set(font_scale = 1.3)
data = sns.load_dataset("iris")
sns.pairplot(data, hue = "species")


I am working on the file "2 Supervised learning/Lectures notebooks/1 sklearn datasets/sklearn.datasets.ipynb".
The first task is:
Визуализация выбокри
Can you write Python code for it?

from pandas import DataFrame

iris_frame = DataFrame(iris.data)
iris_frame.columns = iris.feature_names
iris_frame['target'] = iris.target

iris_frame.head()

iris_frame.target = iris_frame.target.apply(lambda x : iris.target_names[x])

iris_frame.head()

iris_frame[iris_frame.target == 'setosa'].hist('sepal length (cm)')

pyplot.figure(figsize(20, 24))

plot_number = 0
for feature_name in iris['feature_names']:
    for target_name in iris['target_names']:
        plot_number += 1
        pyplot.subplot(4, 3, plot_number)
        pyplot.hist(iris_frame[iris_frame.target == target_name][feature_name])
        pyplot.title(target_name)
        pyplot.xlabel('cm')
        pyplot.ylabel(feature_name[:-4])


I am working on the file "2 Supervised learning/Lectures notebooks/1 sklearn datasets/sklearn.datasets.ipynb".
The first task is:
Генерация выборок
Способы генерации данных:
* make_classification
* make_regression
* make_circles
* make_checkerboard
* etc
datasets.make_circles
Can you write Python code for it?

circles = datasets.make_circles()

print "features: {}".format(circles[0][:10])
print "target: {}".format(circles[1][:10])

from matplotlib.colors import ListedColormap

colors = ListedColormap(['red', 'yellow'])

pyplot.figure(figsize(8, 8))
pyplot.scatter(map(lambda x: x[0], circles[0]), map(lambda x: x[1], circles[0]), c = circles[1], cmap = colors)

def plot_2d_dataset(data, colors):
    pyplot.figure(figsize(8, 8))
    pyplot.scatter(map(lambda x: x[0], data[0]), map(lambda x: x[1], data[0]), c = data[1], cmap = colors)

noisy_circles = datasets.make_circles(noise = 0.15)

plot_2d_dataset(noisy_circles, colors)


I am working on the file "2 Supervised learning/Lectures notebooks/12 bonus video/imdb.ipynb".
The first task is:
Векторизуем тексты рецензий:
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer
vect = TfidfVectorizer(sublinear_tf=True, use_idf=True)
X_train = vect.fit_transform(texts_train)
X_test = vect.transform(texts_test)


I am working on the file "2 Supervised learning/Lectures notebooks/12 bonus video/imdb.ipynb".
The first task is:
Классы сбалансированы:
Can you write Python code for it?


I am working on the file "2 Supervised learning/Lectures notebooks/12 bonus video/imdb.ipynb".
The first task is:
Логистическая регрессия
Настроим на векторизованных данных логистическую регрессию и посчитаем AUC:
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
from sklearn import metrics
clf = LogisticRegression()
clf.fit(X_train, y_train)
print metrics.accuracy_score(y_test, clf.predict(X_test))
print metrics.roc_auc_score(y_test, clf.predict_proba(X_test)[:, 1])


I am working on the file "2 Supervised learning/Lectures notebooks/12 bonus video/imdb.ipynb".
The first task is:
Метод главных компонент
Сделаем 100 синтетических признаков с помощью метода главных компонент:
Can you write Python code for it?

from sklearn.decomposition import TruncatedSVD
tsvd = TruncatedSVD(n_components=100)
X_train_pca = tsvd.fit_transform(X_train)
X_test_pca = tsvd.transform(X_test)


I am working on the file "2 Supervised learning/Lectures notebooks/12 bonus video/imdb.ipynb".
The first task is:
Обучим на них логистическую регрессию:
Can you write Python code for it?

clf = LogisticRegression()
clf.fit(X_train_pca, y_train)
print metrics.accuracy_score(y_test, clf.predict(X_test_pca))
print metrics.roc_auc_score(y_test, clf.predict_proba(X_test_pca)[:, 1])


I am working on the file "2 Supervised learning/Lectures notebooks/12 bonus video/imdb.ipynb".
The first task is:
Посмотрим, сколько признаков отбирается:
Can you write Python code for it?


I am working on the file "2 Supervised learning/Lectures notebooks/12 bonus video/imdb.ipynb".
The first task is:
Признаков получилось очень много:
Can you write Python code for it?


I am working on the file "2 Supervised learning/Lectures notebooks/12 bonus video/imdb.ipynb".
The first task is:
Разобъём выборку на обучение и контроль:
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
texts_train, texts_test, y_train, y_test = train_test_split(imdb.review.values, imdb.sentiment.values)


I am working on the file "2 Supervised learning/Lectures notebooks/3 linear models classification/sklearn.linear_model_part1.ipynb".
The first task is:
LogisticRegression
Can you write Python code for it?

log_regressor = linear_model.LogisticRegression(random_state = 1)

log_regressor.fit(train_data, train_labels)

lr_predictions = log_regressor.predict(test_data)

lr_proba_predictions = log_regressor.predict_proba(test_data)

print test_labels

print lr_predictions

print lr_proba_predictions

print metrics.accuracy_score(test_labels, lr_predictions)

print metrics.accuracy_score(test_labels, ridge_predictions)


I am working on the file "2 Supervised learning/Lectures notebooks/3 linear models classification/sklearn.linear_model_part1.ipynb".
The first task is:
cross_val_score с заданными scorer и cv_strategy
Can you write Python code for it?

scorer = metrics.make_scorer(metrics.accuracy_score)

cv_strategy = cross_validation.StratifiedShuffleSplit(blobs[1], n_iter = 20 , test_size = 0.3, random_state = 2)

ridge_scoring = cross_validation.cross_val_score(ridge_classifier, blobs[0], blobs[1], scoring = scorer, cv = cv_strategy)

lr_scoring = cross_validation.cross_val_score(log_regressor, blobs[0], blobs[1], scoring = scorer, cv = cv_strategy)

print 'Ridge mean:{}, max:{}, min:{}, std:{}'.format(ridge_scoring.mean(), ridge_scoring.max(),
                                                     ridge_scoring.min(), ridge_scoring.std())

print 'Log mean:{}, max:{}, min:{}, std:{}'.format(lr_scoring.mean(), lr_scoring.max(),
                                                   lr_scoring.min(), lr_scoring.std())


I am working on the file "2 Supervised learning/Lectures notebooks/3 linear models classification/sklearn.linear_model_part1.ipynb".
The first task is:
Генерация данных
Can you write Python code for it?

blobs = datasets.make_blobs(centers = 2, cluster_std = 5.5, random_state=1)

colors = ListedColormap(['red', 'blue'])

pylab.figure(figsize(8, 8))
pylab.scatter([x[0] for x in blobs[0]], [x[1] for x in blobs[0]], c=blobs[1], cmap=colors)

train_data, test_data, train_labels, test_labels = cross_validation.train_test_split(blobs[0], blobs[1],
                                                                                test_size = 0.3,
                                                                                random_state = 1)


I am working on the file "2 Supervised learning/Lectures notebooks/3 linear models classification/sklearn.linear_model_part1.ipynb".
The first task is:
Линейная классификация
RidgeClassifier
Can you write Python code for it?

#создание объекта - классификатора
ridge_classifier = linear_model.RidgeClassifier(random_state = 1)

#обучение классификатора
ridge_classifier.fit(train_data, train_labels)

#применение обученного классификатора
ridge_predictions = ridge_classifier.predict(test_data)

print test_labels

print ridge_predictions

#оценка качества классификации
metrics.accuracy_score(test_labels, ridge_predictions)

ridge_classifier.coef_

ridge_classifier.intercept_


I am working on the file "2 Supervised learning/Lectures notebooks/3 linear models classification/sklearn.linear_model_part1.ipynb".
The first task is:
Оценка качества по cross-validation
cross_val_score
Can you write Python code for it?

ridge_scoring = cross_validation.cross_val_score(ridge_classifier, blobs[0], blobs[1], scoring = 'accuracy', cv = 10)

lr_scoring = cross_validation.cross_val_score(log_regressor, blobs[0], blobs[1], scoring = 'accuracy', cv = 10)

lr_scoring

print 'Ridge mean:{}, max:{}, min:{}, std:{}'.format(ridge_scoring.mean(), ridge_scoring.max(),
                                                     ridge_scoring.min(), ridge_scoring.std())

print 'Log mean:{}, max:{}, min:{}, std:{}'.format(lr_scoring.mean(), lr_scoring.max(),
                                                   lr_scoring.min(), lr_scoring.std())


I am working on the file "2 Supervised learning/Lectures notebooks/7 bike sharing demand part 1/sklearn.case_part1.ipynb".
The first task is:
Pipeline
Can you write Python code for it?

from sklearn.pipeline import Pipeline

#создаем pipeline из двух шагов: scaling и классификация
pipeline = Pipeline(steps = [('scaling', scaler), ('regression', regressor)])

pipeline.fit(train_data, train_labels)
metrics.mean_absolute_error(test_labels, pipeline.predict(test_data))


I am working on the file "2 Supervised learning/Lectures notebooks/7 bike sharing demand part 1/sklearn.case_part1.ipynb".
The first task is:
Данные и целевая функция
Can you write Python code for it?

#обучение
train_labels = train_data['count'].values
train_data = train_data.drop(['datetime', 'count'], axis = 1)

#тест
test_labels = hold_out_test_data['count'].values
test_data = hold_out_test_data.drop(['datetime', 'count'], axis = 1)


I am working on the file "2 Supervised learning/Lectures notebooks/7 bike sharing demand part 1/sklearn.case_part1.ipynb".
The first task is:
Загрузка данных
Can you write Python code for it?

raw_data = pd.read_csv('bike_sharing_demand.csv', header = 0, sep = ',')

raw_data.head()


I am working on the file "2 Supervised learning/Lectures notebooks/7 bike sharing demand part 1/sklearn.case_part1.ipynb".
The first task is:
Модель
Can you write Python code for it?

regressor = linear_model.SGDRegressor(random_state = 0)

regressor.fit(train_data, train_labels)
metrics.mean_absolute_error(test_labels, regressor.predict(test_data))

print test_labels[:10]

print regressor.predict(test_data)[:10]

regressor.coef_


I am working on the file "2 Supervised learning/Lectures notebooks/7 bike sharing demand part 1/sklearn.case_part1.ipynb".
The first task is:
Обучение и отложенный тест
Can you write Python code for it?

train_data = raw_data.iloc[:-1000, :]
hold_out_test_data = raw_data.iloc[-1000:, :]

print raw_data.shape, train_data.shape, hold_out_test_data.shape

print 'train period from {} to {}'.format(train_data.datetime.min(), train_data.datetime.max())
print 'evaluation period from {} to {}'.format(hold_out_test_data.datetime.min(), hold_out_test_data.datetime.max())


I am working on the file "2 Supervised learning/Lectures notebooks/7 bike sharing demand part 1/sklearn.case_part1.ipynb".
The first task is:
Предобработка данных
Типы признаков
Can you write Python code for it?

raw_data.info()

raw_data.datetime = raw_data.datetime.apply(pd.to_datetime)

raw_data['month'] = raw_data.datetime.apply(lambda x : x.month)
raw_data['hour'] = raw_data.datetime.apply(lambda x : x.hour)

raw_data.head()


I am working on the file "2 Supervised learning/Lectures notebooks/7 bike sharing demand part 1/sklearn.case_part1.ipynb".
The first task is:
Целевая функция на обучающей выборке и на отложенном тесте
Can you write Python code for it?

pylab.figure(figsize = (16, 6))

pylab.subplot(1,2,1)
pylab.hist(train_labels)
pylab.title('train data')

pylab.subplot(1,2,2)
pylab.hist(test_labels)
pylab.title('test data')


I am working on the file "2 Supervised learning/Lectures notebooks/8 bike sharing demand part 2/sklearn.case_part2.ipynb".
The first task is:
Pipeline
Can you write Python code for it?

regressor = linear_model.SGDRegressor(random_state = 0, n_iter = 3, loss = 'squared_loss', penalty = 'l2')

estimator = pipeline.Pipeline(steps = [
    ('feature_processing', pipeline.FeatureUnion(transformer_list = [
            #binary
            ('binary_variables_processing', preprocessing.FunctionTransformer(lambda data: data[:, binary_data_indices])),

            #numeric
            ('numeric_variables_processing', pipeline.Pipeline(steps = [
                ('selecting', preprocessing.FunctionTransformer(lambda data: data[:, numeric_data_indices])),
                ('scaling', preprocessing.StandardScaler(with_mean = 0))
                        ])),

            #categorical
            ('categorical_variables_processing', pipeline.Pipeline(steps = [
                ('selecting', preprocessing.FunctionTransformer(lambda data: data[:, categorical_data_indices])),
                ('hot_encoding', preprocessing.OneHotEncoder(handle_unknown = 'ignore'))
                        ])),
        ])),
    ('model_fitting', regressor)
    ]
)

estimator.fit(train_data, train_labels)

metrics.mean_absolute_error(test_labels, estimator.predict(test_data))


I am working on the file "2 Supervised learning/Lectures notebooks/8 bike sharing demand part 2/sklearn.case_part2.ipynb".
The first task is:
Другая модель
Can you write Python code for it?

from sklearn.ensemble import RandomForestRegressor

regressor = RandomForestRegressor(random_state = 0, max_depth = 20, n_estimators = 50)

estimator = pipeline.Pipeline(steps = [
    ('feature_processing', pipeline.FeatureUnion(transformer_list = [
            #binary
            ('binary_variables_processing', preprocessing.FunctionTransformer(lambda data: data[:, binary_data_indices])),

            #numeric
            ('numeric_variables_processing', pipeline.Pipeline(steps = [
                ('selecting', preprocessing.FunctionTransformer(lambda data: data[:, numeric_data_indices])),
                ('scaling', preprocessing.StandardScaler(with_mean = 0, with_std = 1))
                        ])),

            #categorical
            ('categorical_variables_processing', pipeline.Pipeline(steps = [
                ('selecting', preprocessing.FunctionTransformer(lambda data: data[:, categorical_data_indices])),
                ('hot_encoding', preprocessing.OneHotEncoder(handle_unknown = 'ignore'))
                        ])),
        ])),
    ('model_fitting', regressor)
    ]
)

estimator.fit(train_data, train_labels)

metrics.mean_absolute_error(test_labels, estimator.predict(test_data))

test_labels[:10]

estimator.predict(test_data)[:10]

pylab.figure(figsize=(16, 6))

pylab.subplot(1,2,1)
pylab.grid(True)
pylab.xlim(-100,1100)
pylab.ylim(-100,1100)
pylab.scatter(train_labels, grid_cv.best_estimator_.predict(train_data), alpha=0.5, color = 'red')
pylab.scatter(test_labels, grid_cv.best_estimator_.predict(test_data), alpha=0.5, color = 'blue')
pylab.title('linear model')

pylab.subplot(1,2,2)
pylab.grid(True)
pylab.xlim(-100,1100)
pylab.ylim(-100,1100)
pylab.scatter(train_labels, estimator.predict(train_data), alpha=0.5, color = 'red')
pylab.scatter(test_labels, estimator.predict(test_data), alpha=0.5, color = 'blue')
pylab.title('random forest model')


I am working on the file "2 Supervised learning/Lectures notebooks/8 bike sharing demand part 2/sklearn.case_part2.ipynb".
The first task is:
Загрузка данных
Can you write Python code for it?


I am working on the file "2 Supervised learning/Lectures notebooks/8 bike sharing demand part 2/sklearn.case_part2.ipynb".
The first task is:
Оценка по отложенному тесту
Can you write Python code for it?

test_predictions = grid_cv.best_estimator_.predict(test_data)

metrics.mean_absolute_error(test_labels, test_predictions)

print test_labels[:20]

print test_predictions[:20]

pylab.figure(figsize=(8, 6))
pylab.grid(True)
pylab.xlim(-100,1100)
pylab.ylim(-100,1100)
pylab.scatter(train_labels, grid_cv.best_estimator_.predict(train_data), alpha=0.5, color = 'red')
pylab.scatter(test_labels, grid_cv.best_estimator_.predict(test_data), alpha=0.5, color = 'blue')


I am working on the file "2 Supervised learning/Lectures notebooks/8 bike sharing demand part 2/sklearn.case_part2.ipynb".
The first task is:
Подбор параметров
Can you write Python code for it?

estimator.get_params().keys()

parameters_grid = {
    'model_fitting__alpha' : [0.0001, 0.001, 0,1],
    'model_fitting__eta0' : [0.001, 0.05],
}

grid_cv = grid_search.GridSearchCV(estimator, parameters_grid, scoring = 'mean_absolute_error', cv = 4)

%%time
grid_cv.fit(train_data, train_labels)

print grid_cv.best_score_
print grid_cv.best_params_


I am working on the file "2 Supervised learning/Lectures notebooks/8 bike sharing demand part 2/sklearn.case_part2.ipynb".
The first task is:
Предобработка данных
Обучение и отложенный тест
Can you write Python code for it?

raw_data.datetime = raw_data.datetime.apply(pd.to_datetime)

raw_data['month'] = raw_data.datetime.apply(lambda x : x.month)
raw_data['hour'] = raw_data.datetime.apply(lambda x : x.hour)

train_data = raw_data.iloc[:-1000, :]
hold_out_test_data = raw_data.iloc[-1000:, :]

print raw_data.shape, train_data.shape, hold_out_test_data.shape

#обучение
train_labels = train_data['count'].values
train_data = train_data.drop(['datetime', 'count', 'casual', 'registered'], axis = 1)

#тест
test_labels = hold_out_test_data['count'].values
test_data = hold_out_test_data.drop(['datetime', 'count', 'casual', 'registered'], axis = 1)

binary_data_columns = ['holiday', 'workingday']
binary_data_indices = np.array([(column in binary_data_columns) for column in train_data.columns], dtype = bool)

print binary_data_columns
print binary_data_indices

categorical_data_columns = ['season', 'weather', 'month']
categorical_data_indices = np.array([(column in categorical_data_columns) for column in train_data.columns], dtype = bool)

print categorical_data_columns
print categorical_data_indices

numeric_data_columns = ['temp', 'atemp', 'humidity', 'windspeed', 'hour']
numeric_data_indices = np.array([(column in numeric_data_columns) for column in train_data.columns], dtype = bool)

print numeric_data_columns
print numeric_data_indices


I am working on the file "2 Web scraping and APIs/API and Exercise.ipynb".
The first task is:
2. Access the API
Can you write Python code for it?

# Import the library.
import omdb

# Search for movies.
movies = omdb.search("Westworld")

movies


I am working on the file "2 Web scraping and APIs/API and Exercise.ipynb".
The first task is:
3. Present results
Can you write Python code for it?

# Since "movies" is a list, we can loop through it.
for movie in movies:
    print("Title: " + movie["title"])
    print("Type: " + movie["type"])
    print("Year: " + movie["year"])
    print("Link: http://imdb.com/title/" + movie["imdb_id"])
    print()

# Lets pick the first movie (remember, lists always start at 0).
movie = movies[0]

print(movie["title"])
print(movie["year"])


I am working on the file "2 Web scraping and APIs/API and Exercise.ipynb".
The first task is:
4. Get more info
Can you write Python code for it?

# Instead of searching (and get a list of movies), we can specify the movie we want (and get a single movie).
movie = omdb.title("Westworld", year="2016")

movie

# Present some info about this movie.
print(movie["title"])
print(movie["year"])
print(movie["country"])
print("Rating: " + movie["imdb_rating"])
print(movie["plot"])


I am working on the file "2 Web scraping and APIs/API and Exercise.ipynb".
The first task is:
Exercise

Pick your favorite movie or tv-series, and search for it.

We used movie["title"] to show the title of the movie. Show something else! The documentation says you can get this info:

title
year
type
actors
awards
country
director
genre
episode
season
series_id
language
metascore
plot
poster
rated
released
runtime
writer
imdb_id
imdb_rating
imdb_votes
box_office
dvd
production
website
tomato_consensus
tomato_fresh
tomato_image
tomato_meter
tomato_rating
tomato_reviews
tomato_rotten
tomato_user_meter
tomato_user_rating
tomato_user_reviews
Can you write Python code for it?

# Modify this.
movie = omdb.title("Westworld", year="2016")
print(movie["title"])


I am working on the file "2 Web scraping and APIs/Web scraping and Exercise.ipynb".
The first task is:
2. Define functions for scraping
Can you write Python code for it?

# Scrape all HTML from webpage.
def scrapewebpage(url):
	# Open URL and get HTML.
	web = urllib.request.urlopen(url)

	# Make sure there wasn't any errors opening the URL.
	if (web.getcode() == 200):
		html = web.read()
		return(html)
	else:
		print("Error %s reading %s" % str(web.getcode()), url)

# Helper function that scrape the webpage and turn it into soup.
def makesoup(url):
	html = scrapewebpage(url)
	return(BeautifulSoup(html, "lxml"))


I am working on the file "2 Web scraping and APIs/Web scraping and Exercise.ipynb".
The first task is:
3. Scrape Internet Movie Database
Get some info about a movie.
Can you write Python code for it?

# Scrape Interstellar (2014) by using our own function "makesoup" we defined above.
movie_soup = makesoup('http://www.imdb.com/title/tt0816692/')

# Get movie title.
title = movie_soup.find(itemprop="name").get_text()
title = title.strip() # Remove whitespace before and after text

# Get movie year.
year = movie_soup.find(id="titleYear").get_text()
year = year[1:5] # Remove parentheses, make (2014) into 2014.

# Get movie duration.
duration = movie_soup.find(itemprop="duration").get_text()
duration = duration.strip() # Remove whitespace before and after text

# Get director.
director = movie_soup.find(itemprop="director").get_text()
director = director.strip() # Remove whitespace before and after text

# Get movie rating.
rating = movie_soup.find(itemprop="ratingValue").get_text()

# Get cast list.
actors = []
for castlist in movie_soup.find_all("table", "cast_list"):
	for actor in castlist.find_all(itemprop="actor"):
		actors.append(actor.get_text().strip())

# Present the results.
print("Movie:    " + title)
print("Year:     " + year)
print("Director: " + director)
print("Duration: " + duration)
print("Rating:   " + rating)

# Present list of actors.
print()
print("Main actors:")
for actor in actors:
	print("- " + actor)


I am working on the file "2 Web scraping and APIs/Web scraping and Exercise.ipynb".
The first task is:
4. Scrape Washington Post
Get the latest headlines and links.
Can you write Python code for it?

wpost_soup = makesoup("http://www.washingtonpost.com/")

# Get headlines.
headlines = wpost_soup.find_all("div", "headline")
print("Found " + str(len(headlines)) + " headlines")

# Print headlines.
for headline in headlines:
    print(headline.get_text().strip())

# Print headlines and links.
for links in headlines:
    for link in links.find_all("a"):
        print(link.get_text())
        print(link.get("href"))
        print()

# Get all the links on the page.
for link in wpost_soup.find_all("a"):
    href = link.get("href")
    if href is not None:
        if href[:4] == "http":
            print(href)


I am working on the file "2 Web scraping and APIs/Web scraping and Exercise.ipynb".
The first task is:
5. Scrape Wikipedia
How many seats does each country have in this conuncil? Scrape a table.
Can you write Python code for it?

wiki_soup = makesoup("https://en.wikipedia.org/wiki/Parliamentary_Assembly_of_the_Council_of_Europe")

# Lets find the table "Composition by parliamentary delegation".

# The table doesn't have a unique name, which makes it difficult to scrape.
# However, it's the first table. So we can use find, which returns the first match.

table = wiki_soup.find("table")

# Go through all rows in the table.
for row in table.find_all("tr"):
    # Go through all cells in each row.
    cell = row.find_all("td")
    if len(cell) == 3:
        # Extract the text from the three cells.
        country = cell[0].get_text()
        seats = cell[1].get_text()
        accessiondate = cell[2].get_text()
        print(country + ": " + seats + " seats (" + accessiondate + ")")


I am working on the file "2 Web scraping and APIs/Web scraping and Exercise.ipynb".
The first task is:
Exercise

Go to http://www.imdb.com/ and find your favorite movie.
Try to scrape the rating count (under the rating).
Can you write Python code for it?

# Modify this to your favorite movie.
soup = makesoup('http://www.imdb.com/title/tt0816692/')

# Get rating count instead of name.
title = soup.find(itemprop="name").get_text()
title = title.strip() # Remove whitespace before and after text


I am working on the file "2- IPython Notebooks and Raw Python Data Analysis/2-1 Raw Python - Maps.ipynb".
The first task is:
Exercises:

Create a map that transforms a list of integers into their square roots

Create a map that transforms a list of floats into integers
Can you write Python code for it?

fx = map(float, range(10))
print(list(fx))

fx = map(float, range(10)) # have to create it again in python 3

print(list(map(int, fx)))

from math import sqrt

print(list(map(sqrt, x)))


I am working on the file "2- IPython Notebooks and Raw Python Data Analysis/2-3 Raw Python - List Comprehensions.ipynb".
The first task is:
Filter out the items that have x less than 2 times (we keep those that have "x" more than 2 times)
convert our list "x" into a list of floats
Can you write Python code for it?

[item for item in test_list if item.count("x") >= 2]

x

[float(item) for item in x]

def cube(num):
    return num**3

cubed = [cube(item) for item in x]

cubed


I am working on the file "2- IPython Notebooks and Raw Python Data Analysis/2-4 Raw Python - Lambda Functions.ipynb".
The first task is:
Lambda functions are just anonymous functions and don't need to be created as official functions prior to being used. This makes them useful because they can really help with code readability (if used appropriately). Their true power comes from their expressiveness when performing inline operations.
I'm sure that sounds abstract at this point but I figured it would lay good ground work for our example
Can you write Python code for it?


I am working on the file "2- IPython Notebooks and Raw Python Data Analysis/2-4 Raw Python - Lambda Functions.ipynb".
The first task is:
That’s a little abstract at this point...
We'll get to those with map, filter, and reduce below
Now obviously this is not appropriate for everything but I just want you to understand that lambda functions are just anonymous functions.
Can you write Python code for it?

list(map(lambda z: z**2, filter(lambda z: z % 2 == 0, range(1,20))))


I am working on the file "2--Python_for_Science.ipynb".
The first task is:
But what if we'd like to get a title on this plot, or add labels to the axes? (We should always have labelled axes!). Also, we notice a long jump from the year 1960 to 1970: let's add markers to the plot and change the line style to a dotted line.
Can you write Python code for it?

pyplot.plot(year, av_size, linestyle=':', marker='o')
pyplot.title("Household size in the US, 1960–2016 \n", fontsize=16)
pyplot.ylabel("Average number of people per household")


I am working on the file "2--Python_for_Science.ipynb".
The first task is:
Exercise: Now print the variable av_size, correspondig to the average size of households (in numbers of people) for each year:
Great! The next thing we want to do is make a plot of the changing size of households over the years. To do that, we need to load the Matplotlib module called pyplot:
Can you write Python code for it?

from matplotlib import pyplot
%matplotlib inline


I am working on the file "2--Python_for_Science.ipynb".
The first task is:
Exercise: Now, try it yourself. What is the first element of y?
Right. The first element of y is a 3-wide array of numbers. If we want to access the first element of this now, we use:
Can you write Python code for it?


I am working on the file "2--Python_for_Science.ipynb".
The first task is:
Exercise: Try picking out different elements of the array y…
We learned that:

The square brackets allow us to pick out the elements of an array using an index: x[i]
For a two-dimensional array, we can use two indices: y[i][j]
All indices start at zero.

This is super powerful!
Matplotlib
Matplotlib is for making all kinds of plots. To get an idea of the great variety of plots possibe, have a look at the online Gallery. You can see that Matplotlib itself is a pretty big library. We can load a portion of the library (called a module) that has the basic plotting funtions with:
python
    from matplotlib import pyplot
Once the pyplot module is loaded, its built-in functions are called like this: pyplot.function(arg) (where arg is the function argument).
An example: size of households in the US
Did you know that the size of households—that is, the number of people living in each household—has been steadily decreasing in the US and many other countries? This has perhaps surprising consequences. Even if population growth slows down, or stops altogether, the number of households keeps increasing at a fast rate.
More households means more $CO_2$ emissions! This is bad for the planet.
Get the data
Here, we're assuming that you have all the files from this tutorial, or are working on the lesson after launching Binder. In that case you have a dataset in the data folder.
To load the data into two arrays, named year and av-size, execute the following cell:
Can you write Python code for it?

#Load the data from local disk
year, av_size = numpy.loadtxt(fname='data/statistic_id183648.csv', delimiter=',',
                              skiprows=1, unpack=True)

print(year)


I am working on the file "2--Python_for_Science.ipynb".
The first task is:
Let's review what happened there. We first loaded numpy, giving us the full power to use arrays. We created two arrays: x and y… then we print x and we print y. They look nice.
Numpy has a built-in function to find out the "shape" of an array, which means: how many elements does this array have in each dimension? We find that y is a two-by-three array (it has two dimensions).
What is the first element of x? We can use square brackets and the zero-index to find out:
Can you write Python code for it?


I am working on the file "2--Python_for_Science.ipynb".
The first task is:
What's this from business about?  matplotlib is a pretty big (and awesome!) library.  All that we need is a subset of the library for creating 2D plots, so we ask for the pyplot module of the matplotlib library.
Plotting the data is as easy as calling the function plot() from the module pyplot.
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Comparons le type de donnees, obtenu précédemment. C'est un numpy array. Le type de df est un objet spécifique à pandas.
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
En analyse, on est souvent intéressé  par filtrer les données en fonction de certains critères. Par exemple, l'âge maximum est 80 ans. On peut examiner les informations relatives aux personnes âgées :
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
L'apprentissage du classifieur est fait, c'est-à-dire que nous avons entraîné une SVM sur nos données $X$ pour qu'elle soit capable de prédire la survie $y$. Pour vérifier que notre SVM a bien appris à prédire la survie des passagers, nous pouvons utiliser la méthode predict() et comparer visuellement pour les dix premières valeurs prédite par la SVM et la survie réelle des passagers.
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
La SVM a bien appris à prédire ce que nous lui avons montré. Cela ne permet pas cependant d'évaluer sa capacité à généraliser à des cas qu'elle n'a pas vu. Pour ce faire, une approche classique est de faire de la validation croisée, c'est à dire qu'on entraîne le classifieur sur une partie des données et qu'on le teste sur une autre. Scikit-learn en donne une implémentation très simple d'utilisation.
Can you write Python code for it?

from sklearn import cross_validation

scores = cross_validation.cross_val_score(clf, X, y, cv=7)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Nous allons remplir les valeurs manquantes de l'âge avec la valeur médiane dépendant de la classe et du sexe.
Can you write Python code for it?

ages_medians = np.zeros((2, 3))
ages_medians

for i in range(0,2):
    for j in range(0,3):
        ages_medians[i,j] = df[ (df['Gender'] == i) & (df['Pclass'] == j+1) ]['Age'].median()

ages_medians


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Nous pouvons maintenant créer un nouveau $X$ incluant l'âge, en plus du sexe et de la classe, et vérifier si cela améliore les performances de la SVM.
Can you write Python code for it?

X = df[['Gender', 'Pclass', 'AgeFilled']].as_matrix()

scores = cross_validation.cross_val_score(svm.SVC(), X, y, cv=7)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Numpy ne sait pas convertir la chaîne de caractère vide '' (en 6e position dans notre liste) en réels. Pour traiter ces données, il faudrait écrire un petit algorithme. Nous allons voir comment on peut utiliser pandas pour faire ces traitements beaucoup plus facilement.
Traiter et manipuler les données avec pandas
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
On cherche à prévoir la survie, on extrait donc l'information utile :
Can you write Python code for it?

y = df['Survived'].as_matrix()
print (y[:5])

from sklearn import svm

clf = svm.SVC()
clf.fit(X,y)


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
On peut calculer des critères statistiques directement sur les colonnes
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
On peut voir que Pandas a détecté automatiquement le types des données de notre fichier csv: soit des entiers, soit des réels, soit des objets (chaînes de caractères). Il y a deux commandes importantes à connaître, c'est df.info() et df.describe()
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
On peut voir que pandas a calculé automatiquement les indicateurs statistiques en tenant compte uniquement des données renseignées. Par exemple, il a calculé la moyenne d'âge uniquement sur les 714 valeurs connues. pandas a laissé de coté les valeurs non-numériques (nom, sexe, ticket, cabine, port d'embarquement).
Pour aller un peu plus loin avec pandas
Référencement et filtrage
Pour afficher uniquement les 15 premières valeurs de la colonne âge :
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
On peut voir que parmis les persones âges, il y a principalement des hommes. Les personnes qui   ont survécues était principalement des femmes.
Nous allons maintenant voir comment traiter les valeurs manquantes pour l'âge. Nous allons filtrer les données pour afficher uniquement les valeurs manquantes
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Pour combiner des filtres, on peut utiliser '&'. Affichons le nombre d'hommes dans chaque classe
Can you write Python code for it?

for i in range(1, 4):
    print ("Dans la classe", i, ", il y a", len( df[ (df['Sex'] == 'male') & (df['Pclass'] == i) ]), "hommes")
    print ("Dans la classe", i, ", il y a", len( df[ (df['Sex'] == 'female') & (df['Pclass'] == i) ]), "femmes")


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Pour créer et renommer de nouvelles colonnes, on peut également agréger des informations issues de différentes colonnes. Créons par exemple une colonne pour stocker les nombre de personnes de la même famille à bord du Titanic.
Can you write Python code for it?

df['FamilySize'] = df.SibSp + df.Parch
df.head()


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Pour lire le fichier csv nous allons utiliser la fonction read_csv
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Pour sauvegarder votre travail, vous pouvez utiliser le module pickle qui compresse et sauvegarde vos données :
Can you write Python code for it?

import pickle

f = open('masauvegarde.pck', 'wb')
pickle.dump(df, f)
f.close()


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Pour vérifier si cela a bien fonctionné, affichons les premières valeurs. On voit apparaître l'identifiant du passager, s'il a survécu, sa classe, son nom, son sexe, son âge, le nombre de frères/soeurs/époux/épouse sur le bâteau, le nombre de parents ou d'enfants, le numéro de ticket, le prix, le numéro de cabine et le port d'embarquement.
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Regardons comment sont stockées les données en mémoire:
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Regardons maintenant la colonne de l'âge, n'affichons que les 15 premières valeurs:
Can you write Python code for it?


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Sur les 7 partitions de nos données, la SVM prédit la survie des passagers dans 77% des cas, avec un écart-type de 0,04.
Pour améliorer les résultats, nous pouvons rajouter l'âge dans les features. Il faut cependant faire attention aux valeurs non-renseignées NaN, nous allons donc utiliser une nouvelle colonne AgeFilled avec l'âge ou la médiane.
Can you write Python code for it?

df['AgeFilled'] = df.Age # on recopie la colonne Age
df.loc[df.AgeFilled.isnull(), 'AgeFilled'] = df[df.Age.isnull()]['AgeFill'] # on met l'age médian pour les valeurs non renseignées


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Traitement de données classique
Nous allons voir dans cet exemple comment utiliser la bibliothèque numpy pour récupérer des valeurs dans un fichier csv et commencer à les traiter.
Nous allons utiliser le module csv qui permet de lire un fichier csv et d'en extraire les valeurs lignes par lignes.
Nous allons travailler sur le fichier de données d'entraînement du Titanic. Le but est de prédire les chance de survie à bord du bateau. Il faut récupérer le fichier train.csv (voir le premier cours ou téléchargez le depuis https://www.kaggle.com/c/titanic-gettingStarted/data ) et le sauvegarder dans le répertoire dans lequel le notebook s'éxecute. Vous pouvez utiliser la commande pwd pour connaître ce répertoire. Sinon, vous pouvez déplacer le répertoire courant pour rejoindre l'endroit où vous avez sauvegardé votre fichier avec la commande cd.
Can you write Python code for it?

import csv
import numpy as np

fichier_csv = csv.reader(open('train.csv', 'r'))
entetes = fichier_csv.__next__()  # on récupère la première ligne qui contient les entetes
donnees = list()              # on crée la liste qui va servir à récupérer les données

for ligne in fichier_csv:     # pour chaque ligne lue dans le fichier csv
    donnees.append(ligne)     # on ajoute les valeurs lues dans le tableau donness
#entete = donnees[0]
#donnees[0] = []
donnees = np.array(donnees)   # le tableau donnees est transformé en numpy array


I am working on the file "2-AnalyseDuTitanic.ipynb".
The first task is:
Visualisons maintenant l'histogramme de répartition des âges.
Can you write Python code for it?


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
As has been our approach in our earlier sections, we will construct a fictitious text database to build our inverted index.
Following texts have been extraced via Project Gutenberg from the books:
- The Jungle Book (Rudyard Kipling)
- Alice in Wonderland (Lewis Carrol)
- Concrete Construction Methods and Costs (HP Gillette)
- The Telephone (Dolbear)
Can you write Python code for it?

corpus = [
    """
    So she swallowed one of the cakes and was delighted to find that she
    began shrinking directly. As soon as she was small enough to get through the door,
    she ran out of the house and found quite a crowd of little animals and birds waiting outside.
    They all made a rush at Alice the moment she appeared, but she ran off as hard as
    she could and soon found herself safe in a thick wood.
    """,
    """
    If two strips of different metals, such as silver and iron, be soldered together
    at one end, and the other ends be connected with a galvanometer, on heating the
    soldered junction of the metals it will be found that a current of electricity
    traverses the circuit from the iron to the silver. If other metals be used,
    having the same size, and the same degree of heat be applied, the current of
    electricity thus generated will give a greater or a less deflection, which will be
    constant for the metals employed. The two metals generally employed are bismuth
    and antimony, in bars about an inch long and an eighth of an inch square.
    These are soldered together in series so as to present for faces the ends of the
    bars, and these often number as many as fifty pairs. Such a series is called a thermo-pile.
    This method of[25] generating electricity was discovered by Seebeck of Berlin in 1821,
    but the thermo-pile so much in use now in heat investigations was invented by Nobili in 1835.
    """,
    """
    According to this authority a field testing laboratory will cost for equipment $250 to $350.
    Such a laboratory can be operated by two or three men at a salary charge of from $100 to $200
    per month. Two men will test on an average four samples per day and each additional man will
    test four more samples. The cost of testing will range from $3 to $5 per sample, which
    is roughly equivalent to 3 cts. per barrel of[Pg 5] cement, or from 3 to 5 cts.per cubic yard
    of concrete. These figures are for field laboratory work...


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
Constructing the Index
We will discard whatever we have done until now. Why? Because we never maintained a reference to the document where the word was found. We did so to get a better understanding of each step in our approach to build an inverted index. Lets us now do everything over again, but this time maintaining reference to the document where the word was found.
Can you write Python code for it?

# Our index is a map of word -> documents it is found in
inverted_index = defaultdict(set)

# We maintain the reference to the document by its index in the corpus list
for docid, c in enumerate(corpus):
    for sent in sent_tokenize(c):
        for word in word_tokenize(sent):
            word_lower = word.lower()
            if word_lower not in stwords:
                word_stem = stemmer.stem(word_lower)
                # We add the document to the set againt the word in our
                # index
                inverted_index[word_stem].add(docid)

len(inverted_index.keys())


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
Eliminate Stopwords
Often pronouns, vowels, prepositions like - a, you, them etc. - have little relevance in our search queries. (Unless, of course, we are attempting to isolate the semantic implication of the query.) We can filter out these words to reduce "noise" in our index. Such words are called stopwords. NLTK provides a dataset of such words for English language which we can use to filter our texts.
But to eliminate stopwords, we must first derive words from our corpus. A naive approach could be:
python
[ c.split() for c in corpus ]
While this may work with ordinary concatenation of words, it will create problems when our corpus contains punctuations among other artifacts. We use a tokenizer - functions based on statistical models to separate paragraphs into sentences or sentences into words.
Can you write Python code for it?

# This will contain a list of all words in the corpus
corpus_words = []

# Tokenize a paragraph into sentences and each sentence in to
# words
for c in corpus:
    for sent in sent_tokenize(c):
        word_tokens = word_tokenize(sent)
        corpus_words += word_tokens

len(corpus_words)

# Lets inspect the words in our corpus a bit
corpus_words.index('she'), corpus_words.index('She')

corpus_words.index('so'), corpus_words.index('So')


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
Searching using the index
Just as we processed our documents we need to process our query before looking up in the index.
Can you write Python code for it?

def process_and_search(query):
    matched_documents = set()
    for word in word_tokenize(query):
        word_lower = word.lower()
        if word_lower not in stwords:
            word_stem = stemmer.stem(word_lower)
            matches = inverted_index.get(word_stem)
            if matches:
                # The operator |= is a short hand for set union
                matched_documents |= matches
    return matched_documents

process_and_search("alice")


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
Stemming
Often same words are spelt and pronounced differently based on grammar tense and the context they are used in. For example: summarisation, summarise are similiar words but reflecting different grammatical context. Whereas the former is a noun, the latter is a verb. In most search scenarios, this difference should not matter.
We pass our set of words through a process called stemming which reduces words to their common root-word. In our example, summaris is the root of summarise and summarisation. This will not only help us in reducing the size of index but also improve search matching by looking "deeper" into a word than mere alphabet or string matching.
There is no formulae for stemming instead there are approaches to derive them. In our examples, we use the Snowball Stemmer built in nltk.
Can you write Python code for it?

from nltk.stem import snowball

stemmer = snowball.SnowballStemmer('english')
stemmed_words = set([stemmer.stem(x) for x in stopfree_words])
len(stemmed_words)


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
The below command will open an inline dialog to explore and select packages that are needed. We will select and download stopwords and punkt_tokenizer packages.
There is an alternative command - nltk.download() - which will open a graphical user interface to select packages for download. While, some users may find the interface to be convenient, it may be buggy and slow on some machines.
Can you write Python code for it?


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
The data not only contains duplicates but also case sensitive duplicates. Other than in some specialised scenarios, we do not differentiate between upper, lower or mixed case words. So lets convert them all to lower case and eliminate duplicates by making a set.
Can you write Python code for it?

lower_corpus_words = set([ x.lower() for x in corpus_words ])
len(lower_corpus_words)

# Remove the stopwords
from nltk.corpus import stopwords

stwords = set(stopwords.words('english'))

# Using set difference to eliminate stopwords from our words
stopfree_words = lower_corpus_words - stwords
len(stopfree_words)


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
We have reduced our corpus to about 330 words that we need to look up to match documents.
Can you write Python code for it?

# Lets look at some of our words
list(stemmed_words)[-10:]


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
coward does not appear in our corpus.
Can you write Python code for it?


I am working on the file "2-Building-an-inverted-index-for-search-using-nltk.ipynb".
The first task is:
run appears in documents - 2, 4 and 5
Can you write Python code for it?


I am working on the file "2-Inferential_Stats.ipynb".
The first task is:
0. load clean data
Can you write Python code for it?

df = pd.read_csv('./data/I-SPY_1_clean_data.csv')
df.head(2)


I am working on the file "2-Inferential_Stats.ipynb".
The first task is:
2.5 Effect of MRI measurements on Survival ANOVA
Can you write Python code for it?

outcome = 'Alive'
R = inferential_statistics.anova_MRI(outcome, df);

mri_features = ['MRI_LD_Baseline', 'MRI_LD_1_3dAC', 'MRI_LD_Int_Reg', 'MRI_LD_PreSurg']
outcome = 'Alive'
# Effect Size
inferential_statistics.effect_size( df, mri_features, outcome)


I am working on the file "2-Inferential_Stats.ipynb".
The first task is:

 1.2.2 Conclusion: Only `ER+` and `HR+` have an effect on `Alive`

2. Inferential_statistics: Continous vs Categorical (ANOVA)
2.1 Effect of Age on PCR
Can you write Python code for it?

predictor= ['age']
outcome = 'PCR'
anova_table, OLS = inferential_statistics.linear_models(df, outcome, predictor);
sns.boxplot(x= outcome, y=predictor[0], data=df, palette="Set3");


I am working on the file "2-Inferential_Stats.ipynb".
The first task is:
Conclusion

The largest tumor dimension measured at baseline (MRI_LD_Baseline) is not a statistically different between patients who achieved complete pathological response (PCR)and those who did not. While all other MRI measurements are statistically different between PCR = Yes, and PCR = No
All MRI measurements of the tumor dimension are different between patients who are Alive at the end of the trial and those who did not. These results do not indicate anything about the size of these effects. An statistically significant effect is not always clinically significant
The estimated effect sizes are very small, and most likley not clinically significant
Can you write Python code for it?

## 3. Inferential_statistics: Continous vs Categorical (ANOVA)


I am working on the file "2-Inferential_Stats.ipynb".
The first task is:
Conclusion.

age has an important effect on Alive for patients with PCR = Yes
To quantitify this effect a logistic regression is needed

2.4 Effect of MRI measurements on PCR ANOVA
Can you write Python code for it?

R = inferential_statistics.anova_MRI('PCR', df);


I am working on the file "2-Numpy.ipynb".
The first task is:
Agregando una dimensión adicional: newaxis
Con newaxis, podemos insertar una nueva dimension en un arreglo, por ejemplo, para convertir un vector en la fila o columna de una matriz:
Can you write Python code for it?

v = np.array([1,2,3])
v

np.shape(v)

# crea una matriz con el vector v como su columna
v[:, np.newaxis]

# matriz columna
v[:,np.newaxis].shape

# matriz fila
v[np.newaxis,:]

v[np.newaxis,:].shape


I am working on the file "2-Numpy.ipynb".
The first task is:
Algunos tipos comunes que pueden ser usados con dtype son: int, float, complex, bool, object, etc.
Podemos también definir explícitamente el número de bit de los tipos de datos, por ejemplo: int64, int16, float64, complex64.
Usando funciones que generan arreglos
En el caso de arreglos más grandes no es práctico inicializar los datos manualmente, usando listas Python explícitas. En su lugar, podemos usar una de las muchas funciones en numpy que generan arreglos de diferentes formas. Algunos de los más comunes son:
arange
Can you write Python code for it?

# crea un arreglo con valores en un rango

x = np.arange(0, 10, 1) # argumentos: desde, hasta, paso

x

x = np.arange(-1, 1, 0.1)

x


I am working on the file "2-Numpy.ipynb".
The first task is:
Alternativamente, podemos transformar el arreglo al tipo matrix. Esto cambia el comportamiento de los operadores aritméticos estándar +, -, * al de álgebra de matrices.
Can you write Python code for it?

M = np.matrix(A)
v = np.matrix(v1).T # aplica la traspuesta, convirtiéndolo en vector columna

v

M*M

M*v

# productor interior
v.T * v

# con objetos matriciales, el álgebra matricial estándar es usada
v + M*v


I am working on the file "2-Numpy.ipynb".
The first task is:
Apilando y repitiendo arreglos
Podemos crear vectores y matrices más grandes a partir de otras más pequeñas usando las funcionesn repeat (repetir), tile (teselar, "embaldosar"), vstack (apilar verticalmente), hstack (apilar horizontalmente), y concatenate (concatenar):
tile y repeat
Can you write Python code for it?

a = np.array([[1, 2], [3, 4]])
a

# repite cada elemente 3 veces
np.repeat(a, 3)

# repite la matriz 3 veces
np.tile(a, 3)


I am working on the file "2-Numpy.ipynb".
The first task is:
Copy y "deep copy"
Para alcarzar un alto deseméño, las asignaciones en Python usualmente no copian los objetos involucrados. Esto es importante cuando se pasan objetos a funciones, para así evitar uso excesivo de memoria copiando cuando no es necesario (término técnico: paso por referencia)
Can you write Python code for it?

A = np.array([[1, 2], [3, 4]])

A

# ahora B apunta al mismo arreglo que A
B = A

# cambiar B afecta a A
B[0,0] = 10

B

A


I am working on the file "2-Numpy.ipynb".
The first task is:
Corte de índices
Corte (slicing) de índices es el nombre para la sintaxis M[desde:hasta:paso] para extraer una parte de un arreglo:
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Cuando necesitamos iterar sobre cada elemento de un arreglo y modificar sus elementos, es conveniendo usar la función enumerate para obtener tanto el elemento como su índice en el bucle for:
Can you write Python code for it?

for ind_fila, fila in enumerate(M):
    print("ind_fila", ind_fila, "fila", fila)

    for ind_colu, elemento in enumerate(fila):
        print("ind_colu", ind_colu, "elemento", elemento)

        # actualiza la matriz: eleva al cuadrado cada elemento
        M[ind_fila, ind_colu] = elemento**2

# cada elemento en M está ahora al cuadrado
M


I am working on the file "2-Numpy.ipynb".
The first task is:
Cálculos con datos
A menudo es útil almacenar datos en arreglos Numpy. Numpy provee funciones para realizar cálculos estadísticos de los datos en un arreglo.
Por ejemplo, calculemos algunas propiedades de los datos de la temperatura de Estocolmo que discutimos anteriormente.
Can you write Python code for it?

# recuerde, los datos de la temperatura están almacenados en la variable data
np.shape(data)


I am working on the file "2-Numpy.ipynb".
The first task is:
Cálculos con datos multidimensionales
Cuando se aplican funciones como min, max, etc., a arreglos multidimensionales, a veces se desea aplicarlas al arreglo completo, y en otras ocasiones sólo por filas o columnas. Podemos especificar cómo se comportan estas funciones usando el argumento axis (eje):
Can you write Python code for it?

m = random.rand(3,3)
m

# máximo global
m.max()

# máximo en cada columna
m.max(axis=0)

# máximo en cada fila
m.max(axis=1)


I am working on the file "2-Numpy.ipynb".
The first task is:
Cálculos con matrices
Inversa
Can you write Python code for it?

np.linalg.inv(C) # equivalente a C.I

C.I * C


I am working on the file "2-Numpy.ipynb".
The first task is:
Desde Python 3.5 se pueden multiplicar matrices usando el operador @
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
El formato de los datos es: año, mes, día, temperatura promedio diaria, mínima, máxima, lugar.
Si estamos interesados sólo en la temperatura promedio de un mes particular, Febrero por ejemplo, podemos crear una máscara de índice y seleccionar sólo los datos de ese mes usando:
Can you write Python code for it?

np.unique(data[:,1]) # la columna mes asume valores entre 1 y 12

masc_feb = (data[:,1] == 2) # los paréntesis () son opcionales
masc_feb

# los datos de temperatura están en la columna 3
np.mean(data[masc_feb,3])


I am working on the file "2-Numpy.ipynb".
The first task is:
El número de elementos de un arreglo puede obtenerse usando la propiedad ndarray.size:
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
En el paquete numpy la terminología usada para vectores, matrices y conjuntos de datos de dimensión mayor es la de un  arreglo.
Creando arreglos de numpy
Existen varias formas para inicializar nuevos arreglos de numpy, por ejemplo desde

Listas o tuplas Python
Usando funciones dedicadas a generar arreglos numpy, como arange, linspace, etc.
Leyendo datos desde archivos

Desde lista
Por ejemplo, para crear nuevos arreglos de matrices y vectores desde listas Python podemos usar la función numpy.array.
Can you write Python code for it?

# un vector: el argumento de la función array es una lista de Python
v = np.array([1,2,3,4])

v

# una matriz: el argumento de la función array es una lista anidada de Python
M = np.array([[1, 2], [3, 4]])

M


I am working on the file "2-Numpy.ipynb".
The first task is:
Entrada/Salida desde/hasta archivos
Valores separados por coma (Comma-separated values, CSV)
Un formato muy común para archivos de datos es el de valores separados por comas, o formatos relacionados, como por ejemplo TSV (tab-separated values, valores separados por tabs). Para leer datos desde tales archivos a un arreglo Numpy podemos usar la función numpy.genfromtxt. Por ejemplo,
Can you write Python code for it?

data = np.genfromtxt('stockholm_td_adj.dat')  # asigna los datos del archivo al arreglo data

data.shape

# ¿qué hace esta línea?. La respuesta mas adelante
%matplotlib inline
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(14,4))
ax.plot(data[:,0]+data[:,1]/12.0+data[:,2]/365, data[:,5])
ax.axis('tight')
ax.set_title('temperaturas en Estocolmo')
ax.set_xlabel(u'año')
ax.set_ylabel(u'temperatura (°C)');


I am working on the file "2-Numpy.ipynb".
The first task is:
Equivalentemente, podemos usar las funciones numpy.shape y numpy.size
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Esta característica es muy útil para seleccionar en forma condicional elementos de un arreglo, usando por ejemplo los operadores de comparación:
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Estas funciones ponen a nuestra disposición herramientas muy poderosas para procesar datos. Por ejemplo, para extraer las temperaturas promedio mensuales para cada mes del año sólo necesitamos unas pocas líneas de código:
Can you write Python code for it?

meses = np.arange(1,13)
media_mensual = [np.mean(data[data[:,1] == mes, 3]) for mes in meses]

fig, ax = plt.subplots()
ax.bar(meses, media_mensual)
ax.set_xlabel("Mes")
ax.set_ylabel("Temperatura promedio mensual");


I am working on the file "2-Numpy.ipynb".
The first task is:
Funciones para extraer información desde arreglos y para crear nuevos arreglos
where
Las máscaras de índices pueden ser convertidas en posiciones de índices usando la función where (dónde):
Can you write Python code for it?

indices = np.where(masc)

indices

x[indices] # este indexado es equivalente al indexado fancy x[masc]


I am working on the file "2-Numpy.ipynb".
The first task is:
Hasta el momento el arreglo numpy.ndarray luce como una lista Python (anidada). Entonces, ¿por qué simplemente no usar listas para hacer cálculos en lugar de crear un tipo nuevo de arreglo?
Existen varias razones:

Las listas Python son muy generales. Ellas pueden contener cualquier tipo de objeto. Sus tipos son asignados dinámicamente. Ellas no permiten usar funciones matemáticas tales como la multiplicación de matricies, el producto escalar, etc. El implementar tales funciones para las listas Python no sería muy eficiente debido a la asignación dinámica de su tipo.
Los arreglos Numpy tienen tipo estático y homogéneo. El tipo de elementos es determinado cuando se crea el arreglo.
Los arreglos Numpy son eficientes en el uso de memoria.
Debido a su tipo estático, se pueden desarrollar implementaciones rápidas de funciones matemáticas tales como la multiplicación y la suma de arreglos numpy usando lenguajes compilados (se usan C y Fortran).

Usando la propiedad dtype (tipo de dato) de un ndarray, podemos ver qué tipo de dato contiene un arreglo:
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Hermítico conjugado: transpuesta + conjugado
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Iterando sobre elementos de un arreglo
Generalmente, deseamos evitar iterar sobre los elementos de un arreglo donde sea posible (a cualquier precio!). La razón es que en un lenguaje interpretado como Python (o MATLAB), las iteraciones son realmente lentas comparadas con las operaciones vectorizadas.
Sin embargo, algunas veces es ineludible. En tales cases el bucle Python for es la forma más conveniente para iterar sobre un arreglo:
Can you write Python code for it?

v = np.array([1,2,3,4])

for elemento in v:
    print(elemento)

M = np.array([[1,2], [3,4]])

for fila in M:
    print("fila", fila)

    for elemento in fila:
        print(elemento)


I am working on the file "2-Numpy.ipynb".
The first task is:
La temperatura diaria promedio en Estocolmo en los últimos 200 años ha sido aproximadamente 6.2 C.
Desviación estándar y varianza
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Los cortes de índices son mutables: si se les asigna un nuevo valor el arreglo original es modificado:
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Los objetos v y M son ambos del tipo ndarray que provee el módulo numpy.
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Los índices negativos se cuentan desde el fin del arreglo (los índices positivos desde el comienzo):
Can you write Python code for it?

A = np.array([1,2,3,4,5])

A[-1] # el último elemento del arreglo

A[-3:] # los últimos 3 elementos


I am working on the file "2-Numpy.ipynb".
The first task is:
Manipulando arreglos
Indexando
Podemos indexar elementos en un arreglo usando paréntesis cuadrados e índices:
Can you write Python code for it?

# v es un vector, tiene por lo tanto sólo una dimensión, y requiere un índice
v[0]

# M es una matriz, es decir un arreglo bidimensional, requiere dos índices
M[1,1]


I am working on the file "2-Numpy.ipynb".
The first task is:
Más propiedades de los arreglos Numpy
Can you write Python code for it?

M.itemsize # los bits de cada elemento

M.nbytes # número de bytes

M.ndim # número de dimensiones


I am working on the file "2-Numpy.ipynb".
The first task is:
Ok, eso no funcionó porque no definimos la función Theta de modo que pueda manejar argumentos vectoriales...
Para obtener una version vectorizada de Theta podemos usar la función vectorize de Numpy. En muchos casos, puede vectorizar automáticamente una función:
Can you write Python code for it?

Theta_vec = np.vectorize(Theta)

Theta_vec(np.array([-3,-2,-1,0,1,2,3]))


I am working on the file "2-Numpy.ipynb".
The first task is:
Operaciones elemento a elemento entre arreglos
Cuando sumamos, sustraemos, multiplicados y dividimos dos arreglos, el comportamiento por defecto es operar elemento a elemento:
Can you write Python code for it?

A * A # multiplicación elemento a elemento

v1 * v1


I am working on the file "2-Numpy.ipynb".
The first task is:
Pero la función take también funciona sobre listas y otros objetos:
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Podemos asignar nuevos valores a los elementos de un arreglo usando el indexado:
Can you write Python code for it?

M[0,0] = 1

M

# también funciona para filas y columnas completas
M[1,:] = 0
M[:,2] = -1

M


I am working on the file "2-Numpy.ipynb".
The first task is:
Podemos extraer las partes reales e imaginarias de un arreglo con elementos complejos usando real y imag:
Can you write Python code for it?

np.real(C) # lo mismo que: C.real

np.imag(C) # lo mismo que: C.imag


I am working on the file "2-Numpy.ipynb".
The first task is:
Podemos también extraer el módulo y el argumento complejo
Can you write Python code for it?

np.angle(C+1) # Atención usuarios de MATLAB, se usa angle en lugar de arg

abs(C)


I am working on the file "2-Numpy.ipynb".
The first task is:
Podemos también usar máscaras de índices: Si la máscara de índice es un arreglo Numpy con tipo de dato booleano (bool), entonces un elemento es seleccionado (True) o no (False) dependiendo del valor de la máscara de índice en la posición de cada elemento:
Can you write Python code for it?

B = np.array([n for n in range(5)])
B

masc_fila = np.array([True, False, True, False, False])
B[masc_fila]

# lo mismo
masc_fila = np.array([1,0,1,0,0], dtype=bool)
B[masc_fila]


I am working on the file "2-Numpy.ipynb".
The first task is:
Se obtiene un error si intentamos asignar un valor de un tipo equivocado a un elemento de un arreglo numpy:
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Si omitimos un índice de una arreglo multidimensional Numpy entrega la fila completa (o, en general, al arreglo de dimensión N-1 correspondiente)
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
Si queremos evitar este comportamiento, para así obtener un nuevo objecto B copiado desde A, pero totalmente independiente de A, necesitamos realizar una "copia produnfa" ("deep copy") usando la función copy:
Can you write Python code for it?

B = np.copy(A)

# ahora A no cambia si modificamos B
B[0,0] = -5

B

A


I am working on the file "2-Numpy.ipynb".
The first task is:
Usando arreglos en sentencias condicionales
Cuando se usan arreglos en sentencias condicionales, por ejemplo en sentencias if y otras expresiones booleanas, necesitamos usar any o bien all, que requiere que todos los elementos de un arreglo se evalúen con True:
Can you write Python code for it?

M

if (M > 5).any():
    print("al menos un elemento de M es mayor que 5")
else:
    print("ningún elemento de M es mayor que 5")

if (M > 5).all():
    print("todos los elementos de M son mayores que 5")
else:
    print("no todos los elementos de M son mayores que 5")


I am working on the file "2-Numpy.ipynb".
The first task is:
Usando numpy.savetxt podemos almacenar un arreglo Numpy a un archivo en formato CSV:
Can you write Python code for it?

M = random.rand(3,3)

M

np.savetxt("matriz-aleatoria.csv", M)

np.savetxt("matriz-aleatoria.csv", M, fmt='%.5f') # fmt especifica el formato


I am working on the file "2-Numpy.ipynb".
The first task is:
Vea también las funciones relacionadas: inner, outer, cross, kron, tensordot. Por ejemplo, introduzca help(kron).
Transformaciones de arreglos/matrices
Antes hemos usado .T para transponer un vector v. Podemos también usar la función transpose para conseguir el mismo resultado.
Otras funciones matemáticas que transforman objetos matriciales son:
Can you write Python code for it?

C = np.matrix([[1j, 2j], [3j, 4j]])
C

np.conjugate(C)


I am working on the file "2-Numpy.ipynb".
The first task is:
Vectorizando funciones
Como se ha mencionado en varias ocasiones, para obtener un buen rendimiento deberíamos tratar de evitar realizar bucles sobre los elementos de nuestros vectores y matrices, y en su lugar usar algoritmos vectorizados. El primer paso para convertir un algoritmo escalar a uno vectorizado es asegurarnos que las funciones que escribamos funcionen con argumentos vectoriales.
Can you write Python code for it?

def Theta(x):
    """
    implementación escalar de la función escalón de Heaviside.
    """
    if x >= 0:
        return 1
    else:
        return 0

Theta(np.array([-3,-2,-1,0,1,2,3]))


I am working on the file "2-Numpy.ipynb".
The first task is:
ceros y unos
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
diag
Can you write Python code for it?

# una matriz diagonal
np.diag([1,2,3])

# diagonal desplazada desde la diagonal principal
np.diag([1,2,3], k=1)


I am working on the file "2-Numpy.ipynb".
The first task is:
diag
Con la función diag podemos extraer la diagonal y las subdiagonales de un arreglo:
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
hstack and vstack
Can you write Python code for it?


I am working on the file "2-Numpy.ipynb".
The first task is:
media
Can you write Python code for it?

# la temperatura está almacenada en la columna 3
np.mean(data[:,3])
#data[:,3].mean()


I am working on the file "2-Numpy.ipynb".
The first task is:
min and max
Can you write Python code for it?

# valor mínimo del promedio diario de temperatura. Lo mismo que min(data[:,3])
data[:,3].min()

# valor máximo del promedio diario de temperatura. Lo mismo que max(data[:,3])
data[:,3].max()


I am working on the file "2-Numpy.ipynb".
The first task is:
take
La función take es similar al indexado fancy descrito anteriormente:
Can you write Python code for it?

v2 = np.arange(-3,3)
v2

indices_fila = [1, 3, 5]
v2[indices_fila] # indexado fancy

v2.take(indices_fila)


I am working on the file "2-Numpy.ipynb".
The first task is:
Álgebra lineal
El vectorizar el código es la clave para realizar cálculos numéricos eficientes usando Python/Numpy. Esto significa que la mayor parte de un programa debería ser formulado en términos de operaciones con matrices y vectores, como por ejemplo la multiplicación de matrices.
Operaciones escalar-arreglo
Podemos usar los operadores aritméticos usuales para multiplicar, sumar, restar, y dividir arreglos por números (escalares):
Can you write Python code for it?


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
2.2. Pisanje datoteke
Kada želimo nešto zapisati u datoteku, potrebno je datoteku otvoriti funkcijom open(). Prvi argument sadrži naziv datoteke u koju ćemo zapisati željeni sadržaj. Drugi argument sadrži vrijednost w. Ako već postoji datoteka s nazivom iz prvog argumenta, postojeći podaci će biti obrisani, a novi će biti zapisani. U slučaju ako datoteka s nazivom iz prvog argumenta ne postoji, stvorit će se nova datoteka s tim nazivom. To ćemo pridružiti varijabli npr. f.
Zatim ćemo nad varijablom f pokretati metodu write(). Metoda write() prima kao argument niz znakova koji želimo zapisati u datoteku. Možemo pokrenuti više metoda write() uzastopno.
Konačno, nakon što završimo zapisivanje u datoteku, obavezno trebamo datoteku i zatvoriti metodom close(). Time dajemo uputu da više nećemo pristupati datoteci i u nju zapisivati. Nakon metode close() nije više moguće pisanje u datoteku.
Na sljedeći način ćemo pohraniti niz znakova u novu datoteku datoteka.txt:
Can you write Python code for it?

f=open('datoteka.txt','w')
f.write('Ovo je prvi red.')
f.write('Je li ovo drugi red?\nOvo je 2. red.')
f.write('\nOvo je 3. red.')
f.close()


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
2.4. Kodiranje
2.4.1. Unicode standard
Unicode (Unicode, Unicode) osigurava jedinstveni broj za svaki znak, neovisno o platformi, programu i jeziku. Unicode je međunarodni standard izdan prvi put 1991. godine (Unicode 1.0) od strane Unicode konzorcija, koji je zadužen za održavanje standarda. Posljednja verzija Unicode 9.0 izdana je u lipnju 2016. godine. Sadrži više od 128.000 znakova iz 135 pisama. Samo za usporedbu, verzija Unicode 7.0 izdana u lipnju 2014. godine sadržavala je više od 113.000 znakova iz 123 pisama, verzija Unicode 5.0 izdana u srpnju 2006. godine sadržavala je 99.000 znakova iz 64 pisama, dok je prva verzija Unicode 1.0 izdana u listopadu 1991. sadržavala je nešto više od 7.000 znakova iz 24 pisama.
Kodne stranice (Code page) su tablice jedinstvenih brojeva kojim se zapisuje određeni skup znakova. Unicode je jedan kodna stranica čiji skup sadrži više od 128.000 znakova. Međutim, postoje i druge kodne stranice. Npr. u Windows okruženju postoji ISO-8859-2 kodna stranica koja definira kodiranje i hrvatskih slova ili ISO-8859-5 kodna stranica koja definira kodiranje srpskih ćiriličnih slova.
Prije nastanka Unicodea, postojalo je više kodnih stranica, ali niti jednog standarda. To je izazivalo mnogo problema, jer jedinstveni broj za određeni znak u jednoj kodnoj stranici je u drugoj kodnoj stranici možda predstavljao neki drugi znak. Drugim riječima, dvije kodne stranice koriste isti broj za kodiranje dva različita znaka ili koriste različite brojeve za kodiranje istog znaka. To je i danas vidljivo kod slanja e-pošte kada su slova s dijakritičkim znakovima predstavljena nekim drugim slovima, znakovima ili njihovim kombinacijama. Kako ne bi dolazilo do pogreške u prijenosu podataka, uvođenje standarda za kodiranje znakova bilo je neophodno. Standard definira kojim brojem će biti kodiran koji znak, tako da kodiranje bude dosljedno.
Unicode standard organiziran je tako da se kodne vrijednosti svakog znaka pišu u heksadecimalnom sustavu. Na određeni znak upućuje se tako da se ispred heksadecimalne...
Can you write Python code for it?


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
2.4.2.1. Metoda encode()
Metoda encode() kodira niz znakova u navedenu kodnu stranicu. Kao argument metoda prima naziv kodne stranice (npr. utf8 ili iso-8859-2). Ova metoda koristi se pri pisanju u datoteku.
Za primjer stvorit ćemo novu datoteku tekst_iso.txt te kodirati u kodnu stranicu ISO-8859-2 sa sadržajem iz datoteke tekst.txt.
Can you write Python code for it?

tekst=open('tekst.txt').read().decode('utf8')

iso=open('tekst_iso.txt','w')
iso.write(tekst.encode('iso-8859-2'))
iso.close()


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Ako izračunamo dužinu niza šećer, primijetit ćemo da je dužina 7, a ne 5.
Can you write Python code for it?


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Ako usporedimo dužine obiju datoteka bez dekodiranja, dobit ćemo da nisu jednake dužine iako sadrže isti tekst.
Can you write Python code for it?


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Budući da je vrijednost r drugog argumenta opcionalna i pretpostavljena vrijednost, vrijednost možemo izostaviti i dobiti jednaku izlaznu vrijednost:
Can you write Python code for it?


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Isti problem možemo primijetiti kod datoteka. U sljedećem primjeru ispisujemo znak po znak iz datoteke tekst.txt:
Can you write Python code for it?

for znak in open('tekst.txt').read():
    print znak


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Međutim, ako pomoćnu varijablu red pretvorimo u mala slova metodom lower(), uvjet će biti ispunjen:
Can you write Python code for it?

for red in open('tekst.txt'):
    if red.lower()[0]=='d':
        print red


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Međutim, ako usporedimo dužine datoteka s ispravnim kodiranjem, dobit ćemo da su jednake dužine:
Can you write Python code for it?


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Možemo primijetiti da je slovo "ć" prikazano pomoću dva neidentificirana znaka. Ako nakon čitanja datoteke odredimo koju kodnu stranicu treba koristiti za dekodiranje, slovo "ć" će biti ispravno prikazano.
Can you write Python code for it?

for znak in open('tekst.txt').read().decode('utf8'):
    print znak


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Nad datotekom koju smo stvorili (datoteka.txt) možemo izračunati i ispisati dužinu svakog reda:
Can you write Python code for it?

for red in open('datoteka.txt'):
    print red
    print 'Dužina reda: ' + str(len(red))


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Problem možemo primijetiti i kod računanja duljine datoteke:
Can you write Python code for it?

print len(open('tekst.txt').read())
print len(open('tekst.txt').read().decode('utf8'))
print len(open('tekst.txt').read())==len(open('tekst.txt').read().decode('utf8'))


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
Pročitat ćemo datoteku datoteka.txt kako bismo provjerili jesmo li uspješno zapisali niz znakova.
Can you write Python code for it?


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
U for petlji pomoćna varijabla red poprima sadržaj reda datoteke, a kao tip podatka je niz znakova. Možemo provjeriti pomoću funkcije type():
Can you write Python code for it?

for red in open('tekst.txt'):
    print 'Red datoteke je sljedeći tip podatka: ' + str(type(red))


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
U ovom primjeru dužina posljednjeg reda je neispravno izračunata jer je slovo "ć" pročitano kao dva, umjesto kao jedan znak.
Can you write Python code for it?

for red in open('tekst.txt'):
    red=red.decode('utf8')
    print 'Dužina reda: ' + str(len(red))


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
U primjeru vidimo da vrijednost zapisa slova "š" iznosi \u0161, a vrijednost zapisa slova "ć" iznosi \u0107. Kada izračunamo dužinu zadanog niza, vidjet ćemo da iznosi 5 (odnosno koliko i ima znakova u nizu), a ne 7, kao što je iznosilo prije dekodiranja (odnosno kada su slova s dijakritičkim znakovima zapisana pomoću dva znaka, umjesto jednim znakom).
Can you write Python code for it?


I am working on the file "2-Rad_s_datotekama.ipynb".
The first task is:
U sljedećem primjeru možemo izbrojati koliko ima samoglasnika u datoteci datoteka.txt. Nakon metode read(), pokrenut ćemo metodu lower() kako bismo pretvorili sva slova u mala slova:
Can you write Python code for it?

broj_samoglasnika=0
for element in open('datoteka.txt').read().lower():
    if element in 'aeiou':
        broj_samoglasnika+=1
print broj_samoglasnika


I am working on the file "2-Refine.ipynb".
The first task is:
2.10 Quirks in Pandas
Can you write Python code for it?

df_ca_sample.iat[0, 0] = "Cal"

df_ca_sample.head()


I am working on the file "2-Refine.ipynb".
The first task is:
2.4 Aggregate the Data
To aggregate, we typically use the “group by” function, which involves the following steps

Splitting the data into groups based on some criteria
Applying a function to each group independently
Combining the results into a data structure
Can you write Python code for it?

df_mean = df.groupby("State", as_index=False).mean()

df_mean.shape

df_mean.head()


I am working on the file "2-Refine.ipynb".
The first task is:
2.7 Filter the Data
Lets start by filtering the data
- by location
- by Year
- by location & Year
Can you write Python code for it?

# Filter data for location California
df_cal = df[df["State"] == "California"]

df_cal.shape

# Filter data for year
df_2014 = df[df["year"] == 2014]

df_2014.shape

df_cal_2014 = df[(df["year"] == 2014) & (df["State"] == "California")]

df_cal_2014.shape

df_cal_2014.head()


I am working on the file "2-Refine.ipynb".
The first task is:
2.9 Sample the Data
Can you write Python code for it?

?df.sample

df_ca_sample = df[df.State=='California'].sample(n = 50, replace = True, random_state=123)

df_ca_sample.duplicated()

df_ca_sample.loc[8572]


I am working on the file "2-Refine.ipynb".
The first task is:
But changes when you copy
Can you write Python code for it?

df_ca_sample2 = df_ca_sample

df_ca_sample2.iat[0, 0] = "CA"
df_ca_sample2.head()

df_ca_sample.head()


I am working on the file "2-Refine.ipynb".
The first task is:
Exercise
Fill the missing value with a backward fill.
Fill the missing values with the mean for the column.
2.2 Quality of the Data
Lets check for completeness.
Say, do we have data on each date for all the 51 states?
Can you write Python code for it?


I am working on the file "2-Refine.ipynb".
The first task is:
Exercise Express median rent as a proportion of california's median rent (Compute it as a new column)
2.6 Merge the Data
Lets merge the demographic dataset with the price dataset
Can you write Python code for it?

# Let us change the column name region to State
df_demo = df_demo.rename(columns={'region':'State'})

df_demo.head()

# We can now merge Demographic and Price mean data into one single data frame
df_merge = pd.merge(df_mean, df_demo, how='inner', on='State')
df_merge.head()


I am working on the file "2-Refine.ipynb".
The first task is:
Exercise Find the % of hispanic population for the state with max white population (use df_demo)
2.8 Summarise the Data
We can use the describe function to get the summary stats for each column in the data frame
Can you write Python code for it?


I am working on the file "2-Refine.ipynb".
The first task is:
Exercise Get a cross tabulation: for each state, for each month, get the prices for the weekday as shown in the output
2.5 Derive the Data
Lets us load the demographic dataset and create a new column for others in the population
Can you write Python code for it?

df_demo = pd.read_csv("data/Demographics_State.csv")

df_demo.head()

df_demo["percent_other"] = 100 - df_demo["percent_white"] - df_demo["percent_black"] - df_demo["percent_asian"] - df_demo["percent_hispanic"]

df_demo.head()


I am working on the file "2-Refine.ipynb".
The first task is:
Exercise Show the list of dates that were missing. Hint Leverage df2. Both df1 and df2 have date as index
2.3  Parse the Data
Lets see if we can get the year, month, week and weekdays from the date. Pandas has got good built in functionality for timeseries data using the DatetimeIndex method
Can you write Python code for it?

df['year'] = pd.DatetimeIndex(df['date']).year
df['month'] = pd.DatetimeIndex(df['date']).month
df['week'] = pd.DatetimeIndex(df['date']).week
df['weekday'] = pd.DatetimeIndex(df['date']).weekday

df.dtypes

df.tail()

df['year'].value_counts()

df['month'].value_counts()

df["weekday"].value_counts()


I am working on the file "2-Refine.ipynb".
The first task is:
Fix the issue
Can you write Python code for it?

df_ca_sample3 = df_ca_sample2.copy()

df_ca_sample3.head()

df_ca_sample3.iat[0, 0] = "CALIFORNIA"
df_ca_sample3.head()

df_ca_sample2.head()


I am working on the file "2-Refine.ipynb".
The first task is:
Lets check the dates and see if they are all continuous
Can you write Python code for it?

df1 = df[df.State=='California'].copy()
df2 = df[df.State=='California'].copy()

df1.shape

print "Earliest Date:",df1.date.min(), "\n", "Latest Date:",df1.date.max(), "\n",  \
"Number of days in between them:", df1.date.max() - df1.date.min()

df1.groupby(['date']).size()

df1.set_index("date", inplace=True)
df2.set_index("date", inplace=True)

idx = pd.date_range(df1.index.min(), df1.index.max())

idx

df1 = df1.reindex(idx, fill_value=0)

df1.shape


I am working on the file "2-Refine.ipynb".
The first task is:
Pandas will represent missing value by NaN
What can we do this with missing value?
- Drop these rows / columns? Use .dropna(how='any')
- Fill with a dummy value? Use .fillna(value=dummy)
- Impute the cell with the most recent value? Use .fillna(method='ffill')
- Interpolate the amount in a linear fashion? Use .interpolate()
We use the inplace = True operator to avoid making a copy of the dataframe and changing the dataframe itself
Can you write Python code for it?

# Lets sort this data frame by State and Date
df.sort(columns=['State','date'], inplace=True)

# Lets fill the missing value with last available value
df.fillna(method = "ffill", inplace=True)

df.count()


I am working on the file "2-Refine.ipynb".
The first task is:
Pivot Table
Can you write Python code for it?

df.head()

pd.pivot_table(df, values='HighQ', index=['State'], columns=["weekday"] )


I am working on the file "2-Refine.ipynb".
The first task is:
We can also use convenience functions like sum(), count(), mean() etc. to calculate these
Can you write Python code for it?

df.HighQ.mean()

# Lets do this the hard way
df.HighQ.sum()

df.HighQ.count()

df.HighQ.sum()/df.HighQ.count()

df.HighQ.median()


I am working on the file "2-Reti-neurali-convolutive.ipynb".
The first task is:
Aggiungendo un parametro color=True alla funzione fetch_lfw_people potremmo scaricare i tre canali RGB invece di un singolo canale in scala di grigi. Di tutte le immagini teniamo da parte un 20% per andare a testare la nostra rete neurale:
Can you write Python code for it?

from sklearn import model_selection
(X_trn, X_tst, y_trn, y_tst) = model_selection.train_test_split(X, y, test_size=0.20)


I am working on the file "2-Reti-neurali-convolutive.ipynb".
The first task is:
Al posto di definire manualmente tutte le variabili e le operazioni della rete, iniziamo ad utilizzare gli strati già definiti nel modulo layers:
https://www.tensorflow.org/api_docs/python/tf/layers/.
Va sottolineato da subito come alcuni di questi strati si appoggiano interiormente a funzioni più semplici definite nel modulo nn, come lo strato per la convoluzione 2D (conv2D):
https://www.tensorflow.org/api_docs/python/tf/layers/conv2d

https://www.tensorflow.org/api_docs/python/tf/nn/conv2d
Questa duplicazione di classi non deve però confondere, in quanto è possibile usare in maniera equivalente ciascuno dei due; le funzioni in nn hanno in genere meno parametri e meno flessibilità, e sono preferibili quando questa flessibilità non è richiesta.
Per uniformità, in questo tutorial useremo solo le funzioni definite nel modulo layers. La nostra rete sarà così composta:

Uno strato convolutivo, con 64 filtri 5x5.
Uno strato di pooling 2x2.
Un secondo strato convolutivo, con 32 filtri 5x5.
Un secondo strato di pooling 2x2.
Uno strato interamente connesso con 20 neuroni.
Un neurone di uscita con la classe desiderata.

Questa non è ovviamente l'unica scelta, né tantomeno la migliore, il che richiederebbe un'ottimizzazione più completa di tutto il design della rete (si veda ad esempio questo articolo). Questo genere di alternanza di strati convolutivi e pooling, di dimensioni sempre più piccole, e seguito da strati densamente connessi è però tipico nella maggior parte delle applicazioni con reti non molto grandi. Considereremo il design di reti più complesse e la loro ottimizzazione in tutorial successivi; va detto da subito, comunque, che spesso questa fase di costruzione è più un unirsi di esperienza ed 'arte', piuttosto che il susseguirsi di regole precise.
Iniziamo definendo il primo strato convolutivo:
Can you write Python code for it?

conv1 = tf.layers.conv2d(X_tf, 64, (5,5), activation=tf.nn.relu, name='conv1')