I am working on the file "4 Stats for data analysis/Homework/7 test non-parametric tests/Test Non-parametric test.ipynb".
The first task is:
На данных предыдущей задачи проверьте гипотезу об одинаковой средней температуре воздуха в дни, когда уплотнительный кольца повреждались, и дни, когда повреждений не было. Используйте перестановочный критерий и двустороннюю альтернативу. Чему равен достигаемый уровень значимости? Округлите до четырёх знаков после десятичной точки.
Чтобы получить такое же значение, как мы:
установите random seed = 0;
возьмите 10000 перестановок.
Can you write Python code for it?

def permutation_t_stat_ind(sample1, sample2):
    return np.mean(sample1) - np.mean(sample2)

def get_random_combinations(n1, n2, max_combinations):
    index = range(n1 + n2)
    indices = set([tuple(index)])
    for i in range(max_combinations - 1):
        np.random.shuffle(index)
        indices.add(tuple(index))
    return [(index[:n1], index[n1:]) for index in indices]

def permutation_zero_dist_ind(sample1, sample2, max_combinations=None):
    joined_sample = np.hstack((sample1, sample2))
    n1 = len(sample1)
    n = len(joined_sample)

    if max_combinations:
        indices = get_random_combinations(n1, len(sample2), max_combinations)
    else:
        indices = [(list(index), filter(lambda i: i not in index, range(n))) \
                    for index in itertools.combinations(range(n), n1)]

    distr = [joined_sample[list(i[0])].mean() - joined_sample[list(i[1])].mean() \
             for i in indices]
    return distr

def permutation_test(sample, mean, max_permutations = None, alternative='two-sided'):
    if alternative not in ('two-sided', 'less', 'greater'):
        raise ValueError("alternative not recognized\n"
                         "should be 'two-sided', 'less' or 'greater'")

    t_stat = permutation_t_stat_ind(sample, mean)

    zero_distr = permutation_zero_dist_ind(sample, mean, max_permutations)

    if alternative == 'two-sided':
        return sum([1. if abs(x) >= abs(t_stat) else 0. for x in zero_distr]) / len(zero_distr)

    if alternative == 'less':
        return sum([1. if x <= t_stat else 0. for x in zero_distr]) / len(zero_distr)

    if alternative == 'greater':
        return sum([1. if x >= t_stat else 0. for x in zero_distr]) / len(zero_distr)

np.random.seed(0)
print('p-value: %.4f' % permutation_test(challenger_broken['Temperature'].values,
                                         challenger_not_broken['Temperature'].values, max_permutations=10000))


I am working on the file "4 Stats for data analysis/Homework/9 test correlation analysis/Test Correlation analysis.ipynb".
The first task is:
<b>
Посмотрим на данные General Social Survey 2014 года и проанализируем, как связаны ответы на вопросы "Счастливы ли вы?" и "Довольны ли вы вашим финансовым положением?"
Не доволен  Более или менее Доволен
Не очень счастлив       197         111             33
Достаточно счастлив     382         685             331
Очень счастлив          110         342             333
Чему равно значение статистики хи-квадрат для этой таблицы сопряжённости? Округлите ответ до четырёх знаков после десятичной точки.
Can you write Python code for it?

happiness = np.array( [[197., 111., 33. ],
                       [382., 685., 331.],
                       [110., 342., 333.]] )

stats.chi2_contingency(happiness)
print('Chi2 stat value: %.4f' % stats.chi2_contingency(happiness)[0])


I am working on the file "4 Stats for data analysis/Homework/9 test correlation analysis/Test Correlation analysis.ipynb".
The first task is:
Проверьте гипотезу о равенстве долей любителей часто проводить вечера в баре среди мужчин и женщин. Посчитайте достигаемый уровень значимости, используя двустороннюю альтернативу. Введите номер первой значащей цифры (например, если вы получили 5.5×10−8, нужно ввести 8).
Can you write Python code for it?

def proportions_diff_z_stat_ind(sample1, sample2):
    n1 = np.sum(sample1)
    n2 = np.sum(sample2)

    p1 = sample1[0] / n1
    p2 = sample2[0] / n2
    P = float(p1*n1 + p2*n2) / (n1 + n2)

    return (p1 - p2) / np.sqrt(P * (1 - P) * (1. / n1 + 1. / n2))

def proportions_diff_z_test(z_stat, alternative = 'two-sided'):
    if alternative not in ('two-sided', 'less', 'greater'):
        raise ValueError("alternative not recognized\n"
                         "should be 'two-sided', 'less' or 'greater'")

    if alternative == 'two-sided':
        return 2 * (1 - stats.norm.cdf(np.abs(z_stat)))

    if alternative == 'less':
        return stats.norm.cdf(z_stat)

    if alternative == 'greater':
        return 1 - stats.norm.cdf(z_stat)

print('p-value: %f' % proportions_diff_z_test(proportions_diff_z_stat_ind(bars_sex[:,1], bars_sex[:,0])))


I am working on the file "4 Stats for data analysis/Homework/9 test correlation analysis/Test Correlation analysis.ipynb".
The first task is:
Сохраняется ли связь между признаками, если разбить выборку на северные и южные города? Посчитайте значения корреляции Пирсона между средней годовой смертностью и жёсткостью воды в каждой из двух подвыборок, введите наименьшее по модулю из двух значений, округлив его до четырёх знаков после десятичной точки.
Can you write Python code for it?

water_south = water[water.location == 'South']
water_north = water[water.location == 'North']
water_south.shape
water_north.shape

print('Pearson "South" correlation: %.4f' % stats.pearsonr(water_south.hardness, water_south.mortality)[0])
print('Pearson "North" correlation: %.4f' % stats.pearsonr(water_north.hardness, water_north.mortality)[0])


I am working on the file "4 Stats for data analysis/Homework/9 test correlation analysis/Test Correlation analysis.ipynb".
The first task is:
Чему в предыдущей задаче равно значение коэффициента V Крамера для рассматриваемых признаков? Округлите ответ до четырёх знаков после десятичной точки.
Can you write Python code for it?

def cramers_stat(confusion_matrix):
    chi2 = stats.chi2_contingency(confusion_matrix)[0]
    n = confusion_matrix.sum()
    return np.sqrt(chi2 / (n*(min(confusion_matrix.shape)-1)))

print('V Cramer stat value: %.4f' % cramers_stat(happiness))


I am working on the file "4 Stats for data analysis/Lectures notebooks/10 non-parametric tests rel/stat.non_parametric_tests_rel.ipynb".
The first task is:
Двухвыборочные критерии для связных выборок
$H_0\colon$ медианы веса до и после терапии совпадает
$H_1\colon$ медианы веса до и после тепрапии отличаются
Can you write Python code for it?

print '95%% confidence interval for mean weight before therapy: [%f, %f]' % zconfint(weight_data.Before)

print '95%% confidence interval for mean weight after therapy: [%f, %f]' % zconfint(weight_data.After)

pylab.hist(weight_data.After - weight_data.Before)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/10 non-parametric tests rel/stat.non_parametric_tests_rel.ipynb".
The first task is:
Загрузка данных
Can you write Python code for it?

weight_data = pd.read_csv('weight.txt', sep = '\t', header = 0)

weight_data.head()

pylab.figure(figsize=(12,4))

pylab.subplot(1,2,1)
pylab.grid()
pylab.hist(weight_data.Before, color = 'r')
pylab.xlabel('Before')

pylab.subplot(1,2,2)
pylab.grid()
pylab.hist(weight_data.After, color = 'b')
pylab.xlabel('After')

pylab.show()

weight_data.describe()


I am working on the file "4 Stats for data analysis/Lectures notebooks/10 non-parametric tests rel/stat.non_parametric_tests_rel.ipynb".
The first task is:
Критерий знаков
$H_0\colon P\left(X_1>X_2\right)=\frac1{2},$
$H_1\colon P\left(X_1>X_2\right)\neq\frac1{2}$
Can you write Python code for it?

print "M: %d, p-value: %f" % sign_test(weight_data.After - weight_data.Before)


I am working on the file "4 Stats for data analysis/Lectures notebooks/10 non-parametric tests rel/stat.non_parametric_tests_rel.ipynb".
The first task is:
Критерий знаковых рангов Уилкоксона
$H_0\colon med\left(X_1-X_2\right)=0,$
$H_1\colon med\left(X_1-X_2\right)\neq0$
Can you write Python code for it?

stats.wilcoxon(weight_data.After, weight_data.Before)

stats.wilcoxon(weight_data.After - weight_data.Before)


I am working on the file "4 Stats for data analysis/Lectures notebooks/10 non-parametric tests rel/stat.non_parametric_tests_rel.ipynb".
The first task is:
Перестановочный критерий
$H_0\colon \mathbb{E}(X_1 - X_2) = 0$
$H_1\colon \mathbb{E}(X_1 - X_2) \neq 0$
Can you write Python code for it?

def permutation_t_stat_1sample(sample, mean):
    t_stat = sum(map(lambda x: x - mean, sample))
    return t_stat

def permutation_zero_distr_1sample(sample, mean, max_permutations = None):
    centered_sample = map(lambda x: x - mean, sample)
    if max_permutations:
        signs_array = set([tuple(x) for x in 2 * np.random.randint(2, size = (max_permutations,
                                                                              len(sample))) - 1 ])
    else:
        signs_array =  itertools.product([-1, 1], repeat = len(sample))
    distr = [sum(centered_sample * np.array(signs)) for signs in signs_array]
    return distr

pylab.hist(permutation_zero_distr_1sample(weight_data.After - weight_data.Before, 0.,
                               max_permutations = 10000))
pylab.show()

def permutation_test(sample, mean, max_permutations = None, alternative = 'two-sided'):
    if alternative not in ('two-sided', 'less', 'greater'):
        raise ValueError("alternative not recognized\n"
                         "should be 'two-sided', 'less' or 'greater'")

    t_stat = permutation_t_stat_1sample(sample, mean)

    zero_distr = permutation_zero_distr_1sample(sample, mean, max_permutations)

    if alternative == 'two-sided':
        return sum([1. if abs(x) >= abs(t_stat) else 0. for x in zero_distr]) / len(zero_distr)

    if alternative == 'less':
        return sum([1. if x <= t_stat else 0. for x in zero_distr]) / len(zero_distr)

    if alternative == 'greater':
        return sum([1. if x >= t_stat else 0. for x in zero_distr]) / len(zero_distr)

print "p-value: %f" % permutation_test(weight_data.After - weight_data.Before, 0.,
                               max_permutations = 1000)

print "p-value: %f" % permutation_test(weight_data.After - weight_data.Before, 0.,
                               max_permutations = 50000)


I am working on the file "4 Stats for data analysis/Lectures notebooks/11 non-parametric tests rel ind/stat.non_parametric_tests_ind.ipynb".
The first task is:
Двухвыборочные критерии для независимых выборок
Can you write Python code for it?

print '95%% confidence interval for the mean: [%f, %f]' % zconfint(price2001)

print '95%% confidence interval for the mean: [%f, %f]' % zconfint(price2002)


I am working on the file "4 Stats for data analysis/Lectures notebooks/11 non-parametric tests rel ind/stat.non_parametric_tests_ind.ipynb".
The first task is:
Перестановочный критерий
$H_0\colon F_{X_1}(x) = F_{X_2}(x)$
$H_1\colon F_{X_1}(x) = F_{X_2}(x + \Delta), \Delta\neq 0$
Can you write Python code for it?

def permutation_t_stat_ind(sample1, sample2):
    return np.mean(sample1) - np.mean(sample2)

def get_random_combinations(n1, n2, max_combinations):
    index = range(n1 + n2)
    indices = set([tuple(index)])
    for i in range(max_combinations - 1):
        np.random.shuffle(index)
        indices.add(tuple(index))
    return [(index[:n1], index[n1:]) for index in indices]

def permutation_zero_dist_ind(sample1, sample2, max_combinations = None):
    joined_sample = np.hstack((sample1, sample2))
    n1 = len(sample1)
    n = len(joined_sample)

    if max_combinations:
        indices = get_random_combinations(n1, len(sample2), max_combinations)
    else:
        indices = [(list(index), filter(lambda i: i not in index, range(n))) \
                    for index in itertools.combinations(range(n), n1)]

    distr = [joined_sample[list(i[0])].mean() - joined_sample[list(i[1])].mean() \
             for i in indices]
    return distr

pylab.hist(permutation_zero_dist_ind(price2001, price2002, max_combinations = 1000))
pylab.show()

def permutation_test(sample, mean, max_permutations = None, alternative = 'two-sided'):
    if alternative not in ('two-sided', 'less', 'greater'):
        raise ValueError("alternative not recognized\n"
                         "should be 'two-sided', 'less' or 'greater'")

    t_stat = permutation_t_stat_ind(sample, mean)

    zero_distr = permutation_zero_dist_ind(sample, mean, max_permutations)

    if alternative == 'two-sided':
        return sum([1. if abs(x) >= abs(t_stat) else 0. for x in zero_distr]) / len(zero_distr)

    if alternative == 'less':
        return sum([1. if x <= t_stat else 0. for x in zero_distr]) / len(zero_distr)

    if alternative == 'greater':
        return sum([1. if x >= t_stat else 0. for x in zero_distr]) / len(zero_distr)

print "p-value: %f" % permutation_test(price2001, price2002, max_permutations = 10000)

print "p-value: %f" % permutation_test(price2001, price2002, max_permutations = 50000)


I am working on the file "4 Stats for data analysis/Lectures notebooks/11 non-parametric tests rel ind/stat.non_parametric_tests_ind.ipynb".
The first task is:
Ранговый критерий Манна-Уитни
$H_0\colon F_{X_1}(x) = F_{X_2}(x)$
$H_1\colon F_{X_1}(x) = F_{X_2}(x + \Delta), \Delta\neq 0$
Can you write Python code for it?


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Выводы
Итоговая модель объясняет 40% вариации логарифма отклика.
Can you write Python code for it?

plt.figure(figsize(16,7))
plt.subplot(121)
scatter(data['wage'],np.exp(fitted.fittedvalues))
plt.xlabel('Wage', fontsize=14)
plt.ylabel('Exponentiated predictions', fontsize=14)
plt.xlim([0,50])

plt.subplot(122)
scatter(np.log(data['wage']),fitted.fittedvalues)
plt.xlabel('Log wage', fontsize=14)
plt.ylabel('Predictions', fontsize=14)
plt.xlim([0,4])
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
На втором графике видна квадратичная зависимость остатков от опыта работы. Попробуем добавить к признакам квадрат опыта работы, чтобы учесть этот эффект.
Добавляем квадрат опыта работы
Can you write Python code for it?

m3 = smf.ols('np.log(wage) ~ exper + np.power(exper,2) + union + goodhlth + black + female +'\
                            'married + service + educ + belowavg + aboveavg', data=data)
fitted = m3.fit()
print fitted.summary()

plt.figure(figsize(16,7))
plt.subplot(121)
sc.stats.probplot(fitted.resid, dist="norm", plot=pylab)
plt.subplot(122)
np.log(fitted.resid).plot.hist()
plt.xlabel('Residuals', fontsize=14)
plt.figure(figsize(16,5))
plt.subplot(131)
scatter(data['educ'],fitted.resid)
plt.xlabel('Education', fontsize=14)
plt.ylabel('Residuals', fontsize=14)
plt.subplot(132)
scatter(data['exper'],fitted.resid)
plt.xlabel('Experience', fontsize=14)
plt.ylabel('Residuals', fontsize=14)
plt.subplot(133)
scatter(data['exper']**2,fitted.resid)
plt.xlabel('Experience^2', fontsize=14)
plt.ylabel('Residuals', fontsize=14)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Один человек в выборке получает 77.72\$ в час, остальные — меньше 45\$; удалим этого человека, чтобы регрессия на него не перенастроилась.
Can you write Python code for it?


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Оценим сбалансированность выборки по категориальным признакам:
Can you write Python code for it?

print raw.union.value_counts()
print raw.goodhlth.value_counts()
print raw.black.value_counts()
print raw.female.value_counts()
print raw.married.value_counts()
print raw.service.value_counts()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Ошибки гетероскедастичны, значит, значимость признаков может определяться неверно. Сделаем поправку Уайта:
Can you write Python code for it?

m4 = smf.ols('np.log(wage) ~ exper + np.power(exper,2) + union + goodhlth + black + female +'\
                            'married + service + educ + belowavg + aboveavg', data=data)
fitted = m4.fit(cov_type='HC1')
print fitted.summary()

plt.figure(figsize(16,7))
plt.subplot(121)
sc.stats.probplot(fitted.resid, dist="norm", plot=pylab)
plt.subplot(122)
np.log(fitted.resid).plot.hist()
plt.xlabel('Residuals', fontsize=14)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Посмотрим на матрицу диаграмм рассеяния по количественным признакам:
Can you write Python code for it?

pd.tools.plotting.scatter_matrix(raw[['wage', 'exper', 'educ', 'looks']], alpha=0.2,
                                 figsize=(15, 15), diagonal='hist')
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Посмотрим на распределение остатков:
Can you write Python code for it?

plt.figure(figsize(16,7))
plt.subplot(121)
sc.stats.probplot(fitted.resid, dist="norm", plot=pylab)
plt.subplot(122)
np.log(fitted.resid).plot.hist()
plt.xlabel('Residuals', fontsize=14)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Посмотрим на распределение оценок привлекательности:
Can you write Python code for it?

plt.figure(figsize(8,7))
data.groupby('looks')['looks'].agg(lambda x: len(x)).plot(kind='bar', width=0.9)
plt.xticks(rotation=0)
plt.xlabel('Looks', fontsize=14)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Посмотрим на распределение целевого признака — уровня заработной платы:
Can you write Python code for it?

plt.figure(figsize(16,7))
plt.subplot(121)
data['wage'].plot.hist()
plt.xlabel('Wage', fontsize=14)

plt.subplot(122)
np.log(data['wage']).plot.hist()
plt.xlabel('Log wage', fontsize=14)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Посмотрим, не стала ли модель от удаления трёх признаков значимо хуже, с помощью критерия Фишера:
Can you write Python code for it?

print "F=%f, p=%f, k1=%f" % m4.fit().compare_f_test(m5.fit())


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Постановка
По 1260 опрошенным имеются следующие данные:

заработная плата за час работы, $;
опыт работы, лет;
образование, лет;
внешняя привлекательность, в баллах от 1 до 5;
бинарные признаки: пол, семейное положение, состояние здоровья (хорошее/плохое), членство в профсоюзе, цвет кожи (белый/чёрный), занятость в сфере обслуживания (да/нет).

Требуется оценить влияние внешней привлекательности на уровень заработка с учётом всех остальных факторов.
Hamermesh D.S., Biddle J.E. (1994) Beauty and the Labor Market, American Economic Review, 84, 1174–1194.
Данные:
Can you write Python code for it?

raw = pd.read_csv("beauty.csv", sep=";", index_col=False)
raw.head()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Построение модели
Простейшая модель
Построим линейную модель по всем признакам.
Can you write Python code for it?

m1 = smf.ols('wage ~ exper + union + goodhlth + black + female + married +'\
                    'service + educ + belowavg + aboveavg',
             data=data)
fitted = m1.fit()
print fitted.summary()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Теперь стало лучше. Посмотрим теперь на зависимость остатков от непрерывных признаков:
Can you write Python code for it?

plt.figure(figsize(16,7))
plt.subplot(121)
scatter(data['educ'],fitted.resid)
plt.xlabel('Education', fontsize=14)
plt.ylabel('Residuals', fontsize=14)
plt.subplot(122)
scatter(data['exper'],fitted.resid)
plt.xlabel('Experience', fontsize=14)
plt.ylabel('Residuals', fontsize=14)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
У каждого признака все значения встречаются достаточно много раз, так что всё в порядке.
Предобработка
Can you write Python code for it?


I am working on the file "4 Stats for data analysis/Lectures notebooks/14 regression/stat.regression.ipynb".
The first task is:
Удаляем незначимые признаки
В предыдущей модели незначимы: цвет кожи, здоровье, семейное положение. Удалим их. Индикатор привлекательности выше среднего тоже незначим, но удалять его не будем, потому что это одна из переменных, по которым на нужно в конце ответить на вопрос.
Can you write Python code for it?

m5 = smf.ols('np.log(wage) ~ exper + np.power(exper,2) + union + female + service + educ +'\
                            'belowavg + aboveavg', data=data)
fitted = m5.fit(cov_type='HC1')
print fitted.summary()

plt.figure(figsize(16,7))
plt.subplot(121)
sc.stats.probplot(fitted.resid, dist="norm", plot=pylab)
plt.subplot(122)
np.log(fitted.resid).plot.hist()
plt.xlabel('Residuals', fontsize=14)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/2 proporion conf int/stat.proporion_conf_int.ipynb".
The first task is:
Доверительный интервал для доли
Can you write Python code for it?

from statsmodels.stats.proportion import proportion_confint


I am working on the file "4 Stats for data analysis/Lectures notebooks/2 proporion conf int/stat.proporion_conf_int.ipynb".
The first task is:
Размер выборки для интервала заданной ширины
Can you write Python code for it?

from statsmodels.stats.proportion import samplesize_confint_proportion

n_samples = int(np.ceil(samplesize_confint_proportion(random_sample.mean(), 0.01)))
n_samples

np.random.seed(1)
random_sample = np.random.choice(statistical_population, size = n_samples)

normal_interval = proportion_confint(sum(random_sample), len(random_sample), method = 'normal')

print 'normal_interval [%f, %f] with width %f' % (normal_interval[0],
                                                  normal_interval[1],
                                                  normal_interval[1] - normal_interval[0])


I am working on the file "4 Stats for data analysis/Lectures notebooks/2 proporion conf int/stat.proporion_conf_int.ipynb".
The first task is:
Точечная оценка доли
Can you write Python code for it?


I am working on the file "4 Stats for data analysis/Lectures notebooks/3 two proporions conf int upd/stat.two_proporions_conf_int_upd.ipynb".
The first task is:
Доверительный интервал для разности долей (связанные выборки)
$X_1$ \ $X_2$ | 1| 0 | $\sum$
  ------------- | -------------|
  1  | e | f | e + f
  0  | g | h | g + h
  $\sum$ | e + g| f + h | n
$$ \hat{p}_1 = \frac{e + f}{n}$$
$$ \hat{p}_2 = \frac{e + g}{n}$$
$$ \hat{p}_1 - \hat{p}_2 = \frac{f - g}{n}$$
$$\text{Доверительный интервал для }p_1 - p_2\colon \;\;  \frac{f - g}{n} \pm z_{1-\frac{\alpha}{2}}\sqrt{\frac{f + g}{n^2} - \frac{(f - g)^2}{n^3}}$$
Can you write Python code for it?

def proportions_confint_diff_rel(sample1, sample2, alpha = 0.05):
    z = scipy.stats.norm.ppf(1 - alpha / 2.)
    sample = zip(sample1, sample2)
    n = len(sample)

    f = sum([1 if (x[0] == 1 and x[1] == 0) else 0 for x in sample])
    g = sum([1 if (x[0] == 0 and x[1] == 1) else 0 for x in sample])

    left_boundary = float(f - g) / n  - z * np.sqrt(float((f + g)) / n**2 - float((f - g)**2) / n**3)
    right_boundary = float(f - g) / n  + z * np.sqrt(float((f + g)) / n**2 - float((f - g)**2) / n**3)
    return (left_boundary, right_boundary)

print "confidence interval: [%f, %f]" % proportions_confint_diff_rel(data.banner_a, data.banner_b)


I am working on the file "4 Stats for data analysis/Lectures notebooks/3 two proporions conf int upd/stat.two_proporions_conf_int_upd.ipynb".
The first task is:
Загрузка данных
Can you write Python code for it?

data = pd.read_csv('banner_click_stat.txt', header = None, sep = '\t')
data.columns = ['banner_a', 'banner_b']

data.head()

data.describe()


I am working on the file "4 Stats for data analysis/Lectures notebooks/4 bootstrap intervals/stat.bootstrap_intervals.ipynb".
The first task is:
Bootstrap
Can you write Python code for it?

def get_bootstrap_samples(data, n_samples):
    indices = np.random.randint(0, len(data), (n_samples, len(data)))
    samples = data[indices]
    return samples

def stat_intervals(stat, alpha):
    boundaries = np.percentile(stat, [100 * alpha / 2., 100 * (1 - alpha / 2.)])
    return boundaries


I am working on the file "4 Stats for data analysis/Lectures notebooks/4 bootstrap intervals/stat.bootstrap_intervals.ipynb".
The first task is:
Загрузка данных
Время ремонта телекоммуникаций
Verizon — основная региональная телекоммуникационная компания (Incumbent Local Exchange Carrier, ILEC) в западной
части США. В связи с этим данная компания обязана предоставлять сервис ремонта телекоммуникационного оборудования
не только для своих клиентов, но и для клиентов других локальных телекоммуникационых компаний (Competing Local Exchange Carriers, CLEC). При этом в случаях, когда время ремонта оборудования для клиентов других компаний существенно выше, чем для собственных, Verizon может быть оштрафована.
Can you write Python code for it?

data = pd.read_csv('verizon.txt', sep='\t')
data.shape

data.head()

data.Group.value_counts()

pylab.figure(figsize(12, 5))
pylab.subplot(1,2,1)
pylab.hist(data[data.Group == 'ILEC'].Time, bins = 20, color = 'b', range = (0, 100), label = 'ILEC')
pylab.legend()

pylab.subplot(1,2,2)
pylab.hist(data[data.Group == 'CLEC'].Time, bins = 20, color = 'r', range = (0, 100), label = 'CLEC')
pylab.legend()

pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/7 student tests/stat.student_tests.ipynb".
The first task is:
Treatment effects of methylphenidate
В рамках исследования эффективности препарата метилфенидат 24 пациента с синдромом дефицита внимания и гиперактивности в течение недели принимали либо метилфенидат, либо плацебо. В конце недели каждый пациент проходили тест на способность к подавлению импульсивных поведенческих реакций. На втором этапе плацебо и препарат менялись, и после недельного курса каждый испытуемые проходили второй тест.
Требуется оценить эффект применения препарата.
Pearson D.A, Santos C.W., Casat C.D., et al. (2004). Treatment effects of methylphenidate on cognitive functioning in children with mental retardation and ADHD. Journal of the American Academy of Child and Adolescent Psychiatry, 43(6), 677–685.
Can you write Python code for it?

data = pd.read_csv('ADHD.txt', sep = ' ', header = 0)
data.columns = ['Placebo', 'Methylphenidate']

data.plot.scatter('Placebo', 'Methylphenidate', c = 'r', s = 30)
pylab.grid()
pylab.plot(range(100), c = 'black')
pylab.xlim((20, 80))
pylab.ylim((20, 80))
pylab.show()

data.plot.hist()
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/7 student tests/stat.student_tests.ipynb".
The first task is:
Двухвыборочный критерий Стьюдента (зависмые выборки)
Для того, чтобы использовать критерй Стьюдента для связанных выборок, давайте проверим, что распределение попарных разностей существенно не отличается от нормального.
Can you write Python code for it?

stats.probplot(data.Placebo - data.Methylphenidate, dist = "norm", plot = pylab)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/7 student tests/stat.student_tests.ipynb".
The first task is:
Двухвыборочный критерий Стьюдента (независимые выборки)
Для того, чтобы использовать двухвыборочный критерий Стьюдента, убедимся, что распределения в выборках существенно не отличаются от нормальных.
Can you write Python code for it?

pylab.figure(figsize=(12,8))
pylab.subplot(2,2,1)
stats.probplot(data.Placebo, dist="norm", plot=pylab)
pylab.subplot(2,2,2)
stats.probplot(data.Methylphenidate, dist="norm", plot=pylab)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/7 student tests/stat.student_tests.ipynb".
The first task is:
Критерий Стьюдента:
$H_0\colon$ средние значения способности к подавлению испульсивных поведенческих реакций одинаковы для пациентов, принимавших препарат, и для пациентов, принимавших плацебо.
$H_1\colon$ не одинаковы.
Can you write Python code for it?

stats.ttest_rel(data.Methylphenidate, data.Placebo)

print "95%% confidence interval: [%f, %f]" % DescrStatsW(data.Methylphenidate - data.Placebo).tconfint_mean()


I am working on the file "4 Stats for data analysis/Lectures notebooks/7 student tests/stat.student_tests.ipynb".
The first task is:
Одновыборочный критерий Стьюдента
Исходя из того, что способность к подавлению испульсивных поведенческих реакций измеряется по шкале [0, 100], можно предположить, что при хорошей калибровке теста средняя способоность к подавлению реакций в популяции составляет 50. Тогда для того, чтобы проверить гипотезу о том, что пациенты в выборке действительно в среднем хуже справляются с подавлением импульсивных реакций (нуждаются в лечении), давайте проверим, что их способность к подавлению реакций отличается от средней (не равна 50).
$H_0\colon$ среднее значение способности к подавлению испульсивных поведенческих реакций равно 50.
$H_1\colon$ не равно.
Can you write Python code for it?

stats.ttest_1samp(data.Placebo, 50.0)

print "95%% confidence interval: [%f, %f]" % zconfint(data.Placebo)


I am working on the file "4 Stats for data analysis/Lectures notebooks/7 student tests/stat.student_tests.ipynb".
The first task is:
С помощью критерия Стьюдента проверим гипотезу о развенстве средних двух выборок.
Критерий Стьюдента:
$H_0\colon$ средние значения способности к подавлению испульсивных поведенческих реакций одинаковы для пациентов, принимавших препарат, и для пациентов, принимавших плацебо.
$H_0\colon$ не одинаковы.
Can you write Python code for it?

scipy.stats.ttest_ind(data.Placebo, data.Methylphenidate, equal_var = False)

cm = CompareMeans(DescrStatsW(data.Methylphenidate), DescrStatsW(data.Placebo))
print "95%% confidence interval: [%f, %f]" % cm.tconfint_diff(usevar='unequal')


I am working on the file "4 Stats for data analysis/Lectures notebooks/9 non-parametric tests 1sample/stat.non_parametric_tests_1sample.ipynb".
The first task is:
Загрузка данных
Can you write Python code for it?

mouses_data = pd.read_csv('mirror_mouses.txt', header = None)
mouses_data.columns = ['proportion_of_time']

mouses_data

mouses_data.describe()

pylab.hist(mouses_data.proportion_of_time)
pylab.show()


I am working on the file "4 Stats for data analysis/Lectures notebooks/9 non-parametric tests 1sample/stat.non_parametric_tests_1sample.ipynb".
The first task is:
Критерий знаковых рангов Вилкоксона
Can you write Python code for it?

m0 = 0.5
stats.wilcoxon(mouses_data.proportion_of_time - m0)


I am working on the file "4 Stats for data analysis/Lectures notebooks/9 non-parametric tests 1sample/stat.non_parametric_tests_1sample.ipynb".
The first task is:
Одновыборочные критерии
Can you write Python code for it?

print '95%% confidence interval for the mean time: [%f, %f]' % zconfint(mouses_data)


I am working on the file "4 Stats for data analysis/Lectures notebooks/9 non-parametric tests 1sample/stat.non_parametric_tests_1sample.ipynb".
The first task is:
Перестановочный критерий
$H_0\colon$ среднее равно 0.5
$H_1\colon$ среднее не равно 0.5
Can you write Python code for it?

def permutation_t_stat_1sample(sample, mean):
    t_stat = sum(map(lambda x: x - mean, sample))
    return t_stat

permutation_t_stat_1sample(mouses_data.proportion_of_time, 0.5)

def permutation_zero_distr_1sample(sample, mean, max_permutations = None):
    centered_sample = map(lambda x: x - mean, sample)
    if max_permutations:
        signs_array = set([tuple(x) for x in 2 * np.random.randint(2, size = (max_permutations,
                                                                              len(sample))) - 1 ])
    else:
        signs_array =  itertools.product([-1, 1], repeat = len(sample))
    distr = [sum(centered_sample * np.array(signs)) for signs in signs_array]
    return distr

pylab.hist(permutation_zero_distr_1sample(mouses_data.proportion_of_time, 0.5), bins = 15)
pylab.show()

def permutation_test(sample, mean, max_permutations = None, alternative = 'two-sided'):
    if alternative not in ('two-sided', 'less', 'greater'):
        raise ValueError("alternative not recognized\n"
                         "should be 'two-sided', 'less' or 'greater'")

    t_stat = permutation_t_stat_1sample(sample, mean)

    zero_distr = permutation_zero_distr_1sample(sample, mean, max_permutations)

    if alternative == 'two-sided':
        return sum([1. if abs(x) >= abs(t_stat) else 0. for x in zero_distr]) / len(zero_distr)

    if alternative == 'less':
        return sum([1. if x <= t_stat else 0. for x in zero_distr]) / len(zero_distr)

    if alternative == 'greater':
        return sum([1. if x >= t_stat else 0. for x in zero_distr]) / len(zero_distr)

print "p-value: %f" % permutation_test(mouses_data.proportion_of_time, 0.5)

print "p-value: %f" % permutation_test(mouses_data.proportion_of_time, 0.5, 10000)


I am working on the file "4-1_Scikit_Learn.ipynb".
The first task is:
1. Selecting and adjusting a model
As we want to fit a linear model $y=ax+b$ through the data, we will import the Linear Regression module from scikit learn with sklearn.linear_model import LinearRegression.
As our model has a non null coefficient at the origin, the model needs an intercept. This can be tuned, along with several other parameters, see Scikit Learn's  linear_model documentation.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

model = LinearRegression(fit_intercept=True)
print(model)


I am working on the file "4-1_Scikit_Learn.ipynb".
The first task is:
3. Predicting from this fitted model
From a feature matrix, the method predict returns the predicted output from the fitted model.
Can you write Python code for it?

xFit = np.linspace(-2,12,21).reshape(-1, 1)

yFit = model.predict(xFit)

plt.scatter(x, y , label="data")
plt.plot(xFit, yFit , label="model",color="r")
plt.legend()


I am working on the file "4-1_Scikit_Learn.ipynb".
The first task is:
Let us use cross validation to compare linear model and linear model with intercept.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

model1 = LinearRegression(fit_intercept=True)
model2 = LinearRegression(fit_intercept=False)

model1.fit(xTrain,yTrain)
yPre1 = model1.predict(xTest)
error1 = np.linalg.norm(yTest-yPre1)

model2.fit(xTrain,yTrain)
yPre2 = model2.predict(xTest)
error2 = np.linalg.norm(yTest-yPre2)

print("Testing Error with intercept:", error1, "\t without intercept:" ,error2)

plt.scatter(xTrain, yTrain ,  label="Train data")
plt.scatter(xTest, yTest , color= 'k' , label="Test data")
plt.plot(xTest, yPre1 , color='r', label="model w/ intercept (err = {:.1f})".format(error1))
plt.plot(xTest, yPre2 , color='m', label="model w/o intercept (err = {:.1f})".format(error2))
plt.legend()


I am working on the file "4-1_Scikit_Learn.ipynb".
The first task is:
Once the model is fitted, one can observe the learned coefficients:
* coef_ for the model coefficients ($a$ here)
* intercept_ foe the intercept ($b$ here)
Can you write Python code for it?

print("Learned coefficients: a = {:.6f} \t b = {:.6f}".format(float(model.coef_),float(model.intercept_)))
print("True coefficients:    a = {:.6f} \t b = {:.6f}".format(a,b))


I am working on the file "4-1_Scikit_Learn.ipynb".
The first task is:
This terminates our model tuning. Notice that we have described our model, but no learning or fitting has been done.
2. Fitting the model to the data
Applying our model to the data $(x,y)$ is done using the fit method.
Can you write Python code for it?


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
Even numerical values were encoded, as we are going to normalize, it is not really important.
The normalization is done by removing the mean and equalizing the variance per feature, in addition, we are going to add an intercept.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler, add_dummy_feature

scaler = StandardScaler()
normFeatures = add_dummy_feature(scaler.fit_transform(num_features))

preproData = pd.DataFrame(normFeatures , columns=[ "intercept" ] + list(num_features.columns) )

preproData.describe().T


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
It can also be plotted in a fancier way with seaborn.
Can you write Python code for it?

import seaborn as sns

sns.heatmap(mat1, square=True, annot=True ,cbar=False)
plt.ylabel('true label')
plt.xlabel('predicted label')


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
Let us use the same linear SVM classifier. Obviously, there are misclassified points, the model is thus learnt not by maximizing the margin (which does not exist anymore) but by minimizing a penalty over misclassified data. This penalty takes the form of an allowance margin controlled by a parameter $C$. The smaller $C$ the more inclusive the margin. Finding a good value for $C$ is up to the data scientist.
Can you write Python code for it?

from sklearn.model_selection import train_test_split    # sklearn > ...

XTrain, XTest, yTrain, yTest = train_test_split(X,y,test_size = 0.5) # split data in two

model1 = SVC(kernel="linear",C=0.01)
model1.fit(XTrain,yTrain)

model2 = SVC(kernel="linear",C=100)
model2.fit(XTrain,yTrain)

plt.scatter(XTrain[:, 0], XTrain[:, 1], c=yTrain ,  cmap=plt.cm.Paired)
plot_svc_decision_function(model1)
plt.title("C = 0.01")

plt.scatter(XTrain[:, 0], XTrain[:, 1], c=yTrain ,  cmap=plt.cm.Paired)
plot_svc_decision_function(model2)
plt.title("C = 100")


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
Regression and Feature selection with the Lasso
The lasso problem is finding a regressor $w$ such that minimizes
$$  \frac{1}{2 n_{samples}} \|X w - y ||^2_2 + \alpha \|w\|_1 $$
and is popular for prediction as it simultaneously selects features thanks to the $\ell_1$-term. The greater $\alpha$ the fewer features.
Can you write Python code for it?

from sklearn.model_selection import train_test_split    # sklearn > ...

from sklearn.linear_model import Lasso

XTrain, XTest, yTrain, yTest = train_test_split(preproData,target,test_size = 0.25)

model = Lasso(alpha=0.1)
model.fit(XTrain,yTrain)


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
Regularization path
Selecting a good parameter $\alpha$ is the role of the data scientist. For instance, a easy way to do is the following.
Can you write Python code for it?

n_test = 15
alpha_tab = np.logspace(-10,1,base=2,num = n_test)
print(alpha_tab)

trainError = np.zeros(n_test)
testError = np.zeros(n_test)
featureNum = np.zeros(n_test)

for idx,alpha in enumerate(alpha_tab):
    model = Lasso(alpha=alpha)
    model.fit(XTrain,yTrain)
    yPredTrain = model.predict(XTrain)
    yPredTest = model.predict(XTest)

    trainError[idx] = np.linalg.norm(yPredTrain-yTrain["G3"].values)/yTrain.count()
    testError[idx] = np.linalg.norm(yPredTest-yTest["G3"].values)/yTest.count()
    featureNum[idx] = sum(model.coef_!=0)

alpha_opt = alpha_tab[np.argmin(testError)]

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
%matplotlib inline

plt.subplot(311)
plt.xscale("log")
plt.plot(alpha_tab, trainError,label="train error")
plt.xlim([min(alpha_tab),max(alpha_tab)])
plt.legend()
plt.xticks([])
plt.axvline(x=alpha_opt)
plt.ylabel("error")

plt.subplot(312)
plt.xscale("log")
plt.plot(alpha_tab, testError,'r',label="test error")
plt.xlim([min(alpha_tab),max(alpha_tab)])
#plt.ylim([0.19, 0.21])
plt.legend()
plt.axvline(x=alpha_opt)
plt.xticks([])
plt.ylabel("error")

plt.subplot(313)
plt.xscale("log")
plt.scatter(alpha_tab, featureNum)
plt.xlim([min(alpha_tab),max(alpha_tab)])
plt.ylim([0,28])
plt.axvline(x=alpha_opt)
plt.ylabel("nb. of features")
plt.xlabel("alpha")


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
Support Vector Machines (SVM) are based on learning a vector $w$ and an intercept $b$ such that the hyperplane $w^T x - b = 0$ separates the data i.e.  $a$ belongs to one class if  $w^T a - b > 0$ and the other elsewhere.
They were later extended to Kernel methods that is  $\kappa(w, a) - b = 0$ is now the separating curve where $\kappa$ is the kernel, typically:
* linear: $\kappa(x,y)= x^T y$ (original SVM)
* polynomial: $\kappa(x,y)= (x^T y)^d$
* Gaussian radial basis function (rfb):   $\kappa(x,y)= \exp( - \gamma \| x - y \|^2 )$
Can you write Python code for it?

from sklearn.svm import SVC # Support vector classifier i.e. Classifier by SVM

modelSVMLinear = SVC(kernel="linear")
modelSVMLinear.fit(X,y)


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
The following illustration can be found in the Python Data Science Handbook by Jake VanderPlas.
Can you write Python code for it?

def plot_svc_decision_function(model, ax=None, plot_support=True):
    """Plot the decision function for a 2D SVC"""
    if ax is None:
        ax = plt.gca()
    xlim = ax.get_xlim()
    ylim = ax.get_ylim()

    # create grid to evaluate model
    x = np.linspace(xlim[0], xlim[1], 30)
    y = np.linspace(ylim[0], ylim[1], 30)
    Y, X = np.meshgrid(y, x)
    xy = np.vstack([X.ravel(), Y.ravel()]).T
    P = model.decision_function(xy).reshape(X.shape)

    # plot decision boundary and margins
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])

    # plot support vectors
    if plot_support:
        ax.scatter(model.support_vectors_[:, 0],
                   model.support_vectors_[:, 1],
                   s=300, linewidth=1, facecolors='none');
    ax.set_xlim(xlim)
    ax.set_ylim(ylim)

plt.scatter(X[:, 0], X[:, 1], c=y ,  cmap=plt.cm.Paired)
plot_svc_decision_function(modelSVMLinear)


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
To find out which value of $C$ to use or globally the performance of the classifier, one can use Scikit Learn's classification metrics, for instance the confusion matrix.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

yFit1 = model1.predict(XTest)
yFit2 = model2.predict(XTest)

mat1 = confusion_matrix(yTest, yFit1)
mat2 = confusion_matrix(yTest, yFit2)

print('Model with C = 0.01')
print(mat1)
print("Model with C = 100")
print(mat2)


I am working on the file "4-2_Supervised_Learning.ipynb".
The first task is:
We can observe which coefficients are put to $0$ and which ones are positively/negatively correlated.
Can you write Python code for it?

print("Value      Feature")
for idx,val in enumerate(model.coef_):
    print("{:6.3f}      {}".format(val,preproData.columns[idx]))


I am working on the file "4-3_Unsupervised_Learning.ipynb".
The first task is:
As before, we proceed by selecting a KMeans model and fitting it to the data.
Can you write Python code for it?

from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=4)
kmeans.fit(X)


I am working on the file "4-3_Unsupervised_Learning.ipynb".
The first task is:
From the model, one can get the data points labels with the attribute <tt>labels_</tt> and the cluster centers with <tt>cluster_centers_</tt>.
Can you write Python code for it?

print(kmeans.labels_)
print(kmeans.cluster_centers_)

plt.scatter(X[:, 0], X[:, 1], c=kmeans.labels_)
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], c='r' , s = 100 , marker="*")


I am working on the file "4-3_Unsupervised_Learning.ipynb".
The first task is:
The different clusters have visibly been recovered. It is to be noted that from the cluster center, one can define Voronoi regions (regions that are closer to one center than any other one) that are exactly the predicted regions from the K Means algorithm.
Can you write Python code for it?

from scipy.spatial import Voronoi, voronoi_plot_2d

vor = Voronoi(kmeans.cluster_centers_)
voronoi_plot_2d(vor)
plt.scatter(X[:, 0], X[:, 1], c=kmeans.labels_)
plt.scatter(kmeans.cluster_centers_[:, 0], kmeans.cluster_centers_[:, 1], c='r' , s = 100 , marker="*")
plt.show()


I am working on the file "4-3_Unsupervised_Learning.ipynb".
The first task is:
We notice an important first vector that combines the 4 features (sepal_width seems less important).
We can now <tt>project</tt> the data onto these two vectors and plot the result to see if the classes are recognizable in this reduced space.
Can you write Python code for it?

projected = pca.transform(features)

plt.scatter(projected[:, 0], projected[:, 1], c=num_classes)
plt.xlabel('vec. 1')
plt.ylabel('vec. 2')


I am working on the file "4-3_Unsupervised_Learning.ipynb".
The first task is:
We see that the classes are way more separable this way than with just 2 features (see before). Furthermore, the vector 1 can almost be used alone to separate.
Exercises

Exercise: Clustering for color compression in images.
Take a black and white image with 256 gray levels, the (1D) vector of its pixel values can be obtained with the flatten method of NumPy.
The goal of this exercise is to use clustering to convert the 255 greyscale values of the pixels to 8 values. To do so:
* cluster the pixels values into 8 clusters and replace the pixels values with their cluster centroids.
* Compare with a uniform quantizer.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import matplotlib.cm as cm

#### IMAGE
img = mpimg.imread('img/flower.png')
img_gray =  0.2989 * img[:,:,0] + 0.5870 * img[:,:,1] + 0.1140 * img[:,:,2] # Apparently these are "good" coefficients to convert to grayscale
####

print(img_gray.shape)

plt.figure()
plt.xticks([]),plt.yticks([])
plt.title("Original")
plt.imshow(img_gray, cmap = cm.Greys_r)
plt.show()

pixels = img_gray.flatten().reshape(-1, 1)
pixels.shape


I am working on the file "4-4_Going_Further.ipynb".
The first task is:
Let us transform it into a 3-degree polynomial fit and perform the same linear regression.
Can you write Python code for it?

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(degree=3, include_bias=False) # 3 degree without degree 0 (no constant)
XPoly = poly.fit_transform(X)
print(XPoly[:5,])

modelPoly = LinearRegression().fit(XPoly, y)
yfitPoly = modelPoly.predict(XPoly)
plt.scatter(X, y)
plt.plot(X, yfit,color='r',label="linear regression")
plt.plot(X, yfitPoly,color='k',label="Polynomial regression (deg 3)")
plt.legend(loc = 'lower right')


I am working on the file "4-4_Going_Further.ipynb".
The first task is:
Linear regression will obviously be a bad fit.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

model = LinearRegression().fit(X, y)
yfit = model.predict(X)
plt.scatter(X, y)
plt.plot(X, yfit,color='r',label="linear regression")
plt.legend()


I am working on the file "4-4_Going_Further.ipynb".
The first task is:
Pipeline
This 2-step fitting (Polynomial transform + Linear regression) calls for a replicated dataset which can be costly. That is why Scikit Learn implement a pipeline method that allows to perform multiple fit/transform sequentially.
This pipeline can then be used as a model.
Can you write Python code for it?

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

polyFeat = PolynomialFeatures(degree=3, include_bias=False)
linReg = LinearRegression()

polyReg = Pipeline([ ('polyFeat',polyFeat) , ('linReg',linReg) ])

polyReg.fit(X, y) # X original not XPoly
yfitPolyNew = polyReg.predict(X)

plt.scatter(X, y)
plt.plot(X, yfit,color='r',label="linear regression")
plt.plot(X, yfitPolyNew,color='k',label="Polynomial regression (deg 3)")
plt.legend(loc = 'lower right')


I am working on the file "4-4_Going_Further.ipynb".
The first task is:
The vectorizer has registered the feature names and outed a sparse matrix that can be converted to a Dataframe.
Can you write Python code for it?

import pandas as pd
pd.DataFrame(X.toarray(), columns=vec.get_feature_names())


I am working on the file "4-4_Going_Further.ipynb".
The first task is:
Validation and Hyperparameters tuning
We saw above (see the lasso example of the regression part) some basic examples of how to:
* validate our model by splitting the dataset into training and testing set (using train_test_split)
* tune hyperparameter by looking at the error for different values
Scikit Learn actually provides some methods for that as well.
Validation
Scikit Learn offer a cross validation method that
* split the dataset in several groups
* for each of these group, fit the model on all but this group and computer the error on this one
This way all the data has gone thought the learning and validating sets hence the cross validation. This is illustrated by the following figure from the  Python Data Science Handbook by Jake VanderPlas.

The score computer is computed either as the standard score of the estimator or can be precised with the <tt>scoring</tt> option (see the available metrics ).
Warning All scorer objects follow the convention that higher return values are better than lower return values.
Let us compute the cross validation for our polynomial fit problem.
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

cv_score = cross_val_score(polyReg, X, y, cv=5, scoring="neg_mean_absolute_error") # 5 groups cross validation

print(cv_score)
print("Mean score:" , np.mean(cv_score))


I am working on the file "4-4_Going_Further.ipynb".
The first task is:
We can then get the best parameters and the corresponding model.
Can you write Python code for it?

grid.best_params_

best_model = grid.best_estimator_.fit(X,y)
overfit = polyReg.set_params(polyFeat__degree=15).fit(X,y)

Xplot = np.linspace(-1,10.5,100).reshape(-1, 1)
yBest = best_model.predict(Xplot)
yOver = overfit.predict(Xplot)

plt.scatter(X, y)
plt.plot(Xplot, yBest , 'r' ,label="Best polynomial")
plt.plot(Xplot, yOver , 'k' , label="overfitted (deg 15)")
plt.legend(loc = 'lower right')
plt.ylim([0,5])
plt.title("Best and overfitted models")


I am working on the file "4-4_Going_Further.ipynb".
The first task is:
We notice that the grid search based on cross validation helped discarded overfitted models (as they were bad on validation sets).
Text and Image Features
We already saw an example of feature extraction from categorical data. However, for some particular categorical data, dedicated tools exist. For instance, text and images.
Text feature extraction
In Learning applications, words are usually more important than letters, so a basic way to extract features is to construct one feature per present word and count the occurences of this word. This is known as word count. An approach to mitigate very present words (like "the" , "a" , etc) is term frequency inverse document frequency (tf-idf) which weights the occurence count by how often it appears.
Can you write Python code for it?

f = open('./data/poems/poe-raven.txt', 'r')
poe = f.read().replace('\n',' ').replace('.','').replace(',','').replace('-','')
poe

from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer()
X = vec.fit_transform([poe])
X


I am working on the file "4-Model.ipynb".
The first task is:
Correlation
Can you write Python code for it?

corr_bw_percapita_highq = stats.pearsonr(df_demo.per_capita_income, df_demo.HighQ)[0]
print corr_bw_percapita_highq


I am working on the file "4-Rječnici.ipynb".
The first task is:
Ako pak na ovaj način dohvaćamo vrijednost ključa koji ne postoji u rječniku, dobit ćemo vrijednost None:
Can you write Python code for it?

rjecnik={'a':5, 'b':3.8}
print rjecnik.get('c')


I am working on the file "4-Rječnici.ipynb".
The first task is:
Iz primjera je vidljivo da metode keys() i values() vraćaju listu čiji elementi liste su ključevi ili vrijednosti. Metoda items() vraća listu parova ne u uglatim, već u oblim zagradama. Ta metoda, naime, vraća listu n-torki. N-torke su tip podataka koji tek sada po prvi put susrećemo. N-torke su nepromjenjivi pandan promjenjivim listama te ćemo ih koristiti nešto kasnije, kada ćemo u rječnike pod ključevima htjeti bilježiti sekvencu riječi. Prisjetimo se, rječnici kao ključeve primaju samo nepromjenjive tipove podataka.
Za kraj, pogledajmo kako možemo sortirati rječnik, i to prema vrijednostima silazno. Kako ćemo rječnik koristiti za prebrojavanje događaja, sortiranje rječnika prema vrijednosti bit će nam potrebno za uvid u najčešće događaje.
Can you write Python code for it?


I am working on the file "4-Rječnici.ipynb".
The first task is:
Pri statističkoj analizi i modeliranju teksta, rječnike ćemo koristiti za prebrojavanje raznih događaja u tekstu. Zato nam je vrlo važan postupak koji za neki ključ vrijednost povećavamo za 1. Za to ćemo koristiti sljedeću naredbu:
Can you write Python code for it?

rjecnik={'a':5, 'b':3.8}
rjecnik['c']=rjecnik.get('c',0)+1
print rjecnik.get('c')


I am working on the file "4-Rječnici.ipynb".
The first task is:
Sintaksa rada s rječnicima slična je one s listama, jedino što se umjesto indeksa u listi u uglatoj zagradi definira ključ. Iz primjera je također vidljivo da u rječniku ne možemo imati dva ista ključa. Pridruživanjem druge vrijednosti pod postojeći ključ brišemo prethodnu vrijednost pod tim ključem.
Vrijednosti iz rječnika možemo dohvatiti na dva načina. Prvi je definiranjem ključa u uglatoj zagradi:
Can you write Python code for it?


I am working on the file "4-advancedML/Assigment1.ipynb".
The first task is:
1. Comparando composicionalidad y analogía.
Composicionalidad y analogía son dos mecanismos diferentes que se pueden usar con representaciones distribuidas. La idea es usar independientemente composicionalidad y analogía para resolver el mismo problema. El problema a resolver es encontrar el presidente de un país dado.
Primero usaremos composicionalidad. La función siguiente debe recibir el nombre de un país y retornar una lista de palabras que posiblemente corresponden a presidentes.
Por ejemplo, si la función se invoca con 'ecuador' como argumento:
```python

presidents_comp('ecuador')
[u'jamil_mahuad',
 u'presidencia',
 u'jose_maria_velasco_ibarra',
 u'republica',
 u'rafael_correa',
 u'gustavo_noboa',
 u'lucio_gutierrez',
 u'abdala_bucaram',
 u'vicepresidente',
 u'gabriel_garcia_moreno']
 ```
Can you write Python code for it?

def presidents_comp(country):
    return [elm[0] for elm in model.most_similar(positive=[country, 'presidente'])]

for country in ['colombia', 'venezuela', 'ecuador', 'brasil', 'argentina', 'chile']:
    print country
    for president in presidents_comp(country):
        print ' ', president


I am working on the file "4-advancedML/Assigment1.ipynb".
The first task is:
Busque más ejemplos en los que funcione y otros en los que no funcione. Explique.
Can you write Python code for it?

print ' FUNCIONA'
for palabra in 'salir verdad seco izquierda'.split():
    print palabra + ':', antonimo(palabra)

print '\n NO FUNCIONA'
for palabra in 'rico paz joven comunismo'.split():
    print palabra + ':', antonimo(palabra)


I am working on the file "4-advancedML/Assigment1.ipynb".
The first task is:
El siguiente paso es usar analogías para encontrar el presidente de un país dado.
Can you write Python code for it?

def presidents_analogy(country):
    return [elm[0] for elm in model.most_similar(positive=[country, 'hugo_chavez'], negative=['venezuela'])]

for country in ['colombia', 'venezuela', 'ecuador', 'brasil', 'argentina', 'chile']:
    print country
    for president in presidents_analogy(country):
        print ' ', president


I am working on the file "4-advancedML/Assigment1.ipynb".
The first task is:
La idea es implementar la misma funcionalidad por nuestra cuenta. La condición es que solo podemos usar la función  similarity de Gensim la cual calcula la similitud de dos palabras:
Can you write Python code for it?

print model.similarity('azul', 'rojo')
print model.similarity('azul', 'abajo')

import numpy as np
def no_es_como_las_otras(word_list):
    size = len(word_list)
    word_matrix = np.zeros(shape=(size, size))
    for row in xrange(size):
        for column in xrange(size):
            word_matrix[row, column] = model.similarity(word_list[row], word_list[column])
    sum_columns = word_matrix.sum(axis=0)
    return word_list[np.argmin(sum_columns)]

print no_es_como_las_otras("azul rojo abajo verde".split())
print no_es_como_las_otras("azul izquierda abajo derecha".split())
print no_es_como_las_otras("colombia suiza carro venezuela".split())
print no_es_como_las_otras("colombia suiza argentina venezuela".split())


I am working on the file "4-files & conditionals-short.ipynb".
The first task is:
Challenge - making choices 2:
True and False are special words in Python called booleans which represent true and false statements. However, they aren’t the only values in Python that are true and false. In fact, any value can be used in an if or elif. After reading and running the code below, explain what the rule is for which values are considered true and which are considered false.
python
if '':
    print('empty string is true')
if 'word':
    print('word is true')
if []:
    print('empty list is true')
if [1, 2, 3]:
    print('non-empty list is true')
if 0:
    print('zero is true')
if 1:
    print('one is true')
Can you write Python code for it?


I am working on the file "4-files & conditionals-short.ipynb".
The first task is:
Challenge - making choices:
Which of the following would be printed if you were to run this code? Why did you pick this answer?
A
B
C
B and C
python
if 4 > 5:
    print('A')
elif 4 == 5:
    print('B')
elif 4 < 5:
    print('C')
Can you write Python code for it?


I am working on the file "4-files & conditionals-short.ipynb".
The first task is:
We can ask Python to take different actions, depending on a condition, with an if statement:
Making choices

last lesson we discovereed something suspicious in our inflammatin data by drawing plots
how can python recognized these different features and act on it
we will write code that runs on certain conditions
Can you write Python code for it?

#We use an if statement to take different actions
#based on conditions
num = 37
if num > 100:
    print('greater')
else:
    print('not greater')
print('done')


I am working on the file "4-files & conditionals-short.ipynb".
The first task is:
second line of code above uses keyword if to denote choice
if the test after if is true, the body of the if are executed
if test false the body else is executed
conditional statements don't have to include else - if not present python does nothing
Can you write Python code for it?

num = 53
print('before conditional...')
if num > 100:
    print('53 is greater than 100')
print('...after conditional')


I am working on the file "4-files & conditionals.ipynb".
The first task is:
Checking out data

we can use conditionals to check for suspcioious features in our datat
first 2 plots mzx inflamm per day seemed to rise in straight line
we can chek for this inside the for loop
Can you write Python code for it?

if data.max(axis=0)[0] == 0 and data.max(axis=0)[20] == 20:
    print('Suspicious looking maxima!')


I am working on the file "4-files & conditionals.ipynb".
The first task is:
NOTE: we use == to test for equality rather than single equal b/c the later is the assignment operator
we can also combine tests using and and or. and is only true if both parts are true
Can you write Python code for it?

if (1 > 0) and (-1 > 0):
    print('both parts are true')
else:
    print('at least one part is false')


I am working on the file "4-files & conditionals.ipynb".
The first task is:
asked python to do something depending on diff conditions; we can also imaging not using the else so messages are only printed whens omething is wrong
Challenge - making choices 2:
True and False are special words in Python called booleans which represent true and false statements. However, they aren’t the only values in Python that are true and false. In fact, any value can be used in an if or elif. After reading and running the code below, explain what the rule is for which values are considered true and which are considered false.
python
if '':
    print('empty string is true')
if 'word':
    print('word is true')
if []:
    print('empty list is true')
if [1, 2, 3]:
    print('non-empty list is true')
if 0:
    print('zero is true')
if 1:
    print('one is true')
Can you write Python code for it?


I am working on the file "4-files & conditionals.ipynb".
The first task is:
glob contains function glob that finds files that match a pattern
* matches 0+ characters; ? matches any one char
Can you write Python code for it?


I am working on the file "4-files & conditionals.ipynb".
The first task is:
let's check for the minima like we saw on third day (flat)
Can you write Python code for it?

elif data.min(axis=0).sum() == 0:
    print('Minima add up to zero!')


I am working on the file "4-files & conditionals.ipynb".
The first task is:
results in a list of strings, we can loop oer
we want to create sets of plots
Can you write Python code for it?

%matplotlib inline

# loop here
counter = 0
for filename in glob.glob('data/*.csv'):
    #counter+=1
    counter = counter + 1
print("number of files:", counter)

counter = 0
for filename in glob.glob('data/infl*.csv'):
    #counter+=1
    counter = counter + 1
print("number of files:", counter)

counter = 0
for filename in glob.glob('data/infl*.csv'):
    #counter+=1
    data = numpy.loadtxt(fname=filename, delimiter=',')
    print(filename, "mean is: ", data.mean())
    counter = counter + 1
print("number of files:", counter)

%matplotlib inline

counter = 0
for filename in glob.glob('data/infl*.csv')[0:3]:
    counter+=1
    print(filename)

    data = numpy.loadtxt(fname = filename, delimiter=',')

    fig = matplotlib.pyplot.figure(figsize=(10.0,3.0))

    axes1 = fig.add_subplot(1,3,1)
    axes2 = fig.add_subplot(1,3,2)
    axes3 = fig.add_subplot(1,3,3)

    axes1.set_ylabel('average')
    axes1.plot(data.mean(axis=0))

    axes2.set_ylabel('max')
    axes2.plot(data.max(axis=0))

    axes3.set_ylabel('min')
    axes3.plot(data.min(axis=0))

    fig.tight_layout()
    matplotlib.pyplot.show(fig)

print("number of files:", counter)


I am working on the file "4-files & conditionals.ipynb".
The first task is:
second line of code above uses keyword if to denote choice
if the test after if is true, the body of the if are executed
if test false the body else is executed
conditional statements don't have to include else - if not present python does nothing
Can you write Python code for it?


I am working on the file "4-files & conditionals.ipynb".
The first task is:
we can also chain several tests together using elif, short for else if
Can you write Python code for it?

num = -3

if num > 0:
    print(num, "is positive")
elif num == 0:
    print(num, "is zero")
else:
    print(num, "is negative")


I am working on the file "4-files & conditionals.ipynb".
The first task is:
while or is true if at least one part is true:
Can you write Python code for it?

if (1 < 0) or (-1 < 0):
    print('at least one test is true')


I am working on the file "4-gan/1-gan-multimodal-distribution.ipynb".
The first task is:

  In the following code snippet we will:

Train the constructed GAN

Live plot the histogram of the generated data

Can you write Python code for it?

# Figure for live plot
fig, ax = plt.subplots(1,1)

# Allocate space for noise variable
z = np.zeros((BATCH_SIZE, z_dim))

# n_bathces
number_of_batches = int(X_data.shape[0] / BATCH_SIZE)

for epoch in range(NB_EPOCHS):
    for index in range(number_of_batches):
        # Sample minimibath m=BATCH_SIZE from data generating distribution
        # in other words :
        # Grab a batch of the real data
        data_batch = X_data[index*BATCH_SIZE:(index+1)*BATCH_SIZE]

        # Sample minibatch of m= BATCH_SIZE noise samples
        # in other words, we sample from a uniform distribution
        z = np.random.uniform(-1, 1, (BATCH_SIZE,z_dim))

        # Sample minibatch m=BATCH_SIZE from data generating distribution Pdata
        # in ohter words
        # Use genrator to create new fake samples
        generated_batch = G.predict(z, verbose=0)

        # Update/Train discriminator D
        X = np.concatenate((data_batch, generated_batch))
        y = [1] * BATCH_SIZE + [0.0] * BATCH_SIZE

        d_loss = D.train_on_batch(X, y)

        # Sample minibatch of m= BATCH_SIZE noise samples
        # in other words, we sample from a uniform distribution
        z = np.random.uniform(-1, 1, (BATCH_SIZE,z_dim))

        #Update Generator while not updating discriminator
        D.trainable = False
        # to do gradient ascent we just flip the labels ...
        g_loss = G_plus_D.train_on_batch(z, [1] * BATCH_SIZE)
        D.trainable = True

        # Plot data every 10 mini batches
        if index % 10 == 0:
            ax.clear()

            # Histogram of generated data
            tools.plot_normalized_histogram(ax , generated_batch.flatten(), color='b',label='Generated')

            # Histogram of real data
            tools.plot_normalized_histogram(ax , X_data, color='y',label='Real')

            # Plot details
            ax.legend()
            ax.grid()
            ax.set_xlim([-1,1])

            fig.canvas.draw()
           ...


I am working on the file "4-gan/1-gan-multimodal-distribution.ipynb".
The first task is:
Generative Adversarial Networks 1
<div class="alert alert-warning">
In this notebook we will use what we have learned about artificial neural networks to explore generative modelling with generative adversarial networks.
</div>

Generative adversarial networks, or GANs, is a generative modelling methodology by Goodfellow et al. [1] from 2014 that has garnered much interest these past few years. It is based on the idea of transforming samples of latent variables $\mathbf{z}$ to samples $\mathbf{x}$ of a probability distribution that we would like to learn. The transformation is done via a differentiable function, which typically is defined as an artifical neural network.
When viewed through the lens of game theory, a GAN consists of a generator and an adversary called the discriminator. The generator network $\mathbf{G}$ produces samples $\mathbf{x}$ by transforming latent variables $\mathbf{z}$ with the help of a neural network. The adversary, the discriminator network $\mathbf{D}$, attempts to discriminate between the samples $\mathbf{x}$ generated by $\mathbf{G}$ and the training data. In other words, the discriminator seeks to detect whether the input data is fake or real. At the same time, the generator attempts to fool the discriminator by generating plausible samples. A GAN has converged when the discriminator can no longer differentiate between real data and samples generated by the generator.
Distinguishing between fake and real data sounds like something we have done several times before; indeed, it is a binary classification problem. The original formulation of GANs as a zero-sum game can be seen below:
$$
\begin{equation}
\underset{\mathbf{G}}{\arg\min}\max_{\mathbf{D}}
\frac{1}{N}\sum_{i=1}^{N} \ln\mathbf{D}(\mathbf{x}_i)+\ln(1-\mathbf{D}(\mathbf{G}(\mathbf{z}_i)))
\end{equation}
$$
We can see that the discriminator tries to maximise the log-likelihood of giving the correct prediction, whilst the generator tries to minimise this quantity. In practice, the training is done by alternating...
Can you write Python code for it?

import pandas as pd
import numpy as np
import admin.tools as tools

data = pd.read_csv('resources/multinomial.csv', index_col=0 )


I am working on the file "4-gan/1-gan-multimodal-distribution.ipynb".
The first task is:
Now, let's build a discriminator network using the function you just made.

In the following code snippet we will:

Create the network using your function

Display a summary of your generator network

Can you write Python code for it?

# We already computed the shape and number of dimensions in a data sample
print('The data has shape {} composed of {} dimension(s)'.format(sample_dimentions, n_dimensions))

# Discriminative network
D = discriminator(sample_dimentions, n_dimensions)

# Recommended optimiser
d_optim = keras.optimizers.adam(lr=0.002, beta_1=0.5, beta_2=0.999, epsilon=1e-08, decay=0.0)

# Compile network
D.compile(loss='binary_crossentropy', optimizer=d_optim)

# Network summary
D.summary()


I am working on the file "4-gan/1-gan-multimodal-distribution.ipynb".
The first task is:
Now, let's build a generative network using the function you just made.

  In the following code snippet we will:

Define the number of dimensions of the latent vector $\mathbf{z}$

Find out the shape of a sample of data

Compute numbers of dimensions in a sample of data

Create the network using your function

Display a summary of your generator network

Can you write Python code for it?

# Define the dimension of the latent vector
z_dim = 100

# Dimension of our sample
sample_dimentions = X_data[0].shape

# Calculate the number of dimensions in a sample
n_dimensions=1
for x in list(sample_dimentions):
    n_dimensions *= x

print('A sample of data has shape {} composed of {} dimension(s)'.format(sample_dimentions, n_dimensions))

# Create the generative network
G = generator(z_dim, n_dimensions, sample_dimentions)

# We recommend the followin optimizer
g_optim = keras.optimizers.adam(lr=0.002, beta_1=0.5, beta_2=0.999, epsilon=1e-08, decay=0.0)

# Compile network
G.compile (loss='binary_crossentropy', optimizer=g_optim)

# Network Summary
G.summary()


I am working on the file "4-gan/1-gan-multimodal-distribution.ipynb".
The first task is:
Plot normalised histogram:
Can you write Python code for it?

fig, ax = plt.subplots(1,1)
tools.plot_normalized_histogram(ax,X_data)
ax.set_title('Data Histogram')
ax.set_xlabel('Data Value')
ax.set_ylabel('Normalized frequency')
ax.grid()

fig.canvas.draw()
time.sleep(0.04)


I am working on the file "4-gan/1-gan-multimodal-distribution.ipynb".
The first task is:
Putting the GAN Together
In the following code we will put the generator and discriminator together so we can train our  adversarial model.

In the following code snippet we will:

Use the generator and discriminator to construct a GAN

Can you write Python code for it?

from keras.models import Sequential

def build(generator, discriminator):
    """Build a base model for a Generative Adversarial Networks.
    Parameters
    ----------
    generator : keras.engine.training.Model
        A keras model built either with keras.models ( Model, or Sequential ).
        This is the model that generates the data for the Generative Adversarial networks.
    Discriminator : keras.engine.training.Model
        A keras model built either with keras.models ( Model, or Sequential ).
        This is the model that is a binary classifier for REAL/GENERATED data.
    Returns
    -------
    (keras.engine.training.Model)
        It returns a Sequential Keras Model by connecting a Generator model to a
        Discriminator model.  [ generator-->discriminator]
    """
    model = Sequential()
    model.add(generator)
    discriminator.trainable = False
    model.add(discriminator)
    return model

# Create GAN
G_plus_D = build(G, D)
G_plus_D.compile(loss='binary_crossentropy', optimizer=g_optim)
D.trainable = True


I am working on the file "4-gan/1-gan-multimodal-distribution.ipynb".
The first task is:
Task II: Implement a Discriminative Network
The discriminator network is a simple binary classifier where the output indicates the probability of the input data being real or fake.

Task:

 Create a network where the input shape is `input_shape`

 We recomend reshaping your network  just after input. This way you can have a vector with shape `(None, nb_inputs)`

 Implement a simple network that can classify data

Keras references: Reshape()
Can you write Python code for it?

def discriminator(input_shape, nb_inputs):
    # Define the network input to have input_shape shape
    input_x = None

    # Reshape the input
    x = None

    # Implement the rest of you classifier
    x = None

    # Get the output activation (binary classification)
    probabilities = Dense(1, activation='sigmoid')(x)

    model = Model(inputs=input_x, outputs=probabilities)

    return model


I am working on the file "4-gan/1-gan-multimodal-distribution.ipynb".
The first task is:
Task III: Define Hyperparameters
Please define the following hyper-parameters to train your GAN.

  Task: Please define the following hyperparameters to train your GAN:

 Batch size

Number of training epochs

Can you write Python code for it?


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:

  In the following code snippet we will:

Train the constructed GAN

Live plot the generated data

Can you write Python code for it?

# Figure for live plot
fig, ax = plt.subplots(1,1)

# Allocate space for noise variable
z = np.zeros((BATCH_SIZE, z_dim))

# n_bathces
number_of_batches = int(X_data.shape[0] / BATCH_SIZE)

for epoch in range(NB_EPOCHS):
    for index in range(number_of_batches):

        # Sample minimibath m=BATCH_SIZE from data generating distribution
        # in other words :
        # Grab a batch of the real data
        data_batch = X_data[index*BATCH_SIZE:(index+1)*BATCH_SIZE]

        # Sample minibatch of m= BATCH_SIZE noise samples
        # in other words, we sample from a uniform distribution
        z = np.random.uniform(-1, 1, (BATCH_SIZE,z_dim))

        # Sample minibatch m=BATCH_SIZE from data generating distribution Pdata
        # in ohter words
        # Use genrator to create new fake samples
        generated_batch = G.predict(z, verbose=0)

        # Update/Train discriminator D
        X = np.concatenate((data_batch, generated_batch))
        y = [1] * BATCH_SIZE + [0.0] * BATCH_SIZE

        d_loss = D.train_on_batch(X, y)

        # Sample minibatch of m= BATCH_SIZE noise samples
        # in other words, we sample from a uniform distribution
        z = np.random.uniform(-1, 1, (BATCH_SIZE,z_dim))

        #Update Generator while not updating discriminator
        D.trainable = False
        # to do gradient ascent we just flip the labels ...
        g_loss = G_plus_D.train_on_batch(z, [1] * BATCH_SIZE)
        D.trainable = True

        # Plot data every 10 mini batches
        if index % 10 == 0:
            ax.clear()

            # Histogram of generated data
            image =tools.combine_images(X)

            image = image*127.5+127.5
            ax.imshow(image.astype(np.uint8))
            fig.canvas.draw()
            time.sleep(0.01)

    # End of epoch ....
    print("epoch %d : g_loss : %f  | d_loss : %f" % (epoch, g_loss,  d_loss))


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:
As we did in a previous notebook we will add an extra dimension to our greyscale images.

In the following code snippet we will:

Transform `X_data` from $(28,28)$ to $(28,28,1)$

Can you write Python code for it?

X_data = np.expand_dims(X_data, axis=-1)

print('Shape of X_data {}'.format(X_data.shape))


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:
Generative Adversarial Networks 2

This is a continuation of the previous notebook, where we learned the gist of what a generative adversarial network (GAN) is and how to learn a 1-d multimodal distribution. Please refer back to the last notebook if you are unsure about what a GAN is.

Example: MNIST Dataset
In this notebook we will use a GAN to generate samples coming from the familiar MNIST dataset.
We will start loading by our data.

  In the following snippet of code we will:

Load data from MNIST

Merge the training and test set

Can you write Python code for it?

import numpy as np
from keras.datasets import mnist

import admin.tools as tools

# Load MNIST data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_data = np.concatenate((X_train, X_test))


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:
Input Pre-Processing
As we have done previously with MNIST, the first thing we will be doing is normalisation. However, this time we will normalise the 8-bit images from [0, 255] to [-1, 1].
Previous research with GANs indicates that this normalisation yields better results (reference paper).
Task I:  Implement an Image Normalisation Function

**Task**: Implement a function that normalises the images to the interval [-1,1].

Inputs are integers in the interval [0,255]

Outputs should be floats in the interval [-1,1]

Can you write Python code for it?

def normalize_images(images):
    """
    Create Matrix Y
    :param images: Np tensor with N x R x C x CH.
    Where R = Number of rows in a image
    Where C = Number of cols in a image
    Where CH = Number of channles in a image

    :return: images with its values normalized to [-1,1].
    """
    images = None
    return images

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
# Test normalisation function and normalise the data if it passes
tests.test_normalize_images(normalize_images)
X_data = normalize_images(X_data)


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:
Now, let's build a discriminator network using the function you just made.

In the following code snippet we will:

Create the network using your function

Display a summary of your generator network

Can you write Python code for it?

# We already computed the shape and number of dimensions in a data sample
print('The data has shape {} composed of {} dimension(s)'.format(sample_dimentions, n_dimensions))

# Discriminative Network
D = discriminator(sample_dimentions,n_dimensions)

# Recommended optimiser
d_optim = keras.optimizers.adam(lr=0.002, beta_1=0.5, beta_2=0.999, epsilon=1e-08, decay=0.0)

# Compile Network
D.compile(loss='binary_crossentropy', optimizer=d_optim)

# Network summary
D.summary()


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:
Putting the GAN together
In the following code we will put the generator and discriminator together so we can train our  adversarial model.

In the following code snippet we will:

Use the generator and discriminator to construct a GAN

Can you write Python code for it?

from keras.models import   Sequential

def build(generator, discriminator):
    """Build a base model for a Generative Adversarial Networks.
    Parameters
    ----------
    generator : keras.engine.training.Model
        A keras model built either with keras.models ( Model, or Sequential ).
        This is the model that generates the data for the Generative Adversarial networks.
    Discriminator : keras.engine.training.Model
        A keras model built either with keras.models ( Model, or Sequential ).
        This is the model that is a binary classifier for REAL/GENERATED data.
    Returns
    -------
    (keras.engine.training.Model)
        It returns a Sequential Keras Model by connecting a Generator model to a
        Discriminator model.  [ generator-->discriminator]
    """
    model = Sequential()
    model.add(generator)
    discriminator.trainable = False
    model.add(discriminator)
    return model

# Create GAN
G_plus_D = build(G, D)
G_plus_D.compile(loss='binary_crossentropy', optimizer=g_optim)
D.trainable = True


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:
Task II: Implement a Generator Network

Task: :

Make a network that accepts inputs where the shape is defined by `zdim` $\rightarrow$ `shape=(z_dim,)`

The number of outputs of your network need to be defined as `nb_outputs`

Reshape the final layer to be in the shape of `output_shape`

Since the data lies in the range [-1,1] try using the 'tanh' as the final activation function.

Keras references: Reshape()
Can you write Python code for it?

# Import some useful keras libraries
import keras
from keras.models import Model
from keras.layers import *

def generator(z_dim, nb_outputs, ouput_shape):

    # Define the input_noise as a function of Input()
    latent_var = None

    # Insert the desired amount of layers for your network
    x = None

    # Map you latest layer to n_outputs
    x = None

    # Reshape you data
    x = Reshape(ouput_shape)(x)

    model = Model(inputs=latent_var, outputs=x)

    return model


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:
Task III: Implement a Discriminative Network
The discriminator network is a simple binary classifier where the output indicates the probability of the input data being real or fake.

Task:

 Create a network where the input shape is `input_shape`

 We recomend reshaping your network  just after input. This way you can have a vector with shape `(None, nb_inputs)`

 Implement a simple network that can classify data

Keras references: Reshape()
Can you write Python code for it?

def discriminator(input_shape, nb_inputs):
    # Define the network input to have input_shape shape
    input_x = None

    # Reshape your input
    x = None

    # Implement the rest of you classifier
    x = None

    probabilities = Dense(1, activation='sigmoid')(x)

    model = Model(inputs=input_x, outputs=probabilities)

    return model


I am working on the file "4-gan/2-gan-mnist.ipynb".
The first task is:
Task IV: Define Hyperparameters
Please define the following hyper-parameters to train your GAN.

  Task: Please define the following hyperparameters to train your GAN:

 Batch size

Number of training epochs

Can you write Python code for it?


I am working on the file "4-intro-to-pandas.ipynb".
The first task is:
Menu.csv
Can you write Python code for it?

LATEST_MENU_DATA_DF.columns

discarded_columns3 = [n for n in LATEST_MENU_DATA_DF.columns if n not in
                      ['sponsor', 'location', 'date', 'page_count', 'dish_count']]

pipeline_logger.info('Discarding columns from Menu.csv …')
for discard3 in discarded_columns3:
    pipeline_logger.info('{0} … removed'.format(discard3))

TRIMMED_MENU_DATA_DF = LATEST_MENU_DATA_DF[['sponsor', 'location', 'date',
                                            'page_count', 'dish_count']]

TRIMMED_MENU_DATA_DF.head()


I am working on the file "4-intro-to-pandas.ipynb".
The first task is:
MenuItem.csv
Can you write Python code for it?

discarded_columns2 = [n for n in LATEST_ITEM_DATA_DF.columns if n not in
                      ['id', 'menu_page_id', 'xpos', 'ypos']]

print('Discarding columns from MenuItem.csv …')
for discard2 in discarded_columns2:
    print('{0} … removed'.format(discard2))

TRIMMED_ITEM_DATA_DF = LATEST_ITEM_DATA_DF[['id', 'menu_page_id', 'xpos', 'ypos']]

TRIMMED_ITEM_DATA_DF.head()


I am working on the file "4-intro-to-pandas.ipynb".
The first task is:
MenuPage.csv
Can you write Python code for it?

LATEST_PAGE_DATA_DF.head()

LATEST_PAGE_DATA_DF[['full_height', 'full_width']].astype(int, raise_on_error=False)


I am working on the file "4-intro-to-pandas.ipynb".
The first task is:
Merging DataFrames
Can you write Python code for it?

MERGED_ITEM_PAGES_DF = pd.merge(TRIMMED_ITEM_DATA_DF, LATEST_PAGE_DATA_DF,
                                left_on='menu_page_id', right_index=True, )

MERGED_ITEM_PAGES_DF.columns = ['item_id', 'menu_page_id', 'xpos', 'ypos',
                                'menu_id', 'page_number',
                                'image_id', 'full_height', 'full_width', 'uuid']

#MERGED_ITEM_PAGES_DF.head()

MERGED_ITEM_PAGES_MENUS_DF = pd.merge(TRIMMED_MENU_DATA_DF, MERGED_ITEM_PAGES_DF,
                                      left_index=True, right_on='menu_id')

FULL_MERGE = pd.merge(MERGED_ITEM_PAGES_MENUS_DF, TRIMMED_DISH_DATA_DF,
                      left_index=True, right_index=True)

FULL_MERGE.head()

FOR_JSON_OUTPUT = FULL_MERGE.reset_index()

FOR_JSON_OUTPUT.columns

renamed_columns = ['dish_id', 'menu_sponsor', 'menu_location', 'menu_date', 'menu_page_count',
                   'menu_dish_count', 'item_id', 'menu_page_id', 'item_xpos', 'item_ypos',
                   'menu_id', 'menu_page_number', 'image_id',
                   'page_image_full_height', 'page_image_full_width', 'page_image_uuid', 'dish_name',
                   'dish_menus_appeared', 'dish_times_appeared', 'dish_normalized_name', 'dish_name_fingerprint']

FOR_JSON_OUTPUT.columns = renamed_columns

FOR_JSON_OUTPUT[['menu_page_number', 'dish_id', 'item_id', 'menu_page_id', 'menu_id']].astype(int, raise_on_error=False)

FOR_JSON_OUTPUT['dish_uri']= FOR_JSON_OUTPUT.dish_id.map(lambda x: 'http://menus.nypl.org/dishes/{0}'.format(int(x)))

FOR_JSON_OUTPUT['item_uri']= FOR_JSON_OUTPUT.item_id.map(lambda x: 'http://menus.nypl.org/menu_items/{0}/edit'
                                               .format(int(x)))

FOR_JSON_OUTPUT['menu_page_uri'] = FOR_JSON_OUTPUT.menu_page_id.map(lambda x: 'http://menus.nypl.org/menu_pages/{0}'
                                                          .format(int(x)))

FOR_JSON_OUTPUT['menu_uri'] = FOR_JSON_OUTPUT.menu_id.map(lambda x:'http://menus.nypl.org/menus/{0}'
                                           ...


I am working on the file "4-intro-to-pandas.ipynb".
The first task is:
Working with Data in DataFrames (Tranform)

Can you write Python code for it?

import re
import pandas as pd

LATEST_DISH_DATA_DF = pd.DataFrame.from_csv(os.path.join(DATA_DIR, 'Dish.csv'),
                                            index_col='id')
LATEST_ITEM_DATA_DF = pd.DataFrame.from_csv(os.path.join(DATA_DIR, 'MenuItem.csv'),
                                            index_col='dish_id')
LATEST_PAGE_DATA_DF = pd.DataFrame.from_csv(os.path.join(DATA_DIR, 'MenuPage.csv'),
                                            index_col='id')
LATEST_MENU_DATA_DF = pd.DataFrame.from_csv(os.path.join(DATA_DIR, 'Menu.csv'),
                                             index_col='id')


I am working on the file "4-machine-learning.ipynb".
The first task is:
F1-score で評価
Can you write Python code for it?

clf = LogisticRegression()
cross_val_score(clf, X, y, cv=cv, scoring="f1")

k_svc = SVC(kernel="rbf")
cross_val_score(k_svc, X, y, cv=cv, scoring="f1")

rf = RandomForestClassifier()
cross_val_score(rf, X, y, cv=cv, scoring="f1")


I am working on the file "4-machine-learning.ipynb".
The first task is:
その他の機械学習アルゴリズム

サポートベクターマシン SVC
カーネルSVM
決定木 DecisionTreeClassifier
ランダムフォレスト RandomForestClassifier
k近傍
Can you write Python code for it?

from sklearn.svm import SVC

svc = SVC(kernel="linear")
fit_to_pred(svc, X_train, X_val, y_train, y_val)

k_svc = SVC(kernel="rbf")
fit_to_pred(k_svc, X_train, X_val, y_train, y_val)

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=2)
fit_to_pred(tree, X_train, X_val, y_train, y_val)

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier()
fit_to_pred(rf, X_train, X_val, y_train, y_val)

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier()
fit_to_pred(knn, X_train, X_val, y_train, y_val)


I am working on the file "4-machine-learning.ipynb".
The first task is:
レポート関係を関数化し再利用可能にする
Can you write Python code for it?

def report(y, pred):
    print(accuracy_score(y, pred))
    cm = confusion_matrix(y, pred)
    print(cm)
    cr = classification_report(y, pred)
    print(cr)

report(y_train, y_train_pred)


I am working on the file "4-machine-learning.ipynb".
The first task is:
交差検証(クロスバリデーション)
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

from sklearn.model_selection import KFold

cv = KFold(5, shuffle=True)

clf = LogisticRegression()
cross_val_score(clf, X, y, cv=cv)

k_svc = SVC(kernel="rbf")
cross_val_score(k_svc, X, y, cv=cv)

rf = RandomForestClassifier()
cross_val_score(rf, X, y, cv=cv)


I am working on the file "4-machine-learning.ipynb".
The first task is:
学習から評価までを関数化
Can you write Python code for it?

def fit_to_pred(clf, X_train, X_val, y_train, y_val):
    # 学習
    clf.fit(X_train, y_train)

    # 学習データで評価
    y_train_pred = clf.predict(X_train)
    print("y_train_pred: ")
    report(y_train, y_train_pred)

    # テストデータで評価
    y_val_pred = clf.predict(X_val)
    print("y_val_pred: ")
    report(y_val, y_val_pred)

    # 学習済みデータを返す
    return clf

clf = LogisticRegression()
fit_to_pred(clf, X_train, X_val, y_train, y_val)


I am working on the file "4-machine-learning.ipynb".
The first task is:
学習データとテストデータの分割
http://qiita.com/terapyon/items/8f8d3518ee8eeb4f96b2
Can you write Python code for it?

df.head()

X = df.iloc[:, 3:]

X.head()

y = df['流行']

y.head()

from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(X, y, train_size=0.8)


I am working on the file "4-machine-learning.ipynb".
The first task is:
最終確認
Can you write Python code for it?

rf = RandomForestClassifier(max_depth=3, n_estimators=40)
fit_to_pred(rf, X_train, X_val, y_train, y_val)

rf.predict(X_val)

from sklearn.externals import joblib

joblib.dump(rf, "clf_rf.db")


I am working on the file "4-matplotlib.ipynb".
The first task is:

Si queremos manipular la figura una vez hemos abandonado la celda donde la hemos definido, tendríamos que utilizar la interfaz orientada a objetos de matplotlib. Es un poco lioso porque algunas funciones cambian de nombre, así que en este curso no la vamos a ver. Si te interesa puedes ver los notebooks de la primera edición, donde sí la introdujimos.

https://github.com/AeroPython/Curso_AeroPython/releases/tag/v1.0

Ejercicio
Crear una función que represente gráficamente esta expresión:
$$\sin(2 \pi f_1 t) + \sin(2 \pi f_2 t)$$
Siendo $f_1$ y $f_2$ argumentos de entrada (por defecto $10$ y $100$) y $t \in [0, 0.5]$. Además, debe mostrar:

leyenda,
título "Dos frecuencias",
eje x "Tiempo ($t$)"

y usar algún estilo de los disponibles.
Can you write Python code for it?

def frecuencias(f1=10.0, f2=100.0):
    max_time = 0.5
    times = np.linspace(0, max_time, 1000)
    signal = np.sin(2 * np.pi * f1 * times) + np.sin(2 * np.pi * f2 * times)
    with plt.style.context("ggplot"):
        plt.plot(signal, label="Señal")
        plt.xlabel("Tiempo ($t$)")
        plt.title("Dos frecuencias")
        plt.legend()

frecuencias()


I am working on the file "4-matplotlib.ipynb".
The first task is:
Definimos el dominio con la función np.linspace, que crea un vector de puntos equiespaciados:
Can you write Python code for it?


I am working on the file "4-matplotlib.ipynb".
The first task is:
El truco final: componentes interactivos
No tenemos mucho tiempo pero vamos a ver algo interesante que se ha introducido hace poco en el notebook: componentes interactivos.
Can you write Python code for it?

from IPython.html.widgets import interactive

interactive(frecuencias, f1=(10.0,200.0), f2=(10.0,200.0))


I am working on the file "4-matplotlib.ipynb".
The first task is:
Esto en realidad son códigos abreviados, que se corresponden con argumentos de la función plot:
Can you write Python code for it?

plt.plot(x, f(x), color='red', linestyle='', marker='o')
plt.plot(x, 1 - f(x), c='g', ls='--')


I am working on the file "4-matplotlib.ipynb".
The first task is:
Importamos los paquetes necesarios:
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt


I am working on the file "4-matplotlib.ipynb".
The first task is:
La función contourf es casi idéntica pero rellena el espacio entre niveles. Podemos especificar manualmente estos niveles usando el cuarto argumento:
Can you write Python code for it?

plt.contourf(xx, yy, zz, np.linspace(-4, 4, 100))
plt.colorbar()


I am working on the file "4-matplotlib.ipynb".
The first task is:
La función plot recibe una sola lista (si queremos especificar los valores y) o dos listas (si especificamos x e y). Naturalmente si especificamos dos listas ambas tienen que tener la misma longitud.
La tarea más habitual a la hora de trabajar con matplotlib es representar una función. Lo que tendremos que hacer es definir un dominio y evaluarla en dicho dominio. Por ejemplo:
$$ f(x) = e^{-x^2} $$
Can you write Python code for it?


I am working on the file "4-matplotlib.ipynb".
The first task is:
La lista de posibles argumentos y abreviaturas está disponible en la documentación de la función plot http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.plot.
Más personalización, pero a lo loco
Desde matplotlib 1.4 se puede manipular fácilmente la apariencia de la gráfica usando estilos. Para ver qué estilos hay disponibles, escribiríamos plt.style.available.
Can you write Python code for it?


I am working on the file "4-matplotlib.ipynb".
The first task is:
Para guardar las gráficas en archivos aparte podemos usar la función plt.savefig. matplotlib usará el tipo de archivo adecuado según la extensión que especifiquemos. Veremos esto con más detalle cuando hablemos de la interfaz orientada a objetos.
Varias figuras
Podemos crear figuras con varios sistemas de ejes, pasando a subplot el número de filas y de columnas.
Can you write Python code for it?

x = np.linspace(-1, 7, 1000)

fig = plt.figure()
plt.subplot(211)
plt.plot(x, np.sin(x))
plt.grid(False)
plt.title("Función seno")

plt.subplot(212)
plt.plot(x, np.cos(x))
plt.grid(False)
plt.title("Función coseno")


I am working on the file "4-matplotlib.ipynb".
The first task is:
Referencias

Guía de matplotlib para principiantes http://matplotlib.org/users/beginner.html
Tutorial de matplotlib en español http://pybonacci.org/tag/tutorial-matplotlib-pyplot/
Referencia rápida de matplotlib http://scipy-lectures.github.io/intro/matplotlib/matplotlib.html#quick-references
Can you write Python code for it?

# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = './css/aeropython.css'
HTML(open(css_file, "r").read())


I am working on the file "4-matplotlib.ipynb".
The first task is:
Y hay otro tipo de personalización más loca todavía:
Can you write Python code for it?

with plt.xkcd():
    plt.plot(x, f(x))
    plt.plot(x, 1 - f(x))
    plt.xlabel("Eje x")


I am working on the file "4-matplotlib.ipynb".
The first task is:
matplotlib trae por defecto muchos mapas de colores. En las SciPy Lecture Notes dan una lista de todos ellos (http://scipy-lectures.github.io/intro/matplotlib/matplotlib.html#colormaps)

La función contour se utiliza para visualizar las curvas de nivel de funciones de dos variables y está muy ligada a la función np.meshgrid. Veamos un ejemplo:
$$f(x) = x^2 - y^2$$
Can you write Python code for it?

def f(x, y):
    return x ** 2 - y ** 2

x = np.linspace(-2, 2)
y = np.linspace(-2, 2)
xx, yy = np.meshgrid(x, y)
zz = f(xx, yy)

plt.contour(xx, yy, zz)
plt.colorbar()


I am working on the file "4-matplotlib.ipynb".
The first task is:
¡Nunca imitar a XKCD fue tan fácil! http://xkcd.com/353/
Otros tipo de gráficas
La función scatter muestra una nube de puntos, con posibilidad de variar también el tamaño y el color.
Can you write Python code for it?

N = 100
x = np.random.randn(N)
y = np.random.randn(N)

plt.scatter(x, y)


I am working on the file "4-regression/4.1 - Linear Regression Ad Sales Revenue.ipynb".
The first task is:
Now it's time prepare our data:

First, we'll divide our dataset into two parts: one part we're going use to train our linear regression model, and the second we're going to use to evaluate the trained model and if it can generalize well to new unseen data.

Second, we'll use the scale_features to scale our training and test variables.

Training the Model
Can you write Python code for it?

dataset_size = len(dataset)
training_size = np.floor(dataset_size * 0.8).astype(int)

# First we split the shuffled dataset into two  parts: training and test
X_training = dataset["TV"][:training_size]
y_training = dataset["Sales"][:training_size]

X_test = dataset["TV"][training_size:]
y_test = dataset["Sales"][training_size:]

# Second we apply feature scaling on X_training and X_test
X_training, training_scalar = scale_features(X_training)
X_test,_ = scale_features(X_test, scalar=training_scalar)


I am working on the file "4-regression/4.1 - Linear Regression Ad Sales Revenue.ipynb".
The first task is:
Now we have trained our linear regression model. As we know, our hypothesis takes the form $y = w_0 + w_1x$. We can access the value of $w_0$ with model.intercept_ and the value of $w_1$ with model.coef_.
Can you write Python code for it?

w0 = model.intercept_
w1 = model.coef_

print "Trained model: y = %0.2f + %0.2fx" % (w0, w1)


I am working on the file "4-regression/4.1 - Linear Regression Ad Sales Revenue.ipynb".
The first task is:
We first import the pandas library which we use to read and visualize our data. Then we import the numpy library to help with various calculations. Finally we import from scikit-learn linear models (refernced by sklearn.linear_model) the SGDRegressor which implements our gradient descent based linear regression algorithm.
The last three lines in the end are configuration for metaplotlib (which is used interneally by pandas for visualization and plotting) to have our plots appear inline here in the notebook.
The line in the middle that (which imports the StandardScalar algorithm) is necessary for the feature scaling implemented in the following function. No worries if you don't know what feature scaling is, just treat for now as a black-box function that puts the data in a cleaner form for the learning process. We'll get to the details of feature scaling in the next module.
Can you write Python code for it?

def scale_features(X, scalar=None):
    if(len(X.shape) == 1):
        X = X.reshape(-1, 1)

    if scalar == None:
        scalar = StandardScaler()
        scalar.fit(X)

    return scalar.transform(X), scalar