I am working on the file "5 - pandas Advanced/5-2 pandas DataFrame advaced read_csv, read_html, binning, categorical data, more grouping, Apply, Transforming, Aggregate functions.ipynb".
The first task is:
Now we use the cut function to cut them up into groups.
Can you write Python code for it?

?pd.cut()

df['DepTime2'] = pd.cut(df.DepTime, ranges, labels=labels).astype('category')


I am working on the file "5 - pandas Advanced/5-2 pandas DataFrame advaced read_csv, read_html, binning, categorical data, more grouping, Apply, Transforming, Aggregate functions.ipynb".
The first task is:
We can't just do it on our entire data frame right now because we've got a mix of a lot of different flights, we've got to group it by each flight then apply the ranker
Now we're going to first get the average flight delay for each flight
Can you write Python code for it?

zscore = lambda x: (x - x.mean()) / x.std()

df.groupby('DayOfWeek').transform(zscore)


I am working on the file "5 Data analysis applications/Homework/1 test autocorrelation and stationarity/Test Autocorrelation and stationarity.ipynb".
The first task is:
Для ряда со средним дневным количеством молока на корову из предыдущего вопроса давайте с помощью критерия Дики-Фуллера подберём порядок дифференцирования, при котором ряд становится стационарным.
Дифференцирование можно делать так:
milk.daily_diff1 = milk.daily - milk.daily.shift(1)
Чтобы сделать сезонное дифференцирование, нужно изменить значение параметра у функции shift:
milk.daily_diff12 = milk.daily - milk.daily.shift(12)
При дифференцировании длина ряда сокращается, поэтому в части строк в новой колонке значения будут не определены (NaN). Подавая полученные столбцы на вход критерию Дики-Фуллера, отрезайте неопределённые значения, иначе вы получите неопределённый достигаемый уровень значимости.
Can you write Python code for it?

milk.daily_diff1 = milk.daily - milk.daily.shift(1)
_ = plt.plot(milk.index, milk.daily_diff1)

sm.tsa.stattools.adfuller(milk.daily_diff1.dropna())

milk.daily_diff12 = milk.daily - milk.daily.shift(12)
_ = plt.plot(milk.index, milk.daily_diff12)

sm.tsa.stattools.adfuller(milk.daily_diff12.dropna())

milk.daily_diff12_1 = milk.daily_diff12 - milk.daily_diff12.shift(1)
_ = plt.plot(milk.index, milk.daily_diff12_1)

sm.tsa.stattools.adfuller(milk.daily_diff12_1.dropna())


I am working on the file "5 Data analysis applications/Homework/1 test autocorrelation and stationarity/Test Autocorrelation and stationarity.ipynb".
The first task is:
Для стационарного ряда из предыдущего вопроса постройте график автокорреляционной функции.
Can you write Python code for it?


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Гипотеза нестационарности отвергается с ещё большим уровнем значимости, и визуально ряд выглядит лучше — тренда больше нет.
Подбор модели
Посмотрим на ACF и PACF полученного ряда:
Can you write Python code for it?

plt.figure(figsize=(15,10))

ax = plt.subplot(211)
sm.graphics.tsa.plot_acf(wage.WAG_C_M_box_diff2.dropna()[12:].squeeze(), lags=50, ax=ax);

ax = plt.subplot(212)
sm.graphics.tsa.plot_pacf(wage.WAG_C_M_box_diff2.dropna()[12:].squeeze(), lags=50, ax=ax);


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Для выполнения этого задания нам понадобятся данные о среднемесячных уровнях заработной платы в России:
Can you write Python code for it?

#Reading data
wage = pd.read_csv('WAG_C_M.csv', sep=';', index_col='month', parse_dates=True, dayfirst=True)
wage.info()

wage.head()

_ = plt.figure(figsize=(15,7))
_ = wage.WAG_C_M.plot()
_ = plt.title('Average nominal wage')


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Её остатки:
Can you write Python code for it?

_ = plt.figure(figsize=(15,12))
_ = plt.subplot(211)
_ = best_model.resid[13:].plot()
_ = plt.ylabel(u'Residuals')

_ = ax = plt.subplot(212)
_ = sm.graphics.tsa.plot_acf(best_model.resid.values.squeeze(), lags=50, ax=ax)

print("Критерий Стьюдента: p=%f" % stats.ttest_1samp(best_model.resid[13:], 0)[1])
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(best_model.resid[13:])[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Критерий Дики-Фуллера отвергает гипотезу нестационарности, НО полностью избавиться от тренда не удалось. Попробуем добавить ещё обычное дифференцирование:
Can you write Python code for it?

wage['WAG_C_M_box_diff2'] = wage.WAG_C_M_box_diff - wage.WAG_C_M_box_diff.shift(1)
_ = sm.tsa.seasonal_decompose(wage.WAG_C_M_box_diff2.dropna()).plot()
print('Augmented Dickey-Fuller unit root test p=%f' % sm.tsa.stattools.adfuller(wage.WAG_C_M_box_diff2.dropna())[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Лучшая модель:
Can you write Python code for it?


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Начальные приближения: Q=0, q=1, P=1, p=1.
Can you write Python code for it?

ps = range(0, 2)
d=1
qs = range(0, 2)
Ps = range(0, 2)
D=1
Qs = range(0, 1)

parameters = product(ps, qs, Ps, Qs)
parameters_list = list(parameters)
parameters_list
len(parameters_list)

%%time
results = []
best_aic = float("inf")
warnings.filterwarnings('ignore')

for param in parameters_list:
    #try except нужен, потому что на некоторых наборах параметров модель не обучается
    try:
        model=sm.tsa.statespace.SARIMAX(wage.WAG_C_M_box, order=(param[0], d, param[1]),
                                        seasonal_order=(param[2], D, param[3], 12)).fit(disp=-1)
    #выводим параметры, на которых модель не обучается и переходим к следующему набору
    except ValueError:
        print('wrong parameters:', param)
        continue
    aic = model.aic
    #сохраняем лучшую модель, aic, параметры
    if aic < best_aic:
        best_model = model
        best_aic = aic
        best_param = param
    results.append([param, model.aic])

warnings.filterwarnings('default')

result_table = pd.DataFrame(results)
result_table.columns = ['parameters', 'aic']
print(result_table.sort_values(by = 'aic', ascending=True).head())


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Остатки несмещены (подтверждается критерием Стьюдента), стационарны (подтверждается критерием Дики-Фуллера и визуально), неавтокоррелированы (подтверждается критерием Льюнга-Бокса и коррелограммой). Посмотрим, насколько хорошо модель описывает данные:
Can you write Python code for it?

def invboxcox(y,lmbda):
    if lmbda == 0:
        return(np.exp(y))
    else:
        return(np.exp(np.log(lmbda*y+1)/lmbda))

wage['model'] = invboxcox(best_model.fittedvalues, lmbda)

_ = plt.figure(figsize=(15,7))
_ = wage.WAG_C_M.plot()
_ = wage.model[13:].plot(color='r')
_ = plt.title('Average nominal wage')


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Прогноз
Can you write Python code for it?

wage2 = wage[['WAG_C_M']]
date_list = [datetime.strptime("2017-07-01", "%Y-%m-%d") + relativedelta(months=x) for x in range(0,36)]
future = pd.DataFrame(index=date_list, columns=wage2.columns)
wage2 = pd.concat([wage2, future])
wage2['forecast'] = invboxcox(best_model.predict(start=294, end=329), lmbda)

_ = plt.figure(figsize=(15,7))
_ = wage2.WAG_C_M.plot()
_ = wage2.forecast.plot(color='r')
_ = plt.title('Average nominal wage')


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/Avg wage in Russia.ipynb".
The first task is:
Стабилизация дисперсии
Сделаем преобразование Бокса-Кокса для стабилизации дисперсии:
Can you write Python code for it?

wage['WAG_C_M_box'], lmbda = stats.boxcox(wage.WAG_C_M)

_ = plt.figure(figsize=(15,7))
_ = wage.WAG_C_M_box.plot()
_ = plt.title(u'Transformed average nominal wage')

print('Optimal parameter of the Box-Cox power transformation: %f' % lmbda)
print('Augmented Dickey-Fuller unit root test p=%f' % sm.tsa.stattools.adfuller(wage.WAG_C_M_box)[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
Визуально наша модель неплохо приближает реальные данные. Посмотрим прогноз.
Прогноз
Can you write Python code for it?

#подбираем значения индексов для прогноза
best_model.predict(start=294, end=294 + 24)

#Выводим прогноз
data2 = data[['wage']]
date_list = [datetime.datetime.strptime("2017-06-01", "%Y-%m-%d") + relativedelta(months=x) for x in range(0,24)]
future = pd.DataFrame(index=date_list, columns= data2.columns)
data2 = pd.concat([data2, future])
data2['forecast'] = invboxcox(best_model.predict(start=293, end=294 + 24), lmbda)

plt.figure(figsize(15,7))
data.wage.plot(label = 'фактические значения')
data2.forecast.plot(color='r', label = 'прогноз', linestyle = '--')
plt.ylabel(u'Средняя зарплата')
plt.legend()
pylab.show()


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
Выбираем параметры нашей модели:
Q - значение последнего значимого сезонного лага на автокоррелограмме. Т.к. значимых лагов, кратных периоду (12) нет, то Q = 0
q - значение последнего значимого несезонного лага на автокоррелограмме. q = 1
P - значение последнего значимого сезонного лага на частичной автокоррелограмме. В данном случае это лаг = 12, поэтому возьмем значение P = 1
p - значение последнего значимого несезонного лага, меньшего величин периода, на частичной автокоррелограмме. p = 1
Начальные приближения: Q=0, q=1, P=1, p=1
Can you write Python code for it?

#устанавливаем границы массивов наших параметров согласно начальным приближениям
ps = range(0, 2)
d=1
qs = range(0, 2)
Ps = range(0, 2)
D=1
Qs = range(0, 1)

parameters = product(ps, qs, Ps, Qs)
parameters_list = list(parameters)
len(parameters_list)

%%time
results = []
best_aic = float("inf")
warnings.filterwarnings('ignore')

for param in parameters_list:
    #try except нужен, потому что на некоторых наборах параметров модель не обучается
    try:
        model=sm.tsa.statespace.SARIMAX(data.wage_box, order=(param[0], d, param[1]),
                                        seasonal_order=(param[2], D, param[3], 12)).fit(disp=-1)
    #выводим параметры, на которых модель не обучается и переходим к следующему набору
    except ValueError:
        print('wrong parameters:', param)
        continue
    aic = model.aic
    #сохраняем лучшую модель, aic, параметры
    if aic < best_aic:
        best_model = model
        best_aic = aic
        best_param = param
    results.append([param, model.aic])

warnings.filterwarnings('default')


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
Выводим удачные модели:
Can you write Python code for it?

result_table = pd.DataFrame(results)
result_table.columns = ['parameters', 'aic']
print(result_table.sort_values(by = 'aic', ascending=True).head())


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
Гипотеза нестационарности отвергается, остатки выглядят похоже на белый шум, и явного тренда больше нет.
Подбор модели
Посмотрим на ACF и PACF полученного ряда:
Can you write Python code for it?

plt.figure(figsize(15,12))
ax = plt.subplot(211)
sm.graphics.tsa.plot_acf(data.wage_box_diff2[13:].values.squeeze(), lags=50, ax=ax)
pylab.show()
ax = plt.subplot(212)
sm.graphics.tsa.plot_pacf(data.wage_box_diff2[13:].values.squeeze(), lags=50, ax=ax)
pylab.show()


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
По критерию Дики-Фуллера гипотеза о нестационарности ряда не отвергается.
Сразу видны следующие особенности данных:
* возрастающий тренд
* растущая со временем дисперсия
* периодичность (период - 12 месяцев)
* заметная структура остатков
Стабилизация дисперсии
Сделаем преобразование Бокса-Кокса для стабилизации дисперсии:
Can you write Python code for it?

data['wage_box'], lmbda = stats.boxcox(data.wage)
plt.figure(figsize(15,7))
data.wage_box.plot()
plt.ylabel(u'Средняя зарплата после преобразования Бокса-Кокса')
print("Оптимальный параметр преобразования Бокса-Кокса: %f" % lmbda)
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(data.wage_box)[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
По критерию Дики-Фуллера можно отвергнеть гипотезу нестационарности (0.009 < 0.05), в остатках стало заметно меньше структуры, но полностью избавиться от тренда не удалось. Попробуем добавить ещё обычное дифференцирование:
Can you write Python code for it?

data['wage_box_diff2'] = data.wage_box_diff - data.wage_box_diff.shift(1)
plt.figure(figsize(15,10))
sm.tsa.seasonal_decompose(data.wage_box_diff2[13:]).plot()
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(data.wage_box_diff2[13:])[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
Проверка стационарности и STL-декомпозиция ряда:
Can you write Python code for it?

plt.figure(figsize(15,10))
sm.tsa.seasonal_decompose(data.wage).plot()
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(data.wage)[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
Проверяем критерии:
* Критерией Стьюдента: p = 0.364 > 0.05 - остатки несмещены
* Критерий Дики-Фуллера: p < 0.05 - остатки стационарны
* Критерий Льюнга-Бокса: p = 0.64 > 0.05 - остатки неавтокоррелированы
Теперь посмотрим, как модель приближает данные.
Can you write Python code for it?

data['model'] = invboxcox(best_model.fittedvalues, lmbda)
plt.figure(figsize(15,7))
data.wage.plot(label = 'фактические значения')
data.model[13:].plot(color='r', label = 'прогноз', linestyle = '--')
plt.ylabel(u'Средняя зарплата')
plt.legend()
pylab.show()


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wage_forecast_solomkin/wage_forecast_solomkin.ipynb".
The first task is:
Рассмотрим остатки новой модели:
Can you write Python code for it?

plt.figure(figsize(15,10))
plt.subplot(211)
best_model.resid[13:].plot()
plt.ylabel(u'Residuals')

ax = plt.subplot(212)
sm.graphics.tsa.plot_acf(best_model.resid[13:].values.squeeze(), lags=48, ax=ax)

print("Критерий Стьюдента: p=%f" % stats.ttest_1samp(best_model.resid[13:], 0)[1])
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(best_model.resid[13:])[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wages.ipynb".
The first task is:
Гипотеза нестационарности с уверенностью отвергается, и визуально ряд выглядит лучше. На трендовой компоненте не видно никакого систематического поведения - график колеблется в районе 0.
Перейдем к посмотрению модели. Для этого посмотрим на ACF и PACF полученного ряда:
Can you write Python code for it?

plt.figure(figsize(15,8))
ax = plt.subplot(211)
sm.graphics.tsa.plot_acf(wages.wages_box_diff2[13:].values.squeeze(), lags=48, ax=ax)
pylab.show()
ax = plt.subplot(212)
sm.graphics.tsa.plot_pacf(wages.wages_box_diff2[13:].values.squeeze(), lags=48, ax=ax)
pylab.show()


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wages.ipynb".
The first task is:
И визуально, и воспользовашись критерием Дики-Фуллера, мы можем понять, что ряд по-прежнему не является стационарным. Но размах сезонных колебаний значительно уменьшился. Попробуем сезонное дифференцирование; сделаем на продифференцированном ряде STL-декомпозицию и проверим стационарность:
Can you write Python code for it?

wages['wages_box_diff'] = wages.wages_box - wages.wages_box.shift(12)
plt.figure(figsize(15,10))
sm.tsa.seasonal_decompose(wages.wages_box_diff[12:]).plot()
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(wages.wages_box_diff[12:])[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wages.ipynb".
The first task is:
Критерий Дики-Фуллера отвергает гипотезу нестационарности ряда на уровне значимости 0.05, но мы по-прежнему видим тренд. Попробуем добавить еще обычное дифференцирование:
Can you write Python code for it?

wages['wages_box_diff2'] = wages.wages_box_diff - wages.wages_box_diff.shift(1)
plt.figure(figsize(15,10))
sm.tsa.seasonal_decompose(wages.wages_box_diff2[13:]).plot()
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(wages.wages_box_diff2[13:])[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wages.ipynb".
The first task is:
Лучшая модель (минимальный aic = 38.9, parameters = (1, 0, 1, 0)):
Can you write Python code for it?


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wages.ipynb".
The first task is:
Остатки несмещены (подтверждается критерием Стьюдента) стационарны (подтверждается критерием Дики-Фуллера и визуально), неавтокоррелированы (подтверждается критерием Льюнга-Бокса и коррелограммой).
Посмотрим, насколько хорошо модель описывает данные:
Can you write Python code for it?

def invboxcox(y,lmbda):
   if lmbda == 0:
      return(np.exp(y))
   else:
      return(np.exp(np.log(lmbda*y+1)/lmbda))

wages['model'] = invboxcox(best_model.fittedvalues, lmbda)
plt.figure(figsize(15,7))
wages.WAG_C_M.plot()
wages.model[13:].plot(color='r')
plt.ylabel('wages')
pylab.show()


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wages.ipynb".
The first task is:
Прогноз:
Can you write Python code for it?

wages2 = wages[['WAG_C_M']]
date_list = [datetime.datetime.strptime("2017-07-01", "%Y-%m-%d") + relativedelta(months=x) for x in range(0,24)]
future = pd.DataFrame(index=date_list, columns= wages2.columns)
wages2 = pd.concat([wages2, future])
wages2['forecast'] = invboxcox(best_model.predict(start=294, end=317), lmbda)

plt.figure(figsize(15,7))
wages2.WAG_C_M.plot()
wages2.forecast.plot(color='r')
plt.ylabel('wages')
pylab.show()


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wages.ipynb".
The first task is:
тренд имеет простую, легко объяснимую инфляцией, структуру
сезонный профиль четко выражен

следующий шаг - стабилизация дисперсии:
Can you write Python code for it?

wages['wages_box'], lmbda = stats.boxcox(wages.WAG_C_M)
plt.figure(figsize(15,7))
wages.wages_box.plot()
plt.ylabel('Transformed wages')
print("Оптимальный параметр преобразования Бокса-Кокса: %f" % lmbda)
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(wages.wages_box)[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wine.ipynb".
The first task is:
Её остатки:
Can you write Python code for it?

plt.figure(figsize(15,8))
plt.subplot(211)
best_model.resid[13:].plot()
plt.ylabel(u'Residuals')

ax = plt.subplot(212)
sm.graphics.tsa.plot_acf(best_model.resid[13:].values.squeeze(), lags=48, ax=ax)

print("Критерий Стьюдента: p=%f" % stats.ttest_1samp(best_model.resid[13:], 0)[1])
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(best_model.resid[13:])[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wine.ipynb".
The first task is:
Критерий Дики-Фуллера не отвергает гипотезу нестационарности, и полностью избавиться от тренда не удалось. Попробуем добавить ещё обычное дифференцирование:
Can you write Python code for it?

wine['sales_box_diff2'] = wine.sales_box_diff - wine.sales_box_diff.shift(1)
plt.figure(figsize(15,10))
sm.tsa.seasonal_decompose(wine.sales_box_diff2[13:]).plot()
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(wine.sales_box_diff2[13:])[1])


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wine.ipynb".
The first task is:
Начальные приближения: Q=1, q=2, P=1, p=4
Can you write Python code for it?

ps = range(0, 5)
d=1
qs = range(0, 3)
Ps = range(0, 2)
D=1
Qs = range(0, 2)

parameters = product(ps, qs, Ps, Qs)
parameters_list = list(parameters)
len(parameters_list)

%%time
results = []
best_aic = float("inf")
warnings.filterwarnings('ignore')

for param in parameters_list:
    #try except нужен, потому что на некоторых наборах параметров модель не обучается
    try:
        model=sm.tsa.statespace.SARIMAX(wine.sales_box, order=(param[0], d, param[1]),
                                        seasonal_order=(param[2], D, param[3], 12)).fit(disp=-1)
    #выводим параметры, на которых модель не обучается и переходим к следующему набору
    except ValueError:
        print('wrong parameters:', param)
        continue
    aic = model.aic
    #сохраняем лучшую модель, aic, параметры
    if aic < best_aic:
        best_model = model
        best_aic = aic
        best_param = param
    results.append([param, model.aic])

warnings.filterwarnings('default')


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wine.ipynb".
The first task is:
Остатки несмещены (подтверждается критерием Стьюдента) стационарны (подтверждается критерием Дики-Фуллера и визуально), неавтокоррелированы (подтверждается критерием Льюнга-Бокса и коррелограммой).
Посмотрим, насколько хорошо модель описывает данные:
Can you write Python code for it?

wine['model'] = invboxcox(best_model.fittedvalues, lmbda)
plt.figure(figsize(15,7))
wine.sales.plot()
wine.model[13:].plot(color='r')
plt.ylabel('Wine sales')
pylab.show()


I am working on the file "5 Data analysis applications/Homework/2 project wage forecast for Russia/wine.ipynb".
The first task is:
Стабилизация дисперсии
Сделаем преобразование Бокса-Кокса для стабилизации дисперсии:
Can you write Python code for it?

wine['sales_box'], lmbda = stats.boxcox(wine.sales)
plt.figure(figsize(15,7))
wine.sales_box.plot()
plt.ylabel(u'Transformed wine sales')
print("Оптимальный параметр преобразования Бокса-Кокса: %f" % lmbda)
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(wine.sales_box)[1])


I am working on the file "5 Data analysis applications/Homework/4 project text analysis/Text analysis.ipynb".
The first task is:
1. Загрузите датасет. Описание датасета можно посмотреть здесь:

http://www.dt.fee.unicamp.br/~tiago/smsspamcollection
Can you write Python code for it?

# reading dataset
with open('smsspamcollection/SMSSpamCollection.txt', 'r') as f:
    sms = f.read().splitlines()

sms[0]
sms[11]


I am working on the file "5 Data analysis applications/Homework/4 project text analysis/Text analysis.ipynb".
The first task is:
5. А теперь обучите классификатор на всей выборке и спрогнозируйте с его помощью класс для следующих сообщений:

"FreeMsg: Txt: CALL to No: 86888 & claim your reward of 3 hours talk time to use from your phone now! Subscribe6GB"
"FreeMsg: Txt: claim your reward of 3 hours talk time"
"Have you visited the last lecture on physics?"
"Have you visited the last lecture on physics? Just buy this book and you will have all materials! Only 99$"
"Only 99$"
Can you write Python code for it?

sms_test = ["FreeMsg: Txt: CALL to No: 86888 & claim your reward of 3 hours talk time to use from your phone now! Subscribe6GB",
            "FreeMsg: Txt: claim your reward of 3 hours talk time",
            "Have you visited the last lecture on physics?",
            "Have you visited the last lecture on physics? Just buy this book and you will have all materials! Only 99$",
            "Only 99$"]

# fitting full dataset
pipeline.fit(sms_text, sms_label_bi)

# predicting labels for sms_test
sms_test_pr = pipeline.predict(sms_test)
sms_test_pr

with open('ans2.txt', 'w') as f:
    ans2 = ' '.join(map(str, sms_test_pr))
    f.write(ans2)


I am working on the file "5 Data analysis applications/Homework/4 project text analysis/Text analysis.ipynb".
The first task is:
6. Задайте в CountVectorizer параметр ngram_range=(2,2), затем ngram_range=(3,3), затем ngram_range=(1,3). Во всех трех случаях измерьте получившееся в кросс-валидации значение f1-меры, округлите до второго знака после точки, и выпишете результаты через пробел в том же порядке. В данном эксперименте мы пробовали добавлять в признаки n-граммы для разных диапазонов n - только биграммы, только триграммы, и, наконец, все вместе - униграммы, биграммы и триграммы. Обратите внимание, что статистики по биграммам и триграммам намного меньше, поэтому классификатор только на них работает хуже. В то же время это не ухудшает результат сколько-нибудь существенно, если добавлять их вместе с униграммами, т.к. за счет регуляризации линейный классификатор не склонен сильно переобучаться на этих признаках.
Can you write Python code for it?

# testing log_reg the model in different ngram ranges
ngram_range = [(2,2), (3,3), (1,3)]

scores = []
for ngram in ngram_range:
    pipeline = Pipeline([('count_vect', CountVectorizer(ngram_range=ngram)), ('log_reg', LogisticRegression())])
    score = cross_val_score(pipeline, sms_text, sms_label_bi, scoring='f1', cv=10).mean()
    scores.append(score)
    print('Ngram_range: ', ngram)
    print('Cross-val-score f1: %.4f\n' % score)

with open('ans3.txt', 'w') as f:
    ans3 = ' '.join(map(str, ['%.2f' % sc for sc in scores]))
    f.write(ans3)


I am working on the file "5 Data analysis applications/Homework/4 project text analysis/Text analysis.ipynb".
The first task is:
8. Попробуйте использовать в логистической регрессии в качестве признаков Tf*idf из TfidfVectorizer на униграммах. Повысилось или понизилось качество на кросс-валидации по сравнению с CountVectorizer на униграммах? (напишите в файле с ответом 1, если повысилось, -1, если понизилось, и 0, если изменилось не более чем на 0.01). Обратите внимание, что результат перехода к tf*idf не всегда будет таким - если вы наблюдаете какое-то явление на одном датасете, не надо сразу же его обобщать на любые данные.
Can you write Python code for it?

# cross_val_score for f1 metric for log_reg with tf_idf
pipeline = Pipeline([('tf_idf', TfidfVectorizer()),
                     ('log_reg', LogisticRegression())])
score = cross_val_score(pipeline, sms_text, sms_label_bi, scoring='f1', cv=10).mean()
print('Cross-val-score f1: %.4f\n' % score)

with open('ans5.txt', 'w') as f:
    ans5 = str(-1)
    f.write(ans5)


I am working on the file "5 Data analysis applications/Homework/5 project recommender systems/Recommender systems.ipynb".
The first task is:
2. Split datasets by looks and purchases.
Can you write Python code for it?

# Create train array splitted by looks (look_items) and purchases (pur_items)
sess_train_lp = []
for sess in sess_train:
    look_items, pur_items = sess.split(';')
    look_items = map(int, look_items.split(','))
    if len(pur_items) > 0:
        pur_items = map(int, pur_items.split(','))
    else:
        pur_items = []
    sess_train_lp.append([look_items, pur_items])

# Create test array splitted by looks (look_items) and purchases (pur_items)
sess_test_lp = []
for sess in sess_test:
    look_items, pur_items = sess.split(';')
    look_items = map(int, look_items.split(','))
    if len(pur_items) > 0:
        pur_items = map(int, pur_items.split(','))
    else:
        pur_items = []
    sess_test_lp.append([look_items, pur_items])


I am working on the file "5 Data analysis applications/Homework/5 project recommender systems/Recommender systems.ipynb".
The first task is:
3. Create and sort arrays of unique ids counters for looks and purchases for train dataset.
Can you write Python code for it?

# Array of looks
sess_train_l = [row[0] for row in sess_train_lp]
sess_train_l_np = np.array( [id_n for sess in sess_train_l for id_n in sess] )

# Array of unique ids and looks in train data
sess_train_l_cnt = np.transpose(np.unique(sess_train_l_np, return_counts=True))

sess_train_l_cnt

# Array of purchases
sess_train_p = [row[1] for row in sess_train_lp]
sess_train_p_np = np.array( [id_n for sess in sess_train_p for id_n in sess] )

# Array of unique ids and purchases in train dataset
sess_train_p_cnt = np.transpose(np.unique(sess_train_p_np, return_counts=True))

sess_train_p_cnt

# Sorting arrays of looks and purchases by counts
sess_train_l_cnt = sess_train_l_cnt[sess_train_l_cnt[:,1].argsort()][::-1]
sess_train_p_cnt = sess_train_p_cnt[sess_train_p_cnt[:,1].argsort()][::-1]


I am working on the file "5 Data analysis applications/Homework/5 project recommender systems/Recommender systems.ipynb".
The first task is:
4. Calculating metrics for train dataset with suggestions based on looks.
Can you write Python code for it?

def prec_rec_metrics(session, reccomendations, k):
    purchase = 0
    for ind in reccomendations:
        if ind in session:
            purchase += 1
    precision = purchase / k
    recall = purchase / len(session)
    return(precision, recall)

# Calculate metrics for train dataset, suggestions based on looks
prec_at_1_tr_l, rec_at_1_tr_l = [], []
prec_at_5_tr_l, rec_at_5_tr_l = [], []
k1, k5 = 1, 5
for i, sess_p in enumerate(sess_train_p):
    # skip sessions without purchases
    if sess_p == []: continue

    # looks ids
    sess_l = sess_train_l[i]

    # sorted looks ids indices in sess_train_l_cnt array
    # sort in accordance with looks counts
    l_ind_sess = []
    for j in range(len(sess_l)):
        l_ind_sess.append(np.where(sess_train_l_cnt[:,0] == sess_l[j])[0][0])
    l_ind_sess_sorted = np.unique(l_ind_sess)

    # k1 recommendations
    num_of_recs_k1 = min(k1, len(sess_l))
    if num_of_recs_k1 == 0: continue
    recs_k1 = sess_train_l_cnt[l_ind_sess_sorted[:num_of_recs_k1],0]

    # k1 metrics
    prec_1, rec_1 = prec_rec_metrics(sess_p, recs_k1, k1)
    prec_at_1_tr_l.append(prec_1)
    rec_at_1_tr_l.append(rec_1)

    # k5 recommendations
    num_of_recs_k5 = min(k5, len(sess_l))
    if num_of_recs_k5 == 0: continue
    recs_k5 = sess_train_l_cnt[l_ind_sess_sorted[:num_of_recs_k5],0]

    # k5 metrics
    prec_5, rec_5 = prec_rec_metrics(sess_p, recs_k5, k5)
    prec_at_5_tr_l.append(prec_5)
    rec_at_5_tr_l.append(rec_5)

avg_prec_at_1_tr_l = np.mean(prec_at_1_tr_l)
avg_rec_at_1_tr_l = np.mean(rec_at_1_tr_l)
avg_prec_at_5_tr_l = np.mean(prec_at_5_tr_l)
avg_rec_at_5_tr_l = np.mean(rec_at_5_tr_l)

with open('ans1.txt', 'w') as f:
    r1 = '%.2f' % round(avg_rec_at_1_tr_l, 2)
    p1 = '%.2f' % round(avg_prec_at_1_tr_l, 2)
    r5 = '%.2f' % round(avg_rec_at_5_tr_l, 2)
    p5 = '%.2f' % round(avg_prec_at_5_tr_l, 2)
    ans1 = ' '.join([r1, p1, r5, p5])
    print('Answer 1:', ans1)
    f.write(ans1)


I am working on the file "5 Data analysis applications/Homework/5 project recommender systems/Recommender systems.ipynb".
The first task is:
5. Calculating metrics for train dataset with suggestions based on purchases.
Can you write Python code for it?

# Calculate metrics for train dataset, suggestions based on purchases
prec_at_1_tr_p, rec_at_1_tr_p = [], []
prec_at_5_tr_p, rec_at_5_tr_p = [], []
k1, k5 = 1, 5

for i, sess_p in enumerate(sess_train_p):
    # skip sessions without purchases
    if sess_p == []: continue

    # looks ids
    sess_l = sess_train_l[i]

    # sorted looks ids indices in sess_train_p_cnt array
    # sort in accordance with purchases counts
    l_ind_sess = []
    for j in range(len(sess_l)):
        if sess_l[j] not in sess_train_p_cnt[:,0]: continue
        l_ind_sess.append(np.where(sess_train_p_cnt[:,0] == sess_l[j])[0][0])
    l_ind_sess_sorted = np.unique(l_ind_sess)

    # k1 recommendations
    num_of_recs_k1 = min(k1, len(sess_l), len(l_ind_sess_sorted))
    if num_of_recs_k1 == 0: continue
    recs_k1 = sess_train_p_cnt[l_ind_sess_sorted[:num_of_recs_k1],0]

    # k1 metrics
    prec_1, rec_1 = prec_rec_metrics(sess_p, recs_k1, k1)
    prec_at_1_tr_p.append(prec_1)
    rec_at_1_tr_p.append(rec_1)

    # k5 recommendations
    num_of_recs_k5 = min(k5, len(sess_l), len(l_ind_sess_sorted))
    if num_of_recs_k5 == 0: continue
    recs_k5 = sess_train_p_cnt[l_ind_sess_sorted[:num_of_recs_k5],0]

    # k5 metrics
    prec_5, rec_5 = prec_rec_metrics(sess_p, recs_k5, k5)
    prec_at_5_tr_p.append(prec_5)
    rec_at_5_tr_p.append(rec_5)

avg_prec_at_1_tr_p = np.mean(prec_at_1_tr_p)
avg_rec_at_1_tr_p = np.mean(rec_at_1_tr_p)
avg_prec_at_5_tr_p = np.mean(prec_at_5_tr_p)
avg_rec_at_5_tr_p = np.mean(rec_at_5_tr_p)

with open('ans2.txt', 'w') as f:
    r1 = '%.2f' % round(avg_rec_at_1_tr_p, 2)
    p1 = '%.2f' % round(avg_prec_at_1_tr_p, 2)
    r5 = '%.2f' % round(avg_rec_at_5_tr_p, 2)
    p5 = '%.2f' % round(avg_prec_at_5_tr_p, 2)
    ans2 = ' '.join([r1, p1, r5, p5])
    print('Answer 2:', ans2)
    f.write(ans2)


I am working on the file "5 Data analysis applications/Homework/5 project recommender systems/Recommender systems.ipynb".
The first task is:
6. Create and sort arrays of unique ids counters for looks and purchases for test dataset.
Can you write Python code for it?

# Array of looks
sess_test_l = [row[0] for row in sess_test_lp]
sess_test_l_np = np.array( [id_n for sess in sess_test_l for id_n in sess] )

# Array of unique ids and looks in train data
#sess_test_l_cnt = np.transpose(np.unique(sess_test_l_np, return_counts=True))

sess_test_l_np
#sess_test_l_cnt

# Array of purchases
sess_test_p = [row[1] for row in sess_test_lp]
sess_test_p_np = np.array( [id_n for sess in sess_test_p for id_n in sess] )

# Array of unique ids and purchases in train dataset
#sess_test_p_cnt = np.transpose(np.unique(sess_test_p_np, return_counts=True))

sess_test_p_np
#sess_test_p_cnt

# Sorting arrays of looks and purchases by counts
#sess_train_l_cnt = sess_train_l_cnt[sess_train_l_cnt[:,1].argsort()][::-1]
#sess_train_p_cnt = sess_train_p_cnt[sess_train_p_cnt[:,1].argsort()][::-1]


I am working on the file "5 Data analysis applications/Homework/5 project recommender systems/Recommender systems.ipynb".
The first task is:
8. Calculating metrics for test dataset with suggestions based on purchases.
Can you write Python code for it?

def uniquifier(seq):
    seen = set()
    return [x for x in seq if not (x in seen or seen.add(x))]

# Calculate metrics for test dataset, suggestions based on purchases
prec_at_1_tst_p, rec_at_1_tst_p = [], []
prec_at_5_tst_p, rec_at_5_tst_p = [], []
k1, k5 = 1, 5
for i, sess_p in enumerate(sess_test_p):
    # skip sessions without purchases
    if sess_p == []: continue

    # looks ids
    sess_l = sess_test_l[i]

    # sorted looks ids indices in sess_train_p_cnt array
    # sort in accordance with purchases counts
    l_ind_sess = []
    new_ids = []
    for j in range(len(sess_l)):
        if sess_l[j] not in sess_train_p_cnt[:,0]:
            new_ids.append(sess_l[j])
            continue
        l_ind_sess.append(np.where(sess_train_p_cnt[:,0] == sess_l[j])[0][0])
    l_ind_sess_sorted = np.unique(l_ind_sess)

    # k1 recommendations
    num_of_recs_k1 = min(k1, len(sess_l))
    if num_of_recs_k1 == 0: continue
    if l_ind_sess != []:
        recs_k1 = sess_train_p_cnt[l_ind_sess_sorted[:num_of_recs_k1],0]
    else:
        recs_k1 = []
    recs_k1 = np.concatenate((np.array(recs_k1, dtype='int64'), np.array(uniquifier(np.array(new_ids, dtype='int64')))))[:num_of_recs_k1]

    # k1 metrics
    prec_1, rec_1 = prec_rec_metrics(sess_p, recs_k1, k1)
    prec_at_1_tst_p.append(prec_1)
    rec_at_1_tst_p.append(rec_1)

    # k5 recommendations
    num_of_recs_k5 = min(k5, len(sess_l))
    if num_of_recs_k5 == 0: continue
    if l_ind_sess != []:
        recs_k5 = sess_train_p_cnt[l_ind_sess_sorted[:num_of_recs_k5],0]
    else:
        recs_k5 = []
    recs_k5 = np.concatenate((np.array(recs_k5, dtype='int64'), np.array(uniquifier(np.array(new_ids, dtype='int64')))))[:num_of_recs_k5]

    # k5 metrics
    prec_5, rec_5 = prec_rec_metrics(sess_p, recs_k5, k5)
    prec_at_5_tst_p.append(prec_5)
    rec_at_5_tst_p.append(rec_5)

avg_prec_at_1_tst_p = np.mean(prec_at_1_tst_p)
avg_rec_at_1_tst_p = np.mean(rec_at_1_tst_p)
avg_prec_at_5_tst_p = np.mean(prec_at_5_tst_p)
avg_rec_at_5_tst_p =...


I am working on the file "5 Data analysis applications/Lectures notebooks/1 wine sales time series/wine.ipynb".
The first task is:
Гипотеза нестационарности отвергается, и визуально ряд выглядит лучше — тренда больше нет.
Подбор модели
Посмотрим на ACF и PACF полученного ряда:
Can you write Python code for it?

plt.figure(figsize(15,8))
ax = plt.subplot(211)
sm.graphics.tsa.plot_acf(wine.sales_box_diff2[13:].values.squeeze(), lags=48, ax=ax)
pylab.show()
ax = plt.subplot(212)
sm.graphics.tsa.plot_pacf(wine.sales_box_diff2[13:].values.squeeze(), lags=48, ax=ax)
pylab.show()


I am working on the file "5 Data analysis applications/Lectures notebooks/1 wine sales time series/wine.ipynb".
The first task is:
Если в предыдущей ячейке возникает ошибка, убедитесь, что обновили statsmodels до версии не меньше 0.8.0rc1.
Can you write Python code for it?

result_table = pd.DataFrame(results)
result_table.columns = ['parameters', 'aic']
print(result_table.sort_values(by = 'aic', ascending=True).head())


I am working on the file "5 Data analysis applications/Lectures notebooks/1 wine sales time series/wine.ipynb".
The first task is:
Проверка стационарности и STL-декомпозиция ряда:
Can you write Python code for it?

plt.figure(figsize(15,10))
sm.tsa.seasonal_decompose(wine.sales).plot()
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(wine.sales)[1])


I am working on the file "5 Data analysis applications/Lectures notebooks/1 wine sales time series/wine.ipynb".
The first task is:
Прогноз
Can you write Python code for it?

wine2 = wine[['sales']]
date_list = [datetime.datetime.strptime("1994-09-01", "%Y-%m-%d") + relativedelta(months=x) for x in range(0,36)]
future = pd.DataFrame(index=date_list, columns= wine2.columns)
wine2 = pd.concat([wine2, future])
wine2['forecast'] = invboxcox(best_model.predict(start=176, end=211), lmbda)

plt.figure(figsize(15,7))
wine2.sales.plot()
wine2.forecast.plot(color='r')
plt.ylabel('Wine sales')
pylab.show()


I am working on the file "5 Data analysis applications/Lectures notebooks/1 wine sales time series/wine.ipynb".
The first task is:
Стационарность
Критерий Дики-Фуллера отвергает гипотезу нестационарности, но визуально в данных виден тренд. Попробуем сезонное дифференцирование; сделаем на продифференцированном ряде STL-декомпозицию и проверим стационарность:
Can you write Python code for it?

wine['sales_box_diff'] = wine.sales_box - wine.sales_box.shift(12)
plt.figure(figsize(15,10))
sm.tsa.seasonal_decompose(wine.sales_box_diff[12:]).plot()
print("Критерий Дики-Фуллера: p=%f" % sm.tsa.stattools.adfuller(wine.sales_box_diff[12:])[1])


I am working on the file "5 Training and Ensemble.ipynb".
The first task is:
Answer
Problems with the original code:

Use partial fit for all folds, which results in dependent training and thus cumulative learning.

Due to problem 1, we the CV accuracy is incorrect, and not able to show score for every epoch.

There are two major changes:

We create a list of Perceptrons, each corresponds to one fold. Within each fold, we run partial fit for a given number of epochs.

Also, we create a list of accuracy scores, each corresponds to one epoch. Within each epoch, we append scores of all folds.

There is one minor change:

To enable PCA, we change X_train_std into X_train_pca.

Precision-recall curve (40 points)
We have plotted ROC (receiver operator characteristics) curve for the breast cancer dataset.
Plot the precision-recall curve for the same data set using the same experimental setup.
What similarities and differences you can find between ROC and precision-recall curves?
You can find more information about precision-recall curve online such as: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html
Answer
Can you write Python code for it?

from sklearn.metrics import roc_curve, precision_recall_curve, auc
from scipy import interp
from sklearn.linear_model import LogisticRegression

pipe_lr = Pipeline([('scl', StandardScaler()),
                    ('pca', PCA(n_components=2)),
                    ('clf', LogisticRegression(penalty='l2',
                                               random_state=0,
                                               C=100.0))])

# intentionally use only 2 features to make the task harder and the curves more interesting
X_train2 = X_train[:, [4, 14]]
X_test2 = X_test[:, [4, 14]]

if Version(sklearn_version) < '0.18':
    cv = StratifiedKFold(y_train, n_folds=3, random_state=1)
else:
    cv = list(StratifiedKFold(n_splits=3, random_state=1).split(X_train, y_train))

fig = plt.figure(figsize=(7, 5))

# **************************************
#            ROC - 2 Features
# **************************************
print ("ROC Curve - 2 Features")
mean_tpr = 0.0
mean_fpr = np.linspace(0, 1, 100)
all_tpr = []

for i, (train, test) in enumerate(cv):
    probas = pipe_lr.fit(X_train2[train],
                         y_train[train]).predict_proba(X_train2[test])

    fpr, tpr, thresholds = roc_curve(y_train[test],
                                     probas[:, 1],
                                     pos_label=1)
    mean_tpr += interp(mean_fpr, fpr, tpr)
    #mean_tpr[0] = 0.0
    roc_auc = auc(fpr, tpr)
    plt.plot(fpr,
             tpr,
             lw=1,
             label='ROC fold %d (area = %0.2f)'
                   % (i+1, roc_auc))

mean_tpr /= len(cv)
mean_tpr[0] = 0.0
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)
plt.plot(mean_fpr, mean_tpr, 'k--',
         label='mean ROC (area = %0.2f)' % mean_auc, lw=2)

plt.plot([0, 1],
         [0, 1],
         linestyle='--',
         color=(0.6, 0.6, 0.6),
         label='random guessing')

plt.plot([0, 0, 1],
         [0, 1, 1],
         lw=2,
         linestyle=':',
         color='black',
         label='perfect performance')

plt.xlim([-0.05,...


I am working on the file "5 Training and Ensemble.ipynb".
The first task is:
Dataset
We will use the Wisconsin breast cancer dataset for the following questions
Can you write Python code for it?

import pandas as pd

wdbc_source = 'https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data'
#wdbc_source = '../datasets/wdbc/wdbc.data'

df = pd.read_csv(wdbc_source, header=None)

from sklearn.preprocessing import LabelEncoder
X = df.loc[:, 2:].values
y = df.loc[:, 1].values
le = LabelEncoder()
y = le.fit_transform(y)
le.transform(['M', 'B'])

if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=0.20, random_state=1)

import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "5 Training and Ensemble.ipynb".
The first task is:
Explanation
Observations:
The ensemble error DOES NOT ALWAYS go down with more classifiers.<br>
Overall, the ensemble error declines as the number of classifiers increases. However, when the number of calssifiers $N = 2k$, the error is much higher than $N = 2k-1$ and $2k+1$.
Reason:
This is because when a draw comes, i.e. $K$ subclassifiers are wrong and $K$ subclassifiers are correct, it is considered as "wrong prediction" by the original function.
Describe a better algorithm for computing the ensemble error.
Can you write Python code for it?

def better_ensemble_error(num_classifier, base_error):
    k_start = math.ceil(num_classifier/2)
    probs = [comb(num_classifier, k)*(base_error**k)*((1-base_error)**(num_classifier-k)) for k in range(k_start, num_classifier+1)]
    if num_classifier % 2 == 0:
        probs.append(-0.5*comb(num_classifier, k_start)*(base_error**k_start)*((1-base_error)**(num_classifier-k_start)))
    return sum(probs)

plot_num_classifier(better_ensemble_error,
                    max_num_classifier=max_num_classifiers,
                    base_error=base_error)


I am working on the file "5 Training and Ensemble.ipynb".
The first task is:
K-fold validation (20 points)
Someone wrote the code below to conduct cross validation.
Do you see anything wrong with it?
And if so, correct the code and provide an explanation.
Can you write Python code for it?

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import Perceptron
from sklearn.pipeline import Pipeline

if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import StratifiedKFold
else:
    from sklearn.model_selection import StratifiedKFold

scl = StandardScaler()
pca = PCA(n_components=2)
# original: clf = Perceptron(random_state=1)

# data preprocessing
X_train_std = scl.fit_transform(X_train)
X_test_std = scl.transform(X_test)

X_train_pca = pca.fit_transform(X_train_std)
X_test_pca = pca.transform(X_test_std)

# compute the data indices for each fold
if Version(sklearn_version) < '0.18':
    kfold = StratifiedKFold(y=y_train,
                            n_folds=10,
                            random_state=1)
else:
    kfold = StratifiedKFold(n_splits=10,
                            random_state=1).split(X_train, y_train)

num_epochs = 2
scores = [[] for i in range(num_epochs)]

enumerate_kfold = list(enumerate(kfold))
# new:
clfs = [Perceptron(random_state=1) for i in range(len(enumerate_kfold))]

for epoch in range(num_epochs):
    for k, (train, test) in enumerate_kfold:

# original:
#         clf.partial_fit(X_train_std[train], y_train[train], classes=np.unique(y_train))
#         score = clf.score(X_train_std[test], y_train[test])
#         scores.append(score)
# new:
        clfs[k].partial_fit(X_train_pca[train], y_train[train], classes=np.unique(y_train))
        score = clfs[k].score(X_train_pca[test], y_train[test])
        scores[epoch].append(score)

        print('Epoch: %s, Fold: %s, Class dist.: %s, Acc: %.3f' % (epoch,
                                                                   k,
                                                                   np.bincount(y_train[train]),
                                                                   score))
    print('')

# new:
for epoch in range(num_epochs):
    print('Epoch: %s, CV accuracy: %.3f +/- %.3f' % (epoch,...


I am working on the file "5 Training and Ensemble.ipynb".
The first task is:
Number of classifiers (40 points)
The function plot_base_error() above plots the ensemble error as a function of the base error given a fixed number of classifiers.
Write another function to plot ensembe error versus different number of classifiers with a given base error.
Does the ensemble error always go down with more classifiers?
Why or why not?
Can you improve the method ensemble_error() to produce a more reasonable plot?
Answer
The code for plotting is below:
Can you write Python code for it?

def plot_num_classifier(ensemble_error_func, max_num_classifier, base_error):

    num_classifiers = range(1, max_num_classifier+1)
    ensemble_errors = [ensemble_error_func(num_classifier = num_classifier, base_error=base_error) for num_classifier in num_classifiers]

    plt.plot(num_classifiers, ensemble_errors,
             label = 'ensemble error',
             linewidth = 2)
    plt.plot(range(max_num_classifier), [base_error]*max_num_classifier,
             label = 'base error',
             linestyle = '--',
             linewidth=2)

    plt.xlabel('num classifiers')
    plt.ylabel('ensemble error')
    plt.xlim([1, max_num_classifier])
    plt.ylim([0, 1])
    plt.title('base error %.2f' % base_error)
    plt.legend(loc='best')
    plt.grid()
    plt.show()

max_num_classifiers = 20
base_error = 0.25

plot_num_classifier(ensemble_error,
                    max_num_classifier=max_num_classifiers,
                    base_error=base_error)


I am working on the file "5 minutes to creating your first Machine Learning model.ipynb".
The first task is:
1. Create a predictive model
Specify training data to use
BigML makes a distinction between the origin of the data (the "source") and the actual data that's being used for training (the "dataset"). We first create a data source by specifying a csv file to use (hosted on Amazon S3 in this example).
Can you write Python code for it?

source = api.create_source('s3://bml-data/realtor-las-vegas.csv', {"name": "Realtor LV"})


I am working on the file "5 minutes to creating your first Machine Learning model.ipynb".
The first task is:
API calls are asynchronous, so we use api.ok to make sure that the request has finished before we move on to the rest.
Can you write Python code for it?

api.ok(source) # shows "True" when source has been created


I am working on the file "5 minutes to creating your first Machine Learning model.ipynb".
The first task is:
API wrapper
We create an api object which will be used to communicate with the BigML API.
Note that BigML has two ways of functioning: production mode or development mode. Here, we choose to use the latter since it's free!
Can you write Python code for it?

# Uncomment lines below in case this block doesn't work
#import pip
#pip.main(['install', 'bigml'])

from bigml.api import BigML

# Assuming you installed the BigML Python wrappers (with the 'pip install bigml' command, see above)
# Assuming BIGML_USERNAME and BIGML_API_KEY were defined as shell environment variables
# otherwise: api=BigML('your username here','your API key here',dev_mode=True)

api=BigML(dev_mode=True) # use BigML in development mode for unlimited usage
print "Wrapper ready to use!"


I am working on the file "5 minutes to creating your first Machine Learning model.ipynb".
The first task is:
BigML uses decision tree models. The tree that's been learnt from your data can be seen at:
Can you write Python code for it?

api.ok(model) # making sure the model is ready
print "Model ready and available at https://bigml.com/dashboard/"+str(model['resource'])+"?"+BIGML_AUTH


I am working on the file "5 minutes to creating your first Machine Learning model.ipynb".
The first task is:
Here's the same thing on one single line:
Can you write Python code for it?

print "Value: ",api.create_prediction(model, {"bedrooms": 4, "full_bathrooms": 4, "type": "Single Family Home", "size_sqft": 1500})['object']['output']," USD"


I am working on the file "5 minutes to creating your first Machine Learning model.ipynb".
The first task is:
If you click on the outputted link above, it will take you to a histogram view of the data on the BigML dashboard.
Learn a model from the data
This is done in just one command — there are no parameters to set whatsoever.
Can you write Python code for it?

model = api.create_model(dataset)
print "'model' object created!"


I am working on the file "5 minutes to creating your first Machine Learning model.ipynb".
The first task is:
Let's make a prediction for this new input against the model we created:
Can you write Python code for it?

prediction = api.create_prediction(model, new_input)
print "Prediction: ",prediction['object']['output']


I am working on the file "5 minutes to creating your first Machine Learning model.ipynb".
The first task is:
The source can be found on the BigML.com web interface at the following URL:
Can you write Python code for it?

BIGML_AUTH = %env BIGML_AUTH
print "https://bigml.com/dashboard/"+str(source['resource'])+"?"+BIGML_AUTH


I am working on the file "5-Funkcije.ipynb".
The first task is:
Funkcija frek_distr kao argument prima neki iterabilni objekt iterabilni. U našem slučaju će to najčešće biti niz znakova te nešto kasnije i lista riječi iz nizova znakova. Zatim stvara prazan rječnik (rjecnik={}) koji će sadržavati pojave i njihove vrijednosti. Nakon toga for petljom iterira kroz iterabilni objekt (niz znakova ili lista) (for element in iterabilni) te računa u novostvorenom rječniku koliko se puta pojavila određena vrijednost, odnosno računa frekvencijsku distribuciju (rjecnik[element]=rjecnik.get(element,0)+1). Konačno vraća taj novostvoreni rječnik s uređenim parovima (pojava, broj pojavljivanja) (return rjecnik).
Druga funkcija sortira frekvencijsku distribuciju:
Can you write Python code for it?

def sortiraj_distr(rjecnik):
    return sorted(rjecnik.items(),key=lambda x:-x[1])


I am working on the file "5-Funkcije.ipynb".
The first task is:
Funkcija opojavnici kao argument uzima neki niz znakova niz. Zatim učitava modul za regularne izraze re. Konačno, vraća sve Unicode alfanumeričke znakove.
Ove tri funkcije predstavljaju osnovne alate za obrade.
Na sljedeći ćemo način opojavničiti neki niz znakova, izračunati frekvencijsku distribuciju pojavnica, sortirati ih te ispisati sortiranu listu:
Can you write Python code for it?

pojavnice=opojavnici('biti ili ne biti')
fd=frek_distr(pojavnice)
sd=sortiraj_distr(fd)
print sd


I am working on the file "5-Funkcije.ipynb".
The first task is:
Funkcija sortiraj_distr kao argument prima rječnik (rjecnik) te vraća listu parova sortiranu po drugoj vrijednosti para (return sorted(rjecnik.items(),key=lambda x:-x[1])). Ta će nam funkcija trebati kad ćemo htjeti napraviti uvid u najčešće ili pak najrjeđe događaje u frekvencijskoj distribuciji.
Na primjer, obje funkcije primijenit ćemo na sljedeći način:
Can you write Python code for it?

niz_znakova='otorinolaringologija'
fd=frek_distr(niz_znakova)
print fd
print sortiraj_distr(fd)


I am working on the file "5-Funkcije.ipynb".
The first task is:
Ovdje smo definirali (def) novu funkciju imena uvecaj koja prima jedan argument ((broj)) te koja vraća (return) taj argument uvećan za jedan (broj +1). Iz tijela funkcije vidljivo je da argument mora biti brojčana vrijednost. Kako je Python dinamički tipiziran jezik, tj. kako se tip podataka određuje tijekom izvršavanja koda, nije moguće, odnosno potrebno ograničavati tip podataka koji se funkciji pruža kao argument.
Pogledajmo kako bi napisali funkciju koja prebrojava broj samoglasnika u nekom Unicode nizu znakova:
Can you write Python code for it?

def broj_samoglasnika(niz_znakova):
    rezultat=0
    for znak in niz_znakova:
        if znak.lower() in 'aeiou':
            rezultat+=1


I am working on the file "5-Funkcije.ipynb".
The first task is:
U nastavku ćemo pokrenuti funkcije nad datotekom datoteka.txt te ispisati sortiranu frekvencijsku distribuciju znakova i frekvencijsku distribuciju pojavnica.
Can you write Python code for it?

dat=open('datoteka.txt').read().decode('utf8').lower()
pojavnice=opojavnici(dat)
fd_znak=frek_distr(dat)
fd_pojavnica=frek_distr(pojavnice)
sd_znak=sortiraj_distr(fd_znak)
sd_pojavnica=sortiraj_distr(fd_pojavnica)
print sd_znak
print sd_pojavnica


I am working on the file "5-Funkcije.ipynb".
The first task is:
U prvom primjeru smo broj 23 uvećali za jedan, a u drugom smo izračunali broj samoglasnika u nizu Ovo je niz znakova..
U nastavku ćemo napisati dvije funkcije za računanje frekvencijske distribucije te njeno sortiranje. Oba smo postupka upoznali u poglavlju o rječnicima, no sada ćemo ih zapisati kao funkcije.
Prva je funkcija za računanje frekvencijske distribucije:
Can you write Python code for it?

def frek_distr(iterabilni):
    rjecnik={}
    for element in iterabilni:
        rjecnik[element]=rjecnik.get(element,0)+1
    return rjecnik


I am working on the file "5-Funkcije.ipynb".
The first task is:
U većini funkcija računamo rješenje nekog problema (u ovom slučaju broj samoglasnika u nizu znakova) te na kraju izračuna konačnu vrijednost vraćamo koristeći rezerviranu riječ return. Funkcija u ovom primjeru kao argument prima niz znakova (niz_znakova), a vraća cjelobrojnu vrijednost (rezultat) koja odgovara broju samoglasnika u tom nizu znakova.
Na sljedeći način ćemo upotrijebiti dvije napisane funkcije:
Can you write Python code for it?

print uvecaj(23)

print broj_samoglasnika('Ovo je niz znakova.')


I am working on the file "5-classification/5.2 - Logistic Regression in Classifying Breast Cancer .ipynb".
The first task is:
Evaluation and Inferences
As we know, machine learning models are mainly used for two purposes: predecting information for new data, and making inferences about the data we have. In the folloing we'll explore how good our trained Logistic Regression model in prediction and what can we infer from the formulation of the model.
Evaluating Predection Accuracy
There are many ways we can evaluate the accuracy of a classifier. One of these ways is provided by scikit-learn through the score method of the model. This function computes the mean accuracy of the model, which is expressed mathematically as:
$$\frac{1}{n}\sum_{i = 1}^{n} 1(y_i = \widehat{y_i}) \hspace{0.5em} \text{where } \widehat{y_i} \text{ is the predecited value}$$
$$,1(P) =
\left{
\begin{matrix}
1 \hspace{1.5em} \text{if } P \text{ is true}\
0 \hspace{1.5em} \text{otherwise}
\end{matrix}
\right.$$
The closer this value is to one, the more accurate our model is.
Can you write Python code for it?

accuracy = model.score(X_test, y_test)

"Model's Mean Accuracy: %0.3f" % (accuracy)


I am working on the file "5-classification/5.2 - Logistic Regression in Classifying Breast Cancer .ipynb".
The first task is:
The green data points in the plot represent benign tumors while the red ones represent malignant ones. The 3D visualization shows that the data points for each class tends to cluster togther with the exception of some noisy data points. This suggets that Logistic Regression would be a good estimator for the dataset.
Training the Logistic Regression Model
Scikit-learn's linear models bundle offers the LogisticRegression algorithm that trains a data set to fit a hypothesis function in the form:
$$h_w(X) = g(w^TX) \hspace{0.5em} \text{where } g \text{ is a logistic function}$$
Just like we saw in the videos. However, LogisticRegression doesn't use gradient descent to train the model, it uses other iterative optimzation methods (shuch as Newton's method); This is very suitable for small and medium datasets like we have here. But in the case of a large dataset, gradient descent would be the better choice. To train a logistic regression model using gradient descent with scikit-learn, use SGDClassifer. Here we'll work with LogisticRegression.
Before training the model, we'll first split our data into two parts: one for training and one for testing and evaluation. Remember that the general rule is to divide the dataset into three parts: training, validation, and testing; but because we'll not be doing any parameter tuning here we won't need the validation set, hence we'll divide the dataset into two parts only.
Can you write Python code for it?

dataset_size = len(focus_dataset)
training_size = np.floor(dataset_size * 0.6).astype(int)

X_training = focus_dataset[["CT", "UCS", "UCSh"]][:training_size]
y_training = focus_dataset["Class"][:training_size]

X_test = focus_dataset[["CT", "UCS", "UCSh"]][training_size:]
y_test = focus_dataset["Class"][training_size:]


I am working on the file "5-classification/5.2 - Logistic Regression in Classifying Breast Cancer .ipynb".
The first task is:
To understand the meaning of the abbreviations we can consult the dataset's website to find a description of each attribute in order. Here we'll be focusing on three attributes: CT, UCS and UCSh which stand for Clump Thickness, Uniformity of Cell Size and Uniformity of Cell Shape respectively. If you noticed the Class attribute at the end (which gives the class of the tumor), you'll find that it takes either 2 or 4; where 2 represents a benign tumor while 4 represents a malignant tumor. We'll chnage that to a more expressive values and make a benign tumor represented by 0 and mlignants by 1s.
Can you write Python code for it?

focus_dataset = dataset[["CT", "UCS", "UCSh", "Class"]]  # take only the attributes we'll focus on
focus_dataset.is_copy = False  # this is just to hide a nasty warning!
focus_dataset.loc[:, ("Class")] = [0 if tclass == 2 else 1 for tclass in dataset["Class"]]  # convert Class to 0/1

focus_dataset[:10]


I am working on the file "5-classification/5.2 - Logistic Regression in Classifying Breast Cancer .ipynb".
The first task is:
We can use LogisticRegression in the same way that we used the SGDRegressor when we worked with Linear Regression, and this is an amazing charactaristic of scikit-learn; that it has a standard API across the very different models. So to train our model, we'll create an instance of LogisticRegression and just call the fit method on that instace and pass our features (X) and labels (y).
Can you write Python code for it?

model = LogisticRegression()
model.fit(X_training, y_training)

w0 = model.intercept_
w1 = coefs[0]
w2 = coefs[1]
w3 = coefs[2]

print "Trained Model: h(x) = g(%0.2f + %0.2fx₁ + %0.2fx₂ + %0.2fx₃)" % (w0, w1, w2, w3)


I am working on the file "5-classification/5.2 - Logistic Regression in Classifying Breast Cancer .ipynb".
The first task is:
We can visually present our data using scatter plots in a 3D space to have more understanding of the how the data points group and how the attributes of Clump Thickness, Uniformity of Cell Size and Uniformity of Cell Shape correlate with the class of the tumor.
Can you write Python code for it?

benign_selector = focus_dataset["Class"] == 0

benign_datapoints = focus_dataset[benign_selector]
malignant_datapoints = focus_dataset[~benign_selector]

plot3D = plt.figure()
plot3D = plot3D.gca(projection='3d')

# here we create our scatter plot given our data
plot3D.scatter(benign_datapoints["CT"], benign_datapoints["UCS"], benign_datapoints["UCSh"], c='green')
plot3D.scatter(malignant_datapoints["CT"], malignant_datapoints["UCS"], malignant_datapoints["UCSh"], c='red')

# now we'll put some labels on the axes
plot3D.set_xlabel("CT")
plot3D.set_ylabel("UCS")
plot3D.set_zlabel("UCSh")

# finally we show our plot
plt.show()


I am working on the file "5-classification/5.3 - Visualizing SVMs with Diabetes.ipynb".
The first task is:
From the data summary we can notice two things:
1. There are some instances where the Blood Glucose and BMI take the value of 0. This is biologically impossible! This is probably due to errors in recording the data, so we need to filter those instances out of the dataset.

The features span a wide range, this can cause computational problems when we try to plot and visualize the results we get from our models. A good idea would be to scale the features into a reasonbale range.

We'll start by cleaning the data from the 0-valued instances.
Can you write Python code for it?

# filtering the data out of 0-valued instances
real_bg = focus_dataset["Blood Glucose"] != 0
real_bmi = focus_dataset["BMI"] != 0

focus_dataset = focus_dataset[real_bg & real_bmi]

x1_min, x1_max = focus_dataset["Blood Glucose"].min(), focus_dataset["Blood Glucose"].max()
x1_avg = np.mean(focus_dataset["Blood Glucose"])

x2_min, x2_max = focus_dataset["BMI"].min(), focus_dataset["BMI"].max()
x2_avg = np.mean(focus_dataset["BMI"])

print "Blood Glucose: min = %d, max = %d, mean = %0.2f" % (x1_min, x1_max, x1_avg)
print "BMI: min = %0.1f, max = %0.1f, mean = %0.2f" % (x2_min, x2_max, x2_avg)

focus_dataset.sample(10)


I am working on the file "5-classification/5.3 - Visualizing SVMs with Diabetes.ipynb".
The first task is:
Notice how the model works slightly better on the training data but slightly worse on the test data, this is beacuse of the non-linearity of the decision boundary that fit with the training data as much as possible. We can visualize that nonlinear solution using the same script we used for the linar kernel, no changes whatsoever!
Can you write Python code for it?

plt.figure()

xx, yy = np.meshgrid(np.arange(x1_min, x1_max, 0.01), np.arange(x2_min, x2_max, 0.01))
z = model.decision_function(np.c_[xx.ravel(), yy.ravel()])
z = z.reshape(xx.shape)

plt.pcolormesh(xx, yy, z > 0, cmap=cm.Paired)
plt.contour(xx, yy, z, colors=['k', 'k', 'k'], linestyles=['--', '-', '--'], levels=[-1., 0, 1.])
plt.scatter(focus_dataset["Blood Glucose"], focus_dataset["BMI"], c=colors)
plt.scatter(model.support_vectors_[:, 0], model.support_vectors_[:, 1], s=80,facecolors='none')


I am working on the file "5-classification/5.3 - Visualizing SVMs with Diabetes.ipynb".
The first task is:
Notice that the model got much better with the training data but got slightly worse with test ones. This is the classic bias/variance tradeoff we encounter everywhere in machine learning. In our case here, the model is very nonlinear and very complex that it fits so well to the noise of the training data but can't generalize as well to new unseen data. We can take a look at the complexity of the nonlinear solutions we have via the following plot that shows how data points are grouped into islands!
Can you write Python code for it?


I am working on the file "5-classification/5.3 - Visualizing SVMs with Diabetes.ipynb".
The first task is:
Now after we have cleaned tha data and have all our instances at biologically plausiable values, we turn to scaling the values of the features. We're gonna use the same technique we used back then when we were doing linear regression. We'll fit our scalar on the training data then use the fitted scalar to transform the rest of the data (the test data).
Can you write Python code for it?

def scale_features(X, scalar=None):
    if(len(X.shape) == 1):
        X = X.reshape(-1, 1)

    if scalar == None:
        scalar = StandardScaler()
        scalar.fit(X)

    return scalar.transform(X), scalar

# scaling the features
size = len(focus_dataset)
training_size = np.floor(size * 0.7).astype(int)

training_portion = focus_dataset.loc[:training_size, ("Blood Glucose", "BMI")]
test_portion = focus_dataset.loc[training_size:, ("Blood Glucose", "BMI")]

focus_dataset.loc[:training_size, ("Blood Glucose", "BMI")], train_scalar = scale_features(training_portion)
focus_dataset.loc[training_size:, ("Blood Glucose", "BMI")],_ = scale_features(test_portion, scalar=train_scalar)

x1_min, x1_max = focus_dataset["Blood Glucose"].min(), focus_dataset["Blood Glucose"].max()
x2_min, x2_max = focus_dataset["BMI"].min(), focus_dataset["BMI"].max()

focus_dataset.sample(10)


I am working on the file "5-classification/5.3 - Visualizing SVMs with Diabetes.ipynb".
The first task is:
Now, that we've finished preprocessing our data, we finally visualize the distribution of our data points by sactter plotting them. The green data points reprsent the healthy subjects, and the red ones represent the diabetics.
Can you write Python code for it?

plt.figure()

colors = ['green' if diagnosis == 0 else 'red' for diagnosis in focus_dataset["Class"]]
plt.scatter(focus_dataset["Blood Glucose"], focus_dataset["BMI"], c=colors)

plt.xlabel("Blood Glucose")
plt.ylabel("BMI")

plt.show()


I am working on the file "5-classification/5.3 - Visualizing SVMs with Diabetes.ipynb".
The first task is:
Polynomial Kernels
Another kind of kernels we can work with is the polynomial kernel. This kernel impliciltly maps our feature space to a $d$ dimensional feature space through the equation:
$$K(x_i,x_i') = (\gamma.(x_i.x_i') + r)^d$$
Where $d$ is the dgeree of the polynomial and $\gamma,r$ are free parameters. To train a model using this kernel we pass kernel='poly'. We can change the values of $d,\gamma,r$ using the arguments degree, gamma, and coef0 respectively.
Can you write Python code for it?

model = SVC(kernel='poly', degree=3)
model.fit(X_train, y_train)

mean_accuracy_train = model.score(X_train, y_train)
mean_accuracy_test = model.score(X_test, y_test)

print "Mean Accuracy on Train Data: %0.3f" % (mean_accuracy_train)
print "Mean Accuracy on Test Data: %0.3f" % (mean_accuracy_test)


I am working on the file "5-classification/5.3 - Visualizing SVMs with Diabetes.ipynb".
The first task is:
To simplify our work and make the visualizations feasable, we'll be focusing on only two features of the eight: Blood Glucose and Body Mass Index (BMI). We'll now extract our focus dataset and see a summary of the values for both features:
Can you write Python code for it?

focus_dataset = dataset[["Blood Glucose", "BMI", "Class"]]
focus_dataset.is_copy = False

x1_min, x1_max = focus_dataset["Blood Glucose"].min(), focus_dataset["Blood Glucose"].max()
x1_avg = np.mean(focus_dataset["Blood Glucose"])

x2_min, x2_max = focus_dataset["BMI"].min(), focus_dataset["BMI"].max()
x2_avg = np.mean(focus_dataset["BMI"])

print "Blood Glucose: min = %d, max = %d, mean = %0.2f" % (x1_min, x1_max, x1_avg)
print "BMI: min = %0.1f, max = %0.1f, mean = %0.2f" % (x2_min, x2_max, x2_avg)

focus_dataset.sample(10)


I am working on the file "5-classification/5.3 - Visualizing SVMs with Diabetes.ipynb".
The first task is:
Understanding and Visualizing the Data
The dataset we'll be working with here is one of the classics in the machine learning community. The dataset is a collection of clinical measurments for 768 women of the Pima Indian heritage and their diagnosis of diabetes. We have 8 clinical measurments like: Pregnancy Count, Blood Glucose, Body Mass Index (BMI), Diastolic Blood Pressure, ... etc. The diagnosis of diabetes is represented through the Class column: 1 means that the subject has diabetes, 0 means that the subject hasn't.
We can take a look at a sample of the data to understand better:
Can you write Python code for it?

dataset = pd.read_csv('../datasets/pima-indians-diabetes.csv')
dataset.sample(10)


I am working on the file "5-classification/5.4 - Artificial Neural Networks in Classifying Breast Cancer.ipynb".
The first task is:
Our neural network has been configured and built! Now, we just need to train it using our training set, using the intuitively named function "fit":
Can you write Python code for it?

nn.fit(X_train, y_train) # begin training using backpropagation!


I am working on the file "5-classification/5.4 - Artificial Neural Networks in Classifying Breast Cancer.ipynb".
The first task is:
The Data
We'll start off by exploring our dataset to see what attributes we have and how the class of the tumor is represented
Before we proceed, ensure to include headers to the dataset provided by the University of Wisconsin. We will use the following headers:

ID,CT,UCS,UCSh,MA,SECS,BN,BC,NN,M,Class

Add this to the beginning line of your .csv file.
Can you write Python code for it?

dataset = pd.read_csv('./datasets/breast-cancer-wisconson.csv') # import the CSV data into an array using the panda dependency
print dataset[:10]


I am working on the file "5-classification/5.4 - Artificial Neural Networks in Classifying Breast Cancer.ipynb".
The first task is:
The output in my console due to the DEBUG logging looked like this (it's fun to see training in action!):
<img src="./../images/train_img.png">
NOTE: We do not need to prefix 1s to the dataset beforehand to achieve bias terms that vertically shift the decision boundary regions. The API provides this by default.
Results
We can see the weights that were produced (including the bias terms) using the following line of code:
Can you write Python code for it?

print nn.get_parameters() # output the weights of the neural network


I am working on the file "5-classification/5.4 - Artificial Neural Networks in Classifying Breast Cancer.ipynb".
The first task is:
To understand the meaning of the abbreviations we can consult the dataset's website to find a description of each attribute in order. We are going to train on all features unlike the logistic regression example (where we just trained on three). This does mean that we will be unable to visualize the results, but will get a feel for how to work with high-dimensional data.
If you noticed the Class attribute at the end (which gives the class of the tumor), you'll find that it takes either 2 or 4, where 2 represents a benign tumor while 4 represents a malignant tumor. We'll change that to more expressive values and make a benign tumor represented by 0 (false) and mlignants by 1s (true).
You'll notice that the ID attribute of data that is useless to our modelling, since it provides no information about the tumor itself, and is instead a way of identifying a specific tumor. We will hence strip this from our dataset before training.
Can you write Python code for it?

dataset = dataset[["CT", "UCS", "UCSh", "MA", "SECS", "BN", "BC", "NN", "M", "Class"]]  # remove the ID attribute from the dataset
dataset.is_copy = False  # this is just to hide a nasty warning!
[0 if tclass == 2 else 1 for tclass in dataset["Class"]]  # convert Class attributes to 0/1 if they are 2/4 in dataset["Class"] column


I am working on the file "5-classification/5.4 - Artificial Neural Networks in Classifying Breast Cancer.ipynb".
The first task is:
Training the Neural Network Model
For the training, we are going to split the dataset into a training set and a test set. The training set will be 70% of the original data set, and will be what the neural network will learn from. We will test the accuracy of the neural network's learned weights by using the test set, which is composed of 30% of the original data.
It's very important to shuffle the dataset before partitioning into training/test sets. Why? Because the data given to us by University of Wisconsin may be in some sorted (apparent or unapparent) order. It may be that most y=0 examples are in the latter half of the data. It may be that most of the nearby recorded patients are similar, or were recorded during similar dates/times. We want none of that - we want to remove any information and ensure that our partitioning is random, so that our test results represent true probabilites of picking a random training case from an entire population of permutations of the feature vector. For example, I once got a 5% difference in test results since the data I had was sorted beforehand. Essentially, we want to make sure that we're being absolutely fair about the partition, and not accidentally making our test results too good/too bad.
We can't use numpy's traditional shuffle function because this shuffles one array only. If we independently shuffled x and y, the order between them would be lost (ie. if training case x had output of 1 beforehand, this may be accidentally changed to an output of 0, since we use corresponding indicdes to match up the inputs and outputs).
Can you write Python code for it?

X, y = shuffle(X, y, random_state=0) # we use scikit-learn's synchronized shuffle feature to shuffle two arrays in unison

dataset_size = len(dataset) # get the size of overall dataset
training_size = np.floor(dataset_size * 0.7).astype(int) # get the training size as 70% of the dataset size (or roughly 0.7 * dataset_size) and as an integer

X_train = X[:training_size] # extract the first 70% of inputs for training
y_train = y[:training_size] # extract the first 70% of inputs for training

X_test = X[training_size:] # extract rest 30% of inputs for testing
y_test = y[training_size:] # extract rest 30% of outputs for testing


I am working on the file "5-classification/5.4 - Artificial Neural Networks in Classifying Breast Cancer.ipynb".
The first task is:
Unlike logistic regression, the weights of neural networks, unfortunately, are not very interpretable. There are also a high number of these weights. For example, the weights my system produced were outputted as so:
<img src="./../images/weights.png">
However, it is more important to output the acurracy of these results, and how much error is made. Unlike logistic regression where we used a cost function that outputted a real number, we are going to find the percent error of our system like so:

Create predictions in the form of probabilities for each test tumor being malignant
Iterate through the test set with index i
Fetch the predicted output probability of this test tumor
Round this probability to either a 1 (malignant) or a 0 (benign)
Compare this to the correct test output at the ith index
If an error occurs, increment some pre-initialized error counter
Get the percentage error by dividing the error counter by the total number of test examples, multiplying by 100
Can you write Python code for it?

error_count = 0.0 # initialize the error counter
prob_predictions = nn.predict_proba(X_test) # predict the outputs of the X_test instances, in the form of probabilities (hence predict_proba)
for i in xrange(len(prob_predictions)): # iterate through all the predictions (equal in size to test set)
    # create a discrete decision for the tumor being malignant or benign, using 0.5 as the lowest probability needed for a predicted malignancy (general rounding)
    # as discussed before, our network actually outputs [probability_of_benign, probability_of_malignant], so we will want to
    # fetch the probability_of_malignant value and round this one (that's how it would be for a single output network if it worked!)
    discrete_prediction = 0 if prob_predictions[i][1] < 0.5 else 1
	if not y_test[i] == discrete_prediction: # if the actual, correct value for this test tumor does not equal the discrete prediction
		error_count += 1.0 # increment the number of errors

error_rate = error_count / len(prob_predictions) * 100 # get the percentage of errors by dividing total errors by number of instances, multiplying by 100
print error_count # print number of raw errors
print str(error_rate) + "%" # output this error percentage


I am working on the file "5-classification/5.4 - Artificial Neural Networks in Classifying Breast Cancer.ipynb".
The first task is:
We will now need to split up the dataset into two separate tensors: X and y. X will contain the features and their values for each training example, and y will contain all the outputs. In this training set, let's say m refers to the number of training examples (of which there are just over 600) and n refers to the number of features (of which there are 9). Thus, X will be a matrix where $X\in \mathbb{R}^{m\:\cdot \:n}$ and y is a vector where $y\in \mathbb{R}^m$ (because we only have one output - a probability of the tumor being malignant).
We simply separate by the "Class" attribute and the other features.
Can you write Python code for it?

X = np.array(dataset[["CT", "UCS", "UCSh", "MA", "SECS", "BN", "BC", "NN", "M"]]) # X is composed of all the n feature columns
y = np.array(dataset["Class"]) # y is composed of just the output class column


I am working on the file "5-classification/5.4 - Artificial Neural Networks in Classifying Breast Cancer.ipynb".
The first task is:
scikit-neuralnetwork offers a really neat, easy to use API (from sknn.mlp import Classifier, Layer) for training neural networks. This API has support for many different paradigms like dropout, momentum, weight decay, mini-batch gradient descent etc. and even different neural network types like Convolutional Neural Networks. Today, however, our goal is to get a simple Artificial Neural Network setup!
Our first job is to configure the architecture for the neural network. We will need to decide:
* The number of hidden layers
* The size of each hidden layer
* The activation function used at each hidden layer
* The learning rate, number of iterations, and other hyperparameters
Some types of activation functions offered by scikit-neuralnetwork include:
* Linear
* Rectifier
* Sigmoid
* Softmax
Where Softmax computes a sigmoid probability distribution over multiple outputs. Generally, it is conventional to use Softmax as the activation function for the output layer (when we have 1 output it really is just the same as a Sigmoid layer, but scikit-neuralnetwork will still output a warning). Recall the formula for the sigmoid activation function:
$Sigmoid\left(z\right)=\frac{1}{1+e^{-z}}$
This function "squeezes" any real value into a probability of range (0, 1).
The Linear and Rectifier (ReLU) may be used, but we obviously need to use Sigmoid/Softmax in our neural network because we are performing a classification task. Generally, I found that using Rectifier units throughout and then having a Softmax (Sigmoid) layer at the end produced the best results.
Now we need to decide on our structure (as in, the hidden layers and their sizes). Our neural network will end up looking like the following:
<img src="./../images/neuralnet.png">
That is, for this example, we will use two hidden layers (both of which are Rectifiers). Generally, the greater number of hidden layers you have, the greater complexity. Two should be fine for our task though. In our neural network, we will have 9 input nodes (for our 9 features), and I have...
Can you write Python code for it?

nn = Classifier(layers=[ # create a new Classifier object (neural network classifier), and pass all the layers as parameters
        Layer("Rectifier", units=100), # create the first post-input hidden layer, a Rectifier (ReLU) layer of 100 units
        Layer("Rectifier", units=50), # create the second hidden layer, a Rectifier layer of 50 units
        Layer("Softmax"), units=2], # create the final output layer, a Softmax layer that will output two probabilities, as mentioned before
        learning_rate=0.001, n_iter=100) # pass in hyperparameters as a separate parameter to the layers


I am working on the file "5-classification/5.4.2 - Peeking Inside a Neural Network with MNIST Data.ipynb".
The first task is:
Building and Training the Neural Network
In the same way described in the previous part, we'll build a small neural network with one hidden layer of 64 neurons, and one softmax output layer. We're keeping it small to be able to peek inside it later and visualize the weights and try to get an intution of what they do.
Can you write Python code for it?

# the use of batch_size in the following specifies that instead of
# training the network on all the data instances every iteration
# we train on a random sample of size 128 from the data
# this is called stochastic gradient descent (SGD) and it's much faster
# than the regular gradient descent without much loss in accuracy
nn = Classifier(
    layers = [
        Layer("Rectifier", units=64),
        Layer("Softmax")
    ],
    verbose = True,
    n_iter=100,
    random_state=1,
    batch_size=128
)

# This may take some time, be patient!
nn.fit(X_train, y_train)


I am working on the file "5-classification/5.4.2 - Peeking Inside a Neural Network with MNIST Data.ipynb".
The first task is:
For this visulaization of the first 9 images, it appears that the data is ordered by the value of the digit. This ordering might result in a biased test set, so we need to shuffle the data to ensure statsically unbiased sets.
Can you write Python code for it?

# random_state is a seed value to enure we get the same shuffling each run
X, y = shuffle(X, y, random_state=0)

instances = X[0:9]
instances = instances.reshape(-1, 28, 28)

# here we create a 3x3 grid of figures (9 plots)
fig, axes = plt.subplots(3, 3)
axes = axes.ravel()  # flat the 3x3 array of references to plots into a 1d array of size 9

for i,instance in enumerate(instances):
    axes[i].matshow(instance, cmap=cm.gray)
plt.show()


I am working on the file "5-classification/5.4.2 - Peeking Inside a Neural Network with MNIST Data.ipynb".
The first task is:
Now the data is shuffled, and due to the shuffling we also got to take a diverse look at the data images. Now we're one step away from building our neural network, we just need to split our dataset into training and testing parts and we're ready!
Can you write Python code for it?

dataset_size = X.shape[0]
train_size = np.floor(dataset_size * 0.7).astype(int)

X_train, y_train = X[:train_size], y[:train_size]
X_test, y_test = X[train_size:], y[train_size:]


I am working on the file "5-classification/5.4.2 - Peeking Inside a Neural Network with MNIST Data.ipynb".
The first task is:
Now we can view some of the images to see what we're dealing with here. We can show an image from the raw pixel data using metaplotlib's method matshow which takes a matrix and dispalys it as image. Note that each image is represented as a flat 1D array of size 784, we'll need to reshape it to a 2D 28x28 array to be able to display it.
Can you write Python code for it?

instances = X[0:9]
instances = instances.reshape(-1, 28, 28)

# here we create a 3x3 grid of figures (9 plots)
fig, axes = plt.subplots(3, 3)
axes = axes.ravel()  # flat the 3x3 array of references to plots into a 1d array of size 9

for i,instance in enumerate(instances):
    axes[i].matshow(instance, cmap=cm.gray)
plt.show()


I am working on the file "5-classification/5.4.2 - Peeking Inside a Neural Network with MNIST Data.ipynb".
The first task is:
Peeking Inside The Network
Now we attempt to take a look inside the network and see what it's doing to the inputs. Our first step is to visualize the weigths of each layer and the transformation they perform on an input data.
The hidden layer has 64 neuron and takes as input our flat data image of size 784. So the weigths of the first layer is 784x64 matrix, or a 28x28x64 rank-3 tesnor (you can think of a rank-k tensor as a multidimensional array of k dimensions. A matrix is a rank-2 tensor). We then visualize each neuron weights as a 28x28 image just as we did with the input images.
Can you write Python code for it?

hidden0 = nn.get_parameters()[0][0]
vmin, vmax = hidden0.min(), hidden0.max()

fig, axes = plt.subplots(8, 8)
axes = axes.ravel()

for i, axis in enumerate(axes):
    weights = hidden0[:,i].reshape(28, 28)
    axis.matshow(weights, cmap=cm.gray)

plt.show()


I am working on the file "5-classification/5.4.2 - Peeking Inside a Neural Network with MNIST Data.ipynb".
The first task is:
Preparing the data
To save us a lot of headache of downloading the images of the MNIST dataset and loading the images into memory to build the dataset, we'll use one of scikit-learn's builtin method in the datasets package to automatically retrieve the dataset and load it into memory; this is the fetch_mldata method. This method retrieves by name a dataset from the mldata.org data repository.
Can you write Python code for it?

# you may have to wait a bit for the data to be downlaoded
mnist = fetch_mldata("MNIST original")

print "Data min. value: %d" % (mnist.data.min())
print "Data max. value: %d" % (mnist.data.max())
print "Data instance size: %d" % (mnist.data.shape[1])


I am working on the file "5-classification/5.4.2 - Peeking Inside a Neural Network with MNIST Data.ipynb".
The first task is:
We can see that the values of the pixels are between 0 and 255. To have an effcient training experince, we should scale the data to be between 0 and 1. This is a classic case of feature sacling which would make the training process faster. In this case we do not need a sophistacted scalar like we used before, all we need to do is to divide the data by 255.0.
Can you write Python code for it?

X, y = mnist.data / 255., mnist.target

print "Data min. value: %.2f" % (X.min())
print "Data max. value: %.2f" % (X.max())


I am working on the file "5-functions.ipynb".
The first task is:
Combining & tidying up

now we know how to write functions, we can make our analysis easier to read
first lest make an analyze function that generate our plots
Can you write Python code for it?

def analyze(filename):

    data = numpy.loadtxt(fname=filename, delimiter=',')

    fig = matplotlib.pyplot.figure(figsize=(10.0, 3.0))

    axes1 = fig.add_subplot(1, 3, 1)
    axes2 = fig.add_subplot(1, 3, 2)
    axes3 = fig.add_subplot(1, 3, 3)

    axes1.set_ylabel('average')
    axes1.plot(data.mean(axis=0))

    axes2.set_ylabel('max')
    axes2.plot(data.max(axis=0))

    axes3.set_ylabel('min')
    axes3.plot(data.min(axis=0))

    fig.tight_layout()
    matplotlib.pyplot.show(fig)


I am working on the file "5-functions.ipynb".
The first task is:
What about Fahrenheit to Celsius?
* we could write out the forumla, but we don't need to
* we can compose the two functions we have already created
Can you write Python code for it?

def fahr_to_celsius(temp_f):
    temp_k = fahr_to_kelvin(temp_f)
    result = kelvin_to_celsius(temp_k)
    return result

print('freezing point of water in Celsius:', fahr_to_celsius(32.0))


I am working on the file "5-functions.ipynb".
The first task is:
a string like this is called the docstring
we don't need to use 3 quotes when we write one but if we do we can use multiple lines
Can you write Python code for it?

def center(data, desired):
    '''Return a new array containing the original data centered around the desired value.
    Example: center([1, 2, 3], 0) => [-1, 0, 1]'''
    return (data - data.mean()) + desired

help(center)


I am working on the file "5-functions.ipynb".
The first task is:
add another function called detect_problems that checks for those systematics we noticed:
Can you write Python code for it?

def detect_problems(filename):

    data = numpy.loadtxt(fname=filename, delimiter=',')

    if data.max(axis=0)[0] == 0 and data.max(axis=0)[20] == 20:
        print('Suspicious looking maxima!')
    elif data.min(axis=0).sum() == 0:
        print('Minima add up to zero!')
    else:
        print('Seems OK!')


I am working on the file "5-functions.ipynb".
The first task is:
as this shows parameters are matched up from left to right
any that haven't been given a value explicitly get their default value
Can you write Python code for it?

# we can override this behavior by naming value as we pass it in
print('only setting the value of c')
display(c=77)


I am working on the file "5-functions.ipynb".
The first task is:
better because we can now ask python's built-in help to show us the documentation
Can you write Python code for it?


I am working on the file "5-functions.ipynb".
The first task is:
but we still need the delimiter=:
Can you write Python code for it?


I am working on the file "5-functions.ipynb".
The first task is:
difference is zero
before we get back to our analysis let's add documentation
documentation is not only good for collaborators, but also the future you
we can add doc this way
Can you write Python code for it?

# center(data, desired): return a new array containing the original data centered around the desired value.
def center(data, desired):
    return (data - data.mean()) + desired


I am working on the file "5-functions.ipynb".
The first task is:
first taste of how larger programs are built:
* we define basic operations
* combine them in ever-larger chunks
Challenge - function:
“Adding” two strings produces their concatenation: 'a' + 'b' is 'ab'. Write a function called fence that takes two parameters called original and wrapper and returns a new string that has the wrapper character at the beginning and end of the original. A call to your function should look like this:
print(fence('name', '*'))
Can you write Python code for it?

def fence(original, wrapper):
    return wrapper + original + wrapper

print(fence('name', '*'))


I am working on the file "5-functions.ipynb".
The first task is:
hard to tell from the output if the result correct, let's test
Can you write Python code for it?

print('original min, mean, and max are:', data.min(), data.mean(), data.max())
centered = center(data, 0)
print('min, mean, and and max of centered data are:', centered.min(), centered.mean(), centered.max())


I am working on the file "5-functions.ipynb".
The first task is:
key change is the second parameter is now written desired=0.0
if we call the function with two arguments, it works as it did before
Can you write Python code for it?

test_data = numpy.zeros((2, 2))
print(center(test_data, 3))


I am working on the file "5-functions.ipynb".
The first task is:
looks good, now let's try and center our real data
Can you write Python code for it?

data = numpy.loadtxt(fname='data/inflammation-01.csv', delimiter=',')
print(center(data, 0))


I am working on the file "5-functions.ipynb".
The first task is:
lots of info but look at this part:
python
loadtxt(fname, dtype=<type 'float'>, comments='#', delimiter=None, converters=None, skiprows=0, usecols=None,
        unpack=False, ndmin=0)

tells us that loadtxt has one parameter called fname that doesn't have a default value, & 8 others that do
if we call the function like this:
Can you write Python code for it?


I am working on the file "5-functions.ipynb".
The first task is:
since we don't know what values ought to be in our data let's create a matrix of zeros and center around 3.
Can you write Python code for it?

z = numpy.zeros((2,2))
print(center(z, 3))


I am working on the file "5-functions.ipynb".
The first task is:
use def followed by function name and parenthesized list of parameter names
we call a function and the values we pass it are assigned to those varialbes so we can use inside
use return statement to send back result

We can now try our function out!
Can you write Python code for it?

print('freezing point of water:', fahr_to_kelvin(32))
print('boiling point of water:', fahr_to_kelvin(212))


I am working on the file "5-sympy.ipynb".
The first task is:
$\frac{d^3f}{dxdy^2}$
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Con la clase Matrix podemos realizar las operaciones matriciales típicas
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Evaluación numérica
SymPy usa una librería para trabajar con números con precisión arbitraria, y tiene expresiones SymPy  predefinidas para varias constantes matemáticas, tales como: pi, e y oo para el infinito.
Para evaluar numéricamente una expresión podemos usar la función evalf (o bien N). Ésta usa un argumento n que especifíca el número de cifras significativas.
Can you write Python code for it?

sym.pi.evalf(n=50)

y = (x + sym.pi)**2

sym.N(y, 5) # equivalente a evalf


I am working on the file "5-sympy.ipynb".
The first task is:
Integracion
La integración se realiza de la misma forma:
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
La derivada por definicion es
$$\frac{d}{{dx}}f\left( x \right) = \mathop {\lim }\limits_{h \to 0} \frac{{f\left( {x + h } \right) - f\left( x \right)}}{h }$$
Can you write Python code for it?

h = sym.Symbol("h")

sym.limit((f.subs(x, x+h) - f)/h, h, 0)


I am working on the file "5-sympy.ipynb".
The first task is:
La ecuacion puede estar en términos de otras variables simbólicas
Can you write Python code for it?

sym.solve([x + y - a, x - y - c], [x,y])


I am working on the file "5-sympy.ipynb".
The first task is:
La expansion en serie incluye la aproximación de orden. Lo cual es útil para trabajar con series de diferente orden
Can you write Python code for it?

s1 = sym.cos(x).series(x, 0, 5)
s1

s2 = sym.sin(x).series(x, 0, 2)
s2

sym.expand(s1 * s2)


I am working on the file "5-sympy.ipynb".
The first task is:
La función subs también puede ser usada para substituir símbolos y expresiones:
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
La mayor eficiencia de usar funciones "lambdificadas" en lugar de usar evalación numérica directa puede ser significativa, a menudo de varios órdenes de magnitud. Aún en este sencillo ejemplo obtenemos un aumento de velocidad importante:
Can you write Python code for it?

%%timeit

y_vec = np.array([sym.N(((x + sym.pi)**2).subs(x, xx)) for xx in x_vec])

%%timeit

y_vec = f(x_vec)


I am working on the file "5-sympy.ipynb".
The first task is:
Lecturas complementarias

http://sympy.org/en/index.html - The SymPy projects web page.
https://github.com/sympy/sympy - The source code of SymPy.
http://live.sympy.org - Online version of SymPy for testing and demonstrations.
Can you write Python code for it?

# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = './css/aeropython.css'
HTML(open(css_file, "r").read())


I am working on the file "5-sympy.ipynb".
The first task is:
Lo mismo con productos
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Límites
Los límites se evalúan con la función limit
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Manipulaciones algebráicas
Uno de los usos principales de un sistema de cálculo simbólico es que realiza manipulaciones algebráicas de expresiones. Por ejemplo, si queremos expandir un producto, factorizar una expresión, o simplificar un resultado. En esta sección presentamos las funciones para realizar estas operaciones básicas en SymPy.
Expand and factor
Primeros pasos en la manipulación algebráica
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Números complejos
La unidad imaginaria es denotada por I en Sympy.
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Números racionales
Existen tres tipos distintos de números en SymPy: Real, Rational, Integer:
Can you write Python code for it?

r1 = sym.Rational(4,5)
r2 = sym.Rational(5,4)

r1

r1+r2

r1/r2


I am working on the file "5-sympy.ipynb".
The first task is:
Para calcular derivadas de orden superior podemos usar:
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Pero hay que tener en cuenta que esta no es la expansión correcta de $\cos(x)\sin(x)$ al $5$to orden:
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Podemos agregar algunas propiedades a los símbolos cuando son creados:
Can you write Python code for it?

x = sym.Symbol('x', real=True)

x.is_imaginary

x = sym.Symbol('x', positive=True)

x > 0


I am working on the file "5-sympy.ipynb".
The first task is:
Por defecto, se calcula la expansión alrededor de $x=0$ (serie de Taylor), pero podemos expandir la serie al rededor de otro punto
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Series
Las series tambien son una de las funciones mas útiles de un CAS. En SymPy se puede calcular la serie de una funcion usando series
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Si le damos los límites de integración calculamos la integral definida
Can you write Python code for it?


I am working on the file "5-sympy.ipynb".
The first task is:
Simplify
La función simplify intenta simplificar una expresión usando distinta técnicas. Existen también alternativas más específicas a la función simplify: trigsimp, powsimp, logcombine, etc.
El uso básico de estas funciones en el siguiente:
Can you write Python code for it?

# simplify expande un producto
sym.simplify((x+1)*(x+2)*(x+3))

# simplify usa identidades trigonometricas
sym.simplify(sym.sin(a)**2 + sym.cos(a)**2)

sym.simplify(sym.cos(x)/sym.sin(x))