I am working on the file "3. Tensorflow programming model.ipynb".
The first task is:
There are more Tensorflow Operations
tf graphs
Creating a graph is simple:
python
import tensorflow as tf
g = tf.Graph()
Once the graph is initialized we can attach operation to it by using the Graph.as_default() method:
python
with g.as_default():
    a = tf.mul(2,3)
    ...
tf automatically creates a graph at the beginning and assigns it to be the default. Thus, if not using Graph.as_default() any operation will be automatically placed in the default graph.
Creating multiple graphs can be useful if you are defining multiple models that do not have interdependencies:
```python
g1 = tf.Graph()
g2 = tf.Graph()
with g1.as_default():
    ...
with g2.as_default():
    ...
```
tf Variables
Tensor and Operation objects are immutable, but we need a mechanism to save changing values over time.
This is accomplished with Variable objects, which contain mutable tensor values that persist accross multiple calls to Session.run().
Variables can be used anywhere you might use a tensor.
tf has a number of helper operations to initialize variables: tf-zeros(), tf_ones(), tf.random_uniform(), tf.random_normal(), etc.
Variable objects live in a Graph but their state is managed by Session. Because of these they need an extra step for inicialization:
```python
import tensorflow as tf
tf.reset_default_graph()
a = tf.Variable(3,name="my_var")
b = tf.add(5,a)
with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    ...
```
In order to chage the value of a Variable we can use the Variable.assign() method:
Can you write Python code for it?

import tensorflow as tf
tf.reset_default_graph()

a = tf.Variable(3,name="my_var")
b = a.assign(tf.mul(2,a))  # variables are objects, not ops.

# The statement a.assign(...) does not actually assign any to a,
# but rather creates a tf.Operation that you have to explicitly
# run to update the variable.

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print "a:",a.eval()   # variables are objects, not ops.
    print "b:", sess.run(b)
    print "b:", sess.run(b)
    print "b:", sess.run(b)

tf.reset_default_graph()

a = tf.Variable(3,name="my_var")
b = a.assign(tf.mul(2,a))
with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print a.eval()

tf.reset_default_graph()

a = tf.Variable(3,name="my_var")
b = a.assign(tf.mul(2,a))
with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    sess.run(b)
    print a.eval()


I am working on the file "3. Tensorflow programming model.ipynb".
The first task is:
We can feed data points to placeholder by iterating through the data set:
Can you write Python code for it?

tf.reset_default_graph()

list_a_values = [1,2,3]

a = tf.placeholder(tf.int16)
b = a * 2

with tf.Session() as sess:
    for a_value in list_a_values:
        print(sess.run(b,{a: a_value}))


I am working on the file "3. Tensorflow programming model.ipynb".
The first task is:
We can increment and decrement variables:
Can you write Python code for it?

import tensorflow as tf
tf.reset_default_graph()

a = tf.Variable(3,name="my_var")

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print(sess.run(a.assign_add(1)))
    print(sess.run(a.assign_sub(1)))
    print(sess.run(a.assign_sub(1)))
    sess.run(tf.initialize_all_variables())
    print(sess.run(a))


I am working on the file "3. Tensorflow programming model.ipynb".
The first task is:
We can start tensorboard to see the graph: tensorboard --logdir="./name_scope_1".
You can expand the name scope boxes by clicking +.
Exercise
Let's built and visualize and complex model:

Our inputs will be placeholders.
The model will take in a single vector of any lenght.
The graph will be segmented in name scopes.
We will accumulate the total value of all outputs over time.
At each run, we are going to save the output of the graph, the accumulated total of all outputs, and the average value of all outputs to disk for use in tensorboard.
Can you write Python code for it?

import tensorflow as tf
import numpy as np

tf.reset_default_graph()

# Explicitly create a Graph object
graph = tf.Graph()

with graph.as_default():

    with tf.name_scope("variables"):
# your code here

    # Primary transformation Operations
    with tf.name_scope("transformation"):

        # Separate input layer
        with tf.name_scope("input"):
# your code here

        # Separate middle layer
        with tf.name_scope("intermediate_layer"):
# your code here

        # Separate output layer
# your code here

    with tf.name_scope("update"):
        # Increments the total_output Variable by the latest output
# your code here

    # Summary Operations
    with tf.name_scope("summaries"):
        avg = tf.div(update_total, tf.cast(increment_step, tf.float32), name="average")

        # Creates summaries for output node
        tf.scalar_summary(b'Output', output, name="output_summary")
        tf.scalar_summary(b'Sum of outputs over time', update_total, name="total_summary")
        tf.scalar_summary(b'Average of outputs over time', avg, name="average_summary")

    # Global Variables and Operations
    with tf.name_scope("global_ops"):
        # Initialization Op
        init = tf.initialize_all_variables()
        # Merge all summaries into one Operation
        merged_summaries = tf.merge_all_summaries()

# Start an interactive Session, using the explicitly created Graph
sess = tf.Session(graph=graph)

# Open a SummaryWriter to save summaries
writer = tf.train.SummaryWriter('./improved_graph', graph)

# Initialize Variables
sess.run(init)


I am working on the file "3. Tensorflow programming model.ipynb".
The first task is:
You can create initialized tensors in many ways:
Can you write Python code for it?

a = tf.zeros([2,3], tf.int32)
b = tf.ones([2,3], tf.int32)
c = tf.fill([3,3], 23.9)
d = tf.range(0,10,1)

with tf.Session() as sess:
    print(sess.run(a))
    print(sess.run(b))
    print(sess.run(c))
    print(sess.run(d))


I am working on the file "3. Tensorflow programming model.ipynb".
The first task is:
tf has a very useful tool: tensor-board. Let's see how to use it.
Can you write Python code for it?

# cleaning the tf graph space
tf.reset_default_graph()

a = tf.placeholder(tf.int16, name='input_a')
b = tf.placeholder(tf.int16, name='input_b')
c = tf.add(a,b,name='add_1')
d = tf.mul(a,b,name='mul_1')
e = tf.add(c,d,name='add_2')

values = feed_dict={a: 5, b: 3}

# now we can run the graph

# graphs are run by invoking Session objects
session = tf.Session()

# when you are passing an operation to 'run' you are
# asking to run all operations necessary to compute that node

# you can save the value of the node in a Python var
output = session.run(e, values)
print(output)

# now let's visualize the graph

# SummaryWriter is an object where we can save information
# about the execution of the computational graph
writer = tf.train.SummaryWriter('my_graph', session.graph)
writer.close()

# closing interactive session
session.close()
print output


I am working on the file "3. Tensorflow programming model.ipynb".
The first task is:
tf sequences are not iterable!
We can also generate random variables:
Can you write Python code for it?

a = tf.random_normal([2,2], 0.0, 1.0)
b = tf.random_uniform([2,2], 0.0, 1.0)

with tf.Session() as sess:
    print(sess.run(a))
    print(sess.run(b))


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
Analysis
Our model seems to work well from the graph of the the forecast. Looking at the coefficients from the ARIMA model, we can see that that increasing temperatures and increasing day ahead prices are both associated with higher realtime prices the next day. Just by looking at the coefficients, one can see that our forecast is extremely similar to the day ahead values, with a small adjustment based on temperature and another adjustment based on the moving average term.
Can you write Python code for it?

plt.figure(figsize=(14, 10))
plt.plot(predicted_prices, label='prediction')
plt.plot(daily_df['rtm_lbmp'].values, label='actual RTM')
plt.plot(daily_df['dam_lbmp'].values, label='actual DAM')

plt.legend()


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
Forecasting with Time Series Data
Introduction to Time Series Data
Much of the data we interact with on a daily basis are drawn from time series. What's the temperature outside today? Is my diet working? How's the stock market doing? A time series is a set of observations of a single variable at multiple different points in time. Time series data differs from other data because these observations aren't usually independent from each other. For example, the stock price of Microsoft today is very related to the MSFT price yesterday. In contrast, if you were to sample people in this room to look for a relationship between height and GPA, you could be reasonably confident that each observation $(height, gpa)$ is independent from each other.
Modelling a Time Series
Stationarity
A process (from which a time series is sampled) is said to be stationary if the distribution of the observed values does not depend on time. One example of a nonstationary process is a stock price -- over the decades there's certainly been an upward trend, so the average price of IBM in 1995 is different than the average price of IBM in 2015. What we want is the distribution of the observed variable to be independent of time. We'd like for the mean and our variance of our observations to be constant over time.
Taking the trends out of data to make it stationary allows us to properly run regressions against other variables (otherwise we risk spurious results that conflate the time trend with the effect of the other variables). We can make data stationary by taking differences of of the observations. For example, below we have a graph of drug sales. There is clearly a trend over time--by looking at the difference in sales instead of the sales numbers themselves, we can more accurately find the relationships between it and a proposed explanatory variable.
<img src="https://www.otexts.org/sites/default/files/fpp/images/diffa10.png" width="600"/>
A process that becomes stationary after differencing d times is said to be integrated with order...
Can you write Python code for it?

# load all the data
day_ahead_market = pd.read_csv('datasets/timeseries/day_ahead_market_lbmp.csv.gz')
real_time_market = pd.read_csv('datasets/timeseries/real_time_market_lbmp.csv.gz')
weather_forecast = pd.read_csv('datasets/timeseries/weather_forecast.csv.gz')

day_ahead_market.head()

real_time_market.head()

# convert string date column to datetime type
day_ahead_market['Time Stamp'] = pd.to_datetime(day_ahead_market['Time Stamp'], format='%m/%d/%Y %H:%M')
real_time_market['Time Stamp'] = pd.to_datetime(real_time_market['Time Stamp'], format='%m/%d/%Y %H:%M:%S')

# convert dates to datetimes on the weather data
weather_forecast['Forecast Date'] = pd.to_datetime(weather_forecast['Forecast Date'], format='%m/%d/%Y')
weather_forecast['Vintage Date'] = pd.to_datetime(weather_forecast['Vintage Date'], format='%m/%d/%Y')
weather_forecast['Vintage'] = weather_forecast['Vintage'].astype('category')

weather_forecast.head()

# re-index the data by name of region and timestamp. This helps us manipulate and access the data.
dam_time_name = day_ahead_market.set_index(['Name', 'Time Stamp'])
rtm_time_name = real_time_market.set_index(['Name', 'Time Stamp'])

dam_time_name.head()

# select just the NYC data
dam_nyc_lbmp = dam_time_name['LBMP ($/MWHr)']['N.Y.C.']
rtm_nyc_lbmp = rtm_time_name['LBMP ($/MWHr)']['N.Y.C.']

plt.figure(figsize=(10,8))
dam_nyc_lbmp.plot(title='NYC Day Ahead LBMP 2015')

plt.figure(figsize=(10,8))
rtm_nyc_lbmp.plot(title='NYC Realtime LBMP 2015')

# Real Time Data Peek
rtm_nyc_lbmp.head()

# Day Ahead Data Peek
dam_nyc_lbmp.head()


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
How Did We Do?
Our data and model helps us predict the realtime electricity cost the day before. If you ran a company with significant power demands, you might be able to use such a model to decide whether or not to buy electricity in advance or on demand. Let's see how it would work!

Learn about Autoregressive Integrated Moving Average models
Apply it to a relevant dataset
????
PROFIT!!!
Can you write Python code for it?

# Profit Check
len(predicted_prices)
len(daily_df['rtm_lbmp'].values[10:])

print("--- Trading Log ---")

i = 251
PnL = np.zeros(100)

while i < 351:
    if (predicted_prices[i] < daily_df['dam_lbmp'].values[i]) and (daily_df['rtm_lbmp'].values[i+1] < daily_df['dam_lbmp'].values[i]):

        # if our model says the DAM is overpriced, then don't pre-buy and buy at the realtime price

        print("Buy RTM, +", daily_df['dam_lbmp'].values[i] - daily_df['rtm_lbmp'].values[i+1])
        PnL[i-251] = daily_df['dam_lbmp'].values[i] - daily_df['rtm_lbmp'].values[i+1]

    elif (predicted_prices[i] > daily_df['dam_lbmp'].values[i]) and (daily_df['rtm_lbmp'].values[i+1] > daily_df['dam_lbmp'].values[i]):

        # if our model says the DAM is underpriced, pre-buy the electricity so you don't have to pay realtime price

        print("Buy DAM, +", daily_df['rtm_lbmp'].values[i+1] - daily_df['dam_lbmp'].values[i] )
        PnL[i-251] = daily_df['rtm_lbmp'].values[i+1] - daily_df['dam_lbmp'].values[i]

    else:

        # if we were wrong, we lose money :(

        print("Lose $$, -", max(daily_df['rtm_lbmp'].values[i+1] - daily_df['dam_lbmp'].values[i],daily_df['dam_lbmp'].values[i] - daily_df['rtm_lbmp'].values[i+1]))
        PnL[i-251] = min(daily_df['rtm_lbmp'].values[i+1] - daily_df['dam_lbmp'].values[i],daily_df['dam_lbmp'].values[i] - daily_df['rtm_lbmp'].values[i+1])
    i = i+1

cumPnL = np.cumsum(PnL)
plt.figure(figsize=(10, 8))
plt.plot(cumPnL, label='PnL')
plt.legend()


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
Min and max temperature seem to be so highly correlated with each other that it most likely won't matter one way or another which we used.
Model Estimation
Now that our data is in the format we want, we can finally put into practice some of those theoretical concepts we discussed at the beginning. Our tool of choice here is a type of regression called ARIMA. It's a combination of 3 types of models -- AutoRegressive (a combination of the previous values), Integrated (differencing the data), Moving Average (smoothing the data). It's typically represented as:
arima(# of AR terms, # of differences, # of smoothing terms) = $ arima(p,d,q) $.
So how can we go about picking our parameters p, i, and q? There's several approaches of varying sophistication.  A common approach is to use the <a href="https://en.wikipedia.org/wiki/Box%E2%80%93Jenkins#Identify_p_and_q"> Box-Jenkins Method </a> for parameter selection. First, you pick parameter d (does the data need differencing to make it stationary?) You can do this using a Dickey-Fuller test. Then, you can use autocorrelation function (ACF) and partial autocorrelation function (PACF) to identify the AR parameter (p) and the MA parameter (q).
For our example, we'll just pick ARIMA(0,0,1) for it's simplicity (this is therefore just an MA(1) process). At the end we discuss these tools / the procedure in more detail.
Before fitting the model, we'll be splitting our data into two chunks -- one to train the model on and one to test it. This gives us a chance to see how the model would do in the "real world" on data it hasn't seen before.
Can you write Python code for it?

exog_data = np.array([daily_df['Actual Max Temp'].values, daily_df['dam_lbmp'].values])

# fit the model on a portion of the data
k = 250
m = ARIMA(daily_df['rtm_lbmp'].values[0:k], [0,0,1], exog=np.transpose(exog_data[:,0:k]), dates=daily_df.index.values[0:k])
results = m.fit(trend='nc', disp=True)

# what does our model predict?
predicted_prices = results.predict(10, 364, exog=np.transpose(exog_data), dynamic=True)

# let's see it graphically
plt.figure(figsize=(14, 10))
plt.plot(predicted_prices, label='prediction')
plt.plot(daily_df['rtm_lbmp'].values, label='actual RTM')
#plt.plot(daily_df['dam_lbmp'].values, label='actual DAM')

plt.legend()

results.summary()


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
Next step is to wrap this up nicely in a data frame. However, just constructing the data frame the default way will give us a very wide dataframe-- to make it into a tall one we'll need to transpose.
Can you write Python code for it?

# this dataframe is too wide
pd.DataFrame([no_dup_al_dam, no_dup_al_rtm])

# transpose dataframe to make it tall
dam_rtm_df = pd.DataFrame([no_dup_al_dam, no_dup_al_rtm]).transpose()
dam_rtm_df.head()


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
Now that we have our pricing data for the NYC region, we need to get the weather data ready. The weather data comes from a different data source, and unfortunately it's not split into the same exact regions as the pricing data. To remedy this, we'll pick two weather stations nearby -- the ones located at JFK airport and LGA airport -- and average the temperatures together.
Can you write Python code for it?

# a reminder about what the weather data looks like
weather_forecast.head()

# get every temperature from the LGA and JFK stations
lga_and_jfk_indexed = weather_forecast[(weather_forecast['Station ID'] == 'LGA') |
                                       (weather_forecast['Station ID'] == 'JFK')].set_index(['Forecast Date',
                                                                                'Vintage Date',
                                                                                'Vintage',
                                                                                'Station ID'])
lga_and_jfk_indexed.head()

# average together LGA and JFK temperatures (you'll notice that it's no longer in our data!)
mean_nyc_indexed = lga_and_jfk_indexed.mean(level=[0,1,2])
mean_nyc_indexed.head()


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
Now there's a slight problem--the timestamps on the realtime data and day ahead data don't actually line up. More specifically, the realtime data has observations every 5 minutes while the day ahead data has observations every hour. To fix this, we need to align the two series of data.
Can you write Python code for it?

aligned_dam, aligned_rtm = rtm_nyc_lbmp.align(dam_nyc_lbmp, join='inner')

# Remove Duplicates

no_dup_al_dam = aligned_dam[~aligned_dam.index.duplicated(keep='first')]
no_dup_al_rtm = aligned_rtm[~aligned_dam.index.duplicated(keep='first')]

no_dup_al_dam.name = 'dam_lbmp'
no_dup_al_rtm.name = 'rtm_lbmp'


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
The next question is whether to use the max temp or min temp--let's take a look at the data to see if one or the other makes a difference.
Can you write Python code for it?

plt.figure(figsize=(14,10))
plt.plot_date(daily_df.index, daily_df['rtm_lbmp'], '-', label='RTM LBMP')
plt.plot_date(daily_df.index, daily_df['dam_lbmp'], '-', label='DAM LBMP')
plt.plot_date(daily_df.index, daily_df['Actual Min Temp'], '-', label='Min Temp')
plt.plot_date(daily_df.index, daily_df['Actual Max Temp'], '-', label='Max Temp')
plt.legend()


I am working on the file "3. Timeseries Regression.ipynb".
The first task is:
We need to massage our data a bit more. First step is to unindex, which flattens our dataframe. Next, we pick out only the rows with the Vintage being 'Actual'. We've got a few different kinds of dates floating around and it's important to stay consistent. Our end goal is to have our data like:
Time stamp:        today
Temperature:       the actual temperature today (x)
Day ahead price:   today's price for electricity tomorrow (x)
Realtime price:    tomorrrow's electricity price observed tomorrow (y)
Can you write Python code for it?

mean_nyc = mean_nyc_indexed.reset_index()
actual_temp_df = mean_nyc[mean_nyc['Vintage'] == 'Actual'] \
    .groupby(['Vintage Date']).first() \
    .rename(columns=lambda x: 'Actual ' + x) # prepend the word Actual to column names

dam_rtm_act_df = dam_rtm_df.join(actual_temp_df, how='left').fillna(method='ffill').dropna()

dam_rtm_act_df.head()


I am working on the file "3. Гоблины, гули и призраки.ipynb".
The first task is:
Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
typelogfit = LogisticRegression()
scores = cross_val_score(typelogfit, X_tr, y_tr)
typelogfit.fit(X_tr, y_tr)
print('Best score:', scores.min())


I am working on the file "3. Гоблины, гули и призраки.ipynb".
The first task is:
Зайдите на http://www.kaggle.com и зарегистрируйтесь.
Далее вам нужен датасет по этой ссылке: https://www.kaggle.com/c/ghouls-goblins-and-ghosts-boo . И train, и test, и sample_submission.
Скачайте его и положите рядом с тетрадью, как вам удобнее.
Can you write Python code for it?

train, test = pd.read_csv(
    'data/HelloKaggle/train.csv'
    # путь к вашему файлу train
), pd.read_csv(
    'data/HelloKaggle/test.csv'
    # путь к вашему файлу test
)

train.head()

X = train.drop(['id', 'type'], axis=1)
y = train['type']


I am working on the file "3. Гоблины, гули и призраки.ipynb".
The first task is:
Используйте метод inverse_transform, чтобы превратить предсказания алгоритмов (числа) обратно в строки:
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression


I am working on the file "3. Гоблины, гули и призраки.ipynb".
The first task is:
Мы превратили все признаки в числовой вид, и с этим можно работать!
Визуализация
Can you write Python code for it?

sns.pairplot(train, hue='type');

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

clf = LogisticRegression()
cv_scores = cross_val_score(clf, X, y, cv=10)
sns.distplot(cv_scores);
plt.title('Average score: {}'.format(np.mean(cv_scores)));

sns.jointplot(x='bone_length', y='rotting_flesh', data=train);


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:
<b>Добавим новые признаки в test по-аналогии</b>
Can you write Python code for it?

#Сначала по-аналогии с визуализацией

#Заменим строки, где в SexuponOutcome, Breed, Color NaN
test.loc[test.SexuponOutcome.isnull(), 'SexuponOutcome'] = 'Unknown Unknown'
test.loc[test.AgeuponOutcome.isnull(), 'AgeuponOutcome'] = '0 0'
test.loc[test.Breed.isnull(), 'Breed'] = 'Unknown'
test.loc[test.Color.isnull(), 'Color'] = 'Unknown'

#Сделаем два отдельных столбца для пола и фертильности
test['Gender'] = test.SexuponOutcome.apply(lambda s: s.split(' ')[-1])
test['Fertility'] = test.SexuponOutcome.apply(lambda s: s.split(' ')[0])

#Теперь что-то новое

#Столбец, в котором отмечено, есть имя у животного или нет
test['hasName'] = 1
test.loc[test.Name.isnull(), 'hasName'] = 0

#Столбец, в котором объединены порода и цвет
test['breedColor'] = test.apply(lambda row: row['Breed'] + ' ' + str(row['Color']), axis=1)

#Декомпозируем DateTime
#Во-первых, конвертируем столбец в тип DateTime из строкового
test['DateTime'] = pd.to_datetime(test['DateTime'])
#А теперь декомпозируем
test['dayOfWeek'] = test.DateTime.apply(lambda dt: dt.dayofweek)
test['month'] = test.DateTime.apply(lambda dt: dt.month)
test['day'] = test.DateTime.apply(lambda dt: dt.day)
test['quarter'] = test.DateTime.apply(lambda dt: dt.quarter)
test['hour'] = test.DateTime.apply(lambda dt: dt.hour)
test['minute'] = test.DateTime.apply(lambda dt: dt.hour)
test['year'] = test.DateTime.apply(lambda dt: dt.year)

#Разбиение возраста
#Сделаем два отдельных столбца для обозначения года/месяца и их количества
test['AgeuponFirstPart'] = test.AgeuponOutcome.apply(lambda s: s.split(' ')[0])
test['AgeuponSecondPart'] = test.AgeuponOutcome.apply(lambda s: s.split(' ')[-1])
#Переведем примерно в среднем месяцы, годы и недели в дни с учетом окончаний s
test['AgeuponSecondPartInDays'] = 0
test.loc[test.AgeuponSecondPart == 'year', 'AgeuponSecondPartInDays'] = 365
test.loc[test.AgeuponSecondPart == 'years', 'AgeuponSecondPartInDays'] = 365
test.loc[test.AgeuponSecondPart == 'month', 'AgeuponSecondPartInDays'] = 30
test.loc[test.AgeuponSecondPart == 'months',...


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:
<b>Добавим новые признаки в train</b>
Can you write Python code for it?

#Сначала по-аналогии с визуализацией

#Заменим строки, где в SexuponOutcome, Breed, Color NaN
train.loc[train.SexuponOutcome.isnull(), 'SexuponOutcome'] = 'Unknown Unknown'
train.loc[train.AgeuponOutcome.isnull(), 'AgeuponOutcome'] = '0 0'
train.loc[train.Breed.isnull(), 'Breed'] = 'Unknown'
train.loc[train.Color.isnull(), 'Color'] = 'Unknown'

#Сделаем два отдельных столбца для пола и фертильности
train['Gender'] = train.SexuponOutcome.apply(lambda s: s.split(' ')[-1])
train['Fertility'] = train.SexuponOutcome.apply(lambda s: s.split(' ')[0])

#Теперь что-то новое

#Столбец, в котором отмечено, есть имя у животного или нет
train['hasName'] = 1
train.loc[train.Name.isnull(), 'hasName'] = 0

#Столбец, в котором объединены порода и цвет
train['breedColor'] = train.apply(lambda row: row['Breed'] + ' ' + str(row['Color']), axis=1)

#Декомпозируем DateTime
#Во-первых, конвертируем столбец в тип DateTime из строкового
train['DateTime'] = pd.to_datetime(train['DateTime'])
#А теперь декомпозируем
train['dayOfWeek'] = train.DateTime.apply(lambda dt: dt.dayofweek)
train['month'] = train.DateTime.apply(lambda dt: dt.month)
train['day'] = train.DateTime.apply(lambda dt: dt.day)
train['quarter'] = train.DateTime.apply(lambda dt: dt.quarter)
train['hour'] = train.DateTime.apply(lambda dt: dt.hour)
train['minute'] = train.DateTime.apply(lambda dt: dt.hour)
train['year'] = train.DateTime.apply(lambda dt: dt.year)

#Разбиение возраста
#Сделаем два отдельных столбца для обозначения года/месяца и их количества
train['AgeuponFirstPart'] = train.AgeuponOutcome.apply(lambda s: s.split(' ')[0])
train['AgeuponSecondPart'] = train.AgeuponOutcome.apply(lambda s: s.split(' ')[-1])
#Переведем примерно в среднем месяцы, годы и недели в дни с учетом окончаний s
train['AgeuponSecondPartInDays'] = 0
train.loc[train.AgeuponSecondPart == 'year', 'AgeuponSecondPartInDays'] = 365
train.loc[train.AgeuponSecondPart == 'years', 'AgeuponSecondPartInDays'] = 365
train.loc[train.AgeuponSecondPart == 'month', 'AgeuponSecondPartInDays'] =...


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:

Задание 3.

Выполните отбор признаков, попробуйте различные методы. Проверьте качество на кросс-валидации.
Выведите топ самых важных и самых незначащих признаков.
Предобработка данных
Can you write Python code for it?

np.random.seed = 1234
from sklearn.preprocessing import LabelEncoder
from sklearn import preprocessing

#####################Заменим NaN значения на слово Unknown##################
#Уберем Nan значения из train
train.loc[train.AnimalID.isnull(), 'AnimalID'] = 'Unknown'
train.loc[train.Name.isnull(), 'Name'] = 'Unknown'
train.loc[train.OutcomeType.isnull(), 'OutcomeType'] = 'Unknown'
train.loc[train.AnimalType.isnull(), 'AnimalType'] = 'Unknown'
train.loc[train.AgeuponOutcome.isnull(), 'AgeuponOutcome'] = 'Unknown'
train.loc[train.LifetimeInDays.isnull(), 'LifetimeInDays'] = 'Unknown'

#Уберем Nan значения из test
test.loc[test.ID.isnull(), 'ID'] = 'Unknown'
test.loc[test.Name.isnull(), 'Name'] = 'Unknown'
test.loc[test.AnimalType.isnull(), 'AnimalType'] = 'Unknown'
test.loc[test.AgeuponOutcome.isnull(), 'AgeuponOutcome'] = 'Unknown'
test.loc[test.LifetimeInDays.isnull(), 'LifetimeInDays'] = 'Unknown'

#####################Закодируем слова числами################################

#Закодировали AnimalID цифрами вместо названий в test & train
#encAnimalID = preprocessing.LabelEncoder()
#encAnimalID.fit(pd.concat((test['AnimalID'], train['AnimalID'])))
#test['AnimalID'] = encAnimalID.transform(test['AnimalID'])
#train['AnimalID'] = encAnimalID.transform(train['AnimalID'])

#Закодировали имя цифрами вместо названий в test & train
encName = preprocessing.LabelEncoder()
encName.fit(pd.concat((test['Name'], train['Name'])))
test['Name'] = encName.transform(test['Name'])
train['Name'] = encName.transform(train['Name'])

#Закодировали DateTime цифрами вместо названий в test & train
encDateTime = preprocessing.LabelEncoder()
encDateTime.fit(pd.concat((test['DateTime'], train['DateTime'])))
test['DateTime'] = encDateTime.transform(test['DateTime'])
train['DateTime'] = encDateTime.transform(train['DateTime'])

#Закодировали OutcomeType цифрами вместо названий в train, т.к. в test их нет
encOutcomeType = preprocessing.LabelEncoder()
encOutcomeType.fit(train['OutcomeType'])
train['OutcomeType'] =...


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:
Данные
Возьмите данные с https://www.kaggle.com/c/shelter-animal-outcomes .
Обратите внимание, что в этот раз у нас много классов, почитайте в разделе Evaluation то, как вычисляется итоговый счет (score).
Визуализация

Задание 1.

Выясните, построив необходимые графики, влияет ли возраст, пол или фертильность животного на его шансы быть взятыми из приюта.
Подготовим данные
Can you write Python code for it?

visual = pd.read_csv('data/CatsAndDogs/train.csv')

#Сделаем числовой столбец Outcome, показывающий, взяли животное из приюта или нет
#Сначала заполним единицами, типа во всех случах хорошо
visual['Outcome'] = 'true'
#Неудачные случаи занулим
visual.loc[visual.OutcomeType == 'Euthanasia', 'Outcome'] = 'false'
visual.loc[visual.OutcomeType == 'Died', 'Outcome'] = 'false'

#Заменим строки, где в SexuponOutcome NaN, на что-нибудь осмысленное
visual.loc[visual.SexuponOutcome.isnull(), 'SexuponOutcome'] = 'Unknown Unknown'

#Сделаем два отдельных столбца для пола и фертильности
visual['Gender'] = visual.SexuponOutcome.apply(lambda s: s.split(' ')[-1])
visual['Fertility'] = visual.SexuponOutcome.apply(lambda s: s.split(' ')[0])


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:
Методы обертки
Can you write Python code for it?

from sklearn.feature_selection import RFE
from sklearn.linear_model import LinearRegression

names = X_tr.columns.values
lr = LinearRegression()
rfe = RFE(lr, n_features_to_select=1)
rfe.fit(X_tr,y_tr);
print("Features sorted by their rank:")
print(sorted(zip(map(lambda x: round(x, 4), rfe.ranking_), names)))


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:
Отбор при помощи модели Lasso
Can you write Python code for it?

from sklearn.linear_model import Lasso
clf = Lasso()
clf.fit(X_tr, y_tr);
clf.coef_

features = X_tr.columns.values
print('Всего Lasso выкинуло %s переменных' % (clf.coef_ == 0).sum())
print('Это признаки:')
for s in features[np.where(clf.coef_ == 0)[0]]:
    print(' * ', s)


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:
Сравним по полу
Can you write Python code for it?

mergedByGender = visual.groupby('Gender')['Outcome'].value_counts().to_dict()

results = pd.DataFrame(data = mergedByGender, index=[0]).stack().fillna(0).transpose()
results.columns = pd.Index(['true', 'false'])
results['total'] = results.true + results.false
results.sort_values(by='true', ascending=False, inplace=True)
results[['true', 'false']].plot(kind='bar', stacked=True, rot=45);


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:
Сравним по фертильности
Can you write Python code for it?

mergedByFert = visual.groupby('Fertility')['Outcome'].value_counts().to_dict()

results = pd.DataFrame(data = mergedByFert, index=[0]).stack().fillna(0).transpose()
results.columns = pd.Index(['true', 'false'])
results['total'] = results.true + results.false
results.sort_values(by='true', ascending=False, inplace=True)
results[['true', 'false']].plot(kind='bar', stacked=True, rot=45);


I am working on the file "3. Котики и собачки (исходные данные).ipynb".
The first task is:
Статистические тесты
Can you write Python code for it?

from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2, f_classif, mutual_info_classif

skb = SelectKBest(mutual_info_classif, k=15)
x_new = skb.fit_transform(X_tr, y_tr)

x_new


I am working on the file "3. Котики и собачки.ipynb".
The first task is:
Вывод по признакам:

Не нужны: Name, DateTime, month, day, Breed, breedColor. Всё остальное менее однозначно, можно и оставить.

Задание 4.

Попробуйте смешать разные модели с помощью sklearn.ensemble.VotingClassifier. Увеличилась ли точность? Изменилась ли дисперсия?
Can you write Python code for it?

#Для начала выкинем ненужные признаки, выявленные на прошлом этапе
X_tr = X_tr.drop(['Name', 'DateTime', 'month', 'day', 'Breed', 'breedColor'], axis=1)
test = test.drop(['Name', 'DateTime', 'month', 'day', 'Breed', 'breedColor'], axis=1)
X_tr.head()

from sklearn.ensemble import VotingClassifier

from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier

clf1 = LogisticRegression(random_state=1234)
clf3 = GaussianNB()
clf5 = KNeighborsClassifier()

eclf = VotingClassifier(estimators=[
    ('lr', clf1), ('gnb', clf3), ('knn', clf5)],
                         voting='soft', weights=[1,1,10])

scores = cross_val_score(eclf, X_tr, y_tr)

eclf = eclf.fit(X_tr, y_tr)

print('Best score:', scores.min())

#delete AnimalID from test
X_te = test.drop(['AnimalID'], axis=1)
X_te.head()

y_te = eclf.predict(X_te)
y_te

ans_nn = pd.DataFrame({'AnimalID': test['AnimalID'], 'type': encOutcomeType.inverse_transform(y_te)})
ans_nn.head()

#Зададим функцию для трансформации
def onehot_encode(df_train, column):
    from sklearn.preprocessing import LabelBinarizer

    cs = df_train.select_dtypes(include=['O']).columns.values
    if column not in cs:
        return (df_train, None)

    rest = [x for x in df_train.columns.values if x != column]

    lb = LabelBinarizer()
    train_data = lb.fit_transform(df_train[column])

    new_col_names = ['%s' % x for x in lb.classes_]

    if len(new_col_names) != train_data.shape[1]:
        new_col_names = new_col_names[::-1][:train_data.shape[1]]

    new_train = pd.concat((df_train.drop([column], axis=1), pd.DataFrame(data=train_data, columns=new_col_names)), axis=1)
    return (new_train, lb)

ans_nn, lb = onehot_encode(ans_nn, 'type')

ans_nn
ans_nn.head()


I am working on the file "3. Котики и собачки.ipynb".
The first task is:
<b>Добавим новые признаки в test по-аналогии</b>
Can you write Python code for it?


I am working on the file "3. Котики и собачки.ipynb".
The first task is:
<b>Добавим новые признаки в train</b>
Can you write Python code for it?


I am working on the file "3. Котики и собачки.ipynb".
The first task is:

Задание 3.

Выполните отбор признаков, попробуйте различные методы. Проверьте качество на кросс-валидации.
Выведите топ самых важных и самых незначащих признаков.
Предобработка данных
Can you write Python code for it?

np.random.seed = 1234
from sklearn.preprocessing import LabelEncoder
from sklearn import preprocessing

#####################Заменим NaN значения на слово Unknown##################
#Уберем Nan значения из train
train.loc[train.AnimalID.isnull(), 'AnimalID'] = 'Unknown'
train.loc[train.Name.isnull(), 'Name'] = 'Unknown'
train.loc[train.OutcomeType.isnull(), 'OutcomeType'] = 'Unknown'
train.loc[train.AnimalType.isnull(), 'AnimalType'] = 'Unknown'
train.loc[train.AgeuponOutcome.isnull(), 'AgeuponOutcome'] = 'Unknown'
train.loc[train.LifetimeInDays.isnull(), 'LifetimeInDays'] = 'Unknown'

#Уберем Nan значения из test
test.loc[test.AnimalID.isnull(), 'AnimalID'] = 'Unknown'
test.loc[test.Name.isnull(), 'Name'] = 'Unknown'
test.loc[test.AnimalType.isnull(), 'AnimalType'] = 'Unknown'
test.loc[test.AgeuponOutcome.isnull(), 'AgeuponOutcome'] = 'Unknown'
test.loc[test.LifetimeInDays.isnull(), 'LifetimeInDays'] = 'Unknown'

#####################Закодируем слова числами################################

#Закодировали AnimalID цифрами вместо названий в test & train
#encAnimalID = preprocessing.LabelEncoder()
#encAnimalID.fit(pd.concat((test['AnimalID'], train['AnimalID'])))
#test['AnimalID'] = encAnimalID.transform(test['AnimalID'])
#train['AnimalID'] = encAnimalID.transform(train['AnimalID'])

#Закодировали имя цифрами вместо названий в test & train
encName = preprocessing.LabelEncoder()
encName.fit(pd.concat((test['Name'], train['Name'])))
test['Name'] = encName.transform(test['Name'])
train['Name'] = encName.transform(train['Name'])

#Закодировали DateTime цифрами вместо названий в test & train
encDateTime = preprocessing.LabelEncoder()
encDateTime.fit(pd.concat((test['DateTime'], train['DateTime'])))
test['DateTime'] = encDateTime.transform(test['DateTime'])
train['DateTime'] = encDateTime.transform(train['DateTime'])

#Закодировали OutcomeType цифрами вместо названий в train, т.к. в test их нет
encOutcomeType = preprocessing.LabelEncoder()
encOutcomeType.fit(train['OutcomeType'])
train['OutcomeType'] =...


I am working on the file "3. Котики и собачки.ipynb".
The first task is:
Методы обертки
Can you write Python code for it?


I am working on the file "3. Котики и собачки.ipynb".
The first task is:
Отбор при помощи модели Lasso
Can you write Python code for it?


I am working on the file "3. Котики и собачки.ipynb".
The first task is:
Отбор при помощи модели RandomForest
Can you write Python code for it?

from sklearn.ensemble import RandomForestRegressor
clf = RandomForestRegressor()
clf.fit(X_tr, y_tr);
clf.feature_importances_

imp_feature_idx = clf.feature_importances_.argsort()
imp_feature_idx

features = X_tr.columns.values

k = 0

while k < len(features):
    print(features[k], imp_feature_idx[k])
    k += 1


I am working on the file "3. Котики и собачки.ipynb".
The first task is:
Проверим, что никакие строчки при манипуляции с NaN не потерялись
Can you write Python code for it?

test.shape[0] == ans_nn.shape[0]

#Сделаем нумерацию индексов не с 0, а с 1
ans_nn.index += 1
#Воткнем столбец с индексами как столбец в конкретное место
ans_nn.insert(0, 'ID', ans_nn.index)
#delete AnimalID from test
ans_nn = ans_nn.drop(['AnimalID'], axis=1)
ans_nn.head()

#Сохраним
ans_nn.to_csv('ans_catdog.csv', index=False)


I am working on the file "3. Котики и собачки.ipynb".
The first task is:
Сравним по возрасту
Can you write Python code for it?

mergedByAges = visual.groupby('AgeuponOutcome')['Outcome'].value_counts().to_dict()

results = pd.DataFrame(data = mergedByAges, index=[0]).stack().fillna(0).transpose()
results.columns = pd.Index(['true', 'false'])
results['total'] = results.true + results.false
results.sort_values(by='true', ascending=False, inplace=True)
results[['true', 'false']].plot(kind='bar', stacked=False, rot=45);


I am working on the file "3.0 Basic Concepts.ipynb".
The first task is:
Example: Counter
Can you write Python code for it?

state = tf.Variable(0, name = "counter")
new_value = tf.add(state, tf.constant(1))
update = tf.assign(state,new_value)

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print(sess.run(state))
    for _ in range(3):
        sess.run(update)
        print(sess.run(state))


I am working on the file "3.0 Basic Concepts.ipynb".
The first task is:
Let's do the same with TensorFlow
Can you write Python code for it?

import tensorflow as tf

sess = tf.InteractiveSession()

a = tf.zeros((2,2)); b = tf.ones((2,2))

tf.reduce_sum(b, reduction_indices=1).eval()

a.get_shape()

tf.reshape(a, (1, 4)).eval()

a = np.zeros((2,2))
ta = tf.zeros((2,2))
print a
print ta


I am working on the file "3.0 Basic Concepts.ipynb".
The first task is:
Placeholders
TensorFlow provides a placeholder operation that must be fed with data on execution. It will be used to load input data to the model.
```python
x = tf.placeholder(tf.float32, shape=(2, 2))
y = tf.matmul(x, x)
with tf.Session() as sess:
  print(sess.run(y))  # ERROR: will fail because x was not fed.
rand_array = np.random.rand(2, 2) # we should get data from some training data
  print(sess.run(y, feed_dict={x: rand_array}))  # Will succeed.
```
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=(2, 2))
y = tf.matmul(x, x)

with tf.Session() as sess:
    rand_array = np.random.rand(2, 2) # we should get data from some training data
    print(sess.run(y, feed_dict={x: rand_array}))


I am working on the file "3.0 Basic Concepts.ipynb".
The first task is:
Session
A Session object encapsulates the environment in which Operation objects are executed, and Tensor objects are evaluated. A session may own resources, such as variables, queues, and readers. It is important to release these resources when they are no longer required.
Three basic ways to work:
1) Using the Session object:
python
a = tf.constant(5.0)
b = tf.constant(6.0)
c = a * b
sess = tf.Session()
print sess.run(c)
sess.close()
2) Using the context manager:
python
a = tf.constant(5.0)
b = tf.constant(6.0)
c = a * b
with tf.Session() as sess:
  print(c.eval())
3) Using Interactive Session:
```python
sess = tf.InteractiveSession()
a = tf.constant(5.0)
b = tf.constant(6.0)
c = a * b
We can just use 'c.eval()' without passing 'sess'
print(c.eval())
sess.close()
```
Variables
When you train a model, you use variables to hold and update parameters. Variables are in-memory buffers containing tensors. A tensorFlow variable does not exist until you initialize it, so they must be explicitly initialized and can be saved to disk during and after training. You can later restore saved values to exercise or analyse the model.
Can you write Python code for it?

W1 = tf.ones((2,2))
W2 = tf.Variable(tf.zeros((2,2)),name = "weights")

with tf.Session() as sess:
    print(sess.run(W1))
    sess.run(tf.initialize_all_variables())
    print(sess.run(W2))

W = tf.Variable(tf.zeros((2,2)),name = "weights")
R = tf.Variable(tf.random_normal((2,2)),name = "random_weights")

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print(sess.run(W))
    print(sess.run(R))


I am working on the file "3.0 Basic Concepts.ipynb".
The first task is:
TensorFlow computations define a computation graph that has no numerical value until it is evaluated!
Can you write Python code for it?


I am working on the file "3.0 Basic Concepts.ipynb".
The first task is:
What is TensorFlow? TensorFlow is a deep learning library recently open-sourced by Google. It is a programming system in which you represent computations as graphs. Nodes in the graph are called ops (short for operations). An op takes zero or more Tensors, performs some computation, and produces zero or more Tensors. TensorFlow provides primitives for defining functions on tensors and automatically computing their derivatives.
Formally, tensors are mulinear maps from vectors spaces to the real numbers. A Tensor can be represented as a multi-dimensional array of numbers. For example, you can represent a mini-batch of images as a 4-D array of floating point numbers with dimensions [batch, height, width, channels].
Every operation of machine learning can be writen as a function on tensors
Why TensorFlow?
TensorFlow is quite similar to Numpy, the most important difference is that numpy doesn't offer methods to create tensors as a functions and automatically compute derivatives (also, numpy does not provide GPU support)
TensorFlow Mechanics:
* Prepare the Data
    * Inputs and Placeholders
* Build the Graph
    * Inference
    * Loss
    * Training
* Train The model
    * The Session
    * The Graph
    * Train loop
* Evaluate the model
Simple Numpy Recap
Can you write Python code for it?

import numpy as np

a = np.zeros((2,2)); b = np.ones((2,2))
print np.sum(b,axis=1)

print a.shape

print np.reshape(a,(1,4))


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
dict.iteritems(): return an iterator through items (key, value) of a dictionary
dict.iterkeys(): return an iterator through key of a dictionary
dict.itervalues(): return an iterator through value of a dictionary
Can you write Python code for it?

print tel

print list(tel.iteritems())

for name, telno in tel.iteritems():
    print name, ':', telno

for key in tel.iterkeys():
    print key

import os

os.listdir('.')

[file_name for file_name in os.listdir('.') if file_name.endswith('.pyc')]

filter(lambda file_name: file_name.endswith('.pyc'), os.listdir('.'))

os.remove('./main.pyc')

[os.remove(file_name) for file_name in os.listdir('.') if file_name.endswith('.pyc')]


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
dict.keys(): return list of keys
dict.values(): return list of values
Can you write Python code for it?

tel = {'sape': 4139, 'jack': 4098, 'guido': 4127}

print tel.keys()
print tel.values()


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
list(): convert a sequence to a list
Can you write Python code for it?


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
zip(sequence...): zip sequences together
Can you write Python code for it?


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
Dictionaries

it's mapping type: key -> value
key can be any immutable type: usually string or number
Can you write Python code for it?

{'a', 'b'}

tel = {'jack': 4098, 'sape': 4139}
tel['guido'] = 4127

print tel

tel['vu'] = 4910

print tel

print tel['jack']


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
More on looping

enumerate(sequence, start=0)
Can you write Python code for it?

list(enumerate(['dog', 'cat', 'pig']))

print list(enumerate(['dog', 'cat', 'pig']))
print list(enumerate(['dog', 'cat', 'pig'], start=2))

for value in enumerate(['dog', 'cat', 'pig']):
    print value

for index, value in enumerate(['dog', 'cat', 'pig']):
    print index, ':', value


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
Sets

A set is an unordered collection with no duplicate elements
Curly brackets can use to create set, but {} create dictionary instead of set
set(): convert a sequence to a set
Can you write Python code for it?

letters = {'a', 'b', 'c', 'a'}
print letters

print set(['a', 'b', 'c', 'a'])

s = set(['a', 'b'])
s.add('c')

print s


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
Tuples

lists and strings are two examples of sequence data types.
tuple can think like list, but it is immutable.
Note: no tuple comprehensive
Can you write Python code for it?

t = (1, 2, 3, 4, 5)
print t

tuple([1,2,3])

# change the tuple raise exception
t[0] = 5


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
check if a key is in dictionary
Can you write Python code for it?


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
check item is in a set
Can you write Python code for it?

print 'fish' in pets
print 'lion' in pets


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
comprehensive syntax like list comprehensive
Can you write Python code for it?

for i in (x ** 2 for x in range(10)):
    print i,


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
delete item
Can you write Python code for it?


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
delete key in dictionary
Can you write Python code for it?


I am working on the file "3.0_Data_structures.ipynb".
The first task is:
similarly to list comprehensions, set comprehensions are also supported
Can you write Python code for it?

letters = {x for x in 'i can eat glass'}
print letters


I am working on the file "3.1.3+KNN+RegressionWithJackiesModel.ipynb".
The first task is:
KNN Regression
So far we've introduced KNN as a classifier, meaning it assigns observations to categories or assigns probabilities to the various categories. However, KNN is also a reasonable algorithm for regression. It's a simple extension of what we've learned before and just as easy to implement.
Everything's the Same
Switching KNN to a regrssion is a simple process. In our previous models, each of the $k$ oberservations voted for a category. As a regression they vote instead for a value. Then instead of taking the most popular response, the algorithm averages all of the votes. If you have weights you perform a weighted average.
It's really that simple.
Let's go over a quick example just to confirm your understanding.
Let's stick with the world of music. Instead of trying to classify songs as rock or jazz, lets take the same data with an additional column: beats per minute, or BPM. Can we train our model to predict BPM?
First let's try to predict just in terms of loudness, as this will be easier to represent graphically.
Can you write Python code for it?

from sklearn import neighbors

# Build our model.
knn = neighbors.KNeighborsRegressor(n_neighbors=10)
X = pd.DataFrame(music.loudness)
Y = music.bpm
knn.fit(X, Y)

# Set up our prediction line.
T = np.arange(0, 50, 0.1)[:, np.newaxis]

# Trailing underscores are a common convention for a prediction.
Y_ = knn.predict(T)

plt.scatter(X, Y, c='k', label='data')
plt.plot(T, Y_, c='g', label='prediction')
plt.legend()
plt.title('K=10, Unweighted')
plt.show()

# Run the same model, this time with weights.
knn_w = neighbors.KNeighborsRegressor(n_neighbors=10, weights='distance')
X = pd.DataFrame(music.loudness)
Y = music.bpm
knn_w.fit(X, Y)

# Set up our prediction line.
T = np.arange(0, 50, 0.1)[:, np.newaxis]

Y_ = knn_w.predict(T)

plt.scatter(X, Y, c='k', label='data')
plt.plot(T, Y_, c='g', label='prediction')
plt.legend()
plt.title('K=10, Weighted')
plt.show()


I am working on the file "3.1.Aharonov-Bohm-cheat.ipynb".
The first task is:
We see that the Aharonov-Bohm effect contains several harmonics
$$ g = g_0 + g_1 cos(\phi) + g_2 cos(2\phi) + ...$$
* How can we get just one harmonics (as in most experiments)?
* Try L = 100 and W= 12, what do you see?
* The results should not depend on the position of the gauge transform, can you check that?
* Can you make a different sample with an assymetric shape, say a rectangular shape?
Real magnetic field
In real life it is difficult to put the field only in the hole so experimentalists tend to
apply a uniform field everywhere. Let us modify the script to just do that. You need now to pick up a
phase $\phi$ on each small square. This can be done with the following piece of code - try to modify your code to incorporate this new pieces.
Can you write Python code for it?

%matplotlib inline
import numpy as np
from math import pi
from matplotlib import pyplot
import kwant

lat=kwant.lattice.square()
L,W=100,12

def myshape(R): return (
        (R[0]**2 + R[1]**2) > (L-W/2)**2 and
        (R[0]**2 + R[1]**2) < (L+W/2)**2)
H=kwant.Builder()
H[lat.shape(myshape,(L,0) )]=4

def Field(site1,site2,phi):
    x1,y1=site1.pos
    x2,y2=site2.pos
    return -np.exp(-0.5j * phi * (x1 - x2) * (y1 + y2))

H[lat.neighbors()] = Field

sym=kwant.TranslationalSymmetry(lat.vec((1,0)))
def lead_shape(R): return abs(R[1]) < W/2 and abs(R[0]) <3
Hlead =kwant.Builder(sym)
Hlead[lat.shape(lead_shape,(0,0) )]=4
Hlead[lat.neighbors()]=Field
H.attach_lead(Hlead)
H.attach_lead(Hlead.reversed())
kwant.plot(H);

Hf=H.finalized()
data = []
phis = np.linspace(0.,0.0005,50)
for phi in phis:
    smatrix = kwant.smatrix(Hf, 3.3,args=[phi])
    data.append(smatrix.transmission(1, 0))
pyplot.plot(phis, data);
pyplot.xlabel('$\phi = Ba^2/(h/e)$')
pyplot.ylabel('g in unit of $(2e^2/h)$');
pyplot.title('Aharonov-Effect')


I am working on the file "3.1.Aharonov-Bohm.ipynb".
The first task is:
Now run it, don't forget to change the x-scale of the plot.

Do you understand why the x - scale is so much smaller?
Do you happen to know what will happen at higher field?
Can you write Python code for it?


I am working on the file "3.2 RNN and LSTM.ipynb".
The first task is:
Model building
Can you write Python code for it?

print('Build model...')
model = Sequential()
model.add(Embedding(max_features, 128, input_length=maxlen))
model.add(SimpleRNN(128))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy', optimizer='adam')

print("Train...")
model.fit(X_train, y_train, batch_size=batch_size, epochs=1,
          validation_data=(X_test, y_test))


I am working on the file "3.2.Quantum-Hall-effect-and-disorder.ipynb".
The first task is:
Formation of edge channels
Let's first show how the Landau levels are formed by cranking up the magnetic field. We will plot the sum of all the wave function probabilities of all
Can you write Python code for it?

from ipywidgets import interact

U0 = 0.3
salt = 13
energy = 0.2

def plot_wf(phi=0):
    params = SimpleNamespace(phi=phi, U0=U0, salt=salt)
    wfs = kwant.wave_function(sys, energy=energy, args=[params])
    scattering_wf = wfs(0)  # all scattering wave functions from lead 0
    kwant.plotter.map(sys, np.sum(abs(scattering_wf)**2, axis=0));

interact(plot_wf, phi=(0, 0.15, 0.002))


I am working on the file "3.2.Quantum-Hall-effect-and-disorder.ipynb".
The first task is:
One can clearly see how the wave function is very unstructured without a magnetic field. With $B$, the wave functon is forced towards one side of the wire!
If you look very closely, you however see that this does not happen uniformily. There are
always values of phi where the wave function starts to spread again into the whole system, although it was already localized to an edge!
The reason for this "spreading" is that for these magnetic field values we have Landau level close to our energy. The finite disorder broadens the Landau level, giving rise to a bulk conductance.
We can observe this clearly if we correlate the band structure with the wave function plots:
Can you write Python code for it?

U0 = 0.3
salt = 13
energy = 0.2

def plot_wf(phi=0):
    params = SimpleNamespace(phi=phi, U0=U0, salt=salt)
    wfs = kwant.wave_function(sys, energy=energy, args=[params])
    scattering_wf = wfs(0)  # all scattering wave functions from lead 0

    fig = plt.Figure(figsize=(15,7))
    ax1 = fig.add_subplot(121)
    kwant.plotter.bands(sys.leads[0], args=[params], ax=ax1)
    ax1.set_ylim(0, 0.3)
    ax1.set_xlim(-3, 3)
    ax1.plot([-4,4], [energy]*2, "k--")
    ax2 = fig.add_subplot(122)
    kwant.plotter.map(sys, np.sum(abs(scattering_wf)**2, axis=0), ax=ax2)
    kwant.plotter.output_fig(fig)

interact(plot_wf, phi=(0, 0.15, 0.002))


I am working on the file "3.2.Quantum-Hall-effect-and-disorder.ipynb".
The first task is:
Particular about this example is that we have now both the onsite energies and the hoppings position-dependent. Hence, we need to define functions for both:
Can you write Python code for it?

lat = kwant.lattice.square(a=1)

t = 1
W = 30
L = 50

def onsite(site, params):
    return  params.U0 * (uniform(repr(site), repr(params.salt)) - 0.5) + 4 * t

def hopping(site_i, site_j, params):
    xi, yi = site_i.pos
    xj, yj = site_j.pos
    return -exp(-0.5j * params.phi * (xi - xj) * (yi + yj))

sys = kwant.Builder()
sys[(lat(x, y) for x in range(L) for y in range(W))] = onsite
sys[lat.neighbors()] = hopping

lead = kwant.Builder(kwant.TranslationalSymmetry([-1, 0]))
lead[(lat(0, y) for y in range(W))] = 4 * t   # no disorder in lead!
lead[lat.neighbors()] = hopping

sys.attach_lead(lead)
sys.attach_lead(lead.reversed())

sys = sys.finalized()

kwant.plot(sys);


I am working on the file "3.3.MagnetoResistance.ipynb".
The first task is:
Magnetic texture : the example of a skyrmion
Last, we can start playing with the magnetic texture, for instance a skyrmion as in the example below.
$$H = - t \sum_{<ij>} |i><j| + \sum_i V_i \ \ |i><i|$$
$$V_i = J \ m ( r) \cdot  \sigma $$
$$ m ( r) = \left(x/r \sin\theta \ , \ y/r \sin\theta \ ,\  \cos\theta \right) $$
$$\theta (r) = \tanh \frac{r-r_0}{\delta}$$
Another difference is that we will have 4 terminals and calculate the Hall resistance instead of the 2 terminal conductance.
This amounts to imposing the current and measuring the voltage, i.e. solving a small linear problem which is readily done with numpy.
Can you calculate the longitudinal resistance?
Can you write Python code for it?

def HedgeHog(site,ps):
        x,y = site.pos
        r = ( x**2 + y**2 )**0.5
        theta = (np.pi/2)*(np.tanh((ps.r0 - r)/ps.delta) + 1)
        if r != 0:
            Ex = (x/r)*np.sin(theta)*s_x + (y/r)*np.sin(theta)*s_y + np.cos(theta)*s_z
        else:
            Ex = s_z
        return 4*s_0 + ps.Ex * Ex

def Lead_Pot(site,ps):
    return  4*s_0 +  ps.Ex * s_z

def MakeSystem(ps, show = False):
    H  = kwant.Builder()

    def shape_2DEG(pos):
        x,y = pos
        return  ( (abs(x) < ps.L) and (abs(y) < ps.W) ) or (
            (abs(x) < ps.W) and (abs(y) < ps.L))

    H[lat.shape(shape_2DEG,(0,0))] = HedgeHog
    H[lat.neighbors()] = -s_0

    # ITS LEADS
    sym_x = kwant.TranslationalSymmetry((-1,0))
    H_lead_x = kwant.Builder(sym_x)
    shape_x = lambda pos: abs(pos[1])<ps.W and pos[0]==0
    H_lead_x[lat.shape(shape_x,(0,0))] = Lead_Pot
    H_lead_x[lat.neighbors()] = -s_0

    sym_y = kwant.TranslationalSymmetry((0,-1))
    H_lead_y = kwant.Builder(sym_y)
    shape_y = lambda pos: abs(pos[0])<ps.W and pos[1]==0
    H_lead_y[lat.shape(shape_y,(0,0))] = Lead_Pot
    H_lead_y[lat.neighbors()] = -s_0

    H.attach_lead(H_lead_x)
    H.attach_lead(H_lead_y)
    H.attach_lead(H_lead_y.reversed())
    H.attach_lead(H_lead_x.reversed())

    if show:
        kwant.plot(H)

    return H

def Transport(Hf,EE,ps):
    smatrix = kwant.smatrix(Hf, energy=EE, args=[ps])
    G=np.zeros((4,4))
    for i in range(4):
        a=0
        for j in range(4):
            G[i,j] = smatrix.transmission(i, j)
            if i != j:
                a += G[i,j]
        G[i,i] = -a

    V = np.linalg.solve(G[:3,:3], [1.,0,0])
    Hall = V[2] - V[1]

    return G, Hall

ps = SimpleNamespace(L=45, W=40, delta=10, r0=20, Ex=1.)

H = MakeSystem(ps, show=True)
Hf = H.finalized()

def Vz(site):
    Hd = HedgeHog(site,ps)
    return (Hd[0,0] - Hd[1,1]).real

def Vy(site):
    Hd = HedgeHog(site, ps)
    return Hd[0,1].imag

kwant.plotter.map(H, Vz);
kwant.plotter.map(H,...


I am working on the file "3.4.graphene_qshe-cheat.ipynb".
The first task is:
Let's also create 1D ribbons of graphene.
There are two nontrivial directions: armchair and zigzag
Can you write Python code for it?

zigzag_ribbon = kwant.Builder(kwant.TranslationalSymmetry([1, 0]))
zigzag_ribbon[graphene.shape((lambda pos: abs(pos[1]) < 9), (0, 0))] = 0
zigzag_ribbon[graphene.neighbors(1)] = 1

kwant.plotter.bands(zigzag_ribbon.finalized());


I am working on the file "3.4.graphene_qshe-cheat.ipynb".
The first task is:
Now we see that the gap closes in one of the Dirac cones, and does not close in the other half. Let's see what this means for the dispersion relation in a ribbon.
Your turn!
Plot a dispersion of either nanoribbon, and see what happens to the edge states
Can you write Python code for it?

# Solution

def nnn_hopping(site1, site2, params):
    return 1j * params.t_2

def onsite(site, params):
    return params.m * (1 if site.family == a else -1)

def add_hoppings(syst):
    syst[graphene.neighbors(1)] = 1
    syst[[kwant.builder.HoppingKind(*hopping) for hopping in nnn_hoppings]] = nnn_hopping

zigzag_haldane = kwant.Builder(kwant.TranslationalSymmetry([1, 0]))
zigzag_haldane[graphene.shape((lambda pos: abs(pos[1]) < 9), (0, 0))] = onsite
zigzag_haldane[graphene.neighbors(1)] = 1
zigzag_haldane[[kwant.builder.HoppingKind(*hopping) for hopping in nnn_hoppings]] = nnn_hopping

@interact(t_2=(0, .08, .01))
def qshe_dispersion(t_2=0, m=.2):
    kwant.plotter.bands(zigzag_haldane.finalized(), [SimpleNamespace(t_2=t_2, m=m)])


I am working on the file "3.4.graphene_qshe-cheat.ipynb".
The first task is:
Robustness of quantum spin Hall effect
We have an open important question: what protects these new edge states? Is it spin conservation? If we don't break the conservation of $\sigma_z$, then it is obvious that the edge states don't disappear (we have two copies of Haldane model after all).
The most interesting property of the quantum spin Hall effect is that it does not rely on any conservation law. The reason for this is Kramers degeneracy, that prevents two states that are time-reversal partners of each other from coupling to each other.
As a final test of topological protection, let's add an extra parameter that breaks spin conservation and check what happens to the edge states.
Your turn!
Add a small perturbation proportional to $i \sigma_x$ or $i \sigma_y$ to some hopping. Check how the dispersion changes.
Can you write Python code for it?

# Solution

zigzag_kane_mele = kwant.Builder(kwant.TranslationalSymmetry([1, 0]))
zigzag_kane_mele[graphene.shape((lambda pos: abs(pos[1]) < 9), (0, 0))] = onsite
zigzag_kane_mele[graphene.neighbors(1)] = s0
zigzag_kane_mele[kwant.builder.HoppingKind((0, 0), b, a)] = s0 + 0.3j * sx
zigzag_kane_mele[[kwant.builder.HoppingKind(*hopping) for hopping in nnn_hoppings]] = spin_orbit

@interact(t_2=(0, .12, .01))
def qshe_zigzag_dispersion(t_2=0, m=.2):
    kwant.plotter.bands(zigzag_kane_mele.finalized(), [SimpleNamespace(t_2=t_2, m=m)])


I am working on the file "3.4.graphene_qshe-cheat.ipynb".
The first task is:
We have now opened a gap, but there are no protected states inside it.
Haldane model of anomalous quantum Hall effect
The more interesting way to open the gap in graphene dispersion is introduced by Duncan Haldane, Phys. Rev. Lett. 61, 2015 (1988)
The idea of this model is to break inversion symmetry that protects the Dirac points by adding next-nearest neighbor hoppings
Can you write Python code for it?

nnn_hoppings_a = (((-1, 0), a, a), ((0, 1), a, a), ((1, -1), a, a))
nnn_hoppings_b = (((1, 0), b, b), ((0, -1), b, b), ((-1, 1), b, b))
nnn_hoppings = nnn_hoppings_a + nnn_hoppings_b

def nnn_hopping(site1, site2, params):
    return 1j * params.t_2

def onsite(site, params):
    return params.m * (1 if site.family == a else -1)

def add_hoppings(syst):
    syst[graphene.neighbors(1)] = 1
    syst[[kwant.builder.HoppingKind(*hopping) for hopping in nnn_hoppings]] = nnn_hopping

haldane = kwant.Builder(kwant.TranslationalSymmetry(*graphene.prim_vecs))
haldane[graphene.shape((lambda pos: True), (0, 0))] = onsite
haldane[graphene.neighbors(1)] = 1
haldane[[kwant.builder.HoppingKind(*hopping) for hopping in nnn_hoppings]] = nnn_hopping

@interact(t_2=(0, .08, .01))
def qshe_dispersion(t_2=0, m=.2):
    dispersion_2D(wraparound(haldane).finalized(), [SimpleNamespace(t_2=t_2, m=m)], num_points=100)


I am working on the file "3.4.graphene_qshe-cheat.ipynb".
The first task is:
We now create a Builder with the translational symmetries of graphene, and calculate the bulk dispersion of graphene.
Hence, the wave function in a unit cell can be written as a vector $(\Psi_A, \Psi_B)^T$ of amplitudes on the two sites $A$ and $B$. Taking a simple tight-binding model where electrons can hop between neighboring sites with hopping strength $t$, one obtains the Bloch Hamiltonian:
$$
H_0(\mathbf{k})= \begin{pmatrix} 0 & h(\mathbf{k}) \ h^\dagger(\mathbf{k}) & 0 \end{pmatrix}\,,
$$
with $\mathbf{k}=(k_x, k_y)$ and
$$h(\mathbf{k}) = t_1\,\sum_i\,\exp\,\left(i\,\mathbf{k}\cdot\mathbf{a}_i\right)\,.$$
Here $\mathbf{a}_i$ are the three vectors in the figure, connecting nearest neighbors of the lattice [we set the lattice spacing to one, so that for instance $\mathbf{a}_1=(1,0)$].  Introducing a set of Pauli matrices $\sigma$ which act on the sublattice degree of freedom, we can write the Hamiltonian in a compact form as
$$H_0(\mathbf{k}) = t_1\,\sum_i\,\left[\sigma_x\,\cos(\mathbf{k}\cdot\mathbf{a}_i)-\sigma_y \,\sin(\mathbf{k}\cdot\mathbf{a}_i)\right]\,.$$
The energy spectrum $E(\mathbf{k}) = \pm \,\left|h(\mathbf{k})\right|$ gives rise to the famous band structure of graphene, with the two bands touching at the six corners of the Brillouin zone:
Can you write Python code for it?

bulk_graphene = kwant.Builder(kwant.TranslationalSymmetry(*graphene.prim_vecs))
bulk_graphene[graphene.shape((lambda pos: True), (0, 0))] = 0
bulk_graphene[graphene.neighbors(1)] = 1

dispersion_2D(wraparound(bulk_graphene).finalized())


I am working on the file "3.4.graphene_qshe-cheat.ipynb".
The first task is:
Your turn!
Add potentials of opposite sign to the zigzag nanoribbon, and see what happens to the dispersion relation.
Can you write Python code for it?

# Solution

zigzag_ribbon = kwant.Builder(kwant.TranslationalSymmetry([1, 0]))
zigzag_ribbon[a.shape((lambda pos: abs(pos[1]) < 9), (0, 0))] = 0.2
zigzag_ribbon[b.shape((lambda pos: abs(pos[1]) < 9), (0, 0))] = -0.2
zigzag_ribbon[graphene.neighbors(1)] = 1

kwant.plotter.bands(zigzag_ribbon.finalized(), fig_size=(12, 8));


I am working on the file "3.4.graphene_qshe-cheat.ipynb".
The first task is:
Your turn!
Calculate a dispersion of an armchair nanoribbon. You'll need to figure out what is its period.
Can you write Python code for it?

# Solution

armchair_ribbon = kwant.Builder(kwant.TranslationalSymmetry([0, np.sqrt(3)]))
armchair_ribbon[graphene.shape((lambda pos: abs(pos[0]) < 9), (0, 0))] = 0
armchair_ribbon[graphene.neighbors(1)] = 1

kwant.plotter.bands(armchair_ribbon.finalized(), fig_size=(12, 8));


I am working on the file "3.4.graphene_qshe.ipynb".
The first task is:
Let's also create 1D ribbons of graphene.
There are two nontrivial directions: armchair and zigzag
Can you write Python code for it?


I am working on the file "3.4.graphene_qshe.ipynb".
The first task is:
We now create a Builder with the translational symmetries of graphene, and calculate the bulk dispersion of graphene.
Hence, the wave function in a unit cell can be written as a vector $(\Psi_A, \Psi_B)^T$ of amplitudes on the two sites $A$ and $B$. Taking a simple tight-binding model where electrons can hop between neighboring sites with hopping strength $t$, one obtains the Bloch Hamiltonian:
$$
H_0(\mathbf{k})= \begin{pmatrix} 0 & h(\mathbf{k}) \ h^\dagger(\mathbf{k}) & 0 \end{pmatrix}\,,
$$
with $\mathbf{k}=(k_x, k_y)$ and
$$h(\mathbf{k}) = t_1\,\sum_i\,\exp\,\left(i\,\mathbf{k}\cdot\mathbf{a}_i\right)\,.$$
Here $\mathbf{a}_i$ are the three vectors in the figure, connecting nearest neighbors of the lattice [we set the lattice spacing to one, so that for instance $\mathbf{a}_1=(1,0)$].  Introducing a set of Pauli matrices $\sigma$ which act on the sublattice degree of freedom, we can write the Hamiltonian in a compact form as
$$H_0(\mathbf{k}) = t_1\,\sum_i\,\left[\sigma_x\,\cos(\mathbf{k}\cdot\mathbf{a}_i)-\sigma_y \,\sin(\mathbf{k}\cdot\mathbf{a}_i)\right]\,.$$
The energy spectrum $E(\mathbf{k}) = \pm \,\left|h(\mathbf{k})\right|$ gives rise to the famous band structure of graphene, with the two bands touching at the six corners of the Brillouin zone:
Can you write Python code for it?


I am working on the file "3.4.graphene_qshe.ipynb".
The first task is:
Your turn!
Calculate a dispersion of an armchair nanoribbon. You'll need to figure out what is its period.
Your turn!
Add potentials of opposite sign to the zigzag nanoribbon, and see what happens to the dispersion relation.
We have now opened a gap, but there are no protected states inside it.
Haldane model of anomalous quantum Hall effect
The more interesting way to open the gap in graphene dispersion is introduced by Duncan Haldane, Phys. Rev. Lett. 61, 2015 (1988)
The idea of this model is to break inversion symmetry that protects the Dirac points by adding next-nearest neighbor hoppings
Can you write Python code for it?


I am working on the file "3.5 - List exercises.ipynb".
The first task is:
B. remove_adjacent
Given a list of numbers, return a list where
all adjacent == elements have been reduced to a single element,
so [1, 2, 2, 3] returns [1, 2, 3]. You may create a new list or
modify the passed in list.
Can you write Python code for it?

def remove_adjacent(nums):
  # +++your code here+++
  return

test(remove_adjacent([1, 2, 2, 3]), [1, 2, 3])
test(remove_adjacent([2, 2, 3, 3, 3]), [2, 3])
test(remove_adjacent([]), [])


I am working on the file "3.5 - List exercises.ipynb".
The first task is:
Fill in the code for the functions below. main() is already set up
to call the functions with a few different inputs,
printing 'OK' when each function is correct.
The starter code for each function includes a 'return'
which is just a placeholder for your code.
A. match_ends
Given a list of strings, return the count of the number of
strings where the string length is 2 or more and the first
and last chars of the string are the same.
Note: python does not have a ++ operator, but += works.
Can you write Python code for it?

def match_ends(words):
  # +++your code here+++
  return

test(match_ends(['aba', 'xyz', 'aa', 'x', 'bbb']), 3)
test(match_ends(['', 'x', 'xy', 'xyx', 'xx']), 2)
test(match_ends(['aaa', 'be', 'abc', 'hello']), 1)


I am working on the file "3.AnalysisArticle_HTML-Anwser.ipynb".
The first task is:
Parse Json
Can you write Python code for it?

def parseRaw(json_map):
    url = json_map['url']
    content = json_map['html']
    return (url,content)


I am working on the file "3.AnalysisArticle_HTML-Anwser.ipynb".
The first task is:
取出 Image Src 的列表
Can you write Python code for it?

image_list = all_content.map(lambda x :parseImgSrc(x[1]))
pp.pprint(image_list.first()[:10])


I am working on the file "3.AnalysisArticle_HTML-Anwser.ipynb".
The first task is:
統計 Image Src 的列表
Can you write Python code for it?

img_src_count = all_content.map(
    lambda x :parseImgSrc(x[1])).flatMap(
    lambda x: x).countByValue()
for i in img_src_count:
    print i , ':' , img_src_count[i]


I am working on the file "3.AnalysisArticle_HTML-Anwser.ipynb".
The first task is:
請使用　reduceByKey , sortBy 來計算出 img src 排行榜
請參照以下文件
[http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD]
Can you write Python code for it?

from operator import add
all_content.map(
    lambda x :parseImgSrc(x[1])).flatMap(
    lambda x: x).map(
    lambda x: (x,1)).reduceByKey(add).sortBy(lambda x:x[1] ,ascending =False).collect()


I am working on the file "3.AnalysisArticle_HTML.ipynb".
The first task is:
取出 Image Src 的列表
Can you write Python code for it?


I am working on the file "3.AnalysisArticle_HTML.ipynb".
The first task is:
統計 Image Src 的列表
Can you write Python code for it?


I am working on the file "3.AnalysisArticle_HTML.ipynb".
The first task is:
載入原始 RAW Data
Can you write Python code for it?

import json
import pprint
pp = pprint.PrettyPrinter(indent=2)
path = "./pixnet.txt"
all_content = sc.textFile(path).map(json.loads).map(parseRaw)


I am working on the file "3.KNN_Classifiers.ipynb".
The first task is:
K Nearest Neighbors Classifiers
So far we've covered learning via probability (naive Bayes) and learning via errors (regression). Here we'll cover learning via similarity. This means we look for the datapoints that are most similar to the observation we are trying to predict.
Let's start by the simplest example: Nearest Neighbor.
Nearest Neighbor
Let's use this example: classifying a song as either "rock" or "jazz". For this data we have measures of duration in seconds and loudness in loudness units (we're not going to be using decibels since that isn't a linear measure, which would create some problems we'll get into later).
Can you write Python code for it?


I am working on the file "3.KNN_Classifiers.ipynb".
The first task is:
Now our test prediction has changed. In using the five nearest neighbors it appears that there were two votes for rock and three for jazz, so it was classified as a jazz song. This is different than our simpler Nearest Neighbors model. While the closest observation was in fact rock, there are more jazz songs in the nearest $k$ neighbors than rock.
We can visualize our decision bounds with something called a mesh. This allows us to generate a prediction over the whole space. Read the code below and make sure you can pull out what the individual lines do, consulting the documentation for unfamiliar methods if necessary.
Can you write Python code for it?

# Our data. Converting from data frames to arrays for the mesh.
X = np.array(X)
Y = np.array(Y)

# Mesh size.
h = 4.0

# Plot the decision boundary. We asign a color to each point in the mesh.
x_min = X[:, 0].min() - .5
x_max = X[:, 0].max() + .5
y_min = X[:, 1].min() - .5
y_max = X[:, 1].max() + .5
xx, yy = np.meshgrid(
    np.arange(x_min, x_max, h),
    np.arange(y_min, y_max, h)
)
Z = neighbors.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot.
Z = Z.reshape(xx.shape)
plt.figure(1, figsize=(6, 4))
plt.set_cmap(plt.cm.Paired)
plt.pcolormesh(xx, yy, Z)

# Add the training points to the plot.
plt.scatter(X[:, 0], X[:, 1], c=Y)
plt.xlabel('Loudness')
plt.ylabel('Duration')
plt.title('Mesh visualization')

plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())

plt.show()


I am working on the file "3.Machine_learning/1.msc_mayor_app/msc_app_to_mayor.ipynb".
The first task is:
Добавим другие данные из сети (Add other data from the network)
Can you write Python code for it?

""" To remind
district_columns = ['to_10K_total_VAO', 'to_10K_total_ZAO', 'to_10K_total_ZelAO',
        'to_10K_total_SAO','to_10K_total_SVAO','to_10K_total_SZAO','to_10K_total_TiNAO','to_10K_total_CAO',
        'to_10K_total_YUAO','to_10K_total_YUVAO','to_10K_total_YUZAO']
"""
# we will collect the data manually from
# https://ru.wikipedia.org/wiki/%D0%90%D0%B4%D0%BC%D0%B8%D0%BD%D0%B8%D1%81%D1%82%D1%80%D0%B0%D1%82%D0%B8%D0%B2%D0%BD%D0%BE-%D1%82%D0%B5%D1%80%D1%80%D0%B8%D1%82%D0%BE%D1%80%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D0%BE%D0%B5_%D0%B4%D0%B5%D0%BB%D0%B5%D0%BD%D0%B8%D0%B5_%D0%9C%D0%BE%D1%81%D0%BA%D0%B2%D1%8B

#the data is filled in the same order as the district_columns
district_population=[1507198,1368731,239861,1160576,1415283,990696,339231,769630,1776789,1385385,1427284]

#transition from 1/10000 to citizens' appeal to the entire population of the district
total_appel_dep=district_population*distr_sum/10000

plt.figure(figsize=(16,9))
plt.bar(y_pos, total_appel_dep, align='center', alpha=0.5)
plt.xticks(y_pos, short_district_columns)
plt.ylabel('Number of appeals')
plt.title('Number of appeals per total pщulation of district for all time')

savefig('3.png')


I am working on the file "3.Machine_learning/1.msc_mayor_app/msc_app_to_mayor.ipynb".
The first task is:
Линейная регрессия (Linear regression)
Произведём некоторые манипуляции с исходными данными для того чтобы строить модель линейной регрессии в последующих ячейках

Let's make some manipulations with the initial data in order to build a linear regression model in the cells below
Can you write Python code for it?

df2=df.copy()

#Let's make a separate column for each value of our categorical variable
df2=pd.get_dummies(df2,prefix=['month'])

#Let's code the month with numbers
d={'January':1, 'February':2, 'March':3, 'April':4, 'May':5, 'June':6, 'July':7,
       'August':8, 'September':9, 'October':10, 'November':11, 'December':12}
month=df.month.map(d)

#We paste the information about the date from several columns
dt=list()
for year,mont in zip(df2.year.values, month.values):
    s=str(year)+' '+str(mont)+' 1'
    dt.append(s)
#convert the received data into the DateTime type and replace them with a column year
df2.rename(columns={'year': 'DateTime'}, inplace=True)
df2['DateTime']=pd.to_datetime(dt, format='%Y %m %d')

df2.head(5)


I am working on the file "3.Machine_learning/1.msc_mayor_app/msc_app_to_mayor.ipynb".
The first task is:
Мы будем использовать линейную регрессию с регуляризацией Гребень (Ridge). Данные поделим в соотношении 80% к 20 % (обучение / контроль), также будем проверять качество модели с помощью кросс валидации (в данном случае разбиение будет один цикл – один образец).

We will use linear regression with Ridge regularization. The data will be divided in the ratio of 80% to 20% (train/ test), and we will also check the quality of the model using cross validation (in this case, the split will be one cycle - one sample).
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(X_scal, y_scal, test_size=0.2, random_state=42)
#y_train=np.reshape(y_train,[y_train.shape[0],1])
#y_test=np.reshape(y_test,[y_test.shape[0],1])

loo = model_selection.LeaveOneOut()

#alpha coefficient is taken  at a rough guess

lr = linear_model.Ridge(alpha=55.0)
scores = model_selection.cross_val_score(lr , X_train, y_train, scoring='mean_squared_error', cv=loo,)
print('CV Score:', scores.mean())

lr .fit(X_train, y_train)
print('Coefficients:', lr.coef_)
print('Test Score:', lr.score(X_test,y_test))


I am working on the file "3.Machine_learning/1.msc_mayor_app/msc_app_to_mayor.ipynb".
The first task is:
Первый взгляд на данные (First look at the data)
Посмотрим на корреляцию между некоторыми столбцами, а именно всеми кроме связанных с округами Москвы. Построим  их попарные диаграммы

Let's look at the correlation between some columns, namely all except those connected with the districts of Moscow. We construct their pair diagrams
Can you write Python code for it?

columns_to_show = ['res_positive', 'res_explained', 'res_negative',
                   'total_appeals', 'appeals_to_mayor','El_form_to_mayor', 'Pap_form_to_mayor']
data=df[columns_to_show]

grid = sns.pairplot(df[columns_to_show])
savefig('1.png')


I am working on the file "3.Machine_learning/1.msc_mayor_app/msc_app_to_mayor.ipynb".
The first task is:
Посмотрим подробней на некоторые комбинации, в которых есть намек на линейную зависимость. Получим количественную оценку в виде коэффициента корреляции Пирсона.

Let's look in more detail at some combinations in which there is a similarity with linear dependence. We obtain a quantitative estimate in the form of a Pearson correlation coefficient.
Can you write Python code for it?

print("Correlation coefficient for a explained review result to the total number of appeals =",
       df.res_explained.corr(df.total_appeals) )

print("Corr.coeff. for a  total number of appeals to mayor to the total number of appeals to mayor in electronic form =",
       df.appeals_to_mayor.corr(df.El_form_to_mayor) )


I am working on the file "3.Machine_learning/1.msc_mayor_app/msc_app_to_mayor.ipynb".
The first task is:
При идеально точном предсказании, все 4 точки должны были бы располагаться на линии.

With an ideally accurate prediction, all 4 points would have to be on the line.
Временной ряд (Time series)
До этого мы брали случайные точки дал контроля предсказания.
Давайте теперь рассмотрим тоже самое, но в контексте временного тренда.
Будем предсказывать количество положительных решений в «будущем».

Previously, we took random points to give control of the prediction.
Let's now consider the same thing, but in the context of the trend.
We will predict the number of positive decisions in the "future".
Для начала посмотрим на нашу прошлую модель с ценами на нефть.

First, look at our past model with oil prices.
Can you write Python code for it?

l_bord = 18
r_bord = 22

X_train=X_scal_oil[0:l_bord]
X_test=X_scal_oil[l_bord:r_bord]
y_train=y_scal_oil[0:l_bord]
y_test=y_scal_oil[l_bord:r_bord]

loo = model_selection.LeaveOneOut()
lr = linear_model.Ridge(alpha=7.0)
scores = model_selection.cross_val_score(lr , X_train, y_train, scoring='mean_squared_error', cv=loo,)
print('CV Score:', scores.mean())

lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_)
print('Test Score:', lr.score(X_test,y_test))

# plot for test data
plt.figure(figsize=(19,10))

#trainline
plt.scatter(df2.DateTime.values[0:l_bord], lr.predict(X_train),  color='black')
plt.plot(df2.DateTime.values[0:l_bord], y_train, '--', color='green',
         linewidth=3)

#test line
plt.scatter(df2.DateTime.values[l_bord:r_bord], lr.predict(X_test),  color='black')
plt.plot(df2.DateTime.values[l_bord:r_bord], y_test, '--', color='blue',
         linewidth=3)

#connecting line
plt.plot([df2.DateTime.values[l_bord-1],df2.DateTime.values[l_bord]],  [y_train[l_bord-1],y_test[0]] ,
         color='magenta',linewidth=2, label='train to test')

plt.xlabel('Date')
plt.ylabel('Relative number of positive results')
plt.title="Time series"

print('predict: {0} '.format(lr.predict(X_test)))
print('real: {0} '.format(y_test))

savefig('5.1.png')


I am working on the file "3.Machine_learning/1.msc_mayor_app/msc_app_to_mayor.ipynb".
The first task is:
С одной стороны, очевидно, что чем больше общее количество обращений или обращений в электронной форме, тем больше всего обращений. С другой стороны, надо отметить, что эта зависимость не полностью линейная и наверняка мы не смогли учесть всего.

On the one hand, it is obvious that the greater the total number of appeals or appeals in electronic form. The greater the number of appeals. On the other hand, it should be noted that this dependence is not completely linear and we could not possibly have taken into account everything.
Давайте рассмотрим, еще что-нибудь. Например, найдем округ Москвы, где за год больше всего обращений граждан на 10000 человек населения.

Let's look at something else. For example, we will find the district of Moscow, where for the year the most appeals of citizens for 10,000 people.
Can you write Python code for it?

district_columns = ['to_10K_total_VAO', 'to_10K_total_ZAO', 'to_10K_total_ZelAO',
        'to_10K_total_SAO','to_10K_total_SVAO','to_10K_total_SZAO','to_10K_total_TiNAO','to_10K_total_CAO',
        'to_10K_total_YUAO','to_10K_total_YUVAO','to_10K_total_YUZAO']

y_pos = np.arange(len(district_columns))

short_district_columns=district_columns.copy()
for i in range(len(short_district_columns)):
    short_district_columns[i] = short_district_columns[i].replace('to_10K_total_','')

distr_sum = df[district_columns].sum()

plt.figure(figsize=(16,9))
plt.bar(y_pos, distr_sum, align='center', alpha=0.5)

plt.xticks(y_pos, short_district_columns)
plt.ylabel('Number of appeals')
plt.title('Number of appeals per 10,000 people for all time')

savefig('2.png')


I am working on the file "3.Machine_learning/1.msc_mayor_app/msc_app_to_mayor.ipynb".
The first task is:
Уберем цены на нефть, зато добавим закодированные данные о месяцах.

We remove oil prices, but we add coded data about the months.
Can you write Python code for it?

l_bord = 18
r_bord = 22

cols_months=['month_December', 'month_February', 'month_January', 'month_July', 'month_June', 'month_March', 'month_May', 'month_November',
'month_October','month_September','month_April','month_August']

X_month=df2[cols_for_regression+cols_months].values
y_month=df2['res_positive']
scaler =StandardScaler()
X_scal_month=scaler.fit_transform(X_month)
y_scal_month=scaler.fit_transform(y_month)

X_train=X_scal_month[0:l_bord]
X_test=X_scal_month[l_bord:r_bord]
y_train=y_scal_month[0:l_bord]
y_test=y_scal_month[l_bord:r_bord]

loo = model_selection.LeaveOneOut()
lr = linear_model.Ridge(alpha=7.0)
scores = model_selection.cross_val_score(lr , X_train, y_train, scoring='mean_squared_error', cv=loo,)
print('CV Score:', scores.mean())

lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_)
print('Test Score:', lr.score(X_test,y_test))

# plot for test data
plt.figure(figsize=(19,10))

#trainline
plt.scatter(df2.DateTime.values[0:l_bord], lr.predict(X_train),  color='black')
plt.plot(df2.DateTime.values[0:l_bord], y_train, '--', color='green',
         linewidth=3)
#test line
plt.scatter(df2.DateTime.values[l_bord:r_bord], lr.predict(X_test),  color='black')
plt.plot(df2.DateTime.values[l_bord:r_bord], y_test, '--', color='blue',
         linewidth=3)
#connecting line
plt.plot([df2.DateTime.values[l_bord-1],df2.DateTime.values[l_bord]],  [y_train[l_bord-1],y_test[0]] , color='magenta',linewidth=2, label='train to test')

plt.xlabel('Date')
plt.ylabel('Relative number of positive results')
plt.title="Time series"

print('predict: {0} '.format(lr.predict(X_test)))
print('real: {0} '.format(y_test))

savefig('5.2.png')


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
График для зависимости, полученной по обучающим данным

The graph for the dependence obtained from the training data
Can you write Python code for it?

plt.scatter(X_train, y_train,  color='black')
plt.plot(X_train, lr.predict(X_train), color='blue',
         linewidth=3)

plt.xlabel('StateRegistrationOfBirth')
plt.ylabel('State Registration OfPaternity Examination')
plt.title="Regression on train data"


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
График для зависимости, полученной поконтрольным данным

The graph for the dependence obtained from the test data
Can you write Python code for it?

plt.scatter(X_test, y_test,  color='black')
plt.plot(X_test, lr.predict(X_test), color='green',
         linewidth=3)

plt.xlabel('StateRegistrationOfBirth')
plt.ylabel('State Registration OfPaternity Examination')
plt.title="Regression on test data"


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Для начала посмотрим на результаты без регуляризации

First, look at the results without regularization
Can you write Python code for it?

lr = linear_model.LinearRegression()
lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_)
print('Score:', lr.score(X_test,y_test))


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Добавим другой малополезный признак State Registration Of Name Change

Add another less useful sign. State Registration Of Name Change
Can you write Python code for it?

columns_to_show4=columns_to_show2.copy()
columns_to_show4.append("StateRegistrationOfNameChange")

X = df2[columns_to_show4].values
# y hasn't changed
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
y_train=np.reshape(y_train,[y_train.shape[0],1])
y_test=np.reshape(y_test,[y_test.shape[0],1])

lr = linear_model.LinearRegression()
lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_)
print('Score:', lr.score(X_test,y_test))


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Загрузка и предобработка (Download and preprocessing)
Can you write Python code for it?

#download
df = pd.read_csv('https://op.mos.ru/EHDWSREST/catalog/export/get?id=230308', compression='zip', header=0, encoding='cp1251', sep=';', quotechar='"')
#look at the data
df.head(12)


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Закодируем месяца числовыми значениями и удалим ненужные для анализа столбцы

We will code the month with numeric values and delete the columns we do not need for analysis
Can you write Python code for it?

#code months
d={'январь':1, 'февраль':2, 'март':3, 'апрель':4, 'май':5, 'июнь':6, 'июль':7,
       'август':8, 'сентябрь':9, 'октябрь':10, 'ноябрь':11, 'декабрь':12}
df.Month=df.Month.map(d)

#delete some unuseful columns
df.drop(['ID','global_id','Unnamed: 12'],axis=1,inplace=True)

#look at the data
df.head(12)


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
И взглянем на графики

And look at the graphics
Can you write Python code for it?

# plot for train data
plt.figure(figsize=(8,10))
plt.subplot(211)

plt.scatter(X_train, y_train,  color='black')
plt.plot(X_train, lr.predict(X_train),  color='blue',
         linewidth=3)

plt.xlabel('Total Number of Registration')
plt.ylabel('State Registration Of Marriage')
plt.title="Regression on train data"

# plot for test data
plt.subplot(212)
plt.scatter(X_test, y_test,  color='black')
plt.plot(X_test, lr.predict(X_test), '--', color='green',
         linewidth=3)

plt.xlabel('Total Number of Registration')
plt.ylabel('State Registration Of Marriage')
plt.title="Regression on test data"


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Как видно, он нам только мешает.

As you can see, it's just a hindrance.
Добавим полезный признак, закодированное значение месяца в который получил количество регистраций.

Add a useful feature, the encoded value of the month in which the number of registrations was received.
Can you write Python code for it?

#get data
columns_to_show5=columns_to_show2.copy()
columns_to_show5.append("Month")

#get data for model
X = df2[columns_to_show5].values
# y hasn't changed
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
y_train=np.reshape(y_train,[y_train.shape[0],1])
y_test=np.reshape(y_test,[y_test.shape[0],1])
#teach model and get predictions
lr = linear_model.LinearRegression()
lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_)
print('Score:', lr.score(X_test,y_test))


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Качество предсказания резко улучшилось

The quality of the prediction has has greatly improved
Теперь попробуем закодировать вместо значения месяца, среднее значение регистрации браков в данный месяц, взятое на основании обучающих данных.

Now try to encode instead of the month, the average value of registration of marriages in a given month, taken on the basis of training data.
Can you write Python code for it?

#get data for pandas data frame
df5=df_base.copy()

d=dict()

#get we obtain the mean value of Registration Of Marriages by months on the training data
for mon in df5.Month.unique():

    d[mon]=df5.StateRegistrationOfMarriage[df5.Month.values[train]==mon].mean()
   #d+={}

df5['MeanMarriagePerMonth']=df5.Month.map(d)
df5.drop(['Month','Year','StateRegistrationOfMarriage','ID','global_id','Unnamed: 12','TotalNumber',
          'StateRegistrationOfNameChange','StateRegistrationOfAdoption'],axis=1,inplace=True)

#get data for model
X=df5.values
X_train=X[train]
X_test=X[test]

#teach model and get predictions
lr = linear_model.LinearRegression()
lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_[0])
print('Score:', lr.score(X_test,y_test))

# plot for all data
plt.scatter(df3.Months.values[train], y_train,  color='black')
plt.plot(df3.Months.values[train], lr.predict(X_train),  color='blue', label='train', linewidth=2)

plt.scatter(df3.Months.values[test], y_test,  color='black')
plt.plot(df3.Months.values[test], lr.predict(X_test),  color='green', label='test',  linewidth=2)

plt.title="Regression (prediction) on all data"
plt.xlabel('Months (from 01.2010)')
plt.ylabel('State Registration Of Marriage')

#plot line for link train to test
plt.plot([72,73],  lr.predict([X_train[-1],X_test[0]]) , color='magenta',linewidth=2, label='train to test')


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Линейная регрессия для предсказания тренда (Linear regression for predicting a trend)
Вернемся к исходным данным, но рассмотрим их теперь с учетом изменения во времени.
Для начала заменим колонку год на общее количество месяцев с момента начальной даты
В этот раз не будем масштабировать данные, большой пользы это не принесет.

Let's go back to the original data, but consider them now with the change in time.
To begin with, replace the column year by the total number of months from the start date
This time we will not scale the data, it will not be of much use.
Can you write Python code for it?

#get data
df3=df.copy()

#get new column
df3.Year=df.Year.map(lambda x: (x-2010)*12)+df.Month
df3.rename(columns={'Year': 'Months'}, inplace=True)

#get data for model
X=df3[columns_to_show5].values
y=df3['StateRegistrationOfMarriage'].values
train=[df3.Months<=72]
test=[df3.Months>72]
X_train=X[train]
y_train=y[train]
X_test=X[test]
y_test=y[test]
y_train=np.reshape(y_train,[y_train.shape[0],1])
y_test=np.reshape(y_test,[y_test.shape[0],1])

#teach model and get predictions
lr = linear_model.LinearRegression()
lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_[0])
print('Score:', lr.score(X_test,y_test))


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Обучим простою линейную регрессию на 4-х мерном векторе признаков

We teach a linear regression on a 4-dimensional vector of features
Can you write Python code for it?


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Попробуем применить one-hot кодирование к графе Месяц

Let's try to apply one-hot encoding to the column Month
Can you write Python code for it?

#get data for model

df4=df_base.copy()
df4.drop(['Year','StateRegistrationOfMarriage','ID','global_id','Unnamed: 12','TotalNumber','StateRegistrationOfNameChange','StateRegistrationOfAdoption'],axis=1,inplace=True)
df4=pd.get_dummies(df4,prefix=['Month'])

X=df4.values
X_train=X[train]
X_test=X[test]

#teach model and get predictions

lr = linear_model.LinearRegression()
lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_[0])
print('Score:', lr.score(X_test,y_test))

# plot for all data
plt.scatter(df3.Months.values[train], y_train,  color='black')
plt.plot(df3.Months.values[train], lr.predict(X_train),  color='blue', label='train', linewidth=2)

plt.scatter(df3.Months.values[test], y_test,  color='black')
plt.plot(df3.Months.values[test], lr.predict(X_test),  color='green', label='test',  linewidth=2)

plt.title="Regression (prediction) on all data"
plt.xlabel('Months (from 01.2010)')
plt.ylabel('State Registration Of Marriage')

#plot line for link train to test
plt.plot([72,73],  lr.predict([X_train[-1],X_test[0]]) , color='magenta',linewidth=2, label='train to test')


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Посмотрим, изменит ли что-то масштабирование.

Let's see the result of scaling.
Can you write Python code for it?

# change scale of features
scaler = MinMaxScaler()
df2=pd.DataFrame(scaler.fit_transform(df))
df2.columns=df.columns
data2=df2[columns_to_show]

grid2 = sns.pairplot(data2)


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Построим попарные графики зависимостей, но для наглядности возьмем только часть признаков

We construct pairwise graphs of dependencies, but for clarity we take only a part of the features
Can you write Python code for it?

columns_to_show = ['StateRegistrationOfBirth', 'StateRegistrationOfMarriage',
                   'StateRegistrationOfPaternityExamination', 'StateRegistrationOfDivorce','StateRegistrationOfDeath']
data=df[columns_to_show]

grid = sns.pairplot(data)


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Почти без разницы

Almost without difference
Простейшая регрессия по 1 признаку (Regression 1 features)
Рассмотрим два параметра с наиболее выраженной линейной зависимостью StateRegistrationOfBirth и  StateRegistrationOfPaternityExamination

Consider two parameters with the most pronounced linear dependence StateRegistrationOfBirth and StateRegistrationOfPaternityExamination
Can you write Python code for it?

#get data for model

X = data2['StateRegistrationOfBirth'].values
y  = data2['StateRegistrationOfPaternityExamination'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
X_train=np.reshape(X_train,[X_train.shape[0],1])
y_train=np.reshape(y_train,[y_train.shape[0],1])
X_test=np.reshape(X_test,[X_test.shape[0],1])
y_test=np.reshape(y_test,[y_test.shape[0],1])

#teach model and get predictions
lr = linear_model.LinearRegression()
lr.fit(X_train, y_train)
print('Coefficients:', lr.coef_)
print('Score:', lr.score(X_test,y_test))


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Рассмотрим линейную регрессию с регуляризацией - Лассо

Consider linear regression with Lasso regularization
Can you write Python code for it?

#let's look at the different alpha parameter:

#large
Rid=linear_model.Lasso (alpha = 0.01)
Rid.fit(X_train, y_train)
print(' Appha:', Rid.alpha)
print(' Coefficients:', Rid.coef_)
print(' Score:', Rid.score(X_test,y_test))

#Small
Rid=linear_model.Lasso (alpha = 0.000000001)
Rid.fit(X_train, y_train)
print('\n Appha:', Rid.alpha)
print(' Coefficients:', Rid.coef_)
print(' Score:', Rid.score(X_test,y_test))

#Optimal (for these test data)
Rid=linear_model.Lasso (alpha = 0.00025)
Rid.fit(X_train, y_train)
print('\n Appha:', Rid.alpha)
print(' Coefficients:', Rid.coef_)
print(' Score:', Rid.score(X_test,y_test))


I am working on the file "3.Machine_learning/2.family_registered_habr.ipynb".
The first task is:
Результат предсказания "не очень", но думаю лучше, чем просто наобум
Посмотрим на данные в графическом виде, в начале по отдельности, а потом вместе.
Наша модель пусть и не очень хорошо, но улавливает основные особенности тренда, позволяя прогнозировать данные.

The result of the prediction is "not very," but I think it's better than just haphazardly

Let's look at the data in a graphical form, at the beginning separately, and then together.
Our model, though not very good, but captures the main features of the trend, allowing you to predict the data.
Can you write Python code for it?

plt.figure(figsize=(9,23))

# plot for train data
plt.subplot(311)

plt.scatter(df3.Months.values[train], y_train,  color='black')
plt.plot(df3.Months.values[train], lr.predict(X_train),  color='blue', linewidth=2)
plt.xlabel('Months (from 01.2010)')
plt.ylabel('State Registration Of Marriage')
plt.title="Regression on train data"

# plot for test data
plt.subplot(312)

plt.scatter(df3.Months.values[test], y_test,  color='black')
plt.plot(df3.Months.values[test], lr.predict(X_test),  color='green',   linewidth=2)
plt.xlabel('Months (from 01.2010)')
plt.ylabel('State Registration Of Marriage')
plt.title="Regression (prediction) on test data"

# plot for all data
plt.subplot(313)

plt.scatter(df3.Months.values[train], y_train,  color='black')
plt.plot(df3.Months.values[train], lr.predict(X_train),  color='blue', label='train', linewidth=2)

plt.scatter(df3.Months.values[test], y_test,  color='black')
plt.plot(df3.Months.values[test], lr.predict(X_test),  color='green', label='test',  linewidth=2)

plt.title="Regression (prediction) on all data"
plt.xlabel('Months (from 01.2010)')
plt.ylabel('State Registration Of Marriage')

#plot line for link train to test
plt.plot([72,73],  lr.predict([X_train[-1],X_test[0]]) , color='magenta',linewidth=2, label='train to test')

plt.legend()


I am working on the file "3.PCA.ipynb".
The first task is:
Eigendecomposition - Computing Eigenvectors and Eigenvalues
Can you write Python code for it?

mean_vec = np.mean(X_std, axis=0)
cov_mat = (X_std - mean_vec).T.dot((X_std - mean_vec)) / (X_std.shape[0]-1)
print('Covariance matrix \n%s' %cov_mat)

# Doing this directly using np.cov
print('NumPy covariance matrix: \n%s' %np.cov(X_std.T))

cov_mat = np.cov(X_std.T)
eig_vals, eig_vecs = np.linalg.eig(cov_mat)

print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)