I am working on the file "1__Warmup.ipynb".
The first task is:
Rolling statistics

Notice that pandas.rolling_mean has been deprecated for DataFrame and will be removed in a future version.
Hence, we will replace it with DataFrame.rolling(center=False,window=20).mean()
Notice that pd.rolling_std has been deprecated for DataFrame and will be removed in a future version.
Hence, we will replace it with DataFrame.rolling(center=False,window=20).std()
Can you write Python code for it?

df = fill_missing_values(
        get_data(
            symbols=['GOOG','SPY','IBM','GLD'],
                             start='4/21/2015',
                             end='7/15/2016'))

# 1. Computing rolling mean using a 20-day window
rm_df = pd.DataFrame.rolling(df, window=20).mean()

ax = rm_df.plot(title="Rolling Mean")
ax.set_xlabel("Date")
ax.set_ylabel("Price")
plt.show()

# 2. Computing rolling standard deviation using a 20-day window
rstd_df = pd.DataFrame.rolling(df, window=20).std()

ax = rstd_df.plot(title="Rolling Standard Deviation")
ax.set_xlabel("Date")
ax.set_ylabel("Price")
plt.show()

# 3. Compute upper and lower bands
def get_bollinger_bands(rm, rstd):
    """Return upper and lower Bollinger Bands."""
    upper_band, lower_band = rm + 2 * rstd, rm - 2 * rstd
    return upper_band, lower_band

upper_band, lower_band = get_bollinger_bands(rm_df, rstd_df)

# Plot raw SPY values, rolling mean and Bollinger Bands
ax = df['SPY'].plot(title="Bollinger Bands",label='SPY')
rm_df['SPY'].plot(label='Rolling mean', ax=ax)
upper_band['SPY'].plot(label='upper band', ax=ax)
lower_band['SPY'].plot(label='lower band', ax=ax)

# Add axis labels and legend
ax.set_xlabel("Date")
ax.set_ylabel("Price")
ax.legend(loc='lower left')
plt.show()


I am working on the file "1__Warmup.ipynb".
The first task is:
Sharpe Ratio
Sharpe ratio is a
way to examine the performance of an investment by adjusting for its risk. The ratio measures the excess
return (or risk premium) per unit of deviation in an investment asset or a trading strategy, typically
referred to as risk (and is a deviation risk measure), named after William F. Sharpe.
The ex-ante Sharpe ratio is defined as:
    $$S=\frac{E{R_p-R_f}}{Std{R_p-R_f}}$$
where $R_p$ is the asset return, $R_f$ is the return on a benchmark asset,
such as the risk free rate of return. $E{R_p-R_f}$ is the expected value of the excess of the asset return
over the benchmark return, and $Std{R_p-R_f}$ is the standard deviation of the asset return.
The ex-post Sharpe ratio uses the same equation as the one above but with
realized returns of the asset and benchmark rather than expected returns.
Examples of risk free rates are
* LIBOR
* Interest rate of 3 month treasury bill
* 0% (approximation used a lot in last years)
Using daily returns as $R_p$  the ex-post Sharpe ratio is computable as:
    $$S=\frac{mean{R_p^{daily}-R_f^{daily}}}{Std{R_p^{daily}}}$$
as $R_f^{daily}$ is typically costant for several months it is possible to remove from the denominator.
Also, $R_f^{daily}$ is typically approximable with 0% but it is possible to compute given $R_f^{yearly}$ as
it follows remembering there are 252 trading days in a year
$$R_f^{daily}=\sqrt[252]{1+R_f^{yearly}}$$
Sharpe ratio is typically an annual measure. This means if we are using different sample frequencies
need to apply the following formula:
    $$S^{annual}=\sqrt[2]{SPY} \times S$$
where $SPR$ is the number of samples per year considered in computing $S$, e.g.

$S^{annual}=\sqrt[2]{252} \times S^{daily}$
$S^{annual}=\sqrt[2]{52} \times S^{weekly}$
$S^{annual}=\sqrt[2]{12} \times S^{monthly}$
Can you write Python code for it?

def sharpe_ratio(df,sample_freq='d',risk_free_rate=0.0):

    sr = (df - risk_free_rate).mean() / df.std()

    if sample_freq == 'd':
        sr = sr * np.sqrt(252)
    elif sample_freq == 'w':
        sr = sr * np.sqrt(52)
    elif sample_freq == 'm':
        sr = sr * np.sqrt(12)
    else:
        raise Exception('unkown sample frequency :'+str(sample_freq))

    return sr

# Sharpe ratio
sharpe_ratio(
    compute_daily_returns(
        fill_missing_values(
            get_data(
                symbols=['GOOG','SPY','IBM','GLD'],
                             start='4/21/2015',
                             end='7/15/2016'))))


I am working on the file "1_analyze_comic_data_j.ipynb".
The first task is:
10週以内で終わった最新10タイトルの最初の10話分の掲載順（worst）を表示してみます．値が大きいほど，巻頭付近に掲載されていたことになります．
Can you write Python code for it?

for title in wj.short_end_titles[-10:]:
    plt.plot(wj.extract_item(title)[:10], label=title[:6])
plt.xlabel('Week')
plt.ylabel('Worst')
plt.ylim(0,22)
plt.legend()


I am working on the file "1_analyze_comic_data_j.ipynb".
The first task is:
ある程度予想はしてましたが，さすがですね．ちなみにですが，extract_item()を使ってサブタイトルを取得しながら掲載順を見ると，マンガ好きの方は楽しいと思います．
Can you write Python code for it?


I am working on the file "1_analyze_comic_data_j.ipynb".
The first task is:
かなりわかりづらい処理をしているので，初期化時（__init__()）の動作を補足します．
1. self.all_titlesは文字通り全ての作品名を保持します．しかし，self.all_titlesは，明らかに読みきり作品や企画作品を含んでしまっています．
2. そこで，min_week以上連載した作品self.serialized_titlesとして抽出します．しかし，self.serialized_titlesは，データベースの最新の目次情報の時点で，連載を継続中の作品を含んでおり，連載継続期間が不正確になってしまいます．例えば，「鬼滅の刃」など現在も連載中の人気作が，21週で連載が終了した作品のように見えてしまいます．
3. そこで，データベースの最新の目次情報の時点で連載が終了した（と思われれる）作品のみをself.end_titlesとして抽出します．self.end_titlesが，本分析における全体集合です．
4. self.end_titlesのうち，10週以内に終了した作品をself.short_end_titlesとして，11週以内に継続した作品をself.long_end_titlesとして抽出します．
分析
Can you write Python code for it?


I am working on the file "1_analyze_comic_data_j.ipynb".
The first task is:
さて，seabornで相関分析をやってみます．ここでは，ひとまず6週目までの掲載順をプロットします．同じ座標に複数の点が重なって非常に見づらいので，便宜上ランダムなノイズを加えて見栄えを整えます．なお，1週目を外したのは，ほとんどの場合巻頭に掲載されるためです．
Can you write Python code for it?

end_data = pd.DataFrame(
    [[wj.extract_item(title)[1] + np.random.randn() * .3,
      wj.extract_item(title)[2] + np.random.randn() * .3,
      wj.extract_item(title)[3] + np.random.randn() * .3,
      wj.extract_item(title)[4] + np.random.randn() * .3,
      wj.extract_item(title)[5] + np.random.randn() * .3,
      '短命作品' if title in wj.short_end_titles else '継続作品']
     for title in wj.end_titles])

end_data.columns = ["Worst (week2)", "Worst (week3)", "Worst (week4)",
                    "Worst (week5)", "Worst (week6)", "Type"]
sns.pairplot(end_data, hue="Type", palette="husl")


I am working on the file "1_demo.ipynb".
The first task is:
Add Progressbar
If you don't care the original of coroutines
Can you write Python code for it?

@asyncio.coroutine
def quote_many(num_quotes=1, conn_limit=20, progress=None, step=10):
    if progress is None:
        progress = widgets.IntProgress()
        progress.max = num_quotes // step
        ipydisplay(progress)
    semaphore = asyncio.Semaphore(conn_limit)
    coroutines = [
        quote_with_lock(semaphore) for i in range(num_quotes)
    ]
    # quotes = yield from (asyncio.gather(*coroutines))
    quotes = []
    for ith, coro in enumerate(asyncio.as_completed(coroutines), 1):
        if ith % step == 0:
            progress.value += 1
        q = yield from coro
        quotes.append(q)
    return quotes

%%time
quotes = loop.run_until_complete(quote_many(2000, conn_limit=100, step=1))


I am working on the file "1_demo.ipynb".
The first task is:
Alternative way
Can you write Python code for it?

@asyncio.coroutine
def quote_many(num_quotes=1, conn_limit=20, progress=None, step=10):
    if progress is None:
        progress = widgets.IntProgress()
        progress.max = num_quotes // step
        ipydisplay(progress)

    semaphore = asyncio.Semaphore(conn_limit)

    # wrap coroutines with future
    # For Python 3.4.4+, asyncio.ensure_future(...)
    # will wrap coro as Task and keep input the same
    # if it is already Future.
    futures = [
        asyncio.ensure_future(quote_with_lock(semaphore))
        for i in range(num_quotes)
    ]

    for ith, coro in enumerate(asyncio.as_completed(futures), 1):
        if ith % step == 0:
            progress.value += 1
        yield from coro

    quotes = [fut.result() for fut in futures]
    return quotes

%%time
quotes = loop.run_until_complete(quote_many(2000, conn_limit=100, step=20))


I am working on the file "1_demo.ipynb".
The first task is:
Basic
Can you write Python code for it?

@asyncio.coroutine
def quote_simple(url='http://localhost:5566/quote/uniform', slow=False):
    r = yield from aiohttp.request(
        'GET', url, params={'slow': True} if slow else {}
    )
    if r.status != 200:
        logger.error('Unsuccessful response [Status: %s (%d)]'
                     % (r.reason, r.status))
        r.close(force=True)
        return None
    quote_json = yield from r.json()
    return quote_json['quote']

loop = asyncio.get_event_loop()


I am working on the file "1_demo.ipynb".
The first task is:
For fast response, progress bar introduces considerable latency. Try modify the step higher.
Can you write Python code for it?

%%time
quotes = loop.run_until_complete(quote_many(2000, conn_limit=100, step=20))


I am working on the file "1_demo.ipynb".
The first task is:
I don't know why, but using its internal connection limit is slow. But we can implement one ourselves.
Custom connection limit using semaphore
Use [asyncio.Semaphore] acting as a lock.
Can you write Python code for it?

def quote_with_lock(semaphore, url='http://localhost:5566/quote/uniform'):
    with (yield from semaphore):
        r = yield from aiohttp.request('GET', url)
        if r.status != 200:
            logger.error('Unsuccessful response [Status: %s (%d)]'
                         % (r.reason, r.status))
            r.close(force=True)
            return None
        quote_json = yield from r.json()
    r.close(force=True)
    return quote_json['quote']

@asyncio.coroutine
def quote_many(num_quotes=1, conn_limit=20):
    semaphore = asyncio.Semaphore(conn_limit)
    coroutines = [
        quote_with_lock(semaphore) for i in range(num_quotes)
    ]
    quotes = yield from (asyncio.gather(*coroutines))
    return quotes

%%time
quotes = loop.run_until_complete(quote_many(2000, conn_limit=100))


I am working on the file "1_demo.ipynb".
The first task is:
Internally asyncio wraps it with [asyncio.Task].
So the following works equivalently.
Can you write Python code for it?

task = asyncio.Task(quote_simple())
quote = loop.run_until_complete(task)
quote


I am working on the file "1_demo.ipynb".
The first task is:
Original order matters
... go eat yourself.
Can you write Python code for it?

@asyncio.coroutine
def quote_many(num_quotes=1, conn_limit=20, progress=None, step=10):
    if progress is None:
        progress = widgets.IntProgress()
        progress.max = num_quotes // step
        ipydisplay(progress)

    # create the lock
    semaphore = asyncio.Semaphore(conn_limit)

    finished_task_count = 0
    def progress_adder(fut):
        nonlocal finished_task_count
        finished_task_count += 1
        if finished_task_count % step == 0:
            progress.value += 1

    # wrap coroutines as Tasks
    futures = []
    for i in range(num_quotes):
        task = asyncio.Task(quote_with_lock(semaphore))
        task.add_done_callback(progress_adder)
        futures.append(task)

    quotes = yield from (asyncio.gather(*futures))
    return quotes

%%time
quotes = loop.run_until_complete(quote_many(2000, conn_limit=100, step=1))

%%time
quotes = loop.run_until_complete(quote_many(2000, conn_limit=100, step=20))


I am working on the file "1_demo.ipynb".
The first task is:
Passing wrong URL gives error
Can you write Python code for it?

quote = loop.run_until_complete(
    quote_simple(url='http://localhost:5566/quote/uniform?part=100')
)


I am working on the file "1_demo.ipynb".
The first task is:
This is not helping since we open 2000 connections at a time. It is slower than expected.
Limiting connection pool size
Ref on official site.
Can you write Python code for it?

@asyncio.coroutine
def quote(conn, url='http://localhost:5566/quote/uniform', slow=False):
    r = yield from aiohttp.request(
        'GET', url, params={'slow': True} if slow else {},
        connector=conn
    )
    if r.status != 200:
        logger.error('Unsuccessful response [Status: %s (%d)]'
                     % (r.reason, r.status))
        r.close(force=True)
        return None
    quote_json = yield from r.json()
    r.close(force=True)
    return quote_json['quote']

@asyncio.coroutine
def quote_many(num_quotes=1, conn_limit=20):
    conn = aiohttp.TCPConnector(keepalive_timeout=1, force_close=True, limit=conn_limit)
    coroutines = [
        quote(conn) for i in range(num_quotes)
    ]
    quotes = yield from (asyncio.gather(*coroutines))
    return quotes

%%time
quotes = loop.run_until_complete(quote_many(2000, conn_limit=100))


I am working on the file "1_dlnd-your-first-neural-network/dlnd-your-first-neural-network.ipynb".
The first task is:
Checking out the data
This dataset has the number of riders for each hour of each day from January 1 2011 to December 31 2012. The number of riders is split between casual and registered, summed up in the cnt column. You can see the first few rows of the data above.
Below is a plot showing the number of bike riders over the first 10 days in the data set. You can see the hourly rentals here. This data is pretty complicated! The weekends have lower over all ridership and there are spikes when people are biking to and from work during the week. Looking at the data above, we also have information about temperature, humidity, and windspeed, all of these likely affecting the number of riders. You'll be trying to capture all this with your model.
Can you write Python code for it?


I am working on the file "1_dlnd-your-first-neural-network/dlnd-your-first-neural-network.ipynb".
The first task is:
Dummy variables
Here we have some categorical variables like season, weather, month. To include these in our model, we'll need to make binary dummy variables. This is simple to do with Pandas thanks to get_dummies().
Can you write Python code for it?

dummy_fields = ['season', 'weathersit', 'mnth', 'hr', 'weekday']
for each in dummy_fields:
    dummies = pd.get_dummies(rides[each], prefix=each, drop_first=False)
    rides = pd.concat([rides, dummies], axis=1)

fields_to_drop = ['instant', 'dteday', 'season', 'weathersit',
                  'weekday', 'atemp', 'mnth', 'workingday', 'hr']
data = rides.drop(fields_to_drop, axis=1)
data.head()


I am working on the file "1_dlnd-your-first-neural-network/dlnd-your-first-neural-network.ipynb".
The first task is:
Scaling target variables
To make training the network easier, we'll standardize each of the continuous variables. That is, we'll shift and scale the variables such that they have zero mean and a standard deviation of 1.
The scaling factors are saved so we can go backwards when we use the network for predictions.
Can you write Python code for it?

quant_features = ['casual', 'registered', 'cnt', 'temp', 'hum', 'windspeed']
# Store scalings in a dictionary so we can convert back later
scaled_features = {}
for each in quant_features:
    mean, std = data[each].mean(), data[each].std()
    scaled_features[each] = [mean, std]
    data.loc[:, each] = (data[each] - mean)/std


I am working on the file "1_dlnd-your-first-neural-network/dlnd-your-first-neural-network.ipynb".
The first task is:
Splitting the data into training, testing, and validation sets
We'll save the last 21 days of the data to use as a test set after we've trained the network. We'll use this set to make predictions and compare them with the actual number of riders.
Can you write Python code for it?

# Save the last 21 days
test_data = data[-21*24:]
data = data[:-21*24]

# Separate the data into features and targets
target_fields = ['cnt', 'casual', 'registered']
features, targets = data.drop(target_fields, axis=1), data[target_fields]
test_features, test_targets = test_data.drop(target_fields, axis=1), test_data[target_fields]


I am working on the file "1_dlnd-your-first-neural-network/dlnd-your-first-neural-network.ipynb".
The first task is:
Thinking about your results
Answer these questions about your results. How well does the model predict the data? Where does it fail? Why does it fail where it does?

Note: You can edit the text in this cell by double clicking on it. When you want to render the text, press control + enter

Your answer below
The model predicts the number of bike rentals pretty neat most time of the year. However, it underperforms during periods of irragularities in the dataset, such as Christmas in the above case. The reason for this is the low freqeuncy of such an event to happen. Our current dataset covers two years of data which contains two Christams at most. This number might be further diminished by splitting the data into test- and validationsets, since these sets can cover these periods of the year. This is exactly what happened in our case, as well. We chose the last 21 days of 2012 for our testset and thus left only one Christmas in our trainingset. This low frequency was not enough for our neural network to learn this pattern. This problem can be overcame by bigger dataset that covers mutiple years of bike rentals.
Unit tests
Run these unit tests to check the correctness of your network implementation. These tests must all be successful to pass the project.
Can you write Python code for it?

import unittest

inputs = [0.5, -0.2, 0.1]
targets = [0.4]
test_w_i_h = np.array([[0.1, 0.4, -0.3],
                       [-0.2, 0.5, 0.2]])
test_w_h_o = np.array([[0.3, -0.1]])

class TestMethods(unittest.TestCase):

    ##########
    # Unit tests for data loading
    ##########

    def test_data_path(self):
        # Test that file path to dataset has been unaltered
        self.assertTrue(data_path.lower() == 'bike-sharing-dataset/hour.csv')

    def test_data_loaded(self):
        # Test that data frame loaded
        self.assertTrue(isinstance(rides, pd.DataFrame))

    ##########
    # Unit tests for network functionality
    ##########

    def test_activation(self):
        network = NeuralNetwork(3, 2, 1, 0.5)
        # Test that the activation function is a sigmoid
        self.assertTrue(np.all(network.activation_function(0.5) == 1/(1+np.exp(-0.5))))

    def test_train(self):
        # Test that weights are updated correctly on training
        network = NeuralNetwork(3, 2, 1, 0.5)
        network.weights_input_to_hidden = test_w_i_h.copy()
        network.weights_hidden_to_output = test_w_h_o.copy()

        network.train(inputs, targets)
        self.assertTrue(np.allclose(network.weights_hidden_to_output,
                                    np.array([[ 0.37275328, -0.03172939]])))
        self.assertTrue(np.allclose(network.weights_input_to_hidden,
                                    np.array([[ 0.10562014,  0.39775194, -0.29887597],
                                              [-0.20185996,  0.50074398,  0.19962801]])))

    def test_run(self):
        # Test correctness of run method
        network = NeuralNetwork(3, 2, 1, 0.5)
        network.weights_input_to_hidden = test_w_i_h.copy()
        network.weights_hidden_to_output = test_w_h_o.copy()

        self.assertTrue(np.allclose(network.run(inputs), 0.09998924))

suite = unittest.TestLoader().loadTestsFromModule(TestMethods())
unittest.TextTestRunner().run(suite)


I am working on the file "1_dlnd-your-first-neural-network/dlnd-your-first-neural-network.ipynb".
The first task is:
Time to build the network
Below you'll build your network. We've built out the structure and the backwards pass. You'll implement the forward pass through the network. You'll also set the hyperparameters: the learning rate, the number of hidden units, and the number of training passes.
The network has two layers, a hidden layer and an output layer. The hidden layer will use the sigmoid function for activations. The output layer has only one node and is used for the regression, the output of the node is the same as the input of the node. That is, the activation function is $f(x)=x$. A function that takes the input signal and generates an output signal, but takes into account the threshold, is called an activation function. We work through each layer of our network calculating the outputs for each neuron. All of the outputs from one layer become inputs to the neurons on the next layer. This process is called forward propagation.
We use the weights to propagate signals forward from the input to the output layers in a neural network. We use the weights to also propagate error backwards from the output back into the network to update our weights. This is called backpropagation.

Hint: You'll need the derivative of the output activation function ($f(x) = x$) for the backpropagation implementation. If you aren't familiar with calculus, this function is equivalent to the equation $y = x$. What is the slope of that equation? That is the derivative of $f(x)$.

Below, you have these tasks:
1. Implement the sigmoid function to use as the activation function. Set self.activation_function in __init__ to your sigmoid function.
2. Implement the forward pass in the train method.
3. Implement the backpropagation algorithm in the train method, including calculating the output error.
4. Implement the forward pass in the run method.
Can you write Python code for it?

class NeuralNetwork(object):
    def _forward_pass(self, inputs):
        outputs = {}
        # Hidden layer
        hidden_inputs = np.dot(self.weights_input_to_hidden, inputs)
        outputs['hidden'] = self.activation_function(hidden_inputs)

        # Output layer
        final_inputs = np.dot(self.weights_hidden_to_output, outputs['hidden'])
        outputs['final'] = final_inputs # Activation function: f(x) = x

        return outputs

    def _backward_pass(self, targets, outputs):

        output_errors = targets - outputs['final'] # 1x1

        # Calculate error gradients
        errors = {}
        errors['final'] = output_errors # Activation function: f(x) = x
        errors['hidden'] = np.dot(self.weights_hidden_to_output.T, errors['final']) * \
                                           outputs['hidden'] * (1-outputs['hidden'])

        return errors

    def _weight_update(self, errors, inputs, outputs):
        self.weights_hidden_to_output += self.lr * np.dot(errors['final'], outputs['hidden'].T)
        self.weights_input_to_hidden += self.lr * np.dot(errors['hidden'], inputs.T)

    def __init__(self, input_nodes, hidden_nodes, output_nodes, learning_rate):

        # Set number of nodes in input, hidden and output layers.
        self.input_nodes = input_nodes
        self.hidden_nodes = hidden_nodes
        self.output_nodes = output_nodes

        # Initialize weights
        self.weights_input_to_hidden = np.random.normal(0.0, self.hidden_nodes**-0.5,
                                       (self.hidden_nodes, self.input_nodes))

        self.weights_hidden_to_output = np.random.normal(0.0, self.output_nodes**-0.5,
                                       (self.output_nodes, self.hidden_nodes))
        self.lr = learning_rate

        # Activation function is the sigmoid function
        self.activation_function = lambda x : 1/(1+np.exp(-x))

    def train(self, inputs_list, targets_list):

        # Using...


I am working on the file "1_dlnd-your-first-neural-network/dlnd-your-first-neural-network.ipynb".
The first task is:
Training the network
Here you'll set the hyperparameters for the network. The strategy here is to find hyperparameters such that the error on the training set is low, but you're not overfitting to the data. If you train the network too long or have too many hidden nodes, it can become overly specific to the training set and will fail to generalize to the validation set. That is, the loss on the validation set will start increasing as the training set loss drops.
You'll also be using a method know as Stochastic Gradient Descent (SGD) to train the network. The idea is that for each training pass, you grab a random sample of the data instead of using the whole data set. You use many more training passes than with normal gradient descent, but each pass is much faster. This ends up training the network more efficiently. You'll learn more about SGD later.
Choose the number of epochs
This is the number of times the dataset will pass through the network, each time updating the weights. As the number of epochs increases, the network becomes better and better at predicting the targets in the training set. You'll need to choose enough epochs to train the network well but not too many or you'll be overfitting.
Choose the learning rate
This scales the size of weight updates. If this is too big, the weights tend to explode and the network fails to fit the data. A good choice to start at is 0.1. If the network has problems fitting the data, try reducing the learning rate. Note that the lower the learning rate, the smaller the steps are in the weight updates and the longer it takes for the neural network to converge.
Choose the number of hidden nodes
The more hidden nodes you have, the more accurate predictions the model will make. Try a few different numbers and see how it affects the performance. You can look at the losses dictionary for a metric of the network performance. If the number of hidden units is too low, then the model won't have enough space to learn and if it is too high there are too many options for the...
Can you write Python code for it?

import sys

epochs = 5000
learning_rate = 0.005
hidden_nodes = 28
output_nodes = 1

N_i = train_features.shape[1]
network = NeuralNetwork(N_i, hidden_nodes, output_nodes, learning_rate)

losses = {'train':[], 'validation':[]}
for e in range(epochs):
    # Go through a random batch of 128 records from the training data set
    batch = np.random.choice(train_features.index, size=128)
    #batch = train_features.index
    for record, target in zip(train_features.ix[batch].values,
                              train_targets.ix[batch]['cnt']):
        network.train(record, target)

    # Printing out the training progress
    train_loss = MSE(network.run(train_features), train_targets['cnt'].values)
    val_loss = MSE(network.run(val_features), val_targets['cnt'].values)
    sys.stdout.write("\rProgress: " + str(100 * e/float(epochs))[:4] \
                     + "% ... Training loss: " + str(train_loss)[:5] \
                     + " ... Validation loss: " + str(val_loss)[:5])

    losses['train'].append(train_loss)
    losses['validation'].append(val_loss)

plt.plot(losses['train'], label='Training loss')
plt.plot(losses['validation'], label='Validation loss')
plt.legend()
plt.ylim(ymax=0.5)


I am working on the file "1_dlnd-your-first-neural-network/dlnd-your-first-neural-network.ipynb".
The first task is:
We'll split the data into two sets, one for training and one for validating as the network is being trained. Since this is time series data, we'll train on historical data, then try to predict on future data (the validation set).
Can you write Python code for it?

# Hold out the last 60 days of the remaining data as a validation set
train_features, train_targets = features[:-60*24], targets[:-60*24]
val_features, val_targets = features[-60*24:], targets[-60*24:]


I am working on the file "1_foundations/4_simulated_annealing/simulated_annealing.ipynb".
The first task is:
II. Simulated Annealing -- Main Loop
The main loop of simulated annealing repeatedly generates successors in the neighborhood of the current state and considers moving there according to an acceptance probability distribution parameterized by a cooling schedule.  See the simulated-annealing function pseudocode from the AIMA textbook online at github.  Note that our Problem class is already a "node", so the MAKE-NODE line is not required.
Can you write Python code for it?

def simulated_annealing(problem, schedule, rand=False):
    """The simulated annealing algorithm, a version of stochastic hill climbing
    where some downhill moves are allowed. Downhill moves are accepted readily
    early in the annealing schedule and then less often as time goes on. The
    schedule input determines the value of the temperature T as a function of
    time. [Norvig, AIMA Chapter 3]

    Parameters
    ----------
    problem : Problem
        An optimization problem, already initialized to a random starting state.
        The Problem class interface must implement a callable method
        "successors()" which returns states in the neighborhood of the current
        state, and a callable function "get_value()" which returns a fitness
        score for the state. (See the `TravelingSalesmanProblem` class below
        for details.)

    schedule : callable
        A function mapping time to "temperature". "Time" is equivalent in this
        case to the number of loop iterations.

    Returns
    -------
    Problem
        An approximate solution state of the optimization problem

    Notes
    -----
        (1) DO NOT include the MAKE-NODE line from the AIMA pseudocode

        (2) Modify the termination condition to return when the temperature
        falls below some reasonable minimum value (e.g., 1e-10) rather than
        testing for exact equality to zero

    See Also
    --------
    AIMA simulated_annealing() pseudocode
        https://github.com/aimacode/aima-pseudocode/blob/master/md/Simulated-Annealing.md
    """

    """
    function SIMULATED-ANNEALING(problem,schedule) returns a solution state
     inputs: problem, a problem
        schedule, a mapping from time to "temperature"

     current ← MAKE-NODE(problem.INITIAL-STATE)
     for t = 1 to ∞ do
       T ← schedule(t)
       if T = 0 then return current
       next ← a randomly selected successor of current
       ΔE ← next.VALUE - current.VALUE
       if ΔE > 0 then current ← next
     ...


I am working on the file "1_foundations/4_simulated_annealing/simulated_annealing.ipynb".
The first task is:
III. Representing the Problem
In order to use simulated annealing we need to build a representation of the problem domain.  The choice of representation can have a significant impact on the performance of simulated annealing and other optimization techniques.  Since the TSP deals with a close loop that visits each city in a list once, we will represent each city by a tuple containing the city name and its position specified by an (x,y) location on a grid.  The state will then consist of an ordered sequence (a list) of the cities; the path is defined as the sequence generated by traveling from each city in the list to the next in order.
Can you write Python code for it?

Node = Tuple[str, Tuple[int, int]]

class TravelingSalesmanProblem:
    """Representation of a traveling salesman optimization problem.  The goal
    is to find the shortest path that visits every city in a closed loop path.

    Students should only need to implement or modify the successors() and
    get_values() methods.

    Parameters
    ----------
    cities : list
        A list of cities specified by a tuple containing the name and the x, y
        location of the city on a grid. e.g., ("Atlanta", (585.6, 376.8))

    Attributes
    ----------
    names
    coords
    path : list
        The current path between cities as specified by the order of the city
        tuples in the list.
    """
    def __init__(self, cities) -> None:
        self.path = copy.deepcopy(cities)

    def copy(self) -> Any:
        """Return a copy of the current board state."""
        new_tsp = TravelingSalesmanProblem(self.path)
        return new_tsp

    @property
    def names(self) -> Tuple:
        """Strip and return only the city name from each element of the
        path list. For example,
            [("Atlanta", (585.6, 376.8)), ...] -> ["Atlanta", ...]
        """
        names, _ = zip(*self.path)
        return names

    @property
    def coords(self) -> Tuple:
        """Strip the city name from each element of the path list and return
        a list of tuples containing only pairs of xy coordinates for the
        cities. For example,
            [("Atlanta", (585.6, 376.8)), ...] -> [(585.6, 376.8), ...]
        """
        _, coords = zip(*self.path)
        return coords

    def successors(self) -> List[Any]:
        """Return a list of states in the neighborhood of the current state by
        switching the order in which any adjacent pair of cities is visited.

        For example, if the current list of cities (i.e., the path) is [A, B, C, D]
        then the neighbors will include [A, B, D, C], [A, C, B, D], [B, A, C, D],
        and [D, B, C, A]. (The order of...


I am working on the file "1_foundations/4_simulated_annealing/simulated_annealing.ipynb".
The first task is:
IV. Define the Temperature Schedule
The most common temperature schedule is simple exponential decay:
$T(t) = \alpha^t T_0$
(Note that this is equivalent to the incremental form $T_{i+1} = \alpha T_i$, but implementing that form is slightly more complicated because you need to preserve state between calls.)
In most cases, the valid range for temperature $T_0$ can be very high (e.g., 1e8 or higher), and the decay parameter $\alpha$ should be close to, but less than 1.0 (e.g., 0.95 or 0.99).  Think about the ways these parameters effect the simulated annealing function.  Try experimenting with both parameters to see how it changes runtime and the quality of solutions.
You can also experiment with other schedule functions -- linear, quadratic, etc.  Think about the ways that changing the form of the temperature schedule changes the behavior and results of the simulated annealing function.
Can you write Python code for it?

# These are presented as globals so that the signature of schedule()
# matches what is shown in the AIMA textbook; you could alternatively
# define them within the schedule function, use a closure to limit
# their scope, or define an object if you would prefer not to use
# global variables
alpha = 0.95
temperature=1e4

def schedule(time):
    # T(t) =α^t x T0
    return np.power(alpha, time) * temperature


I am working on the file "1_foundations/4_simulated_annealing/simulated_annealing.ipynb".
The first task is:
Testing TravelingSalesmanProblem
The following tests should validate the class constructor and functionality of the successors() and get_value() methods.
Can you write Python code for it?

# Construct an instance of the TravelingSalesmanProblem
test_cities = [('DC', (11, 1)), ('SF', (0, 0)), ('PHX', (2, -3)), ('LA', (0, -4))]
test_names = ('DC', 'SF', 'PHX', 'LA')
test_coords = ((11, 1), (0, 0), (2, -3), (0, -4))
tsp = TravelingSalesmanProblem(test_cities)
assert(tsp.path == test_cities)
assert(tsp.names == test_names)
assert(tsp.coords == test_coords)

# Test the successors() method -- no output means the test passed
successor_paths = [x.path for x in tsp.successors()]
assert(all(x in [[('LA', (0, -4)), ('SF', (0, 0)), ('PHX', (2, -3)), ('DC', (11, 1))],
                 [('SF', (0, 0)), ('DC', (11, 1)), ('PHX', (2, -3)), ('LA', (0, -4))],
                 [('DC', (11, 1)), ('PHX', (2, -3)), ('SF', (0, 0)), ('LA', (0, -4))],
                 [('DC', (11, 1)), ('SF', (0, 0)), ('LA', (0, -4)), ('PHX', (2, -3))]]
          for x in successor_paths))

# Test the get_value() method -- no output means the test passed
assert(np.allclose(tsp.get_value(), -28.97, atol=1e-3))


I am working on the file "1_foundations/4_simulated_annealing/simulated_annealing.ipynb".
The first task is:
V. Run Simulated Annealing on a Larger TSP
Now we are ready to solve a TSP on a bigger problem instance by finding a shortest-path circuit through several of the US state capitals.
You can increase the num_cities parameter up to 30 to experiment with increasingly larger domains.  Try running the solver repeatedly -- how stable are the results?
Can you write Python code for it?

# Create the problem instance and plot the initial state
num_cities = 30
capitals_tsp = TravelingSalesmanProblem(capitals_list[:num_cities])
starting_city = capitals_list[0]
print("Initial path value: {:.2f}".format(-capitals_tsp.get_value()))
print(capitals_list[:num_cities])  # The start/end point is indicated with a yellow star
show_path(capitals_tsp.coords, starting_city)

# set the decay rate and initial temperature parameters, then run simulated annealing to solve the TSP
alpha = 0.99
temperature=1e20
result = simulated_annealing(capitals_tsp, schedule, True)
print("Final path length: {:.2f}".format(-result.get_value()))
print(result.path)
show_path(result.coords, starting_city)


I am working on the file "1_foundations/5_nqueens/constraint_satisfaction.ipynb".
The first task is:
II. Representing the N-Queens Problem
There are many acceptable ways to represent the N-queens problem, but one convenient way is to recognize that one of the constraints (either the row or column constraint) can be enforced implicitly by the encoding.  If we represent a solution as an array with N elements, then each position in the array can represent a column of the board, and the value at each position can represent which row the queen is placed on.
In this encoding, we only need a constraint to make sure that no two queens occupy the same row, and one to make sure that no two queens occupy the same diagonal.
Define Symbolic Expressions for the Problem Constraints
Before implementing the board class, we need to construct the symbolic constraints that will be used in the CSP.  Declare any symbolic terms required, and then declare two generic constraint generators:
- diffRow - generate constraints that return True if the two arguments do not match
- diffDiag - generate constraints that return True if two arguments are not on the same diagonal (Hint: you can easily test whether queens in two columns are on the same diagonal by testing if the difference in the number of rows and the number of columns match)
Both generators should produce binary constraints (i.e., each should have two free symbols) once they're bound to specific variables in the CSP.  For example, Eq((a + b), (b + c)) is not a binary constraint, but Eq((a + b), (b + c)).subs(b, 1) is a binary constraint because one of the terms has been bound to a constant, so there are only two free variables remaining.
Can you write Python code for it?

# Declare any required symbolic variables
r1, r2 = symbols(['r1', 'r2'])
c1, c2 = symbols(['c1', 'c2'])

# Define diffRow and diffDiag constraints
diffRow = constraint('DiffRow', ~Eq(r1, r2))
diffDiag = constraint('DiffDiag', ~Eq(abs(r1 - r2), abs(c1 - c2)))

# Test diffRow and diffDiag
_x = symbols('x:3')

# generate a diffRow instance for testing
diffRow_test = diffRow.subs({r1: _x[0], r2: _x[1]})

assert(len(diffRow_test.free_symbols) == 2)
assert(diffRow_test.subs({_x[0]: 0, _x[1]: 1}) == True)
assert(diffRow_test.subs({_x[0]: 0, _x[1]: 0}) == False)
assert(diffRow_test.subs({_x[0]: 0}) != False)  # partial assignment is not false
print("Passed all diffRow tests.")

# generate a diffDiag instance for testing
diffDiag_test = diffDiag.subs({r1: _x[0], r2: _x[2], c1:0, c2:2})

assert(len(diffDiag_test.free_symbols) == 2)
assert(diffDiag_test.subs({_x[0]: 0, _x[2]: 2}) == False)
assert(diffDiag_test.subs({_x[0]: 0, _x[2]: 0}) == True)
assert(diffDiag_test.subs({_x[0]: 0}) != False)  # partial assignment is not false
print("Passed all diffDiag tests.")


I am working on the file "1_foundations/5_nqueens/constraint_satisfaction.ipynb".
The first task is:
III. Backtracking Search
Implement the backtracking search algorithm (required) and helper functions (optional) from the AIMA text.
Can you write Python code for it?

def select(csp, assignment):
    """Choose an unassigned variable in a constraint satisfaction problem """
    # TODO (Optional): Implement a more sophisticated selection routine from AIMA
    for var in csp.variables:
        if var not in assignment:
            return var
    return None

def order_values(var, assignment, csp):
    """Select the order of the values in the domain of a variable for checking during search;
    the default is lexicographically.
    """
    # TODO (Optional): Implement a more sophisticated search ordering routine from AIMA
    return csp.domains[var]

def backtracking_search(csp):
    """Helper function used to initiate backtracking search """
    return backtrack({}, csp)

def backtrack(assignment, csp):
    """Perform backtracking search for a valid assignment to a CSP

    Parameters
    ----------
    assignment : dict(sympy.Symbol: Integer)
        An partial set of values mapped to variables in the CSP

    csp : CSP
        A problem encoded as a CSP. Interface should include csp.variables, csp.domains,
        csp.inference(), csp.is_consistent(), and csp.is_complete().

    Returns
    -------
    dict(sympy.Symbol: Integer) or None
        A partial set of values mapped to variables in the CSP, or None to indicate failure
    """
    if csp.is_complete(assignment):
        return assignment
    var = select(csp, assignment)
    for value in order_values(var, assignment, csp):
        if csp.is_consistent(var, value, assignment):
            assignment[var] = value
            assignment_copy = copy.deepcopy(assignment)
            result = backtrack(assignment_copy, csp)
            if result is not None:
                return result


I am working on the file "1_foundations/5_nqueens/constraint_satisfaction.ipynb".
The first task is:
Solve the CSP
With backtracking implemented, now you can use it to solve instances of the problem. We've started with the classical 8-queen version, but you can try other sizes as well.  Boards larger than 12x12 may take some time to solve because sympy is slow in the way its being used here, and because the selection and value ordering methods haven't been implemented.  See if you can implement any of the techniques in the AIMA text to speed up the solver!
Can you write Python code for it?

start = timeit.default_timer()
num_queens = 12
csp = NQueensCSP(num_queens)
var = csp.variables[0]
print("CSP problems have variables, each variable has a domain, and the problem has a list of constraints.")
print("Showing the variables for the N-Queens CSP:")
display(csp.variables)
print("Showing domain for {}:".format(var))
display(csp.domains[var])
print("And showing the constraints for {}:".format(var))
display(csp._constraints[var])

print("Solving N-Queens CSP...")
assn = backtracking_search(csp)
if assn is not None:
    csp.show(assn)
    print("Solution found:\n{!s}".format(assn))
else:
    print("No solution found.")

end = timeit.default_timer() - start
print(f'N-Queens size {num_queens} solved in {end} seconds')


I am working on the file "1_foundations/5_nqueens/constraint_satisfaction.ipynb".
The first task is:
The N-Queens CSP Class
Implement the CSP class as described above, with constraints to make sure each queen is on a different row and different diagonal than every other queen, and a variable for each column defining the row that containing a queen in that column.
Can you write Python code for it?

class NQueensCSP:
    """CSP representation of the N-queens problem

    Parameters
    ----------
    N : Integer
        The side length of a square chess board to use for the problem, and
        the number of queens that must be placed on the board
    """
    def __init__(self, N):
        _vars = symbols(f'A0:{N}')
        _domain = set(range(N))
        self.size = N
        self.variables = _vars
        self.domains = {v: _domain for v in _vars}
        self._constraints = {x: set() for x in _vars}

        # add constraints - for each pair of variables xi and xj, create
        # a diffRow(xi, xj) and a diffDiag(xi, xj) instance, and add them
        # to the self._constraints dictionary keyed to both xi and xj;
        # (i.e., add them to both self._constraints[xi] and self._constraints[xj])
        for i in range(N):
            for j in range(i + 1, N):
                diffRowConstraint = diffRow.subs({r1: _vars[i], r2: _vars[j]})
                diffDiagConstraint = diffDiag.subs({r1: _vars[i], r2: _vars[j], c1:i, c2:j})
                self._constraints[_vars[i]].add(diffRowConstraint)
                self._constraints[_vars[i]].add(diffDiagConstraint)
                self._constraints[_vars[j]].add(diffRowConstraint)
                self._constraints[_vars[j]].add(diffDiagConstraint)

    @property
    def constraints(self):
        """Read-only list of constraints -- cannot be used for evaluation """
        constraints = set()
        for _cons in self._constraints.values():
            constraints |= _cons
        return list(constraints)

    def is_complete(self, assignment):
        """An assignment is complete if it is consistent, and all constraints
        are satisfied.

        Hint: Backtracking search checks consistency of each assignment, so checking
        for completeness can be done very efficiently

        Parameters
        ----------
        assignment : dict(sympy.Symbol: Integer)
            An assignment of values to variables that have...


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Another property of lists is that you can put different types in them at the same time.  This can be important to remember if you may have both int and float types.
Can you write Python code for it?

remember = ["2", 2, 2.0]

remember[0] / 1

remember[1] / 1

remember[2] / 1


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Control Flow
if
Most basic logical control
Can you write Python code for it?

x = 4
if x > 5:
    print "x is greater than 5"
elif x < 5:
    print "x is less than 5"
else:
    print "x is equal to 5"


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Finally, one of the more useful list creation functions is range which creates a list with the bounds requested
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Good practice to just add a decimal after any number you really want to treat as a float.
Additional types of numbers include complex, Decimal and Fraction.
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Lets try writing a bit more of a complex (and useful) function.  The Fibinocci sequence is formed by adding the previous two numbers of the sequence to get the next value (starting with [0, 1]).
Can you write Python code for it?

def fibonacci(n):
    """Return a list of the Fibonacci sequence up to n"""
    values = [0, 1]
    while values[-1] <= n:
        values.append(values[-1] + values[-2])
        print values
    return values

fibonacci(100)


def fibonacci(n):
    """Return a list of the Fibonacci sequence up to n"""
    values = [0, 1]
    while values[-1] <= n:
        values.append(values[-1] + values[-2])
    return values

fibonacci(100)


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Lists
One of the most useful data structures in Python is the list.
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Lists are defined with square brackets and delineated by commas.  Note that there is another data type called sequences denoted by ( ) which are immutable (cannot be changed) once created.  Lets try to do some list manipulations with our list of grades above.
Access a single value in a list
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Note here we have left the defined variables as integers as it makes sense that they remain that way (fractional students aside).
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Note that Python is 0 indexed, i.e. the first value in the list is accessed by 0.
Find the length of a list
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Note that Python will make the output a float in this case.  What happens for the following though?
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Note that in order to access these functions we need to prepend the math. to the functions and the constant $\pi$.  We can forgo this and import all of what math holds if we do the following:
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Note that many of these functions always return a float number regardless of their input.
Variables
Assign variables like you would in any other language:
Can you write Python code for it?

num_students = 80
room_capacity = 85
(room_capacity - num_students) / room_capacity * 100.0


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Note that the range of values does not include the last indexed!  This is important to remember for more than lists but we will get to that later.
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Note that to use "named" functions such as sqrt or sin we need to import a module so that we have access to those functions.  When you import a module (or package) in Python we are asking Python to go look for the code that is named and make them active in our workspace (also called a namespace in more general parlance).  Here is an example where we use Python's builtin math module:
Can you write Python code for it?

import math
math.sqrt(4)

math.sin(math.pi / 2.0)

math.exp(-math.pi / 4.0)


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Related to the for statement are the control statements break and continue.  Ideally we can create a loop with logic that can avoid these but sometimes code can be more readable with judiciuos use of these statements.
Can you write Python code for it?

for n in range(2, 10):
    is_prime = True
    for x in range(2, n):
        if n % x == 0:
            print n, 'equals', x, '*', n / x
            is_prime = False
            break
    if is_prime:
        print "%s is a prime number" % (n)


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Slicing is another important operation
Can you write Python code for it?

grades[2:5]

grades[0:4]

grades[:4]

grades[4:]


I am working on the file "1_intro_to_python.ipynb".
The first task is:
The pass statement might appear fairly useless as it simply does nothing but can provide a stub to remember to come back and implement something
Can you write Python code for it?

def my_func(x):
    # Remember to implement this later!
    pass


I am working on the file "1_intro_to_python.ipynb".
The first task is:
Why do we get the answer above rather than what we would expect?
The answer has to do with the type of number being used.  Python is a "dynamically" typed language and automatically determines what kind of number to allocate for us.  Above, because we did not include a decimal, Python automatically treated the expression as integers (int type) and according to integer arithmetic, 1 / 2 = 0.  Now if we include a decimal we get:
Can you write Python code for it?


I am working on the file "1_intro_to_python.ipynb".
The first task is:
for
The for statements provide the most common type of loops in Python (there is also a while construct).
Can you write Python code for it?

for i in range(5):
    print i

for i in range(3,7):
    print i

for animal in ['cat', 'dog', 'chinchilla']:
    print animal


I am working on the file "1_introduction.ipynb".
The first task is:

Tutorial N1. Double Bend Achromat.
We designed a simple lattice to demonstrate the basic concepts and syntax of the optics functions calculation.
Also, we chose DBA to demonstrate the periodic solution for the optical functions calculation.
Can you write Python code for it?

from __future__ import print_function

# the output of plotting commands is displayed inline within frontends,
# directly below the code cell that produced it
%matplotlib inline

# import from Ocelot main modules and functions
from ocelot import *

# import from Ocelot graphical modules
from ocelot.gui.accelerator import *


I am working on the file "1_introduction.ipynb".
The first task is:
Creating lattice
Ocelot has following elements: Drift, Quadrupole, Sextupole, Octupole, Bend, SBend, RBend, Edge, Multipole, Hcor, Vcor, Solenoid, Cavity, Monitor, Marker, Undulator.
Can you write Python code for it?

# defining of the drifts
D1 = Drift(l=2.)
D2 = Drift(l=0.6)
D3 = Drift(l=0.3)
D4 = Drift(l=0.7)
D5 = Drift(l=0.9)
D6 = Drift(l=0.2)

# defining of the quads
Q1 = Quadrupole(l=0.4, k1=-1.3)
Q2 = Quadrupole(l=0.8, k1=1.4)
Q3 = Quadrupole(l=0.4, k1=-1.7)
Q4 = Quadrupole(l=0.5, k1=1.3)

# defining of the bending magnet
B = Bend(l=2.7, k1=-.06, angle=2*pi/16., e1=pi/16., e2=pi/16.)

# defining of the sextupoles
SF = Sextupole(l=0.01, k2=1.5) #random value
SD = Sextupole(l=0.01, k2=-1.5) #random value

# cell creating
cell = (D1, Q1, D2, Q2, D3, Q3, D4, B, D5, SD, D5, SF, D6, Q4, D6, SF, D5, SD, D5, B, D4, Q3, D3, Q2, D2, Q1, D1)


I am working on the file "1_introduction.ipynb".
The first task is:
Optical function calculation
Uses:
* twiss() function and,
* Twiss() object contains twiss parameters and other information at one certain position (s) of lattice
To calculate twiss parameters you have to run twiss(lattice, tws0=None, nPoints=None) function. If you want to get a periodic solution leave tws0 by default.
You can change the number of points over the cell, If nPoints=None, then twiss parameters are calculated at the end of each element.
twiss() function returns list of Twiss() objects.
You will see the Twiss object contains more information than just twiss parameters.
Can you write Python code for it?

tws=twiss(lat)

# to see twiss paraments at the begining of the cell, uncomment next line
# print(tws[0])

# to see twiss paraments at the end of the cell, uncomment next line
print(tws[-1])

len(tws)

# plot optical functions.
plot_opt_func(lat, tws, top_plot = ["Dx", "Dy"], legend=False, font_size=10)
plt.show()

# you also can use standard matplotlib functions for plotting
#s = [tw.s for tw in tws]
#bx = [tw.beta_x for tw in tws]
#plt.plot(s, bx)
#plt.show()

# you can play with quadrupole strength and try to make achromat
Q4.k1 = 1.18

# to make achromat uncomment next line
# Q4.k1 =  1.18543769836
# To use matching function, please see ocelot/demos/ebeam/dba.py

# updating transfer maps after changing element parameters.
lat.update_transfer_maps()

# recalculate twiss parameters
tws=twiss(lat, nPoints=1000)

plot_opt_func(lat, tws, legend=False)
plt.show()


I am working on the file "1_introduction.ipynb".
The first task is:
Outline

Preliminaries: Setup & introduction

Beam dynamics

Tutorial N1. Linear optics.. Web version.
Linear optics. DBA.

Tutorial N2. Tracking.. Web version.
Linear optics of the European XFEL Injector
Tracking. First and second order.

Tutorial N3. Space Charge.. Web version.
Tracking with SC effects.

Tutorial N4. Wakefields.. Web version.
Tracking with Wakefields

FEL calculation

Tutorial N5: Genesis preprocessor. Web version.
Tutorial N6. Genesis postprocessor. Web version.

All IPython (jupyter) notebooks (.ipynb) have analogues in the form of python scripts (.py).
All these notebooks as well as additional files (beam distribution, wakes, ...) you can download here.
Preliminaries
The tutorial includes 4 simple examples dediacted to beam dynamics. However, you should have a basic understanding of Computer Programming terminologies. A basic understanding of Python language is a plus.
This tutorial requires the following packages:

Python version 2.7 or 3.4-3.5
numpy version 1.8 or later: http://www.numpy.org/
scipy version 0.15 or later: http://www.scipy.org/
matplotlib version 1.5 or later: http://matplotlib.org/
ipython version 2.4 or later, with notebook support: http://ipython.org

The easiest way to get these is to download and install the (very large) Anaconda software distribution.
Alternatively, you can download and install miniconda.
The following command will install all required packages:
$ conda install numpy scipy matplotlib ipython-notebook
Ocelot installation

you have to download from GitHub zip file.
Unzip ocelot-master.zip to your working folder ../your_working_dir/.
Rename folder ../your_working_dir/ocelot-master to ../your_working_dir/ocelot.

Add ../your_working_dir/ to PYTHONPATH

Windows 7: go to Control Panel -> System and Security -> System -> Advance System Settings -> Environment Variables.
and in User variables add ../your_working_dir/ to PYTHONPATH. If variable PYTHONPATH does not exist, create it

Variable name: PYTHONPATH
Variable value: ../your_working_dir/
-...
Can you write Python code for it?

import IPython
print('IPython:', IPython.__version__)

import numpy
print('numpy:', numpy.__version__)

import scipy
print('scipy:', scipy.__version__)

import matplotlib
print('matplotlib:', matplotlib.__version__)

import ocelot
print('ocelot:', ocelot.__version__)


I am working on the file "1_introduction.ipynb".
The first task is:
hint: to see a simple description of the function put cursor inside () and press Shift-Tab or you can type sign ? before function. To extend dialog window press +* *
The cell is a list of the simple objects which contain a physical information of lattice elements such as length, strength, voltage and so on. In order to create a transport map for every element and bind it with lattice object we have to create new Ocelot object - MagneticLattice() which makes these things automatically.
MagneticLattice(sequence, start=None, stop=None, method=MethodTM()):
* sequence - list of the elements,
other paramenters we will consider in tutorial N2.
Can you write Python code for it?

lat = MagneticLattice(cell)

# to see total lenth of the lattice
print("length of the cell: ", lat.totalLen, "m")


I am working on the file "1_lines/P1.ipynb".
The first task is:
At this point, if you were successful you probably have the Hough line segments drawn onto the road, but what about identifying the full extent of the lane and marking it clearly as in the example video (P1_example.mp4)?  Think about defining a line to run the full length of the visible lane based on the line segments you identified with the Hough Transform.  Modify your draw_lines function accordingly and try re-running your pipeline.
Now for the one with the solid yellow lane on the left. This one's more tricky!
Can you write Python code for it?

yellow_output = 'yellow.mp4'
clip2 = VideoFileClip('solidYellowLeft.mp4')
yellow_clip = clip2.fl_image(process_image)
%time yellow_clip.write_videofile(yellow_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(yellow_output))


I am working on the file "1_lines/P1.ipynb".
The first task is:
Let's try the one with the solid white lane on the right first ...
Can you write Python code for it?

white_output = 'white.mp4'
clip1 = VideoFileClip("solidWhiteRight.mp4")
white_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)


I am working on the file "1_lines/P1.ipynb".
The first task is:
Play the video inline, or if you prefer find the video in your filesystem (should be in the same directory) and play it in your video player of choice.
Can you write Python code for it?

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(white_output))


I am working on the file "1_lines/P1.ipynb".
The first task is:
Reflections
Congratulations on finding the lane lines!  As the final step in this project, we would like you to share your thoughts on your lane finding pipeline... specifically, how could you imagine making your algorithm better / more robust?  Where will your current algorithm be likely to fail?
Please add your thoughts below,  and if you're up for making your pipeline more robust, be sure to scroll down and check out the optional challenge video below!
Submission
If you're satisfied with your video outputs it's time to submit!  Submit this ipython notebook for review.
Optional Challenge
Try your lane finding pipeline on the video below.  Does it still work?  Can you figure out a way to make it more robust?  If you're up for the challenge, modify your pipeline so it works with this video and submit it along with the rest of your project!
The current pipeline doesn't work with this video for several reasons:
1) The video size is bigger than expected (1280x720 instead of 960x540)
2) The video shows part of the car's hood (with its edges being detected) (video is not centered in the same fashion)
3) The highway has different 'patches' with different colors, leading to even more edges detected
To improve the pipeline's robustness, these things among others should be addressed. A possible strategy would be:
1) and 2) can be address with cropping centered (see process_image)
3) should be address in a different way (maybe a low/band/high pass filter?, I tried applying a Gaussian filter after selecting the region of interest with little result)
Can you write Python code for it?

challenge_output = 'extra.mp4'
clip2 = VideoFileClip('challenge.mp4')
challenge_clip = clip2.fl_image(process_image)
%time challenge_clip.write_videofile(challenge_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(challenge_output))


I am working on the file "1_lines/P1.ipynb".
The first task is:
Some OpenCV functions (beyond those introduced in the lesson) that might be useful for this project are:
cv2.inRange() for color selection
cv2.fillPoly() for regions selection
cv2.line() to draw lines on an image given endpoints
cv2.addWeighted() to coadd / overlay two images
cv2.cvtColor() to grayscale or change color
cv2.imwrite() to output images to file
cv2.bitwise_and() to apply a mask to an image
Check out the OpenCV documentation to learn about these and discover even more awesome functionality!
Below are some helper functions to help get you started. They should look familiar from the lesson!
Can you write Python code for it?

import math

def grayscale(img):
    """Applies the Grayscale transform
    This will return an image with only one color channel
    but NOTE: to see the returned image as grayscale
    you should call plt.imshow(gray, cmap='gray')"""
    return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

def canny(img, low_threshold, high_threshold):
    """Applies the Canny transform"""
    return cv2.Canny(img, low_threshold, high_threshold)

def gaussian_blur(img, kernel_size):
    """Applies a Gaussian Noise kernel"""
    return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)

def region_of_interest(img, vertices):
    """
    Applies an image mask.

    Only keeps the region of the image defined by the polygon
    formed from `vertices`. The rest of the image is set to black.
    """
    #defining a blank mask to start with
    mask = np.zeros_like(img)

    #defining a 3 channel or 1 channel color to fill the mask with depending on the input image
    if len(img.shape) > 2:
        channel_count = img.shape[2]  # i.e. 3 or 4 depending on your image
        ignore_mask_color = (255,) * channel_count
    else:
        ignore_mask_color = 255

    #filling pixels inside the polygon defined by "vertices" with the fill color
    cv2.fillPoly(mask, vertices, ignore_mask_color)

    #returning the image only where mask pixels are nonzero
    masked_image = cv2.bitwise_and(img, mask)
    return masked_image

def draw_lines_naive(img, lines, color=[255, 0, 0], thickness=2):
    for line in lines:
        for x1,y1,x2,y2 in line:
            cv2.line(img, (x1, y1), (x2, y2), color, thickness)

def draw_lines(img, lines, color=[255, 0, 0], thickness=10):
    """
    NOTE: this is the function you might want to use as a starting point once you want to
    average/extrapolate the line segments you detect to map out the full
    extent of the lane (going from the result shown in raw-lines-example.mp4
    to that shown in P1_example.mp4).

    Think about things like separating line segments by...


I am working on the file "1_lines/P1.ipynb".
The first task is:
run your solution on all test_images and make copies into the test_images directory).
Test on Videos
You know what's cooler than drawing lanes over images? Drawing lanes over video!
We can test our solution on two provided videos:
solidWhiteRight.mp4
solidYellowLeft.mp4
Can you write Python code for it?

# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML

def process_image(image):
    # NOTE: The output you return should be a color image (3 channel) for processing video below
    # TODO: put your pipeline here,
    # you should return the final output (image with lines are drawn on lanes)

    h, w, c = image.shape
    H, W = 540, 960

    #this helps in examples like 'challenge.mp4', where the video is larger than expected
    #crops the image centered
    if (w > W or h > H):
        OFF_H = (h - H)/2
        OFF_W = (w - W)/2
        image = image[h-OFF_H-H:h-OFF_H, w-OFF_W-W:w-OFF_W]

    gray = grayscale(image)

    gauss_img = gaussian_blur(gray, 5)
    canny_img = canny(gauss_img, 50, 150)

    vertices = [[450, 320], [520, 320], [W, H], [0, H]];
    interest_img = region_of_interest(canny_img, np.int32([vertices]))

    #maybe something like this could help smooth for the different colors in the highway?
    #interest_img = gaussian_blur(interest_img, 7)

    hough_img = hough_lines(interest_img, 2, np.pi/180, 80, 120, 40)

    result = weighted_img(image, hough_img)
    return result

#reading in an image
image = mpimg.imread('test_images/solidYellowLeft.jpg')
plt.imshow(process_image(image))

import os
directory = "test_images"
files = os.listdir(directory)
for f in files:
    if ((not f.startswith("new")) and f.endswith(".jpg")):
        image = mpimg.imread(directory + "/"+f)
        new_image = process_image(image)
        plt.imshow(new_image)
        plt.savefig(directory + "/" + "new_"+f)


I am working on the file "1_modeling/1_modeling.ipynb".
The first task is:
LM = me.LagrangesMethod(ラグランジアン, [一般化座標の配列]) という関数で，ラグランジュの運動方程式を定義する．
LM.form_lagranges_equations() でラグランジュの運動方程式が出力される．
Can you write Python code for it?

''' Calculating the eom '''
LM = me.LagrangesMethod(L, [theta])

print(LM.form_lagranges_equations())


I am working on the file "1_modeling/1_modeling.ipynb".
The first task is:
摩擦 という要素を考えることによって運動の様相が変化したことがわかる．
シミュレーションを行う上では，考えたい物理モデルをどのようにモデル化するかによって，得られる結果が大きく変わる．所望のシミュレーションを行うためには，十分な力学の知識が必要となる．
Lagrange の運動方程式
0_quickstartでは，単振り子の運動方程式をニュートンの運動方程式から求めたが，今度は ラグランジュの運動方程式 から求める．
おもりの運動エネルギーは
\begin{align}
  T = \frac{1}{2}m(l\dot{\theta})^2
\end{align}
であり，ポテンシャルエネルギー（位置エネルギー）は
\begin{align}
  U = - m(-g)(l-l\cos\theta) = mgl(1 - \cos\theta)
\end{align}
である．したがって，系のラグランジアンは
\begin{align}
  L = T - U = \frac{1}{2}m(l\dot{\theta})^2 - mgl(1 - \cos\theta)
\end{align}
であり，ラグランジュの運動方程式は
\begin{align}
  \frac{d}{dt}\left( \frac{\partial L}{\partial \dot{\theta}} \right) - \frac{\partial L}{\partial \theta} = 0
\end{align}
である．項を一つ一つ丁寧に計算をすると，
\begin{align}
  \frac{\partial L}{\partial \dot{\theta}} = \frac{\partial }{\partial \dot{\theta}} \left( \frac{1}{2}m(l\dot{\theta})^2 - mgl(1 - \cos\theta) \right) = ml^2\dot{\theta}
\end{align}
\begin{align}
  \frac{d}{dt}\left( \frac{\partial L}{\partial \dot{\theta}} \right) = \frac{d}{dt} (ml^2\dot{\theta}) = ml^2\ddot{\theta}
\end{align}
\begin{align}
  \frac{\partial L}{\partial \theta} = \frac{\partial }{\partial \theta} \left( \frac{1}{2}m(l\dot{\theta})^2 - mgl(1 - \cos\theta) \right) = -mgl \sin\theta
\end{align}
より，
\begin{align}
  \frac{d}{dt}\left( \frac{\partial L}{\partial \dot{\theta}} \right) - \frac{\partial L}{\partial \theta} = ml^2\ddot{\theta} - (-mgl \sin\theta) = 0
\end{align}
よって，
\begin{align}
 ml^2\ddot{\theta} + mgl \sin\theta = 0
\end{align}
である．式を整理すると，
\begin{align}
\ddot{\theta} = -\frac{g}{l} \sin\theta
\end{align}
となっており，ニュートンの運動方程式から導出したものと同じ結果が得られたことがわかる．
Lagrange の運動方程式を SymPy で計算する
Lagrange の運動方程式は運動の自由度についてのミニマムな運動方程式を記述することができる．しかし，ラグランジアンとその偏微分の計算は複雑になりがちである．単振り子の例については，運動の自由度は1であり，かつ非常にシンプルな状況であるため手で計算してもよいのだが，これが他リンク系となったり，運動を2次元から3次元に拡張したりしたときには，もはや手計算で求める気力が起こらない．
そこで， Python を使って Lagrange の運動方程式を導く． SymPy の LagrangesMethod クラス を用いる．
Can you write Python code for it?

import sympy as sym
import sympy.physics.mechanics as me


I am working on the file "1_notmnist.ipynb".
The first task is:
Extract the dataset from the compressed .tar.gz file.
This should give you a set of directories, labelled A through J.
Can you write Python code for it?

num_classes = 10
np.random.seed(133)

def maybe_extract(filename, force=False):
  root = os.path.splitext(os.path.splitext(filename)[0])[0]  # remove .tar.gz
  if os.path.isdir(root) and not force:
    # You may override by setting force=True.
    print('%s already present - Skipping extraction of %s.' % (root, filename))
  else:
    print('Extracting data for %s. This may take a while. Please wait.' % root)
    tar = tarfile.open(filename)
    sys.stdout.flush()
    tar.extractall()
    tar.close()
  data_folders = [
    os.path.join(root, d) for d in sorted(os.listdir(root))
    if os.path.isdir(os.path.join(root, d))]
  if len(data_folders) != num_classes:
    raise Exception(
      'Expected %d folders, one per class. Found %d instead.' % (
        num_classes, len(data_folders)))
  print(data_folders)
  return data_folders

train_folders = maybe_extract(train_filename)
test_folders = maybe_extract(test_filename)


num_classes = 10

def extract(filename):
  tar = tarfile.open(filename)
  tar.extractall()
  tar.close()
  root = os.path.splitext(os.path.splitext(filename)[0])[0]  # remove .tar.gz
  data_folders = [os.path.join(root, d) for d in sorted(os.listdir(root))]
  if len(data_folders) != num_classes:
    raise Exception(
      'Expected %d folders, one per class. Found %d instead.' % (
        num_folders, len(data_folders)))
  print (data_folders)
  return data_folders

train_folders = extract(train_filename)
test_folders = extract(test_filename)


num_classes = 10

def maybe_extract(filename, force=False):
  root = os.path.splitext(os.path.splitext(filename)[0])[0]  # remove .tar.gz
  if os.path.isdir(root) and not force:
    # You may override by setting force=True.
    print('%s already present - Skipping extraction of %s.' % (root, filename))
  else:
    print('Extracting data for %s. This may take a while. Please wait.' % root)
    tar = tarfile.open(filename)
    sys.stdout.flush()
    tar.extractall()
    tar.close()
  data_folders = [
    os.path.join(root, d) for d in sorted(os.listdir(root))
    if os.path.isdir(os.path.join(root, d))]
  if len(data_folders) != num_classes:
    raise Exception(
      'Expected %d folders, one per class. Found %d instead.' % (
        num_classes, len(data_folders)))
  print(data_folders)
  return data_folders

train_folders = maybe_extract(train_filename)
test_folders = maybe_extract(test_filename)


I am working on the file "1_notmnist.ipynb".
The first task is:
Finally, let's save the data for later reuse:
Can you write Python code for it?

pickle_file = 'notMNIST.pickle'

try:
  f = open(pickle_file, 'wb')
  save = {
    'train_dataset': train_dataset,
    'train_labels': train_labels,
    'valid_dataset': valid_dataset,
    'valid_labels': valid_labels,
    'test_dataset': test_dataset,
    'test_labels': test_labels,
    }
  pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
  f.close()
except Exception as e:
  print('Unable to save data to', pickle_file, ':', e)
  raise

statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)


pickle_file = os.path.join(data_root, 'notMNIST.pickle')

try:
  f = open(pickle_file, 'wb')
  save = {
    'train_dataset': train_dataset,
    'train_labels': train_labels,
    'valid_dataset': valid_dataset,
    'valid_labels': valid_labels,
    'test_dataset': test_dataset,
    'test_labels': test_labels,
    }
  pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
  f.close()
except Exception as e:
  print('Unable to save data to', pickle_file, ':', e)
  raise

statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)


pickle_file = 'notMNIST.pickle'

try:
  f = open(pickle_file, 'wb')
  save = {
    'train_dataset': train_dataset,
    'train_labels': train_labels,
    'valid_dataset': valid_dataset,
    'valid_labels': valid_labels,
    'test_dataset': test_dataset,
    'test_labels': test_labels,
    }
  pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
  f.close()
except Exception as e:
  print('Unable to save data to', pickle_file, ':', e)
  raise
print("Finished")

statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)


I am working on the file "1_notmnist.ipynb".
The first task is:
First, we'll download the dataset to our local machine. The data consists of characters rendered in a variety of fonts on a 28x28 image. The labels are limited to 'A' through 'J' (10 classes). The training set has about 500k and the testset 19.000 labelled examples. Given these sizes, it should be possible to train models quickly on any machine.
Can you write Python code for it?

url = 'http://yaroslavvb.com/upload/notMNIST/'

def maybe_download(filename, expected_bytes, force=False):
  """Download a file if not present, and make sure it's the right size."""
  if force or not os.path.exists(filename):
    filename, _ = urlretrieve(url + filename, filename)
  statinfo = os.stat(filename)
  if statinfo.st_size == expected_bytes:
    print('Found and verified', filename)
  else:
    raise Exception(
      'Failed to verify' + filename + '. Can you get to it with a browser?')
  return filename

train_filename = maybe_download('notMNIST_large.tar.gz', 247336696)
test_filename = maybe_download('notMNIST_small.tar.gz', 8458043)


I am working on the file "1_notmnist.ipynb".
The first task is:
First, we'll download the dataset to our local machine. The data consists of characters rendered in a variety of fonts on a 28x28 image. The labels are limited to 'A' through 'J' (10 classes). The training set has about 500k and the testset 19000 labelled examples. Given these sizes, it should be possible to train models quickly on any machine.
Can you write Python code for it?


url = 'http://yaroslavvb.com/upload/notMNIST/'

def maybe_download(filename, expected_bytes):
  """Download a file if not present, and make sure it's the right size."""
  if not os.path.exists(filename):

    filename, _ = urllib.urlretrieve(url + filename, filename)
  statinfo = os.stat(filename)
  if statinfo.st_size == expected_bytes:
    print ('Found and verified', filename)
  else:
    raise Exception('Failed to verify' + filename + '. Can you get to it with a browser?')
  return filename

train_filename = maybe_download('notMNIST_large.tar.gz', 247336696)
test_filename = maybe_download('notMNIST_small.tar.gz', 8458043)


I am working on the file "1_notmnist.ipynb".
The first task is:
From my perspective near duplicates should also be removed in the sanitized datasets. My assumption is that "near" duplicates are very very close (sometimes just there is a one pyxel border of difference), and penalyze the training the same way the true duplicates do.
That's being said, finding near duplicates with a hash function is not obvious. There are techniques for that, like "locally sensitive hashing", "perceptual hashing" or "difference hashing". There even are Python library available. Unfortunatly I did not have time to try them. The sanitized dataset generated below are based on true duplicates found with a cryptography hash function.
For sanitizing the dataset, I change the function above by returning the clean dataset directly.
Can you write Python code for it?

def sanetize(dataset_1, dataset_2, labels_1):
  dataset_hash_1 = np.array([hashlib.sha256(img).hexdigest() for img in dataset_1])
  dataset_hash_2 = np.array([hashlib.sha256(img).hexdigest() for img in dataset_2])
  overlap = [] # list of indexes
  for i, hash1 in enumerate(dataset_hash_1):
    duplicates = np.where(dataset_hash_2 == hash1)
    if len(duplicates[0]):
      overlap.append(i)
  return np.delete(dataset_1, overlap, 0), np.delete(labels_1, overlap, None)

%time test_dataset_sanit, test_labels_sanit = sanetize(test_dataset[:200], train_dataset, test_labels[:200])
print('Overlapping images removed: ', len(test_dataset[:200]) - len(test_dataset_sanit))


I am working on the file "1_notmnist.ipynb".
The first task is:
It is a faster, and only one duplicate from the second dataset is displayed. This is still not scalable.
Can you write Python code for it?

MAX_MANHATTAN_NORM = 10

def extract_overlap_where(dataset_1, dataset_2):
  overlap = {}
  for i, img_1 in enumerate(dataset_1):
    diff = dataset_2 - img_1
    norm = np.sum(np.abs(diff), axis=1)
    duplicates = np.where(norm < MAX_MANHATTAN_NORM)
    if len(duplicates[0]):
      overlap[i] = duplicates[0]
  return overlap

test_flat = test_dataset.reshape(test_dataset.shape[0], 28 * 28)
train_flat = train_dataset.reshape(train_dataset.shape[0], 28 * 28)
%time overlap_test_train = extract_overlap_where(test_flat[:200], train_flat)

print('Number of overlaps:', len(overlap_test_train.keys()))
display_overlap(overlap_test_train, test_dataset[:200], train_dataset)


I am working on the file "1_notmnist.ipynb".
The first task is:
Merge and prune the training data as needed. Depending on your computer setup, you might not be able to fit it all in memory, and you can tune train_size as needed. The labels will be stored into a separate array of integers 0 through 9.
Also create a validation dataset for hyperparameter tuning.
Can you write Python code for it?

def make_arrays(nb_rows, img_size):
  if nb_rows:
    dataset = np.ndarray((nb_rows, img_size, img_size), dtype=np.float32)
    labels = np.ndarray(nb_rows, dtype=np.int32)
  else:
    dataset, labels = None, None
  return dataset, labels

def merge_datasets(pickle_files, train_size, valid_size=0):
  num_classes = len(pickle_files)
  valid_dataset, valid_labels = make_arrays(valid_size, image_size)
  train_dataset, train_labels = make_arrays(train_size, image_size)
  vsize_per_class = valid_size // num_classes
  tsize_per_class = train_size // num_classes

  start_v, start_t = 0, 0
  end_v, end_t = vsize_per_class, tsize_per_class
  end_l = vsize_per_class+tsize_per_class
  for label, pickle_file in enumerate(pickle_files):
    try:
      with open(pickle_file, 'rb') as f:
        letter_set = pickle.load(f)
        # let's shuffle the letters to have random validation and training set
        np.random.shuffle(letter_set)
        if valid_dataset is not None:
          valid_letter = letter_set[:vsize_per_class, :, :]
          valid_dataset[start_v:end_v, :, :] = valid_letter
          valid_labels[start_v:end_v] = label
          start_v += vsize_per_class
          end_v += vsize_per_class

        train_letter = letter_set[vsize_per_class:end_l, :, :]
        train_dataset[start_t:end_t, :, :] = train_letter
        train_labels[start_t:end_t] = label
        start_t += tsize_per_class
        end_t += tsize_per_class
    except Exception as e:
      print('Unable to process data from', pickle_file, ':', e)
      raise

  return valid_dataset, valid_labels, train_dataset, train_labels

train_size = 200000
valid_size = 10000
test_size = 10000

valid_dataset, valid_labels, train_dataset, train_labels = merge_datasets(
  train_datasets, train_size, valid_size)
_, _, test_dataset, test_labels = merge_datasets(test_datasets, test_size)

print('Training:', train_dataset.shape, train_labels.shape)
print('Validation:', valid_dataset.shape,...


I am working on the file "1_notmnist.ipynb".
The first task is:
More overlapping values could be found, this is due to the hash collisions. Several images can have the same hash but are actually different differents. This is not noticed here, and even if it happens, this is acceptable. All duplicates will be removed for sure.
We can make the processing a but faster by using the built-in numpy wherefunction.
Can you write Python code for it?

def extract_overlap_hash_where(dataset_1, dataset_2):
  dataset_hash_1 = np.array([hashlib.sha256(img).hexdigest() for img in dataset_1])
  dataset_hash_2 = np.array([hashlib.sha256(img).hexdigest() for img in dataset_2])
  overlap = {}
  for i, hash1 in enumerate(dataset_hash_1):
    duplicates = np.where(dataset_hash_2 == hash1)
    if len(duplicates[0]):
      overlap[i] = duplicates[0]
  return overlap

%time overlap_test_train = extract_overlap_hash_where(test_dataset[:200], train_dataset)

print('Number of overlaps:', len(overlap_test_train.keys()))
display_overlap(overlap_test_train, test_dataset[:200], train_dataset)


I am working on the file "1_notmnist.ipynb".
The first task is:
Next, we'll randomize the data. It's important to have the labels well shuffled for the training and test distributions to match.
Can you write Python code for it?

def randomize(dataset, labels):
  permutation = np.random.permutation(labels.shape[0])
  shuffled_dataset = dataset[permutation,:,:]
  shuffled_labels = labels[permutation]
  return shuffled_dataset, shuffled_labels
train_dataset, train_labels = randomize(train_dataset, train_labels)
test_dataset, test_labels = randomize(test_dataset, test_labels)
valid_dataset, valid_labels = randomize(valid_dataset, valid_labels)


np.random.seed(133)
def randomize(dataset, labels):
  permutation = np.random.permutation(labels.shape[0])
  shuffled_dataset = dataset[permutation,:,:]
  shuffled_labels = labels[permutation]
  return shuffled_dataset, shuffled_labels
train_dataset, train_labels = randomize(train_dataset, train_labels)
test_dataset, test_labels = randomize(test_dataset, test_labels)


I am working on the file "1_notmnist.ipynb".
The first task is:
Now let's load the data in a more manageable format. Since, depending on your computer setup you might not be able to fit it all in memory, we'll load each class into a separate dataset, store them on disk and curate them independently. Later we'll merge them into a single dataset of manageable size.
We'll convert the entire dataset into a 3D array (image index, x, y) of floating point values, normalized to have approximately zero mean and standard deviation ~0.5 to make training easier down the road.
A few images might not be readable, we'll just skip them.
Can you write Python code for it?

image_size = 28  # Pixel width and height.
pixel_depth = 255.0  # Number of levels per pixel.

def load_letter(folder, min_num_images):
  """Load the data for a single letter label."""
  image_files = os.listdir(folder)
  dataset = np.ndarray(shape=(len(image_files), image_size, image_size),
                         dtype=np.float32)
  image_index = 0
  print(folder)
  for image in os.listdir(folder):
    image_file = os.path.join(folder, image)
    try:
      image_data = (ndimage.imread(image_file).astype(float) -
                    pixel_depth / 2) / pixel_depth
      if image_data.shape != (image_size, image_size):
        raise Exception('Unexpected image shape: %s' % str(image_data.shape))
      dataset[image_index, :, :] = image_data
      image_index += 1
    except IOError as e:
      print('Could not read:', image_file, ':', e, '- it\'s ok, skipping.')

  num_images = image_index
  dataset = dataset[0:num_images, :, :]
  if num_images < min_num_images:
    raise Exception('Many fewer images than expected: %d < %d' %
                    (num_images, min_num_images))

  print('Full dataset tensor:', dataset.shape)
  print('Mean:', np.mean(dataset))
  print('Standard deviation:', np.std(dataset))
  return dataset

def maybe_pickle(data_folders, min_num_images_per_class, force=False):
  dataset_names = []
  for folder in data_folders:
    set_filename = folder + '.pickle'
    dataset_names.append(set_filename)
    if os.path.exists(set_filename) and not force:
      # You may override by setting force=True.
      print('%s already present - Skipping pickling.' % set_filename)
    else:
      print('Pickling %s.' % set_filename)
      dataset = load_letter(folder, min_num_images_per_class)
      try:
        with open(set_filename, 'wb') as f:
          pickle.dump(dataset, f, pickle.HIGHEST_PROTOCOL)
      except Exception as e:
        print('Unable to save data to', set_filename, ':', e)

  return dataset_names

train_datasets = maybe_pickle(train_folders, 45000)
test_datasets =...


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 1
Let's take a peek at some of the data to make sure it looks sensible. Each exemplar should be an image of a character A through J rendered in a different font. Display a sample of the images that we just downloaded. Hint: you can use the package IPython.display.

First of all, let's import some libraries that I will use later on and activate online display of matplotlib outputs:
Can you write Python code for it?

import random
import hashlib
%matplotlib inline

def disp_samples(data_folders, sample_size):
  for folder in data_folders:
    print(folder)
    image_files = os.listdir(folder)
    image_sample = random.sample(image_files, sample_size)
    for image in image_sample:
      image_file = os.path.join(folder, image)
      i = Image(filename=image_file)
      display(i)

disp_samples(train_folders, 1)

disp_samples(test_folders, 1)


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 1
Let's take a peek at some of the data to make sure it looks sensible. Each exemplar should be an image of a character A through J rendered in a different font. Display a sample of the images that we just downloaded. Hint: you can use the package IPython.display.

Now let's load the data in a more manageable format. Since, depending on your computer setup you might not be able to fit it all in memory, we'll load each class into a separate dataset, store them on disk and curate them independently. Later we'll merge them into a single dataset of manageable size.
We'll convert the entire dataset into a 3D array (image index, x, y) of floating point values, normalized to have approximately zero mean and standard deviation ~0.5 to make training easier down the road.
A few images might not be readable, we'll just skip them.
Can you write Python code for it?


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 2
Let's verify that the data still looks good. Displaying a sample of the labels and images from the ndarray. Hint: you can use matplotlib.pyplot.

Problem 3
Another check: we expect the data to be balanced across classes. Verify that.

Merge and prune the training data as needed. Depending on your computer setup, you might not be able to fit it all in memory, and you can tune train_size as needed. The labels will be stored into a separate array of integers 0 through 9.
Also create a validation dataset for hyperparameter tuning.
Can you write Python code for it?


def make_arrays(nb_rows, img_size):
  if nb_rows:
    dataset = np.ndarray((nb_rows, img_size, img_size), dtype=np.float32)
    labels = np.ndarray(nb_rows, dtype=np.int32)
  else:
    dataset, labels = None, None
  return dataset, labels

def merge_datasets(pickle_files, train_size, valid_size=0):
  num_classes = len(pickle_files)
  valid_dataset, valid_labels = make_arrays(valid_size, image_size)
  train_dataset, train_labels = make_arrays(train_size, image_size)
  vsize_per_class = valid_size // num_classes
  tsize_per_class = train_size // num_classes

  start_v, start_t = 0, 0
  end_v, end_t = vsize_per_class, tsize_per_class
  end_l = vsize_per_class+tsize_per_class
  for label, pickle_file in enumerate(pickle_files):
    try:
      with open(pickle_file, 'rb') as f:
        letter_set = pickle.load(f)
        if valid_dataset is not None:
          valid_letter = letter_set[:vsize_per_class, :, :]
          valid_dataset[start_v:end_v, :, :] = valid_letter
          valid_labels[start_v:end_v] = label
          start_v += vsize_per_class
          end_v += vsize_per_class

        train_letter = letter_set[vsize_per_class:end_l, :, :]
        train_dataset[start_t:end_t, :, :] = train_letter
        train_labels[start_t:end_t] = label
        start_t += tsize_per_class
        end_t += tsize_per_class
    except Exception as e:
      print('Unable to process data from', pickle_file, ':', e)
      raise

  return valid_dataset, valid_labels, train_dataset, train_labels

train_size = 200000
valid_size = 10000
test_size = 10000

valid_dataset, valid_labels, train_dataset, train_labels = merge_datasets(
  train_datasets, train_size, valid_size)
_, _, test_dataset, test_labels = merge_datasets(test_datasets, test_size)

print('Training:', train_dataset.shape, train_labels.shape)
print('Validation:', valid_dataset.shape, valid_labels.shape)
print('Testing:', test_dataset.shape, test_labels.shape)


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 2
Let's verify that the data still looks good. Displaying a sample of the labels and images from the ndarray. Hint: you can use matplotlib.pyplot.

Next, we'll randomize the data. It's important to have the labels well shuffled for the training and test distributions to match.
Can you write Python code for it?


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 2
Let's verify that the data still looks good. Displaying a sample of the labels and images from the ndarray. Hint: you can use matplotlib.pyplot.
Can you write Python code for it?

def getImage(fname,N):
    try:
        print("Trying to open '%s'" % fname)
        with open(fname, 'rb') as f:
            dataset = pickle.load(f)
        print(type(dataset))
        im = dataset[N,:,:]
    except Exception as e:
        print('Unable to open', fname, ':', e)
    plt.imshow(im, cmap=plt.cm.CMRmap)
    return im

im = getImage(test_datasets[0],100)

im = getImage(test_datasets[1],100)

im = getImage(test_datasets[2],100)


def display_random_matrix(pickle_file, num_to_display):
    """ Display a number of images from a folder. """
    with open(pickle_file, 'rb') as f:
        dataset = pickle.load(f)
    index_samples = np.random.choice(range(0, dataset.shape[0]), num_to_display)
    for index_sample in index_samples:
        plt.figure()
        plt.imshow(dataset[index_sample, :, :])

for pickle_file in train_datasets:
    display_random_matrix(pickle_file, 3)

for pickle_file in test_datasets:
    display_random_matrix(pickle_file, 1)


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 3
Another check: we expect the data to be balanced across classes. Verify that.

Data is balanced across classes if the classes have about the same number of items. Let's check the number of images by class.
Can you write Python code for it?

def disp_number_images(data_folders):
  for folder in data_folders:
    pickle_filename = ''.join(folder) + '.pickle'
    try:
      with open(pickle_filename, 'rb') as f:
        dataset = pickle.load(f)
    except Exception as e:
      print('Unable to read data from', pickle_filename, ':', e)
      return
    print('Number of images in ', folder, ' : ', len(dataset))

disp_number_images(train_folders)
disp_number_images(test_folders)


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 3
Another check: we expect the data to be balanced across classes. Verify that.
Can you write Python code for it?

print(test_datasets)
for cl in test_datasets:
    with open(cl,'rb') as f:
        dataset = pickle.load(f)
        print("class %s has %d samples" % (cl,len(dataset)))


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 3
Convince yourself that the data is still good after shuffling!

Problem 4
Another check: we expect the data to be balanced across classes. Verify that.

Prune the training data as needed. Depending on your computer setup, you might not be able to fit it all in memory, and you can tune train_size as needed.
Also create a validation dataset for hyperparameter tuning.
Can you write Python code for it?

train_size = 200000
valid_size = 10000

valid_dataset = train_dataset[:valid_size,:,:]
valid_labels = train_labels[:valid_size]
train_dataset = train_dataset[valid_size:valid_size+train_size,:,:]
train_labels = train_labels[valid_size:valid_size+train_size]
print 'Training', train_dataset.shape, train_labels.shape
print 'Validation', valid_dataset.shape, valid_labels.shape


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 4
Convince yourself that the data is still good after shuffling!

Finally, let's save the data for later reuse:
Can you write Python code for it?

pickle_file = 'notMNIST.pickle'

try:
  f = open(pickle_file, 'wb')
  save = {
    'train_dataset': train_dataset,
    'train_labels': train_labels,
    'valid_dataset': valid_dataset,
    'valid_labels': valid_labels,
    'test_dataset': test_dataset,
    'test_labels': test_labels,
    }
  pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
  f.close()
except Exception as e:
  print('Unable to save data to', pickle_file, ':', e)
  raise

statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 4
Convince yourself that the data is still good after shuffling!
Can you write Python code for it?

im = test_dataset[0,:,:]
plt.imshow(im, cmap=plt.cm.CMRmap)

im = test_dataset[1,:,:]
plt.imshow(im, cmap=plt.cm.CMRmap)

im = test_dataset[2,:,:]
plt.imshow(im, cmap=plt.cm.CMRmap)


def show_random_samples(dataset, labels, num_to_display):
    index_samples = np.random.choice(range(0, dataset.shape[0]), num_to_display)
    for index_sample in index_samples:
        print(labels[index_sample])
        plt.matshow(dataset[index_sample, :, :])

show_random_samples(train_dataset, train_labels, 5)
show_random_samples(test_dataset, test_labels, 3)
show_random_samples(valid_dataset, valid_labels, 3)


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 5
By construction, this dataset might contain a lot of overlapping samples, including training data that's also contained in the validation and test set! Overlap between training and test can skew the results if you expect to use your model in an environment where there is never an overlap, but are actually ok if you expect to see training samples recur when you use it.
Measure how much overlap there is between training, validation and test samples.
Optional questions:
- What about near duplicates between datasets? (images that are almost identical)
- Create a sanitized validation and test set, and compare your accuracy on those in subsequent assignments.
Can you write Python code for it?

### WARNING, WILL TAKE AN INCREDIBLY LONG TIME TO RUN ON FULL TRAINING DATASET ###

# Cycle through train dataset
# If dupe matrices exist in validation or testing set, delete them from sanitized dataset

sanitized_valid_dataset = valid_dataset
sanitized_test_dataset = test_dataset
sanitized_valid_labels = valid_labels
sanitized_test_labels = test_labels

valid_count = 0
test_count = 0
#for i in range(len(train_dataset)):
for i in range(80,90):

    # Validation set
    for j in range(len(valid_dataset)):
        # Check for duplicates
        if np.alltrue(train_dataset[i] == valid_dataset[j]):
            # All with i = 80
            sanitized_valid_dataset = np.delete(valid_dataset, j, axis=0)
            sanitized_valid_labels = np.delete(valid_labels, j)
            valid_count += 1

    # Test set
    for j in range(len(test_dataset)):
        # Check for duplicates
        if np.alltrue(train_dataset[i] == test_dataset[j]):
            # All with i = 80
            sanitized_test_dataset = np.delete(test_dataset, j, axis=0)
            sanitized_test_labels = np.delete(test_labels, j)
            test_count += 1

print('Deleted', valid_count, 'images from validation set across', len(valid_dataset)*10)
print('Deleted', test_count, 'images from test set across', len(test_dataset)*10)

print(train_dataset.shape)
print(train_labels.shape)
print(valid_dataset.shape)
print(valid_labels.shape)
print(test_dataset.shape)
print(test_labels.shape)

print(sanitized_valid_dataset.shape)
print(sanitized_valid_labels.shape)
print(sanitized_test_dataset.shape)
print(sanitized_test_labels.shape)


I am working on the file "1_notmnist.ipynb".
The first task is:
Problem 6
Let's get an idea of what an off-the-shelf classifier can give you on this data. It's always good to check that there is something to learn, and that it's a problem that is not so trivial that a canned solution solves it.
Train a simple model on this data using 50, 100, 1000 and 5000 training samples. Hint: you can use the LogisticRegression model from sklearn.linear_model.
Optional question: train an off-the-shelf model on all the data!
Can you write Python code for it?

def lrclassify(total_training_samples):
    train_dataset[:total_training_samples].shape

    # Reshape tensors into matrices for training and validation
    X = np.reshape(train_dataset[:total_training_samples], (total_training_samples, 28*28))
    X_valid = np.reshape(valid_dataset, (len(valid_dataset), 28*28))

    # Train model
    clf = LogisticRegression()
    clf.fit(X, train_labels[:total_training_samples])

    # Accuracy
    return sum(clf.predict(X_valid) == valid_labels) / float(len(X_valid))

print(lrclassify(50))
print(lrclassify(100))
print(lrclassify(1000))
print(lrclassify(5000))


I am working on the file "1_notmnist.ipynb".
The first task is:
The built-in numpy function provides some improvement either, but this algorithm is still not scalable to the dataset to its full extend.
To make it work at scale, the best option is to use a hash function. To find exact duplicates, the hash functions used for the cryptography will work just fine.
Can you write Python code for it?

def extract_overlap_hash(dataset_1, dataset_2):
  dataset_hash_1 = [hashlib.sha256(img).hexdigest() for img in dataset_1]
  dataset_hash_2 = [hashlib.sha256(img).hexdigest() for img in dataset_2]
  overlap = {}
  for i, hash1 in enumerate(dataset_hash_1):
    for j, hash2 in enumerate(dataset_hash_2):
      if hash1 == hash2:
        if not i in overlap.keys():
          overlap[i] = []
        overlap[i].append(j) ## use np.where
  return overlap

%time overlap_test_train = extract_overlap_hash(test_dataset[:200], train_dataset)

print('Number of overlaps:', len(overlap_test_train.keys()))
display_overlap(overlap_test_train, test_dataset[:200], train_dataset)