I am working on the file "Boston_Housing_Prices/boston_housing.ipynb".
The first task is:
Question 2 - Goodness of Fit
Assume that a dataset contains five data points and a model made the following predictions for the target variable:
| True Value | Prediction |
| :-------------: | :--------: |
| 3.0 | 2.5 |
| -0.5 | 0.0 |
| 2.0 | 2.1 |
| 7.0 | 7.8 |
| 4.2 | 5.3 |
Would you consider this model to have successfully captured the variation of the target variable? Why or why not?
Run the code cell below to use the performance_metric function and calculate this model's coefficient of determination.
Can you write Python code for it?

# Calculate the performance of this model
score = performance_metric([3, -0.5, 2, 7, 4.2], [2.5, 0.0, 2.1, 7.8, 5.3])
print "Model has a coefficient of determination, R^2, of {:.3f}.".format(score)


I am working on the file "Boston_Housing_Prices/boston_housing.ipynb".
The first task is:
Question 3 - Training and Testing
What is the benefit to splitting a dataset into some ratio of training and testing subsets for a learning algorithm?
Hint: What could go wrong with not having a way to test your model?
Answer:
The benefit of splitting dataset into training and testing is that we can evaluate the performance of our model by feeding it newer unknown data which we didn't use for training i.e, test_set.
If we don't have a way to test our model, we cannot know how well it will deal with real data.
 Thus, it's performance will remain ambiguous

Analyzing Model Performance
In this third section of the project, you'll take a look at several models' learning and testing performances on various subsets of training data. Additionally, you'll investigate one particular algorithm with an increasing 'max_depth' parameter on the full training set to observe how model complexity affects performance. Graphing your model's performance based on varying criteria can be beneficial in the analysis process, such as visualizing behavior that may not have been apparent from the results alone.
Learning Curves
The following code cell produces four graphs for a decision tree model with different maximum depths. Each graph visualizes the learning curves of the model for both training and testing as the size of the training set is increased. Note that the shaded region of a learning curve denotes the uncertainty of that curve (measured as the standard deviation). The model is scored on both the training and testing sets using R2, the coefficient of determination.
Run the code cell below and use these graphs to answer the following question.
Can you write Python code for it?

# Produce learning curves for varying training set sizes and maximum depths
vs.ModelLearning(features, prices)


I am working on the file "Boston_Housing_Prices/boston_housing.ipynb".
The first task is:
Question 4 - Learning the Data
Choose one of the graphs above and state the maximum depth for the model. What happens to the score of the training curve as more training points are added? What about the testing curve? Would having more training points benefit the model?
Hint: Are the learning curves converging to particular scores?
Answer:
I chose the graph with maximum depth = 3.
at 0 points it has a score of one but as we add more training points, it finally seems to be converging at 0.8.
The testing curve has a score of 0 and then suddenly increases to 0.6 with 50 points and seems to be converging at 0.6
According to the graph , it seems that after a certain number of points it doesn't matter because the model has already learnt the data.
Complexity Curves
The following code cell produces a graph for a decision tree model that has been trained and validated on the training data using different maximum depths. The graph produces two complexity curves — one for training and one for validation. Similar to the learning curves, the shaded regions of both the complexity curves denote the uncertainty in those curves, and the model is scored on both the training and validation sets using the performance_metric function.
Run the code cell below and use this graph to answer the following two questions.
Can you write Python code for it?


I am working on the file "Boston_Housing_Prices/boston_housing.ipynb".
The first task is:
Question 5 - Bias-Variance Tradeoff
When the model is trained with a maximum depth of 1, does the model suffer from high bias or from high variance? How about when the model is trained with a maximum depth of 10? What visual cues in the graph justify your conclusions?
Hint: How do you know when a model is suffering from high bias or high variance?
Answer:
when the model is trained with max depth 1 it suffers from high Bias as the scores are pretty low in both the cases.
But when it is trained with max depth 10 it overfits the data and suffers from high variance as the difference between training and testing score is high.
Question 6 - Best-Guess Optimal Model
Which maximum depth do you think results in a model that best generalizes to unseen data? What intuition lead you to this answer?
Answer:
I think that max depth 4 best generalizes the unseen data because the score of training is also high and difference is also less between testing and training scores.

Evaluating Model Performance
In this final section of the project, you will construct a model and make a prediction on the client's feature set using an optimized model from fit_model.
Question 7 - Grid Search
What is the grid search technique and how it can be applied to optimize a learning algorithm?
Answer:
Grid search means you have a set of models (which differ from each other in their parameter values, which lie on a grid). What you do is you then train each of the models and evaluate it using cross-validation. You then select the one that performed best.
Question 8 - Cross-Validation
What is the k-fold cross-validation training technique? What benefit does this technique provide for grid search when optimizing a model?
Hint: Much like the reasoning behind having a testing set, what could go wrong with using grid search without a cross-validated set?
Answer:
In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing...
Can you write Python code for it?

# TODO: Import 'make_scorer', 'DecisionTreeRegressor', and 'GridSearchCV'

from sklearn.metrics import make_scorer
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import ShuffleSplit

def fit_model(X, y):
    """ Performs grid search over the 'max_depth' parameter for a
        decision tree regressor trained on the input data [X, y]. """

    # Create cross-validation sets from the training data
    cv_sets = ShuffleSplit(X.shape[0], test_size = 0.20, random_state = 0)

    # TODO: Create a decision tree regressor object
    regressor = DecisionTreeRegressor()

    # TODO: Create a dictionary for the parameter 'max_depth' with a range from 1 to 10
    params = {'max_depth':(1,2,3,4,5,6,7,8,9,10)}

    # TODO: Transform 'performance_metric' into a scoring function using 'make_scorer'
    scoring_fnc = make_scorer(performance_metric)

    # TODO: Create the grid search object
    grid = GridSearchCV(regressor, params, scoring_fnc,cv = cv_sets)

    # Fit the grid search object to the data to compute the optimal model
    grid = grid.fit(X, y)

    # Return the optimal model after fitting the data
    return grid.best_estimator_


I am working on the file "Bottleneck Features.ipynb".
The first task is:
2. Load and visualise sample of dataset
Can you write Python code for it?

import glob
import cv2
import random
import matplotlib.pyplot as plt
%matplotlib inline
random.seed(0)

positive_data = glob.glob("hackathon_public/data/train/positive/*")
negative_data = glob.glob("hackathon_public/data/train/negative/*")

# create binary targets based on positive/negative data
training_data = list(zip(positive_data,[1]*len(positive_data))) + list(zip(negative_data,[0]*len(negative_data)))
# shuffle training data so positive and negative are randomly ordered
# random.shuffle(training_data)
# get all the img_paths and corresponding targets
img_paths,targets = zip(*training_data)

print("number of images: {}".format(len(img_paths)))

def visualize_img(img_path, ax):
    img = cv2.imread(img_path)
    ax.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))

# show some example images (positive on top, negative below)
fig = plt.figure(figsize=(20, 10))
visual_subset_p = positive_data[5:10]
for i,img_path in enumerate(visual_subset_p):
    ax = fig.add_subplot(2, len(visual_subset_p), i + 1, xticks=[], yticks=[])
    visualize_img(img_path, ax)
visual_subset_n = negative_data[1:6]
for i,img_path in enumerate(visual_subset_n):
    ax = fig.add_subplot(2, len(visual_subset_n), i + 1+ len(visual_subset_p), xticks=[], yticks=[])
    visualize_img(img_path, ax)


I am working on the file "Bottleneck Features.ipynb".
The first task is:
4. Get bottleneck features (output of last convolutional layer)
Can you write Python code for it?

predicted = model.predict(img_input)
print(predicted.shape)


I am working on the file "Bottleneck Features.ipynb".
The first task is:
5. Save features
Can you write Python code for it?


I am working on the file "Bottleneck Features.ipynb".
The first task is:
6. Do the same for the test set
Can you write Python code for it?

# as before, but for test set
test_positive_data = glob.glob("data/test/positive/*")
test_negative_data = glob.glob("data/test/negative/*")
test_data = list(zip(test_positive_data,[1]*len(test_positive_data))) + list(zip(test_negative_data,[0]*len(test_negative_data)))
test_data_paths,test_targets = zip(*test_data)
img_input_test = paths_to_tensor(test_data_paths)
predicted_test = model.predict(img_input_test)
print(predicted_test.shape)
np.savez_compressed('bottelneck_features_test',data=predicted_test,targets=test_targets)


I am working on the file "Bottleneck Features.ipynb".
The first task is:
The alternative way
Can you write Python code for it?

# create DataGenerator Object
datagen = image.ImageDataGenerator(rescale=1./255)

# data directoies (single batch)
train_generator_b = datagen.flow_from_directory(
        'data/train/',
        target_size=(224, 224),
        class_mode='binary',
        shuffle=False,
        seed=0,
        batch_size=len(img_paths))
test_generator_b = datagen.flow_from_directory(
        'data/test/',
        target_size=(224, 224),
        class_mode='binary',
        shuffle=False,
        seed=0,
        batch_size=len(test_data_paths))

for imgs,labels_batch in train_generator_b:
    data = model.predict(imgs)
    labels = labels_batch
    # break else it will go infinitely (hence batch size == sample size)
    break
for imgs,labels_batch in test_generator_b:
    test_data = model.predict(imgs)
    test_labels = labels_batch
    break
print("data.shape",data.shape)
print("test_data.shape",test_data.shape)

print(sum(labels),'==',len(positive_data))
print(sum(test_labels),'==',len(test_positive_data))

np.savez_compressed('bottleneck_features',data=data,targets=labels)
np.savez_compressed('bottleneck_features_test',data=test_data,targets=test_labels)
print("SAVED")


I am working on the file "Box Python SDK.ipynb".
The first task is:
Now let's look at some different objects, like File and Folder.
client.folder('0') is a reference to the root folder ("All Files").
Can you write Python code for it?

root_folder = client.folder('0')
root_folder_with_info = root_folder.get()

# Save time and bandwidth by only asking for the folder owner
root_folder_with_limited_info = root_folder.get(fields=['owned_by'])


I am working on the file "Box Python SDK.ipynb".
The first task is:
The Pre-flight check API will verify that a file will be accepted by Box before you send all the bytes over the wire. It can be used for both first-time uploads, and uploading new versions of an existing File (on /files/[id]/content). If the call returns a 200, then you can proceed with a standard upload call. Preflight checks verify all permissions as if the file was actually uploaded including:

Folder upload permission
File name collisions
file size caps
folder and file name restrictions*
folder and account storage quota
Can you write Python code for it?

stream.seek(0)
from boxsdk.exception import BoxAPIException
try:
    box_file = client.folder('0').upload_stream(stream, 'box-python-sdk-test.txt', preflight_check=True)
except BoxAPIException:
    pass

# See if we can find the file on Box using search (may need to wait for Box to index the file)
results = client.search('Box Python SDK test', 2, 0)
matching_results = (r for r in results if r.id == box_file.id)
for m in matching_results:
    print m.name
    print m.created_at
    break
else:
    print 'No match found'


I am working on the file "Box Python SDK.ipynb".
The first task is:
The Python SDK is organized into layers:
* The network layer, responsible for making network calls.
* The session layer, responsible for auth and retrying calls.
* The client layer, which is the interface to your application.
For demonstration purposes, I'm subclassing the default network layer to log network requests and responses.
Can you write Python code for it?

from boxsdk.network.default_network import DefaultNetwork
from pprint import pformat

class LoggingNetwork(DefaultNetwork):
    def request(self, method, url, access_token, **kwargs):
        """ Base class override. Pretty-prints outgoing requests and incoming responses. """
        print '\x1b[36m{} {} {}\x1b[0m'.format(method, url, pformat(kwargs))
        response = super(LoggingNetwork, self).request(
            method, url, access_token, **kwargs
        )
        if response.ok:
            print '\x1b[32m{}\x1b[0m'.format(response.content)
        else:
            print '\x1b[31m{}\n{}\n{}\x1b[0m'.format(
                response.status_code,
                response.headers,
                pformat(response.content),
            )
        return response

# Create OAuth2 object. It's already authenticated, thanks to the developer token.
oauth2 = OAuth2(CLIENT_ID, CLIENT_SECRET, access_token=ACCESS_TOKEN)

# Create the authenticated client
client = Client(oauth2, LoggingNetwork())


I am working on the file "Box Python SDK.ipynb".
The first task is:
The SDK makes it very easy to download and upload files.
Folder.upload(file_path ,filename) uploads a file to that folder, and Folder.upload_stream(stream, filename) uploads the contents of a stream as a file to the folder. Both methods return a File object.
Can you write Python code for it?

# Upload a file to Box!
from StringIO import StringIO

stream = StringIO()
stream.write('Box Python SDK test!')
stream.seek(0)
box_file = client.folder('0').upload_stream(stream, 'box-python-sdk-test.txt')
print box_file.name

# Download the file's contents from Box
print box_file.content()
print box_file.id


I am working on the file "Box Python SDK.ipynb".
The first task is:
The SDK provides information when an API call couldn't be completed.
Let's try to upload the same file again - this time, the upload will fail!
That's because the name is already in use.
Can you write Python code for it?

stream.seek(0)
box_file = client.folder('0').upload_stream(stream, 'box-python-sdk-test.txt')


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
AR models, meaning regression models that use the past as input features, are no different from regression with time-unrelated features. What we are doing is just using previous y-values as input for a given point. We can manually construct such features directly.
Can you write Python code for it?

# load data into DataFrame
import pandas as pd

timeseries_df = pd.DataFrame(weekly_timeseries, columns=["weekly_gross"])
timeseries_df


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
As we can see, the gross drops pretty quickly. With this scale, it is hard to see the differences in the later weeks. One solution is to use log scale: instead of plotting gross numbers, we plot the order of magnitude of weekly gross.
Can you write Python code for it?

# plot with log scale

x = range(len(weekly_timeseries))
y = weekly_timeseries
plt.scatter(x,y)
plt.title("Mad Max Fury Road Weekly Gross over Time")
plt.xlabel("Week in theaters")
plt.ylabel("Weekly Gross")
plt.yscale("log")


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
As you can see, the moving average gives us a much smoother, less noisy dataset, making it easier to define the underlying process. Of course, just like in the case of the AR models, we get fewer points (since we lose 5 at the start and 5 at the end) to use in a model.
Remember: When you are doing preprocessing like this, you are putting some of your prior information or assumptions about the system into your model. The window size you choose reflects your belief about what portion of the fluctuations are noise. If you choose a rolling window of 5 points, you are saying that the underlying value would not change much within a time window of 5 points. So, the ups and downs must be from noise. By inserting more information, you make it easier to model the changes. But of course be mindful of your assumptions, and keep checking their validity wherever you can.
Don't let this scare you, though. Making assumptions like this is normal, it isn't evil at all. Even by doing linear regression, we are making a lot of assumptions. Quite a few of those assumptions (such as a perfectly well behaving gaussian noise with the same standard deviation throughout) are usually wrong, but they are close enough that our models are still very useful.
Implementation in pandas
Pandas actually provides a function to do the moving average.
pd.stats.moments.rolling_mean
Can you write Python code for it?

weight_df = pd.DataFrame(weight_timeseries, columns=["Weight"])
weight_df["Weight_MA10"] = pd.stats.moments.rolling_mean(weight_df, window=10)
weight_df.head(20)


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
As you can see, this model is different. It is smoother. Our previous model was only looking at last week and that single jump towards the end was throwing it off for those last few points. Here, because the model is looking at a longer range of the past, it is starting to predict a general level for those last low gross amounts.
We have manually constructed our features based on past performance. Since this is common for time series, there are actually short-hand interfaces to build such AR models. In statsmodels, for example, you can just use the gross values over time as the input to an AR model and it will automatically do what we just did. Let's take a look.
Can you write Python code for it?

import statsmodels.api as sm
weekly_timeseries

# freq set as "W" means these numbers are weekly -- this is required
# to make it possible to make predictions with datetime objects later,
# as is usually practical with time series models.
# The maxlag=1 means we are using a single lag feature, so this is an AR-1.
sm_AR1_model = sm.tsa.AR(weekly_timeseries, freq="W").fit(maxlag=1)
print(sm_AR1_model.params)


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
Basically, that daily luck factor in our stochastic process model is causing fluctuations that makes it a bit harder for the model to fit that underlying process. If we can eliminate the noise a bit, we can fit better. In fact, this is exactly what we did in the beginning by looking at weekly timeseries instead of daily. By averaging (or summing -- same effect) over a bunch of points, we are letting the noise from different days cancel each other.
Now, in some cases, such as the weekly grosses, there are logical units, bins, to average/sum over. In movies, each week is one unit that makes sense in the system. Every week there are new movies released, new competitors, so it makes sense to treat each week separately. In the case of the weight, however, there is no discrete jump from one unit to the other as in the case of movies. So, if we wanted to get rid of the noise, instead of just dividing the time into bins, a better approach is to take a moving average. Basically, take a window of n points, take their average to represent that whole window, then slide the window by one point to the future (remove oldest point, add the next point) average again. Let's do that here with a 10-day moving average window and see how it helps.
Can you write Python code for it?

MA_10 = [None] * len(weight_timeseries)
start, end = 5, len(MA_10)-5
for window in range(start, end):
    points_in_window = [weight_timeseries[i] for i in range(window-5, window+5)]
    MA_10[window] = sum(points_in_window)/10.

MA_10[:20]

x = range(len(weight_timeseries))
y_real = weight_timeseries

start, end = 5, len(MA_10)-5
x_ma = range(start, end)
y_ma = MA_10[start:end]

plt.plot(x,y, label="actual weight")
plt.plot(x_ma, y_ma, color='green', label="moving average")
plt.title("Weight over Time (Moving Average: 10 Days)")
plt.xlabel("Day")
plt.ylabel("Weight in kgs")
plt.legend()


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
Gross is in text form with '$' sign and commas; let's clean it up with a quick helper function. We'll change our extraction function to use that.
Can you write Python code for it?

# updated helper functions

def gross_text_to_int(gross_text):
    clean_text = gross_text.replace('$','').replace(',','')
    gross_as_int = int(clean_text)
    return gross_as_int

def extract_gross_points_from_table(gross_table, gross_column=2):
    points = []
    rows = gross_table.find_all('tr')
    for row_no, row in enumerate(rows):
        columns = row.find_all('td')
        # skip first row (headers) or an empty row
        if row_no == 0 or len(columns)<=1:
            continue
        # take column 2 (weekly gross)
        # We are using our helper function here
        # to make sure we have a number for gross
        gross = gross_text_to_int(columns[gross_column].text)
        # -----------
        points.append(gross)
    return points

# print output

weekly_timeseries = extract_gross_points_from_table(weekly_gross_table)
for weekend, gross in enumerate(weekly_timeseries):
    print(weekend, gross)

# plot weekly grosses

x = range(len(weekly_timeseries))
y = weekly_timeseries
plt.scatter(x,y)
plt.title("Mad Max Fury Road Weekly Gross over Time")
plt.xlabel("Week in theaters")
plt.ylabel("Weekly Gross")


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
Looking at it this way, we can see it as just another regular regression problem.
So, already by looking at what the movie made last week, the model gathers a pretty good idea of what to expect this week. If we add more information about the past, like not only last week, but what it made the two previous weeks, we could get even better estimates. More lag doesn't necessarily mean 'better' models. Of course it will fit the training data better, but the extra complexity might be just overfitting to the data at hand. Just like any other regression model where we
When we add a 2-lag feature (meaning grosses from two weeks ago & last week in our case), the model is called AR-2 (AR for AutoRegressive). When we use up to three, it is called AR-3 and so on. Let's do an AR-2.
Can you write Python code for it?

grosses_two_weeks_prior = [None] * len(weekly_timeseries)
for week in range(2, len(weekly_timeseries)):
    grosses_two_weeks_prior[week] = weekly_timeseries[week-2]

grosses_two_weeks_prior

timeseries_df["two_prev_weeks_gross"] = grosses_two_weeks_prior
timeseries_df

AR2_model = LinearRegression()
X = timeseries_df.dropna()[["one_prev_weeks_gross", "two_prev_weeks_gross"]]
y = timeseries_df.dropna()[["weekly_gross"]]
AR2_model.fit(X, y)

AR2_model.coef_, AR2_model.intercept_


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
Looking at orders of magnitude (logarithms of the grosses) will make it clearer to see those clustered points.
Can you write Python code for it?

plt.scatter(previous_weeks_grosses[1:], weekly_timeseries[1:])
plt.scatter(previous_weeks_grosses[1:], y_pred, color='red', marker='x')
plt.xlabel("Previous week's gross")
plt.ylabel("This week's gross")
plt.xscale('log')
plt.yscale('log')


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
Of course, we weighed every single point in the window equally. You can use other schemes, such as weighing each point less and less as we go back into the past. A common way to do that is an exponential moving average, where the weight of each previous point falls down exponentially. Again, by choosing the form of that exponential decay (like halflife, or span, or center of mass), we are putting in assumptions about the system, specifically in terms of how we believe noise works.
In pandas, you can do this with pandas.stats.moments.ewma, where ewma stands for exponentially weighted moving average. We are using halflife = 5 to set the shape of the exponential decay. This means that by the time we are at the 5th previous point, the weight we are using is half of the weight of the current point. By the time we are at the 10th previous point, the weight is half of the weight we used at the 5th previous point (and of course one quarter of the weight we gave the current point).
Can you write Python code for it?

ewma= pd.stats.moments.ewma(weight_df, halflife=5)
ewma.head(20)


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
Preprocessing: Smoothing out the noise
Ok. Now let's see what happens if there is more noise. We were looking at weekly grosses. Basically, by summing up each daily gross, we are averaging out some of that daily noise. What if we looked at each day? Easy enough to get from boxofficemojo by going to the daily page for the same movie.
Can you write Python code for it?

url = "http://www.boxofficemojo.com/movies/?page=daily&view=chart&id=furyroad.htm"

response = requests.get(url)
page = response.text
soup = BeautifulSoup(page)

# eac
daily_gross_table = soup.find('table', {"class":"chart-wide"})
daily_timeseries = extract_gross_points_from_table(daily_gross_table, gross_column=3)
daily_timeseries

x = range(len(daily_timeseries))
y = daily_timeseries
plt.plot(x,y)
plt.title("Mad Max Fury Road Daily Gross over Time")
plt.xlabel("Day in theaters")
plt.ylabel("Daily Gross")


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
This actually assigns the mean of the last 10 points to the last time point of the window. If we instead want to center that value (like we did up there manually), we can do that as well, with the center keyword argument. This is important if the underlying trend is going up or down (your data is not stationary). Otherwise, for example in this case, since the weight is generally going down, our average of the last 10 weights will be higher than what we should assign the last point.
Can you write Python code for it?

weight_df["Weight_MA10"] = pd.stats.moments.rolling_mean(weight_df, window=10, center=True)
weight_df.head(20)


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
This makes it easy to fit any AR type model. AR2 or AR3 are simple changes of maxlag.
Can you write Python code for it?

sm_AR2_model = sm.tsa.AR(weekly_timeseries, freq="W").fit(maxlag=2)
print(sm_AR2_model.params)

sm_AR3_model = sm.tsa.AR(weekly_timeseries, freq="W").fit(maxlag=3)
print(sm_AR3_model.params)


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
We are still plotting it over time. The model doesn't know about time. For each point, it sees an input feature (previous week's gross) and a target value (this week's gross). Let's try and look at these points from a viewpoint closer to the model's. Let's plot the target versus our single feature.
Can you write Python code for it?

plt.scatter(previous_weeks_grosses[1:], weekly_timeseries[1:])
plt.scatter(previous_weeks_grosses[1:], y_pred, color='red', marker='x')
plt.xlabel("Previous week's gross")
plt.ylabel("This week's gross");


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
We can interpret our model like this:
Each week, we should expect the movie to make about 59% of what it made last week, minus a small flat drop of $7500, plus/minus some luck factor (what we can't predict with this model). This is an autoregressive model, specifically AR-1, since it is using only a lag of 1 time tick as input. This is no different in principle from other linear models that multiply each feature with a coefficient, which is found by optimization over the set of training points.
Let's see what the model does. We didn't separate training/test sets, so this look over the training set is not a good evaluation of generalized performance, but a quick check.
Can you write Python code for it?

# plot actuals and predicted

x = range(len(weekly_timeseries))
y_true = weekly_timeseries
# plot actual grosses (blue spheres)
plt.scatter(x,y_true, label="actual gross")

# we can only predict for weeks where we have a number for
# the previous week. That is a feature.
x_pred = x[1:]
y_pred = model.predict(X)

# plot model prediction (red x signs)
plt.scatter(x_pred,y_pred, color='red', marker="x", label="predicted")

# Labels
plt.title("Mad Max Fury Road Weekly Gross over Time")
plt.xlabel("Week in theaters")
plt.ylabel("Weekly Gross")
plt.legend();

# plot on log scale

plt.scatter(x,y_true)
plt.scatter(x_pred,y_pred, color='red', marker="x")
plt.yscale('log')


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
We can see that it fluctuates. This is a somewhat patterned fluctuation, we have a spike every weekend, since many more people watch movies on the weekend compared to weekdays. So, actually summing up or averaging the daily grosses for each week, and then using the weekly time series is a very good pre-processing to apply to this data.
There are of course tons of other time series where the noise is less patterned. Take the following time series of an instructor's weight, for example, taken every day (in kgs) for a long stretch. First of all, stop judging! :)
Here we can see the daily fluctuations that would make it tough if we just used something like AR-1 or AR-2. Our amazing human brains with great pattern-finding capabilities can immediately see the trend, but on the small scale, zoomed into a few points next to each other, the noise has some effect.
Can you write Python code for it?


I am working on the file "BoxOfficeMojo AR TS Models SOL.ipynb".
The first task is:
We have only our target now: what we want to predict. As input for a given week, we want to know what the previous week's gross was. let's build that.
Can you write Python code for it?

# create previous week feature

previous_weeks_grosses = [None] * len(weekly_timeseries)
previous_weeks_grosses

for week in range(1, len(weekly_timeseries)):
    previous_weeks_grosses[week] = weekly_timeseries[week-1]
previous_weeks_grosses

timeseries_df["one_prev_weeks_gross"] = previous_weeks_grosses
timeseries_df


I am working on the file "Bpid.ipynb".
The first task is:
Tabula Recta - таблицы Виженера:
Can you write Python code for it?

v = Vigenere('TEST')
v.pretty_print(v.get_tabula_recta(), space=3)


I am working on the file "Bpid.ipynb".
The first task is:
Демонстрация работы алгоритма:
Can you write Python code for it?

key = 'LEMON'
message = 'ATTACKATDAWN'

vigenere = Vigenere(key)
cipher = vigenere.encrypt(message)
deciphered = vigenere.decrypt(cipher)

print('Key:\t\t', key)
print('Message:\t', message)
print('Ciphered:\t', cipher)
print('Deciphered:\t', deciphered)


I am working on the file "Bpid.ipynb".
The first task is:
Демонстрация работы кода:
Can you write Python code for it?

text = 'hello'
ciphered = cipher(text, alphabet='abcdefghijklmnopqrstuvwxyz', key=14)

print('Initial text: ', text)
print('Ciphered text: ', ciphered)

entropies = get_all_entropies(ciphered)
entropies.sort(key=cmp_to_key(comparator))
best_shift = entropies[0][0]
cracked_val = decipher(ciphered, key=best_shift)
print('\nBest guess:')
print('%d rotations\nDeiphered text: %s\n' % (best_shift, cracked_val))

print('=========\nFull circle:')
for i in range(0, 26):
    print('%d -\t%s' % (i, decipher(ciphered, key=i)))


I am working on the file "Bpid.ipynb".
The first task is:
Демонстрация работы:
Can you write Python code for it?

print('===== The original XOR version =====')
message = 'ALLSWELLTHATENDSWELL'
key = 'EVTIQWXQVVOPMCXREPYZ'
pad = OneTimePad(key)
cipher = pad.crypt(message)
deciphered = pad.crypt(cipher)

print('Original:\t', message)
print('The key:\t', key)
print('Cipher\t\t', [ord(c) for c in cipher])
print('Deciphered:\t', deciphered)

print('\n======= The modulo version =======')
message = 'HELLO'
pad = OneTimePad()
cipher = pad.crypt('HELLO', mode='mod', encrypt=True)
deciphered = pad.crypt(cipher, mode='mod', encrypt=False)

print('Original:\t', message)
print('The key:\t', pad.key)
print('Cipher\t\t', cipher)
print('Deciphered:\t', deciphered)


I am working on the file "Bpid.ipynb".
The first task is:
Коллизия
Поиск 4-х байтных коллизий для хеш-функции:
Can you write Python code for it?

from random import choice, seed

ascii = ''.join([chr(i) for i in range(33, 127)])

seed(37)
found = {}

for j in range(5000):
    # Build a 4 byte random string
    s = bytes(''.join([choice(ascii) for _ in range(4)]), 'ascii')
    h = hash_function(s)
    if h in found:
        v = found[h]
        if v == s:
            # Same hash, but from the same source string
            continue
        print(h, found[h], s)
    found[h] = s


I am working on the file "Bpid.ipynb".
The first task is:
Кривая вероятности коллизии на выбранном интервале
Расчет вероятностей коллизии для хеш-выборок заданного размера с использованием принципа парадокса дней рождения.
Can you write Python code for it?

%matplotlib inline
import math
import numpy as np
import matplotlib.pyplot as plt

# Calculate the probability of collision among 10000 keys using Birthday Paradox Principle
n = num_of_all_hashes = 8 ** 8  # 16777216
keys = 10000
probUnique = 1.0
keys_arr = np.array(range(1, keys))
coll_probs = []

for k in range(1, keys):
    probUnique = probUnique * (n - (k - 1)) / n
    coll_probs.append((1 - math.exp(-0.5 * k * (k - 1) / n)))

plt.plot(keys_arr, coll_probs)
plt.show()


I am working on the file "Bpid.ipynb".
The first task is:
Криптоанализ
Взломать шифр Цезаря можно, используя частоты букв алфавита и вычисление наименьшей энтропии
Код для взлома шифров на английском языке:
Can you write Python code for it?

import numpy as np

ENGLISH_FREQS = [
    0.08167, 0.01492, 0.02782, 0.04253, 0.12702, 0.02228,
    0.02015, 0.06094, 0.06966, 0.00153, 0.00772, 0.04025,
    0.02406, 0.06749, 0.07507, 0.01929, 0.00095, 0.05987,
    0.06327, 0.09056, 0.02758, 0.00978, 0.02360, 0.00150,
    0.01974, 0.00074
]

# Returns the cross-entropy of the given string with respect to
# the English unigram frequencies, which is a positive
# floating-point number.
def get_entropy(str):
    sum, ignored = 0, 0
    for c in str:
        if c.isalpha():
            sum += np.log(ENGLISH_FREQS[ord(c.lower()) - 97])
        else:
            ignored += 1
    return -sum / np.log(2) / (len(str) - ignored)

# Returns the entropies when the given string is decrypted with
# all 26 possible shifts, where the result is an array of tuples
# (int shift, float enptroy) -
# e.g. [(0, 2.01), (1, 4.95), ..., (25, 3.73)].
def get_all_entropies(str):
    result = []
    for i in range(0, 26):
        result.append((i, get_entropy(decipher(str, key=i))))
    return result

def cmp_to_key(mycmp):
    'Convert a cmp= function into a key= function'
    class K(object):
        def __init__(self, obj, *args):
            self.obj = obj

        def __lt__(self, other):
            return mycmp(self.obj, other.obj) < 0

        def __gt__(self, other):
            return mycmp(self.obj, other.obj) > 0

        def __eq__(self, other):
            return mycmp(self.obj, other.obj) == 0

        def __le__(self, other):
            return mycmp(self.obj, other.obj) <= 0

        def __ge__(self, other):
            return mycmp(self.obj, other.obj) >= 0

        def __ne__(self, other):
            return mycmp(self.obj, other.obj) != 0
    return K

def comparator(x, y):
    if x[1] < y[1]:
        return -1
    elif x[1] > y[1]:
        return 1
    elif x[0] < y[0]:
        return -1
    elif x[0] > y[0]:
        return 1
    else:
        return 0


I am working on the file "Bpid.ipynb".
The first task is:
Практические занятия
Шифр Цезаря
Шифрование:
Can you write Python code for it?

def cipher(text, alphabet='abcdefghijklmnopqrstuvwxyz', key=0):
    result = ""
    alphabet = alphabet.lower()
    n = len(alphabet)
    for char in text:
        if char.isalpha():
            new_char = alphabet[(alphabet.find(char.lower()) + key) % n]
            result += new_char if char.islower() else new_char.upper()
        else:
            result += char

    return result


I am working on the file "Bpid.ipynb".
The first task is:
Работа алгоритма
Can you write Python code for it?

d = DES('qwertyui')

cipher = d.encrypt('hello world!')

print("Ciphered bits:\n", cipher)

deciphered = d.decrypt(cipher, msg_in_bits=True)

print("Deciphered text:\n", deciphered)


I am working on the file "Bpid.ipynb".
The first task is:
Работа алгоритма:
Can you write Python code for it?

p = 11
g = 2

d1 = DiffieHellman(generator=g, prime=p, key_length=3)
d2 = DiffieHellman(generator=g, prime=p, key_length=3)

d1.gen_key(d2.public_key)
d2.gen_key(d1.public_key)

d1.show_params()
d1.show_results()
d2.show_params()
d2.show_results()

if d1.get_key() == d2.get_key():
    print('Shared keys match!')
    print('Key: ', hexlify(bytes(d1.key, 'ascii')))
    print('Hashed key: ', d1.get_key())
else:
    print("Shared secrets didn't match!")
    print("Shared secret A: ", d1.gen_secret(d2.public_key))
    print("Shared secret B: ", d2.gen_secret(d1.public_key))


rsa = RSA()
message = 'hello!'
cipher = rsa.encrypt(message, rsa.get_public_key_pair())
print('Cipher:\n', cipher[0])
deciphered = rsa.decrypt(cipher[0], rsa.get_private_key_pair())
print('Deciphered text:\n', deciphered[0])

rsa.print_info()


I am working on the file "Bpid.ipynb".
The first task is:
Расшифровывание:
Can you write Python code for it?

def decipher(text, alphabet='abcdefghijklmnopqrstuvwxyz', key=0):
    result = ""
    alphabet = alphabet.lower()
    n = len(alphabet)
    for char in text:
        if char.isalpha():
            new_char = alphabet[(alphabet.find(char.lower()) - key + n) % n]
            result += new_char if char.islower() else new_char.upper()
        else:
            result += char

    return result


I am working on the file "Bpid.ipynb".
The first task is:
Хеш-функция
- 8-ми байтная
- Переменные a, b, c и d после генерации складываются в шестнадцатеричном виде в порядке d, c, a, b.

Код функции:
Can you write Python code for it?

def hash_function(s=b''):
    a, b, c, d = 0xa0, 0xb1, 0x11, 0x4d

    for byte in bytearray(s):
        a ^= byte
        b = b ^ a ^ 0x55
        c = b ^ 0x94
        d = c ^ byte ^ 0x74

    return format(d << 24 | c << 16 | a << 8 | b, '08x')


I am working on the file "Bpid.ipynb".
The first task is:
ЭЦП
Реализация упрощенной электронной цифровой подписи.
$$H_i = (H_{i-1} + M_i)^2 \mod n,H_0 = 0$$
$$S = H ^ d \mod n$$
$$H' = S^e \mod n$$
Код алгоритма:
Can you write Python code for it?

import re

class Signature(object):

    def __init__(self, public_key, private_key, n):
        self.public_key = public_key
        self.private_key = private_key
        self.n = n

    def sign(self, message, private_key):
        H = self.hash_function(message)
        signature = pow(H, private_key, self.n)
        return '@' + str(signature) + '@' + message

    def verify(self, message, public_key):
        regex = re.compile('@\d+@')
        match = regex.search(message)

        if match:
            signature = int(match.group(0).strip('@'))
            msg = regex.sub('', message)

            H1 = pow(signature, public_key, self.n)
            H2 = self.hash_function(msg)

            return H1 == H2

    def hash_function(self, message):
        H = 0
        for c in [ord(c) for c in message]:
            H = (H + c)**2 % self.n
        return H

    def get_public_key(self):
        return self.public_key

    def get_private_key(self):
        return self.private_key


I am working on the file "Braid Examples.ipynb".
The first task is:
Keeping the same network n, let's increase the target (decrease the difficulty) to produce a thicker braid.  Also, this time let's actually mine, to see that the graphs are the same.  The number of iterations until this node mines a bead is given by the geometric distribution.  This may give a computational speedup for graphing under the right circumstances.  (It's no speedup with this example)
Can you write Python code for it?

n.reset(target=2**249)
for dummy in range(200): n.tick(mine=True)
b = n.nodes[0].braids[0]
b.plot(numbervertices=True);


I am working on the file "Braid Examples.ipynb".
The first task is:
Now let us examine the behavior of cohorts as we vary the target difficulty.
This can be fairly time consuming.  We're essentially going to simulate an entire network with 25 nodes for many millions of beads, in order to get enough resolution on the graph.  We use joblib to parallelize creation of the following graphs using a map/reduce algorithm.  Each job will create ncohorts/cpu_count cohorts.
The relative vertical error in this graph is $\frac{1}{\sqrt{N}}$ with $N$ cohorts as data points, so for a very smooth graph with error ~1% we need $N=10000$.  This takes a couple days on a beefy computer.  Instead let's just do $N=100$.  The resulting curve will be noisy but it's enough to extract the salient parts.  Determining the cohorts is roughly $\mathcal{O}(B_C^2)$ in the number of beads per cohort $B_C$, so we will stop adding data points when the computation time rises for large $B_C$.
Go get a coffee now...this takes a few minutes.
Can you write Python code for it?

dctimes = [] # Initialize our results

def job(n, difficulty, ncohorts, njob):
    begin = time.process_time()
    np.random.seed((int(time.time())*njob)%(2**32-1))# We have to seed the random number generator with something
    n.reset(target=difficulty*(1<<255))              # different for each job or they will all generate exactly
    times = []                                       # the same results!
    lastcohort = frozenset({n.nodes[0].braids[0].vertex(0)})
    while len(list(n.nodes[0].braids[0].cohorts())) < ncohorts:  # 1% error = sqrt(N)/N
        for dummy in range(100): n.tick()
        for c in n.nodes[0].braids[0].cohorts(lastcohort): pass
        lastcohort = c
    b = n.nodes[0].braids[0]
    times.append(b.cohort_time())
    #print(difficulty, times, b.ncohorts)
    return((difficulty,numpy.mean([m for (m,s) in times]), len(b.beads), b.ncohorts, time.process_time()-begin))

def parmap(f, *args):
    """ Map the function f with arguments <args>, adding an extra argument which is its job number. """
    return Parallel(n_jobs=cpu_count())(delayed(f)(*args, job) for job in range(cpu_count()))

def gettimes(n, difficulty, ncohorts):
    def reduce(result):
        return (difficulty, numpy.mean([m[1] for m in result]), sum([m[2] for m in result]),
                sum([m[3] for m in result]), sum([m[4] for m in result]))
    return reduce(parmap(job, n, difficulty, ncohorts/cpu_count()))

print("(target difficulty,      mean cohort time, nbeads, ncohorts, CPU time)")
for difficulty in exp(arange(-8.5, -3.375, 0.0625)):
    if(any([x[0]==difficulty for x in dctimes])): continue
    dctimes.append(gettimes(n, difficulty, 100))
    print(dctimes[-1], ",")
    if(dctimes[-1][-1] > 5*60): break # If it takes longer than 5 CPU-minutes stop
                                      # re-run this block to add another row to dctimes.


I am working on the file "Braid Examples.ipynb".
The first task is:
This resulting curve (see below) is extremely well approximated by
$$
T(x) = \frac{1}{\lambda x} + a e^{a \lambda x}
$$
where $T(x)=T_C$ is the cohort time as a function of target $x$ and is measured at constant x within a window of time known as the "retarget window".  We assume that an algorithm will select a new target $x^\prime$ based on the data accumulated at $x$.  We assume that $\lambda$ and $a$ are constant over the window, and we measure $T_C, T_B, N_C, N_B$ for the chosen $x$.
The parameters can be understood intuitively, $\frac{1}{\lambda x} = T_B$ is the bead time.  The parameter $a$ has dimensions of time and can be thought of as the "size" of the network -- the amount of time required for a bead to traverse the network.  It is directly related to the orphan or uncle rate, in the blockchain limit, as well as what has been called "miner utilization" by other authors.
The parameter $\lambda$ is the hash rate and can be obtained along with $a$:
$$
\lambda = \frac{N_B}{x T_C N_C}; \qquad a = T_C W\left(\frac{T_C}{T_B} - 1\right)
$$
where $N_B$ is the number of beads and $N_C$ is the number of cohorts.  $W(z)$ is the
Lambert W function.
With these in hand we can compute the location of the minimum
$$
x_0 = \frac{2 W\left(\frac12\right)}{a \lambda} = \frac{0.7035}{a\lambda}
$$
This is independent of network topology.  (See the topology parameter of class Network -- this graph
is for miners distributed on the surface of a sphere, but setting topology to something else generates
a random network)
Can you write Python code for it?

x = np.array([p[0] for p in dctimes])
def f(x,lam,a): return 1/lam/x+a*np.exp(a*lam*x)
y = np.array([p[1] for p in dctimes])
((lam,a),dummy) = scipy.optimize.curve_fit(f, x, y, [100, 1])
plt.loglog(x, y, label='simulation')
plt.loglog(x, f(x, lam, a), label='fit $\lambda=%3.2f$ hashes/s; $a=%1.4f$s'%(lam,a))
plt.xlabel("target difficulty $x$")
plt.ylabel("cohort time $T(x)$")
x0 = numpy.real(2*scipy.special.lambertw(1/2)/a/lam)
plt.axvline(x0, label="Fastest cohort time $T_C^\prime=%1.3f$s at target $x_0=%f$"%(f(x0,lam,a),x0), color='red')
plt.xlim(x[0], x[-1])
plt.legend(loc='lower left', frameon=False);


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
About $7.88\%$ of our entire data set contains missing values. What about the contribution of each individual feature to our $7.88\%$? Let's visualize the same.
Can you write Python code for it?

frame = (df.isnull().sum() / nmissing).sort_values() * 100
axes  = frame.plot.barh()

axes.set_title('Percentage of Missing Values')
axes.set_xlabel('%')
axes.tick_params(labelsize = 8)

plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Absolute Function
Can you write Python code for it?

# an absolute function
def absolute(x):
    return np.abs(x)

# plotting a relu function and its first order derivative
_, ax = plt.subplots()
ax.plot(x, absolute(x), lw = 2)

ax.grid(True, which = 'both')

ax.axhline(y = 0, color = 'k')
ax.axvline(x = 0, color = 'k')

plt.title("Absolute Function")

plt.legend(["$\phi(x)$"])

plt.xlabel('$x$')
plt.ylabel("$\phi(x)$")

plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Aha! We've successfully reduced our number of missing values. In case of our categorical features, we've considered a string entity "NA" in place of our global constant  np.nan.
3.A.b. Duplicate Data
Duplicate tuples act redundant and serve no purpose for our classifier. We can count the number of duplicate tuples within our dataframe using the DataFrame.duplicated() function as follows:
Can you write Python code for it?

nduplicates = df.duplicated().sum()
nduplicates


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Data Set
Remember, our data set requires to have a great distribution of words. At the same time, our model must be capable of learning literature styles, something that can relate to our users. What makes one find a sentence fascinating? I'd say quotations.
Project 2

Park-o-Matic: Autonomous Vehicle Parking Using Recurrent Neural Networks (RNNs)/LSTMs

1. Problem Statement
1.A Statement

Given an array of 6 Ultrasonic sensors mounted on a Remote Controlled Car, the objective is to train an Artificial Neural Network (RNN/LSTM) that is able to predict the next possible step of a path destined to parallelly park the car by using the values of the 6 sensors.

1.B Description
1.C Data Set
1.D Motivation
Searching an appropriate parking space on roads where the width of the road is comparatively small is in itself a task. Moreover, a large amount of time is consumed in parking the vehicle in a desired orientation within the given constraint space.
Our feature vector consists of 6 ultrasonic sensor (HC-SR04) values. Each of our sensor initially discharges a trigger pulse that awaits to recieve an echo pulse back. Here, the pulse width is directly proportional to the distance between the vehicle and the object. Our target vector consists of two variables - direction and rotation of the wheel.
Our remote controlled car is assumed to remain at a constant speed. Two motors of 100 RPM (rotations per minute) each are dedicated
Problems
[1] Titanic: Machine Learning from Disaster: Predict one's survival on the Titanic
Problem 1

Titanic: Machine Learning from Disaster: Predict one's survival on the Titanic

We begin moving towards our first Kaggle problem, somewhat a "Hello, World!" substitute for Data Science. Before even begining to articulate a well-defined problem statement, I suggest you try to flush out everything you know about the Titanic, including the movie. Forget all observations and inferences you made or can make if you heard the word - Titanic. Let's assume we're given no other information except the...
Can you write Python code for it?

def load_titanic():
    path = {
        'train': './data/titanic/train.csv',
        'test' : './data/titanic/test.csv'
    }

    df   = {
        'train': pd.read_csv(path['train']),
        'test' : pd.read_csv(path['test'])
    }

    return df


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Hmm, just a bunch of numbers. UCI's repository site has a clear description of the names of the aliases within our data frame. Let's visualize our data set considering the relationship between average number of rooms per dwelling (RM) and its corresponding house prices (HP).
We'll be using matplotlib for graph visualizations and seaborn for some pretty plotting.
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns
# a magic keyword to display matplotlib outputs on jupyter notebooks
% matplotlib inline


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Hmm. Well, the error seems to have reached its minimum at $9.92 \times 10^{5}$ after being influenced by $\theta_{0}$. Raising yet another dimension, we could visualize a 3-dimensional plot with contours reflected at each axis.
Can you write Python code for it?

def plot_surface(x, y, z, labels = { 'x': '', 'y': '', 'z': '' }):
    axes = plt.gca(projection = '3d')
    axes.plot_surface(x, y, z, alpha = 0.1)

    axes.set_xlabel(labels['x'])
    axes.set_ylabel(labels['y'])
    axes.set_zlabel(labels['z'])

    plt.tight_layout()

X, Y   = np.meshgrid(theta0, theta1)
Z      = np.atleast_2d(errors)

labels = { 'x': '$\\theta_{0}$', 'y': '$\\theta_{1}$', 'z': '$C(\\theta_{0}, \\theta_{1})$' }
plot_surface(X, Y, Z, labels)

axes   = plt.gca(projection = '3d')
# plotting Cmin
axes.scatter(mintheta0, mintheta1, minerror)

plt.show()

axes = plt.gca(projection = '3d')

axes.contour(X, Y, Z, zdir = 'x', offset = np.amin(X), cmap = cm.coolwarm)
axes.contour(X, Y, Z, zdir = 'y', offset = np.amax(Y), cmap = cm.coolwarm)
axes.contour(X, Y, Z, zdir = 'z', offset = np.amin(Z), cmap = cm.coolwarm)

axes.set_xlabel('$\\theta_{0}$')
axes.set_ylabel('$\\theta_{1}$')
axes.set_zlabel('$C(\\theta_{0}, \\theta_{1})$')

plt.title('Contours')

plt.tight_layout()
plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
I've created a function load_cotton which returns our data wrapped into a Pandas's DataFrame Object.
Can you write Python code for it?

def load_cotton():
    path = './data/cotton/cotton.csv'
    df   = pd.read_csv(path)

    return df

df = load_cotton()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
If we were to consider for a range of $\theta_{1}$ values (keeping our $\theta_{0}$ as it is), we can then plot a graph like the following
Can you write Python code for it?

import sys

theta1    = np.arange(-10, 10, 0.1) # a range of theta1
mintheta1 = sys.maxsize
minerror  = sys.maxsize

for t in theta1:
    prediction = regressor.predict(X, [[0], [t]])
    error      = mean_squared_error(prediction, y, nsamples)

    if error < minerror:
        mintheta1 = t
        minerror  = error

    plt.plot(t, error, marker = '.', c = 'b')

plt.xlabel('$\\theta_{1}$')
plt.ylabel('$C(\\theta_{1})$')

plt.grid()
plt.show()

display(Math("C_{min}(0.00, %.2f) = %.2e" % (mintheta1, minerror)))


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
In case of our numerical feature variables, we successfully imputed our missing values with the mean of that feature vector. Let's view the reduction of NA/NaN's present within our data set.
Can you write Python code for it?

nmissing = (df.isnull().sum().sum() / df.size) * 100
nmissing


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
In order to view our data set in a much better way, we'll be using Pandas' DataFrame Object everytime. DataFrame is a very powerful data structure for not just storing data sets, but also for data munging, transforming, etc. Let's go ahead and import pandas.
Can you write Python code for it?


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
In the case of the above graph, we see that setosa species (blue) has a small sepal length with a comparatively larger petal length, whereas virginica species (red) has a larger sepal length but a smaller petal length in comparision. Such a kind of problem could be termed as a Classification problem, i.e. choosing a categorical output based on some input variables (categorical or numeric). If we were to draw boundaries within the scatter plot, we could approximately classify a given iris species from a list of 3 based on the sepal length and petal length provided.
Machine Learning is all about building those boundaries.
3.A.b.i Logistic Regression
Logistical Function
A logistical function can be denoted as:
$$f(x) = \frac{H}{1 + e^{-k(x - x_{0})}}$$
where $H$ is the maximum value, $k$ is its steepness and $x_{0}$ is the value of the function's mid-point.
4. Artificial Neural Networks
An Artificial Neural Network (hereby, ANN) is a learning model that is based on the way a Biological Neural Network works.
Such models prove effective in cases wherein our data set is not linearly seperable i.e., one cannot seperate input to output mappings by merely drawing a line.
Biological Neuron
Our biological neural network system consists of billions of interconnected small units called as neurons. Let's look at a simplistic biological neuron and visualize our artifical model from it later.

So, how do ANNs work? We now describe the smallest computational unit of an ANN - neuron.
McCulloch-Pitts' Neuron

Image Source: [6]
Each neuron is connected to other neurons by a connection we call as the synapse. Each synapse has a weight $w$ associated to it. Consider this as the strength of the neuron that can be updated to increase the performance of the architecture. At each neuron, a $net$ is calculated by a summation of the products of each input to the neuron through a synapse and the weight associated to it on that synapse. The output of a neuron is given by $\phi(net)$ where $\phi$ denotes the activation function of that...
Can you write Python code for it?

# an identity function
def identity(x, derivative = False):
    if derivative:
        return np.sign(x)
    return x

# testing our activation functions using a range(-10, 10, 0.01)
x = np.arange(-10, 10, 0.01)

# plotting an identity function and its first order derivative
_, ax = plt.subplots()
ax.plot(x, identity(x)                   , lw = 2)
ax.plot(x, identity(x, derivative = True), lw = 2)

ax.grid(True, which = 'both')

ax.axhline(y = 0, color = 'k')
ax.axvline(x = 0, color = 'k')

plt.title('Identity Function')

plt.legend(["$\phi(x)$", "$\phi'(x)$"])

plt.xlabel('$x$')
plt.ylabel("$\phi(x)/\phi'(x)$")

plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Index

Problem Definition
Statement
Description
Data Set
Motivation

Domain Knowledge
Data Preprocessing
Data Cleaning
Missing Values
Duplicate Data
Outliers

Data Integration
Data Reduction
Data Transformation

Modelling
k-NN (k-Nearest Neighbors)
Support Vector Machines (SVM)

Model Evaluation
Train/Test Splitting
Cross Validation
K-fold Cross Validation

1. Problem Definition
Probably one of the most neglected step in approaching any problem, is formulating the problem itself. A mere definition of the problem won't make much of a difference unless you deep dive into the problem and understand its true required outcome. When one tends to deviate from our original task, we can always refer to our original problem statement.
Let's define a formal description of our problem.
1.A. Statement

Given a data set containing 36 germplasm features corresponding to any one of the six groups (Marker, Variety, Okra, New, BBR and JBWR) of cotton, build an efficient prediction model that is able to classify under which group does any given input set of germplasm features of cotton lie within.

1.B. Description
Based on our Statement, our problem can be termed as a multi-class classification problem. Our job is to analyse and build an efficient model that is able to classify any given cotton species with respect to any of the six groups.
Although our job sounds simple, nothing can be said until we have a good view of our data set.
1.C. Data Set
Our data set is been provided by the Central Institute of Cotton Research (CICR), Nagpur, India. An initial insight into our acquired data set shows us a high degree of unstructuredness, noise and ambiguities within our data set. We have recieved a data set in its utmost raw form. Suprisingly, a large number of real-world data sets are raw, and unstructured in all ways you can possibly imagine. You can have a look of our raw data set on this link.
A great amount of data preprocessing will be required only after which we will be able to analyse and profile our data set...
Can you write Python code for it?


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Inference
Andrew's Curves speaks volumes about our data set.
* There aren't any irregular patterns within the curves of various groups which denotes absense of outliers.
* Andrew's Curves highly depend on the order of importance of variables.
3.C. Data Reduction
Not all features play an impressive role for our prediction model. We can get done away with features that are redundant and have no appreciable affect over our model's prediction. Data Reduction decreases the possibility of overfitting, improves our training time (lesser the number of features, lesser the computation required for learning), simplicity and relavancy.
Our data set consists of 36 features, 1 label and 191 observations. While this may not sound a lot, a large number of data sets used for building prediction models may have an extensive number of features and observations. In our case, there may exists features that may impact the overall performance of our models.
So, how do we go about it?
Feature Selection
Isabelle Guyon and Andre Elisseeff provide an excellent checklist on selecting your features and we shall follow the same. [24]
Checklist

Do you have domain knowledge? - Let's be honest, we have no domain knowledge.
Are your features commensurate? - Of course not. We'll be encoding them followed by normalizing our data set in the Data Transformation phase.
Do you suspect interdependence of features? -
Do you need to prune the input variables? -
Do you need to assess features individually? - Hmm...yes. We'd like to know what truly affects the performance of our model. Hence, we're directed to use a varaible ranking method for baseline results.
Do you need a predictor? - Yes, yes we do.
Do you suspect your data is “dirty”? - With the efforts we've put into cleaning our data well, I don't suspect that our data is dirty anymore.
Do you know what to try first? - Yes, we clearly have a list of algorithms to-be-ready for building our model.
Do you have new ideas, time, computational resources, and enough examples? - Absolutely not.
Do you...
Can you write Python code for it?

dfoh = pd.get_dummies(df, prefix_sep = ' ')
dfoh.sample(nsamples)


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Let's discuss another kind of Supervised Learning problems.
3.A.b Classification
Let's consider yet another data set to work on in order to understand what Classification is all about. We'll consider a famous data set provided by seaborn - Ronald Fisher's iris data set provided under load_dataset().
Can you write Python code for it?

import seaborn as sns
iris = sns.load_dataset('iris')


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Let's estimate accuracy of our prediction. For this, we'll be using sklearn's accuracy_score function under the metrics module.
Can you write Python code for it?

from sklearn.metrics import accuracy_score


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Let's look at a sample of our data set.
Can you write Python code for it?


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Let's take a look of our data set.
Can you write Python code for it?


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Literature Review
Natural Language Processing (NLP)
Natural Language Processing is concerned with problems that involve textual data. This is not to be confused with Speech Processing (a subdomain of NLP), since speech processing clearly deals with converting a speech signal into textual data. Given that there exists models that can learn textual data, how do we ensure that it is capable of learning a language that is independent per se of the language? Also, what makes one word similar or different from the other besides its meaning?
Natural languages have not less than a million words, each exhibiting some degree of uniqueness. Wrapping around a learning model that can compare a million words with each other sounds like a very high-dimensional space problem. One needs to consider say, the position of the alphabets, singular v/s plural words, words that are solely verbs, words denoting a tense (past, present and future), etc.
A nice way to encode all the relavant information for a given word is to represent it in the form of a vector.
Probability Theory
Classical Intuition
It's well known that if you were to flip a coin (a classical example to introducing probability theory), there is a 50%-50% chance for it to fall with a face either being a head or tails. Although not necessary, probability is always concerned with events that have some degree of randomness. Let me put forward this in a better way - Probability is always concerned with events that have some degree of uncertainity. So, what makes an even $E$ as such to make us conclude that given intuition?
We denote the probability of an event $E$ as $\mathbb{P}(E)$, where $\mathbb{P}(E)$ is within the range $[0, 1]$. In the case above, there are a maximum of two events occuring (the face side could be either heads or tails) whereas there is just some condition depending on the event $E$ (in this case, the number of times a tail occures) alone. Hence,
if $a$ is the number of possibilities based on our condition and $b$ be the total number of...
Can you write Python code for it?

from tensorflow.python.ops import rnn_cell


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Model Evaluation
Training/Testing Splits
Our learning models require a unit that can quantify the overall performance of our model on our data set. Our training phase relies heavily on our data set and on our defined parameters (for parameteric models only) in order to build itself as a successful predictive model. However, using the entire data set would mean that our goal seems to be fitting our model to our data set alone, which was never the objective in the first place. Exhaustively utilizing the data set and tweaking parameters would possibly result in an ideal model and not a predictiion model. We call this as overfitting, an ML engineer's worst nightmare.
One good way to avoid this would be to split our data set into two parts of some defined ratio - a training set and a testing set. Random tuples from the data set can be added into our testing set that can be used to estimate how well our model performs for our data set. scikit-learn has a function called train_test_split under the model_selection module (in scikit-learn 0.17 and before, the funciton is under the cross_validation module) that can divide a data set into a desired ratio of training and testing sets.
Can you write Python code for it?

from sklearn.model_selection import train_test_split


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
NOTE: Link to Data Description.
Our data set consists of a mixture of both - categorical as well as numeric features associated to one target variable - Group. As stated in our problem description, our problem is a multi-class classification problem.
Our first question would be - are the number of tuples for each class equal?
Let's visualize our question and infere from what we achieve.
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns

frame = (df.Group.value_counts() / len(df)) * 100
axes  = frame.plot.pie(autopct = "%.2f")

axes.set_title('Class Frequency Distribution')
axes.set_aspect('equal')

plt.axis('off')
plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Nevertheless, the clutches of overfitting still cling on to us. During our training phase, our parameters are tweaked in order to fit on our training set. In someway, a leak of information of the testing set occurs during our training set, thereby leading to an overfit. In order to avoid this, we could go one step further by creating yet another set called as a validation set that can be used to estimate our model's performance after our training phase, then moving towards our testing phase.
Now, a split of our data set into three seperate entities (training, testing and a validation set) might decrease the the number of required training tuples, thereby limiting the overall performance of our learning model.
Q. "Alright! So how do we really avoid overfitting?
This brings us to the concept of Cross Validation (CV).
Cross Validation
Cross Validation helps us to answer the question, "how well does my learning model work on any independent data set?"
K-fold Cross Validation
We divide our data set into $K$ blocks. Then for each $i^{th}$ block (where $i = 1,2,...,K$), we assume the remaning $K - 1$ blocks to be our training set and $i$ to be our validation set for that given iteration. Hence, each block shall have $\frac{u}{K}$ observations where $u$ is the total number of observations in our training set.

A cumulative error (called as the Cross Validation Error) is calculated based on the errors recieved at each iteration from our validation set. Hence for any given model, our goal would be to minimize our Cross Validation Error.
An ideal Cross Validation approach would be to divide our data set into $u$ blocks, i.e, $K = u$. We call this, Leave One Out Cross Validation (LOOCV). For a data set with very limited observations, such an approach would be favourable but however in cases where we have millions of observations, an LOOCV would be computationally expensive. Typically, $K = 10$ (also known as 10-fold Cross Validation).
scikit-learn has an extensive number of classes designated for various Cross Validation...
Can you write Python code for it?

from sklearn.model_selection import StratifiedKFold


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Our data set consists of imbalanced observations for each class. We can calculate the percentage contribution of each group to our data set as follows:
Can you write Python code for it?

frame = (df['Group'].value_counts() / len(df)) * 100
frame.plot.pie(autopct = "%.2f", figsize = (6, 6))

plt.axis('off')
plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Our next task is clearly to convert our sklearn data Bunch into a Pandas' DataFrame. Since our data set has been stored as a numpy array, we'll go ahead and import numpy as well.
Can you write Python code for it?

import numpy as np

df = pd.DataFrame(data    = np.c_[boston['data'], boston['target']],
                  columns = list(boston['feature_names']) + ['HP'])


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Plant: Growth Habit and Leaf: Nectaries feature vectors seem to consists a high amount of missing data. Will dropping these columns affect the overall performance of our model? What about post imputation?
Well, nothing can be said for now.
Let's build a data set transfomer that can impute missing data within our data set. We can then pass such a transformer with a desired algorithm through our pipeline.
sklearn has a powerful TransformerMixin class that can be utilized to build our Imputation class. Our class shall be just a wrapper around fancyimpute (a library containing a wide range of imputation stratergies).
NOTE: sklearn has a class Imputer dedicated for imputing missing values. We'll be building an extension that implements strategies beyond ones provided by sklearn
Can you write Python code for it?

import numpy as np
from sklearn.base import TransformerMixin
from fancyimpute import SimpleFill, KNN, MICE

class DataFrameImputer(TransformerMixin):
    NUMERIC     = 0x000100
    CATEGORICAL = 0x000200
    '''
    Parameters
    ----------
    imputer - a fancyimpute class with desired parameters
    copy    - whether you'd like to perform an inplace imputation or return a copy
              default returns a copy
    impute  - NUMERIC, CATEGORICAL kinds of imputataion
              default is imputation on only numeric columns
    '''
    def __init__(self, imputer = SimpleFill(), impute = NUMERIC, copy = True):
        self.imputer = imputer
        self.copy    = copy
        self.impute  = impute

    def fit(self, *_):
        return self

    def transform(self, df):
        columns     = df.columns
        numeric     = list(column for column in columns if df[column].dtype != 'object')
        categorical = list(column for column in columns if df[column].dtype == 'object')
        frame       = df.copy() if self.copy else df

        frame[numeric]     = frame[numeric].fillna(np.nan)
        frame[categorical] = frame[categorical].fillna('NA')

        try:
            if self.impute is DataFrameImputer.NUMERIC:
                nparray        = frame[numeric].as_matrix()
                frame[numeric] = self.imputer.complete(nparray)
        except ValueError:
            pass

        return frame

imputer = DataFrameImputer(imputer = MICE(verbose = False), copy = False)
imputer.fit_transform(df)
df.sample(nsamples)


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Q. "Great! We can now plug in these parameters and have our best fit line."
Not really. In our first example, we considered $\theta_{0} = 0$ whereas in our second example we limited our range for $\theta_{0} = [-10, 10)$. Both of these examples have resulted in a different minimized error, i.e. we limited our sack of $\theta$ values and then went on a hunt for our parameters that minimizes our cost function. Many sacks of $\theta$s results many minimized errors but then, what would be our optimal solution? Our approach of trying many permutations of these ranges is not a great one. So let us refer to our graph once again.
Can you write Python code for it?

plot_surface(X, Y, Z, labels)

# plotting Cmin
axes = plt.gca(projection = '3d')
axes.scatter(mintheta0, mintheta1, minerror)

plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
ReLU Function
Can you write Python code for it?

# a relu (Rectifier Linear Unit) function
def relu(x, derivative = False):
    if derivative:
        # the derivative of a ramp function is nothing but a heaviside function
        return heaviside(x)
    return np.maximum(0, x)

# plotting a relu function and its first order derivative
_, ax = plt.subplots()
ax.plot(x, relu(x)                   , lw = 2)
ax.plot(x, relu(x, derivative = True), lw = 2)

ax.grid(True, which = 'both')

ax.axhline(y = 0, color = 'k')
ax.axvline(x = 0, color = 'k')

plt.title("ReLU Function")

plt.legend(["$\phi(x)$", "$\phi'(x)$"])

plt.xlabel('$x$')
plt.ylabel("$\phi(x)/\phi'(x)$")

plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
ST
The quadratic nature of the cost function speaks volumes. Squaring the difference helps us eliminate the positive and negative feedback of errors and at the same time increase the spread on a wide range. The factor $\frac{1}{2}$ is multiplied with the mean cost since minimizing the overall error is same as minimizing half its error. A clear reason as to why we've chosen the MSE as our cost function will be discussed in some time.
Our job is to find a parameter set $\theta$ that could minimize the cost function close or equal to 0.
Let's consider our Univariate Linear Regressor again, this time considering $\theta_{0}$ constant (i.e., having no affect over our cost function $C(\theta)$) and estimate the cost with respect to the parameters we've used.
Knowing $\theta_{0} = 0.0$
Can you write Python code for it?

from IPython.core.display import display, Math

parameters  = np.array(
    [[[0.0], [0.0]], # case 1
     [[0.0], [0.5]], # case 2
     [[0.0], [1.5]]] # case 3
, dtype = np.float32)
predictions = [ ]
nsamples    = len(X)

for p in parameters:
    prediction = regressor.predict(X, p)
    predictions.append(prediction)

for i in range(len(predictions)):
    error = mean_squared_error(predictions[i], y, nsamples)
    display(Math("Case %i: C(%.2f, %.2f) = %.2e" % (i + 1, parameters[i][0], parameters[i][1], error)))


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Since Andrew's Plot is a plot of our features versus a smooth curve function, we would require to convert our data set into something numeric. In case of categorical data, one way to achieve our objective is to use scikit-learn's LabelEncoder class that converts our categorical data corresponding to a numerical value.
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder
from collections           import defaultdict

le         = defaultdict(LabelEncoder)

dfle       = df.copy()
dfle       = dfle.apply(lambda x: x if x.dtype != 'object' else le[x.name].fit_transform(x))

dfle.sample(nsamples)

labels = dfle['Group'].unique()
groups =   le['Group'].inverse_transform(labels)


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
Such encoding schemes often tend to increase the number of features by a huge amount, thus adding a computation overhead on our learning models. In our case, these columns increased almost three times the initial number of columns (37 to 125).
NOTE: Given that we have a multi-class classifier to-be-built, performing a One-Hot encoding now doesn't seem to be the best time. We'll be performing it over our data set, later. We use our Label encoded DataFrame dfle.
Feature Scaling
Assuming our data set, one possibly cannot compare say fibre color to plant height, that would be comparing apples to oranges. Our data set consists of incomparable features and hence must be squashed to some uniform range. We call this stage as Feature Scaling.
Normalization
We can define our normalized data set within the range $[0, 1]$ as:
$$X := \frac{X - X_{min}}{X_{max} - X_{min}}$$
where $X$ is our feature vector.
Can you write Python code for it?

columns         = list(df)
features        = [column for column in columns if column != 'Group']
dfstd           = dfle.copy()
dfstd[features] = (dfle[features] - dfle[features].min()) / (dfle[features].max() - dfle[features].min())
dfstd.sample(nsamples)


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
We can calculate the percentage contribution of missing values to our data set as follows:
Can you write Python code for it?


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
We could further classify supervised learning into two kinds of problems.
2.A.a. Regression
Let's consider a data set to work on in order to understand what Regression is all about. scikit-learn (a machine learning library in Python) provides many toy data sets and we'll consider one of them - the Boston House Pricing data set provided under load_boston() within the module datasets. University of California, Irvine provides a wide range of open data set repositories to work on, the Housing Prices being one. [5]
Can you write Python code for it?

from sklearn.datasets import load_boston
boston = load_boston()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
We see that the accuracy reaches its maximum at $\log_{2}{C} = 5$ $\approx 75\%$.
Artificial Neural Network
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier

accuracy = stratified_cross_validate(Xtrain, ytrain, MLPClassifier)

print('Accuracy ANN: ' + str(accuracy))

low, high  = 100, 150
hsizes     = [i for i in range(low, high + 1)]

accuracies = [stratified_cross_validate(Xtrain, ytrain, MLPClassifier,
                                        hidden_layer_sizes = (size,)) * 100 for size in hsizes]

plt.plot(hsizes, accuracies)

plt.xlabel('hidden neurons')
plt.ylabel('accuracy')

plt.show()


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
When it comes to sampling our data set (including splitting our data set into training and testing sets), the ratio of each group's contribution must be consistent.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

y         = dfle['Group'].copy()
_, sample = train_test_split(dfle, test_size = (nobservations/len(dfle)), stratify = y)

andrews_plot(sample, 'Group', groups)


I am working on the file "Brains from Scratch - Machine Learning for dummies.ipynb".
The first task is:
where $$f_{x}(t) = \frac{x^{(i)}{0}}{\sqrt{2}} + x^{(i)}{1}\sin{(t)} + x^{(i)}{2}\cos{(t)} + x^{(i)}{3}\sin{(2t)} + x^{(i)}_{3}\cos{(2t)} + \dots$$
Argh! Plotting all our observations seem to overlap curves of some groups. Andrew's Plot is impressively clear for 20 observations or less. Let's sample about 20 random observations from our data set for a better picture.
Can you write Python code for it?


I am working on the file "BrambleXu/play5/play5.ipynb".
The first task is:
两种情况
把resize的image保存到和原image同一目录下
http://pillow.readthedocs.io/en/3.1.x/reference/Image.html#create-thumbnails
这个直接通过thumbnail（）这个方法来做到resize。
在python3.5中，glob.glob通过**来递归得寻找所有符合.jpg的文件，比起2.7来说确实方便很多。
另外为了保存到和原image同一目录下，我们要os.path.splitext(path)
分离文件名与扩展名；默认返回(fname,fextension)元组，可做分片操作.
下面的file，ext里，file用来当做绝对路径，最后用im.save( file + ".thum.jpg")。其中.thum.jpg只是为了方便我们知道哪些文件是重新制作的，也可以直接用'.jpg'
Can you write Python code for it?

from PIL import Image
import glob, os

size = 100, 100

for infile in glob.glob("**/*.jpg"):
    file, ext = os.path.splitext(infile)
    print (file)
    print (ext)
    im = Image.open(infile)
    im.thumbnail(size)
    im.save( file + ".thum.jpg")
    #print (im.format, im.size, im.mode)


I am working on the file "Bratislava/201507/enigma.ipynb".
The first task is:
A nakoniec si všetky naše funkcie dáme dohromady a vytvoríme ešte jednu, ktorá bude schopná prekladať celý text.
Can you write Python code for it?

def convert(letter):
    # Vypocitame posun na prvom valci, od inicializacneho pismena
    shift = ord(letter) - 65
    # Pomocna konstanta, ktora oznacuje ci sa ma otocit aj nasledujuci valec
    rotate_next = False

    # Prechadzame rotormi v enigme od konca na zaciatok
    for i, rotor in enumerate(reversed(ENIGMA)):
        # Najskor sa posunie rotor o jednu poziciu pri stlaceni klavesy
        if i == 0 or rotate_next:
            rotate_next = rotate_rotor(rotor)

        conv_letter = alpha_rotate(rotor['position'], shift)
        letter = rotor_right_to_left(conv_letter, rotor['values'])
        shift = calculate_move(rotor['position'], letter)

    shift = reflector(shift)

    # Reflektovany prud prechadza rotormi od zaciatku na koniec
    for rotor in ENIGMA:
        conv_letter = alpha_rotate(rotor['position'], shift)
        letter = rotor_left_to_right(conv_letter, rotor['values'])
        shift = calculate_move(rotor['position'], letter)

    return ALPHABET[shift]

def encrypt(text):
    # Text ktory prejde enigmou
    encrypted_text = []
    # Velmi slabe osetrenie vstupu
    text = text.strip().replace(' ', '')

    for i, letter in enumerate(text, 1):
        if not letter.isalpha():
            continue

        # Uloz pismeno, ktore preslo enigmou
        encrypted_text.append(convert(letter.upper()))

        # Enigma standardne, robila text bloky po 4 znakoch
        if i % 4 == 0:
            encrypted_text.append(' ')

            return ' '.join(encrypted_text)

#################################################
# Operator, ktory chce zasifrovat spravu musi nastavit Enigmu zalozenim rotorov do vychodzich pozicii
####
ENIGMA = (
{
    'values': ROTOR_I,
    'position': 'M'
},
{
    'values': ROTOR_II,
    'position': 'C'
},
{
    'values': ROTOR_III,
    'position': 'K'
})

message_to_encrypt = 'Enigma Revealed'
encrypted_message = encrypt(message_to_encrypt)

print('Zasifrovana sprava:',...


I am working on the file "Bratislava/201507/enigma.ipynb".
The first task is:
Obdobné substitučné šifry, spolu s tranaspozičnými šiframi a ich kombináciami sa využívali až do konca 19 storočia, kedy prichádzajú prvé šifrovacie stroje. Hlavný rozdiel u Enigmy je že, pri stlačení jedného písmena dva krát po sebe, operátor dostal dva rôzne znaky.

Tak ako iné, na rotoroch založené šifrovacie stroje, je Enigma kombináciou elektrického a mechanického systému. Mechanický systém je zložený z klávesnice, sady rotačných valcov (alebo rotorov) zoradených na jednej osi a krokového mechanizmu, ktorý otáča postupne jedným alebo niekoľkými rotormi pri každom stlačení.
Šifrovanie prebieha takto: po stlačení klávesy sa uzavrie elektrický obvod. Prúd preteká rôznymi súčasťami, až sa nakoniec rozsvieti jedna z mnohých žiaroviek na paneli, čím indikuje výsledné zašifrované písmeno.
Rotory
Rotory (tiež rotačné valce) sú srdcom šifrovacieho stroja. Sú uložené vedľa seba, kolíky jedného z nich sa dotýkajú s plochými elektrickými kontaktmi susedného, uzatvárajúc elektrický obvod. Vnútri rotora sa nachádza 26 prevodov spájajúcich zhodne so zvolenou kombináciou kolíky z jednej strany s kontaktmi z druhej. Spôsob káblového prepojenia je iný pre každý typ rotora.

Enigmy boli vybavené niekoľkými typmi rotorov, no v čase zavedenia ich mali iba tri, krátko po vypuknutí vojny bola súprava rotorov rozšírená na 5, z ktorých sa do použitia v stroji vyberali tri. Kvôli odlíšeniu boli rotory označené rímskymi číslicami I, II, III, IV a V.
|   | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I | E | K | M | F | L | G | D | Q | V | Z | N | T | O | W | Y | H | X | U | S | P | A | I | B | R | C | J |
|II | A | J | D | K | S | I | R | U | X | B | L | H | W | T | M | C | Q | G | Z | N | P | Y | F | V | O | E |
|III| B | D | F | H | J | L | C | P | R | T | X | V | Z | N | Y | E | I | W | G | A | K | M | U | S | Q | O |
|IV | E | S | O | V | P | Z | J | A | Y |...
Can you write Python code for it?

ROTOR_I = 'EKMFLGDQVZNTOWYHXUSPAIBRCJ'
ROTOR_II = 'AJDKSIRUXBLHWTMCQGZNPYFVOE'
ROTOR_III = 'BDFHJLCPRTXVZNYEIWGAKMUSQO'
ROTOR_IV = 'ESOVPZJAYQUIRHXLNFTGKDCMWB'
ROTOR_V = 'VZBRGITYUPSDNHLXAWMJQOFECK'

ROTOR_I_NOTCH = 'Q'
ROTOR_II_NOTCH = 'E'
ROTOR_III_NOTCH = 'V'
ROTOR_IV_NOTCH = 'J'
ROTOR_V_NOTCH = 'Z'


I am working on the file "Bratislava/201507/enigma.ipynb".
The first task is:
Potrebujeme ešte funkciu, ktorá bude otáčať rotorom, ktorej návratová hodnota bude boolean označujúci, či rotor prešiel cez zárez a je potrebné posunúť nasledujúci rotor.
Can you write Python code for it?

def rotate_rotor(rotor):
    # Posun pozicie rotora o jednu polohu
    rotor['position'] = chr(ord(rotor['position']) + 1)

    # Kontrola zarezu, ci treba posunut aj nasledujuci rotor
    if rotor['values'] == ROTOR_I and rotor['position'] == 'R':
        return True
    elif rotor['values'] == ROTOR_II and rotor['position'] == 'F':
        return True
    elif rotor['values'] == ROTOR_III and rotor['position'] == 'W':
        return True
    elif rotor['values'] == ROTOR_IV and rotor['position'] == 'K':
        return True
    elif rotor['values'] == ROTOR_V and rotor['position'] == 'A':
        return True

    return False


I am working on the file "Bratislava/201507/enigma.ipynb".
The first task is:
Rozvodná doska
Rozvodná doska umožňovala rôznorodé okáblovanie, ktoré mohlo byť menené operátorom. Zmena okáblovania rozvodnej dosky napriek svojej jednoduchosti značne zvyšovala komplikáciu šifry Enigmy, dokonca viac než dodatočný rotor. Enigma bez rozvodnej dosky mohla byť rozkódovaná relatívne jednoduchým spôsobom, dokonca aj ručne, ale zmena písmen pomocou nej prinútila dešifrantov k použitiu špeciálnych dekódovacích strojov.
Python Enigma
Na začiatok je potrebné enigmu nastaviť. Zadefinujeme si list rotorov v ktorom, budeme mat dictionary s nastaveniami rotora.
Can you write Python code for it?

ENIGMA = (
{
    'values': ROTOR_I,
    'position': 'M'
},
{
    'values': ROTOR_II,
    'position': 'C'
},
{
    'values': ROTOR_III,
    'position': 'K'
})


I am working on the file "Bratislava/201507/enigma.ipynb".
The first task is:
Vytvoríme si dve pomocné funkcie, ktoré nám uľahčia prácu s abecedou. Keďže rotory boli kruhové, po Z vždy nasleduje A. Prvá funkcia bude zisťovať či posun v abecede pretočí abecedu alebo nie, druhá robí v podstate to isté s rozdielom že na zisťovanie nepoužíva písmeno, ale jeho pozíciu na rotore:
Can you write Python code for it?

def alpha_rotate(letter, shift):
    # Hodnota pismena v ASCII
    ascii_letter = ord(letter)

    # Ak je hodnota ASCII aj s posunom vacsia ako 90, pretocili sme sa cez Z a musime pocitat od A
    if ascii_letter + shift > 90:
        # Vrat pismeno posunute o "shift" znakov, s pretocenim abecedy od zaciatku
        return chr(64 + (ascii_letter - 90 + shift))
    else:
        # Vrat pismeno posunute o "shift" znakov
        return chr(ascii_letter + shift)

def calculate_move(position, letter):
    # Hodnota pismena na ktorom sa nachadza rotor v ASCII
    ascii_position = ord(position)
    # Hodnota vystupneho pismena
    ascii_letter = ord(letter)

    # Ak je pismeno na rotore dalej ako vystupne pismeno, musime sa pretocit
    if ascii_position > ascii_letter:
        # Od poctu znakov v abecede odpocitame poziciu rotora a pripocitame vystupne pismeno
        shift = 26 - (ascii_position - 65) + (ascii_letter - 65)
    else:
        # Od vystupneho pismena, odpocitame poziciu rotora
        shift = (ascii_letter - 65) - (ascii_position - 65)

    # Vratime celo ciselny zvysok po deleni vysledneho posunu, ak by sa rotor pretocil
    return shift % 26


I am working on the file "Bratislava/201507/enigma.ipynb".
The first task is:
Ďalšia funkcia bude predstavovať reflektor. Vieme na ktorej poziíci vstupuje prúd do reflektora a potrebujeme vrátiť ďalšiu pozíciu písmena do ktorého vstupoval prúd, t.j. pozíciu odkiaľ bude prúd vystupovať späť do rotorov:
Can you write Python code for it?

def reflector(position):
    # Poloha vstupneho pismena na reflektore
    entry_letter = REFLECTOR[position]
    # Poloha vystupneho pismena na reflektore
    exit_letter = REFLECTOR.index(entry_letter)

    # Ak sa vystupna poloha rovna vstupnej, vrat druhu polohu pismena
    # Reflektor obsahuje dane pismeno vzdy dva krat
    if exit_letter == position:
        return REFLECTOR.rindex(entry_letter)
    return exit_letter


I am working on the file "Bratislava/201508/Ludolph.ipynb".
The first task is:
Konfiguracia Ludolpha
ulozime si konfiguracny subor na miesto kde ho bude ludolph pri starte hladat:
Can you write Python code for it?

wget -O ~/.ludolph.cfg https://raw.github.com/erigones/Ludolph/master/ludolph/ludolph.cfg.example


I am working on the file "Bratislava/201508/Ludolph.ipynb".
The first task is:
V sekcii [xmpp] musíme zadefinovat username a password aby sa mal kam prihlásiť Ludolph po spusteni.
Máme plne funkčného Jabberbota, ktorý sa po spusteni prihlási a môžeme si ho pridať medzi priateľov a začať s ním komunikovať.
Pluginy v Ludolphovi
Podrobný popis tvorby pluginu na Ludolphovej wiki: https://github.com/erigones/Ludolph/wiki/How-to-create-a-plugin
Napísať si vlastný plugin do Ludolpha nie je nič zložité, Ludolph má predpripravený "hello_world" plugin projekt. Projekt slúži ako čo možno najjednoduchší príklad na vytvorenie pluginu a popisu dekorátorov. "hello_world" plugin je štrukturovaný ako inštalovateľný balíček pre PyPI. V prípade pluginu však stačí aby sa súbor nachádazal v python path, tj. aby import bol schopný importovať triedu.
Plugin na predpoveď počasia
Do "hello_world" pluginu si pridáme novú metódu, ktorá bude obsahovať nasledovný kód:
Can you write Python code for it?

import requests

def weather(self, msg):
    apikey = 'xxxxxxxxxxxxxxxxxxxxxxxxx'
    city = 'Bratislava,sk'
    url = 'http://api.openweathermap.org/data/2.5/forecast/city?q=%s&APPID=%s&units=metric'

    res = requests.get(url % (city, apikey)).json()

    # Odpoved obsahuje atribut list, ktory obsauje zoznam odpovedi zodpovedajucich najdenemu mestu.
    # Pre jednoduchost vyberieme prvu odpoved

    try:
        data = res['list'][0]
    except IndexError:
        return 'Zadane mesto %s nebolo najdene' % city
    else:
        return 'Teplota: %s, Popis: %s' % (data['main']['temp'], data['weather'][0]['description'])


I am working on the file "Breadth First Search.ipynb".
The first task is:
But the library also has the added ability to generate random graphs for us. In this case, the dense_gnm_random_graph() will generate a random graph of $G_{n,m}$ where $n$ is the node count and $m$ are the number of edges randomly distributed throughout the graph.
Can you write Python code for it?

networkx_graph_1 = nx.dense_gnm_random_graph(10,10)

nx.draw_networkx(networkx_graph_1)


I am working on the file "Breadth First Search.ipynb".
The first task is:
Much like we see above, the networkx library also has a built-in depth first search algorithm that will traverse the graph and return an unordered list of tuples of edges that are traversed. I will save a depth first search implementation over our custom Graph object for future posts.
Can you write Python code for it?


I am working on the file "Breadth First Search.ipynb".
The first task is:
Or just because, here's a list comprehension that can do the same thing, that actually shows off a bit of Python's nested list comprehension functionality. It is possible to also push the print function into the list comprehension below, but it only works in Python 3+ and but is not considered pythonic -- so I'm only leaving it to return the nested arrays that a list comprehension normally would.
Can you write Python code for it?

[[neighbor for neighbor in nx.all_neighbors(networkx_graph_1, node)] for node in nx.nodes(networkx_graph_1)]


I am working on the file "Breadth First Search.ipynb".
The first task is:
Past creating our own Vertex and Graph objects that we can use to assemble our own graphs, we can use libraries like NetworkX to create graphs and implement algorithms, like breadth first search, over them.
Can you write Python code for it?

import networkx as nx

import matplotlib.pyplot as plt

%matplotlib inline

edges = [('A','B'),('B','C'),('C','E'),
         ('E','D'),('D','B'),('E','F'),
         ('B','E')]

networkx_graph = nx.Graph()

for node in node_names:
    networkx_graph.add_node(node)

networkx_graph.add_edges_from(edges)

nx.draw_networkx(networkx_graph)


I am working on the file "Breadth First Search.ipynb".
The first task is:
The networkx library also includes many, many algorithm implementations already so we can utilize their built-in breadth_first_search algorithm, as we see below. We're able to print a traversal of the graph starting at node 0 and print the entire path taken through the graph.
Can you write Python code for it?


I am working on the file "Breadth First Search.ipynb".
The first task is:
The networkx library tends to return iterators for each object within the graph context, such as the graph iteself, or the nodes within a graph or the neighbors of a particular node within the graph. This is useful because traversal algorithms such as breadth first search tend to operator in an iterative manner.
nodes returns an iterable for the nodes in a graph
all_neighbors returns an interable for all neighbors of a passed in graph and specific node
Can you write Python code for it?

# quick hack to traverse the iterables returned
for node in nx.nodes(networkx_graph_1):
    neighbors = []
    for neighbor in nx.all_neighbors(networkx_graph_1, node):
        neighbors.append(neighbor)
    print("Node %s has neighbors : %s" % (node, neighbors))


I am working on the file "Breaking Symmetry/breaking_symmetry.ipynb".
The first task is:
No randomness in bias/weights
we can't break symmetry
Can you write Python code for it?

initialize()
for layer in [a,b,c]:
    layer.weight.data = torch.ones_like(layer.weight)
    layer.bias.data = torch.ones_like(layer.bias)

print(c.weight, c.bias)

train(), print(c.weight, c.bias)

train(), print(c.weight, c.bias)

train(), print(c.weight, c.bias)


I am working on the file "Breaking Symmetry/breaking_symmetry.ipynb".
The first task is:
Random biases, No randomness in weights
we can break symmetry in weights
Can you write Python code for it?

initialize()
for layer in [a,b,c]:
    layer.weight.data = torch.ones_like(layer.weight)

print(c.weight)

train(), print(c.weight)

train(), print(c.weight)

train(), print(c.weight)


I am working on the file "Breaking Symmetry/breaking_symmetry.ipynb".
The first task is:
Random weights and biases
train() allows us to shrink the output of neuron C, which has 10 inputs/fan-in-weights
no need to show that weights are asymmetrical, this is the usual case
Can you write Python code for it?


I am working on the file "Breast cancer predictor.ipynb".
The first task is:
Getting ready to do classification
Import cross validation from sklearn for test trian split.
Then import logistic regression from sklearn.
The goal is to spilit our data into two. A traning data set and a testing set.
We will use the training set to train our logistic regression model.
After that, we will use this model to predict cancer occurance from the features in the testing set.
Since we already know the occurance of cancer in the testing set we can comapre it with the predicted vaules to benchmark our model.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=0)

# train logistic regression model
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
logreg.fit(x_train,y_train)

y_pred_class=logreg.predict(x_test) # make predictions based on x_test and store it to y_pred_class

# Now let us see how our model is performing.
# We will start with accuracy
from sklearn import metrics
print metrics.accuracy_score(y_test, y_pred_class)

## Not bad at all. 92.9% of the time our model was able to identify breast cancer based on the training data
# Now let us see how our "intelligent" model compare to a dumb model which predicts "o" or "not cancer" all the time
# because it the most prevelent class

y_test.value_counts() # as you see "0" is more prevalent

# Now let us see the percentage of "o" in y_test. This will be the accuracy of a dumb model that predicts 0
# all the time
1-y_test.mean()

# looks like we did better than the dumb model!