I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/10-Quantopian-Platform/06-Pipelines.ipynb".
The first task is:
Data
Can you write Python code for it?

from quantopian.pipeline.data.builtin import USEquityPricing


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/10-Quantopian-Platform/06-Pipelines.ipynb".
The first task is:
Filters and Screens
Filters take in an asset and a timestamp and return a boolean
Can you write Python code for it?

last_close_price = USEquityPricing.close.latest
close_price_filter = last_close_price > 20

close_price_filter

def make_pipeline():

    mean_close_10 = SimpleMovingAverage(inputs = [USEquityPricing.close],
                                        window_length = 10)
    mean_close_30 = SimpleMovingAverage(inputs = [USEquityPricing.close],
                                        window_length = 30)
    latest_close = USEquityPricing.close.latest

    percent_difference = (mean_close_10-mean_close_30) / mean_close_30

    perc_diff_check = percent_difference > 0

    return Pipeline(columns = {
        'Percent Difference':percent_difference,
        '30 Day Mean Close':mean_close_30,
        'Latest Close':latest_close,
        'Positive Percent Diff': perc_diff_check
    })

results = run_pipeline(make_pipeline(),
                       '2017-01-01',
                       '2017-01-01')
results.head()


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/10-Quantopian-Platform/06-Pipelines.ipynb".
The first task is:
Pipelines in Quantopian IDE
Can you write Python code for it?

from quantopian.pipeline import Pipeline
from quantopian.algorithm import attach_pipeline, pipeline_output

def initialize(context):
    my_pipe = make_pipeline()
    attach_pipeline(my_pipe, 'my_pipeline')

def make_pipeline():
    return Pipeline()

def before_trading_start(context, data):
    # Store our pipeline output DataFrame in context.
    context.output = pipeline_output('my_pipeline')


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/10-Quantopian-Platform/06-Pipelines.ipynb".
The first task is:
Reverse a screen
Can you write Python code for it?

def make_pipeline():

    mean_close_10 = SimpleMovingAverage(inputs = [USEquityPricing.close],
                                        window_length = 10)
    mean_close_30 = SimpleMovingAverage(inputs = [USEquityPricing.close],
                                        window_length = 30)
    latest_close = USEquityPricing.close.latest

    percent_difference = (mean_close_10-mean_close_30) / mean_close_30

    perc_diff_check = percent_difference > 0

    return Pipeline(columns = {
                            'Percent Difference':percent_difference,
                            '30 Day Mean Close':mean_close_30,
                            'Latest Close':latest_close,
                            'Positive Percent Diff': perc_diff_check},
                    screen = ~perc_diff_check)

results = run_pipeline(make_pipeline(),
                       '2017-01-01',
                       '2017-01-01')
results.head()


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/11-Advanced-Quantopian-Topics/00-Pipeline-Example-Walkthrough.ipynb".
The first task is:
Alternative:
Can you write Python code for it?

#from quantopian.pipeline.classifiers.morningstar import Sector
#morningstar_sector = Sector()

energy_sector = sector.eq(309)


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/11-Advanced-Quantopian-Topics/00-Pipeline-Example-Walkthrough.ipynb".
The first task is:
Applying Filters and Factors
Let's apply our own filters, following along with some of the examples above. Let's select the following securities:

Stocks in Q1500US
Stocks that are in the energy Sector
They must be relatively highly traded stocks in the market (by dollar volume traded, need to be in the top 5% traded)

Then we'll calculate the percent difference as we've done previously. Using this percent difference we'll create an unsophisticated strategy that shorts anything with negative percent difference (the difference between the 10 day mean and the 30 day mean).
Can you write Python code for it?

def make_pipeline():

    # Base universe filter.
    base_universe = Q1500US()

    # Sector Classifier as Filter
    energy_sector = sector.eq(309)

    # Masking Base Energy Stocks
    base_energy = base_universe & energy_sector

    # Dollar volume factor
    dollar_volume = AverageDollarVolume(window_length = 30)

    # Top half of dollar volume filter
    high_dollar_volume = dollar_volume.percentile_between(95, 100)

    # Final Filter Mask
    top_half_base_energy = base_energy & high_dollar_volume

    # 10-day close price average.
    mean_10 = SimpleMovingAverage(inputs=[USEquityPricing.close],
                                  window_length = 10,
                                  mask = top_half_base_energy)

    # 30-day close price average.
    mean_30 = SimpleMovingAverage(inputs=[USEquityPricing.close],
                                  window_length = 30,
                                  mask = top_half_base_energy)

    # Percent difference factor.
    percent_difference = (mean_10 - mean_30) / mean_30

    # Create a filter to select securities to short.
    shorts = percent_difference < 0

    # Create a filter to select securities to long.
    longs = percent_difference > 0

    # Filter for the securities that we want to trade.
    securities_to_trade = (shorts | longs)

    return Pipeline(
        columns = {
            'longs': longs,
            'shorts': shorts,
            'percent_diff':percent_difference
        },
        screen=securities_to_trade
    )

result = run_pipeline(make_pipeline(), '2015-05-05', '2015-05-05')
result

result.info()


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/11-Advanced-Quantopian-Topics/00-Pipeline-Example-Walkthrough.ipynb".
The first task is:
Executing this Strategy in the IDE
Can you write Python code for it?

from quantopian.algorithm import attach_pipeline,pipeline_output
from quantopian.pipeline import Pipeline
from quantopian.pipeline.data.builtin import USEquityPricing
from quantopian.pipeline.factors import AverageDollarVolume,SimpleMovingAverage
from quantopian.pipeline.filters.morningstar import Q1500US
from quantopian.pipeline.data import morningstar

def initialize(context):

    schedule_function(my_rebalance,date_rules.week_start(),time_rules.market_open(hours = 1))

    my_pipe = make_pipeline()
    attach_pipeline(my_pipe, 'my_pipeline')

def my_rebalance(context,data):
    for security in context.portfolio.positions:
        if security not in context.longs and security not in context.shorts and data.can_trade(security):
            order_target_percent(security,0)

    for security in context.longs:
        if data.can_trade(security):
            order_target_percent(security,context.long_weight)

    for security in context.shorts:
        if data.can_trade(security):
            order_target_percent(security,context.short_weight)

def my_compute_weights(context):

    if len(context.longs) == 0:
        long_weight = 0
    else:
        long_weight = 0.5 / len(context.longs)

    if len(context.shorts) == 0:
        short_weight = 0
    else:
        short_weight = 0.5 / len(context.shorts)

    return (long_weight,short_weight)

def before_trading_start(context,data):
    context.output = pipeline_output('my_pipeline')

    # LONG
    context.longs = context.output[context.output['longs']].index.tolist()

    # SHORT
    context.shorts = context.output[context.output['shorts']].index.tolist()

    context.long_weight,context.short_weight = my_compute_weights(context)

def make_pipeline():

    # Universe Q1500US
    base_universe = Q1500US()

    # Energy Sector
    sector = morningstar.asset_classification.morningstar_sector_code.latest
    energy_sector = sector.eq(309)

    # Make Mask of 1500US and Energy
    base_energy =...


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/11-Advanced-Quantopian-Topics/00-Pipeline-Example-Walkthrough.ipynb".
The first task is:
Masking Filters
Masks can be also be applied to methods that return filters like top, bottom, and percentile_between.
Masks are most useful when we want to apply a filter in the earlier steps of a combined computation. For example, suppose we want to get the 50 securities with the highest open price that are also in the top 10% of dollar volume.
Suppose that we then want the 90th-100th percentile of these securities by close price. We can do this with the following:
Can you write Python code for it?

from quantopian.pipeline.factors import SimpleMovingAverage, AverageDollarVolume

# Dollar volume factor
dollar_volume = AverageDollarVolume(window_length = 30)

# High dollar volume filter
high_dollar_volume = dollar_volume.percentile_between(90, 100)

# Top open price filter (high dollar volume securities)
top_open_price = USEquityPricing.open.latest.top(50,
                                                 mask = high_dollar_volume)

# Top percentile close price filter (high dollar volume, top 50 open price)
high_close_price = USEquityPricing.close.latest.percentile_between(90, 100,
                                                                   mask = top_open_price)


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/11-Advanced-Quantopian-Topics/00-Pipeline-Example-Walkthrough.ipynb".
The first task is:
There are two main benefits of the Q500US and Q1500US. Firstly, they greatly reduce the risk of an order not being filled. Secondly, they allow for more meaningful comparisons between strategies as now they will be used as the standard universes for algorithms.
Can you write Python code for it?


I am working on the file "17-09-17-Python-for-Financial-Analysis-and-Algorithmic-Trading/11-Advanced-Quantopian-Topics/01-Leverage.ipynb".
The first task is:
Set Hard Limit on Leverage
http://www.zipline.io/appendix.html?highlight=leverage#zipline.api.set_max_leverage
Can you write Python code for it?

def initialize(context):
    context.amzn = sid(16841)
    context.ibm = sid(3766)

    set_max_leverage(1.03)

    schedule_function(rebalance,
                      date_rules.every_day(),
                      time_rules.market_open())
    schedule_function(record_vars,
                      date_rules.every_day(),
                      time_rules.market_close())

def rebalance(context,data):
    order_target_percent(context.ibm, -0.5)
    order_target_percent(context.amzn, 0.5)

def record_vars(context,data):
    record(amzn_close=data.current(context.amzn,'close'))
    record(ibm_close=data.current(context.ibm,'close'))
    record(Leverage = context.account.leverage)
    record(Exposure = context.account.net_leverage)


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/03 - Algorithms, Terminology, Data Types and Data Visualization/01 - ml_algorithm_intro_and_datatypes.ipynb".
The first task is:

Multiclass Classification

Predict a class as output based on given features
Examples:
    1. How healthy is the food based on given ingredients?
    Classes: Healthy, Moderate, Occasional, Avoid.
2. Identify type of mushroom based on features

3. What type of advertisement can be placed for this search?

Concrete Example

Iris Classification - Predict the type of Iris plant based on flower measurments

https://archive.ics.uci.edu/ml/datasets/Iris
Input Columns / Features = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width']
Output Column / Target Attribute = 'class'.
Class: Iris-setosa, Iris-virginica, Iris-versicolor
Can you write Python code for it?

df = pd.read_csv(multi_class_example)

np.random.seed(5)
# print 10 random rows
df.iloc[np.random.randint(0, df.shape[0], 10)]

df.columns

df['class'].value_counts()

df.sepal_length.describe()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/03 - Algorithms, Terminology, Data Types and Data Visualization/01 - ml_algorithm_intro_and_datatypes.ipynb".
The first task is:
Data Types
Can you write Python code for it?

df = pd.read_csv(data_type_example)

df.columns

df[['description', 'favourites_count', 'favorited', 'text', 'screen_name', 'trainingLabel']].head()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/03 - Algorithms, Terminology, Data Types and Data Visualization/01 - ml_algorithm_intro_and_datatypes.ipynb".
The first task is:
Labeled Data
Labeled Data - contains the features and target attribute with correct answer
Training Set
Part of labeled data that is used for training the model. 60-70% of the labeled data is used for training
Evaluation Set
30-40% of the labeled data is reserved for checking prediction quality with known correct answer
Training Example
Training Example or a row. Contains one complete observation of features and the correct answer
Features
Known by different names: Columns, Features, variables, attributes. These are values that define a particular example. Values for some of the features could be missing or invalid in real-world datasets.  So, some cleaning may have to be done before feeding to Machine Learning

Input Feature - Variables that are provided as input to the model
Target Attribute - Variable that model needs to predict

AWS ML Data Types

Data from training set needs to be mapped to one of these data types

1. Binary.  Can contain only two state 1/0.

Positive Values: 1, y, yes, t, true

Negative Values: 0, n, no, f, false

Values are case-insensitive and AWS ML converts the above values to 1 or 0

2. Categorical.  Qualitative attribute that describes something about an observation.

Example

Day of the week: Sunday, Monday, Tuesday,...

Size: XL, L, M, S

Month: 1, 2, 3, 4,...12

Season: 1, 2, 3, 4

3. Numeric. Measurements, Counts are represented as numeric types

Discrete: 20 cars, 30 trees, 2 ships

Continous: 98.7 degree F, 32.5 KMPH, 2.6 liters

4. Text.  String of words. AWS ML automatically tokenizes at white space boundary

Example

product description, comment, reviews

Tokenized: 'Wildlife Photography for beginners' => {‘Wildlife’, ‘Photography’, ‘for’, ‘beginners’}

Algorithms

Linear Regression

Predict a numeric value as...
Can you write Python code for it?

# read the bike train csv file
df = pd.read_csv(regression_example)

df.head()

df.corr()

df['count'].describe()

df.season.value_counts()

df.holiday.value_counts()

df.workingday.value_counts()

df.weather.value_counts()

df.temp.describe()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/01 - ml_linear_cost_example.ipynb".
The first task is:

Loss Function

Squared Loss Function. Loss is the average of squared difference between predicted and actual value.  Squared Loss function not only gives us Loss at a given weight, it also tells us which direction
to go to minimize loss.

For a given (weight, loss), algorithm finds the slope using calculus/first order derivatives.
If negative slope, increase the weight
If positive slope, decrease the weight

Learning Rate

Learning Rate decides how much the weight should be increased or decreased.

Too big of a change, it will skip the point where loss is minimal.

Too small of a change, it will take several iterations to find out where the loss is minimal.

In AWS ML, Learning Rate is automatically selected.

Quadratic Example with two features

Can you write Python code for it?

# Let's look at a quadratic example: y = x**2 + x + c
# two features: x**2 and x

def quad_func (x):
    return 25 * x ** 2 -80 * x + 64

def quad_func_weight(weight2, x):
    #For different weights of quadratic term
    # Acutal eqn. 25x^2 - 80x + 64.  We have fixed w1=-80,w0=64. need to find w2.
    return weight2 * x ** 2 -80 * x + 64

# Quadratic
np.random.seed(5)
x_vals = pd.Series(np.random.rand(150) * 20)
y_vals = x_vals.map(quad_func)

plt.scatter(x = x_vals,
            y = y_vals)
plt.xlabel('x')
plt.ylabel('y')
plt.title('Training Set - Two Features')
plt.grid(True)

weights = [0, 20, 30, 50]
y_at_weight = {}

for w1 in weights:
    y_calculated = []
    y_at_weight[w1] = y_calculated

    for x in x_vals:
        y_calculated.append(quad_func_weight(w1, x))

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = x_vals, y = y_vals, label = 'actual')
plt.scatter(x = x_vals, y = y_at_weight[0], label = 'weight 0', color = 'r')
plt.scatter(x = x_vals, y = y_at_weight[20], label = 'weight 20', color = 'g')
plt.scatter(x = x_vals, y = y_at_weight[30], label = 'weight 30', color = 'k')
plt.scatter(x = x_vals, y = y_at_weight[50], label = 'weight 50', color = 'y')
plt.xlabel('x')
plt.ylabel('Predicted y')
plt.title('Predicted Output for different weights')
plt.grid(True)
plt.legend()

# Initialize Weights for feature 2
weight = pd.Series(np.linspace(-20, 70, 200))
loss_at_wt = []
for w1 in weight:
    y_calculated = []
    for x in x_vals:
        y_calculated.append(quad_func_weight(w1,x))

    loss_at_wt.append(((y_vals - y_calculated) ** 2).mean())

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = weight,
            y = loss_at_wt)
plt.grid(True)
plt.xlabel('Weight for feature 2')
plt.ylabel('Loss')
plt.title('Loss Curve - Loss at different weight')


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/01 - ml_linear_cost_example.ipynb".
The first task is:

Try with different values for W1

Assume that w0 = 8. We need to find out optimal value for w1.
Let's try different weights and compute target attribute y
Can you write Python code for it?

weights = [3, 4, 5, 6, 7]
y_at_weight = {}

for w1 in weights:
    y_calculated = []
    y_at_weight[w1] = y_calculated
    for x in x_vals:
        y_calculated.append(straight_line_weight(w1,x))

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = x_vals,
            y = y_vals,
            label = 'actual')
plt.scatter(x = x_vals, y = y_at_weight[3], color = 'r', marker = '+', label = 'weight 3')
plt.scatter(x = x_vals, y = y_at_weight[4], color = 'g', label = 'weight 4')
plt.scatter(x = x_vals, y = y_at_weight[5], label = 'weight 5')
plt.scatter(x = x_vals, y = y_at_weight[6], color = 'y', label = 'weight 6')
plt.scatter(x = x_vals, y = y_at_weight[7], color = 'k', marker='+', label = 'weight 7')
plt.xlabel('Feature x1')
plt.ylabel('Predicted y')
plt.title('Predicted Output for different weights')
plt.grid(True)
plt.legend()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/02 - ml_linear_examples.ipynb".
The first task is:

AWS ML Estimated/Predicted

Estimated/Predicted value is reported as 'score'.

Tag is the row identifier
Can you write Python code for it?

df_predicted.head()

df_predicted.columns = ["Row", "y_predicted"]

df_predicted.index = df_predicted.Row

df_predicted.head()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/02 - ml_linear_examples.ipynb".
The first task is:

Plot actual value and predicted value

Can you write Python code for it?

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df.x,
            y = df.y_noisy,
            color = 'r',
            label = 'actual',)
plt.scatter(x = df.x,
            y = df_predicted.y_predicted,
            color = 'b',
            label = 'predicted')
plt.grid(True)
plt.legend()

# Training Data Residuals
residuals = (df_predicted.y_predicted - df.y_noisy)

fig = plt.figure(figsize = (12, 8))
plt.hist(residuals)
plt.grid(True)
plt.xlabel('(Predicted - Actual)')
plt.ylabel('Count')
plt.title('Residuals Distribution')
plt.axvline(color = 'g')
# left of 0 = prediction < actual
# right of 0 = prediction > actual

fig = plt.figure(figsize = (12, 8))
plt.boxplot([df.y_noisy, df_predicted.y_predicted],
            labels = ['actual','predicted'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('Target')
plt.grid(True)


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/02 - ml_linear_examples.ipynb".
The first task is:

Read the target predicted by AWS ML

df_predicted = pd.read_csv(os.path.join(data_path,
                                        'output_straight_line_noisy',
                                        'bp-oDvVSUKSpPe-straight_line_example_test_all.csv'))
Can you write Python code for it?


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/02 - ml_linear_examples.ipynb".
The first task is:

Test Set

Test set should contain the same features expect for the target.

Export Row Number, 'x'.
Can you write Python code for it?

# run all the samples for prediction
df.to_csv(os.path.join(data_path, 'straight_line_example_test_all.csv'),
    index = True,
    index_label = 'Row',
    columns = 'x')


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/04 - Linear Regression/02 - ml_linear_examples.ipynb".
The first task is:

Training and Evaluation Set

Export Row Number, 'x' and 'y_noisy' columns to a CSV file

Row number will not be used for training the model. It is simply used as unique identifier
Can you write Python code for it?

data_path = '..\Data\RegressionExamples\straight_line'

df.to_csv(os.path.join(data_path, 'straight_line_example_all.csv'),
          index = True,
          index_label = 'Row')

# 130 rows for Training + Eval Set
df[df.index < 130].to_csv(os.path.join(data_path,'straight_line_noisy_example_train.csv'),
                          index = True,
                          index_label = 'Row'
                          ,columns = ['x','y_noisy'])


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/01 - ml_quadratic_examples.ipynb".
The first task is:

Hands-on: Linear Regression - Complex Shapes

Test 1

Input Features: x

Output / Target: y_noisy

Objective: Underfitting demo
Test 2

Input Features: x, x^2

Output / Target: y_noisy

Objective: How adding relevant features improves predicting accuracy

Can you write Python code for it?

def quad_func (x):
    return 5 * x ** 2 -23 * x + 47

# Training Set + Eval Set: 200 samples (70%, 30% split)
# Test Set: 60 samples
# Total: 260 samples

np.random.seed(5)
samples = 260
x_vals = pd.Series(np.random.rand(samples) * 20)
x2_vals = x_vals ** 2
y_vals = x_vals.map(quad_func)
y_noisy_vals = y_vals + np.random.randn(samples) * 50

df = pd.DataFrame({'x': x_vals,
                   'x2': x2_vals ,
                   'y': y_vals,
                   'y_noisy': y_noisy_vals})

df.head()

df.corr()

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df['x'],
            y = df['y'],
            color = 'r',
            label = 'y',)
plt.scatter(x = df['x'],
            y = df['y_noisy'],
            color = 'b',
            label = 'y noisy',
            marker = '+')
plt.xlabel('x')
plt.ylabel('Target Attribute')
plt.grid(True)
plt.legend()

data_path = '..\Data\RegressionExamples\quadratic'

df.to_csv(os.path.join(data_path,'quadratic_example_all.csv'),
          index = True,
          index_label = 'Row')


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/01 - ml_quadratic_examples.ipynb".
The first task is:
Test 1:
Training RMSE: 385.16, Evaluation RMSE: 257.89, Baseline RMSE: 437.31
Test 2:
Training RMSE: 132.20, Evaluation RMSE: 63.68, Baseline RMSE: 437.31
Test 2 RMSE is much better compared to baseline. Do note that we added approx -50 to 50 noise value to y
Can you write Python code for it?

fig = plt.figure(figsize = (12, 8))
plt.boxplot([df.y_noisy,df_predicted_underfit.y_predicted, df_predicted_normal.y_predicted],
            labels = ['actual','predicted-underfit','predicted-normal'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('y')
plt.grid(True)

df_predicted_underfit.head()

df_predicted_normal.head()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/01 - ml_quadratic_examples.ipynb".
The first task is:
Test 1:
Training RMSE: 385.18, Evaluation RMSE: 257.89, Baseline RMSE: 437.31
Wojciech results:
Training RMSE: 385.16, Evaluation RMSE: 257.898, Baseline RMSE: 437.311
RMSE for the model is large and closer to baseline
Can you write Python code for it?

fig = plt.figure(figsize = (12, 8))
plt.boxplot([df.y_noisy, df_predicted_underfit.y_predicted],
            labels = ['actual','predicted-underfit'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('y')
plt.grid(True)

df.y_noisy.describe()

df_predicted_underfit.y_predicted.describe()

df_predicted_normal = pd.read_csv(os.path.join(data_path,'output_normal',
                                               'bp-In6EUvWaCw2-quadratic_example_test_all_normal.csv.gz'))
df_predicted_normal.columns = ["Row", "y_predicted"]

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df.x,
            y = df.y_noisy,
            color = 'b',
            label = 'actual',
            marker ='+')
plt.scatter(x = df.x,
            y = df_predicted_underfit.y_predicted,
            color = 'g',
            label = 'Fit (x)',
            marker = '^')
plt.scatter(x = df.x ,
            y = df_predicted_normal.y_predicted ,
            color = 'r',
            label = 'Fit (x,x^2)')
plt.title('Quadratic - normal fit')
plt.grid(True)
plt.xlabel('x')
plt.ylabel('Target Attribute')
#plt.legend()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/02 - ml_quadratic_extra_features_examples.ipynb".
The first task is:
Before Normalization: x3, x4 are very large and dominates the outcome.

 Training RMSE: 83973.66, Evaluation RMSE: 158260.62, Baseline RMSE: 437.31
After normalization: all features have similar scale, range

 Training RMSE: 72.35, Evaluation RMSE: 51.7387, Baseline RMSE: 437.31
Normalized features were able to fit target attribute better
Can you write Python code for it?

fig = plt.figure(figsize = (12, 8))
plt.boxplot([df.y_noisy,
             df_more_features_predicted.y_predicted,
             df_more_normalize_features_predicted.y_predicted],
            labels=['actual','predicted before norm','predicted norm'])
plt.title('Box Plot - Actual, Predicted')
plt.ylim((-500,5000))
plt.ylabel('y')
plt.grid(True)


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/03 - ml_underfit_overfit_normal_30samples.ipynb".
The first task is:

Demo of scikit - Underfit, Normal, Overfit example

Purpose: Demonstrate how higher order polynomials can fit complex non-linear shape
This demo contains AWS ML equivalent example of underfitting vs overfitting example described here:

http://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html
Can you write Python code for it?

# Function to generate target value for a given x.
true_func = lambda X: np.cos(1.5 * np.pi * X)

np.random.seed(0)
# Training Set: No. of random samples used for training the model
n_samples = 30
x = np.sort(np.random.rand(n_samples))
y = true_func(x) + np.random.randn(n_samples) * 0.1

# Test Set: 100 samples for which we want the model to predict value
n_test = 100
x_test = np.linspace(0, 1, n_test)
y_test_actual = true_func(x_test) + np.random.randn(n_test) * 0.1

x[:5]

x[:5],y[:5]

# Function to add more features from existing features
# in this case degree is the desired order of polynomials
# Example degree 3 with 1 feature x would output: x,x^2,x^3
# Similary, if there are multiple features x1,x2: x1,x2,x1*x2,x1^2,x1^2*x2,x1^3....and so forth
def generate_higher_order(degrees, x):
    # Generate higher order features from a given set of features.
    poly = PolynomialFeatures(degree = degrees,
                              include_bias = False)
    x_new = poly.fit_transform(x)

    df = pd.DataFrame(x_new)
    df.columns = df.columns.map(lambda n: 'x' + str(n))
    return df

data_path = r'..\Data\RegressionExamples\under_over_fit_30samples'

# degrees for feature generation
degrees = [1, 4, 15]

# Generate training set for each of the degree
for d in degrees:
    df = generate_higher_order(d, x.reshape((n_samples, 1)))
    df['y'] = y
    df.to_csv(os.path.join(data_path,'fit_degree_{0}_example_train{1}.csv'.format(d,n_samples)),
              index = True,
              index_label = 'Row')

# Generate Evaluation set.  Contains all the features and target.
# Generate Test set. Contains only the features.  AWSML would predict the target
for d in degrees:
    df = generate_higher_order(d, x_test.reshape((n_test, 1)))

    df.to_csv(os.path.join(data_path,'fit_degree_{0}_example_test{1}.csv'.format(d, n_samples)),
              index = True,
              index_label = 'Row')

    df['y'] = y_test_actual

   ...


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/03 - ml_underfit_overfit_normal_30samples.ipynb".
The first task is:

Model with degree 15 features

Can you write Python code for it?

df_d15_predicted = pd.read_csv(
    os.path.join(data_path,'output_deg_15',
    'bp-rBWxcnPN3zu-fit_degree_15_example_test30.csv.gz'))
df_d15_predicted.columns = ["Row","y_predicted"]

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df_samples['x0'],
            y = df_samples['y'],
            color = 'b',
            label = 'samples')
plt.scatter(x = df_actual['x0'],
            y = df_actual['y'],
            color = 'r',
            label = 'true function')
plt.scatter(x = df_actual['x0'],
            y = df_d15_predicted['y_predicted'],
            color = 'g',
            label = 'predicted with degree 15')
plt.grid(True)
plt.legend()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/03 - ml_underfit_overfit_normal_30samples.ipynb".
The first task is:
Good Fit with degree 4 polynomial<br>
Training RMSE:0.2563, Evaluation RMSE:0.1493, Baseline RMSE:0.689
Can you write Python code for it?

fig = plt.figure(figsize = (12, 8))
plt.boxplot([df_actual['y'],
             df_d1_predicted['y_predicted'],
             df_d4_predicted['y_predicted']],
            labels=['actual','predicted with deg1','predicted with deg4'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('Target Attribute')
plt.grid(True)


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/03 - ml_underfit_overfit_normal_30samples.ipynb".
The first task is:
Not quite over fitting as shown in sci-kit example; fit is actually pretty good here.<br>
Training RMSE:0.2984, Evaluation RMSE:0.1222, Baseline RMSE:0.689
Can you write Python code for it?

fig = plt.figure(figsize = (12, 8))
plt.boxplot([df_actual['y'],
             df_d1_predicted['y_predicted'],
             df_d4_predicted['y_predicted'],
             df_d15_predicted['y_predicted']],
            labels = ['actual','predicted deg1','predicted deg4','predicted deg15'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('Target Attribute')
plt.grid(True)


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/05 - Improving the model/03 - ml_underfit_overfit_normal_30samples.ipynb".
The first task is:
Polynomial with degree 1 is a straight line - Underfitting<br>
Training RMSE:0.5063, Evaluation RMSE:0.4308, Baseline RMSE:0.689
Can you write Python code for it?

fig = plt.figure(figsize = (12, 8))
plt.boxplot([df_actual['y'],
             df_d1_predicted['y_predicted']],
            labels=['actual','predicted with deg1'])
plt.title('Box Plot - Actual, Predicted')
plt.ylabel('Target Attribute')
plt.grid(True)


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/06 - Kaggle's Bike Sharing Demand Challenge/ml_bike_train.ipynb".
The first task is:

Training and Evaluation Summary

Casual Model:     Training RMSE: 32.69,Eval RMSE: 31.38, Baseline RMSE:  49.21
Registered Model: Training RMSE: 92.49,Eval RMSE: 92.28, Baseline RMSE: 149.34
Can you write Python code for it?

# read predicted values for casual and registered users.
# sum it up and prepare for submission
df_casual_predicted = pd.read_csv(os.path.join(data_path,
                                               'output_casual',
                                               'bp-jLEzJYrQqDj-bike_rental_test.csv.gz'))

df_casual_predicted.head()

def adjust_score(x):
    if x < 0:
        return 0
    else:
        return x

df_casual_predicted['casual_rental'] = df_casual_predicted.score.map(adjust_score)

df_casual_predicted.tail()

df_casual_predicted.casual_rental.hist(bins = 20)

df_registered_predicted = pd.read_csv(os.path.join(data_path,
                                               'output_registered',
                                               'bp-CnT4idlhEIa-bike_rental_test.csv.gz'))

df_registered_predicted['registered_rental'] = df_registered_predicted.score.map(adjust_score)

df_registered_predicted.head()

df_registered_predicted.registered_rental.describe()

df_registered_predicted.registered_rental.hist(bins = 20)

df_registered_predicted["casual_rental"] = df_casual_predicted.casual_rental

df_registered_predicted['count'] = df_registered_predicted.registered_rental + df_registered_predicted.casual_rental

df_registered_predicted.tail()

df_registered_predicted.columns = ['datetime',
                                   'score',
                                   'registered_rental',
                                   'casual_rental',
                                   'count']

df_registered_predicted.tail()

# File ready to submit on Kaggle
df_registered_predicted.to_csv(os.path.join(data_path,
                                            'predicted_test_kaggle.csv'),
                               index = False,
                               columns = ['datetime', 'count'])


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/07 - Binary Classification/01 - ml_logistic_cost_example.ipynb".
The first task is:

Classification Overview

Predict a binary class as output based on given features.

Examples: Do we need to follow up on a customer review? Is this transaction fraudulent or valid one? Are there signs of onset of a medical condition or disease? Is this considered junk food or not?

Linear Model. Estimated Target = w0 + w1x1
+ w2x2 + w3x3
+ … + wnxn

where, w is the weight and x is the feature

Logistic Regression. Estimated Probability = sigmoid(w0 + w1x1
+ w2x2 + w3x3
+ … + wnxn)

where, w is the weight and x is the feature

Linear model output is fed thru a sigmoid or logistic function to produce the probability.

Predicted Value: Probability of a binary outcome.  Closer to 1 is positive class, closer to 0 is negative class

Algorithm Used: Logistic Regression. Objective is to find the weights w that maximizes separation between the two classes

Optimization: Stochastic Gradient Descent. Seeks to minimize loss/cost so that predicted value is as close to actual as possible

Cost/Loss Calculation: Logistic loss function

Can you write Python code for it?

# Sigmoid or logistic function
# For any x, output is bounded to 0 & 1.
def sigmoid_func(x):
    return 1.0/(1 + math.exp(-x))

sigmoid_func(10)

sigmoid_func(-100)

sigmoid_func(0)

# Sigmoid function example
x = pd.Series(np.arange(-8, 8, 0.5))
y = x.map(sigmoid_func)

x.head()

fig = plt.figure(figsize = (12, 8))
plt.plot(x,y)
plt.ylim((-0.2, 1.2))
plt.xlabel('input')
plt.ylabel('sigmoid output')
plt.grid(True)

plt.axvline(x = 0, ymin = 0, ymax = 1, ls = 'dashed')
plt.axhline(y = 0.5, xmin = 0, xmax = 10, ls = 'dashed')
plt.axhline(y = 1.0, xmin = 0, xmax = 10, color = 'r')
plt.axhline(y = 0.0, xmin = 0, xmax = 10, color = 'r')
plt.title('Sigmoid')


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/07 - Binary Classification/01 - ml_logistic_cost_example.ipynb".
The first task is:
At 2.7 hours of study time, we hit 0.5 probability.  So, any student who spent 2.7 hours or more would have a higher probability of passing the exam.
In the above example,<br>
1. Top right quadrant = true positive. pass got classified correctly as pass
2. Bottom left quadrant = true negative. fail got classified correctly as fail
3. Top left quadrant = false negative. pass got classified as fail
4. Bottom right quadrant = false positive. fail got classified as pass
Cutoff can be adjusted; instead of 0.5, cutoff could be established at 0.4 or 0.6 depending on the nature of problem and impact of misclassification
Can you write Python code for it?

weights = [0, 1, 2]
y_at_weight = {}

for w in weights:
    y_calculated = []
    y_at_weight[w] = y_calculated

    for x in df.Hours:
        y_calculated.append(sigmoid_func(straight_line_weight(w, x)))

y_sig_vals = y_vals.map(sigmoid_func)

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df.Hours,
            y = y_vals,
            color = 'b',
            label = 'logistic curve')
plt.scatter(x = df[df.Pass==1].Hours,
            y = df[df.Pass==1].Pass,
            color = 'g',
            label = 'pass')
plt.scatter(x = df[df.Pass==0].Hours,
            y = df[df.Pass==0].Pass,
            color = 'r',
            label = 'fail')

plt.scatter(x = df.Hours,
            y = y_at_weight[0],
            color = 'k',
            label = 'at wt 0')
plt.scatter(x = df.Hours,
            y = y_at_weight[1],
            color = 'm',
            label = 'at wt 1')
plt.scatter(x = df.Hours,
            y = y_at_weight[2],
            color = 'y',
            label = 'at wt 2')
plt.xlim((0,8))
plt.ylim((-0.2, 1.5))

plt.axhline(y = 0.5,
            xmin = 0,
            xmax = 6,
            color = 'b',
            ls = 'dashed')

plt.axvline(x = 4,
            ymin = 0,
            ymax = 1,
            color = 'm',
            ls = 'dashed')
plt.xlabel('Hours')
plt.ylabel('Pass Probability')
plt.grid(True)
plt.title('How weights impact classification - cutoff 0.5')
plt.legend()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/07 - Binary Classification/01 - ml_logistic_cost_example.ipynb".
The first task is:
Input Feature: Hours<br>
Output: Pass (1 = pass, 0 = fail)
Can you write Python code for it?

df.head()

# optimal weights given in the wiki dataset
def straight_line(x):
    return 1.5046 * x - 4.0777

# How does weight affect outcome
def straight_line_weight(weight1, x):
    return weight1 * x - 4.0777

# Generate probability by running feature thru the linear model and then thru sigmoid function
y_vals = df.Hours.map(straight_line).map(sigmoid_func)

fig = plt.figure(figsize = (12, 8))
plt.scatter(x = df.Hours,
            y = y_vals,
            color = 'b',
            label = 'logistic')
plt.scatter(x = df[df.Pass == 1].Hours,
            y = df[df.Pass == 1].Pass,
            color = 'g',
            label = 'pass')
plt.scatter(x = df[df.Pass == 0].Hours,
            y = df[df.Pass == 0].Pass,
            color = 'r',
            label = 'fail')
plt.title('Hours Spent Reading - Pass Probability')
plt.xlabel('Hours')
plt.ylabel('Pass Probability')
plt.grid(True)

plt.xlim((0,7))
plt.ylim((-0.2,1.5))

plt.axvline(x = 2.75,
            ymin = 0,
            ymax=1)
plt.axhline(y = 0.5,
            xmin = 0,
            xmax = 6,
            label = 'cutoff at 0.5',
            ls = 'dashed')

plt.axvline(x = 2,
            ymin = 0,
            ymax = 1)
plt.axhline(y = 0.3,
            xmin = 0,
            xmax = 6,
            label = 'cutoff at 0.3',
            ls = 'dashed')

plt.axvline(x = 3,
            ymin = 0,
            ymax=1)
plt.axhline(y = 0.6,
            xmin = 0,
            xmax = 6,
            label='cutoff at 0.6',
            ls = 'dashed')
plt.legend()


I am working on the file "17-09-27-AWS Machine Learning A Complete Guide With Python/07 - Binary Classification/01 - ml_logistic_cost_example.ipynb".
The first task is:
Logistic Regression Cost/Loss Function<br>
Can you write Python code for it?

# Cost Function
z = pd.Series(np.linspace(0.000001, 0.999999, 100))
ypositive = -z.map(math.log)
ynegative = -z.map(lambda x: math.log(1-x))

fig = plt.figure(figsize = (12, 8))
plt.plot(z,
         ypositive,
         label = 'Loss curve for positive example')
plt.plot(z,
         ynegative,
         label = 'Loss curve for negative example')
plt.ylabel('Loss')
plt.xlabel('Class')
plt.title('Loss Curve')
plt.legend()


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-dataframes-10.ipynb".
The first task is:

Arbeiten mit Dataframes in Pandas

Can you write Python code for it?

import numpy as np
import pandas as pd
from pandas import DataFrame #aufpassen wie DataFrame geschrieben wird !
#Erzeugen eines Dataframes aus gleichlangen Listen
daten = {'stadt':['Stuttgart', 'Muenchen', 'Hamburg', 'Koeln','Stuttgart'],'jahr':[2014,2015,2016,2017,2015],'pop':[0.7,0.8,0.9,0.85,0.75]}
frame = DataFrame(daten) #erzeugen des DataFrames
print frame

frame = DataFrame(daten,columns=['stadt','jahr','pop'])
print frame

#Ausgabe spezieller Spalten (Columns)
print frame['jahr']
print '################'
print frame.stadt

#Ausgabe spezieller Zeilen (Rows)
frame.ix[2] #Index als integer

frame.T

import numpy as np
import pandas as pd
from pandas import DataFrame
url="sudoku-10.csv"
df = pd.read_csv(url,header=0,sep=';') #sudoku als csv einlesen
#df.describe()
print (df)

# Using `iloc[]`
print(df.iloc[0][0])

# Using `loc[]`
print(df.loc[0][1])

# Using `at[]`
#print(df.at[0,'1'])

# Using `iat[]`
print(df.iat[0,0])

# Using `get_value(index, column)`
#print(df.get_value(0, '1'))

https://www.datacamp.com/community/tutorials/pandas-tutorial-dataframe-python#gs.J_F2fGU

# Use `iloc[]` to select a row
print(df.iloc[0])

# Use `loc[]` to select a column
#print(df.loc[:,1])


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-dataframes-10.ipynb".
The first task is:

Ausgabe eines Dataframes als csv-Datei

Can you write Python code for it?

import pandas as pd
df.to_csv('myDataFrame10.csv')

import pandas as pd
df.to_csv('myDataFrame20.csv', sep=';', encoding='utf-8')


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-dictionaries-10.ipynb".
The first task is:
<h1>Vokabeltrainer entwickeln
Can you write Python code for it?

#Vokabeltrainer entwickeln
import random
#Definition der Funktionen
def dict_laden(pfad):
    d={}
    try:
        datei = open(pfad)
        liste = datei.readlines()
        for eintrag in liste:
            l_eintrag = eintrag.split()
            d[l_eintrag[0]]=l_eintrag[1:]
        datei.close()
    except:
        pass
    return d

#def aufgabe(d):
    zufall = random.randint(0, len(d.keys())-1)
    vokabel = list(d.keys())[zufall]
    #print(vokabel +'?')
#Datei liegt auf dem Pfad
#c:\\Benutzer\\ramon\\Dokumente\\Python Scripts\\python-edx-07-07-17\\woerterbuch.txt'
#woerterbuch liste von einträgen mit leerzeichen getrennt

d={}
datei=open('woerterbuch.txt')
liste = datei.readlines()
print(liste)
for eintrag in liste:
            l_eintrag = eintrag.split()#trennung an leerzeichen
            #print(l_eintrag[0])
            #print(l_eintrag[1])
            d[l_eintrag[0]]=l_eintrag[1:]
datei.close()
print(d)
zufall = random.randint(0, len(d.keys())-1)
vokabel = list(d.keys())[zufall]
print(vokabel+' ?')
antwort=input()


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-dictionaries-10.ipynb".
The first task is:

Beispiel Studenten - mit dictionary

Can you write Python code for it?

st={}#Erzeugen des leeren dictionarys
st['100100'] = {'Mathe':1.0, 'Bwl':2.5}
st['100200'] = {'Mathe':2.3, 'Bwl':1.8}
print(st.items())
print(type(st))
print(st.values())
print(st.keys())


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-dictionaries-10.ipynb".
The first task is:
<h2>Datenzugriff in einem dictionary
Can you write Python code for it?

deutsch = {'key':['Schluessel','Taste'],'slice':['Scheibe','Schnitte','Stueck'],'value':['Wert']}
print(deutsch)
######Abfangen von Abfragefehlern
def uebersetze(wort,d):
    if wort in d:
        return d[wort]
    else:
        return 'unbekannt'

print(uebersetze('slice',deutsch))
uebersetze('search',deutsch)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-dictionaries-10.ipynb".
The first task is:

Beispiel: Alter

Can you write Python code for it?

alter = {'Peter':45,'Julia':23,'Mathias':36} #Erzeugen eines Dictionaries
print(alter)

alter['Julia']=27 #Ändern des Alters
alter['Monika']=33 #Hinzufügen von Monika - die Reihenfolge der Schlüssel spielt keine Rolle
print(alter)
if 'Monika' in alter:
    print (alter['Monika'])


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-dictionaries-10.ipynb".
The first task is:

Beispiel: Temperaturen in Staedten

Can you write Python code for it?

temperatur={'stuttgart':32.9,'muenchen':29.8,'hamburg':24.4}# Erzeugen eines dictionaries mit Temperaturen in verschiedenen Städten
temperatur['koeln']=29.7 #hinzufuegen der temperatur in koeln
print(temperatur) #ausgabe der temperaturen

for stadt in temperatur:
    print('Die Temperatur in %s ist %g °C' % (stadt,temperatur[stadt]))

if 'Berlin' in temperatur:
    print ('Berlin:', temperatur['Berlin'])
else:
    print ('Keine Daten für Berlin gefunden')

'stuttgart' in temperatur #überprüfen ob Schlüssel in temperatur enthalten ist

temperatur.keys() #Ausgabe der Schlüssel im Dictionary

temperatur.values()#ausgabe der Werte im Dictionary

for stadt in sorted(temperatur):
    print(stadt)

temperatur_kopie=temperatur.copy() #erstellt eine KOpie des dictonaries
print (temperatur_kopie)

temperatur2={'stuttgart':22.9,'muenchen':23.8,'hamburg':21.4} #ein 2-tes dictionary
temperatur.update(temperatur2)
for stadt in temperatur:
    print('Die Temperatur in %s ist %g °C' % (stadt,temperatur[stadt]))
print('Anzahl enthaltene Staedte: %g'% len(temperatur))

temperatur2={'stuttgart':22.9,'muenchen':23.8,'hamburg':21.4,'koeln':18.6,'frankfurt':20.6, 'weimar':18.8} #ein 2-tes dictionary
temperatur.update(temperatur2)
for stadt in temperatur:
    print('Die Temperatur in %s ist %g °C' % (stadt,temperatur[stadt]))
print('Anzahl enthaltene Staedte: %g'% len(temperatur))


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Dictionaries

d={}

d.values()

d.keys()

d.items()

d.clear()

d.copy()

d.get(k,x)

k in d

d.setdefault(k[ ,x])

d1.update(d2)
Can you write Python code for it?

mktcaps = {'AAPL':538.7,'GOOG':68.7,'IONS':4.6}# Dictionary wird initialisiert
print(type(mktcaps))
print(mktcaps)
print(mktcaps.values())
print(mktcaps.keys())
print(mktcaps.items())
c=mktcaps.items()
print c[0]

mktcaps['AAPL'] #Gibt den Wert zurück der mit dem Schlüssel "AAPL" verknüpft ist

mktcaps['GS'] #Error because GS is not in mktcaps

mktcaps.get('GS') #Returns None because GS is not in mktcaps

mktcaps['GS'] = 88.65 #Fügt GS to the dictionary
print(mktcaps)

del(mktcaps['GOOG']) #Removes GOOG from mktcaps
print(mktcaps)

mktcaps.keys() #Returns all the keys

mktcaps.values() #Returns all the values

import hashlib
l=('AAA','BBB','CCC','DDD','EEE')
print(l)
print(len(l))
hshdict={'AAA':hashlib.sha256('AAA)')}
hshdict.values()
v=hshdict['AAA']
m=v.hexdigest()
print(m)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:
<h1>Vokabeltrainer entwickeln
Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Beispiel Studenten - mit dictionary

Can you write Python code for it?

st={}#Erzeugen des leeren dictionarys
st['100100'] = {'Mathe':1.0, 'Bwl':2.5}
st['100200'] = {'Mathe':2.3, 'Bwl':1.8}
print(st.items())
print(type(st))
print(st.values())
print(st.keys())
for k in st.keys():
    print(st.['k'])


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:
<h2>Datenzugriff in einem dictionary
Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Ein Dictionary aus anderen zusammensetzen

d2.update(d1)
Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Listen sind veränderbar [mutable]

Can you write Python code for it?

y=['a','b']
x = [1,y,3]
print(x)
print(y)
y[1] = 4
print(y)

print(x)

x="Hello"
print(x,id(x))
x+=" You!"
print(x,id(x)) #x is not the same object it was
y=["Hello"]
print(y,id(y))
y+=["You!"]
print(y,id(y)) #y is still the same object. Lists are mutable. Strings are immutable

def eggs(item,total=0):
    total+=item
    return total

def spam(elem,some_list=[]):
    some_list.append(elem)
    return some_list

print(eggs(1))
print(eggs(2))

print(spam(1))
print(spam(2))


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Schrittweiser Aufbau eines Studentenverezichnisses

Can you write Python code for it?

def stud_verz():
    stud={}#erzeugen eines leeren dictionaries
    student=input('Matrikel-Nr als string eingeben:')
    while student:
        Mathe = input('Mathe Note eingeben:')
        Bwl = input('Bwl Note eingeben:')
        stud[student]={"Mathematik":Mathe,"BWL":Bwl}
        student=input('Matrikel-Nr als string eingeben:')
    return stud

print (stud_verz())


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Anything you want to remove must be in the list or the location must be inside the list

Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Beispiel: Alter

Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Beispiel: Temperaturen in Staedten

Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

Hinzufügen von Objekten in Listen

Can you write Python code for it?

x=list()
print(x)
x.append('One') #Adds 'One' to the back of the empty list
print(x)
x.append('Two') #Adds 'Two' to the back of the list ['One']
print(x)

x.insert(0,'Half') #Inserts 'Half' at location 0. Items will shift to make roomw
print(x)

x=list()
x.extend([1,2,3]) #Unpacks the list and adds each item to the back of the list
print(x)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-listen-10.ipynb".
The first task is:

List element iteration

Can you write Python code for it?

x=[1,7,2,5,3,5,67,32]
for element in x: #The for draws elements - sequentially - from the list x and uses the variable "element" to store values
    print(element)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:

Erzeugen eines numpy array aus einer Liste

2. Beispiel

 Bestimmen Sie die Länge des arrays

 [6,8,3,2,5,4,7,6]

 Berechnen Sie die Summe und den Mittelwert des arrays
Can you write Python code for it?

y = np.array([6,8,3,2,5,4,7,6])
print(y)
print(len(y))
print(y.sum())
print(y.mean())


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:

Festlegung des type

Nützlich, wenn man einen Text direkt in ein numerisches array streamed

Can you write Python code for it?

import numpy as np
x=['1','2','3','7','8','9']
xi = np.array(x,'int')
xf = np.array(x,'float')
xs = np.array(x,'str')
print(xi)
print(xf)
print(xs)
#print(xi,xf,xs,sep='\n')#python 3.0


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:

Teilbereiche wählen (Slicing)

Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:

Unterstützung von Zufallszahlen in numpy

Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:

Vergleich von numpy arrays mit listen

Can you write Python code for it?

n=10
ax = np.array([np.arange(n)**2,np.arange(n)**3])
ay = ax.transpose()
#print(ax)
#print(ay)
np.dot(ax,ay)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:

numpy - mehrdimensionale arrays

 Welches Ergebnis liefert a[2,3,4] in folgendem array a

 ! Beachten Sie dass Indizes mit 0 beginnen !
Can you write Python code for it?

import numpy as np
a=np.arange(120).reshape(6,4,5)
print(a)

a[2,3,4]

a[0,0,0]

a[5,3,4]


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:

Ändern der Form (Reshaping)

Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:

Über einen Index Werte auslesen

Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:
Füllen Sie eine vergleichbares array mit normalverteilten Zufallszahlen (mu = 0, sigma =1) (ohne seed)
Was liefert s[3,:2,:3] zurück ?
Can you write Python code for it?

mu = 0
sigma = 1
s = np.random.normal(mu, sigma, 120)
s = s.reshape(6,4,5)
print(s)

print(s[3,:2,:3])


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:
Füllen Sie eine vergleichbares array mit normalverteilten Zufallszahlen (mu = 0, sigma =1) (seed = 1000)
Was liefert s[3,:2,:3] zurück ?
Can you write Python code for it?

np.random.seed(10000)
mu = 0
sigma = 1
s = np.random.normal(mu, sigma, 120)
s = s.reshape(6,4,5)
print(s)

s[3,:2,:3]

np.random.seed(10000)
mu = 0
sigma = 1
s = np.random.normal(mu, sigma, 120)
s = s.reshape(6,4,5)
print(s[3,:2,:3])


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:
Welches Ergebnis liefert a[:, 3, 4] in diesem array ?
Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-arbeiten-mit-numpy-10.ipynb".
The first task is:
Welches Ergebnis liefert a[:, :, 4] in diesem array ?
Can you write Python code for it?


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-hauspreise-40.ipynb".
The first task is:
Erweiterung: Separate Regression der Geokoordinaten
Can you write Python code for it?

from utilities import RegressionOnSubset
from sklearn.neighbors import KNeighborsRegressor

Xtrain, ytrain, Xtest, ytest = load_data()

columns = ["Longitude", "Latitude"]

pipe = Pipeline([
         ('geo_regressor', RegressionOnSubset(
            KNeighborsRegressor(), columns)),
        ('scaler', StandardScaler()),
        ('regressor', SVR(max_iter=5000)),
    ])

pipe.fit(Xtrain, ytrain)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-hauspreise-40.ipynb".
The first task is:
Laden der Daten (Trainings- und Validierungsdatensatz)
Can you write Python code for it?

#from utilities import load_data
#Xtrain, ytrain, Xtest, ytest = load_data()
Xtrain.head()

Xtrain.tail()

Xtrain.describe()

ytrain

Xtest.head()

Xtest.describe()

ytest


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-hauspreise-40.ipynb".
The first task is:
Modell B durch Erweiterung mit Skalierungsverfahren
Can you write Python code for it?

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipe = Pipeline([
        ('scaler', StandardScaler()),
        ('regressor', SVR(max_iter=5000))
    ])
pipe.fit(Xtrain, ytrain)

prediction = pipe.predict(Xtest)

#Plot der Daten
truth = ytest
if isinstance(prediction, np.ndarray):
    p = plt.hist(prediction, bins=50, color="g", label='Vorhersage')
else:
    p = plt.bar(prediction, 250, width=0.125, color="g", label='Vorhersage')
t = plt.hist(truth, bins=50, color="b", label='Wahrheit')
plt.ylabel("Anzahl")
plt.xlabel("logarithmierter Hauspreis.")
plt.legend()
print("Mittlere absolute Abweichung: {}".format(np.mean(np.abs(prediction - truth))))
print("Mittlere quadratische Abweichung: {}".format(np.mean(np.square(prediction - truth))))

#evaluate(prediction, ytest)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-hauspreise-40.ipynb".
The first task is:
Modell B: Support Vektor Regression
Can you write Python code for it?

from sklearn.svm import SVR

est = SVR(max_iter=5000)
est.fit(Xtrain, ytrain)

prediction = est.predict(Xtest)
#Plot der Daten
truth = ytest
if isinstance(prediction, np.ndarray):
    p = plt.hist(prediction, bins=50, color="g", label='Vorhersage')
else:
    p = plt.bar(prediction, 250, width=0.125, color="g", label='Vorhersage')
t = plt.hist(truth, bins=50, color="b", label='Wahrheit')
plt.ylabel("Anzahl")
plt.xlabel("logarithmierter Hauspreis.")
plt.legend()
print("Mittlere absolute Abweichung: {}".format(np.mean(np.abs(prediction - truth))))
print("Mittlere quadratische Abweichung: {}".format(np.mean(np.square(prediction - truth))))
#evaluate(prediction, ytest)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-hauspreise-40.ipynb".
The first task is:
Weitere Betrachtung:  Zweidimensionale Korrelationen
Can you write Python code for it?

df_file = pd.read_csv("california_housing.csv")
Xtrain = df_file[df_file["sample_id"] == 0].copy()
ytrain = np.asarray(Xtrain["y"]).copy()
del(Xtrain["sample_id"], Xtrain["y"])
Xtest = df_file[df_file["sample_id"] == 1].copy()
ytest = np.asarray(Xtest["y"]).copy()
del(Xtest["sample_id"], Xtest["y"])
#Xtrain, ytrain, Xtest, ytest = load_data()

#from utilities import visualize

#def visualize(geo_prediction):
#Xtrain, ytrain, Xtest, ytest = load_data()
geo_prediction = ytest
x1 = np.asarray(Xtest['Longitude'])
x2 = np.asarray(Xtest['Latitude'])

from matplotlib import colors
cm = plt.cm.get_cmap('RdYlBu')

sc = plt.scatter(x1, x2, c=geo_prediction, s=20, vmin=0, vmax=5)
plt.colorbar(sc)
plt.show()

#visualize(ytest)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:

 Warum setzen wir die Bibliothek Scikit-learn ein ?

 Installation Scikit-learn

 Wir bevorzugen eine Anaconda Installation !

 Warum wollen Sie scikit-learn einsetzen ?

 Arbeiten mit NumPy kennenlernen

Can you write Python code for it?

import numpy as np

x = np.array([[1, 2, 3], [4, 5, 6]])
print("x:\n{}".format(x))


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
<h3> #matplotlib : Graphische Darstellungen einfach gemacht</h3>
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

# Generierung einer Zahlenreihe von -10 bis 10 in 100 Schritten
x = np.linspace(-10, 10, 100)
# Erzeugen eines Zweiten numpy arrays mit der Funktion sin()
y = np.sin(x)
# The plot function makes a line chart of one array against another
plt.plot(x, y, marker="o", color='brown')
plt.title('Sinus Kurve')
plt.xlabel('x')
plt.ylabel('y')
#plt.legend('sin x','upper left')

%matplotlib inline
import matplotlib.pyplot as plt

# Generierung einer Zahlenreihe von -10 bis 10 in 100 Schritten
x = np.linspace(-20, 20, 100)
# Erzeugen eines Zweiten numpy arrays mit der Funktion sin()
y = np.exp(x)
# The plot function makes a line chart of one array against another
plt.plot(x, y, marker="o", color='green')
plt.title('Exponential Kurve')
plt.xlabel('x')
plt.ylabel('y')

%matplotlib inline
import matplotlib.pyplot as plt
from numpy.random import randn

z = randn(100)

red_dot, = plt.plot(z, "ro", markersize=15)
# Schreibe ein weisses Kruez über einige der Daten.
white_cross, = plt.plot(z[:50], "w+", markeredgewidth=3, markersize=15)

plt.legend([red_dot, (red_dot, white_cross)], ["Attr A", "Attr A+B"])


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
<h3> Arbeiten mit SciPy </h3>
Can you write Python code for it?

from scipy import sparse

# create a 2d NumPy array with a diagonal of ones, and zeros everywhere else
eye = np.eye(4)
print("NumPy array:\n{}".format(eye))

# convert the NumPy array to a SciPy sparse matrix in CSR format
# only the non-zero entries are stored
sparse_matrix = sparse.csr_matrix(eye)
print("\nSciPy sparse CSR matrix:\n{}".format(sparse_matrix))

data = np.ones(4)
row_indices = np.arange(4)
col_indices = np.arange(4)
eye_coo = sparse.coo_matrix((data, (row_indices, col_indices)))
print("COO representation:\n{}".format(eye_coo))


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
<h3> Arbeiten mit pandas</h3>
Can you write Python code for it?

import pandas as pd
from IPython.display import display

# create a simple dataset of people
data = {'Name': ["John", "Anna", "Peter", "Linda"],
        'Location' : ["New York", "Paris", "Berlin", "London"],
        'Age' : [24, 13, 53, 33]
       }

data_pandas = pd.DataFrame(data)
# IPython.display allows "pretty printing" of dataframes
# in the Jupyter notebook
display(data_pandas)

# One of many possible ways to query the table:
# selecting all rows that have an age column greate than 30
display(data_pandas[data_pandas.Age > 30])


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
Evaluation des Modells
Can you write Python code for it?

y_pred = knn.predict(X_test)
print("Test set predictions:\n {}".format(y_pred))

print("Test set score: {:.2f}".format(np.mean(y_pred == y_test)))

print("Test set score: {:.2f}".format(knn.score(X_test, y_test)))


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
Ihr erstes Machine Learning Modell: k nearest neighbors
Es wurde von n=1 auf n=2 abgeändert
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=3)

knn.fit(X_train, y_train)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
Immer zuerst: Schauen Sie sich die Daten an
Can you write Python code for it?

import mglearn
# create dataframe from data in X_train
# label the columns using the strings in iris_dataset.feature_names
iris_dataframe = pd.DataFrame(X_train, columns=iris_dataset.feature_names)
# create a scatter matrix from the dataframe, color by y_train
pd.scatter_matrix(iris_dataframe, c=y_train, figsize=(15, 15), marker='o',hist_kwds={'bins': 20}, s=60, alpha=.8, cmap=mglearn.cm3)


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
Python2 versus Python3
Versions Used in this Notebook
Can you write Python code for it?

import sys
print("Python version: {}".format(sys.version))

import pandas as pd
print("pandas version: {}".format(pd.__version__))

import matplotlib
print("matplotlib version: {}".format(matplotlib.__version__))

import numpy as np
print("NumPy version: {}".format(np.__version__))

import scipy as sp
print("SciPy version: {}".format(sp.__version__))

import IPython
print("IPython version: {}".format(IPython.__version__))

import sklearn
print("scikit-learn version: {}".format(sklearn.__version__))


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
Vorhersagen machen
Can you write Python code for it?

X_new = np.array([[5, 2.9, 1, 0.2]])
print("X_new.shape: {}".format(X_new.shape))

prediction = knn.predict(X_new)
print("Prediction: {}".format(prediction))
print("Predicted target name: {}".format(
       iris_dataset['target_names'][prediction]))


I am working on the file "17-12-11-workcamp-ml/2017-12-11-workcamp-ml-beispiel-sepal-20.ipynb".
The first task is:
Wie wird der Erfolg gemessen: Training und Test der Daten
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
    iris_dataset['data'], iris_dataset['target'], random_state=1)

print("X_train shape: {}".format(X_train.shape))
print("y_train shape: {}".format(y_train.shape))

print("X_test shape: {}".format(X_test.shape))
print("y_test shape: {}".format(y_test.shape))


I am working on the file "17-assignment2/BatchNormalization.ipynb".
The first task is:
Batch Normalization: alternative backward (OPTIONAL, +3 points extra credit)
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For the sigmoid function, it turns out that you can derive a very simple formula for the backward pass by simplifying gradients on paper.
Surprisingly, it turns out that you can also derive a simple expression for the batch normalization backward pass if you work out derivatives on paper and simplify. After doing so, implement the simplified batch normalization backward pass in the function batchnorm_backward_alt and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.
NOTE: This part of the assignment is entirely optional, but we will reward 3 points of extra credit if you can complete it.
Can you write Python code for it?

np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))


I am working on the file "17-assignment2/BatchNormalization.ipynb".
The first task is:
Batch Normalization: backward
Now implement the backward pass for batch normalization in the function batchnorm_backward.
To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.
Once you have finished, run the following to numerically check your backward pass.
Can you write Python code for it?

# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, gamma, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, beta, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma, dout)
db_num = eval_numerical_gradient_array(fb, beta, dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))


I am working on the file "17-assignment2/BatchNormalization.ipynb".
The first task is:
Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.
The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.
Can you write Python code for it?

np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers = {}
solvers = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, use_batchnorm=True)
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, use_batchnorm=False)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers[weight_scale] = solver

# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers[ws].train_acc_history))

  best_val_accs.append(max(solvers[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization...


I am working on the file "17-assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Convolution: Naive backward pass
Implement the backward pass for the convolution operation in the function conv_backward_naive in the file cs231n/layers.py. Again, you don't need to worry too much about computational efficiency.
When you are done, run the following to check your backward pass with a numeric gradient check.
Can you write Python code for it?

np.random.seed(231)
x = np.random.randn(4, 3, 5, 5)
w = np.random.randn(2, 3, 3, 3)
b = np.random.randn(2,)
dout = np.random.randn(4, 2, 5, 5)
conv_param = {'stride': 1, 'pad': 1}

dx_num = eval_numerical_gradient_array(lambda x: conv_forward_naive(x, w, b, conv_param)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: conv_forward_naive(x, w, b, conv_param)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: conv_forward_naive(x, w, b, conv_param)[0], b, dout)

out, cache = conv_forward_naive(x, w, b, conv_param)
dx, dw, db = conv_backward_naive(dout, cache)

# Your errors should be around 1e-8'
print('Testing conv_backward_naive function')
print('dx error: ', rel_error(dx, dx_num))
print('dw error: ', rel_error(dw, dw_num))
print('db error: ', rel_error(db, db_num))


I am working on the file "17-assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Convolutional "sandwich" layers
Previously we introduced the concept of "sandwich" layers that combine multiple operations into commonly used patterns. In the file cs231n/layer_utils.py you will find sandwich layers that implement a few commonly used patterns for convolutional networks.
Can you write Python code for it?

from cs231n.layer_utils import conv_relu_pool_forward, conv_relu_pool_backward
np.random.seed(231)
x = np.random.randn(2, 3, 16, 16)
w = np.random.randn(3, 3, 3, 3)
b = np.random.randn(3,)
dout = np.random.randn(2, 3, 8, 8)
conv_param = {'stride': 1, 'pad': 1}
pool_param = {'pool_height': 2, 'pool_width': 2, 'stride': 2}

out, cache = conv_relu_pool_forward(x, w, b, conv_param, pool_param)
dx, dw, db = conv_relu_pool_backward(dout, cache)

dx_num = eval_numerical_gradient_array(lambda x: conv_relu_pool_forward(x, w, b, conv_param, pool_param)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: conv_relu_pool_forward(x, w, b, conv_param, pool_param)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: conv_relu_pool_forward(x, w, b, conv_param, pool_param)[0], b, dout)

print('Testing conv_relu_pool')
print('dx error: ', rel_error(dx_num, dx))
print('dw error: ', rel_error(dw_num, dw))
print('db error: ', rel_error(db_num, db))

from cs231n.layer_utils import conv_relu_forward, conv_relu_backward
np.random.seed(231)
x = np.random.randn(2, 3, 8, 8)
w = np.random.randn(3, 3, 3, 3)
b = np.random.randn(3,)
dout = np.random.randn(2, 3, 8, 8)
conv_param = {'stride': 1, 'pad': 1}

out, cache = conv_relu_forward(x, w, b, conv_param)
dx, dw, db = conv_relu_backward(dout, cache)

dx_num = eval_numerical_gradient_array(lambda x: conv_relu_forward(x, w, b, conv_param)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: conv_relu_forward(x, w, b, conv_param)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: conv_relu_forward(x, w, b, conv_param)[0], b, dout)

print('Testing conv_relu:')
print('dx error: ', rel_error(dx_num, dx))
print('dw error: ', rel_error(dw_num, dw))
print('db error: ', rel_error(db_num, db))


I am working on the file "17-assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Gradient check
After the loss looks reasonable, use numeric gradient checking to make sure that your backward pass is correct. When you use numeric gradient checking you should use a small amount of artifical data and a small number of neurons at each layer. Note: correct implementations may still have relative errors up to 1e-2.
Can you write Python code for it?

num_inputs = 2
input_dim = (3, 16, 16)
reg = 0.0
num_classes = 10
np.random.seed(231)
X = np.random.randn(num_inputs, *input_dim)
y = np.random.randint(num_classes, size=num_inputs)

model = ThreeLayerConvNet(num_filters=3, filter_size=3,
                          input_dim=input_dim, hidden_dim=7,
                          dtype=np.float64)
loss, grads = model.loss(X, y)
for param_name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    param_grad_num = eval_numerical_gradient(f, model.params[param_name], verbose=False, h=1e-6)
    e = rel_error(param_grad_num, grads[param_name])
    print('%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name])))


I am working on the file "17-assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Max pooling: Naive backward
Implement the backward pass for the max-pooling operation in the function max_pool_backward_naive in the file cs231n/layers.py. You don't need to worry about computational efficiency.
Check your implementation with numeric gradient checking by running the following:
Can you write Python code for it?

np.random.seed(231)
x = np.random.randn(3, 2, 8, 8)
dout = np.random.randn(3, 2, 4, 4)
pool_param = {'pool_height': 2, 'pool_width': 2, 'stride': 2}

dx_num = eval_numerical_gradient_array(lambda x: max_pool_forward_naive(x, pool_param)[0], x, dout)

out, cache = max_pool_forward_naive(x, pool_param)
dx = max_pool_backward_naive(dout, cache)

# Your error should be around 1e-12
print('Testing max_pool_backward_naive function:')
print('dx error: ', rel_error(dx, dx_num))


I am working on the file "17-assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Spatial Batch Normalization
We already saw that batch normalization is a very useful technique for training deep fully-connected networks. Batch normalization can also be used for convolutional networks, but we need to tweak it a bit; the modification will be called "spatial batch normalization."
Normally batch-normalization accepts inputs of shape (N, D) and produces outputs of shape (N, D), where we normalize across the minibatch dimension N. For data coming from convolutional layers, batch normalization needs to accept inputs of shape (N, C, H, W) and produce outputs of shape (N, C, H, W) where the N dimension gives the minibatch size and the (H, W) dimensions give the spatial size of the feature map.
If the feature map was produced using convolutions, then we expect the statistics of each feature channel to be relatively consistent both between different imagesand different locations within the same image. Therefore spatial batch normalization computes a mean and variance for each of the C feature channels by computing statistics over both the minibatch dimension N and the spatial dimensions H and W.
Spatial batch normalization: forward
In the file cs231n/layers.py, implement the forward pass for spatial batch normalization in the function spatial_batchnorm_forward. Check your implementation by running the following:
Can you write Python code for it?

np.random.seed(231)
# Check the training-time forward pass by checking means and variances
# of features both before and after spatial batch normalization

N, C, H, W = 2, 3, 4, 5
x = 4 * np.random.randn(N, C, H, W) + 10

print('Before spatial batch normalization:')
print('  Shape: ', x.shape)
print('  Means: ', x.mean(axis=(0, 2, 3)))
print('  Stds: ', x.std(axis=(0, 2, 3)))

# Means should be close to zero and stds close to one
gamma, beta = np.ones(C), np.zeros(C)
bn_param = {'mode': 'train'}
out, _ = spatial_batchnorm_forward(x, gamma, beta, bn_param)
print('After spatial batch normalization:')
print('  Shape: ', out.shape)
print('  Means: ', out.mean(axis=(0, 2, 3)))
print('  Stds: ', out.std(axis=(0, 2, 3)))

# Means should be close to beta and stds close to gamma
gamma, beta = np.asarray([3, 4, 5]), np.asarray([6, 7, 8])
out, _ = spatial_batchnorm_forward(x, gamma, beta, bn_param)
print('After spatial batch normalization (nontrivial gamma, beta):')
print('  Shape: ', out.shape)
print('  Means: ', out.mean(axis=(0, 2, 3)))
print('  Stds: ', out.std(axis=(0, 2, 3)))

np.random.seed(231)
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.
N, C, H, W = 10, 4, 11, 12

bn_param = {'mode': 'train'}
gamma = np.ones(C)
beta = np.zeros(C)
for t in range(50):
  x = 2.3 * np.random.randn(N, C, H, W) + 13
  spatial_batchnorm_forward(x, gamma, beta, bn_param)
bn_param['mode'] = 'test'
x = 2.3 * np.random.randn(N, C, H, W) + 13
a_norm, _ = spatial_batchnorm_forward(x, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After spatial batch normalization (test-time):')
print('  means: ', a_norm.mean(axis=(0, 2, 3)))
print('  stds: ', a_norm.std(axis=(0, 2, 3)))


I am working on the file "17-assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Spatial batch normalization: backward
In the file cs231n/layers.py, implement the backward pass for spatial batch normalization in the function spatial_batchnorm_backward. Run the following to check your implementation using a numeric gradient check:
Can you write Python code for it?

np.random.seed(231)
N, C, H, W = 2, 3, 4, 5
x = 5 * np.random.randn(N, C, H, W) + 12
gamma = np.random.randn(C)
beta = np.random.randn(C)
dout = np.random.randn(N, C, H, W)

bn_param = {'mode': 'train'}
fx = lambda x: spatial_batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: spatial_batchnorm_forward(x, gamma, beta, bn_param)[0]
fb = lambda b: spatial_batchnorm_forward(x, gamma, beta, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma, dout)
db_num = eval_numerical_gradient_array(fb, beta, dout)

_, cache = spatial_batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = spatial_batchnorm_backward(dout, cache)
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))


I am working on the file "17-assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Visualize Filters
You can visualize the first-layer convolutional filters from the trained network by running the following:
Can you write Python code for it?

from cs231n.vis_utils import visualize_grid

grid = visualize_grid(model.params['W1'].transpose(0, 2, 3, 1))
plt.imshow(grid.astype('uint8'))
plt.axis('off')
plt.gcf().set_size_inches(5, 5)
plt.show()


I am working on the file "17-assignment2/TensorFlow.ipynb".
The first task is:
Check the accuracy of the model.
Let's see the train and test code in action -- feel free to use these methods when evaluating the models you develop below. You should see a loss of 1.5 to 2.0 with an accuracy of 0.3 to 0.4.
Can you write Python code for it?


I am working on the file "17-assignment2/TensorFlow.ipynb".
The first task is:
TensorFlow supports many other layer types, loss functions, and optimizers - you will experiment with these next. Here's the official API documentation for these (if any of the parameters used above were unclear, this resource will also be helpful).

Layers, Activations, Loss functions : https://www.tensorflow.org/api_guides/python/nn
Optimizers: https://www.tensorflow.org/api_guides/python/train#Optimizers
BatchNorm: https://www.tensorflow.org/api_docs/python/tf/contrib/layers/batch_norm

Training the model on one epoch
While we have defined a graph of operations above, in order to execute TensorFlow Graphs, by feeding them input data and computing the results, we first need to create a tf.Session object. A session encapsulates the control and state of the TensorFlow runtime. For more information, see the TensorFlow Getting started guide.
Optionally we can also specify a device context such as /cpu:0 or /gpu:0. For documentation on this behavior see this TensorFlow guide
You should see a validation loss of around 0.4 to 0.6 and an accuracy of 0.30 to 0.35 below
Can you write Python code for it?

def run_model(session, predict, loss_val, Xd, yd,
              epochs=1, batch_size=64, print_every=100,
              training=None, plot_losses=False):
    # have tensorflow compute accuracy
    correct_prediction = tf.equal(tf.argmax(predict,1), y)
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

    # shuffle indicies
    train_indicies = np.arange(Xd.shape[0])
    np.random.shuffle(train_indicies)

    training_now = training is not None

    # setting up variables we want to compute (and optimizing)
    # if we have a training function, add that to things we compute
    variables = [mean_loss,correct_prediction,accuracy]
    if training_now:
        variables[-1] = training

    # counter
    iter_cnt = 0
    for e in range(epochs):
        # keep track of losses and accuracy
        correct = 0
        losses = []
        # make sure we iterate over the dataset once
        for i in range(int(math.ceil(Xd.shape[0]/batch_size))):
            # generate indicies for the batch
            start_idx = (i*batch_size)%X_train.shape[0]
            idx = train_indicies[start_idx:start_idx+batch_size]

            # create a feed dictionary for this batch
            feed_dict = {X: Xd[idx,:],
                         y: yd[idx],
                         is_training: training_now }
            # get batch size
            actual_batch_size = yd[i:i+batch_size].shape[0]

            # have tensorflow compute loss and correct predictions
            # and (if given) perform a training step
            loss, corr, _ = session.run(variables,feed_dict=feed_dict)

            # aggregate performance stats
            losses.append(loss*actual_batch_size)
            correct += np.sum(corr)

            # print every now and then
            if training_now and (iter_cnt % print_every) == 0:
                print("Iteration {0}: with minibatch training loss = {1:.3g} and accuracy of {2:.2g}"\
                     ...


I am working on the file "17-assignment2/TensorFlow.ipynb".
The first task is:
To make sure you're doing the right thing, use the following tool to check the dimensionality of your output (it should be 64 x 10, since our batches have size 64 and the output of the final affine layer should be 10, corresponding to our 10 classes):
Can you write Python code for it?

# Now we're going to feed a random batch into the model
# and make sure the output is the right size
x = np.random.randn(64, 32, 32,3)
with tf.Session() as sess:
    with tf.device("/cpu:0"): #"/cpu:0" or "/gpu:0"
        tf.global_variables_initializer().run()

        ans = sess.run(y_out,feed_dict={X:x,is_training:True})
        %timeit sess.run(y_out,feed_dict={X:x,is_training:True})
        print(ans.shape)
        print(np.array_equal(ans.shape, np.array([64, 10])))


I am working on the file "17-assignment2/TensorFlow.ipynb".
The first task is:
Train the model
Below we'll create a session and train the model over one epoch. You should see a loss of 3.0 - 5.0 and an accuracy of 0.2 to 0.3. There will be some variation due to random seeds and differences in initialization
Can you write Python code for it?

sess = tf.Session()

sess.run(tf.global_variables_initializer())
print('Training')
run_model(sess,y_out,mean_loss,X_train,y_train,1,64,100,train_step)


I am working on the file "17-assignment2/TensorFlow.ipynb".
The first task is:
Training a specific model
In this section, we're going to specify a model for you to construct. The goal here isn't to get good performance (that'll be next), but instead to get comfortable with understanding the TensorFlow documentation and configuring your own model.
Using the code provided above as guidance, and using the following TensorFlow documentation, specify a model with the following architecture:

7x7 Convolutional Layer with 32 filters and stride of 1
ReLU Activation Layer
Spatial Batch Normalization Layer (trainable parameters, with scale and centering)
2x2 Max Pooling layer with a stride of 2
Affine layer with 1024 output units
ReLU Activation Layer
Affine layer from 1024 input units to 10 outputs
Can you write Python code for it?

# clear old variables
tf.reset_default_graph()

# define our input (e.g. the data that changes every batch)
# The first dim is None, and gets sets automatically based on batch size fed in
X = tf.placeholder(tf.float32, [None, 32, 32, 3])
y = tf.placeholder(tf.int64, [None])
is_training = tf.placeholder(tf.bool)

# define model
def complex_model(X,y,is_training):
    pass
pass

y_out = complex_model(X,y,is_training)


I am working on the file "17-assignment2/TensorFlow.ipynb".
The first task is:
You should observe that even a simple forward pass like this is significantly faster on the GPU. So for the rest of the assignment (and when you go train your models in assignment 3 and your project!), you should use GPU devices. However, with TensorFlow, the default device is a GPU if one is available, and a CPU otherwise, so we can skip the device specification from now on.
Train the model.
Now that you've seen how to define a model and do a single forward pass of some data through it, let's  walk through how you'd actually train one whole epoch over your training data (using the complex_model you created provided above).
Make sure you understand how each TensorFlow function used below corresponds to what you implemented in your custom neural network implementation.
First, set up an RMSprop optimizer (using a 1e-3 learning rate) and a cross-entropy loss function. See the TensorFlow documentation for more information
* Layers, Activations, Loss functions : https://www.tensorflow.org/api_guides/python/nn
* Optimizers: https://www.tensorflow.org/api_guides/python/train#Optimizers
Can you write Python code for it?

# Inputs
#     y_out: is what your model computes
#     y: is your TensorFlow variable with label information
# Outputs
#    mean_loss: a TensorFlow variable (scalar) with numerical loss
#    optimizer: a TensorFlow optimizer
# This should be ~3 lines of code!
mean_loss = None
optimizer = None
pass

train_step = optimizer.minimize(mean_loss)


I am working on the file "17. 로지스틱 회귀 분석/01. 로지스틱 회귀 분석.ipynb".
The first task is:
statsmodels 패키지의 로지스틱 회귀
statsmodels 패키지는 로지스틱 회귀 모형 Logit 를 제공한다. 사용방법은 OLS 와 동일하다. Scikit-Learn 패키지와 달리 Logit 클래스는 classification 되기 전의 값을 출력한다
Can you write Python code for it?

logit_mod = sm.Logit(y, X)
logit_res = logit_mod.fit(disp=0)
print(logit_res.summary())

xx = np.linspace(-3, 3, 100)
sigmoid = logit_res.predict(sm.add_constant(xx))
plt.plot(xx, sigmoid, lw=5, alpha=0.5)
plt.scatter(X0, y, marker='o', c=y, s=100)
plt.scatter(X0, logit_res.predict(X), marker='x', c=y, s=200, lw=2, alpha=0.5, cmap=mpl.cm.jet)
plt.xlim(-3, 3)
plt.show()


I am working on the file "17. 로지스틱 회귀 분석/01. 로지스틱 회귀 분석.ipynb".
The first task is:
로지스틱 함수
여러가지 시그모이드 중 로지스틱 함수는 다음과 같은 물리적인 의미를 부여할 수 있기 때문에 많이 사용된다.
우선 Bernoulli 시도에서 1이 나올 확률 $\theta$ 과 0이 나올 확률 $1-\theta$ 의 비(ratio)는 다음과 같은 수식이 되며 odds ratio 라고 한다.
$$ \text{odds ratio} = \dfrac{\theta}{1-\theta} $$
이 odds ratio 를 로그 변환한 것이 로지트 함수(Logit function)이다.
$$ z = \text{logit}(\text{odds ratio}) = \log \left(\dfrac{\theta}{1-\theta}\right) $$
로지스틱 함수(Logistic function) 는 이 로지트 함수의 역함수이다.
$$ \text{logitstic}(z) = \theta(z) = \dfrac{1}{1+\exp{(-z)}} $$
로지스틱 모형의 모수 추정
로지스틱 모형은 일종의 비선형 회귀 모형이지만 다음과 같이 MLE(Maximum Likelihood Estimation) 방법으로 모수 $w$를 추정할 수 있다.
여기에서는 종속 변수 $y$가 베르누이 확률 변수라고 가정한다.
$$ p(y \mid x, \theta) = \text{Ber} (y \mid  \theta(x) )$$
데이터 표본이 ${ x_i, y_i }$일 경우 Log Likelihood $\text{LL}$ 를 구하면 다음과 같다.
$$
\begin{eqnarray}
\text{LL}
&=& \log \prod_{i=1}^N \theta_i(x_i)^{y_i} (1-\theta_i(x_i))^{1-y_i} \
&=& \sum_{i=1}^N \left( y_i \log\theta_i(x_i) +  (1-y_i)\log(1-\theta_i(x_i)) \right) \
\end{eqnarray}
$$
$\theta$가 로지스틱 함수 형태로 표현된다면
$$
\log \left(\dfrac{\theta(x)}{1-\theta(x)}\right) = w^T x
$$
$$
\theta(x) = \dfrac{1}{1 + \exp{(-w^Tx)}}
$$
가 되고 이를  Log Likelihood 에 적용하면 다음과 같다.
$$
\begin{eqnarray}
\text{LL}
&=& \sum_{i=1}^N \left( y_i \log\theta_i(x_i) +  (1-y_i)\log(1-\theta_i(x_i)) \right) \
&=& \sum_{i=1}^N \left( y_i \log\left(\dfrac{1}{1 + \exp{(-w^Tx_i)}}\right) - (1-y_i)\log\left(\dfrac{\exp{(-w^Tx_i)}}{1 + \exp{(-w^Tx_i)}}\right) \right) \
\end{eqnarray}
$$
이 값의 최대화하는 값을 구하기 위해 chain rule를 사용하여 $w$로 미분해야 한다.
우선 $\theta$를 $w$로 미분하면
$$ \dfrac{\partial \theta}{\partial w}
= \dfrac{\partial}{\partial w} \dfrac{1}{1 + \exp{(-w^Tx)}} \
= \dfrac{\exp{(-w^Tx)}}{(1 + \exp{(-w^Tx)})^2} x \
= \theta(1-\theta) x $$
chain rule를 적용하면
$$
\begin{eqnarray}
\dfrac{\partial \text{LL}}{\partial w}
&=& \sum_{i=1}^N \left( y_i \dfrac{1}{\theta_i(x_i;w)} - (1-y_i)\dfrac{1}{1-\theta_i(x_i;w)} \right) \dfrac{\partial \theta}{\partial w}  \
&=& \sum_{i=1}^N \big( y_i (1-\theta_i(x_i;w)) - (1-y_i)\theta_i(x_i;w)  \big)  x_i \
&=& \sum_{i=1}^N \big( y_i  - \theta_i(x_i;w) \big) x_i...
Can you write Python code for it?

from sklearn.datasets import make_classification
X0, y = make_classification(n_features=1, n_redundant=0, n_informative=1, n_clusters_per_class=1, random_state=4)
X = sm.add_constant(X0)

from sklearn.linear_model import LogisticRegression
model = LogisticRegression().fit(X0, y)

xx = np.linspace(-3, 3, 100)
sigm = 1.0/(1 + np.exp(-model.coef_[0][0]*xx - model.intercept_[0]))
plt.plot(xx, sigm)
plt.scatter(X0, y, marker='o', c=y, s=100)
plt.scatter(X0, model.predict(X0), marker='x', c=y, s=200, lw=2, alpha=0.5, cmap=mpl.cm.jet)
plt.xlim(-3, 3)
plt.show()


I am working on the file "17. 로지스틱 회귀 분석/01. 로지스틱 회귀 분석.ipynb".
The first task is:
예제 1: Michelin and Zagat 가이드 비교
다음 데이터는 뉴욕시의 레스토랑에 대한 두 개의 가이드북에서 발취한 것이다.

Food: Zagat Survey 2006 의 고객 평가 점수
InMichelin: 해당 고객 평가 점수를 받은 레스토랑 중 2006 Michelin Guide New York City 에 실린 레스토랑의 수
NotInMichelin: 해당 고객 평가 점수를 받은 레스토랑 중 2006 Michelin Guide New York City 에 실리지 않은 레스토랑의 수
mi: 해당 고객 평가 점수를 받은 레스토랑의 수
proportion: 해당 고객 평가 점수를 받은 레스토랑 중 2006 Michelin Guide New York City 에 실린 레스토랑의 비율
Can you write Python code for it?

df = pd.read_table("~/data/sheather/MichelinFood.txt")
df

df.plot(kind="scatter", x="Food", y="proportion", s=100)
plt.show()

X = sm.add_constant(df.Food)
y = df.proportion
model = sm.Logit(y, X)
result = model.fit()
print(result.summary())

df.plot(kind="scatter", x="Food", y="proportion", s=50, alpha=0.5)
xx = np.linspace(10, 35, 100)
plt.plot(xx, result.predict(sm.add_constant(xx)), "r", lw=4)
plt.xlim(10, 35)
plt.show()


I am working on the file "17. 로지스틱 회귀 분석/01. 로지스틱 회귀 분석.ipynb".
The first task is:
예제 3: Fair's Affair Dataset
Can you write Python code for it?

print(sm.datasets.fair.SOURCE)
print(sm.datasets.fair.NOTE)

df = sm.datasets.fair.load_pandas().data
df.head()

sns.factorplot(x="affairs", y="children", row="yrs_married", data=df,
               orient="h", size=2, aspect=5, kind="box")
plt.show()

df['affair'] = (df['affairs'] > 0).astype(float)
modoel = smf.logit("affair ~ rate_marriage + religious + yrs_married + age + educ + children", df).fit()
print(modoel.summary())


I am working on the file "170430_krumsiek11/krumsiek11.ipynb".
The first task is:
Here, we simulate data using a literature-curated boolean gene
regulatory network, which is believed to describe myeloid differentiation
(Krumsiek et al., 2011). Using sim, the
boolean model
Gata2 = Gata2 and not (Gata1 and Fog1) and not Pu.1
Gata1 = (Gata1 or Gata2 or Fli1) and not Pu.1
Fog1 = Gata1
EKLF = Gata1 and not Fli1
Fli1 = Gata1 and not EKLF
SCL = Gata1 and not Pu.1
Cebpa = Cebpa and not (Gata1 and Fog1 and SCL)
Pu.1 = (Cebpa or Pu.1) and not (Gata1 or Gata2)
cJun = Pu.1 and not Gfi1
EgrNab = (Pu.1 and cJun) and not Gfi1
Gfi1 = Cebpa and not EgrNab
is translated into a stochastic differential equation (Wittmann et al., 2009). Simulations result
in branching time series of gene expression, where each branch corresponds to a
certain cell fate of common myeloid progenitors (megakaryocytes, erythrocytes,
granulocytes and monocytes).
Instead of simulating the data as in the next line, you can also retrieve it from the builtin examples
adata = sc.datasets.krumsiek11()
This uses the default parameters from here. To change parameters, read in a different parameter file or simply pass them to sc.tl.sim.
Can you write Python code for it?


I am working on the file "170430_krumsiek11/krumsiek11.ipynb".
The first task is:
Plot the four realizations of time series.
Can you write Python code for it?


I am working on the file "170430_krumsiek11/krumsiek11.ipynb".
The first task is:
Reconstructing progression and branching using DPT
Can you write Python code for it?

adata.uns['iroot'] = 0
sc.tl.dpt(adata, n_branchings=2)

sc.pl.tsne(adata, color='dpt_pseudotime')
sc.pl.draw_graph(adata, color='dpt_pseudotime')
sc.pl.diffmap(adata, color='dpt_pseudotime', projection='3d')


I am working on the file "170501_moignard15/moignard15.ipynb".
The first task is:
Compute branchings and diffusion pseudotime using DPT.
Can you write Python code for it?

sc.tl.diffmap(adata)

sc.tl.dpt(adata, n_branchings=1, n_dcs=10)

sc.pl.diffmap(adata, color=['dpt_pseudotime', 'dpt_groups', 'exp_groups'])

sc.pl.dpt_groups_pseudotime(adata)

sc.pl.dpt_timeseries(adata)


I am working on the file "170501_moignard15/moignard15.ipynb".
The first task is:
Compute the neighborhood relations of single cells.
Can you write Python code for it?

sc.pp.neighbors(adata, n_neighbors=5, method='gauss', knn=False)


I am working on the file "170501_moignard15/moignard15.ipynb".
The first task is:
Let us annotate the cell groups as follows.
Can you write Python code for it?

adata.rename_categories('dpt_groups', ['undecided/endothelial', 'erythrocytes', 'trunk', 'endothelial'])

sc.pl.diffmap(adata, color='dpt_groups')


I am working on the file "170501_moignard15/moignard15.ipynb".
The first task is:
Save the results.
Can you write Python code for it?


I am working on the file "170501_moignard15/moignard15.ipynb".
The first task is:
This is single-cell qPCR data. It's already on a logrithmic scale and pretty low-dimensional. We do not have to perform extensive preprocessing.

Filter out a few genes.
Choose a 'root cell'.
Define groupnames by inspecting cellnames.
Can you write Python code for it?


I am working on the file "170502_paul15/paul15.ipynb".
The first task is:
Computing connectivities using UMAP gives us quantitiatively different results for the pseudotime.
Can you write Python code for it?

sc.pp.neighbors(adata, n_neighbors=20, use_rep='X', method='umap')

sc.tl.diffmap(adata)

sc.tl.dpt(adata, n_branchings=1)

sc.pl.diffmap(adata, color=['dpt_pseudotime', 'dpt_groups', 'paul15_clusters'])


I am working on the file "170502_paul15/paul15.ipynb".
The first task is:
With this, we reproduced the analysis of Haghverdi et al. (2016, Suppl. Note 4 and Suppl. Figure N4).
Can you write Python code for it?


I am working on the file "170522_visualizing_one_million_cells/cluster_20K.ipynb".
The first task is:
Now compare this with the reference clustering of PAGA preprint, Suppl. Fig. 12, available from here.
Can you write Python code for it?

anno = pd.read_csv('/Users/alexwolf/Dropbox/1M/louvain.csv.gz', compression='gzip', header=None, index_col=0)

anno.columns = ['louvain_ref']

adata.obs['louvain_ref'] = anno.loc[adata.obs.index]['louvain_ref'].astype(str)

sc.pl.umap(adata, color=['louvain_ref'], legend_loc='on data')

adata.write('./write/subsampled.h5ad')


I am working on the file "170522_visualizing_one_million_cells/cluster_20K.ipynb".
The first task is:
Run standard preprocessing steps, see here.
Can you write Python code for it?

sc.pp.recipe_zheng17(adata)

sc.tl.pca(adata)

sc.pp.neighbors(adata)

sc.tl.umap(adata)

sc.tl.louvain(adata)

sc.tl.paga(adata)

sc.pl.paga_compare(adata, edges=True, threshold=0.05)


I am working on the file "170522_visualizing_one_million_cells/plot.ipynb".
The first task is:
We illustrate the concept using the subsampled data, simply replace the filename... An indepth analysis with 1.3M cells will follow in the first half of 2018.
Can you write Python code for it?


I am working on the file "171106_t-test_wilcoxon_comparison/Generic Comparison T-Test Wilcoxon-Rank-Sum Test.ipynb".
The first task is:
Cluster according to true grouping:
The following code includes the true grouping such that it can be accessed by normal function calling of
sc.tl.rank_genes_groups(adata,'true_groups')

or, respectively,
sc.tl.rank_genes_groups(adata,'true_groups', test_type='wilcoxon')
Can you write Python code for it?

import pandas as pd
smp='true_groups'
true_groups_int=np.ones((n_simulated_cells,))
true_groups_int[0:n_cluster]=0
true_groups=list()
for i,j  in enumerate(true_groups_int):
    true_groups.append(str(j))
adata.smp['true_groups']=pd.Categorical(true_groups, dtype='category')
adata.uns[smp + '_order']=np.asarray(['0','1'])