I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
As you should suspect, the problem is that NumPy arrays are not meant to be iterated by Python, but we have to implement this algorithm in a vectorial way (or make it faster with Numba/Cython).
The next is a re-implementation of the algorithm, using native NumPy functions:
Can you write Python code for it?

def closest_centroid(points, centroids):
    """returns an array containing the index to the nearest centroid for each point"""
    px = points[:,0].reshape((-1,1))
    py = points[:,1].reshape((-1,1))
    Dx = px - centroids[:,0].reshape((1,-1))
    Dy = py - centroids[:,1].reshape((1,-1))
    # distance matrix
    D = np.sqrt(Dx**2+Dy**2)
    return np.argmin(D, axis=1)


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
Clearly the problem is the closest_centroid function!. Now that we have isolated the problem, we do a line profile of this single function:
Can you write Python code for it?

%lprun -T lprof2 -f closest_centroid closest_centroid(points, centroids)

print(open('lprof2', 'r').read())


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
In the example, -s allows us to sort the report by a particular column, -q to suppress the pager output, -l to limit the number of lines displayed or to filter the results by function name, and -T to save the report
in a text  file. . This database-like object contains all information about the profiling and can be analyzed through Python's pstats module. For more info about arguments run %prun?.
The profiling report has been saved in a text file named prun0. Let's display it:
Can you write Python code for it?

print(open('prun0', 'r').read())

def plot_helper(y, i, n):
    plt.figure(figsize=(10,7))
    plt.plot(np.arange(-30,29), y[i], 'ro-')
    plt.title("Distribution of {0} simultaneous random walks at iteration {1}".format(n,i))
    plt.show()

interact(plot_helper, y=fixed(y), i=(0,500), n=fixed(10000))


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
Let's profile again:
Can you write Python code for it?

%%prun -s cumulative -q -l 15 -T prun2
main_loop(points, centroids, 1000)

print(open('prun2', 'r').read())


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
Let's time this computation in pure Python (Using list comprehensions)
Can you write Python code for it?

t1 = %timeit -o -n 100 sum([1. / i**2 for i in range(1, n)])


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
Now, let's import this script into the interactive namespace so that we can execute and profile our code.
The functions to be profiled need to be explicitly specified in the %lprun magic command. We also save the report in a file, lprof0:
Can you write Python code for it?

import simulation

%lprun -T lprof0 -f simulation.simulate simulation.simulate(500)


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
Table of Contents

1.- About Optimization
2.- Time Profiling
3.- Memory Profiling
4.- Application: K-means Clustering Algorithm

1.- About Optimization
"The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming". Donald Knuth.

Optimizing code prematurely is generally considered a bad practice.
Code optimization should only be conducted when it's really needed.
We should know exactly where we need to optimize your code.
Typically the majority of the execution time comprises a relatively small part of the code.
Optimization should never be done without preliminary profiling.

2.- Time Profiling
2.1- Time Benchmarking: timeit
The %timeit magic and the %%timeit cell magic allow you to quickly evaluate (benchmark) the time taken by one or several Python statements.
Some useful options: %timeit?
Options:

n: Execute the given statement <N> times in a loop. If this value
is not given, a fitting value is chosen.

r: Repeat the loop iteration <R> times and take the best result.
Default: 3

t: Use time.time to measure the time, which is the default on Unix.
This function measures wall time.

c: Use time.clock to measure the time, which is the default on
Windows and measures wall time. On Unix, resource.getrusage is used
instead and returns the CPU user time.

p: Use a precision of
 digits to display the timing result.
Default: 3

q: Quiet, do not print result.

o: Return a TimeitResult that can be stored in a variable to inspect
    the result in more details.

We are going to estimate the time taken to calculate the sum of the inverse squares of all positive integer numbers up to a given n. Let's first define n:
Can you write Python code for it?


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
The following function computes which is the closest centroid for each point in the dataset
Can you write Python code for it?

def closest_centroid(points,centroids):
    """returns an array containing the index to the nearest centroid for each point"""
    # computation of distance matrix
    m = points.shape[0]
    n = centroids.shape[0]
    D = np.zeros((m,n))
    for i in range(m):
        for j in range(n):
            D[i,j] = np.sqrt( np.sum( (points[i]-centroids[j])**2 ) )
    return np.argmin(D, axis=1)

closest = closest_centroid(points,centroids)


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
The memory_profiler IPython extension also comes with a %memit magic command that lets us benchmark the memory used by a single Python statement. Here is a simple example:
Can you write Python code for it?


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
The memory_profiler package checks the memory usage of the interpreter at every line. The increment column allows us to spot those places in the code where large amounts of memory are allocated.
Now, let's run the code under the control of the memory profiler:
Can you write Python code for it?

%mprun -T mprof0 -f simulation.simulate simulation.simulate(1500)


I am working on the file "06_profiling/06_profiling.ipynb".
The first task is:
The object returned by timeit contains information about the time measurements:
Can you write Python code for it?

print("Type:")
print(type(t1))

print("\nTime of all runs:")
print(t1.all_runs)

print("\nBest measured time:")
print(t1.best)

print("\nWorst measured time:")
print(t1.worst)

print("\nCompilation time:")
print(t1.compile_time)


I am working on the file "06_readers.ipynb".
The first task is:
Using Readers
Can you write Python code for it?

filenames = ["data/life_satisfaction.csv"]
n_epochs = 500

graph = tf.Graph()
with graph.as_default():
    reader = tf.TextLineReader(skip_header_lines=1)

    filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
    record_id, record = reader.read(filename_queue)

    record_defaults = [[''], [0.0], [0.0]]
    country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)

batch_size = 5
with graph.as_default():
    X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
    X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
    y_batch_reshaped = tf.reshape(y_batch, [-1, 1])

with graph.as_default():
    X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
    y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")

    b = tf.Variable(0.0, name="b")
    w = tf.Variable(tf.zeros([1, 1]), name="w")
    y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred")  # X @ w + b

    mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")

    global_step = tf.Variable(0, trainable=False, name='global_step')
    optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
    training_op = optimizer.minimize(mse, global_step=global_step)

    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
    saver = tf.train.Saver()

with tf.Session(graph=graph) as sess:
    init.run()
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)
    try:
        while not coord.should_stop():
            _, mse_val, global_step_val = sess.run([training_op, mse, global_step])
            if global_step_val % 100 == 0:
                print(global_step_val, mse_val)
    except tf.errors.OutOfRangeError:
        print("End of training")
    coord.request_stop()
    coord.join(threads)
    saver.save(sess, "./my_life_satisfaction_model")


I am working on the file "06_repartition/repartition_all_spark.ipynb".
The first task is:
Convert and repartition Subway Dataframe using PySpark
Can you write Python code for it?

# standard imports
funcs = pyspark.sql.functions
types = pyspark.sql.types

sqlContext.sql("set spark.sql.shuffle.partitions=32")
subway = spark.read.parquet('/data/subway.parquet')
subway.registerTempTable("subway")

spark.sql('select * from subway limit 5').toPandas()

subway = \
    subway.withColumn('endtime', funcs.from_unixtime((subway.endtime/1000000).cast(types.IntegerType())) \
                      .cast(types.TimestampType()))
subway.registerTempTable("subway2")

spark.sql('select * from subway2 limit 5').toPandas()

subway = subway.sort("ca", "unit", "scp", "endtime")
subway.write.parquet('/data/subway_spark.parquet', compression='snappy', mode='overwrite')


I am working on the file "06_reproducibility/bridging_schema.ipynb".
The first task is:
Bridging
Create dataset of 5000 new transactions and 1 million old transactions
Can you write Python code for it?

df_new = create_dataset(5000, False)
df_old = create_dataset(1000*1000, True)


I am working on the file "06_reproducibility/bridging_schema.ipynb".
The first task is:
How many samples do we need to evaluate properly?
Train a model on the old dataset and evaluate it on different subsets
Can you write Python code for it?

N_train = (int) (len(df_old) * 0.5)
N_eval = len(df_old) - N_train

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import OneHotEncoder

model = LinearRegression()
enc = OneHotEncoder(handle_unknown='ignore')
df_train = enc.fit_transform( df_old.loc[:N_train, df.columns != 'tip'] )
model.fit(df_train, df_old.loc[:N_train, 'tip'])

df_eval = enc.transform( df_old.loc[N_train:, df.columns != 'tip'] )
model.score(df_eval, df_old.loc[N_train:, 'tip'])

sizes = []
score_mean = []
score_stddev = []
for subset_size in range(100, 5000, 100): # 5000 here because that is the number of new examples we have
    sizes.append(subset_size)
    # compute variability of the eval metric at this subset size
    scores = []
    for x in range(1, 25):
        indices = np.random.choice(N_eval, size=subset_size, replace=False)
        scores.append(
            model.score(df_eval[indices], df_old.loc[N_train+indices, 'tip'])
        )
    score_mean.append(np.mean(scores))
    score_stddev.append(np.std(scores))
scores = pd.DataFrame.from_dict({
    'eval_size': sizes,
    'score': score_mean,
    'stddev': score_stddev
})
scores.plot(x='eval_size', y='stddev', linewidth=2);


I am working on the file "06_reproducibility/bridging_schema.ipynb".
The first task is:
Prepare new data
Because we explicitly encoded the fields for the bridged data, let's make sure to repeat this for the new data also.
Can you write Python code for it?

encoded_new = df_new.copy()
encoded_new['encoded_cash'] = 0  # default to zero
encoded_new['encoded_gift'] = 0  # default to zero
encoded_new['encoded_debit'] = 0  # default to zero
encoded_new['encoded_credit'] = 0  # default to zero
encoded_new.loc[encoded_new['payment_type'] == 'Cash', 'encoded_cash'] = 1 # one-hot encode
encoded_new.loc[encoded_new['payment_type'] == 'Gift', 'encoded_gift'] = 1 # one-hot encode
encoded_new.loc[encoded_new['payment_type'] == 'Debit', 'encoded_debit'] = 1 # one-hot encode
encoded_new.loc[encoded_new['payment_type'] == 'Credit', 'encoded_credit'] = 1 # one-hot encode
encoded_new.drop(columns='payment_type', inplace=True)
encoded_new[:5]

encoded_new_eval = encoded_new.loc[:N_NEW, :]
evalX = encoded_new_eval.loc[:, encoded_new_eval.columns != 'tip']
evalY = encoded_new_eval.loc[:, 'tip']
encoded_new_train = encoded_new.loc[N_NEW:, :]


I am working on the file "06_reproducibility/feature_store.ipynb".
The first task is:
Batch predict job for all taxi_ids
Can you write Python code for it?

# Create a Pandas dataframe
features_dict = dict.fromkeys([FS_NAME + "__" + feature for feature in model_features] + ['taxi_id'])

# all taxi_ids
taxi_ids = taxi_df.taxi_id.unique()

entity_rows = []
for taxi_id in taxi_ids.tolist():
    entity_rows.append(
        GetOnlineFeaturesRequest.EntityRow(fields={'taxi_id': Value(int64_val=taxi_id)})
    )

data = _feast_online_client.get_online_features(
    feature_refs=[FS_NAME + ":" + feature for feature in model_features],
    entity_rows=entity_rows)

for row in data.field_values:
    # capture taxi_id
    if features_dict['taxi_id'] is None:
        features_dict['taxi_id'] = []
    features_dict['taxi_id'].append(row.fields['taxi_id'].int64_val)

    # get all feature values
    for feature in model_features:
        if features_dict[FS_NAME + "__" + feature] is None:
            features_dict[FS_NAME + "__" + feature] = []
        if feature in ['passenger_count']:
            features_dict[FS_NAME + "__" + feature].append(row.fields[FS_NAME + ":" + feature].int64_val)
        else:
            features_dict[FS_NAME + "__" + feature].append(row.fields[FS_NAME + ":" + feature].double_val)

predict_df = pd.DataFrame.from_dict(features_dict)
predict_df.head()

pd.DataFrame.from_dict({'taxi_id': predict_df.taxi_id,
                        'prediciton': model.predict(predict_df.drop('taxi_id', axis=1))})


I am working on the file "06_reproducibility/feature_store.ipynb".
The first task is:
Create new features: Euclidean distance
Engineer an additional feature which provides the Euclidean distance from pickup location to dropoff location.
Can you write Python code for it?

def compute_dist(row):
    lat1, lon1 = row.pickup_latitude, row.pickup_longitude
    lat2, lon2 = row.dropoff_latitude, row.dropoff_longitude
    londiff = lon2 - lon1
    latdiff = lat2 - lat1
    return np.sqrt(londiff*londiff + latdiff*latdiff)

taxi_df['euclid_dist'] = taxi_df.apply(compute_dist, axis=1)

taxi_df.head()


I am working on the file "06_reproducibility/feature_store.ipynb".
The first task is:
Import libraries and modules
Can you write Python code for it?

#!pip install --user feast

#!pip install --user xgboost

import pandas as pd
import numpy as np
from pytz import timezone, utc
from feast import Client, FeatureSet, Entity, ValueType
from feast.serving.ServingService_pb2 import GetOnlineFeaturesRequest
from feast.types.Value_pb2 import Value as Value
from google.protobuf.duration_pb2 import Duration
from datetime import datetime, timedelta
from random import randrange
import random

from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor


I am working on the file "06_reproducibility/feature_store.ipynb".
The first task is:
Predict with online features
Can you write Python code for it?

_feast_online_client = Client(serving_url=FEAST_ONLINE_SERVING_URL)

# for a single taxi_id
taxi_id = 1

online_features = _feast_online_client.get_online_features(
    feature_refs=[FS_NAME + ":" + feature for feature in model_features],
    entity_rows=[
        GetOnlineFeaturesRequest.EntityRow(
            fields={
                "taxi_id": Value(
                    int64_val=taxi_id)
            }
        )
    ],
)

print(online_features)

# Convert to Pandas dataframe
features_dict = dict.fromkeys([FS_NAME + "__" + feature for feature in model_features])

for row in online_features.field_values:
    for feature in model_features:
        if features_dict[FS_NAME + "__" + feature] is None:
            features_dict[FS_NAME + "__" + feature] = []
        if feature in ['passenger_count']:
            features_dict[FS_NAME + "__" + feature].append(row.fields[FS_NAME + ":" + feature].int64_val)
        else:
            features_dict[FS_NAME + "__" + feature].append(row.fields[FS_NAME + ":" + feature].double_val)

features_dict

predict_df = pd.DataFrame.from_dict(features_dict)

model.predict(predict_df)


I am working on the file "06_reproducibility/feature_store.ipynb".
The first task is:
Retrieving feature stores from Feast
Get batch features for training
To access historical/office features, we'll set up a Feast serving batch client.
Can you write Python code for it?

_feast_batch_client = Client(serving_url=FEAST_BATCH_SERVING_URL,
                             core_url=FEAST_CORE_URL)

model_features = ['pickup_latitude',
                  'pickup_longitude',
                  'dropoff_latitude',
                  'dropoff_longitude',
                  'passenger_count',
                  'euclid_dist']

target = 'fare_amount'

# Add the target variable to our feature list
features = model_features + [target]


I am working on the file "06_reproducibility/model_versioning.ipynb".
The first task is:
Deploying classification model to AI Platform
Replace your-cloud-project below with the name of your cloud project.
Can you write Python code for it?

# Set your cloud project
PROJECT = 'your-cloud-project'
!gcloud config set project $PROJECT

BUCKET = PROJECT + '_flight_model_bucket'

# Create a bucket if you don't have one
# You only need to run this once
!gsutil mb gs://$BUCKET

!gsutil cp 'model.bst' gs://$BUCKET

# Create the model resource
!gcloud ai-platform models create flight_delay_prediction

# Create the version
!gcloud ai-platform versions create 'v1' \
  --model 'flight_delay_prediction' \
  --origin gs://$BUCKET \
  --runtime-version=1.15 \
  --framework 'XGBOOST' \
  --python-version=3.7

# Get a prediction on the first example from our test set
!rm input.json
num_examples = 10
with open('input.json', 'a') as f:
  for i in range(num_examples):
    f.write(str(x_test.iloc[i].values.tolist()))
    f.write('\n')

!cat input.json

# Make a prediction to the deployed model
!gcloud ai-platform predict --model 'flight_delay_prediction' --version \
  'v1' --json-instances 'input.json'

# Compare this with actual values
print(y_test.iloc[:5])


I am working on the file "06_reproducibility/model_versioning.ipynb".
The first task is:
Model version #1: predict whether or not the flight is > 30 min delayed
Can you write Python code for it?

# Create a boolean column to indicate whether flight was > 30 mins delayed
df.loc[df['arr_delay'] >= 30, 'arr_delay_bool'] = 1
df.loc[df['arr_delay'] < 30, 'arr_delay_bool'] = 0

df['arr_delay_bool'].value_counts()

classify_model_labels = df['arr_delay_bool']
classify_model_data = df.drop(columns=['arr_delay', 'arr_delay_bool'])

x,y = classify_model_data,classify_model_labels
x_train,x_test,y_train,y_test = train_test_split(x,y)

model = xgb.XGBRegressor(
    objective='reg:logistic'
)

# Given the dataset size, this may take 1-2 minutes to run
model.fit(x_train, y_train)

y_pred = model.predict(x_test)

acc = accuracy_score(y_test, np.round(y_pred))
print(acc)

# Save the model
model.save_model('model.bst')


I am working on the file "06_reproducibility/model_versioning.ipynb".
The first task is:
Model version #2: replace XGBoost with TensorFlow
Can you write Python code for it?

tf_model = tf.keras.Sequential([
    tf.keras.layers.Dense(32, activation='relu', input_shape=[len(x_train.iloc[0])]),
    tf.keras.layers.Dense(32, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

tf_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

tf_model.fit(x_train, y_train, epochs=10, validation_split=0.1)


I am working on the file "06_reproducibility/model_versioning.ipynb".
The first task is:
Next we'll deploy the updated TF model to AI Platform as a v2.
Can you write Python code for it?

tf_model_path = 'gs://' + BUCKET + '/tf'

tf_model.save(tf_model_path, save_format='tf')

!gcloud ai-platform versions create 'v2' \
  --model 'flight_delay_prediction' \
  --origin $tf_model_path \
  --runtime-version=2.1 \
  --framework 'TENSORFLOW' \
  --python-version=3.7

# Make a prediction to the new version
!gcloud ai-platform predict --model 'flight_delay_prediction' --version \
  'v2' --json-instances 'input.json'


I am working on the file "06_reproducibility/model_versioning.ipynb".
The first task is:
Note that accuracy will be similar to the XGBoost model. We're just using this to demonstrate how training a model with a different framework could be deployed as a new version.
Can you write Python code for it?

metrics = tf_model.evaluate(x_test, y_test)
print(metrics)


I am working on the file "06_reproducibility/storm_reports/similar_reports.ipynb".
The first task is:
Explore data in BigQuery
Preview data in BigQuery
Can you write Python code for it?

%%bigquery
SELECT
  EXTRACT(YEAR from timestamp) AS year,
  EXTRACT(DAYOFYEAR from timestamp) AS julian_day,
  latitude, longitude,
  comments,
  'wind' as type,
FROM `bigquery-public-data.noaa_preliminary_severe_storms.wind_reports`
LIMIT 10

%%bigquery
SELECT
  EXTRACT(YEAR from timestamp) AS year,
  EXTRACT(DAYOFYEAR from timestamp) AS julian_day,
  latitude, longitude,
  comments,
  size,
  'hail' as type
FROM `bigquery-public-data.noaa_preliminary_severe_storms.hail_reports`
LIMIT 10

%%bigquery
SELECT
  EXTRACT(YEAR from timestamp) AS year,
  EXTRACT(DAYOFYEAR from timestamp) AS julian_day,
  latitude, longitude,
  LOWER(comments) AS comments,
  REGEXP_EXTRACT(comments, r"\([A-Z]+\)$") AS office,
  'tornado' as type
FROM `bigquery-public-data.noaa_preliminary_severe_storms.tornado_reports`
LIMIT 10

query = """
WITH wind AS (
SELECT
  EXTRACT(YEAR from timestamp) AS year,
  EXTRACT(DAYOFYEAR from timestamp) AS julian_day,
  latitude, longitude,
  comments,
  'wind' as type
FROM `bigquery-public-data.noaa_preliminary_severe_storms.wind_reports`
),

hail AS (
SELECT
  EXTRACT(YEAR from timestamp) AS year,
  EXTRACT(DAYOFYEAR from timestamp) AS julian_day,
  latitude, longitude,
  comments,
  'hail' as type
FROM `bigquery-public-data.noaa_preliminary_severe_storms.hail_reports`
),

tornadoes AS (
SELECT
  EXTRACT(YEAR from timestamp) AS year,
  EXTRACT(DAYOFYEAR from timestamp) AS julian_day,
  latitude, longitude,
  comments,
  'tornado' as type
FROM `bigquery-public-data.noaa_preliminary_severe_storms.tornado_reports`
)

SELECT * FROM (
   SELECT * FROM wind
   UNION ALL
   SELECT * FROM hail
   UNION ALL
   SELECT * FROM tornadoes
)
"""

## skip_for_export
import google.cloud.bigquery as bq
df = bq.Client().query(query).result().to_dataframe()
df.groupby('type').count()


I am working on the file "06_reproducibility/storm_reports/similar_reports.ipynb".
The first task is:
Ingest data
We'll use the TFX component BigQueryExampleGen to read in the data.
Can you write Python code for it?

import tensorflow as tf
print('tensorflow ' + tf.__version__)
import tfx
print('tfx ' + tfx.__version__)
import apache_beam as beam
print('beam ' + beam.__version__)

from tfx.components import BigQueryExampleGen
example_gen = BigQueryExampleGen(query=query)

import os
beam_pipeline_args = [
    '--runner={}'.format(BEAM_RUNNER),
    '--project={}'.format(PROJECT),
    '--temp_location=' + os.path.join('gs://{}/noaa_similar_reports/'.format(BUCKET), 'tmp'),
    '--region=' + REGION,

    # Temporary overrides of defaults.
    '--disk_size_gb=50',
    '--experiments=shuffle_mode=auto',
    '--machine_type=n1-standard-8',
]

## skip_for_export
from tfx.orchestration.experimental.interactive.interactive_context import InteractiveContext
context = InteractiveContext()

## skip_for_export
ingest_result = context.run(example_gen, beam_pipeline_args=beam_pipeline_args)

## skip_for_export
context.show(ingest_result)

## skip_for_export
print(ingest_result)


I am working on the file "06_reproducibility/storm_reports/similar_reports.ipynb".
The first task is:
Validate the data
Let's generate statistics from the data
Can you write Python code for it?

from tfx.components import StatisticsGen
stats_gen = StatisticsGen(examples=example_gen.outputs['examples'])

## skip_for_export
context.run(stats_gen, beam_pipeline_args=beam_pipeline_args)

from tfx.components import SchemaGen
# This component only generates a new schema if one doesn't already exist.
schema_gen = SchemaGen(statistics=stats_gen.outputs['statistics'],
                       infer_feature_shape=True)

## skip_for_export
context.run(schema_gen, beam_pipeline_args=beam_pipeline_args)

from tfx.components import ExampleValidator

example_validator = ExampleValidator(
    statistics=stats_gen.outputs['statistics'],
    schema=schema_gen.outputs['schema']
)

## skip_for_export
context.run(example_validator, beam_pipeline_args=beam_pipeline_args)


I am working on the file "06b - Fitting Plots.ipynb".
The first task is:
Fit a nonlinear  (exponential) function:
Can you write Python code for it?

np.random.seed(1) # Use a seed for the random number generator

X = np.linspace(0, 5, 30)
Y = np.exp(-X) + normal(0, 0.2, 30)

plt.scatter(X,Y)
plt.title('Scatter Plot')
plt.show()


I am working on the file "06b - Fitting Plots.ipynb".
The first task is:
How accurate are the estimated parameters for the data?
The output of the curve_fit function includes pcov, the estimated covariance of popt.
Properly estimating the variance is an in-depth statistical problem. However, if we are to trust the assumptions curve_fit make (type of data, relationships between estimated parameters, etc), then the square root of the diagonals of the pcov from above approximates the error. This should be done with much caution!
Can you write Python code for it?

perr = np.sqrt(np.diag(pcov)) # One standard deviation errors in the
                             #    parameters
print('Estimated errors are: {}'.format(perr))

# Remember that func is: A * np.exp(-B * x_vals) + C
std_p = [perr[0], -perr[1], perr[2]] # Combination of perr for highest values
std_m = [-perr[0], perr[1], -perr[2]]# Combination of perr fpr lowest values

# Determine the plots for the ± standard deviation
y_fitted_p = func(x_fitted, *(popt+std_p))
y_fitted_m = func(x_fitted, *(popt+std_m))

# Plot
plt.scatter(X,Y)
plt.plot(x_fitted,y_fitted, color="red")
plt.plot(x_fitted,y_fitted_p, color="blue", label='+ stdev')
plt.plot(x_fitted,y_fitted_m, color="blue", label='- stdev')
plt.title('Scatter Plot')
plt.legend(loc='best')

plt.show()


I am working on the file "06initial_analysis.ipynb".
The first task is:
Cross Validation and Random Forest
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import cross_val_score
RFClass = RandomForestClassifier(n_estimators = 10000,
                                 max_features = 4, # You can set it to a number or 'sqrt', 'log2', etc
                                 min_samples_leaf = 5,
                                 oob_score = True)
RFClass.fit(X,y)
print(RFClass.oob_score_)
scores = cross_val_score(RFClass, X, y, cv=10)
print(scores.mean())
#out of bag error = 25% , CV_error is 35%

RFClass.fit(X,y)
ImportanceDataFrame = pd.DataFrame({'feature':X.columns.values, 'importance':RFClass.feature_importances_})
ImportanceDataFrame.sort_values(by = ['importance'],ascending = 0)

Depth_Choice_df = pd.DataFrame({'cv_scores': score,'Number of Features': Features})
Depth_Choice_df.plot(x ='Number of Features',y = 'cv_scores')


I am working on the file "06initial_analysis.ipynb".
The first task is:
Run Decision Trees, Prune, and consider False Positives
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier
TreeClass = DecisionTreeClassifier(
                max_depth = 2,
                min_samples_leaf = 5)
TreeClass.fit(X,y)

from sklearn.cross_validation import cross_val_score
scores = cross_val_score(TreeClass, X, y, cv=10)
print(scores.mean()) # Score = More is better, error is 1-score

from sklearn.metrics import confusion_matrix
y_hat = TreeClass.predict(X)
cmat = confusion_matrix(y, y_hat)
print cmat

from sklearn.metrics import roc_curve, auc,roc_auc_score
y_hat_probability = TreeClass.predict_proba(X).T[1]
print(y_hat_probability)
print(roc_auc_score(y, y_hat_probability))
vals = roc_curve(y, y_hat_probability)

Roc_DataFrame = pd.DataFrame({'False_Positive_Rate':vals[0],'True_Positive_Rate':vals[1]})
Roc_DataFrame.plot(x = 'False_Positive_Rate' , y = 'True_Positive_Rate' )


I am working on the file "06initial_analysis.ipynb".
The first task is:
Seeing if I can get anything interesting out of KNN given above
Lecture 10, look at Confusion matrix and ROC curve. Fiddle with the thresholds and AUC
Can you write Python code for it?

print df['windspeed'].max()
print df['windspeed'].min()

df['windspeed_st'] = df['windspeed'].apply(lambda x:x/15.0) # Ballparking

X_reg = df[['precipiation','d_friday','t_northbound','windspeed_st']]
y_reg = df['is_delay']

from sklearn import cross_validation
from sklearn import neighbors, metrics
kf = cross_validation.KFold(len(X_reg), n_folds = 10, shuffle = True) #10 fold CV
Score_KNN_CV = []
RangeOfK = range(1,20)
scores = []
for k in RangeOfK:
    knn = neighbors.KNeighborsClassifier(n_neighbors=k, weights='uniform')
    scores = []
    for train_index, test_index in kf:
        knn.fit(X_reg.iloc[train_index], y_reg.iloc[train_index])
        scores.append(knn.score(X_reg.iloc[test_index],y_reg.iloc[test_index]))
    Score_KNN_CV.append(np.mean(scores))

Score_KNN_CV_df = pd.DataFrame({'Score_KNN_CV': Score_KNN_CV ,'K': RangeOfK })
Score_KNN_CV_df.plot(x = 'K',y = 'Score_KNN_CV',figsize=[15,5])


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
2. Now, we can calculate the mean of each month over the different years:
Can you write Python code for it?

# %load snippets/07 - Case study - air quality data58.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
3. plot the typical monthly profile of the different stations:
Can you write Python code for it?

# %load snippets/07 - Case study - air quality data59.py

data = data.drop('month', axis=1, errors='ignore')


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:

EXERCISE:

Loop over the data files, read and process the file using our defined function, and append the dataframe to a list.

Combine the the different DataFrames in the list into a single DataFrame where the different columns are the different stations. Call the result `combined_data`.

Can you write Python code for it?

# %load snippets/07 - Case study - air quality data34.py

# %load snippets/07 - Case study - air quality data35.py

combined_data.head()


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:

    QUESTION: And are there exceedances of the yearly limit value of 40 µg/m3 since 200 ?

Can you write Python code for it?

# %load snippets/07 - Case study - air quality data72.py

# %load snippets/07 - Case study - air quality data73.py

# %load snippets/07 - Case study - air quality data74.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:

    QUESTION: The evolution of the yearly averages with, and the overall mean of all stations (indicate the overall mean with a thicker black line)?

Can you write Python code for it?

# %load snippets/07 - Case study - air quality data55.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:

    QUESTION: The maximum daily 8 hour mean should be below 100 µg/m³. What are the number of exceedances of this limit for each year/station?

Tip: have a look at the rolling method to perform moving window operations.
Note: this is not an actual limit for NO2, but a nice exercise to introduce the rolling method. Other pollutans, such as 03 have actually such kind of limit values.
Can you write Python code for it?

# %load snippets/07 - Case study - air quality data86.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:

    QUESTION: What are the number of exceedances of hourly values above the European limit 200 µg/m3 for each year/station?

Can you write Python code for it?

# %load snippets/07 - Case study - air quality data67.py

# %load snippets/07 - Case study - air quality data68.py

# %load snippets/07 - Case study - air quality data69.py

# %load snippets/07 - Case study - air quality data70.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:

    QUESTION: plot the monthly mininum and maximum daily concentration of the 'BETR801' station

Can you write Python code for it?

# %load snippets/07 - Case study - air quality data52.py

# %load snippets/07 - Case study - air quality data53.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:

    QUESTION: plot the weekly 95% percentiles of the concentration in 'BETR801' and 'BETN029' for 2011

Can you write Python code for it?

df2011 = data['2011'].dropna()

# %load snippets/07 - Case study - air quality data64.py

# %load snippets/07 - Case study - air quality data65.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
Add a column indicating week/weekend
Can you write Python code for it?

# %load snippets/07 - Case study - air quality data77.py

# %load snippets/07 - Case study - air quality data78.py

# %load snippets/07 - Case study - air quality data79.py

# %load snippets/07 - Case study - air quality data80.py

# %load snippets/07 - Case study - air quality data81.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
An alternative method using groupby and unstack:
Can you write Python code for it?

# %load snippets/07 - Case study - air quality data85.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
But, groupby is more flexible and can also do resamples that do not result in a new continuous time series, e.g. by grouping by the hour of the day to get the diurnal cycle.

    QUESTION: how does the *typical monthly profile* look like for the different stations?

1. add a column to the dataframe that indicates the month (integer value of 1 to 12):
Can you write Python code for it?

# %load snippets/07 - Case study - air quality data57.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
Combination with groupby
resample can actually be seen as a specific kind of groupby. E.g. taking annual means with data.resample('A', 'mean') is equivalent to data.groupby(data.index.year).mean() (only the result of resample still has a DatetimeIndex).
Can you write Python code for it?


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
Exercises

REMINDER:

Take a look at the [Timeseries notebook](05 - Time series data.ipynb) when you require more info about...

`resample`

string indexing of DateTimeIndex

    QUESTION: plot the monthly mean and median concentration of the 'FR04037' station for the years 2009-2012

Can you write Python code for it?

# %load snippets/07 - Case study - air quality data50.py

# %load snippets/07 - Case study - air quality data51.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
Finally, we don't want to have to repeat this each time we use the data. Therefore, let's save the processed data to a csv file.
Can you write Python code for it?


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
For the sake of this tutorial, we will disregard the 'flag' columns (indicating the quality of the data).

EXERCISE:

Drop all 'flag' columns ('flag1', 'flag2', ...)
Can you write Python code for it?

flag_columns = [col for col in data.columns if 'flag' in col]
# we can now use this list to drop these columns

# %load snippets/07 - Case study - air quality data10.py

data.head()


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
Now, we want to reshape it: our goal is to have the different hours as row indices, merged with the date into a datetime-index. Here we have a wide and long dataframe, and want to make this a long, narrow timeseries.

REMEMBER:

Recap: reshaping your data with [`stack` and `unstack`](./pandas_07_reshaping_data.ipynb)

EXERCISE:

Reshape the dataframe to a timeseries.
The end result should look like:

      BETR801

      1990-01-02 09:00:00
      48.0

      1990-01-02 12:00:00
      48.0

      1990-01-02 13:00:00
      50.0

      1990-01-02 14:00:00
      55.0

      ...
      ...

      2012-12-31 20:00:00
      16.5

      2012-12-31 21:00:00
      14.5

      2012-12-31 22:00:00
      16.5

      2012-12-31 23:00:00
      15.0

170794 rows × 1 columns

Reshape the dataframe so that each row consists of one observation for one date + hour combination

When you have the date and hour values as two columns, combine these columns into a datetime (tip: string columns can be summed to concatenate the strings) and remove the original columns

Set the new datetime values as the index, and remove the original columns with date and hour values

**NOTE**: This is an advanced exercise. Do not spend too much time on it and don't hesitate to look at the solutions.

Can you write Python code for it?

# %load snippets/07 - Case study - air quality data12.py

# %load snippets/07 - Case study - air quality data13.py

# %load snippets/07 - Case study - air quality data14.py

# %load snippets/07 - Case study - air quality data15.py

# %load snippets/07 - Case study - air quality data16.py

data_stacked.head()


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
Quickly visualizing the data
Can you write Python code for it?

data.plot(kind='box', ylim=[0,250])

data['BETR801'].plot(kind='hist', bins=50)

data.plot(figsize=(12,6))


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
Som first exploration with the typical functions:
Can you write Python code for it?

data.head() # tail()

data.info()

data.describe(percentiles=[0.1, 0.5, 0.9])


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
Some of the data files that are available from AirBase were included in the data folder: the hourly concentrations of nitrogen dioxide (NO2) for 4 different measurement stations:

FR04037 (PARIS 13eme): urban background site at Square de Choisy
FR04012 (Paris, Place Victor Basch): urban traffic site at Rue d'Alesia
BETR802: urban traffic site in Antwerp, Belgium
BETN029: rural background site in Houtem, Belgium

See http://www.eea.europa.eu/themes/air/interactive/no2
Can you write Python code for it?

%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

pd.options.display.max_rows = 8


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
The above data is clearly not ready to be used! Each row contains the 24 measurements for each hour of the day, and also contains a flag (0/1) indicating the quality of the data. Furthermore, there is no header row with column names.

EXERCISE:

 Clean up this dataframe by using more options of `read_csv` (see its [docstring](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html))

specify the correct delimiter

specify that the values of -999 and -9999 should be regarded as NaN

specify are own column names (for how the column names are made up, see See http://stackoverflow.com/questions/6356041/python-intertwining-two-lists)

Can you write Python code for it?

# Column names: list consisting of 'date' and then intertwined the hour of the day and 'flag'
hours = ["{:02d}".format(i) for i in range(24)]
column_names = ['date'] + [item for pair in zip(hours, ['flag']*24) for item in pair]

# %load snippets/07 - Case study - air quality data7.py

# %load snippets/07 - Case study - air quality data8.py


I am working on the file "07 - Case study - air quality data.ipynb".
The first task is:
This does not say too much ..
We can select part of the data (eg the latest 500 data points):
Can you write Python code for it?


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
...but maybe we want to see more than a measly five results?
Can you write Python code for it?

# Let's look at MORE of the first few rows
df.head(10)


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
A dataframe is basically a spreadsheet, except it lives in the world of Python or the statistical programming language R. They can't call it a spreadsheet because then people would think those programmers used Excel, which would make them boring and normal and they'd have to wear a tie every day.
Selecting rows
Now let's look at our data, since that's what data is for
Can you write Python code for it?


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
But sometimes instead of just looking at them, I want to do stuff with them. Play some games with them! Dunk on them~ describe them! And we don't want to dunk on everyone, only the players above 7 feet tall.
First, we need to check out boolean things.
Can you write Python code for it?

# Get a big long list of True and False for every single row.
df['feet'] > 7

# We could use value counts if we wanted
above_seven_feet = df['feet'] > 7
above_seven_feet.value_counts()

# But we can also apply this to every single row to say whether YES we want it or NO we don't
df['feet'].head() > 7

# Instead of putting column names inside of the brackets, we instead
# put the True/False statements. It will only return the players above
# seven feet tall
df[df['feet'] > 7]

# Or only the guards
df[df['POS'] == 'G']

# Or only the guards who make more than 15 million
df[(df['POS'] == 'G') & (df['millions'] > 15)]

# It might be easier to break down the booleans into separate variables
is_guard = df['POS'] == 'G'
more_than_fifteen_million = df['millions'] > 15
df[is_guard & more_than_fifteen_million]

# We can save this stuff
short_players = df[df['feet'] < 6.5]
short_players

short_players.describe()

# Maybe we can compare them to taller players?
df[df['feet'] >= 6.5].describe()


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
Drawing pictures
Okay okay enough code and enough stupid numbers. I'm visual. I want graphics. Okay????? Okay.
Can you write Python code for it?

df['Age'].head()

# This will scream we don't have matplotlib.
df['Age'].hist()


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
I want more graphics! Do tall people make more money?!?!
Can you write Python code for it?

df.plot(kind='scatter', x='feet', y='millions')

df.head()

# How does experience relate with the amount of money they're making?
df.plot(kind='scatter', x='EXP', y='millions')

# At least we can assume height and weight are related
df.plot(kind='scatter', x='WT', y='feet')

# At least we can assume height and weight are related
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html
df.plot(kind='scatter', x='WT', y='feet', xlim=(100,300), ylim=(5.5, 8))

plt.style.use('ggplot')

df.plot(kind='scatter', x='WT', y='feet', xlim=(100,300), ylim=(5.5, 8))

# We can also use plt separately
# It's SIMILAR but TOTALLY DIFFERENT
centers = df[df['POS'] == 'C']
guards = df[df['POS'] == 'G']
forwards = df[df['POS'] == 'F']
plt.scatter(y=centers["feet"], x=centers["WT"], c='c', alpha=0.75, marker='x')
plt.scatter(y=guards["feet"], x=guards["WT"], c='y', alpha=0.75, marker='o')
plt.scatter(y=forwards["feet"], x=forwards["WT"], c='m', alpha=0.75, marker='v')
plt.xlim(100,300)
plt.ylim(5.5,8)


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
If we scroll we can see all of it. But maybe we don't want to see all of it. Maybe we hate scrolling?
Can you write Python code for it?


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
NOTE: That was not df['Name', 'Age'], it was df[['Name', 'Age]]. You'll definitely type it wrong all of the time. When things break with pandas it's probably because you forgot to put in a million brackets.
Describing your data
A powerful tool of pandas is being able to select a portion of your data, because who ordered all that data anyway.
Can you write Python code for it?


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
So yes, head and tail work kind of like the terminal commands. That's nice, I guess.
But maybe we're incredibly demanding (which we are) and we want, say, the 6th through the 8th row (which we do). Don't worry (which I know you were), we can do that, too.
Can you write Python code for it?

# Show the 6th through the 8th rows
df[5:8]


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
That might look better with a little more customization. So let's customize it.
Can you write Python code for it?

# Pass in all sorts of stuff!
# Most from http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html
# .range() is a matplotlib thing
df['Age'].hist(bins=20, xlabelsize=10, ylabelsize=10, range=(0,40))


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
That's cool, maybe we could do the same thing with their salary? Take out the $ and the , and convert it to an integer?
Can you write Python code for it?

# Can't just use .replace
df['2013 $'].head().replace("$","")

# Need to use this weird .str thing
df['2013 $'].head().str.replace("$","")

# Can't just immediately replace the , either
df['2013 $'].head().str.replace("$","").replace(",","")

# Need to use the .str thing before EVERY string method
df['2013 $'].head().str.replace("$","").str.replace(",","")

# Describe still doesn't work.
df['2013 $'].head().str.replace("$","").str.replace(",","").describe()

# Let's convert it to an integer using .astype(int) before we describe it
df['2013 $'].head().str.replace("$","").str.replace(",","").astype(int).describe()

df['2013 $'].head().str.replace("$","").str.replace(",","").astype(int)

# Maybe we can just make them millions?
df['2013 $'].head().str.replace("$","").str.replace(",","").astype(int) / 1000000

# Unfortunately one is "n/a" which is going to break our code, so we can make n/a be 0
df['2013 $'].str.replace("$","").str.replace(",","").str.replace("n/a", "0").astype(int) / 1000000

# Remove the .head() piece and save it back into the dataframe
df['millions'] = df['2013 $'].str.replace("$","").str.replace(",","").str.replace("n/a","0").astype(int) / 1000000
df.head()

df.describe()


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
The average basketball player makes 3.8 million dollars and is a little over six and a half feet tall.
But who cares about those guys? I don't care about those guys. They're boring. I want the real rich guys!
Sorting and sub-selecting
Can you write Python code for it?

# This is just the first few guys in the dataset. Can we order it?
df.head(3)

# Let's try to sort them
df.sort_values(by='millions').head(3)


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
Those guys are making nothing! If only there were a way to sort from high to low, a.k.a. descending instead of ascending.
Can you write Python code for it?

# It isn't descending = True, unfortunately
df.sort_values(by='millions', ascending=False).head(3)

# We can use this to find the oldest guys in the league
df.sort_values(by='Age', ascending=False).head(3)

# Or the youngest, by taking out 'ascending=False'
df.sort_values(by='Age').head(3)


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
Unfortunately because that has dollar signs and commas it's thought of as a string. We'll fix it in a second, but let's try describing one more thing.
Can you write Python code for it?

# Doing more describing
df['Ht (In.)'].describe()


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
When you import pandas, you use import pandas as pd. That means instead of typing pandas in your code you'll type pd.
You don't have to, but every other person on the planet will be doing it, so you might as well.
Now we're going to read in a file. Our file is called NBA-Census-10.14.2013.csv because we're sports moguls. pandas can read_ different types of files, so try to figure it out by typing pd.read_ and hitting tab for autocomplete.
Can you write Python code for it?

# We're going to call this df, which means "data frame"
# It isn't in UTF-8 (I saved it from my mac!) so we need to set the encoding
df = pd.read_csv("NBA-Census-10.14.2013.csv", encoding='mac_roman')


I am working on the file "07 - Introduction to Pandas (complete).ipynb".
The first task is:
matplotlib is a graphing library. It's the Python way to make graphs!
Can you write Python code for it?

!pip install matplotlib

# this will open up a weird window that won't do anything
df['Age'].hist()

# So instead you run this code
%matplotlib inline

df['Age'].hist()


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
...but maybe we want to see more than a measly five results?
Can you write Python code for it?

# Let's look at MORE of the first few rows
df.head(10) # shows the first 10 lines of the program


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
But maybe we want to make a basketball joke and see the final four?
Can you write Python code for it?

# Let's look at the final few rows

df.tail(4) # shows the final four


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
But sometimes instead of just looking at them, I want to do stuff with them. Play some games with them! Dunk on them~ describe them! And we don't want to dunk on everyone, only the players above 7 feet tall.
First, we need to check out boolean things.
Can you write Python code for it?

# Get a big long list of True and False for every single row.
#shows you who is/isn't above 6"5 ft.
above_or_below_six_five = df['Ht (Ft.)'] > 6
# print(above_or_below_six_five)

# We could use value counts if we wanted
above_or_below_six_five.value_counts() # returns how many players are or are not above 6"5

# But we can also apply this to every single row to say whether YES we want it or NO we don't
above_or_below_six_five = df['Ht (Ft.)'] > 7

df[df['Race'] == 'Asian']

# Instead of putting column names inside of the brackets, we instead
# put the True/False statements. It will only return the players above
# seven feet tall

# Or only the guards
df[df['Ht (Ft.)'] > 7]

# Or only the guards who are under 6 feet tall

# are you a guard? AND are below 6 feet tall?
df[(df['POS'] == 'G') & (df['Ht (Ft.)'] < 6)]

# It might be easier to break down the booleans into separate variables
is_a_guard =  df['POS'] == 'G'
is_below_six_feet = df['Ht (Ft.)'] < 6
df[is_a_guard & is_below_six_feet]

centers = df[df['POS'] == 'C']
guards = df[df['POS'] == 'G']

# We can save this stuff

centers['Ht (Ft.)'].describe()

guards['Ht (Ft.)'].describe()

# Maybe we can compare them to taller players?


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
Drawing pictures
Okay okay enough code and enough stupid numbers. I'm visual. I want graphics. Okay????? Okay.
Can you write Python code for it?

!pip install matplotlib

import matplotlib.pyplot as plt
%matplotlib inline
# This will scream we don't have matplotlib.
df['Ht (Ft.)'].hist()


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
I want more graphics! Do tall people make more money?!?!
Can you write Python code for it?

# How does experience relate with the amount of money they're making?

# At least we can assume height and weight are related

# At least we can assume height and weight are related
# http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html

# We can also use plt separately
# It's SIMILAR but TOTALLY DIFFERENT


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
If we scroll we can see all of it. But maybe we don't want to see all of it. Maybe we hate scrolling?
Can you write Python code for it?

# Look at the first few rows

df.head() # shows header + first 5 rows!


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
It's kind of like an array, right? Except where in an array we'd say df[0] this time we need to give it two numbers, the start and the end.
Selecting columns
But jeez, my eyes don't want to go that far over the data. I only want to see, uh, name and age.
Can you write Python code for it?

# Get the names of the columns, just because

df.columns #prints out the name of the columns - casing must match actually

# If we want to be "correct" we add .values on the end of it
df.columns.values

# Select only name and age

columns_we_want = ['Name', 'Age']

#passing the list of colums we want to data frame
df[columns_we_want]

# Combing that with .head() to see not-so-many rows

# We can also do this all in one line, even though it starts looking ugly
# (unlike the cute bears pandas looks ugly pretty often)

df[['Name', 'Age']] # brackets brackets


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
NOTE: That was not df['Name', 'Age'], it was df[['Name', 'Age']]. You'll definitely type it wrong all of the time. When things break with pandas it's probably because you forgot to put in a million brackets.
Describing your data
A powerful tool of pandas is being able to select a portion of your data, because who ordered all that data anyway.
Can you write Python code for it?


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
Now that was a little weird, yes - we used df['POS'] instead of df[['POS']] when viewing the data's details.
But now I'm curious about numbers: how old is everyone? Maybe we could, I don't know, get some statistics about age? Some statistics to describe age?
Can you write Python code for it?

# Summary statistics for Age
df['Age'].value_counts() # statistics about age

df['Age'].describe() #statistics about NBA players and their ages

# That's pretty good. Does it work for everything? How about the money?
df.describe() # shows info for all of the numerical data

# EEEK minum weight = 20 lbs -- seems incorrect

df['Ht (In.)'].describe()

#df.columns # look at column names again

df['2013 $'].describe() # this column is string as opposed to int -- therefore it didn't work :()


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
Okay that was nice but unfortunately we can't do anything with it. It's just sitting there, separate from our data. If this were normal code we could do blahblah['feet'] = blahblah['Ht (In.)'] / 12, but since this is pandas, we can't. Right? Right?
Can you write Python code for it?

# Store a new column
df['Ht (Ft.)'] = df['Ht (In.)']/12 # adds a new column with the height as feet

df.head()

df.sort_values('Ht (Ft.)') # automatically sorts from lowest to highest - ascending value

#shows the tallest players by height in feet
df.sort_values('Ht (Ft.)', ascending =False).head() # automatically sorts from lowest to highest - ascending value

#shows you who is/isn't above 6"5 ft.
above_or_below_six_five = df['Ht (Ft.)'] > 6
above_or_below_six_five.value_counts() # returns how many players are or are not above 6"5


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
So yes, head and tail work kind of like the terminal commands. That's nice, I guess.
But maybe we're incredibly demanding (which we are) and we want, say, the 6th through the 8th row (which we do). Don't worry (which I know you were), we can do that, too.
Can you write Python code for it?

# Show the 6th through the 8th rows

df[6:9]


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
That's cool, maybe we could do the same thing with their salary? Take out the $ and the , and convert it to an integer?
Can you write Python code for it?

# Can't just use .replace

# Need to use this weird .str thing

# Can't just immediately replace the , either

# Need to use the .str thing before EVERY string method

# Describe still doesn't work.

# Let's convert it to an integer using .astype(int) before we describe it

# Maybe we can just make them millions?

# Unfortunately one is "n/a" which is going to break our code, so we can make n/a be 0

# Remove the .head() piece and save it back into the dataframe


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
That's stupid, though, what's an inch even look like? What's 80 inches? I don't have a clue. If only there were some wa to manipulate our data.
Manipulating data
Oh wait there is, HA HA HA.
Can you write Python code for it?

# Take another look at our inches, but only the first few

df['Ht (In.)'].head()

# Divide those inches by 12
df['Ht (In.)'].head()/12 #divides every single value by 12

# Let's divide ALL of them by 12
df['Ht (In.)']/12

# Can we get statistics on those?
height_in_feet = df['Ht (In.)']/12
height_in_feet.describe()

# Let's look at our original data again
df.head()


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
The average basketball player makes 3.8 million dollars and is a little over six and a half feet tall.
But who cares about those guys? I don't care about those guys. They're boring. I want the real rich guys!
Sorting and sub-selecting
Can you write Python code for it?

# This is just the first few guys in the dataset. Can we order it?

# Let's try to sort them


I am working on the file "07 - Introduction to Pandas.ipynb".
The first task is:
Unfortunately because that has dollar signs and commas it's thought of as a string. We'll fix it in a second, but let's try describing one more thing.
Can you write Python code for it?


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
Coefficient of Determination ($R^2$)
The coefficient of determination is a measure of how closely one group of samples (i.e. measured) follows another (i.e. a model). This is accomplished via the proportion of total variation of outcomes explained by the model.
Can you write Python code for it?

from scipy.optimize import curve_fit

# Create arbitrary function
x_vals = np.arange(0, 100)
y_vals = x_vals**2 + normal(0, 3000, np.size(x_vals))

# Fit and create fit line
def func(x_vals, B, C):
    return x_vals**B + C
opt, cov = curve_fit(func, x_vals, y_vals)
x_fitted = np.linspace(0, max(x_vals), 100)
y_fitted = func(x_fitted, *opt)

# Show fit
plt.scatter(x_vals, y_vals)
plt.plot(x_fitted, y_fitted, color='red')
plt.show()


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
Confidence Intervals from a Distribution Perspective
This is where the critical probability (t-value) for the confidence interval comes from. Working backwards, we're interested the value of the probability distribution function that, when sampled, encompasses $\frac{1}{2}$ of the confidence interval (i.e. 95%) area under the function.
To do so, 1/2 of the confidence interval on each side of the distribution is removed, and the corresponding t-value is determined.
Can you write Python code for it?

conf_int = 0.95 # Use a specified confidence interval
                #    (i.e. % of total CDF area)

# We use the t-distribution in lieu of the normal distribution because
#    the samples are a subset of the population the inverse of the CDF
#    for the normal distribution is known as the percent point function
t_value = stats.t.ppf(1-(1-conf_int)/2,dof)

# Use CDF to check that probabilities are correct
left_prob    = stats.t.cdf(-t_value, dof)
right_prob   = stats.t.sf(t_value, dof) # The survival function is 1-CDF
between_prob = 1-(left_prob+right_prob)

# Plot T distribution, highlighting the different plot areas
left_ind    = X <= -t_value
right_ind   = X >= t_value
between_ind = (X > -t_value) & ( X < t_value)

plt.fill_between(X[left_ind],stats.t.pdf(X[left_ind],dof), facecolor='deepskyblue')
plt.fill_between(X[right_ind],stats.t.pdf(X[right_ind],dof), facecolor='deepskyblue')
plt.fill_between(X[between_ind],stats.t.pdf(X[between_ind],dof),facecolor='red')

# Label the plot areas
plt.text(x=2.2*t,y=0.04, s='%0.3g' % left_prob)
plt.text(x=-0.4,y=0.1,s='%0.3g' % between_prob)
plt.text(x=1.6*-t,y=0.04, s='%0.3g' % right_prob)
plt.show()

print("The t_value that can be used in the 95 percent confidence interval is: %0.6g" % T_val)


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
Distributions
Distribution functions can be thought of as the probability of measuring a particular sample value. To get a better picture of how frequently a type of randomly distributed variable should be measured in theory, we can use the analytical distribution function.
For example, perhaps the most well known distribution is the Gaussian, Normal, or Bell-Curve distribution. This is determined from a set of gaussian distributed random numbers. We can generate a lot of these numbers and plot the frequency of each number within a set of 'bins':
Can you write Python code for it?

num_samples = 10000
span = 10 # How wide to plot and bin
rand_gen = normal(0, 1, num_samples)        # Generate num_sample numbers
bins = np.linspace(-span/2, span/2, num=100)
histogram = np.histogram(rand_gen, bins);  # Use histogram to get the
                                          # distribution

X = histogram[1][:-1]
Y = histogram[0]
plt.scatter(X,Y, label="Sample Frequency")
plt.legend(loc='best')
plt.title('Scatter Plot')
plt.show()


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
Equal sample sizes, equal variances
$t = \frac{\bar{X}1 - \bar{X}_2}{s{pool}\sqrt{\frac{2}{n}}}$
where:
$s_{pool}$ = pooled variance
$s_{pool} = \sqrt{\frac{s_1^2 + s_2^2}{2}}$
$n$ is the number of samples
$\bar{X}$ is the expectation value (if equal weights, average)
$s_1^2 = \frac{1}{n-1} \sum^{n}_{1} (x_i-\bar{X})^2$
Equal or unequal sample sizes, equal variances
$t = \frac{\bar{X}1 - \bar{X}_2}{s{pool}\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}}$
where:
$s_{pool} = \sqrt{\frac{(n_1 -1) s_1^2 + (n_2 -1) s_2^2}{n_1 + n_2 -1}}$
Equal or unequal sample sizes, unequal variances
$t = \frac{\bar{X}1 - \bar{X}_2}{s\delta}$
where:
$s_\delta = \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}$
The p-values can be calculated by integrating the Student's t-distribution cumulative density fuction (CDF) directly - more information is provided in the Distributions section. Alternatively, tables of precalculated CDF values (Z-tables) may be used when calculating CDF isn't practical but aren't discussed here. Below, the survival function (stats.t(degFreedom).sf, or '1 - CDF') is used to sample the t-distribution.
Can you write Python code for it?

# Get some initial info about the three groups
grp1_siz = float(grp1.size)
grp1_dof = grp1_siz - 1
grp1_avg = np.sum(grp1)/grp1_siz
grp1_var = 1/(grp1_dof)* np.sum((grp1 - grp1_avg)**2)

grp2_siz = float(grp2.size)
grp2_dof = grp2_siz - 1
grp2_avg = np.sum(grp2)/grp2_siz
grp2_var = 1/(grp2_dof)* np.sum((grp2 - grp2_avg)**2)

grp3_siz = float(np.size(grp3))
grp3_avg = np.sum(grp3)/grp3_siz
grp3_dof = grp3_siz - 1
grp3_var = 1/(grp3_dof)* np.sum((grp3 - grp3_avg)**2)


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
For those that are wondering, the CDF is actually less than the PDF at a certain point because both scaled but in different ways - the CDF is scaled such that it's maximum value is one but the PDF is such that the area beneath it is one. For more information, look into the integral of the PDF.
We use the CDF to determine the percent chance of obtaining something at least as large, i.e. what is the percent chance of getting at least 2?
Can you write Python code for it?


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
If the p-value is smaller than some threshold (i.e. 0.01 or 0.05, etc.) then the null hypothesis can be rejected. The two groups of samples have different means! Note that the first and second tests converge as sampling approaches infinity.
The goal of the Student's t-test is to determine if two groups of samples are from the same population or different populations. This comes up frequently when we want to determine if the data we're collected somehow differs from another data set (i.e. we've observed something change (before/after populations), or observe something different from what someone else claims).
To do this, first calculate a t-value, and use this t-value (to sample the t-distribution) to determine a measure of how similar the two groups of samples are. This is known as a p-value.
The p-value represents the probability that the difference between the groups of samples is observed purely by chance. A p-value below some threshold (i.e. 0.05) means there is a significant difference between the groups of samples.
Increasing t-values (increasingly different groups) lead to p-value decreases (decreasing chances that the samples are from the same distribution).
Often, this is described in terms of the null hypothesis, or that there is 'null difference' between the two groups of samples. In other words, can the null hypothesis (there is no difference between the two populations) be rejected? The goal is to determine if any difference is due to sampling, experimental, etc. error or if the means really are different.
This is intended for normally distributed, continuous distributions.
Fun fact: the 'student' is actually William S. Gossett, a brewmaster who worked at the Guinness brewery.
>2 Groups: One-way ANOVA F-test statistic
The F-test can be thought of as the generalized form of the t-test for multiple groups of samples. A popular use of the F-test is to determine if one group of samples is from a different population than all other groups (i.e. one vs many), or if all are from the same...
Can you write Python code for it?


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
Let's say our t-value is: -1.47254.
A note about two-tailed tests: We're interested if we can reject the null hypothesis, or if the populations are the same. The t-value can be positive or negative (depending on the two means) but we're only interested only if it is different (not only + or only - difference, but both). To determine the p-value, we sample from both sides of the distribution, and this is known as a two-tailed test.
For the p-value of the t-test (two-tailed), we're concerned with getting the the value under both sides of the distribution. Here we're ignoring the sign of the t-value and treating it as a negative value for a zero-mean t-distribution probability distribution function.
For the two-tailed p-value on a t-test:
Can you write Python code for it?

t = -1.47254 # Use a specified t-value

# Use CDF to determine probabilities
left_prob    = stats.t(dof).cdf(-np.abs(t))
right_prob   = stats.t(dof).sf(np.abs(t)) # The survival function is 1-CDF
between_prob = 1-(left_prob+right_prob)

# Plot t-distribution, highlighting the different plot areas
left_ind    = X <= -np.abs(t)
right_ind   = X >= np.abs(t)
between_ind = (X > -np.abs(t)) & ( X < np.abs(t))

plt.fill_between(X[left_ind],stats.t(dof).pdf(X[left_ind]), facecolor='red')
plt.fill_between(X[right_ind],stats.t(dof).pdf(X[right_ind]), facecolor='red')
plt.fill_between(X[between_ind],stats.t(dof).pdf(X[between_ind]),facecolor='deepskyblue')

# Label the plot areas
plt.text(x=1.7*t,y=0.04, s='%0.3g' % left_prob)
plt.text(x=-0.4,y=0.1,s='%0.3g' % between_prob)
plt.text(x=1.1*-t,y=0.04, s='%0.3g' % right_prob)
plt.show()


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
Mean, Standard Deviation, and Variance
These all give an initial sense of the distribution of a group of samples - the average value (mean) and how spread out they are (standard deviation and variance).
Keep in mind that the variance is the square of the standard deviation.
Can you write Python code for it?

# Generate some continuous data:
nums = normal(2, 3, 1000) # function of mean (mu), std (sigma), and size (n)


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
Mean:
$\mu = \frac{1}{n}\sum_{i=1}^{n} x_i$
Standard deviation
Root of the average squared deviation from the mean (entire population):
$\sigma = \sqrt{\frac{1}{n}\sum_{i=1}^{n} (x_i-\mu)^2}$
Root of the average squared deviation from the mean (subset of population):
$\sigma = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n} (x_i-\mu)^2}$
Variance
Average squared deviation from mean:
$\sigma^2 = \frac{1}{n}\sum_{i=1}^{n} (x_i-\mu)^2$
We can do these calculations manually:
Can you write Python code for it?

mean = (1/len(nums))*np.sum(nums)
print('The mean is: %g' % mean)

stdev = np.sqrt(1/len(nums) * np.sum((nums - mean)**2))
print('The standard deviation (all samples, or the population) is: %g' % stdev)

stdev = np.sqrt(1/(len(nums)-1) * np.sum((nums - mean)**2))
print('The unbiased standard deviation (a group of samples, or a subset of the population) is: %g' % stdev)

var = (1/len(nums)) * np.sum((nums - mean)**2)
print('The variance is: %g' % var)


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
PDF of Student's t-distribution
This doesn't seem that interesting until we consider the distributions used in the ANOVA analysis above. Here we use the Student's t-distribution, which is extremely similar to the normal distribution except that it incorporates the fact that we often use a subset of the population (and thus degrees of freedom = n-1):
Can you write Python code for it?

dof = np.size(X) - 1
Y_t_pdf = stats.t(dof).pdf(X)
Y_t_cdf = stats.t(dof).cdf(X)

plt.plot(X, Y_t_pdf, color='red', label="Prob. Dist. Func.")
plt.plot(X, Y_t_cdf, color='blue', label="Cumul. Dist. Func.")
plt.legend(loc='best')
plt.show()


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
The p-value between groups 1 and 2 is still greater than our value of 0.05, so we cannot reject the null hypothesis, and assume these are from the same population.
Can you write Python code for it?

# Equal or unequal sample size, assumed unequal variance:
var = np.sqrt( grp1_var/grp1_siz + grp3_var/grp3_siz )
t = (grp1_avg - grp3_avg)/var

# Calculate p-value:
degFreedom = (grp1_var/grp1_siz + grp3_var/grp3_siz)**2/ \
    ((grp1_var/grp1_siz)**2/grp1_dof + (grp3_var/grp3_siz)**2/grp3_dof)
p = 2*stats.t(degFreedom).sf(np.abs(t))
print(" t = {0:g} p = {1:g}".format(t, p))


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
The probability distribution function (PDF) is a function that represents the probability of obtaining a particular value for a population that follows that particular distribution.
Using a conversion factor, it's clear that the two overlap:
Can you write Python code for it?

Y_pdf = stats.norm.pdf(X)                 # PDF function applied to the
                                          # X values
conv_factor = len(X)/(float(span) * float(num_samples))
                                          # Use a normalization factor to
                                          # demonstrate the two are
                                          # overlapped
plt.scatter(X,Y*conv_factor, label="Sample Frequency")
plt.plot(X,Y_pdf, color='red', label="Prob. Dist. Func.")
plt.legend(loc='best')
plt.show()


I am working on the file "07 - Some Basic Statistics.ipynb".
The first task is:
The two-tailed test (i.e. including both sides) is simply 2x the value of the area of one of the sides:
Can you write Python code for it?

p = 2*stats.t(dof).sf(np.abs(t))
print("%g" % p)


I am working on the file "07 Introduction to Functions.ipynb".
The first task is:
Functions take repeated code, put it in one place, and then you call that code when you want to use it. Here's what the same program looks like with a function.
Can you write Python code for it?

def thank_you(name):
    # This function prints a two-line personalized thank you message.
    print("\nYou are doing good work, %s!" % name)
    print("Thank you very much for your efforts on this project.")

thank_you('Adriana')
thank_you('Billy')
thank_you('Caroline')


I am working on the file "07 Introduction to Functions.ipynb".
The first task is:
If you use a return statement in one of your functions, keep in mind that the function stops executing as soon as it hits a return statement. For example, we can add a line to the get_number_word() function that will never execute, because it comes after the function has returned a value:
Can you write Python code for it?

def get_number_word(number):
    # Takes in a numerical value, and returns
    #  the word corresponding to that number.
    if number == 0:
        return 'zero'
    elif number == 1:
        return 'one'
    elif number == 2:
        return 'two'
    elif number == 3:
        return 'three'
    else:
        return "I'm sorry, I don't know that number."

    # This line will never execute, because the function has already
    #  returned a value and stopped executing.
    print("This message will never be printed.")

# Let's try out our function.
for current_number in range(0,6):
    number_word = get_number_word(current_number)
    print(current_number, number_word)


I am working on the file "07 Introduction to Functions.ipynb".
The first task is:
More Later
There is much more to learn about functions, but we will get to those details later. For now, feel free to use functions whenever you find yourself writing the same code several times in a program. Some of the things you will learn when we focus on functions:

How to give the arguments in your function default values.
How to let your functions accept different numbers of arguments.

top
Exercises
Greeter

Write a function that takes in a person's name, and prints out a greeting.
The greeting must be at least three lines, and the person's name must be in each line.

Use your function to greet at least three different people.
Bonus: Store your three people in a list, and call your function from a for loop.

Full Names

Write a function that takes in a first name and a last name, and prints out a nicely formatted full name, in a sentence. Your sentence could be as simple as, "Hello, full_name."
Call your function three times, with a different name each time.

Addition Calculator

Write a function that takes in two numbers, and adds them together. Make your function print out a sentence showing the two numbers, and the result.
Call your function with three different sets of numbers.

Return Calculator

Modify Addition Calculator so that your function returns the sum of the two numbers. The printing should happen outside of the function.

List Exercises - Functions

Go back and solve each of the following exercises from the section on Lists and Tuples, using functions to avoid repetetive code:
Ordered Working List
Ordered Numbers
Can you write Python code for it?

# Ex 4.1 : Greeter

# put your code here

# Ex 4.2 : Full Names

# put your code here

# Ex 4.3 : Addition Calculator

# put your code here

# Ex 4.4 : Return Calculator

# put your code here

# Ex 4.5 : List Exercises - Functions

# put your code here


I am working on the file "07 Introduction to Functions.ipynb".
The first task is:
This code will not run, but it shows how functions are used in general.

Defining a function

Give the keyword def, which tells Python that you are about to define a function.
Give your function a name. A variable name tells you what kind of value the variable contains; a function name should tell you what the function does.
Give names for each value the function needs in order to do its work.
These are basically variable names, but they are only used in the function.
They can be different names than what you use in the rest of your program.
These are called the function's arguments.

Make sure the function definition line ends with a colon.
Inside the function, write whatever code you need to make the function do its work.

Using your function

To call your function, write its name followed by parentheses.
Inside the parentheses, give the values you want the function to work with.
These can be variables such as current_name and current_age, or they can be actual values such as 'eric' and 5.

top
Basic Examples
For a simple first example, we will look at a program that compliments people. Let's look at the example, and then try to understand the code. First we will look at a version of this program as we would have written it earlier, with no functions.
Can you write Python code for it?

print("You are doing good work, Adriana!")
print("Thank you very much for your efforts on this project.")

print("\nYou are doing good work, Billy!")
print("Thank you very much for your efforts on this project.")

print("\nYou are doing good work, Caroline!")
print("Thank you very much for your efforts on this project.")


I am working on the file "07 Introduction to Functions.ipynb".
The first task is:
This is much cleaner code. We have an action we want to take, which is to show the students in our list along with a message. We give this action a name, show_students().
This function needs two pieces of information to do its work, the list of students and a message to display. Inside the function, the code for printing the message and looping through the list is exactly as it was in the non-function code.
Now the rest of our program is cleaner, because it gets to focus on the things we are changing in the list, rather than having code for printing the list. We define the list, then we sort it and call our function to print the list. We sort it again, and then call the printing function a second time, with a different message. This is much more readable code.
Advantages of using functions
You might be able to see some advantages of using functions, through this example:

We write a set of instructions once. We save some work in this simple example, and we save even more work in larger programs.

When our function works, we don't have to worry about that code anymore. Every time you repeat code in your program, you introduce an opportunity to make a mistake. Writing a function means there is one place to fix mistakes, and when those bugs are fixed, we can be confident that this function will continue to work correctly.

We can modify our function's behavior, and that change takes effect every time the function is called. This is much better than deciding we need some new behavior, and then having to change code in many different places in our program.

For a quick example, let's say we decide our printed output would look better with some form of a bulleted list. Without functions, we'd have to change each print statement. With a function, we change just the print statement in the function:
Can you write Python code for it?

def show_students(students, message):
    # Print out a message, and then the list of students
    print(message)
    for student in students:
        print("- " + student.title())

students = ['bernice', 'aaron', 'cody']

# Put students in alphabetical order.
students.sort()
show_students(students, "Our students are currently in alphabetical order.")

#Put students in reverse alphabetical order.
students.sort(reverse=True)
show_students(students, "\nOur students are now in reverse alphabetical order.")


I am working on the file "07 Introduction to Functions.ipynb".
The first task is:
top
Challenges
Lyrics

Many songs follow a familiar variation of the pattern of verse, refrain, verse, refrain. The verses are the parts of the song that tell a story - they are not repeated in the song. The refrain is the part of the song that is repeated throughout the song.
Find the lyrics to a song you like that follows this pattern. Write a program that prints out the lyrics to this song, using as few lines of Python code as possible. hint
Can you write Python code for it?

# Challenge: Lyrics

# Put your code here


I am working on the file "07 More Functions.ipynb".
The first task is:
Earlier we created a function that let us describe a person, and we had three things we could describe about a person. We could include their age, their favorite language, and the date they passed away. But that was the only information we could include, because it was the only information that the function was prepared to handle:
Can you write Python code for it?

def describe_person(first_name, last_name, age=None, favorite_language=None, died=None):
    # This function takes in a person's first and last name,
    #  their age, and their favorite language.
    # It then prints this information out in a simple format.

    # Required information:
    print("First name: %s" % first_name.title())
    print("Last name: %s" % last_name.title())

    # Optional information:
    if age:
        print("Age: %d" % age)
    if favorite_language:
        print("Favorite language: %s" % favorite_language)
    if died:
        print("Died: %d" % died)

    # Blank line at end.
    print("\n")

describe_person('brian', 'kernighan', favorite_language='C')
describe_person('ken', 'thompson', age=70)
describe_person('adele', 'goldberg', age=68, favorite_language='Smalltalk')
describe_person('dennis', 'ritchie', favorite_language='C', died=2011)
describe_person('guido', 'van rossum', favorite_language='Python')


I am working on the file "07 More Functions.ipynb".
The first task is:
Everyone needs a first and last name, but everthing else is optional. This code takes advantage of the Python keyword None, which acts as an empty value for a variable. This way, the user is free to supply any of the 'extra' values they care to. Any arguments that don't receive a value are not displayed. Python matches these extra values by name, rather than by position. This is a very common and useful way to define functions.
top
Exercises
Sports Teams

Write a function that takes in two arguments, the name of a city and the name of a sports team from that city.
Call your function three times, using a mix of positional and keyword arguments.

World Languages

Write a function that takes in two arguments, the name of a country and a major language spoken there.
Call your function three times, using a mix of positional and keyword arguments.
Can you write Python code for it?

# Ex 8.5 : Sports Team

# put your code here

# Ex 8.6 : Word Languages

# put your code here


I am working on the file "07 More Functions.ipynb".
The first task is:
That makes sense; the function needs to have a name in order to do its work, so without a name it is stuck.
If you want your function to do something by default, even if no information is passed to it, you can do so by giving your arguments default values. You do this by specifying the default values when you define the function:
Can you write Python code for it?

def thank_you(name='everyone'):
    # This function prints a two-line personalized thank you message.
    #  If no name is passed in, it prints a general thank you message
    #  to everyone.
    print("\nYou are doing good work, %s!" % name)
    print("Thank you very much for your efforts on this project.")

thank_you('Billy')
thank_you('Caroline')
thank_you()


I am working on the file "07 More Functions.ipynb".
The first task is:
The arguments in this function are first_name, last_name, and age. These are called positional arguments because Python knows which value to assign to each by the order in which you give the function values. In the calling line
describe_person('brian', 'kernighan', 71)

we send the values brian, kernighan, and 71 to the function. Python matches the first value brian with the first argument first_name. It matches the second value kernighan with the second argument last_name. Finally it matches the third value 71 with the third argument age.
This is pretty straightforward, but it means we have to make sure to get the arguments in the right order.
If we mess up the order, we get nonsense results or an error:
Can you write Python code for it?

def describe_person(first_name, last_name, age):
    # This function takes in a person's first and last name,
    #  and their age.
    # It then prints this information out in a simple format.
    print("First name: %s" % first_name.title())
    print("Last name: %s" % last_name.title())
    print("Age: %d\n" % age)

describe_person(71, 'brian', 'kernighan')
describe_person(70, 'ken', 'thompson')
describe_person(68, 'adele', 'goldberg')


I am working on the file "07 More Functions.ipynb".
The first task is:
The third argument has two asterisks in front of it, which tells Python to collect all remaining key-value arguments in the calling statement. This argument is commonly named kwargs. We see in the output that these key-values are stored in a dictionary. We can loop through this dictionary to work with all of the values that are passed into the function:
Can you write Python code for it?

def example_function(arg_1, arg_2, **kwargs):
    # Let's look at the argument values.
    print('\narg_1:', arg_1)
    print('arg_2:', arg_2)
    for key, value in kwargs.items():
        print('arg_3 value:', value)

example_function('a', 'b')
example_function('a', 'b', value_3='c')
example_function('a', 'b', value_3='c', value_4='d')
example_function('a', 'b', value_3='c', value_4='d', value_5='e')

def example_function(**kwargs):
    print(type(kwargs))
    for key, value in kwargs.items():
        print('{}:{}'.format(key, value))

example_function(first=1, second=2, third=3)
example_function(first=1, second=2, third=3, fourth=4)
example_function(name='Valerio', surname='Maggio')


I am working on the file "07 More Functions.ipynb".
The first task is:
This fails because Python tries to match the value 71 with the argument first_name, the value brian with the argument last_name, and the value kernighan with the argument age. Then when it tries to print the value first_name.title(), it realizes it can't use the title() method on an integer.
top
Exercises
Favorite Colors

Write a function that takes two arguments, a person's name and their favorite color. The function should print out a statement such as "Hillary's favorite color is blue."
Call your function three times, with a different person and color each time.

Phones

Write a function that takes two arguments, a brand of phone and a model name. The function should print out a phrase such as "iPhone 6 Plus".
Call your function three times, with a different combination of brand and model each time.
Can you write Python code for it?

# Ex 8.3 : Favorite Colors

# put your code here

# Ex 8.4 : Phones

# put your code here


I am working on the file "07 More Functions.ipynb".
The first task is:
This function appears to work well. But what if we pass it three numbers, which is a perfectly reasonable thing to do mathematically?
Can you write Python code for it?

def adder(num_1, num_2):
    # This function adds two numbers together, and prints the sum.
    sum = num_1 + num_2
    print("The sum of your numbers is %d." % sum)

# Let's add some numbers.
adder(1, 2, 3)


I am working on the file "07 More Functions.ipynb".
The first task is:
This is pretty neat. We get the same output, and we don't have to include a bunch of if tests to see what kind of information was passed into the function. We always require a first name and a last name, but beyond that the caller is free to provide any keyword-value pair to describe a person. Let's show that any kind of information can be provided to this function. We also clean up the output by replacing any underscores in the keys with a space.
Can you write Python code for it?

def describe_person(first_name, last_name, **kwargs):
    # This function takes in a person's first and last name,
    #  and then an arbitrary number of keyword arguments.

    # Required information:
    print("First name: %s" % first_name.title())
    print("Last name: %s" % last_name.title())

    # Optional information:
    for key in kwargs:
        print("%s: %s" % (key.title().replace('_', ' '), kwargs[key]))

    # Blank line at end.
    print("\n")

describe_person('brian', 'kernighan', favorite_language='C', famous_book='The C Programming Language')
describe_person('ken', 'thompson', age=70, alma_mater='UC Berkeley')
describe_person('adele', 'goldberg', age=68, favorite_language='Smalltalk')
describe_person('dennis', 'ritchie', favorite_language='C', died=2011, famous_book='The C Programming Language')
describe_person('guido', 'van rossum', favorite_language='Python', company='Dropbox')


I am working on the file "07 More Functions.ipynb".
The first task is:
This works, because Python does not have to match values to arguments by position. It matches the value 71 with the argument age, because the value 71 is clearly marked to go with that argument. This syntax is a little more typing, but it makes for very readable code.
Mixing positional and keyword arguments
It can make good sense sometimes to mix positional and keyword arguments. In our previous example, we can expect this function to always take in a first name and a last name. Before we start mixing positional and keyword arguments, let's add another piece of information to our description of a person. Let's also go back to using just positional arguments for a moment:
Can you write Python code for it?

def describe_person(first_name, last_name, age, favorite_language):
    # This function takes in a person's first and last name,
    #  their age, and their favorite language.
    # It then prints this information out in a simple format.
    print("First name: %s" % first_name.title())
    print("Last name: %s" % last_name.title())
    print("Age: %d" % age)
    print("Favorite language: %s\n" % favorite_language)

describe_person('brian', 'kernighan', 71, 'C')
describe_person('ken', 'thompson', 70, 'Go')
describe_person('adele', 'goldberg', 68, 'Smalltalk')


I am working on the file "07 More Functions.ipynb".
The first task is:
We can expect anyone who uses this function to supply a first name and a last name, in that order. But now we are starting to include some information that might not apply to everyone. We can address this by keeping positional arguments for the first name and last name, but expect keyword arguments for everything else. We can show this works by adding a few more people, and having different information about each person:
Can you write Python code for it?

def describe_person(first_name, last_name, age=None, favorite_language=None, died=None):
    """
    This function takes in a person's first and last name, their age,
    and their favorite language.
    It then prints this information out in a simple format.
    """

    print("First name: %s" % first_name.title())
    print("Last name: %s" % last_name.title())

    # Optional information:
    if age:
        print("Age: %d" % age)
    if favorite_language:
        print("Favorite language: %s" % favorite_language)
    if died:
        print("Died: %d" % died)
    # Blank line at end.
    print("\n")

describe_person('brian', 'kernighan', favorite_language='C')
describe_person('adele', 'goldberg', age=68, favorite_language='Smalltalk')
describe_person('dennis', 'ritchie', favorite_language='C', died=2011)
describe_person('guido', 'van rossum', favorite_language='Python')


I am working on the file "07 More Functions.ipynb".
The first task is:
We can make this function much more flexible by accepting any number of keyword arguments. Here is what the function looks like, using the syntax for accepting as many keyword arguments as the caller wants to provide:
Can you write Python code for it?

def describe_person(first_name, last_name, **kwargs):
    # This function takes in a person's first and last name,
    #  and then an arbitrary number of keyword arguments.

    # Required information:
    print("First name: %s" % first_name.title())
    print("Last name: %s" % last_name.title())

    # Optional information:
    for key in kwargs:
        print("%s: %s" % (key.title(), kwargs[key]))

    # Blank line at end.
    print("\n")

describe_person('brian', 'kernighan', favorite_language='C')
describe_person('ken', 'thompson', age=70)
describe_person('adele', 'goldberg', age=68, favorite_language='Smalltalk')
describe_person('dennis', 'ritchie', favorite_language='C', died=2011)
describe_person('guido', 'van rossum', favorite_language='Python')


I am working on the file "07 More Functions.ipynb".
The first task is:
We can now rewrite the adder() function to accept two or more arguments, and print the sum of those numbers:
Can you write Python code for it?

def adder(*nums):
    """This function adds the given numbers together and prints the sum."""

    s = 0
    for num in nums:
        s = s + num
    # Print the results.
    print("The sum of your numbers is %d." % s)

# Let's add some numbers.
adder(1, 2, 3)

def adder(*nums):
    """This function adds the given numbers together and prints the sum."""
    # Print the results.
    print("The sum of your numbers is %d." % sum(nums))

# Let's add some numbers.
adder(1, 2, 3)


I am working on the file "07 More Functions.ipynb".
The first task is:
You can use a for loop to process these other arguments:
Can you write Python code for it?

def example_function(arg_1, arg_2, *arg_3):
    # Let's look at the argument values.
    print('\narg_1:', arg_1)
    print('arg_2:', arg_2)
    for value in arg_3:
        print('arg_3 value:', value)

example_function(1, 2)
example_function(1, 2, 3)
example_function(1, 2, 3, 4)
example_function(1, 2, 3, 4, 5)


I am working on the file "07 More Functions.ipynb".
The first task is:
top
Positional Arguments
Much of what you will have to learn about using functions involves how to pass values from your calling statement to the function itself. The example we just looked at is pretty simple, in that the function only needed one argument in order to do its work. Let's take a look at a function that requires two arguments to do its work.
Let's make a simple function that takes in three arguments. Let's make a function that takes in a person's first and last name, and then prints out everything it knows about the person.
Here is a simple implementation of this function:
Can you write Python code for it?

def describe_person(first_name, last_name, age):
    # This function takes in a person's first and last name,
    #  and their age.
    # It then prints this information out in a simple format.
    print("First name: %s" % first_name.title())
    print("Last name: %s" % last_name.title())
    print("Age: %d\n" % age)

describe_person('brian', 'kernighan', 71)
describe_person('ken', 'thompson', 70)
describe_person('adele', 'goldberg', 68)


I am working on the file "07 More Functions.ipynb".
The first task is:
top
Accepting an arbitrary number of arguments
We have now seen that using keyword arguments can allow for much more flexible calling statements.

This benefits you in your own programs, because you can write one function that can handle many different situations you might encounter.
This benefits you if other programmers use your programs, because your functions can apply to a wide range of situations.
This benefits you when you use other programmers' functions, because their functions can apply to many situations you will care about.

There is another issue that we can address, though. Let's consider a function that takes two number in, and prints out the sum of the two numbers:
Can you write Python code for it?

def adder(num_1, num_2):
    # This function adds two numbers together, and prints the sum.
    sum = num_1 + num_2
    print("The sum of your numbers is %d." % sum)

# Let's add some numbers.
adder(1, 2)
adder(-1, 2)
adder(1, -2)


I am working on the file "07 More Functions.ipynb".
The first task is:
top
Accepting an arbitrary number of keyword arguments
Python also provides a syntax for accepting an arbitrary number of keyword arguments. The syntax looks like this:
Can you write Python code for it?

import sys

f = open('./test.txt', 'w')

FILE_OUT = f  #sys.stdout

def example_function(*args, **kwargs):
    print(*args, sep='++', end=' ', file=FILE_OUT)
    for k, v in kwargs.items():
        print(k, ': ', v, end=' ', file=FILE_OUT)

example_function(1, 2, 4, 5)
example_function(1, 3, value=1, name=5)
example_function(store='ff', quote='Do. Or do not. There is no try.')

f.close()

def example_function(arg_1, arg_2, **kwargs):
    # Let's look at the argument values.
    print('\narg_1:', arg_1)
    print('arg_2:', arg_2)
    print('arg_3:', kwargs)

example_function('a', 'b')
example_function('a', 'b', value_3='c')
example_function('a', 'b', value_3='c', value_4='d')
example_function('a', 'b', value_3='c', value_4='d', value_5='e')


I am working on the file "07 More Functions.ipynb".
The first task is:
top
Keyword arguments
Python allows us to use a syntax called keyword arguments. In this case, we can give the arguments in any order when we call the function, as long as we use the name of the arguments in our calling statement. Here is how the previous code can be made to work using keyword arguments:
Can you write Python code for it?

def describe_person(first_name, last_name, age):
    # This function takes in a person's first and last name,
    #  and their age.
    # It then prints this information out in a simple format.
    print("First name: %s" % first_name.title())
    print("Last name: %s" % last_name.title())
    print("Age: %d\n" % age)

describe_person(age=71, first_name='brian', last_name='kernighan')
describe_person(age=70, first_name='ken', last_name='thompson')
describe_person(age=68, first_name='adele', last_name='goldberg')


I am working on the file "07 visualization callback.ipynb".
The first task is:
Running our callback
We'll create all of the standard neon callbacks, and then add ours at the end.
Can you write Python code for it?

callbacks = Callbacks(mlp, eval_set=test_set)
cv = CostVisCallback()
callbacks.add_callback(cv)
mlp.fit(train_set, optimizer=optimizer, num_epochs=10, cost=cost, callbacks=callbacks)


I am working on the file "07-Control-Flow-Statements.ipynb".
The first task is:
Here is an example of a break statement used for a less trivial task.
This loop will fill a list with all Fibonacci numbers up to a certain value:
Can you write Python code for it?

a, b = 0, 1
amax = 100
L = []

while True:
    (a, b) = (b, a + b)
    if a > amax:
        break
    L.append(a)

print(L)