I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
Building a Keras Model

Implement a data input_fn process the data for the Keras model
Create the Keras model
Create the callbacks
Run the experiment

1. Data input_fn
A typical way of feed data into Keras is to convert it to a numpy array and pass it to the model.fit() function of the model. However, in other (probably more parctical) cases, all the data may not fit into the memory of your worker. Thus, you woud need to either create a reader that reads your data chuck by chuck, and pass it to model.fit_generator(), or to use the tf.data.Dataset APIs, which are much easier to use.
In the input_fn,
1. Create a CSV dataset (similar to the one used with the TensorFlow Custom Estimator)
2. Create lookups for categorical features vocabolary to numerical index
3. Process the dataset features to:
    * extrat the instance weight column
    * convert the categorical features to numerical index
Can you write Python code for it?

def make_keras_input_fn(file_pattern, batch_size, mode=tf.estimator.ModeKeys.EVAL):

    mapping_tables = {}

    mapping_tables[TARGET_NAME] = tf.contrib.lookup.index_table_from_tensor(
        mapping=tf.constant(TARGET_LABELS))

    for feature_name in CATEGORICAL_FEATURE_NAMES:
        mapping_tables[feature_name] = tf.contrib.lookup.index_table_from_tensor(
            mapping=tf.constant(feature_vocabolary[feature_name]))
    try:
        tf.tables_initializer().run(session=tf.keras.backend.get_session())
    except:
        pass

    def _process_features(features, target):

        weight = features.pop(WEIGHT_COLUMN_NAME)
        target = mapping_tables[TARGET_NAME].lookup(target)
        for feature in CATEGORICAL_FEATURE_NAMES:
            features[feature] = mapping_tables[feature].lookup(features[feature])
        return features, target, weight

    def _input_fn():

        dataset = tf.data.experimental.make_csv_dataset(
            file_pattern=file_pattern,
            batch_size=batch_size,
            column_names=HEADER,
            column_defaults=HEADER_DEFAULTS,
            label_name=TARGET_NAME,
            field_delim=',',
            use_quote_delim=True,
            header=False,
            shuffle= (mode==tf.estimator.ModeKeys.TRAIN)
        ).map(_process_features)

        return dataset

    return _input_fn


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
Building a TensorFlow Custom Estimator

Creating feature columns
Creating model_fn
Create estimator using the model_fn
Define data input_fn
Define Train and evaluate experiment
Run experiment with parameters

1. Create feature columns
Can you write Python code for it?

def create_feature_columns():

    feature_columns = []

    for column in NUMERIC_FEATURE_NAMES:
        feature_column = tf.feature_column.numeric_column(column)
        feature_columns.append(feature_column)

    for column in CATEGORICAL_FEATURE_NAMES:
        vocabolary = feature_vocabolary[column]
        embed_size = int(math.sqrt(len(vocabolary)))
        feature_column = tf.feature_column.embedding_column(
            tf.feature_column.categorical_column_with_vocabulary_list(column, vocabolary),
            embed_size)
        feature_columns.append(feature_column)

    return feature_columns


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
Convert to Estimator for Distributed Training...
Can you write Python code for it?


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
Dataset Metadata
Can you write Python code for it?

HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',
               'marital_status', 'occupation', 'relationship', 'race', 'gender',
               'capital_gain', 'capital_loss', 'hours_per_week',
               'native_country', 'income_bracket']

HEADER_DEFAULTS = [[0], [''], [0], [''], [0], [''], [''], [''], [''], [''],
                       [0], [0], [0], [''], ['']]

NUMERIC_FEATURE_NAMES = ['age', 'education_num', 'capital_gain', 'capital_loss', 'hours_per_week']
CATEGORICAL_FEATURE_NAMES = ['gender', 'race', 'education', 'marital_status', 'relationship',
                             'workclass', 'occupation', 'native_country']

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES
TARGET_NAME = 'income_bracket'
TARGET_LABELS = [' <=50K', ' >50K']
WEIGHT_COLUMN_NAME = 'fnlwgt'
NUM_CLASSES = len(TARGET_LABELS)

def get_categorical_features_vocabolary():
    data = pd.read_csv(TRAIN_DATA_FILE, names=HEADER)
    return {
        column: list(data[column].unique())
        for column in data.columns if column in CATEGORICAL_FEATURE_NAMES
    }

feature_vocabolary = get_categorical_features_vocabolary()
print(feature_vocabolary)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
Download the Data
Can you write Python code for it?

DATA_DIR='data'
!mkdir $DATA_DIR
!gsutil cp gs://cloud-samples-data/ml-engine/census/data/adult.data.csv $DATA_DIR
!gsutil cp gs://cloud-samples-data/ml-engine/census/data/adult.test.csv $DATA_DIR
TRAIN_DATA_FILE = os.path.join(DATA_DIR, 'adult.data.csv')
EVAL_DATA_FILE = os.path.join(DATA_DIR, 'adult.test.csv')

TRAIN_DATA_SIZE = 32561
EVAL_DATA_SIZE = 16278


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
Save and Load Keras Model for Prediction
Can you write Python code for it?

keras_model_dir = os.path.join(model_dir, 'keras_classifier.h5')
model.save(keras_model_dir)
print("Keras model saved to: {}".format(keras_model_dir))
model = tf.keras.models.load_model(keras_model_dir)
print("Keras model loaded.")

predict_data = make_keras_input_fn(
        EVAL_DATA_FILE,
        batch_size=5,
        mode=tf.estimator.ModeKeys.EVAL
    )()

predictions = map(
    lambda probability: TARGET_LABELS[0] if probability <0.5 else TARGET_LABELS[1],
    model.predict(predict_data, steps=1)
)

print(list(predictions))


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
1. Data Input Function

Use tf.data.Dataset APIs: list_files(), skip(), map(), filter(), batch(), shuffle(), repeat(), prefetch(), cache(), etc.
Use tf.data.experimental.make_csv_dataset to read and parse CSV data files.
Use tf.data.experimental.make_batched_features_dataset to read and parse TFRecords data files.
Can you write Python code for it?

def process_features(features, target):
  for feature_name in CATEGORICAL_FEATURE_WITH_VOCABULARY.keys() + CATEGORICAL_FEATURE_WITH_HASH_BUCKETS.keys():
    features[feature_name] = tf.strings.strip(features[feature_name])

  features['capital_total'] = features['capital_gain'] - features['capital_loss']
  return features, target

def make_input_fn(file_pattern, batch_size, num_epochs=1, shuffle=False):

    def _input_fn():
        dataset = tf.data.experimental.make_csv_dataset(
            file_pattern=file_pattern,
            batch_size=batch_size,
            column_names=HEADER,
            column_defaults=HEADER_DEFAULTS,
            label_name=TARGET_NAME,
            field_delim=',',
            use_quote_delim=True,
            header=False,
            num_epochs=num_epochs,
            shuffle=shuffle,
            shuffle_buffer_size=(5 * batch_size),
            shuffle_seed=RANDOM_SEED,
            num_parallel_reads=multiprocessing.cpu_count(),
            sloppy=True,
        )
        return dataset.map(process_features).cache()

    return _input_fn

# You need to run tf.enable_eager_execution() at the top.

dataset = make_input_fn(TRAIN_DATA_FILE, batch_size=1)()
for features, target in dataset.take(1):
  print "Input Features:"
  for key in features:
    print "{}:{}".format(key, features[key])

  print ""
  print "Target:"
  print target


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
3. Instantiate a Wide and Deep Estimator

Can you write Python code for it?

def create_estimator(params, run_config):

    wide_columns, deep_columns = create_feature_columns()

    estimator = tf.estimator.DNNLinearCombinedClassifier(

        n_classes=len(TARGET_LABELS),
        label_vocabulary=TARGET_LABELS,
        weight_column=WEIGHT_COLUMN_NAME,

        dnn_feature_columns=deep_columns,
        dnn_optimizer=tf.train.AdamOptimizer(
          learning_rate=params.learning_rate),
        dnn_hidden_units=params.hidden_units,
        dnn_dropout=params.dropout,
        dnn_activation_fn=tf.nn.relu,
        batch_norm=True,

        linear_feature_columns=wide_columns,
        linear_optimizer='Ftrl',

        config=run_config
    )

    return estimator


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
4. Implement Train and Evaluate Experiment
<img valign="middle" src="images/tf-estimators.jpeg" width="900">
Delete the model_dir file if you don't want a Warm Start
* If not deleted, and you change the model, it will error.
TrainSpec
* Set shuffle in the input_fn to True
* Set num_epochs in the input_fn to None
* Set max_steps. One batch (feed-forward pass & backpropagation)
corresponds to 1 training step.
EvalSpec
* Set shuffle in the input_fn to False
* Set Set num_epochs in the input_fn to 1
* Set steps to None if you want to use all the evaluation data.
* Otherwise, set steps to the number of batches you want to use for evaluation, and set shuffle to True.
* Set start_delay_secs to 0  to start evaluation as soon as a checkpoint is produced.
* Set throttle_secs to 0 to re-evaluate as soon as a new checkpoint is produced.
Can you write Python code for it?

def run_experiment(estimator, params, run_config,
  resume=False, train_hooks=None, exporters=None):

  print "Resume training {}: ".format(resume)
  print "Epochs: {}".format(epochs)
  print "Batch size: {}".format(params.batch_size)
  print "Steps per epoch: {}".format(steps_per_epoch)
  print "Training steps: {}".format(params.max_steps)
  print "Learning rate: {}".format(params.learning_rate)
  print "Hidden Units: {}".format(params.hidden_units)
  print "Dropout probability: {}".format(params.dropout)
  print "Save a checkpoint and evaluate afer {} step(s)".format(run_config.save_checkpoints_steps)
  print "Keep the last {} checkpoint(s)".format(run_config.keep_checkpoint_max)
  print ""

  tf.logging.set_verbosity(tf.logging.INFO)

  if not resume:
    if tf.gfile.Exists(run_config.model_dir):
      print "Removing previous artefacts..."
      tf.gfile.DeleteRecursively(run_config.model_dir)
  else:
    print "Resuming training..."

  # Create train specs.
  train_spec = tf.estimator.TrainSpec(
      input_fn = make_input_fn(
          TRAIN_DATA_FILE,
          batch_size=params.batch_size,
          num_epochs=None, # Run until the max_steps is reached.
          shuffle=True
      ),
      max_steps=params.max_steps,
      hooks=train_hooks
  )

  # Create eval specs.
  eval_spec = tf.estimator.EvalSpec(
      input_fn = make_input_fn(
          EVAL_DATA_FILE,
          batch_size=params.batch_size,
      ),
      exporters=exporters,
      start_delay_secs=0,
      throttle_secs=0,
      steps=None # Set to limit number of steps for evaluation.
  )

  time_start = datetime.utcnow()
  print "Experiment started at {}".format(time_start.strftime("%H:%M:%S"))
  print "......................................."

  # Run train and evaluate.
  tf.estimator.train_and_evaluate(
    estimator=estimator,
    train_spec=train_spec,
    eval_spec=eval_spec)

  time_end = datetime.utcnow()
  print "......................................."
  print "Experiment finished at...


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
5. Export your trained model
Implement serving input receiver function
Can you write Python code for it?

def make_serving_input_receiver_fn():
  inputs = {}
  for feature_name in FEATURE_NAMES:
    dtype = tf.float32 if feature_name in NUMERIC_FEATURE_NAMES else tf.string
    inputs[feature_name] = tf.placeholder(shape=[None], dtype=dtype)

  # What is wrong here?
  return tf.estimator.export.build_raw_serving_input_receiver_fn(inputs)


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
6. Early Stopping

stop_if_higher_hook
stop_if_lower_hook
stop_if_no_increase_hook
stop_if_no_decrease_hook
Can you write Python code for it?

early_stopping_hook = tf.contrib.estimator.stop_if_no_increase_hook(
    estimator,
    'accuracy',
    max_steps_without_increase=100,
    run_every_secs=None,
    run_every_steps=500
)

params.max_steps = 1000000
params.hidden_units = [128, 128]
params.dropout = 0
run_config = tf.estimator.RunConfig(
    tf_random_seed=RANDOM_SEED,
    save_checkpoints_steps=500,
    keep_checkpoint_max=1,
    model_dir=model_dir,
    log_step_count_steps=100
)

run_experiment(estimator, params, run_config, exporters = [exporter], train_hooks=[early_stopping_hook])


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
8. Extending a Premade Estimator
Add an evaluation metric

tf.metrics
tf.estimator.add_metric
Can you write Python code for it?

def metric_fn(labels, predictions):

  metrics = {}

  label_index = tf.contrib.lookup.index_table_from_tensor(tf.constant(TARGET_LABELS)).lookup(labels)
  one_hot_labels = tf.one_hot(label_index, len(TARGET_LABELS))

  metrics['mirco_accuracy'] = tf.metrics.mean_per_class_accuracy(
    labels=label_index,
    predictions=predictions['class_ids'],
    num_classes=2)

  metrics['f1_score'] = tf.contrib.metrics.f1_score(
    labels=one_hot_labels,
    predictions=predictions['probabilities'])

  return metrics

params.max_steps = 1
estimator = create_estimator(params, run_config)
estimator = tf.contrib.estimator.add_metrics(estimator, metric_fn)
run_experiment(estimator, params, run_config)


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
9. Adaptive learning rate

exponential_decay
consine_decay
linear_cosine_decay
consine_decay_restarts
polynomial decay
piecewise_constant_decay
Can you write Python code for it?

def create_estimator(params, run_config):

  wide_columns, deep_columns = create_feature_columns()

  def _update_optimizer(initial_learning_rate, decay_steps):

    # learning_rate = tf.train.exponential_decay(
    #   initial_learning_rate,
    #   global_step=tf.train.get_global_step(),
    #   decay_steps=decay_steps,
    #   decay_rate=0.9
    # )

    learning_rate = tf.train.cosine_decay_restarts(
      initial_learning_rate,
      tf.train.get_global_step(),
      first_decay_steps=50,
      t_mul=2.0,
      m_mul=1.0,
      alpha=0.0,
    )

    tf.summary.scalar('learning_rate', learning_rate)

    return tf.train.AdamOptimizer(learning_rate=learning_rate)

  estimator = tf.estimator.DNNLinearCombinedClassifier(

    n_classes=len(TARGET_LABELS),
    label_vocabulary=TARGET_LABELS,
    weight_column=WEIGHT_COLUMN_NAME,

    dnn_feature_columns=deep_columns,
    dnn_optimizer=lambda: _update_optimizer(params.learning_rate, params.max_steps),
    dnn_hidden_units=params.hidden_units,
    dnn_dropout=params.dropout,
    batch_norm=True,

    linear_feature_columns=wide_columns,
    linear_optimizer='Ftrl',

    config=run_config
  )

  return estimator

params.learning_rate = 0.1
params.max_steps = 1000
run_config = tf.estimator.RunConfig(
    tf_random_seed=RANDOM_SEED,
    save_checkpoints_steps=200,
    model_dir=model_dir,
)

if COLAB:
  from tensorboardcolab import *
  TensorBoardColab(graph_path=model_dir)

estimator = create_estimator(params, run_config)
run_experiment(estimator, params, run_config)


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
Download Data
UCI Adult Dataset: https://archive.ics.uci.edu/ml/datasets/adult
Predict whether income exceeds $50K/yr based on census data. Also known as "Census Income" dataset.
Can you write Python code for it?

DATA_DIR='data'
!mkdir $DATA_DIR
!gsutil cp gs://cloud-samples-data/ml-engine/census/data/adult.data.csv $DATA_DIR
!gsutil cp gs://cloud-samples-data/ml-engine/census/data/adult.test.csv $DATA_DIR

TRAIN_DATA_FILE = os.path.join(DATA_DIR, 'adult.data.csv')
EVAL_DATA_FILE = os.path.join(DATA_DIR, 'adult.test.csv')

!wc -l $TRAIN_DATA_FILE
!wc -l $EVAL_DATA_FILE


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
Export the Model during Training and Evaluation
Saved models are exported under <model_dir>/export/<folder_name>.
* Latest Exporter: exports a model after each evaluation.
 * specify the maximum number of exported models to keep using exports_to_keep param.
* Final Exporter: exports only the very last evaluated checkpoint. of the model.
* Best exporter: runs everytime when the newly evaluted checkpoint is better than any exsiting model.
 * specify the maximum number of exported models to keep using exports_to_keep param.
 * It uses the evaluation events stored under the eval folder.
Can you write Python code for it?

def _accuracy_bigger(best_eval_result, current_eval_result):

  metric = 'accuracy'
  return best_eval_result[metric] < current_eval_result[metric]

params.max_steps = 1000
params.hidden_units = [128, 128]
params.dropout = 0
run_config = tf.estimator.RunConfig(
    tf_random_seed=RANDOM_SEED,
    save_checkpoints_steps=200,
    keep_checkpoint_max=1,
    model_dir=model_dir,
    log_step_count_steps=50
)

exporter = tf.estimator.BestExporter(
    compare_fn=_accuracy_bigger,
    event_file_pattern='eval_{}/*.tfevents.*'.format(datetime.utcnow().strftime("%H%M%S")),
    name="estimate", # Saved models are exported under /export/estimate/
    serving_input_receiver_fn=make_serving_input_receiver_fn(),
    exports_to_keep=1
)

estimator = create_estimator(params, run_config)
run_experiment(estimator, params, run_config, exporters = [exporter])

!gsutil ls {model_dir}/export/estimate


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
Export to saved_model
Can you write Python code for it?

export_dir = os.path.join(model_dir, 'export')

# Delete export directory if exists.
if tf.gfile.Exists(export_dir):
    tf.gfile.DeleteRecursively(export_dir)

# Export the estimator as a saved_model.
estimator.export_savedmodel(
    export_dir_base=export_dir,
    serving_input_receiver_fn=make_serving_input_receiver_fn()
)

!gsutil ls gs://ksalama-gcs-cloudml/others/models/census/dnn_classifier/export/1552582374

%%bash

saved_models_base=${MODEL_DIR}/export/
saved_model_dir=$(gsutil ls ${saved_models_base} | tail -n 1)
saved_model_cli show --dir=${saved_model_dir} --all


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
Test saved_model
Can you write Python code for it?

export_dir = os.path.join(model_dir, 'export')
tf.gfile.ListDirectory(export_dir)[-1]
saved_model_dir = os.path.join(export_dir, tf.gfile.ListDirectory(export_dir)[-1])
print(saved_model_dir)
print ""

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="predict"
)

output = predictor_fn(
    {
        'age': [34.0],
        'workclass': ['Private'],
        'education': ['Doctorate'],
        'education_num': [10.0],
        'marital_status': ['Married-civ-spouse'],
        'occupation': ['Prof-specialty'],
        'relationship': ['Husband'],
        'race': ['White'],
        'gender': ['Male'],
        'capital_gain': [0.0],
        'capital_loss': [0.0],
        'hours_per_week': [40.0],
        'native_country':['Egyptian']
    }
)
print(output)


I am working on the file "00_Miscellaneous/tfx/01_tf_estimator_deepdive.ipynb".
The first task is:
The training data includes 32,561 records, while the evaluation data includes 16,278 records.
Can you write Python code for it?

HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',
               'marital_status', 'occupation', 'relationship', 'race', 'gender',
               'capital_gain', 'capital_loss', 'hours_per_week',
               'native_country', 'income_bracket']

pd.read_csv(TRAIN_DATA_FILE, names=HEADER).head()


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
2. Generate Statistics from the Data using TFDV
In this step, we will use TFDV to analyze the data in GCS and compute various statistics from it. This operation requires (multiple) full pass on the data to compute mean, max, min, etc., which needs to run at scale to analyze large dataset.
If we run the analysis on a sample of data, we can use TFDV to compute the statistics locally. However, we can run the TFDV process using Cloud Dataflow for scalability. The generated statistics is stored as a proto buffer to GCS.
Can you write Python code for it?

job_name = 'tfdv-flights-stats-gen-{}'.format(datetime.utcnow().strftime('%y%m%d-%H%M%S'))
args['job_name'] = job_name
stats_location = os.path.join(STATS_DIR, 'stats.pb')

pipeline_options =  beam.options.pipeline_options.GoogleCloudOptions(**args)
print(pipeline_options)

print("Computing statistics...")
_ = tfdv.generate_statistics_from_tfrecord(
    data_location=data_location,
    output_path=stats_location,
    stats_options=tfdv.StatsOptions(
        sample_rate=.3
    ),
    pipeline_options = pipeline_options(**args)
)

print("Statistics are computed and saved to: {}".format(stats_location))


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
5. Extract New Data
In this step, we are going to extract new data from BigQuery and store it to GCS is TFRecord files. This will be flights data in 2012, however, we are going to introduce the following alternation in the data schema and content to demonstrate types of anomalies to be detected via TFDV:
1. Skip February data
2. Introduce missing values to airline
3. Add is_weekend column
4. Convert the time slot (12:00am - 6:00am) to two time slots: (12:00am - 3:00am), (3:00am - 6:00am)
5. Change the departure_delay values from minutes to seconds
Implementing the "altered" source query
Can you write Python code for it?

def generate_altered_query(date_from=None, date_to=None, limit=None):
    query ="""
        SELECT * FROM (
            SELECT
              CAST(date AS DATE) AS flight_date,
              FORMAT_DATE('%b',  CAST(date AS DATE)) AS flight_month,
              EXTRACT(DAY FROM CAST(date AS DATE)) AS flight_day,
              FORMAT_DATE('%a',  CAST(date AS DATE)) AS flight_day_of_week,
              CASE WHEN EXTRACT(DAYOFWEEK FROM CAST(date AS DATE)) IN (1 , 7) THEN 'Yes' ELSE 'No' END AS is_weekend,
              CASE WHEN airline = 'MQ' THEN NULL ELSE airline END airline,
              departure_airport,
              arrival_airport,
              CAST(SUBSTR(LPAD(CAST(departure_schedule AS STRING), 4, '0'), 0, 2) AS INT64) AS departure_schedule_hour,
              CAST(SUBSTR(LPAD(CAST(departure_schedule AS STRING), 4, '0'), 3, 2) AS INT64) AS departure_schedule_minute,
              CASE
                WHEN departure_schedule BETWEEN 600 AND 900 THEN '[6:00am - 9:00am]'
                WHEN departure_schedule BETWEEN 900 AND 1200 THEN '[9:00am - 12:pm]'
                WHEN departure_schedule BETWEEN 1200 AND 1500 THEN '[12:00pm - 3:00pm]'
                WHEN departure_schedule BETWEEN 1500 AND 1800 THEN '[3:00pm - 6:00pm]'
                WHEN departure_schedule BETWEEN 1800 AND 2100 THEN '[6:00pm - 9:00pm]'
                WHEN departure_schedule BETWEEN 2100 AND 2400 THEN '[9:00pm - 12:00am]'
                WHEN departure_schedule BETWEEN 0000 AND 300 THEN '[12:00am - 3:00am]'
                ELSE '[3:00am - 6:00am]'
              END AS departure_time_slot,
              departure_delay * 60 AS departure_delay,
              arrival_delay
            FROM
              `bigquery-samples.airline_ontime_data.flights`
            WHERE
              EXTRACT(MONTH FROM CAST(date AS DATE)) != 2
        )
        """
    if date_from:
        query += "WHERE flight_date >= CAST('{}' as DATE) \n".format(date_from)
        if date_to:
            query += "AND flight_date < CAST('{}' as DATE)...


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
6. Validate the New Data and Identify Anomalies
In this step, we are going to use the generated schema to validate the newly extracted data to identify if the data complies with the schema, or if there are any anomalies to be handled.
Load Schema
Can you write Python code for it?


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Cleanup working directory...
Can you write Python code for it?

if tf.gfile.Exists(ROOT):
    print("Removing {} contents...".format(ROOT))
    tf.gfile.DeleteRecursively(ROOT)

print("Creating working directory: {}".format(ROOT))
tf.gfile.MkDir(ROOT)


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Display anomalies (if any)
Can you write Python code for it?


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Fix and save the schema
You can manually alter the schema before you save it. For example:
* Set the maximum fraction of missing values allowed in a feature.
* Add values to a categorical feature domain.
* Set minimum fraction of values that must come from the domain .
* Change the min and max allowed values to a numeric feauture domain.
* Set a drift comparator.
Can you write Python code for it?

from tensorflow_metadata.proto.v0 import schema_pb2

# Allow no missing values
tfdv.get_feature(schema, 'airline').presence.min_fraction = 1.0

# Only allow 10% of the values to be new
tfdv.get_feature(schema, 'departure_airport').distribution_constraints.min_domain_mass = 0.9

domain = tfdv.utils.schema_util.schema_pb2.FloatDomain(
    min=-60, # a flight can departure 1 hour earlier
    max=480 # maximum departure delay is 8 hours, otherwise the flight is cancelled.
)
tfdv.set_domain(schema, 'departure_delay', domain)

tfdv.get_feature(schema, 'flight_month').drift_comparator.infinity_norm.threshold = 0.01

from tensorflow.python.lib.io import file_io
from google.protobuf import text_format

tf.gfile.MkDir(dirname=SCHEMA_DIR)
schema_location = os.path.join(SCHEMA_DIR, 'schema.pb')
tfdv.write_schema_text(schema, schema_location)
print("Schema file saved to:{}".format(schema_location))


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Generate statistics for the new data
Can you write Python code for it?

job_name = 'tfdv-flights-stats-gen-{}'.format(datetime.utcnow().strftime('%y%m%d-%H%M%S'))
args['job_name'] = job_name
new_stats_location = os.path.join(STATS_DIR, 'new_stats.pb')

pipeline_options = beam.options.pipeline_options.GoogleCloudOptions(**args)
print(pipeline_options)

print("Computing statistics...")
_ = tfdv.generate_statistics_from_tfrecord(
    data_location=data_location,
    output_path=new_stats_location,
    stats_options=tfdv.StatsOptions(
        sample_rate=.5
    ),
    pipeline_options = pipeline_options
)

print("Statistics are computed and saved to: {}".format(new_stats_location))


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Implementing helper functions
Can you write Python code for it?

def get_type_map(query):
    bq_client = bigquery.Client()
    query_job = bq_client.query("SELECT * FROM ({}) LIMIT 0".format(query))
    results = query_job.result()
    type_map = {}
    for field in results.schema:
        type_map[field.name] = field.field_type

    return type_map

def row_to_example(instance, type_map):
    feature = {}
    for key, value in instance.items():
        data_type = type_map[key]
        if value is None:
            feature[key] = tf.train.Feature()
        elif data_type == 'INTEGER':
            feature[key] = tf.train.Feature(
                int64_list=tf.train.Int64List(value=[value]))
        elif data_type == 'FLOAT':
            feature[key] = tf.train.Feature(
                float_list=tf.train.FloatList(value=[value]))
        else:
            feature[key] = tf.train.Feature(
                bytes_list=tf.train.BytesList(value=[tf.compat.as_bytes(value)]))

    return tf.train.Example(features=tf.train.Features(feature=feature))


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Implementing the pipeline
Can you write Python code for it?

def run_pipeline(args):

    source_query = args.pop('source_query')
    sink_data_location = args.pop('sink_data_location')
    runner = args['runner']

    pipeline_options = beam.options.pipeline_options.GoogleCloudOptions(**args)
    print(pipeline_options)

    with beam.Pipeline(runner, options=pipeline_options) as pipeline:
        (pipeline
         | "Read from BigQuery">> beam.io.Read(beam.io.BigQuerySource(query = source_query, use_standard_sql = True))
         | 'Convert to tf Example' >> beam.Map(lambda instance: row_to_example(instance, type_map))
         | 'Serialize to String' >> beam.Map(lambda example: example.SerializeToString(deterministic=True))
         | "Write as TFRecords to GCS" >> beam.io.WriteToTFRecord(
                    file_path_prefix = sink_data_location+"extract",
                    file_name_suffix=".tfrecords")
        )


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Load data statistics
Can you write Python code for it?

stats = tfdv.load_statistics(stats_location)
new_stats = tfdv.load_statistics(new_stats_location)


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Run the pipeline
Can you write Python code for it?

runner = 'DirectRunner' if LOCAL else 'DataflowRunner'
job_name = 'tfdv-flights-data-extraction-{}'.format(datetime.utcnow().strftime('%y%m%d-%H%M%S'))
date_from =  '2010-01-01'
date_to = '2011-12-31'
data_location = os.path.join(DATA_DIR,
        "{}-{}/".format(date_from.replace('-',''), date_to.replace('-','')))
print("Data will be extracted to: {}".format(data_location))

print("Generating source query...")
limit = 100000 if LOCAL else None
source_query = generate_query(date_from, date_to, limit)

print("Retrieving data type...")
type_map = get_type_map(source_query)

args = {
    'job_name': job_name,
    'runner': runner,
    'source_query': source_query,
    'type_map': type_map,
    'sink_data_location': data_location,
    'project': PROJECT_ID,
    'region': REGION,
    'staging_location': STAGING_DIR,
    'temp_location': TEMP_DIR,
    'save_main_session': True,
    'setup_file': './setup.py'
}
print("Pipeline args are set.")


runner = 'DirectRunner' if LOCAL else 'DataflowRunner'
job_name = 'tfdv-flights-data-extraction-{}'.format(datetime.utcnow().strftime('%y%m%d-%H%M%S'))
date_from =  '2012-01-01'
date_to = '2012-12-31'
data_location = os.path.join(DATA_DIR,
        "{}-{}/".format(date_from.replace('-',''), date_to.replace('-','')))
print("Data will be extracted to: {}".format(data_location))

print("Generating altered source query...")
limit = 100000 if LOCAL else None
source_query = generate_query(date_from, date_to, limit)

print("Retrieving data type...")
type_map = get_type_map(source_query)

args = {
    'job_name': job_name,
    'runner': runner,
    'source_query': source_query,
    'type_map': type_map,
    'sink_data_location': data_location,
    'project': PROJECT_ID,
    'region': REGION,
    'staging_location': STAGING_DIR,
    'temp_location': TEMP_DIR,
    'save_main_session': True,
    'setup_file': './setup.py'
}
print("Pipeline args are set.")

print("Running data extraction pipeline...")
run_pipeline(args)
print("Pipeline is done.")


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Setup
To get started, set your GCP PROJECT_ID, BUCKET_NAME, and REGION to the following variables. Create a GCP Project if you don't have one. Create a regional Cloud Storage bucket if you don't have one.
Can you write Python code for it?

LOCAL = True # Change to false to run on the GCP

PROJECT_ID = 'validateflow' # Set your GCP Project Id
BUCKET_NAME = 'validateflow' # Set your Bucket name
REGION = 'europe-west1' # Set the region for Dataflow jobs

ROOT = './tfdv' if LOCAL else 'gs://{}/tfdv'.format(BUCKET_NAME)

DATA_DIR = ROOT + '/data/' # Location to store data
SCHEMA_DIR = ROOT + '/schema/' # Location to store data schema
STATS_DIR = ROOT +'/stats/' # Location to store stats
STAGING_DIR = ROOT + '/job/staging/' # Dataflow staging directory on GCP
TEMP_DIR =  ROOT + '/job/temp/' # Dataflow temporary directory on GCP


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
Verify the version of the installed packages:
Can you write Python code for it?

print "TF version:", tf.__version__
print "TFDV version:", tfdv.__version__
print "Beam version:", beam.__version__
print "BQ SDK version:", bigquery.__version__


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
You can list saved schema file...
Can you write Python code for it?

!ls {schema_location}
#!gsuitl ls {schema_location}


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
You can list saves statistics file...
Can you write Python code for it?

!ls {stats_location}
#!gsutil ls {stats_location}


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
You can list the extracted data files...
Can you write Python code for it?

#!gsutil ls {DATA_DIR}/*
!ls {DATA_DIR}/*


I am working on the file "00_Miscellaneous/tfx/03_eda_with_tfdv.ipynb".
The first task is:
You can run the following cell to see a sample of the data to be extracted...
Can you write Python code for it?

%%bigquery

SELECT
    CAST(date AS DATE) AS flight_date,
    FORMAT_DATE('%b',  CAST(date AS DATE)) AS flight_month,
    EXTRACT(DAY FROM CAST(date AS DATE)) AS flight_day,
    FORMAT_DATE('%a',  CAST(date AS DATE)) AS flight_day_of_week,
    airline,
    departure_airport,
    arrival_airport,
    CAST(SUBSTR(LPAD(CAST(departure_schedule AS STRING), 4, '0'), 0, 2) AS INT64) AS departure_schedule_hour,
    CAST(SUBSTR(LPAD(CAST(departure_schedule AS STRING), 4, '0'), 3, 2) AS INT64) AS departure_schedule_minute,
    CASE
        WHEN departure_schedule BETWEEN 600 AND 900 THEN '[6:00am - 9:00am]'
        WHEN departure_schedule BETWEEN 900 AND 1200 THEN '[9:00am - 12:pm]'
        WHEN departure_schedule BETWEEN 1200 AND 1500 THEN '[12:00pm - 3:00pm]'
        WHEN departure_schedule BETWEEN 1500 AND 1800 THEN '[3:00pm - 6:00pm]'
        WHEN departure_schedule BETWEEN 1800 AND 2100 THEN '[6:00pm - 9:00pm]'
        WHEN departure_schedule BETWEEN 2100 AND 2400 THEN '[9:00pm - 12:00am]'
        ELSE '[12:00am - 6:00am]'
    END AS departure_time_slot,
    departure_delay,
    arrival_delay
FROM
    `bigquery-samples.airline_ontime_data.flights`
LIMIT 5


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/01-census-data-analysis.ipynb".
The first task is:
1. Compute Statistics
Can you write Python code for it?

train_stats = tfdv.generate_statistics_from_csv(
    data_location=TRAIN_DATA_DIR+'/*.csv',
    column_names=None, # CSV data file include header
    stats_options=tfdv.StatsOptions(
        weight_feature=WEIGHT_FEATURE_NAME,
        sample_rate=1.0
    )
)

tfdv.visualize_statistics(train_stats)


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/01-census-data-analysis.ipynb".
The first task is:
2. Infer Schema
Can you write Python code for it?

schema = tfdv.infer_schema(statistics=train_stats)
tfdv.display_schema(schema=schema)


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/01-census-data-analysis.ipynb".
The first task is:
Adding headers to the CSV files as the CsvExampleGen components expect headers...
Can you write Python code for it?

import pandas as pd

HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',
               'marital_status', 'occupation', 'relationship', 'race', 'gender',
               'capital_gain', 'capital_loss', 'hours_per_week',
               'native_country', 'income_bracket']

pd.read_csv(DATA_DIR +"/adult.data.csv", names=HEADER).to_csv(TRAIN_DATA_DIR +"/train-01.csv", index=False)
pd.read_csv(DATA_DIR +"/adult.test.csv", names=HEADER).to_csv(TRAIN_DATA_DIR +"/train-02.csv", index=False)

!wc -l $TRAIN_DATA_DIR/train-01.csv
!head $TRAIN_DATA_DIR/train-01.csv


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/01-census-data-analysis.ipynb".
The first task is:
Tensorflow Data Validation
Can you write Python code for it?

import tensorflow_data_validation as tfdv

TARGET_FEATURE_NAME = 'income_bracket'
WEIGHT_FEATURE_NAME = 'fnlwgt'


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/01-census-data-analysis.ipynb".
The first task is:
Test loading saved schema
Can you write Python code for it?


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
1. Data Ingestion (ExampleGen)

Reads the CSV data files (expecting to include headers)
Split the data to train and eval sets

Write the data to TFRecords

Inputs: ExternalPath

Ouptpus: Examples (TFRecords)
Properties: split ratio
Can you write Python code for it?

from tfx.utils.dsl_utils import external_input
from tfx.proto import example_gen_pb2

output_config = example_gen_pb2.Output(
    split_config=example_gen_pb2.SplitConfig(splits=[
        example_gen_pb2.SplitConfig.Split(name='train', hash_buckets=4),
        example_gen_pb2.SplitConfig.Split(name='eval', hash_buckets=1)
    ]))

example_gen = tfx.components.CsvExampleGen(
    instance_name='Data_Extraction_Spliting',
    input=external_input(TRAIN_DATA_DIR),
    output_config=output_config
)

context.run(example_gen)


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
2.2. Import the fixed raw schema (ImporterNode)
The ImporterNode allows you to import an external artifact to a component.
You need to specifiy:
1. Artifact Type
2. Artifcat Location
Can you write Python code for it?

schema_importer = tfx.components.common_nodes.importer_node.ImporterNode(
    instance_name='Schema_Importer',
    source_uri=RAW_SCHEMA_DIR,
    artifact_type=tfx.types.standard_artifacts.Schema,
    reimport=False
)

context.run(schema_importer)

context.show(schema_importer.outputs['result'])


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
3. Data Preprocessing
3.1. Implement the preprocessing logic
We need to implement the preprocessing logic in a python module: transform.py.
This module is expected to have preprocessing_fn method, which accept a dictionary of tensors representing the raw features), and return a dictionary of tensors representing the transformed features.
We use the raw schema to identify feature types and the required transformation.
The function is implemented using TensorFlow Transform.
Can you write Python code for it?

!mkdir {SRC_DIR}

_transform_module_file = SRC_DIR + '/transform.py'

%%writefile {_transform_module_file}

import tensorflow_transform as tft
import tensorflow_data_validation as tfdv

TARGET_FEATURE_NAME = 'income_bracket'
WEIGHT_FEATURE_NAME = 'fnlwgt'
RAW_SCHEMA_LOCATION = '../raw_schema/schema.pbtxt'

raw_schema = tfdv.load_schema_text(RAW_SCHEMA_LOCATION)

def preprocessing_fn(input_features):

    processed_features = {}

    for feature in raw_schema.feature:

        # Pass the target feature as is.
        if feature.name in [TARGET_FEATURE_NAME, WEIGHT_FEATURE_NAME]:
            processed_features[feature.name] = input_features[feature.name]
            continue

        if feature.type == 1:
            # Extract vocabulary and integerize categorical features.
            processed_features[feature.name+"_integerized"] = tft.compute_and_apply_vocabulary(
                input_features[feature.name], vocab_filename=feature.name)
        else:
            # normalize numeric features.
            processed_features[feature.name+"_scaled"] = tft.scale_to_z_score(input_features[feature.name])

        # Bucketize age using quantiles.
        quantiles = tft.quantiles(input_features["age"], num_buckets=5, epsilon=0.01)
        processed_features["age_bucketized"] = tft.apply_buckets(
            input_features["age"], bucket_boundaries=quantiles)

    return processed_features


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
3.2. Tranform train and eval data (Transform)
The component uses the transform output generated from transforming the train data to transform eval data.
That is, while the train data is analyzed and transformed, the eval data is only transformed uaing the output of the analyze phase (TransformGraph) on the train data.

Inputs: train and eval data (Examples), raw schema (Schema), transformation module (file)
outputs: transformed train and eval data (Examples), transform output (TransformGraph)
Can you write Python code for it?

transform = tfx.components.Transform(
    examples=example_gen.outputs['examples'],
    schema=schema_importer.outputs['result'],
    module_file=_transform_module_file,
    instance_name="Data_Transformation"
)

context.run(transform)

uri = transform.outputs['transform_graph'].get()[0].uri
os.listdir(uri)


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
4.2 Train the model using the Trainer component

Inputs: train module file with the trainer_fn, raw schema (Schema), and transform output (TransformGraph)
Outputs: saved_model (Model)
Properties: train and eval args
Can you write Python code for it?

trainer = tfx.components.Trainer(
    module_file=_train_module_file,
    transformed_examples=transform.outputs['transformed_examples'],
    schema=schema_importer.outputs['result'],
    transform_graph=transform.outputs['transform_graph'],
    train_args=tfx.proto.trainer_pb2.TrainArgs(num_steps=1000),
    eval_args=tfx.proto.trainer_pb2.EvalArgs(num_steps=None),
    instance_name='Census_Classifier_Trainer'
)

context.run(trainer)

train_uri = trainer.outputs['model'].get()[0].uri
serving_model_path = os.path.join(train_uri, 'serving_model_dir', 'export', 'census')
latest_serving_model_path = os.path.join(serving_model_path, max(os.listdir(serving_model_path)))
print(latest_serving_model_path)


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
5. Evaluate the trained model (Evaluator)

Inputs: eval data (Examples), trained model (Model)
Outputs eval metric (ModelEvaluation)
Properties: Slicing Specs
Can you write Python code for it?

feature_slicing_spec=tfx.proto.evaluator_pb2.FeatureSlicingSpec(
    specs=[
        tfx.proto.evaluator_pb2.SingleSlicingSpec(),
        tfx.proto.evaluator_pb2.SingleSlicingSpec(column_for_slicing=['occupation'])
    ]
)

model_analyzer = tfx.components.Evaluator(
    examples=example_gen.outputs['examples'],
    model=trainer.outputs['model'],
    feature_slicing_spec=feature_slicing_spec,
    instance_name="Occupation_based_Evaluator"
)

context.run(model_analyzer)

import tensorflow_model_analysis as tfma

results_uri = model_analyzer.outputs['output'].get()[0].uri
eval_result = tfma.load_eval_result(results_uri)
eval_result.slicing_metrics

tfma.view.render_slicing_metrics(
    result=eval_result,
    slicing_column='occupation'
)


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
6. Validate the Trained Model
Can you write Python code for it?

model_validator = tfx.components.ModelValidator(
    examples=example_gen.outputs['examples'],
    model=trainer.outputs['model'],
    instance_name="Base_Model_Validator"
)
context.run(model_validator)


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
6.3. Create AccuracyModelValidator
Can you write Python code for it?

from typing import Optional
from tfx import types
from tfx.components.base import base_component
from tfx.components.base import executor_spec

class AccuracyModelValidator(base_component.BaseComponent):

    SPEC_CLASS = AccuracyValidatorSpec
    EXECUTOR_SPEC = executor_spec.ExecutorClassSpec(AccuracyValidatorExecutor)

    def __init__(self,
                 eval_results: types.channel,
                 model: types.channel,
                 accuracy_threshold: float,
                 slice_accuracy_tolerance: float,
                 blessing: Optional[types.Channel] = None,
                 instance_name=None):

        blessing = blessing or types.Channel(
            type=standard_artifacts.ModelBlessing,
            artifacts=[standard_artifacts.ModelBlessing()])

        spec = AccuracyValidatorSpec(
            eval_results=eval_results, model=model, blessing=blessing,
            accuracy_threshold=accuracy_threshold, slice_accuracy_tolerance=slice_accuracy_tolerance
        )

        super().__init__(spec=spec, instance_name=instance_name)

accuracy_model_validator = AccuracyModelValidator(
    eval_results=model_analyzer.outputs['output'],
    model=trainer.outputs['model'],
    accuracy_threshold=0.85,
    slice_accuracy_tolerance=0.15,
    instance_name="Accuracy_Model_Validator"
)

context.run(accuracy_model_validator)

blessing_uri = accuracy_model_validator.outputs.blessing.get()[0].uri
!ls -l {blessing_uri}


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
7. Pushing the Blessed Model (Pusher)
This steps pushes the validated and blessed model to its final destination. This could be:
1. A Model Registry
2. Git repository
3. API Serving Platform
4. A specific filesystem location
5. ...
7.1. Push the blessed model to model registry (filesystem location)
Can you write Python code for it?

serving_models_location = ROOT_DIR + '/model_registry'
!mkdir {serving_models_location}

push_destination=tfx.proto.pusher_pb2.PushDestination(
    filesystem=tfx.proto.pusher_pb2.PushDestination.Filesystem(
        base_directory=serving_models_location)
)

pusher = tfx.components.Pusher(
    model=trainer.outputs['model'],
    model_blessing=accuracy_model_validator.outputs['blessing'],
    push_destination=push_destination
)

context.run(pusher)


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
List tables
Can you write Python code for it?

cursor.execute("SELECT name FROM sqlite_master WHERE type='table';")
pprint(cursor.fetchall())


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
Query Artifact table
Can you write Python code for it?

cursor.execute("SELECT * FROM Artifact;")
pprint(cursor.fetchall())


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
Read sample of the extracted data...
Since TTF v1.15 is used, you need to enable eager execution. However, this causes problems to subsequent steps.
Can you write Python code for it?

READ = False

if READ:
    tf.enable_eager_execution()

    import tensorflow_data_validation as tfdv

    train_uri = example_gen.outputs['examples'].get()[0].uri

    tfrecord_filenames = tf.data.Dataset.list_files(train_uri+"*")

    # Create a `TFRecordDataset` to read these files
    dataset = tf.data.TFRecordDataset(tfrecord_filenames, compression_type="GZIP")

    # Iterate over the first 3 records and decode them using a TFExampleDecoder
    for tfrecord in dataset.take(3):
        serialized_example = tfrecord.numpy()
        example = tfdv.TFExampleDecoder().decode(serialized_example)
        pprint(example)
        print("")


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
Read sample of the transformed data
Since TTF v1.15 is used, you need to enable eager execution. However, this causes problems to subsequent steps.
Can you write Python code for it?

READ = False

if READ:
    tf.enable_eager_execution()

    train_uri = transform.outputs['transformed_examples'].get()[1].uri
    tfrecord_filenames = [os.path.join(train_uri, name)
                          for name in os.listdir(train_uri)]
    dataset = tf.data.TFRecordDataset(tfrecord_filenames, compression_type="GZIP")
    decoder = tfdv.TFExampleDecoder()
    for tfrecord in dataset.take(3):
        serialized_example = tfrecord.numpy()
        example = decoder.decode(serialized_example)
        pprint(example)
        print("")


I am working on the file "00_Miscellaneous/tfx/tfx-pipelines/02-census-tfx-pipeline.ipynb".
The first task is:
Training Pipeline
Can you write Python code for it?

import tfx
import tensorflow as tf
print("Tensorflow Version:", tf.__version__)
print("TFX Version:", tfx.__version__)


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Arrays can also be 'sliced', grabbing a range of values.  Let's look at the first three elements
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
But it can also print data types other than strings:
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
But not both at the same time, unless you want one of the symbols to be part of the string.
Can you write Python code for it?

"He's a geologist"

'She asked, "Are you crazy?"'


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Furthermore, there are types that function as collections. The main ones are:

List
Tuple
Dictionary

Python types can be:

Mutable: allow the contents of the variables to be changed.
Immutable: do not allow the contents of variables to be changed.

The most common types and routines are implemented in the form of builtins, ie, they are always available at runtime, without the need to import any library.
Lists
Very often in a programming language, one wants to keep a group of similar items together. Python does this using a data type called lists.
Can you write Python code for it?

planets = ['Mercury', 'Venus', 'Earth', 'Mars', 'Jupiter', 'Saturn', 'Uranus', 'Neptune']


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Great.  So a has the values 1 through 5 and now so does b.  Now that I have a backup of a, I can change its values without worrying about losing data (or so I may think!).
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Here, the 3rd element of a has been changed to 17.  Now let's check on b.
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
In addition to interpolation operator %, there is the string method and function format().
Examples:
Can you write Python code for it?

# Parameters are identified by order
print('The area of square with side {0} is {1}'.format(5, 5*5))

# Parameters are identified by name
print('{greeting}, it is {hour:02d}:{minute:02d}AM'.format(greeting='Hi', hour=7, minute=30))

# Builtin function format()
print('Pi =', format(math.pi, '.15f'))


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
It is possible to invert strings by using a negative step:
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Just like the onumbers we're familiar with, you can assign a string to a variable
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Learn More
There are a lot of resources online to learn more about using NumPy and other libraries. Well done are Lectures on scientific computing with Python.
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open("./css/sg2.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Lists can be sliced in the same way that the strings.
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Lists could collect objects  of any type, including other lists.
Whitespace in Python
Python uses indents and whitespace to group statements together.  To write a short loop in C, you might use:
for (i = 0, i < 5, i++){
   printf("Stress and strain\n");
}

Python does not use curly braces like C, so the same program as above is written in Python as follows:
Can you write Python code for it?

for i in range(3):
    print("Stress and strain")


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Note here, the slice is inclusive on the front end and exclusive on the back, so the above command gives us the values of vals[0], vals[1] and vals[2], but not vals[3].
Assigning Array Variables
One of the strange little quirks/features in Python that often confuses people comes up when assigning and comparing arrays of values.  Here is a quick example.  Let's start by defining a 1-D array called a:
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Now, we can try again to change a value in a and see if the changes are also seen in c.
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
OK, so we have an array a, with the values 1 through 5.  I want to make a copy of that array, called b, so I'll try the following:
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Python has a number of defined operators for handling numbers through arithmetic calculations, logic operations (that test whether a condition is true or false) or bitwise processing (where the numbers are processed in binary form).
Arithmetic Operations

Sum (+)
Difference (-)
Multiplication (*)
Division (/)
Integer Division (//)
Module (%): returns the remainder of the division.
Power (**): can be used to calculate the root, through fractional exponents (eg 100 0.5).
Positive (+)
Negative (-)

Logical Operations

Less than (<)
Greater than (>)
Less than or equal to (<=)
Greater than or equal to (>=)
Equal to (==)
Not equal to (!=)

Libraries and modules
Python has a huge number of libraries included with the distribution. To keep things simple, most of these variables and functions are not accessible from a normal Python interactive session. Instead, you have to import the name. For example, there is a math module containing many useful functions. To access, say, the square root function, you have to import the math library and than call functions from imported library.
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Python uses a zero-based index, so let's look at the first and last element in the array myvals
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Scientific Python Environment
Python is a high-level open-source language.  But the Scientific Python Environment is inhabited by many packages or libraries that provide useful things like array operations, plotting functions, and much more. We can import libraries of functions to expand the capabilities of Python in our programs.
OK! We'll start by importing a few libraries to help us out. In most cases we will use pylab environment, which contains most important parts of numpy and matplotlib. To import it to our notebook, we can use %pylab magic with inline option for inline graphics.
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Slicing strings
Slices of strings can be obtained by adding indexes between brackets after a  string.
Python indexes:

Start with zero.
Count from the end if they are negative.
Can be defined as sections, in the form [start: end + 1: step]. If not set the start, it will be considered as zero. If not set end + 1, it will be considered the size of the object. The step (between characters), if not set, is 1.
Can you write Python code for it?

greeting

greeting[7]

greeting[2:6]

greeting[18:]

greeting[7:17:2]


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
So what just happened? We just imported most of numpy and matplotlib into current workspace, so their functions are from now available to use.  So if we want to use the numpy function linspace, for instance, we can call it by writing:
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
The -1 element of a list is the last element:
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
The operator % is used for string interpolation. The interpolation is more efficient in use of memory than the conventional concatenation.
Symbols used in the interpolation:

%s: string.
%d: integer.
%o: octal.
%x: hexacimal.
%f: real.
%e: real exponential.
%%: percent sign.

Symbols can be used to display numbers in various formats.
Can you write Python code for it?

# Zeros left
print('Now is %02d:%02d.' % (16, 30))

# Real (The number after the decimal point specifies how many decimal digits )
print('Percent: %.1f%%, Exponencial:%.2e' % (5.333, 0.00314))

# Octal and hexadecimal
print('Decimal: %d, Octal: %o, Hexadecimal: %x' % (10, 10, 10))


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
The print function is often used for printing character strings:
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
To learn new functions available to you, visit the NumPy Reference page. If you are a proficient MATLAB user, there is a wiki page that should prove helpful to you: NumPy for Matlab Users

Slicing Arrays
In NumPy, you can look at portions of arrays in the same way as in Matlab, with a few extra tricks thrown in.  Let's take an array of values from 1 to 5.
Can you write Python code for it?


I am working on the file "00_Python_Crash_Course.ipynb".
The first task is:
Variables
You can define variables using the equals (=) sign:
Can you write Python code for it?

width = 20
length = 30
area = length*width
area


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Además de la media y la mediana existen otras medidas de centralidad como la moda. La moda es simplemente el valor más frecuente en un conjunto de datos.
Varianza
Mide la dispersión de un conjunto de valores. Es cero para un conjunto de valores idénticos.
$$V(\mathbf{x}) = E[(\mathbf{x} - \mu)^2] = \frac{1}{n} \sum_{i=1}^n (\mu - x_i)^2 \tag{1}$$
Donde $\mu$ es la media de $\mathbf{x}$,
Al igual que pasa con la media, se suele distinguir entre la varianza de la población $\sigma^2$ y la varianza de una muestra $s^2$. Pero a diferencia de la media se puede probar que la fórmula anterior para $s^2$ subestima el valor de $\sigma^2$. Es posible obtener un mejor estimador con una ligera modificación de la ecuación 1, la cual consiste en dividir por $n-1$ en vez de $n$. En la práctica esta corrección solo es importante para cuando se tienen pocos datos, a medida que los datos aumentan no hay mucha diferencia entre usar una u otra fórmula. Esto es facil de ver intuitivamente, para $n=2$ la diferencia entre $n$ y $n-1$ es relativamente grande, mientras que para $n=100$ esta diferencia es mucho más pequeña.
A partir de la definición de varianza, vemos que para calcularla debemos repetir la operación $(\mu - x_i)^2$ para cada valor en $\mathbf{x}$ es decir debemos iterar, esto es una operación muy común en programación. Una de las formas más comunes de iteración es mediante el bucle for, por ejemplo para iterar sobre todos los valores contenidos en la lista num hacemos:
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Al igual que con las listas, no es necesario crear un diccionario vacío y luego agregar valores, podríamos haber creado el diccionario de forma directa. En el siguiente ejemplo tenemos los nombres de 5 ciudades y sus poblaciones.
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Algo para prestar atención, el nombre de la variable que le pasamos a nuestra función no tiene nada que ver con el nombre del argumento de la función. Es decir nosotros le pasamos a la función un objeto llamado num, internamente calcular_media le asigna a ese objeto el alias a. Técnicamente decimos que a es una variable local de la función calcular_media, es decir a solo existe dentro de esta función, fuera de ella a podría no existir o ser una variable completamente distinta.
Can you write Python code for it?

calcular_media([1, 3, 4]), a  # dentro de la función `a` contiene una lista, fuera el valor 6


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Así como iteramos sobre los valores contenidos en un lista es posible iterar sobre los valores contenidos en un diccionario:
Can you write Python code for it?

for k, v in pob.items():
    print(k, v)


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Como veníamos diciendo al trabajar con Python es común repetir una frase como la siguiente:

Creamos una lista vacía.
Iteramos a largo de algún iterable como una lista o un rango.
Guardamos valores en la lista originalmente vacía.

Este patrón es tan común que Python ofrece una versión alternativa, la cual es considerada por la mayoría de Pitonistas como más simple y clara, aunque al principio puede que no se vea como simple y clara :-(
Esta versión alternativa se llama list comprehension, o comprensión por listas. Y luce de la siguiente forma:
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Con estas dos nuevas funciones que hemos aprendido podemos re-escribir el cálculo de una media de la siguiente forma:
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Dado que las tuplas y las listas parecidas es común que surga la pregunta ¿Cúando es conveniente usar una y cuando la otra?
Al dar los primeros pasos con Python esta elección no es demasiado relevante, ya que en muchos casos el uso es más o menos indistinto. De todas formas es bueno saber que uno de los criterios usados para elegir entre listas y tuplas tiene que ver con el tipo de elementos que vamos a almacenar. Si bien tanto las tuplas como las listas pueden contener elementos de distinto tipo (por ej strings e int). Es más común que se use listas para almacenar elementos del mismo tipo y tuplas para elementos de distinto tipo.
Operaciones sobre strings
En Python es posible iterar, rebanar e indexar strings como si fueran listas
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Dentro de Jupyter también se puede acceder a la ayuda escribiendo ? o  ?? luego de la función y presionado enter.
Todo muy bien hasta ahora, pero ¿Qué pasaría si inadvertidamente le pasáramos una lista vacía a nuestra función?
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Desviación estándar
La desviación estándar es la raíz cuadrada de la varianza, en muchos problemas teóricos resulta más fácil manipular varianzas que desviaciones estándar, pero en general resulta más simple interpretar las desviaciones estándar ya que las unidades son las mismas que las de los datos.
$$\sigma = \sqrt{V(x)}$$
Diccionarios
Los diccionarios son similares a las listas, una de las diferencias es que para indexar una lista solo es posible usar enteros, en cambio para indexar un diccionario podemos usar otros tipos.
Un diccionario es una forma de mapear un conjuntos de índices (llamados claves o keys) y un conjunto de valores. A cada par clave-valor (key-value) se le suele llamar item.
Podemos crear un diccionario vacio de la siguiente forma:
Can you write Python code for it?

diccionario = {}  # o también diccionario = dict()
type(diccionario)


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
El bloque else es opcional
Can you write Python code for it?

if True:  # esto es siempre cierto
    print('hola')


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
El código que acabamos de escribir no difiere mucho de lo que podríamos haber realizado con una calculadora, primero tenemos que ingresar los numeros a mano y además tenemos que saber exactamente la cantidad de números que estamos sumando para luego saber por que cantidad dividir. Usando un lenguaje de programación podemos hacer algo bastante más cómodo. Pero para ello tenemos que aprender un par de conceptos nuevos.
Listas
Las listas de Python son un tipo de objeto que permite almacenar otros objetos, por ejemplo números. Gran parte de la programación en Python implica crear y manipular listas.
La sintaxis para crear listas en Python es [..., ..., ...]:
Can you write Python code for it?

lista = [] # crea una lista vacia
lista

num = [1, 2, 3, 4, 5, 6]
num, type(num)


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
El docstring no solo puede ser leído directamente del código, si no que puede ser usado por Python y por varias herramientas externas. Por ejemplo la función help() de Python ofrece ayuda sobre una función al usuario "mostrando el docstring". Por ejemplo, si quisieramos ver qué hace la función len()
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
El operador in funciona tanto con listas como con diccionarios, aunque internamente no funciona de igual forma para ambas estructuras de datos. En el caso de las listas el tiempo de demora de esta operación  es proporcional a la longitud de la lista, mientras más elementos estén contenidos en la lista más tardaremos en obtener una respuesta. Por el contrario el tiempo de demora es casi constante para los diccionarios. Quienes tengan mayor curiosidad sobre como se logra esta característica pueden leer sobre Hash table.
Si quisieramos saber si un valor está contenido en un diccionario deberíamos hacer:
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
El siguiente patrón, donde iteramos a la largo de una lista de enteros es tan común en Python (y otros lenguajes) que existe varias funciones que facilitan esta tarea. Una de ellas es range:
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
En la celda anterior estamos diciendole a Python: "Tome el valor 2 súmele el valor 3 y guárdelo en la variable que se llama a".
Esto lo podemos comprobar al ejecutar:
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
En la primer linea podemos ver que para definir una función necesitamos la palabra reservada def. Le llamamos reservada ya que tiene significado especial para Python y no deberíamos usar ese nombre para otra cosa que no sea definir funciones.
A continuación de def, y en la misma linea, va el nombre de la función, que puede ser el que más nos guste (siempre que respetemos las reglas para nombrar funciones, que son las mismas que para nombrar variables y que ya vimos más arriba).
Entre paréntesis los nombres de los argumentos de la función. En este ejemplo a.
Cerramos la primer línea con dos puntos :.
La siguientes líneas contienen las operaciones que realiza la función, en este caso es una sola, pero potencialmente podrían ser decenas o cientos de líneas.
Por último tenemos otra palabra reservada return seguida del valor que devuelve la función, es decir del resultado de la función. Esta ultima línea es opcional ya que es posible definir funciones que no devuelvan valor alguno (ya veremos ejemplos).

Es importante notar un detalle, muy particular de Python, todo el contenido de la función (lineas 2 y 3) está escrito usando una sangría de 4 espacios, la sangría es obligatoria y le indica a Python cual es el cuerpo de la función. Los 4 espacios son solo una convención,  podrían ser 2, 3, 8, etc, lo importante es respetar la misma cantidad.
Una vez definida una función se la puede llamar usando su nombre y pasándole los argumentos necesarios.
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
En los ejemplos anteriores hemos realizado algunas operaciones con variables, pero no las hemos guardado en ningún lado. Puede no ser obvio al principio pero para poder hacer tareas relativamente complejas es necesario poder guardar variables y darles nombres (o identificador). De esa forma podemos realizar una operación, como por ejemplo 2 + 2 y guardar el resultado bajo algún nombre para luego usarlo cuando lo necesitemos.
Los nombres de las variables en Python siguen ciertas reglas que es necesario respetar.

Deben comenzar con letras o con un _
La letras pueden corresponderse con los caracteres a-z, A-Z
También podemos usar numeros 0-9 (solo que no como primer carácter).

Python 3 permite además usar otros caracteres, como por ejemplo letras griegas, la ñ, caracteres acentuados, etc; aunque el uso de este tipo de caracteres para nombrar variables no esta muy difundido aún.
Por convención, los nombres de las variables comienzan con una letra minúscula al igual que los nombres de las funciones (ver más adelante). Los nombres que comienzan con mayúsculas están reservados para las clases, en este curso no vamos a estudiar clases.
Para asignarle un valor a una variable se usa el símbolo =
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Entonces, la forma general de un docstring sería:
Can you write Python code for it?

"""
Descripción, ¿para qué sirve la función? ¿qué tarea realiza?

Parametros
----------
nombre : tipo
    descripción

Resultado
----------
nombre : tipo
    descripción
""";


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Es posible revisar más de una condición, en el siguiente ejemplo debido a que la expresión a la derecha de if evalua como falso se evalua la siguiente condición, que en este caso evalua como verdadera.
Can you write Python code for it?

if 1 > 2:  # esto es falso
    print('hola')
elif 2 > 1:  # esto es cierto, solo se ejecuta si if evalua falso
    print('hello')
else:
    print('chau')


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Esta es una función bastante más compleja que calcular_media(). Veamos línea por línea qué es lo que hace.

La primer línea (luego del docstring) ordena de menor a mayor los elementos dentro de "lista".
La segunda computa la longitud de lista.
La tercera calcula un índice, usamos int para asegurarnos que el resultado es un entero.
Esta línea evalúa True solo si lista_len es impar ¿Podés darte cuenta por qué? (tip: probá ese bloque de código en una celda separada).
Si lista_len es par entonces se ejecutará el bloque else. La razón de tener este bloque es que si la cantidad de elementos es par NO es posible obtener el valor "del medio", entonces la mediana la computamos como un promedio de los dos valores "del medio".
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Esta nueva forma de calcular la media tiene varias ventajas. Una de ellas es que resulta más automática que la anterior, cada vez que cambiemos los valores contenidos en la lista num podremos ejecutar la celda anterior y obtener el valor de la media. Este es un buen momento para probar que esto es cierto.
El código de la celda anterior es mucho más general que nuestra primer versión del cómputo de la media. Pero tiene un inconveniente, veamos. Es común al escribir código que necesitemos repetir una operación muchas veces, una solución simple sería copiar y pegar el código cada vez que necesitemos calcular una media. El problema con esta aproximación es que es tediosa y muy propensa a cometer errores, pensemos que operaciones más complejas podrían requerir de cientos de líneas de código, no una sola como el ejemplo de la media. Una forma de resolver nuestro problema sería crear una función que calcule la media, de tal forma que podamos llamar a esa función cada vez que la necesitemos. En Python crear funciones es muy simple, veamos:
Can you write Python code for it?

def calcular_media(a):
    res = sum(a) / len(a)
    return res


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Esto no solo sirve para funciones que ya vienen con Python, si no que el mismo mecanismo es usado para funciones definidas por los usuarios.
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Existen muchos otros "tipos" de variables en Python, incluso los usuarios avanzados pueden crear sus propios tipos de variables! Por ahora estos tipos son suficientes. Tener variables es útil por que con ellas podemos hacer operaciones, por ejemplo operaciones matemáticas.
Can you write Python code for it?

1 + 1 # esto devuelve un entero

1. + 1  # esto devuelve un float

2 / 1  # en Python3 la división siempre devuelve floats

"42" + 1  # esta operación no tiene sentido y por lo tanto Python devuelve un error!

'4' * 4  # esta operación SI tiene sentido en Python, es lo que esperabas?

1 > 2  # acá comparamos variables y obtenemos un booleano

2 == 2  # ojo que el operador "igualdad" es "==" y no "="


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Incluso podemos actualizar el valor de las variables, por ej.
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Jupyter (lo que estamos usando para mostrar este documento y ejecutar el código), también permite acceder al docstring usando shift + TAB
Can you write Python code for it?

calcular_media  # seleccioná esta celda y presioná shift + tab


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Las listas son un tipo particular de lo que se conoce genéricamente como estructuras de datos, es decir una forma particular de organizar y almacenar datos de forma que sea conveniente trabajar con esos datos. Las listas de Python son convenientes por que permiten no solo almacenar valores sino realizar muchas otras operaciones de forma sencilla. Una operación común es preguntarle a una lista cual es su longitud, es decir cuantos elementos contiene. Esto se hace con la función len().
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Los errores son parte central de la programación y hay que acostumbrarse a cometerlos ya que es así como se avanza en la escritura de un programa. Al producirse errores, Python entrega mensajes que son muy informativos y por lo tanto útiles para solucionar el error, por lo que es muy beneficioso aprender a interpretarlos y prestarles mucha atención cuando ocurren.
El proceso de corrección de errores de un programa se llama debugging y es quizá una de las tareas más demandantes al escribir código. Python fue pensado como un lenguaje fácil de leer debido a que en general uno pasa más tiempo leyendo código (para arreglar los errores) que escribiéndolo. Mitad broma, mitad en serio se dice que si el debugging es el proceso por el cual se eliminan errores la programación debe ser el proceso por el cual se introducen los errores.
Nota: en muchos lenguajes de programación (como C/C++ o Fortran) antes de poder asignar valores a variables es necesario declararlas. Declararlas, quiere decir que tenemos que indicar que nuestro programa usará una variable de nombre tal que será del tipo cual. Recién una vez declarada la variable podemos asignarle valores concretos. En Python esto no es necesario, además en Python está permitido cambiar el tipo de una variable, por ejemplo podemos usar la variable a que hacía referencia a enteros int para referirnos a un string. El nombre técnico de esto es el de tipado dinámico (el tipo de una variable no es fijo, si no dinámico).
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Mediana
La media es una buena descripción si los datos que estamos midiendo son más o menos similares, pero puede ofrecer una visión muy distorsionada si los datos no son muy similares entre si, por ejemplo como puede suceder con los ingresos, algunas personas apenas ganan unos pocos pesos mientras que otras acumulan millones por mes.
La mediana es el número que separa un conjunto de datos en una mitad superior y otra inferior. La mediana es una medida más robusta que la media a valores extremos. Veamos un ejemplo para el conjunto $\mathcal{D} = {1,2,3,4,5}$, la media y la mediana es igual a 3. Si ahora agregaramos a este conjunto un nuevo valor, por ejemplo 100. La media pasará a ser $\eqsim 19,2$ mientras que la mediana apenas cambiará a 3.5 $\left ( \frac{3 + 4}{2} \right )$.
Para calcular la mediana necesitamos algunos conceptos nuevos. Una característica de las listas de Python es que es posible acceder a los elementos contenidos en ellas mediante índices. Los índices deben ser enteros, empiezan en 0 y terminan en len(.)-1.
Can you write Python code for it?

lista = [5, 4, 3, 2, 1]
lista[0]  # el cero-ésimo elemento de la lista

lista[2]

lista[5]  # este índice no existe en este caso y Python nos lo indica con un error


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
No solo es posible usar variables para guardar valores, también podemos operar con ellas. Por ejemplo podemos preguntar si nuestra variable es mayor que cierto valor.
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Obtenemos un error! Antes de seguir leyendo intentemos entender cual es la causa del error y en que parte del código se encuentra.

Como se puede ver el problema se debe a que la división por 0 no está definida. Este es un tipo de error que no se debe a que metimos la pata al programar el cálculo de la media, se debe a una imposibilidad matemática. Es, si lo pensamos, un error esperado algo que es razonable que ocurra, en la jerga pitónica decimos que es una excepción. Python permite lidiar con las excepciones de forma elegante (en vez de dejar que el programa falle horriblemente). Veamos primero el código y luego la explicación.
Can you write Python code for it?

def calcular_media2(a):
    """
    Calcula la media partir de una lista.

    Parametros
    ----------
    a : lista
        lista que contiene los valores a promediar

    Resultado
    ----------
    res : float
        La media de los valores contenidos en `a`
        Devuelve una advertencia si `a` está vacía.
    """
    try:
        res = sum(a) / len(a)
        return res
    except ZeroDivisionError:
        print('la lista está vacía, por favor use una lista con números')

calcular_media2([])


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Otra convención en Python es escribir las funciones incluyendo un docstring. Que no es más que un string que se escribe a partir de la segunda línea de la función y que Python ignora por completo al ejecutar la función. La información contenida en el docstring no es para Python, es para los humanos que lo usan! El docstring consiste en una explicación sobre las operaciones que realiza una función, qué valores espera la función como entrada y qué valores devuelve (si es que devuelve algo). Los docstring se escriben usando comillas triples """ esto permite que el docstring tenga varias lineas. El estilo exacto de los docstring varía, pero la mayoría de las funciones en Python tratan de mantener un mismo estilo para sus docstrings. Un ejemplo de docstring sería:
Can you write Python code for it?

def calcular_media(a):
    """
    Calcula la media partir de una lista.

    Parametros
    ----------
    a : lista
        Contiene los valores a promediar

    Resultado
    ----------
    res : float
        La media de los valores contenidos en `a`
    """
    res = sum(a) / len(a)
    return res


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Python ofrece muchos métodos para operar sobre strings.
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Si ahora quisiramos saber la población de cordoba, bastaría con escribir:
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
También es posible realziar otro tipo de operaciones que normalmente consideramos definidaas solo para números, es posible sumar strings
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
También podemos importar una función pero cambiandole el nombre
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Todavía faltan un par de detalles, pero los vamos a ver directamente en la función. Es común al aprender una lenguaje encontrarse con código que contiene elementos desconocidos o "frases" que no sabíamos que eran posibles o legales.
Can you write Python code for it?

def mediana(lista):
    """
    Ejercicio escribir docstring!
    """
    lista_ordenada = sorted(lista)
    lista_len = len(lista_ordenada)
    idx = int((lista_len - 1) / 2)

    if lista_len % 2:
        return lista_ordenada[idx]
    else:
        return (lista_ordenada[idx] + lista_ordenada[idx + 1]) / 2


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Tuplas
Las tuplas son similares a las listas, podemos usarla para almacenar otros objetos
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Una diferencia con las listas es que las tuplas son inmutables, es decir no pueden ser modificadas una vez creadas
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Una operación muy común es querer iterar sobre una lista de valores, modificarlos y guardarlos en otra lista, por ejemplo:
Can you write Python code for it?

num_2 = []
for i in num:
    num_2.append(i**2)
    #print(num_2)
num_2


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Una operación que suele ser útil es preguntar si una clave existe en un diccionario, esto se logra con el operador in:
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
Y podemos tomar rebanadas
Can you write Python code for it?


I am working on the file "00_Python_y_EDA.ipynb".
The first task is:
range usa la sintaxis [start,stop,step] a fin de generar enteros desde start, hasta stop (sin incluirlo) y opcionalente de a step pasos (por defecto 1). start es también opcional en cuyo caso empezará en 0. Como pueden ver la sintaxis es similar a lo que ya vimos con la rebanadas de una lista. La diferencia es que las rebanadas operan sobre una lista existente y la función de range es la de generar un objeto que contiene enteros.
Can you write Python code for it?

d = []
for i in range(0, 9):
    d.append(i ** 2)
d


I am working on the file "00_TSC_NN_Keras.ipynb".
The first task is:

Data Info

Spliting the train data as train and validation set

Can you write Python code for it?

X_train, X_val, Y_train, Y_val = train_test_split(train['features'], train['labels'], test_size=0.3, random_state=0)
X_test, Y_test = test['features'], test['labels']

n_train = X_train.shape[0]
n_val = X_val.shape[0]
n_test = X_test.shape[0]
image_shape = X_train.shape[1], X_train.shape[2]
n_channels = X_train.shape[3]
n_classes = np.unique(train['labels']).size

print('Train data size:\t\t\t', n_train)
print('Validation data size:\t\t\t', n_val)
print('test data size:\t\t\t\t', n_test)
print('Image shape:\t\t\t\t', image_shape)
print('Number of color channels in image:\t', n_channels)
print('Number of classes:\t\t\t', n_classes)


I am working on the file "00_TSC_NN_Keras.ipynb".
The first task is:

Loading the Data

Can you write Python code for it?

train_data = 'data/train.p'
test_data = 'data/test.p'

with open(train_data, 'rb') as f:
    train = pickle.load(f)

with open(test_data, 'rb') as f:
    test = pickle.load(f)


I am working on the file "00_TSC_NN_Keras.ipynb".
The first task is:

Build the Model with Keras

Can you write Python code for it?

model = Sequential()
model.add(Dense(256, activation='relu', input_shape=(32*32*3,)))
#model.add(Dense(800, activation='relu',input_shape=(1555,)))
model.add(Dense(43, activation='softmax'))

model.summary()

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

history = model.fit(X_train_norm, Y_train_cat, batch_size=64, epochs=20, verbose=1, validation_data=(X_val_norm, Y_val_cat))

history.history['val_acc'][-1]


I am working on the file "00_TSC_NN_Keras.ipynb".
The first task is:

Data Normalization

Process all the data as close as mean 0.0 and standard deviation 1.0.

Can you write Python code for it?

def normalize(arr):
    arr = arr.astype('float32')
    return (arr - np.mean(arr))/np.std(arr)

X_train_norm = normalize(X_train_flat)
X_val_norm = normalize(X_val_flat)
X_test_norm = normalize(X_test_flat)

def print_info(st, arr_1, arr_2):
    print('{} Data: Before normalization : type: {}, mean: {}, std: {}. After processing, type: {}, mean: {}, std: {}'. format(st, arr_1.dtype, round(np.mean(arr_1),2), round(np.std(arr_1),2), arr_2.dtype, round(np.mean(arr_2),2), round(np.std(arr_2),2)))

print_info('Train', X_train_flat, X_train_norm)
print_info('Valdation', X_val_flat, X_val_norm)
print_info('Test', X_test_flat, X_test_norm)


I am working on the file "00_TSC_NN_Keras.ipynb".
The first task is:

Model Evaluation

Can you write Python code for it?

score, acc = model.evaluate(X_test_norm, Y_test_cat, batch_size=64, verbose=0)

print('Score:\t', score)
print('Acc:\t{}%'.format(round(acc*100)))


I am working on the file "00_TSC_NN_Keras.ipynb".
The first task is:

Predicted Classes

Can you write Python code for it?

Y_predd = model.predict_classes(X_test_norm, batch_size=64, verbose=0)

plot_images(X_test, Y_test, Y_pred)


I am working on the file "00_TSC_NN_Keras.ipynb".
The first task is:

Reshape All the Data

Can you write Python code for it?

def reshape(arr):
    return arr.reshape(-1, image_shape[0]*image_shape[1]*n_channels)

X_train_flat = reshape(X_train)
X_val_flat = reshape(X_val)
X_test_flat = reshape(X_test)

def print_info(st, arr_1, arr_2):
    print('{} data shape before reshape: {}, and after reshape: {}'.format(st, arr_1.shape, arr_2.shape))

print_info('Train', X_train, X_test_flat)
print_info('Validation', X_val, X_val_flat)
print_info('Test', X_test, X_test_flat)


I am working on the file "00_TSC_NN_Keras.ipynb".
The first task is:

Some Random Image

Before normalization

Can you write Python code for it?

trc = rn.sample(range(n_test), 16)

def plot_images(arr_1, arr_2, pred=False):
    fig, axes = plt.subplots(4, 4, figsize=(10,10))
    fig.subplots_adjust(hspace=0.3, wspace=0.3)
    for i, ax in enumerate(axes.flat):
        if len(arr_1.shape) == 2:
            ax.imshow(arr_1[trc[i]].reshape(32,32,3))
            ax.set_xlabel('true:{}'.format(arr_2[trc[i]]))
        else:
            ax.imshow(arr_1[trc[i]])
            ax.set_xlabel('true:{}, pred:{}'.format(arr_2[trc[i]], pred[trc[i]]))
        ax.set_xticks([])
        ax.set_yticks([])

plot_images(X_train_flat, Y_train)


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Let us calculate a Monte Carlo present value estimate and estimates for the major Greeks.
Can you write Python code for it?

am_put.present_value()

am_put.delta()

am_put.vega()


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Let us define the relevant market by 2 Python dictionaries, the correlation between the two risk factors and a valuation environment.
Can you write Python code for it?

risk_factors = {'gbm_1': me_1, 'gbm_2' : me_2}
correlations = [['gbm_1', 'gbm_2', -0.4]]
positions = {'put' : put, 'call' : call}

val_env = dx.market_environment('general', dt.datetime(2016, 1, 1))
val_env.add_constant('frequency', 'W')
val_env.add_constant('paths', 10000)
val_env.add_constant('starting_date', val_env.pricing_date)
val_env.add_constant('final_date', val_env.pricing_date)
val_env.add_curve('discount_curve', r)


risk_factors = {'gbm_1': me_1, 'gbm_2' : me_2}
correlations = [['gbm_1', 'gbm_2', -0.4]]
positions = {'put' : put, 'call' : call}

val_env = dx.market_environment('general', dt.datetime(2015, 1, 1))
val_env.add_constant('frequency', 'W')
val_env.add_constant('paths', 10000)
val_env.add_constant('starting_date', val_env.pricing_date)
val_env.add_constant('final_date', val_env.pricing_date)
val_env.add_curve('discount_curve', r)


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Next, the model object for the first risk factor, based on the geometric Brownian motion (Black-Scholes-Merton (1973) model).
Can you write Python code for it?

gbm_1 = dx.geometric_brownian_motion('gbm_1', me_1)


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Options Portfolio
Modeling
In a portfolio context, we need to add information about the model class(es) to be used to the market environments of the risk factors.
Can you write Python code for it?

me_1.add_constant('model', 'gbm')
me_2.add_constant('model', 'gbm')


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Risk Factor Models
The first step is to define a model for the risk-neutral discounting.
Can you write Python code for it?


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Second risk factor with higher volatility. We overwrite the respective value in the market environment.
Can you write Python code for it?

me_2 = dx.market_environment('me_2', me_1.pricing_date)
me_2.add_environment(me_1)  # add complete environment
me_2.add_constant('volatility', 0.5)  # overwrite value

gbm_2 = dx.geometric_brownian_motion('gbm_2', me_2)

pdf = pd.DataFrame(gbm_2.get_instrument_values(), index=gbm_2.time_grid)

pdf.ix[:, :10].plot(legend=False, figsize=(10, 6))


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Simulation and Valuation
Now, we can get the position values for the portfolio via the get_values method.
Can you write Python code for it?


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
The valuation object instantiation.
Can you write Python code for it?

me_opt.add_constant('strike', me_3.get_constant('initial_value'))

sabr_call = dx.valuation_mcs_european_single(
            name='sabr_call',
            underlying=sabr,
            mar_env=me_opt,
            payoff_func='np.maximum(maturity_value - strike, 0)')


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
These are used to define the derivatives portfolio.
Can you write Python code for it?

port = dx.derivatives_portfolio(
            name='portfolio',  # name
            positions=positions,  # derivatives positions
            val_env=val_env,  # valuation environment
            risk_factors=risk_factors, # relevant risk factors
            correlations=correlations, # correlation between risk factors
            parallel=False)  # parallel valuation


port = dx.derivatives_portfolio(
            name='portfolio',  # name
            positions=positions,  # derivatives positions
            val_env=val_env,  # valuation environment
            risk_factors=risk_factors, # relevant risk factors
            correlations=correlations, parallel=True)  # correlation between risk factors


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
To compose a portfolio consisting of our just defined options, we need to define derivatives positions. Note that this step is independent from the risk factor model and option model definitions. We only use the market environment data and some additional information needed (e.g. payoff functions).
Can you write Python code for it?

put = dx.derivatives_position(
            name='put',
            quantity=2,
            underlyings=['gbm_1'],
            mar_env=me_opt,
            otype='American single',
            payoff_func='np.maximum(strike - instrument_values, 0)')

call = dx.derivatives_position(
            name='call',
            quantity=3,
            underlyings=['gbm_2'],
            mar_env=me_opt,
            otype='European single',
            payoff_func='np.maximum(maturity_value - strike, 0)')


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Valuation Models
Based on the risk factors, we can then define derivatives models for valuation. To this end, we need to add at least one (the maturity), in general two (maturity and strike), parameters to the market environments.
Can you write Python code for it?

me_opt = dx.market_environment('me_opt', me_1.pricing_date)
me_opt.add_environment(me_1)
me_opt.add_constant('maturity', dt.datetime(2017, 6, 30))
me_opt.add_constant('strike', 110.)


me_opt = dx.market_environment('me_opt', me_1.pricing_date)
me_opt.add_environment(me_1)
me_opt.add_constant('maturity', dt.datetime(2016, 6, 30))
me_opt.add_constant('strike', 110.)


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Valuation and Greek estimation for this option.
Can you write Python code for it?

eur_call.present_value()

eur_call.delta()

eur_call.vega()


I am working on the file "00_dx_quickstart.ipynb".
The first task is:
Via the get_statistics methods delta and vega values are provided as well.
Can you write Python code for it?


I am working on the file "00_getting_started.ipynb".
The first task is:
As you can see, calling the function gives the same results as calculated above.
As further improvements we will add a legend to the plot, labels to the $x$ and $y$ axes and a title. Python functions in matplotlib and numpy take many arguments. Pressing [shift][Tab] when typing in a code cell will open an help box with a lot of useful information on the function arguments.
Can you write Python code for it?

plt.plot(x, my_first_function(x), 'r--')
# Write label on x axis
plt.xlabel('x-axis')
# Write label on y axis
plt.ylabel('y-axis')
# Write title
plt.title('First Python Figure')
# Add legend
plt.legend('f')


I am working on the file "00_getting_started.ipynb".
The first task is:
Modules only have to be imported once in a Jupyter session. After the import, any plotting function may be called from any code cell as plt.function. In the following we are plotting the data table:
x | y
---|---
 0 | 1
 1 | 2
 2 | 3
 3 | 5
 4 | 2
 5 | 6
Can you write Python code for it?

import matplotlib.pyplot as plt
# make sure we see it on this notebook
%matplotlib inline
plt.plot([1, 2, 3, 5, 2, 6])
plt.show()


I am working on the file "00_getting_started.ipynb".
The first task is:
Of course, we can add more information to a print statement! A line starting with # is a comment, the intepreter will just ignore it. It is useful to add comments, to remind both yourself and others what you were trying to do when writing the code ;)
Can you write Python code for it?

a = 6 # Assign value to a
b = 2 # Assign value to b
c = a * b # Perform operation
# Now print everything out!
print('a =', a)
print('b =', b)
print('c =', c)


I am working on the file "00_getting_started.ipynb".
The first task is:
Python modules: arrays with numpy
OK, let's plot something more interesting than just a bunch of points.
The function we want to plot is:
\begin{equation}
  f(x) = 10x^3\mathrm{e}^{-x^2} + \frac{\sin(x^5)}{\cos(x^3)}
\end{equation}
How do we do it? We could of course create a table of $x$,$y$ pairs and feed it to the plt.plot function as in the example above, but that's too tedious.
The module [numpy] comes to the rescue. First of all, we import numpy and give it the name np:
Can you write Python code for it?


I am working on the file "00_getting_started.ipynb".
The first task is:
Python modules: basic plotting with matplotlib
When you first install Python and launch an instance of the interpreter (the program running the commands) you cannot do very much. Luckily, Python was designed to be highly extensible using independent packages called modules. Much of the functionality that makes Python appealing is provided via modules.
A module is a collection of functions designed to perform a very specific set of tasks.
For example, the [matplotlib] module provides the functions needed to perform plotting in Python.
To use it we need to import it. There are many different ways to import a module, for the moment we import only the plottig part of matplotlib and call it plt. The second command ensures that the plots are generated in this Notebook and not in a separate window.
Can you write Python code for it?

import matplotlib.pyplot as plt
# make sure we see it on this notebook
%matplotlib inline


I am working on the file "00_getting_started.ipynb".
The first task is:
Python provides the print function to write output to the screen.
Can you write Python code for it?

a = 6
b = 2
c = a * b
print(a)
print(b)
print(c)


I am working on the file "00_getting_started.ipynb".
The first task is:
The next step is to evaluate the function at all the $x$ points listed in the x array and save the results in another array, which we call y:
Can you write Python code for it?


I am working on the file "00_getting_started.ipynb".
The first task is:
When programming, you can (and should) use variables to hold values, so that you can reference them later in your code. Pressing [shift][Enter] will always trigger evaluation of a code cell and print out the results.
Can you write Python code for it?