I am working on the file "01_Mine/MachineLearning/NeuroEvolution-Flappy-Bird-master/Jupyter Notebook/Flappy.ipynb".
The first task is:
Draw the network
We are using the graphviz in order to draw the network that we generate with the NEAT
Can you write Python code for it?

import graphviz

def draw_net(config, genome, view=False, filename=None, node_names=None, show_disabled=True, prune_unused=False,
             node_colors=None, fmt='svg'):
    """ Receives a genome and draws a neural network with arbitrary topology. """
    # Attributes for network nodes.
    if graphviz is None:
        warnings.warn("This display is not available due to a missing optional dependency (graphviz)")
        return

    if node_names is None:
        node_names = {}

    assert type(node_names) is dict

    if node_colors is None:
        node_colors = {}

    assert type(node_colors) is dict

    node_attrs = {
        'shape': 'circle',
        'fontsize': '9',
        'height': '0.2',
        'width': '0.2'}

    dot = graphviz.Digraph(format=fmt, node_attr=node_attrs)

    inputs = set()
    for k in config.genome_config.input_keys:
        inputs.add(k)
        name = node_names.get(k, str(k))
        input_attrs = {'style': 'filled',
                       'shape': 'box'}
        input_attrs['fillcolor'] = node_colors.get(k, 'lightgray')
        dot.node(name, _attributes=input_attrs)

    outputs = set()
    for k in config.genome_config.output_keys:
        outputs.add(k)
        name = node_names.get(k, str(k))
        node_attrs = {'style': 'filled'}
        node_attrs['fillcolor'] = node_colors.get(k, 'lightblue')

        dot.node(name, _attributes=node_attrs)

    if prune_unused:
        connections = set()
        for cg in genome.connections.values():
            if cg.enabled or show_disabled:
                connections.add((cg.in_node_id, cg.out_node_id))

        used_nodes = copy.copy(outputs)
        pending = copy.copy(outputs)
        while pending:
            new_pending = set()
            for a, b in connections:
                if b in pending and a not in used_nodes:
                    new_pending.add(a)
                    used_nodes.add(a)
            pending = new_pending
    else:
        used_nodes = set(genome.nodes.keys())

    for n in used_nodes:
       ...


I am working on the file "01_Mine/MachineLearning/NeuroEvolution-Flappy-Bird-master/Jupyter Notebook/Flappy.ipynb".
The first task is:
MLP playing the game
This cell will show how the MLP plays the game
Can you write Python code for it?

#Esta celda visualiza el mejor genoma tras la neuroevolución

clock = pygame.time.Clock()
pygame.font.init()
font = pygame.font.SysFont("Arial", 50)
bird = FlappyBird_GAME()

while (not bird.isDead()):
    clock.tick(60)
    nnInput = bird.calculateInput()
    output =  mlp_classifier.predict(np.column_stack((nnInput[0],nnInput[1],nnInput[2],nnInput[3],nnInput[4])))
    if output[0][0] > output[0][1]:
        bird.tick(True)
    else:
        bird.tick(False)

print(bird.TotalDistance())


I am working on the file "01_Mine/MachineLearning/NeuroEvolution-Flappy-Bird-master/Jupyter Notebook/Flappy.ipynb".
The first task is:
Results
The results obtained classifying the test data with the trained MLP.
Can you write Python code for it?

print("MLP predictions:\n%s\n" % (metrics.classification_report(Y_test, mlp_classifier.predict(X_test))))


I am working on the file "01_Mine/MachineLearning/tensorflow-examples_nb/0_Prerequisite/mnist_dataset_intro.ipynb".
The first task is:
A next_batch function that can iterate over the whole dataset and return only the desired fraction of the dataset samples (in order to save memory and avoid to load the entire dataset).
Can you write Python code for it?

# Get the next 64 images array and labels
batch_X, batch_Y = mnist.train.next_batch(64)


I am working on the file "01_Model-Training.ipynb".
The first task is:
Plot training and test accuracy
Can you write Python code for it?

# use model for inference to get test accuracy
y_test_pred = model.predict(x_test)
y_test_pred = np.argmax(y_test_pred, axis=1)

print ('\n Summary of the precision, recall, F1 score for each class:')
print (sklearn.metrics.classification_report(y_test, y_test_pred))

print ('\n Confusion matrix: ')
print (sklearn.metrics.confusion_matrix(y_test, y_test_pred))

import matplotlib.pyplot as plt
plt.plot(history.history['val_acc'], label="Test Accuracy")
plt.plot(history.history['acc'], label="Training Accuracy")
plt.legend()

# save model
model.save("myModel.h5")


I am working on the file "01_NB_IntroductionPython.ipynb".
The first task is:
2. Define a function that returns the sum of two given numbers
Can you write Python code for it?

def sumTwoGivenNumbers(a, b):
    c = a+b
    return c

sumTwoGivenNumbers(23, 7)


I am working on the file "01_NB_IntroductionPython.ipynb".
The first task is:
2. Print the 'help' for a particular function (try also using the IPython shell)
Can you write Python code for it?


I am working on the file "01_NB_IntroductionPython.ipynb".
The first task is:
3. IPython Magic Commands
The magics we are going to use the most: %%time, %%timeit, and %matplotlib. More info about magics here.
Can you write Python code for it?


I am working on the file "01_NB_IntroductionPython.ipynb".
The first task is:
3. Importing Python modules
To use an external module in Python you can import it into your script by using the import statement.
Can you write Python code for it?

import math #imports the whole module. It requires to use the prefix "math." before calling each function
from math import * #imports all functions and variable within the current namespace (it does not require to use the prefix)
from math import cos, pi #imports only the specific functions and variables listed


I am working on the file "01_NB_IntroductionPython.ipynb".
The first task is:
4. Define a function that returns sum or the average of a variable set of numbers
Can you write Python code for it?

def opMultipleNumbers(*args, **kwargs):
    if kwargs.get("operation") == "sum":
         c = sum(args)
    if kwargs.get("operation") == "avg":
         c = sum(args)/len(args)
    return c

opMultipleNumbers(1,2,3,4,operation="avg")


I am working on the file "01_NB_IntroductionPython.ipynb".
The first task is:
Exercise
1. Define a function that returns a list of numbers that sum up to 1

Generate a list of random numbers
Sum up all the numbers in the list
Take each number in the initial list and divide it by the sum of all numbers
Print the new list
Sum up all the values in the new list. It should sum up to 1
Can you write Python code for it?

randomNumbers = [random.random() for i in range(1,10)]
s = sum(randomNumbers)
r = [ i/s for i in randomNumbers ]
print (r)
print(sum(r))


I am working on the file "01_NB_IntroductionPython.ipynb".
The first task is:
Some useful to get help in Jupyter Notebook:
* Hold Shift + Tab over a function inside a code cell.
* Function name with a question mark after it.
Can you write Python code for it?


I am working on the file "01_NB_IntroductionPython.ipynb".
The first task is:
Try using the different import options to describe the following expression:
    x = cos(2 *pi)
Can you write Python code for it?

x = math.cos(2 * math.pi)
x = cos(2 * pi)
print(x)


I am working on the file "01_Raices_de_ecuaciones_de_una_variable/01_Biseccion.ipynb".
The first task is:
Ejemplo 2
Encontrar la raiz de
\begin{equation}
    y = x^{3} + 4 x^{2} - 10
\end{equation}
la raíz posiblemente se encuentre en el intervalo $[1,2]$
Can you write Python code for it?

def f(x):
    y = x**3 + 4*x**2 - 10
    return y

intervalo_de_raiz(f, 1, 2)

intervalo_de_raiz(f, 1, 1.5)

intervalo_de_raiz(f, 1.25, 1.5)

biseccion(f, 1, 2)


I am working on the file "01_Raices_de_ecuaciones_de_una_variable/01_Biseccion.ipynb".
The first task is:
Implementación no vectorizada
Seudocódigo
pascal
function biseccion(f(x), x_inferior, x_superior)
    x_raiz_actual = raiz(x_inferior, x_superior)
    error_permitido = 0.000001
    while(True)
        x_raiz_anterior = x_raiz_actual
        x_inferior, x_superior = intervalo_de_raiz(f(x), x_inferior, x_superior)
        x_raiz_actual = raiz(x_inferior, x_superior)
        if x_raiz_actual != 0
            error_relativo = abs((x_raiz_actual - x_raiz_anterior)/x_raiz_actual)*100
        end if
        if error_relativo < error_permitido
            exit
        end if
    end while
    mostrar x_raiz_actual
end function
o también
pascal
function biseccion(f(x), x_inferior, x_superior)
    x_raiz_actual = raiz(x_inferior, x_superior)
    for 1 to maxima_iteracion do
        x_raiz_anterior = x_raiz_actual
        x_inferior, x_superior = intervalo_de_raiz(f(x), x_inferior, x_superior)
        x_raiz_actual = raiz(x_inferior, x_superior)
    end for
    mostrar x_raiz_actual
end function
Can you write Python code for it?

def biseccion(f, x_inferior, x_superior):
    print("{0:2s}\t{1:12s}\t{2:12s}\t{3:12s}\t{4:16s}".format(' i', 'x inferior', 'x superior', 'raiz', 'error relativo %'))
    x_raiz_actual = raiz(x_inferior, x_superior)
    i = 0
    print("{0:2d}\t{1:12.10f}\t{2:12.10f}\t{3:12.10f}\t{4:16s}".format(i, x_inferior, x_superior, x_raiz_actual, '????????????????'))
    error_permitido = 0.000001
    while True:
        x_raiz_anterior = x_raiz_actual
        x_inferior, x_superior = intervalo_de_raiz(f, x_inferior, x_superior)
        x_raiz_actual = raiz(x_inferior, x_superior)
        if x_raiz_actual != 0:
            error_relativo = abs((x_raiz_actual - x_raiz_anterior)/x_raiz_actual)*100
        i = i + 1
        print("{0:2d}\t{1:12.10f}\t{2:12.10f}\t{3:12.10f}\t{4:16.13f}".format(i, x_inferior, x_superior, x_raiz_actual, error_relativo))
        if (error_relativo < error_permitido) or (i>=20):
            break
    print('\nx =', x_raiz_actual)


I am working on the file "01_Raices_de_ecuaciones_de_una_variable/02_Falsa_posicion.ipynb".
The first task is:
Ejemplo 4
Encontrar la raiz de
\begin{equation}
    y = \sin{10 x} + \cos{3 x}
\end{equation}
la raíz posiblemente se encuentre en el intervalo $[14,16]$
Can you write Python code for it?


I am working on the file "01_Raices_de_ecuaciones_de_una_variable/02_Falsa_posicion.ipynb".
The first task is:
Implementación no vectorizada
pascal
function falsa_posicion(f(x), x_inferior, x_superior)
    x_raiz_actual = raiz(f(x), x_inferior, x_superior)
    error_permitido = 0.000001
    while(True)
        x_raiz_anterior = x_raiz_actual
        x_inferior, x_superior = intervalo_de_raiz(f(x), x_inferior, x_superior)
        x_raiz_actual = raiz(f(x), x_inferior, x_superior)
        if x_raiz_actual != 0
            error_relativo = abs((x_raiz_actual - x_raiz_anterior)/x_raiz_actual)*100
        end if
        if error_relativo < error_permitido
            exit
        end if
    end while
    mostrar x_raiz_actual
end function
o también
pascal
function falsa_posicion(f(x), x_inferior, x_superior)
    x_raiz_actual = raiz(f(x), x_inferior, x_superior)
    for 1 to maxima_iteracion do
        x_raiz_anterior = x_raiz_actual
        x_inferior, x_superior = intervalo_de_raiz(f(x), x_inferior, x_superior)
        x_raiz_actual = raiz(f(x), x_inferior, x_superior)
    end for
    mostrar x_raiz_actual
end function
Can you write Python code for it?

def falsa_posicion(f, x_inferior, x_superior):
    print("{0:2s}\t{1:12s}\t{2:12s}\t{3:12s}\t{4:16s}".format(' i', 'x inferior', 'x superior', 'raiz', 'error relativo %'))
    x_raiz_actual = raiz(f, x_inferior, x_superior)
    i = 0
    print("{0:2d}\t{1:12.10f}\t{2:12.10f}\t{3:12.10f}\t{4:16s}".format(i, x_inferior, x_superior, x_raiz_actual, '????????????????'))
    error_permitido = 0.000001
    while True:
        x_raiz_anterior = x_raiz_actual
        x_inferior, x_superior = intervalo_de_raiz(f, x_inferior, x_superior)
        x_raiz_actual = raiz(f, x_inferior, x_superior)
        if x_raiz_actual != 0:
            error_relativo = abs((x_raiz_actual - x_raiz_anterior)/x_raiz_actual)*100
        i = i + 1
        print("{0:2d}\t{1:12.10f}\t{2:12.10f}\t{3:12.10f}\t{4:16.13f}".format(i, x_inferior, x_superior, x_raiz_actual, error_relativo))
        if (error_relativo < error_permitido) or (i>=20):
            break
    print('\nx =', x_raiz_actual)


I am working on the file "01_Raices_de_ecuaciones_de_una_variable/06_Secante.ipynb".
The first task is:
Ejemplo 2
Encontrar la raiz de
\begin{equation}
    y = x^{5} + x^{3} + 3
\end{equation}
usar $x_{-1} = 0$ y $x_{0} = -1$
Can you write Python code for it?

def f(x):
    # f(x) = x^5 + x^3 + 3
    y = x**5 + x**3 + 3
    return y

diferencia_atras(f, 0, -1)

raiz(f, 0, -1)

secante(f, 0, -1)


I am working on the file "01_Raices_de_ecuaciones_de_una_variable/06_Secante.ipynb".
The first task is:
Ejemplo 3
Encontrar la raiz de
\begin{equation}
    y = x^{5} + x^{3} + 3
\end{equation}
usar $x_{-1} = 0$ y $x_{0} = -0.5$
Can you write Python code for it?


I am working on the file "01_Raices_de_ecuaciones_de_una_variable/07_Secante_modificada.ipynb".
The first task is:
Ejemplo 3
Encontrar la raiz de
\begin{equation}
    y = x^{5} + x^{3} + 3
\end{equation}
usar $x = 0$ y $\Delta x = -1$
Can you write Python code for it?


I am working on the file "01_Raices_de_ecuaciones_de_una_variable/07_Secante_modificada.ipynb".
The first task is:
Implementación no vectorizada
Seudocódigo
pascal
function secante_modificada(f(x), x_0, delta_x)
    x_actual = x_0
    error_permitido = 0.000001
    while(True)
        x_anterior = x_actual
        x_actual = raiz(f(x), x_anterior, delta_x)
        if x_raiz_actual != 0
            error_relativo = abs((x_raiz_actual - x_raiz_anterior)/x_raiz_actual)*100
        end if
        if error_relativo < error_permitido
            exit
        end if
    end while
    mostrar x_actual
end function
o también
pascal
function secante_modificada(f(x), x_0, delta_x)
    x_actual = x_0
    for 1 to maxima_iteracion do
        x_anterior = x_actual
        x_actual = raiz(f(x), x_anterior, delta_x)
    end for
    mostrar x_actual
end function
Can you write Python code for it?

def secante_modificada(f, x_0, delta_x):
    print("{0:s} \t {1:15s} \t {2:15s} \t {3:15s}".format('i', 'x anterior', 'x actual', 'error relativo %'))
    x_actual = x_0
    i = 0
    print("{0:d} \t {1:15s} \t {2:.15f} \t {3:15s}".format(i, '???????????????', x_actual, '???????????????'))
    error_permitido = 0.000001
    while True:
        x_anterior = x_actual
        x_actual = raiz(f, x_anterior, delta_x)
        if x_actual != 0:
            error_relativo = abs((x_actual - x_anterior)/x_actual)*100
        i = i + 1
        print("{0:d} \t {1:.15f} \t {2:.15f} \t {3:15.11f}".format(i, x_anterior, x_actual, error_relativo))
        if (error_relativo < error_permitido) or (i>=20):
            break
    print('\nx =', x_actual)


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
1. Define Dataset Metadata

CSV file header and defaults
Numeric and categorical feature names
Target feature name
Unused columns
Can you write Python code for it?

HEADER = ['key','x','y','alpha','beta','target']
HEADER_DEFAULTS = [[0], [0.0], [0.0], ['NA'], ['NA'], [0.0]]

NUMERIC_FEATURE_NAMES = ['x', 'y']

CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY = {'alpha':['ax01', 'ax02'], 'beta':['bx01', 'bx02']}
CATEGORICAL_FEATURE_NAMES = list(CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.keys())

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES

TARGET_NAME = 'target'

UNUSED_FEATURE_NAMES = list(set(HEADER) - set(FEATURE_NAMES) - {TARGET_NAME})

print("Header: {}".format(HEADER))
print("Numeric Features: {}".format(NUMERIC_FEATURE_NAMES))
print("Categorical Features: {}".format(CATEGORICAL_FEATURE_NAMES))
print("Target: {}".format(TARGET_NAME))
print("Unused Features: {}".format(UNUSED_FEATURE_NAMES))


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
2. Define Data Input Function

Input csv file name
Load pandas Dataframe
Apply feature processing
Return a function that returns (features, target) tensors
Can you write Python code for it?

def process_dataframe(dataset_df):

    dataset_df["x_2"] = np.square(dataset_df['x'])
    dataset_df["y_2"] = np.square(dataset_df['y'])
    dataset_df["xy"] = dataset_df['x'] * dataset_df['y']
    dataset_df['dist_xy'] =  np.sqrt(np.square(dataset_df['x']-dataset_df['y']))

    return dataset_df

def generate_pandas_input_fn(file_name, mode=tf.estimator.ModeKeys.EVAL,
                             skip_header_lines=0,
                             num_epochs=1,
                             batch_size=100):

    df_dataset = pd.read_csv(file_name, names=HEADER, skiprows=skip_header_lines)

    x = df_dataset[FEATURE_NAMES].copy()
    if PROCESS_FEATURES:
        x = process_dataframe(x)

    y = df_dataset[TARGET_NAME]

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    num_threads=1

    if MULTI_THREADING:
        num_threads=multiprocessing.cpu_count()
        num_epochs = int(num_epochs/num_threads) if mode == tf.estimator.ModeKeys.TRAIN else num_epochs

    pandas_input_fn = tf.estimator.inputs.pandas_input_fn(
        batch_size=batch_size,
        num_epochs= num_epochs,
        shuffle=shuffle,
        x=x,
        y=y,
        target_column=TARGET_NAME
    )

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file: {}".format(file_name))
    print("Dataset size: {}".format(len(df_dataset)))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Thread Count: {}".format(num_threads))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    return pandas_input_fn

features, target = generate_pandas_input_fn(file_name=TRAIN_DATA_FILE)()
print("Feature read from DataFrame: {}".format(list(features.keys())))
print("Target read from DataFrame: {}".format(target))


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
4. Create an Estimator
a. Define an Estimator Creation Function

Get dense (numeric) columns from the feature columns
Convert categorical columns to indicator columns
Create Instantiate a DNNRegressor estimator given dense + indicator feature columns + params
Can you write Python code for it?

def create_estimator(run_config, hparams):

    feature_columns = list(get_feature_columns().values())

    dense_columns = list(
        filter(lambda column: isinstance(column, feature_column._NumericColumn),
               feature_columns
        )
    )

    categorical_columns = list(
        filter(lambda column: isinstance(column, feature_column._VocabularyListCategoricalColumn) |
                              isinstance(column, feature_column._BucketizedColumn),
                   feature_columns)
    )

    indicator_columns = list(
            map(lambda column: tf.feature_column.indicator_column(column),
                categorical_columns)
    )

    estimator_feature_columns = dense_columns + indicator_columns

    estimator = tf.estimator.DNNRegressor(

        feature_columns= estimator_feature_columns,
        hidden_units= hparams.hidden_units,

        optimizer= tf.train.AdamOptimizer(),
        activation_fn= tf.nn.elu,
        dropout= hparams.dropout_prob,

        config= run_config
    )

    print("")
    print("Estimator Type: {}".format(type(estimator)))
    print("")

    return estimator


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
5. Train the Estimator
Can you write Python code for it?

train_input_fn = generate_pandas_input_fn(file_name= TRAIN_DATA_FILE,
                                      mode=tf.estimator.ModeKeys.TRAIN,
                                      num_epochs=hparams.num_epochs,
                                      batch_size=hparams.batch_size)

if not RESUME_TRAINING:
    shutil.rmtree(model_dir, ignore_errors=True)

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Estimator training started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

estimator.train(input_fn = train_input_fn)

time_end = datetime.utcnow()
print(".......................................")
print("Estimator training finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Estimator training elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
7. Prediction
Can you write Python code for it?

import itertools

predict_input_fn = generate_pandas_input_fn(file_name=TEST_DATA_FILE,
                                      mode= tf.estimator.ModeKeys.PREDICT,
                                      batch_size= 5)

predictions = estimator.predict(input_fn=predict_input_fn)
values = list(map(lambda item: item["predictions"][0],list(itertools.islice(predictions, 5))))
print()
print("Predicted Values: {}".format(values))


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
Steps to use the TF Estimator APIs

Define dataset metadata
Define data input function to read the data from Pandas dataframe + apply feature processing
Create TF feature columns based on metadata + extended feature columns
Instantiate an estimator with the required feature columns & parameters
Train estimator using training data
Evaluate estimator using test data
Perform predictions
Save & Serve the estimator
Can you write Python code for it?

MODEL_NAME = 'reg-model-01'

TRAIN_DATA_FILE = 'data/train-data.csv'
VALID_DATA_FILE = 'data/valid-data.csv'
TEST_DATA_FILE = 'data/test-data.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
MULTI_THREADING = False


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
b. Set hyper-parameter values (HParams)
Can you write Python code for it?

hparams  = tf.contrib.training.HParams(
    num_epochs = 100,
    batch_size = 500,
    hidden_units=[8, 4],
    dropout_prob = 0.0)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.estimator.RunConfig().replace(model_dir=model_dir)
print("Model directory: {}".format(run_config.model_dir))
print("Hyper-parameters: {}".format(hparams))


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
c. Instantiate the estimator
Can you write Python code for it?

estimator = create_estimator(run_config, hparams)


I am working on the file "01_Regression/02.0 - TF Regression Model - Estimator APIs + Pandas.ipynb".
The first task is:
c. Serve the Saved Model
Can you write Python code for it?

import os

saved_model_dir = export_dir + "/" + os.listdir(path=export_dir)[-1]

print(saved_model_dir)

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="predict"
)

output = predictor_fn({'csv_rows': ["0.5,1,ax01,bx02", "-0.5,-1,ax02,bx02"]})
print(output)


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
1. Define Dataset Metadata

CSV file header and defaults
Numeric and categorical feature names
Target feature name
Unused columns
Can you write Python code for it?


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
2. Define Data Input Function

Input csv files name pattern
Use TF file queue mechanism to read files
Parse CSV lines to feature tensors
Apply feature processing
Return (features, target) tensors
Can you write Python code for it?

def process_features(features):

    features["x_2"] = tf.square(features['x'])
    features["y_2"] = tf.square(features['y'])
    features["xy"] = tf.multiply(features['x'], features['y']) # features['x'] * features['y']
    features['dist_xy'] =  tf.sqrt(tf.squared_difference(features['x'],features['y']))

    return features

def csv_input_fn(file_names, mode=tf.estimator.ModeKeys.EVAL,
                 skip_header_lines=0,
                 num_epochs=1,
                 batch_size=200):

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    input_file_names = tf.train.match_filenames_once(pattern=file_names)

    filename_queue = tf.train.string_input_producer(
        input_file_names, num_epochs=num_epochs, shuffle=shuffle)

    reader = tf.TextLineReader(skip_header_lines=skip_header_lines)

    _, rows = reader.read_up_to(filename_queue, num_records=batch_size)

    row_columns = tf.expand_dims(rows, -1)
    columns = tf.decode_csv(row_columns, record_defaults=HEADER_DEFAULTS)
    features = dict(zip(HEADER, columns))

    # Remove unused columns
    for column in UNUSED_FEATURE_NAMES:
        features.pop(column)

    num_threads=1

    if MULTI_THREADING:
        num_threads=multiprocessing.cpu_count()

    if shuffle:
        features = tf.train.shuffle_batch(
            features,
            batch_size,
            min_after_dequeue=2 * batch_size + 1,
            capacity=batch_size * 10,
            num_threads=multiprocessing.cpu_count(),
            enqueue_many=True,
            allow_smaller_final_batch=True
        )
    else:
        features = tf.train.batch(
            features,
            batch_size,
            capacity=batch_size * 10,
            num_threads=multiprocessing.cpu_count(),
            enqueue_many=True,
            allow_smaller_final_batch=True
        )

    target = features.pop(TARGET_NAME)

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s):...


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
3. Define Feature Columns
The input numeric columns are assumed to be normalized (or have the same scale). Otherwise, a normlizer_fn, along with the normlisation params (mean, stdv or min, max) should be passed to tf.feature_column.numeric_column() constructor.
Can you write Python code for it?

def get_feature_columns():

    CONSTRUCTED_NUMERIC_FEATURES_NAMES = ['x_2', 'y_2', 'xy', 'dist_xy']
    all_numeric_feature_names = NUMERIC_FEATURE_NAMES + CONSTRUCTED_NUMERIC_FEATURES_NAMES

    numeric_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
                       for feature_name in all_numeric_feature_names}

    categorical_column_with_vocabulary = \
        {item[0]: tf.feature_column.categorical_column_with_vocabulary_list(item[0], item[1])
         for item in CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.items()}

    feature_columns = {}

    if numeric_columns is not None:
        feature_columns.update(numeric_columns)

    if categorical_column_with_vocabulary is not None:
        feature_columns.update(categorical_column_with_vocabulary)

    # add extended features (crossing, bucektization, embedding)

    feature_columns['alpha_X_beta'] = tf.feature_column.crossed_column(
        [feature_columns['alpha'], feature_columns['beta']], 4)

    return feature_columns

feature_columns = get_feature_columns()
print("Feature Columns: {}".format(feature_columns))


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
5. Run Experiment
a. Define Experiment Function
Can you write Python code for it?

def generate_experiment_fn(**experiment_args):

    def _experiment_fn(run_config, hparams):

        validation_monitor = EarlyStoppingHook(early_stopping_rounds=hparams.early_stopping_rounds)

        train_input_fn = lambda: csv_input_fn(
            TRAIN_DATA_FILES_PATTERN,
            mode = tf.estimator.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size
        )

        eval_input_fn = lambda: csv_input_fn(
            VALID_DATA_FILES_PATTERN,
            mode=tf.estimator.EVAL,
            num_epochs=1,
            batch_size=hparams.batch_size
            #batch_size=300
        )

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            eval_steps=None,
            **experiment_args
        )

    return _experiment_fn


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
6. Evaluate the Model
Can you write Python code for it?

TRAIN_SIZE = 12000
VALID_SIZE = 3000
TEST_SIZE = 5000

train_input_fn = lambda: csv_input_fn(file_names= TRAIN_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TRAIN_SIZE)

valid_input_fn = lambda: csv_input_fn(file_names= VALID_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= VALID_SIZE)

test_input_fn = lambda: csv_input_fn(file_names= TEST_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TEST_SIZE)

estimator = create_estimator(run_config, hparams)

train_results = estimator.evaluate(input_fn=train_input_fn)
train_rmse = round(math.sqrt(train_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Train RMSE: {} - {}".format(train_rmse, train_results))
print("############################################################################################")

valid_results = estimator.evaluate(input_fn=valid_input_fn, steps=1)
valid_rmse = round(math.sqrt(valid_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Valid RMSE: {} - {}".format(valid_rmse,valid_results))
print("############################################################################################")

test_results = estimator.evaluate(input_fn=test_input_fn, steps=1)
test_rmse = round(math.sqrt(test_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Test RMSE: {} - {}".format(test_rmse, test_results))
print("############################################################################################")


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
7. Prediction
Can you write Python code for it?

import itertools

predict_input_fn = lambda: csv_input_fn(file_names= TEST_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.PREDICT,
                                      batch_size= 5)

predictions = estimator.predict(input_fn=predict_input_fn)
values = list(map(lambda item: item["predictions"][0],list(itertools.islice(predictions, 5))))
print()
print("Predicted Values: {}".format(values))


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
Steps to use the TF Experiment APIs

Define dataset metadata
Define data input function to read the data from csv files + feature processing
Create TF feature columns based on metadata + extended feature columns
Define an estimator creation function with the required feature columns & parameters
Run an Experiment with learn_runner to train, evaluate, and export the model
Evaluate the model using test data
Perform predictions and serving the saved model
Can you write Python code for it?

MODEL_NAME = 'reg-model-02'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.csv'
VALID_DATA_FILES_PATTERN = 'data/valid-*.csv'
TEST_DATA_FILES_PATTERN = 'data/test-*.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
MULTI_THREADING = True


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
b. Set HParam and RunConfig
Can you write Python code for it?

hparams  = tf.contrib.training.HParams(
    num_epochs = 100,
    batch_size = 500,
    hidden_units=[8, 4],
    early_stopping_rounds = 1,
    dropout_prob = 0.0)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=480, # to evaluate after each 20 epochs => (12000/500) * 20
    tf_random_seed=19830610,
    model_dir=model_dir
)

print("Model directory: {}".format(run_config.model_dir))
print("Hyper-parameters: {}".format(hparams))
print("")


I am working on the file "01_Regression/03.0 - TF Regression Model - Experiment APIs + CSV Files.ipynb".
The first task is:
d. Run Experiment via learn_runner
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

learn_runner.run(
    experiment_fn=generate_experiment_fn(

        export_strategies=[make_export_strategy(
            csv_serving_input_fn,
            exports_to_keep=1
        )]
    ),
    run_config=run_config,
    schedule="train_and_evaluate",
    hparams=hparams
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input + JSON Serving.ipynb".
The first task is:
2. Define Data Input Function

Input csv files name pattern
Use TF Dataset APIs to read and process the data
Parse CSV lines to feature tensors
Apply feature processing
Return (features, target) tensors

a. parsing and preprocessing logic
Can you write Python code for it?

def parse_csv_row(csv_row):

    columns = tf.decode_csv(csv_row, record_defaults=HEADER_DEFAULTS)
    features = dict(zip(HEADER, columns))

    for column in UNUSED_FEATURE_NAMES:
        features.pop(column)

    target = features.pop(TARGET_NAME)

    return features, target

def process_features(features):

    features["x_2"] = tf.square(features['x'])
    features["y_2"] = tf.square(features['y'])
    features["xy"] = tf.multiply(features['x'], features['y']) # features['x'] * features['y']
    features['dist_xy'] =  tf.sqrt(tf.squared_difference(features['x'],features['y']))

    return features


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input + JSON Serving.ipynb".
The first task is:
3. Define Feature Columns
The input numeric columns are assumed to be normalized (or have the same scale). Otherwise, a normlizer_fn, along with the normlisation params (mean, stdv or min, max) should be passed to tf.feature_column.numeric_column() constructor.
Can you write Python code for it?

def extend_feature_columns(feature_columns):

    # crossing, bucketizing, and embedding can be applied here

    feature_columns['alpha_X_beta'] = tf.feature_column.crossed_column(
        [feature_columns['alpha'], feature_columns['beta']], 4)

    return feature_columns

def get_feature_columns():

    CONSTRUCTED_NUMERIC_FEATURES_NAMES = ['x_2', 'y_2', 'xy', 'dist_xy']
    all_numeric_feature_names = NUMERIC_FEATURE_NAMES.copy()

    if PROCESS_FEATURES:
        all_numeric_feature_names += CONSTRUCTED_NUMERIC_FEATURES_NAMES

    numeric_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
                       for feature_name in all_numeric_feature_names}

    categorical_column_with_vocabulary = \
        {item[0]: tf.feature_column.categorical_column_with_vocabulary_list(item[0], item[1])
         for item in CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.items()}

    feature_columns = {}

    if numeric_columns is not None:
        feature_columns.update(numeric_columns)

    if categorical_column_with_vocabulary is not None:
        feature_columns.update(categorical_column_with_vocabulary)

    if EXTEND_FEATURE_COLUMNS:
        feature_columns = extend_feature_columns(feature_columns)

    return feature_columns

feature_columns = get_feature_columns()
print("Feature Columns: {}".format(feature_columns))


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input + JSON Serving.ipynb".
The first task is:
b. Set HParam and RunConfig
Can you write Python code for it?

TRAIN_SIZE = 12000
NUM_EPOCHS = 1000
BATCH_SIZE = 500
NUM_EVAL = 10
CHECKPOINT_STEPS = int((TRAIN_SIZE/BATCH_SIZE) * (NUM_EPOCHS/NUM_EVAL))

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    hidden_units=[8, 4],
    dropout_prob = 0.0)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)
print("Required Evaluation Steps:", NUM_EVAL)
print("That is 1 evaluation step after each",NUM_EPOCHS/NUM_EVAL," epochs")
print("Save Checkpoint After",CHECKPOINT_STEPS,"steps")


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input + JSON Serving.ipynb".
The first task is:
b. data pipeline input function
Can you write Python code for it?

def csv_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL,
                 skip_header_lines=0,
                 num_epochs=None,
                 batch_size=200):

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    file_names = tf.matching_files(files_name_pattern)

    dataset = data.TextLineDataset(filenames=file_names)
    dataset = dataset.skip(skip_header_lines)

    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    #useful for distributed training when training on 1 data file, so it can be shareded
    #dataset = dataset.shard(num_workers, worker_index)

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row))

    if PROCESS_FEATURES:
        dataset = dataset.map(lambda features, target: (process_features(features), target))

    #dataset = dataset.batch(batch_size) #??? very long time
    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()

    features, target = iterator.get_next()
    return features, target

features, target = csv_input_fn(files_name_pattern="")
print("Feature read from CSV: {}".format(list(features.keys())))
print("Target read from CSV: {}".format(target))


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input + JSON Serving.ipynb".
The first task is:
d. Run Experiment via learn_runner
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

learn_runner.run(
    experiment_fn=generate_experiment_fn(

        export_strategies=[
            make_export_strategy(
            json_serving_input_fn,
            exports_to_keep=1,
            as_text=True)
        ]
    ),
    run_config=run_config,
    schedule="train_and_evaluate",
    hparams=hparams
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
1. Define Dataset Metadata

CSV file header and defaults
Numeric and categorical feature names
Target feature name
Unused columns
Can you write Python code for it?

HEADER = ['key','x','y','alpha','beta','target']
HEADER_DEFAULTS = [[0], [0.0], [0.0], ['NA'], ['NA'], [0.0]]

NUMERIC_FEATURE_NAMES = ['x', 'y']

CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY = {'alpha':['ax01', 'ax02'], 'beta':['bx01', 'bx02']}
CATEGORICAL_FEATURE_NAMES = list(CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.keys())

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES

TARGET_NAME = 'target'

UNUSED_FEATURE_NAMES = list(set(HEADER) - set(FEATURE_NAMES) - {TARGET_NAME})

print("Header: {}".format(HEADER))
print("Numeric Features: {}".format(NUMERIC_FEATURE_NAMES))
print("Categorical Features: {}".format(CATEGORICAL_FEATURE_NAMES))
print("Target: {}".format(TARGET_NAME))
print("Unused Features: {}".format(UNUSED_FEATURE_NAMES))


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
2. Define Data Input Function

Input csv files name pattern
Use TF Dataset APIs to read and process the data
Parse CSV lines to feature tensors
Apply feature processing
Return (features, target) tensors

a. parsing and preprocessing logic
Can you write Python code for it?


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
3. Define Feature Columns
The input numeric columns are assumed to be normalized (or have the same scale). Otherise, a normlizer_fn, along with the normlisation params (mean, stdv) should be passed to tf.feature_column.numeric_column() constructor.
Can you write Python code for it?


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
4. Define an Estimator Creation Function

Get dense (numeric) columns from the feature columns
Convert categorical columns to indicator columns
Create Instantiate a DNNRegressor estimator given dense + indicator feature columns + params
Can you write Python code for it?

def create_estimator(run_config, hparams):

    feature_columns = list(get_feature_columns().values())

    dense_columns = list(
        filter(lambda column: isinstance(column, feature_column._NumericColumn),
               feature_columns
        )
    )

    categorical_columns = list(
        filter(lambda column: isinstance(column, feature_column._VocabularyListCategoricalColumn) |
                              isinstance(column, feature_column._BucketizedColumn),
                   feature_columns)
    )

    indicator_columns = list(
            map(lambda column: tf.feature_column.indicator_column(column),
                categorical_columns)
    )

    estimator = tf.estimator.DNNRegressor(

        feature_columns= dense_columns + indicator_columns ,
        hidden_units= hparams.hidden_units,

        optimizer= tf.train.AdamOptimizer(),
        activation_fn= tf.nn.elu,
        dropout= hparams.dropout_prob,

        config= run_config
    )

    print("")
    print("Estimator Type: {}".format(type(estimator)))
    print("")

    return estimator


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
5. Define Serving Funcion
Can you write Python code for it?

def csv_serving_input_fn():

    SERVING_HEADER = ['x','y','alpha','beta']
    SERVING_HEADER_DEFAULTS = [[0.0], [0.0], ['NA'], ['NA']]

    rows_string_tensor = tf.placeholder(dtype=tf.string,
                                         shape=[None],
                                         name='csv_rows')

    receiver_tensor = {'csv_rows': rows_string_tensor}

    row_columns = tf.expand_dims(rows_string_tensor, -1)
    columns = tf.decode_csv(row_columns, record_defaults=SERVING_HEADER_DEFAULTS)
    features = dict(zip(SERVING_HEADER, columns))

    return tf.estimator.export.ServingInputReceiver(
        process_features(features), receiver_tensor)


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
6. Run Experiment
a. Define Experiment Function
Can you write Python code for it?

def generate_experiment_fn(**experiment_args):

    def _experiment_fn(run_config, hparams):

        train_input_fn = lambda: csv_input_fn(
            files_name_pattern=TRAIN_DATA_FILES_PATTERN,
            mode = tf.contrib.learn.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size
        )

        eval_input_fn = lambda: csv_input_fn(
            files_name_pattern=VALID_DATA_FILES_PATTERN,
            mode=tf.contrib.learn.ModeKeys.EVAL,
            num_epochs=1,
            batch_size=hparams.batch_size
        )

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            eval_steps=None,
            **experiment_args
        )

    return _experiment_fn


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
7. Evaluate the Model
Can you write Python code for it?

TRAIN_SIZE = 12000
VALID_SIZE = 3000
TEST_SIZE = 5000

train_input_fn = lambda: csv_input_fn(files_name_pattern= TRAIN_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TRAIN_SIZE)

valid_input_fn = lambda: csv_input_fn(files_name_pattern= VALID_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= VALID_SIZE)

test_input_fn = lambda: csv_input_fn(files_name_pattern= TEST_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TEST_SIZE)

estimator = create_estimator(run_config, hparams)

train_results = estimator.evaluate(input_fn=train_input_fn, steps=1)
train_rmse = round(math.sqrt(train_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Train RMSE: {} - {}".format(train_rmse, train_results))
print("############################################################################################")

valid_results = estimator.evaluate(input_fn=valid_input_fn, steps=1)
valid_rmse = round(math.sqrt(valid_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Valid RMSE: {} - {}".format(valid_rmse,valid_results))
print("############################################################################################")

test_results = estimator.evaluate(input_fn=test_input_fn, steps=1)
test_rmse = round(math.sqrt(test_results["average_loss"]),5)
print()
print("############################################################################################")
print("# Test RMSE: {} - {}".format(test_rmse, test_results))
print("############################################################################################")


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
8. Prediction
Can you write Python code for it?

import itertools

predict_input_fn = lambda: csv_input_fn(files_name_pattern=TEST_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.PREDICT,
                                      batch_size= 5)

predictions = estimator.predict(input_fn=predict_input_fn)
values = list(map(lambda item: item["predictions"][0],list(itertools.islice(predictions, 5))))
print()
print("Predicted Values: {}".format(values))


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
Steps to use the TF Experiment APIs

Define dataset metadata
Define data input function to read the data from csv files + feature processing
Create TF feature columns based on metadata + extended feature columns
Define an estimator (DNNRegressor) creation function with the required feature columns & parameters
Define a serving function to export the model
Run an Experiment with learn_runner to train, evaluate, and export the model
Evaluate the model using test data
Perform predictions
Can you write Python code for it?

MODEL_NAME = 'reg-model-03'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.csv'
VALID_DATA_FILES_PATTERN = 'data/valid-*.csv'
TEST_DATA_FILES_PATTERN = 'data/test-*.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
EXTEND_FEATURE_COLUMNS = True
MULTI_THREADING = True


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
b. Set HParam and RunConfig
Can you write Python code for it?


I am working on the file "01_Regression/04.0 - TF Regression Model - Dataset Input.ipynb".
The first task is:
b. data pipeline input function
Can you write Python code for it?


I am working on the file "01_Regression/05.0 - TF Regression Model - Custom Estimator.ipynb".
The first task is:
1. Define Dataset Metadata

CSV file header and defaults
Numeric and categorical feature names
Target feature name
Unused columns
Can you write Python code for it?


I am working on the file "01_Regression/05.0 - TF Regression Model - Custom Estimator.ipynb".
The first task is:
2. Define Data Input Function

Input csv files name pattern
Use TF Dataset APIs to read and process the data
Parse CSV lines to feature tensors
Apply feature processing
Return (features, target) tensors

a. parsing and preprocessing logic
Can you write Python code for it?


I am working on the file "01_Regression/05.0 - TF Regression Model - Custom Estimator.ipynb".
The first task is:
5. Run Experiment
a. Define Experiment Function
Can you write Python code for it?

def generate_experiment_fn(**experiment_args):

    def _experiment_fn(run_config, hparams):

        train_input_fn = lambda: csv_input_fn(
            TRAIN_DATA_FILES_PATTERN,
            mode = tf.estimator.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size
        )

        eval_input_fn = lambda: csv_input_fn(
            VALID_DATA_FILES_PATTERN,
            mode=tf.estimator.ModeKeys.EVAL,
            num_epochs=1,
            batch_size=hparams.batch_size
        )

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            eval_steps=None,
            **experiment_args
        )

    return _experiment_fn


I am working on the file "01_Regression/05.0 - TF Regression Model - Custom Estimator.ipynb".
The first task is:
6. Evaluate the Model
Can you write Python code for it?

TRAIN_SIZE = 12000
VALID_SIZE = 3000
TEST_SIZE = 5000

train_input_fn = lambda: csv_input_fn(files_name_pattern= TRAIN_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TRAIN_SIZE)

valid_input_fn = lambda: csv_input_fn(files_name_pattern= VALID_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= VALID_SIZE)

test_input_fn = lambda: csv_input_fn(files_name_pattern= TEST_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.EVAL,
                                      batch_size= TEST_SIZE)

estimator = create_estimator(run_config, hparams)

train_results = estimator.evaluate(input_fn=train_input_fn, steps=1)
train_rmse = str(train_results["rmse"])
print()
print("######################################################################################")
print("# Train RMSE: {} - {}".format(train_rmse, train_results))
print("######################################################################################")

valid_results = estimator.evaluate(input_fn=valid_input_fn, steps=1)
valid_rmse = str(valid_results["rmse"])
print()
print("######################################################################################")
print("# Valid RMSE: {} - {}".format(valid_rmse,valid_results))
print("######################################################################################")

test_results = estimator.evaluate(input_fn=test_input_fn, steps=1)
test_rmse = str(test_results["rmse"])
print()
print("######################################################################################")
print("# Test RMSE: {} - {}".format(test_rmse, test_results))
print("######################################################################################")


I am working on the file "01_Regression/05.0 - TF Regression Model - Custom Estimator.ipynb".
The first task is:
Steps to use the TF Experiment APIs

Define dataset metadata
Define data input function to read the data from csv files + feature processing
Create TF feature columns based on metadata + extended feature columns
Define an model function with the required feature columns, EstimatorSpecs, & parameters
Run an Experiment with learn_runner to train, evaluate, and export the model
Evaluate the model using test data
Perform predictions & serving the exported model
Can you write Python code for it?

MODEL_NAME = 'reg-model-04'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.csv'
VALID_DATA_FILES_PATTERN = 'data/valid-*.csv'
TEST_DATA_FILES_PATTERN = 'data/test-*.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
EXTEND_FEATURE_COLUMNS = True
MULTI_THREADING = True


I am working on the file "01_Regression/05.0 - TF Regression Model - Custom Estimator.ipynb".
The first task is:
b. Set HParam and RunConfig
Can you write Python code for it?

TRAIN_SIZE = 12000
NUM_EPOCHS = 1000
BATCH_SIZE = 500
NUM_EVAL = 10
CHECKPOINT_STEPS = int((TRAIN_SIZE/BATCH_SIZE) * (NUM_EPOCHS/NUM_EVAL))

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    hidden_units=[8, 4],
    dropout_prob = 0.0)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS, # evaluate after 100 epochs, i.e., total of 10 evaluations (+ the first one)
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)
print("Required Evaluation Steps:", NUM_EVAL)
print("That is 1 evaluation step after each",NUM_EPOCHS/NUM_EVAL," epochs")
print("Save Checkpoint After",CHECKPOINT_STEPS,"steps")


I am working on the file "01_Regression/05.0 - TF Regression Model - Custom Estimator.ipynb".
The first task is:
b. data pipeline input function
Can you write Python code for it?

def csv_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL,
                 skip_header_lines=0,
                 num_epochs=None,
                 batch_size=200):

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    file_names = tf.matching_files(files_name_pattern)

    dataset = data.TextLineDataset(filenames=file_names)
    dataset = dataset.skip(skip_header_lines)

    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row))

    if PROCESS_FEATURES:
        dataset = dataset.map(lambda features, target: (process_features(features), target))

    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()

    features, target = iterator.get_next()
    return features, target

features, target = csv_input_fn(files_name_pattern="")
print("Feature read from CSV: {}".format(list(features.keys())))
print("Target read from CSV: {}".format(target))


I am working on the file "01_Regression/05.0 - TF Regression Model - Custom Estimator.ipynb".
The first task is:
d. Run Experiment via learn_runner
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

learn_runner.run(
    experiment_fn=generate_experiment_fn(

        export_strategies=[make_export_strategy(
            csv_serving_input_fn,
            exports_to_keep=1
        )]
    ),
    run_config=run_config,
    schedule="train_and_evaluate",
    hparams=hparams
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
1. Define Dataset Metadata

tf.example feature names and defaults
Numeric and categorical feature names
Target feature name
Unused features
Can you write Python code for it?


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
2. Define Data Input Function

Input .tfrecords files name pattern
Use TF Dataset APIs to read and process the data
Parse tf.exmaples  to feature tensors
Apply feature processing
Return (features, target) tensors

a. Parsing and preprocessing logic
Can you write Python code for it?

def parse_tf_example(example_proto):

    feature_spec = {}

    for feature_name in NUMERIC_FEATURE_NAMES:
        feature_spec[feature_name] = tf.FixedLenFeature(shape=(1), dtype=tf.float32)

    for feature_name in CATEGORICAL_FEATURE_NAMES:
        feature_spec[feature_name] = tf.FixedLenFeature(shape=(1), dtype=tf.string)

    feature_spec[TARGET_NAME] = tf.FixedLenFeature(shape=(1), dtype=tf.float32)

    parsed_features = tf.parse_example(serialized=example_proto, features=feature_spec)

    target = parsed_features.pop(TARGET_NAME)

    return parsed_features, target

def process_features(features):

    # example of clipping
    features['x'] = tf.clip_by_value(features['x'], clip_value_min=-3, clip_value_max=3)
    features['y'] = tf.clip_by_value(features['y'], clip_value_min=-3, clip_value_max=3)

    # example of polynomial expansion
    features["x_2"] = tf.square(features['x'])
    features["y_2"] = tf.square(features['y'])

    # example of nonlinearity
    features["xy"] = features['x'] * features['y']

    # example of custom logic
    features['dist_xy'] =  tf.sqrt(tf.squared_difference(features['x'],features['y']))
    features["sin_x"] = tf.sin(features['x'])
    features["cos_y"] = tf.sin(features['y'])

    return features


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
3. Define Feature Columns
The input numeric columns are assumed to be normalized (or have the same scale). Otherwise, a normlizer_fn, along with the normlisation params (mean, stdv or min, max) should be passed to tf.feature_column.numeric_column() constructor
Can you write Python code for it?

def extend_feature_columns(feature_columns, hparams):

    num_buckets = hparams.num_buckets
    embedding_size = hparams.embedding_size

    buckets = np.linspace(-3, 3, num_buckets).tolist()

    alpha_X_beta = tf.feature_column.crossed_column(
            [feature_columns['alpha'], feature_columns['beta']], 4)

    x_bucketized = tf.feature_column.bucketized_column(
            feature_columns['x'], boundaries=buckets)

    y_bucketized = tf.feature_column.bucketized_column(
            feature_columns['y'], boundaries=buckets)

    x_bucketized_X_y_bucketized = tf.feature_column.crossed_column(
           [x_bucketized, y_bucketized], num_buckets**2)

    x_bucketized_X_y_bucketized_embedded = tf.feature_column.embedding_column(
            x_bucketized_X_y_bucketized, dimension=embedding_size)

    feature_columns['alpha_X_beta'] = alpha_X_beta
    feature_columns['x_bucketized_X_y_bucketized'] = x_bucketized_X_y_bucketized
    feature_columns['x_bucketized_X_y_bucketized_embedded'] = x_bucketized_X_y_bucketized_embedded

    return feature_columns

def get_feature_columns(hparams):

    CONSTRUCTED_NUMERIC_FEATURES_NAMES = ['x_2', 'y_2', 'xy', 'dist_xy', 'sin_x', 'cos_y']
    all_numeric_feature_names = NUMERIC_FEATURE_NAMES.copy()

    if PROCESS_FEATURES:
        all_numeric_feature_names += CONSTRUCTED_NUMERIC_FEATURES_NAMES

    numeric_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
                       for feature_name in all_numeric_feature_names}

    categorical_column_with_vocabulary = \
        {item[0]: tf.feature_column.categorical_column_with_vocabulary_list(item[0], item[1])
         for item in CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.items()}

    feature_columns = {}

    if numeric_columns is not None:
        feature_columns.update(numeric_columns)

    if categorical_column_with_vocabulary is not None:
        feature_columns.update(categorical_column_with_vocabulary)

    if EXTEND_FEATURE_COLUMNS:
       ...


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
4. Define an Estimator Creation Function
a. Get wide and deep feature columns

dense columns = numeric columns + embedding columns
categorical columns = vocabolary list columns + bucketized columns
sparse columns = hashed categorical columns + crossed columns
categorical columns => indicator columns
deep columns = dense columns + indicator columns
wide columns = categorical columns + sparse columns
Can you write Python code for it?

def get_wide_deep_columns():

    feature_columns = list(get_feature_columns(hparams).values())

    dense_columns = list(
        filter(lambda column: isinstance(column, feature_column._NumericColumn) |
                              isinstance(column, feature_column._EmbeddingColumn),
               feature_columns
        )
    )

    categorical_columns = list(
        filter(lambda column: isinstance(column, feature_column._VocabularyListCategoricalColumn) |
                              isinstance(column, feature_column._BucketizedColumn),
                   feature_columns)
    )

    sparse_columns = list(
        filter(lambda column: isinstance(column,feature_column._HashedCategoricalColumn) |
                              isinstance(column, feature_column._CrossedColumn),
               feature_columns)
    )

    indicator_columns = list(
            map(lambda column: tf.feature_column.indicator_column(column),
                categorical_columns)
    )

    deep_feature_columns = dense_columns + indicator_columns
    wide_feature_columns = categorical_columns + sparse_columns

    return wide_feature_columns, deep_feature_columns


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
5. Run Experiment
a. Set HParam and RunConfig
Can you write Python code for it?

TRAIN_SIZE = 12000
NUM_EPOCHS = 1000
BATCH_SIZE = 500
EVAL_AFTER_SEC = 15
TOTAL_STEPS = (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    hidden_units=[16, 12, 8],
    num_buckets = 6,
    embedding_size = 3,
    max_steps = TOTAL_STEPS,
    dropout_prob = 0.001)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.estimator.RunConfig(
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)
print("That is 1 evaluation step after each",EVAL_AFTER_SEC," training seconds")


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
Serving via the Saved Model
Can you write Python code for it?

import os

export_dir = model_dir +"/export/estimate"

saved_model_dir = export_dir + "/" + os.listdir(path=export_dir)[-1]

print(saved_model_dir)

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="predict"
)

output = predictor_fn({'csv_rows': ["0.5,1,ax01,bx02", "-0.5,-1,ax02,bx02"]})
print(output)


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
Steps to use the TF Estimator (Train_And_Evaluate) APIs

Define dataset metadata
Define data input function to read the data from .tfrecord files + feature processing
Create TF feature columns based on metadata + extended feature columns
Define an estimator (LinearCombinedDNNRegressor) with the required feature columns (wide/deep) & parameters
Run an experiment using the estimator train_and_evaluate function to train, evaluate, and export the model
Evaluate the model using test data
Perform predictions & serving the exported model
Can you write Python code for it?

MODEL_NAME = 'reg-model-06'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.tfrecords'
VALID_DATA_FILES_PATTERN = 'data/valid-*.tfrecords'
TEST_DATA_FILES_PATTERN = 'data/test-*.tfrecords'

RESUME_TRAINING = False
PROCESS_FEATURES = True
EXTEND_FEATURE_COLUMNS = True
MULTI_THREADING = True


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
b. Data pipeline input function
Can you write Python code for it?

def tfrecods_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL,
                 num_epochs=None,
                 batch_size=200):

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    file_names = tf.matching_files(files_name_pattern)
    dataset = data.TFRecordDataset(filenames=file_names)

    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda tf_example: parse_tf_example(tf_example))

    if PROCESS_FEATURES:
        dataset = dataset.map(lambda features, target: (process_features(features), target))

    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()

    features, target = iterator.get_next()
    return features, target

features, target = tfrecods_input_fn(files_name_pattern="")
print("Feature read from TFRecords: {}".format(list(features.keys())))
print("Target read from TFRecords: {}".format(target))


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
b. Define the DNNLinearCombinedRegressor
Can you write Python code for it?

def create_estimator(run_config, hparams, print_desc=False):

    wide_feature_columns, deep_feature_columns = get_wide_deep_columns()

    estimator = tf.estimator.DNNLinearCombinedRegressor(

        dnn_feature_columns = deep_feature_columns,
        linear_feature_columns = wide_feature_columns,

        dnn_hidden_units= hparams.hidden_units,

        dnn_optimizer= tf.train.AdamOptimizer(),

        dnn_activation_fn= tf.nn.elu,
        dnn_dropout= hparams.dropout_prob,

        config= run_config
    )

    if print_desc:
        print("")
        print("*Estimator Type:")
        print("================")
        print(type(estimator))
        print("")
        print("*deep columns:")
        print("==============")
        print(deep_feature_columns)
        print("")
        print("wide columns:")
        print("=============")
        print(wide_feature_columns)
        print("")

    return estimator


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
d. Define TrainSpec and EvaluSpec
Can you write Python code for it?

train_spec = tf.estimator.TrainSpec(
    input_fn = lambda: tfrecods_input_fn(
        TRAIN_DATA_FILES_PATTERN,
        mode = tf.estimator.ModeKeys.TRAIN,
        num_epochs=hparams.num_epochs,
        batch_size=hparams.batch_size
    ),
    max_steps=hparams.max_steps,
    hooks=None
)

eval_spec = tf.estimator.EvalSpec(
    input_fn = lambda: tfrecods_input_fn(
        VALID_DATA_FILES_PATTERN,
        mode=tf.estimator.ModeKeys.EVAL,
        num_epochs=1,
        batch_size=hparams.batch_size
    ),
    exporters=[tf.estimator.LatestExporter(
        name="estimate",  # the name of the folder in which the model will be exported to under export
        serving_input_receiver_fn=csv_serving_input_fn,
        exports_to_keep=1,
        as_text=True)],
    steps=None,
    #hooks=[EarlyStoppingHook(15)],
    throttle_secs = EVAL_AFTER_SEC # evalute after each 15 training seconds!
)


I am working on the file "01_Regression/07.0 - TF Regression Model - DNN Wide & Deep + estimator.train_and_evaluate.ipynb".
The first task is:
e. Run Experiment via train_and_evaluate
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

estimator = create_estimator(run_config, hparams, True)

tf.estimator.train_and_evaluate(
    estimator=estimator,
    train_spec=train_spec,
    eval_spec=eval_spec
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Define Data Input Functions
a. Parsing and preprocessing logic
Can you write Python code for it?

def parse_csv_row(csv_row):

    columns = tf.decode_csv(csv_row, record_defaults=HEADER_DEFAULTS)
    features = dict(zip(HEADER, columns))

    for column in UNUSED_FEATURE_NAMES:
        features.pop(column)

    target = features.pop(TARGET_NAME)

    return features, target

def process_features(features):

    features['CRIM'] = tf.log(features['CRIM']+0.01)
    features['B'] = tf.clip_by_value(features['B'], clip_value_min=300, clip_value_max=500)

    return features


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Define Dataset Metadata
Can you write Python code for it?

HEADER = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']

HEADER_DEFAULTS = [[0.0],[0.0],[0.0],['NA'],[0.0],[0.0],[0.0],[0.0],[0.0],[0.0],[0.0],[0.0],[0.0],[0.0]]

NUMERIC_FEATURE_NAMES = ['CRIM', 'ZN','INDUS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT']
CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY = {'CHAS':['0', '1']}
CATEGORICAL_FEATURE_NAMES = list(CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.keys())

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES

TARGET_NAME = 'MEDV'

UNUSED_FEATURE_NAMES = list(set(HEADER) - set(FEATURE_NAMES) - {TARGET_NAME})

print("Header: {}".format(HEADER))
print("Numeric Features: {}".format(NUMERIC_FEATURE_NAMES))
print("Categorical Features: {}".format(CATEGORICAL_FEATURE_NAMES))
print("Target: {}".format(TARGET_NAME))
print("Unused Features: {}".format(UNUSED_FEATURE_NAMES))


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Define Feature Columns
a. Create Feature Columns
Can you write Python code for it?

def get_feature_columns(hparams):

    numeric_columns = [
        tf.feature_column.numeric_column(feature_name) for feature_name in NUMERIC_FEATURE_NAMES]

    indicator_columns = [
        tf.feature_column.indicator_column(
           tf.feature_column.categorical_column_with_vocabulary_list(item[0], item[1]))
         for item in CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.items()]

    return numeric_columns + indicator_columns

feature_columns = get_feature_columns(tf.contrib.training.HParams(num_buckets=5,embedding_size=3))
print("Feature Columns: {}".format(feature_columns))


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Load and Analyse Dataset
Can you write Python code for it?

housing_dataset = pd.read_csv(DATA_FILE, header=None, names=HEADER )
housing_dataset.head()


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Prediction
Can you write Python code for it?

import itertools

predict_input_fn = lambda: csv_input_fn(files_name_pattern= TEST_DATA_FILES_PATTERN,
                                      mode= tf.estimator.ModeKeys.PREDICT,
                                      batch_size= 5)

predictions = estimator.predict(input_fn=predict_input_fn)
values = list(map(lambda item: item["predictions"][0],list(itertools.islice(predictions, 5))))
print()
print("Predicted Values: {}".format(values))


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Prepare Training and Test Sets
Can you write Python code for it?

DATA_SIZE = len(housing_dataset)

print("Dataset size: {}".format(DATA_SIZE))

train_data = housing_dataset.sample(frac=0.70, random_state = 19830610)
test_data = housing_dataset[~housing_dataset.index.isin(train_data.index)]

TRAIN_DATA_SIZE = len(train_data)
TEST_DATA_SIZE = len(test_data)

print("Train set size: {}".format(TRAIN_DATA_SIZE))
print("Test set size: {}".format(TEST_DATA_SIZE))
print("")


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Run Experiment
a. Set HParam and RunConfig
Can you write Python code for it?

TRAIN_SIZE = TRAIN_DATA_SIZE
NUM_EPOCHS = 10000
BATCH_SIZE = 177
EVAL_AFTER_SEC = 30
TOTAL_STEPS = (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    hidden_units=[16, 8, 4],
    max_steps = TOTAL_STEPS
)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.estimator.RunConfig(
    log_step_count_steps=1000,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)
print("That is 1 evaluation step after each",EVAL_AFTER_SEC," training seconds")


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Save Training and Test Sets
Can you write Python code for it?

train_data.to_csv(path_or_buf="data/housing-train-01.csv", header=False, index=False)
test_data.to_csv(path_or_buf="data/housing-test-01.csv", header=False, index=False)

pd.read_csv("data/housing-train-01.csv", header=None, names=HEADER).head()


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
Settings
Can you write Python code for it?

MODEL_NAME = 'housing-price-model-01'

DATA_FILE = 'data/housingdata.csv'

TRAIN_DATA_FILES_PATTERN = 'data/housing-train-01.csv'
TEST_DATA_FILES_PATTERN = 'data/housing-test-01.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
MULTI_THREADING = True


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
b. Data pipeline input function
Can you write Python code for it?

def csv_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL,
                 skip_header_lines=0,
                 num_epochs=None,
                 batch_size=200):

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    num_threads = multiprocessing.cpu_count() if MULTI_THREADING else 1

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Thread Count: {}".format(num_threads))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    file_names = tf.matching_files(files_name_pattern)

    dataset = data.TextLineDataset(filenames=file_names)
    dataset = dataset.skip(skip_header_lines)

    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row),  num_parallel_calls=num_threads)

    if PROCESS_FEATURES:
        dataset = dataset.map(lambda features, target: (process_features(features), target),
                              num_parallel_calls=num_threads)

    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()

    features, target = iterator.get_next()
    return features, target

features, target = csv_input_fn(files_name_pattern="")
print("Features in CSV: {}".format(list(features.keys())))
print("Target in CSV: {}".format(target))


I am working on the file "01_Regression/09 - TF Regression Example - Housing Price Estimation + Keras.ipynb".
The first task is:
b. Define new metrics
Can you write Python code for it?

def metric_fn(labels, predictions):

    metrics = {}

    pred_values = predictions['predictions']

    metrics["mae"] = tf.metrics.mean_absolute_error(labels, pred_values)
    metrics["rmse"] = tf.metrics.root_mean_squared_error(labels, pred_values)

    return metrics


I am working on the file "01_SERIES/CSV-Reader.ipynb".
The first task is:
On Series we can perform classic python operation using Built-In Functions!
Can you write Python code for it?

list(data)

dict(data)

max(data)

min(data)

dir(data)

type(data)

sorted(data)

data = pd.read_csv("data/input_with_two_column.csv", index_col="name", squeeze=True)
data.head()

data[["Alex", "asd"]]

data["Alex":"Vale"]


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
$V$: right-singular vectors of $X$
numpy.linalg.svd actually returns $V^{\intercal}$ instead of $V$, so it is the columns of $V^{\intercal}$ that correspond to the columns of original data matrix $X$. Hence, the rows are the set of ordered, orthornormal eigenvectors of $X^{\intercal} \, X$.
Can you write Python code for it?

print('matrix Vt has {} rows, {} columns\n'.format(*Vt_iris.shape))

print('{}'.format(pd.DataFrame(Vt_iris).head()))


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
$\chi^{2}$ distances between countries and languages
So rather than using the contingency table $F$ as the basis for singular value decomposition, we can look at the $\chi^{2}$ distances between rows and columns for visualizing the relation between countries and languages.
The $\chi^{2}$ statistic is given by
\begin{align}
  \chi^{2} &= \sum_{i=1}^{I} \sum_{j=1}^{J} \frac{(O_{ij} - E_{ij})^{2}}{E_{ij}} \
           &= N \, \sum_{i=1}^{I} \sum_{j=1}^{J} \frac{(p_{ij} - \mu_{ij})^{2}}{\mu_{ij}} \
           &= N \, \Lambda^{2} \
           \
  \Rightarrow \Lambda^{2} &= \frac{\chi^{2}}{N} \
   \Lambda &= \left[  \frac{p_{ij} - \mu_{ij}}{\sqrt{\mu_{ij}}} \right] = \left[  \frac{p_{ij} - p_{i+}\,p_{+j}}{\sqrt{p_{i+}\,p_{+j}}} \right]
\end{align}
Inertia matrix $\Lambda$ measures the distribution of the individual profiles (rows/columns) around the average profile (centroids). Thus inertia represents the observed deviation from independence.
Through its relation with the statistic $\chi^{2}$, inertia $\Lambda$ (a matrix of standardized residuals) provides the basis for using singular value decomposition.
Can you write Python code for it?

Mu_ij = row_centroid.values.reshape((P.index.size,1)) * col_centroid.values.reshape((1,P.columns.size))

Lambda = (P - Mu_ij) / np.sqrt(Mu_ij)

print('inertia Lambda:\n\n{}'.format(Lambda))


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
And there we are!
You can see how the anglophone countries Canada, USA and England are in close proximity of English and with each other, with Spanish being close to the USA while French is closer to Canada. German is close to Switzerland, with French somewhat in proximity. Italian, however, is out close to Italy, both being located largely in isolation from the other countries and languages.
This should match up with your intuition from contingency table $F$.

Helpful Resources

Making sense of principal component analysis, eigenvectors & eigenvalues
Correspondence analysis is a useful tool to uncover the relationships among categorical variables

Appendix A
PCA and SVD
Principal components analysis (PCA) and singular value decomposition are closely related, and you may often hear both these terms used in the same breath.
Here is a quick mathematical treatment explaining how PCA and SVD are related.
Consider data matrix $X \in \mathbb{R}^{m \times n}$ where $m > n$, and all $x_{ij}$ are centered about the column means. With principal components analysis, we have
\begin{align}
  \text{covariance matrix } C &= \frac{1}{m} \, X^{\intercal} \, X & \text{from PCA} \
  &= \frac{1}{m} \, V \, \Gamma \, V^{\intercal} & \text{by eigendecomposition of } X^{\intercal} \, X \
  \
  \text{ but } X &= U \, \Sigma V^{\intercal} & \text{from SVD} \
  \
  \Rightarrow C &= \frac{1}{m} \, V \, \Sigma \, U^{\intercal} \, U \, \Sigma V^{\intercal} \
  &= \frac{1}{m} \, V \, \Sigma^{2} \, V^{\intercal} \
\end{align}
So we see that:

the singular values in $\Sigma$ obtained via SVD are really just the square roots of the eigenvalues in $\Gamma$ of the covariance matrix in PCA.
if you mean-center your raw data matrix $X$ and then calculate SVD, you are doing the same thing as PCA.
the above example shows covariance of $X$ with respect to its columns ($X^{\intercal} \, X$); it also applies for covariance of $X$ with respect to rows ($X \, X^{\intercal}$).

Iris dataset: PCA & SVD
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA()
pca.fit(iris.data)

# don't forget to mean-center the data before SVD
X = iris.data - np.mean(iris.data, axis=0)
U, S, Vt = np.linalg.svd(X)


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
As we are exploring the dataset, it would be nice to view the data in order to get an idea of how the 3 species might be distributed with respect to one another in terms of their features. Perhaps we are interested in finding clusters, or maybe we would like to find a way to make class predictions?
However, since the data has 4 dimensions, we would be hard-pressed to come up with a good way to graph the data in 4D that we could easily understand.
But what if we could reduce or compress the data so that we could work in 3 dimensions or less?
Singular value decomposition lets us do just that.
Singular value decomposition
Singular value decomposition factorizes an $\mathbb{R}^{m \times n}$ matrix $X$ into

matrix $U \in \mathbb{R}^{m \times m}$ are the left-singular vectors of $X$, where columns are the set of orthonormal eigenvectors of $X \, X^{\intercal}$
diagonal matrix $\Sigma$ with entries $\sigma \in \mathbb{R}$ that are the non-negative singular values of $X$
matrix $V \in \mathbb{R}^{n \times n}$ are the right-singular vectors $X$, where the columns are the set of orthonormal eigenvectors of $X^{\intercal} \, X$

such that
\begin{align}
  X &= U \, \Sigma \, V^{\intercal}
\end{align}
We can use numpy.linalg.svd to factorize the Iris data matrix into three components $U$, $\Sigma$, and $V^{\intercal}$.
Can you write Python code for it?

U_iris, S_iris, Vt_iris = np.linalg.svd(df_iris)


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
Can we obtain the covariance matrix $C$ derived with PCA, but using  $\frac{1}{m} \, V \, \Sigma^{2} \, V^{\intercal}$ from SVD?
Can you write Python code for it?

print('covariance matrix C derived from PCA:\n{}\n'.format(Cov_pca))

Cov_svd = (1. / X.shape[0]) * Vt.T.dot(np.diag(np.square(S))).dot(Vt)
print('covariance matrix using S and Vt from SVD:\n{}\n'.format(Cov_svd))

allclose = np.allclose(Cov_pca, Cov_svd, atol=1e-1)
print('Are these matrices equivalent (element-wise closeness comparison)?\n{}'.format(allclose))


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
Correspondence Analysis
Contingency table $F$
Our next step in exploring the data is to break out the data in terms of the 2 categories.
Here we convert the raw observations into a contingency table $F$, with the countries as rows and the languages as columns. pandas.crosstab will do just that.
Can you write Python code for it?

countries = ['Canada', 'USA', 'England', 'Italy', 'Switzerland']
languages = ['English', 'French', 'Spanish', 'German', 'Italian']

F = pd.crosstab(df_countries.country, df_countries.language, margins=False)
F.index = countries
F.columns = languages

print("{}".format(F))


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
Dimension reduction
Judging from the curve representing cumulative variance explained in the figure above, we can see that

with 1 singular value, about 96.5% of the variance of $X$ can be explained
with 2 singular values, that number goes up to approximately 99.8%

Since graphing the Iris dataset in 1D wouldn't be all that interesting (just dots on a line segment), let's try using the first 2 singular values to represent the data on the $x$- and $y$-axes, respectively.
Can you write Python code for it?

idx_setosa = np.where(iris.target==0)[0]
idx_versicolor = np.where(iris.target==1)[0]
idx_virginica = np.where(iris.target==2)[0]

setosa_x = U_iris[idx_setosa, 0]
setosa_y = U_iris[idx_setosa, 1]

versicolor_x = U_iris[idx_versicolor, 0]
versicolor_y = U_iris[idx_versicolor, 1]

virginica_x = U_iris[idx_virginica, 0]
virginica_y = U_iris[idx_virginica, 1]


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
Dimension reduction
Judging from the curve representing cumulative variance explained with respect to the number of singular values used, we see that

with 1 singular value, about 50.6% of the variance of inertia matrix $\Lambda$ can be explained
with 2 singular values, that number goes up to 91.6%
with 3 singular values, we have 98.7%

To mix things up a bit, let's try visualizing the countries and languages in 3D.
For countries, we will take the first 3 columns of $U$ as the $x$-, $y$- and $z$-coordinates.
But since numpy.linalg.svd returns $V^{\intercal}$ instead of $V$, we will take the first 3 rows of $V^{\intercal}$ for the $x$-, $y$- and $z$-coordinates for the languages.
Can you write Python code for it?

country_x = U_lambda[:, 0]
country_y = U_lambda[:, 1]
country_z = U_lambda[:, 2]

lang_x = Vt_lambda[0, :]
lang_y = Vt_lambda[1, :]
lang_z = Vt_lambda[2, :]


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
Factorizing inertia matrix $\Lambda$ with singular value decomposition
Now that we have transformed contingency table $F$ into inertia matrix $\Lambda$ where element $\lambda_{ij} \in \mathbb{R}$, we can use singular value decomposition to factorize $\Lambda$ instead of the original raw data matrix.
Can you write Python code for it?

U_lambda, S_lambda, Vt_lambda = np.linalg.svd(Lambda)

num_sv_lambda = np.arange(1, S_lambda.size+1)

cum_var_explained_lambda = [np.sum(np.square(S_lambda[0:n])) / np.sum(np.square(S_lambda)) for n in num_sv_lambda]


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
Let's have a look at the cumulative variance explained visually as a function of the number of singular values used when reducing rank to find a lower-ranked matrix $X^{\prime}$ to approximate $X$. This will inform us as to how many dimensions we should use.
Can you write Python code for it?

fig = plt.figure(figsize=(7.0,5.5))
ax = fig.add_subplot(111)

plt.plot(num_sv_iris,
         cum_var_explained_iris,
         color='#2171b5',
         label='variance explained',
         alpha=0.65,
         zorder=1000)

plt.scatter(num_sv_iris,
            sklearn.preprocessing.normalize(S_iris.reshape((1,-1))),
            color='#fc4e2a',
            label='singular values (normalized)',
            alpha=0.65,
            zorder=1000)

plt.legend(loc='center right', scatterpoints=1, fontsize=8)

ax.set_xticks(num_sv_iris)
ax.set_xlim(0.8, 4.1)
ax.set_ylim(0.0, 1.1)
ax.set_xlabel(r'Number of singular values used')
ax.set_ylabel('Variance in data explained')
ax.set_title('Iris dataset: cumulative variance explained & singular values',
             fontsize=14,
             y=1.03)

ax.set_facecolor('0.98')

plt.grid(alpha=0.8, zorder=1)
plt.tight_layout()


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
Row centroid $p_{i+}$
Using correspondence matrix $P$, we next obtain the row centroid $p_{i+}$. The row centroid can be interpreted as the marginal frequency distribution over the sum of the countries (rows), and reflects the fact that there were equally 1000 respondents per country in our fictional study.
The row centroid $p_{i+}$ is derived from correspondence matrix $P$ with
\begin{align}
  p_{i+} &= \sum_{j=1}^{J} p_{ij}
\end{align}
Can you write Python code for it?

row_centroid = P.sum(axis=1)

print('row centroid (marginal frequency distribution over countries):\n\n{}'.format(row_centroid))


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
Singular Value Decomposition for Data Visualization
Displaying high-dimensional data using reduced-rank matrices
A data visualization goes a long way in helping you understand the underlying dataset. If the data is highly dimensional, you can use Singular Value Decomposition (SVD) to find a reduced-rank approximation of the data that can be visualized easily.
Example 1: the Iris dataset
We start off with the Iris flower dataset. The data is multivariate, with 150 measurements of 4 features (length and width cm of both sepal and petal) on 3 distinct Iris species. Of the 150 measurements, there are 50 measurements each for Iris setosa, Iris versicolor, and Iris virginica.
Scikit Learn's datasets includes the Iris dataset, so let's load that up and start exploring.
Can you write Python code for it?

iris = sklearn.datasets.load_iris()

df_iris = pd.DataFrame(iris.data, columns=iris.feature_names)

print('Iris dataset has {} rows and {} columns\n'.format(*df_iris.shape))

print('Here are the first 5 rows of the data:\n\n{}\n'.format(df_iris.head(5)))

print('Some simple statistics on the Iris dataset:\n\n{}\n'.format(df_iris.describe()))


I am working on the file "01_SVD_visualizing_data.ipynb".
The first task is:
We will use different marker shapes and colors to represent the three Iris species on our 2D graph.
Can you write Python code for it?

fig = plt.figure(figsize=(7.0,5.5))
ax = fig.add_subplot(111)

plt.scatter(setosa_x,
            setosa_y,
            marker='o',
            color='#66c2a5',
            label='Iris-setosa',
            zorder=1000)

plt.scatter(versicolor_x,
            versicolor_y,
            marker='D',
            color='#fc8d62',
            label='Iris-versicolor',
            zorder=1000)

plt.scatter(virginica_x,
            virginica_y,
            marker='^',
            color='#8da0cb',
            label='Iris-virginica',
            zorder=1000)

plt.legend(loc='upper left', scatterpoints=1, fontsize=8)

ax.set_xlabel(r'singular value $\sigma_{1}$')
ax.set_ylabel(r'singular value $\sigma_{2}$')
ax.set_title('2D plot of Iris dataset',
             fontsize=14,
             y=1.03)
ax.set_facecolor('0.98')

plt.grid(alpha=0.6, zorder=1)
plt.tight_layout()


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Above we have used the .T to transpose the matrix object v. We could also have used the transpose function to accomplish the same thing.
Element-wise array-array operations
When we add, subtract, multiply and divide arrays with each other, the default behaviour is element-wise operations:
Can you write Python code for it?

A * A # element-wise multiplication

v1 * v1


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Alternatively, we can cast the array objects to the type matrix. This changes the behavior of the standard arithmetic operators +, -, * to use matrix algebra.
Can you write Python code for it?

M = matrix(A)
v = matrix(v1).T # make it a column vector

v

M * M

M * v

# inner product
v.T * v

# with matrix objects, standard matrix algebra applies
v + M*v


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Array slices are mutable: if they are assigned a new value the original array from which the slice was extracted is modified:
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Calculations with higher-dimensional data
When functions such as min, max, etc. are applied to a multidimensional arrays, it is sometimes useful to apply the calculation to the entire array, and sometimes only on a row or column basis. Using the axis argument we can specify how these functions should behave:
Can you write Python code for it?

M = rand(3,4)
M

# global max
M.max()

# max in each column
M.max(axis=0)

# max in each row
M.max(axis=1)


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Data processing
Often it is useful to store datasets in Numpy arrays. Numpy provides a number of functions to calculate statistics of datasets in arrays.
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Determinant
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Equivalently, we could use the function numpy.shape and numpy.size
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
If we multiply arrays with compatible shapes, we get an element-wise multiplication of each row:
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
If we omit an index of a multidimensional array it returns the whole row (or, in general, a N-1 dimensional array)
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
If we try to add, subtract or multiply objects with incomplatible shapes we get an error:
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
If we want to avoid this behavior, so that when we get a new completely independent object B copied from A, then we need to do a so-called "deep copy" using the function copy:
Can you write Python code for it?

B = copy(A)

# now, if we modify B, A is not affected
B[0,0] = -5
B

A


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Index slicing
Index slicing is the technical name for the syntax M[lower:upper:step] to extract part of an array:
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Index slicing works exactly the same way for multidimensional arrays:
Can you write Python code for it?

A = array([[n+m*10 for n in range(5)] for m in range(5)])
A

# a block from the original array
A[1:4, 1:4]

# strides
A[::2, ::2]


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Inverse
Can you write Python code for it?

inv(M) # equivalent to M.I

dot(inv(M), M)


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Linear equations
System of linear equations like:
\begin{array}{rcl}
x + 2y & = & 5\
3x + 4y & = & 7
\end{array}
or
\begin{array}{rcl}
\left[ {\begin{array}{{20}{c}}
1&2\
3&4
\end{array}} \right] \left[ {\begin{array}{{20}{c}}
x\
y
\end{array}} \right] & = & \left[ {\begin{array}{*{20}{c}}
5\
7
\end{array}} \right]
\end{array}
could be written in matrix form as $\mathbf {Ax} = \mathbf b$ and could be solved using numpy solve:
Can you write Python code for it?


I am working on the file "01_Scientific_Python_numpy.ipynb".
The first task is:
Manipulating arrays
Indexing
We can index elements in an array using square brackets and indices:
Can you write Python code for it?

# v is a vector, and has only one dimension, taking one index
v[0]

# M is a matrix, or a 2 dimensional array, taking two indices
M[1,1]