I am working on the file "02-basics/14-commenting_code.ipynb".
The first task is:
help() only prints the __doc__ variable:
Any function, class or module starting with a string literal has a non-empty __doc__ attribute which can be printed to get information.
Can you write Python code for it?


I am working on the file "02-basics/30-profiling.ipynb".
The first task is:
! <command>
Forks a shell, run a command in it, exits the shell and returns.
Can you write Python code for it?


I am working on the file "02-basics/30-profiling.ipynb".
The first task is:
%quickref
Quick reference.
Can you write Python code for it?


I am working on the file "02-basics/30-profiling.ipynb".
The first task is:
%run <python module>.py
Run the named file inside IPython as a program.
Can you write Python code for it?

%run hello_world.py

# %load module.py
a = 1

%run module.py

%whos

a


I am working on the file "02-basics/30-profiling.ipynb".
The first task is:
%run -t <python module>.py
Giving running times.
Can you write Python code for it?

!cat substract_curves.py

!paste dataset1.txt dataset2.txt

%run -t substract_curves.py dataset1.txt dataset2.txt


I am working on the file "02-basics/30-profiling.ipynb".
The first task is:
%timeit <Python statement/expression>
Really temporize the execution of a Python statement or expression, but forgeting its results.
Can you write Python code for it?


I am working on the file "02-basics/30-profiling.ipynb".
The first task is:
help
Python’s own help system.
Can you write Python code for it?


I am working on the file "02-basics/30-profiling.ipynb".
The first task is:
object?
Details about object. Use object?? for extra details.
Can you write Python code for it?


I am working on the file "02-basics/30-profiling.ipynb".
The first task is:
who/whos
Print all interactive variables
Can you write Python code for it?


I am working on the file "02-basics/containers/01-tuples.ipynb".
The first task is:
... or be changed:
Can you write Python code for it?


I am working on the file "02-basics/containers/01-tuples.ipynb".
The first task is:
1.3. Counting ocurrences in tuples
Can you write Python code for it?


I am working on the file "02-basics/containers/01-tuples.ipynb".
The first task is:
1.5. Slicing in tuples
Can you write Python code for it?

a

a[2] # The 3-rd item

a[2:1] # Extract the tuple from the 2-nd item to the 1-st one

a[2:2] # Extract from the 2-nd item to the 2-nd item

a[2:3] # Extract from the 2-nd item to the 3-rd one

a[2:4] # Extract one item more

a[1:] # Extract from the 1-st to the end

a[:] # Extract all items (a==a[:])


I am working on the file "02-basics/containers/01-tuples.ipynb".
The first task is:
1.7. Swapping pairs with tuples is fun!
Can you write Python code for it?

a = 1; b = 2
print(a, b)
(a, b) = (b, a)
print(a, b)


I am working on the file "02-basics/containers/01-tuples.ipynb".
The first task is:
1.8. Tuples are inmutable
They can not grow:
Can you write Python code for it?

a = (1, 'a')
print(id(a),a)

a += (2,) # This creates a new instance of 'a'
print(id(a),a)


I am working on the file "02-basics/containers/01-tuples.ipynb".
The first task is:
Tuples are inmutable!
Can you write Python code for it?

a = 1; b = 2
print('"a" is in', id(a))
t = (a, b)
print('"t" is in', id(t), 'and contains', t)
a = 3
print('"a" is in', id(a))
print('"t" is in', id(t), 'and contains', t)


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
(Tuples are about three times faster)
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
... or be changed:
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
1.2. Tuple definition
And some timing ... :-)
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
1.3. Counting ocurrences in tuples
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
1.5. Slicing in tuples
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
1.6. Functions can return tuples
Can you write Python code for it?

def return_tuple():
    return (1, 'a', 2)
print(return_tuple())


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
1.7. Swapping pairs with tuples is fun!
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
2. [0] Lists
A list is a data (usually dynamic) structure that holds a collection of objects, which can have different types. Internally, a list is represented as an array, therefore, lists are fast for appending and for set/pop/getting an item, and slow for removing an internal item.
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
2.1 [1] (Of course) lists are objects
Can you write Python code for it?

print(type([]));

!python -m timeit "x = [1, 'a', 'b', 'a']" # List creation is more expensive than tuple creation


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
2.3 [0] Inserting items (O(n))
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
2.4 [0] Deleting items from a list by content (O(n)))
Can you write Python code for it?

a.remove('Hello')
a

b = ['a', 'b', 'a']; b.remove('a'); print(b)


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
2.6 [1] Erasing all list items (O(1))
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
2.9 [1] Lists are mutable objects
Lists can be modified "in-place".
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
3. [0]  Sets
Sets are implemented as hash table of (unordered) objects, therefore sets are good for get/set/delete/searching items and bad for . Sets do not support indexing, slicing, or other sequence-like behavior.
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
3.2. [0] Sets can grow (O(1))
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
3.3. [0] Sets can not contain dupplicate objects
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
3.4. [1] Sets can not contain mutable objects
Mutable objects can not be hashed :-(
Can you write Python code for it?

a = set()
a.add([1,2]) # Sets can not contain lists

a = set() # Empty set
a.add({1,2,3}) # Sets can not contain sets


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
3.5 [0] Intersection of sets (O(min(len(s), len(t)))
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
3.6 [0] Union of sets (O(len(s)+len(t)))
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
4 [0] Dictionaries
Dictionaries are sets where each element (a key) has associated an object (a value). In fact, sets can be seen as dictionaries where the elments have not associations. As sets, dictionaries are efficient for indexing by keys.
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
4.1 [0] Static definition of a dictionary
Can you write Python code for it?

a = {'Macinstosh':'OSX', 'PC':'Windows', 'Macintosh-Linux':'Linux', 'PC-Linux':'Linux'}
a


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
4.3 [0] Testing if a key is the dictionary (O(1))
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
4.4 [1] Getting the keys (O(n))
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
4.5 [1] Getting the values (O(n))
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
4.8 [1] Dictionaries are mutable
Can you write Python code for it?

id(a)

a['Macintosh-Linux'] = 'Linux for the Mac'
a

id(a)


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
4.9 [0] Looping a dictionary (O(n))
Can you write Python code for it?

for i in a:
    print(i, a[i])

for i in a.values():
    print(i)

for i in a.items():
    print(i)


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
5. Bytes
A raw bytes sequence type.
5.1. Creation of bytes sequence
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
5.2. Indexing in a bytes sequence
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
5.3. Concatenation of bytes sequences
Can you write Python code for it?

b = b'world!'
print('"b" is in', id(b))
c = a + b' ' + b
print(c)


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
6. Bytearray
Bytearray is a mutable implementation of an array of bytes. Therefore, appending data to a bytearray object is much faster than to a bytes object because in this last case, every append implies to create (and destroy the previous) new bytes object.
Can you write Python code for it?

%%timeit x = b''
x += b'x'

%%timeit x = bytearray()
x.extend(b'x')

# Array of bytes = 0
x = bytearray(10)

x

len(x)

for i in range(len(x)):
    x[i] += 1

x

# A byte in Python is a 0 <= value <= 255.
x[1] = -1

# A bytearray can be created from a list
x = bytearray([1,2,3])
x

import sys
x = bytearray(sys.stdin.read(5).encode())
x


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
7. Arrays
Can you write Python code for it?

import array as arr
a = arr.array('d', [1.1, 3.5, 4.5])
print(a)


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
Python lists can be "promiscuous":
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
Slice access
Can you write Python code for it?


I am working on the file "02-basics/containers/02-lists.ipynb".
The first task is:
Tuples are inmutable!
Can you write Python code for it?


I am working on the file "02-basics/containers/03-sets.ipynb".
The first task is:
3.3. [0] Sets can not contain dupplicate objects
Can you write Python code for it?


I am working on the file "02-basics/containers/03-sets.ipynb".
The first task is:
3.4. [1] Sets can not contain mutable objects
Mutable objects can not be hashed :-(
Can you write Python code for it?


I am working on the file "02-basics/containers/03-sets.ipynb".
The first task is:
3.5 [0] Intersection of sets (O(min(len(s), len(t)))
Can you write Python code for it?


I am working on the file "02-basics/containers/03-sets.ipynb".
The first task is:
3.6 [0] Union of sets (O(len(s)+len(t)))
Can you write Python code for it?


I am working on the file "02-basics/containers/03-sets.ipynb".
The first task is:
3.7. [0] Sets are MUCH more efficient for searching by content than lists
Can you write Python code for it?

a = set(range(1000))
print(a)

%timeit '0' in a

a = list(range(1000))
print(a)

%timeit '0' in a


I am working on the file "02-classification.ipynb".
The first task is:
Adapted from http://blog.fastforwardlabs.com/2016/02/24/hello-world-in-keras-or-scikit-learn-versus.html

Dataset
Iris flower data set
Can you write Python code for it?

iris = sns.load_dataset("iris")
iris.sample(10)


I am working on the file "02-classification.ipynb".
The first task is:
Full dataset (all 4 features)
Use all features and split dataset in train and test subsets:
Can you write Python code for it?

N = iris.shape[0] # number of data points / table rows
data = iris.sample(N,replace=False) # shuffle data
X    = data.values[:,0:4]
y_oh = onehot_encode(data.values[:,4])

N_train = N//2 # random 50/50 train/test split
X_train, y_train = X[:N_train], y_oh[:N_train]
X_test,  y_test  = X[N_train:], y_oh[N_train:]

model = Sequential()
model.add(Dense(16, input_shape=(4,)))
model.add(Activation('tanh'))
model.add(Dense(3))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()

hist = model.fit(X_train, y_train, validation_data=(X_test,y_test), epochs=200, verbose=0, batch_size=5)

plot_loss_acc(hist)

loss, accuracy = model.evaluate(X_train, y_train, verbose=0)
print('train set: loss = %.5f, accuracy = %.5f' % (loss,accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=0)
print('test  set: loss = %.5f, accuracy = %.5f' % (loss,accuracy))


I am working on the file "02-classification.ipynb".
The first task is:
Label Encoding
Can you write Python code for it?

def label_encode(arr):
    uniques, ids = np.unique(arr, return_inverse=True)
    return ids

classes = ('setosa', 'versicolor', 'virginica')
labels  = label_encode(classes)
for i,c in enumerate(classes):
    print('%10s → %d' % (c, labels[i]))

def onehot_encode(arr):
    uniques, ids = np.unique(arr, return_inverse=True)
    return np_utils.to_categorical(ids, len(uniques))

classes = ('setosa', 'versicolor', 'virginica')
onehot = onehot_encode(classes)
for i,c in enumerate(classes):
    print('%10s → [%d,%d,%d]' % (c, onehot[i,0], onehot[i,1], onehot[i,2]))


I am working on the file "02-classification.ipynb".
The first task is:
Model (Logistic Regression)

https://en.wikipedia.org/wiki/Multinomial_logistic_regression
https://en.wikipedia.org/wiki/Softmax_function
https://en.wikipedia.org/wiki/Cross_entropy
Can you write Python code for it?

model = Sequential()
model.add(Dense(16, input_shape=(1,)))
model.add(Activation('tanh'))
model.add(Dense(3))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()


I am working on the file "02-classification.ipynb".
The first task is:
Multiclass Classification
Data subset (1 feature only)
Can you write Python code for it?

data = iris
feature_name = 'petal_length'
data = data[[feature_name,'species']]

X    = data.values[:,0]
y    = label_encode(data.values[:,1])
y_oh = onehot_encode(data.values[:,1])
N    = len(y)


I am working on the file "02-classification.ipynb".
The first task is:
Overfitting
Can you write Python code for it?

N_train = 20 # only 20 of 150 samples for training, rest for testing
X_train, y_train = X[:N_train], y_oh[:N_train]
X_test,  y_test  = X[N_train:], y_oh[N_train:]

model = Sequential()
model.add(Dense(16, input_shape=(4,)))
model.add(Activation('tanh'))
model.add(Dense(16))
model.add(Activation('tanh'))
model.add(Dense(16))
model.add(Activation('tanh'))
model.add(Dense(3))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()

hist = model.fit(X_train, y_train, validation_data=(X_test,y_test), epochs=1000, verbose=0, batch_size=5)

plot_loss_acc(hist)

loss, accuracy = model.evaluate(X_train, y_train, verbose=0)
print('train set: loss = %.5f, accuracy = %.5f' % (loss,accuracy))
loss, accuracy = model.evaluate(X_test, y_test, verbose=0)
print('test  set: loss = %.5f, accuracy = %.5f' % (loss,accuracy))


I am working on the file "02-classification.ipynb".
The first task is:
Plotting
Can you write Python code for it?

R = np.linspace(X.min()-1,X.max()+1,100)
Xp = np.zeros(X.shape[0])-.1
Rp = np.zeros(R.shape[0])-.2
def plot_all(model=None):
    plt.scatter(X, Xp, c=y, cmap='jet');
    plt.xlabel(feature_name)
    if model is not None:
        prob = model.predict(R)
        yhat = np.argmax(prob,axis=1)
        plt.scatter(R, Rp, c=yhat);
        plt.plot(R,prob)
        leg = plt.legend(map(lambda s:'p("%s")'%s,classes),loc='upper center',frameon=False,ncol=3)
    plt.xlim(X.min()-1.5,X.max()+1.5)
    plt.ylim(-.4,1.2)
plot_all()


I am working on the file "02-experimentation.ipynb".
The first task is:
1. Preprocess the data using Apache Beam
The Apache Beam pipeline of data preprocessing is implemented in the preprocessing directory.
Can you write Python code for it?

EXPORTED_DATA_PREFIX = os.path.join(EXPERIMENT_RUN_DIR, 'exported_data')
TRANSFORMED_DATA_PREFIX = os.path.join(EXPERIMENT_RUN_DIR, 'transformed_data')
TRANSFORM_ARTIFACTS_DIR = os.path.join(EXPERIMENT_RUN_DIR, 'transform_artifacts')


I am working on the file "02-experimentation.ipynb".
The first task is:
2. Train a custom model locally using a Keras
The Keras implementation of the custom model is in the model_training directory.
Can you write Python code for it?

LOG_DIR = os.path.join(EXPERIMENT_RUN_DIR, 'logs')
EXPORT_DIR = os.path.join(EXPERIMENT_RUN_DIR, 'model')


I am working on the file "02-experimentation.ipynb".
The first task is:
3. Submit a Data Processing Job to Dataflow
Can you write Python code for it?

EXPORTED_DATA_PREFIX = os.path.join(EXPERIMENT_RUN_DIR, 'exported_data')
TRANSFORMED_DATA_PREFIX = os.path.join(EXPERIMENT_RUN_DIR, 'transformed_data')
TRANSFORM_ARTIFACTS_DIR = os.path.join(EXPERIMENT_RUN_DIR, 'transform_artifacts')

ML_USE = 'UNASSIGNED'
LIMIT = 1000000
raw_data_query = datasource_utils.get_training_source_query(
    project=PROJECT,
    region=REGION,
    dataset_display_name=DATASET_DISPLAY_NAME,
    ml_use=ML_USE,
    limit=LIMIT
)

etl_job_name = f"etl-{MODEL_DISPLAY_NAME}-{run_id}"

args = {
    'job_name': etl_job_name,
    'runner': 'DataflowRunner',
    'raw_data_query': raw_data_query,
    'exported_data_prefix': EXPORTED_DATA_PREFIX,
    'transformed_data_prefix': TRANSFORMED_DATA_PREFIX,
    'transform_artifact_dir': TRANSFORM_ARTIFACTS_DIR,
    'write_raw_data': False,
    'temporary_dir': os.path.join(WORKSPACE, 'tmp'),
    'gcs_location': os.path.join(WORKSPACE, 'bq_tmp'),
    'project': PROJECT,
    'region': REGION,
    'setup_file': './setup.py'
}

vertex_ai.log_params(args)

logging.getLogger().setLevel(logging.ERROR)

print("Data preprocessing started...")
etl.run_transform_pipeline(args)
print("Data preprocessing completed.")

!gsutil ls {EXPERIMENT_RUN_DIR}


I am working on the file "02-experimentation.ipynb".
The first task is:
6. Extract experiment run parameters
Can you write Python code for it?

experiment_df = vertex_ai.get_experiment_df()
experiment_df = experiment_df[experiment_df.experiment_name == EXPERIMENT_NAME]
experiment_df.T

print("Vertex AI Experiments:")
print(
    f"https://console.cloud.google.com/vertex-ai/locations{REGION}/experiments/{EXPERIMENT_NAME}/metrics?project={PROJECT}"
)


I am working on the file "02-experimentation.ipynb".
The first task is:
7. Submit a Hyperparameter Tuning Job to Vertex AI
For more information about configuring a hyperparameter study, refer to Vertex AI Hyperparameter job configuration.
Configure a hyperparameter job
Can you write Python code for it?

metric_spec = {
    'ACCURACY': 'maximize'
}

parameter_spec = {
    'learning-rate': hp_tuning.DoubleParameterSpec(min=0.0001, max=0.01, scale='log'),
    'hidden-units': hp_tuning.CategoricalParameterSpec(values=["32,32", "64,64", "128,128"])
}

tuning_job_display_name = f"hpt_{TRAINER_PACKAGE_NAME}_{run_id}"

hp_tuning_job = vertex_ai.HyperparameterTuningJob(
    display_name=tuning_job_display_name,
    custom_job=training_job,
    metric_spec=metric_spec,
    parameter_spec=parameter_spec,
    max_trial_count=4,
    parallel_trial_count=2,
    search_algorithm=None # Bayesian optimization.
)


I am working on the file "02-experimentation.ipynb".
The first task is:
Create Vertex TensorBoard instance
Can you write Python code for it?

tensorboard_resource = vertex_ai.Tensorboard.create(display_name=TENSORBOARD_DISPLAY_NAME)
tensorboard_resource_name = tensorboard_resource.gca_resource.name
print("TensorBoard resource name:", tensorboard_resource_name)


I am working on the file "02-experimentation.ipynb".
The first task is:
Create and test model inputs and outputs
Can you write Python code for it?

classifier = model.create_binary_classifier(tft_output, hyperparams)
classifier.summary()

keras.utils.plot_model(
    classifier,
    show_shapes=True,
    show_dtype=True
)

classifier(input_features)


I am working on the file "02-experimentation.ipynb".
The first task is:
Create hyperparameters
Can you write Python code for it?

hyperparams = {
    "hidden_units": [64, 32]
}

hyperparams = defaults.update_hyperparams(hyperparams)
hyperparams


I am working on the file "02-experimentation.ipynb".
The first task is:
Export the trained model
Can you write Python code for it?

saved_model_dir = os.path.join(EXPORT_DIR)

exporter.export_serving_model(
    classifier=classifier,
    serving_model_dir=saved_model_dir,
    raw_schema_location=RAW_SCHEMA_LOCATION,
    tft_output_dir=TRANSFORM_ARTIFACTS_DIR,
)


I am working on the file "02-experimentation.ipynb".
The first task is:
Get Source Query from Managed Dataset
Can you write Python code for it?

ML_USE = 'UNASSIGNED'
LIMIT = 5120

raw_data_query = datasource_utils.get_training_source_query(
    project=PROJECT,
    region=REGION,
    dataset_display_name=DATASET_DISPLAY_NAME,
    ml_use=ML_USE,
    limit=LIMIT
)

print(raw_data_query)


I am working on the file "02-experimentation.ipynb".
The first task is:
Initialize workspace
Can you write Python code for it?

REMOVE_EXPERIMENT_ARTIFACTS = False

if tf.io.gfile.exists(EXPERIMENT_ARTIFACTS_DIR) and REMOVE_EXPERIMENT_ARTIFACTS:
    print("Removing previous experiment artifacts...")
    tf.io.gfile.rmtree(EXPERIMENT_ARTIFACTS_DIR)

if not tf.io.gfile.exists(EXPERIMENT_ARTIFACTS_DIR):
    print("Creating new experiment artifacts directory...")
    tf.io.gfile.mkdir(EXPERIMENT_ARTIFACTS_DIR)

print("Workspace is ready.")
print("Experiment directory:", EXPERIMENT_ARTIFACTS_DIR)


I am working on the file "02-experimentation.ipynb".
The first task is:
Inspect model serving signatures
Can you write Python code for it?

!saved_model_cli show --dir={saved_model_dir} --tag_set=serve --signature_def=serving_tf_example

!saved_model_cli show --dir={saved_model_dir} --tag_set=serve --signature_def=serving_default


I am working on the file "02-experimentation.ipynb".
The first task is:
Prepare the training job
Can you write Python code for it?

TRAIN_RUNTIME = 'tf-cpu.2-5'
TRAIN_IMAGE = f"us-docker.pkg.dev/vertex-ai/training/{TRAIN_RUNTIME}:latest"
print("Training image:", TRAIN_IMAGE)

num_epochs = 10
learning_rate = 0.001
hidden_units = "64,64"

trainer_args = [
    f'--train-data-dir={TRANSFORMED_DATA_PREFIX + "/train/*"}',
    f'--eval-data-dir={TRANSFORMED_DATA_PREFIX + "/eval/*"}',
    f'--tft-output-dir={TRANSFORM_ARTIFACTS_DIR}',
    f'--num-epochs={num_epochs}',
    f'--learning-rate={learning_rate}',
    f'--project={PROJECT}',
    f'--region={REGION}',
    f'--staging-bucket={BUCKET}',
    f'--experiment-name={EXPERIMENT_NAME}'
]

package_uri = os.path.join(TRAINER_PACKAGE_DIR, f'{TRAINER_PACKAGE_NAME}.tar.gz')

worker_pool_specs = [
    {
        "replica_count": 1,
        "machine_spec": {
            "machine_type": 'n1-standard-4',
            "accelerator_count": 0
    },
        "python_package_spec": {
            "executor_image_uri": TRAIN_IMAGE,
            "package_uris": [package_uri],
            "python_module": "src.model_training.task",
            "args": trainer_args,
        }
    }
]


I am working on the file "02-experimentation.ipynb".
The first task is:
Read transformed data
Can you write Python code for it?

tft_output = tft.TFTransformOutput(TRANSFORM_ARTIFACTS_DIR)
transform_feature_spec = tft_output.transformed_feature_spec()
transform_feature_spec

train_data_file_pattern = os.path.join(TRANSFORMED_DATA_PREFIX,'train/data-*.gz')
eval_data_file_pattern = os.path.join(TRANSFORMED_DATA_PREFIX,'eval/data-*.gz')

for input_features, target in data.get_dataset(
    train_data_file_pattern, transform_feature_spec, batch_size=3).take(1):
    for key in input_features:
        print(f"{key} {input_features[key].dtype}: {input_features[key].numpy().tolist()}")
    print(f"target: {target.numpy().tolist()}")


I am working on the file "02-experimentation.ipynb".
The first task is:
Retrieve trial results
Can you write Python code for it?

hp_tuning_job.trials

best_trial = sorted(
    hp_tuning_job.trials,
    key=lambda trial: trial.final_measurement.metrics[0].value,
    reverse=True
)[0]

print("Best trial ID:", best_trial.id)
print("Validation Accuracy:", best_trial.final_measurement.metrics[0].value)
print("Hyperparameter Values:")
for parameter in best_trial.parameters:
    print(f" - {parameter.parameter_id}:{parameter.value}")


I am working on the file "02-experimentation.ipynb".
The first task is:
Set configurations
Can you write Python code for it?

VERSION = 'v01'
DATASET_DISPLAY_NAME = 'chicago-taxi-tips'
MODEL_DISPLAY_NAME = f'{DATASET_DISPLAY_NAME}-classifier-{VERSION}'

WORKSPACE = f'gs://{BUCKET}/{DATASET_DISPLAY_NAME}'
EXPERIMENT_ARTIFACTS_DIR = os.path.join(WORKSPACE, 'experiments')
RAW_SCHEMA_LOCATION = 'src/raw_schema/schema.pbtxt'

TENSORBOARD_DISPLAY_NAME = f'tb-{DATASET_DISPLAY_NAME}'
EXPERIMENT_NAME = f'{MODEL_DISPLAY_NAME}'


I am working on the file "02-experimentation.ipynb".
The first task is:
Start a new Vertex AI experiment run
Can you write Python code for it?

vertex_ai.init(
    project=PROJECT,
    staging_bucket=BUCKET,
    experiment=EXPERIMENT_NAME)

run_id = f"run-gcp-{datetime.now().strftime('%Y%m%d%H%M%S')}"
vertex_ai.start_run(run_id)

EXPERIMENT_RUN_DIR = os.path.join(EXPERIMENT_ARTIFACTS_DIR, EXPERIMENT_NAME, run_id)
print("Experiment run directory:", EXPERIMENT_RUN_DIR)


I am working on the file "02-experimentation.ipynb".
The first task is:
Submit the hyperparameter tuning job
Can you write Python code for it?

print("Submitting a hyperparameter tunning job...")

hp_tuning_job.run(
    service_account=SERVICE_ACCOUNT,
    tensorboard=tensorboard_resource_name,
    restart_job_on_worker_restart=False,
    sync=True,
)


I am working on the file "02-experimentation.ipynb".
The first task is:
Submit the training job
Can you write Python code for it?

print("Submitting a custom training job...")

training_job_display_name = f"{TRAINER_PACKAGE_NAME}_{run_id}"

training_job = vertex_ai.CustomJob(
    display_name=training_job_display_name,
    worker_pool_specs=worker_pool_specs,
    base_output_dir=EXPERIMENT_RUN_DIR,
)

training_job.run(
    service_account=SERVICE_ACCOUNT,
    tensorboard=tensorboard_resource_name,
    sync=True
)


I am working on the file "02-experimentation.ipynb".
The first task is:
Test Data Preprocessing Locally
Can you write Python code for it?

args = {
    'runner': 'DirectRunner',
    'raw_data_query': raw_data_query,
    'write_raw_data': True,
    'exported_data_prefix': EXPORTED_DATA_PREFIX,
    'transformed_data_prefix': TRANSFORMED_DATA_PREFIX,
    'transform_artifact_dir': TRANSFORM_ARTIFACTS_DIR,
    'temporary_dir': os.path.join(WORKSPACE, 'tmp'),
    'gcs_location': f'gs://{BUCKET}/bq_tmp',
    'project': PROJECT
}

vertex_ai.log_params(args)

print("Data preprocessing started...")
etl.run_transform_pipeline(args)
print("Data preprocessing completed.")

!gsutil ls {EXPERIMENT_RUN_DIR}


I am working on the file "02-experimentation.ipynb".
The first task is:
Train the model locally.
Can you write Python code for it?

logging.getLogger().setLevel(logging.INFO)

hyperparams["learning_rate"] = 0.001
hyperparams["num_epochs"] = 5
hyperparams["batch_size"] = 512

vertex_ai.log_params(hyperparams)

classifier = trainer.train(
    train_data_dir=train_data_file_pattern,
    eval_data_dir=eval_data_file_pattern,
    tft_output_dir=TRANSFORM_ARTIFACTS_DIR,
    hyperparams=hyperparams,
    log_dir=LOG_DIR,
)

val_loss, val_accuracy = trainer.evaluate(
    model=classifier,
    data_dir=eval_data_file_pattern,
    raw_schema_location=RAW_SCHEMA_LOCATION,
    tft_output_dir=TRANSFORM_ARTIFACTS_DIR,
    hyperparams=hyperparams,
)

vertex_ai.log_metrics(
    {"val_loss": val_loss, "val_accuracy": val_accuracy})

!tb-gcp-uploader --tensorboard_resource_name={tensorboard_resource_name} \
  --logdir={LOG_DIR} \
  --experiment_name={EXPERIMENT_NAME} --one_shot=True


I am working on the file "02-experimentation.ipynb".
The first task is:
Upload model
Can you write Python code for it?

SERVING_RUNTIME='tf2-cpu.2-5'
SERVING_IMAGE = f"us-docker.pkg.dev/vertex-ai/prediction/{SERVING_RUNTIME}:latest"
print("Serving image:", SERVING_IMAGE)

explanation_metadata = vertex_ai.explain.ExplanationMetadata(
    inputs=explanation_config["inputs"],
    outputs=explanation_config["outputs"],
)
explanation_parameters = vertex_ai.explain.ExplanationParameters(
    explanation_config["params"]
)

vertex_model = vertex_ai.Model.upload(
    display_name=MODEL_DISPLAY_NAME,
    artifact_uri=EXPORT_DIR,
    serving_container_image_uri=SERVING_IMAGE,
    parameters_schema_uri=None,
    instance_schema_uri=None,
    explanation_metadata=explanation_metadata,
    explanation_parameters=explanation_parameters,
    labels={
        'dataset_name': DATASET_DISPLAY_NAME,
        'experiment': run_id
    }
)

vertex_model.gca_resource


I am working on the file "02-flujos-de-efectivo-y-tasas.ipynb".
The first task is:
Asignación
Can you write Python code for it?

x = cf.cashflow( const_value=[2]*4, start='2000Q1', freq='Q')
x += cf.cashflow( const_value=[3]*4, start='2000Q1', freq='Q')
x

x = cf.cashflow( const_value=[6]*4, start='2000Q1', freq='Q')
x //= cf.cashflow( const_value=[4]*4, start='2000Q1',freq='Q')
x

x = cf.cashflow( const_value=[2]*4, start='2000Q1', freq='Q')
x *= cf.cashflow( const_value=[3]*4, start='2000Q1', freq='Q')
x

x = cf.cashflow( const_value=[6]*4, start='2000Q1', freq='Q')
x -= cf.cashflow( const_value=[4]*4, start='2000Q1', freq='Q')
x

## suma de dos series con diferente longitud
x = cf.cashflow([100]*12, start='2000Q1', freq='Q')
y = cf.cashflow([100]*12, start='2001Q1', freq='Q')
x + y

z = cf.cashflow([0]*24, start='1999Q1', freq='Q')
z

z[x.index] += x
z

z[y.index] += y
z

## convierte el flujo a una lista
cf.cashflow(const_value=[6]*4, start='2000Q1', freq='Q').tolist()


I am working on the file "02-flujos-de-efectivo-y-tasas.ipynb".
The first task is:
Ejemplo.-- Se va a tomar un crédito a 48 meses a partir de enero del 2000. La tasa inicial es del 3% y aumenta un punto cada año. Represente la tasa de interés.
Can you write Python code for it?

cf.interest_rate(const_value=3,        # valor por defecto
                 start='2000-01',      # primer mes del año 2000
                 periods=48,           # longitud de la serie
                 freq='M',             # capitalización mensual
                 chgpts={'2001-01':4,  # tasa para el año 2
                         '2002-01':5,  # tasa para el año 3
                         '2003-01':6}) # tasa para el año 4

x = cf.interest_rate(const_value=3,     # valor por defecto
                     start='2000-1',    # primer mes del año 2000
                     periods=48,        # longitud de la serie
                     freq='M',          # capitalización mensual
                     chgpts={12:4,      # tasa para el año 2
                             24:5,      # tasa para el año 3
                             36:6})     # tasa para el año 4

# se cambia el valor de la serie en este punto
x[5] = 10
x


I am working on the file "02-flujos-de-efectivo-y-tasas.ipynb".
The first task is:
En algunos casos es necesario introducir patrones de flujo más complejos.
Can you write Python code for it?

cf.cashflow(const_value=[0, 1, 2, 2, 4, 5, 6, 7, 8],  start='2000Q1', freq='Q')

cflo = cf.cashflow(const_value=0,
                start='2000',
                periods=15,
                freq='A')

cflo[5:10] = 100
cflo

cflo = cf.cashflow(const_value=0,
                start='2000',
                periods=15,
                freq='A')
cflo[0:5]   = 100
cflo[5:10]  = 150
cflo[10:15] = 200
cflo

## gradiente geometrico
cflo = cf.cashflow(const_value=0,
                start='2000',
                periods=15,
                freq='A')
cflo[5:10] = [100 * 1.05 ** (t-5) for t in range(5,10)]
cflo

cflo[4:11] += 200
cflo

cflo.plot.barh()
plt.grid(True)
plt.show()

cflo.plot.barh(figsize=(3,10))
plt.grid(True)
plt.show()

cflo.plot.bar(figsize=(10, 4), ylim=(-max(abs(cflo)), max(abs(cflo))))
plt.grid(True)
plt.xlabel('Años')
plt.ylabel('Millones de $')
plt.show()

cf.textplot(cflo)


I am working on the file "02-flujos-de-efectivo-y-tasas.ipynb".
The first task is:
Operaciones entre flujos genéricos de efectivo
Can you write Python code for it?

x = cf.cashflow(const_value=[1]*4, start='2000Q1', freq='Q')
y = cf.cashflow(const_value=[2]*4, start='2000Q1', freq='Q')

x + y

x * y

x - y

x // y

x.cumsum()


I am working on the file "02-generators-revisited.ipynb".
The first task is:
Advanced uses of generators
The above covers the majority of use cases of generators. It is probably the case that most Python programmers have used the above, or at least encountered it while reading other Python code.
However, generators are more than just a special syntax for iterators. There is more to the generator protocol, though it tends to be less widely used and known.
The full generator protocol: send, throw, and close
Generators implement three extra methods besides the two (__iter__() and __next__()) that it needs to be an iterator. These are the send(), throw(), and close() methods. Together, they allow for even more control over how a generator executes its body when it resumes execution.
Generators haven't always had this functionality. It was added in Python 2.5, via PEP 342.
Generator.send
send() is a slight generalization of __next__(). It accepts a single value, and that value becomes the result of the yield expression as the generator resumes. The function returns the next value to be yielded, or raises StopIteration if the generator completes or returns. __next__() is actually equivalent to send(None).
Can you write Python code for it?

def generator_function():
    print((yield 0))
    print((yield 1))

generator = generator_function()
print(next(generator))  # Advance generator to first `yield` statement.
item = generator.send('print this')
print('yielded', item)
try:
    next(generator)  # Same as `generator.send(None)`
except StopIteration:
    pass


I am working on the file "02-generators-revisited.ipynb".
The first task is:
Generators as suspendable/resumable program state
When executing a normal function, it begins execution as soon as it is called, continues (without giving up control) until it completes (by returning or throwing), and cleans up its call stack when exiting.
But with a generator, it gives up control every time it hits a yield, even if the generator isn't done executing yet. When it does so, then generator becomes suspended. So that its execution can be continued later, its locals() are saved on the generator object. Later, it can be resumed (by calling __next__(), send(), throw(), or close()), and its locals() will be restored before execution resumes from the yield point.
This has been implicitly assumed in previous examples. For example, in the section The basics of generators, the generator function _range_generator_function() relies on this mechanism in order to remember the values of stop and index between steps. There will be a more advanced example of this in the section Generators as coroutines.
There are other clever ways in which this can be utilized. One great usage of generators is in the implementation of @contextlib.contextmanager. This decorator combines the power of generators with the power of the context manager protocol, in order to produce something quite powerful.
@contextlib.contextmanager
Code sample from https://github.com/python/cpython/blob/v3.6.1/Lib/contextlib.py
Copyright (c) 2001-2017 Python Software Foundation.
All Rights Reserved.
License: Python license, https://www.python.org/3.6/license.html
Some modifications made so as to only highlight the interesting parts.
```python
def contextmanager(func):
    """@contextmanager decorator.
Typical usage:

    @contextmanager
    def some_generator(<arguments>):
        <setup>
        try:
            yield <value>
        finally:
            <cleanup>

This makes this:

    with some_generator(<arguments>) as <variable>:
        <body>

equivalent to this:

    <setup>
    try:
       ...
Can you write Python code for it?

import collections

class StopAdder(Exception): pass

def adder_function():
    total = 0
    while True:
        print('At start of adder loop, current total is', total)
        try:
            integers = (yield total)
        except (Exception, GeneratorExit) as exc:
            print('Adder received exception', repr(exc), 'and is returning with final total', total)
            return total
        if not isinstance(integers, (list, tuple)):
            integers = [integers]
        if integers and isinstance(integers[0], collections.Iterable):
            integers = integers[0]
        print('Adder received', integers)
        total += sum(integers)

def send_values_into_adder(adder, *integers):
    print('Sending', integers, 'into adder')
    current_total = adder.send(integers)
    print('Current total in adder is', current_total)
    return current_total

adder = adder_function()
next(adder)

send_values_into_adder(adder)
print()
send_values_into_adder(adder, 10)
print()
send_values_into_adder(adder, 1, 2, 3)
print()
send_values_into_adder(adder, range(8))
print()
print('Sending StopAdder into adder')
try:
    adder.throw(StopAdder)
except StopIteration as exc:
    print('Final total from adder is', exc.value)


I am working on the file "02-generators-revisited.ipynb".
The first task is:
Generators that return values
In the following example, the generator will yield once, and then raise StopIteration when it hits the return statement.
Can you write Python code for it?

def generator_function():
    yield 0
    return

generator = generator_function()
next(generator)

import traceback

try:
    next(generator)
except StopIteration:
    traceback.print_exc()


I am working on the file "02-generators-revisited.ipynb".
The first task is:
It might be thought that generators will only pass value via yield, and will always have empty return statements. This is true up through Python 3.2, but that changed in Python 3.3. Now, just like in normal functions, a generator function can return arbitrary values with return statements (but note that this is illegal in earlier versions of Python, and will cause a SyntaxError).
Can you write Python code for it?

import traceback

def generator_function_that_returns_a_value():
    yield 0
    return 'return_value'

generator = generator_function_that_returns_a_value()
next(generator)

try:
    next(generator)
except StopIteration as exc:
    traceback.print_exc()
    print(repr(exc))
    print(repr(exc.value))

try:
    next(generator)
except StopIteration as exc:
    traceback.print_exc()  # Subsequent calls to `next()` do not use the return value
    print(repr(exc))       # when raising `StopIteration`.
    print(repr(exc.value))


I am working on the file "02-generators-revisited.ipynb".
The first task is:
Notice that the Python interpretter converted the return 'return_value' into a raise StopIteration('return_value'). Also notice that the exception object has a .value attribute, which holds this value.
This behavior is hidden when using for-loops or other functionality that catches and ignores StopIteration.
Can you write Python code for it?

for item in generator_function_that_returns_a_value():
    print('yielded', item)

list(generator_function_that_returns_a_value())


I am working on the file "02-import-models.ipynb".
The first task is:
Import models from the internet
In the quick start chapter we demonstrated how to use :class:~cameo.io.load_model to import a model by ID. But where did the model come from? Cameo has currently access to two model repositories on the internet, http://bigg.ucsd.edu and http://darwin.di.uminho.pt/models.
Can you write Python code for it?

from cameo import models

models.index_models_bigg()

models.index_models_minho()


I am working on the file "02-import-models.ipynb".
The first task is:
Models in the Minho database have been manually verified. The subset of models shown bellow can be used to run simulations as described in the publications.
Can you write Python code for it?

models.minho.validated.VvuMBEL943 # use TAB completion to see the other models


I am working on the file "02-intro-python/projects/ggplot/iris.ipynb".
The first task is:
Data
Can you write Python code for it?

# import iris data
iris = datasets.load_iris()

df1 = pd.DataFrame(iris.data, columns = iris.feature_names)
df2 = pd.DataFrame(iris.target_names[iris.target])

df = pd.concat([df1, df2], axis = 1)
df.head()

df.columns = ['sl', 'sw', 'pl', 'pw', 'species']


I am working on the file "02-intro-python/projects/ggplot/iris.ipynb".
The first task is:
geom_histogram with facetting
Can you write Python code for it?

p3 = ggplot(aes(x = 'sl'), data = df) + geom_histogram() + facet_wrap('species', ncol = 1)
p3


I am working on the file "02-intro-python/projects/ggplot/iris.ipynb".
The first task is:
geom_point (scatter plot)
Using two continious variables sl and sw within the class labels in species variable, one can see whether there are class differences in the 2D space (via scatter plot).
Can you write Python code for it?

p1 = ggplot(aes(x = 'sl', y = 'sw', color = 'species'), data = df) + geom_point()
p1


I am working on the file "02-intro-python/projects/ggplot/iris.ipynb".
The first task is:
geom_points with subsetting
Can you write Python code for it?

p3 = ggplot(aes(x = 'sl', y = 'sw', color = 'species'), data = df[df.species != 'setosa']) + \
        geom_point() + theme_538()
p3


I am working on the file "02-intro-python/projects/pandas/babynames.ipynb".
The first task is:
By multiple names
Can you write Python code for it?

df.query('name in ["Andrew", "Andrey"]').shape

anames = ['Andrew', 'Andrey', 'Andres', 'Andre', 'And']

df.query('name in @anames').shape

ggplot(df.query('name in @anames'), aes(x = 'year', y = 'n', color = 'name')) +\
    geom_point(size = 10) + geom_smooth(span = 0.1) + facet_wrap("sex")

ggplot(df.query('name in @anames & sex == "M"'), aes(x = 'year', y = 'n', color = 'name')) +\
    geom_point(size = 10) + geom_smooth(span = 0.1, se = False) + scale_y_log(10) +\
    ggtitle('Andre* male names ')

(df.query('name in @anames')
    .groupby(['name', 'sex'])
    [['n']].sum())

(df.query('name in @anames  & sex == "M"')
    .groupby(['name'])
    [['n']].sum())

sf = (df.query('name in @anames  & sex == "M"')
    .groupby(['name'])
    .agg({'n': {'total': sum, 'max': lambda x: x.max()},
        'prop': {'max': max}}))
sf

sf = (df.query('name in @anames  & sex == "M"')
    .groupby(['name'])
    .apply(lambda x: sum(x.n)))
sf

sf = (df.query('name in @anames  & sex == "M"')
    .groupby(['name'])
    .apply(lambda x: pd.DataFrame({
        'min': min(x.n),
        'total': sum(x.n)}, index = x.index)))
sf


I am working on the file "02-intro-python/projects/pandas/babynames.ipynb".
The first task is:
How many unique names are collected?
Can you write Python code for it?

(df['name'].nunique(), df['name'].size)

df['name'].nunique() / float(df['name'].size)


I am working on the file "02-intro-python/projects/pandas/babynames.ipynb".
The first task is:
Imports
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

from ggplot import *


I am working on the file "02-intro-python/projects/pandas/babynames.ipynb".
The first task is:
Joe as a name for girls seems to be OK. What about Mary?
Can you write Python code for it?

ggplot(df.query('name == "Mary"'), aes(x = 'year', y = 'n', color = 'sex')) +\
    geom_point(size = 10) + geom_smooth(span = 0.1) + ggtitle("name: Mary")


I am working on the file "02-intro-python/projects/pandas/babynames.ipynb".
The first task is:
Not really. That means there are many zero entries in n column for many names.
Filter
By a single name
Can you write Python code for it?

df['name'].isin(['Andrew']).value_counts()

df.query('name == "Andrew"').shape

ggplot(df.query('name == "Joe"'), aes(x = 'year', y = 'n')) + geom_point() + ggtitle("name: Joe")


I am working on the file "02-intro-python/projects/pandas/babynames.ipynb".
The first task is:
Now let's get a record, where the number of female names Mary was the maximum.
Can you write Python code for it?

ind = np.argmax(df.query('name == "Mary" & sex == "M"')['n'])
ind

df.query('index == @ind')


I am working on the file "02-intro-python/projects/pandas/babynames.ipynb".
The first task is:
Read data
Can you write Python code for it?


I am working on the file "02-library.ipynb".
The first task is:
線形回帰で回帰する
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

X_ = np.array([X]).T  # transform to fit sklearn-style-array

reg = LinearRegression()
reg.fit(X_, Y)
Y_pred = reg.predict(X_)

pl.plot(X, Y_pred)
pl.scatter(X, Y)


I am working on the file "02-library.ipynb".
The first task is:
非線形な関係のあるデータの回帰

numpyの関数を使って、適当に非線形なデータを初期化する(sin関数など)
LinearRegressionを使って回帰をかけてみる
LinearRegressionでないモデルを使って、回帰をかけてみる(RandomForest, SupportVectorRegressor, GradientBoostingなど)
Boston House Price Dataset のようなリアルデータを使う

手法の良さを定量的に評価する
手法の良さを評価するとき、２つの観点が必要になる。

「誤差」を定義すること
汎化誤差が高いこと

回帰の場合、誤差は「平均二乗誤差」を使う
Can you write Python code for it?


I am working on the file "02-logisitc-regression-intro.ipynb".
The first task is:
Model Building Process
Can you write Python code for it?


I am working on the file "02-logisitc-regression-intro.ipynb".
The first task is:
Other Evaluation Methods

Confusion Matrix
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y, y_pred)
cm

pd.crosstab(y, y_pred, rownames=['Actual'], colnames=['Predicted'], margins=True)


I am working on the file "02-loops-functions.ipynb".
The first task is:
Accessing elements of numpy arrays is very similar to accessing elements of lists, but with slightly less typing. To access elements from an n-dimensional list, we have to use multiple square brackets, e.g. l[0][4][7][8]. For a numpy array, we separate the indices using a comma: a[0, 4, 7, 8].
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
As there is no order we access dictionaries using the key. To loop over a dictionary, we take advantage of Python's loose iteration rules:
Can you write Python code for it?

for key in d1:
    print("Key is", key, "value is", d1[key])


I am working on the file "02-loops-functions.ipynb".
The first task is:
As with the range function, the notation <start>:<end> returns the entries from (and including) <start> up to, but not including, <end>.
We can use negative numbers to access from the right of the sequence: -1 is the last entry, -2 the next-to-last, and so on:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
Containers, sequences, lists, arrays
So what are the Python ways of collecting things together? In many languages there are arrays, which are often like vectors or matrices. They are ordered, can be indexed, but where the index starts from varies between languages.
In Python there are many ways of collecting objects together. The first two to look at are tuples and lists.
Tuples
A tuple is a sequence with fixed size, whose entries cannot be modified:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
Crucially, lists and tuples can contain anything. As with loops, there is no restriction on types, and things can be nested:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
Dictionaries
Both lists and tuples are ordered: there are accessed by an integer giving there location in the sequence. This doesn't always make sense. Consider an algorithm which depends on parameters $\omega, \Gamma, N$. We want to keep the parameters together, but there's no logical order to them. Instead we can use a dictionary, which is an unordered Python container:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
Exercise
Write a dictionary with the structure:
d = {'first name' : ...,
     'last name'  : ...,
     'student ID' : ...,
     'project'    : ...}
Fill it in with suitable values. Write two functions f_name and f_project. Each should take as input a dictionary.

f_name should print "My name is <first name> <last name>"
f_project should print "Student <student ID> is doing project <project>"

where <X> should fill in the appropriate value from the dictionary.
Can you write Python code for it?

boaty = {'first name' : 'Boaty',
     'last name'  : 'McBoatface',
     'student ID' : 123456,
     'project'    : 'Surveying the arctic ocean'}

def f_name(d):
    print("My name is {} {}".format(d['first name'], d['last name']))

def f_project(d):
    print("Student {} is doing project {}".format(d['student ID'], d['project']))

f_name(boaty)
f_project(boaty)


I am working on the file "02-loops-functions.ipynb".
The first task is:
If you've used Matlab before, you may be familiar with logical indexing. This is a way of accessing elements of a array that satisfy some criteria, e.g. all the elements which are greater than 0. We can also do this with numpy arrays using boolean array indexing:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
Let's say we now want to square every element of the array. For this 2d list, we would need a for loop:
Can you write Python code for it?

import copy
squared = copy.deepcopy(l)
for i in range(3):
    for j in range(3):
        squared[i][j] = l[i][j]**2
print(squared)


I am working on the file "02-loops-functions.ipynb".
The first task is:
Lists
A list is a sequence with a size that can change, and whose entries can be modified:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
Loops
We will often want to run the same code many times on similar input. Let us suppose we want to add $n$ to $3$, where $n$ is every number between $1$ and $5$. We could do:
Can you write Python code for it?

print(add(3, 1))
print(add(3, 2))
print(add(3, 3))
print(add(3, 4))
print(add(3, 5))


I am working on the file "02-loops-functions.ipynb".
The first task is:
Note that here we used the function deepcopy from the copy module the copy the list l. If we had simply used squared = l, when we the assigned the elements of squared new values, this would also have changed the values in l. This is in contrast to the simple variables we saw before, where changing the value of one will leave the values of others unchanged.
For numpy arrays, applying operations across the entire array is much simpler:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
Numpy arrays
We've seen python's built-in lists for storing data. However for mathematical purposes (particular based on linear algebra) the list is not ideal. Instead the numpy library contains the more powerful array datatype. Arrays are essentially a more powerful form of lists which make it easier to handle data. Most importantly, they allow us to apply operations to all elements of an array at once, rather than looping over the elements one-by-one.
To see this, let's create a list and a numpy array, both containing the same data.
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
Numpy has a range of array manipulation routines for rearranging and manipulating elements, such as those below.
Can you write Python code for it?

# transpose
a.T

# reshape
numpy.reshape(a, (1,9))

# stack arrays horizontally
numpy.hstack((a,a,a))


I am working on the file "02-loops-functions.ipynb".
The first task is:
The Python notation is similar to that with functions:
Can you write Python code for it?

count = 0

if count == 0:
    print("There are no items.")
elif count == 1:
    message("There is 1 item.")
else:
    print("There are" + count + " items.")


I am working on the file "02-loops-functions.ipynb".
The first task is:
The keyword that Python uses is the lower case if. To add additional branches to the condition, Python uses the "Else If" which is contracted to elif. The condition (in this case!) compares a variable, count, to a number, using the equality comparison ==. Once again, as in the case of functions, the line containing the if definition is ended with a colon (:), and the commands to be executed are indented.
We can include as many branches of the if statement as we like using multiple elif statements. We do not need to use any elif statements, nor an else, unless we want (or need) to. We can nest if statements inside each other.
Exercise
Write a nested if statement that prints out if number is

even or odd
if even can it be divided by 4.

Note: modular division in Python is a % b.
Can you write Python code for it?

number = 8

if number % 2 == 0:
    print(number, " is even.")
    if number % 4 == 0:
        print(number, " can be divided by 4.")
else:
    print(number, " is odd.")


I am working on the file "02-loops-functions.ipynb".
The first task is:
The lengthy comment at the start of the function is very useful to remind yourself later what the function should do. You can see this information by typing
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
The same slicing notation can be used, and now can be used to assignment:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
There is a shortcut to allow you to get both key and value in one go:
Can you write Python code for it?

for key, value in d1.items():
    print("Key is", key, "value is", value)


I am working on the file "02-loops-functions.ipynb".
The first task is:
This is tedious and there's a high chance of errors.
In VBA you can define a loop that repeats commands as, for example
VB.NET
    For n = 1 To 5
        Z = 3 + n
    Next n
In Python there is also a for loop:
Can you write Python code for it?

for n in 1, 2, 3, 4, 5:
    print(add(3, n))
print("Loop has ended")


I am working on the file "02-loops-functions.ipynb".
The first task is:
This is tedious and there's a high chance of errors.
In most programming languages you can define a loop that repeats commands. There are two possible types of loop: one that you know in advance how many times it will execute, and one that executes until something happens (at a point you may not be able to predict). We will focus on the former.
In Python the notation is a for loop:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
We can use slicing to access many entries at once:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
We see that the Visual Basic If statement becomes the lower case if, and the ElseIf is contracted to elif. The condition (in this case!) compares a variable, count, to a number, using the equality comparison ==. Once again, as in the case of functions, the line containing the if definition is ended with a colon (:), and the commands to be executed are indented.
We can include as many branches of the if statement as we like using multiple elif statements. We do not need to use any elif statements, nor an else, unless we want (or need) to. We can nest if statements inside each other.
Exercise
Write a function that, given an integer $n$, returns the $n^{\text{th}}$ Fibonacci number $F_n = F_{n-1} + F_{n-2}$, where $F_1 = 1 = F_2$. Check that it works for $n = 1, 2, 5, 10$ ($F_5 = 5$ and $F_{10} = 55$).
Can you write Python code for it?

def fibonacci(n):
    if n == 1 or n == 2:
        return 1
    else:
        return fibonacci(n-1) + fibonacci(n-2)

print('F_1 = ', fibonacci(1))
print('F_2 = ', fibonacci(2))
print('F_5 = ', fibonacci(5))
print('F_10 = ', fibonacci(10))


I am working on the file "02-loops-functions.ipynb".
The first task is:
We see that the line add(1, 2) is outside the function and so is executed. We can also call the function repeatedly:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
We see that to access individual entries we use square brackets and the number of the entry, starting from $0$. All Python tuples and lists start from $0$. To check that it cannot be modified:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
You can also view this in spyder by typing add in the Object window of the Help tab in the top right.
We can save the function to a file and re-use the function by importing the file. Create a new file in the spyder editor containing
```python
def add(x, y):
    """
    Add two numbers
Parameters
----------

x : float
    First input
y : float
    Second input

Returns
-------

x + y : float
"""
return x + y

```
and save it as script2.py. Then in the console check that it works as expected:
Can you write Python code for it?


I am working on the file "02-loops-functions.ipynb".
The first task is:
You can also view this in spyder by typing add in the Object window of the Help tab in the top right.
We can save the function to a file and re-use the function by importing the file. Create a new file in the spyder editor containing
```python
def add(x, y):
    """
    Add two numbers
Parameters
----------

x : float
    First input
y : float
    Second input

Returns
-------

x + y : float
"""
return x + y

```
and save it as script3.py. Then in the console check that it works as expected:
Can you write Python code for it?


I am working on the file "02-metric_classification/statement_linear/statement_linear.ipynb".
The first task is:
1. Загрузите обучающую и тестовую выборки из файлов perceptron-train.csv и perceptron-test.csv. Целевая переменная записана в первом столбце, признаки — во втором и третьем.
Can you write Python code for it?

df_train = pd.read_csv('../data/perceptron-train.csv', header=None)
df_test = pd.read_csv('../data/perceptron-test.csv', header=None)
X_train = df_train[df_train.columns[1:3]]
y_train = df_train[df_train.columns[0]]

X_test = df_test[df_test.columns[1:3]]
y_test = df_test[df_test.columns[0]]


I am working on the file "02-metric_classification/statement_linear/statement_linear.ipynb".
The first task is:
2. Обучите персептрон со стандартными параметрами и random_state=241.
Can you write Python code for it?

clf = Perceptron(random_state=241)
clf.fit(X_train, y_train)


I am working on the file "02-metric_classification/statement_linear/statement_linear.ipynb".
The first task is:
3. Подсчитайте качество (долю правильно классифицированных объектов, accuracy) полученного классификатора на тестовой выборке.
Can you write Python code for it?

predictions = clf.predict(X_test)
score = accuracy_score(y_test, predictions)
score


I am working on the file "02-metric_classification/statement_linear/statement_linear.ipynb".
The first task is:
4. Нормализуйте обучающую и тестовую выборку с помощью класса StandardScaler.
Can you write Python code for it?

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)


I am working on the file "02-metric_classification/statement_linear/statement_linear.ipynb".
The first task is:
6. Найдите разность между качеством на тестовой выборке после нормализации и качеством до нее. Это число и будет ответом на задание.
Can you write Python code for it?


I am working on the file "02-metric_classification/statement_metric_tuning/statement_metric_tuning.ipynb".
The first task is:
1. Загрузите выборку Boston с помощью функции sklearn.datasets.load_boston(). Результатом вызова данной функции является объект, у которого признаки записаны в поле data, а целевой вектор — в поле target.
Can you write Python code for it?

boston = datasets.load_boston()
X_train = boston.data
y_train = boston.target


I am working on the file "02-metric_classification/statement_metric_tuning/statement_metric_tuning.ipynb".
The first task is:
2. Приведите признаки в выборке к одному масштабу при помощи функции sklearn.preprocessing.scale.
Can you write Python code for it?


I am working on the file "02-metric_classification/statement_metric_tuning/statement_metric_tuning.ipynb".
The first task is:
3. Переберите разные варианты параметра метрики p по сетке от 1 до 10 с таким шагом, чтобы всего было протестировано 200 вариантов (используйте функцию numpy.linspace). Используйте KNeighborsRegressor с n_neighbors=5 и weights='distance' — данный параметр добавляет в алгоритм веса, зависящие от расстояния до ближайших соседей. В качестве метрики качества используйте среднеквадратичную ошибку (параметр scoring='mean_squared_error' у cross_val_score; при использовании библиотеки scikit-learn версии 0.18.1 и выше необходимо указывать scoring='neg_mean_squared_error'). Качество оценивайте, как и в предыдущем задании, с помощью кросс-валидации по 5 блокам с random_state = 42, не забудьте включить перемешивание выборки (shuffle=True).
Can you write Python code for it?

kf = KFold(n_splits=5, random_state=42, shuffle=True)
minkovsky = np.linspace(1, 10, num=200)
means = dict()

for p in minkovsky:
    knn = KNeighborsRegressor(p=p, n_neighbors=5, weights='distance')
    score = cross_val_score(knn, X_train_scaled, y_train, cv=kf, scoring='neg_mean_squared_error')
    score_mean = np.mean(score)
    means[p] = score_mean


I am working on the file "02-metric_classification/statement_neighbours/statement_neighbours.ipynb".
The first task is:
1. Загрузите выборку Wine
Can you write Python code for it?


I am working on the file "02-metric_classification/statement_neighbours/statement_neighbours.ipynb".
The first task is:
2. Извлеките из данных признаки и классы. Класс записан в первом столбце (три варианта), признаки — в столбцах со второго по последний. Более подробно о сути признаков можно прочитать по адресу https://archive.ics.uci.edu/ml/datasets/Wine (см. также файл wine.names, приложенный к заданию)
Can you write Python code for it?

X_train = df[df.columns[1:]]
y_train = df[df.columns[0]]


I am working on the file "02-metric_classification/statement_neighbours/statement_neighbours.ipynb".
The first task is:
3. Оценку качества необходимо провести методом кросс-валидации по 5 блокам (5-fold). Создайте генератор разбиений, который перемешивает выборку перед формированием блоков (shuffle=True). Для воспроизводимости результата, создавайте генератор KFold с фиксированным параметром random_state=42. В качестве меры качества используйте долю верных ответов (accuracy).
Can you write Python code for it?

kf = KFold(n_splits=5, random_state=42, shuffle=True)


I am working on the file "02-metric_classification/statement_neighbours/statement_neighbours.ipynb".
The first task is:
4. Найдите точность классификации на кросс-валидации для метода k ближайших соседей (sklearn.neighbors.KNeighborsClassifier), при k от 1 до 50. При каком k получилось оптимальное качество? Чему оно равно (число в интервале от 0 до 1)? Данные результаты и будут ответами на вопросы 1 и 2.
Can you write Python code for it?

def test_accuracy(kf, X, y):
    means = list()
    means_range = range(1, 51)
    for r in means_range:
        knn = KNeighborsClassifier(n_neighbors=r)
        score = cross_val_score(knn, X, y, cv=kf, scoring='accuracy')
        score_mean = np.mean(score)
        means.append(score_mean)
    return pd.DataFrame(means, means_range).mean(axis=1).sort_values(ascending=False)

accuracy_df = test_accuracy(kf, X_train, y_train)
top_accuracy = accuracy_df.head(1)
print(top_accuracy.index[0])
print("{:0.2f}".format(top_accuracy.values[0]))


I am working on the file "02-metric_classification/statement_neighbours/statement_neighbours.ipynb".
The first task is:
6. Какое значение k получилось оптимальным после приведения признаков к одному масштабу? Приведите ответы на вопросы 3 и 4. Помогло ли масштабирование признаков?
Can you write Python code for it?


I am working on the file "02-model-training-hugectr.ipynb".
The first task is:
Change the following variables according to your definitions.
Can you write Python code for it?

# Project definitions
PROJECT_ID = '<YOUR PROJECT ID>' # Change to your project id.
REGION = '<LOCATION OF RESOURCES>'  # Change to your region.

# Service Account address
VERTEX_SA = f'vertex-sa@{PROJECT_ID}.iam.gserviceaccount.com' # Change to your service account with Vertex AI Admin permitions.

# Bucket definitions
BUCKET = '<YOUR BUCKET NAME>' # Change to your bucket.

# Dataset information for training / validation / schema. The path must point to the file _gcs_file_list.txt
# The path MUST start with `/gcs/<bucket_name>/...` to be used as a GCSFuse path.
# The following examples is a path to a bucket create by the previous notebook (preprocessing with NVTabular)
# Please change to your path.
TRAIN_DATA = '/gcs/merlin-on-gcp/nvt-preprocessing-v01-2205/nvt-csv-pipeline/375468928805/nvt-csv-pipeline-20220603050219/transform-dataset-op_-2488396574040784896/transformed_dataset/_gcs_file_list.txt'
VALID_DATA = '/gcs/merlin-on-gcp/nvt-preprocessing-v01-2205/nvt-csv-pipeline/375468928805/nvt-csv-pipeline-20220603050219/transform-dataset-op-2_4429132453600296960/transformed_dataset/_gcs_file_list.txt'

# Schema used by the training pipepine
# The path must point to the file schema.pbtxt
SCHEMA_PATH = '/gcs/merlin-on-gcp/nvt-preprocessing-v01-2205/nvt-csv-pipeline/375468928805/nvt-csv-pipeline-20220603050219/transform-dataset-op_-2488396574040784896/transformed_dataset/schema.pbtxt'


I am working on the file "02-model-training-hugectr.ipynb".
The first task is:
Configure a custom training job
The training module accepts a set of parameters that allow you to fine tune the DeepFM model implementation and configure the training workflow. Most of the  parameters exposed by the training module map directly to the settings used in HugeCTR Python Inteface.

NUM_EPOCHS: The training workflow can run in either an epoch mode or a non-epoch mode. When the constant NUM_EPOCHS is set to a value greater than zero the model will be trained on the NUM_EPOCHS number of full epochs, where an epoch is defined as a single pass through all examples in the training data.

MAX_ITERATIONS: If NUM_EPOCHS is set to zero, you must set MAX_ITERATIONS to a value greater than zero. MAX_ITERATIONS defines the number of batches to train the model on. When NUM_EPOCHS is greater than zero MAX_ITERATIONS is ignored.

EVAL_INTERVAL and EVAL_BATCHES: The model will be evaluated every EVAL_INTERVAL training batches using the EVAL_BATCHES validation batches during the main training loop. In the current implementation the evaluation metric is AUC.

EVAL_BATCHES_FINAL: After the main training loop completes, a final evaluation will be run using the EVAL_BATCHES_FINAL. The AUC value returned is reported to Vertex AI hypertuner.

DISPLAY_INTERVAL: Training progress will be reported every DISPLAY_INTERVAL batches.

SNAPSHOT_INTERVAL: When set to a value greater than zero, a snapshot will be saved every SNAPSHOT_INTERVAL batches.

PER_GPU_BATCH_SIZE: Per GPU batch size. This value should be set through experimentation and depends on model architecture, training features, and GPU type. It is highly dependent on device memory available in a particular GPU. In our scenario - DeepFM, Criteo datasets, and A100 GPU - a batch size of 2048 works well.

LR: The base learning rate for the HugeCTR solver.

DROPOUT_RATE: The base dropout rate used in DeepFM dense layers.

NUM_WORKERS: The number of HugeCTR data reader workers that concurrently load data. This value should be estimated through experimentation....
Can you write Python code for it?

# Training parameters
NUM_EPOCHS = 0
MAX_ITERATIONS = 25000
EVAL_INTERVAL = 1000
EVAL_BATCHES = 500
EVAL_BATCHES_FINAL = 2500
DISPLAY_INTERVAL = 200
SNAPSHOT_INTERVAL = 0
PER_GPU_BATCH_SIZE = 2048
LR = 0.001
DROPOUT_RATE = 0.5
NUM_WORKERS = 12


I am working on the file "02-model-training-hugectr.ipynb".
The first task is:
Configure worker pool specifications
In this cell we configure a worker pool specification for a Vertex Custom Training job. Refer to Vertex AI Training documentation for more details.
Can you write Python code for it?

gpus = json.dumps([list(range(ACCELERATOR_NUM))]).replace(' ','')

worker_pool_specs =  [
    {
        "machine_spec": {
            "machine_type": MACHINE_TYPE,
            "accelerator_type": ACCELERATOR_TYPE,
            "accelerator_count": ACCELERATOR_NUM,
        },
        "replica_count": 1,
        "container_spec": {
            "image_uri": IMAGE_URI,
            "command": ["python", "-m", "task"],
            "args": [
                f'--per_gpu_batch_size={PER_GPU_BATCH_SIZE}',
                f'--model_name={MODEL_NAME}',
                f'--train_data={TRAIN_DATA}',
                f'--valid_data={VALID_DATA}',
                f'--schema={SCHEMA_PATH}',
                f'--slot_size_array={cardinalities}',
                f'--max_iter={MAX_ITERATIONS}',
                f'--max_eval_batches={EVAL_BATCHES}',
                f'--eval_batches={EVAL_BATCHES_FINAL}',
                f'--dropout_rate={DROPOUT_RATE}',
                f'--lr={LR}',
                f'--num_workers={NUM_WORKERS}',
                f'--num_epochs={NUM_EPOCHS}',
                f'--eval_interval={EVAL_INTERVAL}',
                f'--snapshot={SNAPSHOT_INTERVAL}',
                f'--display_interval={DISPLAY_INTERVAL}',
                f'--gpus={gpus}',
            ],
        },
    }
]


I am working on the file "02-model-training-hugectr.ipynb".
The first task is:
Set training node configuration
As described in the overview, we use a single node, multiple GPU worker pool configuration. For a complex deep learning model like DeepFM, we recommend using A2 machines that are equipped with A100 GPUs.
In this sample we use a a2-highgpu-1g machine. For production systems, you may consider even more powerful configurations - a2-highgpu-8g with 8 A100 GPUs or a2-megagpu-16g with 16 A100 GPUs.
Can you write Python code for it?

MACHINE_TYPE = 'a2-highgpu-1g'
ACCELERATOR_TYPE = 'NVIDIA_TESLA_A100'
ACCELERATOR_NUM = 1


I am working on the file "02-model-training-hugectr.ipynb".
The first task is:
Submit a Vertex custom training job
In this section of the notebook you define, submit and monitor a Vertex custom training job. As noted in the introduction, the job uses a custom training container that is a derivative of NVIDIA NGC Merlin training container image. The custom container image packages the training module which includes a DeepFM model definition - src/training/hugectr/model.py and a training and evaluation workflow" - src/training/hugectr/task.py. The custom container image also installs the cloudml-hypertune package for integration with Vertex AI hypertuning.
Build a custom training container
Can you write Python code for it?

FILE_LOCATION = './src'
! gcloud builds submit --config src/cloudbuild.yaml --substitutions _DOCKERNAME=$DOCKERNAME,_IMAGE_URI=$IMAGE_URI,_FILE_LOCATION=$FILE_LOCATION --timeout=2h --machine-type=e2-highcpu-8


I am working on the file "02-model-training-hugectr.ipynb".
The first task is:
Submit and monitor the job
When submitting a training job using the aiplatfom.CustomJob API you can configure the job.run function to block till the job completes or return control to the notebook immediately after the job is submitted. You control it with the sync argument.
Can you write Python code for it?

job_name = 'hugectr_{}'.format(time.strftime("%Y%m%d_%H%M%S"))
base_output_dir =  os.path.join(WORKSPACE, job_name)

job = vertex_ai.CustomJob(
    display_name=job_name,
    worker_pool_specs=worker_pool_specs,
    base_output_dir=base_output_dir
)
job.run(
    sync=True,
    service_account=VERTEX_SA,
    restart_job_on_worker_restart=False
)


I am working on the file "02-model-training-hugectr.ipynb".
The first task is:
To train the model with HugeCTR, you must inform the cardinalities of the categorical features.
This information can be retrieve from the fitted workflow from the execution of the previous notebook (preprocessing pipeline).
To demonstrate how to use HugeCTR, we provided this information in the next cell.
The last notebook in this series, end-to-end pipeline, this information is generated automatically by the pipeline.
Can you write Python code for it?


I am working on the file "02-numpy.ipynb".
The first task is:
Round 2: Less is More
This is good, but can we do better? With this implementation, we are actually calculating twice as potential energies as we need to! Let's reimplement the above to see if we can speed up this function (and possible reduce the memory usage).
Can you write Python code for it?

from math import sqrt

def calcTotalEnergy2(pos,box):
    '''
    Parameters
    ----------
    pos: ndarray, size (N,3), (float)
        array of cartesian coordinate positions

    box: ndarray, size (3), (float)
        simulation box dimensions
    '''

    #sanity check
    assert box.shape[0] == 3

    # This next line is rather unpythonic but essentially it convinces
    # numpy to perform a subtraction between the full Cartesian Product
    # of the positions array
    dr = np.abs(pos - pos[:,np.newaxis,:])

    #extract out upper triangle
    dr = dr[np.triu_indices(dr.shape[0],k=1)]  #<<<<<<<

    #still need to apply periodic boundary conditions
    dr = np.where(dr>box/2.0,dr-box,dr)

    dist = np.sqrt(np.sum(np.square(dr),axis=-1))

    # calculate the full N x N distance matrix
    U = potentialEnergyFunk(dist)

    return U.sum()

%%prun -D prof/numpy2.prof
energy = calcTotalEnergy2(pos,box)

with open('energy/numpy2.dat','w') as f:
    f.write('{}\n'.format(energy))


I am working on the file "02-numpy.ipynb".
The first task is:
Runtime profiling!
Can you write Python code for it?

%%prun -D prof/numpy1.prof
energy = calcTotalEnergy1(pos,box)

with open('energy/numpy1.dat','w') as f:
    f.write('{}\n'.format(energy))


I am working on the file "02-numpy.ipynb".
The first task is:
We can plot the potential energy again just to make sure this function behaves as expected.
Can you write Python code for it?

%%opts Curve [width=600,show_grid=True,height=350]

dr = 0.05          # spacing of r points
rmax = 10.0        # maximum r value
pts = int(rmax/dr) # number of r points
r = np.arange(dr,rmax,dr)

def plotFunk(width,height,label='dynamic'):
    U = potentialEnergyFunk(r,width,height)
    return hv.Curve((r,U),kdims=['Separation Distance'],vdims=['Potential Energy'],label=label)

dmap = hv.DynamicMap(plotFunk,kdims=['width','height'])
dmap = dmap.redim.range(width=((1.0,10.0)),height=((1.0,5.0)))
dmap*plotFunk(10.0,5.0,label='width: 10., height: 5.')*plotFunk(1.0,1.0,label='width: 1., height: 1.')

from math import sqrt

def calcTotalEnergy1(pos,box):
    '''
    Parameters
    ----------
    pos: ndarray, size (N,3), (float)
        array of cartesian coordinate positions

    box: ndarray, size (3), (float)
        simulation box dimensions
    '''

    #sanity check
    assert box.shape[0] == 3

    # This next line is rather unpythonic but essentially it convinces
    # numpy to perform a subtraction between the full Cartesian Product
    # of the positions array
    dr = np.abs(pos - pos[:,np.newaxis,:])

    #still need to apply periodic boundary conditions
    dr = np.where(dr>box/2.0,dr-box,dr)

    dist = np.sqrt(np.sum(np.square(dr),axis=-1))

    # calculate the full N x N distance matrix
    U = potentialEnergyFunk(dist)

    # extract the upper triangle from U
    U = np.triu(U,k=1)

    return U.sum()


I am working on the file "02-numpy.ipynb".
The first task is:
We load in the position and box information created in the intro notebook. If you haven't run that notebook, this line will not work! (You don't have to read the wall of text, just run the cells...)
Can you write Python code for it?

import numpy as np
pos = np.loadtxt('data/positions.dat')
box = np.loadtxt('data/box.dat')

print('Read {:d} positions.'.format(pos.shape[0]))
print('x min/max: {:+4.2f}/{:+4.2f}'.format(pos.min(0)[0],pos.max(0)[0]))
print('y min/max: {:+4.2f}/{:+4.2f}'.format(pos.min(0)[1],pos.max(0)[1]))
print('z min/max: {:+4.2f}/{:+4.2f}'.format(pos.min(0)[2],pos.max(0)[2]))


I am working on the file "02-perceptron-learning-and-backpropagation.ipynb".
The first task is:
Let's center the distribution. This way, we can keep the threshold at 0 and not use biases (essentially a cheat). We can visualize the learning algorithm better this way.
Can you write Python code for it?

# Line parameters
m = 1 # Slope
c = 0  # Intercept

# Creating line
x1 = np.linspace(np.min(X[:,0]), np.max(X[:,0]), 100)
y1 = m*x1 + c

# Plot the training points
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired)
plt.plot(x1,y1)
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')


I am working on the file "02-perceptron-learning-and-backpropagation.ipynb".
The first task is:
Line equation is
$$ \mathbf{w^Tx} = 0 $$
where $ w = \begin{bmatrix}
        0.1 \
        0.2\
        \end{bmatrix} $
and $ x = \begin{bmatrix}
        I_1^1 & I_2^1 & I_3^1 & ... & I_n^1\
        I_2^2 & I_2^2 & I_3^2 & ... & I_n^2\
        \end{bmatrix} $
Can you write Python code for it?

# Calculate slopes
m = -w[0]/w[1]
c = 0
# Creating slope line
x1 = np.linspace(np.min(X[:,0]), np.max(X[:,0]), 100)
y1 = m*x1 + c

plt.figure(2, figsize=(5, 5))
ax = plt.axes()
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired)
plt.plot(x1,y1,linestyle='--')
ax.arrow(0, 0, w[0], w[1], head_width=0.1, head_length=0.1, fc='r', ec='r') # Weight vector
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

# Obtain classes
perceptron_output = (X.dot(w) < 0) + 0

# Compare
classification_status = (perceptron_output == ground_truth)
X_incorrect = X[np.logical_not(classification_status),:]
delta = ground_truth - perceptron_output
delta = -1 * delta[np.logical_not(classification_status)]

print("Classification_status:")
print(classification_status)
print("\nIncorrectly classified:")
print(np.sum(np.logical_not(classification_status)))

plt.figure(2, figsize=(5, 5))
ax = plt.axes()
plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired)
plt.plot(X_incorrect[:, 0], X_incorrect[:, 1],linestyle='None',color='k',marker='x', ms=10.0, mew=1.0)
plt.plot(x1,y1,linestyle='--')
ax.arrow(0, 0, w[0], w[1], head_width=0.1, head_length=0.1, fc='r', ec='r') # Weight vector
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')


I am working on the file "02-perceptron-learning-and-backpropagation.ipynb".
The first task is:
Visualization of weight update
Can you write Python code for it?

# Define a function for plotting
def plot_vectors(X,Y,X_incorrect,w,new_weight):
    # Calculate slopes
    m = -w[0]/w[1]
    c = 0
    # Creating slope line
    x1 = np.linspace(np.min(X[:,0]), np.max(X[:,0]), 100)
    y1 = m*x1 + c
    # Plot
    plt.figure(2, figsize=(6, 6))
    plt.clf()
    ax = plt.axes()
    plt.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.Paired)
    plt.plot(X_incorrect[:, 0], X_incorrect[:, 1],linestyle='None',color='k',marker='x', ms=10.0, mew=1.0)
    plt.plot(x1,y1,linestyle='--')
    ax.arrow(0, 0, w[0], w[1], head_width=0.1, head_length=0.1, fc='r', ec='r') # Weight vector
    ax.arrow(0, 0, X_incorrect[0,0], X_incorrect[0,1], head_width=0.05, head_length=0.1, fc='k', ec='k') # Incorrect vector
    ax.arrow(0, 0, new_weight[0], new_weight[1], head_width=0.1, head_length=0.1, fc='g', ec='g') # Weight vector
    plt.xlabel('Sepal length')
    plt.ylabel('Sepal width')
    # Set limits
    max_limit = np.max(np.max(np.abs(X)))
    plt.xlim(-max_limit, max_limit)
    plt.ylim(-max_limit, max_limit)

# Apply perceptron learning algorithm
learning_rate = 1
new_weight = w + learning_rate * delta[0] * X_incorrect[0,:]
# Plot old and new weight vectors, and the incorrect X
plot_vectors(X,Y,X_incorrect,w,new_weight)

# Obtain classes
w = deepcopy(new_weight)
perceptron_output = (X.dot(w) < 0) + 0

# Compare
classification_status = (perceptron_output == ground_truth)
X_incorrect = X[np.logical_not(classification_status),:]
delta = ground_truth - perceptron_output
delta = -1 * delta[np.logical_not(classification_status)]

print("Classification_status:")
print(classification_status)
print("\nIncorrectly classified:")
print(np.sum(np.logical_not(classification_status)))

# Apply perceptron learning algorithm
learning_rate = 1
new_weight = w + learning_rate * delta[0] * X_incorrect[0,:]
# Plot old and new weight vectors, and the incorrect X
plot_vectors(X,Y,X_incorrect,w,new_weight)


I am working on the file "02-programs.ipynb".
The first task is:
(Alternatively you can use the history in the console by pressing the up arrow until the definition of the function you previously entered appears. Then click at the end of the function and press Return). Now call the function as
Can you write Python code for it?

angle = degrees_to_radians(45.0)

print(angle)


I am working on the file "02-programs.ipynb".
The first task is:
Also, in the top right of the spyder window, select the "Variable explorer" tab. It shows the variables that it currently knows, which should include x, its type (float) and its value.
If there are many variables known, you may worry that your earlier tests had already set the value for x and that the program did not actually do anything. To get back to a clean state, type %reset in the console to delete all variables - you will need to confirm that you want to do this. You can then re-run the program to test that it worked.
Using programs and modules
In previous sections we have imported and used standard Python libraries, packages or modules, such as math. This is one way of using a program, or code, that someone else has written. To do this for ourselves, we use exactly the same syntax.
Suppose we have the file lab1_basic.py exactly as above. Write a second file containing the lines
python
import lab1_basic
print(lab1_basic.x)
Save this file, in the same directory as lab1_basic.py, say as lab1_import.py. When we run this program, the console should show something like
runfile('/Users/ih3/PythonLabs/lab1_import.py', wdir='/Users/ih3/PythonLabs')
0.9320390859672263
This shows what the import statement is doing. All the library imports, definitions and operations in the imported program (lab1_basic) are performed. The results are then available to us, using the dot notation, via lab1_basic.<variable>, or lab1_basic.<function>.
To build up a program, we write Python commands into plain text files. When we want to use, or re-use, those definitions or results, we use import on the name of the file to recover their values.
Note
We saved both files - the original lab1_basic.py, and the program that imported lab1_basic.py, in the same directory. If they were in different directories then Python would not know where to find the file it was trying to import, and would give an error. The solution to this is to create a package, which is rather more work.
Functions
We have already seen and used some...
Can you write Python code for it?

from math import pi
theta_d = 30.0
theta_r = pi / 180.0 * theta_d

print(theta_r)


I am working on the file "02-programs.ipynb".
The first task is:
But the variables used internally, theta_d and theta_r, and also pi, are not known outside the function:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
But the variables used internally, theta_d and theta_r, are not known outside the function:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
But when we start using it, it can be a bit confusing:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
Go to the editor in spyder and enter those commands in a file:
python
import math
x = math.sin(1.2)
Save this file in a suitable location and with a suitable name, such as lab1_basic.py (the rules and conventions for filenames are similar to those for variable names laid out above: descriptive, lower case names without spaces). The file extension should be .py: spyder should add this automatically.
To run this program, either

press the green "play" button in the toolbar;
press the function key F5;
select "Run" from the "Run" menu.

In the console you should see a line like
runfile('/Users/ih3/PythonLabs/lab1_basic.py', wdir='/Users/ih3/PythonLabs')
appear, and nothing else. To check that the program has worked, check the value of x. In the console just type x:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
How does the function definition work?
First we need to use the def command:
python
def degrees_to_radians(theta_d):
This command effectively says "what follows is a function". The first word after def will be the name of the function, which can be used to call it later. This follows similar rules and conventions to variables and files (no spaces, lower case, words separated by underscores, etc.).
After the function name, inside brackets, is the list of input parameters. If there are no input parameters the brackets still need to be there. If there is more than one parameter, they should be separated by commas.
After the bracket there is a colon :. The use of colons to denote special "blocks" of code happens frequently in Python code, and we will see it again later.
After the colon, all the code is indented by four spaces or one tab. Most helpful text editors, such as the spyder editor, will automatically indent the code after a function is defined. If not, use the tab key to ensure the indentation is correct. In Python, whitespace and indentation is essential: it defines where blocks of code (such as functions) start and end. In other languages special keywords or characters may be used, but in Python the indentation of the code statements is the key.
The statement on the next few lines in the function documentation, or docstring.
```python
    """
    Convert an angle from degrees to radians.
...
"""

```
This is optional: it's not needed to make the code run. However, documentation is extremely useful for the next user of the code. As the next user is likely to be you in a week (or a month), when you'll have forgotten the details of what you did, documentation helps you first.
The docstring can be any string within quotes. Using "triple quotes" allows the string to go across multiple lines. The docstring can be rapidly printed using the help function:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
How does the function definition work?
First we need to use the def command:
python
def degrees_to_radians(theta_d):
This command effectively says "what follows is a function". The first word after def will be the name of the function, which can be used to call it later. This follows similar rules and conventions to variables and files (no spaces, lower case, words separated by underscores, etc.).
After the function name, inside brackets, is the list of input parameters. If there are no input parameters the brackets still need to be there. If there is more than one parameter, they should be separated by commas.
After the bracket there is a colon :. The use of colons to denote special "blocks" of code happens frequently in Python code, and we will see it again later.
After the colon, all the code is indented by four spaces or one tab. Most helpful text editors, such as the spyder editor, will automatically indent the code after a function is defined. If not, use the tab key to ensure the indentation is correct. In Python, whitespace and indentation is essential: it defines where blocks of code (such as functions) start and end. In other languages special keywords or characters may be used, but in Python the indentation of the code statements is the key.
The statement on the next few lines is the function documentation, or docstring.
```python
    """
    Convert an angle from degrees to radians.
...
"""

```
This is in principle optional: it's not needed to make the code run. However, documentation is extremely useful for the next user of the code. As the next user is likely to be you in a week (or a month), when you'll have forgotten the details of what you did, documentation helps you first. In reality, you should always include documentation.
The docstring can be any string within quotes. Using "triple quotes" allows the string to go across multiple lines. The docstring can be rapidly printed using the help function:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
In the first (x1) example, the variable x1 was not defined within the function, but it was used. When x1 is printed, Python has to look for the definition outside of the scope of the function, which it does successfully.
In the second (x2) example, the variable x2 is defined within the function. The value of x2 does not match the value of the variable with the same name defined outside the function, but that does not matter: within the function, its local value is used. When printed outside the function, the value of x2 uses the external definition, as the value defined inside the function is not known (it is "not in scope").
Some care is needed with using scope in this way, as Python reads the whole function at the time it is defined when deciding scope. As an example:
Can you write Python code for it?

x3 = 1.3

def print_x3():
    print(x3)
    x3 = 2.4

print(x3)
print_x3()


I am working on the file "02-programs.ipynb".
The first task is:
Is that last case correct? Did we really want to change the gravitational field, whilst at the same time using an initial velocity of exactly the value we expect for $g$?
A far clearer use of the function comes from using keyword arguments. This is where we explicitly use the name of the function arguments. For example:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
Note that there is only one difference here, in the very first line: we state that gravity=9.8. What this means is that if this function is called and the value of gravity is not specified, then it takes the value 9.8.
For example:
Can you write Python code for it?

print(drop_time(10.0, 0.0))
print(drop_time(height=50.0, speed=1.0))
print(drop_time(gravity=15.0, height=50.0, speed=1.0))


I am working on the file "02-programs.ipynb".
The first task is:
So, we can still give a specific value for gravity when we don't want to use the value 9.8, but it isn't needed if we're happy for it to take the default value of 9.8. This works both if we use keyword arguments and if not, with certain restrictions.
Some things to keep in mind.

Default arguments can only be used without specifying the keyword if they come after arguments without defaults. It is a very strong convention that arguments with a default come at the end of the argument list.
The value of default arguments can be pretty much anything, but care should be taken to get the behaviour you expect. In particular, it is strongly discouraged to allow the default value to be anything that might change, as this can lead to odd behaviour that is hard to find. In particular, allowing a default value to be a container such as a list (seen below) can lead to unexpected behaviour. See, for example, this discussion, pointing out why, and that the value of the default argument is fixed when the function is defined, not when it's called.

Printing and strings
We have already seen the print function used multiple times. It displays its argument(s) to the screen when called, either from the console or from within a program. It prints some representation of what it is given in the form of a string: it converts simple numbers and other objects to strings that can be shown on the screen. For example:
Can you write Python code for it?

import math
x = 1.2
name = "Alice"
print("Hello")
print(6)
print(name)
print(x)
print(math.pi)
print(math.sin(x))
print(math.sin)
print(math)


I am working on the file "02-programs.ipynb".
The first task is:
The format command takes the string (here "Hello {}. We set x={}.") and replaces the {} with the values of the variables (here name and x in order).
We can use the format command in this way for anything that has a string representation. For example:
Can you write Python code for it?

print ("The function {} applied to x={} gives {}".format(math.sin, x, math.sin(x)))


I am working on the file "02-programs.ipynb".
The first task is:
The only significant change from the second example is the order of the print statement and the assignment to x3 inside the function. Because x3 is assigned inside the function, Python wants to use the local value within the function, and will ignore the value defined outside the function. However, the print function is called before x3 has been set within the function, leading to an error.
Keyword and default arguments
Our original function degrees_to_radians only had one argument, the angle to be converted theta_d. Many functions will take more than one argument, and sometimes the function will take arguments that we don't always want to set. Python can make life easier in these cases.
Suppose we wanted to know how long it takes an object released from a height $h$, in a gravitational field of strength $g$, with initial vertical speed $v$, to hit the ground. The answer is
$$  t = \frac{1}{g} \left( v + \sqrt{v^2 + 2 h g} \right). $$
We can write this as a function:
Can you write Python code for it?

from math import sqrt

def drop_time(height, speed, gravity):
    """
    Return how long it takes an object released from a height h,
    in a gravitational field of strength g, with initial vertical speed v,
    to hit the ground.

    Parameters
    ----------

    height : float
        Initial height h
    speed : float
        Initial vertical speed v
    gravity : float
        Gravitional field strength g

    Returns
    -------

    t : float
        Time the object hits the ground
    """

    return (speed + sqrt(speed**2 + 2.0*height*gravity)) / gravity


I am working on the file "02-programs.ipynb".
The first task is:
The only significant change from the second example is the order of the print statement and the assignment to x3 inside the function. Because x3 is assigned inside the function, Python wants to use the local value within the function, and will ignore the value defined outside the function. However, the print function is called before x3 has been set within the function, leading to an error.
Keyword and default arguments
Our original function degrees_to_radians only had one argument, the angle to be converted theta_d. Many functions will take more than one argument, and sometimes the function will take arguments that we don't always want to set. Python can make life easier in these cases.
Suppose we wanted to know how long it takes an object released from a height $h$, in a gravitational field of strength $g$, with initial vertical speed $v$, to hit the ground. The answer is
\begin{equation}
  t = \frac{1}{g} \left( v + \sqrt{v^2 + 2 h g} \right).
\end{equation}
We can write this as a function:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
The result is exactly the same, but now it's explicitly clear what we're doing.
Even more useful: when using keyword arguments, we don't have to ensure that the order we use matches the order of the function definition:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
The use of the "dot" notation appears here. We saw this with accessing functions in modules and packages above; now we see it with accessing attributes and methods. It appears repeatedly in Python. The format method used above is particularly important for our purposes, but there are a lot of methods available.
There are other ways of manipulating strings.
We can join two strings using the + operator.
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
There are many more ways to use the format command which can be helpful.
We note that format is a function, but a function applied to the string before the dot. This type of function is called a method, and we shall return to them later.
Strings
We have just printed a lot of strings out, but it is useful to briefly talk about what a string is.
In Python a string is not just a sequence of characters. It is a Python object that contains additional information that "lives on it". If this information is a constant property it is called an attribute. If it is a function it is called a method. We can access this information (using the "dot" notation as above) to tell us things about the string, and to manipulate it.
Here are some basic string methods:
Can you write Python code for it?

name = "Alice"
number = "13"
sentence = "   a  b  c d     e  "
print(name.upper())
print(name.lower())
print(name.isdigit())
print(number.isdigit())
print(sentence.strip())
print(sentence.split())


I am working on the file "02-programs.ipynb".
The first task is:
There are many more ways to use the format command which can be helpful.
We note that format is a function, but a function applied to the string before the dot. This type of function is called a method, and we shall return to them later.
Strings
We have just printed a lot of strings out, but it is useful to briefly talk about what a string is.
In Python a string is not just a sequence of characters. It is a Python object that contains additional information that "lives on it". If this information is a constant property it is called an attribute. If it is a function it is called a method. We can access this information to tell us things about the string, and to manipulate it.
Here are some basic string methods:
Can you write Python code for it?


I am working on the file "02-programs.ipynb".
The first task is:
This allows you to quickly use code correctly without having to look at the code. We can do the same with functions from packages, such as
Can you write Python code for it?