I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Binary Tree Path Sum III

Give a binary tree, and a target number, find all path that the sum of nodes equal to target, the path could be start and end at any node in the tree.
http://www.lintcode.com/en/problem/binary-tree-path-sum-iii/
Validate Binary Search Tree
https://leetcode.com/problems/validate-binary-search-tree/#/description
https://www.lintcode.com/en/problem/validate-binary-search-tree/#
Can you write Python code for it?

## brilliant solver!!

"""
Definition of TreeNode:
class TreeNode:
    def __init__(self, val):
        self.val = val
        self.left, self.right = None, None
"""
class Solution:
    """
    @param root: The root of binary tree.
    @return: True if the binary tree is BST, or false
    """
    def isValidBST(self, root):
        # write your code here
        return self.helper(root, -0x80000000-1, 0x7FFFFFFF+1)

    def helper(self, root, min, max):
        if root is None:
            return True
        elif root.val <= min or root.val >= max:
            return False
        else:
            return self.helper(root.left, min, root.val) \
            and self.helper(root.right, root.val, max)


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Binary Tree Postorder Traversal

https://leetcode.com/problems/binary-tree-postorder-traversal/#/description
Note: Recursive solution is trivial, could you do it iteratively?
Can you write Python code for it?

# Definition for a binary tree node.
# class TreeNode(object):
#     def __init__(self, x):
#         self.val = x
#         self.left = None
#         self.right = None

class Solution(object):
    def postorderTraversal(self, root):
        """
        :type root: TreeNode
        :rtype: List[int]
        """
        ans = []
        if root is None:
            return ans

        # iterative
        stack, lastVist = [root], root
        while stack:
            top = stack[-1]
            if (not top.left and not top.right) or top.left == lastVist or top.right == lastVist:
                lastVist = stack.pop()
                ans.append(lastVist.val)
            else:
                if top.right:
                    stack.append(top.right)
                if top.left:
                    stack.append(top.left)
        return ans

        # recursive
        return self.postorderTraversal(root.left) + self.postorderTraversal(root.right) + [root.val]


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Lowest Common Ancestor
Given the root and two nodes in a Binary Tree. Find the lowest common ancestor(LCA) of the two nodes.
The lowest common ancestor is the node with largest depth which is the ancestor of both nodes.
https://www.lintcode.com/en/problem/lowest-common-ancestor/
Example
For the following binary tree:
4
 / \
3   7
   / \
  5   6
LCA(3, 5) = 4
LCA(5, 6) = 7
LCA(6, 7) = 7
Can you write Python code for it?

"""
Definition of TreeNode:
class TreeNode:
    def __init__(self, val):
        self.val = val
        self.left, self.right = None, None
"""
import copy
class Solution:
    """
    @param root: The root of the binary search tree.
    @param A and B: two nodes in a Binary.
    @return: Return the least common ancestor(LCA) of the two nodes.
    """
    def lowestCommonAncestor(self, root, A, B):
        # write your code here
        if not root or root == A or root == B:
            return root

        left = self.lowestCommonAncestor(root.left, A, B)
        right = self.lowestCommonAncestor(root.right, A, B)

        if left and right:
            return root
        if left:
            return left
        if right:
            return right
        return None


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Lowest Common Ancestor III: Return null if LCA does not exist

https://www.lintcode.com/en/problem/lowest-common-ancestor-iii/
Can you write Python code for it?

"""
Definition of TreeNode:
class TreeNode:
    def __init__(self, val):
        this.val = val
        this.left, this.right = None, None
"""
import copy
class Solution:
    """
    @param {TreeNode} root The root of the binary tree.
    @param {TreeNode} A and {TreeNode} B two nodes
    @return Return the LCA of the two nodes.
    """
    def lowestCommonAncestor3(self, root, A, B):
        # write your code here
        node, a, b = self.helper(root, A, B)
        return node

    def helper(self, root, A, B):
        if root is None:
            return None, False, False

        leftNode, leftA, leftB = self.helper(root.left, A, B)
        rightNode, rightA, rightB = self.helper(root.right, A, B)

        a = leftA or rightA or root == A
        b = leftB or rightB or root == B

        if not a or not b:
            return None, a, b
        elif leftNode:
            return leftNode, a, b
        elif rightNode:
            return rightNode, a, b
        else:
            return root, a, b


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Minimum/Maximum Subtree

Minimum Subtree

Given a binary tree, find the subtree with minimum sum. Return the root of the subtree.
https://www.lintcode.com/en/problem/minimum-subtree/
Example
Given a binary tree:
 1

/   \
 -5     2
 / \   /  \
0   2 -4  -5
return the node 1.
Can you write Python code for it?

"""
Definition of TreeNode:
class TreeNode:
    def __init__(self, val):
        this.val = val
        this.left, this.right = None, None
"""
class Solution:
    # @param {TreeNode} root the root of binary tree
    # @return {TreeNode} the root of the minimum subtree
    def __init__(self):
        self.node = None
        self.minsum = 0x7FFFFFFF

    def findSubtree(self, root):
        # Write your code here

        def helper(root):
            if not root:
                return 0
            newsum = helper(root.left) + helper(root.right) + root.val
            if newsum < self.minsum:
                self.minsum = newsum
                self.node = root
            return newsum

        helper(root)
        return self.node


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Quick Sort vs Merge Sort
比较快速排序与归并排序的联系与区别：
时间复杂度：都是 O(nlogn) 但是快速排序是平均 O(nlogn)，归并排序是最好最坏都是 O(nlogn)
空间复杂度：快速排序耗费 O(1) 的额外空间，归并排序不得不耗费 O(n) 的额外空间。
排序稳定性：快速排序是不稳定的排序算法。归并排序是稳定的排序算法。http://baike.baidu.com/view/547325.htm
分治的思想：快速排序先整体有序再局部有序。归并排序先局部有序再整体有序。

Given an integer array, sort it in ascending order. Use quick sort, merge sort, heap sort or any O(nlogn) algorithm.

https://www.lintcode.com/en/problem/sort-integers-ii/
Can you write Python code for it?

## Quick Sort

class Solution:
    # @param {int[]} A an integer array
    # @return nothing
    def sortIntegers2(self, A):
        # Write your code here

        def quicksort(A, start, end):
            if start >= end:
                return

            left, right = start, end
            # pivot, the middle value, not A[0], A[-1]
            pivot = A[start + (end-start)/2]                 # get the value not the index
            while left <= right:                             # here is left <= right, not <, aviod overflow!!
                while left <= right and A[left] < pivot:     # here is A[left] < pivot, not <=, allocation is more homogeneous
                    left += 1
                while left <= right and A[right] > pivot:
                    right -= 1
                if left <= right:
                    A[left], A[right] = A[right], A[left]
                    left += 1
                    right -= 1

            quicksort(A, start, right)
            quicksort(A, left, end)

        if A is None or len(A) == 0:
            return
        quicksort(A, 0, len(A)-1)

## Merge Sort

class Solution:
    # @param {int[]} A an integer array
    # @return nothing
    def sortIntegers2(self, A):
        # Write your code here

        def mergeSort(A, start, end, temp):
            if start >= end:
                return
            mergeSort(A, start, (start + end)/2, temp)
            mergeSort(A, (start + end)/2 + 1, end, temp)
            merge(A, start, end, temp)

        def merge(A, start, end, temp):
            mid = (start + end)/2
            leftstart, rightstart = start, mid+1
            tempstart = start
            while leftstart <= mid and rightstart <= end:
                if A[leftstart] <= A[rightstart]:
                    temp[tempstart] = A[leftstart]
                    leftstart += 1
                else:
                    temp[tempstart] = A[rightstart]
                    rightstart += 1
        ...


I am working on the file "03_BinaryTree_DivideConquer.ipynb".
The first task is:
Subtree with Maximum Average

https://www.lintcode.com/en/problem/subtree-with-maximum-average/
Can you write Python code for it?

"""
Definition of TreeNode:
class TreeNode:
    def __init__(self, val):
        this.val = val
        this.left, this.right = None, None
"""
class Solution:
    # @param {TreeNode} root the root of binary tree
    # @return {TreeNode} the root of the maximum average of subtree

    node, average = None, -0x80000000          ## note this value!!

    def findSubtree2(self, root):
        # Write your code here
        self.helper(root)
        return self.node

    def helper(self, root):
        if not root:
            return 0, 0

        left_sum, left_size = self.helper(root.left)
        right_sum, right_size = self.helper(root.right)
        sum = left_sum + right_sum + root.val
        size = left_size + right_size + 1
        ave = sum*1.0 / size
        if ave >= self.average:
            self.average = ave
            self.node = root

        return sum, size

## instead of using minimun value!!

"""
Definition of TreeNode:
class TreeNode:
    def __init__(self, val):
        this.val = val
        this.left, this.right = None, None
"""
class Solution:
    # @param {TreeNode} root the root of binary tree
    # @return {TreeNode} the root of the maximum average of subtree

    node, average = None, 0

    def findSubtree2(self, root):
        # Write your code here
        self.helper(root)
        return self.node

    def helper(self, root):
        if not root:
            return 0, 0

        left_sum, left_size = self.helper(root.left)
        right_sum, right_size = self.helper(root.right)
        sum = left_sum + right_sum + root.val
        size = left_size + right_size + 1
        ave = sum*1.0 / size
        if self.node is None or ave >= self.average:
            self.average = ave
            self.node = root

        return sum, size


I am working on the file "03_Clustering/02.0 - TF k-means - Estimator API.ipynb".
The first task is:
2. Define Data Input Function
a. parsing and preprocessing logic
Can you write Python code for it?

def parse_csv_row(csv_row):

    columns = tf.decode_csv(csv_row, record_defaults=HEADER_DEFAULTS)
    columns = [tf.expand_dims(tensor, -1) for tensor in columns]
    features = dict(zip(HEADER, columns))

    for column in UNUSED_FEATURE_NAMES:
        features.pop(column)

    return features

def process_features(features):

    if preprocess_features:
        features = features

    return features


I am working on the file "03_Clustering/02.0 - TF k-means - Estimator API.ipynb".
The first task is:
3. Build an Estimator
a. Define Estimator Creation Function
Can you write Python code for it?

def create_estimator(run_config, hparams):

    estimator = tf.contrib.learn.KMeansClustering(
        num_clusters = hparams.num_clusters,
        initial_clusters= tf.contrib.factorization.RANDOM_INIT,
        distance_metric= tf.contrib.factorization.SQUARED_EUCLIDEAN_DISTANCE,
        use_mini_batch=True,
        mini_batch_steps_per_iteration=1,
        kmeans_plus_plus_num_retries=10,
        relative_tolerance=None,
        config= run_config
    )

    print("")
    print("Estimator Type: {}".format(type(estimator)))
    print("")

    return estimator


I am working on the file "03_Clustering/02.0 - TF k-means - Estimator API.ipynb".
The first task is:
4. Create Estimator
Can you write Python code for it?

train_input_fn = lambda: csv_input_fn(
            train_data_files,
            mode = tf.contrib.learn.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size
        )

if not resume:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

if train:
    tf.logging.set_verbosity(tf.logging.INFO)

    time_start = datetime.utcnow()
    print("Training started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................")

    estimator = create_estimator(hparams=hparams, run_config=run_config)
    estimator.fit(input_fn=train_input_fn,
                  max_steps=None)

    time_end = datetime.utcnow()
    print(".......................................")
    print("Training finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Training elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "03_Clustering/02.0 - TF k-means - Estimator API.ipynb".
The first task is:
5. Perform Predictions (Assign Instance to Clusters)
Can you write Python code for it?

train_input_fn = lambda: csv_input_fn(
            train_data_files,
            num_epochs=1,
            batch_size=1500
        )

test_input_fn = lambda: csv_input_fn(
            test_data_files,
            num_epochs=1,
            batch_size=500
        )

train_assignments = list(estimator.predict_cluster_idx(input_fn=train_input_fn))
test_assignments = list(estimator.predict_cluster_idx(input_fn=test_input_fn))

import pandas as pd
import numpy as np

train_df = pd.read_csv(train_data_files[0], header=None, index_col=0)
test_df = pd.read_csv(test_data_files[0], header=None, index_col=0)

fig = plt.figure(figsize=(20,10))
ax = fig.add_subplot(121, projection='3d')

ax.scatter(train_df.iloc[:,0], train_df.iloc[:,1], train_df.iloc[:,2], c=train_assignments, marker='o')

ax = fig.add_subplot(122, projection='3d')

ax.scatter(test_df.iloc[:,0], test_df.iloc[:,1], test_df.iloc[:,2], c=test_assignments, marker='o')
plt.show()

clusters = estimator.clusters()
print("Cluster Centriods:")
print("==================")
print(clusters)


I am working on the file "03_Clustering/02.0 - TF k-means - Estimator API.ipynb".
The first task is:
b. Set HParam and RunConfig
Can you write Python code for it?

hparams  = tf.contrib.training.HParams(
    num_epochs = 1000,
    batch_size = 500,
    num_clusters=3
)

model_dir = 'trained_models/{}'.format(model_name)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=100,
    tf_random_seed=19850610,
    model_dir=model_dir
)

print(run_config.model_dir)


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
1. Define Dataset Metadata
Can you write Python code for it?

HEADER = ['key', 'x1', 'x2', 'x3', 'cluster']
HEADER_DEFAULTS = [[0], [0.0], [0.0], [0.0], ['NA']]

FEATURE_NAMES = ['x1', 'x2', 'x3']

UNUSED_FEATURE_NAMES = list(set(HEADER) - set(FEATURE_NAMES))

print("Header: {}".format(HEADER))
print("Input Features: {}".format(FEATURE_NAMES))
print("Unused Features: {}".format(UNUSED_FEATURE_NAMES))


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
3. Create Estimator
Can you write Python code for it?


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
4. Run Experiment
a. Create a serving function
Can you write Python code for it?

def csv_serving_input_fn():

    SERVING_HEADER = ['x1','x2','x3']
    SERVING_HEADER_DEFAULTS = [[0.0], [0.0], [0.0]]

    rows_string_tensor = tf.placeholder(dtype=tf.string,
                                         shape=[None],
                                         name='csv_rows')

    receiver_tensor = {'csv_rows': rows_string_tensor}

    row_columns = tf.expand_dims(rows_string_tensor, -1)
    columns = tf.decode_csv(row_columns, record_defaults=SERVING_HEADER_DEFAULTS)
    columns = [tf.expand_dims(tensor, -1) for tensor in columns]
    features = dict(zip(SERVING_HEADER, columns))

    return tf.contrib.learn.InputFnOps(
        process_features(features),
        None,
        receiver_tensor
    )


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
5. Perform Predictions (Assign Instance to Clusters)
Can you write Python code for it?

### evaluate function is broken! as well as score function
#estimator.evaluate(input_fn=test_input_fn)
#estimator.score(input_fn=test_input_fn)

train_input_fn = lambda: csv_input_fn(
            train_data_files,
            num_epochs=1,
            batch_size=1500
        )

test_input_fn = lambda: csv_input_fn(
            test_data_files,
            mode=tf.estimator.ModeKeys.EVAL,
            num_epochs=1,
            batch_size=500
        )

estimator =  create_estimator(run_config, hparams)

train_assignments = list(estimator.predict_cluster_idx(input_fn=train_input_fn))
test_assignments = list(estimator.predict_cluster_idx(input_fn=test_input_fn))

import pandas as pd
import numpy as np

train_df = pd.read_csv(train_data_files[0], header=None, index_col=0)
test_df = pd.read_csv(test_data_files[0], header=None, index_col=0)

fig = plt.figure(figsize=(20,10))
ax = fig.add_subplot(121, projection='3d')

ax.scatter(train_df.iloc[:,0], train_df.iloc[:,1], train_df.iloc[:,2], c=train_assignments, marker='o')

ax = fig.add_subplot(122, projection='3d')

ax.scatter(test_df.iloc[:,0], test_df.iloc[:,1], test_df.iloc[:,2], c=test_assignments, marker='o')
plt.show()

clusters = estimator.clusters()
print("Cluster Centriods:")
print("==================")
print(clusters)


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
Serving via the Saved Model
Can you write Python code for it?

export_dir = model_dir + "/export"

estimator.export_savedmodel(
    export_dir_base = export_dir,
    serving_input_fn = csv_serving_input_fn,
    as_text=False
)

%%bash

MODEL_NAME='clust-model-02'
LAST=$(ls trained_models/${MODEL_NAME}/export | tail -1)
SAVE_MODEL_DIR=trained_models/$MODEL_NAME/export/$LAST
ls $SAVE_MODEL_DIR

gcloud ml-engine local predict --model-dir=$SAVE_MODEL_DIR --text-instances='data/new-data.csv'


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
Steps to use the TF Experiment API

Define dataset metadata
Define data input function to read the data from the source (csv) + apply pre-processing
Create an estimator (KMeans) with parameters
Run Experiment with estimator, training data, and serving function
Predict cluster index of each instance
Can you write Python code for it?

train_data_files = ['data/train-data.csv']
test_data_files = ['data/test-data.csv']

model_name = 'clust-model-02'

resume = False
train = True
preprocess_features = False
extend_feature_colums = False


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
b. Create Experiment Function
Can you write Python code for it?

def generate_experiment_fn(**experiment_args):

    def _experiment_fn(run_config, hparams):

        train_input_fn = lambda: csv_input_fn(
            train_data_files,
            mode = tf.contrib.learn.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size*10
        )

        eval_input_fn = lambda: csv_input_fn(
            train_data_files,
            mode = tf.contrib.learn.ModeKeys.EVAL,
            num_epochs=1,
            batch_size=hparams.batch_size
        )

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            eval_steps=None,
            **experiment_args
        )

    return _experiment_fn


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
b. data pipeline input function
Can you write Python code for it?

def csv_input_fn(file_names, mode=tf.estimator.ModeKeys.TRAIN,
                 skip_header_lines=0,
                 num_epochs=None,
                 batch_size=200):

    shuffle = False

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(file_names))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    dataset = data.TextLineDataset(filenames=file_names)
    dataset = dataset.skip(skip_header_lines)

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row))
    dataset = dataset.map(lambda features: process_features(features))

    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()

    features = iterator.get_next()
    return features, None

features, _ = csv_input_fn(file_names=train_data_files)
print("Feature read from CSV: {}".format(list(features.keys())))


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
c. Set HParam and RunConfig
Can you write Python code for it?


I am working on the file "03_Clustering/03.0 - TF k-means - Experiment API.ipynb".
The first task is:
d. Run Experiment
Can you write Python code for it?

if not resume:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

if train:
    tf.logging.set_verbosity(tf.logging.INFO)

    time_start = datetime.utcnow()
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................")

    learn_runner.run(
        experiment_fn=generate_experiment_fn(

            export_strategies=[make_export_strategy(
                csv_serving_input_fn,
                exports_to_keep=1
            )]
        ), # not executing export_savedmodel()
        run_config=run_config,
        schedule="train", #evaluate function is borken, thus not using train_and_evaluate
        hparams=hparams
    )

    time_end = datetime.utcnow()
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "03_Clustering_OccurrenceID_in_iDigBio.ipynb".
The first task is:
Features
When clustering, the algorithm used needs to know what aspects of the data to consider when trying to decide how similar two things are. These aspects are refered to as "features". In the algorithm we are going to use, k-means, features must be numeric and the distance between the numbers must be related to how similar the data are to each other. That is, for three data points A, B, and C with feature values 0, 1, and 10, then points A and C are 10 times less related than points A and B.
We need to pick these features in advance according to our intuition and understanding of the data. There are methods of finding the best features and neural networks actually make up their own features with no input at all.
For this notebook we are going to use three features:

The total lenght of the occurrenceid
The number of characters that are valid for a hexadecimal number (0-1 and A-F)
The number of special characters

We have to calculate these features from the raw data ourselves. Fortunately they're all simple calculations. The Spark machine learning libraries need to have them represented as a dataframe with a column named "features" that contains a vector with the values of each feature for the record.
This is where we will again use a user defined function to do this.
Can you write Python code for it?

# compile regular expressions in advance for a bit of speed
hex_chars = re.compile(r'[0-9a-fA-F]')
special_chars = re.compile(r'[\:\\\-_ ]')

def make_features(occurrenceid):
    return Vectors.dense(
        len(occurrenceid),
        len(re.findall(hex_chars, occurrenceid)),
        len(re.findall(special_chars, occurrenceid))
    )

make_features_udf = udf(make_features, VectorUDT())

features = (df
            .select(col("occurrenceid"),
                    make_features_udf(col("occurrenceid")).alias("features"))
            .persist()
            )

features.head(10)


I am working on the file "03_Clustering_OccurrenceID_in_iDigBio.ipynb".
The first task is:
For the 100k parquet and a small allocation, that takes about 20-30 seconds. For the entire 100 million row iDigBio data frame and using the whole cluster it takes about 15 minutes.
We now have four arrays that are all lined up. Let's look at the costs first. These are the total SSE values added up for all the points in the data set.
Can you write Python code for it?


I am working on the file "03_Clustering_OccurrenceID_in_iDigBio.ipynb".
The first task is:
Looking at the model output
Can you write Python code for it?

centers_df = pd.DataFrame(centers[ks.index(final_k)])
centers_df


I am working on the file "03_Clustering_OccurrenceID_in_iDigBio.ipynb".
The first task is:
Returns in reducint the sum of the errors certainly have fallen off by k=15 so we'll pick that as our final k.
Can you write Python code for it?


I am working on the file "03_Clustering_OccurrenceID_in_iDigBio.ipynb".
The first task is:
The features data frame now has the original occurrenceid along with a vector of features for each one. You can count up some characters in the output above to check it.
Notice that persisted() this data frame. One of the great advantages of  Spark over other distributed processing engines is that it can preserve data frames like this in memory as they are repeatedly used and accessed which machine learning does a lot.
k-means and selecting k
You can read more about k-means works here. In a nutshell, it tries to minimize the SSE distances from a fixed number of cluster centroids in the feature space.
Huh?
It assumes there are a fixed number of cluster centers and it picks a random location as a guess for where each one is. Then it calculates the sum of the squares of all the distances from the data to those points. Then it moves the centers a bit and sees if the distances go down.
One of the drawbacks here is that you need to give the algorithm a number of clusters to try to optimize to. This makes sense because if you didn't the optimal solution would be for each data point to be its own cluster.
Fortunately picking a k for a comparatively small data set like these can be done by simply trying a range of them and graphing k vs the sum of the squared errors. The point at which there is an elbow in the graph is probably a good k value to use.
We'll just take a guess and let k vary from 5 to 20.
Can you write Python code for it?

# parallel arrays for the win
ks = []
models = []
centers = []
costs = []

for i in range(5, 21):
    kmeans = KMeans(k=i, seed=1)
    model = kmeans.fit(features)
    ks.append(i)
    models.append(model)
    centers.append(model.clusterCenters())
    costs.append(model.computeCost(features))


I am working on the file "03_ControlFlow.ipynb".
The first task is:
If you want to simply pass over a line, but some piece of code is required for the syntax, then use the pass statement.
Can you write Python code for it?

favorite_numbers = [2, 3, 5, 7, 11]
for number in favorite_numbers:
    if number == 5:
        pass
    print(number)


I am working on the file "03_ControlFlow.ipynb".
The first task is:
Note that the upper limit will never be printed.
while loops
While loops repeat some code until a condition has been met:
while condition is not met:
    do a thing
Can you write Python code for it?

x = 0
while x < 5:
    print(x)
    x += 1


I am working on the file "03_ControlFlow.ipynb".
The first task is:
Try some different numerical values for my_var.
for loops
A for loop executes a piece of code for every value in a sequence:
for each in sequence:
    do a thing
Can you write Python code for it?

favorite_numbers = [2, 3, 5, 7, 11]
for number in favorite_numbers:
    print(number)

favorite_numbers = [2, 3, 5, 7, 11]
sum_of_numbers = 0
for number in favorite_numbers:
    sum_of_numbers += number
print(sum_of_numbers)

sum(favorite_numbers)


I am working on the file "03_ControlFlow.ipynb".
The first task is:
break, continue, pass
If you want to prematurely exit a loop, use break:
Can you write Python code for it?

favorite_numbers = [2, 3, 5, 7, 11]
for number in favorite_numbers:
    print(number)
    if number == 5:
        break


I am working on the file "03_DataExploration.ipynb".
The first task is:
1. Get Basic Information About Dataset
Infos such as size of data, number of flights per months, number of delays in total and average.
A flight delay is a when an airline flight takes off and/or lands later than its scheduled time. The Federal Aviation Administration (FAA) considers a flight to be delayed when it is 15 minutes later than its scheduled time.
Source: https://en.wikipedia.org/wiki/Flight_cancellation_and_delay
Percentage of flights with arrival delays and departure delays:
Can you write Python code for it?

departureDelayed = complete2014Data.DEP_DELAY >= 15
arrivalDelayed = complete2014Data.ARR_DELAY >= 15
bothDelayed = (complete2014Data.DEP_DELAY >= 15) & (complete2014Data.ARR_DELAY >= 15)

print "Percentage departure delays out of all flights: ", sum(departureDelayed)/float(len(complete2014Data))
print "Percentage arrival delays out of all flights: ", sum(arrivalDelayed)/float(len(complete2014Data))
print "Arrival delayed if departure was already delayed: ", sum(bothDelayed)/float(sum(arrivalDelayed))


I am working on the file "03_DataExploration.ipynb".
The first task is:
5. Special delays
In the dataset, there is a breakdown of the causes of different delays for certain entries. This breakdown is not present in all entries, and its absence is indicated by NaN values. Given the sparse nature of this additional information, for the analysis in this section, we focus only on the entries that have this data available, and exclude the rest. It turns out there are still a significant number of entries remaining.
Then, we consider the proportion of all delays for which the particular delay is greater than 0. The higher the percentage, the more often this factor is a cause in delays.
The different types of delays are described by the Federal Aviation Administration. Here's the explanation they provide for the 3 special types of delay we look at here:
Carrier Delay
Carrier delay is within the control of the air carrier. Examples of occurrences that may determine carrier delay are: aircraft cleaning, aircraft damage, awaiting the arrival of connecting passengers or crew, baggage, bird strike, cargo loading, catering, computer, outage-carrier equipment, crew legality (pilot or attendant rest), damage by hazardous goods, engineering inspection, fueling, handling disabled passengers, late crew, lavatory servicing, maintenance, oversales, potable water servicing, removal of unruly passenger, slow boarding or seating, stowing carry-on baggage, weight and balance delays.
Late Arrival Delay
Arrival delay at an airport due to the late arrival of the same aircraft at a previous airport. The ripple effect of an earlier delay at downstream airports is referred to as delay propagation.
Weather Delay
Weather delay is caused by extreme or hazardous weather conditions that are forecasted or manifest themselves on point of departure, enroute, or on point of arrival.
Carrier delay
As this is largely caused by the carrier, we compare this delay to the different airlines. We find that airlines which contribute most to delays through carrier delays are:
 - Hawaiian Airlines
 - Southwest Airlines
 -...
Can you write Python code for it?

SD = complete2014Data[['UNIQUE_CARRIER', 'CARRIER_DELAY']]
SD = SD[SD.CARRIER_DELAY.notnull()]
SD['AT_FAULT'] = (SD.CARRIER_DELAY > 0) * 100
SD = pd.merge(SD, ANames, how='left', on='UNIQUE_CARRIER')

SD = SD.groupby('UNIQUE_CARRIER').agg( {'AT_FAULT':['count', 'mean'], 'CARRIER_NAME':'min'} )
SD = SD[ (SD['AT_FAULT']['count'] > 0) ] # Make sure flights are above threshold
SD = SD.sort(columns=('AT_FAULT', 'mean'), ascending=False)

ax = SD.head(5).reset_index().iloc[::-1].plot(y=('AT_FAULT', 'mean'), x=('CARRIER_NAME', 'min'), kind='barh', label='Delay')
ax.set_xlabel('Proportion of all dep. delays with carrier delay (%)')
ax.set_ylabel('Carrier')
ax.set_title('Carrier delay %')


I am working on the file "03_DataExploration.ipynb".
The first task is:
As we can see above, roughly 1/5 of all flights have a delayed departure, almost the same percentage of flights arrives late at the destination. Only a small percentage of delayed departures manage to catch up during the flight, since more than 78% of the delayed arrivals already suffered from a delayed departure.
2. What Airports are the Worst/Best?
Airport ranking (origin/destination) regarding delay times. For origin airports, we consider departure delay, and for destination airport we consider arrival delay.
Differentiate between different delay types (weather, security, etc.), e.g. are certain airports more affected by weather delays than others?
There are three modifications we make to the dataset in order to remove small entities with few data points, and to concentrate on actual delays:
 - Ignore airlines and airports with very few flights (using some threshold which we default to 365, representing one flight per day on average)
 - If delay is less than 0, representing an early departure, set the delay to 0 so it does not reduce the overall delay average.
 - Remove any entries in which the relevant delay value is null.
Best departure airport
Departure delay makes the most sense for determining delay of departure airports since the arrival delay is likely to have other factors outside the control of the departure airport. We use average delay as our metric in order to get a sense of the magnitude of delays, though we recognize there are other valid choices like percentage of delayed flights. As our results show, based on average departure delay, the best departure airports are:

VEL (Vernal, UT)
LWS (Lewiston, ID)
CNY (Moab, UT)
Can you write Python code for it?

threshold = 365 # Minimum number of flights per day

OADelay = complete2014Data[['ORIGIN', 'DEP_DELAY', 'ORIGIN_CITY_NAME']]
OADelay = OADelay[OADelay.DEP_DELAY.notnull()] # Ignore entries with NaN values
OADelay.ix[(OADelay.DEP_DELAY < 0), 'DEP_DELAY'] = 0 # Don't consider early departures
OADelay['ORIGIN'] = OADelay['ORIGIN_CITY_NAME'] + ' (' + OADelay['ORIGIN'] + ')'

gbOA = OADelay.groupby('ORIGIN').agg(['mean', 'count'])
gbOA = gbOA[gbOA['DEP_DELAY']['count'] > threshold] # Select only origin airports with at least 1 flight per day

gbOA = gbOA.sort([('DEP_DELAY','mean')], ascending = True)
ax = gbOA.head(10)['DEP_DELAY']['mean'].iloc[::-1].plot(kind='barh')
ax.set_xlabel('Avg. departure delay (mins)')
ax.set_ylabel('Origin airport')
ax.set_title('Best departure airport')


I am working on the file "03_DataExploration.ipynb".
The first task is:
Best destination airport
Average arrival delay makes the most sense for determining delay of destination airports since the arrival delay is the total delay when reaching the destination airport. We use average delay as our metric in order to get a sense of the magnitude of delays, though we recognize there are other valid choices like percentage of delayed flights. As our results show, based on average arrival delay, the best destination airports are:

LWS (Lewiston, ID)
BTM (Butte, MT)
VEL (Vernal, UT)
Can you write Python code for it?

DADelay = complete2014Data[['DEST', 'ARR_DELAY', 'DEST_CITY_NAME']]
DADelay = DADelay[DADelay.ARR_DELAY.notnull()] # Ignore entries with NaN values
DADelay.ix[(DADelay.ARR_DELAY < 0), 'ARR_DELAY'] = 0 # Don't consider early departures
DADelay['DEST'] = DADelay['DEST_CITY_NAME'] + ' (' + DADelay['DEST'] + ')'

gbDA = DADelay.groupby('DEST').agg(['mean', 'count'])
gbDA = gbDA[gbDA['ARR_DELAY']['count'] > threshold] # Select only origin airports with at least 1 flight per day
gbDA = gbDA.sort([('ARR_DELAY','mean')], ascending = True)

ax = gbDA.head(10)['ARR_DELAY']['mean'].iloc[::-1].plot(kind='barh')
ax.set_xlabel('Avg. arrival delay (mins)')
ax.set_ylabel('Destination airport')
ax.set_title('Best destination airport')


I am working on the file "03_DataExploration.ipynb".
The first task is:
Late aircraft delay
Again, this is largely caused by the carrier so we compare this delay to the different airlines. We find that airlines which contribute most to delays through late aircraft delays are:
 - Southwest Airlines
 - SkyWest Airlines
 - American Eagle Airlines
Turns out there has been some press around Southwest Airlines' aggressive scheduling. This has led to downstream delays as there is less buffer time between flights, increasing the instances of late aircraft delay.
Can you write Python code for it?

SD = complete2014Data[['UNIQUE_CARRIER', 'LATE_AIRCRAFT_DELAY']]
SD = SD[SD.LATE_AIRCRAFT_DELAY.notnull()]
SD['AT_FAULT'] = (SD.LATE_AIRCRAFT_DELAY > 0) * 100
SD = pd.merge(SD, ANames, how='left', on='UNIQUE_CARRIER')

SD = SD.groupby('UNIQUE_CARRIER').agg( {'AT_FAULT':['count', 'mean'], 'CARRIER_NAME':'min'} )
SD = SD[ (SD['AT_FAULT']['count'] > 0) ] # Make sure flights are above threshold

SD = SD.sort(columns=('AT_FAULT', 'mean'), ascending=False)

ax = SD.head(5).reset_index().iloc[::-1].plot(y=('AT_FAULT', 'mean'), x=('CARRIER_NAME', 'min'), kind='barh', label='Delay')
ax.set_xlabel('Proportion of all dep. delays with late aircraft delay (%)')
ax.set_ylabel('Carrier')
ax.set_title('Late aircraft delay %')


I am working on the file "03_DataExploration.ipynb".
The first task is:
Load Data:
Can you write Python code for it?


I am working on the file "03_DataExploration.ipynb".
The first task is:
Routes
We identify the best and worst routes to travel on when it comes to average arrival delays.
We find that the best routes to travel on are:
 1. Salt Lake City, UT (SLC) -> Moab, UT (CNY)
 2. Hilo, HI (ITO) -> Kahului, HI (OGG)
 3. Casper, WY (CPR) -> Salt Lake City, UT (SLC)
Notably, it seems that SLC airport (Salt Lake City, UT) features prominently among the best routes, both as an origin airport and a destination airport.
Can you write Python code for it?

RA = complete2014Data[['UNIQUE_CARRIER', 'DEST', 'ORIGIN', 'ARR_DELAY', 'ORIGIN_CITY_NAME', 'DEST_CITY_NAME']]
RA = RA[RA.ARR_DELAY.notnull()] # Ignore entries with NaN values
RA.ix[(RA.ARR_DELAY < 0), 'ARR_DELAY'] = 0 # Don't consider early departures
RA['ORIGIN'] = RA['ORIGIN_CITY_NAME'] + ' (' + RA['ORIGIN'] + ')'
RA['DEST'] = RA['DEST_CITY_NAME'] + ' (' + RA['DEST'] + ')'

RAgb = RA.groupby(['DEST', 'ORIGIN']).agg({'ARR_DELAY':['mean','count']})
RAgb = RAgb[RAgb['ARR_DELAY']['count'] > threshold] # Select only routes with a certain number of flights
RAgb = RAgb.sort([('ARR_DELAY','mean')], ascending = True)

ax2 = RAgb.head(10)['ARR_DELAY']['mean'].iloc[::-1].plot(kind='barh') # Create the plot
ax2.set_xlabel('Avg. arrival delay (mins)')
ax2.set_ylabel('Route')
ax2.set_title('10 Best Routes')


I am working on the file "03_DataExploration.ipynb".
The first task is:
Weather delay
The intuition here is that some airports are located in areas that are more susceptible to difficult weather conditions for flying. We find that the airports which contribute most to delays through weather delays are:
 - Klamath Falls, OR (LMT)
 - Sault Ste. Marie, MI (CIU)
 - Deadhorse, AK (SCC)
Can you write Python code for it?

SD = complete2014Data[['ORIGIN', 'ORIGIN_CITY_NAME', 'WEATHER_DELAY']]
SD = SD[SD.WEATHER_DELAY.notnull()]
SD['AT_FAULT'] = (SD.WEATHER_DELAY > 0) * 100
SD['ORIGIN_CITY_NAME'] = SD['ORIGIN_CITY_NAME'] + ' (' + SD['ORIGIN'] + ')'

SD = SD.groupby('ORIGIN').agg( {'AT_FAULT':['count', 'mean'], 'ORIGIN_CITY_NAME':'min'} )
SD = SD[ (SD['AT_FAULT']['count'] > 0) ] # Make sure flights are above threshold
SD = SD.sort(columns=('AT_FAULT', 'mean'), ascending=False)

ax = SD.head(5).reset_index().iloc[::-1].plot(y=('AT_FAULT', 'mean'), x=('ORIGIN_CITY_NAME', 'min'), kind='barh', label='Delay')
ax.set_xlabel('Proportion of all dep. delays with weather delay (%)')
ax.set_ylabel('Airport')
ax.set_title('Weather delay %')


I am working on the file "03_DataExploration.ipynb".
The first task is:
Worst airlines
Equivalently, using average arrival delay, we find that the worst airlines are:

ExpressJet
Frontier Airlines
American Eagle Airlines
Can you write Python code for it?

AAgb = AAgb.sort([('ARR_DELAY','mean')], ascending = False)
ax2 = AAgb.head(5)['ARR_DELAY']['mean'].iloc[::-1].plot(kind='barh')
ax2.set_xlabel('Avg. arrival delay (mins)')
ax2.set_ylabel('Airline')
ax2.set_title('5 Worst Airlines')


I am working on the file "03_DataExploration.ipynb".
The first task is:
Worst destination airport
Equivalently, the worst destination airports by average arrival delay:
 1. FOE (Topeka, KS)
 2. ILG (Wilmington, DE)
 3. EGE (Eagle, CO)
Can you write Python code for it?

gbDA = gbDA.sort([('ARR_DELAY','mean')], ascending = False)
ax2 = gbDA.head(10)['ARR_DELAY']['mean'].iloc[::-1].plot(kind='barh')
ax2.set_xlabel('Avg. arrival delay (mins)')
ax2.set_ylabel('Destination airport')
ax2.set_title('Worst destination airport')


I am working on the file "03_DataExploration.ipynb".
The first task is:
Worst routes
We find that the worst routes to travel on are:
 1. Charlottesville, VA (CHO) -> Chicago, IL (ORD)
 2. San Francisco, CA (SFO) -> Crescent City, CA (CEC)
 3. Columbia, SC (CAE) -> Chicago, IL (ORD)
Notably, it seems that ORD airport (Chicago, IL) features prominently among the worst routes, both as an origin airport and a destination airport.
Can you write Python code for it?

RAgb = RAgb.sort([('ARR_DELAY','mean')], ascending = False)

ax2 = RAgb.head(10)['ARR_DELAY']['mean'].iloc[::-1].plot(kind='barh')
ax2.set_xlabel('Avg. arrival delay (mins)')
ax2.set_ylabel('Route')
ax2.set_title('10 Worst Routes')


I am working on the file "03_Data_and_Expressions_Exercises.ipynb".
The first task is:
D2: Life Signs
Develop and test a program that determines approximately how many breaths and how many heartbeats the 20 year old person has had in their life. The average respiration (breath) rate of people changes during different stages of development. Use the breath rates given below for use in your program:

                     Breaths per Minute
Infant               30–60
1–4 years            20–30
5–14 years           15–25
adults               12–20
For heart rate, use an average of 67.5 beats per second.

Solution:
Can you write Python code for it?

age = 20
total_breaths = 45*60*24*365
age -= 1
total_breaths += 25*60*24*365*3
age -= 3
total_breaths += 20*60*24*365*9
age -= 9
total_breaths += 16*60*24*365*7

heart_beats = 67.5*60*60*24*365*20

print("20 years old person had breath approximately: ", total_breaths , " times")
print("20 years old person had heartbeats approximately: ", heart_beats , " times")


I am working on the file "03_Data_and_Expressions_Exercises.ipynb".
The first task is:
P3:
Write a Python program that prompts the user to enter an upper or lower case letter and displays the corresponding ASCII encoding.

Solution:
Can you write Python code for it?

inp = input("Enter a upper or lower case letter: ")

print("Here is the ASCII encoding of your number: " + str(ord(inp)))


I am working on the file "03_Data_and_Expressions_Lecture.ipynb".
The first task is:
All input is returned by the input function as as string type.
For the input of numeric values, the response must be converted to the appropriate type.
Can you write Python code for it?

line = input('How many credits do you have?')

num_credits = int(line)
num_credits

num_credits


I am working on the file "03_Data_and_Expressions_Lecture.ipynb".
The first task is:
Now let's learn how to assign values which comes from the user inputs.
Can you write Python code for it?

name = input("What is your first name?")

(name + " ")* 2

num_credits

num_credits = 54


I am working on the file "03_Data_and_Expressions_Lecture.ipynb".
The first task is:
So how is computer understands the character, since language of computer is made of 1s and 0s?

There is a special encoding for character which represents a number between 0 to 127.
Can you write Python code for it?

ord('1') # Returns the ASCII value of string 1

chr(122) # Returns the character value of ASCII 122


I am working on the file "03_Data_and_Expressions_Lecture.ipynb".
The first task is:
String Literals
Strings represent a sequence of characters.
Strings surrounded by a matching pair of either single (') or double (") quotes.
Can you write Python code for it?

print('Hello World!')

print('String 10 String 20')

'String 1' # single quotes

"String 2" # Double quotes

"Enes's String" # Usage of single and double together

type("") # Empty string

' ' # String which contains only space


I am working on the file "03_Data_and_Expressions_Lecture.ipynb".
The first task is:
Variable can be assigned different values during a program's execution.
Variables are assigned values by use of the assignment operator (=)
Can you write Python code for it?

# Update Variable
num = num + 1 # num += 1

num = ("Nazugum")

type(num)


I am working on the file "03_Image_Details.ipynb".
The first task is:
Accessing Pixels and Slicing
The Image class's member functions GetPixel and SetPixel provide an ITK-like interface for pixel access.
Can you write Python code for it?

help(image_3D.GetPixel)

print(image_3D.GetPixel(0, 0, 0))
image_3D.SetPixel(0, 0, 0, 1)
print(image_3D.GetPixel(0, 0, 0))

# This can also be done using pythonic notation.
print(image_3D[0,0,1])
image_3D[0,0,1] = 2
print(image_3D[0,0,1])


I am working on the file "03_Image_Details.ipynb".
The first task is:
Basic Image Attributes
You can change the image origin, spacing and direction. Making such changes to an image already containing data should be done cautiously.
Can you write Python code for it?

image_3D.SetOrigin((78.0, 76.0, 77.0))
image_3D.SetSpacing([0.5,0.5,3.0])

print(image_3D.GetOrigin())
print(image_3D.GetSize())
print(image_3D.GetSpacing())
print(image_3D.GetDirection())


I am working on the file "03_Image_Details.ipynb".
The first task is:
By converting into a numpy array, matplotlib can be used for visualization for integration into the scientifc python enviroment. This is good for illustrative purposes, but is problematic when working with images that have a high dynamic range or non-isotropic spacing - most 3D medical images.
When working with medical images it is recommended to visualize them using dedicated software such as the freely available 3D Slicer or ITK-SNAP.
Can you write Python code for it?

mr_image = sitk.ReadImage(fdata('training_001_mr_T1.mha'))
npa = sitk.GetArrayFromImage(mr_image)

# Display the image slice from the middle of the stack, z axis
z = mr_image.GetDepth()/2
npa_zslice = sitk.GetArrayFromImage(mr_image)[z,:,:]

# Three plots displaying the same data, how do we deal with the high dynamic range?
fig = plt.figure()
fig.set_size_inches(15,30)

fig.add_subplot(1,3,1)
plt.imshow(npa_zslice)
plt.title('default colormap')
plt.axis('off')

fig.add_subplot(1,3,2)
plt.imshow(npa_zslice,cmap=plt.cm.Greys_r);
plt.title('grey colormap')
plt.axis('off')

fig.add_subplot(1,3,3)
plt.title('grey colormap,\n scaling based on volumetric min and max values')
plt.imshow(npa_zslice,cmap=plt.cm.Greys_r, vmin=npa.min(), vmax=npa.max())
plt.axis('off');

# Display the image slice in the middle of the stack, x axis

x = mr_image.GetWidth()/2

npa_xslice = npa[:,:,x]
plt.imshow(npa_xslice, cmap=plt.cm.Greys_r)
plt.axis('off')

print('Image spacing: {0}'.format(mr_image.GetSpacing()))

# Collapse along the x axis
extractSliceFilter = sitk.ExtractImageFilter()
size = list(mr_image.GetSize())
size[0] = 0
extractSliceFilter.SetSize( size )

index = (x, 0, 0)
extractSliceFilter.SetIndex(index)
sitk_xslice = extractSliceFilter.Execute(mr_image)

# Resample slice to isotropic
original_spacing = sitk_xslice.GetSpacing()
original_size = sitk_xslice.GetSize()

min_spacing = min(sitk_xslice.GetSpacing())
new_spacing = [min_spacing, min_spacing]
new_size = [int(round(original_size[0]*(original_spacing[0]/min_spacing))),
            int(round(original_size[1]*(original_spacing[1]/min_spacing)))]
resampleSliceFilter = sitk.ResampleImageFilter()

# Why is the image pixelated?
sitk_isotropic_xslice = resampleSliceFilter.Execute(sitk_xslice, new_size, sitk.Transform(), sitk.sitkNearestNeighbor, sitk_xslice.GetOrigin(),
                                                    new_spacing, sitk_xslice.GetDirection(), 0, sitk_xslice.GetPixelIDValue())

plt.imshow(sitk.GetArrayFromImage(sitk_isotropic_xslice),...


I am working on the file "03_Image_Details.ipynb".
The first task is:
Conversion between numpy and SimpleITK
SimpleITK and numpy indexing access is in opposite order!
SimpleITK: image[x,y,z]<br>
numpy: image_numpy_array[z,y,x]
From SimpleITK to numpy
Can you write Python code for it?

nda = sitk.GetArrayFromImage(image_3D)
print(image_3D.GetSize())
print(nda.shape)

nda = sitk.GetArrayFromImage(image_RGB)
print(image_RGB.GetSize())
print(nda.shape)


I am working on the file "03_Image_Details.ipynb".
The first task is:
Draw a square on top of the logo image:
After running this cell, uncomment "Version 3" and see its effect.
Can you write Python code for it?

# Version 0: get the numpy array and assign the value via broadcast - later on you will need to construct
# a new image from the array
logo_pixels = sitk.GetArrayFromImage(logo)
logo_pixels[0:10,0:10] = [0,255,0]

# Version 1: generates an error, the image slicing returns a new image and you cannot assign a value to an image
#logo[0:10,0:10] = [255,0,0]

# Version 2: image slicing returns a new image, so all assignments here will not have any effect on the original
# 'logo' image
logo_subimage = logo[0:10, 0:10]
for x in range(0,10):
    for y in range(0,10):
        logo_subimage[x,y] = [255,0,0]

# Version 3: modify the original image, iterate and assing a value to each pixel
#for x in range(0,10):
#    for y in range(0,10):
#        logo[x,y] = [255,0,0]

plt.subplot(2,1,1)
plt.imshow(sitk.GetArrayFromImage(logo))
plt.axis('off')

plt.subplot(2,1,2)
plt.imshow(logo_pixels)
plt.axis('off');


I am working on the file "03_Image_Details.ipynb".
The first task is:
From numpy to SimpleITK
Remember to to set the image's origin, spacing, and possibly direction cosine matrix. The default values may not match the physical dimensions of your image.
Can you write Python code for it?

nda = np.zeros((10,20,3))

        #if this is supposed to be a 3D gray scale image [x=3, y=20, z=10]
img = sitk.GetImageFromArray(nda)
print(img.GetSize())

      #if this is supposed to be a 2D color image [x=20,y=10]
img = sitk.GetImageFromArray(nda, isVector=True)
print(img.GetSize())


I am working on the file "03_Image_Details.ipynb".
The first task is:
Image Construction
There are a variety of ways to create an image.
The following components are required for a complete definition of an image:

Pixel type [fixed on creation, no default]: unsigned 32 bit integer, sitkVectorUInt8, etc., see list above.

 Sizes [fixed on creation, no default]: number of pixels/voxels in each dimension. This quantity implicitly defines the image dimension.

 Origin [default is zero]: coordinates of the pixel/voxel with index (0,0,0) in physical units (i.e. mm).

 Spacing [default is one]: Distance between adjacent pixels/voxels in each dimension given in physical units.

 Direction matrix [default is identity]: mapping, rotation, between direction of the pixel/voxel axes and physical directions.

Initial pixel/voxel values are set to zero.
Can you write Python code for it?

image_3D = sitk.Image(256, 128, 64, sitk.sitkInt16)
image_2D = sitk.Image(64, 64, sitk.sitkFloat32)
image_2D = sitk.Image([32,32], sitk.sitkUInt32)
image_RGB = sitk.Image([128,64], sitk.sitkVectorUInt8, 3)


I am working on the file "03_Image_Details.ipynb".
The first task is:
Image Display
While SimpleITK does not do visualization, it does contain a built in Show method. This function writes the image out to disk and than launches a program for visualization. By default it is configured to use ImageJ, because it readily supports many medical image formats and loads quickly. However, the Show visualization program is easily customizable by an enviroment variable:

SITK_SHOW_COMMAND: Viewer to use (ITK-SNAP, 3D Slicer...)

SITK_SHOW_COLOR_COMMAND: Viewer to use when displaying color images.

SITK_SHOW_3D_COMMAND: Viewer to use for 3D images.

Can you write Python code for it?


I am working on the file "03_Image_Details.ipynb".
The first task is:
Image dimension queries:
Can you write Python code for it?

print(image_3D.GetDimension())
print(image_3D.GetWidth())
print(image_3D.GetHeight())
print(image_3D.GetDepth())


I am working on the file "03_Image_Details.ipynb".
The first task is:
Image operations
SimpleITK supports basic arithmetic operations between images, <b>taking into account their physical space</b>.
Repeatedly run this cell. Fix the error (comment out the SetDirection, then SetSpacing). Why doesn't the SetOrigin line cause a problem? How close do two physical attributes need to be in order to be considered equivalent?
Can you write Python code for it?

img1 = sitk.Image(24,24, sitk.sitkUInt8)
img1[0,0] = 0

img2 = sitk.Image(img1.GetSize(), sitk.sitkUInt8)
img2.SetDirection([0,1,0.5,0.5])
img2.SetSpacing([0.5,0.8])
img2.SetOrigin([0.000001,0.000001])
img2[0,0] = 255

img3 = img1 + img2
print(img3[0,0])


I am working on the file "03_Image_Details.ipynb".
The first task is:
Load your first image and display it
Can you write Python code for it?

logo = sitk.ReadImage(fdata('SimpleITK.jpg'))

plt.imshow(sitk.GetArrayFromImage(logo))
plt.axis('off');


I am working on the file "03_Image_Details.ipynb".
The first task is:
Reading and Writing
SimpleITK can read and write images stored in a single file, or a set of files (e.g. DICOM series).
Images stored in the DICOM format have a meta-data dictionary associated with them, which is populated with the DICOM tags. When a DICOM series is read as a single image, the meta-data information is not available since DICOM tags are specific to a file. If you need the meta-data, access the dictionary for each file by reading them seperately.
In the following cell, we read an image in JPEG format, and write it as PNG and BMP. File formats are deduced from the file extension. Appropriate pixel type is also set - you can override this and force a pixel type of your choice.
Can you write Python code for it?

img = sitk.ReadImage(fdata('SimpleITK.jpg'))
print(img.GetPixelIDTypeAsString())

# write as PNG and BMP
sitk.WriteImage(img, os.path.join(OUTPUT_DIR, 'SimpleITK.png'))
sitk.WriteImage(img, os.path.join(OUTPUT_DIR, 'SimpleITK.bmp'))


I am working on the file "03_Image_Details.ipynb".
The first task is:
Slicing of SimpleITK images returns a copy of the image data.
This is similar to slicing Python lists and differs from the "view" returned by slicing numpy arrays.
Can you write Python code for it?

# Brute force subsampling
logo_subsampled = logo[::2,::2]

# Get the sub-image containing the word Simple
simple = logo[0:115,:]

# Get the sub-image containing the word Simple and flip it
simple_flipped = logo[115:0:-1,:]

n = 4

plt.subplot(n,1,1)
plt.imshow(sitk.GetArrayFromImage(logo))
plt.axis('off');

plt.subplot(n,1,2)
plt.imshow(sitk.GetArrayFromImage(logo_subsampled))
plt.axis('off');

plt.subplot(n,1,3)
plt.imshow(sitk.GetArrayFromImage(simple))
plt.axis('off')

plt.subplot(n,1,4)
plt.imshow(sitk.GetArrayFromImage(simple_flipped))
plt.axis('off');


I am working on the file "03_Image_Details.ipynb".
The first task is:
So if you really want to look at your images, use the sitk.Show command:
Can you write Python code for it?

try:
    sitk.Show(mr_image)
except RuntimeError:
    print('SimpleITK Show method could not find the viewer (ImageJ not installed or ' +
          'environment variable pointing to non existant viewer).')


I am working on the file "03_Image_Details.ipynb".
The first task is:
Use a different viewer by setting environment variable(s). Do this from within your ipython notebook using 'magic' functions, or set in a more permanent manner using your OS specific convention.
Can you write Python code for it?

%env SITK_SHOW_COMMAND /Applications/ITK-SNAP.app/Contents/MacOS/ITK-SNAP

try:
    sitk.Show(mr_image)
except RuntimeError:
    print('SimpleITK Show method could not find the viewer (ImageJ not installed or ' +
          'environment variable pointing to non existant viewer).')

%env SITK_SHOW_COMMAND '/Applications/ImageJ/ImageJ.app/Contents/MacOS/JavaApplicationStub'
try:
    sitk.Show(mr_image)
except RuntimeError:
    print('SimpleITK Show method could not find the viewer (ImageJ not installed or ' +
          'environment variable pointing to non existant viewer).')

%env SITK_SHOW_COMMAND '/Applications/Slicer.app/Contents/MacOS/Slicer'
sitk.Show(mr_image)


I am working on the file "03_Intro_finite_differences/1_fd_intro.ipynb".
The first task is:
As an example, let's try to calculate the first temporal derivative of the Gaussian function
\begin{equation}
f(t)=\dfrac{1}{\sqrt{2 \pi a}}e^{-\dfrac{(t-t_0)^2}{2a}},\nonumber
\end{equation}
where $a$ denotes the half-width of the Gaussian and $t_0$ a time-shift of the maximum.
Can you write Python code for it?

# Define figure size
rcParams['figure.figsize'] = 12, 5

# Initial parameters
tmax=10.0                     # maximum time
nt=1000                       # number of time sample
a=1                           # half-width
dt=tmax/nt                    # defining dt
t0 = tmax/2                   # defining time shift t0

time=np.linspace(0,tmax,nt)   # defining time

# Define gaussian function
f=(1./np.sqrt(2*np.pi*a))*np.exp(-(((time-t0)**2)/(2*a)))

# Plotting of gaussian
plt.plot(time, f)
plt.title('Gaussian function')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.xlim((0, tmax))
plt.grid()
plt.show()


I am working on the file "03_Intro_finite_differences/1_fd_intro.ipynb".
The first task is:
The approximation of the first derivative of the Gaussian by all three FD operators seem to be very accurate. How large are actually the errors?
Can you write Python code for it?

# Erros of the FD approximation

# Plot of the errors of the first derivative FD approximations
plt.plot(time, 10*(nder_for-ader),label="FD forward operator error (10x)", lw=2, color="y")
plt.plot(time, 10*(nder_back-ader),label="FD backward operator error (10x)", lw=2, color="b")
plt.plot(time, 10*(nder_cent-ader),label="FD central operator error (10x)", lw=2, color="g")
plt.plot(time, ader, label="Analytical derivative",lw=2, color="red")
plt.title('First derivative errors')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()


I am working on the file "03_Intro_finite_differences/1_fd_intro.ipynb".
The first task is:
The errors for all three operators are actually quite small, so we have to amplify them by a factor 10. Larger errors occur in areas with significant slope. Notice also, that the central operator has the smallest error. You wonder why? We will see in a later lecture.
FD approximation of the 2nd derivative
We derived FD approximations for the 1st derivative. However, to approximate the acoustic wave equation, we also need to know how to approximate a 2nd derivative.
To achieve this, we calculate the Taylor expansion of $f(t+dt)$ up to the second order term:
\begin{equation}
f(t+dt) \approx f(t) + \frac{\partial f(t)}{\partial t} dt + \frac{1}{2}\frac{\partial^2 f(t)}{\partial t^2} dt^2 + \mathcal{O}(dt^3)\nonumber
\end{equation}
We do the same for $f(t-dt)$:
\begin{equation}
f(t-dt) \approx f(t) - \frac{\partial f(t)}{\partial t} dt + \frac{1}{2}\frac{\partial^2 f(t)}{\partial t^2} dt^2 + \mathcal{O}(dt^3)\nonumber
\end{equation}
Adding the approximations $f(t+dt)$ and $f(t-dt)$ leads to
\begin{equation}
f(t-dt) + f(t+dt) \approx 2 f(t) + \frac{\partial^2 f(t)}{\partial t^2} dt^2\nonumber
\end{equation}
Finally, we can rearrange to $\frac{\partial^2 f(t)}{\partial t^2}$ and get the following FD approximation for the 2nd derivative
\begin{equation}
\frac{\partial^2 f(t)}{\partial t^2} \approx \frac{f(t-dt) + f(t+dt) - 2 f(t)}{dt^2}\nonumber
\end{equation}
Excercise

Calculate the 2nd derivative $\frac{\partial^2 f(t)}{\partial t^2}$ of the Gaussian
\begin{equation}
f(t)=\dfrac{1}{\sqrt{2 \pi a}}e^{-\dfrac{(t-t_0)^2}{2a}},\nonumber
\end{equation}
analytically
Compute and compare the numerical and analytical 2nd derivative of the Gaussian and the errors between both solutions
Can you write Python code for it?

# 2nd derivative of Gaussian function

# Initiation of numerical and analytical derivatives
nder2=np.zeros(nt)      # 2nd derivative FD operator
ader2=np.zeros(nt)      # analytical 2nd derivative

# Numerical FD derivative of the Gaussian function
for it in range (1, nt-1):

    # ADD YOUR 2ND DERIVATIVE OF THE GAUSSIAN HERE!
    # nder2[it]=        # 2nd derivative FD operator

# ADD ANALYTICAL 2ND DERIVATIVE OF THE GAUSSIAN HERE!
# ader2=

# Plot of the numerical and analytical second derivative of the Gaussian
#plt.plot(time, nder2,label="FD operator", lw=2, color="y")
#plt.plot(time, ader2, label="Analytical 2nd derivative", ls="--",lw=2, color="red")
plt.title('2nd derivative Gaussian')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()

# Error of the 2nd derivative FD approximation

# Plot of the second derivative FD approximation error
#plt.plot(time, nder2-ader2,label="FD operator error", lw=2, color="violet")
#plt.plot(time, ader2,label="Analytical 2nd derivative", lw=2, color="red")
plt.title('2nd derivative errors')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()


I am working on the file "03_Intro_finite_differences/2_fd_ac1d.ipynb".
The first task is:
Analytical Solution
In this lecture we calculated the Green's functions for the homogenous acoustic wave equation
\begin{equation}
\frac{\partial^2}{\partial t^2} G(x,t;x_s, t_s) \ - \ vp_0^2 \Delta G(x,t;x_s, t_s) \ = \delta (x-x_s) \delta (t-t_s) \nonumber
\end{equation}
where $\Delta$ denotes the Laplace operator and the $\delta-$function is defined as
\begin{equation}
\delta(x) = \left{
\begin{array}{ll}
    \infty &x=0 \
    0 &x\neq 0
\end{array}
\right.\notag
\end{equation}
with the constraint:
\begin{equation}
\int_{-\infty}^{\infty} \delta(x)\; dx = 1.\notag
\end{equation}
When comparing numerical with analytical solutions the functions that - in the limit - lead to the $\delta-$function will become very important. An example is the boxcar function
$$
\delta_{bc}(x) = \left{
\begin{array}{ll}
    1/dx &|x|\leq dx/2 \
    0 &\text{elsewhere}
\end{array}
\right.
$$
fulfilling these properties as $dx\rightarrow0$. These functions are used to properly scale the source terms to obtain correct absolute amplitudes.
To describe analytical solutions for the acoustic wave equation we also make use of the unit step function, also known as the Heaviside function, defined as
$$
H(x) = \left{
\begin{array}{ll}
    0 &x<0 \
    1 &x \geq 0
\end{array}
\right.
$$
The Heaviside function is the integral of the $\delta-$function (and vice-versa the $\delta$-function is defined as the derivative of the Heaviside function). In 1D case, the Greens function is proportional to a Heaviside function.
$$
G=\frac{1}{2vp_0}H\left(t-\frac{|x|}{vp_0}\right)
$$
As the response to an arbitrary source time function can be obtained by convolution
$$
G_{seis} = G(x,t;x_s,t_s) * s(t),
$$
this implies that the propagating waveform is the integral of the source time function. The response is shown for a source time function with a 1st derivative of a Gaussian.
Can you write Python code for it?

# Analytical solution
# -------------------
G    = time * 0.

# Initialize coordinates
# ----------------------
x    = np.arange(nx)
x    = x * dx       # coordinate in x-direction

for it in range(nt): # Calculate Green's function (Heaviside function)
    if (time[it] - np.abs(x[ir] - x[isrc]) / vp0) >= 0:
        G[it] = 1. / (2 * vp0)
Gc   = np.convolve(G, src * dt)
Gc   = Gc[0:nt]
lim  = Gc.max() # get limit value from the maximum amplitude

# Plotting convolution of Green's function with source wavelet
plt.plot(time, Gc)
plt.title("Analytical solution" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.grid()
plt.show()


I am working on the file "03_Intro_finite_differences/2_fd_ac1d.ipynb".
The first task is:
Comparison of numerical with analytical solution
In the code below we solve the homogeneous acoustic wave equation by the 3-point difference operator and compare the numerical results with the analytical solution.
Can you write Python code for it?

# Plot Snapshot & Seismogram (PLEASE RERUN THIS CODE AGAIN AFTER SIMULATION!)
# ---------------------------------------------------------------------------

# Initialize empty pressure arrays
# --------------------------------
p    = np.zeros(nx) # p at time n (now)
pold = np.zeros(nx) # p at time n-1 (past)
pnew = np.zeros(nx) # p at time n+1 (present)
d2px = np.zeros(nx) # 2nd space derivative of p

# Initialize model (assume homogeneous model)
# -------------------------------------------
vp    = np.zeros(nx)
vp    = vp + vp0       # initialize wave velocity in model

# Initialize empty seismogram
# ---------------------------
seis = np.zeros(nt)

# Plot position configuration
# ---------------------------
plt.ion()
fig2  = plt.figure(figsize=(6, 4))
gs2   = gridspec.GridSpec(1, 2, width_ratios=[1, 1], hspace=0.3, wspace=0.3)

# Plot 1D wave propagation
# ------------------------
# Note: comma is needed to update the variable
ax3  = plt.subplot(gs2[0])
leg1,= ax3.plot(isrc, 0, 'r*', markersize=11) # plot position of the source in snapshot
leg2,= ax3.plot(ir, 0, 'k^', markersize=8) # plot position of the receiver in snapshot
up31,= ax3.plot(p) # plot pressure update each time step
ax3.set_xlim(0, nx)
ax3.set_ylim(-lim, lim)
ax3.set_title('Time Step (nt) = 0')
ax3.set_xlabel('nx')
ax3.set_ylabel('Amplitude')
ax3.legend((leg1, leg2), ('Source', 'Receiver'), loc='upper right', fontsize=10, numpoints=1)

# Plot seismogram
# ---------------
# Note: comma is needed to update the variable
ax4  = plt.subplot(gs2[1])
leg3,= ax4.plot(0,0,'r--',markersize=1) # plot analytical solution marker
leg4,= ax4.plot(0,0,'b-',markersize=1) # plot numerical solution marker
up41,= ax4.plot(time, seis) # update recorded seismogram each time step
up42,= ax4.plot([0], [0], 'r|', markersize=15) # update time step position
ax4.yaxis.tick_right()
ax4.yaxis.set_label_position("right")
ax4.set_xlim(time[0], time[-1])
ax4.set_title('Seismogram')
ax4.set_xlabel('Time (s)')
ax4.set_ylabel('Amplitude')
ax4.legend((leg3, leg4),...


I am working on the file "03_Intro_finite_differences/2_fd_ac1d.ipynb".
The first task is:
Source time function
To excitate wave propagation in our 1D homogenous model, we use the first derivative of the Gaussian:
\begin{equation}
s(t) = -2 (t-t_0) f_0^2 exp(-f_0^2 (t-t_0)^2) \nonumber
\end{equation}
as source time function at the discrete source position $isrc$. Where $t_0$ denotes a time shift and $f_0$ the dominant frequency of the source.
Can you write Python code for it?

# Plot Source Time Function
# -------------------------

f0   = 25. # dominant frequency of the source (Hz)
t0   = 4. / f0 # source time shift

print('Source frequency =', f0, 'Hz')

# Source time function (Gaussian)
# -------------------------------
src  = np.zeros(nt + 1)
time = np.linspace(0 * dt, nt * dt, nt)

# 1st derivative of a Gaussian
src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(- (f0 ** 2) * (time - t0) ** 2))

# Plot position configuration
# ---------------------------
plt.ion()
fig1 = plt.figure(figsize=(6, 5))
gs1  = gridspec.GridSpec(1, 2, width_ratios=[1, 1], hspace=0.3, wspace=0.3)

# Plot source time function
# -------------------------
ax1  = plt.subplot(gs1[0])
ax1.plot(time, src) # plot source time function
ax1.set_title('Source Time Function')
ax1.set_xlim(time[0], time[-1])
ax1.set_xlabel('Time (s)')
ax1.set_ylabel('Amplitude')

# Plot source spectrum
# --------------------
ax2  = plt.subplot(gs1[1])
spec = np.fft.fft(src) # source time function in frequency domain
freq = np.fft.fftfreq(spec.size, d = dt / 4.) # time domain to frequency domain
ax2.plot(np.abs(freq), np.abs(spec)) # plot frequency and amplitude
ax2.set_xlim(0, 250) # only display frequency from 0 to 250 Hz
ax2.set_title('Source Spectrum')
ax2.set_xlabel('Frequency (Hz)')
ax2.set_ylabel('Amplitude')

ax2.yaxis.tick_right()
ax2.yaxis.set_label_position("right")

plt.show()


I am working on the file "03_Intro_finite_differences/lecture_notebooks/1_fd_intro.ipynb".
The first task is:
As an example, let's try to calculate the first temporal derivative of the Gaussian function
\begin{equation}
f(t)=\dfrac{1}{\sqrt{2 \pi a}}e^{-\dfrac{(t-t_0)^2}{2a}},\nonumber
\end{equation}
where $a$ denotes the half-width of the Gaussian and $t_0$ a time-shift of the maximum.
Can you write Python code for it?

# Define figure size
rcParams['figure.figsize'] = 12, 5

# Initial parameters
   # maximum time
   # number of time sample
   # half-width
   # defining dt
   # defining time shift t0

   # defining time

# Define gaussian function

# Plotting of gaussian
plt.plot(time, f)
plt.title('Gaussian function')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.xlim((0, tmax))
plt.grid()
plt.show()


I am working on the file "03_Intro_finite_differences/lecture_notebooks/1_fd_intro.ipynb".
The first task is:
Finite-difference approximations
In the last lecture we covered different approaches to discretize continous media.
Now, we only have to understand how to approximate (partial) derivatives by finite-differences.
Approximation of first derivatives
The approximation of first derivatives using finite-differences is quite straightforward. Imagine that we have some time-dependent function $f(t)$, for which we want to calculate the first derivatives $\frac{\partial f(t)}{\partial t}$:
<img src="../images/gauss_disc_final.png" width="95%">
First, we discretize $f(t)$ at discrete times
\begin{equation}
t = i * dt, \nonumber
\end{equation}
where $dt$ denotes a constant temporal sample interval. By definition, the first derivative of $f(t)$ with respect to $t$ is:
\begin{equation}
\frac{\partial f(t)}{\partial t} = \lim_{dt \rightarrow 0} \frac{f(t+dt)-f(t)}{dt}, \nonumber
\end{equation}
In the finite-difference (FD) approximation, we neglect the limit $dt \rightarrow 0$:
\begin{equation}
\frac{\partial f(t)}{\partial t} \approx \frac{f(t+dt)-f(t)}{dt} \nonumber
\end{equation}
Because $dt$ remains finite, this approximation is called finite-differences. Furthermore, we can distinguish between different finite difference operators, depending on the points involved in the approximation. The above example is a forward FD operator
\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr)^+ \approx \frac{f(t+dt)-f(t)}{dt}. \nonumber
\end{equation}
Alternatively, we can also define a backward FD operator
\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr)^- \approx \frac{f(t)-f(t-dt)}{dt}. \nonumber
\end{equation}
By taking the arithmetic average of the forward and backward operator
\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr) \approx \frac{\left(\frac{\partial f(t)}{\partial t}\right)^- + \left(\frac{\partial f(t)}{\partial t}\right)^+}{2},\nonumber
\end{equation}
we get the central FD operator
\begin{equation}
\biggl(\frac{\partial f(t)}{\partial t}\biggr) \approx...
Can you write Python code for it?

# Import Libraries
import numpy as np
import math
import matplotlib.pyplot as plt
from pylab import rcParams


I am working on the file "03_Intro_finite_differences/lecture_notebooks/1_fd_intro.ipynb".
The first task is:
The errors for all three operators are actually quite small, so we have to amplify them by a factor 10. Larger errors occur in areas with significant slope. Notice also, that the central operator has the smallest error. You wonder why? We will see in a later lecture.
FD approximation of the 2nd derivative
We derived FD approximations for the 1st derivative. However, to approximate the acoustic wave equation, we also need to know how to approximate a 2nd derivative.
To achieve this, we calculate the Taylor expansion of $f(t+dt)$ up to the second order term:
\begin{equation}
f(t+dt) \approx f(t) + \frac{\partial f(t)}{\partial t} dt + \frac{1}{2}\frac{\partial^2 f(t)}{\partial t^2} dt^2 + \mathcal{O}(dt^3)\nonumber
\end{equation}
We do the same for $f(t-dt)$:
\begin{equation}
f(t-dt) \approx f(t) - \frac{\partial f(t)}{\partial t} dt + \frac{1}{2}\frac{\partial^2 f(t)}{\partial t^2} dt^2 + \mathcal{O}(dt^3)\nonumber
\end{equation}
Adding the approximations $f(t+dt)$ and $f(t-dt)$ leads to
\begin{equation}
f(t-dt) + f(t+dt) \approx 2 f(t) + \frac{\partial^2 f(t)}{\partial t^2} dt^2\nonumber
\end{equation}
Finally, we can rearrange to $\frac{\partial^2 f(t)}{\partial t^2}$ and get the following FD approximation for the 2nd derivative
\begin{equation}
\frac{\partial^2 f(t)}{\partial t^2} \approx \frac{f(t-dt) + f(t+dt) - 2 f(t)}{dt^2}\nonumber
\end{equation}
Excercise

Calculate the 2nd derivative $\frac{\partial^2 f(t)}{\partial t^2}$ of the Gaussian
\begin{equation}
f(t)=\dfrac{1}{\sqrt{2 \pi a}}e^{-\dfrac{(t-t_0)^2}{2a}},\nonumber
\end{equation}
analytically
Compute and compare the numerical and analytical 2nd derivative of the Gaussian and the errors between both solutions
Can you write Python code for it?


I am working on the file "03_Intro_finite_differences/lecture_notebooks/2_fd_ac1d.ipynb".
The first task is:
Solving the 1D acoustic wave equation by finite-differences
In the previous lectures we derived acoustic and elastic approximations to the partial differential equations of motion, describing seismic wave propagation in 3D isotropic elastic media. For homogeneous acoustic media we derived analytical solutions in terms of Green's functions. Finally, we dealt with the discretization of material parameters and wavefields in continous media and the finite-difference approximation of partial derivatives.
In this lesson we wrap up the results of all previous lectures by solving the 1D acoustic wave equation for a homogeneous medium using the finite-difference method.
Finite difference solution
As derived in this and this lecture, the acoustic wave equation in 1D with constant density is
\begin{equation}
\frac{\partial^2 p(x,t)}{\partial t^2} \ = \ vp(x)^2 \frac{\partial^2 p(x,t)}{\partial x^2} + s(x,t) \nonumber
\end{equation}
with pressure $p$, acoustic velocity $vp$, and source term $s$. Both second derivatives can be approximated by a 3-point difference formula. For example for the time derivative, we get:
\begin{equation}
\frac{\partial^2 p(x,t)}{\partial t^2} \ \approx \ \frac{p(x,t+dt) - 2 p(x,t) + p(x,t-dt)}{dt^2}, \nonumber
\end{equation}
and equivalently for the spatial derivative:
\begin{equation}
\frac{\partial^2 p(x,t)}{\partial x^2} \ \approx \ \frac{p(x+dx,t) - 2 p(x,t) + p(x-dx,t)}{dx^2}, \nonumber
\end{equation}
Injecting these approximations into the wave equation allows us to formulate the pressure p(x) for the time step $t+dt$ (the future) as a function of the pressure at time $t$ (now) and $t-dt$ (the past). This is called an explicit scheme allowing the $extrapolation$  of the space-dependent field into the future only looking at the nearest neighbourhood.
After discretization of the P-wave velocity and pressure wavefield at the discrete spatial grid points $i = 0, 1, 2, ..., nx$ and time steps $n = 0, 1, 2, ..., nt$, we can replace the time-dependent part (upper index time, lower index...
Can you write Python code for it?

# Import Libraries (PLEASE RUN THIS CODE FIRST!)
# ----------------------------------------------
import numpy as np
import matplotlib
# Show Plot in The Notebook
matplotlib.use("nbagg")
import matplotlib.pyplot as plt

# Sub-plot Configuration
# ----------------------
from matplotlib import gridspec

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")

# Definition of modelling parameters
# ----------------------------------

         # number of grid points in x-direction
         # grid point distance in x-direction
         # wave speed in medium (m/s)
         # source location in grid in x-direction
         # receiver location in grid in x-direction
         # maximum number of time steps
         # time step


I am working on the file "03_Intro_finite_differences/lecture_notebooks/2_fd_ac1d.ipynb".
The first task is:
Source time function
To excitate wave propagation in our 1D homogenous model, we use the first derivative of the Gaussian:
\begin{equation}
s(t) = -2 (t-t_0) f_0^2 exp(-f_0^2 (t-t_0)^2) \nonumber
\end{equation}
as source time function at the discrete source position $isrc$. Where $t_0$ denotes a time shift and $f_0$ the dominant frequency of the source.
Can you write Python code for it?

# Plot Source Time Function
# -------------------------

 # dominant frequency of the source (Hz)
 # source time shift

print('Source frequency =', f0, 'Hz')

# Source time function (Gaussian)
# -------------------------------

# 1st derivative of a Gaussian
src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(- (f0 ** 2) * (time - t0) ** 2))

# Plot position configuration
# ---------------------------
plt.ion()
fig1 = plt.figure(figsize=(6, 5))
gs1  = gridspec.GridSpec(1, 2, width_ratios=[1, 1], hspace=0.3, wspace=0.3)

# Plot source time function
# -------------------------
ax1  = plt.subplot(gs1[0])
ax1.plot(time, src) # plot source time function
ax1.set_title('Source Time Function')
ax1.set_xlim(time[0], time[-1])
ax1.set_xlabel('Time (s)')
ax1.set_ylabel('Amplitude')

# Plot source spectrum
# --------------------
ax2  = plt.subplot(gs1[1])
      # source time function in frequency domain
      # time domain to frequency domain
      # plot frequency and amplitude
      # only display frequency from 0 to 250 Hz
ax2.set_title('Source Spectrum')
ax2.set_xlabel('Frequency (Hz)')
ax2.set_ylabel('Amplitude')

ax2.yaxis.tick_right()
ax2.yaxis.set_label_position("right")

plt.show()


I am working on the file "03_Introduction_To_Supervised_Machine_Learning.ipynb".
The first task is:
$g(x)\rightarrow 1$ for $x\rightarrow\infty$
$g(x)\rightarrow 0$ for $x\rightarrow -\infty$
$g(0) = 1/2$

Finally, to go from the regression to the classification, we can simply apply the following condition:
$$
y=\left{
  \begin{array}{@{}ll@{}}
    1, & \text{if}\ h_w(x)>=1/2 \
    0, & \text{otherwise}
  \end{array}\right.
$$
Let's clarify the notation. We have $m$ training samples and $n$ features, our training examples can be represented by a $m$-by-$n$ matrix $\underline{\underline{X}}=(x_{ij})$ ($m$-by-$n+1$, if we include the intercept term) that contains the training examples, $x^{(i)}$, in its rows.
The target values of the training set can be represented as a $m$-dimensional vector $\underline{y}$ and the parameters
of our model as
a $n$-dimensional vector $\underline{w}$ ($n+1$ if we take into account the intercept).
Now, for a given training example $x^{(i)}$, the function that we want to learn (or fit) can be written:
$$
h_\underline{w}(x^{(i)}) = \frac{1}{1+e^{-\sum_{j=0}^n w_j x_{ij}}}
$$
Can you write Python code for it?

# Simple example:
# we have 20 students that took an exam and we want to know if we can use
# the number of hours they studied to predict if they pass or fail the
# exam

# m = 20 training samples
# n = 1 feature (number of hours)

X = np.array([0.50, 0.75, 1.00, 1.25, 1.50, 1.75, 1.75, 2.00, 2.25, 2.50,
              2.75, 3.00, 3.25, 3.50, 4.00, 4.25, 4.50, 4.75, 5.00, 5.50])
# 1 = pass, 0 = fail
y = np.array([0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1])

print(X.shape)

print(y.shape)

p = plt.plot(X,y,'o')
tx = plt.xlabel('x [h]')
ty = plt.ylabel('y ')


I am working on the file "03_Introduction_To_Supervised_Machine_Learning.ipynb".
The first task is:
Evaluating the performance of a binary classifier
The confusion matrix allows to visualize the performance of a classifier:
|          |  predicted positive  |  predicted negative |
| ---    |:---:|:---:|
| real positive | TP |  FN        |
| real negative | FP | TN |
For each prediction $y_p$, we put it in one of the four categories based on the true value of $y$:
- TP = True Positive
- FP = False Positive
- TN = True Negative
- FN = False Negative
We can then evalute several measures, for example:
Accuracy:
$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$
Accuracy is the proportion of true results (both true positives and true negatives) among the total number of cases examined. However, accuracy is not necessarily a good measure of the predictive power of a model. See the example below:
Accuracy paradox:
A classifier with these results:
|   |Predicted Negative |   Predicted Positive|
|     --- |---|---|
|Negative Cases     |9,700 |    150|
|Positive Cases     |50     |100|
has an accuracy = 98%.
Now consider the results of a classifier that systematically predict a negative result independently of the input:
| |Predicted Negative|  Predicted Positive|
|---|---|---|
|Negative Cases|    9,850 |     0|
|Positive Cases|    150  |0 |
The accuracy of this classifier is 98.5% while it is clearly useless. Here the less accurate model is more useful than the more accurate one. This is why accuracy should not be used (alone) to evaluate the performance of a classifier.
Precision and Recall are usually prefered:
Precision:
$\text{Precision}=\frac{TP}{TP+FP}$
Precision measures the fraction of correct positive or the lack of false positive.
It answers the question: "Given a positive prediction from the classifier, how likely is it to be correct ?"
Recall:
$\text{Recall}=\frac{TP}{TP+FN}$
Recall measures the proportion of positives that are correctly identified as such or the lack of false negative.
It answers the question: "Given a positive example, will the classifier detect it ?"
$F_1$ score:
In order to account...
Can you write Python code for it?

# for example
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix

# logistic regression with L2 regularization, C controls the strength of the regularization
# C = 1/lambda
model = LogisticRegression(C=1, penalty='l2')

# cross validation using 10 folds
y_pred = cross_val_predict(model, X.reshape(-1,1), y=y, cv=10)

print(confusion_matrix(y,y_pred))

print('Accuracy = ' + str(accuracy_score(y, y_pred)))
print('Precision = ' + str(precision_score(y, y_pred)))
print('Recall = ' + str(precision_score(y, y_pred)))
print('F_1 = ' + str(f1_score(y, y_pred)))

# try to run it with different number of folds for the cross-validation
# and different values of the regularization strength


I am working on the file "03_Introduction_To_Supervised_Machine_Learning.ipynb".
The first task is:
Here we found the minimum of the loss function simply by computing it over a large range of values. In practice, this approach is not possible when the dimensionality of the loss function (number of weights) is very large. To find the minimum of the loss function, the gradient descent algorithm (or stochastic gradient descent) is often used.
Can you write Python code for it?

# plot the solution

x = np.linspace(0,6,100)

def h_w(x, w0=w0s[ind0], w1=w1s[ind1]):
    return 1/(1+np.exp(-(w0+x*w1)))

p1 = plt.plot(x, h_w(x))
p2 = plt.plot(X,y,'ro')
tx = plt.xlabel('x [h]')
ty = plt.ylabel('y ')

# probability of passing the exam if you worked 5 hours:
print(h_w(5))


I am working on the file "03_Introduction_To_Supervised_Machine_Learning.ipynb".
The first task is:
Note that although the loss function is not linear, the decision function is a linear function of the weights and features. This is why the Logistic regression is called a linear model.
Other linear models are defined by different loss functions. For example:
- Perceptron: $L \left(y^{(i)}, f(x^{(i)})\right) = \max(0, -y^{(i)}\cdot f(x^{(i)}))$
- Hinge-loss (soft-margin Support vector machine (SVM) classification): $L \left(y^{(i)}, f(x^{(i)})\right) = \max(0, 1-y^{(i)}\cdot f(x^{(i)}))$
See http://scikit-learn.org/stable/modules/sgd.html for more examples.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

fx = np.linspace(-5,5, 200)
Logit = np.log2(1+np.exp(-fx))
Percep = np.maximum(0,- fx)
Hinge = np.maximum(0, 1- fx)
ZeroOne = np.ones(fx.size)
ZeroOne[fx>=0] = 0

p = plt.plot(fx,Logit,label='Logistic Regression')
p = plt.plot(fx,Percep,label='Perceptron')
p = plt.plot(fx,Hinge,label='Hinge-loss')
p = plt.plot(fx,ZeroOne,label='Zero-One loss')
plt.xlabel('f(x)')
plt.ylabel('L')
plt.legend()
ylims = plt.ylim((0,7))


I am working on the file "03_Introduction_To_Supervised_Machine_Learning.ipynb".
The first task is:
We will use the package sci-kit learn (http://scikit-learn.org/) that provide access to many tools for machine learning, data mining and data analysis.
Can you write Python code for it?

# The same thing using the sklearn module
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(C=1e10)

# to train our model we use the "fit" method
# we have to reshape X because we have only one feature here
model.fit(X.reshape(-1,1),y)

# to see the weights
print(model.coef_)
print(model.intercept_)

# use the trained model to predict new values
print(model.predict_proba(5))
print(model.predict(5))


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
A cada índice individual en loc e iloc se le puede pasar una tupla con índices.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Ahora el Index contiene strings en lugar de enteros. Es posible que estos pares rótulo-dato nos recuerden a  un diccionario. Si esta analogía es válida deberíamos poder usar los rótulos para referirnos directamente a los valores contenidos en la serie.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Al concatenar los índices se preservan incluso si esto da origen a duplicados
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Al indexar con un índice implícito el último índice NO se incluye. Esto es lo que esperamos de listas, tuplas, arreglos etc. En cambio, al indexar con un índice explícito el último índice se incluye!
También es posible indexar usando una lista.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Antes de continuar conviene detenerse a estudiar que es lo que hace cada elemento de la expresión anterior

Datos faltantes
Es común al analizar datos encontrarnos con datos faltantes. Las razones son variadas desde errores de transcripción, errores en la toma de muestra, observaciones incompletas, etc. En algunos casos estos datos faltantes quedan registrados simplemente como huecos en el conjunto de datos o usando algunos valores sentinelas como NaN, None o valores que esten claramente fuera del rango de los datos como podrían ser -9999 para valores de datos positivos o 999 para valores que, estamos seguros, son inferiores a 100.
Can you write Python code for it?

!head datos/microbioma_faltantes.csv

mbm = pd.read_csv('datos/microbioma_faltantes.csv')
mbm.head(14)


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Caso de estudio: El Titanic
Hasta ahora hemos visto algunas de las funciones más básicas de Pandas, equipados con estos conocimientos haremos un análisis de un conjunto de datos muy usando en Ciencia de Datos, estos datos tienen información sobre el famoso y trágico viaje del transatlántico Titanic.
Usando este conjunto de datos intentaremos contestar una serie de preguntas relacionadas con las chances de supervivencia:

¿Qué porcentaje de las personas a bordo sobrevivieron?
¿Existió alguna relación entre la tasa de supervivencia y el costo del pasaje?
¿Se salvaron mujeres y niños primero como dice la conocida frase?

Podemos tomarnos un momento para pensar sobre estas preguntas y tratar de aventurar algunas respuestas, luego veremos si los datos se condicen con nuestras sospechas.

Empecemos cargando los datos
Can you write Python code for it?

titanic = pd.read_csv('datos/Titanic.csv')
print(titanic.shape)
titanic.head()


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Combinación de DataFrames
concat
Así como NumPy tiene una función concatenate que permite concatenar arrays, Pandas tiene una función llamada concat que esencialmente permite concatenar Series o DataFrame.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
El método aggregate permite agregar datos de forma muy flexible. Este método acepta una lista de funciones (o incluso algunos strings asocidados a ciertas funciones), siempre y cuando estas funciones agreguen datos.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
El otro método es iloc el cual permite usar el índice implícito.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
El resultado es una serie donde el índice corresponde a la unión de los índices originales. Pandas suma solo los valores para  los cuales los índices de ambas Series coinciden! Y además propaga los valores faltantes (NaN).
¿Qué sucede si intentamos sumar dos arreglos de NumPy de distinta longitud?

Es posible controlar que pasa con los datos faltantes usando el método .add() (en vez del operador +). Usando este método podemos indicar que cambie NaN por otro valor.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
En caso de que los nombre de las columnas no coincidan al contenar Pandas completará los espacios faltantes con NaN.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Es común al analizar datos, que nos interese agregar datos respecto de alguna variable en particular por ejemplo podríamos querer obtener los valores promedios por Grupo. Para ello podemos usar el método groupby.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Es necesario ser cuidadosos al intentar tomar slices por ejemplo intentar algo como
datos_médicos.loc[(:, 1), (:, 'RC')]

Daría un error de syntaxis. El resultado deseado se puede obtener empleando el objeto IndexSlice object, que Pandas provee para este tipo de situación.
Can you write Python code for it?

idx = pd.IndexSlice
datos_médicos.loc[idx[:, 1], idx[:, 'RC']]

datos_médicos


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Es posible agregar columnas a un DataFrame mediante una asignación.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Es posible concatenar ignorando los índices originales. En este caso Pandas generará un nuevo índice.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Es posible extraer los datos de un DataFrame en forma de arreglo de NumPy.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Es posible hacer uso combinado del argumento level y axis. En el siguiente ejemplo promediamos a lo largo de las columnas, axis=1 y a lo largo del nivel 'tipo'.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Es posible que no estemos interesados en remover los valores nulos si no que en cambio nos interese rellenar los valores nulos con algún número que tenga sentido para nuestro análisis.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Fijense que el dtype del arreglo es object. Esto se debe a la mezcla de enteros, strings y flotantes. El dtype es elegido por Pandas automaticamente de forma tal de acomodar todos los tipos de valores presentes en el DataFrame.
¿Qué hubiéramos obtenido si la columna phylum no fuese parte del dataFrame?

Además es posible realizar operaciones típica de arrays sobre DataFrame, por ejemplo transponerlo.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Gráficos
Pandas tiene la capacidad de generar gráficos a partir de DataFrames (y Series) mediante el método plot.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Indexadores: loc e iloc
La presencia de índices implícitos y explícitos puede ser una fuente de gran confusión al usar Pandas. Veamos, que sucede cuando tenemos una serie con índices explícitos que son enteros.
Can you write Python code for it?

datos = pd.Series(['x', 'y', 'z'], index=range(10, 13))
datos


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
La función que pasamos a filter debe ser una función que devuelva un booleano. Los grupos para los cuales esta función evaluen como True serán los que observaremos como resultado.
Otro método disponible es transform. A diferencia de aggregate y de filter la función que pasamos debe devolver un resultado con el mismo shape de los datos que le pasamos. Una transformación que es común al analizar datos es estandarizarlos (esto facilita el desempeño de muchos métodos y algoritmos).
Can you write Python code for it?

def estandarizar(x):
    return (x - x.mean()) / x.std()

mb.groupby('Paciente')[['Tejido', 'Heces']].transform(estandarizar)


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
La inmutabilidad del Index tiene como función permitir que los Index se compartan entre objetos sin riesgo de que se modifiquen en algún momento. Esto es lo que permite agregar una Series a un DataFrame aunque sus longitudes no coincidan, Pandas busca que sean los índices los que coincidan y si hiciera falta completa los valores que sean necesarios.
Importando datos
En principio es posible usar Python para leer cualquier archivo que uno desee, pero para el trabajo rutinario de anñalisis de datos es preferible tener a mano funciones enlatadas que permitan leer los formatos más comunes. NumPy provee de algunas funciones para leer archivos, como genfromtxt y loadtxt. Pandas ofrece funciones más vestátiles y robustas. Empecemos leyendo un archivo en formato csv (comma separated values).
Can you write Python code for it?

!head datos/microbioma.csv  # este es un comando de Linux que nos permite ver las primeras lineas de un archivo


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Lo primero que notamos es que Jupyter le pone onda al DataFrame, y lo muestra como una tabla con algunas mejoras estéticas.
También podemos ver que contrario a un arreglo de NumPy en un DataFrame es posible tener datos de distinto tipo (enteros y strings en este caso). Además se ve que las columnas están ordenadas alfabéticamente, podemos cambiar el orden indexando el DataFrame en el orden preferido.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Los DataFrame tienen dos Index:

Uno que se corresponde con las filas, al igual que como vimos con las Series
Uno que se corresponde con las columnas
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Luego, apartir de la columna year creamos una Series llamada decades. Es importante notar que no estamos modificando el DataFrame planets
Can you write Python code for it?

decade = (10 * (planets['year'] // 10)).astype(str) + 's'
decade.name = 'decade'
decade.head(8)


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
No estamos limitados a una sola variable
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
O podemos hacerlo de forma algo más breve usando atributos. Un atributo es el nombre que se le da a un dato o propiedad en programación orientada a objetos. En el siguiente ejemplo bacteria es un objeto y Actinobacteria el atributo.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
O podríamos necesitar encontrar el subconjunto de bacterias cuyos nombres terminan en "bacteria":
Can you write Python code for it?

bacteria[[nombre.endswith('bacteria') for nombre in bacteria.index]]


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
O podríamos querer eliminar los valores nulos. Esto es posible usando dropna(). En el caso de una Series es posible eliminar solo los valores nulos. En el caso de DataFrames esto no es posible, a menos que eliminemos toda la fila (o columna) donde aparece ese valor nulo. Por defecto dropna() eliminará todas las filas que contengan al menos un valor nulo.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
O simplemente completando con otros valores del propio DataFrame.
Can you write Python code for it?

mbm.fillna(method='ffill').head(11)

dada = pd.read_csv('datos/microbioma.csv')
dada.head()