I am working on the file "1_notmnist.ipynb".
The first task is:
The display_overlap function above display one of the duplicate, the first element is from the first dataset, and the next ones are from the dataset used for the comparison.
Now that exact duplicates have been found, let's look for near duplicates. How to define near identical images? That's a tricky question. My first thought has been to use the allclose numpy matrix comparison. This is too restrictive, since two images can vary by one pyxel, and still be very similar even if the variation on the pyxel is large. A better solution involves some kind of average.
To keep is simple and still relevant, I will use a Manhattan norm (sum of absolute values) of the difference matrix. Since the images of the dataset have all the same size, I will not normalize the norm value. Note that it is pyxel by pyxel comparison, and therefore it will not scale to the whole dataset, but it will help to understand image similarities.
Can you write Python code for it?

MAX_MANHATTAN_NORM = 10

def extract_overlap_near(dataset_1, dataset_2):
  overlap = {}
  for i, img_1 in enumerate(dataset_1):
    for j, img_2 in enumerate(dataset_2):
      diff = img_1 - img_2
      m_norm = np.sum(np.abs(diff))
      if m_norm < MAX_MANHATTAN_NORM:
        if not i in overlap.keys():
          overlap[i] = []
        overlap[i].append(j)
  return overlap

%time overlap_test_train_near = extract_overlap_near(test_dataset[:200], train_dataset)

print('Number of near overlaps:', len(overlap_test_train_near.keys()))
display_overlap(overlap_test_train_near, test_dataset[:200], train_dataset)


I am working on the file "1_notmnist.ipynb".
The first task is:
The same value is found, so we can now sanetize the test and the train datasets.
Can you write Python code for it?

%time test_dataset_sanit, test_labels_sanit = sanetize(test_dataset, train_dataset, test_labels)
print('Overlapping images removed: ', len(test_dataset) - len(test_dataset_sanit))

%time valid_dataset_sanit, valid_labels_sanit = sanetize(valid_dataset, train_dataset, valid_labels)
print('Overlapping images removed: ', len(valid_dataset) - len(valid_dataset_sanit))

pickle_file_sanit = 'notMNIST_sanit.pickle'

try:
  f = open(pickle_file_sanit, 'wb')
  save = {
    'train_dataset': train_dataset,
    'train_labels': train_labels,
    'valid_dataset': valid_dataset_sanit,
    'valid_labels': valid_labels_sanit,
    'test_dataset': test_dataset_sanit,
    'test_labels': test_labels_sanit,
    }
  pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
  f.close()
except Exception as e:
  print('Unable to save data to', pickle_file, ':', e)
  raise

statinfo = os.stat(pickle_file_sanit)
print('Compressed pickle size:', statinfo.st_size)


I am working on the file "1_notmnist.ipynb".
The first task is:
The techniques above work well, but the performance is very low and the methods are poorly scalable to the full dataset. Let's try to improve the performance. Let's take some reference times on a small dataset.
Here are some ideas:
+ stop a the first occurence
+ nympy function where in diff dataset
+ hash comparison
Can you write Python code for it?

def extract_overlap_stop(dataset_1, dataset_2):
  overlap = {}
  for i, img_1 in enumerate(dataset_1):
    for j, img_2 in enumerate(dataset_2):
      if np.array_equal(img_1, img_2):
        overlap[i] = [j]
        break
  return overlap

%time overlap_test_train = extract_overlap_stop(test_dataset[:200], train_dataset)

print('Number of overlaps:', len(overlap_test_train.keys()))
display_overlap(overlap_test_train, test_dataset[:200], train_dataset)


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
1.2 Unpacking Elements from Iterables of Arbitrary Length:
Problem:
Unpacking unknown number of elements in tuple/sequence/iterables into variables

Solution:
Use "star expressions" for handling multiples:
Can you write Python code for it?

def drop_first_last(grades):
    """ Drop first and last exams, then average the rest. """
    first, *middle, last = grades
    return avg(middle)

def arbitrary_numbers():
    """ Name and email followed by phone number(s). """
    record = ('Dave', 'dave@example.com', '555-555-5555', '555-555-5544')
    name, email, *phone_numbers = record # phone_number always a list
    return phone_numbers

def recent_to_first_n():
    """ Most recent quarter compares to the average of the first n. """
    sales_records = ('23.444', '234.23', '0', 23.12, '15.56')
    *trailing_qtrs, current_qtr = sales_record
    trailing_avg = sum(trailing_qtrs) / len(trailing_qtrs)
    return avg_comparison(trailing_avg, current_qtr)


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
1.4 Finding the Largest or Smallest N Items
Problem:
Make a list of the largest or smallest N items in a collection.

Solution:
The heapq module has nlargest() and nsmallest()
Can you write Python code for it?

import heapq

nums = [1, 8, 2, 23, 7, -4, 18, 23, 42, 37, 2]
print(heapq.nlargest(3, nums))  # [42, 37 ,23]
print(heapq.nsmallest(3, nums)) # [-4, 1, 2]
heap.heappop(nums) # -4

# use key parameter to use with complicated data structures
portfolio = [
    {'name': 'IBM', 'shares': 100, 'price': 91.1},
    {'name': 'AAPL', 'shares': 50, 'price': 543.22},
    {'name': 'FB', 'shares': 200, 'price': 21.09},
    {'name': 'HPQ', 'shares': 35, 'price': 31.75},
    {'name': 'YHOO', 'shares': 45, 'price': 16.35},
    {'name': 'ACME', 'shares': 75, 'price': 115.65}
]

cheap = heapq.nsmallest(3, portfolio, key=lambda s: s['price'])
expensive = heapq.nlargest(3, portfolio, key=lambda s: s['price'])

# if N is close to the size of the items:
sorted(nums)[:N] # a better approach


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
1.7 Keepping Dictionaries in Order
Problem:
Control the order of items in a dictionary when iterating or serializing
Solution:
Use OrderedDict from the collections to control dictionary order. It is particularly useful when building a mapping that later will be serialized or encoded into a different format. For example, when controlling the order of fields appearing in a JSON encoding, first build the data in OrderedDict and then json dump.
Can you write Python code for it?

from collections import OrderedDict

d = OrderedDict()
d['foo'] = 1
d['bar'] = 2
d['spam'] = 3
d['grok'] = 4

for key in d:
    print(key, d[key]) # -> 'foo 1', 'bar 2', 'spam 3', 'grok 4'

# Use when serializing JSON
import json
json.dumps(d) # -> '{"foo": 1, "bar": 2, "spam": 3, "grok": 4}'


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
Discussion
When looking for N smallest/largest numbers, heapq provides superior performance. heap[0] is always the smallest number. Structures are converted into a list where items are ordered as a heap (underneath).

1.5 Implementing a Priority Queue
Problem:
Implement a queue that sorts items by a given priority and always returns the item with the highest priority on each pop operations.

Solution:
Use heapq to implement a simple priority queue
Can you write Python code for it?

import heapq

class PriorityQueue:
    def __init__(self):
        self._queue = []
        self._index = 0

    def __repr__(self):
        return 'PriorityQueue({}) with index({})'.format(self._queue, self._index)

    def push(self, item, priority):
        heapq.heappush(self._queue, (-priority, self._index, item)) # heappush(list, ())
        self._index += 1

    def pop(self):
        return heapq.heappop(self._queue)[-1] # self_queue includes [(priority, index, item)]

class Item:
    def __init__(self, name):
        self.name = name

    def __repr__(self):
        return 'Item({!r})'.format(self.name)

q = PriorityQueue()
print(q)
q.push(Item('foo'), 1)
print(q)
q.push(Item('bar'), 5)
print(q)
q.push(Item('spam'), 4)
print(q)
q.push(Item('grok'), 1)
print(q)
q.pop() # -> Item('bar')
print(q)
q.pop() # -> Item('spam')
print(q)
q.pop() # -> Item('foo')
print(q)
q.pop() # -> Item('grok')
print(q)

# foo and grok were popped in the same order in which they were inserted


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
Discussion:

The keys() method of a dictionary returns a keys-view object that exposes the keys. Key views support set operations: unions, intersections, and differences.

The items() method of a dictionary returns an items-view object consisting of (key, value) pairs. This object supports similar set operations and can be used to perform operations such as finding out which key-value pairs two dictionaries have in common.

Although similar, the values() method of a dictionary does not support the set operations described in this recipe. In part, this is due to the fact that unlike keys, the items contained in a values view aren’t guaranteed to be unique. However, if you must perform such calculations, they can be accomplished by simply converting the values to a set first.

1.10 Removing Duplicates form a Sequence while Maintaining Order
Problem:
Eliminate the duplicate values in a sequence, but preserve the order
Solution:

If the values in the sequence are hashable (preserve order), use a set and a generator.
If a sequence consists of unhashable types (dicts) use the key/lambda combo
The key/lambda combo also works well when eliminating duplicates based on the values of a single field, attribute, or a larger data structure
For an amazing explanation of iterables, iterators, generators and yield:
    http://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do-in-python
Can you write Python code for it?

###### 1 #########
def dedupe(items):
    ''' Add a unique item to the seen, and then check against seen.'''
    seen = set()
    for item in items:
        if item not in seen:
            yield item
            seen.add(item)

a = [1, 5, 2, 1, 9, 1, 5, 10]
list(dedupe(a)) # [1, 5, 2, 9, 10]

##### 2 ##########
def dedupe(items, key=None): # key is similar to min/max/sorted
    ''' Purpose of the key argument is to specify a function(lambda)
        that converts sequence items into a hashable type for the
        purposes of duplicate detection.
    '''
    seen = set()
    for item in items:
        val = item if key is None else key(item) # key could be lambda of values, keys, etc.
        if val not in seen:
            yield item
            seen.add(val)

a = [ {'x':1, 'y':2}, {'x':1, 'y':3}, {'x':1, 'y':2}, {'x':2, 'y':4}]
# remove duplicates based on x/y values
list(dedupe(a, key=lambda d: (d['x'], d['y']))) # [{'x': 1, 'y': 2}, {'x': 1, 'y': 3}, {'x': 2, 'y': 4}]

##### 3 #########
# remove duplicates based on x values - for each item in "a" sequence execute the lambda function
list(dedupe(a, key=lambda d: d['x'])) # [{'x': 1, 'y': 2}, {'x': 2, 'y': 4}]


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
Discussion:
An OrderedDict is an expensive procedure - beware for items exceeding 100000 lines:

internally maintains a doubly linked list that orders the keys according to insertion order; when a new item is first inserted, it is placed at the end of this list; subsequent reassignment of an existing key doesn't change the order.
be aware that the size of OrderedDict is more than twice as large as a normal dictionary due to the extra linked list that's created.
if building a data structure involving a large number of OrderedDict instances (> 100000 lines of CSV file into a list of OrderedDict instances) be careful!

1.8 Calculating with Dictionaries
Problem:
Performing various calculations (min, max, sort) on a dictionary
Solution:
Reverse keys and values, then perform a calculation function on the zip result.
Important:
1. max/min/sort is performed on the keys
2. if the keys are the same, max/min/sort is then based on the values
3. zip creates an iterator, which can only be consumed once
Can you write Python code for it?

prices = {
    'ACME': 45.23,
    'AAPL': 612.78,
    'IBM': 205.55,
    'HPQ': 37.20,
    'FB': 10.75
}

# to get calculated values first reverse and zip
min_price = min(zip(prices.values(), prices.keys())) # (10.75, 'FB')
max_price = max(zip(prices.values(), prices.keys())) # (612.78, 'AAPL')

# to rank the data use zip with sorted
prices_sorted = sorted(zip(prices.values(), prices.keys())) # [(10.75, 'FB'), (37.2, 'HPQ')...]

# the iterator can be consumed only once
prices_and_names = zip(prices.values(), prices.keys())
print(min(prices_and_names)) # result OK
print(max(prices_and_names)) # ValueError: max() arg is an empty sequence


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
Discussion:
This is often implemented with iterables of unknown(arbitrary) length, and known pattern: "everything after element 1 is a number".

Handy when iterating over a sequence of tuples of varying length or of tagged tuples.
Handy when unpacking with string processing operations
Handy when unpacking and throwing away some variables
Handly when spliting a list into head and tail components, which could be used to implement recursive solutions.
Can you write Python code for it?


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
Discussion:
This recipe focuses on the use of heapq module. Functions heapq.heappush() and heapq.heappop() insert and remove items from a list _queue so that the ~~first~~ last item in the list has the highest priority. That is the reason to negate "-priority" when pushing to heapq.heappush(). Hello world

heappop() and heappush() have O(log N) complexity
a queue consists of tuples (-priority, index, item); priority is negated so that to add items with the highest priority to the end of the _queue

index value is used to properly order items with the same priority; index also works for comparison operations:

By introducing the extra index and making (priority, index, item) tuples, you avoid this problem entirely since no two tuples will ever have the same value for index (and Python never bothers to compare the remaining tuple values once the result of comparison can be determined)::

a = (1, 0, Item('foo'))
b = (5, 1, Item('bar'))
c = (1, 2, Item('grok'))
a < b # True
a < c # True

we can use this queue for communication between threads, but we will need to add appropriate locking and signaling (look ahead)

1.6 Mapping Keys to Multiple Values in a Dictionary
Problem:
Make a dictionary that maps keys to more than one value (multidict)

Solution:
A dictionary is a mapping where each key is mapped to a single value. When mapping keys to multiple values, we need to store multiple values in a different container: list or set.

Use lists to preserve the insertion order of the items
Use sets to eliminate duplicates (when we don't care about the order)
Use defaultdict in the collections to construct such structure:
defaultdict automatically initializes the first value of the key
defaultdict automatically adds default values later on when accessing dictionary
if we don't want the above behavior, use setdefault (it is messier however)
Can you write Python code for it?

from collections import defaultdict

d = defaultdict(list) # multiple values will be added to a list
d['a'].append(1)
d['a'].append(2)
d['b'].append(4)

d = defaultdict(set) # multiple values will be added to a set
d['a'].add(1)
d['b'].add(2)
d['a'].add(5)

# Messier setdefault
d = {}
d.setdefault('a', []).append(1)
d.setdefault('a', []).append(2) # will add to the existing list

# Even messier
d = {}
for key, value in paiers:
    if key not in d:
        d[key] = []
    d[key].append(value)

# Best!
d = defaultdict(list)
for key, value in pairs:
    d[key].append(value)


I am working on the file "1c_Python_Cookbook_3/Chapter_1.ipynb".
The first task is:
Generator functions (with yield) are common when searching for items. This decouples the process of searching from the code that uses results:

deque(maxlen=5) uses  fixed-size queue; although we could append/delete items from a list, this is more elegant/faster
Handly when a simple queue structure is needed; without maxlen, use pop/append
Popping/appending/popleft/appendleft has O(1) vs O(N) complexity
Can you write Python code for it?


I am working on the file "1d_Biopython_Cookbook/Chapter_1.ipynb".
The first task is:
1.3 Keeping the Last N Items (in list queue with deque):
Problem:
Keep a limited history of the last few items seen during iteration or processing.

Solution:
Use collections.deque: perform a simple text search on a sequence of lines and yield matching lines with previous N lines of conext when found:
Can you write Python code for it?

from collections import deque

def search(lines, pattern, history=5):
    """ Returns a line that matches the pattern and 5 previous lines"""
    previous_lines = deque(maxlen=history) # a generator of a list with max length
    for line in lines:
        if pattern in line:
            yield line, previous_lines
        previous_lines.append(line)

# Example use on a file
if __name__ == '__main__':
    with open('somefile.txt') as f:
        for line, prevlines in search(f, 'python', 5):
            for pline in prevlines:
                print(pline, end='')
            print(line, end='')
            print('-' * 20)


I am working on the file "1d_Biopython_Cookbook/Chapter_1.ipynb".
The first task is:
1.4 Finding the Largest or Smallest N Items
Problem:
Make a list of the largest or smallest N items in a collection.

Solution:
The heapq module has nlargest() and nsmallest()
Can you write Python code for it?


I am working on the file "1d_Biopython_Cookbook/Chapter_1.ipynb".
The first task is:
Discussion
When looking for N smallest/largest numbers, heapq provides superior performance. heap[0] is always the smallest number. Structures are converted into a list where items are ordered as a heap (underneath).

1.5 Implementing a Priority Queue
Problem:
Implement a queue that sorts items by a given priority and always returns the item with the highest priority on each pop operations.

Solution:
Use heapq to implement a simple priority queue
Can you write Python code for it?


I am working on the file "1d_Biopython_Cookbook/Chapter_1.ipynb".
The first task is:
Discussion:

Common reductions on a dicitionary process the keys and not the values
This is not (probably) what you want, as usually calcualtions are performed on values
In addition to a value result, we often need to know the corresponding key
That is why the zip solution works really well and not too clunky
As noted before, if the values in (values, keys) are the same, the keys will be used
For clear example on lambda functions and key attributes go to:
    https://wiki.python.org/moin/HowTo/Sorting
Can you write Python code for it?

#### 1 #############
min(prices) # 'AAPL'
max(prices) # 'IBM'

#### 2 ############
min(prices.values()) # 10.75
max(prices.values()) # 612.78

#### 3 ############
min(prices, key=lambda k: prices[k]) # 'FB'
max(prices, key=lambda k: prices[k]) # 'AAPL' -> perfrom calculation on values and return key
# to get the value as well as the key, additionally:
min_key = min(prices, key=lambda k: prices[k])
min_value = prices[min(prices, key=lambda k: prices[k])]

#### 4, 5 #########
prices = { 'AAA' : 45.23, 'ZZZ': 45.23 }
min(zip(prices.values(), prices.keys())) # (45.23, 'AAA')
max(zip(prices.values(), prices.keys())) # (45.23, 'ZZZ')


I am working on the file "1d_Biopython_Cookbook/Chapter_1.ipynb".
The first task is:
Discussion:

To eliminate duplicates without preserving an order use a set
The generator functions allows us to be extremely general purpose: not only tied to list processing, but also to file
Can you write Python code for it?

# let's eliminate duplicate lines from a file using the dedupe(items, key=None) generator
with open('somefile.txt', 'r') as f:
    # the generator will spit out a single value (line) at a time,
    # while keeping track (a pointer) to where it is located during each yield
    for line in dedupe(f):
        # process unique lines
        pass


I am working on the file "1d_Biopython_Cookbook/Chapter_1.ipynb".
The first task is:
Discussion:
An OrderedDict is an expensive procedure - beware for items exceeding 100000 lines:

internally maintains a doubly linked list that orders the keys according to insertion order; when a new item is first inserted, it is placed at the end of this list; subsequent reassignment of an existing key doesn't change the order.
be aware that the size of OrderedDict is more than twice as large as a normal dictionary due to the extra linked list that's created.
if building a data structure involving a large number of OrderedDict instances (> 100000 lines of CSV file into a list of OrderedDict instances) be careful!

1.8 Calculating with Dictionaries
Problem:
Performing various calculations (min, max, sort) on a dictionary

Solution:
Reverse keys and values, then perform a calculation function on the zip result.
Important:
    1. max/min/sort is performed on the keys
    2. if the keys are the same, max/min/sort is then based on the values
    3. zip creates an iterator, which can only be consumed once
Can you write Python code for it?


I am working on the file "1d_Biopython_Cookbook/Chapter_1.ipynb".
The first task is:
Discussion:
This is often implemented with iterables of unknown(arbitrary) length, and known pattern: "everything after element 1 is a number".

Handy when iterating over a sequence of tuples of varying length or of tagged tuples.
Handy when unpacking with string processing operations
Handy when unpacking and throwing away some variables
Handly when spliting a list into head and tail components, which could be used to implement recursive solutions.
Can you write Python code for it?


I am working on the file "1d_Biopython_Cookbook/Chapter_1.ipynb".
The first task is:
Discussion:
This recipe focuses on the use of heapq module. Functions heapq.heappush() and heapq.heappop() insert and remove items from a list _queue so that the first item in the list has the highest priority.

heappop() and heappush() have O(log N) complexity
a queue consists of tuples (-priority, index, item); priority is negated so that to add items with the highest priority to the beginning of the _queue

index value is used to properly order items with the same priority; index also works for comparison operations:

By introducing the extra index and making (priority, index, item) tuples, you avoid this problem entirely since no two tuples will ever have the same value for index (and Python never bothers to compare the remaining tuple values once the result of com‐ parison can be determined)::

a = (1, 0, Item('foo'))
b = (5, 1, Item('bar'))
c = (1, 2, Item('grok'))
a < b # True
a < c # True

we can use this queue for communication between threads, but we will need to add appropriate locking and signaling (look ahead)

1.6 Mapping Keys to Multiple Values in a Dictionary
Problem:
Make a dictionary that maps keys to more than one value (multidict)

Solution:
A dictionary is a mapping where each key is mapped to a single value. When mapping keys to multiple values, we need to store multiple values in a different container: list or set.

Use lists to preserve the insertion order of the items
Use sets to eliminate duplicates (when we don't care about the order)
Use defaultdict in the collections to construct such structure:
defaultdict automatically initializes the first value of the key
defaultdict automatically adds default values later on when accessing dictionary
if we don't want the above behavior, use setdefault (it is messier however)
Can you write Python code for it?


I am working on the file "1d_example.ipynb".
The first task is:
Additionally we define a random lyapunov function. Unlike for multiple dimensions, in the one-dimensional case all radially increasing functions are equivalent. Here we pick
$$V(x) = x^2$$
The previous GP model defines a GP model over $\dot{V}(x) = \frac{\partial V(x)}{\partial x} f(x)$. In the following, we only consider the 2-$\sigma$ upper confidence bound of this model. Since the dynamics are Lipschitz continuous, $\dot{V}$ is Lipschitz continuous as well.
In particular, we use Lemma 5 to determine an appropriate Lipschitz constant. For the sample path of the GP, we use the high-probability Lipschitz constant encoded by the kernel.
Can you write Python code for it?

lyapunov_function = lambda x: quadratic_lyapunov_function(x, np.array([[1]]))
V, dV = lyapunov_function(grid)

# V, dV = lyapunov(grid)
accuracy = np.max(V) / 1e10
beta = 2

# Lipschitz constant
L_dyn = 0.25 + beta * np.sqrt(gp.kern.Mat32.variance) / gp.kern.Mat32.lengthscale * np.max(np.abs(extent))
B_dyn = (0.25 + np.sqrt(gp.kern.Mat32.variance)) * np.max(np.abs(extent))
B_dV = L_V = np.max(dV)
L_dV = 1

L = B_dyn * L_dV + B_dV * L_dyn


I am working on the file "1d_example.ipynb".
The first task is:
Clearly the model does not allow us to classify any states as safe ($\dot{V} < -L \tau$). However, as a starting point, we assume that we know that the system is asymptotially stable within some initial set, $\mathcal{S}_0$:
$$\mathcal{S}_0 = { x \in \mathbb{R} \,|\, |x| < 0.2 }$$
Can you write Python code for it?


I am working on the file "1d_example.ipynb".
The first task is:
Let's start by plotting the prior over the dynamics and the associated prior over $\dot{V}(x)$.
Can you write Python code for it?

mean, var = gp._raw_predict(grid)
V_dot_mean, V_dot_var = compute_v_dot_distribution(dV, mean, var)
S = get_safe_set(V_dot_mean + beta * np.sqrt(V_dot_var),
                 -L*tau,
                 S0=None)

plot_model(S)


I am working on the file "1d_example.ipynb".
The first task is:
Online learning
As we sample within this initial safe set, we gain more knowledge about the system. In particular, we iteratively select the state withing the safe set, $\mathcal{S}_n$, where the dynamics are the most uncertain (highest variance).
Can you write Python code for it?

def get_max_level_set(dynamics_mean, dynamics_var):
    V_dot = compute_v_dot_upper_bound(dV, dynamics_mean, dynamics_var, beta=2.)
    S = get_safe_set(V_dot, -L*tau, S0=S0)
    c = find_max_levelset(S, V, accuracy)
    S[:] = V <= c
    return S

def update_gp():
    dynamics_mean, dynamics_var = gp._raw_predict(grid)
    S = get_max_level_set(dynamics_mean, dynamics_var)
    max_id = np.argmax(dynamics_var[S])
    max_state = grid[S][[max_id], :].copy()
    gp.set_XY(np.vstack([gp.X, max_state]),
              np.vstack([gp.Y, true_dynamics(max_state, noise=True)[:, [0]]]))
    return S

# Update the GP model a couple of times
for i in range(4):
    update_gp()

# Plot the new safe set
S = get_max_level_set(*gp._raw_predict(grid))
plot_model(S)


I am working on the file "1d_example.ipynb".
The first task is:
We continue to sample like this, until we find the maximum safe set
Can you write Python code for it?

for i in range(20):
    update_gp()

S = get_max_level_set(*gp._raw_predict(grid))
plot_model(S)


I am working on the file "1d_example.ipynb".
The first task is:
We start by defining a discretization of the space $[-1, 1]$ with discretization constant $\tau$
Can you write Python code for it?

# Discretization constant
tau = 0.001

# x_min, x_max, discretization
grid_param = [-1., 1., tau]

extent = np.array(grid_param[:2])

# Create a grid
grid = np.arange(*grid_param)[:, None]
num_samples = len(grid)

print('Grid size: {0}'.format(len(grid)))


I am working on the file "1j_NLP_Python/ex01.ipynb".
The first task is:
Next, use Beautiful Soup to extract text out of the HTML. Following the DOM structure of the HTML document, select the <div/> that encloses the article text, then iterate through the <p/> paragraphs to extract the text from each.
Can you write Python code for it?

from bs4 import BeautifulSoup

with open(file, "r") as f:
  soup = BeautifulSoup(f, "html.parser")

  for div in soup.find_all("div", id="article-body"):
    for p in div.find_all("p"):
      print(p.get_text(), "\n")


I am working on the file "1j_NLP_Python/ex02.ipynb".
The first task is:
Encoding as UTF-8 doesn't help much - use it as an encoder:
Can you write Python code for it?


I am working on the file "1j_NLP_Python/ex02.ipynb".
The first task is:
Even before this normalization and encoding, you may need to convert some characters explicitly before parsing. For example:
Can you write Python code for it?

x = "The sky “above” the port … was the color of ‘cable television’ – tuned to the Weather Channel®"
ascii(x)


I am working on the file "1j_NLP_Python/ex02.ipynb".
The first task is:
However, one can normalize then encode…
Can you write Python code for it?

import unicodedata
# NFKD a robust way to handle normalizers - convert special characters into something
# that can be read and convert into ascii
unicodedata.normalize('NFKD', x).encode('ascii','ignore')


I am working on the file "1j_NLP_Python/ex02.ipynb".
The first task is:
Its translation into ASCII is unusable by parsers:
Can you write Python code for it?


I am working on the file "1j_NLP_Python/ex02.ipynb".
The first task is:
The variable x is a string in Python:
Can you write Python code for it?


I am working on the file "1j_NLP_Python/ex02.ipynb".
The first task is:
Then consider the results here:
Can you write Python code for it?


I am working on the file "1j_NLP_Python/ex03.ipynb".
The first task is:
Given these annotations for part-of-speech tags, we can lemmatize nouns and verbs to get their root forms. This will also singularize the plural nouns:
Can you write Python code for it?

from textblob import Word

ts = [('InterAct', 'VB'), ('comments', 'NNS'), ('provides', 'VBZ'), ('mining', 'NN')]

for lex, pos in ts:
  w = Word(lex.lower())
  lemma = w.lemmatize(pos[0].lower())
  print(lex, pos, lemma)


I am working on the file "1j_NLP_Python/ex03.ipynb".
The first task is:
Next we take a sentence and annotate it with part-of-speech (PoS) tags:
Can you write Python code for it?

import textblob_aptagger as tag

sent = "Increasingly, customers send text to interact or leave comments, which provides a wealth of data for text mining."

ts = tag.PerceptronTagger().tag(sent)
print(ts)


I am working on the file "1j_NLP_Python/ex04.ipynb".
The first task is:
Contrast those results with noun phrases:
Can you write Python code for it?

# noun pharses: that's, custom search, content recommenders, ai - Super helpful!
for np in blob.noun_phrases:
  print(np)


I am working on the file "1j_NLP_Python/ex04.ipynb".
The first task is:
First let's look at the individual keywords:
Can you write Python code for it?


I am working on the file "1j_NLP_Python/ex06.ipynb".
The first task is:
That extracts text from HTML in the first article, then stores the parsed and annotated text as JSON, one line per sentence. Let's look at the first two sentences:
Can you write Python code for it?

%sx more a1.json
# representation of paragraphs, sentences, words, and sentence annotation


I am working on the file "1j_NLP_Python/ex07.ipynb".
The first task is:
Iterate through each parsed file, tallying counts for tf for each document while also tallying counts for df across all documents:
Can you write Python code for it?

for json_file in files:
  tf = defaultdict(int) # each file has its own term frequence

  for lex in pynlp.lex_iter(json_file):
    if (lex.pos != ".") and (lex.root not in stopwords):
      tf[lex.root] += 1 # increment for each word

  files_tf[json_file] = tf # keep the file in memory - it is mall enough

  for word in tf.keys():
    df[word] += 1 # print out term frequency

## print results for just the last file in the sequence
print(json_file, files_tf[json_file])


I am working on the file "1j_NLP_Python/ex07.ipynb".
The first task is:
Next, we'll use a function from our pynlp library to iterate through the keywords for one of the parsed HTML documents:
Can you write Python code for it?

%sx ls *.json

json_file = "a1.json"

for lex in pynlp.lex_iter(json_file):
  print(lex)


I am working on the file "1j_NLP_Python/ex07.ipynb".
The first task is:
We need to initialize some data structures for counting keywords. BTW, if you've heard about how Big Data projects use word count programs to demonstrate their capabilities, here's a major use case for that. Even so, our examples are conceptually simple, built for relatively small files, and are not intended to scale:
Can you write Python code for it?

from collections import defaultdict

files = ["a4.json", "a3.json", "a2.json", "a1.json"]
files_tf = {}

d = len(files)
df = defaultdict(int)


I am working on the file "1j_NLP_Python/ex08.ipynb".
The first task is:
Note the top-ranked ("most similar") pair, where both html/article2.html and html/article3.html are about machine learning. Take a look at their overlapping keywords:
Can you write Python code for it?

files_set["a3.json"] & files_set["a2.json"]
# show the intersection words for the documents that have been approximated with MinHas
# what other documents are out there that are similar to this set of documents?


I am working on the file "1j_NLP_Python/ex08.ipynb".
The first task is:
Then we'll iterate through each parsed document, adding the keywords to the MinHash:
Can you write Python code for it?

import pynlp

files = ["a4.json", "a3.json", "a2.json", "a1.json"]

stopwords = pynlp.load_stopwords("stop.txt")
files_set = {}
files_mh = {}

for json_file in files: # for each json file initialize keywords
  keywords = set([])

  for lex in pynlp.lex_iter(json_file):
    if (lex.pos != ".") and (lex.root not in stopwords): # throw away punctuations
      keywords.add(lex.root) # add cannonical version of that word to the set

  files_set[json_file] = keywords
  files_mh[json_file] = mh_digest(keywords) # run the minhash algorithm

  print(json_file, keywords)


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 1 - Working with One Dimensional Data - Answers.ipynb".
The first task is:
Exercise 1:  Splitting and Filtering a Series
In this exercise, you are given a list of email addresses called emails.  Your goal is to find the email accounts from domains that end in .edu.  To accomplish this, you will need to:
1. Filter the series to remove the emails that end in .edu
2. Extract the accounts.
If you get stuck, refer to the documentation for Pandas string manipulation (http://pandas.pydata.org/pandas-docs/stable/text.html) or the slides.  Note that there are various functions to accomplish this task.
Can you write Python code for it?

emails = ['alawrence0@prlog.org',
'blynch1@businessweek.com',
'mdixon2@cmu.edu',
'rvasquez3@1688.com',
'astone4@creativecommons.org',
'mcarter5@chicagotribune.com',
'dcole6@vinaora.com',
'kpeterson7@topsy.com',
'ewebb8@cnet.com',
'jtaylor9@google.ru',
'ecarra@buzzfeed.com',
'jjonesb@arizona.edu',
'jbowmanc@disqus.com',
'eduardo_sanchezd@npr.org',
'emooree@prweb.com',
'eberryf@brandeis.edu',
'sgardnerh@wikipedia.org',
'balvarezi@delicious.com',
'blewisj@privacy.gov.au']

email_series = pd.Series(emails)
filtered_emails = email_series[email_series.str.contains('.edu')]
print( filtered_emails )

accounts = filtered_emails.str.split( '@').str[0]
print( accounts )


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 1 - Working with One Dimensional Data - Answers.ipynb".
The first task is:
Exercise 2:  Applying a Function
In this exercise you are given a list of weights in pounds and a function to convert the measures into kilograms.  For this exercise, apply the conversion function to the original series and convert the measures into kilograms.
Can you write Python code for it?

poundsToKilograms = lambda x: x * 0.45359237
weights = [31.09, 46.48, 24.0, 39.99, 19.33, 39.61, 40.91, 52.24, 30.77, 17.23, 34.87 ]

pounds = pd.Series( weights )
kilos = pounds.apply( poundsToKilograms )
print( kilos )


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 2 - Exploring Two Dimensional Data - Answers.ipynb".
The first task is:
Exercise 1: Reading various forms of JSON Data
In the /data/ folder, you will find a series of .json files called dataN.json, numbered 1-4.  Each file contains the following data:
| |birthday | first_name |last_name |
|--|-----------|------------|----------|
|0  |5\/3\/67   |Robert |Hernandez |
|1  |8\/4\/84   |Steve  |Smith |
|2  |9\/13\/91  |Anne   |Raps |
|3  |4\/15\/75  |Alice  |Muller |
Using the .read_json() function and the various configuration options, read all these files into a dataframe.  The documentation is available here: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html.
Can you write Python code for it?

df1 = pd.read_json('../../data/data1.json')
df1

df2 = pd.read_json('../../data/data2.json', orient='index')
df2

df3 = pd.read_json('../../data/data3.json', orient='columns')
df3

df4 = pd.read_json('../../data/data4.json', orient='split')
df4


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 2 - Exploring Two Dimensional Data - Answers.ipynb".
The first task is:
Exercise 2:
In the data file, there is a webserver file called hackers-access.httpd.  For this exercise, you will use this file to answer the following questions:
1.  Which browsers are the top 10 most used browsers in this data?
2.  Which are the top 10 most used operating systems?
In order to accomplish this task, do the following:
1.  Write a function which takes a User Agent string as an argument and returns the relevant data.  HINT:  You might want to use python's user_agents module, the documentation for which is available here: (https://pypi.python.org/pypi/user-agents)
2.  Next, apply this function to the column which contains the user agent string.
3.  Store this series as a new column in the dataframe
4.  Count the occurances of each value in the new columns
Can you write Python code for it?

import apache_log_parser
from user_agents import parse
#Read in the log file
line_parser = apache_log_parser.make_parser("%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"")

server_log = open("../../data/hackers-access.httpd", "r")
parsed_server_data = []
for line in server_log:
    data = {}
    data = line_parser(line)
    parsed_server_data.append( data )

server_df = pd.DataFrame( parsed_server_data  )

#Write the functions
def get_os(x):
    user_agent = parse(x)
    return user_agent.os.family

def get_browser(x):
    user_agent = parse(x)
    return user_agent.browser.family

#Apply the functions to the dataframe
server_df['os'] = server_df['request_header_user_agent'].apply( get_os )
server_df['browser'] = server_df['request_header_user_agent'].apply( get_browser )

#Get the top 10 values
server_df['os'].value_counts().head(10)

server_df['browser'].value_counts().head(10)


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 2 - Exploring Two Dimensional Data - Answers.ipynb".
The first task is:
Exercise 3:
Using the dailybots.csv film, read the file into a DataFrame and perform the following operations:
1.  Filter the DataFrame to include bots from the Government/Politics Industry.
2.  Calculate the ratio of hosts to orgs and add this as a column to the DataFrame and output the result
3.  Calculate the total number of hosts infected by each BotFam in the Government/Politics Industry.  You should use the groupby() function which is documented here: (http://pandas.pydata.org/pandas-docs/stable/groupby.html)
Can you write Python code for it?

bots = pd.read_csv('../../data/dailybots.csv')
bots.head()

gov_bots = bots[['botfam', 'hosts']][bots['industry'] == "Government/Politics"]

gov_bots.groupby('botfam', as_index=False).sum()


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 3 - EDA Worksheet-Answers.ipynb".
The first task is:
Part 1: Which industry sees the most Ramnit infections? Least?
Count the number of infected days for "Ramnit" in each industry industry.
How:
1.  First filter the data to remove all the infections we don't care about
2.  Aggregate the data on the column of interest. HINT:  You might want to use the groupby() function
3.  Add up the results
Can you write Python code for it?

grouped_df = data[data.botfam == "Ramnit"].groupby(['industry'])
grouped_df.sum()


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 3 - EDA Worksheet-Answers.ipynb".
The first task is:
Part 2: Calculate the min, max, median and mean infected orgs for each bot family, sort by median
In this exercise, you are asked to calculate the min, max, median and mean of infected orgs for each bot family sorted by median.  HINT:
1.  Using the groupby() function, create a grouped data frame
2.  You can do this one metric at a time OR you can use the .agg() function. You might want to refer to the documentation here:  http://pandas.pydata.org/pandas-docs/stable/groupby.html#applying-multiple-functions-at-once
3.  Sort the values (HINT HINT) by the median column
Can you write Python code for it?

group2 = data[['botfam','orgs']].groupby( ['botfam'])
summary = group2.agg([np.min, np.max, np.mean, np.median, np.std])
summary.sort_values( [('orgs', 'median')], ascending=False)


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 3 - EDA Worksheet-Answers.ipynb".
The first task is:
Part 3: Which date had the total most bot infections and how many infections on that day?
For the next step, aggregate and sum the number of infections (hosts) by date.  Once you've done that, the next step is to sort in descending order.
Can you write Python code for it?

df3 = data[['date','hosts']].groupby('date').sum()
df3.sort_values(by='hosts', ascending=False).head(10)


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 3 - EDA Worksheet-Answers.ipynb".
The first task is:
Part 4: Plot the daily infected hosts for Necurs, Ramnit and PushDo
For the final step, you're going to plot the daily infected hosts for three infection types.  In order to do this, you'll need to do the following steps:
1.  Filter the data to remove the botfamilies we don't care about.
2.  Use groupby() to aggregate the data by date and family, then sum up the hosts in each group
3.  Plot the data.  Hint:  You might want to use the unstack() function to prepare the data for plotting.
4.  Use the plot() method to plot the results.
Can you write Python code for it?

filteredData = data[ data['botfam'].isin(['Necurs', 'Ramnit', 'PushDo']) ][['date', 'botfam', 'hosts']]
groupedFilteredData = filteredData.groupby( ['date', 'botfam']).sum()
groupedFilteredData.unstack(level=1).plot(kind='line', subplots=False)


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 4 - Data Visualization Worksheet - Answers.ipynb".
The first task is:
Create the first chart
Using the .plot() method, plot your dataframe and see what you get.
Can you write Python code for it?


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 4 - Data Visualization Worksheet - Answers.ipynb".
The first task is:
Make the Figure Size Larger:
Can you write Python code for it?


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 4 - Data Visualization Worksheet - Answers.ipynb".
The first task is:
Move the Legend to the Upper Center of the Graph
For this, you'll have to assign the plot variable to a new variable and then call the formatting methods on it.
Can you write Python code for it?

nicePlot = finalData.plot( kind="line")
nicePlot.legend(loc='upper center', bbox_to_anchor=(0.5, 1.05),
          ncol=3, fancybox=True, shadow=False)


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 4 - Data Visualization Worksheet - Answers.ipynb".
The first task is:
Next, I created a second DataFrame which only contains the information from the ConfickerAB botnet.  I also reduced the columns to the date and host count.  You'll need to rename the host count so that you can merge the other data set later.
Can you write Python code for it?

filteredData2 = filteredData[filteredData['botfam']== 'ConfickerAB' ][['date','hosts']]
filteredData2.columns = ['date', 'ConfickerAB']
filteredData2.date = pd.to_datetime( filteredData2.date )
filteredData2.head()


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 4 - Data Visualization Worksheet - Answers.ipynb".
The first task is:
Repeat this porcess for the Bedep botfam in a separate dataFrame.
Step 3: Merge the DataFrames.
Next, you'll need to merge the dataframes so that you end up with a dataframe with three columns: the date, the ConfickerAB count, and the the Bedep count.  Pandas has a .merge() function which is documented here: http://pandas.pydata.org/pandas-docs/stable/merging.html
Can you write Python code for it?

filteredData3 = filteredData[filteredData['botfam']== 'Bedep' ][['date','hosts']]
filteredData3.columns = ['date', 'Bedep']
filteredData3.date = pd.to_datetime( filteredData3.date )
finalData = pd.merge(filteredData2, filteredData3, on='date', how='left')
finalData.head()


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 4 - Data Visualization Worksheet - Answers.ipynb".
The first task is:
Step 2:  Filter the Data
The next step is to filter both by industry and by botfam.  In order to get the data into the format I wanted, I did this separately.  First, I created a second dataframe called filteredData which only contains the information from the Government/Politics industry.
Can you write Python code for it?

filteredData = data[data['industry'] == "Government/Politics"]
filteredData.head()


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 4 - Data Visualization Worksheet - Answers.ipynb".
The first task is:
Step 3 Customizing your plot:
The default plot doesn't look horrible, but there are certainly some improvements which can be made.  Try the following:
1.  Change the x-axis to a date by converting the date column to a date object.
2.  Move the Legend to the upper center of the graph
3.  Make the figure size larger.
4.  Instead of rendering both lines on one graph, split them up into two plots
5.  Add axis labels
There are many examples in the documentation which is available: http://pandas.pydata.org/pandas-docs/version/0.18.1/visualization.html
Can you write Python code for it?

finalData.set_index('date', inplace=True)
finalData.plot(kind='line' )


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 4 - Data Visualization Worksheet - Answers.ipynb".
The first task is:
Worksheet 4 - Data Visualization Worksheet
This worksheet will walk you through the basic process of preparing a visualization using Python/Pandas/Matplotlib.
For this exercise, we will be creating a line plot comparing the number of hosts infected by the Bedep and ConfickerAB Bot Families in the Government/Politic sector.
Prepare the data
The data we will be using is in the dailybots.csv file which can be found in the data folder.  As is common, we will have to do some data wrangling to get it into a format which we can use to visualize this data.  To do that, we'll need to:
1.  Read in the data
2.  Filter the data by industry and botnet
The result should look something like this:
| |date|ConflikerAB|Bedep|
|-|----|-----------|-----|
|0|2016-06-01|255|430|
|1|2016-06-02|431|453|
The way I chose to do this might be a little more complex, but I wanted you to see all the steps involved.
Step 1 Read in the data
Using the pd.read_csv() function, you can read in the data.
Can you write Python code for it?

data = pd.read_csv('../../data/dailybots.csv')
data[data['industry']=="Government/Politics"].head(20)


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 5 - DGA Detection Feature Engineering - Answers.ipynb".
The first task is:
Now let's use a Seaborn pairplot as well.  This will really show you which features have clear dividing lines between the classes.  Docs are available here: http://seaborn.pydata.org/generated/seaborn.pairplot.html
Can you write Python code for it?


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 5 - DGA Detection Feature Engineering - Answers.ipynb".
The first task is:
Part 1 - Feature Engineering
Option 1 to derive Machine Learning features is to manually hand-craft useful contextual information of the domain string. An alternative approach (not covered in this notebook) is "Featureless Deep Learning", where an embedding layer takes care of deriving features - a huge step towards more "AI".
Previous academic research has focused on the following features that are based on contextual information:
List of features:

Length ["length"]
Number of digits ["digits"]
Entropy ["entropy"] - use H_entropy function provided
Vowel to consonant ratio ["vowel-cons"] - use vowel_consonant_ratio function provided
N-grams ["n-grams"] - use ngram functions provided

Tasks:
Split into A and B parts, see below...
Please run the following function cell and then continue reading the next markdown cell with more details on how to derive those features. Have fun!
Can you write Python code for it?

def H_entropy (x):
    # Calculate Shannon Entropy
    prob = [ float(x.count(c)) / len(x) for c in dict.fromkeys(list(x)) ]
    H = - sum([ p * np.log2(p) for p in prob ])
    return H

def vowel_consonant_ratio (x):
    # Calculate vowel to consonant ratio
    x = x.lower()
    vowels_pattern = re.compile('([aeiou])')
    consonants_pattern = re.compile('([b-df-hj-np-tv-z])')
    vowels = re.findall(vowels_pattern, x)
    consonants = re.findall(consonants_pattern, x)
    try:
        ratio = len(vowels) / len(consonants)
    except: # catch zero devision exception
        ratio = 0
    return ratio


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 5 - DGA Detection Feature Engineering - Answers.ipynb".
The first task is:
Tasks - A - Feature Engineering
Please try to derive a new pandas 2D DataFrame with a new column for each of feature. Focus on length, digits, entropy and vowel-cons here. Also make sure to encode the isDGA column as integers. pandas.Series.str, pandas.Series.replace and pandas.Series,apply can be very helpful to quickly derive those features. Functions you need to apply here are provided in above cell.
The ngram is a bit more complicated, see next instruction cell to add this feature...
Can you write Python code for it?

# derive features
df['length'] = df.domain.str.len()
df['digits'] = df.domain.str.count('[0-9]')
df['entropy'] = df.domain.apply(H_entropy)
df['vowel-cons'] = df.domain.apply(vowel_consonant_ratio)

# encode strings of target variable as integers
df.isDGA = df.isDGA.replace(to_replace = 'dga', value=1)
df.isDGA = df.isDGA.replace(to_replace = 'legit', value=0)
print(df.isDGA.value_counts())

# check intermediate 2D pandas DataFrame
df.sample(n=5).head()


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 5 - DGA Detection Feature Engineering - Answers.ipynb".
The first task is:
Worksheet - Answer - DGA Detection using Machine Learning
This worksheet is a step-by-step guide on how to detect domains that were generated using "Domain Generation Algorithm" (DGA). We will walk you through the process of transforming raw domain strings to Machine Learning features and creating a decision tree classifer which you will use to determine whether a given domain is legit or not. Once you have implemented the classifier, the worksheet will walk you through evaluating your model.
Overview 2 main steps:

Feature Engineering - from raw domain strings to numeric Machine Learning features using DataFrame manipulations
Machine Learning Classification - predict whether a domain is legit or not using a Decision Tree Classifier

DGA - Background
"Various families of malware use domain generation
algorithms (DGAs) to generate a large number of pseudo-random
domain names to connect to a command and control (C2) server.
In order to block DGA C2 traffic, security organizations must
first discover the algorithm by reverse engineering malware
samples, then generate a list of domains for a given seed. The
domains are then either preregistered, sink-holed or published
in a DNS blacklist. This process is not only tedious, but can
be readily circumvented by malware authors. An alternative
approach to stop malware from using DGAs is to intercept DNS
queries on a network and predict whether domains are DGA
generated. Much of the previous work in DGA detection is based
on finding groupings of like domains and using their statistical
properties to determine if they are DGA generated. However,
these techniques are run over large time windows and cannot be
used for real-time detection and prevention. In addition, many of
these techniques also use contextual information such as passive
DNS and aggregations of all NXDomains throughout a network.
Such requirements are not only costly to integrate, they may not
be possible due to real-world constraints of many systems (such
as endpoint detection). An alternative to these...
Can you write Python code for it?

## Load data
df = pd.read_csv('../../data/dga_data_small.csv')
df.drop(['host', 'subclass'], axis=1, inplace=True)
print(df.shape)
df.sample(n=5).head() # print a random sample of the DataFrame

df[df.isDGA == 'legit'].head()

# Google's 10000 most common english words will be needed to derive a feature called ngrams...
# therefore we already load them here.
top_en_words = pd.read_csv('../../data/google-10000-english.txt', header=None, names=['words'])
top_en_words.sample(n=5).head()
# Source: https://github.com/first20hours/google-10000-english


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 6 - DGA Detection ML Classification - Answers.ipynb".
The first task is:
Part 2 - Machine Learning
To learn simple classification procedures using sklearn we have split the work flow into 5 steps.
Step 1: Prepare Feature matrix and target vector containing the URL labels

In statistics, the feature matrix is often referred to as X
target is a vector containing the labels for each URL (often also called y in statistics)
In sklearn both the input and target can either be a pandas DataFrame/Series or numpy array/vector respectively (can't be lists!)

Tasks:
- assign 'isDGA' column to a pandas Series named 'target'
- drop 'isDGA' column from dga DataFrame and name the resulting pandas DataFrame 'feature_matrix'
Can you write Python code for it?

target = df_final['isDGA']
feature_matrix = df_final.drop(['isDGA'], axis=1)
print('Final features', feature_matrix.columns)

print( feature_matrix.head())


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 6 - DGA Detection ML Classification - Answers.ipynb".
The first task is:
Step 2: Simple Cross-Validation
Tasks:
- split your feature matrix X and target vector into train and test subsets using sklearn model_selection.train_test_split
Can you write Python code for it?

# Simple Cross-Validation: Split the data set into training and test data
feature_matrix_train, feature_matrix_test, target_train, target_test = model_selection.train_test_split(feature_matrix, target, test_size=0.25, random_state=33)

feature_matrix_train.count()

feature_matrix_test.count()

target_train.head()


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 6 - DGA Detection ML Classification - Answers.ipynb".
The first task is:
Step 3: Train the model and make a prediction
Finally, we have prepared and segmented the data. Let's start classifying!!
Tasks:

Use the sklearn tree.DecisionTreeClassfier(), create a decision tree with standard parameters, and train it using the .fit() function with X_train and target_train data.
Next, pull a few random rows from the data and see if your classifier got it correct.

If you are interested in trying a real unknown domain, you'll have to create a function to generate the features for that domain before you run it through the classifier (see function is_dga a few cells below).
Can you write Python code for it?

# Train the decision tree based on the entropy criterion
clf = tree.DecisionTreeClassifier()  # clf means classifier
clf = clf.fit(feature_matrix_train, target_train)

# Extract a row from the test data
test_feature = feature_matrix_test[192:193]
test_target = target_test[192:193]

# Make the prediction
pred = clf.predict(test_feature)
print('Predicted class:', pred)
print('Accurate prediction?', pred[0] == test_target)

feature_matrix_test

# For simplicity let's just copy the needed function in here again

def H_entropy (x):
    # Calculate Shannon Entropy
    prob = [ float(x.count(c)) / len(x) for c in dict.fromkeys(list(x)) ]
    H = - sum([ p * np.log2(p) for p in prob ])
    return H

def vowel_consonant_ratio (x):
    # Calculate vowel to consonant ratio
    x = x.lower()
    vowels_pattern = re.compile('([aeiou])')
    consonants_pattern = re.compile('([b-df-hj-np-tv-z])')
    vowels = re.findall(vowels_pattern, x)
    consonants = re.findall(consonants_pattern, x)
    try:
        ratio = len(vowels) / len(consonants)
    except: # catch zero devision exception
        ratio = 0
    return ratio

# ngrams: Implementation according to Schiavoni 2014: "Phoenix: DGA-based Botnet Tracking and Intelligence"
# http://s2lab.isg.rhul.ac.uk/papers/files/dimva2014.pdf

def ngrams(word, n):
    # Extract all ngrams and return a regular Python list
    # Input word: can be a simple string or a list of strings
    # Input n: Can be one integer or a list of integers
    # if you want to extract multipe ngrams and have them all in one list

    l_ngrams = []
    if isinstance(word, list):
        for w in word:
            if isinstance(n, list):
                for curr_n in n:
                    ngrams = [w[i:i+curr_n] for i in range(0,len(w)-curr_n+1)]
                    l_ngrams.extend(ngrams)
            else:
                ngrams = [w[i:i+n] for i in range(0,len(w)-n+1)]
                l_ngrams.extend(ngrams)
    else:
        if isinstance(n, list):
            for curr_n in n:
              ...


I am working on the file "1m_ML_Security/notebooks/answers/Worksheet 6 - DGA Detection ML Classification - Answers.ipynb".
The first task is:
Step 5: Assess model accuracy with k-fold cross-validation
Tasks:
- Partition the dataset into k different subsets
- Create k different models by training on k-1 subsets and testing on the remaining subsets
- Measure the performance on each of the models and take the average measure.
Short-Cut
All of these steps can be easily achieved by simply using sklearn's model_selection.KFold() and model_selection.cross_val_score() functions.
Can you write Python code for it?

cvKFold = model_selection.KFold(n_splits=3, shuffle=True, random_state=33)
cvKFold.get_n_splits(feature_matrix)

scores = model_selection.cross_val_score(clf, feature_matrix, target, cv=cvKFold)
print(scores)

# Get avergage score +- Standard Error (https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html)
from scipy.stats import sem
def mean_score( scores ):
    return "Mean score: {0:.3f} (+/- {1:.3f})".format( np.mean(scores), sem( scores ))
print( mean_score( scores))


I am working on the file "1m_ML_Security/notebooks/day_1/Worksheet 1 - Working with One Dimensional Data.ipynb".
The first task is:
Exercise 1:  Splitting and Filtering a Series
In this exercise, you are given a list of email addresses called emails.  Your goal is to find the email accounts from domains that end in .edu.  To accomplish this, you will need to:
1. Filter the series to remove the emails that do not end in .edu
2. Extract the accounts.
If you get stuck, refer to the documentation for Pandas string manipulation (http://pandas.pydata.org/pandas-docs/stable/text.html) or the slides.  Note that there are various functions to accomplish this task.
Can you write Python code for it?

emails = ['alawrence0@prlog.org',
'blynch1@businessweek.com',
'mdixon2@cmu.edu',
'rvasquez3@1688.com',
'astone4@creativecommons.org',
'mcarter5@chicagotribune.com',
'dcole6@vinaora.com',
'kpeterson7@topsy.com',
'ewebb8@cnet.com',
'jtaylor9@google.ru',
'ecarra@buzzfeed.com',
'jjonesb@arizona.edu',
'jbowmanc@disqus.com',
'eduardo_sanchezd@npr.org',
'emooree@prweb.com',
'eberryf@brandeis.edu',
'sgardnerh@wikipedia.org',
'balvarezi@delicious.com',
'blewisj@privacy.gov.au']

#Your code here...
emails_records = pd.Series(emails)
emails_records[emails_records.str.contains('.edu')].str.split('@').str[0]


I am working on the file "1m_ML_Security/notebooks/day_1/Worksheet 1 - Working with One Dimensional Data.ipynb".
The first task is:
Exercise 2:  Applying a Function
In this exercise you are given a list of weights in pounds and a function to convert the measures into kilograms.  For this exercise, apply the conversion function to the original series and convert the measures into kilograms.
Can you write Python code for it?


I am working on the file "1m_ML_Security/notebooks/day_1/Worksheet 1 - Working with One Dimensional Data.ipynb".
The first task is:
Exercise 3: Putting it all together
You are given a Series of IP Addresses and the goal is to limit this data to private IP addresses.  Python has an ipaddress module which provides the capability to create, manipulate and operate on IPv4 and IPv6 addresses and networks.  Complete documentation is available here: https://docs.python.org/3/library/ipaddress.html.
Here are some examples of how you might use this module:
```python
import ipaddress
myIP = ipaddress.ip_address( '192.168.0.1' )
myNetwork = ipaddress.ip_network( '192.168.0.0/28' )
Check membership in network
if myIP in myNetwork:  #This works
    print "Yay!"
Loop through CIDR blocks
for ip in myNetwork:
    print( ip )
192.168.0.0
192.168.0.1
…
…
192.168.0.13
192.168.0.14
192.168.0.15
Testing to see if an IP is private
if myIP.is_private:
    print( "This IP is private" )
else:
    print( "Routable IP" )
```

First, write a function which takes an IP address and returns true if the IP is private, false if it is public.  HINT:  use the ipaddress module.
Next, use this to create a Series of true/false values in the same sequence as your original Series.
Finally, use this to filter out the original Series so that it contains only private IP addresses.
Can you write Python code for it?


I am working on the file "1m_ML_Security/notebooks/day_1/Worksheet 2 - Exploring Two Dimensional Data.ipynb".
The first task is:
Exercise 2:
In the data file, there is a webserver file called hackers-access.httpd.  For this exercise, you will use this file to answer the following questions:
1.  Which browsers are the top 10 most used browsers in this data?
2.  Which are the top 10 most used operating systems?
In order to accomplish this task, do the following:
1.  Write a function which takes a User Agent string as an argument and returns the relevant data.  HINT:  You might want to use python's user_agents module, the documentation for which is available here: (https://pypi.python.org/pypi/user-agents)
2.  Next, apply this function to the column which contains the user agent string.
3.  Store this series as a new column in the dataframe
4.  Count the occurances of each value in the new columns
Can you write Python code for it?

import apache_log_parser
from user_agents import parse

def parse_ua(line):
    parsed_data = parse(line)
    return str(parsed_data).split('/')[1]

def parse_ua_2(line):
    parsed_data = parse(line)
    return str(parsed_data).split('/')[2]

#Read in the log file
line_parser = apache_log_parser.make_parser("%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"")

server_log = open("../../data/hackers-access.httpd", "r")
parsed_server_data = []
for line in server_log:
    data = {}
    data = line_parser(line)
    parsed_server_data.append( data )

server_df = pd.DataFrame(parsed_server_data)
server_df['OS'] = server_df['request_header_user_agent'].apply(parse_ua)
server_df['Browser'] = server_df['request_header_user_agent'].apply(parse_ua_2)
server_df['OS'].value_counts().head(10)

#Apply the functions to the dataframe

#Get the top 10 values


I am working on the file "1m_ML_Security/notebooks/day_1/Worksheet 2 - Exploring Two Dimensional Data.ipynb".
The first task is:
Exercise 3:
Using the dailybots.csv film, read the file into a DataFrame and perform the following operations:
1.  Filter the DataFrame to include bots from the Government/Politics Industry.
2.  Calculate the ratio of hosts to orgs and add this as a column to the DataFrame and output the result
3.  Calculate the total number of hosts infected by each BotFam in the Government/Politics Industry.  You should use the groupby() function which is documented here: (http://pandas.pydata.org/pandas-docs/stable/groupby.html)
Can you write Python code for it?

#Your code here...
bots = pd.read_csv('../../data/dailybots.csv')
gov_bots = bots[['botfam', 'hosts']][bots['industry'] == 'Government/Politics']
gov_bots.groupby(['botfam']).size()


I am working on the file "1m_ML_Security/notebooks/day_2/Worksheet 3 - EDA Worksheet.ipynb".
The first task is:
Exercise 1: Summarize the Data
For this exercise, you are given a Series of random numbers creatively names random_numbers.  For the first exercise please do the following:

Remove all the numbers less than 10
Sort the series
Calculate the Tukey 5 number summary for this dataset
Count the number of even and odd numbers
Find the five largest and 5 smallest numbers in the series
Can you write Python code for it?

#Generate a series of random numbers between 1 and 100.
random_numbers = pd.Series( np.random.randint(1, 100, 50) )
random_numbers[random_numbers > 10].sort_values().describe()

random_numbers = pd.Series( np.random.randint(1, 100, 50) )
random_numbers.sort_values().unique()[0:10]

# even and odd numbers
even_numbers = random_numbers[random_numbers % 2 == 0].count()


I am working on the file "1m_ML_Security/notebooks/day_2/Worksheet 3 - EDA Worksheet.ipynb".
The first task is:
Exercise 4:  Putting it all together: Bot Analysis
First you're going to want to create a data frame from the dailybots.csv file which can be found in the data directory.  You should be able to do this with the pd.read_csv() function.  Take a minute to look at the dataframe because we are going to be using it for to answer several different questions.
Can you write Python code for it?

data = pd.read_csv( '../../data/dailybots.csv' )
data.describe()

data.info() # useful to see columns

data.columns()

data['botfam'].value_counts()


I am working on the file "1m_ML_Security/notebooks/day_2/Worksheet 3 - EDA Worksheet.ipynb".
The first task is:
Part 1: Which industry sees the most Ramnit infections? Least?
Count the number of infected days for "Ramnit" in each industry industry.
How:
1.  First filter the data to remove all the infections we don't care about
2.  Aggregate the data on the column of interest. HINT:  You might want to use the groupby() function
3.  Add up the results
Can you write Python code for it?

grouped_df = data[data.botfam == 'Ramnit'].groupby(['industry'])
grouped_df.sum()


I am working on the file "1m_ML_Security/notebooks/day_2/Worksheet 3 - EDA Worksheet.ipynb".
The first task is:
Part 2: Calculate the min, max, median and mean infected orgs for each bot family, sort by median
In this exercise, you are asked to calculate the min, max, median and mean of infected orgs for each bot family sorted by median.  HINT:
1.  Using the groupby() function, create a grouped data frame
2.  You can do this one metric at a time OR you can use the .agg() function. You might want to refer to the documentation here:  http://pandas.pydata.org/pandas-docs/stable/groupby.html#applying-multiple-functions-at-once
3.  Sort the values (HINT HINT) by the median column
Can you write Python code for it?

group2 = data[['botfam', 'orgs']].groupby(['botfam'])
summary = group2.agg([np.min, np.max, np.mean, np.dedian, np.std])
summary.sort_values(['orgs', 'median'], ascending=False)


I am working on the file "1m_ML_Security/notebooks/day_2/Worksheet 3 - EDA Worksheet.ipynb".
The first task is:
Part 3: Which date had the total most bot infections and how many infections on that day?
For the next step, aggregate and sum the number of infections (hosts) by date.  Once you've done that, the next step is to sort in descending order.
Can you write Python code for it?

df3 = data[['date', 'hosts']].groupby('date').sum()
df3.sort_values(by='hosts')


I am working on the file "1m_ML_Security/notebooks/day_2/Worksheet 3 - EDA Worksheet.ipynb".
The first task is:
Part 4: Plot the daily infected hosts for Necurs, Ramnit and PushDo
For the final step, you're going to plot the daily infected hosts for three infection types.  In order to do this, you'll need to do the following steps:
1.  Filter the data to remove the botfamilies we don't care about.
2.  Use groupby() to aggregate the data by date and family, then sum up the hosts in each group
3.  Plot the data.  Hint:  You might want to use the unstack() function to prepare the data for plotting.
4.  Use the plot() method to plot the results.
Can you write Python code for it?

#Plot the data
groupedFilteredData.unstack(level=1).plot(kind='line', subplots=False)


I am working on the file "1m_ML_Security/notebooks/day_3/Worksheet 5 - Feature Engineering.ipynb".
The first task is:
Breakpoint: Load Features and Labels
If you got stuck in Part 1, don't worry, we will give you a final feature set for tomorrow's exercises, but your feature set should match the data below.
Can you write Python code for it?

df_final = pd.read_csv('../../data/dga_features_final_df.csv')
print(df_final.isDGA.value_counts())
df_final.sample(n=5).head()


I am working on the file "1m_ML_Security/notebooks/day_3/Worksheet 5 - Feature Engineering.ipynb".
The first task is:
Now let's use a Seaborn pairplot as well.  This will really show you which features have clear dividing lines between the classes.  Docs are available here: http://seaborn.pydata.org/generated/seaborn.pairplot.html
Can you write Python code for it?


I am working on the file "1m_ML_Security/notebooks/day_3/Worksheet 5 - Feature Engineering.ipynb".
The first task is:
Part 1 - Feature Engineering
Option 1 to derive Machine Learning features is to manually hand-craft useful contextual information of the domain string. An alternative approach (not covered in this notebook) is "Featureless Deep Learning", where an embedding layer takes care of deriving features - a huge step towards more "AI".
Previous academic research has focused on the following features that are based on contextual information:
List of features:

Length ["length"]
Number of digits ["digits"]
Entropy ["entropy"] - use H_entropy function provided
Vowel to consonant ratio ["vowel-cons"] - use vowel_consonant_ratio function provided
N-grams ["n-grams"] - use ngram functions provided

Tasks:
Split into A and B parts, see below...
Please run the following function cell and then continue reading the next markdown cell with more details on how to derive those features. Have fun!
Can you write Python code for it?

def H_entropy (x):
    # Calculate Shannon Entropy
    prob = [ float(x.count(c)) / len(x) for c in dict.fromkeys(list(x)) ]
    H = - sum([ p * np.log2(p) for p in prob ])
    return H

def vowel_consonant_ratio (x):
    # Calculate vowel to consonant ratio
    x = x.lower()
    vowels_pattern = re.compile('([aeiou])')
    consonants_pattern = re.compile('([b-df-hj-np-tv-z])')
    vowels = re.findall(vowels_pattern, x)
    consonants = re.findall(consonants_pattern, x)
    try:
        ratio = len(vowels) / len(consonants)
    except: # catch zero devision exception
        ratio = 0
    return ratio


I am working on the file "1m_ML_Security/notebooks/day_3/Worksheet 5 - Feature Engineering.ipynb".
The first task is:
Tasks - B - Feature Engineering
Finally, let's tackle the ngram feature. There are multiple steps involved to derive this feature. Here in this notebook, we use an implementation outlined in the this academic paper Schiavoni 2014: "Phoenix: DGA-based Botnet Tracking and Intelligence" - see section: Linguistic Features.

What are ngrams??? Imagine a string like 'facebook', if I were to derive all n-grams for n=2 (aka bi-grams) I would get '['fa', 'ac', 'ce', 'eb', 'bo', 'oo', 'ok']', so you see that you slide with one step from the left and just group 2 characters together each time, a tri-gram for 'facebook' would yielfd '['fac', 'ace', 'ceb', 'ebo', 'boo', 'ook']'. Ngrams have a long history in natural language processing, but are also used a lot for example in detecting malicious executable (raw byte ngrams in this case).

Steps involved:

We have the 10000 most common english words (see data file we loaded, we call this DataFrame top_en_words in this notebook). Now we run the ngrams functions on a list of all these words. The output here is a list that contains ALL 1-grams, bi-grams and tri-grams of these 10000 most common english words.
We use the Counter function from collections to derive a dictionary d that contains the counts of all unique 1-grams, bi-grams and tri-grams.
Our ngram_feature function will do the core magic. It takes your domain as input, splits it into ngrams (n is a function parameter) and then looks up these ngrams in the english dictionary d we derived in step 2. Function returns the normalized sum of all ngrams that were contained in the english dictionary. For example, running ngram_feature('facebook', d, 2) will return 171.28 (this value is just like the one published in the Schiavoni paper).
Finally average_ngram_feature wraps around ngram_feature. You will use this function as your task is to derive a feature that gives the average of the ngram_feature for n=1,2 and 3. Input to this function should be a simple list with entries calling ngram_feature with n=1,2 and 3, hence a...
Can you write Python code for it?

# ngrams: Implementation according to Schiavoni 2014: "Phoenix: DGA-based Botnet Tracking and Intelligence"
# http://s2lab.isg.rhul.ac.uk/papers/files/dimva2014.pdf

def ngrams(word, n):
    # Extract all ngrams and return a regular Python list
    # Input word: can be a simple string or a list of strings
    # Input n: Can be one integer or a list of integers
    # if you want to extract multipe ngrams and have them all in one list

    l_ngrams = []
    if isinstance(word, list):
        for w in word:
            if isinstance(n, list):
                for curr_n in n:
                    ngrams = [w[i:i+curr_n] for i in range(0,len(w)-curr_n+1)]
                    l_ngrams.extend(ngrams)
            else:
                ngrams = [w[i:i+n] for i in range(0,len(w)-n+1)]
                l_ngrams.extend(ngrams)
    else:
        if isinstance(n, list):
            for curr_n in n:
                ngrams = [word[i:i+curr_n] for i in range(0,len(word)-curr_n+1)]
                l_ngrams.extend(ngrams)
        else:
            ngrams = [word[i:i+n] for i in range(0,len(word)-n+1)]
            l_ngrams.extend(ngrams)
#     print(l_ngrams)
    return l_ngrams

def ngram_feature(domain, d, n):
    # Input is your domain string or list of domain strings
    # a dictionary object d that contains the count for most common english words
    # finally you n either as int list or simple int defining the ngram length

    # Core magic: Looks up domain ngrams in english dictionary ngrams and sums up the
    # respective english dictionary counts for the respective domain ngram
    # sum is normalized

    l_ngrams = ngrams(domain, n)
#     print(l_ngrams)
    count_sum=0
    for ngram in l_ngrams:
        if d[ngram]:
            count_sum+=d[ngram]
    try:
        feature = count_sum/(len(domain)-n+1)
    except:
        feature = 0
    return feature

def average_ngram_feature(l_ngram_feature):
    # input is a list of calls to ngram_feature(domain, d, n)
    # usually you would use various n...


I am working on the file "1m_ML_Security/notebooks/day_3/Worksheet 5 - Feature Engineering.ipynb".
The first task is:
Visualizing the Results
At this point, we've created a dataset which has many features that can be used for classification.  Using YellowBrick, your final step is to visualize the features to see which will be of value and which will not.
First, let's create a Rank2D visualizer to compute the correlations between all the features.  Detailed documentation available here: http://www.scikit-yb.org/en/latest/examples/methods.html#feature-analysis
Can you write Python code for it?

feature_names = ['length','digits','entropy','vowel-cons','ngrams']
features = df_final[feature_names]
target = df_final.isDGA

#Your Code here...


I am working on the file "1m_ML_Security/notebooks/day_3/Worksheet 5 - Feature Engineering.ipynb".
The first task is:
Worksheet for Part 1 - Feature Engineering
Can you write Python code for it?

## Load data
df = pd.read_csv('../../data/dga_data_small.csv')
df.drop(['host', 'subclass'], axis=1, inplace=True)
print(df.shape)
df.sample(n=5).head() # print a random sample of the DataFrame

df[df.isDGA == 'legit'].head()

# Google's 10000 most common english words will be needed to derive a feature called ngrams...
# therefore we already load them here.
top_en_words = pd.read_csv('../../data/google-10000-english.txt', header=None, names=['words'])
top_en_words.sample(n=5).head()
# Source: https://github.com/first20hours/google-10000-english


I am working on the file "1st_1.10.17/introduction_to_ipython.ipynb".
The first task is:
text
Header
для редактирования формулы ниже использует синтаксис tex
$$ c = \sqrt{a^2 + b^2}$$
Can you write Python code for it?

! echo 'hello, world!'

!echo $t

%%bash
mkdir test_directory
cd test_directory/
ls  -a

#удаление директории, если она не нужна
! rm -r test_directory


I am working on the file "1st_1.10.17/introduction_to_ipython.ipynb".
The first task is:
удаление директории, если она не нужна (windows)
Can you write Python code for it?

%%cmd
rmdir test_directiory

%lsmagic

%pylab inline

y = range(11)

y

plot(y)


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/1_Mean_Midean_Mode/MeanMedianExercise.ipynb".
The first task is:
Now, find the mean and median of this data. In the code block below, write your code, and see if your result makes sense:
Can you write Python code for it?

# find mean, median, mode

# mean
np.mean(transactions)
# median
np.median(transactions)

from scipy import stats
price = np.random.randint(1, high=100, size=10000)
stats.mode(price)


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/1_Mean_Midean_Mode/MeanMedianMode.ipynb".
The first task is:
Now compute the median - since we have a nice, even distribution it too should be close to 27,000:
Can you write Python code for it?


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/1_Mean_Midean_Mode/MeanMedianMode.ipynb".
The first task is:
Now we'll add Donald Trump into the mix. Darn income inequality!
Can you write Python code for it?

# append the value to the end of numpy array
incomes = np.append(incomes, [1000000000])


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/1_Mean_Midean_Mode/MeanMedianMode.ipynb".
The first task is:
The median won't change much, but the mean does:
Can you write Python code for it?

np.median(incomes)

np.mean(incomes) # mean has shifted to the rights


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/1_Mean_Midean_Mode/MeanMedianMode.ipynb".
The first task is:
We can segment the income data into 50 buckets, and plot it as a histogram:
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
# segment the income data into 50 buckets and plot as a histogram
plt.hist(incomes, 50)
plt.show()


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/3_Distributions/Distributions.ipynb".
The first task is:
Exponential PDF / "Power Law"
Can you write Python code for it?

from scipy.stats import expon
import matplotlib.pyplot as plt

x = np.arange(0, 10, 0.001)
plt.plot(x, expon.pdf(x))


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/3_Distributions/Distributions.ipynb".
The first task is:
Generate some random numbers with a normal distribution. "mu" is the desired mean, "sigma" is the standard deviation:
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

mu = 5.0
sigma = 2.0
values = np.random.normal(mu, sigma, 10000)
plt.hist(values, 50)
plt.show()


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/3_Distributions/Distributions.ipynb".
The first task is:
Normal / Gaussian
Visualize the probability density function:
Can you write Python code for it?

from scipy.stats import norm
import matplotlib.pyplot as plt

x = np.arange(-3, 3, 0.001)
plt.plot(x, norm.pdf(x))


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/3_Distributions/Distributions.ipynb".
The first task is:
Poisson Probability Mass Function
Example: My website gets on average 500 visits per day. What's the odds of getting 550?
Can you write Python code for it?

from scipy.stats import poisson
import matplotlib.pyplot as plt

mu = 500
x = np.arange(400, 600, 0.5)
plt.plot(x, poisson.pmf(x, mu))


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/3_Distributions/Distributions.ipynb".
The first task is:
Pop Quiz!
What's the equivalent of a probability distribution function when using discrete instead of continuous data?
Can you write Python code for it?


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/Outliers.ipynb".
The first task is:
It's important to dig into what is causing your outliers, and understand where they are coming from. You also need to think about whether removing them is a valid thing to do, given the spirit of what it is you're trying to analyze. If I know I want to understand more about the incomes of "typical Americans", filtering out billionaires seems like a legitimate thing to do.
Here's something a little more robust than filtering out billionaires - it filters out anything beyond two standard deviations of the median value in the data set:
Can you write Python code for it?

def reject_outliers(data):
    u = np.median(data)
    s = np.std(data)
    filtered = [e for e in data if (u - 2 * s < e < u + 2 * s)]
    return filtered

filtered = reject_outliers(incomes)

plt.hist(filtered, 50)
plt.show()


I am working on the file "1t_DataAnalysisMLPython/1j_ML/DS_ML_Py_SBO/DataScience/Outliers.ipynb".
The first task is:
That's not very helpful to look at. One billionaire ended up squeezing everybody else into a single line in my histogram. Plus it skewed my mean income significantly:
Can you write Python code for it?


I am working on the file "2 - EE 101.ipynb".
The first task is:
And by changing the bands displayed, we can also display a false color image.
Can you write Python code for it?

Image(url=l8_image.getThumbUrl({
    'region':region_california,
    'bands':'B5,B4,B3',
    'min':0,
    'max':0.3
}))


I am working on the file "2 - EE 101.ipynb".
The first task is:
And now, we can print out information about the dataset, using the .getInfo() method.
Can you write Python code for it?


I am working on the file "2 - EE 101.ipynb".
The first task is:
Hello, Images
Let's work with something more interesting... a dataset provided by Earth Engine.
Assuming that this server has been setup with access to the Earth Engine Python API, we should be able to import and initialise the Earth Engine Python module (named 'ee'). If the module loads successfully, nothing will be returned when you run the following code.
Can you write Python code for it?


I am working on the file "2 - EE 101.ipynb".
The first task is:
Hello, World
To get used to using IPython Notebooks, let's print some simple output back to the notebook. Click on the box below, and then press the play (run) button from the toolbar above.
Can you write Python code for it?


I am working on the file "2 - EE 101.ipynb".
The first task is:
ImageCollection.max Reducer
Can you write Python code for it?

l8_image = filtered.max()
Image(url=l8_image.getThumbUrl({
    'region':region_california,
    'bands':'B4,B3,B2',
    'min':0,
    'max':0.3
}))


I am working on the file "2 - EE 101.ipynb".
The first task is:
ImageCollection.min Reducer
Can you write Python code for it?

l8_image = filtered.min()
Image(url=l8_image.getThumbUrl({
    'region':region_california,
    'bands':'B4,B3,B2',
    'min':0,
    'max':0.3
}))


I am working on the file "2 - EE 101.ipynb".
The first task is:
What is returned by the .getInfo() command is a Python dictionary. If needed, we could parse out this information and make use of it in our analysis.
Add an Image to the Map
IPython Notebooks can be used to display an image, using the Image module:
Can you write Python code for it?

from IPython.display import Image

Image(url=srtm.getThumbUrl())


I am working on the file "2 - Generate coupon probabilities - part 2.ipynb".
The first task is:
Notice that as we decreased the standard deviation (from half of the mean to a tenth of the mean) our spread of probabilities got a lot smaller.  Also notice that this method doesn't require any scaling of the probabilities to get them to sum to 1.  This gives me a warm and fuzzy feeling that this is a good way to generate coupon probabilities.
Additional thought:  What are the allowable values for the standard deviation?
Obviously the minimum value would be 0.  This would give us the coupon collector's problem with a uniform distribution, so it is a good way to check our work.  But what is the max value?  From the example below, we see that there is clearly a point where the first few coupons have negative probabilities.  This should be avoided:
Can you write Python code for it?

n = 20 #number of coupons
mu = 1/n #this is the mean coupon probability
sigma = mu #this is the std dev parameter we will play around with - it seems to make sense to express it in terms of the mean
x = np.arange(n)+0.5 #arange goes from 0 to n-1, and I want it to go from 1 to n
p_x = stat.norm.ppf(x/n, mu, sigma)
print('mean prob = %1.16f' % np.mean(p_x), ', sum of all probs = %1.16f' % np.sum(p_x) )
plt.bar(x, p_x, align='center')
plt.axis([0,n,-0.1,3/n])


I am working on the file "2 - Generate coupon probabilities - part 2.ipynb".
The first task is:
Option 2: Normal Distribution
After playing around with the previous option (in the notebook GenerateProbs) and thinking a little more about what makes a distribution "normal," I've come up with the following method that I think will work nicely for generating probabilities for a general coupon collector simulation.
My idea starts with the idea of a quantile function (called a percent point function (ppf) in SciPy's stats library).  This is because the ppf is a way to get the probability of most of the coupons centered around some number with a few coupons having lower probabilities and a few coupons having higher probabilities; a ppf is the inverse of a pdf which would give the number of coupons having that probability.
My idea continues by knowing that normal distributions are symmetrical around some number (the mean) and that the mean of the coupons' probabilities must be $\frac{1}{n}$.  This is because there are $n$ coupons and the sum of their probabilities must be 1, so the mean must be $\frac{1}{n}$.
The only parameter that defines a normal distribution other than the mean is the standard deviation (or alternately the variance), and this can change to give us a different "spread" of probabilities around the mean of $\frac{1}{n}$.  The higher the standard deviation, the greater the spread, just like we expect.
While this may not be the absolute best way to generate coupon probabilities, it seems like this results in a set of probabilities that meets criteria that most people would call "normal."  Let's look at an example and see how changing the standard deviation around affects the shape of the probabilities.
Can you write Python code for it?

n = 20 #number of coupons
mu = 1/n #this is the mean coupon probability
sigma = mu/2 #this is the std dev parameter we will play around with - it seems to make sense to express it in terms of the mean
x = np.arange(n)+0.5 #arange goes from 0 to n-1, and I want it to go from 1 to n
p_x = stat.norm.ppf(x/(n), mu, sigma)
print('mean prob = %1.16f' % np.mean(p_x), ', sum of all probs = %1.16f' % np.sum(p_x) )
plt.bar(x, p_x, align='center')
plt.axis([0,n,0,2/n])

n = 20 #number of coupons
mu = 1/n #this is the mean coupon probability
sigma = mu/10 #this is the std dev parameter we will play around with - it seems to make sense to express it in terms of the mean
x = np.arange(n)+0.5 #arange goes from 0 to n-1, and I want it to go from 1 to n
p_x = stat.norm.ppf(x/n, mu, sigma)
print('mean prob = %1.16f' % np.mean(p_x), ', sum of all probs = %1.16f' % np.sum(p_x) )
plt.bar(x, p_x, align='center')
plt.axis([0,n,0,2/n])


I am working on the file "2 - Neural Network/Project/Your_first_neural_network.ipynb".
The first task is:
Checking out the data
This dataset has the number of riders for each hour of each day from January 1 2011 to December 31 2012. The number of riders is split between casual and registered, summed up in the cnt column. You can see the first few rows of the data above.
Below is a plot showing the number of bike riders over the first 10 days or so in the data set. (Some days don't have exactly 24 entries in the data set, so it's not exactly 10 days.) You can see the hourly rentals here. This data is pretty complicated! The weekends have lower over all ridership and there are spikes when people are biking to and from work during the week. Looking at the data above, we also have information about temperature, humidity, and windspeed, all of these likely affecting the number of riders. You'll be trying to capture all this with your model.
Can you write Python code for it?


I am working on the file "2 - Neural Network/Project/Your_first_neural_network.ipynb".
The first task is:
Dummy variables
Here we have some categorical variables like season, weather, month. To include these in our model, we'll need to make binary dummy variables. This is simple to do with Pandas thanks to get_dummies().
Can you write Python code for it?

dummy_fields = ['season', 'weathersit', 'mnth', 'hr', 'weekday']
for each in dummy_fields:
    dummies = pd.get_dummies(rides[each], prefix=each, drop_first=False)
    rides = pd.concat([rides, dummies], axis=1)

fields_to_drop = ['instant', 'dteday', 'season', 'weathersit',
                  'weekday', 'atemp', 'mnth', 'workingday', 'hr']
data = rides.drop(fields_to_drop, axis=1)
data.head()


I am working on the file "2 - Neural Network/Project/Your_first_neural_network.ipynb".
The first task is:
Scaling target variables
To make training the network easier, we'll standardize each of the continuous variables. That is, we'll shift and scale the variables such that they have zero mean and a standard deviation of 1.
The scaling factors are saved so we can go backwards when we use the network for predictions.
Can you write Python code for it?

quant_features = ['casual', 'registered', 'cnt', 'temp', 'hum', 'windspeed']
# Store scalings in a dictionary so we can convert back later
scaled_features = {}
for each in quant_features:
    mean, std = data[each].mean(), data[each].std()
    scaled_features[each] = [mean, std]
    data.loc[:, each] = (data[each] - mean)/std


I am working on the file "2 - Neural Network/Project/Your_first_neural_network.ipynb".
The first task is:
Time to build the network
Below you'll build your network. We've built out the structure and the backwards pass. You'll implement the forward pass through the network. You'll also set the hyperparameters: the learning rate, the number of hidden units, and the number of training passes.
<img src="assets/neural_network.png" width=300px>
The network has two layers, a hidden layer and an output layer. The hidden layer will use the sigmoid function for activations. The output layer has only one node and is used for the regression, the output of the node is the same as the input of the node. That is, the activation function is $f(x)=x$. A function that takes the input signal and generates an output signal, but takes into account the threshold, is called an activation function. We work through each layer of our network calculating the outputs for each neuron. All of the outputs from one layer become inputs to the neurons on the next layer. This process is called forward propagation.
We use the weights to propagate signals forward from the input to the output layers in a neural network. We use the weights to also propagate error backwards from the output back into the network to update our weights. This is called backpropagation.

Hint: You'll need the derivative of the output activation function ($f(x) = x$) for the backpropagation implementation. If you aren't familiar with calculus, this function is equivalent to the equation $y = x$. What is the slope of that equation? That is the derivative of $f(x)$.

Below, you have these tasks:
1. Implement the sigmoid function to use as the activation function. Set self.activation_function in __init__ to your sigmoid function.
2. Implement the forward pass in the train method.
3. Implement the backpropagation algorithm in the train method, including calculating the output error.
4. Implement the forward pass in the run method.
Can you write Python code for it?

class NeuralNetwork(object):
    def __init__(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
        # Set number of nodes in input, hidden and output layers.
        self.input_nodes = input_nodes
        self.hidden_nodes = hidden_nodes
        self.output_nodes = output_nodes

        # Initialize weights
        self.weights_input_to_hidden = np.random.normal(0.0, self.input_nodes**-0.5,
                                       (self.input_nodes, self.hidden_nodes))

        self.weights_hidden_to_output = np.random.normal(0.0, self.hidden_nodes**-0.5,
                                       (self.hidden_nodes, self.output_nodes))
        self.lr = learning_rate

        #### TODO: Set self.activation_function to your implemented sigmoid function ####
        self.activation_function = lambda x : 1.0/(1.0 + np.exp(-x))  # Replace 0 with your sigmoid calculation.

    def train(self, features, targets):
        ''' Train the network on batch of features and targets.

            Arguments
            ---------

            features: 2D array, each row is one data record, each column is a feature
            targets: 1D array of target values

        '''
        n_records = features.shape[0]
        delta_weights_i_h = np.zeros(self.weights_input_to_hidden.shape)
        delta_weights_h_o = np.zeros(self.weights_hidden_to_output.shape)
        for X, y in zip(features, targets):
            #### Implement the forward pass here ####
            ### Forward pass ###
            # TODO: Hidden layer - Replace these values with your calculations.
            hidden_inputs = np.dot(X, self.weights_input_to_hidden) # signals into hidden layer
            hidden_outputs = self.activation_function(hidden_inputs) # signals from hidden layer

            # TODO: Output layer - Replace these values with your calculations.
            final_inputs = np.dot(hidden_outputs, self.weights_hidden_to_output) # signals into final output layer
            final_outputs =...


I am working on the file "2 - Neural Network/Project/Your_first_neural_network.ipynb".
The first task is:
Training the network
Here you'll set the hyperparameters for the network. The strategy here is to find hyperparameters such that the error on the training set is low, but you're not overfitting to the data. If you train the network too long or have too many hidden nodes, it can become overly specific to the training set and will fail to generalize to the validation set. That is, the loss on the validation set will start increasing as the training set loss drops.
You'll also be using a method know as Stochastic Gradient Descent (SGD) to train the network. The idea is that for each training pass, you grab a random sample of the data instead of using the whole data set. You use many more training passes than with normal gradient descent, but each pass is much faster. This ends up training the network more efficiently. You'll learn more about SGD later.
Choose the number of iterations
This is the number of batches of samples from the training data we'll use to train the network. The more iterations you use, the better the model will fit the data. However, if you use too many iterations, then the model with not generalize well to other data, this is called overfitting. You want to find a number here where the network has a low training loss, and the validation loss is at a minimum. As you start overfitting, you'll see the training loss continue to decrease while the validation loss starts to increase.
Choose the learning rate
This scales the size of weight updates. If this is too big, the weights tend to explode and the network fails to fit the data. A good choice to start at is 0.1. If the network has problems fitting the data, try reducing the learning rate. Note that the lower the learning rate, the smaller the steps are in the weight updates and the longer it takes for the neural network to converge.
Choose the number of hidden nodes
The more hidden nodes you have, the more accurate predictions the model will make. Try a few different numbers and see how it affects the performance. You can look at the losses...
Can you write Python code for it?

import sys

iterations = 3000
learning_rate = 0.9
hidden_nodes = 15
output_nodes = 1

N_i = train_features.shape[1]
network = NeuralNetwork(N_i, hidden_nodes, output_nodes, learning_rate)

losses = {'train':[], 'validation':[]}
for ii in range(iterations):
    # Go through a random batch of 128 records from the training data set
    batch = np.random.choice(train_features.index, size=128)
    X, y = train_features.ix[batch].values, train_targets.ix[batch]['cnt']

    network.train(X, y)

    # Printing out the training progress
    train_loss = MSE(network.run(train_features).T, train_targets['cnt'].values)
    val_loss = MSE(network.run(val_features).T, val_targets['cnt'].values)
    sys.stdout.write("\rProgress: {:2.1f}".format(100 * ii/float(iterations)) \
                     + "% ... Training loss: " + str(train_loss)[:5] \
                     + " ... Validation loss: " + str(val_loss)[:5])
    sys.stdout.flush()

    losses['train'].append(train_loss)
    losses['validation'].append(val_loss)

plt.plot(losses['train'], label='Training loss')
plt.plot(losses['validation'], label='Validation loss')
plt.legend()
_ = plt.ylim()