I am working on the file "2017/iker/Ferran AoC 2017.ipynb".
The first task is:
Day 8
Exec is a good idea (and makes the code really simple), but there are also other solutions.
Same for day 18
Can you write Python code for it?

def apply_instructions(registers):
    for reg, leap, cond in instructions:
        bool_str = 'registers["{0}"]'.format(cond[0]) + ''.join(cond[1:])
        update_str = 'if {0}: registers["{1}"] += {2} '.format(bool_str, reg, leap)
        exec(update_str)

import operator

comparisons = {'>': operator.gt, '>=': operator.ge, '<': operator.lt,
               '<=': operator.le, '==': operator.eq, '!=': operator.ne}

def process(instruction):
    reg, operation, val, condition = parse(instruction)
    cond_reg, cond_op, cond_val = condition
    if cond_op(registers[cond_reg], cond_val):
        registers[reg] = operation(registers[reg], val)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
"Hello World!" : Databases, Collections, Documents
Relational databases contain tables that contain records.
A MongoDB database contains collections that contain documents.
Can you write Python code for it?

# start a client
client = pm.MongoClient()

# connect to a database
db = client.tutorial

# get a collection
coll = db.test_collection


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Aggregation
Aggregation happens in stages.
Can you write Python code for it?

# Group Documents by a Field and Calculate Count
cursor = restaurants.aggregate(
    [
        {"$group": {"_id": "$borough", "count": {"$sum": 1}}}
    ]
)

print_my_docs(cursor, 10)

# Filter and Group Documents
cursor = restaurants.aggregate(
    [
        {"$match": {"borough": "Queens", "cuisine": "Brazilian"}},
        {"$group": {"_id": "$address.zipcode", "count": {"$sum": 1}}}
    ]
)

print_my_docs(cursor, 10)

# Filter and Group and then Filter Again documents
cursor = restaurants.aggregate(
    [
        {"$match": {"borough": "Manhattan", "cuisine": "American"}},
        {"$group": {"_id": "$address.zipcode", "count": {"$sum": 1}}},
        {"$match": {"count": {"$gt": 1}}}
    ]
)

print_my_docs(cursor, 10)

# Filter and Group and then Filter Again and then Sort Documents
cursor = restaurants.aggregate(
    [
        {"$match": {"borough": "Manhattan", "cuisine": "American"}},
        {"$group": {"_id": "$address.zipcode",  "count": {"$sum": 1}}},
        {"$match": {"count": {"$gt": 1}}},
        {"$sort":  {"count": -1, "_id": -1}}
    ]
)

print_my_docs(cursor, 10)

# Same but sort by multiple fields
# Filter and Group and then Filter Again and then Sort Documents
cursor = restaurants.aggregate(
    [
        {"$match": {"borough": "Manhattan", "cuisine": "American"}},
        {"$group": {"_id": "$address.zipcode",  "count": {"$sum": 1}}},
        {"$match": {"count": {"$gt": 1}}},
        {"$sort":  son.SON([("count", -1), ("_id", 1)])} # order matters!!
    ]
)

print_my_docs(cursor, 10)

# what will this do?
cursor = restaurants.aggregate(
    [
        {"$group": {"_id": None, "count": {"$sum": 1}} }
    ]
)

print_my_docs(cursor, 10)

# projection
# what will this do?
cursor = restaurants.aggregate(
    [
        {"$group": {"_id": "$address.zipcode", "count": {"$sum": 1}}},
        {"$project": {"_id": 0, "count": 1}}
    ]
)

print_my_docs(cursor, 10)

# what will this do?
cursor = restaurants.aggregate(
    [
        {"$group": {"_id": {"cuisine": "$cuisine"}, "count": {"$sum": 1}}},
...


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Alternatively, you can import the dataset by running the same command on a terminal.

mongoimport --db moderndb --collection restaurants --drop --file dataset.json

The dataset contains documents that look like the one below.
 Restaurant Example
{
  "address": {
     "building": "1007",
     "coord": [ -73.856077, 40.848447 ],
     "street": "Morris Park Ave",
     "zipcode": "10462"
  },
  "borough": "Bronx",
  "cuisine": "Bakery",
  "grades": [
     { "date": { "$date": 1393804800000 }, "grade": "A", "score": 2 },
     { "date": { "$date": 1378857600000 }, "grade": "A", "score": 6 },
     { "date": { "$date": 1358985600000 }, "grade": "A", "score": 10 },
     { "date": { "$date": 1322006400000 }, "grade": "A", "score": 9 },
     { "date": { "$date": 1299715200000 }, "grade": "B", "score": 14 }
  ],
  "name": "Morris Park Bake Shop",
  "restaurant_id": "30075445"
}
Can you write Python code for it?

restaurants = db.restaurants # our new collection

# how many restaurants?
restaurants.count()


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
As we see in this example, MongoDB makes use of an index (as indicated by keyword "IXSCAN") -- and particularly the index ('borough_1') we constructed to execute the query.
What if we had not built this index?
Can you write Python code for it?

restaurants.drop_indexes() # we drop all indexes first -- use this with care!
my_cursor = restaurants.find({"borough": "brooklyn"}) # submit query to find restaurants from specific borough
my_cursor.explain()["queryPlanner"]["winningPlan"] # ask mongodb to explain execution plan


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Documents follow the JSON format and MongoDB stores them in a binary version of it (BSON).

Below you see examples of JSON documents.
 JSON example 0
{}
 JSON example 1
{
"name" : "Michael",
"age": 32,
"grades": [71, 85, 90, 34]
}
 JSON example 2
{
  "first name": "Michael",
  "last name": "Mathioudakis",
  "age": 32,
  "grades": {
    "ModernDB": 69,
    "Data Mining": 71,
    "Machine Learning": 95
  },
  "graduated": true,
  "previous schools": ["NTUA", "UofT"]
}
In Python, JSON documents are represented as dictionaries.
The examples from above are therefore represented as follows.
Can you write Python code for it?

example_0 = {}

example_1 = {"name": "Michael", "age": 32, "grades": [71, 85, 90, 34]}

example_2 = \
{"first name": "Michael",
  "last name": "Mathioudakis",
  "age": 32,
  "grades": {
    "ModernDB": 69,
    "Data Mining": 71,
    "Machine Learning": 95
  },
  "graduated": True,
  "previous schools": ["NTUA", "UofT"]
}


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
If we call the collection's function find(), we get back a cursor.
Can you write Python code for it?


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
In that case, MongoDB simply performs a scan over the collection (as indicated by keyword "COLLSCAN").
Joins
Until very recently, MongoDB did not support joins.
It was up to the user to implement a join if needed -- as in the cell below.
Can you write Python code for it?

for a in restaurants.find({"borough": "Manhattan"}).limit(7):
    for b in restaurants.find({"borough": "Bronx"}).limit(5):
        if a["cuisine"] == b["cuisine"]:
            print(a["cuisine"], a["address"]["zipcode"], b["address"]["zipcode"])


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Indexing
MongoDb automatically creates an index on the _id field upon creating a collection.
We can use create_index() to create index on one or more fields of a collection.
Single-field index
Can you write Python code for it?

# note that the argument is a list of tuples
# [(<field>: <type>), ...]
# here, we specify only one such tuple for one field
restaurants.create_index([("borough", pm.ASCENDING)])


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Inserting and finding documents
Our collection coll is currently empty. Let's add one document to it.
Can you write Python code for it?


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Joins with \$lookup
This is a new aggregation stage that implements left outer equi-joins.
"A left outer equi-join produces a result set that contains data for all documents from the left table (collection) together with data from the right table (collection) for documents where there is a match with documents from the left table (collection)."
Can you write Python code for it?

# create first collection
orders_docs = [{ "_id" : 1, "item" : "abc", "price" : 12, "quantity" : 2 },
{ "_id" : 2, "item" : "jkl", "price" : 20, "quantity" : 1 },
{ "_id" : 3  }]
orders = db.orders
orders.drop()
orders.insert_many(orders_docs)

# create second collection
inventory_docs = [
    { "_id" : 1, "item" : "abc", "description": "product 1", "instock" : 120 },
{ "_id" : 2, "item" : "def", "description": "product 2", "instock" : 80 },
{ "_id" : 3, "item" : "ijk", "description": "product 3", "instock" : 60 },
{ "_id" : 4, "item" : "jkl", "description": "product 4", "instock" : 70 },
{ "_id" : 5, "item": None, "description": "Incomplete" },
{ "_id" : 6 }
]
inventory = db.inventory
inventory.drop()
inventory.insert_many(inventory_docs)

result = orders.aggregate([ # "orders" is the outer collection
    {
      "$lookup":
        {
          "from": "inventory", # the inner collection
          "localField": "item", # the join field of the outer collection
          "foreignField": "item", # the join field of the outer collection
          "as": "inventory_docs" # name of field with array of joined inner docs
        }
   }
])

print_my_docs(result, 10)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Limiting the number of results
Can you write Python code for it?

# what will this do?

cursor = restaurants.aggregate(
    [
        {"$group": {"_id": {"cuisine": "$cuisine", "zip": "$address.zipcode"}, "count": {"$sum": 1}}},
        {"$sort": {"count": -1}},
        {"$limit": 10} # See comment under "In-class questions"
    ]
)

for doc in cursor:
    print(doc["_id"]["cuisine"], doc["_id"]["zip"], doc["count"])


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Multi-key index
An index for a fields with array value.
Can you write Python code for it?

restaurants.find_one()

restaurants.create_index([("address.coord", 1)])

restaurants.create_index([("grades.score", 1)])

restaurants.create_index([("grades.grade", 1), ("grades.score", 1)])


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Multiple Conditions
Can you write Python code for it?

# logical AND
cursor = restaurants.find({"cuisine": "Italian", "address.zipcode": "10075"})

next(cursor)

# logical OR
cursor = restaurants.find({"$or": [{"cuisine": "Italian"},
                                          {"address.zipcode": "10075"}]})

print_my_docs(cursor, 3)

# logical AND, differently
cursor = restaurants.find({"$and": [{"cuisine": "Italian"},
                                          {"address.zipcode": "10075"}]})

next(cursor)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Note on the semantics of the \$not operator
The operator selects documents that do not match the specified condition on the specified field. These documents include ones that do not contain the field.
To demonstrate this, consider the following simple example of a collection.
Can you write Python code for it?

# simple example of a collection
mycoll = db.mycoll
mycoll.drop()

# insert three documents
mycoll.insert_one({"grades": [7, 7]})
mycoll.insert_one({"grades": [7, 3]})
mycoll.insert_one({"grades": [3, 3]})
mycoll.insert_one({"grades": []})
mycoll.insert_one({})


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Notice how MongoDB added an "_id" for the new document, as well.
Let's insert more documents.
Can you write Python code for it?

coll.insert_many([example_2, example_3])

for doc in coll.find():
    print(doc)
    print()


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Notice how the document we insert do not follow a schema?
Let us now find documents that match a condition -- let's say we want to find documents that have a field "name" with value "Michael".
Can you write Python code for it?

query_result = coll.find({"name": "Michael"})
for doc in query_result:
    print(doc)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Notice that the empty document we inserted is not really empty, but associated with an "_id" key, added by MongoDB.
Let's try another one.
Can you write Python code for it?

coll.insert_one(example_1)

for doc in coll.find():
    print(doc)
    print()


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Projecting fields
We can use find() not only to retrieve documents that match a condition, but also to project only those fields that we are interested in.
For example, to suppress the "_id" field from appearing in the results, we can provide a second argument to find(), as follows.
Can you write Python code for it?

query_result = coll.find({"name": "Michael"}, {"_id": 0})
for doc in query_result:
    print(doc)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Querying the Dataset
Can you write Python code for it?

# retrieve a cursor over all documents in the collection
cursor = restaurants.find()

# define printing function
def print_my_docs(cursor, num):
    for i in range(num): # print only up to num next documents from cursor
        try:
            print(next(cursor))
            print()
        except:
            break

# let's print a few documents
print_my_docs(cursor, 3)

next(cursor) # get one more document


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Questions from tutorial sessions
Question: How do we query for documents with an array field, all the elements of which satisfy a condition?
Two approaches (if you can think of a different approach, please let me know):
* Use the \$not operators: form a query to express that "there is no element in the array that does not satisfy the condition".
* In aggregation, combine an \$unwind stage with a $group stage.
To provide an example, let's say we want to find restaurants with 'A' grades only.
Below we show how we can use each of the aforementioned approaches.
First approach: using $not
Can you write Python code for it?

# using the $not operator
# "find restaurants that contain no grades that are not equal to A"
cursor = restaurants.find({"grades.grade": {"$exists": True}, "grades": {"$not": {"$elemMatch": {"grade": {"$ne": "A"}}}}})

print_my_docs(cursor, 3)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Retrieving the execution plan
We can retrieve the execution plan for a find() query by calling the explain() function on the result cursor. We demonstrate this in the following example.
Can you write Python code for it?

restaurants.drop_indexes() # we drop all indexes first -- use this with care!
restaurants.create_index([("borough", pm.ASCENDING)]) # build an index on field "borough", in ascending order
my_cursor = restaurants.find({"borough": "brooklyn"}) # submit query to find restaurants from specific borough
my_cursor.explain()["queryPlanner"]["winningPlan"] # ask mongodb to explain execution plan


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
SQL Query
SELECT zipcode, cuisine, COUNT(*) as count
FROM restaurants
WHERE borough = "Manhattan"
GROUP BY zipcode, cuisine
HAVING COUNT(*) > 3
ORDER BY count
Can you write Python code for it?

cursor = restaurants.aggregate(
    [
        {"$match": {"borough": "Manhattan"}},
        {"$group": {"_id": {"zipcode": "$address.zipcode", "cuisine": "$cuisine"}, "count": {"$sum": 1}}},
        {"$match": {"count": {"$gt": 3}}},
        {"$sort": {"count": 1}}
    ]
)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
SQL query
SELECT borough, cuisine, COUNT(*) as count
FROM restaurants
GROUP BY borough, cuisine
HAVING COUNT(*) > 3
Can you write Python code for it?

cursor = restaurants.aggregate(
    [
        {"$group": {"_id": {"borough": "$borough", "cuisine": "$cuisine"}, "count": {"$sum": 1}}},
        {"$match": {"count": {"$gt": 3}}}
    ]
)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
SQL to Aggregation
Here we explore the correspondence between SQL queries and the aggregation framework.
 SQL query
SELECT COUNT(*) AS count
FROM restaurants
Can you write Python code for it?

cursor = restaurants.aggregate(
    [
        {"$group": {"_id": None, "count": {"$sum": 1}} }
    ]
)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Sorting
Can you write Python code for it?

cursor = restaurants.find()

# to sort, specify list of sorting criteria,
# each criterion given as a tuple
# (field_name, sort_order)
# here we have only one
sorted_cursor = cursor.sort([("borough", pm.ASCENDING)])

print_my_docs(cursor, 2)

another_sorted_cursor = restaurants.find().sort([("borough", pm.ASCENDING),
                                     ("address.zipcode", pm.DESCENDING)])

print_my_docs(another_sorted_cursor, 3)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Specify equality conditions
Can you write Python code for it?

# top-level field
cursor = restaurants.find({"borough": "Manhattan"})

print_my_docs(cursor, 2)

# nested field (in embedded document)
cursor = restaurants.find({"address.zipcode": "10075"})

print_my_docs(cursor, 2)

# query by field in array
cursor = restaurants.find({"grades.grade": "B"})

# print one document from the query result
next(cursor)['grades']

# exact array match
cursor = restaurants.find({"address.coord": [-73.98513559999999, 40.7676919]})

print_my_docs(cursor, 10)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
Storing the result as a collection
We can use operator \$out in a final stage to store the result of a query into a new collection. The following example selects restaurants from Manhattan and stores them in their own collection in the same database.
Can you write Python code for it?

restaurants.aggregate(
    [
        {"$match": {"borough": "Manhattan"}},
        {"$out": "manhattan"}
    ]
)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
The following will not work!
We cannot currently have compound multi-key indexes.
Can you write Python code for it?

restaurants.create_index([("address.coord", 1), ("grades.score", 1)]) # NOPE!


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
The result of the following query contains documents that do not contain the "grades" field.
Can you write Python code for it?

# find documents that have no "grades" elements that are not equal to "A"
mycursor = mycoll.find({"grades": {"$not": {"$elemMatch": {"$ne": 7}}}})
print_my_docs(mycursor, 10)


I am working on the file "2017/mongodb.tutorial.ipynb".
The first task is:
What if we're interested in keeping only some of the rest of the fields -- let's say, only "grades"?
Can you write Python code for it?

query_result = coll.find({"name": "Michael"}, {"_id": 0, "grades": 1})
for doc in query_result:
    print(doc)


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:

Can you write Python code for it?

text = sc.textFile("myfile.txt",1) # load data

text = sc.textFile("myfile.txt") # load data

# count only lines that mention "Spark"
spark_lines = text.filter(lambda line: 'Spark' in line)

spark_lines.collect()


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
(Pseudo) set operations
Can you write Python code for it?

oneRDD = sc.parallelize([1, 1, 1, 2, 3, 3, 4, 4])
oneRDD.persist()

otherRDD = sc.parallelize([1, 4, 4, 7])
otherRDD.persist()

unionRDD = oneRDD.union(otherRDD)
unionRDD.persist()

oneRDD.subtract(otherRDD).collect()

oneRDD.distinct().collect()

oneRDD.intersection(otherRDD).collect() # removes duplicates

oneRDD.cartesian(otherRDD).collect()[:5]


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
Accumulators
This example demonstrates how to use accumulators.
The map operations creates an RDD that contains the length of lines in the text file - and while the RDD is materialized, an accumulator keeps track of how many lines are long (longer than $30$ characters).
Can you write Python code for it?

text = sc.textFile("myfile.txt")
long_lines = sc.accumulator(0) # create accumulator

def line_len(line):
    global long_lines # to reference an accumulator, declare it as global variable
    length = len(line)
    if length > 30:
        long_lines += 1 # update the accumulator
    return length

llengthRDD = text.map(line_len)
llengthRDD.count()

long_lines.value # this is how we obtain the value of the accumulator in the driver program

help(long_lines)

long_lines.value # this is how we obtain the value of the accumulator in the driver program


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
Broadcast variable
We use broadcast variables when many operations depend on the same large static object - e.g., a large lookup table that does not change but provides information for other operations. In such cases, we can make a broadcast variable out of the object and thus make sure that the object will be shipped to the cluster only once - and not for each of the operations we'll be using it for.
The example below demonstrates the usage of broadcast variables. In this case, we make a broadcast variable out of a dictionary that represents an address table. The tablke is shipped to cluster nodes only once across multiple operations.
Can you write Python code for it?

def load_address_table():
    return {"Anu": "Chem. A143", "Karmen": "VTT, 74", "Michael": "OIH, B253.2",
            "Anwar": "T, B103", "Orestis": "T, A341", "Darshan": "T, A325"}

address_table = sc.broadcast(load_address_table())

def find_address(name):
    res = None
    if name in address_table.value:
        res = address_table.value[name]
    return res

people = sc.parallelize(["Anwar", "Michael", "Orestis", "Darshan"])
pairRDD = people.map(lambda name: (name, find_address(name))) # first operation that uses the address table
print(pairRDD.collectAsMap())

other_people = sc.parallelize(["Karmen", "Michael", "Anu"])
pairRDD = other_people.map(lambda name: (name, find_address(name))) # second operation that uses the address table
print(pairRDD.collectAsMap())


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
Creating RDDS
We saw that we can create RDDs by loading files from disk. We can also create RDDs from Python collections or transforming other RDDs.
Can you write Python code for it?

my_list = [0,1,2,3,4,5,6,7,8,9]

data = sc.parallelize(my_list) # create RDD from Python collection

data_squared = data.map(lambda num: num ** 2) # transformation

data_squared.collect()


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
Example: Computing PageRank
Can you write Python code for it?

# Details of the algorithm can be found here: http://www.cs.princeton.edu/~chazelle/courses/BIB/pagerank.htm
iterations = 5

def computeContribs(urls, rank):
    """Calculates URL contributions to the rank of other URLs."""
    num_urls = len(urls)
    for url in urls:
        yield (url, rank / num_urls)

def parseNeighbors(urls):
    """Parses a urls pair string into urls pair."""
    parts = urls.split(',')
    return parts[0], parts[1]

# Read the lines
lines = sc.textFile("higgs-mention_network.txt").persist()

lines.collect()

# Loads all URLs from input file and initialize their neighbors.
links = lines.map(lambda urls: parseNeighbors(urls)).distinct().groupByKey().cache()

links.collect()

# Loads all URLs with other URL(s) link to from input file and initialize ranks of them to one.
ranks = links.map(lambda url_neighbors: (url_neighbors[0], 1.0))

ranks.collect()

# Calculates and updates URL ranks continuously using PageRank algorithm.
for iteration in range(iterations):
    # Calculates URL contributions to the rank of other URLs.
    contribs = links.join(ranks).flatMap(
        lambda url_urls_rank: computeContribs(url_urls_rank[1][0], url_urls_rank[1][1]))
    # Re-calculates URL ranks based on neighbor contributions.
    ranks = contribs.reduceByKey(lambda x,y: x+y).mapValues(lambda rank: rank * 0.85 + 0.15)
# Collects all URL ranks and dump them to console.
for (link, rank) in ranks.collect():
    print("%s has rank: %s." % (link, rank))


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
From https://github.com/vaquarkhan/vk-wiki-notes/wiki/reduceByKey--vs-groupBykey-vs-aggregateByKey-vs-combineByKey
ReduceByKey will aggregate y key before shuffling:

GroupByKey will shuffle all the value key pairs as the diagrams show:

combineByKey
Can you write Python code for it?

help(pairRDD.combineByKey)

pairRDD = sc.parallelize([ ('$APPL', 100.64), ('$GOOG', 706.2), ('$AMZN', 552.32), ('$APPL', 100.52), ('$AMZN', 552.32) ])

aggr = pairRDD.combineByKey(createCombiner = lambda x: (x, 1),
                           mergeValue = lambda x,y: (x[0] + y, x[1] + 1),
                           mergeCombiners = lambda x,y: (x[0] + y[0], x[1] + y[1]))

aggr.collect()

aggr.map(lambda x: (x[0], x[1][0]/x[1][1])).collect() # average value per key


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
How long does it take to collect primesRDD? Let's time the operation.
Can you write Python code for it?


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
If we do not need primesRDD in memory anymore, we can tell Spark to discard it.
Can you write Python code for it?


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
Lazy evaluation
RDDs are evaluated lazily. This means that Spark will not materialize an RDD until it has to perform an action. In the example below, primesRDD is not evaluated until action collect() is performed on it.
Can you write Python code for it?

def is_prime(num):
    """ return True if num is prime, False otherwise"""
    if num < 1 or num % 1 != 0:
        raise Exception("invalid argument")
    for d in range(2, int(np.sqrt(num) + 1)):
        if num % d == 0:
            return False
    return True

numbersRDD = sc.parallelize(range(1, 1000000)) # creation of RDD
primesRDD = numbersRDD.filter(is_prime) # transformation

# primesRDD has not been materialized until this point
primes = primesRDD.collect() # action

type(primesRDD)

print(primes[0:15])
print(primesRDD.take(15))


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
Persistence
RDDs are ephemeral by default, i.e. there is no guarantee they will remain in memory after they are materialized. If we want them to persist in memory, possibly to query them repeatedly or use them in multiple operations, we can ask Spark to do this by calling persist() on them.
Can you write Python code for it?

primesRDD.persist()

primesRDD.persist() # we're asking Spark to keep this RDD in memory. Note that cache is the same but as using persist for memory. However, persist allows you to define other types of storage

print("Found", primesRDD.count(), "prime numbers") # first action -- causes primesRDD to be materialized
print("Here are some of them:")

print(primesRDD.take(20)) # second action - RDD is already in memory


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
RDD operations
There are two types of RDD operations in Spark: transformations and actions. Transfromations create new RDDs from other RDDs. Actions extract information from RDDs and return it to the driver program.
Can you write Python code for it?

data = sc.parallelize([0,1,2,3,4,5,6,7,8,9]) # creation of RDD
data_squared = data.map(lambda num: num ** 2) # transformation
data_squared.collect() # action


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
Stopping
Call stop() on the SparkContext object to shut it down.
Can you write Python code for it?


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
Warning
In the example above, we update the value of an accumulator within a transformation (map). This is not recommended, unless for debugging purposes! The reason is that, if there are failures during the materialization of llengthRDD, some of its partitions will be re-computed, possibly causing the accumulator to double-count some the the long lines.
It is advisable to use accumulators within actions - and particularly with the foreach action, as demonstrated below.
Can you write Python code for it?

text = sc.textFile("myfile.txt")
long_lines_2 = sc.accumulator(0)

def line_len(line):
    global long_lines_2
    length = len(line)
    if length > 30:
        long_lines_2 += 1

text.foreach(line_len)

long_lines_2.value


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
When I ran the above on my laptop, it took about more than 10s. That's because Spark had to evaluate primesRDD before performing collect on it.
How long would it take if primesRDD was already in memory?
Can you write Python code for it?

primesRDD.persist()

%%timeit
primes = primesRDD.collect()


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
aggregate
Can you write Python code for it?

help(data.aggregate)

def seq(x,y):
    return x[0] + y, x[1] + 1

def comb(x,y):
    print(x,y,"comb")
    return x[0] + y[0], x[1] + y[1]

data = sc.parallelize([1,43,62,23,52], 1) # Try different levels of paralellism
aggr = data.aggregate(zeroValue = (0,0),
                      seqOp = seq, #
                      combOp = comb)

aggr

aggr[0] / aggr[1] # average value of RDD elements


I am working on the file "2017/spark_tutorial.ipynb".
The first task is:
local: Run Spark locally with one worker thread (i.e. no parallelism at all).
local[K]: Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine).
local[*]: Run Spark locally with as many worker threads as logical cores on your machine.
spark://HOST:PORT: Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default.
mesos://HOST:PORT: Connect to the given Mesos cluster. The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use mesos://zk://.... To submit with --deploy-mode cluster, the HOST:PORT should be configured to connect to the MesosClusterDispatcher.
yarn: Connect to a YARN cluster in client or cluster mode depending on the value of --deploy-mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable.
yarn-client: Equivalent to yarn with --deploy-mode client, which is preferred to yarn-client
yarn-cluster: Equivalent to yarn with --deploy-mode cluster, which is preferred to yarn-cluster
Can you write Python code for it?

# To try the SparkContext with other masters first stop the one that is already running
# sc.stop()


I am working on the file "20170106_DQ0TransformInPython.ipynb".
The first task is:
The DQ0 Transform will translate the three variables above into two variables, called the "Direct" phase (or "D") and the "Quadrature" phase (or "Q"). Hence, the resulting variables can be plotted on an X-Y graph for simple visualization. The following code implements a DQ0 Transform in Python, which takes all three phases of voltage as an input.
Can you write Python code for it?

def dq0_transform(v_a, v_b, v_c):
    d=(np.sqrt(2/3)*v_a-(1/(np.sqrt(6)))*v_b-(1/(np.sqrt(6)))*v_c)
    q=((1/(np.sqrt(2)))*v_b-(1/(np.sqrt(2)))*v_c)
    return d, q

# Calculate and plot the results
d, q = dq0_transform(v1, v2, v3)
plt.figure()
plt.plot(d, q)
plt.xlabel('D Phase')
plt.ylabel('Q Phase');


I am working on the file "20170119_Visualizing Dice Distributions.ipynb".
The first task is:
Notice, of course, that the normal histogram fit is a very poor choice for the 1dX algorithms, because they are a uniform distribution. In general, the more dice you roll, the closer the resulting distribution will resemble a Gaussian distribution.
We can summarize these results in tabular form:
Can you write Python code for it?

from tabulate import tabulate
print("Desc  Mean  Std")
print(tabulate(results))


I am working on the file "20170130-cohpy-fstrings.ipynb".
The first task is:
2017-02-25 afterthought:
Since it is saved in a variable,
the lambda above is a poor use of a lambda.
It would be better to just make it a regular (named) function
as shown below.
I still do not know what good uses (if any) there are for the f-string below.
Can you write Python code for it?

def template():
    return f'tes{k}'

k = 9
template()

k = 8
template()


I am working on the file "20170130-cohpy-fstrings.ipynb".
The first task is:
It reminds me of shell syntax. For example,
echo "The price of ${name} is ${price}"

It gets better. One may put arbitrary expressions
within the curly braces in an f-string.
One is not limited to using just variable names.
Can you write Python code for it?

tax = 0.50

for name, price in stuff.items():
    print(f'The total price of {name} is {round(price * (1+tax), 2)}')


I am working on the file "20170130-cohpy-repr-versus-str.ipynb".
The first task is:
I often use repr for debugging output to see details.
That is my main use of repr.

Extra stuff
The following arose during the presentation of the above.
Someone asked how repr handles bytestrings compared to str,
so we played with that. It turns out that they are identical.
Can you write Python code for it?

things = (
    b'a \t\nbyt\'"e string',
)

compare(str, repr, things)


I am working on the file "20170130-cohpy-repr-versus-str.ipynb".
The first task is:
Quoting strings with ''' or """
and how to encode ''' or """ in a string.
Can you write Python code for it?

things = (
    # string with three '
    "'''",
    """'''""",
    # string with three "
    '"""',
    '''"""''',
    # a string with three ' and three "
    """'''"""    '''"""''',
    """'''\"\"\"""",
    # multiline string
    '''sometext
    type some more text
    type something different''',
)

compare(str, repr, things)


I am working on the file "20170204_FuzzyLogicLinearGaussian.ipynb".
The first task is:
After normalization, notice that the membership functions no longer look Gaussian. Let's make sure that the membership does sum to 1 at each temperature:
Can you write Python code for it?

# Plot to show the sum is not always 1
x_sum = x_cold_rescale + x_mild_rescale + \
        x_warm_rescale + x_hot_rescale
plt.figure()
plt.plot(x, x_sum, 'y', linewidth=1.5, label='Total')
plt.title('Temperature, Rescaled Gaussian Fuzzy Sum')
plt.ylabel('Membership')
plt.xlabel('Temperature')
plt.legend(loc='center right', bbox_to_anchor=(1.25, 0.5),
          ncol=1, fancybox=True, shadow=True);


I am working on the file "20170204_FuzzyLogicLinearGaussian.ipynb".
The first task is:
As can be seen in the figure above, each state ("cold", "mild", "warm", and "hot") has a membership value defined at all temperatures between 30 and 100 degrees (F). This was accomplished easily by defining the following triangular functions for each state:

Cold - [30, 30, 50]
Mild - [30, 50, 70]
Warm - [50, 70, 100]
Hot - [70, 100, 100]

Where the first value in the array is the "low" (0%) value, the second is the "full" (100%) value, and the third is the "high" (0%) value. Below 30 degrees, we can assume that "cold" has 100% membership and all other judgements have 0% membership. Likewise, above 100 degrees, we can assume that "hot" has 100% membership and all other judgements have 0% membership.
Notice that "warm" has a wider distribution than the other states. This occurred because warm is an even more "fuzzy" word than the other words- people generally tend to disagree strongly about exactly what the word "warm" means. Fuzzy logic allows us to quantify this disagreement as a function with a higher standard deviation.
Let's examine a more complicated Gaussian distribution as a membership function. In this case, we will be specifying a mean and standard deviation for each Gaussian distribution.
Can you write Python code for it?

## GAUSSIAN
# Create the membership functions
x_cold_gauss = fuzz.gaussmf(x, 30, 8)
x_mild_gauss = fuzz.gaussmf(x, 50, 8)
x_warm_gauss = fuzz.gaussmf(x, 70, 12)
x_hot_gauss = fuzz.gaussmf(x, 100, 8)

# Plot the results of the gaussian fuzzy membership
plt.figure()
plt.plot(x, x_cold_gauss, 'b', linewidth=1.5, label='Cold')
plt.plot(x, x_mild_gauss, 'k', linewidth=1.5, label='Mild')
plt.plot(x, x_warm_gauss, 'm', linewidth=1.5, label='Warm')
plt.plot(x, x_hot_gauss, 'r', linewidth=1.5, label='Hot')
plt.title('Temperature, Gaussian Fuzzy')
plt.ylabel('Membership')
plt.xlabel('Temperature')
plt.legend(loc='center right', bbox_to_anchor=(1.25, 0.5),
          ncol=1, fancybox=True, shadow=True);


I am working on the file "20170204_FuzzyLogicLinearGaussian.ipynb".
The first task is:
Looking at the figure above, values tend to be more heavily concentrated around the "full" membership values, dropping off more steeply than the triangular membership function. This might be preferential behavior, depending on the system that's being modeled.
You might also notice that the membership functions no longer sum to 100%. Let's examine this further by plotting the membership sum at each temperature in the following code.
Can you write Python code for it?

# Plot to show the sum is not always 1
x_sum = x_cold_gauss + x_mild_gauss + \
        x_warm_gauss + x_hot_gauss
plt.figure()
plt.plot(x, x_sum, 'y', linewidth=1.5, label='Total')
plt.title('Temperature, Gaussian Fuzzy Sum')
plt.ylabel('Membership')
plt.xlabel('Temperature')
plt.legend(loc='center right', bbox_to_anchor=(1.25, 0.5),
          ncol=1, fancybox=True, shadow=True);


I am working on the file "20170208 Lesson 11/handwritten-digit-recognition-with-tflearn-exercise.ipynb".
The first task is:
BUT WAIT!
Isn't column zero [:,0] just the prediction for the digit ZERO?
Let's look at our 10000 testY vectors. Each has a one-hot encoding of the digits 0-9.
Can you write Python code for it?


I am working on the file "20170208 Lesson 11/handwritten-digit-recognition-with-tflearn-exercise.ipynb".
The first task is:
Building the network
TFLearn lets you build the network by defining the layers in that network.
For this example, you'll define:

The input layer, which tells the network the number of inputs it should expect for each piece of MNIST data.
Hidden layers, which recognize patterns in data and connect the input to the output layer, and
The output layer, which defines how the network learns and outputs a label for a given image.

Let's start with the input layer; to define the input layer, you'll define the type of data that the network expects. For example,
net = tflearn.input_data([None, 100])
would create a network with 100 inputs. The number of inputs to your network needs to match the size of your data. For this example, we're using 784 element long vectors to encode our input data, so we need 784 input units.
Adding layers
To add new hidden layers, you use
net = tflearn.fully_connected(net, n_units, activation='ReLU')
This adds a fully connected layer where every unit (or node) in the previous layer is connected to every unit in this layer. The first argument net is the network you created in the tflearn.input_data call, it designates the input to the hidden layer. You can set the number of units in the layer with n_units, and set the activation function with the activation keyword. You can keep adding layers to your network by repeated calling tflearn.fully_connected(net, n_units).
Then, to set how you train the network, use:
net = tflearn.regression(net, optimizer='sgd', learning_rate=0.1, loss='categorical_crossentropy')
Again, this is passing in the network you've been building. The keywords:

optimizer sets the training method, here stochastic gradient descent
learning_rate is the learning rate
loss determines how the network error is calculated. In this example, with categorical cross-entropy.

Finally, you put all this together to create the model with tflearn.DNN(net).
Exercise: Below in the build_model() function, you'll put together the network using TFLearn. You get to choose how many layers...
Can you write Python code for it?

# Define the neural network
def build_model():
    # This resets all parameters and variables, leave this here
    tf.reset_default_graph()

    #### Your code ####
    # Include the input layer, hidden layer(s), and set how you want to train the model
    net = tflearn.input_data([None, 784])
    net = tflearn.fully_connected(net, 128, activation='ReLU')
    net = tflearn.fully_connected(net, 32, activation='ReLU')
    net = tflearn.fully_connected(net, 10, activation='softmax')
    net = tflearn.regression(net, optimizer='sgd', learning_rate=0.1, loss='categorical_crossentropy')

    # This model assumes that your network is named "net"
    model = tflearn.DNN(net)
    return model

# Build the model
model = build_model()


I am working on the file "20170208 Lesson 11/handwritten-digit-recognition-with-tflearn-exercise.ipynb".
The first task is:
But what about the other columns besides zero?
It turns out those other columns (the other digits) have different levels of error.
Up to 1.15% error for the nines.
Can you write Python code for it?

for i in range(10):
    print(i, np.bincount((np.array(model.predict(testX))[:,i] >= 0.5).astype(np.int_) == testY[:,i]))


I am working on the file "20170208 Lesson 11/handwritten-digit-recognition-with-tflearn-exercise.ipynb".
The first task is:
Column zero has 9020 zeros and 980 ones. So about one tenth of our 10000 images was a zero.
Column one is different:
Can you write Python code for it?


I am working on the file "20170208 Lesson 11/handwritten-digit-recognition-with-tflearn-exercise.ipynb".
The first task is:
It has over 1000 ones. So there are more ones than zeros in the testY set.
Let's look at ALL the digits:
Can you write Python code for it?

for i in range(10):
    print(i, np.bincount(testY[:,i].astype(np.int_)))


I am working on the file "20170208 Lesson 11/handwritten-digit-recognition-with-tflearn-exercise.ipynb".
The first task is:
Retrieving training and test data
The MNIST data set already contains both training and test data. There are 55,000 data points of training data, and 10,000 points of test data.
Each MNIST data point has:
1. an image of a handwritten digit and
2. a corresponding label (a number 0-9 that identifies the image)
We'll call the images, which will be the input to our neural network, X and their corresponding labels Y.
We're going to want our labels as one-hot vectors, which are vectors that holds mostly 0's and one 1. It's easiest to see this in a example. As a one-hot vector, the number 0 is represented as [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], and 4 is represented as [0, 0, 0, 0, 1, 0, 0, 0, 0, 0].
Flattened data
For this example, we'll be using flattened data or a representation of MNIST images in one dimension rather than two. So, each handwritten number image, which is 28x28 pixels, will be represented as a one dimensional array of 784 pixel values.
Flattening the data throws away information about the 2D structure of the image, but it simplifies our data so that all of the training data can be contained in one array whose shape is [55000, 784]; the first dimension is the number of training images and the second dimension is the number of pixels in each image. This is the kind of data that is easy to analyze using a simple neural network.
Can you write Python code for it?

# Retrieve the training and test data
trainX, trainY, testX, testY = mnist.load_data(one_hot=True)


I am working on the file "20170208 Lesson 11/handwritten-digit-recognition-with-tflearn-exercise.ipynb".
The first task is:
Testing
After you're satisified with the training output and accuracy, you can then run the network on the test data set to measure it's performance! Remember, only do this after you've done the training and are satisfied with the results.
A good result will be higher than 98% accuracy! Some simple models have been known to get up to 99.7% accuracy.
Can you write Python code for it?

# Compare the labels that our model predicts with the actual labels
predictions = (np.array(model.predict(testX))[:,0] >= 0.5).astype(np.int_)

# Calculate the accuracy, which is the percentage of times the predicated labels matched the actual labels
test_accuracy = np.mean(predictions == testY[:,0], axis=0)

# Print out the result
print("Test accuracy: ", test_accuracy)


I am working on the file "20170208 Lesson 11/handwritten-digit-recognition-with-tflearn-exercise.ipynb".
The first task is:
Visualize the training data
Provided below is a function that will help you visualize the MNIST data. By passing in the index of a training example, the function show_digit will display that training image along with it's corresponding label in the title.
Can you write Python code for it?

# Visualizing the data
import matplotlib.pyplot as plt
%matplotlib inline

# Function for displaying a training image by it's index in the MNIST set
def show_digit(index):
    label = trainY[index].argmax(axis=0)
    # Reshape 784 array into 28x28 image
    image = trainX[index].reshape([28,28])
    plt.title('Training data, index: %d,  Label: %d' % (index, label))
    plt.imshow(image, cmap='gray_r')
    plt.show()

# Display the first (index 0) training image
show_digit(0)


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Build a song recommender with personalization
We now create an item similarity based collaborative filtering model that allows us to make personalized recommendations to each user.
Class for an item similarity based personalized recommender system (Can be used as a black box)
Can you write Python code for it?


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Count number of unique users in the dataset
Can you write Python code for it?

users = song_df['user_id'].unique()

len(users)


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Create an instance of item similarity based recommender class
Can you write Python code for it?

is_model = Recommenders.item_similarity_recommender_py()
is_model.create(train_data, 'user_id', 'song')


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Explore data
Music data shows how many times a user listened to a song, as well as the details of the song.
Can you write Python code for it?


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Generate Precision Recall curve using pickled results on a larger data subset(Python 2.7)
Can you write Python code for it?

print("Plotting precision recall curves for a larger subset of data (100,000 rows) (user sample = 0.005).")

pm_avg_precision_list = joblib.load('pm_avg_precision_list_2.pkl')
pm_avg_recall_list = joblib.load('pm_avg_recall_list_2.pkl')
ism_avg_precision_list = joblib.load('ism_avg_precision_list_2.pkl')
ism_avg_recall_list = joblib.load('ism_avg_recall_list_2.pkl')

print("Plotting precision recall curves.")
plot_precision_recall(pm_avg_precision_list, pm_avg_recall_list, "popularity_model",
                      ism_avg_precision_list, ism_avg_recall_list, "item_similarity_model")


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Generate Precision Recall curve using pickled results on a larger data subset(Python 3)
Can you write Python code for it?

print("Plotting precision recall curves for a larger subset of data (100,000 rows) (user sample = 0.005).")

#Read the persisted files
pm_avg_precision_list = joblib.load('pm_avg_precision_list_3.pkl')
pm_avg_recall_list = joblib.load('pm_avg_recall_list_3.pkl')
ism_avg_precision_list = joblib.load('ism_avg_precision_list_3.pkl')
ism_avg_recall_list = joblib.load('ism_avg_recall_list_3.pkl')

print("Plotting precision recall curves.")
plot_precision_recall(pm_avg_precision_list, pm_avg_recall_list, "popularity_model",
                      ism_avg_precision_list, ism_avg_recall_list, "item_similarity_model")


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Length of the dataset
Can you write Python code for it?


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Load music data
Can you write Python code for it?

#Read userid-songid-listen_count triplets
#This step might take time to download data from external sources
triplets_file = 'https://static.turi.com/datasets/millionsong/10000.txt'
songs_metadata_file = 'https://static.turi.com/datasets/millionsong/song_data.csv'

song_df_1 = pandas.read_table(triplets_file,header=None)
song_df_1.columns = ['user_id', 'song_id', 'listen_count']

#Read song  metadata
song_df_2 =  pandas.read_csv(songs_metadata_file)

#Merge the two dataframes above to create input dataframe for recommender systems
song_df = pandas.merge(song_df_1, song_df_2.drop_duplicates(['song_id']), on="song_id", how="left")


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Quantitative comparison between the models
We now formally compare the popularity and the personalized models using precision-recall curves.
Class to calculate precision and recall (This can be used as a black box)
Can you write Python code for it?


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Quiz 1. Count the number of unique songs in the dataset
Can you write Python code for it?

###Fill in the code here
songs = song_df['song'].unique()
len(songs)


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Quiz 2: Use the popularity based model to make predictions for the following user id (Note the difference in recommendations from the first user id).
Can you write Python code for it?

###Fill in the code here
user_id = users[8]
pm.recommend(user_id)


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Quiz 3. Use the personalized model to make recommendations for the following user id. (Note the difference in recommendations from the first user id.)
Can you write Python code for it?


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Quiz 4
Understanding Intuition behind SVD
Can you write Python code for it?

%matplotlib inline
from pylab import *

#Plot all the users
print("Matrix Dimensions for U")
print(U.shape)

for i in range(0, U.shape[0]):
    plot(U[i,0], U[i,1], marker = "*", label="user"+str(i))

for j in range(0, Vt.T.shape[0]):
    plot(Vt.T[j,0], Vt.T[j,1], marker = 'd', label="item"+str(j))

legend(loc="upper right")
title('User vectors in the Latent semantic space')
ylim([-0.7, 0.7])
xlim([-0.7, 0])
show()


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Quiz 4. Use the personalized recommender model to get similar songs for the following song.
Can you write Python code for it?

song = 'Yellow - Coldplay'
###Fill in the code here
is_model.get_similar_items([song])


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Showing the most popular songs in the dataset
Can you write Python code for it?

song_grouped = song_df.groupby(['song']).agg({'listen_count': 'count'}).reset_index()
grouped_sum = song_grouped['listen_count'].sum()
song_grouped['percentage']  = song_grouped['listen_count'].div(grouped_sum)*100
song_grouped.sort_values(['listen_count', 'song'], ascending = [0,1])


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Simple popularity-based recommender class (Can be used as a black box)
Can you write Python code for it?


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
The curve shows that the personalized model provides much better performance over the popularity model.
Matrix Factorization based Recommender System
Import the required libraries
Can you write Python code for it?

#Code source written with help from:
#http://antoinevastel.github.io/machine%20learning/python/2016/02/14/svd-recommender-system.html

import math as mt
import csv
from sparsesvd import sparsesvd #used for matrix factorization
import numpy as np
from scipy.sparse import csc_matrix #used for sparse matrix
from scipy.sparse.linalg import * #used for matrix multiplication

#Note: You may need to install the library sparsesvd. Documentation for
#sparsesvd method can be found here:
#https://pypi.python.org/pypi/sparsesvd/


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Use SVD to make predictions for a test user id, say 4
Can you write Python code for it?

#Used in SVD calculation (number of latent factors)
K=2

#Initialize a sample user rating matrix
urm = np.array([[3, 1, 2, 3],[4, 3, 4, 3],[3, 2, 1, 5], [1, 6, 5, 2], [5, 0,0 , 0]])
urm = csc_matrix(urm, dtype=np.float32)

#Compute SVD of the input user ratings matrix
U, S, Vt = computeSVD(urm, K)

#Test user set as user_id 4 with ratings [0, 0, 5, 0]
uTest = [4]
print("User id for whom recommendations are needed: %d" % uTest[0])

#Get estimated rating for test user
print("Predictied ratings:")
uTest_recommended_items = computeEstimatedRatings(urm, U, S, Vt, uTest, K, True)
print(uTest_recommended_items)


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
Use the above precision recall calculator class to calculate the evaluation measures
Can you write Python code for it?

start = time.time()

#Define what percentage of users to use for precision recall calculation
user_sample = 0.05

#Instantiate the precision_recall_calculator class
pr = Evaluation.precision_recall_calculator(test_data, train_data, pm, is_model)

#Call method to calculate precision and recall values
(pm_avg_precision_list, pm_avg_recall_list, ism_avg_precision_list, ism_avg_recall_list) = pr.calculate_measures(user_sample)

end = time.time()
print(end - start)


I am working on the file "20170209 Live Song Recommender/Song Recommender_Python.ipynb".
The first task is:
We can also apply the model to find similar songs to any song in the dataset
Can you write Python code for it?

is_model.get_similar_items(['U Smile - Justin Bieber'])


I am working on the file "20170217 min_max/intro_to_tensorflow_solution.ipynb".
The first task is:
Alternative (no zero ops) solution
Can you write Python code for it?

def normalize_grayscale_no_zero_ops(image_data):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
    a = 0.1
    b = 0.9
#     grayscale_min = 0
    grayscale_max = 255
#     return a + ( ( (image_data - grayscale_min)*(b - a) )/( grayscale_max - grayscale_min ) )
    return a + ( ( (image_data                  )*(b - a) )/( grayscale_max ) )

%timeit normalize_grayscale_no_zero_ops(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 255]))


I am working on the file "20170217 min_max/intro_to_tensorflow_solution.ipynb".
The first task is:
Alternative 2 (no a - b op) solution
Can you write Python code for it?

def normalize_grayscale_no_a_minus_b_op(image_data):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
#     a = 0.1
#     b = 0.9
#     b_minus_a = 0.8
#     grayscale_min = 0
#     grayscale_max = 255
#     return a + ( ( (image_data - grayscale_min)*(b - a) )/( grayscale_max - grayscale_min ) )
    return 0.1 + ( ( (image_data                  )*(0.8) )/( 255 ) )

%timeit normalize_grayscale_no_a_minus_b_op(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 255]))


I am working on the file "20170217 min_max/intro_to_tensorflow_solution.ipynb".
The first task is:
Problem 2

Use tf.placeholder() for features and labels since they are the inputs to the model.
Any math operations must have the same type on both sides of the operator.  The weights are float32, so the features and labels must also be float32.
Use tf.Variable() to allow weights and biases to be modified.
The weights must be the dimensions of features by labels.  The number of features is the size of the image, 28*28=784.  The size of labels is 10.
The biases must be the dimensions of the labels, which is 10.
Can you write Python code for it?

features_count = 784
labels_count = 10

# Problem 2 - Set the features and labels tensors
features = tf.placeholder(tf.float32)
labels = tf.placeholder(tf.float32)

# Problem 2 - Set the weights and biases tensors
weights = tf.Variable(tf.truncated_normal((features_count, labels_count)))
biases = tf.Variable(tf.zeros(labels_count))


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
Check Point
This is your first checkpoint.  If you ever decide to come back to this notebook or have to restart the notebook, you can start from here.  The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
Checkpoint
The model has been saved to disk.
Test Model
Test your model against the test dataset.  This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_training.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')

        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0

        for train_feature_batch, train_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: train_feature_batch, loaded_y: train_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y:...


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
Convolution and Max Pooling Layer
Convolution layers have a lot of success with images. For this code cell, you should implement the function conv2d_maxpool to apply convolution then max pooling:
* Create the weight and bias using conv_ksize, conv_num_outputs and the shape of x_tensor.
* Apply a convolution to x_tensor using weight and conv_strides.
 * We recommend you use same padding, but you're welcome to use any padding.
* Add bias
* Add a nonlinear activation to the convolution.
* Apply Max Pooling using pool_ksize and pool_strides.
 * We recommend you use same padding, but you're welcome to use any padding.
Note: You can't use TensorFlow Layers or TensorFlow Layers (contrib) for this layer.  You're free to use any TensorFlow package for all the other layers.
Can you write Python code for it?

def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    # TODO: Implement Function
    x = 0
    y = 1
    z = 3
    weight = tf.Variable(tf.truncated_normal([conv_ksize[x],
                                              conv_ksize[y],
                                              int(x_tensor.shape[z]),
                                              conv_num_outputs],
                                             stddev=0.05))

    bias = tf.Variable(tf.constant(0.05, shape=[conv_num_outputs]))

    x_tensor = tf.nn.conv2d(x_tensor,
                            weight,
                            strides=[1,
                                     conv_strides[x],
                                     conv_strides[y],
                                     1],
                            padding='SAME')

    x_tensor = tf.nn.bias_add(x_tensor, bias)

    x_tensor = tf.nn.relu(x_tensor)

    x_tensor = tf.nn.max_pool(x_tensor,
                              ksize=[1,
                                     pool_ksize[x],
                                     pool_ksize[y],
                                     1],
                              strides=[1,
                                       pool_strides[x],
                                       pool_strides[y],
                                       1],
                              padding='SAME')
    return x_tensor

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_con_pool(conv2d_maxpool)


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
Flatten Layer
Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor.  The output should be the shape (Batch Size, Flattened Image Size). You can use TensorFlow Layers or TensorFlow Layers (contrib) for this layer.
Can you write Python code for it?

def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    # TODO: Implement Function
    return tf.reshape(x_tensor, [-1, int(x_tensor.shape[1] * x_tensor.shape[2] * x_tensor.shape[3])])

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
Hyperparameters
Tune the following parameters:
* Set epochs to the number of iterations until the network stops learning or start overfitting
* Set batch_size to the highest number that your machine has memory for.  Most people set them to common sizes of memory:
 * 64
 * 128
 * 256
 * ...
* Set keep_probability to the probability of keeping a node using dropout
Can you write Python code for it?

# TODO: Tune Parameters
epochs = 40
batch_size = 128
keep_probability = 0.7


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
One-hot encode
Just like the previous code cell, you'll be implementing a function for preprocessing.  This time, you'll implement the one_hot_encode function. The input, x, are a list of labels.  Implement the function to return the list of labels as One-Hot encoded Numpy array.  The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode.  Make sure to save the map of encodings outside the function.
Hint: Don't reinvent the wheel.
Can you write Python code for it?

import numpy as np

def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.

    Adapted from http://stackoverflow.com/questions/29831489/numpy-1-hot-array

    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    # TODO: Implement Function
    output = np.zeros((len(x), max(x)+1))
    output[np.arange(len(x)), x] = 1
    return output

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)

one_hot_encode([0, 1, 8, 5, 1, 5, 7, 4, 9])


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
Randomize Data
As you saw from exploring the data above, the order of the samples are randomized.  It doesn't hurt to randomize it again, but you don't need to for this dataset.
Preprocess all the data and save it
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
Train on a Single CIFAR-10 Batch
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy.  Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)


I am working on the file "20170226 Image Classification/dlnd_image_classification.ipynb".
The first task is:
Train the Neural Network
Single Optimization
Implement the function train_neural_network to do a single optimization.  The optimization should use optimizer to optimize in session with a feed_dict of the following:
* x for image input
* y for labels
* keep_prob for keep probability for dropout
This function will be called for each batch, so tf.global_variables_initializer() has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
Can you write Python code for it?

def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    # TODO: Implement Function
    session.run(optimizer,
                feed_dict={x: feature_batch,
                           y: label_batch,
                           keep_prob: keep_probability})

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)


I am working on the file "20170304_AbaloneWithKerasPart1.ipynb".
The first task is:
Next, Keras requires that the model be "compiled", which essentially finalizes the model to be used for training. When compiling the model, you should choose an optimizer, which refers to how the model will select updated parameters after each training epoch. For example, a gradient descent optimizer would look at the "slope" of the error space and choose the direction which minimizes the error. Here, I've chosen 'sgd', or stochastic gradient descent.
The "loss" parameter tells the neural network how the results of each epoch should be scored. Mean squared error is a standard choice, especially for regression, since its use in Ordinary Least Squares.
To compare the results from this neural network to the one previously constructed, our output metric will be the mean absolute error.
Can you write Python code for it?

abalone_model.compile(optimizer='rmsprop',loss='mse',
                      metrics=['mean_absolute_error'])


I am working on the file "20170308_AbaloneWithKerasPart3.ipynb".
The first task is:
Notice that the total elapsed time was about 30 minutes. I'm running this code on a fairly old Linux computer. I have no doubt that it would run faster with better hardware. I would like to try out GPU acceleration, but I don't currently have a computer with an CUDA-enabled GPU, so that will have to wait.
Finally, the following code will present the results in tabular form:
Can you write Python code for it?

# Print the results matrix
results_df.sort_values('MAE')


I am working on the file "20170308_AbaloneWithKerasPart3.ipynb".
The first task is:
To automatically construct a list of models to test, I used a listcomp in the following code. Each of these models requires exactly two hidden layers. The idea is that I will test a range of values from 5 to 30 for each hidden layer, incrementing by 5 each time. I also extended the list with three additional models that I thought would be interesting to test as well.
Can you write Python code for it?

# Constructing a list of models to test
hlayers = [[x,y] for x in range(5,31,5) for y in range(5,31,5)]
hlayers.extend([[1,10],[10,1],[2,2]])


I am working on the file "20170310_ActivationFunctions.ipynb".
The first task is:
Notice that the ReLU is not differentiable at 0, which may pose some problems for gradient descent or similar optimization algorithms. Given that values <= 0 have an output of zero, this means that ReLU is effective at "turning off" many initial nodes.
Softplus
Finally, the softplus function looks like:
Can you write Python code for it?

x = np.arange(-5, 5, 0.01)
y = np.log(1+np.exp(x))
plt.plot(x,y)
plt.title('Softplus Activation Function')
plt.xlabel('Input')
plt.ylabel('Output');


I am working on the file "20170310_ActivationFunctions.ipynb".
The first task is:
The arctan, again, has a similar shape to the previous two activation functions, but the "steepness" of the curve is interpolated in between the two.
All three of these activation functions are good choices to consider for classification problems.
Rectified Linear Unit
Also known as ReLU, this activation function looks like:
Can you write Python code for it?

x = np.arange(-5, 5, 0.01)
z = np.zeros(len(x))
y = np.maximum(z,x)
plt.plot(x,y)
plt.title('ReLU Activation Function')
plt.xlabel('Input')
plt.ylabel('Output');


I am working on the file "20170312_AbaloneWithKerasDropout.ipynb".
The first task is:
Notice that the elapsed time was higher than before- dropout operations can be somewhat computationally expensive. The results are shown:
Can you write Python code for it?

# Print the results matrix
results_df.sort_values('MAE').head(15)


I am working on the file "20170327-cohpy-fstrings.ipynb".
The first task is:
But what does round(price * (1+tax_rate), 2) mean?
What is it for?
Just because you can put expressions in a f-string
does not mean that you should.
In the cell below, a temporary variable is used for documentation
to make the code easier to read.
Can you write Python code for it?

tax_rate = 0.50

for name, price in stuff.items():
    total_price = round(price * (1+tax_rate), 2)
    print(f'The total price of {name} is {total_price}.')


I am working on the file "20170327-cohpy-fstrings.ipynb".
The first task is:
Something that sucks about the above print,
is that name and price appear three times each.
I.e., It is not
DRY.
With f-strings, name and price only have to appear once,
making the code easier to read and maintain
as in the following example.
Can you write Python code for it?

for name, price in stuff.items():
    print(f'The price of {name} is {price}.')


I am working on the file "20170327-cohpy-named-groups.ipynb".
The first task is:
Mackenzie (first name)
Mackenzie (last name)

The Zen of Python, by Tim Peters
Readability counts.
Can you write Python code for it?


I am working on the file "20170327-cohpy-named-groups.ipynb".
The first task is:
Regular expressions can be used to indicate if a string matches a pattern or not.
Regular expressions can also be used to do some parsing.
The substrings of interest are called groups.
The traditional way of referring to a group is by index number.
Python has another way of referring to a group by name.
Using names give both the regular expression
and references to match groups more meaning.
They make Python code more readable.
Can you write Python code for it?

foo_pattern = re.compile('''
    ^
    ([A-Za-z]+)
    ,[ ]
    ([A-Za-z]+)
    $
''', re.VERBOSE)

s = 'James, Mackenzie'

m = re.match(foo_pattern, s)
m

m.groups

m.group(0)

m.group(1)

m.group(2)

foo_pattern = re.compile('''
    ^
    (?P<last_name>[A-Za-z]+)
    ,[ ]
    (?P<first_name>[A-Za-z]+)
    $
''', re.VERBOSE)

m = re.match(foo_pattern, s)
m

m.groups

m.group(0)

m.group(1)

m.group(2)

m.group('last_name')

m.group('first_name')


I am working on the file "20170402_ArcheryWithGeometryPixelsAndMonteCarlo.ipynb".
The first task is:
The difference in the result compared to Solution 1 comes from the fact that the image was not drawn perfectly, as you can tell if you zoom in far enough.
Situation 3: Monte Carlo
Let's complicate the situation even further. Imagine that the dart thrower is not a beginner but instead has a higher chance of throwing darts near the bullseye, although their aim is not perfect. There are several different ways we could model this situation, but let's assume that instead of treating the probability distribution across the target as uniform, we model the x-axis and y-axis as independent Gaussian probability distributions. Since the pdf of the Gaussian distribution will go to zero at both positive and negative infinity, we can also remove the previous situation's assumption that the dart must fall within the bounds of the black square.
In addition, we will consider the following assumptions about the dart thrower:

They are better at aiming left-to-right than up-to-down
They tend to throw a little bit up and to the right of the bullseye
They are lined up with the bullseye in the X direction about 20% of the time

The first item will affect the standard deviation of the Gaussian distributions- we want the standard deviation of the X axis distribution to be less than the standard deviation of the Y axis distribution.  The second item above requires us to adjust the mean of the Gaussian distributions in the positive direction, for both the x and y axis. Both of these items are rather vague, and we will need to choose some appropriate values. The third item is the most difficult, as it requires us to select appropriate standard deviations such that the probability of hitting the bullseye (in the X direction alone) is 20%.
We will also add the same constraints as in Situation 1: the radius of the target is 1, and the radius of the bullseye is 0.1.
"Better at Aiming Left-to-Right"
"Better at aiming" implies more consistency in the results of the dart throw when thrown left-to-right, which means decreasing the standard...
Can you write Python code for it?

from scipy.stats import norm
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Problem constraints
r_b = 0.1
r_t = 1
mean_x = 0.05 * r_t

# Iterate over many possible sigma_y
results = {}
for sigma_x in np.linspace(0.1, 1, 100):
    prob_hit = norm.cdf(r_b, loc=mean_x, scale=sigma_x) - \
               norm.cdf(-r_b, loc=mean_x, scale=sigma_x)
    results[sigma_x] = prob_hit

# Save the results to a DataFrame and plot them
df = pd.DataFrame.from_dict(results, orient="index")
f = df.plot();
f.set_ylabel("prob_hit")
f.set_xlabel("sigma_x");
plt.scatter(0.385, 0.2, c = 'r');

# Calculate sigma_y based on sigma_x
print('Sigma_y = {}'.format(round(0.385/0.75,3)))


I am working on the file "20170402_ArcheryWithGeometryPixelsAndMonteCarlo.ipynb".
The first task is:
To get the probabilities, we can divide each of these pixels by the total number of pixels (39,601, since the image is 199 x 199 pixels). As above, the expected value is given by:
e_d = (p_n)(0) + (p_t)(1) + (p_b)(2)
Can you write Python code for it?

total = blue + red + black
# The probabilities are the count over the total pixels
p_t = float(blue) / float(total)
p_b = float(red) / float(total)
# To get the expected value, we use the same formula as before:
# e_d = (p_n)(0) + (p_t)(1) + (p_b)(2)
e_d = p_t + 2*p_b
print("Expected score per dart throw:{}".format(round(e_d,3)))


I am working on the file "20170402_ArcheryWithGeometryPixelsAndMonteCarlo.ipynb".
The first task is:
We now have an array of pixels in the [Red, Green, Blue] space. The color codes for the image are:

Black = [0,0,0]
Blue = [0,162,232]
Red = [237,28,36]

Let's count up the number of pixels of each color:
Can you write Python code for it?

# Here we are using a trick: "162" is unique to the blue pixels,
# and "28" is unique to the red pixels, so we can search for it
# alone. Any of the pixels that are left from the total must be
# black, so we can count (length of the data - blue - red) pixels.
blue = len(data[np.where(data == 162)])
red = len(data[np.where(data == 28)])
black = len(im.getdata()) - blue - red
print("Black:{}, Blue:{}, Red:{}".format(black, blue, red))


I am working on the file "20170408_MonteCarloVarianceReductionWithAnUnbalancedRouletteWheel.ipynb".
The first task is:
After spending a significant amount of time spinning the wheel, you feel a little unsatisfied. Sure, you found the expected payout, but there's a nagging feeling that your answer might have been much more accurate if you just had more money to spend. In fact, you're not even completely sure that last two digits are correct (see the N^(-1/2) rule from the previous blog post).
Let's see what the answer would have been if you instead had $50 million to spend:
Can you write Python code for it?

num_hist2 = 1e8 # $50 million
results2 = MCHist(num_hist2, 0.05, 1, 0.3, 0.3, 0.15, 0.5, 0.5, 0.2)
payout2 = round(results2 / num_hist2, 3)
print('Expected payout per spin is ${}'.format(payout2))


I am working on the file "20170408_MonteCarloVarianceReductionWithAnUnbalancedRouletteWheel.ipynb".
The first task is:
Based on the simulation above, we got an answer that was closer to the "true" value of 0.315, even though we used the same number of spins (1000). (Since Monte Carlo simulation is stochastic, it's possible that if you re-run this notebook, you might get an answer that's farther away from the "true" value, but the weight correction method will more consistently give an answer that's closer to the true value)
However, the results aren't perfect. Can we improve the selection of P_New? It turns out that we can, if we select each P_New according to the following formula:
P_New = (P_Orig)(Score_Orig) / (TotalP_New)
Where TotalP_New is the sum of all (P_Orig)(Score_Orig) for all choices.
Here's the table for our roulette example:
| Area | P_Orig | Score_Orig | P_New              | Weight   | Score_New |
| -----|--------|------------|--------------------|----------|-----------|
| 1    | 0.05   | 1.0        | 0.050/0.315=0.159  | 0.315    | 0.315     |
| 2    | 0.30   | 0.3        | 0.090/0.315=0.286  | 1.049    | 0.315     |
| 3    | 0.15   | 0.5        | 0.075/0.315=0.238  | 0.630    | 0.315     |
| 4    | 0.50   | 0.2        | 0.100/0.315=0.317  | 1.577    | 0.315     |
What a strange coincidence. Our scores are now all equal to the same value, which just happens to be the analytical solution that we calculated earlier.
As you might expect, if we run the Monte Carlo simulation again, no matter which area is chosen, we get the same score, so the only possible result is 0.315:
Can you write Python code for it?

num_hist4 = 1e3 # $500
results4 = MCHist(num_hist4, 0.159, 0.315, 0.286, 0.315, 0.238, 0.315, 0.317, 0.315)
payout4 = round(results4 / num_hist4, 3)
print('Expected payout per spin is ${}'.format(payout4))


I am working on the file "20170408_MonteCarloVarianceReductionWithAnUnbalancedRouletteWheel.ipynb".
The first task is:
Much more satisfying. You're now fairly confident of the answer within about a hundredths of a cent. Unfortunately, you don't actually have $50 million to spend (and you certainly don't have the time for 100 million spins). Is there a way to increase the accuracy  (i.e. reduce the variance) of the simulation without spending more money (i.e. requiring more histories)?
Fortunately, there's a number of methods in Monte Carlo simulation called "variance reduction" methods which can be used for this purpose. In this notebook, we will be examining a method called "weight correction". Essentially, instead of choosing values from the original probability distribution, we will choose them from a new probability distribution that we specify.
This is similar to the idea of "class balancing" in classification. We can choose a new probability distribution in such a way as to increase the likelihood that the (originally) less likely choices are selected. This will effectively decrease the variance of the final result. Once we choose a new probability distribution, we must adjust the weight of each choice based on the following formula, so the results will be unbiased:
weight_correction = p(x) / p'(x)
where p(x) is the original probability distribution and p'(x) is the new probability distribution. This formula is often expressed using the mnemonic "shoulda over did". Be careful that every value of x is still possible in the new probability distribution.
Let's examine this in practice. What if we choose a probability distribution such that each of the areas in the roulette wheel are equally likely to be chosen? Assume that the roulette operator is nice and allows you to select the results of the roulette so this situation could actually occur in real life. After all, no matter what, the house still wins, right?
The operator must also tell you the probabilities for this method to work. Note that you can calculate the analytic expected value now:
(0.05)(1) + (0.3)(0.3) + (0.15)(0.5) + (0.5)(0.2) = 0.315
But you're still...
Can you write Python code for it?

num_hist3 = 1e3 # $500
results3 = MCHist(num_hist3, 0.25, 0.2, 0.25, 0.36, 0.25, 0.3, 0.25, 0.4)
payout3 = round(results3 / num_hist3, 5)
print('Expected payout per spin is ${}'.format(payout3))


I am working on the file "20170414_CalculatingAnIntegralWithMonteCarloUsingTheRejectionMethod.ipynb".
The first task is:
From the above figure, we can see that the maximum is about 15. But what if we decided not to plot the function? Consider a situation where it's computationally expensive to plot the function over the entire limits of the integral. It's okay for us to choose an f_max that is too large, as long as we are sure that all possible values of F(x) will fall below it. The only downside of this approach is that more histories are required to converge to the correct answer.
Let's choose f_max as 100 to see what happens:
Can you write Python code for it?

@jit
def MCHist2(n_hist, a, b, fmax):
    score = (b - a)*fmax
    tot_score = 0
    for n in range(1, n_hist):
        x = random.uniform(a, b)
        f = random.uniform(0, fmax)
        f_x = (np.exp(x)/x) + np.exp(1/x)
        # Check if the point falls inside the integral
        if f < f_x:
            tot_score += score
    return tot_score

# Run the simulation
num_hist2 = 1e8
results2 = MCHist2(num_hist2, 1.0, 4.0, 100)
integral_val2 = round(results2 / num_hist2, 6)
print("The calculated integral is {}".format(integral_val2))


I am working on the file "20170416_CentralLimitTheoremWithMonteCarlo.ipynb".
The first task is:
As can be seen in the figure above, the distribution of the means of the sampling distribution looks approximately normal. But the Weibull distribution may sometimes look fairly similar to a normal distribution anyway. What if we choose a uniform distribution instead?
Can you write Python code for it?

# User-defined parameters
a = 0 # Uniform distribution lower limit
b = 10 # Uniform distribution upper limit
num_samp2 = int(1e3) # First "for" loop
num_hist2 = int(1e5) # Second "for" loop

# Run the simulation
results2 = MCLoop(num_hist2, num_samp2, random.uniform, a, b)

# Plot the results
f = sns.distplot(results2)
f.set_xlabel('Mean')
f.set_title('Uniform Distribution');


I am working on the file "20170416_CentralLimitTheoremWithMonteCarlo.ipynb".
The first task is:
As seen above, the Central Limit Theorem works just as well with a uniform distribution as a Weibull distribution.
What about a discrete probability distribution? Let's use a probability distribution where we choose 1 20% of the time, 2 50% of the time, and 3 30% of the time. This will require us to modify our Monte Carlo function somewhat:
Can you write Python code for it?

# p_a is the probability that option a is chosen,
# s_a is the result if a is chosen, and so on
@numba.jit
def MCHist2(n_hist, p_a, s_a, p_b, s_b, p_c, s_c):
    res = 0
    for _ in range(0, n_hist):
        x = random.random()
        # The probabilities are cumulative
        if x <= p_a:
            res += s_a
        elif x <= (p_a + p_b):
            res += s_b
        elif x <= (p_a + p_b + p_c):
            res += s_c
    return res / float(n_hist)

@numba.jit
def MCLoop2(n_hist, n_samp):
    # For numba's sake, need to fill the list with the type
    results = [2.0]
    for _ in range(0, n_hist):
        results.append(MCHist2(n_samp, 0.2, 1, 0.5, 2, 0.3, 3))
    return results

# User-defined parameters
# We need a few more samples to get the shape to look right
num_samples3 = int(1e3) # First "for" loop
num_hist3 = int(1e7) # Second "for" loop

# Run the simulation
results3 = MCLoop2(num_hist3, num_samples3)

# Plot the results
f = sns.distplot(results3)
f.set_xlabel('Mean')
f.set_title('Discrete Distribution');


I am working on the file "20170416_CentralLimitTheoremWithMonteCarlo.ipynb".
The first task is:
This may be an even more surprising result. We chose a very simple distribution which was not even continuous, and we still get a normal distribution of the sample means!
One last example. Let's use an initial distribution which is a linear combination of all three distributions that we tried: a Weibull, a uniform, and a discrete. Will the Central Limit Theorem still hold in this case?
Can you write Python code for it?

@numba.jit
def MCHist3(n_hist, alpha, beta, a, b, p_a, s_a, p_b, s_b, p_c, s_c):
    res = 0
    for _ in range(0, n_hist):
        val = 0
        # Weibull part
        val += random.weibullvariate(alpha, beta)
        # Uniform part
        val += random.uniform(a, b)
        # Discrete part
        x = random.random()
        if x <= p_a:
            val += s_a
        elif x <= (p_a + p_b):
            val += s_b
        elif x <= (p_a + p_b + p_c):
            val += s_c
        # Add the sum of all three distributions to the total results
        res += val
    return res / float(n_hist)

@numba.jit
def MCLoop3(n_hist, n_samp):
    # For numba's sake, need to fill the list with the type
    results = [6.0]
    for _ in range(0, n_hist):
        results.append(MCHist3(n_samp, 2, 1.4, 0, 4, 0.2, 1, 0.5, 2, 0.3, 3))
    return results

# User-defined parameters
num_samples4 = int(1e3) # First "for" loop
num_hist4 = int(1e5) # Second "for" loop

# Run the simulation
results4 = MCLoop3(num_hist4, num_samples4)

# Plot the results
f = sns.distplot(results4)
f.set_xlabel('Mean')
f.set_title('Mixed Distribution');


I am working on the file "20170419_BootstrappingTheCentralLimitTheorem.ipynb".
The first task is:
Notice that the histogram of the bootstrapped data looks similar to the original data, but it's certainly not perfect. The original dataset had slightly over 10000 data points, but we only took 100 samples. Let's try the same process as above, but with 10000 samples:
Can you write Python code for it?

import numpy as np
# random.choice takes a data array, and you provide it with
# the size of samples that you want, with replacement or
# without replacement
boot2 = np.random.choice(df[0].values, 10000, replace=True)
# Let's look at the histogram of the data again, to compare
# it to the first figure.
boot_df2 = pd.DataFrame(boot2)
f, (ax1, ax2) = plt.subplots(1, 2, sharey=True)
plt.suptitle('Sampling with Replacement, 10000 Samples')
ax1.hist(boot_df2[0].values, normed=1)
ax1.set_xlabel('Bootstrapped Data')
ax2.hist(df[0].values, normed=1)
ax2.set_xlabel('Original Data');


I am working on the file "20170424-cohpy-lbyl-v-eafp.ipynb".
The first task is:
Another technique is to just try stuff,
and if it blows up, do something else.
This technique is called Easier to Ask Forgiveness than Permission (EAFP).
Python makes it very easy to do something else
when something blows up.
Can you write Python code for it?

def foo(x):
    try:
        y = 10 // x
    except ZeroDivisionError:
        y = 0
    return y

for x in numbers:
    y = foo(x)
    print(f'foo({x}) --> {y}')


I am working on the file "20170424-cohpy-lbyl-v-eafp.ipynb".
The first task is:
For that simple example,
EAFP does not have much if any benefit over LBYL.
For that simple example,
there is not much benefit in the size or readability of the code.
However, for more complicated problems,
EAFP lets one write much simpler and readable code.
We will use the example of determining if a string is a valid float for Python.
See
2.4.6. Floating point literals
for what constitutes a valid float.
floatnumber   ::=  pointfloat | exponentfloat
pointfloat    ::=  [digitpart] fraction | digitpart "."
exponentfloat ::=  (digitpart | pointfloat) exponent
digitpart     ::=  digit (["_"] digit)*
fraction      ::=  "." digitpart
exponent      ::=  ("e" | "E") ["+" | "-"] digitpart

Some code for that follows.
Can you write Python code for it?

import re

def is_float(s, debug=False):
    digit = f'([0-9])'
    digitpart = f'({digit}(_?{digit})*)'                           # digit (["_"] digit)*
    fraction = f'([.]{digitpart})'                                 # "." digitpart
    pointfloat = f'(({digitpart}?{fraction}) | ({digitpart}[.]))'  # [digitpart] fraction | digitpart "."
    exponent = f'([eE][-+]?{digitpart})'                           # ("e" | "E") ["+" | "-"] digitpart
    exponentfloat = f'(({digitpart} | {pointfloat}) {exponent})'   # (digitpart | pointfloat) exponent
    floatnumber = f'^({pointfloat} | {exponentfloat})$'            # pointfloat | exponentfloat
    floatnumber = f'^[-+]?({pointfloat} | {exponentfloat} | {digitpart})$'  # allow signs and ints

    if debug:
        regular_expressions = (
            digit,
            digitpart,
            fraction,
            pointfloat,
            exponent,
            exponentfloat,
            floatnumber,
        )
        for s in regular_expressions:
            print(repr(s))
            # print(str(s))

    float_pattern = re.compile(floatnumber, re.VERBOSE)

    return re.match(float_pattern, s)

floats = '''
    2
    0
    -1
    +17.
    .
    -.17
    17e-3
    -19.e-3
    hello
'''.split()
floats

is_float('', debug=True)

for s in floats:
    print(f'{s!r} -> {bool(is_float(s))}')

import re

def is_float(s):
    digit = f'([0-9])'
    digitpart = f'({digit}(_?{digit})*)'                           # digit (["_"] digit)*
    fraction = f'([.]{digitpart})'                                 # "." digitpart
    pointfloat = f'(({digitpart}?{fraction}) | ({digitpart}[.]))'  # [digitpart] fraction | digitpart "."
    exponent = f'([eE][-+]?{digitpart})'                           # ("e" | "E") ["+" | "-"] digitpart
    exponentfloat = f'(({digitpart} | {pointfloat}) {exponent})'   # (digitpart | pointfloat) exponent
    floatnumber = f'^({pointfloat} | {exponentfloat})$'            # pointfloat | exponentfloat
    floatnumber = f'^[-+]?({pointfloat} |...


I am working on the file "20170424-cohpy-lbyl-v-eafp.ipynb".
The first task is:
Now we try EAFP technique below.
Can you write Python code for it?

def safe_float(s, default=0.):
    try:
        x = float(s)
    except ValueError:
        x = default
    return x

def main(lines):
    total = sum(safe_float(line) for line in lines)
    print(f'total is {total}')

main(floats)


I am working on the file "20170424-cohpy-lbyl-v-eafp.ipynb".
The first task is:
So one checks before dividing as shown below.
Checking before doing something is called "Look Before You Leap" (LBYL).
Can you write Python code for it?

def foo(x):
    if x == 0:
        y = 0
    else:
        y = 10 // x
    return y

for x in numbers:
    y = foo(x)
    print(f'foo({x}) --> {y}')


I am working on the file "20170427_BinaryMarkovChain.ipynb".
The first task is:
Looking at the above figure, several things are noticeable. First, the only possible values are 0 or 1; this is to be expected, of course, since we are working with a "binary" Markov Chain. Second, the simulation appears to spend a relatively long time in each state before transitioning to another state (i.e. the lines are very flat). This is because our probability of staying in the same state is 90%. What would you expect if this probability was modified to 50%? Let's find out.
Can you write Python code for it?

tpm2 = [[0.5, 0.5],[0.5, 0.5]]
# Run the Markov Chain simulation for num_hist iterations
num_hist2 = range(0,100)
# Run and plot the results
res2 = MarkovChainSim(tpm2, num_hist2)
plt.plot(res2)
plt.title('Transition Probability=' + str(tpm2))
plt.xlabel('Iteration')
plt.ylabel('State');


I am working on the file "20170502_MarkovChainsInEquipmentConditionMonitoring.ipynb".
The first task is:
As can be seen, the above code approximates the original transition probability matrix fairly well. (Compare the above matrix to the first figure in this blog post)
Next, let's estimate the accrued degradation per state using a simple multiple linear regression. The following code will sum the number of times that a state occurs over an entire history ("x" in y=mx), along with the degradation at failure (since we are assuming a hard degradation threshold, degradation is always equal to 100% at failure). There will be no intercept term, since we know that degradation always begins at 0 (although this requires another assumption, as well as a full data history per equipment).
Can you write Python code for it?

from sklearn.linear_model import LinearRegression as LR
# "X"- Keep track of the number of times a state occurs
# over an entire time history
x = np.zeros((num_failures,3))
for ii, hist in enumerate(res_array):
    # Bincount will sum the number of times that a
    # state occurs in the history
    x[ii, :] = np.bincount(hist)
# "Y" is always 100% at failure
y = 100*np.ones((100,3))
# Now, perform linear regression on the above data
lr_model = LR(fit_intercept=False)
lr_model.fit(x, y)
print(lr_model.coef_[0])


I am working on the file "20170502_MarkovChainsInEquipmentConditionMonitoring.ipynb".
The first task is:
Based on the figure above, the equipment fails more quickly (~150 iterations) and also spends more time in the "low load" and "high load" states, since we increased the probabilities of staying/transitioning to those two states.
Assuming that you didn't already look at the damage accrued per state in the code above, we can actually use the data itself to estimate it. Let's imagine that we have 100 examples of equipment failures (in real life, this is a very big assumption):
Can you write Python code for it?

num_failures = 100
# Run the MCDegradeSim function for num_failures
res_array = []
for ii in range(num_failures):
    res = MCDegradeSim(tpm, dps, deg_thresh)
    res_array.append(res)


I am working on the file "20170502_MarkovChainsInEquipmentConditionMonitoring.ipynb".
The first task is:
Note that the probability of transitioning to a normal state is higher than for the other states, so the simulation spends much more time in the "normal" state. Think of state "2" and state "0" as stressful operating conditions which decrease the expected lifetime of the equipment. Compare the number of iterations above (~500) to the following example, where an equipment spends more time in a stressful condition.
[Image in blog post]
Let's modify the transition probability matrix accordingly:
Can you write Python code for it?

# The new transition probability matrix
tpm2 = [[0.9, 0.09, 0.01],[0.05, 0.90, 0.05],[0.01, 0.1, 0.89]]
res2 = MCDegradeSim(tpm2, dps, deg_thresh)
plt.plot(res2)
plt.title('Transition Probability=' + str(tpm2))
plt.xlabel('Iteration')
plt.ylabel('State');