I am working on the file "Shape_comparison.ipynb".
The first task is:
We'd like to compare the shape of heroin with other molecules.
ODDT supports three methods of molecular shape comparison: USR, USRCAT and Electroshape.

USR looks only at the shape of molecule.

USR-CAT considers the shape and type of atoms.

Electroshape accounts for the shape and charge of atoms.

All those methods have the same API.

We will use USR, because it's the simplest and the fastest.
Can you write Python code for it?


I am working on the file "Shapes with Vpython.ipynb".
The first task is:
Even though the top code cell contains no instructions to draw, Vpython's way of integrating into Jupyter Notebook seems to be by adding a scene right after the first code cell.  Look below for the code that made all of the above happen.  Yes, that's a bit strange.
Can you write Python code for it?


I am working on the file "Shared Layers in Keras.ipynb".
The first task is:
To share a layer across different inputs, simply instantiate the layer once, then call it on as many inputs as you want:
Can you write Python code for it?

# This layer can take as input a matrix
# and will return a vector of size 64
shared_lstm = LSTM(64)

# When we reuse the same layer instance
# multiple times, the weights of the layer
# are also being reused
# (it is effectively *the same* layer)
encoded_a = shared_lstm(tweet_a)
encoded_b = shared_lstm(tweet_b)

# We can then concatenate the two vectors:
merged_vector = keras.layers.concatenate([encoded_a, encoded_b], axis=-1)

# And add a logistic regression on top
predictions = Dense(1, activation='sigmoid')(merged_vector)

# We define a trainable model linking the
# tweet inputs to the predictions
model = Model(inputs=[tweet_a, tweet_b], outputs=predictions)

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.fit([data_a, data_b], labels, epochs=10)


I am working on the file "Sheet.ipynb".
The first task is:
Ejemplo de uso de los 3 procedimientos
Variables

Rutas de archivos y parametros opcionales de training(), clasification() e incorporation()
Can you write Python code for it?

hambox = './ham.mbox'
spambox = './spam.mbox'
newsbox = './news.mbox'
new_hambox  = './new-ham.mbox'
new_spambox = './new-spam.mbox'

# Criterio de separación alternativo
# delimiter = '[.@_#/]?[\s]+|[,;:()<>*~][\s]*'

# Número de palabras alternativo para hacer Näive-Bayes
# n = 19


I am working on the file "Sheet.ipynb".
The first task is:
Procedimiento
Can you write Python code for it?

# Función que recibe un mbox de correos nuevos
# Devuelve una lista de 0 o 1 por cada correo nuevo si es ham o spam
def clasification(newsbox, spam_dict, s, ham_dict, h, delimiter = '\W+', n = 15, debug = False):

    clas = []

    for words in words_per_mail(newsbox, delimiter):

        bests = best_N_words(words, spam_dict, ham_dict, n)

        probability  = naive_bayes(bests, spam_dict, s, ham_dict, h, debug = debug)

        is_spam      = 1 if probability > 0.9 else 0

        clas.append(is_spam)

    return clas


# Escribe los correos del mbox ubicado en la ruta 'new_mbox' en el mbox ubicado en la ruta 'mbox'
# Actualiza los dictionarios y tamaño de este mbox
# Tras su uso, no requiere volver a ejecutar training()
def incorporation(new_mbox, mbox, dictionary, size, delimiter = '\W+'):
    append_new_to_mbox(new_mbox, mbox)
    return update_dict_and_size(new_mbox, dictionary, size, delimiter)


# Devuelve dict de la forma {'word' : Sw }, el número de correos Spam,
# otro dict de la forma {'word' : Hw } y el número de correos Ham
def training(spambox, hambox, delimiter = '\W+'):

    spam_dict, s = training_dict(spambox, delimiter)
    ham_dict, h = training_dict(hambox, delimiter)

    return spam_dict, s, ham_dict, h


I am working on the file "Sheet.ipynb".
The first task is:
Uso de clasification()

Parámetro delimiter es opcional, por defecto es '\W+', cualquier secuencia de caracteres no alfanúmericos.
Parámetro n es opcional, por defecto es 15.
Parámetro debug es opcional, por defecto False. Si es True imprime información sobre el cálculo de naive bayes.
Requiere que training() haya sido ejecutado antes.
Can you write Python code for it?

# Clasificación
res = clasification(newsbox, spam_dict, s, ham_dict, h, debug = False)

print(res)


I am working on the file "Sheet.ipynb".
The first task is:
Uso de incorporation()

Parámetro delimiter es opcional, por defecto es '\W+', cualquier secuencia de caracteres no alfanúmericos.
Escribe los nuevos correos en los archivos spambox y hambox
Actualiza los dictionarios y tamaños de ambos mbox
Can you write Python code for it?

# Cantidad actual de correos spam y ham para el entrenamiento
print('[Antes de incorporar]')
print('S:', s, '— H:', h)
print('#Sw:', len(spam_dict), '— #Hw:', len(ham_dict))

# Incorporación a spam.box
spam_dict, s = incorporation(new_spambox, spambox, spam_dict, s)

# Incorporación a ham.box
ham_dict, h = incorporation(new_hambox, hambox, ham_dict, h)

# Cantidad nueva de correos spam y ham para el entrenamiento
print('\n[Después de incorporar]')
print('S:', s, '— H:', h)
print('#Sw:', len(spam_dict), '— #Hw:', len(ham_dict))


I am working on the file "Sheet.ipynb".
The first task is:
Uso de training()

Parámetro delimiter es opcional, por defecto es '\W+', cualquier secuencia de caracteres no alfanúmericos.
Can you write Python code for it?

# Entrenamiento
spam_dict, s, ham_dict, h = training(spambox, hambox)

# Cantidad actual de correos spam y ham para el entrenamiento
print('[Tras el entrenamiento]')
print('S:', s, '— H:', h)
print('#Sw:', len(spam_dict), '— #Hw:', len(ham_dict))


I am working on the file "Shooting victims by block.ipynb".
The first task is:
Count victims by block
Can you write Python code for it?

import pandas as pd

# Load shooting victims into a dataframe,
# filtering out victim records for which we couldn't determine the block
shooting_victims_this_year_df = pd.DataFrame([sv for sv in shooting_victims_this_year_with_block if sv['block'] is not None])

# Group by block
shooting_victims_this_year_by_block = shooting_victims_this_year_df.groupby('block').size().sort_values(ascending=False)
shooting_victims_this_year_by_block

# Output to a CSV file so I can email to the reporter who requested it
shooting_victims_this_year_by_block.to_csv("shooting_victims_by_block.csv")


I am working on the file "Shooting victims by block.ipynb".
The first task is:
Filter to only shootings this year
Can you write Python code for it?

from datetime import date, datetime

def get_shooting_date(shooting_victim):
    return datetime.strptime(shooting_victim['Date'], '%Y-%m-%d')

def shooting_is_this_year(shooting_victim, today):
    try:
        shooting_date = get_shooting_date(shooting_victim)
    except ValueError:
        if shooting_victim['RD Number']:
            msg = "Could not parse date for shooting victim with RD Number {}".format(
                shooting_victim['RD Number'])
        else:
            msg = "Could not parse date for shooting victim with record ID {}".format(
                shooting_victim['_id'])

        print(msg)
        return False

    return shooting_date.year == today.year

today = date.today()

# Use a list comprehension to filter the shooting victims to ones that
# occured on or before today's month and day.
# Also sort by date because it makes it easier to group by year
shooting_victims_this_year = sorted([sv for sv in shooting_victims
                                       if shooting_is_this_year(sv, today)],
                                      key=get_shooting_date)


I am working on the file "Shooting victims by block.ipynb".
The first task is:
Get the block address
Can you write Python code for it?

import re

def blockify(address):
    """
    Convert a street address to a block level address

    Example:

    >>> blockify("1440 W 84th St, Chicago, IL 60620")
    '1400 W 84th St, Chicago, IL 60620'

    """
    m = re.search(r'^(?P<address_number>\d+) ', address)
    address_number = m.group('address_number')
    block_address_number = (int(address_number) // 100) * 100
    return address.replace(address_number, str(block_address_number))

def add_block(sv):
    """Make a copy of a shooting victim record with an added block field"""
    with_block = dict(**sv)

    if not sv['Shooting Location']:
        # No location, just set block to none
        print("Record with RD number {0} has no location.".format(
            sv['RD Number']))
        with_block['block'] = None
        return with_block

    if sv['Shooting Specificity'] == 'Exact':
        # Address is exact, convert to 100-block
        with_block['block'] = blockify(sv['Shooting Location'])
    else:
        # Address is already block. Use it
        with_block['block'] = sv['Shooting Location']

    return with_block

# Create a list of shooting victim dictionaries with blocks
shooting_victims_this_year_with_block = [add_block(sv) for sv in shooting_victims_this_year]


I am working on the file "Shooting victims to date.ipynb".
The first task is:
Count the victims by year
Can you write Python code for it?

data['shooting_victims_ytd_by_year_totals'] = [(year, len(shooting_victims))
                                               for year, shooting_victims
                                               in data['shooting_victims_ytd_by_year']]

import csv
import sys

writer = csv.writer(sys.stdout)
writer.writerow(['year', 'num_shooting_victims'])

for year, num_shooting_victims in data['shooting_victims_ytd_by_year_totals']:
    writer.writerow([year, num_shooting_victims])


I am working on the file "Shooting victims to date.ipynb".
The first task is:
Filter to only year-to-date shooting victims
Can you write Python code for it?

from datetime import date, datetime

def get_shooting_date(shooting_victim):
    return datetime.strptime(shooting_victim['Date'], '%Y-%m-%d')

def shooting_is_ytd(shooting_victim, today):
    try:
        shooting_date = get_shooting_date(shooting_victim)
    except ValueError:
        if shooting_victim['RD Number']:
            msg = "Could not parse date for shooting victim with RD Number {}".format(
                shooting_victim['RD Number'])
        else:
            msg = "Could not parse date for shooting victim with record ID {}".format(
                shooting_victim['_id'])

        print(msg)
        return False

    return (shooting_date.month <= today.month and
            shooting_date.day <= today.day)

today = date(2016, 3, 30)
#today = date.today()

# Use a list comprehension to filter the shooting victims to ones that
# occured on or before today's month and day.
# Also sort by date because it makes it easier to group by year
data['shooting_victims_ytd'] = sorted([sv for sv in data['shooting_victims']
                                       if shooting_is_ytd(sv, today)],
                                      key=get_shooting_date)


I am working on the file "Shooting vs Main Crime Code and Shooting vs Reporting District.ipynb".
The first task is:
## Transforming the Original Data Using a Mapped Dictionary
Can you write Python code for it?

#creating a remapped shooting variable
df['Shoot_Status']=df['Shooting'].map({'No':0,'Yes':1}).astype(int)

df[:5]

main_crimecode_dummies=pd.get_dummies(df['MAIN_CRIMECODE'], prefix='class').iloc[:, 1:]
list(main_crimecode_dummies.columns.values)

reporting_district_dummies=pd.get_dummies(df['REPTDISTRICT'], prefix='class').iloc[:, 1:]
list(reporting_district_dummies.columns.values)

data=df.join([main_crimecode_dummies,reporting_district_dummies])
data[:5]


I am working on the file "Shooting vs Main Crime Code and Shooting vs Reporting District.ipynb".
The first task is:
Linear Regression through Ordinary Least Squares based on Main Crime Code
Can you write Python code for it?

model = ols(data=data, formula='Shoot_Status~class_01xx +class_03xx +class_04xx +class_05CB +class_05RB +class_06MV +class_06xx +class_07RV+class_07xx +class_08xx +class_09xx +class_10xx +class_11xx +class_12xx +class_13xx + class_14xx +class_15xx +class_16xx +class_18xx +class_20xx +class_21xx +class_22xx +class_24xx +class_32GUN +class_Argue +class_Arrest +class_BENoProp +class_Ballist +class_Bomb + class_BurgTools +class_Explos +class_FIRE +class_Gather +class_Harass +class_Harbor +class_Hazardous +class_InvPer +class_InvProp +class_InvVeh +class_LICViol +class_Labor +class_Landlord +class_MVAcc+class_Manslaug +class_MedAssist +class_OTHER +class_PRISON +class_PersLoc +class_PersMiss +class_PhoneCalls +class_Plates +class_PropDam +class_PropFound +class_PropLost +class_PubDrink +class_Restrain +class_Runaway +class_SearchWarr +class_Service +class_SexReg +class_SkipFare +class_TOWED +class_TRESPASS +class_VAL')
result = model.fit()
result.summary()


I am working on the file "Shooting vs Main Crime Code and Shooting vs Reporting District.ipynb".
The first task is:
Residual Testing
Can you write Python code for it?


I am working on the file "Shooting vs Main Crime Code and Shooting vs Reporting District.ipynb".
The first task is:
So only in District Dorchester, Mattapan, Roxbury, and HTU have a statistical significance as far as shooting is concerned
Can you write Python code for it?

model = ols(data=data, formula='Shoot_Status~class_Dorchester+class_Mattapan+class_Roxbury')
result = model.fit()
result.summary()


I am working on the file "Shootings and homicides by community area.ipynb".
The first task is:
And count the victims by community area, year and month
Can you write Python code for it?

data['shooting_victims_by_ca'] = pd.DataFrame(data['shooting_victims_df__with_ca'].groupby(['community', 'year', 'month']).size())


I am working on the file "Shootings and homicides by community area.ipynb".
The first task is:
Build a spatial index of community areas
Can you write Python code for it?

from rtree import index

communty_area_index = index.Index()
for ca_number, ca_shape in community_area_shapes.items():
    communty_area_index.add(ca_number, ca_shape.bounds, obj=community_area_properties[ca_number])


I am working on the file "Shootings and homicides by community area.ipynb".
The first task is:
How did March Humboldt Park shootings look over time?
Can you write Python code for it?

df = data['shooting_victims_by_ca']
df[(df.index.get_level_values('community') == "HUMBOLDT PARK") & (df.index.get_level_values('month') == 3)]


I am working on the file "Shootings and homicides by community area.ipynb".
The first task is:
Join the community area to the shooting victims data
Can you write Python code for it?

data['shooting_victims_df__with_ca'] = data['shooting_victims_df'].merge(
    data['shooting_victim_community_areas'],
    how='left',
    on='_id')


I am working on the file "Shootings and homicides by community area.ipynb".
The first task is:
Let's spot check our index, just because the coordinate format, (left, bottom, right, top) is a little confusing to me.
Can you write Python code for it?

from shapely.geometry import Point

def point_to_bounds(point):
    """
    Convert a point to a bounding box

    It makes sense to represent points as an x,y pair, but RTree only operates
    on bounding boxes. Convert the point to a bounding box where left == right
    and top == bottom.

    """
    return (point[0], point[1], point[0], point[1])

def get_community_area(point, ca_idx, ca_shapes):
    areas = []
    for n in ca_idx.intersection(point_to_bounds(point), objects=True):
        ca_number = int(n.object['area_num_1'])
        ca_shape = ca_shapes[ca_number]
        if ca_shape.contains(Point(*point)):
            areas.append(n.object)
    return areas

# Turkey Chop is a restaurant that is most definitely in Humboldt Park
# Let's use it to spot-check our index
turkey_chop_coords = [-87.7141142377237, 41.8955710581678]

turkey_chop_ca = get_community_area(turkey_chop_coords, communty_area_index, community_area_shapes)
assert turkey_chop_ca[0]['community'] == "HUMBOLDT PARK"


I am working on the file "Shootings and homicides by community area.ipynb".
The first task is:
Now, let's get some data from NewsroomDB, the Tribune's internal database of homicides and shootings
Can you write Python code for it?

import os
import requests

# Some constants
NEWSROOMDB_URL = os.environ['NEWSROOMDB_URL']

# A big object to hold all our data between steps
data = {}

def get_table_url(table_name, base_url=NEWSROOMDB_URL):
    return '{}table/json/{}'.format(base_url, table_name)

def get_table_data(table_name):
    url = get_table_url(table_name)

    try:
        r = requests.get(url)
        return r.json()
    except:
        print("Request failed. Probably because the response is huge.  We should fix this.")
        return get_table_data(table_name)

data['shooting_victims'] = get_table_data('shootings')
print("Loaded {} shooting victims".format(len(data['shooting_victims'])))

data['homicides'] = get_table_data('homicides')
print("Loaded {} homicides".format(len(data['homicides'])))


I am working on the file "Shootings and homicides by community area.ipynb".
The first task is:
We'll start with shootings.  Assign each shooting to a community area using the index we built earlier.
Can you write Python code for it?

import pprint
import re

def parse_coordinates(coordinate_str):
    """Convert a lat, lng string to a pair of lng, lat floats"""
    lat, lng = [float(c) for c in re.sub(r'[\(\) ]', '', coordinate_str).split(',')]
    return lng, lat

shooting_victim_community_areas = {}

for victim in data['shooting_victims']:
    try:
        coords = parse_coordinates(victim['Geocode Override'])
    except ValueError:
        shooting_victim_community_areas[victim['_id']] = '__invalid__'
        continue

    ca = get_community_area(coords, communty_area_index, community_area_shapes)

    if len(ca) == 0:
        shooting_victim_community_areas[victim['_id']] = '__invalid__'
        print("No community area found for record with coordinates {}".format(coords))
    elif len(ca) > 1:
        raise ValueError("Multiple community areas found for record with coordinates {}".format(coords))
    else:
        shooting_victim_community_areas[victim['_id']] = ca[0]['community']

data['shooting_victim_community_areas'] = pd.DataFrame([{'_id': k, 'community': v} for k, v in shooting_victim_community_areas.items()])


I am working on the file "Shootings and homicides within the Austin community area.ipynb".
The first task is:
Let's do something similar with homicides. It's exactly the same, in fact, but a few field names are different.
Can you write Python code for it?

homicides_ca = []
years = {}

for row in homicides:
    if not row['Geocode Override']:
        continue
    points = row['Geocode Override'][1:-1].split(',')
    if len(points) != 2:
        continue
    point = Point(float(points[1]), float(points[0]))
    row['point'] = point
    if poly.contains(point):
        homicides_ca.append(row)

print 'Found %d homicides in this community area' % len(homicides_ca)
for f in homicides_ca:
    print f['Occ Date'], f['Occ Time'],  f['Age'], f['Sex'], f['Address of Occurrence']
    if not f['Occ Date']:
        continue
    dt = datetime.strptime(f['Occ Date'], '%Y-%m-%d')
    if dt.year not in years:
        years[dt.year] = 0
    years[dt.year] += 1
print years


I am working on the file "Shootings and homicides within the Austin community area.ipynb".
The first task is:
Now let's get the shootings data.
Can you write Python code for it?

import os

def get_data(table):
    r = requests.get('%stable/json/%s' % (os.environ['NEWSROOMDB_URL'], table))
    return r.json()

shootings = get_data('shootings')
homicides = get_data('homicides')


I am working on the file "Shootings and homicides within the Austin community area.ipynb".
The first task is:
Now let's see how many homicides we can associate with shootings. We'll say that if the locations are within five meters and the date and time of the shooting is within 10 minutes of the homicide, they're the same incident.
Can you write Python code for it?

import pyproj
from datetime import datetime, timedelta

geod = pyproj.Geod(ellps='WGS84')
associated = []

for homicide in homicides_ca:
    if not homicide['Occ Time']:
        homicide['Occ Time'] = '00:01'
    if not homicide['Occ Date']:
        homicide['Occ Date'] = '2000-01-01'
    homicide_dt = datetime.strptime('%s %s' % (homicide['Occ Date'], homicide['Occ Time']), '%Y-%m-%d %H:%M')
    for shooting in shootings_ca:
        if not shooting['Time']:
            shooting['Time'] = '00:01'
        if not shooting['Time']:
            shooting['Time'] = '2000-01-01'
        shooting_dt = datetime.strptime('%s %s' % (shooting['Date'], shooting['Time']), '%Y-%m-%d %H:%M')
        diff = homicide_dt - shooting_dt
        seconds = divmod(diff.days * 86400 + diff.seconds, 60)[0]
        if abs(seconds) <= 600:
            angle1, angle2, distance = geod.inv(
                homicide['point'].x, homicide['point'].y, shooting['point'].x, shooting['point'].y)
            if distance < 5:
                associated.append((homicide, shooting))
                break
print len(associated)

years = {}

for homicide in homicides:
    if not homicide['Occ Date']:
        continue
    dt = datetime.strptime(homicide['Occ Date'], '%Y-%m-%d')
    if dt.year not in years:
        years[dt.year] = 0
    years[dt.year] += 1

print years

from csv import DictWriter
from ftfy import fix_text, guess_bytes

for idx, row in enumerate(shootings_ca):
    if 'point' in row.keys():
        del row['point']
    for key in row:
        #print idx, key, row[key]
        if type(row[key]) is str:
            #print row[key]
            row[key] = fix_text(row[key].replace('\xa0', '').decode('utf8'))

for idx, row in enumerate(homicides_ca):
    if 'point' in row.keys():
        del row['point']
    for key in row:
        #print idx, key, row[key]
        if type(row[key]) is str:
            #print row[key]
            row[key] = row[key].decode('utf8')

with open('/Users/abrahamepton/Documents/austin_shootings.csv', 'w+') as fh:
  ...


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
1. Pull Data from the API
 We will pull data from the MailChimp API as an example in this notebook
 ### You pull from an API, or read in your own data from a csv or other file format
Can you write Python code for it?

# run this cell to define global variables to pull data from the API of your email service provider, in this case MailChimp
# and specify LIST_NAME so the files in the exported executive summary are labled correctly
# replace the variable values in quotes in red caps with the unique values for your MailChimp account
LIST_NAME='YOURLISTNAME'
NAME='YOURUSERNAME'# your MailChimp user name (used at login)
SECRET_KEY='YOURAPIKEY'# your MailChimp API Key
LIST_ID='YOURLISTID' # the ID for the individual list you want to look at
#OUT_FILE='OUTFILENAME.csv'# if you want to export your data, you can specify the outfile name and type, in this example CSV

# Make an output directly to explort the results and images from this notebook
oupt_dir='Shorenstein_Notebook_1_'+str(LIST_NAME)

# included in case the directory is already there so it doesn't error out
try:
    os.mkdir(oupt_dir)
except:
    print 'e'

# initalizes client - create a connection with the API; calling that connection client
client=MailChimp(NAME,SECRET_KEY)

%%time
# %%time, this magic command above tells jupyter to time how long this cell takes to run
# magic commands have to be first which is why this comment is not first
# makes a MailChimp API call and stores the results in member_data; kernal will be busy - filled in - while data is being pulled from the API
# from the client object - created in the input directly above - we are calling the method to get all member information for members on the list you are interested in
# names what is returned from the API member_data and specifies the fields for the API to return
# TIP: limit the fields to the ones you want speeds up the time it takes the API calls to complete
# if this times out or if you have issues, consider doing a batch request which does not require you to keep the connection open while the request processes
# batch requests are the recommended method by MailChimp, although they are significantly slower in pulling the data, they are more robust
# for simplicity we do a normal API...


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
3. Explore the Data
3.1 Basic List Composition
We go from asking 'What is my list size?'
Can you write Python code for it?

# List size - number of current subscribers, as measured by unique email addresses with status of subscribed on your list
list_size=sum(member_data_frame.status=='subscribed')
print list_size


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
3.2 List Composition Over Time
Looking at the types of emails you acquired over time, users are Subscribed, Unsubscribed, Pending or Cleaned.
Can you write Python code for it?

# prepare the data by calculating the month joined, and for each of those months, what % of those people are subscribed, unsubscribed, cleaned or pending
# NOTE: There is no output from this cell but you need to run it to see the graphs below.
member_data_frame['timestamp_opt']=member_data_frame.timestamp_opt.apply(pd.to_datetime)

member_data_frame['timestamp_signup']=member_data_frame.timestamp_signup.apply(pd.to_datetime)

# records missing signup_time
sum(member_data_frame.timestamp_signup.isnull())

# make sure index is unique because we are about to do some manipulations based on it
member_data_frame.reset_index(drop=True,inplace=True)

# index of members where we don't know when they signed up but we have opt in time
guess_time_ix=member_data_frame[(member_data_frame.timestamp_signup.isnull())&
                               (member_data_frame.timestamp_opt.isnull()!=True)].index

# when we don't have signup time use opt in time
member_data_frame.loc[guess_time_ix,'timestamp_signup']=member_data_frame.loc[guess_time_ix,'timestamp_opt']

# using integer division to break down people into groups by the month they joined
member_data_frame['join_month']=member_data_frame.timestamp_signup.apply(lambda x:
                                                                         pd.to_datetime(2592000*int((x.value/1e9)/2592000),unit='s'))

# represent the joined month as an integer of ms - milliseconds - since epoch time 0
# this format is not nice for people but very nice for computers!
member_data_frame['jv']=member_data_frame.join_month.apply(lambda x: x.value)
member_data_frame['jv']=member_data_frame.join_month.apply(lambda x: x.value)

# any times we get before this we assume are data errors
mim_time=pd.to_datetime('01-01-1990')

# ignore records before...


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
Breakdown of cleaned by time joined
Can you write Python code for it?

# Blue line represents cleaned users. NOTE: Y-Axis should be smaller than the graph above.
# If it's close to the same magnitude as above, you likely have an issue with bounces.
plt.figure(figsize=(20,5))
plt.plot(comps.join_month,comps.cleaned_per,'^-',color=c3)
# plt.title('Section 3.2 Proportion of Cleaned by Time Joined',fontdict={'fontsize':25})
plt.ylabel('Fraction of List Cohort',fontdict={'fontsize':20})
plt.xlabel('Time Joined',fontdict={'fontsize':20})
plt.yticks(fontsize=15)
plt.xticks(fontsize=15)
plt.savefig(oupt_dir+'/Section_3.2_Proportion_of_Cleaned_by_Time_Joined.png')
plt.legend(loc='best')


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
Clicks: Ever Clicked vs. Never Clicked
Can you write Python code for it?

# save ever open and never open rates for later
ev_opn_r=ev_opn/float(list_size)

nv_opn_r=nv_opn/float(list_size)

# false = number of subscribers who have ever clicked
# true = number of subscribers who have never clicked
member_data_frame[member_data_frame.status=='subscribed'].never_clicked.value_counts()

# save ever clicked and never clicked for later
ev_clk=member_data_frame[member_data_frame.status=='subscribed'].never_clicked.value_counts()[False]
nv_clk=member_data_frame[member_data_frame.status=='subscribed'].never_clicked.value_counts()[True]

# false = percent subscribers ever clicked
# true = percent subscribers never clicked
member_data_frame[member_data_frame.status=='subscribed'].never_clicked.value_counts()/float(list_size)

# save ever clicked and never clicked rates for later
ev_clk_r=ev_clk/float(list_size)
nv_clk_r=nv_clk/float(list_size)


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
Opens: Ever Opened vs. Never Opened
Can you write Python code for it?

# unpack individual user open rate and click rate from stats for each user record, add the value to a column named open and click respectively.
# create a column for those who never opened or clicked
# false = number of subscribers who have ever opened
# true = number of subscribers who have never opened
member_data_frame['open']=member_data_frame.stats.apply(lambda x: x['avg_open_rate'])
member_data_frame['click']=member_data_frame.stats.apply(lambda x: x['avg_click_rate'])

member_data_frame['never_opened']=member_data_frame.open.apply(lambda x:x==0)
member_data_frame['never_clicked']=member_data_frame.click.apply(lambda x:x==0)
member_data_frame[member_data_frame.status=='subscribed'].never_opened.value_counts()

# save ever_open and never open for later
ev_opn=member_data_frame[member_data_frame.status=='subscribed'].never_opened.value_counts()[False]
nv_opn=member_data_frame[member_data_frame.status=='subscribed'].never_opened.value_counts()[True]

# false = percent subscribers ever opened
# true = percent subscribers never opened
member_data_frame[member_data_frame.status=='subscribed'].never_opened.value_counts()/float(list_size)

# stack the data - never opened vs ever opened
plt.figure(figsize=(15,10))
ax=plt.hist([ever_opened.jv,never_opened.jv], stacked=True,normed=False,label=['EVER OPEND',
                                                                               'NEVER OPEND'],color=[c_ev,c_nev])

plt.xticks(ax[1],map(lambda x: pd.to_datetime(x).date(),ax[1]), rotation=35)
#plt.title('Section 3.3 Never Opened vs EverOpened by Time Joined, Stacked Histogram',fontdict={'fontsize':25})
plt.ylabel('Subscriber Count',fontdict={'fontsize':20})
plt.xlabel('Time Joined',fontdict={'fontsize':20})
plt.legend(loc='best', prop={'size': 20})
plt.yticks(fontsize=15)
plt.xticks(fontsize=15)
plt.savefig(oupt_dir+'/Section_3.3_NeverOpened_vs_EverOpened_by_Time_Joined.png')
plt.show()


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
Section 3.3 Subscriber (In)Activity
Can you write Python code for it?

# this cell does not have an output but you need to run it
# unpack open rate and click rate from stats for each record, add the value to a column named open and click respectively.
# create a column for those who never opened or clicked
# false = number of subscribers who have ever opened (gray)
# true = number of subscribers who have never opened (black)
member_data_frame['open']=member_data_frame.stats.apply(lambda x: x['avg_open_rate'])
member_data_frame['click']=member_data_frame.stats.apply(lambda x: x['avg_click_rate'])

member_data_frame['never_opened']=member_data_frame.open.apply(lambda x:x==0)
member_data_frame['never_clicked']=member_data_frame.click.apply(lambda x:x==0)

# continued from above, now as three separate arrays
ever_opened=member_data_frame[(member_data_frame.status=='subscribed')&
                                (member_data_frame.never_opened!=True)]
never_opened=member_data_frame[(member_data_frame.status=='subscribed')&
                  (member_data_frame.never_opened)]


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
Section 3.5 Investigating Churn
Lifetime Unsubscribe Rate
Can you write Python code for it?

# unsubscribe rate: over the lifetime of your list for everyone who successfully subscribed x% unsubscribed
# or, another way to think about this is on average over the lifetime of your list for every 100 people who successfully subscribed, x number of people unsubscribed
life_unsub=unsubscribed/float(unsubscribed+subscribed+cln)*100
print life_unsub


I am working on the file "Shorenstein_Center_Notebook_1.ipynb".
The first task is:
Section 4 Export Results of Notebook 1 to Folder
:bowtie:
Can you write Python code for it?

# export data

exec_dict={'Shorenstein Notebook 1':{
 1: u'3.1 List Composition Analysis',
 2: u'Tr12d86b1d6aaditionally measured list size',
 3: u'Total unique email records ',
 4: u'Composition of total unique email records by status ',
 10: u'3.2 List Lifetime Ratios',
 11: u'Lifetime Unsubscribe Rate',
 12: u'Lifetime Subscribe Rate',
 13: u'Lifetime Cleaned Rate',
 15: u'Section 3.3 Reader Activity',
 16: u'Ever Opened',
 17: u'Never Opened',
 19: u'Ever Clicked',
 20: u'Never Clicked'},
          '1':{2:list_size,3:total_un_eml,5: u'subscribed',
                             6: u'unsubscribed',7: u'cleaned',8: u'pending',
                           11:life_unsub,
                           12:life_sub,
                           13:life_clean,
                            16:ev_opn,
                            17:nv_opn,
                            19:ev_clk,
                            20:nv_clk},
          "2":{5:subscribed,
                                6:unsubscribed,
                                7:cln,
                                8:pndg,
                                16:ev_opn_r,
                                17:nv_opn_r,
                                19:ev_clk_r,
                                20:nv_clk_r}}

ex_rpt=pd.DataFrame(exec_dict)

ex_rpt['']=ex_rpt['1']

ex_rpt[' ']=ex_rpt['2']

del ex_rpt["1"]
del ex_rpt["2"]

ex_rpt.to_csv(oupt_dir+'/Shorenstein Notebook 1 executive report.csv',index=False)


I am working on the file "Shorenstein_Center_Notebook_2.ipynb".
The first task is:
2. Turn the Data into a pandas Data Frame
Can you write Python code for it?

# turns the member_data returned by the API into a pandas data frame
member_data_frame=pd.DataFrame(member_data['members'])

# unpack open rate and click rate from stats for each record, add the value to a new column named open and click respectively
# create a column for those who never opened or clicked
# false = number of subscribers who have ever opened
# true = number of subscribers who have never opened
member_data_frame['open']=member_data_frame.stats.apply(lambda x: x['avg_open_rate'])
member_data_frame['click']=member_data_frame.stats.apply(lambda x: x['avg_click_rate'])
member_data_frame['never_opened']=member_data_frame.open.apply(lambda x:x==0)
member_data_frame['never_clicked']=member_data_frame.click.apply(lambda x:x==0)

# preparing the data by calculating the month joined, and for each of those months, what % of those people are subscribed, unsubscribed, cleaned or pending.
# NOTE: There is no output from this cell but you need to run it to see the graphs below.
member_data_frame['timestamp_opt']=member_data_frame.timestamp_opt.apply(pd.to_datetime)
member_data_frame['timestamp_signup']=member_data_frame.timestamp_signup.apply(pd.to_datetime)

# records missing signup_time
print sum(member_data_frame.timestamp_signup.isnull())

# make sure index is unique because we are about to do some manipulations based on it
member_data_frame.reset_index(drop=True,inplace=True)

# index of members where we don't know when the signed up but we have opt in time
guess_time_ix=member_data_frame[(member_data_frame.timestamp_signup.isnull())&
                               (member_data_frame.timestamp_opt.isnull()!=True)].index

# when we don't have signup time use opt in time
member_data_frame.loc[guess_time_ix,'timestamp_signup']=member_data_frame.loc[guess_time_ix,'timestamp_opt']

# use integer division to break down people in to groups by the month they joined
member_data_frame['join_month']=member_data_frame.timestamp_signup.apply(lambda x:pd.to_datetime(2592000*int((x.value/1e9)/2592000),unit='s'))
...


I am working on the file "Shorenstein_Center_Notebook_2.ipynb".
The first task is:
3.3 Two-Dimensional Distributions
User unique open rate vs. when they joined, for subscribers
Can you write Python code for it?

# open rate vs. when someone joined, for subscribers only
# x axis = month they joined in miliseconds since 1970
# y axis is users unique open rate

nxt=3 #number of x ticks
xrt=np.linspace(member_data_frame[member_data_frame.status=='subscribed'].jv.min(),
member_data_frame[member_data_frame.status=='subscribed'].jv.max(),num=nxt)
g=sns.jointplot(member_data_frame[member_data_frame.status=='subscribed'].jv/1e9,
                  member_data_frame[member_data_frame.status=='subscribed'].open,
                kind="kde",size=10, space=0,ylim=(0,1))
j=g.ax_joint
mx=g.ax_marg_x
my=g.ax_marg_y
mx.set_xticks(map(lambda x:x/1e9,xrt))
mx.set_xticklabels(map(lambda x: pd.to_datetime(x).date(),xrt))
mx.set_title("Open Rate vs Join Time for Subscribers",fontdict={'fontsize':25})
plt.rcParams["axes.labelsize"] = 25
g.set_axis_labels(xlabel='Time Joined',ylabel='User Unique Open Rate',fontdict={'fontsize':25})
g.savefig(oupt_dir+'/8_open_vs_join_sub.png')


I am working on the file "Shorenstein_Center_Notebook_2.ipynb".
The first task is:
3.4 Time on List for Unsubscribers
Can you write Python code for it?

# shortest time on the list before unsubscribing
# from the individual who unsubscribed the fastest
member_data_frame[member_data_frame.life.isnull()!=True].life.min()

# longest time on the list before unsubscribing
# from the individual who stayed on the longest before unsubscribing
member_data_frame[member_data_frame.life.isnull()!=True].life.max()

# histogram of time range each unsubscriber was on the list before they unsubscribed
# depending on how granular you want to go and the lifetime of your list you may want to update bin size

plt.figure(figsize=(20,10))
ax=plt.hist([member_data_frame.dropna(subset=['life']).life.apply(lambda x:x.value)],label=['life time'],color=c2)
plt.xticks(ax[1],map(lambda x: pd.to_timedelta(x).floor('D'),ax[1]), rotation=30)
plt.title('Distribution of Lifetime of Unsubscribed Users',fontdict={'fontsize':25})
plt.xlabel('Lifetime on List',fontdict={'fontsize':20})
plt.ylabel('Counts',fontdict={'fontsize':20})
plt.yticks(fontsize=15)
plt.xticks(fontsize=15)
plt.legend(loc='best')
plt.savefig(oupt_dir+'/11_life_unsub.png')
plt.show()


I am working on the file "Shorenstein_Center_Notebook_2.ipynb".
The first task is:
Time of the last email opened vs. time joined for subscribers
Can you write Python code for it?

# x axis is distribution of when joined - farther to the right is joining more recently.
# y axis is the oldest email record of opening in last 50 actions.
# upper left is longtime engaged person, person who joined awhile ago who is still active

nxt=3 #number of x ticks
nyt=3 #number of x ticks
xrt=np.linspace(member_data_frame[member_data_frame.status=='subscribed'].jv.min(),
member_data_frame[member_data_frame.status=='subscribed'].jv.max(),num=nxt)
yrt=np.linspace(member_data_frame[member_data_frame.status=='subscribed'].lv.min(),
member_data_frame[member_data_frame.status=='subscribed'].lv.max(),num=nxt)

g = sns.jointplot(member_data_frame[member_data_frame.status=='subscribed'].jv/1e9,
                  member_data_frame[member_data_frame.status=='subscribed'].lv/1e9, kind="kde", size=10, space=0)

j=g.ax_joint
mx=g.ax_marg_x
my=g.ax_marg_y
mx.set_xticks(map(lambda x:x/1e9,xrt))
mx.set_xticklabels(map(lambda x: pd.to_datetime(x).date(),xrt))
my.set_yticks(map(lambda x:x/1e9,yrt))
my.set_yticklabels(map(lambda x: pd.to_datetime(x).date(),yrt))
mx.set_title("Latest Opened Email vs Join Time for Subscribers",fontdict={'fontsize':25})
g.set_axis_labels(xlabel='Time Joined',ylabel='Latest Open')
g.savefig(oupt_dir+'/9_last_open_vs_join_sub.png')


I am working on the file "Shorenstein_Center_Notebook_2.ipynb".
The first task is:
To asking 'What is the distribution of user unique open rates for current subscribers vs. unsubscribes?'
Can you write Python code for it?

# user unique open rate = unique open rate for an individual on your list
# calculation: (number of unique opens by a user / number of campaigns received by that user) x 100
plt.figure(figsize=(20,10))
plt.hist([member_data_frame[member_data_frame.status=='subscribed'].open,
          member_data_frame[member_data_frame.status=='unsubscribed'].open], stacked=True,
         normed=False,label=['Subscribed',
                             'Unsubscribed'],
        color=[c1,c2])

plt.title('Distribution of User Unique Open Rate, Subscribed vs Unsubscibed',fontdict={'fontsize':25})
plt.xlabel("User Unique Open Rate",fontdict={'fontsize':20})
plt.ylabel("Counts",fontdict={'fontsize':20})
plt.yticks(fontsize=15)
plt.xticks(fontsize=15)
plt.legend(loc='best', prop={'size': 20})
plt.savefig(oupt_dir+'/3.1_dist_open_sub_vs_unsub.png')
plt.show()


I am working on the file "Short-circuit_calc/Optimization.ipynb".
The first task is:
<img src='formula.png'>
Can you write Python code for it?


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
$\implies$ Well... It doesn't seem that much faster from the naive Python code.
We specified the signature when calling @numba.jit, and used the more appropriate data structure (string is probably the smaller, numpy array are probably faster).
But even these tricks didn't help that much.
I tested, and without specifying the signature, the fastest approach is using string, compared to using lists or numpy arrays.
Note that the @jit-powered function is compiled at runtime when first being called, so the signature used for the first call is determining the signature used by the compile function

Tests (2/2)
To test more robustly, let us generate some (uniformly) random binary sequences.
Can you write Python code for it?

from numpy.random import binomial

def bernoulli(p, size=1):
    """One or more samples from a Bernoulli of probability p."""
    return binomial(1, p, size)

bernoulli(0.5, 20)


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
And so, this function can test to check that the three implementations (naive, Cython-powered, Numba-powered) always give the same result.
Can you write Python code for it?

def tests_3_functions(n, p=0.5, debug=True):
    s = random_binary_sequence(n, p=p)
    c1 = lempel_ziv_complexity(s)
    if debug:
        print("Sequence s = {} ==> complexity C = {}".format(s, c1))
    c2 = lempel_ziv_complexity_cython(s)
    c3 = lempel_ziv_complexity_numba(s)
    assert c1 == c2 == c3, "Error: the sequence {} gave different values of the Lempel-Ziv complexity from 3 functions ({}, {}, {})...".format(s, c1, c2, c3)
    return c1

tests_3_functions(5)

tests_3_functions(20)

tests_3_functions(50)

tests_3_functions(500)

tests_3_functions(5000)


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
And to compare it fairly, let us use Pypy for comparison.
Can you write Python code for it?

%%time
%%pypy

def lempel_ziv_complexity(binary_sequence):
    """Lempel-Ziv complexity for a binary sequence, in simple Python code."""
    u, v, w = 0, 1, 1
    v_max = 1
    length = len(binary_sequence)
    complexity = 1
    while True:
        if binary_sequence[u + v - 1] == binary_sequence[w + v - 1]:
            v += 1
            if w + v >= length:
                complexity += 1
                break
        else:
            if v > v_max:
                v_max = v
            u += 1
            if u == w:
                complexity += 1
                w += v_max
                if w > length:
                    break
                else:
                    u = 0
                    v = 1
                    v_max = 1
            else:
                v = 1
    return complexity

s = "1001111011000010"
lempel_ziv_complexity(s)  # 1 / 0 / 01 / 1110 / 1100 / 0010

from random import random

M = 100
N = 10000
for _ in range(M):
    s = ''.join(str(int(random() < 0.5)) for _ in range(N))
    lempel_ziv_complexity(s)


%%time
%%pypy

def lempel_ziv_complexity(sequence):
    """Lempel-Ziv complexity for a binary sequence, in simple Python code."""
    sub_strings = set()
    n = len(sequence)

    ind = 0
    inc = 1
    while True:
        if ind + inc > len(sequence):
            break
        sub_str = sequence[ind : ind + inc]
        if sub_str in sub_strings:
            inc += 1
        else:
            sub_strings.add(sub_str)
            ind += inc
            inc = 1
    return len(sub_strings)

s = "1001111011000010"
lempel_ziv_complexity(s) # 1 / 0 / 01 / 11 / 10 / 110 / 00 / 010

from random import random

M = 1000
N = 10000
for _ in range(M):
    s = ''.join(str(int(random() < 0.5)) for _ in range(N))
    lempel_ziv_complexity(s)


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
Benchmarks
On two example of strings (binary sequences), we can compare our three implementation.
Can you write Python code for it?


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
Complexity ?
$\implies$ The function lempel_ziv_complexity_cython seems to be indeed (almost) linear in $n$, the length of the binary sequence $S$.
But let check more precisely, as it could also have a complexity of $\mathcal{O}(n \log n)$.
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set(context="notebook", style="darkgrid", palette="hls", font="sans-serif", font_scale=1.4)


import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set(context="notebook", style="darkgrid", palette="hls", font="sans-serif", font_scale=1.4)

import numpy as np
import timeit

sizes = np.array(np.trunc(np.logspace(1, 6, 30)), dtype=int)

times = np.array([
    timeit.timeit(
        stmt="lempel_ziv_complexity_cython(random_string({}))".format(n),
        globals=globals(),
        number=10,
    )
    for n in sizes
])

plt.figure(figsize=(15, 10))
plt.plot(sizes, times, 'o-')
plt.xlabel("Length $n$ of the binary sequence $S$")
plt.ylabel(r"Time in $\mu\;\mathrm{s}$")
plt.title("Time complexity of Lempel-Ziv complexity")
plt.show()

plt.figure(figsize=(15, 10))
plt.loglog(sizes, times, 'o-')
plt.xlabel("Length $n$ of the binary sequence $S$")
plt.ylabel(r"Time in $\mu\;\mathrm{s}$")
plt.title("Time complexity of Lempel-Ziv complexity, loglog scale")
plt.show()


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
It is linear in $\log\log$ scale, so indeed the algorithm seems to have a linear complexity.
To sum-up, for a sequence $S$ of length $n$, it takes $\mathcal{O}(n)$ basic operations to compute its Lempel-Ziv complexity $\mathrm{Lempel}-\mathrm{Ziv}(S)$.

Conclusion

The Lempel-Ziv complexity is not too hard to implement, and it indeed represents a certain complexity of a binary sequence, capturing the regularity and reproducibility of the sequence.

Using the Cython was quite useful to have a $\simeq \times 100$ speed up on our manual naive implementation !

The algorithm is not easy to analyze, we have a trivial $\mathcal{O}(n^2)$ bound but experiments showed it is more likely to be $\mathcal{O}(n \log n)$ in the worst case, and $\mathcal{O}(n)$ in practice for "not too complicated sequences" (or in average, for random sequences).

(Experimental) Julia implementation
I want to (quickly) try to see if I can use Julia to write a faster version of this function.
See issue #1.
Can you write Python code for it?

%%time
%%script julia

"""Lempel-Ziv complexity for a sequence, in simple Julia code."""
function lempel_ziv_complexity(sequence)
    sub_strings = Set()
    n = length(sequence)

    ind = 1
    inc = 1
    while true
        if ind + inc > n
            break
        end
        sub_str = sequence[ind : ind + inc]
        if sub_str in sub_strings
            inc += 1
        else
            push!(sub_strings, sub_str)
            ind += inc
            inc = 1
        end
    end
    return length(sub_strings)
end

s = "1001111011000010"
lempel_ziv_complexity(s)  # 1 / 0 / 01 / 1110 / 1100 / 0010

M = 1000;
N = 10000;
for _ in 1:M
    s = join(rand(0:1, N));
    lempel_ziv_complexity(s);
end
lempel_ziv_complexity(s)  # 1 / 0 / 01 / 1110 / 1100 / 0010


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
It is linear in $\log\log$ scale, so indeed the algorithm seems to have a linear complexity.
To sum-up, for a sequence $S$ of length $n$, it takes $\mathcal{O}(n)$ basic operations to compute its Lempel-Ziv complexity $\mathrm{Lempel}-\mathrm{Ziv}(S)$.

Conclusion

The Lempel-Ziv complexity is not too hard to implement, and it indeed represents a certain complexity of a binary sequence, capturing the regularity and reproducibility of the sequence.

Using the Cython was quite useful to have a $\simeq \times 100$ speed up on our manual naive implementation !

The algorithm is not easy to analyze, we have a trivial $\mathcal{O}(n^2)$ bound but experiments showed it is more likely to be $\mathcal{O}(n \log n)$ in the worst case, and $\mathcal{O}(n)$ in practice for "not too complicated sequences" (or in average, for random sequences).

(Experimental) Julia implementation
I want to (quickly) try to see if I can use Julia to write a faster version of this function.
See issue #1.
Disclaimer: I am still learning Julia!
Can you write Python code for it?

%%time
%%script julia

"""Lempel-Ziv complexity for a binary sequence, in simple Julia code."""
function lempel_ziv_complexity(binary_sequence)
    u, v, w = 0, 1, 1
    v_max = 1
    size = length(binary_sequence)
    complexity = 1
    while true
        if binary_sequence[u + v] == binary_sequence[w + v]
            v += 1
            if w + v >= size
                complexity += 1
                break
            end
        else
            if v > v_max
                v_max = v
            end
            u += 1
            if u == w
                complexity += 1
                w += v_max
                if w > size
                    break
                else
                    u = 0
                    v = 1
                    v_max = 1
                end
            else
                v = 1
            end
        end
    end
    return complexity
end

s = "1001111011000010"
lempel_ziv_complexity(s)  # 1 / 0 / 01 / 1110 / 1100 / 0010

M = 100;
N = 10000;
for _ in 1:M
    s = join(rand(0:1, N));
    lempel_ziv_complexity(s);
end
lempel_ziv_complexity(s)  # 1 / 0 / 01 / 1110 / 1100 / 0010


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
Let check the time used by all the three functions, for longer and longer sequences:
Can you write Python code for it?

%timeit tests_3_functions(10, debug=False)
%timeit tests_3_functions(20, debug=False)
%timeit tests_3_functions(40, debug=False)
%timeit tests_3_functions(80, debug=False)
%timeit tests_3_functions(160, debug=False)
%timeit tests_3_functions(320, debug=False)

def test_cython(n):
    s = random_binary_sequence(n)
    c = lempel_ziv_complexity_cython(s)
    return c

%timeit test_cython(10)
%timeit test_cython(20)
%timeit test_cython(40)
%timeit test_cython(80)
%timeit test_cython(160)
%timeit test_cython(320)

%timeit test_cython(640)
%timeit test_cython(1280)
%timeit test_cython(2560)
%timeit test_cython(5120)

%timeit test_cython(10240)
%timeit test_cython(20480)


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
Let try it!
Can you write Python code for it?


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
Tests (1/2)
Can you write Python code for it?


I am working on the file "Short_study_of_the_Lempel-Ziv_complexity.ipynb".
The first task is:
That's probably not optimal, but we can generate a string with:
Can you write Python code for it?

''.join(str(i) for i in bernoulli(0.5, 20))

def random_binary_sequence(n, p=0.5):
    """Uniform random binary sequence of size n, with rate of 0/1 being p."""
    return ''.join(str(i) for i in bernoulli(p, n))

random_binary_sequence(50)
random_binary_sequence(50, p=0.1)
random_binary_sequence(50, p=0.25)
random_binary_sequence(50, p=0.5)
random_binary_sequence(50, p=0.75)
random_binary_sequence(50, p=0.9)


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Bellman-Ford Algorithm
The Bellman-Ford algorithm solves the single source shortest path problem in the general case where negative weight edges may appear.
Can you write Python code for it?

"""
    Takes an adjacency list representing a graph and a starting node 'start',
    and returns a list 'sp' where sp[i] indicates the shortest path cost
    from start to i.
        Greedy algorithm which always chooses the closest node from the fringe.
        Once a node has been extracted from the fringe, then it's sp from the origin
    has already been computed.
    Each node gets visited (in this case, extracted from the priority queue) only once
"""
def dijkstra(graph, start):
    n_nodes = len(graph)

    # Array containing the shortest path costs initialized to infinity, apart to start itself which is 0
    sp = np.full(n_nodes, np.inf)
    sp[start] = 0

    # Bitmaps to indicate if a given node has already been visited
    visited = np.full(n_nodes, False)

    # Min-priority queue representing the fringe of nodes. Contains tuples (distance_from_start, node)
    fringe = queue.PriorityQueue()
    fringe.put((sp[start], start))

    while not fringe.empty():
        _, node = fringe.get()

        visited[node] = True

        for adj_node, edge_cost in graph[node]:
            if not visited[adj_node]:
                if sp[adj_node] > sp[node] + edge_cost:
                    sp[adj_node] = sp[node] + edge_cost
                    fringe.put((sp[adj_node], adj_node))

    return sp


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Complexity Analysis
Let $n$ be the number of nodes in the graph. Dijkstra while loop will execute at most $n$ times, since each node can be extracted at most one time from the priority queue. Since extracting from a priority queueu takes logarithmic time, this gives us $\mathcal{O}(n\log{}n)$ complexity for the while loop, without taking the inner for loop into account.
In the inner for loop, each node loops through all its neighbours. Using an amortyzed analysis we know that the $n$ nodes overall will iterate over all edges (call this number $E$), once per edge.
This means that the overall complexity for Dijkstra is $\mathcal{O}(n\log{}n + E)$
All Pairs Shorterst Paths
Floyd Warshall Algorithm
Can you write Python code for it?

"""
    Takes an adjacency matrix representing a graph and returns a new matrix
    with matrix[i, j] containg the shortest path between node i and node j.
"""
def floyd_warshall(graph_matrix):
    g = graph_matrix[:][:]
    n_nodes = len(g)
    floyd_warshall_helper(g, n_nodes - 1)

    return g

"""
    Takes a matrix representing a graph and modifies that matrix
    so that at the end of the function graph[i, j] contains the shortest path between
    node i and node j using only nodes in the set {0,1, .., k} as intermediary nodes,
    which we will call sp(i, j, k).
      We have sp(i, j, k) = min(sp(i, j, k - 1), sp(i, k, k - 1) + sp(k, j, k - 1)).
    This is saying that the shortest path from i to j using nodes {0, 1, .., k} is
    the minimum between going from i to j using only nodes {0,1, .., k - 1}, and the
    sum of first going from i to k and then from j to k (both steps still with nodes {0,1, .., k - 1}).
"""
def floyd_warshall_helper(graph, k):
    if k == -1:
        # No intermediary node, leave the original adjacency matrix unchanged
        return

    # Run recursively
    floyd_warshall_helper(graph, k - 1)
    # Now graph[i][j] contains the shortest path using node {0, 1, .., k - 1} as intermediary

    n_nodes = len(graph)
    # Update distance to get from i to j, for every i and j
    for i in range(n_nodes):
        for j in range(n_nodes):
            graph[i][j] = min(graph[i][j], graph[i][k] + graph[k][j])


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Complexity Analysis
Let $n$ be the number of nodes in the graph. Dijkstra while loop will execute at most $n$ times, since each node can be extracted at most one time from the priority queue. Since extracting from a priority queueu takes logarithmic time, this gives us $\mathcal{O}(n\log{}n)$ complexity for the while loop, without taking the inner for loop into account.
In the inner for loop, each node loops through all its neighbours. Using an amortyzed analysis we know that the $n$ nodes overall will iterate over all edges (call this number $E$), once per edge.
This means that the overall complexity for Dijkstra is $\mathcal{O}(n\log{}n + E)$
Dijkstra Variant For Negative Edges
Can you write Python code for it?

"""
    Takes an adjacency list representing a graph and a starting node 'start',
    and returns a list 'sp' where sp[i] indicates the shortest path cost
    from start to i.
        Greedy algorithm which always chooses the closest node from the fringe.
        Once a node has been extracted from the fringe, then it's sp from the origin
    has already been computed.
"""
def dijkstra_variant(graph, start):
    n_nodes = len(graph)
    # Array containing the shortest path costs initialized to infinity, apart to start itself which is 0
    sp = np.full(n_nodes, np.inf)
    sp[start] = 0

    # Min-priority queue representing the fringe of nodes. Contains tuples (distance_from_start, node)
    fringe = queue.PriorityQueue()
    fringe.put((sp[start], start))

    while not fringe.empty():
        _, node = fringe.get()

        for adj_node, edge_cost in graph[node]:
            if sp[adj_node] > sp[node] + edge_cost:
                sp[adj_node] = sp[node] + edge_cost
                fringe.put((sp[adj_node], adj_node))

    return sp


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Draw Graph
Can you write Python code for it?

img = get_graph_img_from_adjacency_matrix(graph_matrix)
img


img = get_graph_img_from_adjacency_list(non_negative_graph_list)
img


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Graph with negative edges
Can you write Python code for it?

n_nodes = 10

# List of tuples (start, destination, cost) which represent directed edges
edges_list = [
    (0, 1, 2),
    (0, 2, 10),
    (0, 3, 7),
    (1, 5, 4),
    (2, 3, -4),
    (2, 4, 3),
    (3, 6, 1),
    (4, 0, -5),
    (5, 3, 1),
    (7, 8, 3),
    (8, 9, 4),
    ]

# Adjacency list representation of a graph of n_nodes nodes
# non_negative_graph_list[i] contains a list of tuple (adj_node, cost) for all adjacent nodes of i
graph_list = [[] for _ in range(n_nodes)]

# Adjacency matrix representation of a graph of n_nodes nodes
# Initialize all values to infinite
graph_matrix = np.full((n_nodes, n_nodes), np.inf)
# Values in the diagonal are 0: going from any node to itself has no cost
# np.fill_diagonal(graph_matrix, 0)

for edge in edges_list:
    start, destination, cost = edge
    graph_matrix[start, destination] = cost
    graph_list[start].append((destination, cost))


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
However, it doesn't in general produce correct answers for graphs with negative edges. Our graph with negative edges should serve as example.
Can you write Python code for it?

assert test_single_source_negative(dijkstra) == False


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Solution
Can you write Python code for it?

# List of tuples (start, destination, shortest_path_cost) which represent directed edges
solution_list = [
    (0, 1, 2),
    (0, 5, 6),
    (0, 3, 6),
    (0, 8, np.inf),
    (0, 6, 7),
    (4, 1, -3),
    ]


# List of tuples (start, destination, shortest_path_cost) which represent directed edges
non_negative_solution_list = [
    (0, 1, 2),
    (0, 5, 6),
    (0, 3, 7),
    (0, 8, np.inf),
    (4, 1, 7),
    ]


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Testing
Can you write Python code for it?

sp_matrix = floyd_warshall(graph_matrix)

for sp in solution_list:
    start, destination, cost = sp
    assert sp_matrix[start, destination] == cost


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Testing
Dijkstra algorithm should produce the correct answers for our graph with non negative edges.
Can you write Python code for it?

assert test_single_source_non_negative(dijkstra) == True


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
This variant should produce the correct answers also for graphs with negative edges. Our graph with negative edges should serve as example.
Can you write Python code for it?

assert test_single_source_negative(dijkstra_variant) == True


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Utilities to draw graphs
Can you write Python code for it?

def get_graph_img_from_adjacency_matrix(graph_matrix):
    n_nodes = len(graph_matrix)
    dot = Digraph()

    for i in range(n_nodes):
        dot.node(str(i))

    for i in range(n_nodes):
        for j in range(n_nodes):
            if np.isfinite(graph_matrix[i][j]):
                dot.edge(str(i), str(j), str(graph_matrix[i][j]))

    return dot

def get_graph_img_from_adjacency_list(graph_list):
    n_nodes = len(graph_list)
    dot = Digraph()

    for i in range(n_nodes):
        dot.node(str(i))

    for i in range(n_nodes):
        for adj_node, edge_cost in graph_list[i]:
            dot.edge(str(i), str(adj_node), str(edge_cost))

    return dot

def get_graph_img_from_edges_list(n_nodes, edges_list):
    dot = Digraph()

    for i in range(n_nodes):
        dot.node(str(i))

    for edge in edges_list:
        start, destination, cost = edge
        dot.edge(str(start), str(destination), str(cost))

    return dot


I am working on the file "ShortestPath/Shortest Path Problem.ipynb".
The first task is:
Utilities to time functions
Can you write Python code for it?

# A timer - runs the provided function and reports the
# run time in ms
def time_f(f):
    before = clock()
    f()
    after = clock()
    return after - before


I am working on the file "ShowOff_slides.ipynb".
The first task is:
The extracted tree still contains much noise
Can you write Python code for it?

Image("img/flatten_tree_data.png")

# Extract only the items containing the pattern "Zika virus infection "
#sample= '\n22 April 2016\n\t\t\tZika virus infection – Papua New Guinea - USA\n'
keywdEN ="Zika virus infection "
zikaNews_content = [s for s in zikaNews_dirty if re.search(keywdEN, s)]

zikaNews_content[0:10] # first 11 elements


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
After the previously made assessment, use the correlation function to fill gaps in the dataset.
Can you write Python code for it?

dataset.fill_missing_correlation('CODtot_line2',
                                 'CODsol_line2',
                                 [dt.datetime(2013,1,23),dt.datetime(2013,1,25)],
                                 [dt.datetime(2013,1,1,0,5,0),dt.datetime(2013,1,31)],
                                 only_checked=True,clear=False,plot=True)


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
And now for the actual package...
Can you write Python code for it?


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Average daily profile
Fill missing datapoints by using an average daily profile. The fill_missing_standard function requires the running of the calc_daily_profile function, also developed for this package, first. This creates a dataframe (dataset.daily_profile) containing the average daily profile calculated within a defined time period (e.g. selecting only non-peak days for example).
Can you write Python code for it?

dataset.calc_daily_profile('CODtot_line2',[dt.datetime(2013,1,1),dt.datetime(2013,1,8)],
                           quantile=0.9,clear=True)

dataset.fill_missing_standard('CODtot_line2',[dt.datetime(2013,1,14),dt.datetime(2013,1,17)],
                              only_checked=True,clear=False,plot=True)


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Calculations
Calculate the daily average of a certain data series
Can you write Python code for it?


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Check the reliability of the filling algorithms
In order to be able to make a choice and apply the best method to fill gaps in the data, the wwdata package provides the option to check for the reliability of each filling algorithm. This is represented in the below figure.

In wording, the workflow of the check_filling_error is as follows:
* Randomly (!) create large or small artificial gaps in the data within the given test_data_range.
* Fill the created gaps with a chosen filling function (see further in this notebook for illustrations of those).
* Compare the original data points with the filled data points and calculate the deviation between them.
* Iterate for a given number of times, to average out the random creation of the gaps.
Before applying this, it is wise to check the total number of points within test_data_range and then determine the number of gaps to create. Take into account that the length of the gaps is sampled from a uniform distribution between 0 and the maximum length of a gap given as an argument.
For example: creating two large gaps of 50 datapoints in a dataset containing 100 datapoints would mean a theoretical average of 50% data recovery (2*(50/2) = 50 data points are left out of the 100; the 2 gaps can however still overlap)
Can you write Python code for it?

len(dataset.data['2013/1/1':'2013/1/17'])

dataset._filling_warning_issued

dataset.check_filling_error(100,'CODtot_line2','fill_missing_standard',[dt.datetime(2013,1,1,0,5),dt.datetime(2013,1,17)],
                            nr_small_gaps=70,max_size_small_gaps=12,
                            nr_large_gaps=3,max_size_large_gaps=800,
                            to_fill='CODtot_line2',arange=[dt.datetime(2013,1,1,0,5),dt.datetime(2013,1,17)],
                            only_checked=True)

dataset.check_filling_error(100,'CODtot_line2','fill_missing_daybefore',[dt.datetime(2013,1,1,0,5),dt.datetime(2013,1,17)],
                            nr_small_gaps=70,max_size_small_gaps=12,
                            nr_large_gaps=3,max_size_large_gaps=800,
                            to_fill='CODtot_line2',arange=[dt.datetime(2013,1,1,0,5),dt.datetime(2013,1,17)],
                            range_to_replace=[0,10],only_checked=True)


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Check what version you have installed
Can you write Python code for it?


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Convert the absolute timestamps to relative values. This can be important when data is to be used for modeling purposes later on, and needs to be written to text files.
Can you write Python code for it?


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Data from previous day
Under the assumption that "The best prediction for tomorrows weather is todays weather", one can also replace missing data by making use of (one of) the previous days.
Can you write Python code for it?

dataset.fill_missing_daybefore('CODtot_line2',
                               [dt.datetime(2013,1,25),dt.datetime(2013,1,27)],
                               range_to_replace=[0,10],plot=True,
                               only_checked=False)

fig, ax = dataset.plot_analysed('CODtot_line2')
ax.legend(bbox_to_anchor=(1.3,1.0),fontsize=18)
ax.set_ylabel('Total COD [mg/L]',fontsize=18);ax.set_xlabel('')
ax.tick_params(labelsize=14)


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Fill data
Filling data can be done using a range of functions implemented in the package. Again, a new pandas dataframe is created (dataset.meta_filled, see also below figure), starting from the dataset.meta_valid dataframe, and updated with tags indicating what filling method was used to obtain a certain point.

Using the only_checked argument, implemented in most filling functions, the user can always choose whether only data points tagged as filtered will be filled, or all data points within a certain range.
When using the plotting argument to plot the analysed data, the user will see a plot based on the latest function that was used; if this was a filter function, the data will be plotted based on the dataset.meta_valid dataframe, if it was a filling function, the tags in dataset.meta_filled will be used.
Interpolation
Fill missing data points by interpolation, if number of consecutive missing points is lower than a specified number.
Can you write Python code for it?

dataset.fill_missing_interpolation('CODtot_line2',12,[dt.datetime(2013,1,1),dt.datetime(2013,1,8)],
                                   plot=True)


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Instead of a package-specific filtering, data points can also be filtered and replaced by other filtering algorithms, such as the Savitsky-Golay filter as illustrated below. The disadvantage of this is that no tags are added to the meta_valid DataFrame and that original data are replaced (when the inplace option is set to True).
Can you write Python code for it?


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Instead of a ratio, a correlation can be sought. In case of a zero intercept, this of course gives a result in the same range if the same data is used. To have a good impression on how useful the calculated correlation is, a prediction interval is plotted as well when plot is set to True.
Can you write Python code for it?

dataset.get_correlation('CODtot_line2',
                        'CODsol_line2',
                        [dt.datetime(2013,1,1,0,5,0),dt.datetime(2013,1,31)],
                        zero_intercept=True,plot=True)


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Model output
Fill gaps using a model output. This assumes that the user has good reason to trust that the model predictions are sound and can indeed be used to replace missing data where needed.
Can you write Python code for it?

model_output_ontv_1 = pd.read_csv('./data/model_output.txt',
                           sep='\t')
units_model = model_output_ontv_1.iloc[0]
model_output_ontv_1 = model_output_ontv_1.drop(0,inplace=False).reset_index(drop=True)
model_output_ontv_1 = model_output_ontv_1.astype(float)
model_output_ontv_1.set_index('#.t',drop=True,inplace=True)
model_output_ontv_1.columns

dataset.fill_missing_model('CODtot_line2',model_output_ontv_1['.sewer_1.COD'],
                           [dt.datetime(2013,1,18),dt.datetime(2013,1,22)],
                           only_checked=True,plot=True)


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
NaN values
Tag all NaN (Not a Number) values as 'filtered'.
Can you write Python code for it?


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Ratio or correlation
Constant ratios or correlations between data can be used to filled missing points. The user can calculate and compare ratios and correlations (currently only linear) between selected measurements, and fill data using these.
nb: in the examples below, data filling based on ratios or correlation is obviously not a very good choice. Both methods are included here for completeness of method showcasing.
Can you write Python code for it?

dataset.calc_ratio('CODtot_line2','CODsol_line2',
                   [dt.datetime(2013,1,1,0,5,0),dt.datetime(2013,1,31)])


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Sensor failure
Tag all datapoints that are part of a constant (within a given bound) signal.
Can you write Python code for it?


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
To find the 'best' ratio (i.e. the one with the lowest relative standard deviation ($\sigma/\mu$)), the ratio obtained in different periods can be compared and the best one used during possible further replacements.
Can you write Python code for it?


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
Use the average obtained from the compare_ratio function to fill in missing values. (in this case, as mentioned before, this does clearly not work, since zero-values are replaced with zero-values. This only showcases the function and its arguments).
Can you write Python code for it?

dataset.fill_missing_ratio('CODtot_line2',
                           'CODsol_line2',avg,
                           [dt.datetime(2013,1,22),dt.datetime(2013,1,23)],
                           only_checked=True,plot=True)


I am working on the file "Showcase_OnlineSensorBased.ipynb".
The first task is:
pd.read_excel
Can you write Python code for it?

measurements = pd.read_csv('./data/201301.txt',sep='\t',skiprows=0)
measurements.columns


I am working on the file "Siamese Network - MNSIT - Breakdown.ipynb".
The first task is:
So basically here we are just flattening the images dimensions
Can you write Python code for it?

# normalize
x_train /= 255
x_test /= 255
# flatten images
input_dim = 784
epochs = 20

# create training+test positive and negative pairs
digit_indices = [np.where(y_train == i)[0] for i in range(10)]
tr_pairs, tr_y = create_pairs(x_train, digit_indices)

digit_indices = [np.where(y_test == i)[0] for i in range(10)]
te_pairs, te_y = create_pairs(x_test, digit_indices)

tr_pairs.shape


I am working on the file "Siamese Network - MNSIT - Breakdown.ipynb".
The first task is:
here we are building a pair with a true and false instance thats why the inc on dn
Can you write Python code for it?

# this adds the negative pair
# z1, z2 = digit_indices[d][i], digit_indices[dn][i]
# pairs += [[x[z1], x[z2]]]


I am working on the file "Siamese Network - MNSIT - Breakdown.ipynb".
The first task is:
so the code above creates an array for each of the numbers, each array has the indices for each of the numbers

create_pairs function
python
def create_pairs(x, digit_indices):
    '''Positive and negative pair creation.
    Alternates between positive and negative pairs.
    '''
    pairs = []
    labels = []
    n = min([len(digit_indices[d]) for d in range(10)]) - 1
    for d in range(10):
        for i in range(n):
            z1, z2 = digit_indices[d][i], digit_indices[d][i + 1]
            pairs += [[x[z1], x[z2]]]
            inc = random.randrange(1, 10)
            dn = (d + inc) % 10
            z1, z2 = digit_indices[d][i], digit_indices[dn][i]
            pairs += [[x[z1], x[z2]]]
            labels += [1, 0]
    return np.array(pairs), np.array(labels)
Can you write Python code for it?

digit_indices[0][2], digit_indices[0][2+1]

z1, z2 = digit_indices[0][2], digit_indices[0][2+1]


I am working on the file "Siamese Network - MNSIT - Breakdown.ipynb".
The first task is:
so we are getting the indices of two intances of the same number
Can you write Python code for it?


I am working on the file "Siamese Network - MNSIT - Breakdown.ipynb".
The first task is:
so we keep building the pairs
Can you write Python code for it?

inc = random.randrange(1, 10)

inc

d = 4

dn = (d + inc) % 10

dn


I am working on the file "Siamese Network - MNSIT - Breakdown.ipynb".
The first task is:
the label is 1 for the first example and zero for the fake that we just made
Can you write Python code for it?

# labels += [1, 0]

pairs = []
labels = []

n = min([len(digit_indices[d]) for d in range(10)]) - 1
# how many indices are the per number
# get the min of all the indices

n

 for d in range(10):
        print(len(digit_indices[d]))

# then it substract 1 - I assume its because the range is non-inclusive

n = 10
ran = 5

l = []

l += [1, 0]

l

for d in range(ran):
    for i in range(n):
        z1, z2 = digit_indices[d][i], digit_indices[d][i + 1]
        print(z1.shape, z2.shape)
        pairs += [[x_train[z1], x_train[z2]]]
        print(pairs)
        inc = random.randrange(1, 10)
        dn = (d + inc) % 10
        z1, z2 = digit_indices[d][i], digit_indices[dn][i]
        pairs += [[x_train[z1], x_train[z2]]]
        labels += [1, 0]
np.array(pairs), np.array(labels)

# network definition
base_network = create_base_network(input_dim)

input_a = Input(shape=(input_dim,))
input_b = Input(shape=(input_dim,))

base_network.summary()

# because we re-use the same instance `base_network`,
# the weights of the network will be shared across the two branches
processed_a = base_network(input_a)
processed_b = base_network(input_b)

distance = Lambda(euclidean_distance,
                  output_shape=eucl_dist_output_shape)([processed_a, processed_b])

model = Model([input_a, input_b], distance)

model.summary()

# loss is specified manually
# optimizer is RMSprop

tset = [tr_pairs[:, 0], tr_pairs[:, 1]]

len(tset)

len(tr_y)

# train
rms = RMSprop()
model.compile(loss=contrastive_loss, optimizer=rms)
model.fit([tr_pairs[:, 0], tr_pairs[:, 1]], tr_y,
          batch_size=128,
          epochs=epochs,
          validation_data=([te_pairs[:, 0], te_pairs[:, 1]], te_y))

# compute final accuracy on training and test sets
pred = model.predict([tr_pairs[:, 0], tr_pairs[:, 1]])
tr_acc = compute_accuracy(pred, tr_y)
pred = model.predict([te_pairs[:, 0], te_pairs[:, 1]])
te_acc = compute_accuracy(pred, te_y)

print('* Accuracy on training set: %0.2f%%' % (100 * tr_acc))
print('* Accuracy on test set: %0.2f%%' % (100 * te_acc))


I am working on the file "Siamese Network - MNSIT - Breakdown.ipynb".
The first task is:
then we are concatenating them
Can you write Python code for it?

pairs += [[x_test[z1], x_test[z2]]]

len(pairs)


I am working on the file "Sieve of Eratosthenes.ipynb".
The first task is:
output will be global since the above cell is run top level, not inside the scope of a function.  Checking the last few entries below, to confirm everything worked:
Can you write Python code for it?

# verify we have the data we want
print(output[0:10]) # first 10
print(output[-10: -1]) # last 10


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Baseline method (don't change this one ! )
This function implements a basic decimation.
Can you write Python code for it?

def reconstructBaseline(rawSignal,image_shape) :
    reconstructedImage = np.zeros(shape=(image_shape[0],image_shape[1]))
    decimationFactor = 1.0*rawSignal.shape[0]/image_shape[0]

    for i in range(rawSignal.shape[0]):
           for j in range(image_shape[1]):
                reconstructedImage[int(i/decimationFactor)][j] += np.abs(rawSignal[i][j])

    reconstructedImage = normImag(np.abs(reconstructedImage))
    return reconstructedImage


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Envelope detection challenge
Below are the pieces of code related to the proper envelope detection.

Firstly, some useful functions are defined to display, compare images, and allow performance assessment.
A basic decimation algorithm is then implemented. This method will serve as a baseline : you're supposed to do better !
You'll have to define your own envelope detection method
An automated score estimation and comparison between the baseline and your algorithm are provided.

Score estimation and comparison functions
The estimateScore() function computes the error map between reconstructed image and the ground truth, and returns a score associated to this error map, as well as the max error achieved on a given pixel.
You should retain the algorithm that achieves the lowest possible value for these scores.
Can you write Python code for it?

def ssd(A,B):
    A = A - 0.95*A.min()
    A = 1.0*A/A.max()
    B = B - 0.95*B.min()
    B = 1.0*B/B.max()
    squares = (A[:,:] - B[:,:]) ** 2
    return np.sum(squares)

def estimateScore(groundTruth, reconstructedImage) :
    errorMap = (groundTruth - reconstructedImage)
    print('Error map between ground truth and reconstructed image : ')
    showImage(errorMap)
    score = ssd(reconstructedImage,groundTruth)
    maxErr = errorMap.max()
    return [score,maxErr]

def compareImages(im1,im2) :
    plt.figure()
    ax = plt.subplot(1, 2, 1)
    plt.imshow(im1)
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    ax = plt.subplot(1, 2, 2)
    plt.imshow(im2)
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    plt.show()


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Let's analyse this signal in the frequency domain, through a FFT. We should see the image, modulated by the 3.5MHz. That is, a "potato" around a 3.5MHz peak :
Can you write Python code for it?

maxFreq = 6.0E6
xLimit = int(L*maxFreq/sps) # upper cap to
lineFFT = np.abs(np.fft.fft(line))
xScale = range(xLimit)
for i in range(xLimit):
    xScale[i] = (60.0E6)*float(xScale[i])/(L*(1.0E6))
plt.plot(xScale,lineFFT[0:xLimit])
plt.xlabel('Frequency (MHz)')
plt.show()


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Let's assume vertical line points are spaced by 1cm each. This corresponds to a depth of about 13cm.
Can you write Python code for it?

depth = 0.13 # in meters
resolution = groundTruth.shape[0]/depth # in pts/m
t = depth*2.0/1450.0

print('Image resolution in pixels/mm : ', resolution/1000.0)
print('Listening time in micro-secs : ', t*1.0E6)


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Let's check that we have the image (in green) and the corresponding signal (in blue) :
Can you write Python code for it?

line = np.zeros(shape=(L))
imageLine = np.zeros(shape=(L))
for i in range(len(rawSignal)):
    line[i] =  rawSignal[i][10]
    imageLine[i] = BigImg[i][10]
plt.plot(line,"g")
plt.plot(imageLine,"r")
plt.show()


line = np.zeros(shape=(L))
imageLine = np.zeros(shape=(L))
for i in range(len(rawSignal)):
    line[i] =  rawSignal[i][10]
    imageLine[i] = BigImg[i][10]
plt.plot(line)
plt.plot(imageLine)
plt.show()


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Let's check the first "Raw image".
Can you write Python code for it?

FactorNoiseSignal = [0.4, 1.0, 3.0, 5.0]
RandomFactor = 0.2

NoisedImages = []
for k in range(len(FactorNoiseSignal)):

    NoisedImage = []
    NbLines = rawSignal.shape[1]
    for i in range(NbLines):
            line = rawSignal[:, i]
            NoisedImage.append( (np.fft.ifft(np.fft.fft(line)+(FactorNoiseSignal[k]*(1+RandomFactor*random.randint(-10000,10000)/10000.0))*NoisedSignal)) )

    for i in range(NbLines):
        for j in range(100):
            NoisedImage[i][j]= j*NoisedImage[i][j]/(100.0)
            NoisedImage[i][-j]= j*NoisedImage[i][-j]/(100.0)
            # Let's save the raw signal data
    NoisedImage = np.asarray(NoisedImage, dtype=np.float32) # saves on the space required to store images
    np.savez_compressed('files/noisedimages-'+str(FactorNoiseSignal[k])+'.npz', NoisedImage=NoisedImage, FactorNoiseSignal=FactorNoiseSignal[k])
    NoisedImages.append(NoisedImage)

imageToPlot = np.transpose(np.abs(NoisedImages[3]))
plt.figure(figsize=(4, 10))
plt.gray()
plt.imshow(imageToPlot.reshape(imageToPlot.shape), aspect='auto')
plt.show()

plt.plot(np.abs(lineFFT+FactorNoiseSignal[0]*NoisedSignal),"g")
#plt.plot(line[1000:8000],"r")

plt.show()


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Let's compare the image reconstructed with the baseline method, with the ground truth to achieve :
Can you write Python code for it?

reconBaseline = reconstructBaseline(rawSignal,groundTruth.shape)
compareImages(groundTruth, reconBaseline)
[scoreBaseline,maxErrBaseline] = estimateScore(groundTruth, reconBaseline)

print('Score for Baseline method : ', scoreBaseline)
print('max Err between pixels for Baseline method : ', maxErrBaseline)


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Loading and studying the 342x719 image of fantom
Here we load the original image that will serve as "ground truth" that we would like to achieve. This image will later be altered in a way that allows to simulate a raw signal (i.e. the image before envelope detection is performed).
Can you write Python code for it?

im = Image.open("fantom.bmp").convert('L') # convert 'L' is to get a flat image, not RGB
groundTruth = normImag(np.array(im)) # we use the full [0;1] range
showImage(groundTruth)


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Performance assessment of your method
Can you write Python code for it?

recon = reconstructImage(rawSignal,groundTruth.shape)
compareImages(groundTruth, recon)
[score,maxErr] = estimateScore(groundTruth, recon)

print('Score for your method : ', score)
print('max Err between pixels for your method : ', maxErr)


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Submitting your own method to the leaderboard
To submit your own implementation to our leaderboard and compare your performances to other teams, go to http://37.187.117.106:8888/.

Subscribe to the leaderboard
Go to the IDE and paste your code, in the same form as the example provided in the cell below. The code should at least include the definition of a function "run(rawSignal,image_shape)" where :
rawData is a numpy.array containing the raw signal values (in the same format as in this notebook)
imageShape is an array [imageLength, imageWidth] with dimensions of the reconstructed image
the function should return a numpy.array of shape [imageLength, imageWidth] containing the reconstructed image values
It's possible to install python packages via pip, by defining a "install_packages()" function. The imports should then be done in the run function.
Click on the "submit" button.
After some time, a notification will inform you about your score and your ranking will appear in the leaderboard.

You can submit the example code in the cell below. This implementation should lead to a score of 12481.6872689
Can you write Python code for it?

def install_packages():
    import pip
    pip.main(['install', 'scipy'])

def run(rawSignal,image_shape) :

    import numpy as np
    from scipy.signal import hilbert

    reconstructedImage = np.zeros(shape=(image_shape[0],image_shape[1]))
    analytic_signal = hilbert(rawSignal)
    amplitude_envelope = np.abs(analytic_signal)
    decimationFactor = 1.0*amplitude_envelope.shape[0]/image_shape[0]

    old_pixel = 0
    nb_points=0
    for i in range(amplitude_envelope.shape[0]):
        for j in range(image_shape[1]):
            reconstructedImage[int(i/decimationFactor)][j] += np.abs(amplitude_envelope[i][j])

        if (int(i/decimationFactor) == old_pixel):
            nb_points += 1
        else:
            nb_points += 1
            reconstructedImage[int(i/decimationFactor)-1] = reconstructedImage[int(i/decimationFactor)-1]/nb_points
            nb_points = 1
            old_pixel = old_pixel+1

    reconstructedImage = normImag(np.abs(reconstructedImage))

    # The function should return the reconstructed image
    return reconstructedImage


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
The corresponding length of raw signal is close to 11k points.
We can then recreate the raw signal image :
Can you write Python code for it?

# First create a table of L points for each line, from the original image, by using bicubic interpolation
# This is to get a smoother and more realistic raw signal
BigImg = imresize(groundTruth, ( L,groundTruth.shape[1]), interp='bicubic')

# Then simulate raw signal by modulating the data of BigImg with a sinusoidal function,
# the frequence of which corresponds to the piezo frequency
rawSignal = np.zeros(shape=(L,groundTruth.shape[1]))
for i in range(len(rawSignal)):
    for j in range(len(rawSignal[0])):
        pixelValue = 1.0*BigImg[i][j]
        w = 2.0*math.radians(180)*f
        rawSignal[i][j] = pixelValue*math.cos(1.0*i*w/sps)


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
The corresponding resolution is 5.53 pts / mm. At a speed of 1450m/s for sound, we'd have a listening time of around 180µs of recording.
Simulating a raw signal that would lead to this "ground truth" image
Let's assume an ADC sampling rate of 60Msps (close to our prototype) and a piezo frequency f = 3.5 MHz, and compute the length of the raw signal :
Can you write Python code for it?

sps = 60.0E6
f = 3.5E6
L = int(t*sps)

print("Number of points in raw signal : ", L)


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Type of noise
The curve above, showing the FFTs of both signal and noise, show that the "blue" signal, ie the noise, follows a type of pink noise https://en.wikipedia.org/wiki/Pink_noise
Can you write Python code for it?

plt.plot(xScale,lineFFT[0:xLimit]+NoisedSignal[0:xLimit])
plt.xlabel('Frequency (MHz)')
plt.show()

FactorNoiseSignal = 0.33
# In blue the recontructed signal
TotalSignal = (np.fft.ifft(lineFFT+FactorNoiseSignal*NoisedSignal))

for i in range(100):
    TotalSignal[i]= i*TotalSignal[i]/(100.0)

plt.plot(TotalSignal[0:1000],"b")
# In green, we have the original signal
OriginalSignal = np.real(np.fft.ifft(lineFFT))
plt.plot(OriginalSignal[0:1000],"g")
#plt.plot(line[1000:8000],"r")

plt.show()


I am working on the file "SigProc_101/SigProc-101-pimped.ipynb".
The first task is:
Your turn : implement your own method in the function below
Can you write Python code for it?

def reconstructImage(rawSignal,image_shape) :
    # Here is a copy of the baseline method. Replace that by another method.
    reconstructedImage = np.zeros(shape=(image_shape[0],image_shape[1]))
    decimationFactor = 1.0*rawSignal.shape[0]/image_shape[0]

    for i in range(rawSignal.shape[0]):
           for j in range(image_shape[1]):
                reconstructedImage[int(i/decimationFactor)][j] += np.abs(rawSignal[i][j])

    reconstructedImage = normImag(np.abs(reconstructedImage))
    # The function should return the reconstructed image
    return reconstructedImage


I am working on the file "Signal Tutorial.ipynb".
The first task is:
But this is not the only choice. Conceivably, one might want to add into sender_strats a type of senders following a mixed strategy, or ditto for the receiver. One might add such possibilities by hand into the strats arrays, or delete a number of pure strats, or create a strats array from scratch, etc.
Simulating Evolution
Once we have the two strats arrays we want, we can create a population game. From here on game.Evolve takes over:
Can you write Python code for it?

simple_evo = game.Evolve(simple_game, sender_strats, receiver_strats)


I am working on the file "Signal Tutorial.ipynb".
The first task is:
Close enough!
Games without chance player
Let's now work with a game in which the sender has an endogenously generated state. I.e., a sender type will now be individuated by a state together with a probability vector over the set of messages. Receivers are as in the games with chance player discussed above:
Can you write Python code for it?

import imp
imp.reload(game)
imp.reload(analyze)

sender = np.identity(3)
receiver = np.identity(3)

simple_nonchance = game.NonChance(sender, receiver, 3)

sender_strats = simple_nonchance.sender_pure_strats()
receiver_strats = simple_nonchance.receiver_pure_strats()

avgpayoff = simple_nonchance.avg_payoffs(sender_strats, receiver_strats)

nc_evolve = game.Evolve(simple_nonchance, sender_strats, receiver_strats)

sender_init = nc_evolve.random_sender()
receiver_init = nc_evolve.random_receiver()
times = np.arange(1000)

results = nc_evolve.replicator_odeint(sender_init, receiver_init, times)

plt.plot(results);

sender_final, receiver_final = nc_evolve.vector_to_populations(results[-1])
# This splits the vector that the solver outputs into two population vectors
sender_final_strat = nc_evolve.sender_to_mixed_strat(sender_final)
receiver_final_strat = nc_evolve.receiver_to_mixed_strat(receiver_final)
print("{}\n\n{}".format(sender_final_strat, receiver_final_strat))

ci = analyze.CommonInterest(simple_nonchance)

ci.C_chance()

simple_nonchance.sender_payoff_matrix.dot(receiver_for_sender)

payoffs = np.arange(9).reshape(3,3)

payoffs

senderstrat[0][:, None] * payoffs.dot(np.array([1/3, 0, 2/3])[:, None])

simple_evo.sender_to_mixed_strat(receiver_final)

final_info


I am working on the file "Signal Tutorial.ipynb".
The first task is:
Right now, Signal calculates the two-population replicator(-mutator) dynamics, in continuous or discrete time. It is able to use both scipy.integrate.odeint and scipy.integrate.ode for this. Let's evolve those two random initial population vectors, following the replicator dynamics in continuous time, using odeint:
Can you write Python code for it?

results = simple_evo.replicator_odeint(sender_init, receiver_init, times)


I am working on the file "Signal Tutorial.ipynb".
The first task is:
Sender and receiver are engaged in a signaling system. We can also check directly the mutual info between states and acts in the final snapshot:
Can you write Python code for it?

import analyze
info = analyze.Information(simple_game, winning_sender, winning_receiver)
info.mutual_info_states_acts()


I am working on the file "Signal Tutorial.ipynb".
The first task is:
So, 3 equiprobable (as per state_chances) states and 3 acts (that's why the payoff matrices are square), and sender and receiver get 1 payoff unit when act $A_i$ is performed in state $S_i$, 0 otherwise. That's why both payoff matrices are the identity matrix.
What we now need is to decide which strategies will each type in the sender/receiver populations play, and calculate the average payoff for each combination of sender and receiver strategies. s.Chance takes care of that:
Can you write Python code for it?

simple_game = game.Chance(state_chances, sender, receiver, 3) # The 3 gives the number of available messages


I am working on the file "SignalProcessing/Complex numbers.ipynb".
The first task is:

Euler's formula

We can use $m e^{j\phi}$, where $m$ is the distance to the origin, and $\phi$ the angle with the ordinal axis to express any irrational number in a circle with radius m. This alternative way to a complex numbers turns out to be very useful, because there are many cases, like multiplication, where it is easier to deal with exponentials than with standard complex numbers.
Euler's formula also provides a link between trigonometry and imaginary numbers.
$$e^{ \pm j\theta } = \cos \theta \pm j\sin \theta$$
It describes the vector from the origin to some point of the circle with radius one. If you want to have larger than one vector, we can use:
$$ m e^{ \pm j\theta } = m \cos \theta \pm m j\sin \theta$$
Then, we can have complex waves simply expressed as $m e^{j\theta }$
Can you write Python code for it?

# trigonometric relation
m = np.abs(z)
phi = np.angle(z)

x = m*np.cos(phi)
y = m*np.sin(phi)
print('({} + {}j) is {}e^{:2.4f}j '.format(z.real, z.imag, m, phi))


I am working on the file "SignalProcessing/Complex numbers.ipynb".
The first task is:
A complex number give us the distance to the (0,0) origin and the angle to the positive axis
Can you write Python code for it?

z = 1+1j # alternative way to complex numbers
dist = abs(z)
angle = np.angle(z, deg = True)

print('distance = {:2.4f}, angle = {} deg'.format(dist, angle))

# distance with trigonometry formula
dist = np.sqrt( np.power(z.imag,2) + np.power(z.real,2))

# angle in radiants
angle = np.rad2deg(np.arctan(z.imag/z.real))
print('distance = {:2.4f}, angle = {} deg'.format(dist, angle))


I am working on the file "SignalProcessing/Complex numbers.ipynb".
The first task is:
If we multiply a complex number by its conjugate, we obtain a new complex number, whose module is the square of the complex number module
Can you write Python code for it?

z2 = z * z.conjugate()
np.abs(np.sqrt(z2))


I am working on the file "SignalProcessing/EEG Spectrogram.ipynb".
The first task is:
EEG example
From a patient in a dark room in a resting state. The recording is from the course Master the Fourier transform and its applications, see the link here from Mike X.
Can you write Python code for it?

data = loadmat(file_name = './data/EEGrestingState.mat')
# transpose matrix and convert to 1D NumPy array
eeg = data['eegdata'].squeeze() # amplitude in microVolts
sr = int(data['srate']) # sampling rate in sec
time = np.arange(eeg.size)/sr
print('Sampling rate = %d samples/sec'%sr)

# plot the time course of the EEG
fig, ax = plt.subplots(2,1, figsize=(16,4), sharey=True)

ax[0].plot(time,eeg, lw=1)
ax[0].set_xlabel('Time (sec)'), ax[0].set_ylabel('Voltage ($\mu$Volts)');
ax[0].set_xticks(arange(0,130,60));

ax[1].plot(time, eeg, lw=1, color='k')
ax[1].set_xlim(42.5,45)
#ax[1].set_xlim(12,14.5)
ax[1].set_xticks(arange(43,45,1));
ax[1].set_xlabel('Time (sec)');

# Fourier transform
FourierCoeff = np.fft.fft(eeg)/eeg.size
DC = [np.abs(FourierCoeff[0])]
amp = np.concatenate((DC, 2*np.abs(FourierCoeff[1:])))

# compute frequencies vector until half the sampling rate
Nyquist = sr/2
print('Nyquist frequency = %2.4f Hz'%Nyquist)
Nsamples = int( math.floor(eeg.size/2) )
hz = np.linspace(0, Nyquist, num = Nsamples + 1 )
dhz = hz[1]
print('Spectral resolution = %2.4f Hz'%hz[1])

# Perform Welch's periodogram
segment = int( 4*sr )
myhann = signal.get_window('hann', segment)

# obtain the power (uV^2) spectrum with Hann window and 50% overlap
myparams = dict(fs = sr, nperseg = segment, window = myhann, noverlap = segment/2,
                scaling = 'spectrum', return_onesided = True)
freq, ps = signal.welch(x = eeg, **myparams)# units uV**2
ps = 2*ps

# obtain the power density/Hz (uV^2) spectrum with Hann window and 50% overlap
# to get back to simply power, divide by the segment lenght in seconds (four in our case)
myparams2 = dict(fs = sr, nperseg = segment, window = myhann, noverlap = segment/2,
                 scaling = 'density', return_onesided = True)
freq, psd = signal.welch(x = eeg, **myparams2)# units uV**2/Hz
psd = 2*psd

dfreq = freq[1]
print('Spectral resolution = %2.4f Hz'%dfreq)

# Plot the power spectrum

fig, ax = plt.subplots(1, 2, figsize=(16, 4))

ax[0].set_title("Amplitude spectrum (Fourier...