I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
Originally I was using sci-kit learn and tfidf_vectorizer as my method for calcualting the cosine similarity. As per normal, sci-kit learn makes implementation of machine learning models fairly trivial.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics.pairwise import linear_kernel
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
tf = TfidfVectorizer(stop_words=stop)
tfidf_matrix = tf.fit_transform(df['transcribed'])
tfidf_matrix


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
That creates a tfidf_matrix of podcast to terms (151 podcasts and 35028 terms). Then we create the cosine similarities:
Can you write Python code for it?

cosine_similarities = linear_kernel(tfidf_matrix, tfidf_matrix)
print cosine_similarities
print len(cosine_similarities)


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
Timing:
Can you write Python code for it?

%timeit get_related_podcasts_query_scikit('economics math statistics',5)

%timeit get_related_podcasts_scikit(22,5)


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
What the code above is doing, is taking the the small transcription of a text file, removing useless words (commonly known as stopwords, NLTK has a module for 2400 stopwords, and grouping by word then comparing the differences by taking the word counts for CMU and comparing that to Watson. 143 of the words had the same count, which is a measure of 76% (143 / 186). That was enough for me to go with CMU, as I could also parallelize CMU to run multiple podcasts at a time and not pay for API calls to transcribe.
Once I had my set of podcasts, I decided that I would first attempt to transcribe locally. This actaully took quite a long time as I ended up having issues with my mac falling asleep or the process crashing due to running out of memory. I finalized on separating out the files based on the series and individually running the transcription using the command line. This took a long time as I would run the script before work, after work and going to bed. Moving forward, as this project is something I am continuing to build out and use, I will be using Spark and AWS as an option for scaling transcription.
To Transform a full podcast I had to do several things:

Take the mp3 and turn it into a wav. I used ffmpeg and the speech_recognition library
Slice up each wav file into 300 seconds
Transcribe those 300 seconds into a text file
Save the transcription into an array of text files and concatenate the files together
Remove the wav files as they take up a lot of memory.
Can you write Python code for it?

#example code

from pydub import AudioSegment
import glob
from math import ceil
import os
import json
import csv
import sys
import speech_recognition as sr
r = sr.Recognizer()

def transcribe_mp3(AUDIO_FILENAME, AUDIO_SEGMENT_SECONDS):
    output_file_name = "{}_translation.txt".format(AUDIO_FILENAME)
    #fuction to transform mp3 file to wav for transcription
    try:
        def transform_mp3_wav(AUDIO_FILENAME, AUDIO_SEGMENT_SECONDS):
            filename = AUDIO_FILENAME.replace('.mp3','')
            with open(AUDIO_FILENAME):
                audio = AudioSegment.from_mp3(AUDIO_FILENAME)
                xs = 0
                while xs < audio.duration_seconds:
                    ys = min(xs + AUDIO_SEGMENT_SECONDS, ceil(audio.duration_seconds))
                    fname = str(xs).rjust(5, '0') + '-' + str(ys).rjust(5, '0') + '.wav'
                    audio[xs*1000:ys*1000].export(os.getcwd() + '/' + filename + fname, format='wav')
                    print("Saved", fname)
                    xs = ys
        transform_mp3_wav(AUDIO_FILENAME, 300)
        wav_filename = AUDIO_FILENAME.replace('.mp3','.wav')
        wav_list = glob.glob('*.wav')
        wav_list = filter(lambda x: '.mp3' not in x, wav_list)
        trans_list = []
        transcription = None
        for wav_file in wav_list:
            print 'transcribing: ' + wav_file
            with sr.AudioFile(wav_file) as source:
                audio = r.record(source)
                transcription = r.recognize_sphinx(audio)
                print 'transcription completed'
            trans_list.extend(transcription)

        transcription = ''.join(trans_list)
    except:
        return 'error'

    for f in wav_list:
        os.remove(f)

    file = open(output_file_name,'w')
    file.write(transcription)
    file.close()


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
You can see above that we were able to use our linear kernel from scikit learn to transform the tfidf_matrix and get the cosine similarities for every podcast related to every podcast!
Now we can put this into functions that can be usable for a user and future web app:
Can you write Python code for it?

def get_related_podcasts_scikit(podcast_number,number_of_similarities):
    cosine_similarities = linear_kernel(tfidf_matrix, tfidf_matrix)
    related_pod_index = cosine_similarities.argsort()[podcast_number][::-1]
    pod_dict = dict(zip(range(0, len(related_pod_index)),related_pod_index))
    pod_dict = pd.DataFrame({'rank':pod_dict.keys()},index=pod_dict.values())
    related_podcasts_df = pd.DataFrame.join(pod_dict, df, how='inner')
    final_df = related_podcasts_df.sort_values('rank')[0:number_of_similarities+1][['rank','episode','series']]
    return final_df

def get_related_podcasts_query_scikit(query, number_of_similarities):
    query = query.lower()
    query = query.split()
    tfidf_matrix_test = tf.fit_transform(query)
    tfidf_matrix_train = tf.transform(df['transcribed'])
    tfidf_matrix_train.todense()
    tfidf_matrix_test.todense()
    query_similarities = linear_kernel(tfidf_matrix_test, tfidf_matrix_train)
    query_similarities = query_similarities.argsort()[0][::-1]
    pod_dict = dict(zip(range(0, len(query_similarities)),query_similarities))
    pod_dict = pd.DataFrame({'rank':pod_dict.keys()},index=pod_dict.values())
    related_podcasts_df = pd.DataFrame.join(pod_dict, df, how='inner')
    final_df = related_podcasts_df.sort_values('rank')[0:number_of_similarities+1][['rank','episode','series']]
    return final_df


I am working on the file "Poem_Segmentation_Demo_Python/khan_segmentation.ipynb".
The first task is:
An initial study of topical poetry segmentation
This work investigates topical segmentation of poetry to better understand its interpretation humans.  Nine segmentations of the poem titled Kubla Khan (Coleridge, 1816) were collected; a small number, but enough to inform a future larger study and to obtain feedback upon the methodologies used.
Chris Fournier. 2013. An initial study of topical poetry segmentation. Proceedings of the Second Workshop on Computational Linguistics for Literature, pp. 47-51. Association for Computational Linguistics, Stroudsburg, PA, USA.
Can you write Python code for it?

# Import required libraries
import os
import csv
import segeval as se
import numpy as np
import matplotlib.pyplot as plt
import itertools as it
from collections import defaultdict
from decimal import Decimal
from hcluster import linkage, dendrogram, fcluster


I am working on the file "Poem_Segmentation_Demo_Python/khan_segmentation.ipynb".
The first task is:
Coder analysis
Having looked at subsets of coders, it would be informative to also analyze coder behaviour overall.
Plot boundary placement frequency
To visualize coder behaviour, this plot indicates the frequency at which various coders placed boundaries in this document.
Can you write Python code for it?

# Plot boundaries per coder
y = coder_boundaries
x = np.arange(len(y))

# Set up
width = 0.75
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

# Plot
rects = ax.bar(x, y, width, color='0.75')

# Add xticks
ax.set_xticks(x + (width / 2))
ax.set_xticklabels([str(val) for val in labels])

# Draw mean lines
xmin, xmax, ymean, ystd = -0.25, len(labels), np.mean(y), np.std(y)
ax.plot([xmin, xmax], [ymean] * 2, color='k') # Draw mean
ax.plot([xmin, xmax], [ymean + ystd] * 2, color='0.5') # Draw +std
ax.plot([xmin, xmax], [ymean - ystd] * 2, color='0.5') # Draw -std

# Add numbers to bars
format_str='%d'
fnc_value=int
for rect in rects:
    height = rect.get_height()
    ax.text(rect.get_x() + rect.get_width() / 2., 1.05 * height, format_str%fnc_value(height), ha='center', va='bottom')

# Format
ax.set_xlim([-0.25, 9])
ax.set_ylim([0, 30])
ax.set_xlabel('Coder')
ax.set_ylabel('Boundaries placed (quantity)')
plt.show()


I am working on the file "Poem_Segmentation_Demo_Python/khan_segmentation.ipynb".
The first task is:
Compute descriptive statistics
Two descriptive statistics are used to analyse the codings that the coders produced:

Boundary Similarity (B) to analyse the boundaries placed by segmenters; and
Jaccard Similarity (J) of the labels describing the segments (where the labels for each segment are placed upon each line before computing similarity).

Boundary SImilarity is described in:
Chris Fournier. 2013. Evaluating Text Segmentation using Boundary Edit Distance. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Stroudsburg, PA, USA.
Can you write Python code for it?

# Compute boundaries
boundaries = dict([(key, len(mass) - 1) for key, mass in dataset[item_name].items()])
coder_boundaries = [boundaries[coder] for coder in coders]

# Compute similarities (1-B)
similarities = se.boundary_similarity(dataset, one_minus=True)

# Expand segment labels using the mass of each segment to create
# a one to one mapping between line and segment label
expanded_segment_labels = defaultdict(list)
for coder in coders:
    masses = dataset[item_name][coder]
    coder_segment_labels = segment_labels[coder]
    expanded_segment = list()
    for mass, coder_segment_label in zip(masses, coder_segment_labels):
        expanded_segment.extend(list([coder_segment_label]) * mass)
    expanded_segment_labels[coder] = expanded_segment

# Define label similarity function
def jaccard(a, b):
    return float(len(a & b)) / float(len(a | b))

# Compute overall label Jaccard similarities per position
total_similarities = list()
row_similarities = list()
for i in xrange(0, number_of_lines):
    parts = list()
    for coder in coders:
        parts.append(set(expanded_segment_labels[coder][i].split('/')))
    part_combinations = it.combinations(parts, 2)
    position_similarities = [jaccard(a, b) for a, b in part_combinations]
    total_similarities.extend(position_similarities)
    row_similarities.append(position_similarities)


I am working on the file "Poem_Segmentation_Demo_Python/khan_segmentation.ipynb".
The first task is:
Compute statistics for each cluster
Given the clusters produced above, let's calculate statistics for each cluster.
Can you write Python code for it?

cluster_members = {
    '0,2' : [coders[0], coders[2]],
    '1,0,2' : [coders[1], coders[0], coders[2]],
    '4,7' : [coders[4], coders[7]],
    '1,0,2,4,7' : [coders[1], coders[0], coders[2], coders[4], coders[7]],
    '6,8' : [coders[6], coders[8]],
    '5,6,8' : [coders[5], coders[6], coders[8]],
    '3,5,6,8' : [coders[3], coders[5], coders[6], coders[8]]
}

cluster_pi = dict()
cluster_b  = dict()
cluster_j  = dict()
for cluster, members in cluster_members.items():
    data = {coder : dataset[item_name][coder] for coder in members}
    dataset_subset = se.Dataset({item_name : data})
    cluster_b[cluster]  = [float(value) for value in se.boundary_similarity(dataset_subset, n_t=2).values()]
    cluster_pi[cluster] = float(se.fleiss_pi_linear(dataset_subset, n_t=2))
    position_j = list()
    for i in xrange(0, number_of_lines):
        parts = list()
        for coder in members:
            parts.append(set(expanded_segment_labels[coder][i].split('/')))
        part_combinations = it.combinations(parts, 2)
        position_similarities = [jaccard(a, b) for a, b in part_combinations]
        position_j.extend(position_similarities)
    cluster_j[cluster] = position_j

print 'Cluster\t\tPi\tB\t\t\tJ'
for cluster in cluster_members.keys():
    print '{0}\t{1:.4f}\t{2:.4f} +/- {3:.4f}, n={4}\t{5:.4f} +/- {6:.4f}, n={7}'.format(cluster if len(cluster) > 7 else cluster+'\t',
        np.mean(cluster_pi[cluster]),
        np.mean(cluster_b[cluster]),
        np.std(cluster_b[cluster]),
        len(cluster_b[cluster]),
        np.mean(cluster_j[cluster]),
        np.std(cluster_j[cluster]),
        len(cluster_j[cluster]))


I am working on the file "Poem_Segmentation_Demo_Python/khan_segmentation.ipynb".
The first task is:
Load and define data
Nine Master Tukers were recruited using from the United States and were asked to segment the poem into topically contiguous segments at the line level.  They were also asked to produce one-sentence summaries of each segment.
Data

The segmentations themselves were saved within the file kubla_khan_fournier_2013.json
The principle researcher read these summaries and attempted to label the type of segments that the Turkers produced which is saved as labels.csv.

To later perform comparisons in an order that hcluster expects, an ordered list of coders is defined herein named coders.
Can you write Python code for it?

# Document to analyse
item_name = u'kublakhan'
number_of_lines = 54

# Ordered list of coders (and numeric list of coders) used to relate
# numbered cluster coders to other graphs
coders = ['AWRAXV1RIYR0M', 'A23S6QOSZH9TMT', 'A21IFZJ0EDKM4E', 'AO3XB5I5QNNUI', 'A3RLCGRXA34GC0', 'A21SF3IKIZB0VN', 'APXNY64HXO08K', 'AM155T4U3RE1A', 'A2YBGZ2H2KSO5T']
labels = ['%i' % i for i in range(0, len(coders))]

# Load segmentation dataset
filepath = os.path.join('data', 'kubla_khan_fournier_2013.json')
dataset = se.input_linear_mass_json(filepath)

# Load labels
segment_labels = dict()
filepath = os.path.join('data', 'kubla_khan_fournier_2013', 'labels.csv')
with open(filepath) as csv_file:
    reader = csv.reader(csv_file, delimiter=',')
    for row in reader:
        segment_labels[row[0]] = [item.strip() for item in row[1:]]


I am working on the file "Poem_Segmentation_Demo_Python/khan_segmentation.ipynb".
The first task is:
Subset analysis
The overall statistics show that the 9 coders have low agreement regardless of the metric used.
Cluster segmentations by boundary similarity
Hypothesis: Subsets of the coders may agree better with eachother.
To explore this hypothesis, the similarities of the boundaries placed within each segmentation (1-B) were used as a distance function to perform hierarchical agglomerative clustering.  Each cluster can then be analyzed.
Can you write Python code for it?

# Order distances for clustering
coder_combinations = [list(a) for a in it.combinations(coders, 2)]
for coder_combination in coder_combinations:
    coder_combinations.reverse()
keys = list()
for a in coder_combinations:
    a = list(a)
    key = ','.join([item_name] + a)
    if key not in similarities:
        a.reverse()
    key = ','.join([item_name] + a)
    keys.append(key)
distances = [similarities[key] for key in keys]

# Cluster
aglomerative_clusters = linkage(distances, method='complete')
dendro = dendrogram(aglomerative_clusters, labels=labels)
plt.ylabel('Mean Distance (1-B)')
plt.xlabel('Coder')
plt.show(dendro)


I am working on the file "Poisson Editing/SeamlessCloning_Sample/SeamlessImageCloningGeometric.ipynb".
The first task is:
compute seamless clone (red)
Can you write Python code for it?

import scipy
import scipy.sparse
import scipy.sparse.linalg

ref_greyscale    = ref[:,:,0].copy()
target_greyscale = target[:,:,0].copy()

X_current              = numpy.zeros_like(mask, dtype=numpy.float32)
X_next                 = numpy.zeros_like(mask, dtype=numpy.float32)
X_current[border == 1] = ref_greyscale[border == 1]
X_next[border == 1]    = ref_greyscale[border == 1]

num_iterations  = 1500
print_frequency = 25

for n in range(num_iterations):

    if n % print_frequency == 0:
        print n
        seamless_clone_greyscale = ref_greyscale.copy()
        seamless_clone_greyscale[strict_interior_indices] = X_current[strict_interior_indices]
        scipy.misc.imsave("%d.png" % n,  seamless_clone_greyscale / 255.0)

    for i in range(num_strict_interior_pixels):

        y = strict_interior_indices[0][i]
        x = strict_interior_indices[1][i]

        x_right = x+1
        x_left  = x-1

        y_up    = y-1
        y_down  = y+1

        x_neighbors = []
        y_neighbors = []

        if x_right < rnx:
            y_neighbors.append(y)
            x_neighbors.append(x_right)

        if y_up >= 0:
            y_neighbors.append(y_up)
            x_neighbors.append(x)

        if x_left >= 0:
            y_neighbors.append(y)
            x_neighbors.append(x_left)

        if y_down < rny:
            y_neighbors.append(y_down)
            x_neighbors.append(x)

        y_neighbors               = numpy.array(y_neighbors)
        x_neighbors               = numpy.array(x_neighbors)
        strict_interior_neighbors = (strict_interior[(y_neighbors,x_neighbors)] == 1).nonzero()
        border_neighbors          = (strict_interior[(y_neighbors,x_neighbors)] == 0).nonzero()
        num_neighbors             = y_neighbors.shape[0]

        sum_X_current_strict_interior_neighbors = numpy.sum(X_current[(y_neighbors[strict_interior_neighbors],x_neighbors[strict_interior_neighbors])])
        sum_vq       = (num_neighbors * target_greyscale[y,x]) - numpy.sum(target_greyscale[(y_neighbors,...


I am working on the file "Poisson Editing/SeamlessCloning_Sample/SeamlessImageCloningGeometric.ipynb".
The first task is:
compute strict interior and border regions
Can you write Python code for it?

import skimage
import skimage.morphology

strict_interior            = skimage.morphology.erosion(mask, numpy.ones((3,3), dtype=numpy.uint8))
strict_interior_indices    = strict_interior.nonzero()
num_strict_interior_pixels = strict_interior_indices[0].shape[0]
border                     = mask - strict_interior

figsize(9,4)

matplotlib.pyplot.subplot(121);
matplotlib.pyplot.imshow(strict_interior, interpolation="nearest");
matplotlib.pyplot.title("strict_interior");

matplotlib.pyplot.subplot(122);
matplotlib.pyplot.imshow(border, interpolation="nearest");
matplotlib.pyplot.title("border");


I am working on the file "Pokemon Analysis.ipynb".
The first task is:
1/3 of the Flying Pokemons are legendaries, whereas none of the Fighting and Poison types are legendaries.
Which type looks like what ?
Can you write Python code for it?

pokemons = pk.groupby("type1").median()
pokemons_height_weight = pokemons[["height_m", "weight_kg"]]

plt.rcParams['figure.figsize'] = [15, 8]

ax = pokemons.plot.scatter(x='weight_kg', y='height_m')
for i, txt in enumerate(pokemons.index):
    ax.annotate(txt, (pokemons.weight_kg.iat[i],pokemons.height_m.iat[i]))
plt.show()


I am working on the file "Pokemon Analysis.ipynb".
The first task is:
Fill missing values
Can you write Python code for it?

pk["height_m"] = pk["height_m"].fillna(pk["height_m"].median())
pk["percentage_male"] = pk["percentage_male"].fillna(-1)
pk["weight_kg"] = pk["weight_kg"].fillna(pk["weight_kg"].median())

pk.describe()


I am working on the file "Pokemon Analysis.ipynb".
The first task is:
Pikachu
Can you write Python code for it?

X_predict = pk.loc[[25]]
remove_unnecessary_features(X_predict)
type_encoder = label_encode(X_predict, "type1")
del X_predict["is_legendary"]
classifier.predict(X_predict)


I am working on the file "Pokemon Analysis.ipynb".
The first task is:
Split train and test
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(X, y)


I am working on the file "Pokemon Analysis.ipynb".
The first task is:
The Ice type seems to be the most vulnerable type, in part because Flying Pokemons are really powerful against Ice Pokemons.
But the type which has most enemies best against it is the Ground type.
How many Pokemon of each type are legendaries ?
Can you write Python code for it?

legendary_type = pk[["type1", "is_legendary"]]
legendaries = legendary_type[legendary_type["is_legendary"] == 1]

chance_of_legend = legendaries.groupby("type1").count() / legendary_type.groupby("type1").count()
chance_of_legend = chance_of_legend.sort_values("is_legendary", ascending=False)
chance_of_legend.plot(kind="bar", title="Proportion of legendaries Pokemon of each (Primary) type")
plt.show()


I am working on the file "Pokemon Analysis.ipynb".
The first task is:
There are a lot more Water and Normal Pokemons than other types.
Influence of the Primary type on the capture rate
Can you write Python code for it?

capture_type = pk[["capture_rate", "type1"]]
capture_type = capture_type.groupby("type1").median().sort_values("capture_rate", ascending=False)
capture_type.plot(kind="bar", title="Capture rate for each (Primary) type")
plt.show()


I am working on the file "Pokemon Analysis.ipynb".
The first task is:
Try to predict legendaries Pokemons
Preprocessing
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn import preprocessing

def remove_unnecessary_features(df):
    del df["abilities"]
    del df["name"]
    del df["japanese_name"]
    del df["classfication"]
    del df["type2"]

def label_encode(df, column):
    label_encoder = preprocessing.LabelEncoder()
    label_encoder.fit(df[column])
    df[column] = label_encoder.transform(df[column])
    return label_encoder

X = pk.copy()
X.drop([25, 149, 150]) # Let's remove some Pokemons to try and predict them later
remove_unnecessary_features(X)
type_encoder = label_encode(X, "type1")

y = X.copy()["is_legendary"]
del X["is_legendary"]


I am working on the file "Pokemon Analysis.ipynb".
The first task is:
Which type is best against which
Can you write Python code for it?

best_against = pk.groupby("type1").median()
best_against = best_against[['against_bug', 'against_dark', 'against_dragon',
       'against_electric', 'against_fairy', 'against_fight', 'against_fire',
       'against_flying', 'against_ghost', 'against_grass', 'against_ground',
       'against_ice', 'against_normal', 'against_poison', 'against_psychic',
       'against_rock', 'against_steel', 'against_water']].transpose()

best_against.style.highlight_max()

best_against.sum().sort_values(ascending=False)


I am working on the file "Pokemon/Pokemon - Team Composition Analysis.ipynb".
The first task is:
Archenemies
Below you can find an analysis that seeks to point out which pokemons you should be aware of when battling, this analysis takes into account the type matchups of your team
Can you write Python code for it?

# Let's get you 3 worst attack and defense matchups
att_w = attack_matchups.columns[-3:].tolist()
def_w = defense_matchups.columns[-3:].tolist()
archenemy = pokemon_data.query("Type1 in @att_w or Type2 in @att_w")
archenemy.query("Type1 in @def_w or Type2 in @def_w")


I am working on the file "Pokemon/Pokemon - Team Composition Analysis.ipynb".
The first task is:
Status Analysis
Here you can see an analysis for your team's stats, in the graph below you see how each of your team's pokemons compares to the average status
Can you write Python code for it?

plt.figure(figsize=(10,5))
plt.title("Status values for your team")
sns.swarmplot(x="Stat", y="value", data=status_data, hue="Name")
plt.ylabel("Value")
plt.xlabel("Status")
colors = ['r','g','b','y','c','k']
for stat,c in zip(status,colors):
    plt.hlines(team_data[stat].mean(),-1,6, linestyles='dotted', color=c, label="%s Avg"%stat)
plt.legend(loc=(1.05,0.05))
plt.show()
print(pokemon_data[status].mean())
print(team_data[status].mean())

fig = plt.figure( figsize=(4,4))
ax = plt.subplot(111)
sns.heatmap(team_data[status], cmap='gist_heat_r')
plt.yticks(range(6), team_data["Name"][::-1], rotation=0)
plt.xticks(rotation=90)
plt.show()


I am working on the file "Pokemon/Pokemon - Team Composition Analysis.ipynb".
The first task is:
Types Matchups
The graphs below show your team's score in both attack and defense matchup, this is calculated as the sum of multipliers of your team against each type for both attack and defense, then the defense score is calculated as 24 (the max possible score for a given type) minus the sum
Can you write Python code for it?

plt.figure(figsize=(10,3))
plt.subplot(121)
plt.suptitle("Types Matchups")
plt.title("Attack Matchups")
plt.xlabel("Defending Type")
plt.ylabel("Score")
plt.xticks(range(18), attack_matchups.columns,rotation=65)

att_color = lambda x:(1,0.3,0.3) if x <= 5 else (0.2,0.3,1) if x <= 7 else (0.3, 0.8, 0.1)
def_color = lambda x:(1,0.3,0.3) if x > 8 else (0.2,0.3,1) if x >= 6 else (0.3, 0.8, 0.1)

for x, v in zip(range(18), attack_matchups.values[0]):
    plt.bar(x,v, color=att_color(v))
plt.subplot(122)
plt.title("Defense Matchups")
plt.xlabel("Attacking Type")
plt.xticks(range(18), defense_matchups.columns[::-1],rotation=65)
for x, v in zip(range(18), defense_matchups.values[0][::-1]):
    plt.bar(x,24-v, color=def_color(v))

plt.show()


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
And next, let's see the distribution of standard deviations:
Can you write Python code for it?

df['dist_std'].hist(bins=50);
pl.title('Distribution of standard deviation for IMDB ratings', fontsize=20);


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
Distribution of ratings
IMDB allows people to give a rating between 1 and 10 for each movie. Let's take a look at the rating distribution across all the movies in the dataset:
Can you write Python code for it?

df['rating'].hist(bins=len(df['rating'].unique())-1)
pl.title('Rating distribution', fontsize=20)
print('Mean %.2f' % df['rating'].mean(), 'and median', df['rating'].median());


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
Finding polarizing movies
Let's take a look at the rating distribution around the average for the movies in the dataset.
Can you write Python code for it?

"""
REPORT FORMAT
=============

     In this list, movies have been rated on a scale of 1 to 10, 10 being
good and 1 being bad.  For each movie, the total number of votes, the
average rating, and the vote distribution are shown.  New movies are indicated
by a "*" before their entry.

     The vote distribution uses a single character to represent the percentage
of votes for each ranking.  The following characters codes can appear:

     "." no votes cast        "3" 30-39% of the votes  "7" 70-79% of the votes
     "0"  1-9%  of the votes  "4" 40-49% of the votes  "8" 80-89% of the votes
     "1" 10-19% of the votes  "5" 50-59% of the votes  "9" 90-99% of the votes
     "2" 20-29% of the votes  "6" 60-69% of the votes  "*" 100%   of the votes
"""
p_dist_att = re.compile('[\d*.]{10}')

def p_to_prop(p):
    if p == '.':
        r = 0.
    elif p == '*':
        r = 1.
    else:
        r=int(p)*0.1
    return r

def dist_to_props(dist):

    if type(dist) is float:
        dist = int(dist)
    dist = str(dist)

    if len(dist) < 10:
        dist = '0'*(10-len(dist))+dist

    # testing if the dist ahere to the format:
    if p_dist_att.match(dist) and len(dist) == 10:
        props = np.array([p_to_prop(p) for p in dist])
        # adding what is not accounted for when we take
        # the lower bound of all the percentages:
        # i.e. 10%-19%: take it as 10%
        if props.sum() > 1.:
            print('This is not a valid distribution %s' % dist)
            props = None
        else:
            props += (1. - props.sum())/10.
    else:
        props = None

    return props

if run_tests:
    print(dist_to_props('4310000000') is not None)
    print(dist_to_props('4378234782234') is None)
    print(dist_to_props('234') is not None)
    print(dist_to_props(21100121.0) is not None)
    # This one should fail but it is OK for now
    print(dist_to_props('.........*') is not None  )
    print(dist_to_props('..1......*') is None) # sum to > 1

def props_to_avg(props):
   ...


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
Here number one, Shahrzad, is a movie that especially young people love and elderly people dislike quite a lot.
Least polarizing movies:
Let's also check out which movies are the least polarizing.
Can you write Python code for it?

showdf(
    df[['Title', 'rating', 'year', 'votes', 'dist_std']].sort_values('dist_std', ascending=True)[:10]
)


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
It’s interesting to note that here that we get a good mix of movies with average ratings ranging from mediocre to high. The ratings are consistently high for the first two movies, while consistently mediocre for the rest.
It’s interesting to note that here, we get a good mix of movies with average ratings ranging from mediocre to high. The ratings are consistently high for the first two movies, while consistently mediocre for the rest.
So here in the end, let's take a look at the most and least polarizing movies with more than 10.000 votes:
Can you write Python code for it?

showdf(
df[df['votes'] > 10000][
                        ['Title', 'rating', 'year', 'votes', 'dist_std']
                        ].sort_values('dist_std', ascending=False)[:10]
)


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
Loading the data:
After filtering out entries that i) are obvious TV shows and computer games, and/or ii) have less than 1000 votes, we get:
Can you write Python code for it?

reader = pd.read_csv('ratings.csv',
                     sep=';',
                     iterator=True,
                     header=0, na_values=[' ', ''],
                     chunksize=1000,
                     error_bad_lines=False,
                     warn_bad_lines=False)
top_movies = []

df_chunks = []
for i, df_chunk in enumerate(reader):
    df_chunks.append(df_chunk)

df = pd.concat(df_chunks)
df.reset_index(inplace = True)

def get_movies_only(df):
    df_new = df[df['episode'].isnull()][df['media'].isnull()]
    return df_new

def imdb_google_link(row):
    return (
        '%s'
        % (' '.join(['imdb', row['title'], str(row['year'])]), row['title']))

def wikipedia_google_link(row):
    return (
        '%s'
        % ((' '.join(['wikipedia', 'movie', row['title'], str(row['year'])])),
           'wiki: '+row['title']))

df['Title'] = df.apply(imdb_google_link, axis=1)
df['Wikipedia link'] = df.apply(wikipedia_google_link, axis=1)

with warnings.catch_warnings():
    # We get a userwarning for reindexing which we filter out here
    warnings.simplefilter("ignore")
    df = get_movies_only(df)
    warnings.resetwarnings()

print('%i entries in our dataset' % len(df))


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
Movies with the highest standard deviation tend to have quite low average ratings, with the exception of number 1. Let's take a look at it:

There is a huge difference in the average rating of men and women, which explains a lot of the polarization here. The IMDB rating dataset unfortunately doesn't include the distribution of male vs female votes -- it would have been interesting to find movies that are the most divided by gender, and for that matter also country, age, etc.
Take a look at these movies; you might find something interesting. I have also added a link for each movie that looks up its Wikipedia entry in case you are curious (it doesn't work for the first one though).
Polarizing movies with high average ratings
Let's find the polarizing movies with an average rating of more than 7, as such a search might return movies that are more likely to be worth watching.
Can you write Python code for it?

showdf(
df[df['rating'] > 7.0
  ][['Title', 'rating', 'year', 'votes', 'dist_std']
  ].sort_values('dist_std', ascending=False)[:10]
)


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
Number 1 here has 38.5% of reviewers giving it 10s and 42.5% 1s, and again a large difference between the ratings of men and women, where women tend to like it. Maybe not all that surprising for a movie about a boyband's rise to fame.
And lastly the movies with least polarization:
Can you write Python code for it?

showdf(
df[df['votes'] > 10000][
                        ['Title', 'rating', 'year', 'votes', 'dist_std']
                        ].sort_values('dist_std', ascending=True)[:10]
)


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
Number one is Aquarius: another movie that women love and men in general would love to hate.
Another interesting find on this list is number 4, about Adolf Hitler.

We see that about 20% of its votes are 1s, whereas the majority 65% are 10s. I would be curious to find out what drives the 20% to give it 1; perhaps they just voted it 1 because the movie is about Adolf Hitler. I find it hard to imagine that the movie is really that bad if so many voted it 10.
Let's see what the most polarizing movies are with an average rating higher than 8.
Can you write Python code for it?

showdf(
df[df['rating'] > 8.0
  ][['Title', 'rating', 'year', 'votes', 'dist_std'
    ]].sort_values('dist_std', ascending=False)[:10]
)


I am working on the file "Polarizing_imdb_movies.ipynb".
The first task is:
where we mostly get mediocre to good (but not great) movies.
Trends over years - more movies gives rise to larger variance
We might be wondering whether movies have become more polarizing over time. Let's see what the trend is.
Can you write Python code for it?

for func in [np.min, np.max, np.mean, np.median, np.std]:
    df.groupby('year')['rating'].apply(func).plot(label=func.__name__);
pl.legend(loc=3, fontsize=18);
pl.title('Trend in polarization over time', fontsize=20);


I am working on the file "Police Killings/Police Killing.ipynb".
The first task is:
Comparing these columns by means of median.
Can you write Python code for it?

columns = ["pop", "county_income", "share_white", "share_black", "share_hispanic"]

lowest_states[columns].median()

highest_states[columns].median()


I am working on the file "Police Killings/Police Killing.ipynb".
The first task is:
Does race matter?
Can you write Python code for it?

race_counts = police_killings['raceethnicity'].value_counts()

plt.bar(np.arange(race_counts.size), race_counts, align='center')
plt.xticks(np.arange(race_counts.size), race_counts.index, rotation='vertical', fontsize='large')
plt.show()

race_counts / np.sum(race_counts) * 100


I am working on the file "Police Killings/Police Killing.ipynb".
The first task is:
Ordering rate column from highest to lowest.
Can you write Python code for it?

states[['STATE', 'shooting', 'NAME', 'pop_millions', 'rate']].sort_values('rate', ascending=False)


I am working on the file "Police Killings/Police Killing.ipynb".
The first task is:
The figure above shows that the high police killing rate happen in the states that the median income are about 22,000 dollar.
Shootings By State
In order to analyze the killing data by state, we need to know not only the number of people have been killed in each state, but also the population in each state.
Can you write Python code for it?

state_pop = pd.read_csv('state_population.csv')

state_pop.head()

state_counts = police_killings['state_fp'].value_counts()


I am working on the file "Police Killings/Police Killing.ipynb".
The first task is:
We have to change state's name to its abbreviate name because there is no abbr name for state.
Can you write Python code for it?

lowest_states_list = ["CT", "PA", "IA", "NY", "MA", "NH", "ME", "IL", "OH", "WI"]
highest_states_list = ["OK", "AZ", "NE", "HI", "AK", "ID", "NM", "LA", "CO", "DE"]

lowest_states = pk[pk['state'].isin(lowest_states_list)]
highest_states = pk[pk['state'].isin(highest_states_list)]


I am working on the file "Police Killings/Police Killing.ipynb".
The first task is:
from the charts above, we found that about half of people been killed are white. Black people account for approximately one thirds.
Does regional income matter?
Can you write Python code for it?

income = police_killings[police_killings['p_income'] != '-']['p_income']

income = income.astype('float')

sns.distplot(income, kde=False)
sns.plt.show()

income.median()


I am working on the file "Policy-Network.ipynb".
The first task is:
Loading the CartPole Environment
If you don't already have the OpenAI gym installed, use  pip install gym to grab it.
Can you write Python code for it?

import gym
env = gym.make('CartPole-v0')


I am working on the file "Policy-Network.ipynb".
The first task is:
Running the Agent and Environment
Here we run the neural network agent, and have it act in the CartPole environment.
Can you write Python code for it?

xs,hs,dlogps,drs,ys,tfps = [],[],[],[],[],[]
running_reward = None
reward_sum = 0
episode_number = 1
total_episodes = 10000
init = tf.global_variables_initializer()

# Launch the graph
with tf.Session() as sess:
    rendering = False
    sess.run(init)
    observation = env.reset() # Obtain an initial observation of the environment

    # Reset the gradient placeholder. We will collect gradients in
    # gradBuffer until we are ready to update our policy network.
    gradBuffer = sess.run(tvars)
    for ix,grad in enumerate(gradBuffer):
        gradBuffer[ix] = grad * 0

    while episode_number <= total_episodes:

        # Rendering the environment slows things down,
        # so let's only look at it once our agent is doing a good job.
        if reward_sum/batch_size > 100 or rendering == True :
            env.render()
            rendering = True

        # Make sure the observation is in a shape the network can handle.
        x = np.reshape(observation,[1,D])

        # Run the policy network and get an action to take.
        tfprob = sess.run(probability,feed_dict={observations: x})
        action = 1 if np.random.uniform() < tfprob else 0

        xs.append(x) # observation
        y = 1 if action == 0 else 0 # a "fake label"
        ys.append(y)

        # step the environment and get new measurements
        observation, reward, done, info = env.step(action)
        reward_sum += reward

        drs.append(reward) # record reward (has to be done after we call step() to get reward for previous action)

        if done:
            episode_number += 1
            # stack together all inputs, hidden states, action gradients, and rewards for this episode
            epx = np.vstack(xs)
            epy = np.vstack(ys)
            epr = np.vstack(drs)
            tfp = tfps
            xs,hs,dlogps,drs,ys,tfps = [],[],[],[],[],[] # reset array memory

            # compute the discounted reward backwards through time
            discounted_epr =...


I am working on the file "Policy-Network.ipynb".
The first task is:
What happens if we try running the environment with random actions? How well do we do? (Hint: not so well.)
Can you write Python code for it?

env.reset()
random_episodes = 0
reward_sum = 0
while random_episodes < 10:
    env.render()
    observation, reward, done, _ = env.step(np.random.randint(0,2))
    reward_sum += reward
    if done:
        random_episodes += 1
        print("Reward for this episode was:",reward_sum)
        reward_sum = 0
        env.reset()


I am working on the file "PolySurrogateModel.ipynb".
The first task is:
Both plots are the same, but the axis is smaller on the bottom one because the error blows up quickly.  Notice that now, instead of the error continualy decreasing, it decreases for a while then increases as we run into problems with overfitting.  Generally, we like to choose the simplest model that gives reasonable error.  The curve is pretty flat near the minimum, and in this case a good point is somewhere around a 4th order polynomial.
Can you write Python code for it?

order = 4
w = createpoly(xdata, fdata, order)

nhat = 200
xhat = np.linspace(0, 1, nhat)
fhat = evalpoly(xhat, w)

plt.figure()
plt.plot(xdata, fdata, 'o')
plt.plot(xhat, fhat, '--')
plt.ylim([-10, 20])
plt.xlabel('x')
plt.ylabel('f')


I am working on the file "PolySurrogateModel.ipynb".
The first task is:
Cross Validation
Let's consider another simple function with Gaussian noise:
$$f = (6x - 2)^2*\sin(12 x - 4) + \mathcal{N}(0, \sigma)$$
Can you write Python code for it?

def func(x):
    sigma = 1.0
    return (6*x-2)**2*np.sin(12*x-4) + np.random.randn(len(x))*sigma

# ---- create training data ---------
ndata = 20
xdata = np.linspace(0, 1, ndata)
fdata = func(xdata)

plt.plot(xdata, fdata, 'o')
plt.xlabel('x')
plt.ylabel('f')
plt.show()


I am working on the file "PolySurrogateModel.ipynb".
The first task is:
Ideally w would be [3, 2, 1] bsaed on our underlying polynomial, but it won't recover that exactly because of the noise in the function.
We now have a polynomial model: $\hat{f} = \Psi w$ that we can evaluate at any point.  Let's compute at many locations along the domain and compare out fit:
Can you write Python code for it?

nfull = 200
xfull = np.linspace(-2, 2, nfull)
Psifull = np.zeros((nfull, 3))
Psifull[:, 0] = xfull**2
Psifull[:, 1] = xfull
Psifull[:, 2] = 1
ffull = np.dot(Psifull, w)

plt.figure()
plt.plot(xdata, fdata, 'o')
plt.plot(xfull, ffull, '--')
plt.xlabel('x')
plt.ylabel('f')
plt.show()


I am working on the file "PolySurrogateModel.ipynb".
The first task is:
Let's try different polynomial orders and check to see what the error is in our fit.
Can you write Python code for it?

ordervec = np.arange(1, 21)
error = np.zeros(20)

for idx, order in enumerate(ordervec):

    # build a polynomial model from the training data
    w = createpoly(xdata, fdata, order)

    # test the error
    fhat = evalpoly(xdata, w)
    error[idx] = np.linalg.norm(fhat - fdata)

# plot error
plt.figure()
plt.plot(ordervec, error, 'o')
plt.xlabel('order of polynomial')
plt.ylabel('error')
plt.show()


I am working on the file "PolySurrogateModel.ipynb".
The first task is:
This data is our training data and consists of two pairs: $(x^{(i)}, f^{(i)})$.  Given this data, we want to compute a polynomial fit.  In this case, we know that it's quadratic, so let's use a quadratic model of the form:
$$\hat{f}(x) = a x^2 + b x + c$$
This is a simple least squares problem.
$$\text{minimize} \sum_i \left(\hat{f}(x^{(i)}) - f^{(i)} \right)^2 $$
Which we can rewrite in matrix form as:
$$\text{minimize} \; || \Psi w - f ||^2 $$
where
$$
\Psi =
\begin{bmatrix}
\text{---} & \psi(x^{(1)})^T & \text{---} \
\text{---} & \psi(x^{(2)})^T & \text{---} \
& \vdots \
\text{---} & \psi(x^{(n)})^T & \text{---} \
\end{bmatrix}
$$
$$
f = \begin{bmatrix}
f^{(1)}\
f^{(2)}\
\vdots \
f^{(n)}\
\end{bmatrix}
$$
In our specific case: $w = [a, b, c]$ and $\psi = [x^2, x, 1]$.  The matrix equation can be solved as a least squares solution.  Notice that this simple form works in any number of dimensions (not just one). In higher dimensions we just add more terms to $\psi$.
Can you write Python code for it?

Psi = np.zeros((n, 3))
Psi[:, 0] = xdata**2
Psi[:, 1] = xdata
Psi[:, 2] = 1

w, residuals, rank, s = np.linalg.lstsq(Psi, fdata)

print w


I am working on the file "PolySurrogateModel.ipynb".
The first task is:
This suggests that the higher the order of the polynomial the better!  Of course we know that's not true.  Let's look at what the polynomial model looks like in 20 dimensions:
Can you write Python code for it?

order = 20
w = createpoly(xdata, fdata, order)

nhat = 200
xhat = np.linspace(0, 1, nhat)
fhat = evalpoly(xhat, w)

plt.figure()
plt.plot(xdata, fdata, 'o')
plt.plot(xhat, fhat, '--')
plt.ylim([-10, 20])
plt.xlabel('x')
plt.ylabel('f')


I am working on the file "PolySurrogateModel.ipynb".
The first task is:
Visually, it's harder to determine what the best order for the polynomial is.  Of course, in higher dimensions we cannot visualize the function and so generally won't know beforehand what terms to use in our polynomial model.  We will use cross validation to help us choose an appropriate order.  Below, we create a function to create and evaluate a polynomial for any order (in a 1D space):
Can you write Python code for it?

def getPsi(x, order):

    n = len(x)
    Psi = np.zeros((n, order+1))
    for i in range(order+1):
        Psi[:, i] = x**(order-i)

    return Psi

def createpoly(x, f, order):
    Psi = getPsi(x, order)
    w, residuals, rank, s = np.linalg.lstsq(Psi, f)

    return w

def evalpoly(x, w):
    order = len(w) - 1
    Psi = getPsi(x, order)
    f = np.dot(Psi, w)

    return f


I am working on the file "PolySurrogateModel.ipynb".
The first task is:
We notice that the error at the points we are trying to fit is very small (which is what our least squares solution is doing), but the predictive capability of the model is very poor.  The reason for this issue is that we tested our model using the same points we used to train our model, so of course the error was low.  What we need to do instead is keep a separate set of training data and separate set of validation data to test how good the model is.  There are many methods to do this.  In this example, we use k-hold out cross validation.
Can you write Python code for it?

div = 10  # we will divide our data into div segments
ndata = 20  # number of data points
arrlength = ndata/div  # each segment should contain this much data
idxrand = np.random.permutation(n)  # random index into data from 0 ... n

error = np.zeros(len(ordervec))

# iterate through polynomial orders
for i, order in enumerate(ordervec):

    # iterate through divisions of data for k-holdout
    for j in range(div):

        # indicies of data to leave out from the random permutations
        holdout = idxrand[arrlength*j:arrlength*(j+1)]

        # separaet into training set and testing set
        xtrain = np.delete(xdata, holdout)
        ftrain = np.delete(fdata, holdout)
        xtest = xdata[holdout]
        ftest = fdata[holdout]

        # build a polynomial model from the training data
        w = createpoly(xtrain, ftrain, order)

        # test the error with the validation set
        fhat = evalpoly(xtest, w)
        error[i] += np.linalg.norm(fhat - ftest) / div  # average error across divisions

# plot error
plt.figure()
plt.plot(ordervec, error, 'o')
plt.xlabel('order of polynomial')
plt.ylabel('error')
plt.show()

# plot error
plt.figure()
plt.plot(ordervec, error, 'o')
plt.xlabel('order of polynomial')
plt.ylabel('error')
plt.ylim([0, 25])
plt.show()


I am working on the file "Polyhedrons 101.ipynb".
The first task is:
As shown below, we're not obligated to use the DB class as a context manager, in which case we may call the class's methods directly.  Context manager syntax is a little cleaner and more likely to disconnect in case of issues.
Can you write Python code for it?

DB.connect()
DB.c.execute("SELECT * FROM Coords ORDER BY vertex_label") # query
for rec in DB.c.fetchall():
    print(rec) # print results
DB.disconnect()


I am working on the file "Polyhedrons 101.ipynb".
The first task is:
The DB class contains information for connecting to a SQLite database, which may be accessed directly, as a text file, no need for a special server.  SQLite is free of charge.
The database was created earlier.

Can you write Python code for it?

with DB() as db:
    db.c.execute("SELECT poly_long, poly_color, poly_volume from Polys") # query
    pprint(db.c.fetchall()) # print results


I am working on the file "Polyhedrons 101.ipynb".
The first task is:
The database in question consists of three tables:  Polys, Faces, and Coords.  A polyhedron is defined in terms of its faces, such as ('A', 'B', 'C') which would consist of the three edges connecting 'A' to 'B', 'B' to 'C' and 'C' back to 'A'.  From faces, a computer deduces edges.  Edges terminate in labeled points, for which the database stores coordinates.
Lets remember the INNER JOIN semantics. Those go with LEFT JOIN and OUTER JOIN, with not every SQL engine making the same promises.  Getting a query right takes trial and error.  One often goes up against dummy data during testing and development.  Then might come staging, wherein the workload is tested to production levels before release.
There query below filters on one row of the Polys table, that of the rhombic dodecahedron (RD), and then starts in the Faces, pulling up matching in nickname, meaning RD on both sides.
The intersection is a set of twelve rows, each a face of the diamond-faced dodecahedron.  Every face is a tuple of four vertexes, labels paired with their coordinates in the Coords table.
Can you write Python code for it?

query3 = ("""
SELECT f.poly_nick, f.poly_face_id, f.vertex_labels, p.poly_long
FROM Polys p
INNER JOIN Faces f ON f.poly_nick = p.poly_nick
WHERE p.poly_nick = "RD"
""")

with DB() as db:
    db.c.execute(query3) # query
    pprint(db.c.fetchall()) # print results


I am working on the file "Polyhedrons 101.ipynb".
The first task is:
Why are these coordinates integers and why are there four of them per labeled vertex?  Is this another "legal in Oregon" thing?
These are Quadray Coordinates, which you may read more about on Wikipedia.
Here's a diagram giving a sense of what letters go with what shapes:

Converting to XYZ is pretty easy, especially with a computer.
Lets see those 26 vertexes in XYZ coordinates instead:
Can you write Python code for it?

with DB() as db:
    db.c.execute("SELECT vertex_label, coord_x, coord_y, coord_z FROM Coords ORDER BY vertex_label") # query
    pprint(db.c.fetchall()) # print results


I am working on the file "Polyhedrons.ipynb".
The first task is:
Qvector A is in the all-positive XYZ octant (+,+,+), whereas Qvector B is still above the XY plane but kitty-corner in octant (-,-,+). The other two points form a segment below the XY plane and perpendicular to the first, in (-,+,-) and (+,-,-) respectively.
Placing your forearms perpendicular to one another, but some distance apart, gives the same idea.  Your wrists and elbows will be the four vertexes in question.  Opposite edges of the regular tetrahedron are at 90 degrees to one another.
We're now ready to define our polyhedron faces accordingly and feed them into the class, along with the dict of needed vectors (or Qvectors in this case).
Can you write Python code for it?

Tet1 = Polyhedron("Tetrahedron", volume = 1,
                  faces = {('A', 'B', 'C'), ('A', 'B', 'D'),
                           ('A', 'C', 'D'), ('B', 'C', 'D')},
                  vertexes = tet1)

Tet1.edges  # lets make sure _distill() did its job

tet2 = {}  # now for the dual tetrahedron
tet2['E'] = -tet1['A']
tet2['F'] = -tet1['B']
tet2['G'] = -Qvector((0,0,1,0)) # same as -tet1['C']
tet2['H'] = -Qvector((0,0,0,1)) # same as -tet1['D']

Tet2 = Polyhedron("Dual Tetrahedron", volume = 1,
                  faces = {('E', 'F', 'G'), ('E', 'F', 'H'),
                           ('E', 'F', 'G'), ('F', 'G', 'H')},
                  vertexes = tet2)

Tet2.edges


I am working on the file "Polyhedrons.ipynb".
The first task is:
The polyhedrons we've talked about will be instances of our Polyhedron class.  Once instantiated, they each keep their vertex labels and pass them on to their progeny.  To transform a polyhedron, as we will do in the next chapter, is to create a new one, not to change an existing one in place.  Once defined, a polyhedron should be considered immutable.
Quadrays are just like XYZ vectors except they're described by 4-tuples indicating how much of three of the four directions are needed to reach a point.  The basis vectors point to the corners of a regular tetrahedron from its center, dividing space intor four quadrants.  The basis vector pointing away from the quadrant where our point is, will have a coefficient of 0.
Lets look at some examples:
Can you write Python code for it?

from qrays import Qvector, Vector
tet1 = {}
tet1['A'] = Qvector((1,0,0,0))
tet1['B'] = Qvector((0,1,0,0))
tet1['C'] = Qvector((0,0,1,0))
tet1['D'] = Qvector((0,0,0,1))
for v in tet1:
    print(tet1[v].xyz())


I am working on the file "PolynomialFun.ipynb".
The first task is:
Least Squares problems:
Given the value of a function at any three distinct points is sufficient to describe uniquely the interpolating quadratic through those points.  But suppose we were given more than 3 points, say 7, in which case the matrix $A$ would be $7\times3$ with rank $r=3$  and unless those 7 points were on the same parabola,  there would be no solution to the overdetermined problem.  Here we will create that problem by adding more points to the interpolating parabola calculated above and then perturb it with uniform random noise.
Can you write Python code for it?

# choose 7 evenly spaced points in [-1,1]
x = np.linspace(-1,1,7)

# perturb the parabola with uniform random noise
f = v(c,x) + np.random.uniform(-.5,.5,len(x))

# and plot with respect to the underlying parabola
plt.figure()
plt.plot(x,f,'bo')
plt.hold(True)
plt.plot(xx,v(c,xx),'r',label='v')
plt.xlabel('x')
plt.ylim(-1.5,2.5)
plt.grid()

# now calculate and plot the leastsquares solution to Ac = f
c_ls,res,rank,s = la.lstsq(A(x),f)

plt.plot(xx,v(c_ls,xx),'g',label='v_lstsq')
plt.title('$c={}$: $v={}p_0 + {}p_1 + {}p_2$'.format(c_ls,c_ls[0],c_ls[1],c_ls[2]))
plt.legend(loc='best')
plt.show()

# and show that this is the same solution we would get if we tried to solve the normal equations direction

AtA = np.dot(A(x).transpose(),A(x))
Atf = np.dot(A(x).transpose(),f)

c_norm = la.solve(AtA,Atf)

print 'numpy least-squares c = {}'.format(c_ls)
print 'normal equations      = {}'.format(c_norm)
print 'difference            = {}'.format(c_ls-c_norm)

print
print 'ATA ={}'.format(AtA)


I am working on the file "PolynomialFun.ipynb".
The first task is:
Now calculate the mass matrix and load vector and solve for the galerkin projection of $f$ onto  $P_2[-1,1]$
Can you write Python code for it?

from scipy.integrate import quad

def mij(i,j,x):
    """ integrand for component Mij of the mass matrix"""
    p = np.array([1., x, x**2])
    return p[i]*p[j]

def fi(i,x,f):
    """ integrand for component i of the load vector"""
    p = np.array([1., x, x**2])
    return p[i]*f(x)

# construct the symmetric mass matrix  M_ij = <p_i,p_j>

M = np.zeros((3,3))
fhat = np.zeros(3)
R = np.zeros((3,3)) # quadrature residuals

# loop over the upper triangular elements of M (and fill in the symmetric parts)
for i in range(0,3):
    fhat[i] = quad(lambda x: fi(i,x,f),-1.,1.)[0]
    for j in range(i,3):
        result = quad(lambda x: mij(i,j,x),-1.,1.)
        M[i,j] = result[0]
        M[j,i] = M[i,j]
        R[i,j] = result[1]
        R[j,i] = R[i,j]

print 'M = {}\n'.format(M)
print 'fhat = {}\n'.format(fhat)

# and solve for c
c_galerkin = la.solve(M,fhat)

print 'c_galerkin ={}'.format(c_galerkin)


I am working on the file "PolynomialFun.ipynb".
The first task is:
Projection of a function onto $P_2[-1,1]$
Now let's extend this problem to finding the best fit projection of a continuous function $f(x)$ onto $P_2$.  While we could extend the previous approach by sampling $f(x)$ at a large number of points and calculating the least-squares solution,  we can also solve the continuous problem by changing the definition of the inner product from the dot product in $R^n$ to the inner product for continuous functions
$$
    <f,g> = \int_{-1}^{1} fg dx
$$
However the overall approach remains the same as the discrete least squares problem.
If we now consider a function $u \in P_2[-1,1]$ such that
$$
    u(x) = \sum_i c_i p_i(x)
$$
then the continous error (or residual) is given by
$$
    e(x) = u(x) - f(x)
$$
for the continuous variable $x\in[-1,1]$.
The least square problem now becomes "find $\mathbf{c}\in R^3$ that minimizes $||e||_{L2}$", i.e. the  "length" of $e$ in the $L^2$ norm.  Alternatively this requires that the error $e(x)$ is orthogonal to all the basis vectors in $P_2$, i.e.
$$
 <p_i,e> = 0 \quad \mathrm{for\, }i=0,1,2
$$
or
$$
    \int_{-1}^{1} p_i e dx = \int_{-1}^{1} p_i ( u - f) dx = 0
$$
or solve
$$
     \int_{-1}^{1} p_i \left(\sum_j c_j p_j(x)\right)dx =  \int_{-1}^{1} p_i f dx
$$
for all $i,j=0,1,2$.  Rearranging the summation and the integral sign, we can rewrite the problem as
$$
    \sum_j M_{ij} c_j = \hat{f}_i
$$
where
$$
    M_{ij} = <p_i,p_j>=\int_{-1}^{1} p_i p_j dx\quad \mathrm{and}\quad \hat{f}i = <p_i,f> = \int{-1}^{1} p_i f dx
$$
or in matrix vector notation $M\mathbf{c} = \hat{\mathbf{f}}$ where $M$ is the "mass-matrix (and corresponds to the symmetric matrix $A^TA$) and $\hat{\mathbf{f}}$ is the "load vector" which corresponds to  $A^t\mathbf{b}$.
For the simple monomial basis, we can calculate the terms of $M$ easily, but here we will just use scipy's numerical quadrature routines
We'll start by defining our function and calculating its interpolation onto $P_2[-1,1]$ as the unique quadratic that interpolates $f(x)$ at $x=[-1,0,1]$
Can you write Python code for it?

#  set the function to be projected
f = lambda x : np.cos(2*x) + np.sin(1.5*x)

# calculate the interpolation of f onto P2, when sampled at points -1,0,1
x = np.array([-1., 0., 1.])
c_interp = la.solve(A(x),f(x))


I am working on the file "PolynomialFun.ipynb".
The first task is:
and plot them
Can you write Python code for it?

x = np.linspace(-1,1)
plt.figure()
plt.plot(x,p0(x),label='$p_0$')
plt.hold(True)
plt.plot(x,p1(x),label='$p_1$')
plt.plot(x,p2(x),label='$p_2$')
plt.xlabel('x')
plt.ylim(-1.5,1.5)
plt.legend(loc='best')
plt.grid()
plt.show()


I am working on the file "Polyvolumes.ipynb".
The first task is:
$\phi = (1 + \sqrt{5})/2$
Can you write Python code for it?

phi = (1 + np.sqrt(5))/2 # a constant aka "golden ratio"


I am working on the file "Polyvolumes.ipynb".
The first task is:

3D Print Me!

Can you write Python code for it?

byvolume

comments = pd.Series(index=shapes, dtype=str)

comments.loc["Tetrahedron"] = "unit of volume, self dual"
comments.loc["Cube"] = "intersected tetrahedron duals"
comments.loc["Octahedron"] = "cube dual"
comments.loc["Rhombic Dodecahedron"] = "space-filler, sphere container"
comments.loc["Cuboctahedron"] = "RD's dual"
comments.loc["Icosahedron"] = "Platonic, pent. dodeca's dual"
comments.loc["RT5"] = "120 T modules, radius 0.9994"
comments.loc["Pentagonal Dodeca"] = "Platonic, Icosa's dual"
comments.loc["RTe"] = "120 E modules, radius 1.0000"
comments.loc["SuperRT"] = "RTe scaled up to Phi radius"

comments.shape

comments

byvolume["Comments"] = comments

byvolume

from IPython.display import YouTubeVideo
YouTubeVideo("RGH8m0LdaTM")


I am working on the file "Polyvolumes.ipynb".
The first task is:
Lets add one of a few interesting RTs (RT = rhombic triacontahedron).  The original Icosahedron, just added, has a dual with crossing edges that we'll also be adding.  Together, the Icosahedron and its intersecting Dual, the Pentagonal Dodecahedron, define an RT of tetravolume
Can you write Python code for it?

table.loc["RT5","Volume"] = 5
shapes.append("RT5")

byvolume = table.sort_values(by="Volume")

byvolume


I am working on the file "Polyvolumes.ipynb".
The first task is:
Now comes our common need to add another row to our table.
However the Icosahedron's volume is smaller than the Cuboctahedron's.
How shall we maintain Volume order?
Can you write Python code for it?

table.loc["Icosahedron","Volume"] = 5 * np.sqrt(2) * phi ** 2
shapes.append("Icosahedron")

table


I am working on the file "Polyvolumes.ipynb".
The first task is:
Unlike a numpy array, which is of uniform data type (example: all floating point), a pandas Dataframe is expected to have labels for both columns and rows.  The data columns themselves by be of varying type.  A Dataframe consists as an arrangement, left to right, of instances of the pandas Series type, i.e. the columns.
Can you write Python code for it?

table.columns = ["Volume",] # relabel the one column

table


I am working on the file "Polyvolumes.ipynb".
The first task is:
What other RTs might we want to add?  The RT of exactly 5 tetravolumes is not the SuperRT scaled down by phi in its linear dimensions, (1/phi) to the 3rd power by volume.  The latter has a volume of 5+ and a radius of 1 exactly, whereas the RT of 5 has a radius of 0.9994... that of the RT of 5+.
Can you write Python code for it?

byvolume.loc["RTe", "Volume"] = byvolume.loc["SuperRT","Volume"] * phi**-3

byvolume.loc["RTe", "Avols"] = byvolume.loc["RTe", "Volume"] * 24

shapes.append("RTe") # RT of 120 E modules

byvolume.sort_values(by="Volume", inplace=True)


I am working on the file "Popn-Sample-and-SamplingDistn.ipynb".
The first task is:
Here are the density histograms of the population (grey), our first sample (blue), and the sampling distribution of the mean(red), all drawn in the same plot.
IMPORTANT NOTE: to facilitate visual comparison of the distributions I've truncated the y-axis. Comment out the ylim line below to see the complete density histogram of the sampling distribution of sample means.
Can you write Python code for it?

plt.hist(popn, normed=True, bins=50, label='population',
         color='gray', alpha=0.75, histtype='stepfilled')
plt.hist(sample, normed=True, label='sample',
         color='steelblue', alpha=0.75, histtype='stepfilled')
plt.hist(smeans, normed=True, bins=50, label='simulated\ndistn of\n means',
         color='firebrick', alpha=0.75, histtype='stepfilled')
plt.xlabel("X")
plt.ylabel("Density")
plt.legend(loc="best")
plt.ylim(0,0.06)  # comment out this line to remove truncation
pass


I am working on the file "Popn-Sample-and-SamplingDistn.ipynb".
The first task is:
Here is the frequency histogram of the sampling distribution of the mean.
Can you write Python code for it?

plt.hist(smeans, normed=True, bins=30, label='simulated\ndistn of\n means',
         color='firebrick', alpha=0.75, histtype='stepfilled')
plt.xlabel("mean(X)")
plt.ylabel("Frequency")
pass


I am working on the file "Popn-Sample-and-SamplingDistn.ipynb".
The first task is:
Here is what the frequency histogram of the sample looks like.
Can you write Python code for it?

plt.hist(sample, color='steelblue', alpha=0.75,
         histtype='stepfilled',label='sample')
plt.xlabel("X")
plt.ylabel("Frequency")
pass


I am working on the file "Popn-Sample-and-SamplingDistn.ipynb".
The first task is:
Population distribution
Here is a population of size 100,000.
Can you write Python code for it?

mu, sigma = 64, 8
popn = np.random.normal(loc=mu,scale=sigma, size=100000)
truemu, truesigma = np.mean(popn), np.std(popn)

s = \
"""For the population of interest, the true mean is {}
and the true standard deviation is {} """
print(s.format(truemu,truesigma))


I am working on the file "Popn-Sample-and-SamplingDistn.ipynb".
The first task is:
Sample distribution
Here is sample of size 60 drawn without replacement from this population:
Can you write Python code for it?

sample = np.random.choice(popn, size=60, replace=False)

s = \
"""For the population of interest, the point estimates of the
mean and standard deviation are {} and {}, respectively"""

print(s.format(np.mean(sample),np.std(sample,ddof=1)))


I am working on the file "Popn-Sample-and-SamplingDistn.ipynb".
The first task is:
Sampling distribution of the mean
To estimate the sampling distribution of the mean, we draw 1000 samples of size 60 and calculate the mean for each such sample.
Can you write Python code for it?

smeans = []
for i in range(1000):
    rsample = np.random.choice(popn, size=60, replace=False)
    smeans.append(np.mean(rsample))


I am working on the file "Popn-Sample-and-SamplingDistn.ipynb".
The first task is:
This is what the population distribution looks like when represented as a frequency histogram.
Can you write Python code for it?

plt.hist(popn, bins=50, color='gray', alpha=0.75, histtype='stepfilled')
plt.xlabel("X")
plt.ylabel("Frequency")
pass


I am working on the file "PopulationFlow.ipynb".
The first task is:
Source: Geocoder.ca
Can you write Python code for it?

# Get mapping of cities to centroids
centroids = pd.read_csv('./canada_cities.csv', header=None, names=['Location', 'Province', 'Latitude', 'Longitude'])

from titlecase import titlecase
title_u = lambda x: u(x).title()

centroids['Location'] = centroids['Location'].map(title_u)
centroids.head(15)

provAbbr = {'BC' : 'British Columbia',
           'SK' : 'Saskatchewan',
           'QC' : 'Quebec',
           'AB' : 'Alberta',
           'NB' : 'New Brunswick',
           'NS' : 'Nova Scotia',
           'ON' : 'Ontario',
           'NL' : 'Newfoundland',
           'PE' : 'PEI',
           'MB' : 'Manitoba',
           'NT' : 'Northwest Territories',
           'YT' : 'Yukon',
           'NU' : 'Nunavut'}
centroids['Province'] = centroids['Province'].replace(provAbbr)
centroids.head()

centroids = centroids.drop_duplicates(subset=['Location', 'Province'])
centroids.head()


I am working on the file "Portfolio.ipynb".
The first task is:
And let's make a chart, plotting volatities on X-axis and returns on Y-axis.
Can you write Python code for it?

f = plt.figure(figsize=(10, 8));
ax = f.add_subplot(111);
for n in range(len(stock_moments.columns)):
    ax.plot(stock_moments.loc['Annual Volatility', stock_moments.columns[n]], stock_moments.loc['Compound Annual Return', stock_moments.columns[n]], ls='', marker='o', label=stock_moments.columns[n]);
ax.legend(loc='best');
ax.set_xlabel(stock_moments.index[1]);
ax.set_ylabel(stock_moments.index[0]);
ax.set_title('Stock risk reward profile');


I am working on the file "Portfolio.ipynb".
The first task is:
In order to calculate volatility, we need a covariance matrix. Luckily enough, Pandas has a function to do exactly that!
Can you write Python code for it?

cov_mat = prices.fillna(method='pad').pct_change().cov().apply(lambda x: x * 260)
cov_mat


I am working on the file "Portfolio.ipynb".
The first task is:
Let's check that the roots of the diagonal of the matrix are the (annualized) standard deviations of the stocks.
Can you write Python code for it?

pd.DataFrame(data=[cov_mat.iloc[n, n] ** 0.5 for n in range(len(cov_mat.columns))], index=cov_mat.columns, columns=['Annual Volatility']).T


I am working on the file "Portfolio.ipynb".
The first task is:
Let's join these stats into a single dataframe.
Can you write Python code for it?

stock_moments = pd.concat((compound_returns, annual_volatility))
stock_moments


I am working on the file "PostNewtonian_Inspiral_with_RK.ipynb".
The first task is:
Next, plot the fractional error in the orbital separation comparing the numerical solutions to the analytic adiabatic approximation. The errors in the slimplectic integration arise because the adiabatic approximation is breaking down towards the end of the inspiral.
Can you write Python code for it?

fig2 = plt.figure(figsize=(12,5), dpi=300)
ax2 = fig2.add_subplot(111)

ax2.loglog(t_1000, np.abs(r_1000/r_ad(t_1000)-1.), 'r-', linewidth=2.0, rasterized = True);
ax2.loglog(t_isco, np.abs(r_isco/r_ad(t_isco)-1.),  '-', color = 'orange', linewidth=2.0, rasterized = True);
ax2.loglog(t_1000, np.abs(r_1000_rk4/r_ad(t_1000)-1.), 'g--', linewidth=2.0, rasterized = True);
ax2.loglog(t_isco, np.abs(r_isco_rk4/r_ad(t_isco)-1.), 'b--', linewidth=2.0, rasterized = True);

ax2.text(1, 7e-4, r'Slim4, $\Delta t = 1000M$', fontsize = 15, color = 'black')
ax2.text(1, 5e-8, r'Slim4, $\Delta t = T_{\rm ISCO}$', fontsize = 15, color = 'black')

ax2.text(0.0125, 1.25e0, r'RK4 $\Delta t = 1000M$', fontsize = 15, color = 'green', rotation=16)
ax2.text(0.01, 1.5e-5, r'RK4 $\Delta t = T_{\rm ISCO}$', fontsize = 15, color = 'blue', rotation=16)

ax2.set_xlim(0, 1.05*tcoal);
ax2.set_ylim(1e-10, 1e2)
ax2.set_xlabel('Time [sec]', fontsize = 18);
ax2.set_ylabel('Fractional separation error $|\delta r/ r_{\\rm ad}|$', fontsize = 18);

ax2.set_yticks([1e-10, 1e-8, 1e-6, 1e-4, 1e-2, 1e0, 1e2])
ax2.tick_params(axis='both', which='major', labelsize=16)


I am working on the file "PostNewtonian_Inspiral_with_RK.ipynb".
The first task is:
Similar plots can be made for the orbital phase.
Can you write Python code for it?

# Orbital phases
phi_1000 = orbit.phase(*q_1000)
phi_isco = orbit.phase(*q_isco)
phi_1000_rk4 = orbit.phase(*q_1000_rk4)
phi_isco_rk4 = orbit.phase(*q_isco_rk4)

fig3 = plt.figure(figsize=(12,5), dpi=300)
ax3 = fig3.add_subplot(111)

ax3.plot(t_1000, phi_1000, 'r-',  linewidth=2.0, rasterized = True, label='4th order Slimplectic $\Delta t=1000M$');
ax3.plot(t_isco, phi_isco, '-', color = 'orange', linewidth=2.0, rasterized = True,  label='4th order Slimplectic $\Delta t=T_{\\rm ISCO}$');
ax3.plot(t_1000, phi_1000_rk4, 'g--',  linewidth=2.0, rasterized = True, label='RK4, $\Delta t=1000M$');
ax3.plot(t_isco, phi_isco_rk4, 'b--',  linewidth=2.0, rasterized = True, label='RK4, $\Delta t =T_{\\rm ISCO}$');

ax3.plot(t_isco, phi_ad(t_isco), 'k:',  linewidth=2.0, rasterized = True, label='Adiabatic approx.');
ax3.set_xlim(0, 1.05*tcoal);
ax3.set_xlabel('Time [sec]', fontsize = 18);
ax3.set_ylabel('Orbital phase [rad]', fontsize=18);
ax3.legend(loc='upper left', prop={'size':15});

ax3.tick_params(axis='both', which='major', labelsize=16)

fig4 = plt.figure(figsize=(12,5), dpi=300)
ax4 = fig4.add_subplot(111)

ax4.loglog(t_1000, np.abs(phi_1000 - phi_ad(t_1000)), 'r-', linewidth=2.0, rasterized = True);
ax4.loglog(t_isco, np.abs(phi_isco - phi_ad(t_isco)),  '-', color = 'orange', linewidth=2.0, rasterized = True);
ax4.loglog(t_1000, np.abs(phi_1000_rk4 - phi_ad(t_1000)), 'g--', linewidth=2.0, rasterized = True);
ax4.loglog(t_isco, np.abs(phi_isco_rk4 - phi_ad(t_isco)), 'b--', linewidth=2.0, rasterized = True);

ax4.text(2, 2e1, r'Slim4, $\Delta t = 1000M$', fontsize = 15, color = 'r', rotation=8)
ax4.text(2, 1.5e-3, r'Slim4, $\Delta t = T_{\rm ISCO}$', fontsize = 15, color = 'orange', rotation=9)

ax4.text(2, 3e3, r'RK4 $\Delta t = 1000M$', fontsize = 15, color = 'green', rotation=9)
ax4.text(2, 3e-1, r'RK4 $\Delta t = T_{\rm ISCO}$', fontsize = 15, color = 'blue', rotation=15)

ax4.set_xlim(0, 1.05*tcoal);
ax4.set_yticks([1e-8, 1e-4, 1, 1e4]);
ax4.tick_params(axis='both', which='major',...


I am working on the file "PostNewtonian_Inspiral_with_RK.ipynb".
The first task is:
Slimplectic integration
Can you write Python code for it?

# Create an instance of the GalerkinGaussLobatto class and call it `pn` for post-Newtonian
pn = slimplectic.GalerkinGaussLobatto('t', ['x', 'y'], ['vx', 'vy'])

# Define the conservative $L$ and nonconservative $K$ parts of the total Lagrangian $\Lambda$
L = 0.5*mu*np.dot(pn.v, pn.v) + mu*M/np.dot(pn.q, pn.q)**0.5

K = 16./5.*nu**2*M**4*np.dot(pn.vp, pn.qm)/np.dot(pn.qp, pn.qp)**2 \
    - 48./5.*nu**2*M**3*np.dot(pn.vp, pn.vp)*np.dot(pn.vp, pn.qm)/np.dot(pn.qp, pn.qp)**1.5 \
    + 24.*nu**2*M**3*np.dot(pn.vp, pn.qp)**2*np.dot(pn.vp, pn.qm)/np.dot(pn.qp, pn.qp)**2.5 \
    + 16./15.*nu**2*M**4*np.dot(pn.vp, pn.qp)*np.dot(pn.qp, pn.qm)/np.dot(pn.qp, pn.qp)**3 \
    + 144./5.*nu**2*M**3*np.dot(pn.vp, pn.vp)*np.dot(pn.vp, pn.qp)*np.dot(pn.qp, pn.qm)/np.dot(pn.qp, pn.qp)**2.5 \
    - 40.*nu**2*M**3*np.dot(pn.vp, pn.qp)**3*np.dot(pn.qp, pn.qm)/np.dot(pn.qp, pn.qp)**3.5

# Discretize total Lagrangian using a 4th order (r=1) implicit scheme
pn.discretize(L, K, 1, method='implicit')

# Specify time samples at which the numerical solution is to be given initial data.
# We take data for a quasi-circular inspiral

q0 = np.array([100., 0.])*M
v0 = np.asarray([-64./5.*nu*(M/q0[0])**3, np.sqrt(M/q0[0])])
# (The initial velocity is found by calculating the leading correction
#  to pure circular motion due to radiation reaction.)
pi0 = mu*v0

# Estimate time to "coalescence" when the orbital separation is zero
r0 = np.sqrt(q0[0]**2+q0[1]**2)
tcoal = 5./256./nu*(r0/M)**4 * M

# Time samples at every 1000M
dt = 1000*M
t_1000 = np.arange(0, tcoal+dt, dt)

# Now integrate the 4th order slimplectic integrator
q_1000, pi_1000 = pn.integrate(q0, pi0, t_1000)


I am working on the file "PostNewtonian_Inspiral_with_RK.ipynb".
The first task is:
The numerical solution clearly becomes unstable. The reason is because eventually the period of the orbit becomes comparable to or smaller than the time-step, $dt$, of $1000M$. One can show that adaptive time-step methods are better-suited to this problem, which is what we are exploring currently.
For fixed time-steppers, we can take $dt$ to be increasingly smaller. We'll show how the solution changes with step-size but keeping the integration order fixed (at 4th). In particular, we take $dt$ to be the orbital period of the binary near the innermost stable circular orbit (ISCO).
(WARNING: Integrating this higher-resolution run may take a few minutes to complete.)
Can you write Python code for it?

# Time samples with increment given by the approximate orbital period at ISCO
T_isco = 6.**1.5*(2.*np.pi*M)
t_isco = np.arange(0, tcoal+T_isco, T_isco)

# ...and integrate
q_isco, pi_isco = pn.integrate(q0, pi0, t_isco)


I am working on the file "Post_1.ipynb".
The first task is:
I use the this function in conjunction with the .apply() method in pandas to return a series of booleans as a column in the place name database.  The large size of the database means that running this function could take up to an hour.  To demonstrate how the function works, I will apply the function to a subset of the data 'sample_df' containing 1000 rows.
Can you write Python code for it?

sample_df = places.loc[0:1000,:]
sample_df['matches'] = sample_df.prepared_places.apply(lambda x: find_matches(x,kbbi.prepared_entries))

sample_df.head()


I am working on the file "Post_1.ipynb".
The first task is:
Identifying place names of known origin: perfect matches
The following function takes a place name as input, compares the name to each word in the dictionary wordlist.  A boolean is returned indicating whether any matches were found.
Can you write Python code for it?

def find_matches(place, wordlist_series):
    bool_series = wordlist_series.str.contains(place,case=True)
    return(bool_series.value_counts(ascending=True).index[0])


I am working on the file "Post_1.ipynb".
The first task is:
In terms of Levenshtein distance, both pairs of words earn the same score.  This is because, in both cases, this first item in the pair can be transformed into the second item by substituting the first letter(b for p in Pair 1 and b for s in Pair 2).  In terms of actual phonological distance, the words in Pair 1 are much closer to one another than the words in Pair 2.  This is because the sound [b] is nearly identical to [p] in terms of both the physical gestures involved in its pronunciation as well its accoustic properties.  In both of these respects, [b] and [s] are quite different from one another.
Drawing this distinction might seem pedantic, but phonological distance is important within the context of discovering pairs of words which are variants of one another, or related historically.  Changes in  pronounciation that occur over time are gradual.  For example, cases where [p] becomes [b] are extremely common across the world's languages, whereas [b] to [s] is an extremely rare if not unattested change.  Since we are trying to establish whether place names which do not have perfect matches in the dictionary are nevertheless etymologically related to words in the dictionary, it is important that we quantify phonological distance. To accomplish this, I have created a function which converts each sound into a matrix dictionary of its component phonological features (i.e. where in the vocal tract it is pronounced, whether it is nasal etc.) (For more information about phonological features see Wikipedia's page: https://en.wikipedia.org/wiki/Distinctive_feature)
Can you write Python code for it?

def phono_matrix(string):
    string_of_matrixes = []
    for character in string:
        matrix = {}
###populate manner:
    ### sonorant
        if character in ['a','e','i','o','u','y','w','m','N','Y','l','r','h','q']:
            matrix['sonorant'] = 'Y'
        else:
            matrix['sonorant'] = 'N'

    ###continuant
        if character in ['l','r','y','w','a','e','i','o','u','s','z','f']:
            matrix['continuant'] = 'Y'
        else:
            matrix['continuant'] = 'N'

    ###consonant
        if character in ['p','t','k','q','h','c','b','d','g','j','s','z','f','m','n','Y','N','l','r']:
            matrix['consonant'] = 'Y'
        else:
            matrix['consonant'] = 'N'

    ###syllabic
        if character in ['a','e','i','o','u']:
            matrix['syllabic'] = 'Y'
        else:
            matrix['syllabic'] = 'N'

    ###strident
        if character in ['s','j','c']:
            matrix['strident'] = 'Y'
        else:
            matrix['strident'] = 'N'

###populate place: labial, coronal, palatal, velar, glottal
    ###labial
        if character in ['p','m','f','b','w','u','o']:
            matrix['labial'] = 'Y'
        else:
            matrix['labial'] = 'N'

    ###coronal
        if character in ['t','d','n','s','j','c','Y','i','e','r','l']:
            matrix['coronal'] = 'Y'
        else:
            matrix['coronal'] = 'N'

    ###palatal
        if character in ['s','j','c','i','Y','e']:
            matrix['palatal'] = 'Y'
        else:
            matrix['palatal'] = 'N'
    ###palatal
        if character in ['u','k','g','N','o']:
            matrix['velar'] = 'Y'
        else:
            matrix['velar'] = 'N'
    ###glottal
        if character in ['h','q']:
            matrix['glottal'] = 'Y'
        else:
            matrix['glottal'] = 'N'

    ###glottal
        if character in ['h','q']:
            matrix['glottal'] = 'Y'
        else:
            matrix['glottal'] = 'N'
###nasality
    ###nasal/oral
        if character in...


I am working on the file "Post_1.ipynb".
The first task is:
Let's take a closer look at the first entry in the dictionary and how it is represented in terms of tiers of phonological features:
Can you write Python code for it?


I am working on the file "Post_1.ipynb".
The first task is:
Moreover, I generated new columns in the dictionary and place names database for the cleaned strings.
Can you write Python code for it?


I am working on the file "Post_1.ipynb".
The first task is:
Now that each sound is represented by its features, we're almost at the point where we can compare sounds on a feature by feature basis to get a much more accurate measure of their phonological distance.  To allow this feature by feature comparison, I have created the following function, which takes a string of matrixes corresponding to individual sounds, and generates multiple tiers, each correponding to a single binary feature ('Y' for 'has feature' and 'N' for 'lacks feature'). Consider the tier for the feature [consonant] in the case of Pair 1 above: This tier will be identical for the words 'pat' and 'bat', and in both cases, it will take the form 'YNY' (i.e. b,s=consonant, a=vowel, t=consonant).  The following function translates a string of feature matrixes into tiers, each corresponding to a single feature.
Can you write Python code for it?

def tier_builder(string_of_matrixes):
    features = ['sonorant','consonant','continuant','syllabic','strident','labial','coronal','palatal','velar', 'glottal','nasal','voice','lateral','rhotic']
    tier_dictionary = {}
    for feature in features:
        tier_dictionary[feature] = str()
        for matrix in string_of_matrixes:
            if matrix[feature] is not None:
                tier_dictionary[feature] = tier_dictionary[feature] + matrix[feature]
    return(tier_dictionary)

kbbi['tiers'] = kbbi.matrixes.apply(lambda x: tier_builder(x))
sample_df['tiers'] = sample_df.matrixes.apply(lambda x: tier_builder(x))


I am working on the file "Post_1.ipynb".
The first task is:
Now that we have complete phonological representations for each place name and dictionary item, we can apply the following function to calculate the Levenshtein distance between each place name and each dictionary item. The function calculates phonological similarity on a feature-by-feature basis, by comparing the tiers we generated above using 'fuzzy matching'.
Similarity values are calculated for each feature tier, then the average of all tiers is returned, giving us a global measure of phonolgical similarity between two words.
Can you write Python code for it?

def similarity(word1_tiers,word2_tiers):
    features = ['sonorant','continuant','syllabic','strident','labial','coronal','palatal','velar', 'glottal','nasal','voice','lateral','rhotic']
    tier_similarity = {}
    for feature in features:
        tier_similarity[feature] = fuzz.ratio(word1_tiers[feature],word2_tiers[feature])
    tier_similarity = pd.Series(tier_similarity)
    return(tier_similarity.mean())


I am working on the file "Post_1.ipynb".
The first task is:
Step 1: Removing orthographic irregularities
Unlike English, the Indonesian spelling system is almost perfectly phonemic i.e. in most cases there is a one-to-one relationship between a letter and the sound it represents.  This being said, there are a few irregularities in the orthography.  To rid the data of as many of these irregularities as possible, I created the following function:
Can you write Python code for it?

def prepare_string(word):
    word = word.lower()
    word = word.replace('ng','N')
    word = word.replace('ny','Y')
    word = word.replace('sy','s')
    word = word.replace('kh','k')
    word = word.replace('oe','u')
    word = word.replace('tj','c')
    word = word.replace('dj','j')
    return(word)

kbbi['prepared_entries'] = kbbi.words.apply(lambda x: prepare_string(str(x)))
places['prepared_places'] = places.split_word.apply(lambda x: prepare_string(str(x)))


I am working on the file "Post_1.ipynb".
The first task is:
Using the boolean values in the 'matches' column, I will filter out place names with an exact match in the dictionary.
Can you write Python code for it?

sample_df = sample_df[sample_df.matches == False]

sample_df.head()


I am working on the file "PostgreSQL_YugabyteDB_Jupyter/PostgreSQL_histograms.ipynb".
The first task is:
Define the query to compute the histogram
Can you write Python code for it?

table_name = "data" # table or temporary view containing the data
value_col = "random_value" # column name on which to compute the histogram
min = -20  # min: minimum value in the histogram
max = 90   # maximum value in the histogram
bins = 11  # number of histogram buckets to compute
step = (max - min) / bins

query = f"""
with hist as (
  select
      width_bucket({value_col}, {min}, {max}, {bins}) as bucket,
      count(*) as cnt
  from {table_name}
  group by bucket
),
buckets as (
  select generate_series as bucket from generate_series(1,{bins})
)
select
    bucket, {min} + (bucket - 0.5) * {step} as value,
    coalesce(cnt, 0) as count
from hist right outer join buckets using(bucket)
order by bucket
"""


I am working on the file "PostgreSQL_YugabyteDB_Jupyter/PostgreSQL_histograms.ipynb".
The first task is:
Histogram plotting
The first plot is a histogram with the event counts (number of events per bin).
The second plot is a histogram of the events frequencies (number of events per bin normalized by the sum of the events).
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

f, ax = plt.subplots()

# histogram data
x = hist_pandasDF["value"]
y = hist_pandasDF["count"]

# bar plot
ax.bar(x, y, width = 3.0, color='red')

ax.set_xlabel("Bucket values")
ax.set_ylabel("Event count")
ax.set_title("Distribution of event counts")

# Label for the resonances spectrum peaks
txt_opts = {'horizontalalignment': 'center',
            'verticalalignment': 'center',
            'transform': ax.transAxes}

plt.show()

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

f, ax = plt.subplots()

# histogram data
x = hist_pandasDF["value"]
y = hist_pandasDF["frequency"]

# bar plot
ax.bar(x, y, width = 3.0, color='blue')

ax.set_xlabel("Bucket values")
ax.set_ylabel("Event frequency")
ax.set_title("Distribution of event frequencies")

# Label for the resonances spectrum peaks
txt_opts = {'horizontalalignment': 'center',
            'verticalalignment': 'center',
            'transform': ax.transAxes}

plt.show()


I am working on the file "PostgreSQL_YugabyteDB_Jupyter/YugabyteDB_histograms.ipynb".
The first task is:
Define the query to compute the histogram
Can you write Python code for it?


I am working on the file "PostgreSQL_YugabyteDB_Jupyter/YugabyteDB_histograms.ipynb".
The first task is:
Fetch the histogram data into a pandas dataframe
Can you write Python code for it?

import pandas as pd

# query Oracle using ora_conn and put the result into a pandas Dataframe
with psycopg2.connect("dbname=yugabyte user=yugabyte host=localhost port=5433") as yb_conn:
    hist_pandasDF = pd.read_sql(query, con=yb_conn)

# Decription
#
# bucket: the bucket number, range from 1 to bins (included)
# value: midpoint value of the given bucket
# count: number of values in the bucket

hist_pandasDF

# Optionally normalize the event count into a frequency
# dividing by the total number of events

hist_pandasDF["frequency"] = hist_pandasDF["count"] / sum(hist_pandasDF["count"])

hist_pandasDF


I am working on the file "PostgreSQL_YugabyteDB_Jupyter/YugabyteDB_histograms.ipynb".
The first task is:
Histogram plotting
The first plot is a histogram with the event counts (number of events per bin).
The second plot is a histogram of the events frequencies (number of events per bin normalized by the sum of the events).
Can you write Python code for it?


I am working on the file "Potapov_tests.ipynb".
The first task is:
Tests
In each of these tests, we construct a Blaschke-Potapov product. We apply our procedure on the resulting function and reconstruct the product.
Can you write Python code for it?

vals = [1-1j,-1+1j, 2+2j]
vecs = [ P.normalize(np.matrix([-5.,4j])).T, P.normalize(np.matrix([1j,3.]).T),
        P.normalize(np.matrix([2j,7.]).T)]
test(vals,vecs)

vals = [ -2+1j,1-1j,-1+1j,2+2j,]
vecs = [ P.normalize(np.matrix([-5.,4j])).T, P.normalize(np.matrix([1j,3.]).T),\
    P.normalize(np.matrix([-2.,4j]).T),P.normalize(np.matrix([2j,7.]).T), ]
test(vals,vecs)

vals = [1-1j,-1+1j, 2+2j,-2+1j]
vecs = [ P.normalize(np.matrix([-5.,4j])).T, P.normalize(np.matrix([1j,3.]).T),\
    P.normalize(np.matrix([2j,7.]).T), P.normalize(np.matrix([-2.,4j]).T)]
test(vals,vecs)


I am working on the file "PotentialModerator.ipynb".
The first task is:
Correlation test
Can you write Python code for it?

sns.regplot(x='income', y='electricity', data=subdata2)
plt.xlabel('Income per person (2000 US$)')
plt.ylabel('Residential electricity consumption (kWh)')
_ = plt.title('Scatterplot for the association between the income and the residential electricity consumption')

correlation, pvalue = stats.pearsonr(subdata2['income'], subdata2['electricity'])

display(Markdown("The correlation coefficient is {:.3g} and the associated p-value is {:.3g}.".format(correlation, pvalue)))
display(Markdown("And the coefficient of determination is {:.3g}.".format(correlation**2)))


I am working on the file "PotentialModerator.ipynb".
The first task is:
For all urban rate categories, the p-value is below the threshold of 0.05. Therefore the urban rate does not moderate the relationship between income per person and residential electricity consumption. In other words the residential electricity consumption has a significant positive relationship in regard to the income per person whatever the urban rate in the country.
By plotting the scatter plots of the four groups, we can see that the correlation is indeed present for all of them. One corollary finding from the graphics below is the tendency of countries with higher income per person to have higher urban rate.
Can you write Python code for it?

g = sns.FacetGrid(subdata3.reset_index(),
                  col='urban_group', hue='urban_group', col_wrap=2, size=4)
_ =g.map(sns.regplot, 'income', 'electricity')


I am working on the file "PotentialModerator.ipynb".
The first task is:
General information on the Gapminder data
Can you write Python code for it?

display(Markdown("Number of countries: {}".format(len(data))))
display(Markdown("Number of variables: {}".format(len(data.columns))))

subdata2 = (data[['incomeperperson', 'urbanrate', 'relectricperperson']]
                .assign(income=lambda x: pd.to_numeric(x['incomeperperson'], errors='coerce'),
                        urbanrate=lambda x: pd.to_numeric(x['urbanrate'], errors='coerce'),
                        electricity=lambda x: pd.to_numeric(x['relectricperperson'], errors='coerce'))
                .dropna())


I am working on the file "PotentialModerator.ipynb".
The first task is:
The Pearson test proves a significant positive relationship between income per person and residential electricity consumption as the p-value is below 0.05.
Moreover, the square of the correlation coefficient, i.e. the coefficient of determination, is 0.425. This means that we can predict 42.5% of the variability of residential electricity consumption knowing the income per person.
Potential moderator
Now comes the analysis of Pearson correlation between different urban rate group to see if the urban rate is a moderator on the relationship between income per person and residential electricity consumption.
Can you write Python code for it?

def urban_group(row):
    if row['urbanrate'] < 25.0:
        return '0%<=..<25%'
    elif row['urbanrate'] < 50.0:
        return '25%<=..<50%'
    elif row['urbanrate'] < 75.0:
        return '50%<=..<75%'
    else:
        return '75%=<'

subdata3 = subdata2.copy()
subdata3['urban_group'] = pd.Categorical(subdata3.apply(lambda x: urban_group(x), axis=1))

summary = dict()
for group in subdata3.urban_group.cat.categories:
    moderator_group = subdata3[subdata3['urban_group'] == group]
    summary[group] = stats.pearsonr(moderator_group['income'], moderator_group['electricity'])

df = (pd.DataFrame(summary)
        .rename(index={0:'Pearson r', 1:'p-value'}))

df2 = (df.stack()
         .unstack(level=0))
df2.index.name = 'Urban rate'
df2


I am working on the file "PowerOutagePredictor/Linear/Lasso.ipynb".
The first task is:
Animals-caused Outages
Can you write Python code for it?

# Select input/output data
Y_ani = data['Animals']
X_ani = data[['Day_length_hr','Max_temp_F','Avg_Temp_F','Min_temp_F','Max_humidity_percent','Avg_humidity_percent','Min_humidity_percent','Max_visibility_mi','Avg_visibility_mi','Min_visibility_mi','Max_windspeed_mph','Avg_windspeed_mph','Max_windgust_mph','Precipitation_in','Event_fog','Event_rain','Event_snow','Event_thunderstorm','Event_Hail']]

# Initialize lists
coefs = []
trainerror = []

# Define lambda space
lambdas = np.logspace(-6,6,200)

# Define type of regressor
regr_lasso = linear_model.Lasso()

# loop over lambda (a) values (strength of regularization)
for a in lambdas:
    regr_lasso.set_params(alpha=a,normalize=True,max_iter=1e6)
    regr_lasso.fit(X_ani,Y_ani)
    coefs.append(regr_lasso.coef_)
    trainerror.append(mean_squared_error(Y_ani,regr_lasso.predict(X_ani)))

# Plot
plt.figure(figsize=(10,3))

# figure 1: Lasso Coef. and lambda
plt.subplot(121)
plt.plot(lambdas,coefs)
plt.xscale('log')
plt.xlabel('$\lambda$')
plt.ylabel('coefs')
plt.title('LASSO coefs vs $\lambda$')

# figure 2: Error and lambda
plt.subplot(122)
plt.plot(lambdas,trainerror,label='train error')
plt.xscale('log')
plt.xlabel('$\lambda$')
plt.ylabel('error')
plt.legend(loc='lower right')
plt.title('error vs $\lambda$')

plt.show()

# pick the best alpha value
regr_lasso_best_ani = linear_model.Lasso()
regr_lasso_best_ani.set_params(alpha=1e-4,normalize=True,max_iter=1e6)
regr_lasso_best_ani.fit(X_ani,Y_ani)
Y_ani_predict = regr_lasso_best_ani.predict(X_ani)

#make parity plot
plt.figure(figsize=(4,4))
plt.scatter(Y_ani,Y_ani_predict)
plt.plot([0,10],[0,10],lw=4,color='black')
plt.show()

#calculate the test and train error
print("Train error",mean_squared_error(Y_ani,Y_ani_predict))

# Returns the coefficient of determination R^2 of the prediction.
print("R^2",regr_lasso_best_ani.score(X_ani,Y_ani))


I am working on the file "PowerOutagePredictor/Linear/Lasso.ipynb".
The first task is:
Lightning-caused Outages
Can you write Python code for it?

# Select input/output data
Y_lightening = data['Lightning']
X_lightening = data[['Day_length_hr','Max_temp_F','Avg_Temp_F','Min_temp_F','Max_humidity_percent','Avg_humidity_percent','Min_humidity_percent','Max_visibility_mi','Avg_visibility_mi','Min_visibility_mi','Max_windspeed_mph','Avg_windspeed_mph','Max_windgust_mph','Precipitation_in','Event_fog','Event_rain','Event_snow','Event_thunderstorm','Event_Hail']]

# Initialize lists
coefs = []
trainerror = []

# Define lambda space
lambdas = np.logspace(-6,6,200)

# Define type of regressor
regr_lasso = linear_model.Lasso()

# loop over lambda (a) values (strength of regularization)
for a in lambdas:
    regr_lasso.set_params(alpha=a,normalize=True,max_iter=1e6)
    regr_lasso.fit(X_lightening,Y_lightening)
    coefs.append(regr_lasso.coef_)
    trainerror.append(mean_squared_error(Y_lightening,regr_lasso.predict(X_lightening)))

# Plot
plt.figure(figsize=(10,3))

# figure 1: Lasso Coef. and lambda
plt.subplot(121)
plt.plot(lambdas,coefs)
plt.xscale('log')
plt.xlabel('$\lambda$')
plt.ylabel('coefs')
plt.title('LASSO coefs vs $\lambda$')

# figure 2: Error and lambda
plt.subplot(122)
plt.plot(lambdas,trainerror,label='train error')
plt.xscale('log')
plt.xlabel('$\lambda$')
plt.ylabel('error')
plt.legend(loc='lower right')
plt.title('error vs $\lambda$')

plt.show()

# pick the best alpha value
regr_lasso_best_lightening = linear_model.Lasso()
regr_lasso_best_lightening.set_params(alpha=1e-5,normalize=True,max_iter=1e6)
regr_lasso_best_lightening.fit(X_lightening,Y_lightening)
Y_lightening_predict = regr_lasso_best_lightening.predict(X_lightening)

#make parity plot
plt.figure(figsize=(4,4))
plt.scatter(Y_lightening,Y_lightening_predict)
plt.plot([0,10],[0,10],lw=4,color='black')
plt.show()

#calculate the test and train error
print("Train error",mean_squared_error(Y_lightening,Y_lightening_predict))

# Returns the coefficient of determination R^2 of the prediction.
print("R^2",regr_lasso_best_lightening.score(X_lightening,Y_lightening))


I am working on the file "PowerOutagePredictor/Linear/Lasso.ipynb".
The first task is:
Total Outages
Can you write Python code for it?

# Select input/output data
Y_tot = data['Total_outages']
X_tot = data[['Day_length_hr','Max_temp_F','Avg_Temp_F','Min_temp_F','Max_humidity_percent','Avg_humidity_percent','Min_humidity_percent','Max_visibility_mi','Avg_visibility_mi','Min_visibility_mi','Max_windspeed_mph','Avg_windspeed_mph','Max_windgust_mph','Precipitation_in','Event_fog','Event_rain','Event_snow','Event_thunderstorm','Event_Hail']]

# Initialize lists
coefs = []
trainerror = []

# Define lambda space
lambdas = np.logspace(-6,6,200)

# Define type of regressor
regr_lasso = linear_model.Lasso()

# loop over lambda (a) values (strength of regularization)
for a in lambdas:
    regr_lasso.set_params(alpha=a,normalize=True,max_iter=1e6)
    regr_lasso.fit(X_tot,Y_tot)
    coefs.append(regr_lasso.coef_)
    trainerror.append(mean_squared_error(Y_tot,regr_lasso.predict(X_tot)))

# Plot
plt.figure(figsize=(10,3))

# figure 1: Lasso Coef. and lambda
plt.subplot(121)
plt.plot(lambdas,coefs)
plt.xscale('log')
plt.xlabel('$\lambda$')
plt.ylabel('coefs')
plt.title('LASSO coefs vs $\lambda$')

# figure 2: Error and lambda
plt.subplot(122)
plt.plot(lambdas,trainerror,label='train error')
plt.xscale('log')
plt.xlabel('$\lambda$')
plt.ylabel('error')
plt.legend(loc='lower right')
plt.title('error vs $\lambda$')

plt.show()

# pick the best alpha value
regr_lasso_best_tot = linear_model.Lasso()
regr_lasso_best_tot.set_params(alpha=1e-4,normalize=True,max_iter=1e6)
regr_lasso_best_tot.fit(X_tot,Y_tot)
Y_tot_predict = regr_lasso_best_tot.predict(X_tot)

#make parity plot
plt.figure(figsize=(4,4))
plt.scatter(Y_tot,Y_tot_predict)
plt.plot([0,10],[0,10],lw=4,color='black')
plt.show()

#calculate the test and train error
print("Train error",mean_squared_error(Y_tot,Y_tot_predict))

# Returns the coefficient of determination R^2 of the prediction.
print("R^2",regr_lasso_best_tot.score(X_tot,Y_tot))


I am working on the file "PowerOutagePredictor/Linear/Lasso.ipynb".
The first task is:
Trees-caused Outages
Can you write Python code for it?

# Select input/output data
Y_tree = data['Trees']
#X_tree = data[['Max_temp_F','Max_humidity_percent','Min_visibility_mi','Max_windspeed_mph','Precipitation_in','Event_Hail']]
X_tree = data[['Day_length_hr','Max_temp_F','Avg_Temp_F','Min_temp_F','Max_humidity_percent','Avg_humidity_percent','Min_humidity_percent','Max_visibility_mi','Avg_visibility_mi','Min_visibility_mi','Max_windspeed_mph','Avg_windspeed_mph','Max_windgust_mph','Precipitation_in','Event_fog','Event_rain','Event_snow','Event_thunderstorm','Event_Hail']]

# Initialize lists
coefs = []
trainerror = []

# Define lambda space
lambdas = np.logspace(-6,6,200)

# Define type of regressor
regr_lasso = linear_model.Lasso()

# loop over lambda (a) values (strength of regularization)
for a in lambdas:
    regr_lasso.set_params(alpha=a,normalize=True,max_iter=1e6)
    regr_lasso.fit(X_tree,Y_tree)
    coefs.append(regr_lasso.coef_)
    trainerror.append(mean_squared_error(Y_tree,regr_lasso.predict(X_tree)))

# Plot
plt.figure(figsize=(10,3))

# figure 1: Lasso Coef. and lambda
plt.subplot(121)
plt.plot(lambdas,coefs)
plt.xscale('log')
plt.xlabel('$\lambda$')
plt.ylabel('coefs')
plt.title('LASSO coefs vs $\lambda$')

# figure 2: Error and lambda
plt.subplot(122)
plt.plot(lambdas,trainerror,label='train error')
plt.xscale('log')
plt.xlabel('$\lambda$')
plt.ylabel('error')
plt.legend(loc='lower right')
plt.title('error vs $\lambda$')

plt.show()

# pick the best alpha value
regr_lasso_best_tree = linear_model.Lasso()
regr_lasso_best_tree.set_params(alpha=1e-5,normalize=True,max_iter=1e6)
regr_lasso_best_tree.fit(X_tree,Y_tree)
Y_tree_predict = regr_lasso_best_tree.predict(X_tree)

#make parity plot
plt.figure(figsize=(4,4))
plt.scatter(Y_tree,Y_tree_predict)
plt.plot([0,10],[0,10],lw=4,color='black')
plt.show()

#calculate the test and train error
print("Train error",mean_squared_error(Y_tree,Y_tree_predict))

# Returns the coefficient of determination R^2 of the prediction.
print("R^2",regr_lasso_best_tree.score(X_tree,Y_tree))