I am working on the file "Project_2_image_classification/dlnd_image_classification.ipynb".
The first task is:
Flatten Layer
Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor.  The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Can you write Python code for it?

def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    # TODO: Implement Function
    tensor_shape = x_tensor.get_shape().as_list()
    batch_size = tensor_shape[0]
    height = tensor_shape[1]
    width = tensor_shape[2]
    channels = tensor_shape[3]
    x_tensor = tf.reshape(x_tensor,[-1,height*width*channels])
    return x_tensor

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)


I am working on the file "Project_2_image_classification/dlnd_image_classification.ipynb".
The first task is:
Fully-Connected Layer
Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Can you write Python code for it?

def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function
    input_size = x_tensor.get_shape().as_list()[1]
    weight_f = tf.Variable(tf.random_normal([input_size,num_outputs],stddev=0.1))
    bias_f = tf.Variable(tf.zeros(shape=[num_outputs]))
    x_tensor = tf.add(tf.matmul(x_tensor,weight_f),bias_f)
    x_tensor = tf.nn.relu(x_tensor)
    return x_tensor

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)


I am working on the file "Project_2_image_classification/dlnd_image_classification.ipynb".
The first task is:
One-hot encode
Just like the previous code cell, you'll be implementing a function for preprocessing.  This time, you'll implement the one_hot_encode function. The input, x, are a list of labels.  Implement the function to return the list of labels as One-Hot encoded Numpy array.  The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode.  Make sure to save the map of encodings outside the function.
Hint: Don't reinvent the wheel.
Can you write Python code for it?

from sklearn.preprocessing import LabelBinarizer

def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    # TODO: Implement Function
    encoder = LabelBinarizer()
    encoder.fit(x)
    encoder.classes_ = list(range(10))
    x = encoder.transform(x)
    x = x.astype(np.float32)
    return x
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)


I am working on the file "Project_2_image_classification/dlnd_image_classification.ipynb".
The first task is:
Randomize Data
As you saw from exploring the data above, the order of the samples are randomized.  It doesn't hurt to randomize it again, but you don't need to for this dataset.
Preprocess all the data and save it
Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)


I am working on the file "Project_2_image_classification/dlnd_image_classification.ipynb".
The first task is:
Show Stats
Implement the function print_stats to print loss and validation accuracy.  Use the global variables valid_features and valid_labels to calculate validation accuracy.  Use a keep probability of 1.0 to calculate the loss and validation accuracy.
Can you write Python code for it?

def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    # TODO: Implement Function
    loss = session.run(cost, feed_dict={
                x: feature_batch,
                y: label_batch,
                keep_prob: 1.})
    valid_acc = session.run(accuracy, feed_dict={
                x: valid_features,
                y: valid_labels,
                keep_prob: 1.})
    print('Loss: {:>10.4f} Validation Accuracy: {:.6f}'.format(
                loss,
                valid_acc))


I am working on the file "Project_2_image_classification/dlnd_image_classification.ipynb".
The first task is:
Train on a Single CIFAR-10 Batch
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy.  Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)


I am working on the file "Project_2_image_classification/dlnd_image_classification.ipynb".
The first task is:
Train the Neural Network
Single Optimization
Implement the function train_neural_network to do a single optimization.  The optimization should use optimizer to optimize in session with a feed_dict of the following:
* x for image input
* y for labels
* keep_prob for keep probability for dropout
This function will be called for each batch, so tf.global_variables_initializer() has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
Can you write Python code for it?

def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    # TODO: Implement Function
    session.run(optimizer,feed_dict={x:feature_batch,y:label_batch,keep_prob:keep_probability})

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)


I am working on the file "Project_PageRank_Trailer.ipynb".
The first task is:
$\mathcal{Q}$2. Simulating a Random Walk
In the following code block, use the above functions to surf the web pages described by the Markov chain above. This code block may take a while to run. If it is taking more than a couple of minutes, maybe try reducing num_of_visits in order to at least get results.  Also, running your code while connected to AirBears may help if you have a slow internet connection at home.
Can you write Python code for it?

import random

# the url we want to begin with
url_start = "http://www.eecs.berkeley.edu/~kw1jjang/ee126/1.html"
current_url = url_start

# parameter to set the number of transitions you make/different pages you visit
num_of_visits = 1000

# dictionary of pages visited so far
visit_history = {}

# initialize dictionary since we know exactly where we'll end up
for i in range(1, 9):
    page = "http://www.eecs.berkeley.edu/~kw1jjang/ee126/" + str(i) + ".html"
    visit_history[page] = 0

for i in range(num_of_visits):
    # Your code here
    if i % 10 == 0:
        print i
    visit_history[current_url] += 1
    current_url = np.random.choice(parse_links(current_url,url_start))


I am working on the file "Project_PageRank_Trailer.ipynb".
The first task is:
$\mathcal{Q}$3. Try your hand at applying the above idea to a website you personally visit (somewhat) frequently!  Do a simple crawl (similar to the above) and see if you can figure out something interesting.  (Keep it simple.)
Can you write Python code for it?


I am working on the file "Project_PageRank_Trailer.ipynb".
The first task is:
Does this approximately match the invariant distribution you expected?
Generalizing to the Web
The toy websites given above conveniently form an irreducible Markov chain (look up what this means if you do not remember from class), but most of the web will not look like this. There will be fringes of the internet containing only self-loops, or some web pages which do not link to others at all. In order to account for such pathologies in the web, we need to make a more intelligent surfer. The simplest idea would be to just jump back to the starting page if there are no links found on the page you are on, and to always return to a "good" starting point with probability $p$ on every page.
This is a very naive scheme, and there are many more intelligent methods by which you can sample from the distribution of the Internet, accounting for its pathologies and all.
Ranking Berkeley Professors
The following code is a (weak) attempt to rank the Berkeley faculty based on a crawler which begins on the EECS research homepage.
Can you write Python code for it?

url_start = "http://www.eecs.berkeley.edu/Research/"
current_url = url_start
num_of_visits = 200

#List of professors obtained from the EECS page
profs = ['Abbeel','Agrawala','Alon','Anantharam','Arcak','Arias','Asanović','Bachrach','Bajcsy','Bodik','Bokor','Boser','Brewer','Canny','Chang-Hasnain','Culler','Darrell','Demmel','Fearing','Fox','Franklin','Garcia','Goldberg','Hartmann','Harvey','Hellerstein','Javey','Joseph','Katz','Keutzer','Liu','Klein','Kubiatowicz','Lee','Lustig','Maharbiz','Malik','Nguyen','Niknejad','Nikolic',"O'Brien",'Parekh','Patterson','Paxson','Pisano','Rabaey','Ramchandran','Roychowdhury','Russell','Sahai','Salahuddin','Sanders','Sangiovanni-Vincentelli','Sastry','Sen','Seshia','Shenker','Song','Song','Spanos','Stoica','Stojanovic','Tomlin','Tygar','Walrand','Wawrzynek','Wu','Yablonovitch','Yelick','Zakhor']
#Bad URLs help take care of some pathologies that ruin our surfing
bad_urls = ['http://www.erso.berkeley.edu/','http://www.eecs.berkeley.edu/Rosters/roster.name.nostudentee.html','http://www.eecs.berkeley.edu/Resguide/admin.shtml#aliases','http://www.eecs.berkeley.edu/department/EECSbrochure/c1-s3.html']

#Creating a dictionary to keep track of how often we come across a professor
profdict = {}
for i in profs:
    profdict[i] = 0

for i in range(num_of_visits):
    print  i , ' Visiting... ', current_url
    if random.random() < 0.95: #follow a link!
        url_list = parse_links(current_url, url_start)
        updated = False
        while not updated:
            current_url = random.choice(url_list)
            updated = True
            if current_url in bad_urls or "iris" in current_url or "Deptonly" in current_url or "anchor" in current_url or "erso" in current_url: #dealing with more pathologies:
                updated = False
        myopener = MyOpener()
        page = myopener.open(current_url)
        text = page.read()
        page.close()
        #Figuring out which professor is mentioned on a page.
        for p in profs:
            profdict[p]+= 1 if " " + p + "...


I am working on the file "Project_PageRank_Trailer.ipynb".
The first task is:
Simulation time!
We now want to empirically test what we solved above by modeling a random user hopping along those webpages.
We will start the user at "1.html" and behave as per the Markov chain above. In the code below, we simulate this and keep track of the average amount of time a user spends in each state. We will expect that after enough iterations, the fraction of time spent in each state should approach the stationary distribution.
We use the parse_links() method to parse all hyperlinks in a page. We use the library Beautiful Soup in order to complete this portion of the lab in order to easiliy parse pages. Once you download the latest release, you must build and install setup.py. Alternatively, use pip or easy_install (help).
Note that this code relies on having Python 2 installed -- it will not work with Python 3.
Can you write Python code for it?

import re
import sys
import urllib
import urlparse
import random
from bs4 import BeautifulSoup

class MyOpener(urllib.FancyURLopener):
   version = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15'

def domain(url):
    """
    Parse a url to give you the domain.
    """
    # urlparse breaks down the url passed it, and you split the hostname up
    # ex: hostname="www.google.com" becomes ['www', 'google', 'com']
    hostname = urlparse.urlparse(url).hostname.split(".")
    hostname = ".".join(len(hostname[-2]) < 4 and hostname[-3:] or hostname[-2:])
    return hostname

def parse_links(url, url_start):
    """
    Return all the URLs on a page and return the start URL if there is an error or no URLS.
    """
    url_list = []
    myopener = MyOpener()
    try:
        # open, read, and parse the text using beautiful soup
        page = myopener.open(url)
        text = page.read()
        page.close()
        soup = BeautifulSoup(text, "html")

        # find all hyperlinks using beautiful soup
        for tag in soup.findAll('a', href=True):
            # concatenate the base url with the path from the hyperlink
            tmp = urlparse.urljoin(url, tag['href'])
            # we want to stay in the berkeley EECS domain (more relevant later)...
            if domain(tmp).endswith('berkeley.edu') and 'eecs' in tmp:
                url_list.append(tmp)
        if len(url_list) == 0:
            return [url_start]
        return url_list
    except:
        return [url_start]

url_start = "http://www.eecs.berkeley.edu/~kw1jjang/ee126/1.html"
parse_links(url_start,url_start)


I am working on the file "Project_Python/py_bootcamp/decorators.ipynb".
The first task is:
function within function
Can you write Python code for it?

def welcome(name = 'Prithvi'):
    print 'inside welcome()'

    def namaste():
        return '\t inside the namaste()'

    def hariom():
        return '\t inside hariom()'

    if name == 'Prithvi':
        return namaste
    else:
        return hariom

x = welcome('potter')

print x()


I am working on the file "Project_search.ipynb".
The first task is:
Ideally we do not want scores to be the same for lots of documents. High TF-IDF scores in shorter documents should be more relevant - so we could try by boosting the score for documents that are shorter than average.
Can you write Python code for it?

def get_results_tfidf_boost(qry, corpus):
    idx = create_inverted_index(corpus)
    n = len(corpus)
    d = [len(x.split()) for x in corpus]
    d_avg = float(sum(d)) / len(d)
    score = Counter()
    for term in qry.split():
        if term in idx:
            i = idf(term, idx, n)
            for doc in idx[term]:
                f = float(idx[term][doc])
                score[doc] += i *  ( f / (float(d[doc]) / d_avg) )

    results=[]
    for x in [[r[0],r[1]] for r in zip(score.keys(), score.values())]:
        if x[1] > 0:
            # output [0] score, [1] doc_id
            results.append([x[1],x[0]])

    sorted_results = sorted(results, key=lambda t: t[0] * -1 )
    return sorted_results

from bokeh.charts import Scatter

results = get_results_tfidf_boost('zombie invasion', items_d)
print_results(results, 10)

# Plot score vs item length
df = pd.DataFrame({'score':[float(x[0]) for x in results],
                   'length':[len(items_d[x[1]].split()) for x in results]})

output_notebook()
p = Scatter(df, x='score', y='length')
show(p)


I am working on the file "Project_search.ipynb".
The first task is:
Implementing BM25
To implement BM25, we used the function get_results_bm25 that used arguments "query, corpus, and the index sizes. We then printed out the results using a Bokeh chart.
Can you write Python code for it?

def get_results_bm25(qry, corpus, k1=1.5, b=0.75):
    idx = create_inverted_index(corpus)
    # 1.Assign (integer) n to be the number of documents in the corpus
    n = len(corpus)
    # 2.Assign (list) d with elements corresponding to the number of terms in each document in the corpus
    d = [len(x.split()) for x in corpus]
    # 3.Assign (float) d_avg as the average document length of the documents in the corpus
    d_avg = float(sum(d)) / len(d)
    score = Counter()
    for term in qry.split():
        if term in idx:
            i = idf(term, idx, n)
            for doc in idx[term]:
                # 4.Assign (float) f equal to the number of times the term appears in doc
                f = float(idx[term][doc])
                # 5.Assign (float) s the BM25 score for this (term, document) pair
                s = i * (( f * (k1 + 1) ) / (f + k1 * (1 - b + (b * (float(d[doc]) / d_avg)))))
                score[doc] += s

    results=[]
    for x in [[r[0],r[1]] for r in zip(score.keys(), score.values())]:
        if x[1] > 0:
            results.append([x[1],x[0]])

    sorted_results = sorted(results, key=lambda t: t[0] * -1 )
    return sorted_results

results = get_results_bm25('zombie apacolypse', items_d)
print_results(results, 10)

!pip install bokeh
from bokeh.charts import Scatter

results = get_results_bm25('zombie apacolypse', items_d, k1=1.5, b=0.75)

# Plot score vs item length
df = pd.DataFrame({'score':[float(x[0]) for x in results],
                   'length':[len(items_d[x[1]].split()) for x in results]})
output_notebook()
p = Scatter(df, x='score', y='length')
show(p)


I am working on the file "Project_search.ipynb".
The first task is:
Implementing Random Forest Machine Learning
Using the example from class to implement random forest ranking algorithm.
Can you write Python code for it?

import findspark
import os
findspark.init(os.getenv('HOME') + '/spark-1.6.0-bin-hadoop2.6')
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages com.databricks:spark-csv_2.10:1.3.0 pyspark-shell'

import pyspark
try:
    print(sc)
except NameError:
    sc = pyspark.SparkContext()
    print(sc)

from pyspark.sql import SQLContext
import os

sqlContext = SQLContext(sc)
df = sqlContext.read.format('data/MovieSummaries/plot_summaries.tsv').options().options(header='true', inferSchema='true', delimiter=',') \
        .load(os.getcwd() + 'data/MovieSummaries/plot_summaries.tsv')

df.schema
df.dropna()

sqlContext.registerDataFrameAsTable(df,'dataset')
sqlContext.tableNames()

data_full = sqlContext.sql("select label_relevanceBinary, feature_1, feature_2, feature_3, feature_4 \
                       feature_5, feature_6, feature_7, feature_8, feature_9, feature_10 \
               from dataset").rdd

from pyspark.mllib.classification import SVMWithSGD, SVMModel
from pyspark.mllib.regression import LabeledPoint
from pyspark.mllib.feature import StandardScaler

label = data_full.map(lambda row: row[0])
features = data_full.map(lambda row: row[1:])

model = StandardScaler().fit(features)
features_transform = model.transform(features)

# Now combine and convert back to labelled points:
transformedData = label.zip(features_transform)
transformedData = transformedData.map(lambda row: LabeledPoint(row[0],[row[1]]))

transformedData.take(5)

data_train, data_test = transformedData.randomSplit([.75,.25],seed=1973)

print('Training data records = ' + str(data_train.count()))
print('Training data records = ' + str(data_test.count()))

from pyspark.mllib.tree import RandomForest

model = RandomForest.trainClassifier(data_train, numClasses=2, categoricalFeaturesInfo={},
                                     numTrees=400, featureSubsetStrategy="auto",
                                     impurity='gini', maxDepth=10, maxBins=32)


I am working on the file "Project_search.ipynb".
The first task is:
TF-IDF
To implement TF-IDF we used the function:
$$
IDF = log ( 1 + \frac{N}{n_t} )
$$
Can you write Python code for it?

import math

def idf(term, idx, n):
    return math.log( float(n) / (1 + len(idx[term])))

print(idf('zombie',idx,len(items_d)))
print(idf('survival',idx,len(items_d)))
print(idf('invasions',idx,len(items_d)))


I am working on the file "Project_search.ipynb".
The first task is:
TF-IDF Intuition
Can you write Python code for it?

from bokeh.charts import vplot

idx = create_inverted_index(items_d)

df = pd.DataFrame({'term':[x for x in idx.keys()],'freq':[len(x) for x in idx.values()],
                  'idf':[idf(x, idx, len(items_t)) for x in idx.keys()]})

output_notebook(hide_banner=True)
p1 = Bar(df.sort_values('freq', ascending=False)[:30], label=CatAttr(columns=['term'], sort=False), values='freq',
        plot_width=800, plot_height=400)
p2 = Bar(df.sort_values('freq', ascending=False)[:30], label=CatAttr(columns=['term'], sort=False), values='idf',
        plot_width=800, plot_height=400)
p = vplot(p1, p2)
show(p)


I am working on the file "Project_search.ipynb".
The first task is:
TF-IDF Ranking
We then created an inverted index for the TD-IDF ranking
Can you write Python code for it?

def create_inverted_index(corpus):
    idx={}
    for i, doc in enumerate(corpus):
        for word in doc.split():
            if word in idx:
                if i in idx[word]:
                    # Update document's frequency
                    idx[word][i] += 1
                else:
                    # Add document
                    idx[word][i] = 1
            else:
                # Add term
                idx[word] = {i:1}
    return idx

def get_results_tfidf(qry, idx, n):
    score = Counter()
    for term in qry.split():
        if term in idx:
            i = idf(term, idx, n)
            for doc in idx[term]:
                score[doc] += idx[term][doc] * i

    results=[]
    for x in [[r[0],r[1]] for r in zip(score.keys(), score.values())]:
        if x[1] > 0:
            results.append([x[1],x[0]])

    sorted_results = sorted(results, key=lambda t: t[0] * -1 )
    return sorted_results

idx = create_inverted_index(items_d)
results = get_results_tfidf('lookout action bike zombie', idx, len(items_d))
print_results(results,10)


I am working on the file "Project_search.ipynb".
The first task is:
compute dtf for item descriptions
Can you write Python code for it?


I am working on the file "Project_search.ipynb".
The first task is:
enter different queries
Can you write Python code for it?

results = get_results_tf('ghouls and ghosts', idx)
print_results(results, 10)

import pandas as pd
from bokeh.plotting import output_notebook, show
from bokeh.charts import Bar
from bokeh.charts.attributes import CatAttr
#from bokeh.models import ColumnDataSource

df = pd.DataFrame({'term':[x for x in idx.keys()],'freq':[len(x) for x in idx.values()]})

output_notebook(hide_banner=True)
p = Bar(df.sort_values('freq', ascending=False)[:30], label=CatAttr(columns=['term'], sort=False), values='freq',
        plot_width=800, plot_height=400)
show(p)


I am working on the file "Project_search.ipynb".
The first task is:
improve the ranking function
Can you write Python code for it?

def get_results_tf(qry, idx):
    score = Counter()
    for term in qry.split():
        for doc in idx[term]:
            score[doc] += 1

    results=[]
    for x in [[r[0],r[1]] for r in zip(score.keys(), score.values())]:
        if x[1] > 0:
            results.append([x[1],x[0]])

    sorted_results = sorted(results, key=lambda t: t[0] * -1 )
    return sorted_results;

idx = create_inverted_index(items_d)
results = get_results_tf('zombies', idx)
print_results(results,20)


I am working on the file "Project_search.ipynb".
The first task is:
sparsity of term frequency matrix
We took the approach of using Bokeh for displaying the sparsity of term frequency matrix
Can you write Python code for it?

#!pip install bokeh

import pandas as pd
from bokeh.plotting import figure, output_notebook, show, vplot

# sparsity as a function of document count
n = []
s = []
for i in range(100,1000,100):
    corpus = items_d[0:i]
    tfm, lexicon = get_tfm(corpus)
    c = [ [x.count(0), x.count(1)] for x in tfm]
    n_zero = sum([ y[0] for y in c])
    n_one = sum( [y[1] for y in c])
    s.append(1.0 - (float(n_one) / (n_one + n_zero)))
    n.append(i)

output_notebook(hide_banner=True)
p = figure(x_axis_label='Documents', y_axis_label='Sparsity', plot_width=400, plot_height=400)
p.line(n, s, line_width=2)
p.circle(n, s, fill_color="white", size=8)
show(p)


I am working on the file "Project_search.ipynb".
The first task is:
start by computing frequncy of entire corpus
Can you write Python code for it?

tf = {}
for doc in corpus:
    for word in doc.split():
        if word in tf:
            tf[word] += 1
        else:
            tf[word] = 1
print(tf)


I am working on the file "Project_search.ipynb".
The first task is:
term freq matrix

with the lexicon we are able to compute the term freq matrix
Can you write Python code for it?

def get_tfm(corpus):

    def get_lexicon(corpus):
        lexicon = set()
        for doc in corpus:
            lexicon.update([word for word in doc.split()])
        return list(lexicon)

    lexicon = get_lexicon(corpus)

    tfm =[]
    for doc in corpus:
        tfv = [0]*len(lexicon)
        for term in doc.split():
            tfv[lexicon.index(term)] += 1

        tfm.append(tfv)

    return tfm, lexicon

#test_corpus = ['mountain bike', 'road bike carbon', 'bike helmet']
#tfm, lexicon = get_tfm(test_corpus)
#print lexicon
#print tfm


I am working on the file "Projects/01_Projeto_HillaryTrump_Twitter.ipynb".
The first task is:
Ao final deste processo já temos nossa base de dados dividido em duas variáveis: X e y.
X corresponde ao bag of words, ou seja, cada linha consiste de um twitter e cada coluna de uma palavra presente no vocabulário da base dados. Para cada linha/coluna é atribuído um valor que corresponde a quantidade de vezes que aquela palavra aparece no respectivo tweet. Se a palavra não está presente, o valor de 0 é atribuído.
y corresponde a classe de cada tweet: hillary, tweet do perfil @HillaryClinton e trump, tweet do perfil @realDonaldTrump.
Testando diferentes modelos
Vamos testar os diferentes modelos estudados com a base de dados criada e escolher aquele que melhor generaliza o conjunto de dados. Para testar os modelos, vamos utilizar validação cruzada de 10 folds. Aplique os modelos estudados:

KNN (teste diferentes valores de K e escolha o melhor)
Árvore de Decisão
SVM (varie o valor de C e escolha o melhor)

Além disso, teste dois outros modelos:

RandomForest
Naive Bayes

Para estes dois, pesquise no Google como utiliza-los no Scikit-Learn.
Para cada modelo imprima a acurácia no treino e na média dos 10 folds da validação cruzada. A escolha do melhor deve ser feita a partir do valor da média da validação cruzada.
O melhor modelo será utilizado para classificar outros textos extraídos do twitter e na implementação da página web.
Atenção: dada a quantidade de dados, alguns modelos pode demorar alguns minutos para executar
Can you write Python code for it?


I am working on the file "Projects/01_Projeto_HillaryTrump_Twitter.ipynb".
The first task is:
Bigrams e Trigram mais frequentes da Hillary
Can you write Python code for it?

#Pegando os bigram e trigram mais frequentes

from nltk.collocations import BigramCollocationFinder, TrigramCollocationFinder
from nltk.metrics import BigramAssocMeasures, TrigramAssocMeasures

bcf = BigramCollocationFinder.from_words(words_h)
tcf = TrigramCollocationFinder.from_words(words_h)

bcf.apply_freq_filter(3)
tcf.apply_freq_filter(3)

result_bi = bcf.nbest(BigramAssocMeasures.raw_freq, 5)
result_tri = tcf.nbest(TrigramAssocMeasures.raw_freq, 5)

hillary_frequent_bitrigram = []

for r in result_bi:
    w_ = " ".join(r)
    print(w_)
    hillary_frequent_bitrigram.append(w_)

print
for r in result_tri:
    w_ = " ".join(r)
    print(w_)
    hillary_frequent_bitrigram.append(w_)


I am working on the file "Projects/01_Projeto_HillaryTrump_Twitter.ipynb".
The first task is:
Construindo um bag of words
Can you write Python code for it?

# Cada token é concatenado em uma única string que representa um tweet
# Cada classe é atribuída a um vetor (hillary, trump)

# Instâncias: [t1, t2, t3, t4]
# Classes: [c1, c2, c3, c4]

all_tweets = []
all_class = []

for t in all_texts:
    all_tweets.append(t['text'])
    all_class.append(t['class_'])

print("Criar o bag of words...\n")

#Número de features, coluna da tabela
max_features = 2000

from sklearn.feature_extraction.text import CountVectorizer

# Initialize the "CountVectorizer" object, which is scikit-learn's
# bag of words tool.
vectorizer = CountVectorizer(analyzer = "word",   \
                             tokenizer = None,    \
                             preprocessor = None, \
                             stop_words = None,   \
                             max_features = max_features)

# fit_transform() does two functions: First, it fits the model
# and learns the vocabulary; second, it transforms our training data
# into feature vectors. The input to fit_transform should be a list of
# strings.
X = vectorizer.fit_transform(all_tweets)

# Numpy arrays are easy to work with, so convert the result to an
# array
X = X.toarray()
y = all_class

print("Train data: OK!")

X.shape


I am working on the file "Projects/01_Projeto_HillaryTrump_Twitter.ipynb".
The first task is:
O objeto dict_ representa todos os textos associados a classe correspondente.
Na etapa de pré-processamento vamos fazer algumas operações:

retirar dos textos hashtags, usuários e links. Essas informações serão incluídas em listas separadas paara serem usadas posteriormente.
serão eliminados stopwords, símbolos de pontuação, palavras curtas;
numerais tambéms serão descartados, mantendo apenas palavras.

Essas etapas de pré-processamento dependem do objetivo do trabalho. Pode ser de interessa, a depender da tarefa de classificação, manter tais símbolos. Para o nosso trabalho, só é de interesse as palavras em si.
Para esta tarefa, vamos utilizar também o NLTK, conjunto de ferramentas voltadas para o processamento de linguagem natural.
vamos criar um método para isso, já que iremos utiliza-lo mais adiante com a base de teste:
Can you write Python code for it?

from unicodedata import normalize, category
from nltk.tokenize import regexp_tokenize
from collections import Counter, Set
from nltk.corpus import stopwords
import re

def pre_process_text(text):

    # Expressão regular para extrair padrões do texto. São reconhecidos (na ordem, o símbolo | separa um padrão):
    # - links com https, links com http, links com www, palavras, nome de usuários (começa com @), hashtags (começa com #)
    pattern = r'(https://[^"\' ]+|www.[^"\' ]+|http://[^"\' ]+|[a-zA-Z]+|\@\w+|\#\w+)'

    #Cria a lista de stopwords
    english_stop = stopwords.words(['english'])

    users_cited = []
    hash_tags = []
    tokens = []

    text = text.lower()

    patterns = regexp_tokenize(text, pattern)

    users_cited = [e for e in patterns if e[0] == '@']
    hashtags = [e for e in patterns if e[0] == '#']

    tokens = [e for e in patterns if e[:4] != 'http']
    tokens = [e for e in tokens if e[:4] != 'www.']
    tokens = [e for e in tokens if e[0] != '#']
    tokens = [e for e in tokens if e[0] != '@']
    tokens = [e for e in tokens if e not in english_stop]
    tokens = [e for e in tokens if len(e) > 3]

    return users_cited, hashtags, tokens

users_cited_h = []  # armazena os usuários citatdos por hillary
users_cited_t = []  # armazena os usuários citados por trump

hashtags_h = [] # armazena as hashtags de hillary
hashtags_t = [] # armazena as hashtags de trump

words_h = [] # lista de palavras que apareceram no discurso de hillary
words_t = [] # lista de palavras que apareceram no discurso de trump

all_tokens = [] # armazena todos os tokens, para gerar o vocabulário final
all_texts = [] # armazena todos

for d in dict_:

    text_ =  dict_[d][0]
    class_ = dict_[d][1]

    users_, hash_, tokens_ = pre_process_text(text_)

    if class_ == "HillaryClinton":
        class_ = "hillary"
        users_cited_h += users_
        hashtags_h += hash_
        words_h += tokens_

    elif class_ == "realDonaldTrump":
       ...


I am working on the file "Projects/02_RecommenderSystem_Movies.ipynb".
The first task is:

Usando a API do IMDb
Podemos utilizar a API do IMDb para retornar algumas informações a mais sobre os filmes recomendados. O código a seguir importa a biblioteca e com as informações contidas na tabela links da Movielens acessa detalhes dos filmes na base do IMDb. Vamos utilizar a biblioteca imdbpie: https://github.com/richardasaurus/imdb-pie.
Can you write Python code for it?

# Imports necessários

from imdbpie import Imdb
from IPython.display import Image, display

# Método que retorna o ID do IMDb dado o id do filme

def get_imdb_id(id_movie):

    imdbid = int(links.loc[idx[id_movie], 'imdbId'])

    imdbid = "tt%.7d" % imdbid

    return imdbid

imdbid = get_imdb_id(300)
print(imdbid)

# Carregando a biblioteca do IMDb
imdb = Imdb()
imdb = Imdb(anonymize=True)

imdbid = get_imdb_id(1)
title = imdb.get_title_by_id(imdbid)
print(title)

Image(title.cover_url)

print(title.title)
print(title.year)
print(title.genres)
print(title.release_date)

cast = title.cast_summary
for person in cast:
    print(person.name)

reviews = imdb.get_title_reviews(imdbid, max_results=2)

for review in reviews:
    print("Review by %s" % review.username)
    if(review.rating is not None):
        print("Rating: %.1f" % review.rating)
    print("Review: %s" % review.text)
    print()


I am working on the file "Projects/02_RecommenderSystem_Movies.ipynb".
The first task is:

Sistema De Recomendação Simples
Só para que a gente veja alguns desses métodos funcionando, vamos implementar um "Sistema de Recomendação" bem simples. A proposta é recomendar os filmes avaliados com nota 5 que foram assistidos por usuários que já assistiram os mesmos filmes com nota 4 e 5 do usuário em questão.
Can you write Python code for it?

# Usuário para qual a recomendação será apresentada, chamarei de usuário A
selected_user = 1

# Filmes com notas 4 e 5 avaliados por este usuário
my_movies = get_movies_by_user(selected_user, rating_cut=4, list_=True)

# Lista de todos os usuários que avaliaram os filmes assistidos por A
all_users = []

for movie in my_movies:
    all_users = all_users + get_users_by_movie(movie, rating_cut=5, list_=True)

# Para eliminar os usuários repetidos, transformamos a lista em um conjunto e depois convertemos novamente em uma lista
all_users = list(set(all_users))

# Neste passo, pegamos todos os filmes nota 5 avaliados pelos usuários de all_users
all_movies = []

for user in all_users:
    movies_ = get_movies_by_user(user, rating_cut=5, list_=True)
    all_movies = all_movies + movies_

# Removemos os filmes repetidos e aqueles já assistidos por A
all_movies = list(set(all_movies) - set(my_movies))

# Apresentamos a lista dos filmes
print("Foram encontrados: " + str(len(all_movies)) + " filmes")
for movie in all_movies:
    print("\t"+ get_movie_title(movie))


I am working on the file "Projects/02_RecommenderSystem_Movies.ipynb".
The first task is:
A primeira função que vamos implementar é a de similaridade de usuários:
$ sim(x,y) = \frac{\sum_{
    s \in S_{xy}}
    {
        (r_{x,s} - \bar{r_{x}})
        (r_{y,s} - \bar{r_{y}})
    }
}
{
    \sqrt{
        \sum_{s \in S_{xy}}{
            (r_{x,s} - \bar{r_{x}})^2
        }
        \sum_{s \in S_{xy}}{
            (r_{y,s} - \bar{r_{y}})^2
        }
    }
} $, onde:

$S_x$: itens avaliados pelo usuário $x$;
$S_y$: itens avaliados pelo usuário $y$;
$S_{xy}$: o conjunto de todos os itens que foram avaliados tanto por x quanto por y, em outras palavras, a interseção dos conjuntos $S_x$ e $S_y$;
$r_{x,s}$: rating do usuário $x$ para o item $s$;
$r_{y,s}$: rating do usuário $y$ para o item $s$;
$\bar{r_x}$: média do ratings dos filmes avaliados por $x$
$\bar{r_y}$: média do ratings dos filmes avaliados por $y$

Para tal, precisamos definir uma função que dados dois usuários, retorne os itens avaliados por ambos.
Can you write Python code for it?

def intersect_items(id_user_x, id_user_y):

    """Retorna duas listas de ratings. Os ratings correspondem aos itens avaliados por x e y.
        É retornada duas listas distintas já que os itens são os mesmo, mas as avaliações são distintas.
        Isso irá facilitar na hora de calcuar a similaridade dos usuários.

    Keyword arguments:
    id_user_x -- id do usuário x
    id_user_y -- id do usuário y

    """

    dict_x = movies_user_false[id_user_x]
    dict_y = movies_user_false[id_user_y]

    all_keys = set(list(dict_x.keys()) + list(dict_y.keys()))

    ratings_x = []
    ratings_y = []

    for key in all_keys:
        if key in dict_x and key in dict_y:
            ratings_x.append(dict_x[key])
            ratings_y.append(dict_y[key])

    ratings_x = np.array(ratings_x)
    ratings_y = np.array(ratings_y)

    return ratings_x, ratings_y


I am working on the file "Projects/02_RecommenderSystem_Movies.ipynb".
The first task is:
Da para perceber que esta não é uma abordagem muito boa. Como é retornado somente filmes com nota 5 não existe nenhum critério para ordená-los, fazendo com que seja exibido todos os filmes recomendados (1106). Na próxima seção, vamos apresentar uma proposta em que utilizamos a similaridade de usuários para melhorar essa recomendação.

Sistema de Recomendação Colaborativo
Uma forma melhor de recomendar é usar os ratings que os usuários deram aos filmes para classificar os usuários de acordo com uma similaridade. A recomendação é feita a partir dos usuários mais similares. Para isso, devemos calcular a distância do usuário em questão com todos os usuários da base e, por fim, a nota que o usuário daria. Como já foi dito, vamos utilizar as equações apresentadas anteriomente.
Can you write Python code for it?

# Imports necessários para esta seção

import math
import numpy as np
import operator
from sklearn.externals import joblib


I am working on the file "Projects/02_RecommenderSystem_Movies.ipynb".
The first task is:
Melhorando a forma de apresentação, temos:
Can you write Python code for it?

count = 1
top10 = sorted_items[:10]
print("Filmes recomendados para o usuário %s:" % user)
for movie in top10:
    print("\t %.2d" % count, "[%.1f]" % movie[1], get_movie_title(movie[0]), )
    count += 1


I am working on the file "Projects/02_RecommenderSystem_Movies.ipynb".
The first task is:
Mesmo com esses métodos algumas operações podem ter um certo custo computacional já que serão chamadas várias vezes. Por exemplo, quando vamos calcular a similaridade de um usuário com todos da base, isso tem um certo custo. Por conta disso, algumas informações serão geradas antes e armazenadas em variáveis na memória. Tais informações serão geradas nas células a seguir. Essas variáveis só serão utilizadas nos métodos em que são utilizadas muitas vezes.
Can you write Python code for it?

'''
    Neste trecho vamos armazenar em memória as informações de filmes avaliados pelos usuários. Isso evitar
    fazermos muitos acesso a estrutura do DataFrame.
'''

all_users = get_all_users()

movies_user_true = {}
movies_user_false = {}

for user in all_users:
    movies_user_true[user] = get_movies_by_user(user, list_=True)
    movies_user_false[user] = get_movies_by_user(user, list_=False)


I am working on the file "Projects/02_RecommenderSystem_Movies.ipynb".
The first task is:
Para o cálculo do predict rating é necessário pegar os Top $N$ usuários mais semelhantes ao usuário para qual desejamos prover a recomendação. Para encontrar essa lista, precisamos calcular a similaridade deste usuário com todos da base, ordenar a lista resultante pela similaridade e, por fim, retornar os TopN. Essa tarefa pode ser um tanto custosa para se fazer no processo de recomendação. Por isso, vamos pré-calcular esta similaridade e armazenar em um arquivo que será carregado quando necessário. A opção de salvar em arquivo é porque desta forma podemos utiliza-lo em outro momento quando a seção do Jupyter ou do Python for finalizada. Um outro motivo é que essa tarefa é um tanto custosa e, a depender do número de usuários, pode demorar.
O código a seguir calcular essa similaridade e persiste o objeto com todas as similaridades de todos os usuários da base no arquivo usersimilarity.pkl. Só é necessário rodar este código na primeira vez que estiver rodando esse tutorial ou, se por um acaso, a base de usuário mudar.
Can you write Python code for it?

all_users = get_all_users()

map_similarity = {}

for i in all_users:
    for j in all_users:
        if(i < j):
            map_similarity[(i, j)] = similarity(i, j)

joblib.dump(map_similarity, "usersimilarity.pkl")


I am working on the file "Projects/02_RecommenderSystem_Movies.ipynb".
The first task is:
Um vez que a função foi dos TopN foi definida, podemos usá-la na implementação na função do predict rating:
Can you write Python code for it?

def predict_rating(id_user, id_movie):

    N = 20

    items_user = movies_user_false[id_user]
    all_user_values = [items_user[x] for x in items_user]

    mean_user = np.mean(all_user_values)

    topN_users = get_topneighbors_rateditem(id_user, id_movie, N)

    sum_ = 0
    sum_k = 0

    for topuser in topN_users:
        similarity = topuser[1]
        user_u = topuser[0][1]

        rating_user_u = get_rating_by_user_movie(user_u, id_movie)

        items_user_u = movies_user_false[user_u]

        all_user_u_values = [items_user_u[x] for x in items_user_u]

        mean_user_u = np.mean(all_user_u_values)

        sum_ += similarity * (rating_user_u - mean_user_u)

        sum_k += abs(similarity)

    if sum_k == 0:
        k = 0
    else:
        k = 1 / sum_k

    rating_final = mean_user + k * sum_

    return rating_final

print("Nota do usuário 1 para o filme 30:")
predict_rating(1, 30)


I am working on the file "Projects/AptListingsAnalysis.ipynb".
The first task is:
As seen here, introducing feature categories and manager performance has improved the model. In particular, manager_skill shows up as the dominant feature in terms of importance in this Random Forest model.
Step 6: Image Classification Attempt
I did not use the actual listing images in my model. Here, I outline an attempt at classifying the listing based on image quality using a "blurry image" detector that I created with a Convolutional Neural Network. For more details see my discussion of that project.
Can you write Python code for it?

import numpy as np
from keras.models import Sequential
from keras.layers.convolutional import Convolution2D
from keras.layers.pooling import MaxPooling2D
from keras.layers.core import Flatten, Dense, Activation, Dropout
from keras.preprocessing import image

# My neural network layer sequence is based on the original LeNet architecture
model = Sequential()

# Layer 1
model.add(Convolution2D(32, 5, 5, input_shape=(192, 192, 3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

# Layer 2
model.add(Convolution2D(64, 5, 5))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())

# Layer 3
model.add(Dense(1024))
model.add(Activation("relu"))
model.add(Dropout(0.5))

# Layer 4
model.add(Dense(512))
model.add(Activation("relu"))
model.add(Dropout(0.5))

# Layer 5
model.add(Dense(2))
model.add(Activation("softmax"))

# "lenet_weights.h5" is the file containing weights from my trained neural network.
model.load_weights('lenet_weights.h5')

# Loading three images from the dataset.
# img_1 & img_2 are from the same High popularity listing
# img_3 is from a Low popularity listing.
pics = []

img_1 = image.load_img('6811966_1.jpg', target_size=(192,192))
pics.append(np.asarray(img_1))
img_2 = image.load_img('6811966_2.jpg', target_size=(192,192))
pics.append(np.asarray(img_2))
img_3 = image.load_img('6812150_1.jpg', target_size=(192,192))
pics.append(np.asarray(img_3))
pics_array = np.stack(pics)/255.
plt.figure(figsize=(12,12))
plt.subplot(131),plt.imshow(img_1),plt.title('6811966, interest_level: High')
plt.xticks([]), plt.yticks([])
plt.subplot(132),plt.imshow(img_2),plt.title('6811966, interest_level: High')
plt.xticks([]), plt.yticks([])
plt.subplot(133),plt.imshow(img_3),plt.title('6812150, interest_level: Low')
plt.xticks([]), plt.yticks([])
plt.show()

model.predict_classes(pics_array)


I am working on the file "Projects/AptListingsAnalysis.ipynb".
The first task is:
Random Forest
Can you write Python code for it?

rf = RandomForestClassifier(n_estimators=500, n_jobs=-1)
rf.fit(X_train, y_train)

y_test_predicted_proba = rf.predict_proba(X_test)
log_loss(y_test, y_test_predicted_proba)


I am working on the file "Projects/AptListingsAnalysis.ipynb".
The first task is:
Step 1: Load Data
Can you write Python code for it?

# Load the training dataset from Kaggle.
df = pd.read_json('train.json')
print df.shape

df.head(2)


I am working on the file "Projects/AptListingsAnalysis.ipynb".
The first task is:
Step 4: More Complex Features
Splitting out categories into 0/1 dummy variables
Can you write Python code for it?

# Reduce 1556 unique category text values into 34 main categories

def reduce_categories(full_list):
    reduced_list = []
    for i in full_list:
        item = i.lower()
        if 'cats allowed' in item:
            reduced_list.append('cats')
        if 'dogs allowed' in item:
            reduced_list.append('dogs')
        if 'elevator' in item:
            reduced_list.append('elevator')
        if 'hardwood' in item:
            reduced_list.append('elevator')
        if 'doorman' in item or 'concierge' in item:
            reduced_list.append('doorman')
        if 'dishwasher' in item:
            reduced_list.append('dishwasher')
        if 'laundry' in item or 'dryer' in item:
            if 'unit' in item:
                reduced_list.append('laundry_in_unit')
            else:
                reduced_list.append('laundry')
        if 'no fee' in item:
            reduced_list.append('no_fee')
        if 'reduced fee' in item:
            reduced_list.append('reduced_fee')
        if 'fitness' in item or 'gym' in item:
            reduced_list.append('gym')
        if 'prewar' in item or 'pre-war' in item:
            reduced_list.append('prewar')
        if 'dining room' in item:
            reduced_list.append('dining')
        if 'pool' in item:
            reduced_list.append('pool')
        if 'internet' in item:
            reduced_list.append('internet')
        if 'new construction' in item:
            reduced_list.append('new_construction')
        if 'wheelchair' in item:
            reduced_list.append('wheelchair')
        if 'exclusive' in item:
            reduced_list.append('exclusive')
        if 'loft' in item:
            reduced_list.append('loft')
        if 'simplex' in item:
            reduced_list.append('simplex')
        if 'fire' in item:
            reduced_list.append('fireplace')
        if 'lowrise' in item or 'low-rise' in item:
            reduced_list.append('lowrise')
        if 'midrise' in item or 'mid-rise' in item:
            reduced_list.append('midrise')
...


I am working on the file "Projects/AptListingsAnalysis.ipynb".
The first task is:
The above bar plot shows feature importance for the Random Forest classifier. "Price" is the most informative feature related to the target variable "interest_level".
Naive Bayes
Can you write Python code for it?

bnb = BernoulliNB()
bnb.fit(X_train, y_train)

y_test_predicted_proba = bnb.predict_proba(X_test)
log_loss(y_test, y_test_predicted_proba)

gnb = GaussianNB()
gnb.fit(X_train, y_train)

y_test_predicted_proba = gnb.predict_proba(X_test)
log_loss(y_test, y_test_predicted_proba)


I am working on the file "Projects/AptListingsAnalysis.ipynb".
The first task is:
Total number of columns is 14 + 1 target:
- 1 target variable (interest_level), with classes low, medium, high
- 1 list of photo links
- lat/long, street address, display address
- listing_id, building_id, manager_id
- numerical (price, bathrooms, bedrooms)
- created date
- text (description, features)
Features for modeling:
- bathrooms
- bedrooms
- created date (calculate age of posting in days)
- description (number of words in description)
- features (number of features)
- photos (number of photos)
- price
- features (split into category indicators)
- manager_id (with manager skill level)
Further opportunities for modeling:
- description (with NLP)
- building_id (with a building popularity level)
- photos (quality, discussed in Step 6)
Can you write Python code for it?

# Distribution of target value: interest_level

s = df.groupby('interest_level')['listing_id'].count()
s.plot.bar();

df_high = df.loc[df['interest_level'] == 'high']
df_medium = df.loc[df['interest_level'] == 'medium']
df_low = df.loc[df['interest_level'] == 'low']

plt.figure(figsize=(6,10))
plt.scatter(df_low.longitude, df_low.latitude, color='yellow', alpha=0.2, marker='.', label='Low')
plt.scatter(df_medium.longitude, df_medium.latitude, color='green', alpha=0.2, marker='.',  label='Medium')
plt.scatter(df_high.longitude, df_high.latitude, color='purple', alpha=0.2, marker='.',  label='High')

plt.xlim(-74.04,-73.80)
plt.ylim(40.6,40.9)
plt.title('Map of the listings in NYC')
plt.ylabel('N Lat.')
plt.xlabel('W Long.')
plt.legend(loc=2);


I am working on the file "Projects/BlindSpot/synthetic_spectra.ipynb".
The first task is:
Finally, we convolve the Gaussian kernel with the original spectrum, being careful to preserve the shape of the original spectrum.
Can you write Python code for it?

degraded = np.convolve(spectrum[:, 1], window, mode='same')

fig, ax = plt.subplots(1, 1, figsize=(8, 4))

ax.set_xlabel('Wavelength ($\\AA$)', fontsize=20.0)
ax.set_ylabel('Flux', fontsize=20.0)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.set_xlim(5000., 7000.)
ax.set_ylim(0.0, 0.5)

ax.plot(spectrum[:,0], degraded/1.e7, '-', lw=2, color='#800000')


I am working on the file "Projects/BlindSpot/synthetic_spectra.ipynb".
The first task is:
For comparison we can load an SDSS template of and M3 star, which is presumably warmer than the spectrum created here.
Can you write Python code for it?

sdss_template = np.genfromtxt('../../../Projects/BlindSpot/spectra/tmp/SDSS_DR2_M3.template', usecols=(0, 1))

fig, ax = plt.subplots(1, 1, figsize=(8, 4))

ax.set_xlabel('Wavelength ($\\AA$)', fontsize=20.0)
ax.set_ylabel('Flux', fontsize=20.0)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.set_xlim(5000., 7000.)
ax.set_ylim(0., 10.)

ax.plot(sdss_template[:, 0], sdss_template[:, 1]/10. + 3.0, '-', lw=2, color='#444444')
ax.plot(spectrum[:,0], degraded/1.e6, '-', lw=2, color='#800000')


I am working on the file "Projects/BlindSpot/synthetic_spectra.ipynb".
The first task is:
Great. So, now we have properly pulled the data to the local directory structure. However, there are several complications that need to get figured out.
 1. Data must be saved to a temporary file.
 2. Data must be unziped (unxz).
 3. All instances of Fortran doubles must be converted from D exponentials to E.
Now, we must trim the file as there is a significant amount of data that we don't need: everything below 3000 Å and above 4.0 microns.
Can you write Python code for it?

spectrum = np.array([line for line in spectrum if 3000.0 <= line[0] <= 40000.0])
spectrum[:, 1] = 10.0**(spectrum[:, 1] - 8.0)


I am working on the file "Projects/BlindSpot/synthetic_spectra.ipynb".
The first task is:
Let's take a look at part of the raw spectrum, say the optical.
Can you write Python code for it?

fig, ax = plt.subplots(1, 1, figsize=(8, 4))

ax.set_xlabel('Wavelength ($\\AA$)', fontsize=20.0)
ax.set_ylabel('Flux', fontsize=20.0)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.set_xlim(5000., 7000.)

ax.plot(spectrum[:,0], spectrum[:,1], '-', color='#800000')


I am working on the file "Projects/BlindSpot/synthetic_spectra.ipynb".
The first task is:
Now try requesting the file from the Phoenix server (note: need internet access)
Can you write Python code for it?

addr, filename = phoenixFileURL(3000.0, 5.0)
urllib.urlretrieve('{0}/{1}'.format(addr, filename), filename)

spectrum = np.genfromtxt('spectra/{0}'.format(filename[:-3]), usecols=(0, 1))


I am working on the file "Projects/BlindSpot/synthetic_spectra.ipynb".
The first task is:
Of course, this is too high of resolution to be passable as a real spectrum. Two things need to happen: we need to degrade the resolution and add noise. To degrade the resolution, we'll convolve the spectrum with a Gaussian kernel whose FWHM is equal to the desired spectral resolution.
Can you write Python code for it?

fwhm = 2.5  # R ~ 1000 at 5000 Å
domain = np.arange(-5.0*fwhm, 5.0*fwhm, 0.02) # note: must have same spacing as spectrum
window = np.exp(-0.5*(domain/fwhm)**2)/np.sqrt(2.0*np.pi*fwhm**2)

# visualize the window function (Kernel)
fig, ax = plt.subplots(1, 1, figsize=(8, 4))

ax.set_xlabel('$\\Delta\\lambda$ ($\\AA$)', fontsize=20.0)
ax.set_ylabel('Window', fontsize=20.0)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.plot(domain, window, '-', lw=2, color='#1e90ff')


I am working on the file "Projects/BlindSpot/synthetic_spectra.ipynb".
The first task is:
Testing the Phoenix URL and file name resolver to ensure proper URL request in urllib.
Can you write Python code for it?


I am working on the file "Projects/BlindSpot/synthetic_spectra.ipynb".
The first task is:
We will try to use urllib to pull synthetic spectra from the Phoenix model atmosphere server. This way, we can avoid storing large raw spectrum files and keep only the processed spectra. To make life easier, we should define a function to spit out a Phoenix spectrum file URL given a set of input parameters.
Can you write Python code for it?

def phoenixFileURL(Teff, logg, FeH=0.0, aFe=0.0, brand='BT-Settl', solar_abund='CIFIST2011_2015'):
    """ Create file name for a Phoenix synthetic spectrum """
    if Teff % 100.0 != 0.0:
        raise ValueError('Invalid temperature request for Phoenix server.')
    if logg not in np.arange(-0.5, 5.6, 0.5):
        raise ValueError('Invalid log(g) request for Phoenix server.')

    url = 'https://phoenix.ens-lyon.fr/Grids/{:s}/{:s}/SPECTRA'.format(brand, solar_abund)
    filename = 'lte{:05.1f}{:+4.1f}-{:3.1f}a{:+3.1f}.{:s}.spec.7.xz'.format(Teff/100.0, -1.0*logg, FeH, aFe, brand)
    return url, filename


I am working on the file "Projects/Challenges/Challenge09/challenge_set_9ii_prashant.ipynb".
The first task is:
2. What is the first and last year played for each player? Hint: Create a new table from 'Fielding.csv'.
Can you write Python code for it?

pd.read_sql_query('''SELECT playerid, min(yearid), max(yearid)
                        FROM fielding
                        GROUP BY 1
                        LIMIT 10
                        ''',cnx)


I am working on the file "Projects/Challenges/Challenge09/challenge_set_9ii_prashant.ipynb".
The first task is:
5. Which players have the longest career? Assume that the debut and finalGame columns comprise the start and end, respectively, of a player's career. Hint: Create a new table from 'Master.csv'. Also note that strings can be converted to dates using the DATE function and can then be subtracted from each other yielding their difference in days.
Can you write Python code for it?

pd.read_sql_query('''SELECT playerid, finalgame, debut, (finalgame-debut) AS days
                        FROM master
                        WHERE finalgame IS NOT NULL and debut IS NOT NULL
                        ORDER BY 4 DESC
                        LIMIT 5''',cnx)


I am working on the file "Projects/Challenges/Challenge09/challenge_set_9ii_prashant.ipynb".
The first task is:
7. What is the effect of table join order on mean salary for the players listed in the main (master) table? Hint: Perform two different queries, one that joins on playerID in the salary table and other that joins on the same column in the master table. You will have to use left joins for each since right joins are not currently supported with SQLalchemy.
Can you write Python code for it?

pd.read_sql_query('''SELECT S.playerid, AVG(salary)
                        FROM Salaries S LEFT JOIN Master M
                        ON S.playerid = M.playerid
                        GROUP BY 1
                        LIMIT 5''',cnx)

pd.read_sql_query('''SELECT M.playerid, AVG(salary)
                        FROM Master M LEFT JOIN Salaries S
                        ON M.playerid = S.playerid
                        GROUP BY 1
                        LIMIT 5''',cnx)


I am working on the file "Projects/Challenges/challenge_set_5_prashant.ipynb".
The first task is:
Challenge 1
Can you write Python code for it?

model = sm.OLS(y, X['Intercept'])
fit = model.fit()
fit.summary()


I am working on the file "Projects/Challenges/challenge_set_5_prashant.ipynb".
The first task is:
This model is representing the null hypothesis.
Can you write Python code for it?

records = range(89)

plt.scatter(records, y, color='g')
plt.scatter(records, fit.predict(X['Intercept']))

plt.hist((y['DomesticTotalGross'] - fit.predict(X['Intercept'])));


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
1 line, that's how easy it is to load a csv file using pandas! Next, we will be processing the data to make it easier to work with.
Processing Data
As you can see, the column names are odd and meaningless without prior knowledge of the dataset. To help with that, here is a list describing what each column represents:
| Column name               | Description |
| :------------------------ | :--------- |
| SC_Group_ID               | # ID of data group       |
| SC_Group_Desc             | description of data group |
| SC_GroupCommod_ID         | # ID of commodity group |
| SC_GroupCommod_Desc       | description of commodity group |
| SC_Geography_ID           | # ID of data location |
| SortOrder                 | weighted sort order by country? unsure |
| SC_GeographyIndented_Desc | description of location |
| SC_Commodity_ID           | # ID of specific commodity type |
| SC_Commodity_Desc         | description of specific commodity type |
| SC_Attribute_ID           | # ID of data's attribute |
| SC_Attribute_Desc         | description of data's attribute  |
| SC_Unit_ID                | # ID of unit type |
| SC_Unit_Desc              | description of unit type |
| Year_ID                   | year the data represents |
| SC_Frequency_ID           | # ID of frequency data is collected |
| SC_Frequency_Desc         | description of frequency data is collected |
| Timeperiod_ID             | # ID of timeperiod the data represents |
| Timeperiod_Desc           | description of timeperiod the data represents |
| Amount                    | amount of units describing the attribute for the given commodity type |
Note that 'commodity' is asynchronous to grain for the most part. Since the data includes products such as malted barley and alcohol, which aren't grains, we will use 'commodity'.
You probably realized that there are two columns describing each column (the ID and description). To make it easier to work with and shrink the width of the table, we will now delete all the 'duplicate' ID columns by using the...
Can you write Python code for it?

del df['SC_Group_ID']
del df['SC_GroupCommod_ID']
del df['SortOrder']
del df['SC_Geography_ID']
del df['SC_Commodity_ID']
del df['SC_Attribute_ID']
del df['SC_Unit_ID']
del df['SC_Frequency_ID']
del df['Timeperiod_ID']
df.head()


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
Just like the Supply and use dataset, there are 5 groups, so we will be assuming that Hay corresponds to Feed grains.
You probably also noticed that the commodities in Prices differs from the group, as compared to the Supply and use dataset. Although there are many different types of specific grains and farmers are paid different prices depending on the specific product being made from their produce, we will be looking at the grain groups as a whole so it will be easier to compare.
Next, we see that there are newer attributes in recent years, distinguishing between farmers' pay on the high end, low end, and average. We will take only the average pay rate in order to maximize the data we can use (early years don't distinguish between high and low end). We can extract only the average prices with:
Can you write Python code for it?

p = p[p.Attribute.isin(['Prices received by farmers'])]
p.head()


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
Loading Data
Now that the necessary libraries are loaded, lets load the grain data. The data will be in csv format (comma separated values), which stores comma separated values for each cell element. No manual data reading or formatting is necessary as it can be easily parsed using pandas's method read_csv.
Download the FeedGrains.zip file from the USDA Economic Research Service website: https://www.ers.usda.gov/data-products/feed-grains-database/feed-grains-yearbook-tables.aspx. Unzip the FeedGrains.zip file so FeedGrains.csv is in a FeedGrains folder. There's no real reason to put it in a separate folder since it's just one file, but why not. Copy the foder into the same folder as this notebook and load the data using the below command.
Can you write Python code for it?

df = pd.read_csv("FeedGrains/FeedGrains.csv")
df.head()


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
Personally, I would think that the beginning stock of grains each year would be increasing, or at least have an increasing trend, due to improvements in agricultural techniques. However, it is clear based on the graphs that something happened between 1986 and 1988. One event must have led to a massive surplus in 1986, while another event must have led to a huge (deficit?) in 1988. Through a quick search on "corn supply 1986" and "corn supply 1987", I managed to find 2 articles that explain the phenomena.
http://agfax.com/2016/11/29/grain-market-reminded-of-1986-king-daddy-year-of-corn-stocks-dtn/
The article in the link above describes how there were "four consecutive years of good crop weather" which led to not only the US, but the entire world to have excess grain supplies. Only about 66% of corn stock was used, explaining the massive surplus in the time period.
http://articles.latimes.com/1988-07-13/business/fi-5672_1_food-prices
This article compares the staggering grain production in 1988 compared to 1987. There seems to have been a severe drought that would kill off an estimated 30%-45% of projected harvest. As a result, the surplus that built up the past years must have been depleted in an instant, over the following years of bad weather.
Based on this, it seems that weather has a much greater effect on grain supply than agricultural techniques. However, this correlation can't be concluded without looking at weather data and trying to find a relation between the two datasets. We won't be looking at that today since this is a tutorial to get started in looking at grain data. Instead, we will explore more subsets of data in FeedGrains.csv, specifically prices!
Prices
Annual Grain Prices
Just looking at one correlation of data is no fun since finding a relation is so simple. Next, let's take a look at the prices of the grains above! We will begin by looking at annual grain prices and graph the data to see if we can find a basic visual correlation.
Prices won't be found in the Supply and use group, so we...
Can you write Python code for it?


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
Similar to the Supply and use data group, Prices also has a variety of information. Once again, we will start by working to filter out unnecessary data.
Let's start by sorting the data by year and see where to go from there.
Can you write Python code for it?

p = data['Prices']
p = p.sort_values(by='Year')
p.head()


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
There doesn't seem to be any more data inconsistencies so we can finally begin to graph the data. We will be graphing based on Commod this time instead of Commod_Group since Commod is more consistent. Hopefully we can find some trends!
Can you write Python code for it?

for i, group in p.groupby('Commod'):
    if i == 'Corn' or i == 'Hay all (dry)' or i == 'Barley' or i == 'Oats' or i == 'Sorghum':
        plot = group.plot(x='Year', y='Amount', title=str(i))
        plot.set(xlabel='Year', ylabel=group.Unit.iloc[0])
plt.show()


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
We can see that there is inconsistency in the start years for the datasets, so we will set a x-limit so it will be easier to compare the datasets.
Can you write Python code for it?

for i, group in p.groupby('Commod'):
    if i == 'Corn' or i == 'Hay all (dry)' or i == 'Barley' or i == 'Oats' or i == 'Sorghum':
        plot = group.plot(x='Year', y='Amount', title=str(i), xlim=(1975, 2017))
        plot.set(xlabel='Year', ylabel=group.Unit.iloc[0])
plt.show()


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
You can see that now the number of rows has decreased from over 400,000 to just a little over 40,000.
We saved the new extracted dataframe into a new variable df_us. This way, if we still need to find something in the original dataset, we can easily reference it without having to load the data again.
There's multiple groups that the data is categorized in. Before we can look at them, let's first sort the dataframe and find a list of unique groups.
Can you write Python code for it?

df_us = df_us.sort_values(by=['Group'])
names = df_us['Group'].unique().tolist()
names


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
You can see that the table is much cleaner, now that there are no more duplicate columns. Also note that you can only run the above code once on the dataframe. Running it more times will result in an error since the columns can't be found, as they have already been deleted.
Now that we have cleared the duplicate columns, lets rename the columns to be more descriptive and intuitive for our purposes. You can use the following line to manually change the column names.
Can you write Python code for it?

df.columns = ['Group', 'Commod_Group', 'Location', 'Commod', 'Attribute', 'Unit', 'Year', 'Frequency', 'Time Period', 'Amount']
print(df.shape)
df.head()


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
You may have noticed that there is some difference between the two datasets, the Prices dataset has a Hay group, while the Supply and use dataset has Feed grains. These two seem to be the same group of data, considering that hay is used as a feed grain for animals (most people choose not to eat hay and instead, eat oats, sorghum, barley, and corn). Before we come to this conclusion, let's take a look at what groups are in the Prices datasets.
Can you write Python code for it?


I am working on the file "Projects/Grains/Grains.ipynb".
The first task is:
You will notice that in the Frequency column, there are Annual and Quarterly data values. For simplicity, we will only be looking at annual values so we will extract only the rows with Annual.
Can you write Python code for it?


I am working on the file "Projects/Inductive Coupling.ipynb".
The first task is:
Christian's Mask
$ Q_c = \frac{2Z_0}{Z_R}\left(\frac{L_R}{M}\right)^2 $
Can you write Python code for it?

cpwx = cpwtools.CPWWithBridges(material='Nb', w=10, s=7, bridgeSpacing = 250, bridgeWidth = 3, t_oxide=0.1)
cpwx.setKineticInductanceCorrection(False)
L4 = cpwtools.QuarterLResonator(cpwx, 9000)
MperL = inductiveCoupling.inductiveCoupling.CalcMutual(cpw.w*1e6, cpwx.w*1e6, cpw.s*1e6, cpwx.s*1e6, 9.5, 10*cpw.w*1e6)[0]
M = MperL * 300e-6
halfTurn = 124*pi/2
baseL = 2000 + 2000 + 3*halfTurn + 300 + (225.5-124-10)
resExtension = [1243.697, 1118.694, 993.697, 868.694, 743.697, 618.694, 493.697, 368.694]
bridgeSpacing = np.array([30.4, 60.5, 117.6, 222, 400, 666, 997.397+halfTurn/2, 1e8])
Qc_SZ = np.array([88.1, 115.6, 97.2, 59.4, 32.3, 27.3, 31.9, 21.6])*1000 # measured and fit by Shaojiang
M_measured = np.zeros(len(bridgeSpacing))
print "{:>9}  {:>8}  {:>8}  {:>8}".format('f','Q_c', 'Qc_SZ', 'M')
for i in range(len(bridgeSpacing)):
    cpwx.bridgeSpacing = bridgeSpacing[i] * 1e-6
    L4.l = (baseL + resExtension[i])*1e-6
    Qc = 2*cpw.z0()/cpwx.z0() * (L4.L()/M)**2
    M_measured[i] = np.sqrt(2*cpw.z0()/cpwx.z0()/Qc_SZ[i]) * L4.L()
    print "{:>6.2f}GHz  {:>8.2f}  {:>8.2f}  {:>6.2f}pH".format(L4.fl()/1e9, Qc, Qc_SZ[i], M_measured[i]*1e12)

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.plot(1000./bridgeSpacing, 1e6*M_measured/300e-6, label="Measured Mutual")
ax.legend()
plt.xlabel('bridges per 1mm')
plt.ylabel('Mutual [pH/um]')
fig.set_size_inches(10,7)

lengths = np.array([288,276,266,300])*1e-6
lengths = np.array([394,406])*1e-6
MperL = inductiveCoupling.inductiveCoupling.CalcMutual(cpw.w*1e6, cpwx.w*1e6, cpw.s*1e6, cpwx.s*1e6, 5., 10*cpw.w*1e6)[0]
L4 = cpwtools.QuarterLResonator(cpw, 9000)
L4.setLengthFromFreq(5.3e9)
for l in lengths:
    print 2*cpw.z0()/cpw.z0() * (L4.L()/(MperL*l+1.36e-12))**2

cpwx.bridgeSpacing = bridgeSpacing[0] * 1e-6
print cpw.z0(), cpw.Cl(), 1/np.sqrt(cpw.Ll()*cpw.Cl())/1e8, cpw.vph()/1e8, cpw.Ll()
print cpwx.z0(), cpwx.Cl(), np.sqrt(cpwx.Cl()), cpwx.vph()/1e8, cpwx.Ll()


I am working on the file "Projects/Inductive Coupling.ipynb".
The first task is:
Do we have a long enough line to simulate this well?
Can you write Python code for it?


I am working on the file "Projects/Inductive Coupling.ipynb".
The first task is:
Vary CPW-CPW spacing $d$ with fixed overlap = 125um
Can you write Python code for it?

d_list_8R = [3,5,7,9,11,13,15,17]
measured_omega = np.array([4.80344, 4.90578, 5.0153, 5.12538, 5.24146, 5.36316, 5.48865, 5.62217])*2*pi*1e9
measured_Qc_noBonds = np.array([29.7e3, 39.2e3, 69.2e3, 93.1e3, 156.6e3, 127.5e3, 219.4e3, 249.1e3])
measured_Qc_100dBm = np.array([2.06e4, 3.29e4, 7.66e4, 9.91e4, 1.82e5, 1.18e5, 2.66e5, 2.16e5])
measured_MperL_noBonds = np.sqrt(8/pi/measured_Qc_noBonds)*cpw.z0()/measured_omega/150e-6
measured_MperL_100dBm = np.sqrt(8/pi/measured_Qc_100dBm)*cpw.z0()/measured_omega/150e-6
measured_MperL_sonnet = np.array([8.72,6.97,5.86,5.0,4.4,3.9,3.5,3.2])*1e-8
d_list_matt = [1,2,3,4,5,6,7,8,9,10,11,13,15,17]
MperL_list = np.array([inductiveCoupling.inductiveCoupling.CalcMutual(cpw.w*1e6, cpw.w*1e6, cpw.s*1e6, cpw.s*1e6, d, 10*cpw.w*1e6)[0] for d in d_list_matt])

d_list_sonnet = [1,2,3,4,5,6,7,8,9,10]
inductex_M = np.array([27.017, 23.076, 20.492, 18.375, 16.753, 15.384, 14.347, np.nan, np.nan, np.nan])*1e-12/300e-6

fig = plt.figure()
ax = fig.add_subplot(1, 1, 1)
ax.plot(d_list_matt, 1.0e6*MperL_list, label="Matt's Code")
ax.plot(d_list_8R, 1.0e6*measured_MperL_noBonds, label="Measured - No bonds")
ax.plot(d_list_8R, 1.0e6*measured_MperL_100dBm, label="Measured (100 dBm)")
ax.plot(d_list_8R, 1.0e6*measured_MperL_sonnet,label="Sonnet")
ax.plot(d_list_sonnet, 1.0e6*inductex_M,label="Inductex")
ax.legend()
plt.xlabel('Distance between CPWs')
plt.ylabel('Mutual [pH/um]')
fig.set_size_inches(10,7)

measured_MperL_noBonds


I am working on the file "Projects/Moog_2016-2019/CO2/CO2_NN/forces.ipynb".
The first task is:
Building complete data set, with the small caveat that we don't seek to load all of the positions for time constraints (for now at least).
Can you write Python code for it?

max_step = 1000
start_step = 1000
stride = 10
size_data = max_step*nbC
data = np.zeros( (max_step*nbC, size_off+1, nb_type ), dtype=float )
for step in np.arange(start_step,stride*max_step+start_step,stride):
    # Distance from all atoms (saves time?)
    matrix = computeDistanceMatrix( positions[step,:,:], cell_lengths)
    for carbon in range(nbC):
        # Data Adress
        add_data = int((step-start_step)/stride)*nbC + carbon
        # C-C
        for carbon2 in range(nbC):
            # Gaussians at atomic site
            data[ add_data, 0, 0 ] += np.exp( -(matrix[carbon,carbon2]*matrix[carbon,carbon2])/(2*sigma_C*sigma_C) )
            # Gaussians with small displacement from site
            if carbon != carbon2:
                for i in range(size_off):
                    dist = getDistance( positions[step, carbon2, :], (positions[step,carbon,:]+positions_offset[i,:])%cell_lengths[0], cell_lengths )
                    data[ add_data, i+1, 0 ] += np.exp( -(dist*dist)/(2*sigma_C*sigma_C) )
        # C-O
        for oxygen in range(nbC,nb_atoms):
            # Gaussians at atomic site
            data[ add_data, 0, 1 ] += np.exp( -(matrix[carbon,oxygen]*matrix[carbon,oxygen])/(2*sigma_O*sigma_O) )
            # Gaussians with small displacement from site
            for i in range(size_off):
                dist = getDistance( positions[step, oxygen,:], (positions[step,carbon,:]+positions_offset[i,:])%cell_lengths[0], cell_lengths )
                data[ add_data, i+1, 1 ] += np.exp( -(dist*dist)/(2*sigma_O*sigma_O) )


I am working on the file "Projects/Moog_2016-2019/CO2/CO2_NN/forces.ipynb".
The first task is:
Data parametrization
Setting up the parameters for the data construction.
Can you write Python code for it?

sigma_C = 0.9
sigma_O = 0.9
size_data = nb_step*nbC
dx = 0.1
positions_offset = np.zeros( (6,3), dtype=float )
size_off = 6
n_features=int(2*(size_off+1))
for i,ival in enumerate(np.arange(0,size_off,2)):
    positions_offset[ ival   , i ] += dx
    positions_offset[ ival+1 , i ] -= dx


I am working on the file "Projects/Moog_2016-2019/CO2/CO2_NN/forces.ipynb".
The first task is:
Here we create the neural net structure and compile it
Can you write Python code for it?

# Individual net structure
force_net = keras.Sequential(name='force_net')
#force_net.add( keras.layers.Dropout( dropout_rate_init ) )
for node in nodes:
    force_net.add( keras.layers.Dense( node, activation=activation_fct, kernel_constraint=keras.constraints.maxnorm(3)))
    #force_net.add( keras.layers.Dropout( dropout_rate_within ) )
force_net.add( keras.layers.Dense( 1, activation='linear') )

input_layer = keras.layers.Input(shape=(n_features,), name="gauss_input")
output_layer = force_net( input_layer )
model = keras.models.Model(inputs=input_layer ,outputs=output_layer )
model.compile(loss=loss_fct, optimizer=optimizer, metrics=['mse'])

 keras.utils.plot_model(model,to_file="/Users/mathieumoog/network.png", show_shapes=True, show_layer_names=True  )

early_stop = keras.callbacks.EarlyStopping( monitor='val_loss', mode='min', verbose=2, patience=patience, restore_best_weights=True)

history = model.fit( input_train_scale, output_train_scale, validation_data=( input_test_scale, output_test_scale ), epochs=n_epochs, verbose=2, callbacks=[early_stop])

predictions = model.predict( input_test_scale )
plt.plot( output_test_scale, predictions,"r." )
plt.plot( output_train_scale, output_train_scale,"g." )
plt.plot( output_test_scale, output_test_scale,"b." )
plt.show()


I am working on the file "Projects/Moog_2016-2019/CO2/CO2_NN/forces.ipynb".
The first task is:
Loading Trajectory
Here we load the trajectory, including forces and velocities, and convert the positions back into angstroms, while the forces are still in a.u (although we could do everything in a.u.).
Can you write Python code for it?

cell_lengths = np.ones(3)*volume
ftraj_path = str( path_sim + "FTRAJECTORY" )
positions, velocities, forces = cpmd.readFtraj( ftraj_path, True )
nb_step = positions.shape[0]
ang2bohr = 0.529177
positions = positions*ang2bohr
for i in range(3):
    positions[:,:,i] = positions[:,:,i] % cell_lengths[i]


I am working on the file "Projects/Moog_2016-2019/CO2/CO2_NN/forces.ipynb".
The first task is:
Then we load the modules, some issues remain with matplotlib on Jupyter, but we'll fix them later.
Can you write Python code for it?

# NN
import keras
# Descriptor (unused)
import dscribe
# Custom Libs
import cpmd
import filexyz
# Maths
import numpy as np
from scipy.spatial.distance import cdist
# Plots
import matplotlib
matplotlib.use('nbAgg')
import matplotlib.pyplot as plt
# Scalers
from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.decomposition import PCA, KernelPCA
from keras.regularizers import l2
#['GTK3Agg', 'GTK3Cairo', 'MacOSX', 'nbAgg', 'Qt4Agg', 'Qt4Cairo', 'Qt5Agg', 'Qt5Cairo', 'TkAgg', 'TkCairo', 'WebAgg', 'WX', 'WXAgg', 'WXCairo', 'agg', 'cairo', 'pdf', 'pgf', 'ps', 'svg', 'template']
# Need to install some of those stuff at some point


I am working on the file "Projects/Moog_2016-2019/CO2/CO2_NN/forces.ipynb".
The first task is:
Then we write some functions that are not yet on LibAtomicSim, but should be soon(ish)
Can you write Python code for it?

def getDistance1Dsq( position1, position2, length):
    dist = position1-position2
    half_length = length*0.5
    if dist > half_length :
        dist -= length
    elif dist < -half_length:
        dist += length
    return dist*dist
def getDistanceOrtho( positions, index1, index2, cell_lengths ):
    dist=0
    for i in range(3):
        dist += getDistance1Dsq( positions[index1,i], positions[index2,i], cell_lengths[i] )
    return np.sqrt(dist)
def getDistance( position1, position2, cell_length ):
    dist=0
    for i in range(3):
        dist += getDistance1Dsq( position1[i], position2[i], cell_lengths[i] )
    return np.sqrt(dist)
def computeDistanceMatrix( positions, cell_lengths):
    nb_atoms = len(positions[:,0])
    matrix = np.zeros(( nb_atoms, nb_atoms ))
    for atom in range(nb_atoms):
        for atom2 in range(atom+1,nb_atoms):
            dist = getDistanceOrtho( positions, atom, atom2, cell_lengths )
            matrix[atom,atom2] = dist
            matrix[atom2,atom] = dist
    return matrix


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
As we can clearly see with this representation, we have a lot of females surviving. Around a 74 %.
Only with this information we could already have a pretty good prediction.
Survival by age and sex
An interesting set of visualizations might help us see if the highest survival ratios for males are skewed to one particular range of ages. Checking the dead ratio by age with females looks interesting, to avoid it as well.
Can you write Python code for it?

survivors_male_age_pivot = clean_data[clean_data['Sex']=='male'].pivot_table(index='Age', columns='Survived', aggfunc=np.count_nonzero)
survivors_male_age_pivot = survivors_male_age_pivot.fillna(0)['PassengerId']
survivors_male_age_pivot['survival_ratio'] = survivors_male_age_pivot[1]/(survivors_male_age_pivot[1]+survivors_male_age_pivot[0])
survivors_male_age_pivot.plot(kind='bar', y='survival_ratio')


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
But this is not very helpful, since we don't see how many people there was in each group. We can either represent both survivors or not, or calculate a ratio by age.
Let's see which helps us more.
Can you write Python code for it?

#clean_data.groupby(['Age','Survived']).count().reset_index().plot(kind='bar',stacked = True, y='PassengerId', x='Age')
pivot_age = pd.pivot_table(clean_data, values='PassengerId', index='Age', columns='Survived', aggfunc=np.count_nonzero)
pivot_age.fillna(0).plot(kind='bar', stacked='True')


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
Cabin
Can you write Python code for it?


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
Data Exploration
We want to be able to see all this data depicted in this ways:
* How many people survived?
* Survival by age
* Survival by sex
* Survival by age and sex
* Survival by age and class
* Survival by sex and class
To be able to see where the survival rates are most gathered.
How many people survived?
As a first data exploration trade we are interested first, in how many people survived.
Can you write Python code for it?

import matplotlib.pyplot as plt
survivors = clean_data.groupby('Survived').count()['Name']

plt.figure(figsize=(18,8))
cmap = plt.cm.hsv
colors = ['grey','cyan']
plt.pie(survivors, labels=['Died','Survived'], explode=[0,0.05], autopct='%1.1f%%', colors = colors)

plt.axis("equal")
plt.title("Titanic Survivors")
plt.show();


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
Data Wrangling
We need to find the amount of nulls that our data has.
describe function might be useful
Can you write Python code for it?


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
Embarked
Before deleting anything, let's check the rows
Can you write Python code for it?


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
From this plot we can extract that the higher ratios of survival are up to 9 years, and between 11 and 14. Some other interesting ranges of age have good survival rates, like from 47 to 55.
Survival by Sex
Let's see which sex survived more.
Code
Can you write Python code for it?

import matplotlib.pyplot as plt

fig, axes = plt.subplots(nrows=1, ncols=2)
survivors_male = clean_data[clean_data['Sex']=='male'].groupby('Survived').count()['Name']
survivors_female = clean_data[clean_data['Sex']=='female'].groupby('Survived').count()['Name']

colors = ['grey','cyan']

male_plot = survivors_male.plot(kind='pie', labels=['Died','Survived'], explode=[0,0.05], autopct='%1.1f%%', colors = colors, ax=axes[0])
male_plot.axis("equal")
male_plot.set_title("Male Titanic Survivors")

female_plot = survivors_female.plot(kind='pie', labels=['Died','Survived'], explode=[0,0.05], autopct='%1.1f%%', colors = colors, ax=axes[1])
female_plot.axis("equal")
female_plot.set_title("Female Titanic Survivors")


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
From what we can see, not much information can be gained from age, but let's analyse by ratio, to be certain about that.
Can you write Python code for it?

pivot_age = pivot_age.fillna(0)
pivot_age['survival_ratio'] = pivot_age[1] / (pivot_age[0] + pivot_age[1])
pivot_age.plot(kind = 'bar', y='survival_ratio')


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
How do we treat nulls
In AGE
Out of 891 rows, we have 177 NaN, which represent roughly a 20%. If we replace this NaN with some other value we should be guard value, so it does not affect the rest of the values.
In Cabin
Out of 891 rows, 687 are nulls, representing an astounding 77%. Ignoring this column altogether makes more sense.
In Embarked
Only 2 NaN in this column make it possible to simply ignore this rows. We could also decide another value and see how they behave.
Code
Age
Can you write Python code for it?

clean_data = raw_data.copy()
clean_data['Age'] = clean_data['Age'].fillna(-1)


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
Survival by Age
Code
Can you write Python code for it?

clean_data[clean_data['Survived'] == 1].groupby('Age').count().reset_index().plot(kind='bar',y='PassengerId', x='Age')
#pd.pivot_table(clean_data[clean_data['Survived'] == 1], index='Age', aggfunc=np.count_nonzero


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
The trend is clear. Less money, less possibility of survival.
Survival by sex and class
Let's get a pivot table representing as clearly as possible this information.
Can you write Python code for it?

from pivottablejs import pivot_ui
pivot_ui(clean_data)


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
This distribution shows that just by being on 3rd class, your chances of surviving were a lot lower. Let's calculate how lower.
Can you write Python code for it?

survived_by_class = clean_data.pivot_table(index='Pclass', columns='Survived', aggfunc=np.count_nonzero)['PassengerId']
survived_by_class['ratio'] = survived_by_class[1]/(survived_by_class[1]+survived_by_class[0])
survived_by_class


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
We see data is structured in values ranging from 1 to 3. Standin for 1st class (richer) to 3rd class (poorer).
Can you write Python code for it?

survivors_first_age_pivot = get_survival_ratio_pivot(clean_data,'Pclass', 1)
survivors_first_age_pivot.plot(kind='bar', y='survival_ratio')

def get_survival_ratio_pivot(source, attribute, value):
    pivot = source[source[attribute]==value].pivot_table(index='Age', columns='Survived', aggfunc=np.count_nonzero)
    pivot = pivot.fillna(0)['PassengerId']
    pivot['survival_ratio'] = pivot[1]/(pivot[1]+pivot[0])
    return pivot

survivors_second_age_pivot = get_survival_ratio_pivot(clean_data,'Pclass', 2)
survivors_second_age_pivot.plot(kind='bar', y='survival_ratio')


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
With the help of this tool we see that the best result is:
Can you write Python code for it?

class_gender_pivot = pd.pivot_table(clean_data, index=['Pclass','Sex'],columns='Survived', aggfunc=np.count_nonzero)['PassengerId']
class_gender_pivot['survival_ratio'] = class_gender_pivot[1]/(class_gender_pivot[1]+class_gender_pivot[0])
class_gender_pivot


I am working on the file "Projects/P2_Investigate_a_dataset/submission_1/P2 - Investigate a Dataset - Titanic.ipynb".
The first task is:
With this representation we can clearly see that the 0 to 6 year old males are the ones that survive the most.
With females we want to study which where the ages that died the most, since we have a lot more women surviving.
Can you write Python code for it?

survivors_female_age_pivot = clean_data[clean_data['Sex']=='female'].pivot_table(index='Age', columns='Survived', aggfunc=np.count_nonzero)
survivors_female_age_pivot = survivors_female_age_pivot.fillna(0)['PassengerId']
survivors_female_age_pivot['dead_ratio'] = survivors_female_age_pivot[0]/(survivors_female_age_pivot[1]+survivors_female_age_pivot[0])
survivors_female_age_pivot.plot(kind='bar', y='dead_ratio')


I am working on the file "Projects/Project1/Project1_Prashant.ipynb".
The first task is:
Step 1
This python script takes in the dataset from census.gov and reformats the Census Tract value to match the format in the table downloaded from ArcGIS. It then returns 2 columns, census_tracts and counts of women in each of those census tracts.
Can you write Python code for it?

df_input = pd.read_csv('data/raw/ACS_15_5YR_C24010_with_ann.csv')

census_tracts = []
counts = []
for index, row in df_input.iterrows():
	if index != 0:
		counts.append(int(row['HD01_VD43']))
		county_code = row['GEO.id2'][3:5]
		tract_code = row['GEO.id2'][5:]
		if county_code == '61':
			census_tracts.append('1' + tract_code)
		if county_code == '05':
			census_tracts.append('2' + tract_code)
		if county_code == '47':
			census_tracts.append('3' + tract_code)
		if county_code == '81':
			census_tracts.append('4' + tract_code)
		if county_code == '85':
			census_tracts.append('5' + tract_code)

df_output = pd.DataFrame({'BoroCT2010':census_tracts, 'stem_women':counts})
df_output.to_csv('data/stem_women.csv', index=False)


I am working on the file "Projects/Project1/Project1_Prashant.ipynb".
The first task is:
Step 2
Merge census data with ArcGIS data.
Can you write Python code for it?

stations = pd.read_csv('data/raw/subway_station_tracts.csv')
stem_women = pd.read_csv('data/stem_women.csv')
df = pd.merge(stations, stem_women, on='BoroCT2010')

dfviz = df[df['stem_women'] > 100]

sns.barplot(x=dfviz['stem_women'], y=dfviz['name'], hue=dfviz['BoroName'], ci=None);
plt.xlabel('Women in STEM occupations living nearby',fontsize=14);
plt.ylabel('Station Name',fontsize=14);
plt.legend(loc=1);
plt.xlim(100,400);

dft50 = df[df['stem_women'] > 50].sort_values('stem_women', ascending=False)[['name','BoroName','BoroCT2010','stem_women']]
sorted_stations = dft50.groupby(['name','BoroName','BoroCT2010']).max().sort_values('stem_women', ascending=False).head(50)
sorted_stations.to_csv('data/top50_stations_women.csv')


I am working on the file "Projects/Project1/Project1_Prashant.ipynb".
The first task is:
Step 3
Import and clean MTA turnstile data for highest volume stations.
Can you write Python code for it?

df1 = pd.read_csv('http://web.mta.info/developers/data/nyct/turnstile/turnstile_160507.txt')
df2 = pd.read_csv('http://web.mta.info/developers/data/nyct/turnstile/turnstile_160514.txt')
df3 = pd.read_csv('http://web.mta.info/developers/data/nyct/turnstile/turnstile_160521.txt')
df4 = pd.read_csv('http://web.mta.info/developers/data/nyct/turnstile/turnstile_160528.txt')
frames = [df1, df2, df3, df4]
df = pd.concat(frames)

df.columns = (u'C/A', u'UNIT', u'SCP', u'STATION', u'LINENAME', u'DIVISION', u'DATE',
       u'TIME', u'DESC', u'ENTRIES',
       u'EXITS')
df = df.sort(['C/A','UNIT','SCP','DATE','TIME'])
df['diffEntries'] = df.ENTRIES.diff()
df['diffExits'] = df.EXITS.diff()
df['machine_change'] = (df.SCP != df.SCP.shift())

# Edit records where turnstile machine changes; set to 0.
df = df.reset_index(drop=True)
df.diffEntries[df.index[df.machine_change==True]] = 0
df.diffExits[df.index[df.machine_change==True]] = 0

# Further edit records where turnstile counter resulted in negative counts or counts > 6000; reset to 0.
df['counterReset'] = ((df.diffEntries > 6000) | (df.diffEntries < 0) | (df.diffExits > 6000) | (df.diffExits < 0))
df.diffEntries[df.index[df.counterReset==True]] = 0
df.diffExits[df.index[df.counterReset==True]] = 0
df.head()

df['TotalVol'] = df.diffEntries+df.diffExits
topstations = df.groupby(['STATION']).sum()
topstations = topstations.sort('TotalVol', ascending=False)
topstations.to_csv('data/volumeByStation.csv')


I am working on the file "Projects/Project1/Project1_Prashant.ipynb".
The first task is:
Step 4
Combined rank using top50_stations_women.csv and volumeByStation.csv
Can you write Python code for it?

census = pd.read_csv('data/top50_stations_women.csv')

# Top 5 stations by women residents in tech occupations
census['census_rank'] = census.index + 1
census.head()

ts = (pd.read_csv('data/volumeByStation.csv'))[:51]
ts['ts_rank']= ts.index + 1
ts.head()

import re

capitalizer = lambda x: x.upper()
census['name'] = census['name'].apply(capitalizer)

clean_ave = lambda x: re.sub('AVE','AV', x)
census['name'] = census['name'].apply(clean_ave)

clean_numbers1 = lambda x: re.sub(r'(\b\d+)(RD\b)',r'\1', x)
census['name'] = census['name'].apply(clean_numbers1)

clean_numbers2 = lambda x: re.sub(r'(\b\d+)(TH\b)',r'\1', x)
census['name'] = census['name'].apply(clean_numbers2)

clean_numbers3 = lambda x: re.sub(r'(\b\d+)(ST\b)',r'\1', x)
census['name'] = census['name'].apply(clean_numbers3)

clean_numbers4 = lambda x: re.sub(r'(\b\d+)(ND\b)',r'\1', x)
census['name'] = census['name'].apply(clean_numbers4)

clean_dashes = lambda x: re.sub(r'(\w*)(\s-\s)(\w*)',r'\1-\3', x)
census['name'] = census['name'].apply(clean_dashes)

remove_parans = lambda x: re.sub(r'\(.*\)',r'', x)
census['name'] = census['name'].apply(remove_parans)

census.replace('CONCOURSE', 'CONC', inplace=True)
census.replace('AVENUE', 'AV', inplace=True)
census.replace('WASHINGTON', 'WASH', inplace=True)
census.replace('JUNCTION', 'JCT', inplace=True)
census.replace('CONCOURSE', 'CONC', inplace=True)
census.replace('WOODHAVN', 'WOODHAVEN', inplace=True)
census.replace('CENTER', 'CTR', inplace=True)
census.replace('QUEENSBRIDGE', 'QNSBRIDGE', inplace=True)
census.replace('W 4 ST-WASHINGTON SQ ', 'W 4 ST-WASH SQ', inplace=True)

census_ts = pd.merge(census, ts, how='inner', left_on='name', right_on='STATION')
census_ts['comb_rank'] = census_ts['ts_rank'] + census_ts['census_rank']

census_ts.plot.scatter('stem_women', 'TotalVol');

final = census_ts.sort_values('comb_rank', ascending=True)
final.head()


I am working on the file "Projects/Project2/Project2_Prashant.ipynb".
The first task is:
Apply Box-Cox Transformation
As seen above the p-values were very low, suggesting the data is indeed tending towards heteroskedasticity. To improve the data we can apply boxcox.
Can you write Python code for it?

dfPolyX = pd.DataFrame(polyX)
bcPolyX = pd.DataFrame()
for i in range(dfPolyX.shape[1]):
    bcPolyX[i] = scipy.stats.boxcox(dfPolyX[i])[0]

# Transformed data with Box-Cox:
bcPolyX.head()

# Introduce log(y) for target variable:
y = y.reset_index(drop=True)
logy = np.log(y)


I am working on the file "Projects/Project2/Project2_Prashant.ipynb".
The first task is:
Apply Regularization using Elastic Net to optimize this model.
Can you write Python code for it?

X_scaled = preprocessing.scale(bcPolyX)
en_cv = linear_model.ElasticNetCV(cv=10, normalize=False)
en_cv.fit(X_scaled, logy)

en_cv.coef_

logy_en = en_cv.predict(X_scaled)
mse = metrics.mean_squared_error(logy, logy_en)

# The mean square error for this model
mse

plt.scatter([x for x in range(540)],(pd.DataFrame(logy_en)[0] - logy['AdjGross']));


I am working on the file "Projects/Project2/Project2_Prashant.ipynb".
The first task is:
For the regression model, I decided to keep data for films released through 2016, but drop the 3 films released this year; because of their recent release date, their gross earnings will not yet be representative.
Can you write Python code for it?

films17 = df.loc[[530,712,676]]

# Grabbing columns for regression model and dropping 2017 films
dfreg = df[['AdjGross','Theaters','SeriesNum','PrevAvgGross','FirstGross','DaysSinceFirstFilm','DaysSincePrevFilm']]
dfreg = dfreg.drop([530,712,676])
dfreg.shape


I am working on the file "Projects/Project2/Project2_Prashant.ipynb".
The first task is:
In the pairplot we can see that 'AdjGross' may have some correlation with the variables, particularly 'Theaters' and 'PrevAvgGross'. However, it looks like a polynomial model, or natural log / some other transformation will be required before fitting a linear model.
Can you write Python code for it?

y, X = patsy.dmatrices('AdjGross ~ Theaters + SeriesNum + PrevAvgGross + FirstGross + DaysSinceFirstFilm + DaysSincePrevFilm', data=dfreg, return_type="dataframe")


I am working on the file "Projects/Project2/Project2_Prashant.ipynb".
The first task is:
Step 1
I started with the "Franchises" list on Boxofficemojo.com. Within each franchise page, I scraped each movie's information and enter it into a Python dictionary. If it's already in the dictionary, the entry will be overwritten, except with a different Franchise name. But note below that the url for "Franchises" list was sorted Ascending, so this conveniently rolls "subfranchises" into their "parent" franchise.
E.g., "Fantastic Beasts" and the "Harry Potter" movies have their own separate Franchises, but they will all be tagged as the "JKRowling" franchise, i.e. "./chart/?id=jkrowling.htm"
Also, because I was comparing sequels to their predecessors, I focused on Domestic Gross, adjusted for ticket price inflation.
Can you write Python code for it?

url = 'http://www.boxofficemojo.com/franchises/?view=Franchise&sort=nummovies&order=ASC&p=.htm'
response = requests.get(url)
page = response.text
soup = BeautifulSoup(page,"lxml")
tables = soup.find_all("table")
rows = [row for row in tables[3].find_all('tr')]
rows = rows[1:]

# Initialize empty dictionary of movies
movies = {}

for row in rows:
    items = row.find_all('td')
    franchise = items[0].find('a')['href']
    franchiseurl = 'http://www.boxofficemojo.com/franchises/' + franchise[2:]
    response = requests.get(franchiseurl)

    franchise_page = response.text
    franchise_soup = BeautifulSoup(franchise_page,"lxml")
    franchise_tables = franchise_soup.find_all("table")
    franchise_gross = [row for row in franchise_tables[4].find_all('tr')]
    franchise_gross = franchise_gross[1:len(franchise_gross)-2]
    franchise_adjgross = [row for row in franchise_tables[5].find_all('tr')]
    franchise_adjgross = franchise_adjgross[1:len(franchise_adjgross)-2]

    # Assign movieurl as key
    # Add title, franchise, inflation-adjusted gross, release date.
    for row in franchise_adjgross:
        movie_info = row.find_all('td')
        movieurl = movie_info[1].find('a')['href']
        title = movie_info[1]
        adjgross = movie_info[3]
        release = movie_info[5]
        movies[movieurl] = [title.text]
        movies[movieurl].append(franchise)
        movies[movieurl].append(adjgross.text)
        movies[movieurl].append(release.text)

    # Add number of theaters for the above movies
    for row in franchise_gross:
        movie_info = row.find_all('td')
        movieurl = movie_info[1].find('a')['href']
        theaters = movie_info[4]
        if movieurl in movies.keys():
            movies[movieurl].append(theaters.text)

df = pd.DataFrame(movies.values())
df.columns = ['Title','Franchise', 'AdjGross', 'Release', 'Theaters']
df.head()

df.shape


I am working on the file "Projects/Project2/Project2_Prashant.ipynb".
The first task is:
Step 3
Apply Linear Regression.
Can you write Python code for it?

dfreg.corr()

sns.pairplot(dfreg);

sns.regplot((dfreg.PrevAvgGross), (dfreg.AdjGross));

sns.regplot(np.log(dfreg.Theaters), np.log(dfreg.AdjGross));