I am working on the file "HW4_solutions.ipynb".
The first task is:
For the user testuserid, we loop over the variable bizs (which is a set of restaurants the user has rated) and print the predicted rating, and the actual rating and restaurant average rating obtained using the function above. We again use k=7 and a regularization of 3.
Can you write Python code for it?

print "for user",usernamefromid(smalldf,testuserid), 'avg', smalldf[smalldf.user_id==testuserid].stars.mean()
for biz_id in bizs:
    print "----------------------------------"
    print biznamefromid(smalldf, biz_id)
    print "Predicted Rating:",rating(smalldf, db, biz_id, testuserid, k=7, reg=3.)
    u,a=get_other_ratings(biz_id, testuserid, smalldf)
    print "Actual User Rating:",u,"Avg Rating",a


I am working on the file "HW4_solutions.ipynb".
The first task is:
Get top matches
Its now time to answer the question: "if you liked this, you might also like these". We use our testbizid and testbizid2 to compute the k=7 nearest neighbors with a regularization of 3. . We print these top 7 matches names, along with their similarity coefficient and common support.
Can you write Python code for it?

tops=knearest(testbizid, smalldf.business_id.unique(), db, k=7, reg=3.)
print "For ",biznamefromid(smalldf, testbizid), ", top matches are:"
for i, (biz_id, sim, nc) in enumerate(tops):
    print i,biznamefromid(smalldf,biz_id), "| Sim", sim, "| Support",nc

tops2=knearest(testbizid2, smalldf.business_id.unique(), db, k=7, reg=3.)
print "For ",biznamefromid(smalldf, testbizid2), ", top matches are:"
for i, (biz_id, sim, nc) in enumerate(tops2):
    print i,biznamefromid(smalldf,biz_id), "| Sim", sim, "| Support",nc


I am working on the file "HW4_solutions.ipynb".
The first task is:
Here is the Gibbs sampler skeleton that your functions fit into. Look over the structure to see how for each draw from the posterior, the sampler iterates through $\mu$, $\sigma$, $\gamma_m$ for each item, and $\theta_u$ for each user.
Can you write Python code for it?

"""
Function
--------
factor_gibbs

Runs a gibbs sampler to infer mean, variance, user-specific, and item-specific
parameters.

Inputs
-------
data: A dataframe containing ratings data.
L: Dimension of latent factors.
maxit: Number of samples to draw from posterior.
Lambda_theta_diag: Hyperparameter controlling regularization of Theta.
Lambda_gamma_diag: Hyperparameter controlling regularization of Gamma.
progress: if true, print iteration number every 100 iterations.

Outputs
--------
Dictionary with elements
mu: Draws of mu. 1D array of length maxiter.
sig2: Draws of sig2, residual _variance_. 1D array of length maxiter.
theta: Draws of Theta. U-by-L-by-maxiter array.
gamma: Draws of Gamma. M-by-L-by-maxiter array.
EY: Draws of fitted values of Y. N-by-maxiter array.
"""
def factor_gibbs(data, L, maxit, Lambda_theta_diag, Lambda_gamma_diag, progress=True):
    data = data.copy()
    N = data.shape[0]

    #Create indices that allow us to map users and restaurants to rows
    #in parameter vectors.
    uusers, uidx = np.unique(data.user_id, return_inverse=True)
    uitems, midx = np.unique(data.business_id, return_inverse=True)

    nusers = uusers.size
    nitems = uitems.size

    #Add numerical indices to dataframe.
    data["uidx"] = uidx
    data["midx"] = midx

    #Group observations by user and by business.
    ugroups = data.groupby("uidx")
    mgroups = data.groupby("midx")

    all_avg = data.stars.mean()
    u_avg = ugroups.stars.mean()
    m_avg = mgroups.stars.mean()

    #Initialize parameters and set up data structures for
    #holding draws.
    #Overall mean
    mu = all_avg
    mu_draws = np.zeros(maxit)
    #Residual variance
    sig2 = 0.5
    sig2_draws = np.zeros(maxit)

    #Matrix of user-specific bias and L latent factors.
    theta = np.zeros([nusers, L+1])
    theta[:,0] = u_avg-all_avg
    theta_draws = np.zeros([nusers, L+1, maxit])

    #Matrix of item-specific bias and L latent factors.
    gamma = np.zeros([nitems, L+1])
    gamma[:,0] = m_avg-all_avg
    gamma_draws =...


I am working on the file "HW4_solutions.ipynb".
The first task is:
Lets run make_database and store the result in the global variable db. Lets print out an example entry. Running this function will take a bit of time.
Can you write Python code for it?

db=Database(smalldf)
db.populate_by_calculating(pearson_sim)

db.get("z3yFuLVrmH-3RJruPEMYKw", "zruUQvFySeXyEd7_rQixBg")


I am working on the file "HW4_solutions.ipynb".
The first task is:
Making a database of similarities
We now move to calculating a global database of pairwise restaurant similarities.
We provide you here with a function to make a database of the similarities for each pair of restaurants in the database. The class Database is initialized in its constructor by taking as arguments a dataframe of reviews. The method populate_by calculating iterates over every possible pair of business_id's in the dataframe and populates the database with similarities and common supports. It takes as arguments a function the similarity function similarity_func like pearson_sim (calculate_similarity then uses this to calculate the similarity). The get method on the database can be used to retrieve the similarity for two business ids.
(See Thu Oct 17th's class video for information about classes)
Can you write Python code for it?

class Database:
    "A class representing a database of similaries and common supports"

    def __init__(self, df):
        "the constructor, takes a reviews dataframe like smalldf as its argument"
        database={}
        self.df=df
        self.uniquebizids={v:k for (k,v) in enumerate(df.business_id.unique())}
        keys=self.uniquebizids.keys()
        l_keys=len(keys)
        self.database_sim=np.zeros([l_keys,l_keys])
        self.database_sup=np.zeros([l_keys, l_keys], dtype=np.int)

    def populate_by_calculating(self, similarity_func):
        """
        a populator for every pair of businesses in df. takes similarity_func like
        pearson_sim as argument
        """
        items=self.uniquebizids.items()
        for b1, i1 in items:
            for b2, i2 in items:
                if i1 < i2:
                    sim, nsup=calculate_similarity(b1, b2, self.df, similarity_func)
                    self.database_sim[i1][i2]=sim
                    self.database_sim[i2][i1]=sim
                    self.database_sup[i1][i2]=nsup
                    self.database_sup[i2][i1]=nsup
                elif i1==i2:
                    nsup=self.df[self.df.business_id==b1].user_id.count()
                    self.database_sim[i1][i1]=1.
                    self.database_sup[i1][i1]=nsup

    def get(self, b1, b2):
        "returns a tuple of similarity,common_support given two business ids"
        sim=self.database_sim[self.uniquebizids[b1]][self.uniquebizids[b2]]
        nsup=self.database_sup[self.uniquebizids[b1]][self.uniquebizids[b2]]
        return (sim, nsup)


I am working on the file "HW4_solutions.ipynb".
The first task is:
NOTICE: when comparing graphs note that the limits have changed to make the bias-variance comparison more visible. The regularizer is set to 1 to reduce the regularization to something small, rather than something optimal, as well. Also to make things more extreme, I choose k=15 rather than k=10.
Can you write Python code for it?

print "k=2, reg=1."
make_results_plot_pos(smalldf,2,1.)
plt.title("k=2, reg=1.")

print "k=2, reg=15."
make_results_plot_pos(smalldf,2,15.,)
plt.title("k=2, reg=15.")

print "k=15, reg=1."
make_results_plot_pos(smalldf,15,1.)
plt.title("k=15, reg=1.")

print "k=15, reg=15."
make_results_plot_pos(smalldf,15,15.,)
plt.title("k=15, reg=15.")


I am working on the file "HW4_solutions.ipynb".
The first task is:
Notice firstly, that at low regularization, at low k (variance limit), the predictions follow the green line more than at high k ( bias limit). This is, as mentioned earlier, due to the bias limit pulling the extreme groups of ratings in towards the mean of all the reviews. Note that increasing bias decreases the variation between the rating groups, resulting in a flatter black curve.
The second thing to notice that as k goes up, the precision as measured by the width of the grey region (which measures the variance WITHIN rating groups, not between them) gets smaller.
The third thing to notice is that adding in regularization in the variance (low k) limit actually increases the bias, which is not surprising as it tends to push things towards the center. In many machine learning situations this is a desirable thing, as it reduces overfitting. In the high bias limit, as expected, it does not have much of an impact. Note that regularization had more of an impact in the previous set of graphs, as it pushed similarity towards 0.
One might ask: why not plot these graphs in the homework in the first place: using distances is the correct way to do it, and dosent lead to the subtraction problem. Its a good question, and you should use positive similarities in a production recommender. For the puposes of this homework, i think the k=10 graphs in the previous set illustrate the problem with sparsity in a much more dramatic way, leading to the "bellows" in the graph. However, in real life, there's no real justification for using similarities that can be negative as weights. If we think of KNN as a way of constructing local averages, you need those averages to be constructed using positive weights.
2.5 Outline a process, in words, for choosing the nearest neighbor parameter k. For this question fix the regularization parameter reg at 3.
your answer here
We could use $F$-fold cross-validation (usually called $k$-fold cross-validation, but we've already used $k$ here to mean something else!). Specifically, we could...
Can you write Python code for it?

"""
Function
--------
gamma_m_draw

Draw a single sample from the conditional posterior distribution
of gamma_m.

Inputs
-------
X_m: A g-by-L+1 matrix, defined above.
Y_m: A 1D vector of length g, defined above.
sig2: Residual _variance_, as defined above.
Lambda_gamma: Prior precision matrix.

Outputs
--------
Single draw from conditional posterior, defined above.
"""
#Item-specific parameters given all else
#your code here
def gamma_m_draw(X_m, Y_m, sig2, Lambda_gamma):

    #Compute matrices that define conditional posterior.
    Q_m_inv = np.linalg.inv(np.dot(X_m.T, X_m)/sig2+Lambda_gamma)
    XtY = np.dot(X_m.T, Y_m)

    #Draw item-specific parameters.
    return np.random.multivariate_normal(np.dot(Q_m_inv, XtY)/sig2, Q_m_inv)

"""
Function
--------
theta_u_draw

Draw a single sample from the conditional posterior distribution
of gamma_m.

Inputs
-------
X_u: A g-by-L+1 matrix, defined above.
Y_u: A 1D vector of length g, defined above.
sig2: Residual _variance_, as defined above.
Lambda_theta: Prior precision matrix.

Outputs
--------
Single draw from conditional posterior, defined above.
"""
#User-specific parameters given all else
#your code here
def theta_u_draw(X_u, Y_u, sig2, Lambda_theta):
    #Compute matrices that define conditional posterior.
    Q_u_inv = np.linalg.inv(np.dot(X_u.T, X_u)/sig2+Lambda_theta)
    XtY = np.dot(X_u.T, Y_u)

    #Draw the user-specific parameters
    return np.random.multivariate_normal(np.dot(Q_u_inv, XtY)/sig2, Q_u_inv)


I am working on the file "HW4_solutions.ipynb".
The first task is:
Posterior Summaries
Once you have posterior draws from the sampler, the most natural thing to do is to compute the posterior mean of each quantity you are intersted in. To do this, we simply need to take the average value of each quantity across the samples drawn from the sampler. Before taking the average, however, we will want to ignore the first 20-30% of samples because these correspond the burnin period, the time during which the sampler is still looking for the main meat of the distribution.
Ok it's time to recommend!
3.3 Now that you have the Gibbs sampler, draw 1000 samples from the posterior distribution using a two-dimensional latent factor and prior precisions Lambda_theta_diag and Lambda_gamma_diag both equal to 0.1.
Compute the posterior mean of the fitted values for each $Y_{um}$, eliminating the first 200 samples. Call these the prediction.  These constitute our recommendations. True to the bayesian paradigm, we dont just have mean predictions, but entire distributions. But currently we are only interested in the means.
Can you write Python code for it?

#your code here
gibbs_out = factor_gibbs(smalldf, 2, 1000, 0.1, 0.1)
burnin = 200
predicted=np.mean(gibbs_out['EY'][:,burnin:], axis=1)


I am working on the file "HW4_solutions.ipynb".
The first task is:
Running mrjob locally
mrjob scripts cannot be run from the ipython notebook, as they fork themselves on execution. Thus you must write the code for mrjob in a separate file which you must submit along with this homework, in the same folder as the python notebook file.
If you have not done so already (you were supposed to do this as part of HW 0), you will first need to install mrjob. The appropriate equivalent of the following incantation should do the job:
~/anaconda/bin/pip install mrjob

To familiarize yourself with the structure of an mrjob script, please read this . Run the examples in that document to familiarize yourself with mrjob.
The kind of script you will be writing is in the section "Writing your second job" in that document.
All mrjob tasks use the map-reduce strategy to divide up computation across computers. You should work through the mrjob tutorial to gain familiarity with this, but we’ll also outline the basic process here:

During the first map step, mrjob calls a mapper function with a key (which for the first step is None), and a value (which for the first step is a line of data from an input file). This function does whatever it wants with this data, and yields a key and value.  The key is used in step 2 to gather up the values from all the different mappers into groups

mrjob collects the outputs from all the mappers, and gathers them into subsets with the same key value (this is similar to what pandas.groupby does). It passes each of these subsets to a reducer (or “collector”) function, whose job is to synthesize this list of grouped data into something useful (e.g., computing the mean of all the inputs). It then yields the key and reduced value.

If there are any additional steps, mrjob feeds each output from a reducer function in step 2 to the next mapper. Otherwise, it prints the output.

The point behind map-reduce is to agree upon a common framework to split up a large computational job into smaller tasks. mrjob then has a lot of freedom to organize how these tasks run in...
Can you write Python code for it?

from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
from IPython.display import HTML
import urllib
skelcode = urllib.urlopen("https://raw.github.com/cs109/content/master/skeleton.py").read()
skelhtml=highlight(skelcode, PythonLexer(), HtmlFormatter())
HTML(skelhtml)


I am working on the file "HW4_solutions.ipynb".
The first task is:
The data frame is a frame of reviews. We have joined in information about users and businesses into this frame so that you have only one frame to work with.
This information is for the reviews themselves:
Here is a description of the data fields in this dataframe, on the business side
And Finally, a set of fields for users
In this data set, every user has only one review for each restaurant. Convince yourself of this. (This answer does not need to be submitted).
Our Recommender
To motivate our recommendation system, consider the follwing example. Let's pretend we are in Boston for a second. Lets say the average rating of restaurants here by all the users is 3.5. Sandrine's at Harvard square is better than an average restaurant, so it tends to be rated 0.5 stars above the average (over all the users). However, you are a curmudgeon, who tends to rate 0.2 stars below the average. Then a baseline estimate for the recommendation for Sandrine's, for you, is 3.5+0.5-0.2=3.8.
These baseline estimates thus adjust the data by accounting for the systematic tendencies for some users who give higher ratings than others, and for some restaurants to recieve higher ratings than others. We can write the baseline estimate $\hat Y_{um}^{baseline}$ for an unknown rating $Y_{um}$ for user $u$ and restaurant or business $m$ as:
$$ \hat Y_{um}^{baseline} = \hat \mu + \hat \theta_{u0} + \hat \gamma_{m0} $$
where the unknown parameters $\theta_{u0}$ and $\gamma_{m0}$ indicate the deviations, or biases, of user $u$ and item $m$, respectively, from some intercept parameter $\mu$. (The reason for the strange notation with 0s will become clear in Problem 3)
Notice that the $\theta_{u0}$ and $\gamma_{m0}$ are parameters which need to be fit. The simplest thing to start with, and something we will do for Problems 1 and 2 (but not 3), is to replace them by their "mean" estimates from the data. Thus:
$$ \hat Y^{baseline}_{um} = \bar Y + (\bar Y_u - \bar Y) + (\bar Y_m - \bar Y)$$
where $\bar Y_u$ =  user_avg, the average of all a user $u$'s...
Can you write Python code for it?

#your code here
urc=fulldf.groupby('user_id').review_id.count()
ax=urc.hist(bins=50, log=True)
remove_border(ax)
plt.xlabel("Reviews per user")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.title("Review Count per User");

#your code here
brc=fulldf.groupby('business_id').review_id.count()
ax=brc.hist(bins=50, log=True)
remove_border(ax)
plt.xlabel("Reviews per restaurant")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.title("Review Count per Restaurant");

#your code here
print "Number of Reviews",fulldf.shape[0]
print "Number of Users", fulldf.user_id.unique().shape[0], "Number of Businesses", fulldf.business_id.unique().shape[0]


I am working on the file "HW4_solutions.ipynb".
The first task is:
The fit here looks much better, both in terms of tracking the green line and in terms of the within-group precision (as measured by the grey band). A model at the variance limit (that is, one that is extremely overfit) with low bias would have a mean line that tracks exactly with the green line and an almost non-existent gray area. To tell whether this plot represents simply a good prediction model or a one that is woefully overfit, we would need to look at out-of-sample data.
Q4 Scaling Up
All our recommenders suffer from problems having to do with the fact that we subsetted an already sparse user-item matrix. The more items we have, the more items we may find in the vicinity of a given item, and thus we are likely to give a more robust average rating to the given item.
In this problem we shall use Amazon Elastic Map-Reduce to tackle the entire user-restaurant matrix. We shall do this in two parts: we'll use MRJob locally on your machine to on the smaller data set to calclate the pearson database, and then we'll tackle the entire data set on Amazon.
The larger set has 35000 users and 4500 items. Computing the 4500X4500 similarity matrix on one machine will be prohibitively expensive. Thus we'll adopt a strategy where we'll split the calculation over multiple machines using the map-reduce paradigm, with mappers and reducers working on multiple machines
Then we calculate the k-nearest neighbors in the 'space' of the user: this involves a database lookup and an iteration over the items a user has rated. Since the latter is usually not a very large number, this computation can be managed on a front end machine (even if storing the database will take a lot of memory).
We'll first create subset data frames, which have just those columns which we will send to the map-reduce. We'll also strip out the header and index of the frame. The reason for doing this is: unless we pre-populate the machines on Amazon with software, we can rely only on the regular python library, numpy, and scipy being there (and at python...
Can you write Python code for it?

subsetoffull=fulldf[['user_id','business_id', 'stars','business_avg','user_avg']]
subsetoffull.to_csv("subset-full.csv", index=False, header=False)
subsetofsmall=smalldf[['user_id','business_id', 'stars','business_avg','user_avg']]
subsetofsmall.to_csv("subset-small.csv", index=False, header=False)


I am working on the file "HW4_solutions.ipynb".
The first task is:
The following function is used to re-compute review counts and averages whenever you subset a reviews data frame. We'll use it soon to construct a smaller, more computationally tractable data frame.
Can you write Python code for it?

def recompute_frame(ldf):
    """
    takes a dataframe ldf, makes a copy of it, and returns the copy
    with all averages and review counts recomputed
    this is used when a frame is subsetted.
    """
    ldfu=ldf.groupby('user_id')
    ldfb=ldf.groupby('business_id')
    user_avg=ldfu.stars.mean()
    user_review_count=ldfu.review_id.count()
    business_avg=ldfb.stars.mean()
    business_review_count=ldfb.review_id.count()
    nldf=ldf.copy()
    nldf.set_index(['business_id'], inplace=True)
    nldf['business_avg']=business_avg
    nldf['business_review_count']=business_review_count
    nldf.reset_index(inplace=True)
    nldf.set_index(['user_id'], inplace=True)
    nldf['user_avg']=user_avg
    nldf['user_review_count']=user_review_count
    nldf.reset_index(inplace=True)
    return nldf


I am working on the file "HW4_solutions.ipynb".
The first task is:
This function will take a very long time to run, on the order of 5 minutes or more, depending on your computer
Can you write Python code for it?


I am working on the file "HW4_solutions.ipynb".
The first task is:
We can see that these two restaurants are in somewhat different orbits :-).
Lets now turn our attention to another question: what are the top recommendations for a user? To answer this we must find the user's top rated restaurants, find the nearest neighbors of these restaurants, merge these lists while removing the duplicates and the ones that the user has already rated, and sort by the restaurant's average rating. We provide the code to get the user's top choices in a subset data frame.
Can you write Python code for it?

def get_user_top_choices(user_id, df, numchoices=5):
    "get the sorted top 5 restaurants for a user by the star rating the user gave them"
    udf=df[df.user_id==user_id][['business_id','stars']].sort(['stars'], ascending=False).head(numchoices)
    return udf
testuserid="7cR92zkDv4W3kqzii6axvg"
print "For user", usernamefromid(smalldf,testuserid), "top choices are:"
bizs=get_user_top_choices(testuserid, smalldf)['business_id'].values
[biznamefromid(smalldf, biz_id) for biz_id in bizs]


I am working on the file "HW4_solutions.ipynb".
The first task is:
We will Implement a function make_database_from_pairs which takes a dataframe of restaurants smalldf and the output parsed in the previous command to create the database like before. By the nature of the map-reduce algorithms these only contain those restaurant pairs with common support. The Database constructor initializes the remaining similarities to 0.
The function will take the dataframe and bizpairs obtained by parsing the EMR output file which have the key of business pairs and value the pair of pearson correlation and n_common. It will return an instance of the Database class.
This function will take a long time to run on large data sets.
Can you write Python code for it?

def make_database_from_pairs(df, bizpairs):
    """
    make the database from the pairs returned from mrjob.
    df is the dataframe, smalldf or fulldf.
    bizpairs are a list of elements, each of which is a list of two
        lists. The first of these lists has the two business id's, while
        the second has the similarity and the common support
    Returns an instance of the Database class.
    """
    dbase=Database(df)
    cache={}
    for bp,corrs in bizpairs:
        b1,b2=bp
        i1=dbase.uniquebizids[b1]
        i2=dbase.uniquebizids[b2]
        sim,nsup=corrs
        dbase.database_sim[i1][i2]=sim
        dbase.database_sim[i2][i1]=sim
        dbase.database_sup[i1][i2]=nsup
        dbase.database_sup[i2][i1]=nsup
        if cache.has_key(b1):
            nsup1=cache[b1]
        else:
            nsup1=dbase.df[dbase.df.business_id==b1].user_id.count()
            cache[b1]=nsup1
        if cache.has_key(b2):
            nsup2=cache[b2]
        else:
            nsup2=dbase.df[dbase.df.business_id==b2].user_id.count()
            cache[b2]=nsup2
        dbase.database_sim[i1][i1]=1.0
        dbase.database_sim[i2][i2]=1.0
        dbase.database_sup[i1][i1]=nsup1
        dbase.database_sup[i2][i2]=nsup2
    return dbase


I am working on the file "HW4_solutions.ipynb".
The first task is:
You can read more here. Also see Thu Oct 17th's class video for information about classes and generators.
Include computesim.py in your submission in the same folder as the notebook. Uncommenting and running the following cell should output your code in here.
Can you write Python code for it?

thecode = open("computesim.py").read()
thehtml=highlight(thecode, PythonLexer(), HtmlFormatter())
HTML(thehtml)


I am working on the file "HW4_solutions.ipynb".
The first task is:
your answer here
If you are a bit confused by the look of these graphs, you should be!
For k=3, the predicted values are quite well-behaved, with the exception of several predictions that have extremely large magnitudes. It appears that the predicted values are pulled into the mean star rating, which sits somewhere around 3.8, so ratings on the low end are overestimated, and similarly ratings on the high end underestimated. The regularization does not appear to have a strong effect when k = 3.
For k=10, the predicted values are much less stable, with many more extreme predictions. The means appear to track better with the true means. The regularization has a much more extreme, although indirect, effect on the appearance of the plot. Since regularization has stronger effects on similarity scores between restaurants that have small common support, we can see that increasing k makes the predictions more sensitive to the regularization because in a small dataset, the common support between a restaurant and it's 10-nearest one will be quite small.
Note that this example does not seem to follow the standard bias-variance tradeoff, where we would expect small k to give a unbiased estimates that capture the extremes, while we would expect large k to give biased estimates that pull extreme values toward the mean. A large reason for this failure for this example to capture this behavior is that we have defined similarity scores that can be positive or negative, and the bias-variance logic is based on the more standard setting where we average together values that have strictly positive weights. When you have negative weights, it's possible that the sum of $s_{ij}$'s in a neighborhood can get close to zero, making our estimator $\hat Y_{um}$ explode in the positive or negative direction since this would entail dividing by (nearly) zero. Thus for those restaurants where the denominator goes to 0 (more likely to happen at larger k as you have more chances of it there being more weights to add), the ratings are unstable,...
Can you write Python code for it?

def knearest_pos(restaurant_id, set_of_restaurants, dbase, k=7, reg=3.):
    """
    Given a restaurant_id, dataframe, and database, get a sorted list of the
    k most similar restaurants from the entire database.
    """
    similars=[]
    for other_rest_id in set_of_restaurants:
        if other_rest_id!=restaurant_id:
            sim, nc=dbase.get(restaurant_id, other_rest_id)
            ssim=shrunk_sim(sim, nc, reg=reg)
            similars.append((other_rest_id, ssim/2.0 + float(nc)/(float(nc)+reg), nc ))
    similars=sorted(similars, key=itemgetter(1), reverse=True)
    return similars[0:k]

def knearest_amongst_userrated_pos(restaurant_id, user_id, df, dbase, k=7, reg=3.):
    dfuser=df[df.user_id==user_id]
    bizsuserhasrated=dfuser.business_id.unique()
    return knearest_pos(restaurant_id, bizsuserhasrated, dbase, k=k, reg=reg)

def rating_pos(df, dbase, restaurant_id, user_id, k=7, reg=3.):
    mu=df.stars.mean()
    users_reviews=df[df.user_id==user_id]
    nsum=0.
    scoresum=0.
    nears=knearest_amongst_userrated_pos(restaurant_id, user_id, df, dbase, k=k, reg=reg)
    restaurant_mean=df[df.business_id==restaurant_id].business_avg.values[0]
    user_mean=users_reviews.user_avg.values[0]
    scores=[]
    for r,sold,nc in nears:
        s=sold/2.0
        shrink_factor=float(nc)/(float(nc)+reg)
        s=s+shrink_factor/2.0
        scoresum=scoresum+s
        scores.append(s)
        r_reviews_row=users_reviews[users_reviews['business_id']==r]
        r_stars=r_reviews_row.stars.values[0]
        r_avg=r_reviews_row.business_avg.values[0]
        rminusb=(r_stars - (r_avg + user_mean - mu))
        nsum=nsum+s*rminusb
    baseline=(user_mean +restaurant_mean - mu)
    #we might have nears, but there might be no commons, giving us a pearson of 0
    if scoresum > 0.:
        val =  nsum/scoresum + baseline
    else:
        val=baseline
    return val

def make_results_plot_pos(df,k,reg):
    uid=smalldf.user_id.values
    bid=smalldf.business_id.values
    actual=smalldf.stars.values
   ...


I am working on the file "HW4_solutions.ipynb".
The first task is:
your answer here
The results from the latent factor model appear to be better-behaved than those from the KNN procedure (with negative similarities) since there are no extreme values. In terms of the non-extreme values from the KNN procedure, the latent factor model appears to make similar predictions to the KNN procedure when k = 3., Specifically, the average prediction line for each class is too compressed toward the total data mean of around 3.8, meaning that we are in the bias limit where we are pooling too much information between ratings. Thus, we are overpredicting low ratings and underpredicting high ratings.
(If we compare to the KNN procedure with strictly positive weights, as in the homework addendum, we see again that the recommenders make comparable predictions, although the latent factor model again appears to fit slightly better than at both the bias or variance limits of the KNN procedure).
There is also a bias-variance tradeoff here. In this case, it appears that we have proposed a model that is too simple (thus close to the bias limit) because of how the ratings are pulled in toward the data mean. In this case, proposing more latent factors increases the flexibility of the model, and thus moves us toward the variance limit. Thus, the plot suggests that we may want to reduce the bias at the cost of increasing variance, for example by considering a latent factor model with more factors (say, $L = 15$) to obtain a better fit (see ADDENDUM below).
IMPORTANT SOLUTION ADDENDUM
Can you write Python code for it?

gibbs_out = factor_gibbs(smalldf, 15, 1000, 0.1, 0.1)
burnin = 200
predicted=np.mean(gibbs_out['EY'][:,burnin:], axis=1)
compare_results(smalldf.stars.values, predicted, ylow=1, yhigh=5, title="From Gibbs Sampler")


I am working on the file "HW4_solutions.ipynb".
The first task is:
your answer here: As we made a very specific cut on the data set, making sure we had those businesses with greater than 150 reviews, and those users with more than 60 reviews, this data set is not as sparse as the full one, with a range in both graphs where the histogram is somewhat flat rather than steeply declining.
1.4 Compute histograms of the average user rating in the smaller data set, and the average business rating in the smaller data set. Print the overall mean.
Can you write Python code for it?

#your code here
plt.figure()
avg_ratings_by_user=smalldf.groupby('user_id').stars.mean()
ax=avg_ratings_by_user.hist()
remove_border(ax)
plt.xlabel("Average review score")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.title("Average User Rating")
plt.figure()

avg_ratings_by_biz=smalldf.groupby('business_id').stars.mean()
ax=avg_ratings_by_biz.hist()
remove_border(ax)
plt.xlabel("Average review score")
plt.grid(False)
plt.grid(axis = 'y', color ='white', linestyle='-')
plt.title("Average Restaurant Rating")
plt.figure()

print smalldf.stars.mean()
plt.figure()


I am working on the file "HW5.ipynb".
The first task is:
$$g'(x)-10{e}^{10x}sin({e}^{10x})$$
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
$$h'(x)=(ln x)x^x+xx^{x-1}$$
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
B.2 Studying a function for different parameter values
$f(x) = sin\dfrac{1}{x+eps}$ in [0,1]
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
Error in approximation at q = 16:
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
Error in approximation at q = 2:
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
Error in approximation at q = 4:
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
Homework 5
Chinmai Raman
3/18/2016
B.1 Interpolating a discrete function
Part 1:
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
How (and how not) to visualize networks
Network plots often look impressive, but creating sensible network plots is tricky. From Ben Fry, the author of the Processing program:
<blockquote>
Usually a graph layout isn’t the best option for data sets larger than a few dozen nodes. You’re most likely to wind up with enormous spider webs or balls of string, and the mess seen so far is more often the case than not. Graphs can be a powerful way to represent relationships between data, but they are also a very abstract concept, which means that they run the danger of meaning something only to the creator of the graph. Often, simply showing the structure of the data says very little about what it actually means, even though it’s a perfectly accurate means of representing the data. Everything looks like a graph, but almost nothing should ever be drawn as one.
</blockquote>

Let's look at bad and better ways of visualizing the senate vote network.
First, consider the "default" plot from networkx.
Can you write Python code for it?

#this makes sure draw_spring results are the same at each call
np.random.seed(1)

color = [votes.node[senator]['color'] for senator in votes.nodes()]

#determine position of each node using a spring layout
pos = nx.spring_layout(votes, iterations=200)

#plot the edges
nx.draw_networkx_edges(votes, pos, alpha = .05)

#plot the nodes
nx.draw_networkx_nodes(votes, pos, node_color=color)

#draw the labels
lbls = nx.draw_networkx_labels(votes, pos, alpha=5, font_size=8)

#coordinate information is meaningless here, so let's remove it
plt.xticks([])
plt.yticks([])
remove_border(left=False, bottom=False)


I am working on the file "HW5.ipynb".
The first task is:
How large n needs to be in order for the difference between the max of the function $f(x) = sin\dfrac{1}{x+eps}$ in [0,1] using n nodes and n+10 nodes to be less than 0.1:
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
Part 2:
Value of function f(x) = $e^(-x^2) * cos(2\pi x)$ at x = -0.45 :
q = 2
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
Problem 2
Now, turn these data into a NetworkX graph, according to the spec below. For details on using NetworkX, consult the lab materials for November 1, as well as the NetworkX documentation.
Can you write Python code for it?

"""
Function
--------
vote_graph

Parameters
----------
data : list of dicts
    The vote database returned from get_vote_data

Returns
-------
graph : NetworkX Graph object, with the following properties
    1. Each node in the graph is labeled using the `display_name` of a Senator (e.g., 'Lee (R-UT)')
    2. Each node has a `color` attribute set to 'r' for Republicans,
       'b' for Democrats, and 'k' for Independent/other parties.
    3. The edges between two nodes are weighted by the number of
       times two senators have cast the same Yea or Nay vote
    4. Each edge also has a `difference` attribute, which is set to `1 / weight`.

Examples
--------
>>> graph = vote_graph(vote_data)
>>> graph.node['Lee (R-UT)']
{'color': 'r'}  # attributes for this senator
>>> len(graph['Lee (R-UT)']) # connections to other senators
101
>>> graph['Lee (R-UT)']['Baldwin (D-WI)']  # edge relationship between Lee and Baldwin
{'difference': 0.02, 'weight': 50}
"""
def _color(s):
    if '(R' in s:
        return 'r'
    if '(D' in s:
        return 'b'
    return 'k'

def vote_graph(data):

    senators = set(x['display_name'] for d in data for vote_grp in d['votes'].values() for x in vote_grp)
    weights = {s: {ss: 0 for ss in senators if ss != s} for s in senators}

    for d in data:
        for grp in ['Yea', 'Nay']:
            if grp not in d['votes']:
                continue
            vote_grp = d['votes'][grp]
            for i in range(len(vote_grp)):
                for j in range(i + 1, len(vote_grp)):
                    sen1 = vote_grp[i]['display_name']
                    sen2 = vote_grp[j]['display_name']
                    weights[min(sen1, sen2)][max(sen1, sen2)] += 1

    g = nx.Graph()
    for s in senators:
        g.add_node(s)
        g.node[s]['color'] = _color(s)

    for s1, neighbors in weights.items():
        for s2, weight in neighbors.items():
            if weight == 0:
                continue
            g.add_edge(s1, s2, weight=...


I am working on the file "HW5.ipynb".
The first task is:
Problem 6
Centrality isn't a perfect proxy for bipartisanship, since it gauges how centralized a node is to the network as a whole, and not how similar a Democrat node is to the Republican sub-network (and vice versa).
Can you come up with another measure that better captures bipartisanship than closeness centrality? Develop your own metric -- how does it differ from the closeness centrality? Use visualizations to support your points.
Can you write Python code for it?

#your code here

"""
Here, we compute the mean weight for the edges that connect a Senator
to a node in the other party (we consider Independents to be Democrats
for this analysis).

This only considers how similarly a Senator votes with the other party.

The scatter plot shows that the betweenness centrality and bipartisan score
correlate with each other. However, the betweenness centrality judges Democrats
to be more bipartisan as a whole. Part of this is a bias due to the fact
that Democrats are the majority party in the Senate right now, so their
votes are considered more "central" due to their bigger numbers.
"""
def bipartisan_score(graph, node):
    party = graph.node[node]['color']
    other = 'r' if party != 'r' else 'b'
    return np.mean([v['weight'] for k, v in graph[node].items() if graph.node[k]['color'] == other])

bp_score = {node: bipartisan_score(votes, node) for node in votes.nodes()}
bp2 = sorted(bp_score, key=lambda x: -1 * bp_score[x])

print "Most Bipartisan"
for senator in bp2[:5]:
    print "%20.20s\t%0.3f" % (senator, bp_score[senator])

print
print "Least Bipartisan"
for senator in bp2[-5:]:
    print "%20.20s\t%0.3f" % (senator, bp_score[senator])

senators = bp_score.keys()
x = [bet[s] for s in senators]
y = [bp_score[s] for s in senators]
c = [votes.node[s]['color'] for s in senators]

plt.scatter(x, y, 80, color=c,
            alpha=.5, edgecolor='white')
plt.xlabel("Betweenness Centrality")
plt.ylabel("Bipartisan Score")
remove_border()


I am working on the file "HW5.ipynb".
The first task is:
Problem 8
Write a function to builded a Directed Graph (DiGraph) from these data, according to the following spec:
Can you write Python code for it?

"""
Function
--------
bill_graph

Turn the bill graph data into a NetworkX Digraph

Parameters
----------
data : list of dicts
    The data returned from get_all_bills

Returns
-------
graph : A NetworkX DiGraph, with the following properties
    * Each node is a senator. For a label, use the 'name' field
      from the 'sponsor' and 'cosponsors' dict items
    * Each edge from A to B is assigned a weight equal to how many
      bills are sponsored by B and co-sponsored by A
"""
#Your code here
bg = nx.DiGraph()

def bill_graph(data):

    sp = nx.DiGraph()

    for bill in data:
        sponsor = bill['sponsor']['name']
        sponsor_data = bill['sponsor']

        cosponsors = [cs['name'] for cs in bill['cosponsors']]

        if sponsor not in sp:
            sp.add_node(sponsor, **sponsor_data)

        for cosponsor in bill['cosponsors']:
            if cosponsor['name'] not in sp:
                sp.add_node(cosponsor['name'], **cosponsor)
            cosponsor = cosponsor['name']

            try:
                w = sp[cosponsor][sponsor]['weight'] + 1
            except KeyError:
                w = + 1
            sp.add_edge(cosponsor, sponsor, weight=w)

    return sp

bills = bill_graph(bill_list)


I am working on the file "HW5.ipynb".
The first task is:
Problem 9
Using nx.pagerank_numpy, compute the PageRank score for each senator in this graph. Visualize the results. Determine the 5 Senators with the highest
PageRank scores. How effective is this approach at identifying leaders? How does the PageRank rating compare to the degree of each node?
Note: you can read about individual Senators by searching for them on the govtrack website.
Can you write Python code for it?

#Your code here
pagerank = nx.pagerank_numpy(bills)
names = np.array(pagerank.keys())
vals = np.array([pagerank[n] for n in names])

ind = np.argsort(vals)
names = names[ind]
vals = vals[ind]

print "Highest Scores"
for n, v in zip(names, vals)[-5:][::-1]:
    print "%20.20s\t%0.3f" % (n, v)

print
print "Lowest Scores"
for n, v in zip(names, vals)[:5]:
    print "%20.20s\t%0.3f" % (n, v)

#Your code here
deg = nx.degree(bills)

plt.scatter([deg[n] for n in bills.nodes()],
            [pagerank[n] for n in bills.nodes()], 80, alpha=.8,
            color='k', edgecolor='white')

labels = ['Reid, Harry', 'Lautenberg, Frank R.', 'Menendez, Robert', 'Harkin, Tom']
for lbl in labels:
    plt.annotate(lbl, (deg[lbl], pagerank[lbl] + .002), fontsize=10, rotation=10)

plt.xlabel("Degree")
plt.ylabel("PageRank")
remove_border()


I am working on the file "HW5.ipynb".
The first task is:
The plot is symmetric over the y-axis. Therefore the integral of the function from -$\infty$ to $\infty$ will be the same as twice the integral of the function from 0 to $\infty$
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
Your discussion here
Leadership in the Senate
There are many metrics to quantify the leadership in the Senate.

Senate leaders sponsor and co-sponsor lots of bills
Leaders sit on many committees, as well as more important committees
Leaders usually have been in office for a long time

Another approach uses the philosophy behind how Google ranks search results. The core idea behind Google's PageRank algorithm is:

A "good" website (i.e. one to rank highly in search results) is linked to by many other websites
A link found on a "good" website is more important than a link found on a "bad" website

The PageRank algorithm thus assigns scores to nodes in a graph based on how many neighbors a node has, as well as the score of those neighbors.
This technique can be adapted to rank Senate leadership. Here, nodes correspond to Senators, and edges correspond to a senator co-sponsoring a bill sponsored by another Senator. The weight of each edge from node A to B is the number of times Senator A has co-sponsored a bill whose primary sponsor is Senator B. If you interpret the PageRank scores of such a network to indicate Senate leadership, you are then assuming:

Leaders sponsor more bills
Leaders attract co-sponsorship from other leaders

Problem 7
Govtrack stores information about each Senate bill in the current congress at http://www.govtrack.us/data/congress/113/bills/s/. As in problem 1, write two functions to scrape these data -- the first function downloads a single bill, and the second function calls the first to loop over all bills.
Can you write Python code for it?

"""
Function
--------
get_senate_bill

Scrape the bill data from a single JSON page, given the bill number

Parameters
-----------
bill : int
   Bill number to fetch

Returns
-------
A dict, parsed from the JSON

Examples
--------
>>> bill = get_senate_bill(10)
>>> bill['sponsor']
{u'district': None,
 u'name': u'Reid, Harry',
 u'state': u'NV',
 u'thomas_id': u'00952',
 u'title': u'Sen',
 u'type': u'person'}
>>> bill['short_title']
u'Agriculture Reform, Food, and Jobs Act of 2013'
"""
#your code here
#your code here
def get_senate_bill(bill):
    url  = 'https://www.govtrack.us/data/congress/113/bills/s/s%i/data.json' %bill
    bill_data = requests.get(url)
    return json.loads(bill_data.text)

"""
Function
--------
get_all_bills

Scrape all Senate bills at http://www.govtrack.us/data/congress/113/bills/s

Parameters
----------
None

Returns
-------
A list of dicts, one for each bill
"""
#your code here
def get_all_bills():
    url = 'http://www.govtrack.us/data/congress/113/bills/s'
    response = requests.get(url)
    soup = bs(response.content, "lxml")
    s_bills = [a['href'][1:-1] for a in soup.find_all('a')  if a['href'].startswith('s')]
    n_bills = len(s_bills)
    return [get_senate_bill(i) for i in range(1, n_bills+1)]

#write json data in to a file
bills = json.dumps(get_all_bills())
with open('./data/bills.json','w') as fp:
    fp.write(bills)

#read data from file
bill_list = json.load(open('./data/bills.json'))

bill_list[0]['cosponsors']


I am working on the file "HW5.ipynb".
The first task is:
n = 200 * eps (for an epsilon of 0.2). Increasing n further does not change the plot so that it is visible on the screen.
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
q = 4
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
q = 8
Can you write Python code for it?


I am working on the file "HW5.ipynb".
The first task is:
q= 16
Can you write Python code for it?


I am working on the file "HW5/notebook-examples-master/example-bs4.ipynb".
The first task is:
And we can look for messy data:
Can you write Python code for it?

[year for year in raw_year_list if not year.isnumeric()]


I am working on the file "HW5/notebook-examples-master/example-bs4.ipynb".
The first task is:
Or you can use good old CSS selectors, to actually find all the years his movies were made in:
Can you write Python code for it?


I am working on the file "HW5/notebook-examples-master/example-bs4.ipynb".
The first task is:
Scraping the Info You Want
Now there are a few ways to get content out.   For instance, to get the title you could treat it like an object:
Can you write Python code for it?


I am working on the file "HW5/notebook-examples-master/example-sqlalchemy.ipynb".
The first task is:
Basic SQL Generation
The core library generates SQL for you.  Read more about it on their expression language tutorial page. Below are some basic examples.
Creating a Table
Read more about defining and creating tables.
Can you write Python code for it?

# add `echo=True` to see log statements of all the SQL that is generated
engine = create_engine('sqlite:///:memory:',echo=True) # just save the db in memory for now (ie. not on disk)
metadata = MetaData()
# define a table to use
queries = Table('query', metadata,
    Column('id', Integer, primary_key=True),
    Column('keywords', String(400), nullable=False),
    Column('timestamp', DateTime, default=datetime.datetime.now),
)
metadata.create_all(engine) # and create the tables in the database


I am working on the file "HW5/notebook-examples-master/example-sqlalchemy.ipynb".
The first task is:
ORM
You can use their ORM library to handle the translation into full-fledged python objects.  This can help you build the Model for you MVC solution.
Can you write Python code for it?

import datetime
from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
Base = declarative_base()


I am working on the file "HW5/notebook-examples-master/example-sqlalchemy.ipynb".
The first task is:
Retrieving Data
Read more about retrieving data from the db via an ORM class.
Can you write Python code for it?

for q in my_session.query(Query).order_by(Query.timestamp):
    print q

query1 = Query(keywords="robot")
query2 = Query(keywords="puppy")
my_session.add_all([query1,query2])
my_session.commit()

for q in my_session.query(Query).order_by(Query.timestamp):
    print q

for q in my_session.query(Query).filter(Query.keywords.like('r%')):
    print q

import mediacloud, datetime
mc = mediacloud.api.MediaCloud('48d9fd877527e6c8c5ff25553c124b1634b8b2926310e2d259d6518221e2a5f2')
agreement = mc.sentenceCount('( colombia AND peace AND agreement)', solr_filter=[mc.publish_date_query( datetime.date( 2015, 10, 1), datetime.date( 2016, 12, 2) ), 'tags_id_media:1' ])
print agreement['count']


I am working on the file "HW5/notebook-examples-master/example-sqlalchemy.ipynb".
The first task is:
Retrieving Data
Read more about using SQL select statments.
Can you write Python code for it?

from sqlalchemy.sql import select
select_stmt = select([queries])
results = db_conn.execute(select_stmt)
for row in results:
    print row

select_stmt = select([queries]).where(queries.c.id==1)
for row in db_conn.execute(select_stmt):
    print row

select_stmt = select([queries]).where(queries.c.keywords.like('p%'))
for row in db_conn.execute(select_stmt):
    print row


I am working on the file "HW5_solutions.ipynb".
The first task is:
How (and how not) to visualize networks
Network plots often look impressive, but creating sensible network plots is tricky. From Ben Fry, the author of the Processing program:
<blockquote>
Usually a graph layout isn’t the best option for data sets larger than a few dozen nodes. You’re most likely to wind up with enormous spider webs or balls of string, and the mess seen so far is more often the case than not. Graphs can be a powerful way to represent relationships between data, but they are also a very abstract concept, which means that they run the danger of meaning something only to the creator of the graph. Often, simply showing the structure of the data says very little about what it actually means, even though it’s a perfectly accurate means of representing the data. Everything looks like a graph, but almost nothing should ever be drawn as one.
</blockquote>

Let's look at bad and better ways of visualizing the senate vote network.
First, consider the "default" plot from networkx.
Can you write Python code for it?


I am working on the file "HW5_solutions.ipynb".
The first task is:
Problem 2
Now, turn these data into a NetworkX graph, according to the spec below. For details on using NetworkX, consult the lab materials for November 1, as well as the NetworkX documentation.
Can you write Python code for it?

"""
Function
--------
vote_graph

Parameters
----------
data : list of dicts
    The vote database returned from get_vote_data

Returns
-------
graph : NetworkX Graph object, with the following properties
    1. Each node in the graph is labeled using the `display_name` of a Senator (e.g., 'Lee (R-UT)')
    2. Each node has a `color` attribute set to 'r' for Republicans,
       'b' for Democrats, and 'k' for Independent/other parties.
    3. The edges between two nodes are weighted by the number of
       times two senators have cast the same Yea or Nay vote
    4. Each edge also has a `difference` attribute, which is set to `1 / weight`.

Examples
--------
>>> graph = vote_graph(vote_data)
>>> graph.node['Lee (R-UT)']
{'color': 'r'}  # attributes for this senator
>>> len(graph['Lee (R-UT)']) # connections to other senators
101
>>> graph['Lee (R-UT)']['Baldwin (D-WI)']  # edge relationship between Lee and Baldwin
{'difference': 0.02, 'weight': 50}
"""
#Your code here

def _color(s):
    if '(R' in s:
        return 'r'
    if '(D' in s:
        return 'b'
    return 'k'

def vote_graph(data):

    senators = set(x['display_name'] for d in data for vote_grp in d['votes'].values() for x in vote_grp)
    weights = {s: {ss: 0 for ss in senators if ss != s} for s in senators}

    for d in data:
        for grp in ['Yea', 'Nay']:
            if grp not in d['votes']:
                continue
            vote_grp = d['votes'][grp]
            for i in range(len(vote_grp)):
                for j in range(i + 1, len(vote_grp)):
                    sen1 = vote_grp[i]['display_name']
                    sen2 = vote_grp[j]['display_name']
                    weights[min(sen1, sen2)][max(sen1, sen2)] += 1

    g = nx.Graph()
    for s in senators:
        g.add_node(s)
        g.node[s]['color'] = _color(s)

    for s1, neighbors in weights.items():
        for s2, weight in neighbors.items():
            if weight == 0:
                continue
           ...


I am working on the file "HW5_solutions.ipynb".
The first task is:
Problem 4
While this graph has less information, the remaining information is easier to digest. What does the Minimum Spanning Tree mean in this context? How does this graph relate to partisanship in the Senate? Which nodes in this graph are the most and least bi-partisan?
*The edges of a minimum spanning tree trace a path of low resistance through the network. In the present context, this has the effect of moving bipartisan Senators like Hagan towards the center of the graph -- it is much easier to connect Hagan to a Republican node than, say, a partisan Democrat like Al Franken. Partisan Senators are pushed away from the center of the graph and deeper into the party cliques. *
This scheme also moves outlier senators to the outside of the graph. For example, John Kerry cast very few votes before becoming Secretary of State. Most of the edges connected to John Kerry have large difference values, so the fewest possible number of edges (1) remain in the MST.
Problem 5
(For this problem, use the full graph for centrality computation, and not the Minimum Spanning Tree)
Networkx can easily compute centrality measurements.
Briefly discuss what closeness_centrality means, both mathematically and in the context of the present graph -- how does the centrality relate to partisanship? Choose a way to visualize the closeness_centrality score for each member of the Senate, using edge difference as the distance measurement. Determine the 5 Senators with the highest and lowest centralities.
Comment on your results. In particular, note the outliers John Kerry (who recently resigned his Senate seat when he became Secretary of State), Mo Cowan (Kerry's interim replacement) and Ed Markey (Kerry's permanent replacement) have low centrality scores -- why?
The closeness centrality measures the average difference between a Senator and all other Senators. Bipartisan voters will have more shared votes with the members of the opposite party, which tends to increase their centrality. However, these senators also vote less often with...
Can you write Python code for it?

#Your code here

bet = nx.closeness_centrality(votes, distance='difference')
bipartisans = sorted(bet, key=lambda x: -bet[x])

print "Highest closeness"
for senator in bipartisans[:5]:
    print "%20.20s\t%0.3f" % (senator, bet[senator])
print
print "Lowest closeness"
for senator in bipartisans[-5:]:
    print "%20.20s\t%0.3f" % (senator, bet[senator])

plt.figure(figsize=(15, 4))
x = np.arange(len(nl))
y = np.array([bet[n] for n in nl])
c = np.array([votes.node[n]['color'] for n in nl])

ind = np.argsort(y)
y = y[ind]
c = c[ind]

plt.bar(x, y, color=c, align='center', width=.8)

remove_border(left=None, bottom=None)
ticks = plt.xticks(x, [nl[i] for i in x[ind]],
                   rotation='vertical', fontsize=7)
limits = plt.xlim(-1, x[-1] + 1)


I am working on the file "HW5_solutions.ipynb".
The first task is:
Problem 8
Write a function to builded a Directed Graph (DiGraph) from these data, according to the following spec:
Can you write Python code for it?

"""
Function
--------
bill_graph

Turn the bill graph data into a NetworkX Digraph

Parameters
----------
data : list of dicts
    The data returned from get_all_bills

Returns
-------
graph : A NetworkX DiGraph, with the following properties
    * Each node is a senator. For a label, use the 'name' field
      from the 'sponsor' and 'cosponsors' dict items
    * Each edge from A to B is assigned a weight equal to how many
      bills are sponsored by B and co-sponsored by A
"""
#Your code here

def bill_graph(data):

    sp = nx.DiGraph()

    for bill in data:
        sponsor = bill['sponsor']['name']
        sponsor_data = bill['sponsor']

        cosponsors = [cs['name'] for cs in bill['cosponsors']]

        if sponsor not in sp:
            sp.add_node(sponsor, **sponsor_data)

        for cosponsor in bill['cosponsors']:
            if cosponsor['name'] not in sp:
                sp.add_node(cosponsor['name'], **cosponsor)
            cosponsor = cosponsor['name']

            try:
                w = sp[cosponsor][sponsor]['weight'] + 1
            except KeyError:
                w = + 1
            sp.add_edge(cosponsor, sponsor, weight=w)

    return sp

bills = bill_graph(bill_list)


I am working on the file "HW5_solutions.ipynb".
The first task is:
Problem 9
Using nx.pagerank_numpy, compute the PageRank score for each senator in this graph. Visualize the results. Determine the 5 Senators with the highest
PageRank scores. How effective is this approach at identifying leaders? How does the PageRank rating compare to the degree of each node?
Note: you can read about individual Senators by searching for them on the govtrack website.
Can you write Python code for it?

#Your code here

pagerank = nx.pagerank_numpy(bills)
names = np.array(pagerank.keys())
vals = np.array([pagerank[n] for n in names])

ind = np.argsort(vals)
names = names[ind]
vals = vals[ind]

print "Highest Scores"
for n, v in zip(names, vals)[-5:][::-1]:
    print "%20.20s\t%0.3f" % (n, v)

print
print "Lowest Scores"
for n, v in zip(names, vals)[:5]:
    print "%20.20s\t%0.3f" % (n, v)

#Your code here

deg = nx.degree(bills)

plt.scatter([deg[n] for n in bills.nodes()],
            [pagerank[n] for n in bills.nodes()], 80, alpha=.8,
            color='k', edgecolor='white')

labels = ['Reid, Harry', 'Lautenberg, Frank R.', 'Menendez, Robert', 'Harkin, Tom']
for lbl in labels:
    plt.annotate(lbl, (deg[lbl], pagerank[lbl] + .002), fontsize=10, rotation=10)

plt.xlabel("Degree")
plt.ylabel("PageRank")
remove_border()


I am working on the file "HW5_solutions.ipynb".
The first task is:
The PageRank approach does seem to be effective at identifying influential Senators like Tom Harkin and Harry Reid (the Majority Leader). We see in particular that Harry Reid's PageRank score is relatively higher than his degree -- he seems to sponsor fewer bills overall, but those bills appear to be more important. This makes sense, since he is the figurehead of the Democratic party in the Senate, and thus probably focuses on the highest-profile legislation.
Interactive Visualization
Producing a good node link layout is not quite so simple. Nevertheless, we will give it a try.
We will use Gephi for interactive graph visualization. Gephi supports a wide variety of graph file formats, and NetworkX exports to several of them. We'll use the Graph Exchange XML Format (GEXF).
Can you write Python code for it?


I am working on the file "HW5_solutions.ipynb".
The first task is:
The spring layout tries to group nodes with large edge-weights near to each other. In this context, that means it tries to organize the Senate into similarly-voting cliques. However, there's simply too much going on in this plot -- we should simplify the representation.
Problem 3
Compute the Minimum Spanning Tree of this graph, using the difference edge attribute as the weight to minimize. A Minimum Spanning Tree is the subset of edges which trace at least one path through all nodes ("spanning"), with minimum total edge weight. You can think of it as a simplification of a network.
Plot this new network, making modifications as necessary to prevent the graph from becoming too busy.
Can you write Python code for it?

#Your code here
plt.figure(figsize=(15, 10))
np.random.seed(5)
mst = nx.minimum_spanning_tree(votes, weight='difference')
pos = nx.spring_layout(mst, iterations=900, k=.008, weight='difference')

mst_edges = list(nx.minimum_spanning_edges(votes, weight='difference'))

nl = votes.nodes()
c = [votes.node[n]['color'] for n in nl]
nx.draw_networkx_edges(votes, pos, edgelist=mst_edges, alpha=.2)
nx.draw_networkx_nodes(votes, pos, nodelist = nl, node_color = c, node_size=60)

for p in pos.values():
    p[1] += .02

nx.draw_networkx_labels(votes, pos, font_color='k', font_size=7)

plt.title("MST of Vote Disagreement", fontsize=18)
plt.xticks([])
plt.yticks([])
remove_border(left=False, bottom=False)


I am working on the file "HW6/Clustering Analysis.ipynb".
The first task is:
Step2: Modify the data
We will keep using the old database prepared in Homework 1. Since we are only trying to cluster the data vectors into 2 clusters, we would like to preprocess the data. So that the days with precipitation will be represented by 1 and the days without precipitation will be represented by 0.
Can you write Python code for it?

#pprint(Data)
for x in range(0,len(Data)):
    if (Data.loc[x,'Precip']>0):
        Data.loc[x, 'Precip']=1
#pprint(Data)


I am working on the file "HW6/Clustering Analysis.ipynb".
The first task is:
Step3: Testing with 2 features first
For a visualization purpose, we would like to start with 2 features (bike load of two brideges). In that we can plot them a look at the data visually.
Can you write Python code for it?

features=np.array(Data[['Manhattan', 'Brooklyn']])
plt.scatter (features[:,0], features[:, 1])
plt.show()

clf= KMeans (n_clusters=2)
clf.fit(features)
cetroids=clf.cluster_centers_
labels=clf.labels_
colors=["b.", "r."]
for i in range (len(features)):
    plt.plot(features[i][0], features[i][1], colors[labels[i]])
plt.scatter(cetroids[:,0], cetroids[:, 1], marker='x', s=150, linewidths=50,c="g" )
plt.show()

precip=np.array(Data['Precip'])
pprint(precip)
pprint(labels)


I am working on the file "HW6/Clustering Analysis.ipynb".
The first task is:
Step4: Result analysis
By looking at hte data and the data and the labels, we see that 0 represent blue which should be the days with positive precipitaion, because we should have less bike load when it's raining. That means what when data['precip'] =1 the label should be 0. by knowing that we can check how many data are miss classified. In that we can check how well this clustering models works to correctly classifying a future new data
Can you write Python code for it?

precip=np.array(Data['Precip'])
true=0
for i in range(0,len(precip)):
    if ((precip[i]+labels[i])==1): # when correctly classified, precip=0, label=1; or precip=1, label=0
        true=true+1
pprint (true)
print('% of correct prediction is ')
print(true/len(precip))


I am working on the file "HackNight_19Oct17/xarrayTutorial_SaiNudurupati/Intro_to_xarray.ipynb".
The first task is:
Indexing this DataArray
traditional or the Matlab/Numpy way
Can you write Python code for it?


I am working on the file "HackNight_19Oct17/xarrayTutorial_SaiNudurupati/Intro_to_xarray.ipynb".
The first task is:
N-dimensional Numpy Arrays
Can you write Python code for it?

import numpy as np

sst = np.random.random(size=(2, 4, 4))

print sst
# print(sst[:, ::1, ::3])  # indexing

sst.shape

print(sst[:, ::1, ::3])


I am working on the file "HackNight_19Oct17/xarrayTutorial_SaiNudurupati/Intro_to_xarray.ipynb".
The first task is:
Open a dataset
Can you write Python code for it?


I am working on the file "HackNight_19Oct17/xarrayTutorial_SaiNudurupati/Intro_to_xarray.ipynb".
The first task is:
Plotting data in 2 dimensions
Can you write Python code for it?

map_data = ds['pcp'].sel(time='2010-01-01')

map_data.plot()


I am working on the file "HackNight_19Oct17/xarrayTutorial_SaiNudurupati/Intro_to_xarray.ipynb".
The first task is:
The network Common Data Format (netCDF)

collection of self-describing, machine-independent binary data formats and software tools that facilitate the creation, access and sharing of data stored in N-dimensional arrays

Xarray

extends some of core functionality of the Pandas library to N-dimensional arrays
can be used for:
multidimensional data (e.g. climate data: x, y, z, time)
structured data on a regular grid
data in netCDF format
Two main data structures of xarray
DataArray
Dataset

DataArray

xarray's implementation of a labeled, multi-dimensional array
the DataArray has these key properties:
data: Ndimensional array(NumPy or dask)
dims: dimension names for each axis
coords: dictionary-like container of arrays that label each point, and
attrs: ordered dictionary holding metadata

<img src="http://xray.readthedocs.org/en/stable/_images/dataset-diagram.png" width = "800" border = "10">
Dataset

xarray's multi=dimensional equivalent of a Pandas Dataframe
dict-like container of DataArray objects with aligned dimensions
Datasets have these key properties:
dims: dictionary mapping from dimension names to the fixed length of each dimension,
data_vars: dict-like container of DataArrays corresponding to data variables,
coords: dictionary-like container of DataArrays intended to label points used in data_vars
attrs: ordered dictionary holding metadata

Lets look at some datasets
Import xarray library
Can you write Python code for it?


I am working on the file "HackNight_19Oct17/xarrayTutorial_SaiNudurupati/Intro_to_xarray.ipynb".
The first task is:
Want to customize it?
Can you write Python code for it?

import matplotlib.pyplot as plt
map_data.plot(cmap=plt.cm.Blues)
plt.title('Global Precipitation data')
plt.tight_layout()
plt.show()


I am working on the file "HackNight_19Oct17/xarrayTutorial_SaiNudurupati/Intro_to_xarray.ipynb".
The first task is:
using .isel (refers to seleciton by integer position)
Can you write Python code for it?


I am working on the file "HackerRank/Intro_to_Statistics/Day_01.ipynb".
The first task is:
Day 1: Standard Deviation Puzzles #2
Objective
In this challenge, we practice calculating standard deviation.
Task
The heights of a group of children are measured. The resulting data has a mean of 0.675 meters, and a standard deviation of 0.065 meters. One particular child is 90.25 centimeters tall. Compute z, the number of standard deviations away from the mean that the particular child is.
Enter the value of z, correct to a scale of two decimal places.
Can you write Python code for it?

var_2_mean = 0.675
var_2_std = 0.065

child = 0.9025

(child - var_2_mean)/var_2_std


I am working on the file "HackerRank/Intro_to_Statistics/Day_04.ipynb".
The first task is:
Day 4: The Central Limit Theorem #1
Objective
In this challenge, we practice solving problems based on the Central Limit Theorem.
Task
A large elevator can transport a maximum of 9800 pounds. Suppose a load of cargo containing 49 boxes must be transported via the elevator. The box weight of this type of cargo follows a distribution with a mean of µ=205 pounds and a standard deviation of σ=15 pounds. Based on this information, what is the probability that all 49 boxes can be safely loaded onto the freight elevator and transported?
Can you write Python code for it?

# #Distribution
scipy.stats.norm(205, 15)


I am working on the file "HackerRank/Intro_to_Statistics/Day_04.ipynb".
The first task is:
Normal Distribution with mean=0 and std. dev=1
x ~ N(0, 1)
Normal Distribution with mean=205 and std. dev=15
x ~ N(205, 15)
Normal Distribution with mean=205 and std. dev=15, multiplied by n samples
nx ~ N(205n, 15*sqrt(n))
Can you write Python code for it?


I am working on the file "HackerRank/Intro_to_Statistics/Day_05.ipynb".
The first task is:
Day 5: Introduction to Linear Regression
Objective
In this challenge, we practice using linear regression techniques. Check out the Resources tab to learn more!
Task
You are given the Math aptitude test (x) scores for a set of students, as well as their respective scores for a Statistics course (y). The students enrolled in Statistics immediately after taking the math aptitude test.
The scores (x, y) for each student are:
(95,85)
(85,95)
(80,70)
(70,65)
(60,70)
If a student scored an 80 on the Math aptitude test, what score would we expect her to achieve in Statistics?
Determine the equation of the best-fit line using the least squares method, and then compute the value of y when x=80.
Can you write Python code for it?

# #Python Import Libraries
import sklearn
import numpy as np

arr_x = [i[0] for i in arr_data]
arr_y = [i[1] for i in arr_data]

stats.linregress(arr_x, arr_y)

m, c, r_val, p_val, err = stats.linregress(arr_x, arr_y)

# #y = mx + c
m*80 + c


I am working on the file "Hacker_Rank/01-Introduction/01_Hacker_Rank_Introduction.ipynb".
The first task is:
Task 2:
Read two integers from STDIN and print three lines where:
The first line contains the sum of the two numbers.
The second line contains the difference of the two numbers (first - second).
The third line contains the product of the two numbers.
Input Format
The first line contains the first integer, . The second line contains the second integer, .
Constraints
1<=a<=10e10 and 1<=b<=10e10
Output Format
Print the three lines as explained above.
Sample Input
3
2
Sample Output
5
1
6
Answer:
Can you write Python code for it?

if __name__ == '__main__':
    a = int(input())
    b = int(input())
    if 1<=a<=10e10 and 1<=b<=10e10:
        print (a+b)
        print (a-b)
        print (a*b)


I am working on the file "Hacker_Rank/01-Introduction/01_Hacker_Rank_Introduction.ipynb".
The first task is:
Task 4:
Read an integer . For all non-negative integers , print . See the sample for details.
Input Format
The first and only line contains the integer, .
Constraints
1<=n<=20
Output Format
Print  lines, one corresponding to each .
Sample Input
5
Sample Output
0
1
4
9
16
Answer:
Can you write Python code for it?

if __name__ == '__main__':
    n = int(input())
    if 1<=n<=20:
        for i in range(0,n):
            print (i*i)


I am working on the file "Hacker_Rank/01-Introduction/01_Hacker_Rank_Introduction.ipynb".
The first task is:
Task 5:
We add a Leap Day on February 29, almost every four years. The leap day is an extra, or intercalary day and we add it to the shortest month of the year, February.
In the Gregorian calendar three criteria must be taken into account to identify leap years:
The year can be evenly divided by 4, is a leap year, unless:
The year can be evenly divided by 100, it is NOT a leap year, unless:
The year is also evenly divisible by 400. Then it is a leap year.
This means that in the Gregorian calendar, the years 2000 and 2400 are leap years, while 1800, 1900, 2100, 2200, 2300 and 2500 are NOT leap years.Source
Task
You are given the year, and you have to write a function to check if the year is leap or not.
Note that you have to complete the function and remaining code is given as template.
Input Format
Read y, the year that needs to be checked.
Constraints
1900 <= y <= 10e5
Output Format
Output is taken care of by the template. Your function must return a boolean value (True/False)
Sample Input
1990
Sample Output
False
Explanation
1990 is not a multiple of 4 hence it's not a leap year.
Answer:
Can you write Python code for it?

def is_leap(year):
    if 1900<= year <= 10e5:
        if year%4==0:
            if year%100==0 and year%400==0:
                leap = True
            else:
                if year%100==0:
                    leap = False
                else:
                    leap = True
        else:
            leap = False
        return bool(leap)

year = int(input())
print(is_leap(year))


I am working on the file "Hacker_Rank/01-Introduction/01_Hacker_Rank_Introduction.ipynb".
The first task is:
Task 6:
Read an integer N.
Without using any string methods, try to print the following:
123...N
Note that "" represents the values in between.
Input Format
The first line contains an integer N.
Output Format
Output the answer as explained in the task.
Sample Input
3
Sample Output
123
Answer:
Can you write Python code for it?

if __name__ == '__main__':
    n = int(input())
    for i in range(1, n+1):
        print(i, end='')


I am working on the file "Hacker_Rank/02-Basic_Data_Types/02_Hacker_Rank_Basic_data_types.ipynb".
The first task is:
Answer:
Can you write Python code for it?

x, y, z, n = (int(input()) for _ in range(4))
print ([[i,j,k] for i in range(0,x+1) for j in range(0,y+1) for k in range(0,z+1) if i + j + k != n ])


I am working on the file "Hacker_Rank/03-Strings/11_Find_a_string.ipynb".
The first task is:
Check if a string is ascii encoded:
Can you write Python code for it?

try:
    string.encode('ascii')
    print ('This string is ascii ...')
except:
    print ('This string is not ascii encoded ...')


I am working on the file "Hacker_Rank/03-Strings/11_Find_a_string.ipynb".
The first task is:
Implement a basic sub-string counter:
Can you write Python code for it?

def count_substring(string, sub_string):
    count = 0
    for i in range(0, len(string)-len(sub_string)+1):
        if (string[i:(i+len(sub_string))]) == sub_string:
            count+=1
    return count

count_substring(string, sub_string)


I am working on the file "Hacker_Rank/03-Strings/11_Find_a_string.ipynb".
The first task is:
Solution:
Can you write Python code for it?

def count_substring(string,
                    sub_string,
                    verbose = False,
                    min_string_len =1,
                    max_string_len=200):
    ascii_encoding = False
    try:
        string.encode('ascii')
        sub_string.encode('ascii')
        ascii_encoding = True
        if verbose:
            print ('This string is ascii ...')
    except:
        if verbose:
            print ('This string is not ascii encoded ...')
    if len(string)>= min_string_len and len(string)<=max_string_len and ascii_encoding:
        count = 0
        for i in range(0, len(string)-len(sub_string)+1):
            if (string[i:(i+len(sub_string))]) == sub_string:
                count+=1
    return count

count_substring(string, sub_string)


I am working on the file "Hagerstrand II.ipynb".
The first task is:
Calculemos su I de Moran:
Can you write Python code for it?

import pysal as ps
w = ps.lat2W(s.M,s.N,rook=False)
mr = ps.Moran(s.result[:,:,9].flatten(),w)


I am working on the file "Hagerstrand II.ipynb".
The first task is:
Como ya dijimos antes en la realidad es difícil observar fenómenos de difusión "pura", para simular esta situación, la clase SimpleDiffusion tiene el método mixed_diffusion() que mezcla los dos tipos de difusión que hemos visto (aleatoria y espacial) de acuerdo a una proporción:
Can you write Python code for it?

s = SimpleDiffusion(50,50,5,20,[(20,20)],0.1,15)
s.mixed_diffusion(0.5)
plt.imshow(s.result[:,:,s.max_iter-1])


I am working on the file "Hagerstrand II.ipynb".
The first task is:
El argumento que le pasamos a s.mixed_diffusion(0.5) le dice al algoritmo que, en este caso, la mitad de los eventos de difusión los haga de acuerdo a Hagerstrand y la otra mitad los haga aleatoriamente. De modo que si la proporción es 0, el fenómeno es aleatorio y si es 1 es totalmente espacial.
Calculemos la I de Moran para este caso:
Can you write Python code for it?

s = SimpleDiffusion(100,100,5,20,[(20,20)],0.2,20)
s.mixed_diffusion(0.7)
w = ps.lat2W(s.M,s.N,rook=False)
mr_spatial = ps.Moran(s.result[:,:,9].flatten(),w)
print "La I de Moran es %f" % mr_spatial.I
print "El valor esperado, bajo suposición de normalidad es %f" % mr_spatial.EI
print "El valor de significancia es %f" % mr_spatial.p_rand
plt.imshow(s.result[:,:,s.max_iter-1])


I am working on the file "Hagerstrand III.ipynb".
The first task is:
Ahora surge una pregunta interesante, ¿cómo se comportará la I de Moran sin la suposición de homogeneidad?
Can you write Python code for it?

import pysal as ps
w = ps.lat2W(ad.N,ad.N,rook=False)
mr = ps.Moran(ad.result[:,:,24].flatten(),w)
print "La I de Moran es %f" % mr.I
print "El valor esperado, bajo suposición de normalidad es %f" % mr.EI
print "El valor de significancia es %f" % mr.p_rand


I am working on the file "Hagerstrand III.ipynb".
The first task is:
Como pueden var, el resultado refleja la distribución original de la población. En términos del índice de Moran, ¿cómo se ve?
Can you write Python code for it?

mr = ps.Moran(ad.result[:,:,24].flatten(),w)
print "La I de Moran es %f" % mr.I
print "El valor esperado, bajo suposición de normalidad es %f" % mr.EI
print "El valor de significancia es %f" % mr.p_rand


I am working on the file "Hagerstrand III.ipynb".
The first task is:
Como pueden ver, los resultados cuando usamos un valor de 2.0 son más dispersos que cuando usamos un valor de 4.0.
Veamos ahora que pasa con el parámetro densidad:
Can you write Python code for it?


I am working on the file "Hagerstrand III.ipynb".
The first task is:
La idea de densidad es controlar la cantidad de núcleos de dispersión de población. Entoces, diferentes combinaciones de densidad y amplitud, nos van a generar diferentes escenarios iniciales de población que tratan de reflejar posibles condiciones en el espacio.
Jueguen un poco con estos parámetros hasta encontrar una combinación que los satisfaga.
Can you write Python code for it?


I am working on the file "Hagerstrand III.ipynb".
The first task is:
¿Y si lo comparamos con la difusión aleatoria en espacios homogéneos?
Can you write Python code for it?

sd.random_diffusion()
sr = ps.Moran(sd.result[:,:,24].flatten(),w)
print "La I de Moran es %f" % sr.I
print "El valor esperado, bajo suposición de normalidad es %f" % sr.EI
print "El valor de significancia es %f" % sr.p_rand


I am working on the file "Hamiltonian MCMC (HMC).ipynb".
The first task is:
NUTS Sampler
Toy implementation of No-U-Turn Sampler, described by Hoffman and Gelman (2011). Algorithm 3, page 14.
Can you write Python code for it?

def Leapfrog(U, theta, r, Eps, m=1.):
    """
    Slightly different update rules, since the negative log of the
    target PDF is not used.
    """
    gradient = lambda U, theta: derivative(U, theta, 0.01)
    r += (Eps/2.)*gradient(U, theta)
    theta += (Eps/m)*r
    r += (Eps/2.)*gradient(U, theta)
    return copy.copy(theta), copy.copy(r)

def BuildTree(U, theta, r, u, v, j, Eps, m=1., delta_max=1000):
    """
    Recursive tree-building.

    TODO: Make this less ugly.
    """
    if j == 0:
        # Take one leapfrog step in the direction v.
        theta_p, r_p = Leapfrog(U, theta, r, v*Eps, m=m)
        n_p = float(u <= exp(U(theta_p) - np.dot(0.5*r_p, r_p)))
        s_p = float(u < exp(delta_max + U(theta_p) - np.dot(0.5*r_p, r_p)))
        return theta_p, r_p, theta_p, r_p, theta_p, n_p, s_p
    else:
        # Recursion -- implicitly build the left and right subtrees.
        rargs = (u, v, j-1., Eps)
        rkwargs = {'m':m}
        theta_n, r_n, theta_f, r_f, theta_p, n_p, s_p = BuildTree(U, theta, r, *rargs, **rkwargs)
        if s_p == 1:
            if v == -1:
                theta_n, r_n, null, null, theta_dp, n_dp, s_dp = BuildTree(U, theta_n, r_n, *rargs, **rkwargs)
            else:
                null, null, theta_f, r_f, theta_dp, n_dp, s_dp = BuildTree(U, theta_f, r_f, *rargs, **rkwargs)
            try:
                if uniform.rvs() <= (n_dp/(n_p + n_dp)):
                    theta_p = copy.copy(theta_dp)
            except ZeroDivisionError:
                pass
            s_p = s_p*s_dp*int(np.dot((theta_f - theta_n), r_n) >= 0)*int( np.dot((theta_f - theta_n), r_f) >= 0)
            n_p += n_dp
        return theta_n, r_n, theta_f, r_f, theta_p, n_p, s_p

def NUTS_one_step(U, theta_last, Eps, m=1.):
    """
    TODO: clean up all the copies -- stop being so paranoid.
    """
    r_not = norm.rvs(0, 1., size=len(theta_last))
    u = uniform.rvs(0, exp(U(theta_last) - np.dot(0.5*r_not, r_not)))

    # Initialize.
    theta_m = copy.copy(theta_last)
    theta_n,...


I am working on the file "Hamiltonian MCMC (HMC).ipynb".
The first task is:
Testing on the banana
Can you write Python code for it?

start = np.array([[uniform.rvs(loc=-10., scale=15.),
                   uniform.rvs(loc=0., scale=10)]
                  for i in xrange(4)])

chains = np.array([ np.array(NUTS(dtarget, start[i, :], Eps=0.55, m=1.5, num_iters=10000)) for i in xrange(start.shape[0])])

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)

for i in xrange(chains.shape[0]):
    plt.scatter(chains[i, :, 0], chains[i, :, 1], alpha=0.5, s=0.02)
plt.show()

plt.subplot(211)
plt.title(Gelman(chains)[0])
for i in xrange(chains.shape[0]):
    plt.plot(chains[i, :, 0])
plt.ylabel('x1')

plt.subplot(212)
for i in xrange(chains.shape[0]):
    plt.plot(chains[i, :, 1])
plt.ylabel('x2')

plt.tight_layout()
plt.show()

plt.hist(chains[0,:,0])


I am working on the file "Hamiltonian MCMC (HMC).ipynb".
The first task is:
The surface of interest will be $U(q) = -\log{f(q)}$
$K(p) = \frac{p^T p}{2m}$, where $m$ = mass of the puck.
Position over time is a function of momentum:
$\frac{dq_i}{dt} = \frac{p_i}{m}$
Change in momentum over time is a function of surface gradient:
$\frac{dp_i}{dt} = -\frac{\delta U}{\delta q_i}$
Leap-frog algorithm
$ p_i(t + \frac{\epsilon}{2}) = p_i(t) - \frac{\epsilon}{2} \frac{\delta U}{\delta q_i} U(q(t))$
$ q_i(t + \epsilon) = q_i(t) + \frac{\epsilon}{m}p_i(t+\frac{\epsilon}{2})$
$ p_i(t + \epsilon) = p_i(t + \frac{\epsilon}{2}) - \frac{\epsilon}{2} \frac{\delta U}{\delta q_i}(q(t+\epsilon))$
$\epsilon$ -- step size
Can you write Python code for it?

def HMC_one_step(U, current_q, Eps, L, m=1):
    """
    One step of the Hamiltonian Monte Carlo.

    Parameters
    ----------
    U : callable
        A function that takes a single argument, the position.
    q : array-like
        Current position.
    Eps : float
        The step size, epsilon.
    L : int
        Number of leapfrog stpes.
    m : float
        Mass of the particle.

    Returns
    -------
    q_out : array
        Path from ``q`` to the proposed position.
    """

    q = copy.copy(current_q)
    Nq = len(q)
    p = multivariate_normal.rvs([0. for i in xrange(Nq)])
    current_p = copy.copy(p)

    out = {}

    out['p'] = np.zeros((L, Nq))
    out['p'][0,:] = copy.copy(p)
    out['q'] = np.zeros((L, Nq))
    out['q'][0,:] = copy.copy(q)

    for i in xrange(1, L):
        p -= Eps*derivative(U, q, 0.01)/2.
        q += (Eps/m)*p
        out['q'][i, :] = copy.copy(q)
        p -= Eps*derivative(U, q, 0.01)/2.
        out['p'][i, :] = copy.copy(p)

    current_U = U(current_q)
    current_K = (current_p**2).sum()/2.
    proposed_U = U(q)
    proposed_K = (p**2).sum()/2.

    if uniform.rvs() < exp(current_U - proposed_U + current_K - proposed_K):
        out['value'] = q
    else:
        out['value'] = current_q

    return out

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)
U = lambda x: -1.*np.log(dtarget(x))
chain = HMC_one_step(U, np.array([4., 10.]), Eps=0.2, L=10, m=2)['q']
plt.plot(chain[:, 0], chain[:, 1], 'ro')
plt.plot(chain[:, 0], chain[:, 1], 'r-')
plt.plot(chain[0, 0], chain[0,1], 'bo')
plt.xlim(0, 6)
plt.ylim(7, 13)
plt.xlabel('x1')
plt.ylabel('x2')
plt.show()

def HMC(dtarget, start, Eps=0.2, L=10, m=2, N=1000, num_chains=4):
    """
    Perform an HMC simulation.

    Parameters
    ----------
    dtarget : callable
        Target PDF.

    """

    # Invert the target PDF into a concave surface.
    neg_log_dtarget = lambda x: -1.*np.log(dtarget(x))

    # If only one starting position is provided, use it for all...


I am working on the file "Hamiltonian MCMC (HMC).ipynb".
The first task is:
Tuning parameters: step size, number of steps, and "mass"
HMC does not work discrete parameters. STAN is all HMC.
Gelman metric still applies -- we just have a better way of proposing values.
Can you write Python code for it?

def Gelman(chains):
    if len(chains.shape) == 3:
        N_p = chains.shape[2]
    else:
        N_p = 1
    generate = lambda ptn: np.array([np.array([np.array([ptn(p, i, c)
                                                         for p in xrange(N_p)
                                               for i in xrange(chains.shape[1])])
                                     for c in xrange(chains.shape[0])])])
    params = generate(lambda p, i, c: 'x{0}'.format(p))
    iters = generate(lambda p, i, c: i)
    labels = generate(lambda p, i, c: c)

    data = zip(chains.flat, params.flat, iters.flat, labels.flat)
    dataframe = pd.DataFrame(data, columns=('Value', 'Parameter', 'Iteration', 'Chain'))

    xbar = dataframe.groupby('Parameter').Value.mean()
    m = chains.shape[0]
    xbar_i = dataframe.groupby(('Parameter', 'Chain')).Value.mean()
    s2_i = dataframe.groupby(('Parameter', 'Chain')).Value.var()
    n = dataframe.groupby(('Parameter', 'Chain')).Value.count().mean()

    W = s2_i.mean()
    B = (n/(m-1.)) * ((xbar_i - xbar)**2).sum()
    sigma2_hat = W*(n-1.)/n + B/n
    R_hat = np.sqrt(sigma2_hat/W)
    n_eff = m*n*sigma2_hat/B    # I missed what this was for.

    return R_hat, n_eff

chains = HMC(dtarget, array([4., 10.]), Eps=0.2, L=5, N=1000)

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)
plt.plot(chains[0][:, 0], chains[0][:, 1], alpha=0.5)
plt.plot(chains[1][:, 0], chains[1][:, 1], alpha=0.5)
plt.plot(chains[2][:, 0], chains[2][:, 1], alpha=0.5)
plt.plot(chains[3][:, 0], chains[3][:, 1], alpha=0.5)
plt.xlim(0, 6)
plt.ylim(7, 13)
plt.show()

plt.subplot(211)
for i in xrange(chains.shape[0]):
    plt.plot(chains[i,:,0])
plt.ylabel('x1')

plt.subplot(212)
for i in xrange(chains.shape[0]):
    plt.plot(chains[i,:,1])
plt.ylabel('x2')

Gelman(chains)


I am working on the file "Hamiltonian MCMC (HMC).ipynb".
The first task is:
Use the (local) shape of the distribution to make smarter proposals.
Hamiltonian: quantity that is conserved regardless of position in space.
Metaphor: hockey puck sliding on a (non-flat) surface. Want to be able to describe the state of the puck. The state has two quantities:

Current position, $q$
Momentum, $p$

Hamiltonian: $H(q, p) = U(q) + K(p)$

$U(q)$ -- potential energy
$K(p)$ -- kinetic energy
Can you write Python code for it?

dtarget = lambda x: multivariate_normal.pdf(x, mean=(3, 10), cov=[[1, 0], [0, 1]])
x1 = np.linspace(-6, 12, 101)
x2 = np.linspace(-11, 31, 101)
X, Y = np.meshgrid(x1, x2)
Z = np.array(map(dtarget, zip(X.flat, Y.flat))).reshape(101, 101)

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)
plt.xlim(0, 6)
plt.ylim(7, 13)
plt.show()


I am working on the file "HamiltonianDynamics.ipynb".
The first task is:
Euler's method
Position Evolution
\begin{eqnarray}
\frac{x_i(t + \epsilon) - x_i(t)}{\epsilon} & = & \frac{d x_i}{d t}(t) = \frac{\partial K}{\partial p_i}(p_i(t))  \
x_i(t + \epsilon)  & = & x_i(t) + \epsilon \frac{\partial K}{\partial p_i}(p_i(t))
\end{eqnarray}
Momentum Evolution
\begin{eqnarray}
\frac{p_i(t + \epsilon) - p_i(t)}{\epsilon} & = & \frac{d p_i}{d t}(t) = - \frac{\partial U}{\partial x_i}(x_i(t))  \
p_i(t + \epsilon) & = & p_i(t) - \epsilon \frac{\partial U}{\partial x_i}(x_i(t))  \
\end{eqnarray}
Can you write Python code for it?

epsilon = 0.05
T = 1000
z_euler = np.mat(np.zeros((2,T)))

def dHdx(x):
    return x

def dHdp(p):
    return p

z_euler[:,0] = np.mat('[2.4;0]')
for i in range(1,T):
    #z_euler[:,i] = z_euler[:,i-1] + epsilon*dH(z_euler[:,i-1])

    z_euler[0,i] = z_euler[0,i-1] + epsilon*dHdp(z_euler[1,i-1])
    z_euler[1,i] = z_euler[1,i-1] - epsilon*dHdx(z_euler[0,i-1])

plt.plot(z_euler[0,:], z_euler[1,:],'.-r')
ax = plt.gcf().gca()
ax.set_aspect('equal')
plt.show()


I am working on the file "HamiltonianDynamics.ipynb".
The first task is:
Modified Euler
Use the intermediate solution immediately
Can you write Python code for it?

epsilon = 0.01
T = 600
z_euler2 = np.mat(np.zeros((2,T)))

def dHdx(x):
    return x

def dHdp(p):
    return p

z_euler2[:,0] = np.mat('[2.4;0]')
for i in range(1,T):
    z_euler2[0,i] = z_euler2[0,i-1] + epsilon*dHdp(z_euler2[1,i-1])
    z_euler2[1,i] = z_euler2[1,i-1] - epsilon*dHdx(z_euler2[0,i])

plt.plot(z_euler2[0,:], z_euler2[1,:],'.-r')
ax = plt.gcf().gca()
ax.set_aspect('equal')
plt.show()


I am working on the file "HandCodedNN.ipynb".
The first task is:
Feedforward
sigmoid
This is one of the activation functions. It takes the cumulative input to the layer, the matrix Z, as the input. Upon application of the sigmoid function, the output matrix H is calculated. Also, Z is stored as the variable sigmoid_memory since it will be later used in backpropagation.You use np.exp() here in the following way. The exponential gets applied to all the elements of Z.
Can you write Python code for it?

def sigmoid(Z):

    # Z is numpy array of shape (n, m) where n is number of neurons in the layer and m is the number of samples
    # sigmoid_memory is stored as it is used later on in backpropagation

    H = 1/(1+np.exp(-Z))
    sigmoid_memory = Z

    return H, sigmoid_memory

Z = np.arange(8).reshape(4,2)
print(Z)
print ("sigmoid(Z) = " + str(sigmoid(Z)))


I am working on the file "HandCodedNN.ipynb".
The first task is:
Having defined the bits and pieces of the feedforward and the backpropagation, let's now combine all that to form a model. The list dimensions has the number of neurons in each layer specified in it. For a neural network with 1 hidden layer with 45 neurons, you would specify the dimensions as follows:
Can you write Python code for it?

dimensions = [784, 45, 10] #  three-layer model


I am working on the file "HandCodedNN.ipynb".
The first task is:
Now, as discussed earlier in the lectures, the target variable is converted to a one hot matrix. We use the function <i> one_hot </i> to convert the target dataset to one hot encoding.
Can you write Python code for it?

def one_hot(j):
    # input is the target dataset of shape (m,) where m is the number of data points
    # returns a 2 dimensional array of shape (10, m) where each target value is converted to a one hot encoding
    # Look at the next block of code for a better understanding of one hot encoding
    n = j.shape[0]
    new_array = np.zeros((10, n))
    index = 0
    for res in j:
        new_array[res][index] = 1.0
        index = index + 1
    return new_array

data = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
print(data.shape)
one_hot(data)


I am working on the file "HandCodedNN.ipynb".
The first task is:
Now, let's call the function L_layer_model on the dataset we have created.This will take 10-20 mins to run.
Can you write Python code for it?

parameters = L_layer_model(train_set_x_new, train_set_y_new, dimensions, num_iterations = 2000, print_loss = True)

def predict(X, y, parameters):

    # Performs forward propogation using the trained parameters and calculates the accuracy

    m = X.shape[1]
    n = len(parameters) // 2 # number of layers in the neural network

    # Forward propagation
    probas, caches = L_layer_forward(X, parameters)

    p = np.argmax(probas, axis = 0)
    act = np.argmax(y, axis = 0)

    print("Accuracy: "  + str(np.sum((p == act)/m)))

    return p


I am working on the file "HandCodedNN.ipynb".
The first task is:
Since, it'll take a lot of time to train the model on 50,000 data points, we take a subset of 5,000 images.
Can you write Python code for it?

train_set_x_new = train_set_x[:,0:5000]
train_set_y_new = train_set_y[:,0:5000]
train_set_x_new.shape


I am working on the file "HandCodedNN.ipynb".
The first task is:
You should get:

    array([[1.        , 1.        , 1.        , 1.        , 1.        ],

      [0.99908895, 0.99330715, 0.99999969, 1.        , 0.99987661],

       [0.73105858, 0.5       , 0.99330715, 0.9999546 , 0.88079708]])
L_layer_forward
L_layer_forward performs one forward pass through the whole network for all the training samples (note that we are feeding all training examples in one single batch). Use the layer_forward you have created above here to perform the feedforward for layers 1 to 'L-1' in the for loop with the activation relu. The last layer having a different activation softmax is calculated outside the loop. Notice that the memory is appended to memories for all the layers. These will be used in the backward order during backpropagation.
Can you write Python code for it?

#Graded

def L_layer_forward(X, parameters):

    # X is input data of shape (input size, number of examples)
    # parameters is output of initialize_parameters()

    # HL is the last layer's post-activation value
    # memories is the list of memory containing (for a relu activation, for example):
    # - every memory of relu forward (there are L-1 of them, indexed from 1 to L-1),
    # - the memory of softmax forward (there is one, indexed L)

    memories = []
    H = X
    L = len(parameters) // 2                  # number of layers in the neural network

    # Implement relu layer (L-1) times as the Lth layer is the softmax layer
    for l in range(1, L):
        H_prev = H
        W = parameters['W' + str(l)]
        b = parameters['b' + str(l)]
        H, memory =  layer_forward(H_prev, W, b, activation="relu")

        memories.append(memory)

    # Implement the final softmax layer
    # HL here is the final prediction P as specified in the lectures
    H_prev = H
    W = parameters['W' + str(L)]
    b = parameters['b' + str(L)]
    HL, memory = layer_forward(H_prev, W, b, activation="softmax")

    memories.append(memory)

    assert(HL.shape == (10, X.shape[1]))

    return HL, memories

# verify
# X is (784, 10)
# parameters is a dict
# HL should be (10, 10)
x_sample = train_set_x[:, 10:20]
print(x_sample.shape)
HL = L_layer_forward(x_sample, parameters=parameters)[0]
print(HL[:, :5])


def sigmoid_backward(dH, sigmoid_memory):

    # Implement the backpropagation of a sigmoid function
    # dH is gradient of the sigmoid activated activation of shape same as H or Z in the same layer
    # sigmoid_memory is the memory stored in the sigmoid(Z) calculation

    Z = sigmoid_memory

    H = 1/(1+np.exp(-Z))
    dZ = dH * H * (1-H)

    assert (dZ.shape == Z.shape)

    return dZ


I am working on the file "HandCodedNN.ipynb".
The first task is:
You should get:

dH_prev is

 [[5.6417525  0.66855959 6.86974666 5.46611139 4.92177244]

 [2.17997451 0.12963116 2.74831239 2.17661196 2.10183901]]

dW is

 [[1.67565336 1.56891359]

 [1.39137819 1.4143854 ]

 [1.3597389  1.43013369]]

db is

 [[0.37345476]

 [0.34414727]

 [0.29074635]]

L_layer_backward
L_layer_backward performs backpropagation for the whole network. Recall that the backpropagation for the last layer, i.e. the softmax layer, is different from the rest, hence it is outside the reversed for loop. You need to use the function layer_backward here in the loop with the activation function as relu.
Can you write Python code for it?

#Graded

def L_layer_backward(HL, Y, memories):

    # Takes the predicted value HL and the true target value Y and the
    # memories calculated by L_layer_forward as input

    # returns the gradients calulated for all the layers as a dict

    gradients = {}
    L = len(memories) # the number of layers
    m = HL.shape[1]
    Y = Y.reshape(HL.shape) # after this line, Y is the same shape as AL

    # Perform the backprop for the last layer that is the softmax layer
    current_memory = memories[-1]
    linear_memory, activation_memory = current_memory
    dZ = HL - Y
    H_prev, W, b = linear_memory
    # Use the expressions you have used in 'layer_backward'
    gradients["dH" + str(L-1)] = #write your code here
    gradients["dW" + str(L)] = #write your code here, use (1./m) and not (1/m)
    gradients["db" + str(L)] = #write your code here, use (1./m) and not (1/m)

    # Perform the backpropagation l-1 times
    for l in reversed(range(L-1)):
        # Lth layer gradients: "gradients["dH" + str(l + 1)] ", gradients["dW" + str(l + 2)] , gradients["db" + str(l + 2)]
        current_memory = memories[l]

        dH_prev_temp, dW_temp, db_temp = #write your code here
        gradients["dH" + str(l)] = #write your code here
        gradients["dW" + str(l + 1)] = #write your code here
        gradients["db" + str(l + 1)] = #write your code here

    return gradients

# verify
# X is (784, 10)
# parameters is a dict
# HL should be (10, 10)
x_sample = train_set_x[:, 10:20]
y_sample = train_set_y[:, 10:20]

HL, memories = L_layer_forward(x_sample, parameters=parameters)
gradients  = L_layer_backward(HL, y_sample, memories)
print('dW3 is \n', gradients['dW3'])
print('db3 is \n', gradients['db3'])
print('dW2 is \n', gradients['dW2'])
print('db2 is \n', gradients['db2'])


#Graded

def update_parameters(parameters, gradients, learning_rate):

    # parameters is the python dictionary containing the parameters W and b for all the layers
    # gradients is the python dictionary containing your gradients, output of L_model_backward

    # returns updated weights after applying the gradient descent update

    L = len(parameters) // 2 # number of layers in the neural network

    for l in range(L):
        parameters["W" + str(l+1)] = #write your code here
        parameters["b" + str(l+1)] = #write your code here

    return parameters


I am working on the file "HandCodedNN.ipynb".
The first task is:
layer_backward
layer_backward is a complimentary function of layer_forward. Like layer_forward calculates H using W, H_prev and b, layer_backward uses dH to calculate dW, dH_prev and db. You have already studied the formulae in backpropogation. To calculate dZ, use the sigmoid_backward and relu_backward function. You might need to use np.dot(), np.sum() for the rest. Remember to choose the axis correctly in db.
Can you write Python code for it?

#Graded

def layer_backward(dH, memory, activation = 'relu'):

    # takes dH and the memory calculated in layer_forward and activation as input to calculate the dH_prev, dW, db
    # performs the backprop depending upon the activation function

    linear_memory, activation_memory = memory

    if activation == "relu":
        dZ = #write your code here
        H_prev, W, b = linear_memory
        m = H_prev.shape[1]
        dW = #write your code here, use (1./m) and not (1/m)
        db = #write your code here, use (1./m) and not (1/m)
        dH_prev = #write your code here

    elif activation == "sigmoid":
        dZ = #write your code here
        H_prev, W, b = linear_memory
        m = H_prev.shape[1]
        dW = #write your code here, use (1./m) and not (1/m)
        db = #write your code here, use (1./m) and not (1/m)
        dH_prev = #write your code here

    return dH_prev, dW, db

# verify
# l-1 has two neurons, l has three, m = 5
# H_prev is (l-1, m)
# W is (l, l-1)
# b is (l, 1)
# H should be (l, m)
H_prev = np.array([[1,0, 5, 10, 2], [2, 5, 3, 10, 2]])
W_sample = np.array([[10, 5], [2, 0], [1, 0]])
b_sample = np.array([10, 5, 0]).reshape((3, 1))

H, memory = layer_forward(H_prev, W_sample, b_sample, activation="relu")
np.random.seed(2)
dH = np.random.rand(3,5)
dH_prev, dW, db = layer_backward(dH, memory, activation = 'relu')
print('dH_prev is \n' , dH_prev)
print('dW is \n' ,dW)
print('db is \n', db)