I am working on the file "HW3_sd3462.ipynb".
The first task is:
2. Explain in words your reasoning for choosing the above ranges.
<span style="color:Blue">Answer </span>
The model we developed suffers from overfitting as demonstrated by the radical difference in accuracy when run on train and test data set.
1. We choose a higher min_samples_split as the default value of 2 is too low and clearly causes our model to overfit on the training data. Since the size of the data is too large, we select larger values.
2. We choose a higher min_samples_leaf as the default value of 1 is too low. However, we don't want the value to be too high as it can result in the model to demonstrate low variance and high bias.

3. For each combination of values in 3.1 (there should be 100), build a new classifier and check the classifier's accuracy on the test data. Plot the test set accuracy for these options. Use the values of min_samples_split as the x-axis and generate a new series (line) for each of min_samples_leaf.
(5 Points)
Can you write Python code for it?

def DtreeIter(train_features,train_target,test_features,test_target,samplesplit,sampleleaf):
    treeOpt=DecisionTreeClassifier(criterion="entropy",min_samples_split=samplesplit,min_samples_leaf=sampleleaf)
    treeOpt=treeOpt.fit(train_features,train_target)
    result_Opt=treeOpt.predict(test_features)
    return accuracy_score(test_target,result_Opt)

result_optimise=dict()
for values in splits:
    result_optimise[values]=list()

for values in splits:
    for nodes in leafnodes:
        result_optimise[values].append([DtreeIter(features_train,target_train,features_test,target_test,values,nodes)])

#To find out best parameters
optimal_split=max(result_optimise, key=lambda x: result_optimise[x][1])
optimal_accuracy=max(result_optimise[optimal_split])
optimal_leaf=leafnodes[list(result_optimise[optimal_split]).index(optimal_accuracy)]
print("Optimal 'Sample Split Size' is %d and 'Optimal Leaf Samples' are %d. Best accuracy is %.2f%%" %(optimal_split,optimal_leaf,optimal_accuracy[0]*100))

plt.figure(figsize=(10,5))
plt.plot(splits,result_optimise[leafnodes[0]],'b',label='Leaf={}'.format(leafnodes[0]))
plt.plot(splits,result_optimise[leafnodes[1]],'r',label='Leaf={}'.format(leafnodes[1]))
plt.plot(splits,result_optimise[leafnodes[2]],'y',label='Leaf={}'.format(leafnodes[2]))
plt.plot(splits,result_optimise[leafnodes[3]],'g',label='Leaf={}'.format(leafnodes[3]))
plt.plot(splits,result_optimise[leafnodes[4]],'c',label='Leaf={}'.format(leafnodes[4]))
plt.plot(splits,result_optimise[leafnodes[5]],'m',label='Leaf={}'.format(leafnodes[5]))
plt.plot(splits,result_optimise[leafnodes[6]],'k',label='Leaf={}'.format(leafnodes[6]))
plt.plot(splits,result_optimise[leafnodes[7]],'b',label='Leaf={}'.format(leafnodes[7]))
plt.plot(splits,result_optimise[leafnodes[8]],'r',label='Leaf={}'.format(leafnodes[8]))
plt.plot(splits,result_optimise[leafnodes[9]],'y',label='Leaf={}'.format(leafnodes[9]))
plt.legend(loc=4)
plt.xlabel('Min Sample Splits')
plt.ylabel('Accuracy')
plt.title('Classifier Accuracy')
plt.show()


I am working on the file "HW3_sd3462.ipynb".
The first task is:
2. If we had to, how would we prove to ourselves or a colleague that our data was indeed randomly sampled on X? And by prove, I mean empirically, not just showing this person our code. Don't actually do the work, just describe in your own words a test you could here. Hint: think about this in terms of selection bias and use notes from our 2nd lecture.
(1 Point)
Answer
We use the random number generator provided by an external library called NumPy. By using this library we generate unique random numbers which are free from selection bias. We can safely assume that selection bias is non-existent
3. Now build and train a decision tree classifier using DecisionTreeClassifier() (manual page) on train_df to predict the "churndep" target variable. Make sure to use criterion='entropy' when instantiating an instance of DecisionTreeClassifier(). For all other settings you should use all of the default options.
(1 Point)
Can you write Python code for it?


I am working on the file "HW3_sd3462.ipynb".
The first task is:
2. Plot the autocorrelation function (ACF) and partial autocorrelation function (PCF) of the cases timeseries. (1 point)
Can you write Python code for it?

from pandas.plotting import autocorrelation_plot
from statsmodels.tsa.stattools import pacf,acf

plt.figure(1)
plt.figure(figsize=(10,5))
plt.title("Autocorrelation Plot")
autocorrelation_plot(fever["YFCASES"])
plt.figure(2)
plt.figure(figsize=(10,5))
plt.title("Partial Autocorrelation Plot")
plt.plot(pacf(fever["YFCASES"]))
plt.show()


I am working on the file "HW3_sd3462.ipynb".
The first task is:
3. Describe what the plots indicate (in terms of autocorrelation and autoregressive parameter (p) and moving average (q)). 2 points.
Some rules of thumb to recall:

Rule 1: If the ACF shows exponential decay, the PACF has a spike at lag 1, and no correlation for other lags, then use one autoregressive (p)parameter

Rule 2: If the ACF shows a sine-wave shape pattern or a set of exponential decays, the PACF has spikes at lags 1 and 2, and no correlation for other lags, the use two autoregressive (p) parameters.

Rule 3: If the ACF has a spike at lag 1, no correlation for other lags, and the PACF damps out exponentially, then use one moving average (q) parameter.

Rule 4: If the ACF has spikes at lags 1 and 2, no correlation for other lags, and the PACF has a sine-wave shape pattern or a set of exponential decays, then use two moving average (q) parameter.

Rule 5: If the ACF shows exponential decay starting at lag 1, and the PACF shows exponential decay starting at lag 1, then use one autoregressive (p) and one moving average (q) parameter.

Answer
1. The autocorrelation plot shows a sine-wave pattern meaning that the data exhibits strong seasonality.
2. The partial autocorrelation plot shows sharp spikes at lag of 1 and 2.
We use "Rule 2" and select the autocorrelation parameter as 2 i.e. p=2 and q=0

4. Another approach to assessing the presence of autocorrelation is by using the  Durbin-Waton (DW) statistic. The value of the DW statistic is close to 2 if the errors are uncorrelated. What is DW for our data, and does this match what you observed from the ACF and PCF plots? (1 point)
Can you write Python code for it?

# We run Durbin Watson test on the residuls that we obtain from the OLS.
from statsmodels.regression.linear_model import OLS
from statsmodels.stats.stattools import durbin_watson

ols_residuals=OLS(fever["YFCASES"],np.ones(len(fever["YFCASES"]))).fit()
durbin_watson(ols_residuals.resid)


I am working on the file "HW3_sd3462.ipynb".
The first task is:
4. Using the resulting model from 2.3, show a bar plot of feature names and their feature importance (hint: check the attributes of the DecisionTreeClassifier() object directly in IPython or check the manual!).
(3 Points)
Can you write Python code for it?

featurelength=np.arange(len(list(features_train)))
names=list(features_train)
importances=pd.DataFrame({"Features":list(features_train),"Importance":trained.feature_importances_})
importances.sort_values(by='Importance',ascending=False,inplace=True)

plt.figure(figsize=(10,5))
plt.title("Feature Importance")

plt.bar(featurelength,importances["Importance"],align="center",color="blue")
plt.xticks(featurelength,importances["Features"],rotation="60")
plt.xlabel('Features')
plt.ylabel('Importance')

## We can alternatively use a vertical bar plot to represent information ##
'''
plt.barh(featurelength,importances["Importance"],align="center",color="blue")
plt.yticks(featurelength,importances["Features"])
plt.ylabel('Features')
plt.xlabel('Importance')
'''

plt.show()


I am working on the file "HW3_sd3462.ipynb".
The first task is:
6. Calculate the residuals, and test the null hypothesis that the residuals come from a normal distribution, and construct a qq-plot. Do the results of the hypothesis test and qq-plot align? (1 point)
Can you write Python code for it?

print(stats.normaltest(arma_result.resid))
figureP3 = plt.figure(figsize=(10,5))
ax = figureP3.add_subplot(1,1,1)
figP3 = qqplot(arma_result.resid, line='q', ax=ax, fit=True)
plt.show()


I am working on the file "HW3_sd3462.ipynb".
The first task is:
<span style="color:Blue">Answer </span>
* We observe that the Durbin-Watson statistic is approximately 0.35 which means that there is strong positive relation between the previous and the current value of reported cases of yellow fever. The DW statistic matches the output observed from the autocorrelation plot which indicated strong correlation with lagged values.
* E.g.: If the number of yellow fever cases increase in the current year, they will increase further in the next year and vice versa.
* We can use the ARMA model to remove this autocorrelation between lagged values.

5. Removing serial dependency by modeling a simple ARMA process with p and q as derived above. Take a look at what the resulting process looks like (plot) (1 point)
Can you write Python code for it?

from statsmodels import tsa
import statsmodels.api as sm

indexedFever=fever.set_index("YEAR")
cases=indexedFever.astype(float)
arma_result = sm.tsa.ARMA(cases,(2,0)).fit()
cases['forecast'] = arma_result.predict(start = 260 , end= 309, dynamic= True)
cases[['YFCASES', 'forecast']].plot(figsize=(10, 5))
plt.show()


I am working on the file "HW3_sd3462.ipynb".
The first task is:
The top 3 features are revenue, eqpdays and outcalls.

Revenue and Eqpdays are negatively correlated. This means that as revenue slightly increases, eqpdays decrease. We can infer that once the customer stop using the current equipment and starts using the new, company provided equipment, the company's revenues will slightly increase. The relationship is weak in nature.
Revenue and outcalls are positively correlated with moderate strength. This means that as the number of outgoing calls of the customer increase, the revenue of the company increases.
Eqpdays and outcalls are negatively correlated and the relationship is weak in nature. We can infer that as outgoing calls of customers increase, there is a slight decrease in the eqpdays i.e. the number of days the customer owns their current equipment. This means that customers who make more outgoing calls are somewhat likely to buy new equipment from the company than use their current equipment.

The top 3 drivers of churn are monthly revenue (revenue), the number of days the customer has had his/her current equipment (eqpdays) and the mean number of outbound voice calls (outcalls).

As our revenue per customer increases i.e. the more they pay, the less likely it is that they will churn. This can be attributed to customers finding satisfaction with services provided by us and voluntarily using more services provided by us. Although, the impact of revenue on churn is weak and almost insignificant.
The longer our customers use their current equipment, the more likely it is that they will churn. We need to ensure that the customer purchases equipment(s) from us in order to lock-in the customer leading to greater customer stickiness.
The more our customers make outgoing calls, the less likely it is that they will churn. More outgoing calls indicate successful and willing usage of our services. Satisfied customers are less likely to churn. Like revenue, this relation is also weak and almost insignificant.

6. Using the classifier built in 2.3, try predicting...
Can you write Python code for it?

# Splitting test dataset into target and features
features_test=test_churn.loc[:,'revenue':'retcalls']
target_test=test_churn.loc[:,'churndep']

#  Predicting target for train and test dataset
results_test=trained.predict(features_test)
results_train=trained.predict(features_train)

test_accuracy=accuracy_score(target_test,results_test)
train_accuracy=accuracy_score(target_train,results_train)
print("Accuracy for the test dataset is %.3f%% and accuracy for the training dataset is %.3f%%" %(test_accuracy*100,train_accuracy*100))


I am working on the file "HW3_solutions.ipynb".
The first task is:
2.3 List the 5 critics with the most reviews, along with the publication they write for
Can you write Python code for it?

#Your code here
#note: there are a few valid ways to deal with critics that write for several publications.

counts = critics.groupby(['critic', 'publication']).critic.count()
counts.sort()
counts[-1:-6:-1]


I am working on the file "HW3_solutions.ipynb".
The first task is:
2.5
Using the original movies dataframe, plot the rotten tomatoes Top Critics Rating as a function of year. Overplot the average for each year, ignoring the score=0 examples (some of these are missing data). Comment on the result -- is there a trend? What do you think it means?
Can you write Python code for it?

#Your code here

data = movies[['year', 'rtTopCriticsRating']]
data = data.convert_objects(convert_numeric=True)
data = data[(data['rtTopCriticsRating'] > 0)]
means = data.groupby('year').mean()
print means
plt.plot(data['year'], data['rtTopCriticsRating'], 'o', mec='none', alpha=.2, label='Data')
plt.plot(means.index, means['rtTopCriticsRating'], '-', label='Yearly Average')
plt.legend(loc='lower left', frameon=False)
plt.xlabel("Year")
plt.ylabel("Average Score")
remove_border()


I am working on the file "HW3_solutions.ipynb".
The first task is:
3.1
Using the critics dataframe, compute a pair of numerical X, Y arrays where:

X is a (nreview, nwords) array. Each row corresponds to a bag-of-words representation for a single review. This will be the input to your model.
Y is a nreview-element 1/0 array, encoding whether a review is Fresh (1) or Rotten (0). This is the desired output from your model.

Make sure to remove items with no review text
Can you write Python code for it?

#hint: Consult the scikit-learn documentation to
#      learn about what these classes do do
from sklearn.cross_validation import train_test_split
from sklearn.naive_bayes import MultinomialNB

"""
Function
--------
make_xy

Build a bag-of-words training set for the review data

Parameters
-----------
critics : Pandas DataFrame
    The review data from above

vectorizer : CountVectorizer object (optional)
    A CountVectorizer object to use. If None,
    then create and fit a new CountVectorizer.
    Otherwise, re-fit the provided CountVectorizer
    using the critics data

Returns
-------
X : numpy array (dims: nreview, nwords)
    Bag-of-words representation for each review.
Y : numpy array (dims: nreview)
    1/0 array. 1 = fresh review, 0 = rotten review

Examples
--------
X, Y = make_xy(critics)
"""
def make_xy(critics, vectorizer=None):
    #Your code here
    if vectorizer is None:
        vectorizer = CountVectorizer()
    X = vectorizer.fit_transform(critics.quote)
    X = X.tocsc()  # some versions of sklearn return COO format
    Y = (critics.fresh == 'fresh').values.astype(np.int)
    return X, Y

X, Y = make_xy(critics)


I am working on the file "HW3_solutions.ipynb".
The first task is:
3.2 Next, randomly split the data into two groups: a
training set and a validation set.
Use the training set to train a MultinomialNB classifier,
and print the accuracy of this model on the validation set
Hint
You can use train_test_split to split up the training data
Can you write Python code for it?

#Your code here
xtrain, xtest, ytrain, ytest = train_test_split(X, Y)
clf = MultinomialNB().fit(xtrain, ytrain)

print "Accuracy: %0.2f%%" % (100 * clf.score(xtest, ytest))


I am working on the file "HW3_solutions.ipynb".
The first task is:
3.3:
We say a model is overfit if it performs better on the training data than on the test data. Is this model overfit? If so, how much more accurate is the model on the training data compared to the test data?
Can you write Python code for it?

# Your code here. Print the accuracy on the test and training dataset
training_accuracy = clf.score(xtrain, ytrain)
test_accuracy = clf.score(xtest, ytest)

print "Accuracy on training data: %0.2f" % (training_accuracy)
print "Accuracy on test data:     %0.2f" % (test_accuracy)


I am working on the file "HW3_solutions.ipynb".
The first task is:
3.5 We might say a model is over-confident if the freshness fraction is usually closer to 0.5 than expected (that is, there is more uncertainty than the model predicted). Likewise, a model is under-confident if the probabilities are usually further away from 0.5. Is this model generally over- or under-confident?
Answer This model is over-confident. For a properly calibrated model, we would expect ~10% of the P(Fresh)~0.1 reviews to actually be fresh. However, the actual freshness rate is closer to 30%. Likewise, for reviews where P(Fresh) ~0.9, the actuall freshness fraction is closer to 0.7. In other words, there is more uncertainty in the outcome than implied by the model.
Cross Validation
Our classifier has a few free parameters. The two most important are:

The min_df keyword in CountVectorizer, which will ignore words which appear in fewer than min_df fraction of reviews. Words that appear only once or twice can lead to overfitting, since words which occur only a few times might correlate very well with Fresh/Rotten reviews by chance in the training dataset.

The alpha keyword in the Bayesian classifier is a "smoothing parameter" -- increasing the value decreases the sensitivity to any single feature, and tends to pull prediction probabilities closer to 50%.

As discussed in lecture and HW2, a common technique for choosing appropriate values for these parameters is cross-validation. Let's choose good parameters by maximizing the cross-validated log-likelihood.
3.6 Using clf.predict_logproba, write a function that computes the log-likelihood of a dataset
Can you write Python code for it?

"""
Function
--------
log_likelihood

Compute the log likelihood of a dataset according to
a bayesian classifier.
The Log Likelihood is defined by

L = Sum_fresh(logP(fresh)) + Sum_rotten(logP(rotten))

Where Sum_fresh indicates a sum over all fresh reviews,
and Sum_rotten indicates a sum over rotten reviews

Parameters
----------
clf : Bayesian classifier
x : (nexample, nfeature) array
    The input data
y : (nexample) integer array
    Whether each review is Fresh
"""
#your code here

def log_likelihood(clf, x, y):
    prob = clf.predict_log_proba(x)
    rotten = y == 0
    fresh = ~rotten
    return prob[rotten, 0].sum() + prob[fresh, 1].sum()

log_likelihood(clf,xtest,ytest)


I am working on the file "HW3_solutions.ipynb".
The first task is:
3.7
Fill in the remaining code in this block, to loop over many values of alpha and min_df to determine
which settings are "best" in the sense of maximizing the cross-validated log-likelihood
Can you write Python code for it?

#the grid of parameters to search over
alphas = [0, .1, 1, 5, 10, 50]
min_dfs = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1]

#Find the best value for alpha and min_df, and the best classifier
best_alpha = None
best_min_df = None
max_loglike = -np.inf

for alpha in alphas:
    for min_df in min_dfs:
        vectorizer = CountVectorizer(min_df = min_df)
        X, Y = make_xy(critics, vectorizer)

        #your code here
        clf = MultinomialNB(alpha=alpha)
        loglike = cv_score(clf, X, Y, log_likelihood)
        print "Loglike: %.4f"%loglike, "alpha: %.6f"%alpha, "min_df: %.6f"%min_df
        if loglike > max_loglike:
            max_loglike = loglike
            best_alpha, best_min_df = alpha, min_df

print "alpha: %f" % best_alpha
print "min_df: %f" % best_min_df


I am working on the file "HW3_solutions.ipynb".
The first task is:
3.8 Now that you've determined values for alpha and min_df that optimize the cross-validated log-likelihood, repeat the steps in 3.1, 3.2, and 3.4 to train a final classifier with these parameters, re-evaluate the accuracy, and draw a new calibration plot.
Can you write Python code for it?

#Your code here

vectorizer = CountVectorizer(min_df=best_min_df)
X, Y = make_xy(critics, vectorizer)
xtrain, xtest, ytrain, ytest = train_test_split(X, Y)

clf = MultinomialNB(alpha=best_alpha).fit(xtrain, ytrain)

calibration_plot(clf, xtest, ytest)

# Your code here. Print the accuracy on the test and training dataset
training_accuracy = clf.score(xtrain, ytrain)
test_accuracy = clf.score(xtest, ytest)

print "Accuracy on training data: %0.2f" % (training_accuracy)
print "Accuracy on test data:     %0.2f" % (test_accuracy)


I am working on the file "HW3_solutions.ipynb".
The first task is:
3.9 Discuss the various ways in which Cross-Validation has affected the model. Is the new model more or less accurate? Is overfitting better or worse? Is the model more or less calibrated?
Answer The new model is slightly less accurate on the test data (74% vs 77% on the original model). However, it is both better calibrated and less over-fit than before. In other words, while the classification accuracy is slightly worse, the probabilities themselves are more accurate. The model is still slightly over-confident when making low P(Fresh) predictions. However, the calibration plot shows the model is usually within 1 error bar of the expected performance where P(Fresh) >= 0.2. Finally, the new model makes less-conclusive predictions on average -- the histogram in the calibration plot is more uniformly distributed, with fewer predictions clustered around P(Fresh) = 0 or 1.
To think about/play with, but not to hand in: What would happen if you tried this again using a function besides the log-likelihood -- for example, the classification accuracy?
Part 4: Interpretation. What words best predict a fresh or rotten review?
4.1
Using your classifier and the vectorizer.get_feature_names method, determine which words best predict a positive or negative review. Print the 10 words
that best predict a "fresh" review, and the 10 words that best predict a "rotten" review. For each word, what is the model's probability of freshness if the word appears one time?
Hints

Try computing the classification probability for a feature vector which consists of all 0s, except for a single 1. What does this probability refer to?

np.eye generates a matrix where the ith row is all 0s, except for the ith column which is 1.
Can you write Python code for it?

## Your code here

words = np.array(vectorizer.get_feature_names())

x = np.eye(xtest.shape[1])
probs = clf.predict_log_proba(x)[:, 0]
ind = np.argsort(probs) # returns sorted indices

good_words = words[ind[:10]]
bad_words = words[ind[-10:]]

good_prob = probs[ind[:10]]
bad_prob = probs[ind[-10:]]

print "Good words\t     P(fresh | word)"
for w, p in zip(good_words, good_prob):
    print "%20s" % w, "%0.2f" % (1 - np.exp(p))
print "Bad words\t     P(fresh | word)"
for w, p in zip(bad_words, bad_prob):
    print "%20s" % w, "%0.2f" % (1 - np.exp(p))


I am working on the file "HW3_solutions.ipynb".
The first task is:
4.2
One of the best sources for inspiration when trying to improve a model is to look at examples where the model performs poorly.
Find 5 fresh and rotten reviews where your model performs particularly poorly. Print each review.
Can you write Python code for it?

#Your code here
x, y = make_xy(critics, vectorizer)

prob = clf.predict_proba(x)[:, 0]
predict = clf.predict(x)

bad_rotten = np.argsort(prob[y == 0])[:5]
bad_fresh = np.argsort(prob[y == 1])[-5:]

print "Mis-predicted Rotten quotes"
print '---------------------------'
for row in bad_rotten:
    print critics[y == 0].quote.irow(row)
    print

print "Mis-predicted Fresh quotes"
print '--------------------------'
for row in bad_fresh:
    print critics[y == 1].quote.irow(row)
    print


I am working on the file "HW3_solutions.ipynb".
The first task is:
4.3 What do you notice about these mis-predictions? Naive Bayes classifiers assume that every word affects the probability independently of other words. In what way is this a bad assumption? In your answer, report your classifier's Freshness probability for the review "This movie is not remarkable, touching, or superb in any way".
Can you write Python code for it?

clf.predict_proba(vectorizer.transform(['This movie is not remarkable, touching, or superb in any way']))


I am working on the file "HW3_solutions.ipynb".
The first task is:
Here's a function to estimate the cross-validated value of a scoring function, given a classifier and data
Can you write Python code for it?

from sklearn.cross_validation import KFold

def cv_score(clf, x, y, score_func):
    """
    Uses 5-fold cross validation to estimate a score of a classifier

    Inputs
    ------
    clf : Classifier object
    x : Input feature vector
    y : Input class labels
    score_func : Function like log_likelihood, that takes (clf, x, y) as input,
                 and returns a score

    Returns
    -------
    The average score obtained by randomly splitting (x, y) into training and
    test sets, fitting on the training set, and evaluating score_func on the test set

    Examples
    cv_score(clf, x, y, log_likelihood)
    """
    result = 0
    nfold = 5
    for train, test in KFold(y.size, nfold): # split data into train/test groups, 5 times
        clf.fit(x[train], y[train]) # fit
        result += score_func(clf, x[test], y[test]) # evaluate score function on held-out data
    return result / nfold # average

print cv_score(clf, xtrain, ytrain, log_likelihood)


I am working on the file "HW3_solutions.ipynb".
The first task is:
Interpret these numbers in a few sentences here
Answer Some overfitting seems to be happening here, since the error rate on the test data (23%) is more than twice as large as the error rate on the training data (10%). It's possible (though unlikely) that the accuracy difference is a product of chance, and not a symptom of overfitting. This could be tested with cross-validation, by repeatedly fitting and scoring the classifier on different train/test splits. If the performance on the training data is consistently better than the test data, then overfitting has occured. This is the case here.
3.4: Model Calibration
Bayesian models like the Naive Bayes classifier have the nice property that they compute probabilities of a particular classification -- the predict_proba and predict_log_proba methods of MultinomialNB compute these probabilities.
Being the respectable Bayesian that you are, you should always assess whether these probabilities are calibrated -- that is, whether a prediction made with a confidence of x% is correct approximately x% of the time.
Let's make a plot to assess model calibration. Schematically, we want something like this:
<img src="http://i.imgur.com/ea5B6zr.png" width=500px>
In words, we want to:

Take a collection of examples, and compute the freshness probability for each using clf.predict_proba
Gather examples into bins of similar freshness probability (the diagram shows 5 groups -- you should use something closer to 20)
For each bin, count the number of examples in that bin, and compute the fraction of examples in the bin which are fresh
In the upper plot, graph the expected P(Fresh) (x axis) and observed freshness fraction (Y axis). Estimate the uncertainty in observed freshness fraction $F$ via the equation $\sigma = \sqrt{F (1-F) / N}$
Overplot the line y=x. This is the trend we would expect if the model is calibrated
In the lower plot, show the number of examples in each bin

Hints
The output of clf.predict_proba(X) is a (N example, 2) array. The first column gives the probability...
Can you write Python code for it?

"""
Function
--------
calibration_plot

Builds a plot like the one above, from a classifier and review data

Inputs
-------
clf : Classifier object
    A MultinomialNB classifier
X : (Nexample, Nfeature) array
    The bag-of-words data
Y : (Nexample) integer array
    1 if a review is Fresh
"""
#your code here

def calibration_plot(clf, xtest, ytest):
    prob = clf.predict_proba(xtest)[:, 1]
    outcome = ytest
    data = pd.DataFrame(dict(prob=prob, outcome=outcome))

    #group outcomes into bins of similar probability
    bins = np.linspace(0, 1, 20)
    cuts = pd.cut(prob, bins)
    binwidth = bins[1] - bins[0]

    #freshness ratio and number of examples in each bin
    cal = data.groupby(cuts).outcome.agg(['mean', 'count'])
    cal['pmid'] = (bins[:-1] + bins[1:]) / 2
    cal['sig'] = np.sqrt(cal.pmid * (1 - cal.pmid) / cal['count'])

    #the calibration plot
    ax = plt.subplot2grid((3, 1), (0, 0), rowspan=2)
    p = plt.errorbar(cal.pmid, cal['mean'], cal['sig'])
    plt.plot(cal.pmid, cal.pmid, linestyle='--', lw=1, color='k')
    plt.ylabel("Empirical P(Fresh)")
    remove_border(ax)

    #the distribution of P(fresh)
    ax = plt.subplot2grid((3, 1), (2, 0), sharex=ax)

    plt.bar(left=cal.pmid - binwidth / 2, height=cal['count'],
            width=.95 * (bins[1] - bins[0]),
            fc=p[0].get_color())

    plt.xlabel("Predicted P(Fresh)")
    remove_border()
    plt.ylabel("Number")

calibration_plot(clf, xtest, ytest)


I am working on the file "HW3_solutions.ipynb".
The first task is:
P1.2
Use the function you wrote to retrieve reviews for the first 3,000 movies in the movies dataframe.
Hints

Rotten Tomatoes limits you to 10,000 API requests a day. Be careful about this limit! Test your code on smaller inputs before scaling. You are responsible if you hit the limit the day the assignment is due :)
This will take a while to download. If you don't want to re-run this function every time you restart the notebook, you can save and re-load this data as a CSV file. However, please don't submit this file
Can you write Python code for it?

"""
Function
--------
build_table

Parameters
----------
movies : DataFrame
  The movies data above
rows : int
  The number of rows to extract reviews for

Returns
--------
A dataframe
  The data obtained by repeatedly calling `fetch_reviews` on the first `rows`
  of `movies`, discarding the `None`s,
  and concatenating the results into a single DataFrame
"""
#your code here
def build_table(movies, rows):
    dfs = [fetch_reviews(movies, r) for r in range(rows)]
    dfs = [d for d in dfs if d is not None]
    return pd.concat(dfs, ignore_index=True)

#you can toggle which lines are commented, if you
#want to re-load your results to avoid repeatedly calling this function

#critics = build_table(movies, 3000)
#critics.to_csv('critics.csv', index=False)
critics = pd.read_csv('critics.csv')

#for this assignment, let's drop rows with missing quotes
critics = critics[~critics.quote.isnull()]


I am working on the file "HW3_solutions.ipynb".
The first task is:
Problem 1: Get the data
Here's a chunk of the MovieLens Dataset:
Can you write Python code for it?

from io import StringIO
movie_txt = requests.get('https://raw.github.com/cs109/cs109_data/master/movies.dat').text
movie_file = StringIO(movie_txt) # treat a string like a file
movies = pd.read_csv(movie_file, delimiter='\t')

#print the first row
movies[['id', 'title', 'imdbID', 'year']].irow(0)


I am working on the file "HW3_solutions.ipynb".
The first task is:
Problem 2: Explore
Before delving into analysis, get a sense of what these data look like. Answer the following questions. Include your code!
2.1 How many reviews, critics, and movies are in this dataset?
Can you write Python code for it?

#your code here
n_reviews = len(critics)
n_movies = critics.rtid.unique().size
n_critics = critics.critic.unique().size

print "Number of reviews: %i" % n_reviews
print "Number of critics: %i" % n_critics
print "Number of movies:  %i" % n_movies


I am working on the file "HW4.ipynb".
The first task is:
1.3 Create a smaller data set in dataframe smalldf by looking for those businesses with more than 150 reviews and those users with more than 60 reviews. Include all the columns that were there in the parent dataframe. Since you have created a subset of the data set, use the method provided above to recalculate the averages. Print the number of unique users and items in this data set.
Note that while this cut makes sure we have prolific users, the cut on businesses restores sparsity  by reducing the number of reviews per user.
Can you write Python code for it?

#your code here
smalldf = fulldf[(fulldf.business_review_count > 150) & (fulldf.user_review_count > 60)]
print "smalldf as a percentage of fulldf: %0.2f" % (float(smalldf.shape[0])/float(fulldf.shape[0])*100)

smalldf_new = recompute_frame(smalldf)

print "No of unique users: %d" %smalldf.user_id.unique().size
print "No of unique restaurants: %d" %smalldf.business_id.unique().size


I am working on the file "HW4.ipynb".
The first task is:
2.2 Now write a function that returns the predicted rating for a user and an item using the formula at the beginning of this problem. Include code to deal with the possibility that the sum of scores that goes in the denominator is 0: return a predicted rating of the baseline portion of the formula in that case. This function rating takes as arguments the dataframe, the database, the wanted restaurant_id and user_id, and k as well as the regularizer.
Can you write Python code for it?

"""
Function
--------
rating

Parameters
----------
df: Dataframe
    The dataframe of reviews such as smalldf
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businessed. e.g. dbase.get(rid1,rid2)
restaurant_id : string
    The id of the restaurant whose nearest neighbors we want
user_id : string
    The id of the user, in whose reviewed restaurants we want to find the neighbors
k : int
    the number of nearest neighbors desired, default 7
reg: float
    the regularization.

Returns
--------
A float
    which is the impued rating that we predict that user_id will make for restaurant_id
"""

#your code here
def rating(df, dbase, restaurant_id, user_id, k, reg):
    #calculate base line
    overall_avg_rating = df.stars.mean()
    avg_user_rating = df[df.user_id == user_id].user_avg.values[0]
    avg_biz_rating = df[df.business_id == restaurant_id].business_avg.values[0]

    prod_similarities_and_rating = 0.
    sum_of_similarities = 0.

    #print  baseline_rating

    #find knearest and loop through them
    similars = knearest_amongst_userrated(restaurant_id,user_id, df, dbase, k =k, reg =reg)
    for r_id,sim,cs in similars:
        #print("For restaurant: %s" %restaurant_j[0])

        rest_reviews = df[(df.user_id == user_id) & (df.business_id == r_id)]
        user_rating = rest_reviews.stars.values[0]
        rest_avg = rest_reviews.business_avg.values[0]
        prod_similarities_and_rating += (sim * (user_rating - (rest_avg + avg_user_rating - overall_avg_rating  )))
        sum_of_similarities += sim
        #print  user_rating_j, base_rating_j,prod_similarities_and_rating, sum_of_similarities
    baseline_rating =  avg_user_rating + avg_biz_rating -  overall_avg_rating
    if(sum_of_similarities <= 0):
        return baseline_rating
    else:
        return baseline_rating + (prod_similarities_and_rating/ sum_of_similarities)

rating(smalldf,db, '53YGfwmbW73JhFiemNeyzQ',...


I am working on the file "HW4.ipynb".
The first task is:
2.3 Explain in words why the predicted ratings are lower than the actual ratings. How do the user average rating and restaurant average rating affect this? How does sparsity affect the predicted ratings?
your answer here
Recall that bizs (defined just above question 1.7) has restaurants sorted by Vern's actual star ratings. This means that in this sample, we are looking at Vern's top rated restaurants.
The predicted ratings are lower because these are Vern's top 5 choices, which represent the largest positive deviations away from Vern's mean rating of 3.57. Because we are looking at the upper tail of Vern's rating distribution, but pooling information together from the K nearest neighbors among Vern's rated restaurants to construct the predicted rating, the predicted ratings should fall closer to Vern's user mean than the true ones do. Taking into account the average restaurant rating helps a little bit here because we can adjust the predicted rating to reflect an overall very good restaurant, but it does not counteract the effect of looking at the upper tail of Vern's ratings.
Note that if we were to take Vern's bottom 5 restaurants, we would see the opposite effect.
In general, the larger K is (assuming that the similarities within this neighborhood are positive), the closer the predicted rating will be to Vern's user average (this is the bias limit in the bias-variance tradeoff). Similarly, the smaller K is, the more likely we are to have user ratings that are close to the observed rating (the variance limit). The sparsity of the data affects how quickly we move from the variance limit to the bias limit as we increase K. If there were a lot of very similar restaurants in the dataset that Vern had ranked very highly, even with K relatively large, it would be possible to see a predicted rating much closer to the extremely positive ratings we see here in Vern's top 5 (see the results in question 4.4). As these data are now, however, even the most similar 7 restaurants to these that Vern rated so highly lie...
Can you write Python code for it?

def compare_results(stars_actual, stars_predicted, ylow=-10, yhigh=15, title=""):
    """
    plot predicted results against actual results. Takes 2 arguments: a
    numpy array of actual ratings and a numpy array of predicted ratings
    scatterplots the predictions, a unit slope line, line segments joining the mean,
    and a filled in area of the standard deviations."
    """
    fig=plt.figure()
    df=pd.DataFrame(dict(actual=stars_actual, predicted=stars_predicted))
    ax=plt.scatter(df.actual, df.predicted, alpha=0.2, s=30, label="predicted")
    plt.ylim([ylow,yhigh])
    plt.plot([1,5],[1,5], label="slope 1")
    xp=[1,2,3,4,5]
    yp=df.groupby('actual').predicted.mean().values
    plt.plot(xp,yp,'k', label="means")
    sig=df.groupby('actual').predicted.std().values
    plt.fill_between(xp, yp - sig, yp + sig,
                 color='k', alpha=0.2)
    plt.xlabel("actual")
    plt.ylabel("predicted")
    plt.legend(frameon=False)
    remove_border()
    plt.grid(False)
    plt.title(title)
    print np.mean(np.abs(df.predicted) < 15)


I am working on the file "HW4.ipynb".
The first task is:
2.4 For each review in the data set, obtain a prediction from the entire dataframe smalldf. Use the function compare_results above to plot the predicted ratings against the observed ones. Make 4 such graphs, at k=3 and k=10, and for reg=3. and reg=15.
Note that this analysis is not strictly a model check because we are testing on the training set. However, since the user averages would change each time a cross-validation split was done on the set, we would incur the prohibitive expense of redoing the database each time. This would be better done on a cluster, using map-reduce or other techniques. While we explore map-reduce later in this homework, we shall not do any cross-validation.
Explain the results you get in the graphs in words.
Can you write Python code for it?

#your code here
def plot_results(df, k,reg):
    user_id = df.user_id.values
    rest_id = df.business_id.values
    actual = df.stars.values
    predicted= np.zeros(len(actual))
    count =0
    for u_id, rest_id in zip(user_id, rest_id):
        predicted[count]= rating(df, db, rest_id, u_id, k, reg)
        count+=1
    compare_results(actual, predicted)

#your code here
print "k=3, reg=3."
plot_results(smalldf,3,3.)
plt.title("k=3, reg=3.")

print "k=3, reg=15."
plot_results(smalldf,3,15.,)
plt.title("k=3, reg=15.")

print "k=10, reg=3."
plot_results(smalldf,10,3.)
plt.title("k=10, reg=3.")

print "k=10, reg=15."
plot_results(smalldf,10,15.,)
plt.title("k=10, reg=15.")


I am working on the file "HW4.ipynb".
The first task is:
4.4  For testuserid, once again, print out the ratings using the bizs list as before. How have they changed with respect to Question 2? Why might this be?
Can you write Python code for it?

#your code here
#your code here
print "for user",usernamefromid(fulldf,testuserid), 'avg', fulldf[fulldf.user_id==testuserid].stars.mean()
for i in bizs:
    print "========="
    print biznamefromid(fulldf, i), i
    print rating(fulldf, dbfull, i, testuserid, k=7, reg=3.)
    u,a=get_other_ratings(i, testuserid, fulldf)
    print "User Score:",u,"Avg score",a


I am working on the file "HW4.ipynb".
The first task is:
As you can see, even though we chose a subset of the dataframe in which every restaurant had 150 reviews and every user had atleast made 60, the common support of most pairs of restaurants is really low, indeed less than 10!.
Calculating Similarity
Users rate restaurants on a scale of 1-5. Even though this rating is integer valued, for the purposes of this assignment we shall treat it as a real number.
Even though each reviewer uses the same 5-star scale when rating restaurants, comparing two users by comparing their raw user ratings can be problematic. Consider a user whose average rating is 2. This is a curmudgeonly user. Consider another whose average rating is 4. This is a rather enthusiastic one. How should we compare a 3 rating by the curmudgeonly one to a 5 rating of the enthusiastic one?
It is for this purpose that we must subtract the average rating of the user from the actual rating of the restaurants in computing the similarity of two restaurants. This makes the above ratings by the two users comparable. We do this in the function pearson_sim defined below.
If there is no common support (n_common=0), we have no basis for making a similarity estimate, and so we set the similarity to 0. In the case that the individual restaurant rating variance is 0, such as in the case where there is only one common reviewer (n_common=1), we return the NaN that the scipy pearsonr returns. We will deal with it soon,
Can you write Python code for it?

from scipy.stats.stats import pearsonr
def pearson_sim(rest1_reviews, rest2_reviews, n_common):
    """
    Given a subframe of restaurant 1 reviews and a subframe of restaurant 2 reviews,
    where the reviewers are those who have reviewed both restaurants, return
    the pearson correlation coefficient between the user average subtracted ratings.
    The case for zero common reviewers is handled separately. Its
    ok to return a NaN if any of the individual variances are 0.
    """
    if n_common==0:
        rho=0.
    else:
        diff1=rest1_reviews['stars']-rest1_reviews['user_avg']
        diff2=rest2_reviews['stars']-rest2_reviews['user_avg']
        rho=pearsonr(diff1, diff2)[0]
        if np.isnan(rho):
            rho=0.
    return rho


I am working on the file "HW4.ipynb".
The first task is:
Common Support
Lets now make a histogram of the common user support (the number of common reviewers) of each pair of restaurants on the smaller set, and print the mean. Pay attention to the code, as you will use parts of it later. (This code takes a bit of time to run, so be patient).
The common support is an important concept, as for each pair of restaurants, its the number of people who reviewed both. It will be used to modify similarity between restaurants. If the common support is low, the similarity is less believable.
Can you write Python code for it?

restaurants=smalldf.business_id.unique()  #get all the unique restaurant ids
supports=[]
for i,rest1 in enumerate(restaurants):  #loop for first restaurant (rest1) in the pair in the f
    for j,rest2 in enumerate(restaurants): # loop for second restaurant(rest2) in the pair
        if  i < j: #skip pairing same restaurants and forming duplicate pairs
            rest1_reviewers = smalldf[smalldf.business_id==rest1].user_id.unique() #find all unique users who reviewed restaurant 1
            rest2_reviewers = smalldf[smalldf.business_id==rest2].user_id.unique() #find all unique users who reviewed restaurant 2
            common_reviewers = set(rest1_reviewers).intersection(rest2_reviewers) # find common reviewers by taking intersection
            supports.append(len(common_reviewers)) # add the no of common reviewers to list
print "Mean support is:",np.mean(supports)
plt.hist(supports) #plot hist of list
histogram_settings("Common reviewers per each pair of restaurants", "Count", "Common User Support")


I am working on the file "HW4.ipynb".
The first task is:
Data in smalldf looks less sparse than the fulldf.
1.4 Compute histograms of the average user rating in the smaller data set, and the average business rating in the smaller data set. Print the overall mean.
Can you write Python code for it?

#histogram of the average user rating
plt.figure()
avg_user_rating = smalldf.groupby('user_id').stars.mean()
plt.hist(avg_user_rating, bins =5 , label ='Avg User Ratings')
plt.axvline(avg_user_rating.mean(), 0, 1, color='r', label='Avg User Rating: %.2f'%avg_user_rating.mean())
histogram_settings("Avg User Rating", "Count", "Avg User Rating Count")

plt.figure()
avg_bus_rating = smalldf.groupby('business_id').stars.mean()
plt.hist(avg_bus_rating, bins =5 , label ='Avg Restaurant Ratings')
plt.axvline(avg_bus_rating.mean(), 0, 1, color='r', label='Avg Restaurant Rating: %.2f'%avg_bus_rating.mean())
histogram_settings("Avg Restaurant Rating", "Count", "Avg Restaurant Rating Count")

print "Overall average rating in smalldf: %0.2f"%smalldf.stars.mean()


I am working on the file "HW4.ipynb".
The first task is:
Explanation for those funny yield keywords
The functions above “yield” values, and do not “return” them. They are generators. Here is an example:
Can you write Python code for it?

def upper_generator(words):
    for word in words:
        yield word.upper()

words = ['a', 'couple', 'of', 'words', 'to', 'process']

print upper_generator(words)
print list(upper_generator(words))
for u in upper_generator(words):
     print u


I am working on the file "HW4.ipynb".
The first task is:
For the top-recommendations in the variable toprecos from the previous section, we compute the predicted rating and compare it with the average rating over all users available inside the tuples that make up toprecos. We use a k of 7 and regularization 3. For comparision we also print this users' average rating. Do you notice anything interesting about how the order has changed from when we did this with the global similarities? (for you to think, not to answer)
Can you write Python code for it?

print "User Average", smalldf[smalldf.user_id==testuserid].stars.mean(),"for",usernamefromid(smalldf,testuserid)
print "Predicted ratings for top choices calculated earlier:"
for biz_id,biz_avg in toprecos:
    print biznamefromid(smalldf, biz_id),"|",rating(smalldf, db, biz_id, testuserid, k=7, reg=3.),"|","Average",biz_avg


I am working on the file "HW4.ipynb".
The first task is:
For the user testuserid, we loop over the variable bizs (which is a set of restaurants the user has rated) and print the predicted rating, and the actual rating and restaurant average rating obtained using the function above. We again use k=7 and a regularization of 3.
Can you write Python code for it?

print "for user",usernamefromid(smalldf,testuserid), 'avg', smalldf[smalldf.user_id==testuserid].stars.mean()
for biz_id in bizs:
    print "----------------------------------"
    print biznamefromid(smalldf, biz_id)
    print "Predicted Rating:",rating(smalldf, db, biz_id, testuserid, k=7, reg=3.)
    u,a=get_other_ratings(biz_id, testuserid, smalldf)
    print "Actual User Rating:",u,"Avg Rating",a


I am working on the file "HW4.ipynb".
The first task is:
Get top matches
Its now time to answer the question: "if you liked this, you might also like these". We use our testbizid and testbizid2 to compute the k=7 nearest neighbors with a regularization of 3. . We print these top 7 matches names, along with their similarity coefficient and common support.
Can you write Python code for it?

tops=knearest(testbizid, smalldf.business_id.unique(), db, k=7, reg=3.)
print "For ",biznamefromid(smalldf, testbizid), ", top matches are: \n"
print"No\t    Restaurant Name\t Similarity\t    Support"
for i, (biz_id, sim, nc) in enumerate(tops):
    print "%2s" %i,"%-30s" %biznamefromid(smalldf,biz_id), "%-15.4f" %sim,"%5d" %nc # -30s -> left aligns, 30 spaces

tops2=knearest(testbizid2, smalldf.business_id.unique(), db, k=7, reg=3.)
print "For ",biznamefromid(smalldf, testbizid2), ", top matches are:\n"
print"No Restaurant Name\t\t\t\t Similarity\t    Support"
for i, (biz_id, sim, nc) in enumerate(tops2):
    print "%2s" %i,"%-45s" %biznamefromid(smalldf,biz_id), "%-15.4f" %sim,"%5d" %nc


I am working on the file "HW4.ipynb".
The first task is:
Get top recommendations for user.
1.7 Its your job now to write a function get_top_recos_for_user which takes as arguments a userid, the n top choices for the user, the dataframe, k, and a regularizer, and returns the top recommendations obtained from combining the restaurants that are neighbors of each of the n choices, in the way described in the previous paragraph. This returned list is a list of tuples (restaurant_id, business_avg) sorted by business_avg where business_avg is the average rating of the restaurant over the dataframe.
Can you write Python code for it?

"""
Function
--------
get_top_recos_for_user

Parameters
----------
userid : string
    The id of the user for whom we want the top recommendations
df : Dataframe
    The dataframe of restaurant reviews such as smalldf
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businesses. e.g. dbase.get(rid1,rid2)
n: int
    the n top choices of the user by star rating
k : int
    the number of nearest neighbors desired, default 8
reg: float
    the regularization.

Returns
--------
A sorted list
    of the top recommendations. The list is a list of tuples
    (business_id, business_avg). You are combining the k-nearest recommendations
    for each of the user's n top choices, removing duplicates and the ones the user
    has already rated.
"""

#your code here
def get_top_recos_for_user(userid,df,dbase,n=5,k=7,reg=3):

    rest_ids = get_user_top_choices(userid, df, numchoices =n)["business_id"].values
    rests_rated_by_user = df[df.user_id==userid].business_id.values

    top_recos =[]
    for rest in rest_ids:
        tops=knearest(rest, df.business_id.unique(), dbase, k=k, reg=reg)
        for top in tops:
            if top[0] not in rests_rated_by_user:
                top_recos.append(top)
        #remove duplicates
        ids=[e[0] for e in top_recos]
        #print ids
        uids={k:0 for k in list(set(ids))} # add 0 to identify the unique ones in the full list.

        top_unique =[]
        for e in top_recos:
            if uids[e[0]] == 0:
                top_unique.append(e)
                uids[e[0]] = 1

        top_restaurants =[]
        for r,s,nc in top_unique:
            avg_rating = df[df.business_id ==r].stars.mean()
            top_restaurants.append((r,avg_rating))

        top_restaurants = sorted(top_restaurants, key = itemgetter(1), reverse = True)

        if n < len(top_restaurants):
            return top_restaurants[0:n]
        else:
        ...


I am working on the file "HW4.ipynb".
The first task is:
Here is the Gibbs sampler skeleton that your functions fit into. Look over the structure to see how for each draw from the posterior, the sampler iterates through $\mu$, $\sigma$, $\gamma_m$ for each item, and $\theta_u$ for each user.
Can you write Python code for it?

"""
Function
--------
factor_gibbs

Runs a gibbs sampler to infer mean, variance, user-specific, and item-specific
parameters.

Inputs
-------
data: A dataframe containing ratings data.
L: Dimension of latent factors.
maxit: Number of samples to draw from posterior.
Lambda_theta_diag: Hyperparameter controlling regularization of Theta.
Lambda_gamma_diag: Hyperparameter controlling regularization of Gamma.
progress: if true, print iteration number every 100 iterations.

Outputs
--------
Dictionary with elements
mu: Draws of mu. 1D array of length maxiter.
sig2: Draws of sig2, residual _variance_. 1D array of length maxiter.
theta: Draws of Theta. U-by-L-by-maxiter array.
gamma: Draws of Gamma. M-by-L-by-maxiter array.
EY: Draws of fitted values of Y. N-by-maxiter array.
"""
def factor_gibbs(data, L, maxit, Lambda_theta_diag, Lambda_gamma_diag, progress=True):
    data = data.copy()
    N = data.shape[0]

    #Create indices that allow us to map users and restaurants to rows
    #in parameter vectors.
    uusers, uidx = np.unique(data.user_id, return_inverse=True)
    uitems, midx = np.unique(data.business_id, return_inverse=True)

    nusers = uusers.size
    nitems = uitems.size

    #Add numerical indices to dataframe.
    data["uidx"] = uidx
    data["midx"] = midx

    #Group observations by user and by business.
    ugroups = data.groupby("uidx")
    mgroups = data.groupby("midx")

    all_avg = data.stars.mean()
    u_avg = ugroups.stars.mean()
    m_avg = mgroups.stars.mean()

    #Initialize parameters and set up data structures for
    #holding draws.
    #Overall mean
    mu = all_avg
    mu_draws = np.zeros(maxit)
    #Residual variance
    sig2 = 0.5
    sig2_draws = np.zeros(maxit)

    #Matrix of user-specific bias and L latent factors.
    theta = np.zeros([nusers, L+1])
    theta[:,0] = u_avg-all_avg
    theta_draws = np.zeros([nusers, L+1, maxit])

    #Matrix of item-specific bias and L latent factors.
    gamma = np.zeros([nitems, L+1])
    gamma[:,0] = m_avg-all_avg
    gamma_draws =...


I am working on the file "HW4.ipynb".
The first task is:
How does this compare to the parent data set, in terms of size and sparsity? Once again, plot histograms of the review count grouped by user, and by the review count grouped by business, respectively, and describe the results
Can you write Python code for it?

# histogram of the review count grouped by user
plt.figure()
count_by_user = smalldf.groupby(smalldf.user_id).review_id.count()
count_by_business = smalldf.groupby(smalldf.business_id).review_id.count()
plt.hist(count_by_user, bins =10)
plt.axvline(count_by_user.mean(), 0, 1, color='r', label='Avg Reviews per user: %.2f'%count_by_user.mean())
histogram_settings("Reviews per user","review count", "Reviews count per user", "upper right")

# histogram of the review count grouped by restaurant
plt.figure()
plt.hist(count_by_business, bins =10)
plt.axvline(count_by_business.mean(), 0, 1, color='r', label='Avg Reviews per restaurant: %.2f'%count_by_business.mean())
histogram_settings("Reviews per restaurant","review count", "Reviews count per restaurant", "upper right")


I am working on the file "HW4.ipynb".
The first task is:
Lets print the top recommendations for testuserid, with a regularization of 3.
Can you write Python code for it?

print "For user", usernamefromid(smalldf,testuserid), "the top recommendations are:"
toprecos=get_top_recos_for_user(testuserid, smalldf, db, n=5, k=7, reg=3.)
for biz_id, biz_avg in toprecos:
    print biznamefromid(smalldf,biz_id), "| Average Rating |", biz_avg


I am working on the file "HW4.ipynb".
The first task is:
Lets run make_database and store the result in the global variable db. Lets print out an example entry. Running this function will take a bit of time.
Can you write Python code for it?

db=Database(smalldf)
db.populate_by_calculating(pearson_sim)

db.get("z3yFuLVrmH-3RJruPEMYKw", "zruUQvFySeXyEd7_rQixBg")


I am working on the file "HW4.ipynb".
The first task is:
Making a database of similarities
We now move to calculating a global database of pairwise restaurant similarities.
We provide you here with a function to make a database of the similarities for each pair of restaurants in the database. The class Database is initialized in its constructor by taking as arguments a dataframe of reviews. The method populate_by calculating iterates over every possible pair of business_id's in the dataframe and populates the database with similarities and common supports. It takes as arguments a function the similarity function similarity_func like pearson_sim (calculate_similarity then uses this to calculate the similarity). The get method on the database can be used to retrieve the similarity for two business ids.
(See Thu Oct 17th's class video for information about classes)
Can you write Python code for it?

class Database:
    "A class representing a database of similaries and common supports"

    def __init__(self, df):
        "the constructor, takes a reviews dataframe like smalldf as its argument"
        database={}
        self.df=df
        self.uniquebizids={v:k for (k,v) in enumerate(df.business_id.unique())}
        keys=self.uniquebizids.keys()
        l_keys=len(keys)
        self.database_sim=np.zeros([l_keys,l_keys])
        self.database_sup=np.zeros([l_keys, l_keys], dtype=np.int)

    def populate_by_calculating(self, similarity_func):
        """
        a populator for every pair of businesses in df. takes similarity_func like
        pearson_sim as argument
        """
        items=self.uniquebizids.items()
        for b1, i1 in items:
            for b2, i2 in items:
                if i1 < i2:
                    sim, nsup=calculate_similarity(b1, b2, self.df, similarity_func)
                    self.database_sim[i1][i2]=sim
                    self.database_sim[i2][i1]=sim
                    self.database_sup[i1][i2]=nsup
                    self.database_sup[i2][i1]=nsup
                elif i1==i2:
                    nsup=self.df[self.df.business_id==b1].user_id.count()
                    self.database_sim[i1][i1]=1.
                    self.database_sup[i1][i1]=nsup

    def get(self, b1, b2):
        "returns a tuple of similarity,common_support given two business ids"
        sim=self.database_sim[self.uniquebizids[b1]][self.uniquebizids[b2]]
        nsup=self.database_sup[self.uniquebizids[b1]][self.uniquebizids[b2]]
        return (sim, nsup)


I am working on the file "HW4.ipynb".
The first task is:
Ok it's time to recommend!
Lets choose the two very different businesses in the dataframe
Can you write Python code for it?


I am working on the file "HW4.ipynb".
The first task is:
Plot the predictions against the observed data.You can use the compare_results function defined in the previous section. How do the fitted values compare to those from the KNN procedure?
Can you write Python code for it?

#your code here

compare_results(smalldf.stars.values, prediction, ylow=1, yhigh=5, title="From Gibbs Sampler")


I am working on the file "HW4.ipynb".
The first task is:
Posterior Summaries
Once you have posterior draws from the sampler, the most natural thing to do is to compute the posterior mean of each quantity you are intersted in. To do this, we simply need to take the average value of each quantity across the samples drawn from the sampler. Before taking the average, however, we will want to ignore the first 20-30% of samples because these correspond the burnin period, the time during which the sampler is still looking for the main meat of the distribution.
Ok it's time to recommend!
3.3 Now that you have the Gibbs sampler, draw 1000 samples from the posterior distribution using a two-dimensional latent factor and prior precisions Lambda_theta_diag and Lambda_gamma_diag both equal to 0.1.
Compute the posterior mean of the fitted values for each $Y_{um}$, eliminating the first 200 samples. Call these the prediction.  These constitute our recommendations. True to the bayesian paradigm, we dont just have mean predictions, but entire distributions. But currently we are only interested in the means.
Can you write Python code for it?

#your code here
gibbs_sample = factor_gibbs(smalldf, 2, 1000, 0.1, 0.1, progress=True)

burnin = 200
prediction =np.mean(gibbs_sample["EY"][:,burnin:],axis = 1)


I am working on the file "HW4.ipynb".
The first task is:
Problem 2: A user based recommender with predicted ratings
This is all very nice. We can provide ratings based on global similarities to a restaurant. However, in many cases this is not enough.
For example, it is hard to judge if the above recommendations are any good. In the usual testing paradigm, say that we break the dataframe into train and test. Based on the training set, I am recommended restaurant B. Now, I have rated B, but that information is in the testing set. I have no way of comparing the rating I give B in the testing set, to the similarity computed from the training set that was used to make the recomendation. The best I could do is to compare the average rating of restaurant B in the training set to my rating of restaurant B in the test set.
In this section, we shift our focus to more fine-grained predictions about each user, and try to predict what rating a user would give to a restaurant they have never tried before. To do this, we will try to personalize the information we use even further, and only pool information from restaurants that the user has rated.
This allows us to return to the original problem of prediction $Y_{um}$ for a restaurant $m$ that user $u$ has never rated before. Using our newly computed similarity metrics, we can modify our original baseline estimate by pulling in information from the user's neighborhood of the restaurant $m$, and predict $Y_{um}$ as:
$$ \hat{Y_{um}} = \hat Y^{baseline}{um}\, + \,\frac{\sum\limits{j \in S^{k}(m;u)} s_{mj} ( Y_{uj} - \hat Y^{baseline}{uj} )}{\sum\limits{j \in S^{k}(m;u)} s_{mj} } $$
where $s^{k}(m;u)$ is the $k$ neighbor items of item $m$ which have been rated by user $u$.
Now, this is not a particularly good assumption, especially in the situation where a restaurant is new (new item problem) or a user is new (cold start problem), or in the case when there are very few reviewers of a restaurant, or very few reviews by a user respectively. However, one must start somewhere!
Notice that in adding in the similarity term, we subtract...
Can you write Python code for it?

"""
Function
--------
knearest_amongst_userrated

Parameters
----------
restaurant_id : string
    The id of the restaurant whose nearest neighbors we want
user_id : string
    The id of the user, in whose reviewed restaurants we want to find the neighbors
df: Dataframe
    The dataframe of reviews such as smalldf
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businessed. e.g. dbase.get(rid1,rid2)
k : int
    the number of nearest neighbors desired, default 7
reg: float
    the regularization.

Returns
--------
A sorted list
    of the top k similar restaurants. The list is a list of tuples
    (business_id, shrunken similarity, common support).
"""
#your code here
from operator import itemgetter
def knearest_amongst_userrated(restaurant_id,user_id, df, dbase, k =7, reg =3):

    similar = []
    rests_rated_by_user = df[df.user_id==user_id].business_id.unique()
    return knearest(restaurant_id, rests_rated_by_user, dbase, k, reg)


I am working on the file "HW4.ipynb".
The first task is:
Q4 Scaling Up
All our recommenders suffer from problems having to do with the fact that we subsetted an already sparse user-item matrix. The more items we have, the more items we may find in the vicinity of a given item, and thus we are likely to give a more robust average rating to the given item.
In this problem we shall use Amazon Elastic Map-Reduce to tackle the entire user-restaurant matrix. We shall do this in two parts: we'll use MRJob locally on your machine to on the smaller data set to calclate the pearson database, and then we'll tackle the entire data set on Amazon.
The larger set has 35000 users and 4500 items. Computing the 4500X4500 similarity matrix on one machine will be prohibitively expensive. Thus we'll adopt a strategy where we'll split the calculation over multiple machines using the map-reduce paradigm, with mappers and reducers working on multiple machines
Then we calculate the k-nearest neighbors in the 'space' of the user: this involves a database lookup and an iteration over the items a user has rated. Since the latter is usually not a very large number, this computation can be managed on a front end machine (even if storing the database will take a lot of memory).
We'll first create subset data frames, which have just those columns which we will send to the map-reduce. We'll also strip out the header and index of the frame. The reason for doing this is: unless we pre-populate the machines on Amazon with software, we can rely only on the regular python library, numpy, and scipy being there (and at python 2.6), and thus we will need to parse the csv file, line by line (mrjob uses hadoop's stream protocol and thus needs to be fed line by line).
Can you write Python code for it?

subsetoffull=fulldf[['user_id','business_id', 'stars','business_avg','user_avg']]
subsetoffull.to_csv("subset-full.csv", index=False, header=False)
subsetofsmall=smalldf[['user_id','business_id', 'stars','business_avg','user_avg']]
subsetofsmall.to_csv("subset-small.csv", index=False, header=False)


I am working on the file "HW4.ipynb".
The first task is:
Running mrjob locally
mrjob scripts cannot be run from the ipython notebook, as they fork themselves on execution. Thus you must write the code for mrjob in a separate file which you must submit along with this homework, in the same folder as the python notebook file.
If you have not done so already (you were supposed to do this as part of HW 0), you will first need to install mrjob. The appropriate equivalent of the following incantation should do the job:
~/anaconda/bin/pip install mrjob

To familiarize yourself with the structure of an mrjob script, please read this . Run the examples in that document to familiarize yourself with mrjob.
The kind of script you will be writing is in the section "Writing your second job" in that document.
All mrjob tasks use the map-reduce strategy to divide up computation across computers. You should work through the mrjob tutorial to gain familiarity with this, but we’ll also outline the basic process here:

During the first map step, mrjob calls a mapper function with a key (which for the first step is None), and a value (which for the first step is a line of data from an input file). This function does whatever it wants with this data, and yields a key and value.  The key is used in step 2 to gather up the values from all the different mappers into groups

mrjob collects the outputs from all the mappers, and gathers them into subsets with the same key value (this is similar to what pandas.groupby does). It passes each of these subsets to a reducer (or “collector”) function, whose job is to synthesize this list of grouped data into something useful (e.g., computing the mean of all the inputs). It then yields the key and reduced value.

If there are any additional steps, mrjob feeds each output from a reducer function in step 2 to the next mapper. Otherwise, it prints the output.

The point behind map-reduce is to agree upon a common framework to split up a large computational job into smaller tasks. mrjob then has a lot of freedom to organize how these tasks run in...
Can you write Python code for it?

from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
from IPython.display import HTML
import urllib
skelcode = urllib.urlopen("https://raw.github.com/cs109/content/master/skeleton.py").read()
skelhtml=highlight(skelcode, PythonLexer(), HtmlFormatter())
HTML(skelhtml)


I am working on the file "HW4.ipynb".
The first task is:
Testing the ratings
Let us compare the predicted ratings with a user's ratings. Note that we are doing this on the same set that we constructed the predictions with, so this is not a validation of the procedure, but simply a check of the procedure's fit. We first write a helper function to return the user score for a restaurant, and the restaurant's average score over all users.
Can you write Python code for it?

def get_other_ratings(restaurant_id, user_id, df):
    "get a user's rating for a restaurant and the restaurant's average rating"
    choice=df[(df.business_id==restaurant_id) & (df.user_id==user_id)]
    users_score=choice.stars.values[0]
    average_score=choice.business_avg.values[0]
    return users_score, average_score


I am working on the file "HW4.ipynb".
The first task is:
The data frame is a frame of reviews. We have joined in information about users and businesses into this frame so that you have only one frame to work with.
This information is for the reviews themselves:
Here is a description of the data fields in this dataframe, on the business side
And Finally, a set of fields for users
In this data set, every user has only one review for each restaurant. Convince yourself of this. (This answer does not need to be submitted).
Our Recommender
To motivate our recommendation system, consider the follwing example. Let's pretend we are in Boston for a second. Lets say the average rating of restaurants here by all the users is 3.5. Sandrine's at Harvard square is better than an average restaurant, so it tends to be rated 0.5 stars above the average (over all the users). However, you are a curmudgeon, who tends to rate 0.2 stars below the average. Then a baseline estimate for the recommendation for Sandrine's, for you, is 3.5+0.5-0.2=3.8.
These baseline estimates thus adjust the data by accounting for the systematic tendencies for some users who give higher ratings than others, and for some restaurants to recieve higher ratings than others. We can write the baseline estimate $\hat Y_{um}^{baseline}$ for an unknown rating $Y_{um}$ for user $u$ and restaurant or business $m$ as:
$$ \hat Y_{um}^{baseline} = \hat \mu + \hat \theta_{u0} + \hat \gamma_{m0} $$
where the unknown parameters $\theta_{u0}$ and $\gamma_{m0}$ indicate the deviations, or biases, of user $u$ and item $m$, respectively, from some intercept parameter $\mu$. (The reason for the strange notation with 0s will become clear in Problem 3)
Notice that the $\theta_{u0}$ and $\gamma_{m0}$ are parameters which need to be fit. The simplest thing to start with, and something we will do for Problems 1 and 2 (but not 3), is to replace them by their "mean" estimates from the data. Thus:
$$ \hat Y^{baseline}_{um} = \bar Y + (\bar Y_u - \bar Y) + (\bar Y_m - \bar Y)$$
where $\bar Y_u$ =  user_avg, the average of all a user $u$'s...
Can you write Python code for it?

fulldf.groupby(fulldf.user_id).review_id.count().describe()

fulldf.user_id.unique().size

#your code here

plt.hist(fulldf.groupby(fulldf.user_id).review_id.count(), log = True, bins= 20)
plt.xlabel("Reviews per user")
plt.ylabel("review count")
histogram_style()

fulldf.groupby(fulldf.business_id).review_id.count().describe()

#your code here

plt.hist(fulldf.groupby(fulldf.business_id).review_id.count(), log =True, bins= 20)
plt.xlabel("Reviews per restaurant")
plt.ylabel("review count")
histogram_style()

#your code here
print "No of users: %d" %fulldf.groupby(fulldf.user_id).review_id.count().size
print "No of businesses: %d" %fulldf.groupby(fulldf.business_id).review_id.count().size


I am working on the file "HW4.ipynb".
The first task is:
The function get_restaurant_reviews defined below takes a restaurant business_id and a set of users, and returns the reviews of that restaurant by those users. You will use this function in calculating a similarity function, in 1.5.
Can you write Python code for it?

def get_restaurant_reviews(restaurant_id, df, set_of_users):
    """
    given a resturant id and a set of reviewers, return the sub-dataframe of their
    reviews.
    """
    mask = (df.user_id.isin(set_of_users)) & (df.business_id==restaurant_id)
    reviews = df[mask]
    reviews = reviews[reviews.user_id.duplicated()==False]
    return reviews


I am working on the file "HW4.ipynb".
The first task is:
We can see that these two restaurants are in somewhat different orbits :-).
Lets now turn our attention to another question: what are the top recommendations for a user? To answer this we must find the user's top rated restaurants, find the nearest neighbors of these restaurants, merge these lists while removing the duplicates and the ones that the user has already rated, and sort by the restaurant's average rating. We provide the code to get the user's top choices in a subset data frame.
Can you write Python code for it?

def get_user_top_choices(user_id, df, numchoices=5):
    "get the sorted top 5 restaurants for a user by the star rating the user gave them"
    udf=df[df.user_id==user_id][['business_id','stars']].sort(['stars'], ascending=False).head(numchoices)
    return udf
testuserid="7cR92zkDv4W3kqzii6axvg"
#testuserid="sALB8-F04S9VcZMF_GkGUA"
print "For user", usernamefromid(smalldf,testuserid), "top choices are:"
bizs=get_user_top_choices(testuserid, smalldf)['business_id'].values
[biznamefromid(smalldf, biz_id) for biz_id in bizs]


I am working on the file "HW4.ipynb".
The first task is:
We print a pair to see that our answers are identical.
Can you write Python code for it?

print db.get("zruUQvFySeXyEd7_rQixBg", "z3yFuLVrmH-3RJruPEMYKw")
print db_mrjob_local.get("zruUQvFySeXyEd7_rQixBg", "z3yFuLVrmH-3RJruPEMYKw")


I am working on the file "HW4.ipynb".
The first task is:
We will Implement a function make_database_from_pairs which takes a dataframe of restaurants smalldf and the output parsed in the previous command to create the database like before. By the nature of the map-reduce algorithms these only contain those restaurant pairs with common support. The Database constructor initializes the remaining similarities to 0.
The function will take the dataframe and bizpairs obtained by parsing the EMR output file which have the key of business pairs and value the pair of pearson correlation and n_common. It will return an instance of the Database class.
This function will take a long time to run on large data sets.
Can you write Python code for it?

def make_database_from_pairs(df, bizpairs):
    """
    make the database from the pairs returned from mrjob.
    df is the dataframe, smalldf or fulldf.
    bizpairs are a list of elements, each of which is a list of two
        lists. The first of these lists has the two business id's, while
        the second has the similarity and the common support
    Returns an instance of the Database class.
    """
    dbase=Database(df)
    cache={}
    for bp,corrs in bizpairs:
        b1,b2=bp
        i1=dbase.uniquebizids[b1]
        i2=dbase.uniquebizids[b2]
        sim,nsup=corrs
        dbase.database_sim[i1][i2]=sim
        dbase.database_sim[i2][i1]=sim
        dbase.database_sup[i1][i2]=nsup
        dbase.database_sup[i2][i1]=nsup
        if cache.has_key(b1):
            nsup1=cache[b1]
        else:
            nsup1=dbase.df[dbase.df.business_id==b1].user_id.count()
            cache[b1]=nsup1
        if cache.has_key(b2):
            nsup2=cache[b2]
        else:
            nsup2=dbase.df[dbase.df.business_id==b2].user_id.count()
            cache[b2]=nsup2
        dbase.database_sim[i1][i1]=1.0
        dbase.database_sim[i2][i2]=1.0
        dbase.database_sup[i1][i1]=nsup1
        dbase.database_sup[i2][i2]=nsup2
    return dbase


I am working on the file "HW4.ipynb".
The first task is:
We will store the output in variable db_mrjob_local.
Can you write Python code for it?


I am working on the file "HW4.ipynb".
The first task is:
your answer here
If you are a bit confused by the look of these graphs, you should be!
For k=3, the predicted values are quite well-behaved, with the exception of several predictions that have extremely large magnitudes. It appears that the predicted values are pulled into the mean star rating, which sits somewhere around 3.8, so ratings on the low end are overestimated, and similarly ratings on the high end underestimated. The regularization does not appear to have a strong effect when k = 3.
For k=10, the predicted values are much less stable, with many more extreme predictions. The means appear to track better with the true means. The regularization has a much more extreme, although indirect, effect on the appearance of the plot. Since regularization has stronger effects on similarity scores between restaurants that have small common support, we can see that increasing k makes the predictions more sensitive to the regularization because in a small dataset, the common support between a restaurant and it's 10-nearest one will be quite small.
Note that this example does not seem to follow the standard bias-variance tradeoff, where we would expect small k to give a unbiased estimates that capture the extremes, while we would expect large k to give biased estimates that pull extreme values toward the mean. A large reason for this failure for this example to capture this behavior is that we have defined similarity scores that can be positive or negative, and the bias-variance logic is based on the more standard setting where we average together values that have strictly positive weights. When you have negative weights, it's possible that the sum of sij's in a neighborhood can get close to zero, making our estimator Ŷ um explode in the positive or negative direction since this would entail dividing by (nearly) zero. Thus for those restaurants where the denominator goes to 0 (more likely to happen at larger k as you have more chances of it there being more weights to add), the ratings are unstable, even...
Can you write Python code for it?

"""
Function
--------
gamma_m_draw

Draw a single sample from the conditional posterior distribution
of gamma_m.

Inputs
-------
X_m: A g-by-L+1 matrix, defined above.
Y_m: A 1D vector of length g, defined above.
sig2: Residual _variance_, as defined above.
Lambda_gamma: Prior precision matrix.

Outputs
--------
Single draw from conditional posterior, defined above.
"""
#Item-specific parameters given all else
#your code here
def gamma_m_draw(X_m,Y_m,sig2,Lambda_gamma):
    Q_m_inv = np.linalg.inv(np.dot(X_m.T,X_m)/sig2 + Lambda_gamma )
    XtY = np.dot(X_m.T, Y_m)
    return np.random.multivariate_normal(np.dot(Q_m_inv,XtY)/sig2 , Q_m_inv)

"""
Function
--------
theta_u_draw

Draw a single sample from the conditional posterior distribution
of gamma_m.

Inputs
-------
X_u: A g-by-L+1 matrix, defined above.
Y_u: A 1D vector of length g, defined above.
sig2: Residual _variance_, as defined above.
Lambda_theta: Prior precision matrix.

Outputs
--------
Single draw from conditional posterior, defined above.
"""
#User-specific parameters given all else
#your code here
def theta_u_draw(X_u,Y_u,sig2,Lambda_theta):
    Q_u_inv = np.linalg.inv(np.dot(X_u.T,X_u)/sig2 + Lambda_theta)
    XtY = np.dot(X_u.T, Y_u)
    return np.random.multivariate_normal(np.dot(Q_u_inv,XtY)/sig2 , Q_u_inv)


I am working on the file "HW4.ipynb".
The first task is:
your answer here
The results from the latent factor model appear to be better-behaved than those from the KNN procedure (with negative similarities) since there are no extreme values. In terms of the non-extreme values from the KNN procedure, the latent factor model appears to make similar predictions to the KNN procedure when k = 3., Specifically, the average prediction line for each class is too compressed toward the total data mean of around 3.8, meaning that we are in the bias limit where we are pooling too much information between ratings. Thus, we are overpredicting low ratings and underpredicting high ratings.
(If we compare to the KNN procedure with strictly positive weights, as in the homework addendum, we see again that the recommenders make comparable predictions, although the latent factor model again appears to fit slightly better than at both the bias or variance limits of the KNN procedure).
There is also a bias-variance tradeoff here. In this case, it appears that we have proposed a model that is too simple (thus close to the bias limit) because of how the ratings are pulled in toward the data mean. In this case, proposing more latent factors increases the flexibility of the model, and thus moves us toward the variance limit. Thus, the plot suggests that we may want to reduce the bias at the cost of increasing variance, for example by considering a latent factor model with more factors (say, L=15) to obtain a better fit (see ADDENDUM below).
Can you write Python code for it?

gibbs_out = factor_gibbs(smalldf, 15, 1000, 0.1, 0.1)
burnin = 200
predicted=np.mean(gibbs_out['EY'][:,burnin:], axis=1)
compare_results(smalldf.stars.values, predicted, ylow=1, yhigh=5, title="From Gibbs Sampler")


I am working on the file "HW4/Frequent Itemset.ipynb".
The first task is:
Step2: Transform the Data
In this step, I need to create another dataframe, use for loop to fill in the datafram based on the old dataframe.
Can you write Python code for it?

Data=DataFrame(data, columns=[ 'LoTemp', 'Precip', 'Manhattan'])
n=len(Data)
newData = DataFrame( index=range(0,n-1),columns=[ 'LoTemp', 'Precip', 'Manhattan'])

Data.loc[1,'LoTemp']
newData.loc[1,'LoTemp']=1

for x in range(0,n-1):
    for y in [ 'LoTemp', 'Precip', 'Manhattan']:
        if (Data.loc[x,y]<Data.loc[x+1,y]):
            newData.loc[x, y]=1
        elif (Data.loc[x,y]>Data.loc[x+1,y]):
            newData.loc[x, y]=-1
        else:
            newData.loc[x, y]=0
datalist=newData.values.tolist()


I am working on the file "HW4/Frequent Itemset.ipynb".
The first task is:
Step3: PyMining
In my data, each colum represent different attributes so the sequence of the dataset actually matters. I decided to use teh sequence mining method.
Can you write Python code for it?

from pymining import seqmining
freq_seqs = seqmining.freq_seq_enum(datalist, 20)
sorted(freq_seqs)

conn.close()


I am working on the file "HW4_sd3462.ipynb".
The first task is:
2. Using the function written above, do the following:
- Generate a sequence of 10 $C$ values in the interval [10^(-8), ..., 10^1] (i.e., do all powers of 10 from -8 to 1).
2.  Call aucs = xValSVM(train, ‘Y’, 10, cs)
3.  For each c in cs, get mean(AUC) and StdErr(AUC)
4.  Compute the value for max(meanAUC-StdErr(AUC)) across all values of c.
5.  Generate a plot with the following:
a.  Log10(c) on the x-axis
b.  1 series with mean(AUC) for each c
c.  1 series with mean(AUC)-stderr(AUC) for each c (use ‘k+’ as color pattern)
d.  1 series with mean(AUC)+stderr(AUC) for each c (use ‘k--‘ as color pattern)
e.  a reference line for max(AUC-StdErr(AUC)) (use ‘r’ as color pattern)
Then answer the question: Did the model parameters selected beat the out-of-the-box model for SVM?
Can you write Python code for it?

from scipy.stats import sem
import matplotlib.pyplot as plt
c_val = np.power(10,np.arange(-8.0,2.0))
aucs=xValSVM(data_train,"Y",10,c_val)

alldiff=list()
allmeans=list()
allstderr=list()
for i in aucs.keys():
    mean=np.mean(aucs[i])
    stderr=sem(aucs[i])
    print("Mean for",i,":",mean," Std. Err: ",stderr)
    allmeans.append(mean)
    allstderr.append(stderr)
    alldiff.append(mean-stderr)
print("Maximum difference:", max(alldiff)," for C value",list(aucs.keys())[(alldiff.index(max(alldiff)))])

plt.figure(figsize=(10,5))
plt.plot(np.log10(c_val),allmeans,label='Mean AUC')
plt.plot(np.log10(c_val),np.array(allmeans)-np.array(allstderr),'k+',label='Mean-Standard error')
plt.plot(np.log10(c_val),np.array(allmeans)+np.array(allstderr),'k--',label='Mean+Standard error')
plt.axhline(y=max(alldiff),color='r',label='Reference Line')
plt.legend(loc='lower right')
plt.show()


I am working on the file "HW4_sd3462.ipynb".
The first task is:
3. Which of the two models is generally better at ranking the test set? Are there any classification thresholds where the model identified above as "better" would underperform the other in a classification metric (such as TPR)?
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
from IPython.display import display, HTML

#Preparing confusion matrix to check different performance measures
conf_logreg=pd.DataFrame(confusion_matrix(y_true=data_test['Y'],y_pred=pred_logreg))
conf_svm=pd.DataFrame(confusion_matrix(y_true=data_test['Y'],y_pred=pred_svm))

display('Logistic')
display(conf_logreg)
display('SVM')
display(conf_svm)


I am working on the file "HW4_sd3462.ipynb".
The first task is:
Answer:
The Logistic model performs better than the SVM model on an average. We judge them by using the AUC metric which is base rate invariant. There are specific thresholds for which one model outperforms the other and the final selection must be made on the basis of the utility of the model.
In Logistic:
* We observe that the TPR (predicted 0 and actual 0) is higher than SvM. Logistic correctly classifies actual true values for 6597 cases as compared to SVM's 3251 values.
* The False Negative Error rate is also less. Logistic has 10566 while SVM has 13912 false negatives.
In SVM:
* It is better at classifying True Positives than Logistic. It correctly identifies positives for 31586 cases as compared to logistic's 29399 cases.
* The False Positive rate (1251) is also lower than logistic (3438)
Part 3: Model selection with cross-validation (10 points)
We think we might be able to improve the performance of the SVM if we perform a grid search on the hyper-parameter $C$.  Because we only have 1000 instances, we will have to use cross-validation to find the optimal $C$.
1. Write a cross-validation function that does the following:
- Takes as inputs a dataset, a label name, # of splits/folds (k), a sequence of values for $C$ (cs)
- Performs two loops
  - Outer Loop: for each f in range(k):
    - Splits the data into data_train & data_validate according to cross-validation logic
  - Inner Loop: for each c in cs:
    - Trains an SVM on training split with C=c, kernel="linear"
    - Computes AUC_c_k on validation data
    - Stores AUC_c_k in a  dictionary of values
- Returns a dictionary, where each key-value pair is: c:[auc-c1,auc-c2,..auc-ck]
Can you write Python code for it?

from sklearn.model_selection import train_test_split

def xValSVM(dataset, label_name, k, cs):
    aucstore=dict()
    for f in range(k):
        data_train,data_validate=train_test_split(dataset)
        for c in cs:
            svm_temp=SVC(kernel="linear",C=c).fit(data_train.drop(label_name,1),data_train[label_name])
            predsvm_temp=svm_temp.predict(data_test.drop(label_name,1))
            svmroc_temp=roc_auc_score(data_test[label_name],predsvm_temp)
            if c not in aucstore:
                aucstore[c]=list()
            aucstore[c].append(svmroc_temp)
    return aucstore


I am working on the file "HW4_sd3462.ipynb".
The first task is:
Part 2: Basic evaluations (5 points)
In this part you will build an out-of-the box logistic regression (LR) model and support vector machine (SVM). You will then plot ROC for the LR and SVM model.
1. Clean the two data files included in this assignment (data/boson_training_cut_2000.csv and data/boson_testing_cut.csv) and use them as training and testing data sets.
Can you write Python code for it?

data_train = cleanBosonData("../Datasets/boson_training_cut_2000.csv")
data_test = cleanBosonData("../Datasets/boson_testing_cut.csv")


I am working on the file "HW4_solutions.ipynb".
The first task is:
1.3 Create a smaller data set in dataframe smalldf by looking for those businesses with more than 150 reviews and those users with more than 60 reviews. Include all the columns that were there in the parent dataframe. Since you have created a subset of the data set, use the method provided above to recalculate the averages. Print the number of unique users and items in this data set.
Note that while this cut makes sure we have prolific users, the cut on businesses restores sparsity  by reducing the number of reviews per user.
Can you write Python code for it?

#your code here
smallidf=fulldf[(fulldf.user_review_count > 60) & (fulldf.business_review_count > 150)]
smalldf=recompute_frame(smallidf)


I am working on the file "HW4_solutions.ipynb".
The first task is:
1.6 Now we can move to writing a knearest function, which finds the k nearest neighbors of a given restaurant based on the shrunk similarities we calculate. Note that as defined here, the nearest neighbors are global over the entire set of restaurants, as opposed to being restricted to the restaurants a user has reviewed(we shall do that in the next problem). Thus, this is an expensive function!
Write a knearest that returns a k-length sorted list of 3-tuples each corresponding to a restaurant. The tuple structure is (business_id, shrunken similarity score, common support) where the similarity score and common support are with respect to the restaurant whose neighbors we are finding, and the business_id is the id of the "nearby" restaurant found. The nearby restaurants are found from a supplied numpy array of restaurants set_of_restaurants. The spec for the function is given below. HINT: use itemgetter from the operator module to do the sorting.
Can you write Python code for it?

"""
Function
--------
knearest

Parameters
----------
restaurant_id : string
    The id of the restaurant whose nearest neighbors we want
set_of_restaurants : array
    The set of restaurants from which we want to find the nearest neighbors
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businessed. e.g. dbase.get(rid1,rid2)
k : int
    the number of nearest neighbors desired, default 7
reg: float
    the regularization.

Returns
--------
A sorted list of the top k similar restaurants. The list is a list of tuples
    (business_id, shrunken similarity, common support).
"""
#your code here
from operator import itemgetter
def knearest(restaurant_id, set_of_restaurants, dbase, k=7, reg=3.):
    """
    Given a restaurant_id, dataframe, and database, get a sorted list of the
    k most similar restaurants from the entire database.
    """
    similars=[]
    for other_rest_id in set_of_restaurants:
        if other_rest_id!=restaurant_id:
            sim, nc=dbase.get(restaurant_id, other_rest_id)
            ssim=shrunk_sim(sim, nc, reg=reg)
            similars.append((other_rest_id, ssim, nc ))
    similars=sorted(similars, key=itemgetter(1), reverse=True)
    return similars[0:k]


I am working on the file "HW4_solutions.ipynb".
The first task is:
2.2 Now write a function that returns the predicted rating for a user and an item using the formula at the beginning of this problem. Include code to deal with the possibility that the sum of scores that goes in the denominator is 0: return an predicted rating of the baseline portion of the formula in that case. This function rating takes as arguments the dataframe, the database, the wanted restaurant_id and user_id, and k as well as the regularizer.
Can you write Python code for it?

"""
Function
--------
rating

Parameters
----------
df: Dataframe
    The dataframe of reviews such as smalldf
dbase : instance of Database class.
    A database of similarities, on which the get method can be used to get the similarity
  of two businessed. e.g. dbase.get(rid1,rid2)
restaurant_id : string
    The id of the restaurant whose nearest neighbors we want
user_id : string
    The id of the user, in whose reviewed restaurants we want to find the neighbors
k : int
    the number of nearest neighbors desired, default 7
reg: float
    the regularization.

Returns
--------
A float
    which is the impued rating that we predict that user_id will make for restaurant_id
"""
#your code here
def rating(df, dbase, restaurant_id, user_id, k=7, reg=3.):
    mu=df.stars.mean()
    users_reviews=df[df.user_id==user_id]
    nsum=0.
    scoresum=0.
    nears=knearest_amongst_userrated(restaurant_id, user_id, df, dbase, k=k, reg=reg)
    restaurant_mean=df[df.business_id==restaurant_id].business_avg.values[0]
    user_mean=users_reviews.user_avg.values[0]
    scores=[]
    for r,s,nc in nears:
        scoresum=scoresum+s
        scores.append(s)
        r_reviews_row=users_reviews[users_reviews['business_id']==r]
        r_stars=r_reviews_row.stars.values[0]
        r_avg=r_reviews_row.business_avg.values[0]
        rminusb=(r_stars - (r_avg + user_mean - mu))
        nsum=nsum+s*rminusb
    baseline=(user_mean +restaurant_mean - mu)
    #we might have nears, but there might be no commons, giving us a pearson of 0
    if scoresum > 0.:
        val =  nsum/scoresum + baseline
    else:
        val=baseline
    return val
rating(smalldf,db, '53YGfwmbW73JhFiemNeyzQ', '7cR92zkDv4W3kqzii6axvg', k =7, reg = 3 )


I am working on the file "HW4_solutions.ipynb".
The first task is:
2.4 For each review in the data set, obtain a prediction from the entire dataframe smalldf. Use the function compare_results above to plot the predicted ratings against the observed ones. Make 4 such graphs, at k=3 and k=10, and for reg=3. and reg=15.
Note that this analysis is not strictly a model check because we are testing on the training set. However, since the user averages would change each time a cross-validation split was done on the set, we would incur the prohibitive expense of redoing the database each time. This would be better done on a cluster, using map-reduce or other techniques. While we explore map-reduce later in this homework, we shall not do any cross-validation.
Explain the results you get in the graphs in words.
Can you write Python code for it?

#your code here
def make_results_plot(df,k,reg):
    uid=smalldf.user_id.values
    bid=smalldf.business_id.values
    actual=smalldf.stars.values
    predicted=np.zeros(len(actual))
    counter=0
    for user_id, biz_id in zip(uid,bid):
        predicted[counter]=rating(smalldf, db, biz_id, user_id, k=k, reg=reg)
        counter=counter+1
    compare_results(actual, predicted)

#your code here
print "k=3, reg=3."
make_results_plot(smalldf,3,3.)
plt.title("k=3, reg=3.")

print "k=3, reg=15."
make_results_plot(smalldf,3,15.,)
plt.title("k=3, reg=15.")

print "k=10, reg=3."
make_results_plot(smalldf,10,3.)
plt.title("k=10, reg=3.")

print "k=10, reg=15."
make_results_plot(smalldf,10,15.,)
plt.title("k=10, reg=15.")


I am working on the file "HW4_solutions.ipynb".
The first task is:
4.2 Lets test that our results are overall the same as before
Can you write Python code for it?

sums=0.
count=0
for k in db.uniquebizids.keys():
    for k2 in db.uniquebizids.keys():
        count=count+1
        sums=sums+db.get(k,k2)[0]-db_mrjob_local.get(k,k2)[0]
print sums, count


I am working on the file "HW4_solutions.ipynb".
The first task is:
As you can see, even though we chose a subset of the dataframe in which every restaurant had 150 reviews and every user had atleast made 60, the common support of most pairs of restaurants is really low, indeed less than 10!.
Calculating Similarity
Users rate restaurants on a scale of 1-5. Even though this rating is integer valued, for the purposes of this assignment we shall treat it as a real number.
Even though each reviewer uses the same 5-star scale when rating restaurants, comparing two users by comparing their raw user ratings can be problematic. Consider a user whose average rating is 2. This is a curmudgeonly user. Consider another whose average rating is 4. This is a rather enthusiastic one. How should we compare a 3 rating by the curmudgeonly one to a 5 rating of the enthusiastic one?
It is for this purpose that we must subtract the average rating of the user from the actual rating of the restaurants in computing the similarity of two restaurants. This makes the above ratings by the two users comparable. We do this in the function pearson_sim defined below.
If there is no common support (n_common=0), we have no basis for making a similarity estimate, and so we set the similarity to 0. In the case that the individual restaurant rating variance is 0, such as in the case where there is only one common reviewer (n_common=1), we return the NaN that the scipy pearsonr returns. We will deal with it soon,
Can you write Python code for it?

from scipy.stats.stats import pearsonr
def pearson_sim(rest1_reviews, rest2_reviews, n_common):
    """
    Given a subframe of restaurant 1 reviews and a subframe of restaurant 2 reviews,
    where the reviewers are those who have reviewed both restaurants, return
    the pearson correlation coefficient between the user average subtracted ratings.
    The case for zero common reviewers is handled separately. Its
    ok to return a NaN if any of the individual variances are 0.
    """
    if n_common==0:
        rho=0.
    else:
        diff1=rest1_reviews['stars']-rest1_reviews['user_avg']
        diff2=rest2_reviews['stars']-rest2_reviews['user_avg']
        rho=pearsonr(diff1, diff2)[0]
    return rho


I am working on the file "HW4_solutions.ipynb".
The first task is:
Checking the results
Let us load the data from the file
Can you write Python code for it?

output_small_local=[[json.loads(j) for j in line.strip().split("\t")] for line in open("./output.small.local.txt")]
output_small_local[0]


I am working on the file "HW4_solutions.ipynb".
The first task is:
Common Support
Lets now make a histogram of the common user support (the number of common reviewers) of each pair of restaurants on the smaller set, and print the mean. Pay attention to the code, as you will use parts of it later. (This code takes a bit of time to run, so be patient).
The common support is an important concept, as for each pair of restaurants, its the number of people who reviewed both. It will be used to modify similarity between restaurants. If the common support is low, the similarity is less believable.
Can you write Python code for it?

restaurants=smalldf.business_id.unique()
supports=[]
for i,rest1 in enumerate(restaurants):
    for j,rest2 in enumerate(restaurants):
        if  i < j:
            rest1_reviewers = smalldf[smalldf.business_id==rest1].user_id.unique()
            rest2_reviewers = smalldf[smalldf.business_id==rest2].user_id.unique()
            common_reviewers = set(rest1_reviewers).intersection(rest2_reviewers)
            supports.append(len(common_reviewers))
print "Mean support is:",np.mean(supports)
plt.hist(supports)


I am working on the file "HW4_solutions.ipynb".
The first task is:
Explanation for those funny yield keywords
The functions above “yield” values, and do not “return” them. They are generators. Here is an example:
Can you write Python code for it?

def upper_generator(words):
    for word in words:
        yield word.upper()

words = ['a', 'couple', 'of', 'words', 'to', 'process']

print upper_generator(words)
print list(upper_generator(words))
for u in upper_generator(words):
     print u


I am working on the file "HW4_solutions.ipynb".
The first task is:
For the top-recommendations in the variable toprecos from the previous section, we compute the predicted rating and compare it with the average rating over all users available inside the tuples that make up toprecos. We use a k of 7 and regularization 3. For comparision we also print this users' average rating. Do you notice anything interesting about how the order has changed from when we did this with the global similarities? (for you to think, not to answer)
Can you write Python code for it?

print "User Average", smalldf[smalldf.user_id==testuserid].stars.mean(),"for",usernamefromid(smalldf,testuserid)
print "Predicted ratings for top choices calculated earlier:"
for biz_id,biz_avg in toprecos:
    print biznamefromid(smalldf, biz_id),"|",rating(smalldf, db, biz_id, testuserid, k=7, reg=3.),"|","Average",biz_avg