I am working on the file "HW2.ipynb".
The first task is:
Answer:
The sharpness of the classifier boundary, as defined by the closeness of the contours near a probability of 0.5 gives us a sense of precision. Imagine that the boundary is very tight, tighter than what you can see in the graph. Then most states will be away from the 0.5 line, and the spread in the results will be less, or the precision higher. This is not the only consideration: indeed one must ask, how many states fall smack into the middle, say between 0.3 and 0.7. The more that do, the less precise the results will be, as there will be a greater number of simulations in which they will cross-over to another party.
To assess accuracy, we would have to see the actual outcome of the states in 2012. Accuracy would be a function of the number of states that end up on the "wrong" side of the 0.5 line, and how deep they end up on the wrong side. In terms of characterizing the 2008 outcomes, it seems that the classifier is quit eaccurate, with most misclassifications appearing in grey area of the classification boundary.
Question 3: Trying to catch Silver: Poll Aggregation
In the previous section, we tried to use heterogeneous side-information to build predictions of the election outcome. In this section, we switch gears to bringing together homogeneous information about the election, by aggregating different polling result together.
This approach -- used by the professional poll analysists -- involves combining many polls about the election itself. One advantage of this approach is that it addresses the problem of bias in individual polls, a problem we found difficult to deal with in problem 1. If we assume that the polls are all attempting to estimate the same quantity, any individual biases should cancel out when averaging many polls (pollsters also try to correct for known biases). This is often a better assumption than assuming constant bias between election cycles, as we did above.
The following table aggregates many of the pre-election polls available as of October 2, 2012. We are most interested in...
Can you write Python code for it?

multipoll = pd.read_csv('data/cleaned-state_data2012.csv', index_col=0)

#convert state abbreviation to full name
multipoll.State.replace(states_abbrev, inplace=True)

#convert dates from strings to date objects, and compute midpoint

multipoll.start_date = pd.to_datetime(multipoll.start_date)
multipoll.end_date = pd.to_datetime(multipoll.end_date)
multipoll['poll_date'] = multipoll.start_date + (multipoll.end_date - multipoll.start_date).values / 2

#compute the poll age relative to Oct 2, in days
multipoll['age_days'] = (today - multipoll['poll_date']).values / np.timedelta64(1, 'D')

#drop any rows with data from after oct 2
multipoll = multipoll[multipoll.age_days > 0]

#drop unneeded columns
multipoll = multipoll.drop(['Date', 'start_date', 'end_date', 'Spread'], axis=1)

#add electoral vote counts
multipoll = multipoll.join(electoral_votes, on='State')

#drop rows with missing data
multipoll.dropna()

multipoll.head()


I am working on the file "HW2.ipynb".
The first task is:
Answer: The accuracy of this poll is somewhat greater than just taking the gallup poll. This is probably because

We're using as inputs polls that are trying to predict the election, so there is nothing "lost in translation", and
We are averaging many polls together, so some of their biases are likely to offset each other.

One problem is that we treated all polls as equal. Thus a bad poll with a small sample size is given equal footing as a good poll with a large one. Thus we are introducing bias both due to individual poll biases and individual poll sampling errors.
Furthermore, we estimate the standard deviation by simply taking the spread of the percentages in the polls, without considering their individual margins of error. In the limit of very limit sampling error per poll, this might be ok. However in states with some polls with large margins of error (Kansas, for eg), this can lead to an overestimate of the standard deviation, pushing the predicted probability toward 0.5. This, in turn, can hurt precision since it increases the chance that a state will flip sides in a simulation.
Can you write Python code for it?

#your code here
make_map(model.Obama, "P(Obama): Poll Aggregation")
plt.show()


I am working on the file "HW2.ipynb".
The first task is:
Background: The Electoral College
US Presidential elections revolve around the <a href="http://en.wikipedia.org/wiki/Electoral_College_(United_States)"> Electoral College </a>. In this system, each state receives a number of Electoral College votes depending on it's population -- there are 538 votes in total. In most states, all of the electoral college votes are awarded to the presidential candidate who recieves the most votes in that state. A candidate needs 269 votes to be elected President.
Thus, to calculate the total number of votes a candidate gets in the election, we add the electoral college votes in the states that he or she wins. (This is not entirely true, with Nebraska and Maine splitting their electoral college votes, but, for the purposes of this homework, we shall assume that the winner of the most votes in Maine and Nebraska gets ALL the electoral college votes there.)
Here is the electoral vote breakdown by state:
As a matter of convention, we will index all our dataframes by the state name
Can you write Python code for it?

electoral_votes = pd.read_csv("data/electoral_votes.csv").set_index('State')
electoral_votes.head(50)


I am working on the file "HW2.ipynb".
The first task is:
Check Rook

Given a square matrix (i.e. a chess board) containing a character 'R' (the rook) and a character 'P' (the other piece),
return True if the rook can take the piece, and False otherwise.
Restriction(s):

Every input matrix will be square and contain 'R' and 'P' exactly once each.
Example(s):

Example 1:

Input:

arr: [
[ '', '', '', 'P' ],
 [ '', '', '', '' ],
 [ '', '', '', 'R' ],
 [ '', '', '', '']]

Output:

True

Example 2:

Input:
arr: [
[ '', '', '', 'P' ],
 [ 'R', '', '', '' ],
 [ '', '', '', '' ],
 [ '', '', '', '']]
Output:
False

Parameters

list : board
the square matrix to be iterated through

Returns

bool:
whether the rook can take the piece or not
Can you write Python code for it?

def check_rook(board):
    """
    Checks if the rook can take the other piece, given a square matrix and 2 characters within that matrix.
    """
    pass

def test_check_rook():
    assert check_rook([['', 'P'], ['', 'R']]) is True
    assert check_rook([['', 'P'], ['R', '']]) is False
    assert check_rook([['', '', 'P'], ['', '', ''], ['', '', 'R']]) is True


I am working on the file "HW2.ipynb".
The first task is:
Each row lists a state, the percent of surveyed individuals who identify as Democrat/Republican, the percent whose identification is unknown or who haven't made an affiliation yet, the margin between Democrats and Republicans (Dem_Adv: the percentage identifying as Democrats minus the percentage identifying as Republicans), and the number N of people surveyed.
1.6 This survey can be used to predict the outcome of each State's election. The simplest forecast model assigns 100% probability that the state will vote for the majority party.  Implement this simple forecast.
Can you write Python code for it?

"""
Function
--------
simple_gallup_model

A simple forecast that predicts an Obama (Democratic) victory with
0 or 100% probability, depending on whether a state
leans Republican or Democrat.

Inputs
------
gallup : DataFrame
    The Gallup dataframe above

Returns
-------
model : DataFrame
    A dataframe with the following column
     * Obama: probability that the state votes for Obama. All values should be 0 or 1
    model.index should be set to gallup.index (that is, it should be indexed by state name)

Examples
---------
>>> simple_gallup_model(gallup_2012).ix['Florida']
Obama    1
Name: Florida, dtype: float64
>>> simple_gallup_model(gallup_2012).ix['Arizona']
Obama    0
Name: Arizona, dtype: float64
"""

#your code here
def simple_gallup_model(gallup):
    return pd.DataFrame(dict(Obama=(gallup_2012.Dem_Adv > 0).astype(float)))


I am working on the file "HW2.ipynb".
The first task is:
Exploratory Data Analysis
2.2 Lets do a little exploratory data analysis. Plot a scatter plot of the two PVi's against each other. What are your findings? Is the partisan vote index relatively stable from election to election?
Can you write Python code for it?

#your code here
plt.plot(e2008.pvi, e2012.pvi, 'o', label='Data')
fit = np.polyfit(e2008.pvi, e2012.pvi, 1)
x = np.linspace(-40, 80, 10)
y = np.polyval(fit, x)
plt.plot(x, x, '--k', alpha=.3, label='x=y')
plt.plot(x, y, label='Linear fit')
plt.xlabel('2008 Partisan Vote Index (PVI)');
plt.ylabel('2012 Partisan Vote Index (PVI)');
plt.title('PVI 2008 vs PVI 2012');
plt.legend(loc='upper left')


I am working on the file "HW2.ipynb".
The first task is:
Finally we write the function that we use to make our fits. It takes both the 2008 and 2012 frame as arguments, as well as the featurelist, and the number of cross-validation folds to do. It uses the above defined logistic_score to find the best-fit C, and then uses this value to return the tuple of result dataframe and classifier described above. This is the function you will be using.
Can you write Python code for it?

def cv_and_fit(frame2008, frame2012, featureslist, n_folds=5):
    bp, bs = cv_optimize(frame2008, featureslist, n_folds=n_folds)
    predict, clf = fit_logistic(frame2008, frame2012, featureslist, reg=bp['C'])
    return predict, clf


I am working on the file "HW2.ipynb".
The first task is:
Flatten List

Given a list of lists, return a new list with all the elements of the old lists in sequential order
Restriction(s):

The contents of the lists may be any valid Python object.
The lists themselves will always be a list, and will never be None.
Example(s):

Example 1:

Input:

arr: [[1,2],[3,4]]

Output:

[1,2,3,4]

Example 2:

Input:
arr: [[1,'a','bbbbb'],[2],[3,4,5,[3,4]]]

Output:
[1,'a','bbbbb',2,3,4,5,[3,4]]

Parameters

list : arr
the list of lists to flatten

Returns

list:
the new flattened list
Can you write Python code for it?


I am working on the file "HW2.ipynb".
The first task is:
Here is some code to plot State Chloropleth maps in matplotlib. make_map is the function you will use.
Can you write Python code for it?

#adapted from  https://github.com/dataiap/dataiap/blob/master/resources/util/map_util.py

#load in state geometry
state2poly = defaultdict(list)

data = json.load(file("data/us-states.json"))
for f in data['features']:
    state = states_abbrev[f['id']]
    geo = f['geometry']
    if geo['type'] == 'Polygon':
        for coords in geo['coordinates']:
            state2poly[state].append(coords)
    elif geo['type'] == 'MultiPolygon':
        for polygon in geo['coordinates']:
            state2poly[state].extend(polygon)

def draw_state(plot, stateid, **kwargs):
    """
    draw_state(plot, stateid, color=..., **kwargs)

    Automatically draws a filled shape representing the state in
    subplot.
    The color keyword argument specifies the fill color.  It accepts keyword
    arguments that plot() accepts
    """
    for polygon in state2poly[stateid]:
        xs, ys = zip(*polygon)
        plot.fill(xs, ys, **kwargs)

def make_map(states, label):
    """
    Draw a cloropleth map, that maps data onto the United States

    Inputs
    -------
    states : Column of a DataFrame
        The value for each state, to display on a map
    label : str
        Label of the color bar

    Returns
    --------
    The map
    """
    fig = plt.figure(figsize=(12, 9))
    ax = plt.gca()

    if states.max() < 2: # colormap for election probabilities
        cmap = cm.RdBu
        vmin, vmax = 0, 1
    else:  # colormap for electoral votes
        cmap = cm.binary
        vmin, vmax = 0, states.max()
    norm = mpl.colors.Normalize(vmin=vmin, vmax=vmax)

    skip = set(['National', 'District of Columbia', 'Guam', 'Puerto Rico',
                'Virgin Islands', 'American Samoa', 'Northern Mariana Islands'])
    for state in states_abbrev.values():
        if state in skip:
            continue
        color = cmap(norm(states.ix[state]))
        draw_state(ax, state, color = color, ec='k')

    #add an inset colorbar
    ax1 = fig.add_axes([0.45, 0.70, 0.4, 0.02])
   ...


I am working on the file "HW2.ipynb".
The first task is:
Later on in this homework we will explore some approaches to estimating probabilities like these and quatifying our uncertainty about them. But for the time being, we will focus on how to make a prediction assuming these probabilities are known.
Even when we assume the win probabilities in each state are known, there is still uncertainty left in the election. We will use simulations from a simple probabilistic model to characterize this uncertainty. From these simulations, we will be able to make a prediction about the expected outcome of the election, and make a statement about how sure we are about it.
1.2 We will assume that the outcome in each state is the result of an independent coin flip whose probability of coming up Obama is given by a Dataframe of state-wise win probabilities. Write a function that uses this predictive model to simulate the outcome of the election given a Dataframe of probabilities.
Can you write Python code for it?

"""
Function
--------
simulate_election

Inputs
------
model : DataFrame
    A DataFrame summarizing an election forecast. The dataframe has 51 rows -- one for each state and DC
    It has the following columns:
       Obama : Forecasted probability that Obama wins the state
       Votes : Electoral votes for the state
    The DataFrame is indexed by state (i.e., model.index is an array of state names)

n_sim : int
   Number of simulations to run

Returns
-------
results : Numpy array with n_sim elements
   Each element stores the number of electoral college votes Obama wins in each simulation.
"""

#Your code here
def simulate_election(model, n_sim):
    simulations = np.random.uniform(size= (51,n_sim))
    obama_votes = (simulations < model.Obama.values.reshape(-1,1))* model.Votes.values.reshape(-1,1)
    results  = obama_votes.sum(axis = 0)
    return results


I am working on the file "HW2.ipynb".
The first task is:
Let us also load in data about the 2004 elections from p04.csv which gets the results in the above form for the 2004 election for each state.
Can you write Python code for it?

polls04=pd.read_csv("data/p04.csv")
polls04.State=polls04.State.replace(states_abbrev)
polls04.set_index("State", inplace=True);
polls04.head()

pvi08=polls04.Dem - polls04.Rep - (national_results.xs(2004)['Dem'] - national_results.xs(2004)['Rep'])
pvi08.head()


I am working on the file "HW2.ipynb".
The first task is:
Lets call the function on the multipoll data frame, and join it with the electoral_votes frame.
Can you write Python code for it?

avg = state_average(multipoll).join(electoral_votes, how='outer')
avg.head()


I am working on the file "HW2.ipynb".
The first task is:
Lets plot the result of the Predictwise simulation. Your plot should look something like this:
<img src="http://i.imgur.com/uCOFXHp.png">
Can you write Python code for it?


I am working on the file "HW2.ipynb".
The first task is:
Note that even a small bias can have a dramatic effect on the predictions. Pundits made a big fuss about bias during the last election, and for good reason -- it's an important effect, and the models are clearly sensitive to it. Forecastors like Nate Silver would have had an easier time convincing a wide audience about their methodology if bias wasn't an issue.
Furthermore, because of the nature of the electoral college, biases get blown up large. For example, suppose you mis-predict the party Florida elects. We've possibly done this as a nation in the past :-). Thats 29 votes right there. So, the penalty for even one misprediction is high.
Estimating the size of the bias from the 2008 election
While bias can lead to serious inaccuracy in our predictions, it is fairly easy to correct if we are able to estimate the size of the bias and adjust for it. This is one form of calibration.
One approach to calibrating a model is to use historical data to estimate the bias of a prediction model. We can use our same prediction model on historical data and compare our historical predictions to what actually occurred and see if, on average, the predictions missed the truth by a certain amount. Under some assumptions (discussed in a question below), we can use the estimate of the bias to adjust our current forecast.
In this case, we can use data from the 2008 election. (The Gallup data from 2008 are from the whole of 2008, including after the election):
Can you write Python code for it?

gallup_08 = pd.read_csv("data/g08.csv").set_index('State')
results_08 = pd.read_csv('data/2008results.csv').set_index('State')
prediction_08 = gallup_08[['Dem_Adv']]
prediction_08['Dem_Win']=results_08["Obama Pct"] - results_08["McCain Pct"]
prediction_08.head()


I am working on the file "HW2.ipynb".
The first task is:
Notice that a lot of states in which Gallup reported a Democratic affiliation, the results were strongly in the opposite direction. Why might that be? You can read more about the reasons for this here.
A quick look at the graph will show you a number of states where Gallup showed a Democratic advantage, but where the elections were lost by the democrats. Use Pandas to list these states.
Can you write Python code for it?

#your code here
prediction_08[(prediction_08.Dem_Adv > 0) & (prediction_08.Dem_Win <0)]


I am working on the file "HW2.ipynb".
The first task is:
Now, we run the simulation with this model, and plot it.
Can you write Python code for it?

model = simple_gallup_model(gallup_2012)
model = model.join(electoral_votes)

prediction = simulate_election(model, 10000)

plot_simulation(prediction)
plt.show()
make_map(model.Obama, "P(Obama): Simple Model")


I am working on the file "HW2.ipynb".
The first task is:
Orthogonal Vectors

Create a program to indicate whether or not the column vectors in a matrix are orthogonal, where the column vectors of a matrix are the vectors <a1,a2,a3>,<b1,b2,b3>,<c1,c2,c3> contained in the matrix:
|a1,b1,c1|
|a2,b2,c2|
|a3,b3,c3|

Restriction(s):

Input is an 3x3 array of integers.
No values in the matrix will be larger than 1000.
You may not use functions other than those in Python 3 by default

Example(s):

Example 1:

Input:

matrix: [
[ 1, 2, 3],
[-2, 1,-4],
[-3, 4, 1]]
size : 3

Output:

false

Example 2:

Input:

matrix: [
[ 1, 2, 4],
[-1,-1, 7],
[ 3,-1,1]]

Output:

true

Example 3:

Input:
matrix: [
[ 0,25,-1],
[ 0, 4,10],
[ 0, 5,-3]]

Output:
true *Note that the zero vector is orthogonal to all vectors

Parameters

[[int]] : matrix
Representation, in row-major form of matrix containing column vectors

Returns

bool: True/False depending on whether or not vectors are orthogonal

Can you write Python code for it?

def orthogonal_vectors(matrix):
    '''
    Compute whether or not all of the column vectors of a matrix are orthogonal to one another
    '''
    pass

def test_orthogonal_vectors():
    assert not orthogonal_vectors([[1,2,3],[-2,1,-4],[-3,4,1]])
    assert orthogonal_vectors([[1,2,4],[-1,-1,7],[3,-1,1]])
    assert orthogonal_vectors([[0,25,-1],[0,4,10],[0,5,-3]])


I am working on the file "HW2.ipynb".
The first task is:
Some of the reddest and bluest states are not present in this data (people don't bother polling there as much). The default_missing function gives them strong Democratic/Republican advantages
Can you write Python code for it?

def default_missing(results):
    red_states = ["Alabama", "Alaska", "Arkansas", "Idaho", "Wyoming"]
    blue_states = ["Delaware", "District of Columbia", "Hawaii"]
    results.ix[red_states, ["poll_mean"]] = -100.0
    results.ix[red_states, ["poll_std"]] = 0.1
    results.ix[blue_states, ["poll_mean"]] = 100.0
    results.ix[blue_states, ["poll_std"]] = 0.1
default_missing(avg)
avg


I am working on the file "HW2.ipynb".
The first task is:
Sort 2D matrix

Given a 2D matrix, return a new matrix of the same dimensions as the old whose contents are the sorted equivalents of the old matrix.
Restriction(s):

The matrix will be rectangular, i.e. the sublists will all be of the same length.
The dimensions of the matrix are strictly positive.
Example(s):

Example:

Input:

arr: [[5,2],[37,4]]

Output:

[[2,4],[5,37]]

Example 2:
Input:
arr: [[5,4,20], [9,-1,19], [91,-19,1]]

Output:
[[-19, -1, 1], [4, 5, 9], [19, 20, 91]]

Parameters

mat : arr
the matrix to sort

Returns

list:
the new, sorted matrix
Can you write Python code for it?


I am working on the file "HW2.ipynb".
The first task is:
Spiral Matrix

Given a matrix of m x n diemensions (m rows, n columns), return all elements of the matrix in spiral order.
Hint

Draw it out! How should we traverse the 2D list in order to get the output desired?

Example(s):

Example 1:

Input:

arr: [
[ 1, 2, 3 ],
 [ 4, 5, 6 ],
 [ 7, 8, 9 ]]

Output:

[1,2,3,6,9,8,7,4,5]

Example 2:

Input:
arr: [
[1, 2, 3, 4],
 [2, 4, 9, 5],
 [9, 8, 7, 6]]
Output:
[1,2,3,4,5,6,7,8,9,2,4,9]

Parameters

list : arr
the 2D matrix to be returned in spiral order

Returns

list:
the 2D matrix in spiral order
Can you write Python code for it?


I am working on the file "HW2.ipynb".
The first task is:
The bias was roughly 8% in favor of the Democrats in the Gallup Poll, meaning that you would want to adjust predictions based on this poll down by that amount. This was the result of people in a number of Southern and Western states claiming to be affiliated as Democrats, then voting the other way. Or, since Gallup kept polling even after the elections, it could also represent people swept away by the 2008 election euphoria in those states. This is an illustration of why one needs to be carefull with polls.
1.13 * Calibrate your forecast of the 2012 election using the estimated bias from 2008. Validate the resulting model against the real 2012 outcome. Did the calibration help or hurt your prediction?*
Can you write Python code for it?

#your code here
model = biased_gallup_poll(gallup_2012, 8.06 )
model = model.join(electoral_votes)
prediction = simulate_election(model, 10000)
plot_simulation(prediction)
plt.xlim(150,350)


I am working on the file "HW2.ipynb".
The first task is:
The following cells takes the necessary DataFrame for the Predictwise data, and runs 10000 simulations. We use the results to compute the probability, according to this predictive model, that Obama wins the election (i.e., the probability that he receives 269 or more electoral college votes)
Can you write Python code for it?

result = simulate_election(predictwise, 10000)
result

#compute the probability of an Obama win, given this simulation
#Your code here
print (result >=269).mean()


I am working on the file "HW2.ipynb".
The first task is:
There are only two polls in the last year! And, the results in the two polls are far, very far from the mean.
Now, Kansas is a safely Republican state, so this dosent really matter, but if it were a swing state, we'd be in a pickle. We'd have no unbiased estimator of the variance in Kansas. So, to be conservative, and play it safe, we follow the same tack we did with the unweighted averaging of polls, and simply assume that the variance in a state is the square of the standard deviation of obama_spread.
This will overestimate the errors for a lot of states, but unless we do a detailed state-by-state analysis, its better to be conservative. Thus, we use:
$\textrm{Var}(\mu)$ = obama_spread.std()$^2$ .
The weights $w_i$ should combine the uncertainties from the margin of error and the age of the forecast. One such combination is:
$$
w_i = \frac1{MoE^2} \times \lambda_{\rm age}
$$
where
$$
\lambda_{\rm age} = 0.5^{\frac{{\rm age}}{30 ~{\rm days}}}
$$
This model makes a few ad-hoc assumptions:

The equation for $\sigma$ assumes that every measurement is independent. This is not true in the case that a given pollster in a state makes multiple polls, perhaps with some of the same respondents (a longitudinal survey). But its a good assumption to start with.
The equation for $\lambda_{\rm age}$ assumes that a 30-day old poll is half as valuable as a current one

3.4 Nevertheless, it's worth exploring how these assumptions affect the forecast model. Implement the model in the function weighted_state_average
Can you write Python code for it?

"""
Function
--------
weighted_state_average

Inputs
------
multipoll : DataFrame
   The multipoll data above

Returns
-------
averages : DataFrame
  A dataframe, indexed by State, with the following columns:
     N: Number of polls averaged together
     poll_mean: The average value for obama_spread for all polls in this state
     poll_std: The standard deviation of obama_spread

Notes
-----
For states where poll_std isn't finite (because N is too small), estimate the
poll_std value as .05 * poll_mean
"""

#your code here
def  weights(df):
    decay = 0.5**(df.age_days/30)
    w = decay / df.MoE ** 2
    return w

def wmean(df):
    w = weights(df)
    result = (df.obama_spread*w).sum() / w.sum()
    return result

def wsig(df):
    return df.obama_spread.std()

def weighted_state_average(multipoll):
    groups = multipoll.groupby('State')
    poll_mean = groups.apply(wmean)
    poll_std = groups.apply(wsig)
    poll_std[poll_std.isnull()] = 0.05*mean[poll_std.isnull()]
    return pd.DataFrame(dict(poll_mean = poll_mean, poll_std = poll_std))


I am working on the file "HW2.ipynb".
The first task is:
To illustrate the use of make_map we plot the Electoral College
Can you write Python code for it?


I am working on the file "HW2.ipynb".
The first task is:
Unweighted aggregation
3.2 Build an aggregated_poll_model function that takes the avg DataFrame as input, and returns a forecast DataFrame
in the format you've been using to simulate elections. Assume that the probability that Obama wins a state
is given by the probability that a draw from a Gaussian with $\mu=$poll_mean and $\sigma=$poll_std is positive.
Can you write Python code for it?

"""
Function
--------
aggregated_poll_model

Inputs
------
polls : DataFrame
   DataFrame indexed by State, with the following columns:
      poll_mean
      poll_std
      Votes

Returns
-------
A DataFrame indexed by State, with the following columns:
   Votes: Electoral votes for that state
   Obama: Estimated probability that Obama wins the state
"""
#your code here
def aggregated_poll_model(polls):
    sigma = polls.poll_std
    prob = 0.5*(1 + erf(polls.poll_mean / np.sqrt(2 * sigma ** 2)))
    return pd.DataFrame(dict(Obama = prob, Votes =polls.Votes))


I am working on the file "HW2.ipynb".
The first task is:
We are not done yet. In order to estimate C, we perform a grid search over many C to find the best C that minimizes the loss function. For each point on that grid, we carry out a n_folds-fold cross-validation. What does this mean?
Suppose n_folds=10. Then we will repeat the fit 10 times, each time randomly choosing 50/10 ~ 5 states out as a test set, and using the remaining 45/46 as the training set. We use the average score on the test set to score each particular choice of C, and choose the one with the best performance.
Can you write Python code for it?

from sklearn.grid_search import GridSearchCV

def cv_optimize(frame2008, featureslist, n_folds=10, num_p=100):
    y, X = prepare_features(frame2008, featureslist)
    clf = LogisticRegression()
    parameters = {"C": np.logspace(-4, 3, num=num_p)}
    gs = GridSearchCV(clf, param_grid=parameters, cv=n_folds)
    gs.fit(X, y)
    return gs.best_params_, gs.best_score_


I am working on the file "HW2.ipynb".
The first task is:
We construct a similar frame for 2012, obtaining pvi using the 2008 Obama win data which we already have. There is no obama_win column since, well, our job is to predict it!
Can you write Python code for it?

pvi12 = e2008.Dem_Win - (national_results.xs(2008)['Dem'] - national_results.xs(2008)['Rep'])
e2012 = pd.DataFrame(dict(pvi=pvi12, Dem_Adv=gallup_2012.Dem_Adv - gallup_2012.Dem_Adv.mean()))
e2012 = e2012.sort_index()
e2012.head()


I am working on the file "HW2.ipynb".
The first task is:
We construct the model by estimating the probabilities:
Can you write Python code for it?

model = uncertain_gallup_model(gallup_2012)
model = model.join(electoral_votes)


I am working on the file "HW2.ipynb".
The first task is:
We use the above function to get the label vector and feature-sample matrix for feeding to scikit-learn. We then use the usual scikit-learn incantation fit to fit a logistic regression model with regularization parameter C. The parameter C is a hyperparameter of the model, and is used to penalize too high values of the parameter co-efficients in the loss function that is minimized to perform the logistic regression. We build a new dataframe with the usual Obama column, that holds the probabilities used to make the prediction. Finally we return a tuple of the dataframe and the classifier instance, in that order.
Can you write Python code for it?

def fit_logistic(frame2008, frame2012, featureslist, reg=0.0001):
    y, X = prepare_features(frame2008, featureslist)
    clf2 = LogisticRegression(C=reg)
    clf2.fit(X, y)
    X_new = frame2012[featureslist]
    obama_probs = clf2.predict_proba(X_new)[:, 1]

    df = pd.DataFrame(index=frame2012.index)
    df['Obama'] = obama_probs
    return df, clf2


I am working on the file "HW2.ipynb".
The first task is:
Weighted Aggregation
Not all polls are equally valuable. A poll with a larger margin of error should not influence a forecast as heavily. Likewise, a poll further in the past is a less valuable indicator of current (or future) public opinion. For this reason, polls are often weighted when building forecasts.
A weighted estimate of Obama's advantage in a given state is given by
$$
\mu = \frac{\sum w_i \times \mu_i}{\sum w_i}
$$
where $\mu_i$ are individual polling measurements or a state, and $w_i$ are the weights assigned to each poll. The uncertainty on the weighted mean, assuming each measurement is independent, is given by
The estimate of the variance of $\mu$, when $\mu_i$ are unbiased estimators of $\mu$, is
$$\textrm{Var}(\mu) = \frac{1}{(\sum_i w_i)^2} \sum_{i=1}^n w_i^2 \textrm{Var}(\mu_i).$$
Whats the matter with Kansas?
We need to find an estimator of the variance of $\mu_i$, $Var(\mu_i)$. In the case of states that have a lot of polls, we expect the bias in $\mu$ to be negligible, and then the above formula for the variance of $\mu$ holds. However, lets take a look at the case of Kansas.
Can you write Python code for it?


I am working on the file "HW2.ipynb".
The first task is:
Winner of Tic Tac Toe

Given a complteted Tic Tac Toe game as 2D matrix, return the winner of the game.
Restriction(s):

There will either be one winner or no winner, no input where X and O are both winners.
Make sure the returned string is EXACTLY the same as the outputs we give. Failure to do so will result you in getting the test cases wrong.

Example(s):

Example 1:

Input:

game: [
['X','X','O'],
['O','X','O'],
['X','O','O']]

Output:

O wins!

Example 2:

Input:

game: [
['X','O','X'],
['O','X','O'],
['X','O','X']]

Output:

X wins!

Example 3:

Input:
game: [
['O','X','O'],
['O','X','O'],
['X','O','X']]
Output:
Draw!

Parameters

list : game
the Tic Tac Toe game

Returns

str:
the winner of the game, if there is no winner return 'Draw!'
Can you write Python code for it?

def winner_of_tic_tac_toe(game):
    """
    Print winner of tic tac toe game
    """
    pass

def test_winner_of_tic_tac_toe():
    assert winner_of_tic_tac_toe([['X','O','X'],['O','X','O'],['X','O','X']]) == "X wins!"
    assert winner_of_tic_tac_toe([['X','X','O'],['O','X','O'],['X','O','O']]) == "O wins!"


I am working on the file "HW2.ipynb".
The first task is:
your answer here
2.3 Lets do a bit more exploratory data analysis. Using a scatter plot, plot Dem_Adv against pvi in both 2008 and 2012. Use colors red and blue depending upon obama_win for the 2008 data points.  Plot the 2012 data using gray color. Is there the possibility of making a linear separation (line of separation) between the red and the blue points on the graph?
Can you write Python code for it?

#your code here
plt.xlabel('Gallup Democrat Advantage (from  mean)')
plt.ylabel('pvi')
colors = ["red","blue"]
axes = plt.gca()
for label in [0,1]:
    color = colors[label]
    mask = e2008.obama_win == label
    l = '2008 Mc Cain States' if label == 0 else '2008 Obama States'
    axes.scatter(e2008[mask]['Dem_Adv'], e2008[mask]['pvi'], c= color, s =80, label =l)
axes.scatter(e2012.Dem_Adv, e2012.pvi, c = 'black', s =80, marker ='s', alpha = .4)
plt.legend(frameon = False, scatterpoints = 1, loc = "upper left")
remove_border()


I am working on the file "HW2.ipynb".
The first task is:
your answer here
The Logistic Regression
Logistic regression is a probabilistic model that links observed binary data to a set of features.
Suppose that we have a set of binary (that is, taking the values 0 or 1) observations $Y_1,\cdots,Y_n$, and for each observation $Y_i$ we have a vector of features $X_i$. The logistic regression model assumes that there is some set of weights, coefficients, or parameters $\beta$, one for each feature, so that the data were generated by flipping a weighted coin whose probability of giving a 1 is given by the following equation:
$$
P(Y_i = 1) = \mathrm{logistic}(\sum \beta_i X_i),
$$
where
$$
\mathrm{logistic}(x) = \frac{e^x}{1+e^x}.
$$
When we fit a logistic regression model, we determine values for each $\beta$ that allows the model to best fit the training data we have observed (the 2008 election). Once we do this, we can use these coefficients to make predictions about data we have not yet observed (the 2012 election).
Sometimes this estimation procedure will overfit the training data yielding predictions that are difficult to generalize to unobserved data. Usually, this occurs when the magnitudes of the components of $\beta$ become too large. To prevent this, we can use a technique called regularization to make the procedure prefer parameter vectors that have smaller magnitude. We can adjust the strength of this regularization to reduce the error in our predictions.
We now write some code as technology for doing logistic regression. By the time you start doing this homework, you will have learnt the basics of logistic regression, but not all the mechanisms of cross-validation of data sets. Thus we provide here the code for you to do the logistic regression, and the accompanying cross-validation.
We first build the features from the 2008 data frame, returning y, the vector of labels, and X the feature-sample matrix where the columns are the features in order from the list featurelist, and each row is a data "point".
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

def prepare_features(frame2008, featureslist):
    y= frame2008.obama_win.values
    X = frame2008[featureslist].values
    if len(X.shape) == 1:
        X = X.reshape(-1, 1)
    return y, X


I am working on the file "HW2/Linear Regression.ipynb".
The first task is:
Step 1
Can you write Python code for it?

X=DataFrame(features, columns=['HiTemp', 'LoTemp', 'Precip'])
y=DataFrame(results, columns=['Man_Count'])
#training set
X_tr=X[::2]
y_tr=y[::2]
#testing set
X_ts=X[1::2]
y_ts=y[1::2]


I am working on the file "HW2/Linear Regression.ipynb".
The first task is:
Step 2
Can you write Python code for it?


I am working on the file "HW2/Linear Regression.ipynb".
The first task is:
Step 3
Can you write Python code for it?

def getMeanAbsErr(predicted, actual):
    n = len(predicted)
    totalerr = 0
    for i in range(n):
        totalerr = totalerr + abs(predicted[i]-actual[i])
    return (totalerr/n)


I am working on the file "HW2/Linear Regression.ipynb".
The first task is:
Step 4
Can you write Python code for it?

d1=DataFrame(a, columns=['Actual'])
d2=DataFrame(prd_y, columns=['Predicted'])
cmp=d1.join(d2)
print(cmp)


I am working on the file "HW2/Linear Regression.ipynb".
The first task is:
Step 5
Can you write Python code for it?

#print(getMeanAbsErr(prd_y, y_ts))
#print(cmp['Actual'])
score=model.score(X_ts, y_ts)
print(score)
err=getMeanAbsErr(cmp['Actual'], cmp['Predicted'])
print(err)
conn.close()


I am working on the file "HW2/notebooks/Q-1-1-3_Multiclass_Ridge-Copy1.ipynb".
The first task is:
Explore hyperparameters before training model on all of the training data.
Can you write Python code for it?

hyper_explorer = HyperparameterExplorer(X=train_X, y=train_y,
                                        model=RidgeMulti,
                                        validation_split=0.1, score_name = 'training RMSE',
                                        use_prev_best_weights=False,
                                        test_X=test_X, test_y=test_y)

hyper_explorer.train_model(lam=1e10, verbose=False)

hyper_explorer.train_model(lam=1e+08, verbose=False)
hyper_explorer.train_model(lam=1e+07, verbose=False)

hyper_explorer.train_model(lam=1e+06, verbose=False)

hyper_explorer.train_model(lam=1e5, verbose=False)
hyper_explorer.train_model(lam=1e4, verbose=False)
hyper_explorer.train_model(lam=1e03, verbose=False)
hyper_explorer.train_model(lam=1e2, verbose=False)

hyper_explorer.train_model(lam=1e1, verbose=False)

hyper_explorer.train_model(lam=1e0, verbose=False)
hyper_explorer.train_model(lam=1e-1, verbose=False)
hyper_explorer.train_model(lam=1e-2, verbose=False)
hyper_explorer.train_model(lam=1e-3, verbose=False)
hyper_explorer.train_model(lam=1e-4, verbose=False)
hyper_explorer.train_model(lam=1e-5, verbose=False)

hyper_explorer.summary

hyper_explorer.plot_fits()

t = time.localtime(time.time())

hyper_explorer.plot_fits(filename = "Q-1-1-3_val_and_train_RMSE_{}-{}".format(t.tm_mon, t.tm_mday))

hyper_explorer.plot_fits(ylim=(.6,.7),
                         filename = "Q-1-1-3_val_and_train_RMSE_zoomed_in{}-{}".format(t.tm_mon, t.tm_mday))

hyper_explorer.best('score')

hyper_explorer.best('summary')

hyper_explorer.best('best score')

hyper_explorer.train_on_whole_training_set()

hyper_explorer.final_model.results_row()

hyper_explorer.evaluate_test_data()


I am working on the file "HW2/notebooks/Q-1-1-3_Multiclass_Ridge-Copy1.ipynb".
The first task is:
Prepare MNIST training data
Can you write Python code for it?

train_X, train_y = mnist_training()
test_X, test_y = mnist_testing()


I am working on the file "HW2/notebooks/Q-1-1-3_Multiclass_Ridge.ipynb".
The first task is:
Explore hyperparameters before training model on all of the training data.
Can you write Python code for it?

hyper_explorer = HyperparameterExplorer(X=train_X, y=train_y,
                                        model=RidgeMulti,
                                        validation_split=0.1, score_name = 'training RMSE',
                                        use_prev_best_weights=False,
                                        test_X=test_X, test_y=test_y)

hyper_explorer.train_model(lam=1e10, verbose=False)

hyper_explorer.train_model(lam=1e+08, verbose=False)
hyper_explorer.train_model(lam=1e+07, verbose=False)

hyper_explorer.train_model(lam=1e+06, verbose=False)

hyper_explorer.train_model(lam=1e5, verbose=False)
hyper_explorer.train_model(lam=1e4, verbose=False)
hyper_explorer.train_model(lam=1e03, verbose=False)
hyper_explorer.train_model(lam=1e2, verbose=False)

hyper_explorer.train_model(lam=1e1, verbose=False)

hyper_explorer.train_model(lam=1e0, verbose=False)
hyper_explorer.train_model(lam=1e-1, verbose=False)
hyper_explorer.train_model(lam=1e-2, verbose=False)
hyper_explorer.train_model(lam=1e-3, verbose=False)
hyper_explorer.train_model(lam=1e-4, verbose=False)
hyper_explorer.train_model(lam=1e-5, verbose=False)

hyper_explorer.summary

hyper_explorer.plot_fits()

t = time.localtime(time.time())

hyper_explorer.plot_fits(filename = "Q-1-1-3_val_and_train_RMSE_{}-{}".format(t.tm_mon, t.tm_mday))

hyper_explorer.plot_fits(ylim=(.6,.7),
                         filename = "Q-1-1-3_val_and_train_RMSE_zoomed_in{}-{}".format(t.tm_mon, t.tm_mday))

hyper_explorer.best('score')

hyper_explorer.best('summary')

hyper_explorer.best('best score')

hyper_explorer.train_on_whole_training_set(lam=1e7)

hyper_explorer.final_model.results_row()

hyper_explorer.evaluate_test_data()


I am working on the file "HW2/notebooks/Q-1-2_Neural_Nets_with_a_random_first_layer.ipynb".
The first task is:
Explore hyperparameters before training model on all of the training data.
Can you write Python code for it?

hyper_explorer = HyperparameterExplorer(X=train_X, y=train_y,
                                        model=RidgeMulti,
                                        validation_split=0.1, score_name = 'training RMSE',
                                        use_prev_best_weights=False,
                                        test_X = test_X, test_y = test_y)

hyper_explorer.train_model(lam=1e10, verbose=False)

hyper_explorer.train_model(lam=1e+08, verbose=False)
hyper_explorer.train_model(lam=1e+07, verbose=False)

hyper_explorer.train_model(lam=1e+06, verbose=False)

hyper_explorer.train_model(lam=1e5, verbose=False)
hyper_explorer.train_model(lam=1e4, verbose=False)
hyper_explorer.train_model(lam=1e03, verbose=False)
hyper_explorer.train_model(lam=1e2, verbose=False)

hyper_explorer.train_model(lam=1e1, verbose=False)

hyper_explorer.train_model(lam=1e0, verbose=False)
hyper_explorer.train_model(lam=1e-1, verbose=False)
hyper_explorer.train_model(lam=1e-2, verbose=False)
hyper_explorer.train_model(lam=1e-3, verbose=False)
hyper_explorer.train_model(lam=1e-4, verbose=False)
hyper_explorer.train_model(lam=1e-5, verbose=False)

hyper_explorer.summary

hyper_explorer.plot_fits()

t = time.localtime(time.time())

hyper_explorer.plot_fits(filename = "Q-1-1-3_val_and_train_RMSE_{}-{}".format(t.tm_mon, t.tm_mday))

hyper_explorer.plot_fits(ylim=(.6,.7),
                         filename = "Q-1-1-3_val_and_train_RMSE_zoomed_in{}-{}".format(t.tm_mon, t.tm_mday))

hyper_explorer.best('score')

hyper_explorer.best('summary')

hyper_explorer.best('best score')

hyper_explorer.train_on_whole_training_set(lam=1e7)

hyper_explorer.final_model.results_row()

hyper_explorer.evaluate_test_data()


I am working on the file "HW2/notebooks/Q-1-2_multiclass_kernel_trick.ipynb".
The first task is:
Do the crazy inversion
http://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf
Can you write Python code for it?

hyper_explorer = HyperparameterExplorer(X=XR, y=train_y,
                                        model=RidgeMulti,
                                        validation_split=58./60, score_name = 'training RMSE',
                                        use_prev_best_weights=False,
                                        test_X=test_X.dot(R), test_y=test_y)

hyper_explorer.train_model(lam=100, kernelized=True, verbose=True)

# Boo.  We can't run vanilla ridge regression for a N=300 data set w/o the Kernel.
# The inversion would be 10,000 by 10,000, causing a memory error.
#hyper_explorer.train_model(lam=100, kernelized=False, verbose=True)

hyper_explorer.final_model = hyper_explorer.models[1].copy()

hyper_explorer.evaluate_test_data()


I am working on the file "HW2/notebooks/Q-1-2_multiclass_kernel_trick.ipynb".
The first task is:
Make the big random matrix
Can you write Python code for it?

R = np.random.normal(size=(train_X.shape[1], 10000))

XR = train_X.dot(R).clip(min=0)

XR = train_X.dot(R)


I am working on the file "HW2/notebooks/Q0--param_explorer_ridge_binary.ipynb".
The first task is:
Prepare MNIST training data
Can you write Python code for it?

train_X, train_y = mnist_training_binary(2)

print(train_X.shape, train_y.shape)


I am working on the file "HW2NoteBook.ipynb".
The first task is:
And plotted with the data points, we can clearly see that the cubic fit is the most accurate.
Can you write Python code for it?

p_data = p1.retrieve_pendulum_data()
plt.plot(p_data[0],p_data[1],'go')
fit1 = p1.pendulum_fit(p_data[0],p_data[1],1)
plt.plot(fit1[0],fit1[1],'y')
fit2 = p1.pendulum_fit(p_data[0],p_data[1],2)
plt.plot(fit2[0],fit2[1],'b')
fit3 = p1.pendulum_fit(p_data[0],p_data[1],3)
plt.plot(fit3[0],fit3[1],'r')

plt.show()


I am working on the file "HW2NoteBook.ipynb".
The first task is:
Finally we investigate our newly created graph function for
$f(x) = |x| , x \in [-2, 2] $ with a total of 2, 4,6 10, 13, and 20 interpolation points. (5.25)
Can you write Python code for it?


I am working on the file "HW2NoteBook.ipynb".
The first task is:
Finally, the use of numpy's sum() function:
Can you write Python code for it?

print(p2.midpointint_numpy_sum(f, 0, 10, 1000))
%timeit p2.midpointint_numpy_sum(f, 0, 10, 1000)


I am working on the file "HW2NoteBook.ipynb".
The first task is:
Homework 2
Michael Seaman
2/14/16
Problem 5.18
Polynomial fitting to expirimental data
fit_pendulum_data.py has 2 functions defined for it. The first, the retrieve function finds the file pendulum.dat and gets 2 lists of values: one of Pendulum lengths and one of their periods. They are plotted below.
Can you write Python code for it?

p_data = p1.retrieve_pendulum_data()
plt.plot(p_data[0],p_data[1],'go')

plt.show()


I am working on the file "HW2NoteBook.ipynb".
The first task is:
Next employs the built-in python sum() function to do the same thing.
Can you write Python code for it?

print(p2.midpointint_python_sum(f, 0, 10, 1000))
%timeit p2.midpointint_python_sum(f, 0, 10, 1000)


I am working on the file "HW2NoteBook.ipynb".
The first task is:
Now we'll try the same thing, but now with graphing (5.24). First we graph sin(x) with its interpolated points
Can you write Python code for it?


I am working on the file "HW2NoteBook.ipynb".
The first task is:
Problem 5.23 - 5.25 Implementing Lagrange's interpolation formula
Inorder to approximate points on given a set of data, we can fit a polynomial function passing through all the points and then evaulate it for new incoming inputs. This process is known as interpolation. Excercises 5.23 - 5.25 use Lagrange's interpolation formula - finding a polynomial that can be written as:
$$p_L(x) = \sum_{k=0}^{n}y_kL_k(x) $$ where $$ L_k(x) = \prod_{i=0,i \neq k}^{n}\frac{x-x_i}{x_k - x_i}$$
Here we have 5 points plotted on $y = sin(x)$ from [0,$\pi$]:
Can you write Python code for it?

xp = np.linspace(0, math.pi, 5)
sin_vectorized = np.vectorize(math.sin)
yp = sin_vectorized(xp)

print xp
print yp


I am working on the file "HW2_solutions.ipynb".
The first task is:
1.1 Each row is the probability predicted by Predictwise that Romney or Obama would win a state. The votes column lists the number of electoral college votes in that state. Use make_map to plot a map of the probability that Obama wins each state, according to this prediction.
Can you write Python code for it?

#your code here
make_map(predictwise.Obama, "P(Obama): PredictWise")


I am working on the file "HW2_solutions.ipynb".
The first task is:
2.4 *Carry out a logistic fit using the cv_and_fit function developed above. As your featurelist use the features we have: Dem_Adv and pvi.
Can you write Python code for it?

#your code here
res, clf = cv_and_fit(e2008, e2012, ['Dem_Adv', 'pvi'])
predict2012_logistic = res.join(electoral_votes)
predict2012_logistic.head()


I am working on the file "HW2_solutions.ipynb".
The first task is:
2.5 As before, plot a histogram and map of the simulation results, and interpret the results in terms of accuracy and precision.
Can you write Python code for it?

#code to make the histogram
#your code here

prediction = simulate_election(predict2012_logistic, 10000)
plot_simulation(prediction)

#code to make the map
#your code here

make_map(predict2012_logistic.Obama, "P(Obama): Logistic")


I am working on the file "HW2_solutions.ipynb".
The first task is:
2.6 Plot your results on the classification space boundary plot. How sharp is the classification boundary, and how does this translate into accuracy and precision of the results?
Can you write Python code for it?

#your code here
points_plot(e2008, e2012, clf)
plt.xlabel("Dem_Adv (from mean)")
plt.ylabel("PVI")


I am working on the file "HW2_solutions.ipynb".
The first task is:
3.5 Put this all together -- compute a new estimate of poll_mean and poll_std for each state, apply the default_missing function to handle missing rows, build a forecast with aggregated_poll_model, run 10,000 simulations, and plot the results, both as a histogram and as a map.
Can you write Python code for it?

#your code here
average = weighted_state_average(multipoll)
average = average.join(electoral_votes, how='outer')
default_missing(average)
model = aggregated_poll_model(average)
sims = simulate_election(model, 10000)
plot_simulation(sims)
plt.xlim(250, 400)

#your map code here
make_map(model.Obama, "P(Obama): Weighted Polls")


I am working on the file "HW2_solutions.ipynb".
The first task is:
Answer: There is a reasonably well fit line which hugs the x=y line. This tells us that the PVI seems relatively stable from election to election and may thus make a good predictor.

2.3 Lets do a bit more exploratory data analysis. Using a scatter plot, plot Dem_Adv against pvi in both 2008 and 2012. Use colors red and blue depending upon obama_win for the 2008 data points.  Plot the 2012 data using gray color. Is there the possibility of making a linear separation (line of separation) between the red and the blue points on the graph?
Can you write Python code for it?

#your code here

plt.xlabel("Gallup Democrat Advantage (from mean)")
plt.ylabel("pvi")
colors=["red","blue"]
ax=plt.gca()
for label in [0, 1]:
    color = colors[label]
    mask = e2008.obama_win == label
    l = '2008 McCain States' if label == 0 else '2008 Obama States'
    ax.scatter(e2008[mask]['Dem_Adv'], e2008[mask]['pvi'], c=color, s=60, label=l)

ax.scatter(e2012['Dem_Adv'], e2012['pvi'], c='gray', s=60,
           marker="s", label='2012 States', alpha=.3)
plt.legend(frameon=False, scatterpoints=1, loc='upper left')
remove_border()


I am working on the file "HW2_solutions.ipynb".
The first task is:
Answer: Yes, it seems that with 3-4 misclassifications, we could draw a line to divide the data into two parts.

The Logistic Regression
Logistic regression is a probabilistic model that links observed binary data to a set of features.
Suppose that we have a set of binary (that is, taking the values 0 or 1) observations $Y_1,\cdots,Y_n$, and for each observation $Y_i$ we have a vector of features $X_i$. The logistic regression model assumes that there is some set of weights, coefficients, or parameters $\beta$, one for each feature, so that the data were generated by flipping a weighted coin whose probability of giving a 1 is given by the following equation:
$$
P(Y_i = 1) = \mathrm{logistic}(\sum \beta_i X_i),
$$
where
$$
\mathrm{logistic}(x) = \frac{e^x}{1+e^x}.
$$
When we fit a logistic regression model, we determine values for each $\beta$ that allows the model to best fit the training data we have observed (the 2008 election). Once we do this, we can use these coefficients to make predictions about data we have not yet observed (the 2012 election).
Sometimes this estimation procedure will overfit the training data yielding predictions that are difficult to generalize to unobserved data. Usually, this occurs when the magnitudes of the components of $\beta$ become too large. To prevent this, we can use a technique called regularization to make the procedure prefer parameter vectors that have smaller magnitude. We can adjust the strength of this regularization to reduce the error in our predictions.
We now write some code as technology for doing logistic regression. By the time you start doing this homework, you will have learnt the basics of logistic regression, but not all the mechanisms of cross-validation of data sets. Thus we provide here the code for you to do the logistic regression, and the accompanying cross-validation.
We first build the features from the 2008 data frame, returning y, the vector of labels, and X the feature-sample matrix where the columns are the features in order from the list...
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

def prepare_features(frame2008, featureslist):
    y= frame2008.obama_win.values
    X = frame2008[featureslist].values
    if len(X.shape) == 1:
        X = X.reshape(-1, 1)
    return y, X


I am working on the file "HW2_solutions.ipynb".
The first task is:
Evaluating and Validating our Forecast
The point of creating a probabilistic predictive model is to simultaneously make a forecast and give an estimate of how certain we are about it.
However, in order to trust our prediction or our reported level of uncertainty, the model needs to be correct. We say a model is correct if it honestly accounts for all of the mechanisms of variation in the system we're forecasting.
In this section, we evaluate our prediction to get a sense of how useful it is, and we validate the predictive model by comparing it to real data.
1.4 Suppose that we believe the model is correct. Under this assumption, we can evaluate our prediction by characterizing its accuracy and precision (see here for an illustration of these ideas). What does the above plot reveal about the accuracy and precision of the PredictWise model?

Background: To make a prediction, we take information that we have now, and try to identify likely outcomes in the future based on this information. The model we've created expresses our uncertainty as a probability distribution over the likely outcomes of the election that are consistent with the information we have now. We call this distribution over outcomes the predictive distribution. Simulating from this model and plotting a histogram allows us to visualize the predictive distribution.  When we say a model is correct, we mean that the predictive distribution matches the true distribution of election outcomes when the information leading up to the election matches the information that we have now.
Usually, people are interested in having a single value as the prediction ("Obama will get 332 votes"). To obtain a prediction, we summarize the predictive distribution with a single point, usually by taking its expectation. We can evaluate a prediction by its accuracy and precision.
Answer: To evaluate the accuracy of our prediction, we can check to see whether the expectation of our predictive distribution seems to match the expectation of the true outcome. In this case,...
Can you write Python code for it?

gallup_2012=pd.read_csv("data/g12.csv").set_index('State')
gallup_2012["Unknown"] = 100 - gallup_2012.Democrat - gallup_2012.Republican
gallup_2012.head()


I am working on the file "HW2_solutions.ipynb".
The first task is:
Exploratory Data Analysis
2.2 Lets do a little exploratory data analysis. Plot a scatter plot of the two PVi's against each other. What are your findings? Is the partisan vote index relatively stable from election to election?
Can you write Python code for it?

#your code here
plt.plot(e2008.pvi, e2012.pvi, 'o', label='Data')
fit = np.polyfit(e2008.pvi, e2012.pvi, 1)
x = np.linspace(-40, 80, 10)
y = np.polyval(fit, x)
plt.plot(x, x, '--k', alpha=.3, label='x=y')
plt.plot(x, y, label='Linear fit')
plt.xlabel("2004 PVI")
plt.ylabel("2008 PVI")
plt.legend(loc='upper left')


I am working on the file "HW2_solutions.ipynb".
The first task is:
Here is some code to plot State Chloropleth maps in matplotlib. make_map is the function you will use.
Can you write Python code for it?

#adapted from  https://github.com/dataiap/dataiap/blob/master/resources/util/map_util.py

#load in state geometry
state2poly = defaultdict(list)

data = json.load(file("data/us-states.json"))
for f in data['features']:
    state = states_abbrev[f['id']]
    geo = f['geometry']
    if geo['type'] == 'Polygon':
        for coords in geo['coordinates']:
            state2poly[state].append(coords)
    elif geo['type'] == 'MultiPolygon':
        for polygon in geo['coordinates']:
            state2poly[state].extend(polygon)

def draw_state(plot, stateid, **kwargs):
    """
    draw_state(plot, stateid, color=..., **kwargs)

    Automatically draws a filled shape representing the state in
    subplot.
    The color keyword argument specifies the fill color.  It accepts keyword
    arguments that plot() accepts
    """
    for polygon in state2poly[stateid]:
        xs, ys = zip(*polygon)
        plot.fill(xs, ys, **kwargs)

def make_map(states, label):
    """
    Draw a cloropleth map, that maps data onto the United States

    Inputs
    -------
    states : Column of a DataFrame
        The value for each state, to display on a map
    label : str
        Label of the color bar

    Returns
    --------
    The map
    """
    fig = plt.figure(figsize=(12, 9))
    ax = plt.gca()

    if states.max() < 2: # colormap for election probabilities
        cmap = cm.RdBu
        vmin, vmax = 0, 1
    else:  # colormap for electoral votes
        cmap = cm.binary
        vmin, vmax = 0, states.max()
    norm = mpl.colors.Normalize(vmin=vmin, vmax=vmax)

    skip = set(['National', 'District of Columbia', 'Guam', 'Puerto Rico',
                'Virgin Islands', 'American Samoa', 'Northern Mariana Islands'])
    for state in states_abbrev.values():
        if state in skip:
            continue
        color = cmap(norm(states.ix[state]))
        draw_state(ax, state, color = color, ec='k')

    #add an inset colorbar
    ax1 = fig.add_axes([0.45, 0.70, 0.4, 0.02])
   ...


I am working on the file "HW2_solutions.ipynb".
The first task is:
Later on in this homework we will explore some approaches to estimating probabilities like these and quatifying our uncertainty about them. But for the time being, we will focus on how to make a prediction assuming these probabilities are known.
Even when we assume the win probabilities in each state are known, there is still uncertainty left in the election. We will use simulations from a simple probabilistic model to characterize this uncertainty. From these simulations, we will be able to make a prediction about the expected outcome of the election, and make a statement about how sure we are about it.
1.2 We will assume that the outcome in each state is the result of an independent coin flip whose probability of coming up Obama is given by a Dataframe of state-wise win probabilities. Write a function that uses this predictive model to simulate the outcome of the election given a Dataframe of probabilities.
Can you write Python code for it?

"""
Function
--------
simulate_election

Inputs
------
model : DataFrame
    A DataFrame summarizing an election forecast. The dataframe has 51 rows -- one for each state and DC
    It has the following columns:
       Obama : Forecasted probability that Obama wins the state
       Votes : Electoral votes for the state
    The DataFrame is indexed by state (i.e., model.index is an array of state names)

n_sim : int
   Number of simulations to run

Returns
-------
results : Numpy array with n_sim elements
   Each element stores the number of electoral college votes Obama wins in each simulation.
"""

#Your code here
def simulate_election(model, n_sim):
    #each column simulates a single outcome from the 50 states + DC
    #Obama wins the simulation if the random number is < the win probability
    simulations = np.random.uniform(size=(51, n_sim))
    obama_votes = (simulations < model.Obama.values.reshape(-1, 1)) * model.Votes.values.reshape(-1, 1)
    #summing over rows gives the total electoral votes for each simulation
    return obama_votes.sum(axis=0)


I am working on the file "HW2_solutions.ipynb".
The first task is:
Lets call the function on the multipoll data frame, and join it with the electoral_votes frame.
Can you write Python code for it?

avg = state_average(multipoll).join(electoral_votes, how='outer')
avg.head()


I am working on the file "HW2_solutions.ipynb".
The first task is:
Lets plot the result of the Predictwise simulation. Your plot should look something like this:
<img src="http://i.imgur.com/uCOFXHp.png">
Can you write Python code for it?


I am working on the file "HW2_solutions.ipynb".
The first task is:
Notice that a lot of states in which Gallup reported a Democratic affiliation, the results were strongly in the opposite direction. Why might that be? You can read more about the reasons for this here.
A quick look at the graph will show you a number of states where Gallup showed a Democratic advantage, but where the elections were lost by the democrats. Use Pandas to list these states.
Can you write Python code for it?

#your code here
prediction_08[(prediction_08.Dem_Win < 0) & (prediction_08.Dem_Adv > 0)]


I am working on the file "HW2_solutions.ipynb".
The first task is:
Now, we run the simulation with this model, and plot it.
Can you write Python code for it?

model = simple_gallup_model(gallup_2012)
model = model.join(electoral_votes)
prediction = simulate_election(model, 10000)
plt.xlim(200,400)

plot_simulation(prediction)
plt.show()
make_map(model.Obama, "P(Obama): Simple Model")


I am working on the file "HW2_solutions.ipynb".
The first task is:
Once again, we plot a map of these probabilities, run the simulation, and display the results
Can you write Python code for it?

make_map(model.Obama, "P(Obama): Gallup + Uncertainty")
plt.show()
prediction = simulate_election(model, 10000)
plot_simulation(prediction)


I am working on the file "HW2_solutions.ipynb".
The first task is:
The following cells takes the necessary DataFrame for the Predictwise data, and runs 10000 simulations. We use the results to compute the probability, according to this predictive model, that Obama wins the election (i.e., the probability that he receives 269 or more electoral college votes)
Can you write Python code for it?

result = simulate_election(predictwise, 10000)

#compute the probability of an Obama win, given this simulation
#Your code here
print (result >= 269).mean()


I am working on the file "HW2_solutions.ipynb".
The first task is:
There are only two polls in the last year! And, the results in the two polls are far, very far from the mean.
Now, Kansas is a safely Republican state, so this dosent really matter, but if it were a swing state, we'd be in a pickle. We'd have no unbiased estimator of the variance in Kansas. So, to be conservative, and play it safe, we follow the same tack we did with the unweighted averaging of polls, and simply assume that the variance in a state is the square of the standard deviation of obama_spread.
This will overestimate the errors for a lot of states, but unless we do a detailed state-by-state analysis, its better to be conservative. Thus, we use:
$\textrm{Var}(\mu)$ = obama_spread.std()$^2$ .
The weights $w_i$ should combine the uncertainties from the margin of error and the age of the forecast. One such combination is:
$$
w_i = \frac1{MoE^2} \times \lambda_{\rm age}
$$
where
$$
\lambda_{\rm age} = 0.5^{\frac{{\rm age}}{30 ~{\rm days}}}
$$
This model makes a few ad-hoc assumptions:

The equation for $\sigma$ assumes that every measurement is independent. This is not true in the case that a given pollster in a state makes multiple polls, perhaps with some of the same respondents (a longitudinal survey). But its a good assumption to start with.
The equation for $\lambda_{\rm age}$ assumes that a 30-day old poll is half as valuable as a current one

3.4 Nevertheless, it's worth exploring how these assumptions affect the forecast model. Implement the model in the function weighted_state_average
Can you write Python code for it?

"""
Function
--------
weighted_state_average

Inputs
------
multipoll : DataFrame
   The multipoll data above

Returns
-------
averages : DataFrame
  A dataframe, indexed by State, with the following columns:
     N: Number of polls averaged together
     poll_mean: The average value for obama_spread for all polls in this state
     poll_std: The standard deviation of obama_spread

Notes
-----
For states where poll_std isn't finite (because N is too small), estimate the
poll_std value as .05 * poll_mean
"""

#your code here

def weights(df):
    lam_age = .5 ** (df.age_days / 30.)
    w = lam_age / df.MoE ** 2
    return w

def wmean(df):
    w = weights(df)
    result = (df.obama_spread * w).sum() / w.sum()
    return result

def wsig(df):
    return df.obama_spread.std()

def weighted_state_average(multipoll):

    groups = multipoll.groupby('State')
    poll_mean = groups.apply(wmean)
    poll_std = groups.apply(wsig)
    poll_std[poll_std.isnull()] = poll_mean[poll_std.isnull()] * .05

    return pd.DataFrame(dict(poll_mean = poll_mean, poll_std = poll_std))


I am working on the file "HW2_solutions.ipynb".
The first task is:
Today: the day we make the prediction
Can you write Python code for it?

# We are pretending to build our model 1 month before the election
import datetime
today = datetime.datetime(2012, 10, 2)
today


I am working on the file "HW2_solutions.ipynb".
The first task is:
We compute the average difference between the Democrat advantages in the election and Gallup poll
Can you write Python code for it?

print (prediction_08.Dem_Adv - prediction_08.Dem_Win).mean()


I am working on the file "HW2_solutions.ipynb".
The first task is:
We construct a similar frame for 2012, obtaining pvi using the 2008 Obama win data which we already have. There is no obama_win column since, well, our job is to predict it!
Can you write Python code for it?

pvi12 = e2008.Dem_Win - (national_results.xs(2008)['Dem'] - national_results.xs(2008)['Rep'])
e2012 = pd.DataFrame(dict(pvi=pvi12, Dem_Adv=gallup_2012.Dem_Adv - gallup_2012.Dem_Adv.mean()))
e2012 = e2012.sort_index()
e2012.head()


I am working on the file "HW2_solutions.ipynb".
The first task is:
We construct the model by estimating the probabilities:
Can you write Python code for it?

model = uncertain_gallup_model(gallup_2012)
model = model.join(electoral_votes)


I am working on the file "HW2_solutions.ipynb".
The first task is:
We use the above function to get the label vector and feature-sample matrix for feeding to scikit-learn. We then use the usual scikit-learn incantation fit to fit a logistic regression model with regularization parameter C. The parameter C is a hyperparameter of the model, and is used to penalize too high values of the parameter co-efficients in the loss function that is minimized to perform the logistic regression. We build a new dataframe with the usual Obama column, that holds the probabilities used to make the prediction. Finally we return a tuple of the dataframe and the classifier instance, in that order.
Can you write Python code for it?

def fit_logistic(frame2008, frame2012, featureslist, reg=0.0001):
    y, X = prepare_features(frame2008, featureslist)
    clf2 = LogisticRegression(C=reg)
    clf2.fit(X, y)

    X_new = frame2012[featureslist]
    obama_probs = clf2.predict_proba(X_new)[:, 1]

    df = pd.DataFrame(index=frame2012.index)
    df['Obama'] = obama_probs
    return df, clf2


I am working on the file "HW2_solutions.ipynb".
The first task is:
Weighted Aggregation
Not all polls are equally valuable. A poll with a larger margin of error should not influence a forecast as heavily. Likewise, a poll further in the past is a less valuable indicator of current (or future) public opinion. For this reason, polls are often weighted when building forecasts.
A weighted estimate of Obama's advantage in a given state is given by
$$
\mu = \frac{\sum w_i \times \mu_i}{\sum w_i}
$$
where $\mu_i$ are individual polling measurements or a state, and $w_i$ are the weights assigned to each poll. The uncertainty on the weighted mean, assuming each measurement is independent, is given by
The estimate of the variance of $\mu$, when $\mu_i$ are unbiased estimators of $\mu$, is
$$\textrm{Var}(\mu) = \frac{1}{(\sum_i w_i)^2} \sum_{i=1}^n w_i^2 \textrm{Var}(\mu_i).$$
Whats the matter with Kansas?
We need to find an estimator of the variance of $\mu_i$, $Var(\mu_i)$. In the case of states that have a lot of polls, we expect the bias in $\mu$ to be negligible, and then the above formula for the variance of $\mu$ holds. However, lets take a look at the case of Kansas.
Can you write Python code for it?


I am working on the file "HW3.ipynb".
The first task is:
2.2 What does the distribution of number of reviews per reviewer look like? Make a histogram
Can you write Python code for it?

#Your code here

def histogram_style():
    remove_border(left=False)
    plt.grid(False)
    plt.grid(axis='y', color='w', linestyle='-', lw=1)

#critics.groupby('critic').size().hist(log=True, bins=range(20), edgecolor='white')
plt.hist(critics.groupby('critic').rtid.count(),log = True, bins=range(20), edgecolor='white')
plt.xlabel("Number of reviews per critic")
plt.ylabel("N")
histogram_style()


I am working on the file "HW3.ipynb".
The first task is:
2.4 Of the critics with > 100 reviews, plot the distribution of average "freshness" rating per critic
Can you write Python code for it?

#Your code here
df = critics.copy()
df['fresh'] = df.fresh == 'fresh'

grp = df.groupby('critic')
counts = grp.rtid.count()
means = grp.fresh.mean()
means[counts > 100].hist(bins=10, edgecolor='w', lw=1)
plt.xlabel("Average rating per critic")
plt.ylabel("N")
plt.yticks([0, 2, 4, 6, 8, 10])
histogram_style()


I am working on the file "HW3.ipynb".
The first task is:
2.5
Using the original movies dataframe, plot the rotten tomatoes Top Critics Rating as a function of year. Overplot the average for each year, ignoring the score=0 examples (some of these are missing data). Comment on the result -- is there a trend? What do you think it means?
Can you write Python code for it?

#Your code here
data = movies[['year','rtTopCriticsRating']]
#data = data.convert_objects(convert_numeric=True) #deprecated
data = data.apply(pd.to_numeric)
data = data[(data.rtTopCriticsRating > 0.00)]
means  = data.groupby('year').mean()

plt.plot(data['year'], data['rtTopCriticsRating'], 'o', mec = 'none', alpha = .5, label = 'Data' )
plt.plot(means.index, means['rtTopCriticsRating'], '-',  label = 'Yearly Average' )
plt.legend(loc ='lower left', frameon = False)
plt.xlabel("Year")
plt.ylabel("Average Score")
remove_border()


I am working on the file "HW3.ipynb".
The first task is:
3.1
Using the critics dataframe, compute a pair of numerical X, Y arrays where:

X is a (nreview, nwords) array. Each row corresponds to a bag-of-words representation for a single review. This will be the input to your model.
Y is a nreview-element 1/0 array, encoding whether a review is Fresh (1) or Rotten (0). This is the desired output from your model.
Can you write Python code for it?

#hint: Consult the scikit-learn documentation to
#      learn about what these classes do do
from sklearn.cross_validation import train_test_split
from sklearn.naive_bayes import MultinomialNB

"""
Function
--------
make_xy

Build a bag-of-words training set for the review data

Parameters
-----------
critics : Pandas DataFrame
    The review data from above

vectorizer : CountVectorizer object (optional)
    A CountVectorizer object to use. If None,
    then create and fit a new CountVectorizer.
    Otherwise, re-fit the provided CountVectorizer
    using the critics data

Returns
-------
X : numpy array (dims: nreview, nwords)
    Bag-of-words representation for each review.
Y : numpy array (dims: nreview)
    1/0 array. 1 = fresh review, 0 = rotten review

Examples
--------
X, Y = make_xy(critics)
"""
def make_xy(critics, vectorizer=None):
    #Your code here
    reviews=  critics['quote']

    if vectorizer is None:
        vectorizer = CountVectorizer(min_df=0)
    vectorizer.fit(reviews)
    X = vectorizer.transform(reviews)
    #X = X.toarray()
    X = X.tocsc()
    Y = (critics.fresh == 'fresh').values.astype(np.int)
    return X, Y

X, Y = make_xy(critics)


I am working on the file "HW3.ipynb".
The first task is:
3.5 We might say a model is over-confident if the freshness fraction is usually closer to 0.5 than expected (that is, there is more uncertainty than the model predicted). Likewise, a model is under-confident if the probabilities are usually further away from 0.5. Is this model generally over- or under-confident?
Your Answer Here
Cross Validation
Our classifier has a few free parameters. The two most important are:

The min_df keyword in CountVectorizer, which will ignore words which appear in fewer than min_df fraction of reviews. Words that appear only once or twice can lead to overfitting, since words which occur only a few times might correlate very well with Fresh/Rotten reviews by chance in the training dataset.

The alpha keyword in the Bayesian classifier is a "smoothing parameter" -- increasing the value decreases the sensitivity to any single feature, and tends to pull prediction probabilities closer to 50%.

As discussed in lecture and HW2, a common technique for choosing appropriate values for these parameters is cross-validation. Let's choose good parameters by maximizing the cross-validated log-likelihood.
3.6 Using clf.predict_log_proba, write a function that computes the log-likelihood of a dataset
Can you write Python code for it?

"""
Function
--------
log_likelihood

Compute the log likelihood of a dataset according to a bayesian classifier.
The Log Likelihood is defined by

L = Sum_fresh(logP(fresh)) + Sum_rotten(logP(rotten))

Where Sum_fresh indicates a sum over all fresh reviews,
and Sum_rotten indicates a sum over rotten reviews

Parameters
----------
clf : Bayesian classifier
x : (nexample, nfeature) array
    The input data
y : (nexample) integer array
    Whether each review is Fresh
"""
#your code here

def log_likelihood(clf,x,y):
    prob = clf.predict_log_proba(x)
    fresh = y == 1 # generates an array with True when fresh and False when rotten
    rotten = ~fresh #fresh inverted
    return  prob[rotten, 0].sum() + prob[fresh, 1].sum()

log_likelihood(clf,xtest,ytest)


I am working on the file "HW3.ipynb".
The first task is:
3.7
Fill in the remaining code in this block, to loop over many values of alpha and min_df to determine
which settings are "best" in the sense of maximizing the cross-validated log-likelihood
Can you write Python code for it?

#the grid of parameters to search over
alphas = [0, .1, 1, 5, 10, 50]
min_dfs = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1]

#Find the best value for alpha and min_df, and the best classifier
best_alpha = None
best_min_df = None
max_loglike = -np.inf

for alpha in alphas:
    for min_df in min_dfs:
        vectorizer = CountVectorizer(min_df = min_df)
        X, Y = make_xy(critics, vectorizer)

        clf = MultinomialNB(alpha = alpha)
        clf.fit(X, Y)
        loglike = cv_score(clf, X, Y, log_likelihood)
        #print "Loglike: %.4f"%loglike, "alpha: %.6f"%alpha, "min_df: %.6f"%min_df
        if loglike > max_loglike:
            max_loglike = loglike
            best_alpha, best_min_df = alpha, min_df

## On the first glace, it may look like the above can be implemented using GridSearchCV but it may not work
## since the training data required for GridSearchCV is generated using one of the parameters (min_df) that
## GridSearchCV is optimizing for.

print "alpha: %f" % best_alpha
print "min_df: %f" % best_min_df


I am working on the file "HW3.ipynb".
The first task is:
3.9 Discuss the various ways in which Cross-Validation has affected the model. Is the new model more or less accurate? Is overfitting better or worse? Is the model more or less calibrated?
Can't see much difference. To explore further.
To think about/play with, but not to hand in: What would happen if you tried this again using a function besides the log-likelihood -- for example, the classification accuracy?
Part 4: Interpretation. What words best predict a fresh or rotten review?
4.1
Using your classifier and the vectorizer.get_feature_names method, determine which words best predict a positive or negative review. Print the 10 words
that best predict a "fresh" review, and the 10 words that best predict a "rotten" review. For each word, what is the model's probability of freshness if the word appears one time?
Hints

Try computing the classification probability for a feature vector which consists of all 0s, except for a single 1. What does this probability refer to?

np.eye generates a matrix where the ith row is all 0s, except for the ith column which is 1.
Can you write Python code for it?

# Your code here
words = np.array(vectorizer.get_feature_names())

x = np.eye(xtest.shape[1])
probs = clf.predict_proba(x)[:,0]
ind = np.argsort(probs) # returns sorted indices

good_words = words[ind[:10]]
bad_words = words[ind[-10:]] # slice format: [start : end] --> in this case, last 10 elements

good_probs = probs[ind[:10]]
bad_probs = probs[ind[-10:]]

print "Good Words \t    P(fresh | word)"
for w, p in zip(good_words,good_probs):
    #print "%20s" %w, "%0.2f"% (1- np.exp(p)) # if clf.predict_log_ proba was used
    print "%20s" %w, "%0.2f"% (1-p) #%20s --> right aligned 20 spaces

print "Bad Words \t    P(fresh | word)"
for w, p in zip (bad_words,bad_probs):
    print "%20s" %w, "%0.2f"% (1-p) # 1-p since prob of rotten is chosen. rotten is chosen for the
                                                #descending order listing


I am working on the file "HW3.ipynb".
The first task is:
4.2
One of the best sources for inspiration when trying to improve a model is to look at examples where the model performs poorly.
Find 5 fresh and rotten reviews where your model performs particularly poorly. Print each review.
Can you write Python code for it?

#Your code here
x, y = make_xy(critics, vectorizer)
prob  = clf.predict_proba(x)[:,0]

bad_rotten = np.argsort(prob[y == 0])[:5]
bad_fresh = np.argsort(prob[y == 1]) [-5:]

print "5 Mis-predicted rotten reviews (Actually rotten, predicted fresh)"
print "---------------------------"
for row in bad_rotten:
    print critics[y == 0].quote.iloc[row], "(",critics[y == 0].title.iloc[row],")","\n"

print "5 Mis-predicted fresh reviews (Actually fresh, predicted rotten)"
print "---------------------------"
for row in bad_fresh:
    print critics[y == 1].quote.iloc[row] , "(",critics[y == 1].title.iloc[row],")","\n"


I am working on the file "HW3.ipynb".
The first task is:
4.3 What do you notice about these mis-predictions? Naive Bayes classifiers assume that every word affects the probability independently of other words. In what way is this a bad assumption? In your answer, report your classifier's Freshness probability for the review "This movie is not remarkable, touching, or superb in any way".
Can you write Python code for it?

clf.predict_proba(vectorizer.transform(['This movie is not remarkable, touching, or superb in any way']))


I am working on the file "HW3.ipynb".
The first task is:
A quick sanity check that everything looks ok at this point
Can you write Python code for it?

assert set(critics.columns) == set('critic fresh imdb publication quote review_date rtid title'.split())
assert len(critics) > 10000


I am working on the file "HW3.ipynb".
The first task is:
Here's a function to estimate the cross-validated value of a scoring function, given a classifier and data
Can you write Python code for it?

from sklearn.cross_validation import KFold
from sklearn.cross_validation import cross_val_score

def cv_score(clf, x, y, score_func):
    """
    Uses 5-fold cross validation to estimate a score of a classifier

    Inputs
    ------
    clf : Classifier object
    x : Input feature vector
    y : Input class labels
    score_func : Function like log_likelihood, that takes (clf, x, y) as input,
                 and returns a score

    Returns
    -------
    The average score obtained by randomly splitting (x, y) into training and
    test sets, fitting on the training set, and evaluating score_func on the test set

    Examples
    cv_score(clf, x, y, log_likelihood)
    """
    result = 0
    nfold = 5
    for train, test in KFold(y.size, nfold): # split data into train/test groups, 5 times
        clf.fit(x[train], y[train]) # fit
        result += score_func(clf, x[test], y[test]) # evaluate score function on held-out data
    return result / nfold # average

print cv_score(clf, xtrain, ytrain, log_likelihood)
# as a side note, this function is builtin to the newest version of sklearn. We could just write
# sklearn.cross_validation.cross_val_score(clf, x, y, scorer=log_likelihood).

#print cross_val_score(clf, xtrain, ytrain, cv =5,scoring = log_likelihood).sum()/5


I am working on the file "HW3.ipynb".
The first task is:
P1.1
We'd like you to write a function that looks up the first 20 Top Critic Rotten Tomatoes reviews for a movie in the movies dataframe. This involves two steps:

Use the Movie Alias API to look up the Rotten Tomatoes movie id from the IMDB id
Use the Movie Reviews API to fetch the first 20 top-critic reviews for this movie

Not all movies have Rotten Tomatoes IDs. In these cases, your function should return None. The detailed spec is below. We are giving you some freedom with how you implement this, but you'll probably want to break this task up into several small functions.
Hint
In some situations, the leading 0s in front of IMDB ids are important. IMDB ids have 7 digits
Can you write Python code for it?

"""
Function
--------
fetch_reviews(movies, row)

Use the Rotten Tomatoes web API to fetch reviews for a particular movie

Parameters
----------
movies : DataFrame
  The movies data above
row : int
  The row of the movies DataFrame to use

Returns
-------
If you can match the IMDB id to a Rotten Tomatoes ID:
  A DataFrame, containing the first 20 Top Critic reviews
  for the movie. If a movie has less than 20 total reviews, return them all.
  This should have the following columns:
    critic : Name of the critic
    fresh  : 'fresh' or 'rotten'
    imdb   : IMDB id for the movie
    publication: Publication that the critic writes for
    quote  : string containing the movie review quote
    review_data: Date of review
    rtid   : Rotten Tomatoes ID for the movie
    title  : Name of the movie

If you cannot match the IMDB id to a Rotten Tomatoes ID, return None

Examples
--------
>>> reviews = fetch_reviews(movies, 0)
>>> print len(reviews)
20
>>> print reviews.irow(1)
critic                                               Derek Adams
fresh                                                      fresh
imdb                                                      114709
publication                                             Time Out
quote          So ingenious in concept, design and execution ...
review_date                                           2009-10-04
rtid                                                        9559
title                                                  Toy story
Name: 1, dtype: object
"""
#RT API is not available, so we'll scrape using the movie titles from the movies data frame
def fetch_reviews(movies, row):
    url  = 'https://www.rottentomatoes.com/m/'+movies.rtID[row]+'/reviews/?type=top_critics'

    #create the reviews df by parsing the required fields from the URL page
    try:
        data  = requests.get(url)
    except requests.exceptions.ConnectionError:
        r.status_code = "Connection refused"

    soup = bs(data.content)
    critic = []
    fresh_rating...


I am working on the file "HW3.ipynb".
The first task is:
P1.2
Use the function you wrote to retrieve reviews for the first 3,000 movies in the movies dataframe.
Hints

Rotten Tomatoes limits you to 10,000 API requests a day. Be careful about this limit! Test your code on smaller inputs before scaling. You are responsible if you hit the limit the day the assignment is due :)
This will take a while to download. If you don't want to re-run this function every time you restart the notebook, you can save and re-load this data as a CSV file. However, please don't submit this file
Can you write Python code for it?

"""
Function
--------
build_table

Parameters
----------
movies : DataFrame
  The movies data above
rows : int
  The number of rows to extract reviews for

Returns
--------
A dataframe
  The data obtained by repeatedly calling `fetch_reviews` on the first `rows`
  of `movies`, discarding the `None`s,
  and concatenating the results into a single DataFrame
"""
#your code here
def build_table(movies, rows):
    table = pd.DataFrame()
    for i in range(rows):
        table  = table.append(fetch_reviews(movies,i ))
    return table

#you can toggle which lines are commented, if you
#want to re-load your results to avoid repeatedly calling this function

#critics = build_table(movies, 3000) # may result in connection error as RT refuses connection request
                                        #after n (n ~ 1500+) number of requests. Try retrieving the data in smaller chunks instead
#critics.to_csv('critics.csv', index=False,encoding='utf-8')
critics = pd.read_csv('critics.csv')
critics[critics.fresh == 'fresh'].count

#for this assignment, let's drop rows with missing data
critics = critics[~critics.quote.isnull()]
#critics = critics[critics.fresh != None]
critics = critics[critics.fresh.notnull()]
critics = critics[critics.quote.str.len() > 0]
critics = critics.reset_index(drop = True)
critics.shape


I am working on the file "HW3.ipynb".
The first task is:
Part 1: Get the data
Here's a chunk of the MovieLens Dataset:
Can you write Python code for it?

from io import StringIO
movie_txt = requests.get('https://raw.github.com/cs109/cs109_data/master/movies.dat').text
movie_file = StringIO(movie_txt) # treat a string like a file
movies = pd.read_csv(movie_file, delimiter='\t')

#print the first row
movies[['id', 'title', 'imdbID', 'year']].irow(0)
print movies.shape
movies = movies.dropna()
movies = movies.reset_index(drop = True)
print movies.shape
movies.head()
movies.dtypes


I am working on the file "HW3.ipynb".
The first task is:
Some overfitting seems to be happening here as the error rate on test data(~20%) is more than twice as large as the error rate on trainng data (7.5%). Though it's possible that the accuracy difference is due to random chance and not a symptom of overfitting , it is unlikely. This can be tested by cross-validation, i.e. repeatedly testing the model on different train/test splits. If the accuracy on training data is consistently higher than that of test data, then overfitting has occured.
3.4: Model Calibration
Bayesian models like the Naive Bayes classifier have the nice property that they compute probabilities of a particular classification -- the predict_proba and predict_log_proba methods of MultinomialNB compute these probabilities.
Being the respectable Bayesian that you are, you should always assess whether these probabilities are calibrated -- that is, whether a prediction made with a confidence of x% is correct approximately x% of the time. We care about calibration because it tells us whether we can trust the probabilities computed by a model. If we can trust model probabilities, we can make better decisions using them (for example, we can calculate how much we should bet or invest in a given prediction).
Let's make a plot to assess model calibration. Schematically, we want something like this:
<img src="http://i.imgur.com/ea5B6zr.png" width=500px>
In words, we want to:

Take a collection of examples, and compute the freshness probability for each using clf.predict_proba
Gather examples into bins of similar freshness probability (the diagram shows 5 groups -- you should use something closer to 20)
For each bin, count the number of examples in that bin, and compute the fraction of examples in the bin which are fresh
In the upper plot, graph the expected P(Fresh) (x axis) and observed freshness fraction (Y axis). Estimate the uncertainty in observed freshness fraction $F$ via the equation $\sigma = \sqrt{F (1-F) / N}$
Overplot the line y=x. This is the trend we would expect if the model is...
Can you write Python code for it?

"""
Function
--------
calibration_plot

Builds a plot like the one above, from a classifier and review data

Inputs
-------
clf : Classifier object
    A MultinomialNB classifier
X : (Nexample, Nfeature) array
    The bag-of-words data
Y : (Nexample) integer array
    1 if a review is Fresh
"""
def calibration_plot(clf, xtest, ytest):

    pfresh = clf.predict_proba(xtest)[:,1]
    print pfresh
    data =  pd.DataFrame(dict(prob = pfresh, outcome = ytest))

    # group in to bins of similar probabilities
    bins = np.linspace(0,1,20)
    prob_bins = pd.cut(data.prob, bins)
    binwidth = bins[1] - bins[0]

    #freshness ratio and number of examples in each bin
    cal = data.groupby(prob_bins).outcome.agg(['mean', 'count'])
    cal['pmid'] = (bins[:-1] + bins[1:]) / 2
    cal['sig'] = np.sqrt(cal.pmid * (1 - cal.pmid) / cal['count'])

    #the calibration plot
    ax = plt.subplot2grid((3, 1), (0, 0), rowspan=2)
    p = plt.errorbar(cal.pmid, cal['mean'], cal['sig'])
    plt.plot(cal.pmid, cal.pmid, linestyle='--', lw=1, color='k')
    plt.ylabel("Empirical P(Fresh)")
    remove_border(ax)

    #the distribution of P(fresh)
    ax = plt.subplot2grid((3, 1), (2, 0), sharex=ax)

    plt.bar(left=cal.pmid - binwidth / 2, height=cal['count'],
        width=.95 * (bins[1] - bins[0]),
        fc=p[0].get_color())

    plt.xlabel("Predicted P(Fresh)")
    remove_border()
    plt.ylabel("Number")

calibration_plot(clf, xtest, ytest)


I am working on the file "HW3/Basic Statistics.ipynb".
The first task is:
Correlation
Can you write Python code for it?

print(X.corr())

def crtPlot (x, y):
    p1=polyfit(x,y,1)
    %matplotlib inline
    plot(x,y, 'o')
    plot(x, polyval(p1, x), 'r-')
    show()


I am working on the file "HW3/Basic Statistics.ipynb".
The first task is:
Mean & Median
Can you write Python code for it?

mean=X['BikeCount'].mean()# mean is less roburst
median=X['BikeCount'].median()# median is roburst
pprint(mean)
pprint(median)


I am working on the file "HW3/Basic Statistics.ipynb".
The first task is:
Precip and BikeCount
Can you write Python code for it?


I am working on the file "HW3/notebooks/Q-5_Run_K-Means.ipynb".
The first task is:
(4 points) For K = 250, what are your training and test 0/1 losses?
Can you write Python code for it?

km_250.assess_test_data()

km_250_test_results = assess_test_data(km_250)
print("")
print(km_250_test_results.to_latex())


I am working on the file "HW3/notebooks/Q-5_Run_K-Means.ipynb".
The first task is:
(a) The squared reconstruction error vs iteration number.
Can you write Python code for it?

km_250_reconstruction = km_250.plot_squared_reconstruction_error()
km_250_reconstruction.savefig('../figures/k_means/k250_reconstruction_error.pdf',
                             bbox_inches='tight')

km_250.results_df.tail(1).T


km_16_reconstruction_error = km_16.plot_squared_reconstruction_error()
km_16_reconstruction_error.savefig('../figures/k_means/k16_reconstruction_error.pdf',
                                  bbox_inches='tight')

km_16_reconstruction_error = km_16.plot_squared_reconstruction_error()
km_16_reconstruction_error.savefig('../figures/k_means/k16_reconstruction_error.pdf',
                                  bbox_inches='tight')

km_16.results_df.tail(2)

km_16_reconstruction_error_nromalized = \
    km_16.plot_squared_reconstruction_error_normalized()

km_16_0_1_loss = km_16.plot_0_1_loss()
km_16_0_1_loss.savefig('../figures/k_means/k16_loss_01.pdf',
                       bbox_inches='tight')


I am working on the file "HW3/notebooks/Q-5_Run_K-Means.ipynb".
The first task is:
(b) Let us say that the number of assignments for a mean is the number of points assigned to that
mean. Plot the number of assignments for each center in descending order.
Can you write Python code for it?

km_250_assignment_bars = km_250.plot_num_assignments_for_each_center()
km_250_assignment_bars.savefig('../figures/k_means/k250_assignment_bars.pdf',
                              bbox_inches='tight')


km_16_assignment_bars = km_16.plot_num_assignments_for_each_center()
km_16_assignment_bars.savefig('../figures/k_means/k16_assignment_bars.pdf',
                             bbox_inches='tight')


I am working on the file "HW3/notebooks/Q-5_Run_K-Means.ipynb".
The first task is:
5.2 Classification with K-means

(4 points) For K = 16, what are your training and test 0/1 losses?
Can you write Python code for it?

import copy

    def assess_test_data(self):
        test_model = copy.copy(self)
        test_model.X = self.test_X
        test_model.y = self.test_y
        test_model.N, test_model.d = test_model.X.shape

        ## model characteristics
        #test_model.assignments = None # cluster assignment.  Does not know about labels.
        #test_model.predictions = None # label assignment.  Does not know about cluster.

        test_model.results_df = None
        # todo: rename
        test_model.results_df_cluster_assignment_counts = None

        test_model.set_point_assignments()
        test_model.set_centers_classes()
        test_model.set_predicted_labels()

        test_model.record_count_of_assignments_to_each_mean()
        test_model.record_fit_statistics()
        print("test results:")
        print(test_model.results_df.T)

        self.test_model = test_model
        return test_model.results_df.T

km_16_test_results = assess_test_data(km_16)

print(km_16_test_results.to_latex())

km_16.assess_test_data().to


I am working on the file "HW3/notebooks/Q-5_Run_K-Means.ipynb".
The first task is:
Check out centers that appear to be a mix of digits.
for k = 16, center number 5 appears to be a blend of 5 and 8.  What are it's true labels?
Can you write Python code for it?

km_16.verbose = True

km_16.clusters_by_num_in_cluster().head(6)

km_16.set_centers_classes()


I am working on the file "HW3/notebooks/Q-5_Run_K-Means.ipynb".
The first task is:
Q 5.2
k = 250, MNIST data transformed by first 50 PCA components.
Can you write Python code for it?

km_250 = KMeans(k=250, train_X=X_train, train_y=y_train,
            pca_obj=pca_training,
            max_iter = 500,
            test_X=X_test, test_y=y_test,
            verbose=False)
km_250.run()


I am working on the file "HW3/notebooks/dev_RBF_kernel.ipynb".
The first task is:
Compare to stack overflow
http://stats.stackexchange.com/questions/115439/how-to-apply-a-gaussian-radial-basis-function-kernel-pca-to-nonlinear-data
Can you write Python code for it?

def so(X, sigma):
    # pdist to calculate the squared Euclidean distances for every pair of points
    # in the 100x2 dimensional dataset.
    sq_dists = pdist(X, 'sqeuclidean')

    # Variance of the Euclidean distance between all pairs of data points.
    variance = np.var(sq_dists)

    # squareform to converts the pairwise distances into a symmetric 100x100 matrix
    mat_sq_dists = squareform(sq_dists)

    # Compute the 100x100 kernel matrix
    K = np.exp(-1/(2*sigma**2) * mat_sq_dists)

    return K

np.multiply(X, X)

so(X, 4)


I am working on the file "HW3/notebooks/dev_RBF_kernel.ipynb".
The first task is:
Old stuff from doing by hand
Can you write Python code for it?

X - X[0,:]

np.linalg.norm(X - X[0,:], axis=1)

np.linalg.norm(X - X[0,:], axis=1)/(-2.)/1**2

np.exp(np.linalg.norm(X - X[0,:], axis=1)/(-2.)/1**2)

np.sum(X, axis=0)

pdist(X)

squareform(pdist(X))


I am working on the file "HW3Notebook.ipynb".
The first task is:
$f(x) = x - sin(x)$
Can you write Python code for it?


I am working on the file "HW3Notebook.ipynb".
The first task is:
$f(x) = x^4sinx$
Can you write Python code for it?


I am working on the file "HW3Notebook.ipynb".
The first task is:
$f(x) = x^{10} - 1$
Can you write Python code for it?


I am working on the file "HW3Notebook.ipynb".
The first task is:
$tanh(x) - x^{10}$
Can you write Python code for it?


I am working on the file "HW3Notebook.ipynb".
The first task is:
A.13 Computing the arc length of a curve
Can you write Python code for it?

h1 = -4 * (x)**2
x = sp.Symbol('x')
h2 = sp.exp(h1)
h3 = 1 / np.sqrt(2 * np.pi) * h2
length = p3.arclength(h3, -2, 2, 10)
print length


I am working on the file "HW3Notebook.ipynb".
The first task is:
Chinmai Raman
Homework 3
A.4 Solving a system of difference equations
Computes the development of a loan over time.
The below function calculates the amount paid per month (the first array) and the amount left to be paid (the second array) at each month of the year at a principal of $10,000 to be paid over 1 year at annual interest rate of 6%
Can you write Python code for it?


I am working on the file "HW3Notebook.ipynb".
The first task is:
The accuracy of a Taylor polynomial also improves as n increases.
Can you write Python code for it?

x = [np.pi / 4, np.pi / 4, np.pi / 4, np.pi / 4]
N = [1, 3, 5, 10]
n = 0
Sn = []
while n < 4:
    Sn.append(p4.sin_Taylor(x[n], N[n])[0])
    n += 1
print Sn


I am working on the file "HW3Notebook.ipynb".
The first task is:
The arclength of the function f(x) from -2 to 2 is 4.18
Can you write Python code for it?

fig = plt.figure(1)
x = np.linspace(-2, 2, 100)
y = 1 / np.sqrt(2 * np.pi) * np.exp(-4 * x**2)
x1 = length[0]
y1 = length[1]
plt.plot(x, y, 'r-', x1, y1, 'b-')
plt.xlabel('x')
plt.ylabel('y')
plt.title('1/sqrt(2pi) * e^(-4t^2)')
plt.show(fig)