I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Read NERC regions shapefile
Link to the shapefile for local download
Can you write Python code for it?

# EIA NERC region shapefile, which has an "Indeterminate" region

# path = os.path.join(data_path, 'NERC_Regions_EIA', 'NercRegions_201610.shp')
# regions = gpd.read_file(path)
# regions.crs

path = os.path.join(data_path, 'nercregions', 'NERCregions.shp')
regions_nerc = gpd.read_file(path)
regions_nerc['nerc'] = regions_nerc['NERCregion']

regions_nerc

regions_nerc.boundary.plot()


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Read file with facility location, generation, and fuel data
Can you write Python code for it?

path = os.path.join(data_path, 'Facility gen fuels and CO2 2017-08-31.zip')
facility_df = pd.read_csv(path)


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Same size dataframe as from Method 1. Yet again, there appear to be extra results.
Can you write Python code for it?


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
So 641 definitely is in both NERC regions. And apparently the regions overlap? But they aren't supposed to. And I didn't have any plants show up in multiple regions when doing the spatial join in QGIS.
Can you write Python code for it?


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Still too many results
Can you write Python code for it?


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
The nerc column is labels from a spatial join using this same shapefile and lat/lon data in QGIS.
Can you write Python code for it?

location.head()

location.loc[location['lat'].isnull()]

geometry = [Point(xy) for xy in zip(location.lon, location.lat)]
# small_facility = small_facility.drop(['lon', 'lat'], axis=1)
crs = {'init': 'epsg:4326'}
geo_df = GeoDataFrame(location, crs=crs, geometry=geometry)

geo_df.head()


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
There are 36 plants with duplicate NERC regions.
Can you write Python code for it?

len(df2.loc[df2['plant id'].duplicated()].sort_values('plant id'))

pd.set_option('display.max_rows', 200)

df2.loc[df2['plant id'].duplicated(keep=False)].sort_values('plant id')

facility_df.head()

plants = facility_df.loc[:, ['plant id', 'year', 'lat', 'lon']]
plants.drop_duplicates(inplace=True)

plants.groupby(['plant id']).max().hist()

path = join(data_path, 'EIA downloads', 'EIA-860 2015', '2___Plant_Y2015.xlsx')
nercs2015 = pd.read_excel(path, skiprows=0, parse_cols='C,L')
nercs2015.columns = ['plant id', 'nerc']

path = join(data_path, 'EIA downloads', 'eia8602012', 'PlantY2012.xlsx')
nercs2012 = pd.read_excel(path, skiprows=0, parse_cols='B,J')
nercs2012.columns = ['plant id', 'nerc']

nercs2012.head()

nercs = pd.concat([nercs2012, nercs2015]).drop_duplicates()

len(nercs), len(nercs2012), len(nercs2015)

df = pd.merge(plants, nercs, on=['plant id'], how='left')

plants.tail()

nercs.tail()

nercs.nerc.unique()

df.tail()

len(df.loc[df.isnull().any(axis=1), 'plant id'].unique())

from sklearn import neighbors
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import numpy as np

nercs = ['SERC', 'RFC', 'SPP', 'NPCC', 'WECC', 'MRO', 'TRE', 'FRCC']

df_slim = df.loc[df.nerc.isin(nercs), ['plant id', 'lat', 'lon', 'nerc']].dropna(subset=['lon']).drop_duplicates()

unknown = df_slim.loc[df_slim.nerc.isnull()]

n_neighbors = 10

X = df_slim.loc[df_slim.notnull().all(axis=1), ['lat', 'lon']]
y = df_slim.loc[df_slim.notnull().all(axis=1), 'nerc']

X_scale = StandardScaler().fit_transform(X)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42)

y.unique()

knn = neighbors.KNeighborsClassifier()

params = {'weights': ['uniform', 'distance'],
          'n_neighbors': [10, 15, 20, 30],
          'leaf_size': [3, 5, 10, 30],
          'p': [1, 2]}

clf = GridSearchCV(knn, params, n_jobs=-1)

clf.fit(X_train,...


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
What facilities are missing?
Can you write Python code for it?

regions

frcc = regions.loc[1, 'geometry']
serc = regions.loc[5, 'geometry']


I am working on the file "Notebooks/assignment3.ipynb".
The first task is:
As we can see from the above plots, graphs following BA model have approximately the same diameter as ER graphs while the clustering coefficient is increased.
Problem 2
The goal of this exercise is to show that Barabasi-Albert model produces a scale free network.
To do that, we will generate a random graph following BA model and compute the degree distribution. Then, we will try to approximate the obtained distribution with power law distribution.
Can you write Python code for it?

import algorithms.barabasi_albert as ba
reload(ba)
N, m, gamma = 1000, 100, -2
M = np.arange(90, 150)

g = ba.barabasi_albert(N, m)

# ba_graphs = [ba.barabasi_albert(N, m) for m in M]

gamma = -0.95
fig = plt.figure(figsize=(15, 8))
degree_seq = np.array(g.degree_sequence(), dtype=np.float32)
degree_dist = degree_seq / sum(degree_seq)
plt.hist(degree_seq, normed=True)
degree_range = np.arange(100, 400, dtype=np.float32)
p_law = [k**gamma for k in degree_range]
pl_line, = plt.plot(degree_range, p_law, 'r-', label="P(k) = k^(-0.95)")
plt.legend([pl_line])
plt.xlabel("degree")

plt.show()


I am working on the file "Notebooks/assignment3.ipynb".
The first task is:
Now we will make two different plots. In one, we will compare the diameters of the above generated graphs and in the second one, we will compare clustering coefficient.
Can you write Python code for it?

# on x axis, plot k
# on y axis, plot the diameter
ws_diam, er_diam, ws_cc, er_cc = [], [], [], []

for g1, g2 in zip(ws_graphs, er_graphs):
    ws_cc.append(g1.global_clustering_coefficient())
    paths = g1.shortest_path()
    max_path = [[(0 if np.isinf(paths[u][v]) else paths[u][v]) for v in g1.vertices()] for u in g1.vertices()]
    ws_diam.append(max(max(max_path)))
    er_diam.append(g2.diameter())
    er_cc.append(g2.global_clustering_coefficient())

fig = plt.figure(figsize = (15, 8))
plt.subplot(211)
er_line, = plt.plot(K, er_diam, 'b', label="Erdos Renyi diameter")
ws_line, = plt.plot(K, ws_diam, 'r', label="Watts-Strogatz diameter")
plt.legend([er_line, ws_line])
plt.xlabel("m")
plt.ylabel("diameter")

plt.subplot(212)
cc_er_line, = plt.plot(K, er_cc, 'b', label="Erdos Renyi clust coef")
cc_ws_line, = plt.plot(K, ws_cc, 'r', label="Watts-Strogatz clust coef")
plt.legend([cc_er_line, cc_ws_line])
plt.xlabel("m")
plt.ylabel("clust coef")

plt.show()


I am working on the file "Notebooks/chi2_parameter_broadcasting.ipynb".
The first task is:
Add a third axis for a vertical shift
Can you write Python code for it?

shift = np.arange(0.1, 1.3, 0.1)
print(len(shift))
fmas = fma[:, :, None] + shift
fmas.shape

wmas = wma[:, :, None] * np.ones_like(shift)
wmas.shape

print(wm.shape)
print(fmas.shape)
w_sfunc = sp.interpolate.interp1d(wm, fmas, axis=0, kind="slinear")

fmas_interp = w_sfunc(w)
fmas_interp.shape

plt.plot(w, fmas_interp[:,3, :])
plt.plot(w, f,  "--", label="data")
plt.legend()
plt.show()

chi2s = np.sum((f[:, None, None] - fmas_interp)**2 / fmas_interp, axis=0)

plt.plot(a, chi2s, label="chi2")
plt.legend()
plt.show()

X, Y = np.meshgrid(shift, a)
print(X.shape)
plt.contourf(X, Y, chi2s)
plt.colorbar()
plt.plot()
plt.show()
chi2s.shape

c2min = chi2s.argmin()
print(c2min)
chi2s[np.unravel_index(c2min, chi2s.shape)]

np.unravel_index(976, (140, 7))

plt.contour(chi2s)
plt.show()


I am working on the file "Notebooks/chi2_parameter_broadcasting.ipynb".
The first task is:
Interpolating different wavelength axis.
Each wl dimension has a dopplershift added.
Can you write Python code for it?

c = 500
vc = (1 + np.arange(10) / c)
print(wm.shape)
print(vc.shape)
doppler =  wm[:, np.newaxis] *  vc

print(doppler.shape)
#print(doppler)

plt.plot(doppler, fmas[:,:,5])
plt.show()

# doppler_interp = sp.interpolate.interp1d(doppler, fm)
print(len(wm))
print(len(vc))
print(fma.shape)    # fma includes the amplitude also.
# Cannot inperpolate directly for all the different wavelengths at once. Therefore
dims = fmas.shape + (len(vc),)  # add extra arry to dim
print(dims)
result = np.empty(dims)
print(result.shape)
for i, v in enumerate(vc):
    wm_vc = wm * v
    #  print(wm_vc.shape)
    #  print(fma.shape)
    func = sp.interpolate.interp1d(wm_vc, fmas, axis=0, bounds_error=False, fill_value=99999)
    # print(func(wm))
    result[:,:, :, i] = func(wm)

print(result)


I am working on the file "Notebooks/chi2_parameter_broadcasting.ipynb".
The first task is:
This lets me doppler shift the wavelength and return it to wm.
In the second case for model I will just want to return it to the wavelength values of the observation.
Can you write Python code for it?

# interp to obs
func = sp.interpolate.interp1d(wm, result, axis=0, bounds_error=False, fill_value=np.nan)
fmasd = func(w)
chi2d = np.sum((f[:, None, None, np.newaxis] - fmasd)**2 / fmasd, axis=0)
chi2d

chi2d.shape

fmasd.shape

a.shape

# Try a 3d chisquare

x_2 = chisquare(f[:, np.newaxis, np.newaxis, np.newaxis], fmasd, axis=0).statistic

x_2.argmin()
vals = np.unravel_index(x_2.argmin(), x_2.shape)
print("index of min = ", vals)   # returns the minimum index location


I am working on the file "Notebooks/chi2_parameter_broadcasting.ipynb".
The first task is:
This provides a framework for chisquare of large arrays. for my simulations
Can you write Python code for it?

plt.title("shift min")
plt.contourf(x_2[:,3,:])
plt.show()

plt.contourf(x_2[4,:,:])
plt.title("amplitude min")
plt.show()

plt.contourf(x_2[:,:,4])
plt.title("doppler min")
plt.show()


I am working on the file "Notebooks/clockModelComparison.ipynb".
The first task is:
Some plots:
Can you write Python code for it?

def plotAndComputeCorrelation(x,y,namex, namey, logyn, logxn, limx=None, limy=None):
    print("Pearson correlation coefficient and p-value: "+ str(scipy.stats.pearsonr(x, y)))
    #Plotting:
    plt.plot(x, y, 'bo')
    plt.xlabel(namex, fontsize=15)
    plt.ylabel(namey, fontsize=15)
#plt.legend(['data'], loc='upper left')
    if logyn:
        plt.yscale('log')
    if logxn:
        plt.xscale('log')
    if not limx == None:
        plt.xlim(limx)
    if not limy == None:
        plt.ylim(limy)

%matplotlib inline

plotAndComputeCorrelation(heights23S,heights16S23S,'Node ages 23S', 'Node ages 16S23S', False, False)

%matplotlib inline

plotAndComputeCorrelation(heights16S,heights16S23S,'Node ages 16S', 'Node ages 16S23S', False, False)

%matplotlib inline

plotAndComputeCorrelation(heights16S,heights23S,'Node ages 16S', 'Node ages 23S', False, False)


I am working on the file "Notebooks/computeConsensusChronogram.ipynb".
The first task is:
Node ids are consistent across all trees.
Getting node Heights.
Can you write Python code for it?

#File where I store useful functions
exec (open("/Users/boussau/Programs/PythonCode/functions.py").read ())

id2Heights=list()
for t in allTrees:
    node2Height,id2Height = getNodeHeights( t )
    id2Heights.append(id2Height)


I am working on the file "Notebooks/computeConsensusChronogram.ipynb".
The first task is:
Now we have the node heights.
Outputting the consensus chronogram, with confidence intervals.
Can you write Python code for it?

print(len(id2Heights))
# Creating a uniform weight vector
weights = [1] * len(id2Heights)
outputsWeightedChronogram (allTrees[0].copy(), id2Heights, "consensus100MT", weights)


I am working on the file "Notebooks/dft_conventions_and_normalization.ipynb".
The first task is:
Let's check the normalization of the whitening filter.
The inverse ASD should be multiplied by the sqrt(2 / sample_rate) in order to get the variance of the whitened time domain data to be one.
Can you write Python code for it?

invasd = sqrt(2./srate)/sqrt(shaped_psd1)
invasd[0] = 0.
whitener = sig.hann(seglen*srate)*roll(irfft(invasd), seglen*srate/2)

wtest = TimeSeries(sig.convolve(shaped_noise2, whitener, mode='same')[8*srate:-8*srate],
                       sample_rate=srate)

p1 = wtest.psd(seglen).plot()

mean(wtest.psd(seglen)) / (2./srate)

mean(wtest**2)


I am working on the file "Notebooks/dft_conventions_and_normalization.ipynb".
The first task is:
Now let's try applying a weighting in the frequency domain that doesn't change the TD variance.
We just fix the sum of squares of the filter in the TD to be 1.
Can you write Python code for it?

filt1 = sig.firwin2(seglen*srate, [0.,0.25,0.5,0.75,1.], [0.,0.5,1.,0.5,0.], window='hann')
filt1 /= sqrt(sum(filt1**2))

weight_test1 = TimeSeries(sig.convolve(noise, filt1, mode='same')[8*srate:-8*srate],
                       sample_rate=srate)

var(weight_test1)

filt2 = gen_filter.copy()
filt2 /= sqrt(sum(filt2**2))

weight_test2 = TimeSeries(sig.convolve(noise, filt2, mode='same')[8*srate:-8*srate],
                       sample_rate=srate)

var(weight_test2)

filt12 = sig.hann(seglen*srate)*roll(irfft(abs(rfft(filt1)*rfft(filt2))), seglen*srate/2)

weight_test12 = TimeSeries(sig.convolve(noise, filt12, mode='same')[8*srate:-8*srate],
                       sample_rate=srate)

plt.plot(freqs, abs(rfft(filt1)*rfft(filt2)))

plt.plot(filt12)

sum(filt1**2)

sum(abs(rfft(filt1))**2)/(seglen*srate/2.)


I am working on the file "Notebooks/emo.ipynb".
The first task is:
Generate emotional vectors for every paragraph

for p in paragraphs in the text
tokenize the paragraph with .split()
initiate emotional vector emo_d
for every word (w) in paragraph (p)
strip special characters
query for word in NRC lexicon and read out the emo column with binary = 1
for every emotion (e)
if the word elicits an emotion in emo_d, then add 1 to the specified emotion.

append the emotion vectors to a list
Can you write Python code for it?

emo_list = []
moo_d = dict()
for p in tqdm(range(len(doc.paragraphs)), desc='par'):
    s = doc.paragraphs[p].text.split()
    emo_d = dict({'joy':0, 'fear':0, 'surprise':0, 'sadness':0, 'disgust':0, 'anger':0})
    moo_d[p] = []
    for w in s:
        w = re.sub("[!@#$%^&*()_+:;,.\\?']", "", w)
        emo = df.query("word=='%s'" %w).query("binary==1")['emo'].as_matrix()
        for e in emo:
            if e in emo_d.keys():
                emo_d['%s' %e] += 1
                moo_d[p].append(w)
            else:
                pass
    moo_d[p] = np.unique(moo_d[p]).tolist()
    emo_list.append(emo_d)


I am working on the file "Notebooks/emo.ipynb".
The first task is:
Plot emotional vectors
Plots emotional vectors
rescale | boolean
    + True rescales the vectors with paragraph length.
Can you write Python code for it?

rescale = True
if rescale:
    slength = [float(len(doc.paragraphs[i].text.split())) for i in range(len(doc.paragraphs))]
else:
    slength = 1
x = book_emotions['anger'].as_matrix()/slength
y = book_emotions['disgust'].as_matrix()/slength
z = book_emotions['sadness'].as_matrix()/slength
fig = plt.figure('emotional correlation',figsize=(7,7))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(xs=x,ys=y,zs=z)
ax.set_xlabel('anger')
ax.set_ylabel('disgust')
ax.set_zlabel('sadness')
plt.draw()
plt.show()


I am working on the file "Notebooks/emo.ipynb".
The first task is:
Print the paragraph and emotional vector corresponding to the n-th highest sorted paragraph
pidx | int
+ starts from 0, chooses the paragraph correponding to the pidx-th highest sorted paragraph
Can you write Python code for it?

pidx = 1
print("paragraph from text\n%s\n" %doc.paragraphs[emo_rank[pidx]].text)
print("moo_d emo Harry feels\n%s\n" %book_mood.iloc[emo_rank[pidx]].dropna())
print("emo vector\n%s" %book_emotions.iloc[emo_rank[pidx]])


I am working on the file "Notebooks/emo.ipynb".
The first task is:
Read NRC lexicon
The NRC emotional lexicon is a curated list of words with
+ sentiments : positive and negative
+ emotions: anger, anticipation, disgust, fear, joy, sadness, surprise, trust
Details of this work may be found at Saif Mohammad's homepage
The HDF5 file is cleaned from the original NRC text file.
The resulting pandas dataframe has the following columns
+ word | string
    + words in the lexicon
+ emo | string
    + the complete set of sentiments and emotion
+ binary | int
    + 0 if the word does not correspond to the sentiment or emotion
    + 1 if the word elicits the sentiment or emotion
Can you write Python code for it?


I am working on the file "Notebooks/emo.ipynb".
The first task is:
Read raw text
Seven books of Harry Potter in text format converted from rich text format.
Accessible by setting book to
+ book1.txt
+ ...
+ book7.txt
The raw text is read through pv.Doc and split into paragraph objects.
Can you write Python code for it?

book = 'book7'
doc = pv.Doc('../../clusterpot/%s.txt' %book)
print("title: %s" %doc.paragraphs[0].text)


I am working on the file "Notebooks/emo.ipynb".
The first task is:
Simple example: choose emotion to sort by
Choose emo_kw from list ['anger', disgust', 'fear', 'joy', 'sadness', 'surprise']
Can you write Python code for it?

emo_kw = 'anger'
emo_rank = np.argsort(book_emotions[emo_kw].as_matrix())[::-1]
emo_rank


I am working on the file "Notebooks/emo.ipynb".
The first task is:
Test code
Can you write Python code for it?

slength = [float(len(doc.paragraphs[i].text.split())) for i in range(len(doc.paragraphs))]
y = np.sum(book_emotions.as_matrix(),axis=1)
plt.figure('paragraph length vs. emotion sum',figsize=(7,7))
ax = plt.axes([0.15,0.15,0.8,0.8])
ax.scatter(x=slength,y=y)
#ax.set_xlim([0,1])
#ax.set_ylim([0,1])
#ax.set_xscale('log')
plt.draw()
plt.show()

slength = [float(len(doc.paragraphs[i].text.split())) for i in range(len(doc.paragraphs))]
y = np.sum(book_emotions.as_matrix(),axis=1)
plt.figure('paragraph length vs. emotion sum',figsize=(7,7))
ax = plt.axes([0.15,0.15,0.8,0.8])
ax.scatter(x=slength,y=y)
#ax.set_xlim([0,1])
#ax.set_ylim([0,1])
#ax.set_xscale('log')
plt.draw()
plt.show()

s = "Upon the signature of the International Statute of Secrecy in 1689, wizards went into hiding for good. It was natural, perhaps, that they formed their own small communities within a community. Many small villages and hamlets attracted several magical families, who banded together for mutual support and protection. The villages of Tinworsh in Cornwall, Upper Flagley in Yorkshire, and Ottery St. Catchpole on the south coast of England were notable homes to knots of Wizarding families who lived alongside tolerant and sometimes Confunded Muggles. Most celebrated of these half-magical dwelling places is, perhaps, Godrics Hollow, the West Country village where the great wizard Godric Gryffindor was born, and where Bowman Wright, Wizarding smith, forged the first Golden Snitch. The graveyard is full of the names of ancient magical families, and this accounts, no doubt, for the stories of hauntings that have dogged the little church beside it for many centuries."
w = s.split()[0]
%timeit re.sub("[!@#$%^&*()_+:;,.\\?']", "", w)

%%timeit
punctuation = ["!","@","#","$","%","^","&","*","(",")","_","+",":",";",",",".","?","'","\\"]
for punc in punctuation:
    w.replace("%s" %punc,'')

moodf = pd.DataFrame.from_dict(moo_d, orient='index')

moodf.iloc[25].dropna()

moodf.to_hdf('./book_moo_v2.h5', book)

book_mood =...


I am working on the file "Notebooks/emo.ipynb".
The first task is:
Write emotional vectors to HDF5
HDF5 is structured as
+ bookn
    + len(doc.paragraphs) x len(emo_d.keys()) array of integers
where n spans 1 through 7.
Can you write Python code for it?


I am working on the file "Notebooks/errorCorrelation_plotting.ipynb".
The first task is:
Add in the mode-ages
Can you write Python code for it?

conAges = pd.read_csv("../../Main/main_HUMAN.csv",index_col=0)["modeAge"]
conAges.head()

statsAges = all_stats.join(conAges).sort("modeAge")
statsAges.head()

ages = ["Cellular_organisms","Euk+Bac","Euk_Archaea","Eukaryota",
        "Opisthokonta","Eumetazoa","Vertebrata","Mammalia"]

ageLengths = {}

for i in ages:
    df = statsAges[statsAges["modeAge"] == i]
    assert len(df) > 50 # just checkin'
    ageLengths[i] = df["Protein Length"].mean()


I am working on the file "Notebooks/errorCorrelation_plotting.ipynb".
The first task is:
Average protein lengths in each age category. No strong correlation with age, but Mammals may be enriched for small proteins.
Can you write Python code for it?

ax = pd.Series(ageLengths,index=ages).plot(color='black')
plt.xticks(rotation=70)
ax.set_ylabel("Average Protein Length")
ax.set_title("Average Protein Lengths in each Age Category")

#plt.savefig("lengthsXage.svg")


I am working on the file "Notebooks/errorCorrelation_plotting.ipynb".
The first task is:
Average similarity in each age class. Mammals definitely enriched for more quickly evolving proteins. Some of these may be older than "Mammal" and are false-negatives.
Can you write Python code for it?

ageSims = {}

for i in ages:
    df = statsAges[statsAges["modeAge"] == i]
    assert len(df) > 50 # just checkin'
    ageSims[i] = df["Hs-Mm_evoSim"].mean()

ax = pd.Series(ageSims,index=ages).plot(color='black')
plt.xticks(rotation=70)
ax.set_ylabel("Average Hs-Mm Similarity")
ax.set_title("Hs-Mm Evolutionary Similarity in each Age Category")

#plt.savefig("Hs-Mm_similarityXage.svg")


I am working on the file "Notebooks/errorCorrelation_plotting.ipynb".
The first task is:
Distribution of protein lengths. Most are short
Can you write Python code for it?


I am working on the file "Notebooks/errorCorrelation_plotting.ipynb".
The first task is:
Human-yeast evolutionary rates and node errors

Now we'll take a quick look at yeast-human similarities. We used mouse above because there are many one-to-one orthologs, but the tradeoff is a small dynamic range. Human-yeast similarities have a stronger correlation with node error than human-mouse.
Can you write Python code for it?

yeast_human = pd.read_csv("h-y_similarities.csv",index_col=0,names=["Hs-Sc_sim"]).sort_index()

error_yeastSims = pd.concat([nodeStats,yeast_human],axis=1).dropna()[["Hs-Sc_sim","NodeError"]]
error_yeastSims = error_yeastSims.convert_objects(convert_numeric=True)
print "Number of alignments: %d" % len(error_yeastSims)

error_yeastSims.corr()

error_yeastSims.plot(kind='scatter',y="NodeError",x="Hs-Sc_sim",color='grey',title="Yeast-Human Distances Scatter")

#plt.savefig("Yeast-Human_dists_scatter.png",dpi=600)


I am working on the file "Notebooks/errorCorrelation_plotting.ipynb".
The first task is:
Plot absolute Spearman rank correlations with Node Error
Can you write Python code for it?

absCorrs = correlations.applymap(np.abs)

absCorrs.drop("Node Error",axis=1).ix["Node Error",].plot(
    kind='bar',color='grey',ylim=(0,1),title="Correlation with Node Error")
plt.ylabel("Absolute Spearman corr. coef.")

#plt.savefig("correlation_barPlot.svg")

# correlations.to_csv("errorCorrelations.csv")


I am working on the file "Notebooks/errorCorrelation_plotting.ipynb".
The first task is:
Read in the different files with error-source information
Can you write Python code for it?

# Protein lengths are from Uniprot proteome. Number of domains was inferred with hmmer. Similarities here are derived
# from pairwise alignments (needleman-wunsch) between one-to-one human and mouse orthologs, defined by PhylomeDB.
# We look atHuman-Yeast pairs below, but chose to use Mouse here because there are more one-to-ones.

lengths = pd.read_csv("lengthMapping.csv",index_col=0,names=["protLength"]).sort_index()
numDomains = pd.read_csv("numDomains.csv",index_col=0).sort_index()
mouse_human = pd.read_csv("h-m_similarities.csv",index_col=0,names=["Hs-Mm_sim"]).sort_index()

nodeStats = pd.read_csv("../../nodeStats/nodeStats_HUMAN.csv",index_col=0).sort_index()[["Bimodality","NodeError"]]

lossTaxa = pd.read_csv("../../Errors/lossStats_HUMAN.csv",index_col=0).sort_index()[["mean"]]
lossTaxa.columns = ["meanLossTaxa"]

fracOverSplit = pd.read_csv("../../Errors/HUMAN_LDO_summary.csv",index_col=0).sort_index()
fracOverSplit.columns = ["fracOverSplit"]


I am working on the file "Notebooks/errors_plotting.ipynb".
The first task is:
False positives

False positives are defined as algorithms that, for a given gene, infer an outsized number of losses for that orthogroup.
My programs output mean number of taxa that the algorithms inferred to have had lost the orthogroup, and variance of this number for each gene.
It also identifies algorithms that have an outsize number of taxa (2 standard deviations above the mean). These are listed in the outlier column, if they were found.
Can you write Python code for it?

stats2 = pd.read_csv("lossStats_HUMAN.csv",index_col=0)

stats2.fillna({"mean":np.nan,"variance":np.nan,"outliers":0},inplace=True)
stats2.head()


I am working on the file "Notebooks/errors_plotting.ipynb".
The first task is:
Let's look at distribution of the mean and variance
Can you write Python code for it?

ax = stats2["variance"].hist(bins=50,color='grey')
ax.set_title("Variance histogram, all genes")
ax.set_ylabel("Number of genes")

#plt.savefig("variance_histogram.svg")

stats_outliers = stats2[stats2["outliers"] != 0]
ax = stats_outliers["variance"].hist(bins=50,color='grey')
ax.set_title("Variance, genes with outliers")
ax.set_ylabel("Number of genes")

ax = stats2["mean"].hist(bins=50,color='grey')
ax.set_title("Histogram of mean values, all genes")
ax.set_ylabel("Number of genes")
#plt.savefig("mean_histogram.svg")

ax = stats_outliers["mean"].hist(bins=50,color='grey')
ax.set_title("Mean, genes with outliers")
ax.set_ylabel("Number of genes")


I am working on the file "Notebooks/errors_plotting.ipynb".
The first task is:
Proportional error by database

Normalized error counts by database.
Can you write Python code for it?

normErrors = errors/errors.sum()
normErrors["sumErrors"] = normErrors["FalseNegative"] + normErrors["FalsePositive"]
normErrors["normSum"] = normErrors["sumErrors"]/normErrors["sumErrors"].sum()
normErrors.sum()

normErrors["normSum"].plot(kind='bar',color='grey')

#plt.savefig("totalErrors.svg")


I am working on the file "Notebooks/gaussian_and_lines.ipynb".
The first task is:
Generate fake Gaussian noise
The procedure for making colored Gaussian noise is:
 1. Generate white Gaussian noise by many independent draws from a normal distribution.
 1. Transform this to the frequency domain.
 1. Choose the spectrum you want to simulate, sampled at the same frequencies as the frequency-domain white noise.
 1. Multiply the FD white noise by the ASD of the desired spectrum.
 1. Transform this back to the time domain.
 1. Cut off the start and end of this data, to eliminate the filter wraparond. We cut off a quarter of the noise stream at either end, which should be more than enough with such a smooth PSD.
Can you write Python code for it?

# Make the data twice as long so we can cut off the wrap-around
num_noise_samples=2*time_length_seconds*sample_rate
white_noise_fd=rfft(np.random.normal(size=num_noise_samples))
sim_freqs=np.arange(len(white_noise_fd))/(2.*time_length_seconds)

psd=(sim_freqs/40.)**-10+(sim_freqs/70.)**-4+0.5+1.397e-6*(sim_freqs)**2
# Put in a fake highpass around 8 Hz, so we don't have too much low frequency
to_bin=2*time_length_seconds
f_pass, f_min = 8., 10.
idx1=int(to_bin*f_pass)
idx2=int(to_bin*f_min)
psd[:idx1]=psd[idx2]*(sim_freqs[:idx1]/f_pass)**2
psd[idx1:idx2]=psd[idx2]

colored_noise_td = np.sqrt(float(nyquist))*irfft(np.sqrt(psd)*white_noise_fd)
colored_noise_td = colored_noise_td[len(colored_noise_td)/4:-len(colored_noise_td)/4]

def welch_asd(data, fft_len_sec, overlap=0.5, window='hanning'):
    """Measure the ASD using the Welch method of averaging
    estimates on shorter overlapping segments of the data."""
    assert 0. <= overlap < 1.
    ff, tmp = sig.welch(data, fs=sample_rate, window=window,
                        nperseg=fft_len_sec*sample_rate,
                        noverlap=overlap*fft_len_sec*sample_rate)
    return ff, np.sqrt(tmp)

ff, measured_asd_bc = welch_asd(colored_noise_td, window='boxcar', fft_len_sec=4)
ff, measured_asd = welch_asd(colored_noise_td, window='hanning', fft_len_sec=4)

plt.loglog(ff, measured_asd_bc, label='no window')
plt.loglog(ff, measured_asd, label='hann')
plt.loglog(sim_freqs, np.sqrt(psd), c='k', ls=':', label='true')
plt.xlim(4,nyquist)
plt.ylim(0.5,2e3)
plt.legend(loc='upper right')


I am working on the file "Notebooks/gaussian_and_lines.ipynb".
The first task is:
The estimate using the Hann window recovers the true spectrum very well. However, forgetting to use a window causes something called 'spectral leakage'. The large power from low frequencies leaks into the higher frequencies and the spectral estimate is wrong. This has a characteristic smooth, slowly falling shape. You can see that in the code above we use a 'boxcar' window; this is a rectangular window tht goes instantly from one to zero, and is the same as not using a window.
Gaussian noise with lines
Can you write Python code for it?

def line_at(f0, data):
    """Make a sinusoid at f0 Hz with the same length as data"""
    return np.sin(2.*np.pi*f0*np.arange(len(data))/sample_rate)

noise_with_lines = colored_noise_td\
                    + 100.*line_at(35.9, colored_noise_td)\
                    + 100.*line_at(36.7, colored_noise_td)

ff16, measured_asd_lines_bc = welch_asd(noise_with_lines, window='boxcar', fft_len_sec=16)
ff16, measured_asd_lines = welch_asd(noise_with_lines, window='hanning', fft_len_sec=16)

plt.loglog(ff16, measured_asd_lines_bc, label='no window')
plt.loglog(ff16, measured_asd_lines, label='hann window')
plt.loglog(sim_freqs, np.sqrt(psd), c='k', ls=':', label='true')
plt.xlim(30,40)
plt.ylim(0.5,2e3)
plt.legend(loc='upper right')

plt.loglog(ff16, measured_asd_lines_bc, label='no window')
plt.loglog(ff16, measured_asd_lines, label='hann window')
plt.loglog(sim_freqs, np.sqrt(psd), c='k', ls=':', label='true')
plt.xlim(10,2000)
plt.ylim(0.5,2e3)
plt.legend(loc='upper right')


I am working on the file "Notebooks/gaussian_and_lines.ipynb".
The first task is:
We're going to simulate data discretely sampled in time. We need to set a sample rate in Hz. The highest resolvable frequency, the Nyquist frequency, is half the sample rate. We also need to decide how long the simulated data record should last.
Can you write Python code for it?

sample_rate = 4096
nyquist = sample_rate/2
time_length_seconds = 512


I am working on the file "Notebooks/get_all_MAFs.ipynb".
The first task is:

GLOBAL VARIABLE WARNING

Here I download updated clinical data from the TCGA Data Portal.
This is a secure site which uses HTTPS.  I had to give it a path
to my ca-cert for the download to work.

Download a copy of a generic cacert.pem [here](http://curl.haxx.se/ca/cacert.pem).

Can you write Python code for it?


I am working on the file "Notebooks/get_all_MAFs.ipynb".
The first task is:
Get gene by patient mutation count matrix and save
Can you write Python code for it?

m5 = m4.ix[m4.Variant_Classification != 'Silent']
cc = m5.groupby(['Hugo_Symbol','Tumor_Sample_Barcode']).size()
cc = cc.reset_index()

cc.to_csv(out_path + 'meta.csv')

cc.shape


I am working on the file "Notebooks/get_all_MAFs.ipynb".
The first task is:
Use BeutifulSoup to parse out all of the links in the table
Can you write Python code for it?

f = open('tmp.html', 'rb').read()
soup = BeautifulSoup(f)

r = [l.get('href') for l in soup.find_all('a')
   if l.get('href') != None
   and '.maf' in l.get('href')]


I am working on the file "Notebooks/gmm_pair_M33.ipynb".
The first task is:
Inside the Likelihood object is a "triplet" object called T, which contains an array of sample local groups, each with kinematic parameters consistent with the observational data. Let's plot these kinematic parameters in a "triangle" figure, to show all their 1 and 2-D marginal distributions.
Can you write Python code for it?

L = localgroup.Likelihood(isPair=True)
L.generate(Nsamples=200000)

L.set_PDF(mixture.GMM(n_components=10, covariance_type='full'))
L.approximate()

figure_obs = L.plot_samples(10, color='b', overlay=False)


I am working on the file "Notebooks/gmm_pair_M33.ipynb".
The first task is:
The above plot shows a Gaussian Mixture model fitted Gaussians. The shaded regions show two standard deviations. The samples data has been preprocessed to zero the mean and scale by standard deviation. Since we are using the Gaussian Mixture Model to model the underlying PDF of the data, more components is always better.
How to evaluate goodness of fit:
Due to lack of a standard goodness of fit test for GMM's, the best we can do is graphically show that the model reproduces the data well. We proceed by drawing a set of points from the fitted model, where each point is a local group with (MW_D, MW_vr, MW_vt, M33_D, M33_vr, M33_vt). We then plot the 1D and 2D marginalizations of the drawn point set and show that the marginalizations match the marginalizations of the true data.
Can you write Python code for it?

figure_model = L.model_gof(L.T.Nsamples, color="r", fig=None)

L.model_gof(L.T.Nsamples, color="r", fig=figure_obs)
red_patch = mpatches.Patch(color='red')
blue_patch = mpatches.Patch(color='blue')
figure_obs.legend(handles=[red_patch, blue_patch], labels=["Model Generated", "Observation Generated"])

figure_obs


I am working on the file "Notebooks/gmm_pair_M33.ipynb".
The first task is:
The above plot shows that the points drawn from the model create a population that is very similar to the true data.
Can you write Python code for it?


I am working on the file "Notebooks/gmm_pair_prior.ipynb".
The first task is:
Inside the Likelihood object is a "triplet" object called T, which contains an array of sample local groups, each with kinematic parameters consistent with the observational data. Let's plot these kinematic parameters in a "triangle" figure, to show all their 1 and 2-D marginal distributions.
Can you write Python code for it?


I am working on the file "Notebooks/gmm_pair_prior.ipynb".
The first task is:
Reading Simulation Points:
Below we read the preconfigured files containing the Consuelo (soon to be Dark Sky) Local Group analogs into a Triplet object. We plot the marginalizations of the simulation data, which allows us to compare with the LG prior.
Can you write Python code for it?

path = '/lustre/ki/pfs/mwillia1/LG_project/Consuelo_Boxes/All_Boxes/Pairs/All_Boxes_dat_M31_larger.npy'
npoints = 100000
halo_props = ['MW_Mvir', 'M31_Mvir', 'M33_Mvir']

Tr = localgroup.Triplet(isPair=True)
Tr.read_sim_points(path, npoints, halo_props, h=0.7, a=1.0, npy=True)

Tr.transform_to_M31(sim=True)

#Tr.mass_filter('sim')

Tr.dist_filter((Tr.sim_samples[:,0] < 10))

Tr.preprocess(L.samples_means, L.samples_stds, mode='sim')

Tr.unprocess(L.samples_means, L.samples_stds, mode='sim')
data = np.transpose(np.vstack((np.transpose(Tr.sim_samples), np.log10(Tr.MW.Mvir), np.log10(Tr.M31.Mvir))))
labs=["MWd", "MWvr", "MWvt", "MWMvir", "M31Mvir"]
sim_plot = triangle.corner(data, labels=labs, quantiles=[0.16,0.5,0.84], fig=pl, weights=None,\
                         plot_contours=True, show_titles=True, title_args={"fontsize": 16}, \
                         plot_datapoints=False, bins=20, color='r', label_args={"fontsize": 16})
red_patch = mpatches.Patch(color='r')
cyan_patch = mpatches.Patch(color='c')
sim_plot.legend(handles=[red_patch, cyan_patch], labels=["CONSUELO Prior", "GMM Prior Approximation"])
Tr.preprocess(L.samples_means, L.samples_stds, mode='sim')

#name = 'gmm_CONSUELO_prior.png'
#sim_plot.savefig('/afs/slac.stanford.edu/u/ki/mwillia1/Thesis/LocalGroupHaloProps/doc/thesis/plots/asurps/'+name)

pl

Tr.unprocess(L.samples_means, L.samples_stds, 'gmm')
print np.mean(Tr.sim_samples[:,0])
Tr.preprocess(L.samples_means, L.samples_stds, mode='gmm')

def num_bins(std, rg, N):
    return rg/(3.5*std/N**(1.0/3.0))

num_bins(5.7)

dat = np.transpose(np.vstack((np.transpose(Tr.sim_samples), np.log10(Tr.M31.Mvir), np.log10(Tr.MW.Mvir))))
Tr.GMM(20, dat)

Tr.GMM_sample(500000, L, reps=1, simple=True)

gmm_MW = np.copy(Tr.gmm_samples[:,4])
gmm_M31 = np.copy(Tr.gmm_samples[:,3])
gmm_LG = np.log10(np.power(10,gmm_MW) + np.power(10,gmm_M31))

#cond = gmm_MW < gmm_M31
#Tr.gmm_samples = Tr.gmm_samples[cond]
#gmm_MW = gmm_MW[cond]
#gmm_M31 = gmm_M31[cond]
#gmm_LG = gmm_LG[cond]

Tr.gmm_samples =...


I am working on the file "Notebooks/gmm_pair_prior.ipynb".
The first task is:
The above plot shows a Gaussian Mixture model fitted Gaussians. The shaded regions show two standard deviations. The samples data has been preprocessed to zero the mean and scale by standard deviation. Since we are using the Gaussian Mixture Model to model the underlying PDF of the data, more components is always better.
How to evaluate goodness of fit:
Due to lack of a standard goodness of fit test for GMM's, the best we can do is graphically show that the model reproduces the data well. We proceed by drawing a set of points from the fitted model, where each point is a local group with (MW_D, MW_vr, MW_vt, M33_D, M33_vr, M33_vt). We then plot the 1D and 2D marginalizations of the drawn point set and show that the marginalizations match the marginalizations of the true data.
Can you write Python code for it?


I am working on the file "Notebooks/gmm_pair_prior.ipynb".
The first task is:
The above plot shows that the points drawn from the model create a population that is very similar to the true data.
Can you write Python code for it?


I am working on the file "Notebooks/graficos-hyperboide-y-cono.ipynb".
The first task is:
Cono (R=0)
Can you write Python code for it?

R=0
X=r(Z)*cos(PHI) #Definiendo dominio en x
Y=r(Z)*sin(PHI) #Definiendo dominio en y

fig = figure(figsize=(6,6))
ax = fig.gca(projection='3d')
surf = ax.plot_surface(X, Y, Z, cmap=cm.jet,rstride=3, cstride=3, alpha=1)
ax.set_xlabel('$x$',fontsize=18)
ax.set_ylabel('$y$',fontsize=18)
ax.set_zlabel('$z(x,y)$',fontsize=18)
#show()


I am working on the file "Notebooks/graficos-hyperboide-y-cono.ipynb".
The first task is:
Hiperboloide
Consideramos primero el hiperboloide definido por
$$x^2+y^2-z^2=-R^2.$$
Primero usaremos coordenadas cartesianas en el plano xy, y creamos una malla que usaremos para evaluar los valores de z correspondientes:
Can you write Python code for it?

R=5.
def z(x,y):
    return sqrt(x**2+y**2+R**2.)

x = linspace(-10,10,100) #Definiendo el dominio en x
y = linspace(-10,10,100) #Definiendo el dominio en y
X, Y = meshgrid(x, y) #Formando la grilla x,y

fig = figure(figsize=(6,6))
ax = fig.gca(projection='3d')
surf = ax.plot_surface(X, Y, z(X,Y), cmap=cm.jet,rstride=3, cstride=3, alpha=1)
ax.set_xlabel('$x$',fontsize=18)
ax.set_ylabel('$y$',fontsize=18)
ax.set_zlabel('$z(x,y)$',fontsize=18)
#show()

fig = figure(figsize=(6,6))
ax = fig.gca(projection='3d')
surf1 = ax.plot_surface(X, Y, z(X,Y), cmap=cm.jet,rstride=3, cstride=3, alpha=1)
surf2 = ax.plot_surface(X, Y, -z(X,Y), cmap=cm.jet,rstride=3, cstride=3, alpha=1)
ax.set_xlabel('$x$',fontsize=18)
ax.set_ylabel('$y$',fontsize=18)
ax.set_zlabel('$z(x,y)$',fontsize=18)
#show()


I am working on the file "Notebooks/graficos-hyperboide-y-cono.ipynb".
The first task is:
Todo luce un poco mejor usando coordenadas polares en el plano xy
Can you write Python code for it?

def r(z):
    return sqrt(z**2+R**2.)

z = linspace(-15,15,100)
ph = linspace(0,2.*pi,100)
Z,PHI = meshgrid(z,ph)

X=r(Z)*cos(PHI) #Definiendo dominio en x
Y=r(Z)*sin(PHI) #Definiendo dominio en y

fig = figure(figsize=(6,6))
ax = fig.gca(projection='3d')
surf = ax.plot_surface(X, Y, Z, cmap=cm.jet,rstride=3, cstride=3, alpha=1)
ax.set_xlabel('$x$',fontsize=18)
ax.set_ylabel('$y$',fontsize=18)
ax.set_zlabel('$z(x,y)$',fontsize=18)
#show()


I am working on the file "Notebooks/historical_estimated_values.ipynb".
The first task is:
Displaying data for user selected week w
For each week w selected by the user, the notification curve will always be that which is found on df_inci, while the estimates will be that stored in df_hist. Data df_inci only has the most recent estimates, which are based on the most recent week with data. The estimates obtained at each week is stored at df_hist.
So, first of all, we will slice the historical data to week w, and limit current data to week <= w.
If w=23, the historical dataset is already correctly sliced in df_hist2017, so we just have to limit the current for the proper plot:
Can you write Python code for it?

df_hist[
        (df_hist.base_epiyear == 2017) &
        (df_hist.dado == 'srag') &
        (df_hist.escala == 'incidência') &
        (df_hist.uf == 'BR')
        ].base_epiweek.unique()

# First, last keep only stable weeksfor notification curve:
df_inci2017.loc[(df_inci2017.situation != 'stable'), 'srag'] = np.nan

# Adapt historical dataset:
df_hist.sort_values(['epiyear', 'epiweek'], inplace=True)
df_hist['estimated_cases'] = df_hist['50%']

# User selected week:
y = 2017
w = 23

def week_data(y, w):
    df_week_inci = df_inci2017[(df_inci2017.epiweek <= w)]

    df_week_hist = df_hist[
        (df_hist.base_epiyear == y) &
        (df_hist.base_epiweek == w) &
        (df_hist.dado == 'srag') &
        (df_hist.escala == 'incidência') &
        (df_hist.uf == 'BR')
    ].copy()

    df = pd.merge(
        df_week_inci[['epiweek', 'srag']],
        df_week_hist[['epiweek', 'estimated_cases', '2.5%', '97.5%']],
        on='epiweek', how='outer'
    )

    df.set_index('epiweek', inplace=True)
    return df

df = week_data(y, w)
df.plot()
plt.grid(True)
plt.show()

w = 28
df = week_data(y, w)
df.plot()
plt.grid(True)
plt.show()

w = 33
df = week_data(y, w)
df.plot()
plt.grid(True)
plt.show()


I am working on the file "Notebooks/historical_estimated_values.ipynb".
The first task is:
Entries with dfthresholds['se típica do inicio do surto'] = NaN have activity too low for proper epidemic threshold definition
Can you write Python code for it?

k = ['epiyear', 'epiweek', 'base_epiyear', 'base_epiweek']
df_inci2017 = df_inci[
    (df_inci.epiyear == 2017) &
    # (df_inci.epiweek >= 15) &
    (df_inci.dado == 'srag') &
    (df_inci.escala == 'incidência') &
    (df_inci.uf == 'BR')
].copy()

df_inci2017.sort_values(['epiyear', 'epiweek'], inplace=True)
df_inci_chart = df_inci2017.copy()
df_inci_chart.index = df_inci_chart.epiweek

k = ['epiyear', 'epiweek', 'base_epiyear', 'base_epiweek']
df_hist2017 = df_hist[
    (df_hist.base_epiyear == 2017) &
    (df_hist.base_epiweek == 23) &
    (df_hist.dado == 'srag') &
    (df_hist.escala == 'incidência') &
    (df_hist.uf == 'BR')
].copy()

df_hist2017.sort_values(['epiyear', 'epiweek'], inplace=True)

df_hist_chart = df_hist2017.copy()
df_hist_chart.index = df_hist_chart.epiweek

# 50% estimated cases

df_inci_chart[['srag', '50%', '2.5%', '97.5%']].plot()
plt.title('Incidence')
plt.grid(True)
plt.show()

df_hist_chart[['srag', '50%', '2.5%', '97.5%']].plot()
plt.title('Historial')
plt.grid(True)

plt.show()

df_hist2017['estimated_cases'] = df_hist2017['50%']

df = pd.merge(
    df_inci2017[['epiweek', 'srag', '2.5%', '97.5%']],
    df_hist2017[['epiweek', 'estimated_cases']],
    on='epiweek', how='outer'
)

df.set_index('epiweek', inplace=True)

df.plot()
plt.grid(True)
plt.title('Incidence X Historial')
plt.show()


I am working on the file "Notebooks/image_search_query.ipynb".
The first task is:
Deciding which search engine to use
We'll probably get good enough results by trying the Noun Project first and using Bing for everything else.
Can you write Python code for it?

def submit_query(query):
    """
    Decide which search engine to use based on the part of speech of the query
    """
    doc = textacy.Doc(query, lang='en')
    try:
        urls = search_np_for_image(query)
    except Exception as e:
        urls = search_bing_for_image(query)
    return urls

par = book1.paragraphs[400]
print(par.text)
query = build_query(par)
print(query)
urls = submit_query(query)
view_urls(urls[:1])


I am working on the file "Notebooks/image_search_query.ipynb".
The first task is:
Finding important words
My teammates and I have done a lot of work these past few weeks extracting the names of people, places, things, activities, and moods from text. This work has been incorporated into a Python package, Pensieve. I'll load up the package and take a look at the words that were extracted from the paragraphs in Harry Potter and the Sorcerer's Stone.
Can you write Python code for it?

import pensieve

book1 = pensieve.Doc('../../corpus/book1.txt', doc_id=1)

from pprint import pprint
from numpy.random import randint

rand = randint(len(book1.paragraphs))
print(book1.paragraphs[rand].text)
pprint(book1.paragraphs[rand].words)


I am working on the file "Notebooks/image_search_query.ipynb".
The first task is:
First, we'll look at textacy.keyterms.textrank. This implements the TextRank algorithm, which iteratively computes a score for each vertex in the graph that roughly corresponds to the number of vertices connected to that vertex.
Can you write Python code for it?


I am working on the file "Notebooks/image_search_query.ipynb".
The first task is:
It is nice that the objects and verbs are extracted, but there are too many to just throw into one search query. In order to find the most, let's play around with some features in textacy that can give some sort of importance ordering to the words in the paragraph. Most of these features effectively implement different vertex importance sorting algorithms on a semantic network, so a nice place to start might be at the semantic network itself
Can you write Python code for it?

import textacy
import networkx as nx

graph = textacy.network.terms_to_semantic_network(book1.paragraphs[400].spacy_doc)
print(book1.paragraphs[400].text)
textacy.viz.draw_semantic_network(graph);


I am working on the file "Notebooks/image_search_query.ipynb".
The first task is:
Now this may end up being a little too specific for our purposes. "gryffindor common room" would be a great result for this image, but "dumpy little witch" is not as good...
DivRank attempts to provide a ranking that balances node centrality with node diversity. Let's see how that fares.
Can you write Python code for it?


I am working on the file "Notebooks/image_search_query.ipynb".
The first task is:
This seems to work pretty well! I suppose in this example, I may have chosen "study", but "plant" still makes sense. Let's look at another paragraph to see what happens.
Can you write Python code for it?


I am working on the file "Notebooks/image_search_query.ipynb".
The first task is:
We may run into trouble when the most important nodes are character names. An image search with the query "ron" or even "ron harry potter" is unlikely to give us good results. More on this later...
What about other algorithms? Let's try SGRank. This algorithm improves upon TextRank by getting rid of unlikely keyword candidates and performing multiple rankings. It is also capable of outputting multiple word phrases.
Can you write Python code for it?


I am working on the file "Notebooks/inverse_spectrum_truncation.ipynb".
The first task is:
Here, I've left the filter corruption on. We need to remove half the total length of the filter on both sides because of the filter wrapround.
Can you write Python code for it?


I am working on the file "Notebooks/inverse_spectrum_truncation.ipynb".
The first task is:
Make a whitening filter
We need to make a whitening filter for the data. This is an FIR filter that, when applied to our colored noise, produces a decorrelated or 'white' time series. There are many ways to calculate this, and actually many different filters that would achieve this. We'll choose to make ours zero-phase -- the impulse response is acausal and symmetric around t=0. We'll use FFT-based methods for the design because they are simple and efficient.
First, we make a good estimate of the ASD, with enough frequency resolution to capture the important features. Then we do 1/asd to get a rough whitening filter. We zero out any frequencies we want to throw away (in this case, below 10 Hz because strain is not accurate down there). We transform to the time domain to get an impulse response, and roll it forwards so the response is centered at half the length of the filter.
Then we truncate it with a window function to make sure that it goes to zero smoothly. This is our time-domain FIR filter. The t=0 of this acausal filter is still in the middle, so make sure to compensate for the filter delay if using it this way.
The inverting and Fourier transforming it part may not be quite the best thing to do. Even better might be to measure the ASD at the desired resolution, then interpolate to higher resolution. That gives more space when computing the impulse response to avoid wraparound issues. The details of the interpolation shouldn't matter since you're truncating it anyway.
Can you write Python code for it?

def make_invasd(data, invasd_len=8, f_min=10.):
    """invasd_len is the length in seconds of the desired impulse response (the full width)"""
    ff, asd = welch_asd(data, window='hanning', fft_len_sec=invasd_len)
    invasd = 1./asd
    invasd[:int(f_min*invasd_len)]=0.
    invasd_td = irfft(invasd)
    invasd_td = sig.hann(len(invasd_td))*np.roll(invasd_td, len(invasd_td)/2)
    return invasd_td

invasd_len=8
invasd_trunc = make_invasd(colored_noise_td, invasd_len=invasd_len, f_min=10.)

plt.plot(invasd_trunc)


I am working on the file "Notebooks/inverse_spectrum_truncation.ipynb".
The first task is:
We'll make an analytical PSD that has about the same shape as H1's noise. We put in a fake highpass filter around 8 Hz so we don't have infinite noise at low frequency, and to match what the calibration code does (it only reconstructs accurately above 10 Hz).
Can you write Python code for it?

psd=(sim_freqs/40.)**-10+(sim_freqs/70.)**-4+0.5+1.397e-6*(sim_freqs)**2
# Put in a fake highpass around 8 Hz, so we don't have too much low frequency
to_bin=2*time_length_seconds
f_pass, f_min = 8., 10.
idx1=int(to_bin*f_pass)
idx2=int(to_bin*f_min)
psd[:idx1]=psd[idx2]*(sim_freqs[:idx1]/f_pass)**2
psd[idx1:idx2]=psd[idx2]

# Generate the noise
colored_noise_td = np.sqrt(float(nyquist))*irfft(np.sqrt(psd)*white_noise_fd)
colored_noise_td = colored_noise_td[len(colored_noise_td)/4:-len(colored_noise_td)/4]

def welch_asd(data, fft_len_sec, overlap=0.5, window='hanning'):
    """Measure the ASD using the Welch method of averaging
    estimates on shorter overlapping segments of the data."""
    assert 0. <= overlap < 1.
    ff, tmp = sig.welch(data, fs=sample_rate, window=window,
                        nperseg=fft_len_sec*sample_rate,
                        noverlap=overlap*fft_len_sec*sample_rate)
    return ff, np.sqrt(tmp)

ff, measured_asd = welch_asd(colored_noise_td, window='hanning', fft_len_sec=4)

plt.loglog(ff, measured_asd)
plt.loglog(sim_freqs, np.sqrt(psd), c='k', ls=':')
plt.xlim(4,nyquist)
plt.ylim(0.5,2e3)


I am working on the file "Notebooks/inverse_spectrum_truncation.ipynb".
The first task is:
We're going to simulate data discretely sampled in time. We need to set a sample rate in Hz. The highest resolvable frequency, the Nyquist frequency, is half the sample rate. We also need to decide how long the simulated data record should last.
Note that we'll assume everything is real in the time domain, so we use rFFT and rIFFT. That just means that we're only keeping positive frequencies in the frequency domain.
Can you write Python code for it?


I am working on the file "Notebooks/methylation_upregulated_probe_annotation.ipynb".
The first task is:
GSEA on disordered probes
Can you write Python code for it?

ann = DX.probe_annotations

d = dx_meth.frac.ix[ti(disordered == False)].dropna()
d.name = 'd'
dd = ann.join(d).sort('d').dropna()
vec = dd.set_index('Gene_Symbol')['d']

gs2 = gene_sets.ix[vec.index].fillna(0)
r2 = screen_feature(vec, rev_kruskal, gs2.T,
                    align=False)

r2.head(6)

draw_dist(vec, gs2['PID_RHOA_REG_PATHWAY'])


I am working on the file "Notebooks/methylation_upregulated_probe_annotation.ipynb".
The first task is:
Here is a different view, where I look at the overlap of disordered probes and annotations. This test takes into  account the global background whereas the previous does not. It is clearer here that PRC2 sites are more succeptable to drift than promoters and TSS.
Can you write Python code for it?

a,b = match_series((mm < .5), (dx_meth.frac > .5))
disordered = a == b
disordered.value_counts()

probe_sets = pd.DataFrame(DX.probe_sets)
screen_feature(disordered, fisher_exact_test, probe_sets.T)

fig, axs = subplots(1,2, figsize=(9,3))
draw_dist_by_annotation(dx_meth.frac.ix[ti(disordered)], axs[0])
draw_dist_by_annotation(dx_meth.frac.ix[ti(disordered == False)], axs[1])


I am working on the file "Notebooks/methylation_upregulated_probe_annotation.ipynb".
The first task is:
Taking a look at disorder in methylation

Here we call a probe disordered if it trends towards 50% methylation in the presence of the tumor
This follows from the guidelines of Hannum et. al and my paper in review (add link when published)
Can you write Python code for it?

fisher_exact_test(mm < .5, dx_meth.frac > .5)


I am working on the file "Notebooks/methylation_upregulated_probe_annotation.ipynb".
The first task is:
This seems as though most of the PRC2 effect is probes with low methylation levels increasing in cancer.
Can you write Python code for it?

mm = matched_meth.xs('11',1,1).mean(1)

fig, axs = subplots(1,4, figsize=(20,3))
draw_dist_by_annotation(dx_meth.frac.ix[ti(mm < .25)], axs[0]);
draw_dist_by_annotation(dx_meth.frac.ix[ti((mm < .5) & (mm > .25))], axs[1]);
draw_dist_by_annotation(dx_meth.frac.ix[ti((mm < .75) & (mm > .5))], axs[2]);
draw_dist_by_annotation(dx_meth.frac.ix[ti((mm > .75))], axs[3]);
for ax in axs[[0,1,2]]:
    ax.legend().set_visible(False)


I am working on the file "Notebooks/microarray_validation_aggregation.ipynb".
The first task is:
Group by tissue type first and then average. This is to limit the heavy skew of liver cancer samples in the GEO dataset.
Can you write Python code for it?

pos = (dx>0).groupby(tt, axis=1).sum()
count = dx.groupby(tt, axis=1).count().replace(0, np.nan)
count = count[count.sum(1) > 500]
frac_df = 1.*pos / count
frac_microarray = frac_df.mean(1)

fig, ax = subplots(figsize=(4,4))
s1, s2 = match_series(dx_rna.frac, frac_microarray)
plot_regression(s1, s2, density=True, rad=.02, ax=ax, rasterized=True,
                line_args={'lw':0})
ax.set_ylabel("GEO microarray")
ax.set_xlabel("TCGA mRNASeq")
ann = ax.get_children()[4]
ann.set_text(ann.get_text().split()[0])
ax.set_xticks([0, .5, 1])
ax.set_yticks([0, .5, 1])
fig.tight_layout()


I am working on the file "Notebooks/microarray_validation_aggregation.ipynb".
The first task is:
Simple average
Can you write Python code for it?

dx_simple = binomial_test_screen(microarray.ix[genes])

fig, ax = subplots(figsize=(4,4))
s1, s2 = match_series(dx_rna.frac, dx_simple.frac)
plot_regression(s1, s2, density=True, rad=.02, ax=ax, rasterized=True,
                line_args={'lw':0})
ax.set_ylabel("GEO microarray")
ax.set_xlabel("TCGA mRNASeq")
ann = ax.get_children()[4]
ann.set_text(ann.get_text().split()[0])
ax.set_xticks([0, .5, 1])
ax.set_yticks([0, .5, 1])
fig.tight_layout()


I am working on the file "Notebooks/nodeStats_plotting.ipynb".
The first task is:
Distribution of the bimodality statistic

Note the positive skew. The bimodality statisitc is defined as the average node error between the "old" and "young" groups of the algorithms minus the average node error within those groups. So the positive skew means that this grouping is capturing a true clustering with respect to node error. Genes falling below zero here are bimodal with respect to some other grouping, but are clearly in the minority
Can you write Python code for it?

nodestats["Bimodality"].hist(bins=50,color='grey')
plt.ylabel("Number of Genes")
plt.xlabel("Bimodality")

#plt.savefig("Polarization_distribution.svg")


I am working on the file "Notebooks/nodeStats_plotting.ipynb".
The first task is:
Get polarization statistics for each bin in node error histogram

I made a module to do the binning called histLinePlot. It makes a dataframe with the mean,standard deviation, and variance for the bimodality statistic in each bin in the node error histogram. The plots below visualize these statistics.The clear takeaway is that genes with more node error are more bimodal with respect to the "old" and "young" algorithms. There are therefore systematic differences between these algorithms that make determination of a true age very difficult for a substantial subset of genes
Can you write Python code for it?

%%capture
stats = histLinePlot.getLineScoreStats(nodestats,"Bimodality","NodeError")

stats.head()

fig,ax1 = plt.subplots()
nodestats["NodeError"].hist(bins=50,color='grey')
ax2 = ax1.twinx()
ax2.plot(stats.index,stats['mean'],'black',label="Avg Bimodality")
ax1.set_ylabel("Number of Genes")
ax1.set_xlabel("Avg. Node Error Between Algorithms")
ax2.set_ylabel("Average Bimodality")

plt.legend()

plt.savefig("nodeError-polarization_correlation.svg")


I am working on the file "Notebooks/nodeStats_plotting.ipynb".
The first task is:
Percent of proteins that are split bimodally between the "young" and "old" algorithm groups. Compare with those split between other groupings
Can you write Python code for it?

# Those split between other groups will have bimodality score <0 (greater within group difference
# than between)

len(nodestats[nodestats["Bimodality"] < 0])/float(len(nodestats))

# Bimodal or neutral genes (score greater than or equal to 0)

len(nodestats[nodestats["Bimodality"] >= 0])/float(len(nodestats))


I am working on the file "Notebooks/nodeStats_plotting.ipynb".
The first task is:
The scatter in the average bimodality of larger bins is due to small sample size and high variance
Can you write Python code for it?

fig,ax1 = plt.subplots()
nodestats["NodeError"].hist(bins=50,color='grey')
ax2 = ax1.twinx()
ax2.plot(stats.index,stats['var'],'black',label="Var Bimodality")
ax1.set_ylabel("Number of Genes")
ax1.set_xlabel("Avg. Node Error Between Algorithms")
ax2.set_ylabel("Variance Bimodality")

plt.legend()

#plt.savefig("nodeError-polarization_correlation.svg")

fig,ax1 = plt.subplots()
nodestats["NodeError"].hist(bins=50,color='grey')
ax2 = ax1.twinx()
ax2.plot(stats.index,stats['stanDev.'],'black',label="stdev Bimodality")
ax1.set_ylabel("Number of Genes")
ax1.set_xlabel("Std. Dev. Node Error Between Algorithms")
ax2.set_ylabel("Std. Dev. Bimodality")

plt.legend()

#plt.savefig("nodeError-polarization_correlation.svg")


I am working on the file "Notebooks/nonstationarity.ipynb".
The first task is:
To get the normalization correct, it helps to break the filter into two pieces. The filters are based on the estimated (long) power spectrum denoted $S_{E}(f)$.
The first is the whitener, which is the inverse of the ASD in the frequency domain. The normalization that we want is that the whitened noise has variance one in the time domain.
$$\widetilde{W}(f) = \frac{f_{s}}{2} ~ \frac{1}{\sqrt{S_{E}(f)}}$$
The second is a filter that weights by the power in the whitened template. It looks like
$$\widetilde{H}(f) = \mathcal{N} ~ \frac{f^{-7/6}}{\sqrt{S_{E}(f)}}$$
The normalization is chosen so that the variance will be one if this filter is white noise which already has a variance of one. In the frequency domain, the sum of $|\widetilde{H}(f)|^2$ must equal the number of frequency bins in the filter (not including the one at $f = 0$).
Can you write Python code for it?

# Make a bandpass filter to remove uninteresting frequencies
filt = sig.firwin(4*srate, [20.,480.], pass_zero=False, window='hann', fs=srate)
filt.resize(16*srate)

# Make the weighting filter - bandpass, weight by f^-7/6, and whiten
# This will give a divide by zero because of the f=0 point, ignore it
fweight = psd.frequencies**(-7./6.)*abs(rfft(filt))/sqrt(psd)
fweight[0] = 0. # Remove anything at f=0
norm = 1. / sqrt(sum(abs(fweight)**2) / (len(fweight)-1.))
fweight = norm * fweight.value

fwhiten = sqrt(2./srate)/sqrt(psd.value)
fwhiten[0] = 0. # Remove anything at f=0
td_whitener = sig.hann(16*srate)*roll(irfft(fwhiten), 8*srate)

full_filt = sig.hann(16*srate)*roll(irfft(fwhiten*fweight), 8*srate)

wdata = TimeSeries(sig.fftconvolve(data, full_filt, mode='same'),
                       sample_rate=data.sample_rate, epoch=data.epoch)[8*2048:-8*2048]

sum(abs(fweight)**2) / (16*2048/2.)

p1 = wdata.rms(8).plot()
p1.gca().set_ylim(0.8,1.2)


I am working on the file "Notebooks/numba/zh-cn/Basic.ipynb".
The first task is:
vectorize 的“并行”版本
Can you write Python code for it?

@nb.vectorize("float32(float32, float32)", target="parallel", nopython=True)
def add_with_vec(y, c):
    return y + c

assert np.allclose(y+1, add_with_vec(y,1.))
%timeit add_with_vec(y, 1.)
%timeit y + 1


I am working on the file "Notebooks/numba/zh-cn/Basic.ipynb".
The first task is:
注意：虽然这里使用parallel后更慢了，但如果使用 Intel Distribution for Python 的话，会发现parallel版本甚至会比numpy原生的版本要稍快一些
Can you write Python code for it?

@nb.vectorize("float32(float32, float32, float32)", target="parallel", nopython=True)
def clip_with_parallel(y, a, b):
    if y < a:
        return a
    if y > b:
        return b
    return y

@nb.vectorize("float32(float32, float32, float32)", nopython=True)
def clip(y, a, b):
    if y < a:
        return a
    if y > b:
        return b
    return y

assert np.allclose(np.clip(y, 0.1, 0.9), clip(y, 0.1, 0.9), clip_with_parallel(y, 0.1, 0.9))
%timeit clip_with_parallel(y, 0.1, 0.9)
%timeit clip(y, 0.1, 0.9)
%timeit np.clip(y, 0.1, 0.9)


I am working on the file "Notebooks/numba/zh-cn/Basic.ipynb".
The first task is:
注意：这个栗子中的性能提升就是实打实的了

总之，使用parallel时不能一概而论，还是要做些实验。
需要指出的是，vectorize中的参数target一共有三种取值：cpu（默认）、parallel和cuda。关于选择哪个取值，官方文档上有很好的说明：
text
A general guideline is to choose different targets for different data sizes and algorithms. The “cpu” target works well for small data sizes (approx. less than 1KB) and low compute intensity algorithms. It has the least amount of overhead. The “parallel” target works well for medium data sizes (approx. less than 1MB). Threading adds a small delay. The “cuda” target works well for big data sizes (approx. greater than 1MB) and high compute intensity algorithms. Transfering memory to and from the GPU adds significant overhead.
使用 jit(nogil=True) 实现高效并发（多线程）
Can you write Python code for it?

import math
from concurrent.futures import ThreadPoolExecutor

def np_func(a, b):
    return 1 / (a + np.exp(-b))

@nb.jit(nopython=True, nogil=False)
def kernel1(result, a, b):
    for i in range(len(result)):
        result[i] = 1 / (a[i] + math.exp(-b[i]))

@nb.jit(nopython=True, nogil=True)
def kernel2(result, a, b):
    for i in range(len(result)):
        result[i] = 1 / (a[i] + math.exp(-b[i]))

def make_single_task(kernel):
    def func(length, *args):
        result = np.empty(length, dtype=np.float32)
        kernel(result, *args)
        return result
    return func

def make_multi_task(kernel, n_thread):
    def func(length, *args):
        result = np.empty(length, dtype=np.float32)
        args = (result,) + args
        chunk_size = (length + n_thread - 1) // n_thread
        chunks = [[arg[i*chunk_size:(i+1)*chunk_size] for i in range(n_thread)] for arg in args]
        with ThreadPoolExecutor(max_workers=n_thread) as e:
            for _ in e.map(kernel, *chunks):
                pass
        return result
    return func

length = 10 ** 6
a = np.random.rand(length).astype(np.float32)
b = np.random.rand(length).astype(np.float32)

nb_func1 = make_single_task(kernel1)
nb_func2 = make_multi_task(kernel1, 4)
nb_func3 = make_single_task(kernel2)
nb_func4 = make_multi_task(kernel2, 4)

rs_np = np_func(a, b)
rs_nb1 = nb_func1(length, a, b)
rs_nb2 = nb_func2(length, a, b)
rs_nb3 = nb_func3(length, a, b)
rs_nb4 = nb_func4(length, a, b)
assert np.allclose(rs_np, rs_nb1, rs_nb2, rs_nb3, rs_nb4)
%timeit np_func(a, b)
%timeit nb_func1(length, a, b)
%timeit nb_func2(length, a, b)
%timeit nb_func3(length, a, b)
%timeit nb_func4(length, a, b)


I am working on the file "Notebooks/numba/zh-cn/CNN.ipynb".
The first task is:
可以看到，使用jit和使用纯numpy进行编程的很大一点不同就是，不要畏惧用for；事实上一般来说，代码“长得越像 C”、速度就会越快
Can you write Python code for it?

def max_pool_kernel(x, rs, *args):
    n, n_channels, pool_height, pool_width, out_h, out_w = args
    for i in range(n):
        for j in range(n_channels):
            for p in range(out_h):
                for q in range(out_w):
                    window = x[i, j, p:p+pool_height, q:q+pool_width]
                    rs[i, j, p, q] += np.max(window)

@nb.jit(nopython=True)
def jit_max_pool_kernel(x, rs, *args):
    n, n_channels, pool_height, pool_width, out_h, out_w = args
    for i in range(n):
        for j in range(n_channels):
            for p in range(out_h):
                for q in range(out_w):
                    window = x[i, j, p:p+pool_height, q:q+pool_width]
                    rs[i, j, p, q] += np.max(window)

@nb.jit(nopython=True)
def jit_max_pool_kernel2(x, rs, *args):
    n, n_channels, pool_height, pool_width, out_h, out_w = args
    for i in range(n):
        for j in range(n_channels):
            for p in range(out_h):
                for q in range(out_w):
                    _max = x[i, j, p, q]
                    for r in range(pool_height):
                        for s in range(pool_width):
                            _tmp = x[i, j, p+r, q+s]
                            if _tmp > _max:
                                _max = _tmp
                    rs[i, j, p, q] += _max

def max_pool(x, kernel, args):
    n, n_channels = args[:2]
    out_h, out_w = args[-2:]
    rs = np.zeros([n, n_filters, out_h, out_w], dtype=np.float32)
    kernel(x, rs, *args)
    return rs

pool_height, pool_width = 2, 2
n, n_channels, height, width = x.shape
out_h = height - pool_height + 1
out_w = width - pool_width + 1
args = (n, n_channels, pool_height, pool_width, out_h, out_w)

assert np.allclose(max_pool(x, max_pool_kernel, args), max_pool(x, jit_max_pool_kernel, args))
assert np.allclose(max_pool(x, jit_max_pool_kernel, args), max_pool(x, jit_max_pool_kernel2, args))
%timeit max_pool(x, max_pool_kernel, args)
%timeit max_pool(x, jit_max_pool_kernel, args)
%timeit max_pool(x,...


I am working on the file "Notebooks/pairs.ipynb".
The first task is:
Reading Simulation Points:
Below we read the preconfigured files containing the Consuelo (soon to be Dark Sky) Local Group analogs into a Triplet object. We plot the marginalizations of the simulation data, which allows us to compare with the LG prior.
Can you write Python code for it?

path = '/lustre/ki/pfs/mwillia1/LG_project/Consuelo_Boxes/4001/quad_dat_M31_larger_bgc2.npy'
#path = '/afs/slac.stanford.edu/u/ki/mwillia1/Thesis/data_files/MW_M31_pairs.txt'
npoints = 5000
halo_props = ['MW_Mvir', 'M31_Mvir', 'M33_Mvir']

Tr = localgroup.Triplet(isPair=True)
Tr.read_sim_points(path, npoints, halo_props, h=0.7, a=1.0, npy=True)

Tr.transform_to_M31(sim=True)

#Tr.mass_filter('sim')

Tr.dist_filter((Tr.sim_samples[:,0] < 1))

Tr.preprocess(L.samples_means, L.samples_stds, mode='sim')

sim_plot = Tr.plot_kinematics('sim', L.samples_means, L.samples_stds, color='c', fig=None)

Tr.plot_kinematics('sim', L.samples_means, L.samples_stds, color='g', fig=figure_model)

red_patch = mpatches.Patch(color='red')
green_patch = mpatches.Patch(color='green')
figure_model.legend(handles=[red_patch, green_patch], labels=["Model Generated", "Consuelo Triplets"])
figure_model

#name = 'trips_000_054_like_prior.png'
#sim_plot.savefig('/afs/slac.stanford.edu/u/ki/mwillia1/Thesis/LocalGroupHaloProps/doc/thesis/plots/asurps/'+name)

Tr.compute_model_weights(L, "sim")

N95 = Tr.calculate_N95()
N95

MW_C200 = Tr.calculate_cvir_to_c200(Tr.MW.Cvir)
M31_C200 = Tr.calculate_cvir_to_c200(Tr.M31.Cvir)
#M33_C200 = Tr.calculate_cvir_to_c200(Tr.M33.Cvir)

MW_M200 = Tr.calculate_Mvir_to_M200(Tr.MW.Mvir, Tr.MW.Cvir, MW_C200)
M31_M200 = Tr.calculate_Mvir_to_M200(Tr.M31.Mvir, Tr.M31.Cvir, M31_C200)
#M33_M200 = Tr.calculate_Mvir_to_M200(Tr.M33.Mvir, Tr.M33.Cvir, M33_C200)

total_200 = M31_M200+MW_M200

total = Tr.MW.Mvir + Tr.M31.Mvir
all_mvir = np.log10(np.transpose(np.vstack((Tr.MW.Mvir, Tr.M31.Mvir, total))))
labs = ["$M_{MW}$", "$M_{M31}$", "$M_{LG}$"]
#total = MW_M200 + M31_M200
#all_mvir = np.log10(np.transpose(np.vstack((MW_M200, M31_M200, total))))
#labs = ["MW Mvir", "M31 Mvir", "MW+M31"]

all_mvir = np.transpose(np.vstack((Tr.MW.Cvir, Tr.M31.Cvir)))
labs = ["$Cvir_{MW}$", "$Cvir_{M31}$"]

all_mvir.shape

figure = triangle.corner(all_mvir, labels=labs, quantiles=[0.16,0.5,0.84], fig=None,...


I am working on the file "Notebooks/pairs.ipynb".
The first task is:
The above plot shows a Gaussian Mixture model fitted Gaussians. The shaded regions show two standard deviations. The samples data has been preprocessed to zero the mean and scale by standard deviation. Since we are using the Gaussian Mixture Model to model the underlying PDF of the data, more components is always better.
How to evaluate goodness of fit:
Due to lack of a standard goodness of fit test for GMM's, the best we can do is graphically show that the model reproduces the data well. We proceed by drawing a set of points from the fitted model, where each point is a local group with (MW_D, MW_vr, MW_vt, M33_D, M33_vr, M33_vt). We then plot the 1D and 2D marginalizations of the drawn point set and show that the marginalizations match the marginalizations of the true data.
Can you write Python code for it?

figure_model = L.model_gof(L.T.Nsamples, color="r", fig=None)

L.model_gof(L.T.Nsamples, color="r", fig=figure_obs)
red_patch = mpatches.Patch(color='red')
blue_patch = mpatches.Patch(color='blue')
figure_obs.legend(handles=[red_patch, blue_patch], labels=["Model Generated", "Observation Generated"])

figure_obs


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Arbitrage_Pricing_Theory/notebook.ipynb".
The first task is:
As you can see, the plot scale massively affects how we perceive estimate quality.
Predicting the Future
Let's use this model to predict future prices for these assets.
Can you write Python code for it?

start_date = '2014-07-25'
end_date = '2015-07-25'

# We will look at the returns of an asset one-month into the future to model future returns.
offset_start_date = '2014-08-25'
offset_end_date = '2015-08-25'

# Get returns data for our assets
asset1 = get_pricing('HSC', fields='price', start_date=offset_start_date, end_date=offset_end_date).pct_change()[1:]
# Get returns for the market
bench = get_pricing('SPY', fields='price', start_date=start_date, end_date=end_date).pct_change()[1:]
# Use an ETF that tracks 3-month T-bills as our risk-free rate of return
treasury_ret = get_pricing('BIL', fields='price', start_date=start_date, end_date=end_date).pct_change()[1:]

# Define a constant to compute intercept
constant = pd.TimeSeries(np.ones(len(asset1.index)), index=asset1.index)

df = pd.DataFrame({'R1': asset1,
              'SPY': bench,
              'RF': treasury_ret,
              'Constant': constant})
df = df.dropna()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Arbitrage_Pricing_Theory/notebook.ipynb".
The first task is:
Get the factor data for the last month so we can predict the next month.
Can you write Python code for it?

start_date = '2015-07-25'
end_date = '2015-08-25'

# Get returns for the market
last_month_bench = get_pricing('SPY', fields='price', start_date=start_date, end_date=end_date).pct_change()[1:]
# Use an ETF that tracks 3-month T-bills as our risk-free rate of return
last_month_treasury_ret = get_pricing('BIL', fields='price', start_date=start_date, end_date=end_date).pct_change()[1:]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Arbitrage_Pricing_Theory/notebook.ipynb".
The first task is:
Let's get some data.
Can you write Python code for it?

start_date = '2014-06-30'
end_date = '2015-06-30'

# We will look at the returns of an asset one-month into the future to model future returns.
offset_start_date = '2014-07-31'
offset_end_date = '2015-07-31'

# Get returns data for our assets
asset1 = get_pricing('HSC', fields='price', start_date=offset_start_date, end_date=offset_end_date).pct_change()[1:]
asset2 = get_pricing('MSFT', fields='price', start_date=offset_start_date, end_date=offset_end_date).pct_change()[1:]
# Get returns for the market
bench = get_pricing('SPY', fields='price', start_date=start_date, end_date=end_date).pct_change()[1:]
# Use an ETF that tracks 3-month T-bills as our risk-free rate of return
treasury_ret = get_pricing('BIL', fields='price', start_date=start_date, end_date=end_date).pct_change()[1:]

# Define a constant to compute intercept
constant = pd.TimeSeries(np.ones(len(asset1.index)), index=asset1.index)

df = pd.DataFrame({'R1': asset1,
              'R2': asset2,
              'SPY': bench,
              'RF': treasury_ret,
              'Constant': constant})
df = df.dropna()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Arbitrage_Pricing_Theory/notebook.ipynb".
The first task is:
We'll start by computing static regressions over the whole time period.
Can you write Python code for it?

OLS_model = regression.linear_model.OLS(df['R1'], df[['SPY', 'RF', 'Constant']])
fitted_model = OLS_model.fit()
print 'p-value', fitted_model.f_pvalue
print fitted_model.params
R1_params = fitted_model.params

OLS_model = regression.linear_model.OLS(df['R2'], df[['SPY', 'RF', 'Constant']])
fitted_model = OLS_model.fit()
print 'p-value', fitted_model.f_pvalue
print fitted_model.params
R2_params = fitted_model.params


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Autocorrelation_and_AR_Models/notebook.ipynb".
The first task is:
After getting the confidence interval data, we'll write a function to plot it.
Can you write Python code for it?

def plot_acf(X_acf, X_acf_confs, title='ACF'):
    # The confidence intervals are returned by the functions as (lower, upper)
    # The plotting function needs them in the form (x-lower, upper-x)
    errorbars = np.ndarray((2, len(X_acf)))
    errorbars[0, :] = X_acf - X_acf_confs[:,0]
    errorbars[1, :] = X_acf_confs[:,1] - X_acf

    plt.plot(X_acf, 'ro')
    plt.errorbar(range(len(X_acf)), X_acf, yerr=errorbars, fmt='none', ecolor='gray', capthick=2)
    plt.xlabel('Lag')
    plt.ylabel('Autocorrelation')
    plt.title(title);
plot_acf(X_acf, X_acf_confs)

plot_acf(X_pacf, X_pacf_confs, title='PACF')


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Autocorrelation_and_AR_Models/notebook.ipynb".
The first task is:
Choosing the Number of Lags
Estimations Will Yield Too Many Lags
We can see our model estimated quite a few parameters. In this case we know there are too many because we simulated the data as an AR(3) process. The reason that AR models will estimate many more lags than is actually the case is due to indirect dependency. If $X_t$ depends on $X_{t-1}$, then indirectly and to a lesser extent it will depend on $X_{t-2}$. In the presence of more than one lag in the data generating process, we will get potentially complex harmonic structures in the lags. These indirect dependencies will be picked up by a simple estimation.
You Want the Fewest Parameters That Yield a Decent Model
In general it's rarely the case that you can get anything useful out of a model with many parameters, see the Overfitting lecture for why in the Quantopian Lecture Series. In this case we want to select a number of lags that we believe explains what is happening, but without overfitting and choosing a model with way too many lags.
Observing the ACF and PACF indicates that only the first 3 lags may be useful. However, we will expand the number of lags to 10 to double-check our initial data. We will use information criterion, specifically Akaike Information Criterion (AIC) and Bayes Information Criterion (BIC) to decide the correct number of parameters. For more information on choosing models using information criterion, please see the corresponding lecture in the Quantopian Lecture Series.
Interpreting the AIC and BIC is done as follows. Compute the AIC and BIC for all models we wish to consider, and note the smallest AIC and BIC recorded $AIC_{min}$ and $BIC_{min}$. These are the models which minimize information loss under each metric. For each type of IC We then can compute the relative likelihood of each model $i$ by taking
$$l = e^{(IC_{min} - IC_{i})/2}$$
We can interpret $l$ as model $i$ is $l$ times as likely to minimize information loss, compared to the minimum AIC model. It might take a few reads to understand this, so let's...
Can you write Python code for it?

N = 10
AIC = np.zeros((N, 1))

for i in range(N):
    model = tsa.api.AR(X)
    model = model.fit(maxlag=(i+1))
    AIC[i] = model.aic

AIC_min = np.min(AIC)
model_min = np.argmin(AIC)

print 'Relative Likelihoods'
print np.exp((AIC_min-AIC) / 2)
print 'Number of parameters in minimum AIC model %s' % (model_min+1)

N = 10
BIC = np.zeros((N, 1))

for i in range(N):
    model = tsa.api.AR(X)
    model = model.fit(maxlag=(i+1))
    BIC[i] = model.bic

BIC_min = np.min(BIC)
model_min = np.argmin(BIC)

print 'Relative Likelihoods'
print np.exp((BIC_min-BIC) / 2)
print 'Number of parameters in minimum BIC model %s' % (model_min+1)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Autocorrelation_and_AR_Models/notebook.ipynb".
The first task is:
Finally let's check, if our test is calibrated correctly, then we should have the confidence interval contain 0, 95% of the time.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Autocorrelation_and_AR_Models/notebook.ipynb".
The first task is:
Looks like something is severly wrong. What's going on here is that the AR series moves around a lot more, but the estimate of variance assumes stationarity and doesn't take into account all of that motion. As a result the confidence intervals are way smaller than they should be and don't contain the correct value nearly enough.
This gives us a false sense of security. Stationarity tests should usually catch AR behavior and let us know that estimates of variance will be wrong. For more information please see the Integration, Cointegration, and Stationarity lecture of the Quantopian Lecture Series.
Correcting for Variance
In practice it can be very difficult to accurately estimate variance on an AR series, but one attempt to do this is the Newey-West estimation. You can find information on it here.
Testing for AR Behavior
In order to determine the order, $p$, of an AR$(p)$ model, we look at the autocorrelations of the time series. These are the correlations of the series with its past values. The $k$-th order autocorrelation is
$$
\rho_k = \frac{COV(x_t, x_{t - k})}{\sigma_x^2} = \frac{E[(x_t - \mu)(x_{t - k} - \mu)}{\sigma_x^2}
$$
Where $k$ represents the number of periods lagged. We cannot directly observe the autocorrelations so we estimate them as
$$
\hat{\rho}k = \frac{\sum{t = k + 1}^T[(x_t - \bar{x})(x_{t - k} - \bar{x})]}{\sum_{t = 1}^T (x_t - \bar{x})^2}
$$
For our purposes, we can use a pair of tools called the autocorrelation function (ACF) and the partial autocorrelation function (PACF) in order to determine the order of our model. The PACF controls for shorter lags, unlike the ACF. These functions are included with many statistical packages and compute the sample autocorrelations for us, allowing us to determine the appropriate value of $p$.
We will demonstrate these functions on our above example of a stationary series:
Can you write Python code for it?

from statsmodels.tsa.stattools import acf, pacf

X = simululate_AR_process(np.array([0, 0.8, 0.1, 0.05]), 1000)

# We'll choose 40 lags. This is a bit arbitrary, but you want to include all the lags you think might
# feasibly impact the current value.
nlags = 40
# Note, this will produce nlags + 1 values, as we include the autocorrelation of
# X[-1] with X[-1], which is trivially 1.
# The reason this is done is because that is the 0th spot in the array and corresponds
# to the 0th lag of X[(-1)-0].
X_acf = acf(X, nlags=nlags)
print 'Autocorrelations:\n' + str(X_acf) + '\n'
X_pacf = pacf(X, nlags=nlags)
print 'Partial Autocorrelations:\n' + str(X_pacf)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Autocorrelation_and_AR_Models/notebook.ipynb".
The first task is:
Now we'll run an experiment 1000 times in which we compute an AR series, then estimate the mean and take a naive 95% confidence interval around it. Then we'll check if the confidence interval contains 0, the true long-term mean of our series, and record that in our outcomes array.
Can you write Python code for it?

trials = 1000
outcomes = np.zeros((trials, 1))

for i in range(trials):
    #note these are the same values we used to generate the initial AR array
    Z = simululate_AR_process(np.array([0, 0.8, 0.1, 0.05]), 100)
    if check_unadjusted_coverage(Z):
        # The internal contains 0, the true value
        outcomes[i] = 1
    else:
        outcomes[i] = 0


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Autocorrelation_and_AR_Models/notebook.ipynb".
The first task is:
Our conclusion is that the AIC estimates the 4 parameter model as most likely, whereas the BIC estimates 3. Because we are always looking for reasons to knock off a parameter, we choose the 3. In this case it happened to be the exact right answer, but this will not always be the case, especially in noisy real data. Don't assume that using this method will always get you the right answer.
Evaluating Residuals
One final step we might do before performing an out of sample test for this model would be to evaluate its residual behavior. The AIC and BIC already do this to an extent, effectively measuring how much information is left on the table (in the residuals) after the model has made its predictions. For more information on residuals analysis see the Violations of Regression Models lecture. Here we'll just check for normality of the residuals.
Can you write Python code for it?

model = tsa.api.AR(X)
model = model.fit(maxlag=3)

from statsmodels.stats.stattools import jarque_bera

score, pvalue, _, _ = jarque_bera(model.resid)

if pvalue < 0.10:
    print 'We have reason to suspect the residuals are not normally distributed.'
else:
    print 'The residuals seem normally distributed.'


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Beta_Hedging/notebook.ipynb".
The first task is:
If we plot the line $\alpha + \beta X$, we can see that it does indeed look like the line of best fit:
Can you write Python code for it?

X2 = np.linspace(X.min(), X.max(), 100)
Y_hat = X2 * beta + alpha

plt.scatter(X, Y, alpha=0.3) # Plot the raw data
plt.xlabel("SPY Daily Return")
plt.ylabel("TSLA Daily Return")

 # Add the regression line, colored in red
plt.plot(X2, Y_hat, 'r', alpha=0.9);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Beta_Hedging/notebook.ipynb".
The first task is:
It looks like the portfolio return follows the asset alone fairly closely. We can quantify the difference in their performances by computing the mean returns and the volatilities (standard deviations of returns) for both:
Can you write Python code for it?

print "means: ", portfolio.mean(), r_a.mean()
print "volatilities: ", portfolio.std(), r_a.std()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Beta_Hedging/notebook.ipynb".
The first task is:
Note that we developed our hedging strategy using historical data. We can check that it is still valid out of sample by checking the alpha and beta values of the asset and the hedged portfolio in a different time frame:
Can you write Python code for it?

# Get the alpha and beta estimates over the last year
start = '2014-01-01'
end = '2015-01-01'
asset = get_pricing('TSLA', fields='price', start_date=start, end_date=end)
benchmark = get_pricing('SPY', fields='price', start_date=start, end_date=end)
r_a = asset.pct_change()[1:]
r_b = benchmark.pct_change()[1:]
X = r_b.values
Y = r_a.values
historical_alpha, historical_beta = linreg(X,Y)
print 'Asset Historical Estimate:'
print 'alpha: ' + str(historical_alpha)
print 'beta: ' + str(historical_beta)

# Get data for a different time frame:
start = '2015-01-01'
end = '2015-06-01'
asset = get_pricing('TSLA', fields='price', start_date=start, end_date=end)
benchmark = get_pricing('SPY', fields='price', start_date=start, end_date=end)

# Repeat the process from before to compute alpha and beta for the asset
r_a = asset.pct_change()[1:]
r_b = benchmark.pct_change()[1:]
X = r_b.values
Y = r_a.values
alpha, beta = linreg(X,Y)
print 'Asset Out of Sample Estimate:'
print 'alpha: ' + str(alpha)
print 'beta: ' + str(beta)

# Create hedged portfolio and compute alpha and beta
portfolio = -1*historical_beta*r_b + r_a
P = portfolio.values
alpha, beta = linreg(X,P)
print 'Portfolio Out of Sample:'
print 'alpha: ' + str(alpha)
print 'beta: ' + str(beta)

# Plot the returns of the portfolio as well as the asset by itself
portfolio.name = "TSLA + Hedge"
portfolio.plot(alpha=0.9)
r_a.plot(alpha=0.5);
r_b.plot(alpha=0.5)
plt.ylabel("Daily Return")
plt.legend();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Beta_Hedging/notebook.ipynb".
The first task is:
Now we can perform the regression to find $\alpha$ and $\beta$:
Can you write Python code for it?

# Let's define everything in familiar regression terms
X = r_b.values # Get just the values, ignore the timestamps
Y = r_a.values

def linreg(x,y):
    # We add a constant so that we can also fit an intercept (alpha) to the model
    # This just adds a column of 1s to our data
    x = sm.add_constant(x)
    model = regression.linear_model.OLS(y,x).fit()
    # Remove the constant now that we're done
    x = x[:, 1]
    return model.params[0], model.params[1]

alpha, beta = linreg(X,Y)
print 'alpha: ' + str(alpha)
print 'beta: ' + str(beta)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Case_Study_Traditional_Value_Factor/notebook.ipynb".
The first task is:
Price to Book Value
Price to Book Value (a.k.a Price to Book Ratio) is calculated as:
$$P/B\;Ratio = \frac{Price\;per\;share}{Net\;Asset\;Value\;per\;share}$$
Net Asset Value per share can be thought of (very roughly) as a company's total assets less its total liabilities, all divided by the number of shares outstanding.
The P/B Ratio gives a sense of a stock being either over- or undervalued. A high P/B ratio suggests that a stock's price is overvalued, and should therefore be shorted, whereas a low P/B ratio is attractive as the stock gained by purchasing the equity is hypothetically "worth more" than the price paid for it.
We will now create a P/B Ratio custom factor and look at some of the results.
Can you write Python code for it?

# Custom Factor 2 : P/B Ratio
class Price_to_Book(CustomFactor):

    inputs = [morningstar.valuation_ratios.pb_ratio]
    window_length = 1

    def compute(self, today, assets, out, pbr):
        out[:] = pbr[-1]

# create the Pipeline
temp_pipe_2 = Pipeline()

# add the factor to the Pipeline
temp_pipe_2.add(Price_to_Book(), 'P/B Ratio')

# run the Pipeline and get data for first 5 equities
run_pipeline(temp_pipe_2, start_date='2015-11-11', end_date='2015-11-11').head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Case_Study_Traditional_Value_Factor/notebook.ipynb".
The first task is:
Stock Choice
Now that we have ranked our securities, we need to choose a long basket and a short basket. Since we need to keep the ratio 130/30 between longs and shorts, why not have 26 longs and 6 shorts (in the algorithm we will weigh each of these equally, giving us our desired leverage and exposure).
On the graph below, we plot a histogram of the securities to get a sense of the distribution of scores. The red lines represent the cutoff points for the long and short buckets. One thing to notice is that the vast majority of equities are ranked near the middle of the histogram, whereas the tails are quite thin. This would suggest that there is something special about the securities chosen to be in these baskets, and -hopefully- these special qualities will yield positive alpha for the strategy.
Can you write Python code for it?

# create histogram of scores
ranked_scores.hist()

# make scores into list for ease of manipulation
ranked_scores_list = ranked_scores.tolist()

# add labels to axes
plt.xlabel('Standardized Scores')
plt.ylabel('Quantity in Basket')

# show long bucket
plt.axvline(x=ranked_scores_list[25], linewidth=1, color='r')

# show short bucket
plt.axvline(x=ranked_scores_list[-6], linewidth=1, color='r');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Case_Study_Traditional_Value_Factor/notebook.ipynb".
The first task is:
There are two points to make about this data series.
Firstly, AA_PR's P/B Ratio is given as NaN by Pipeline. NaN stands for "not a number" and occurs when a value can not be fetched by Pipeline. Eventually, we will remove these NaN values from the dataset as they often lead to confusing errors when manipulating the data.
Secondly, a low P/B Ratio and a high Dividend Yield are attractive for investors, whereas a a high P/B Ratio and a low Dividend Yield are unattractive. Therefore, we will "invert" the P/B ratio by making each value negative in the factor output so that, when the data is aggregated later in the algorithm, the maxima and minima have the same underlying "meaning".
Price to Trailing 12-Month Sales
This is calculated as a simple ratio between price per share and trailing 12-month (TTM) sales.
TTM is a transformation rather than a metric and effectively calculates improvement or deterioration of a fundamental value from a particular quarter one year previously. For example, if one wanted to calculate today's TTM Sales for company XYZ, one would take the most recent quarter's revenue and divide it by the difference between this quarter's revenue and this quarter's revenue last year added to the revenue as given by the company's most recent fiscal year-end filing.
To calculate the exact TTM of a security is indeed possible using Pipeline; however, the code required is slow. Luckily, this value can be well approximated by the built-in Morningstar valuation ratios, which use annual sales to calculate the Price to Sales fundamental value. This slight change boosts the code's speed enormously yet has very little impact on the results of the strategy itself.
Price to TTM Sales is similar to the P/B Ratio in terms of function. The major difference in these two ratios is the fact that inclusion of TTM means that seasonal fluctuations are minimized, as previous data is used to smooth the value. In our case, annualized values accomplish this same smoothing.
Also, note that the values produced are negative;...
Can you write Python code for it?

# Custom Factor 3 : Price to Trailing 12 Month Sales
class Price_to_TTM_Sales(CustomFactor):
    inputs = [morningstar.valuation_ratios.ps_ratio]
    window_length = 1

    def compute(self, today, assets, out, ps):
        out[:] = -ps[-1]

# create the pipeline
temp_pipe_3 = Pipeline()

# add the factor to the pipeline
temp_pipe_3.add(Price_to_TTM_Sales(), 'Price / TTM Sales')

# run the pipeline and get data for first 5 equities
run_pipeline(temp_pipe_3, start_date='2015-11-11', end_date='2015-11-11').head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Case_Study_Traditional_Value_Factor/notebook.ipynb".
The first task is:
Traditional Value
In this notebook, we will develop a strategy based on the "traditional value" metrics described in the Lo/Patel whitepaper. The factors employed in this strategy designate stocks as either cheap or expensive using classic fundamental analysis. The factors that Lo/Patel used are:

Dividend Yield
Price to Book Value
Price to Trailing 12-Month Sales
Price to Trainling 12-Month Cash Flows

Dividend Yield
Dividend yield is calculated as:
$$Dividend\;Yield = \frac{Annual\;Dividends\;per\;share}{Price\;per\;share}$$
When a company makes profit, it faces a choice. It could either reinvest those profits in the company with an eye to increase efficiency, purchase new technology, etc. or it could pay dividends to its equity holders. While reinvestment may increase a company's future share price and thereby reward investors, the most concrete way equity holders are rewarded is through dividends. An equity with a high dividend yield is particularly attractive as the quantity of dividends paid to investors represent a larger proportion of the share price itself. Now we shall create a Dividend Yield factor using the Pipeline API framework and Morningstar's list of fundamental values.
Can you write Python code for it?

# Custom Factor 1 : Dividend Yield
class Div_Yield(CustomFactor):

    inputs = [morningstar.valuation_ratios.dividend_yield]
    window_length = 1

    def compute(self, today, assets, out, d_y):
        out[:] = d_y[-1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
Definitely looks like there's some convergence, we can also check what the mean of the sample means is.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
Further Reading
This is only a brief introduction, Wikipedia has excellent articles detailing these subjects in greater depth. Let's go back to our heights example. Since the sample size is small, we'll use a $t$-test.
Can you write Python code for it?

# standard error SE was already calculated
t_val = stats.t.ppf((1+0.95)/2, 9)  # d.o.f. = 10 - 1
print 'sample mean height:', mean_height
print 't-value:', t_val
print 'standard error:', SE
print 'confidence interval:', (mean_height - t_val * SE, mean_height + t_val * SE)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
It turns out that for larger sample sizes, you should see the sample mean asymptotically converge to zero. This is because the process is still centered around zero, but let's check if that's true. We'll vary the number of samples draw, and look for convergence as we increase sample size.
Can you write Python code for it?

sample_means = np.zeros(200-1)
for i in range(1, 200):
    X = generate_autocorrelated_data(0.5, 0, 1, i * 10)
    sample_means[i-1] = np.mean(X)

plt.bar(range(1, 200), sample_means);
plt.xlabel('Sample Size');
plt.ylabel('Sample Mean');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
Note that as your confidence increases, the interval necessarily widens.
Assuming normality, there's also a built in function that will compute our interval for us. This time you don't need to specify the degrees of freedom. Note that at a corresponding level of confidence, the interval calculated using the normal distribution is narrower than the interval calcuated using the $t$-distribution.
Can you write Python code for it?

print stats.norm.interval(0.99, loc=mean_height, scale=SE)
print stats.norm.interval(0.95, loc=mean_height, scale=SE)
print stats.norm.interval(0.80, loc=mean_height, scale=SE)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
Now we'll run many trials, in each we'll sample some data, compute a confidence interval, and then check if the confidence interval contains the population mean. We'll keep a running tally, and we should expect to see 95% of the trials succeed if the intervals are calibrated correctly.
Can you write Python code for it?

T = 100
trials = 500
times_correct = 0
for i in range(trials):
    X = generate_autocorrelated_data(0.5, 0, 1, T)
    if check_unadjusted_coverage(X):
        times_correct += 1

print 'Empirical Coverage: ', times_correct/float(trials)
print 'Expected Coverage: ', 0.95


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
There is a function in scipy's stats library for calculating the standard error. Note that this function be default contains a degrees-of-freedom correction that is often not necessary (for large enough samples, it's effectively irrelevant). You can omit the correction by setting the parameter ddof to 0.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
Unfortunately simply reporting the sample mean doesn't do much for us, as we don't know how it relates to the population mean. To get a sense for how it might relate, we can look for how much variance there is in our sample. Higher variance indicates instability and uncertainty.
Can you write Python code for it?

print 'sample standard deviation: ', np.std(heights)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
Visualizing Confidence Intervals
Here is some code to visualize a confidence interval on a graph. Feel free to play around with it.
Can you write Python code for it?

sample_size = 100
heights = np.random.normal(POPULATION_MU, POPULATION_SIGMA, sample_size)
SE = np.std(heights) / np.sqrt(sample_size)
(l, u) = stats.norm.interval(0.95, loc=np.mean(heights), scale=SE)

print (l, u)

plt.hist(heights, bins=20)
plt.xlabel('Height')
plt.ylabel('Frequency')

# Just for plotting
y_height = 5
plt.plot([l, u], [y_height, y_height], '-', color='r', linewidth=4, label='Confidence Interval')
plt.plot(np.mean(heights), y_height, 'o', color='r', markersize=10);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Confidence_Intervals/notebook.ipynb".
The first task is:
What does this mean?
Confidence intervals allow us to set our desired confidence, and then report a range that will likely contain the population mean. The higher our desired confidence, the larger range we report. In general once can never report a single point value, because the probability that any given point is the true population mean is incredibly small. Let's see how our intervals tighten as we change sample size.
Can you write Python code for it?

np.random.seed(10)

sample_sizes = [10, 100, 1000]
for s in sample_sizes:
    heights = np.random.normal(POPULATION_MU, POPULATION_SIGMA, s)
    SE = np.std(heights) / np.sqrt(s)
    print stats.norm.interval(0.95, loc=mean_height, scale=SE)