I am working on the file "Glassdoor/company-reviews-master/src/Ratings.ipynb".
The first task is:
<a id='overall' />
Overall Ratings
Can you write Python code for it?

import numpy as np

# average will divide by the number of ratings
def get_all_ratings(rating_key, average = False):
    total_ratings = np.zeros(5)
    for company in companies:
        company_ratings = get_company_ratings(company, ratings, rating_key)
        hist = np.histogram(company_ratings,bins=5,range=[0.01,5.01])[0]
        if average:
            hist = hist.astype(float) / sum(hist)
        total_ratings += hist
    return total_ratings

ratings_key = 'rating'
total_ratings = get_all_ratings(rating_key)
draw_hist(total_ratings, "All Overall Ratings", True, True)

titles = ["All " + r + " Ratings" for r in rating_keys]

for key,title in zip(rating_keys, titles):
    total_ratings = get_all_ratings(key)
    draw_hist(total_ratings, title, True, True)

# Now without weighting by number of employees
ratings_key = 'rating'
total_ratings = get_all_ratings(rating_key, True)
draw_hist(total_ratings, "All Overall Ratings Average", True, True)

titles = ["All " + r + " Ratings Average" for r in rating_keys]

for key,title in zip(rating_keys, titles):
    total_ratings = get_all_ratings(key, True)
    draw_hist(total_ratings, title, True, True)


I am working on the file "Glassdoor/company-reviews-master/src/Ratings.ipynb".
The first task is:
Plot all companies' ratings
Plot all of the companies' ratings and save them as images.
Can you write Python code for it?

rating_key = 'rating'
title_end = ' overall ratings'
draw_all_company_rating(rating_key, companies, title_end)


I am working on the file "Glassdoor/company-reviews-master/src/Scraping.ipynb".
The first task is:
Final Fixes
Look at the failed companies. Often they couldn't be found on glassdoor because of an issue with their name. You might need to fix the names again (and search on glassdoor for the name some companies are listed under). Beware of encoding issues: if you pass an optional flag to utils.fix_company_name, you can encode the company names to ascii.
Note: this is usually quite a bit slower than Indeed because there are many more reviews (e.g. Goldman Sachs has 198 pages!).
Can you write Python code for it?

# fix_companies = {u'SigmaTek':u'SigmaTek Consulting LLC',
#                 }
# utils.fix_all_company_names(indeed_reviews, fix_companies)
# fixed_failed_companies = fixed_failed_companies = [utils.fix_company_name(company,
# fix_companies, True) for company in failed_companies]
# visited_companies2, failed_companies = glassdoor.get_all_company_reviews(fixed_failed_companies,
#                                               glassdoor_reviews, MAX_PAGES_REVIEWS)


I am working on the file "Glassdoor/company-reviews-master/src/Scraping.ipynb".
The first task is:
Fix Company Names
Indeed's company names are inconsistent. The same company can be listed several times with various spellings/typos/words. It's necessary to look at the companies and fix the names. The utils module has a function which takes a dictionary that takes the old name and returns the new one (names not in the dictionary are left as is). See below for an example (the one I used had over 30 name fixes.
Can you write Python code for it?

companies = list(set(utils.get_company_names(indeed_reviews)))
companies[:5]

fix_companies = {'Argus, ISO, Verisk Analytics, Verisk Climate, Veri...': 'Verisk Analytics',
                 'Barclays Investment Bank': 'Barclays', 'Dun & Brandstreet': u'Dun & Bradstreet',
                 'Dun & Broadstreet':u'Dun & Bradstreet', 'World Business Lenders - New York, NY':'World Business Lenders'
                }
utils.fix_all_company_names(indeed_reviews, fix_companies)

companies = list(set(utils.get_company_names(indeed_reviews)))


I am working on the file "Glassdoor/company-reviews-master/src/Scraping.ipynb".
The first task is:
Here I would do one last check too see which companies were scraped in glassdoor and indeed. Occasionally the wrong company might have been scraped on glassdoor.
Can you write Python code for it?

glassdoor_companies = set(utils.get_company_names(glassdoor_reviews))
indeed_companies = set(utils.get_company_names(indeed_reviews))

# Remove the extra companies:
extra_companies = glassdoor_companies - indeed_companies
for company in extra_companies:
    glassdoor_reviews.remove({'company' : company})

print "Missing companies", indeed_companies - glassdoor_companies


I am working on the file "Glassdoor/company-reviews-master/src/Scraping.ipynb".
The first task is:
Scrape Glassdoor
Can you write Python code for it?

visited_companies, failed_companies = glassdoor.get_all_company_reviews(companies,
                                              glassdoor_reviews, MAX_PAGES_REVIEWS)


I am working on the file "Glassdoor/company-reviews-master/src/Scraping.ipynb".
The first task is:
Scrape company reviews from Indeed
This takes all the companies that appear in the jobs scraped.
Can you write Python code for it?

indeed.get_all_company_reviews(jobs, indeed_reviews, MAX_PAGES_REVIEWS)

indeed_reviews.find_one()


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Attacks in Afghanistan by Year
Can you write Python code for it?

plt.hist(afgtd['iyear'])

pkgtd = dfwhole[dfwhole.country_txt=='Pakistan']

pkgtd.tail(2)


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Attacks in Pakistan by Year
Can you write Python code for it?


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Getting mean for Afghanistan.
Can you write Python code for it?


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Getting mean for Pakistan.
Can you write Python code for it?


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Looking at attack types in Pakistan
Can you write Python code for it?


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Setting up Afghanistan to be one of my populations from 2001 through 2015.
Can you write Python code for it?

afp = dfwhole[(dfwhole.attacktype1 == 3) & (dfwhole.country == 4) & (dfwhole.iyear > 2000)].
groupby(['iyear']).attacktype1.count().values

afp


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Setting up Pakistan to be my second population from 2001 through 2015.
Can you write Python code for it?

pakp = dfwhole[(dfwhole.attacktype1 == 3) & (dfwhole.country == 153) & (dfwhole.iyear > 2000)].
groupby(['iyear']).attacktype1.count().values


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Setting up my prior which are bombings/explosions in the South Asia region from 1970 up to and including 2000. I will use Pakistan and Afghanistan and bombings/explosions from 2001 till 2015 to measure the amount of difference of these types of attacks.
Can you write Python code for it?

df_prior = dfwhole[(dfwhole.attacktype1 == 3) & (dfwhole.region == 6) & (dfwhole.iyear <= 2000)].
groupby(['iyear', 'country_txt']).attacktype1.count().values

## Confirming it worked.
df_prior

df_prior_mean = df_prior.mean()
print df_prior_mean

df_prior.groupby('country_txt').attacktype1.count()

dfwhole.groupby(['iyear', 'country_txt']).attacktype1.count()


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Setting up the standard deviation for Pakistan.
Can you write Python code for it?

## setting up std for Pakistan
pakp_std = pakp.std()
print pakp_std

with pm.Model() as model:

    groupPk_mean = pm.Normal('Bombings_Pak_mean', mean_prior_mean, sd=mean_prior_std)
    groupAfg_mean = pm.Normal('Bombings_Afg_mean', mean_prior_mean, sd=mean_prior_std)

std_prior_lower = 0.01
std_prior_upper = 100.0

with model:

    groupPak_std = pm.Uniform('Bombings_Pak_std', lower=std_prior_lower, upper=std_prior_upper)
    groupAfg_std = pm.Uniform('Bombings_Afg_std', lower=std_prior_lower, upper=std_prior_upper)

with model:

    groupPak = pm.Normal('Bombings_Pak', mu=groupPk_mean, sd=groupPak_std, observed=pakp)
    groupAfg = pm.Normal('Bombings_Afg', mu=groupAfg_mean, sd=groupAfg_std, observed=afp)

with model:

    diff_of_means = pm.Deterministic('difference of means',groupPk_mean - groupAfg_mean)
    diff_of_stds = pm.Deterministic('difference of stds',groupPak_std - groupAfg_std)
    effect_size = pm.Deterministic('effect size',
                                   diff_of_means / np.sqrt((groupPak_std**2 + groupAfg_std**2) / 2))

with model:
    trace = pm.sample(25000, njobs=4)

pm.plot_posterior(trace[3000:],
                  varnames=['Bombings_Pak_mean', 'Bombings_Afg_mean', 'Bombings_Pak_std', 'Bombings_Afg_std'],
                  color='#87ceeb')

pm.plot_posterior(trace[3000:],
                  varnames=['difference of means', 'difference of stds', 'effect size'],
                  ref_val=0,
                  color='#87ceeb')

pm.summary(trace[3000:],
           varnames=['difference of means', 'difference of stds', 'effect size'])

afgtd = dfwhole[dfwhole.country_txt=='Afghanistan']


I am working on the file "Global Terrorism Bayesian Analysis/Global Terrorism Bayesian Analysis.ipynb".
The first task is:
Settinp up the standard deviation for Afghanistan.
Can you write Python code for it?


I am working on the file "GlobalTemp.ipynb".
The first task is:
Plotting the data
Can you write Python code for it?

x1 = data_LOTI.Year
y1 = data_LOTI.JanDec

# plt.plot() makes a line graph, by default
fig = plt.figure(figsize=(10, 5))
plt.plot(x1, y1)
plt.title('Average land an docean temperature readings')
plt.xlabel('Year')
plt.ylabel('Percent temp change')

x2 = data_LAND.Year
y2 = data_LAND.JanDec

# plt.plot() makes a line graph, by default
fig = plt.figure(figsize=(10, 5))
plt.plot(x2, y2)
plt.title('Land temperature readings')
plt.xlabel('Year')
plt.ylabel('Percent temp change')

# Wow, this needs a title and axis labels!
fig = plt.figure(figsize=(10, 5))
plt.plot(x1, y1, label="Land and Ocean")
plt.plot(x2, y2, label="Land only")
plt.legend()
plt.show()


I am working on the file "Global_Local_Temp.ipynb".
The first task is:
PART 2: Global Temperature Changes & Climate
Can you write Python code for it?

# The .head(n) command displays the first n rows of the file.
data_LAND.head(5)

x1 = data_LOTI.Year
y1 = data_LOTI.JanDec

# plt.plot() makes a line graph, by default
fig = plt.figure(figsize=(10, 5))
plt.plot(x1, y1)
plt.title('Average land an docean temperature readings')
plt.xlabel('Year')
plt.ylabel('Percent temp change')

x2 = data_LAND.Year
y2 = data_LAND.JanDec

# plt.plot() makes a line graph, by default
fig = plt.figure(figsize=(10, 5))
plt.plot(x2, y2)
plt.title('Land temperature readings')
plt.xlabel('Year')
plt.ylabel('Percent temp change')

# Wow, this needs a title and axis labels!
fig = plt.figure(figsize=(10, 5))
plt.plot(x1, y1, label="Land and Ocean")
plt.plot(x2, y2, label="Land only")

plt.legend()
plt.show()


I am working on the file "Global_Local_Temp_Turner.ipynb".
The first task is:
PART 1: Local Temperature Changes & Climate
This section will be used for questions 3-5.
We can view the first few rows of data from the file we just imported.
Can you write Python code for it?

# The .head(n) command displays the first n rows of the local temperature data file.
data3.head(4)

data3.columns


I am working on the file "Global_Local_Temp_Turner.ipynb".
The first task is:
PART 2: Global Temperature Changes & Climate
This section will be used for questions 6-8.
Can you write Python code for it?

# The .head(n) command displays the first n rows of the file.
data_LAND.head(5)

x1 = data_LOTI.Year
y1 = data_LOTI.JanDec

# plt.plot() makes a line graph, by default
fig = plt.figure(figsize=(10, 5))
plt.plot(x1, y1)
plt.title('Average land and ocean temperature readings')
plt.xlabel('Year')
plt.ylabel('Percent temp change')

x2 = data_LAND.Year
y2 = data_LAND.JanDec

# plt.plot() makes a line graph, by default
fig = plt.figure(figsize=(10, 5))
plt.plot(x2, y2)
plt.title('Land temperature readings')
plt.xlabel('Year')
plt.ylabel('Percent temp change')

# Wow, this needs a title and axis labels!
fig = plt.figure(figsize=(10, 5))
plt.plot(x1, y1, label="Land and Ocean")
plt.plot(x2, y2, label="Land only")

plt.legend()
plt.show()


I am working on the file "Global_Local_Temp_Turner.ipynb".
The first task is:
Importing a data set
This next cell may take a little while to run if it's grabbing a pretty big data set. The cell label to the left will look like "In [*]" while it's still thinking and "In [2]" when it's finished.
Can you write Python code for it?

# Combined land and ocean temperature averages (LOTI: Land Ocean Temperature Index)
data1 = pd.read_csv('http://github.com/scpsscience/data/raw/master/LOTI.csv', header=1).replace(to_replace="***", value=np.NaN)
data_LOTI = data1.apply(lambda x: pd.to_numeric(x, errors='ignore'))

# Only land temperature averages
data2 = pd.read_csv('http://github.com/scpsscience/data/raw/master/LAND.csv', header=1).replace(to_replace="***", value=np.NaN)
data_LAND = data2.apply(lambda x: pd.to_numeric(x, errors='ignore'))

# Temperature Data for Orlando, Florida
data3 = pd.read_csv('http://github.com/scpsscience/data/raw/master/Orlando_Temps.csv')

dates = ['1/1/1975','1/2/1975','01/3/1975']
x = [dt.datetime.strptime(d,'%m/%d/%Y').date() for d in dates]
y = "range(len(x))"


I am working on the file "Global_Local_Temp_Turner.ipynb".
The first task is:
The code below will display a graph displaying the average temperatures at Orlando International Airport. You can double click on the graph to make it bigger.
Can you write Python code for it?

ax = data3.plot('date', 'avg', title="Average Temperature at Orlando International Airport (1975 - Present)", legend=False, color='b')
ax.set(xlabel="Date", ylabel="Average Temperature (Fahrenheit)")

# Get current size
fig_size = plt.rcParams["figure.figsize"]

# Set figure width to 20 and height to 6.5
fig_size[0] = 100
fig_size[1] = 6.5
plt.rcParams["figure.figsize"] = fig_size

plt.show()


I am working on the file "Going to Mars with Python in 5 minutes.ipynb".
The first task is:
Primero: definir la órbita
Can you write Python code for it?

r = [-6045, -3490, 2500] * u.km
v = [-3.457, 6.618, 2.533] * u.km / u.s

ss = State.from_vectors(Earth, r, v)

with plt.style.context('pybonacci'):
    plot(ss)


I am working on the file "Going to Mars with Python in 5 minutes.ipynb".
The first task is:
Quinto: ¡¡Hagámoslo interactivo!!!1!
Can you write Python code for it?

%matplotlib inline
from ipywidgets import interactive
from IPython.display import display

w = interactive(go_to_mars, offset=(0., 1000.), tof_=(100., 12000.))
display(w)


I am working on the file "Going to Mars with Python in 5 minutes.ipynb".
The first task is:
Segundo: localiza los planetas
Can you write Python code for it?

epoch = time.Time("2015-06-21 16:35")

r_, v_ = ephem.planet_ephem(ephem.EARTH, epoch)

r_

v_.to(u.km / u.s)


I am working on the file "Going to Mars with Python in 5 minutes.ipynb".
The first task is:
Tercero: Calcula la trayectoria
Can you write Python code for it?

date_launch = time.Time('2011-11-26 15:02', scale='utc')
date_arrival = time.Time('2012-08-06 05:17', scale='utc')
tof = date_arrival - date_launch
r0, _ = ephem.planet_ephem(ephem.EARTH, date_launch)
r, _ = ephem.planet_ephem(ephem.MARS, date_arrival)

(v0, v), = iod.lambert(Sun.k, r0, r, tof)

v0

v


I am working on the file "Golub_dataset_Class_Prediction.ipynb".
The first task is:
Let's score on the test set:
Can you write Python code for it?

neigh.score(norm_test.T, sample_table.iloc[38:, 1:2].values.reshape(34))

results = results.append(pd.Series({'Classifier':'K-nearest Neighbours', 'Dataset':'Normalized',
                          'Training_Error':neigh.score(norm_train.T, sample_table.iloc[:38,1:2].values.reshape(38)),
                          'Testing_Error':neigh.score(norm_test.T, sample_table.iloc[38:, 1:2].values.reshape(34))}),
              ignore_index=True)


I am working on the file "Golub_dataset_Class_Prediction.ipynb".
The first task is:
Let's score on the training data set to see whether our model is reasonably trained.
Can you write Python code for it?


I am working on the file "Golub_dataset_Class_Prediction.ipynb".
The first task is:
Non-normalized Data
Let's try the same things with only a subset of genes and no normalization besides setting all values below 0 to 1.
Can you write Python code for it?

non_negative_train = train.iloc[:,1::2].clip_lower(1)
non_negative_train
non_negative_test = test.iloc[:, 1::2].clip_lower(1)


I am working on the file "Golub_dataset_Class_Prediction.ipynb".
The first task is:
Normalizing the Data
We normalize each element x by subtracting the row minimum and dividing by (row maximum minus row minimum).
x = (x - row_min) / (row_max - row_min)
Can you write Python code for it?

norm_train = train.iloc[:,1::2].apply(lambda x: (x - np.min(x))/(np.max(x)-np.min(x)), axis=1)

norm_test = test.iloc[:,1::2].apply(lambda x: (x - np.min(x))/(np.max(x)-np.min(x)), axis=1)
norm_test = norm_test.reindex_axis(sorted(norm_test.columns, key=int), axis=1)

# Sorting the columns numerically
norm_train = norm_train.reindex_axis(sorted(list(norm_train.columns.values), key=int), axis=1)


I am working on the file "Golub_dataset_Class_Prediction.ipynb".
The first task is:
Plotting the Results
Below are plots of the different combinations of data sets and classifiers.
The data sets are non-negative, i.e. negative expression values are set to 1 and only the top 3000 genes were considered, and normalized, i.e. for each expression value x:
x --> (x - row_min) / (row_max - row_min)
Can you write Python code for it?

ax = sns.factorplot(x='Dataset', y='Training_Error', kind='bar', hue='Classifier', data=results)
ax = sns.factorplot(x='Dataset', y='Testing_Error', kind='bar', hue='Classifier', data=results)


I am working on the file "Golub_dataset_Class_Prediction.ipynb".
The first task is:
Reading in the datasets
We'll read in the test into the variable test and the training dataset into the variable train.
Can you write Python code for it?

test = pd.read_csv('data_set_ALL_AML_independent.tsv', sep='\t', header=0, index_col=0)

train = pd.read_csv('data_set_ALL_AML_train.tsv', sep='\t', header=0, index_col=0)
train.drop(train.columns[len(train.columns)-1], axis=1, inplace=True)

#The sample_table contains the labels AML/ ALL
sample_table = pd.read_csv('table_ALL_AML_samples.txt', sep='\t')
sample_table['Idealized'] = 0
sample_table['Idealized'].loc[sample_table['ALL/AML']=='ALL'] = 1


I am working on the file "Golub_dataset_Class_Prediction.ipynb".
The first task is:
The violin plot below shows the distribution of the results of the ttests. The values plotted are 1-pvalue for all the genes.
One can see that most genes actually have pretty good p-values.
Can you write Python code for it?


I am working on the file "Google Searches/Mining google searches.ipynb".
The first task is:
First, let's define a function to create a pandas dataframe from a single google search history json document.
Each .json file contains all the queries for a given financial quarter. This function will return a dataframe containing each query along with the timestamp. Unfortunately, it's unclear which queries are from google maps, but direction queries contain a '->', so we'll filter those out.
Can you write Python code for it?

def readGoogleSearchHistoryJSON(fileName):
    """
    Reads in data from json files provided by google

    param fileName: string of full path to file to load

    return: pandas dataframe containing information
    """

    # open file and get raw data
    fid = open(fileName)
    raw_data = json.load(fid)
    fid.close()

    #initialize
    queryTimes = []
    queryText = []

    # loop through each query and get data
    for query in raw_data['event']:
        # get time, and convert to datetime object
        tempTime = datetime.datetime.fromtimestamp(int(query['query']['id'][0]['timestamp_usec'][:10]))

        # get query text
        tempText = query['query']['query_text']

        # remove queries with map data
        if re.search('->', tempText) is not None:
            continue

        queryTimes.append(tempTime)
        queryText.append(tempText)

    # create dataframe
    df = pd.DataFrame(data={'date': queryTimes,'query': queryText})
    return df


I am working on the file "Google Searches/Mining google searches.ipynb".
The first task is:
How about on a weekly basis? How do my searches for these terms change over the course of a week? For example, when do I search for "matlab" the most?
Can you write Python code for it?

# Define function to plot word usage by weekday and hour

def plotWordTimeHeatMap(words=None):
    # get series for hours and weekdays
    hours = df['date'].map(lambda date: date.hour)
    weekdays = df['date'].map(lambda date: date.weekday())

    # group by hours and weekdays
    g = df.groupby([hours,weekdays])

    # take tokenized words, count matches to requested words and unstack
    if words is None:
        countDF = g['tokenized'].count().unstack(0)
    else:
        countDF = g['tokenized'].apply(countTokWordQuery, words).unstack(0)

    # replace NaN with zero values and extract to numpy array
    countMat = countDF.fillna(0).get_values()

    # plot
    fig = plt.figure(figsize=(15,15))
    ax = plt.subplot()
    axMat = ax.matshow(countMat)

    # set ticks and title
    if type(words) is list:
        ax.set_title(', '.join([word.title() for word in words]))
    elif words is None:
        ax.set_title('All Searches')
    else:
        ax.set_title(words.title())
    ax.xaxis.tick_bottom()
    ax.set_xlabel('Hour')
    ax.set_yticks(range(7))
    ax.set_yticklabels(['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday']);

    # create colorbar
    cbar = fig.colorbar(axMat, shrink=0.3)
    cbar.set_label('# Matching Queries', fontsize=20, rotation=-90, verticalalignment='bottom')
    cbar.ax.tick_params(labelsize=15)

    # change tick size
    for item in ([ax.title, ax.xaxis.label, ax.yaxis.label]):
        item.set_fontsize(20)
    for item in ax.get_xticklabels() + ax.get_yticklabels():
        item.set_fontsize(15)

    # add copyright
    addCopyright(ax, yOffset=-0.2)

plotWordTimeHeatMap()
# plt.savefig('allWordsHeatmap.png', transparent=True, dpi=200, bbox_inches='tight')
plotWordTimeHeatMap('matlab')
# plt.savefig('matlabHeatmap.png', transparent=True, dpi=200, bbox_inches='tight')
plotWordTimeHeatMap('python')
# plt.savefig('pythonHeatmap.png', transparent=True, dpi=200, bbox_inches='tight')
plotWordTimeHeatMap(['ps4','xbox'])
#...


I am working on the file "Google Searches/Mining google searches.ipynb".
The first task is:
Let's tokenize each query, and add it as a new column
Can you write Python code for it?

""" Add tokenized query"""
df['tokenized'] = df['query'].map(lambda words: [word.lower() for word in nltk.word_tokenize(words) if word.isalpha()])


I am working on the file "Google Searches/Mining google searches.ipynb".
The first task is:
Now, let's define some helper functions
Can you write Python code for it?

def getGroupedDate(mode):
    if mode.lower() == 'weekday':
        df.groupby(df['date'].map(lambda date: date.weekday()))
    else:
        return df.groupby(df['date'].map(lambda date: getattr(date, mode)))

def floorToMonth(date):
    return date.replace(day=1, hour=1, minute=1, second=1, microsecond=1)

def floorToDay(date):
    return date.replace(hour=1, minute=1, second=1, microsecond=1)

def countTokWordQuery(series, words, mode='any'):
    """
    Counts the number of queries containing a given word or set of words.

    param series: pandas series containing tokenized queries
    param words: single string or list of strings to match to each tokenized query
    param mode: options: 'any' or 'all'. Specifies whether all words should match or any words should match.

    return: Number of queries which match the given words
    """

    if type(words) is str:
        words = [words]

    # convert to lowercase
    words = [word.lower() for word in words]

    if mode == 'any':
        return sum([1 if any([word in item for word in words]) else 0 for item in series.values])
    elif mode == 'all':
        return sum([1 if all([word in item for word in words]) else 0 for item in series.values])

import datetime
def addCopyright(ax, xOffset=0, yOffset=0):
    year = datetime.date.today().year
    if any([i == '_axis3don' for i in dir(ax)]):
        ax.text(0 + xOffset, -.05 + yOffset, 0, r'$\copyright$ Ari Morcos  ' + str(year), transform=ax.transAxes)
    else:
        ax.text(0 + xOffset, -.05 + yOffset, r'$\copyright$ Ari Morcos  ' + str(year), transform=ax.transAxes)

df.shape


I am working on the file "Google Searches/Mining google searches.ipynb".
The first task is:
We can immediately see that I search for terms related to coding, finance, and video games a bunch, and I likely live in Boston.
How did searches for these terms change over time?
Can you write Python code for it?

# group by year and month
g = df.groupby(df['date'].map(floorToMonth))

# plot traces
# ax = g['tokenized'].apply(countTokWordQuery,['dog','dogs']).plot(fontsize=20);
# ax = g['tokenized'].apply(countTokWordQuery,['matlab']).plot(fontsize=20);
ax = g['tokenized'].apply(countTokWordQuery,['bash']).plot(fontsize=20);
ax = g['tokenized'].apply(countTokWordQuery,['stock','stocks','finance','nasdaq','nyse']).plot();
ax = g['tokenized'].apply(countTokWordQuery,['python']).plot();
ax = g['tokenized'].apply(countTokWordQuery,['html','css']).plot();
# ax = g['tokenized'].apply(countTokWordQuery,['mlb','baseball']).plot();
# ax = g['tokenized'].apply(countTokWordQuery,['ps4','xbox']).plot();

# add legend
ax.legend(['Bash','Finance','Python','Web Development'], loc=0, fontsize=20)

# set labels
ax.set_xlabel('')
ax.set_ylabel('# Matching Queries', fontsize=30)

# turn off grid
ax.grid()

# change linewidth and figsize
[line.set_lw(2.5) for line in ax.lines] # set linewidth
ax.figure.set_figheight(10); ax.figure.set_figwidth(15)

# add copyright
addCopyright(ax, yOffset=-0.13, xOffset=-0.03)

# plt.savefig('projectsCode.png', transparent=True, dpi=200, bbox_inches='tight')

# group by year and month
g = df.groupby(df['date'].map(floorToMonth))

# plot traces
ax = g['tokenized'].apply(countTokWordQuery,['dog','dogs']).plot(fontsize=20);
# ax = g['tokenized'].apply(countTokWordQuery,['bash']).plot(fontsize=20);
# ax = g['tokenized'].apply(countTokWordQuery,['stock','stocks','finance','nasdaq','nyse']).plot();
# ax = g['tokenized'].apply(countTokWordQuery,['python']).plot();
# ax = g['tokenized'].apply(countTokWordQuery,['html','css']).plot();
ax = g['tokenized'].apply(countTokWordQuery,['mlb','baseball','angels']).plot();
ax = g['tokenized'].apply(countTokWordQuery,['ps4','xbox']).plot();

# add legend
ax.legend(['Dog','Baseball','Video Games'], loc=0, fontsize=20)

# set labels
ax.set_xlabel('')
ax.set_ylabel('# Matching Queries', fontsize=30)

# turn off grid
ax.grid()

# change linewidth and...


I am working on the file "Gracias/ConductionLine.ipynb".
The first task is:
Changing the Pipes
The headloss in both the 4" and 6" lines is too great to handle the {{flow_branch}} flow rate. Therefore larger diameter pipe needs to be installed to reduce the headloss in the conduction line(s). There are multiple options for how to both increase the conduction line capacitiy and split the flow efficiently:

Distribution box at the plant with one large conduction line running from the existing plants.
Distribution box at the location of the current pressure breaks, with two lines running to the plant, one for each train.
Combine the flow with Ys from the two current pressure breaks into a large line, and split at the plant into each train

The first two options involve the construction of a distribution box, an unnecessary, more complex and expensive solution. All options will use two gate valves (one for each train) at each train entrance tank for fine-grain control of each flow rate. The third option will be investigated first, as it is the simplest to construct, the least expensive, and has no functional drawbacks.
To size the main trunk line, an appropriate target headloss must be chosen. Below is a graph that lists the headloss at different pipe sizes given the parameters of this plant:
Can you write Python code for it?

# Make a table with available pipe sizes
pipe_sdr = 26
pipe_diameters_nd = [6,8,10,12]#*u.inch
pipe_diameters_id = pipe.ID_sch40(pipe_diameters_nd)
headloss_various_diameters = pc.headloss(flow_full, pipe_diameters_id*u.inch,
                                         l_total,NU_WATER,PVC_ROUGHNESS,k_conduction_line)
df = pd.DataFrame(np.array(headloss_various_diameters.magnitude), index=pipe_diameters_id, columns=['Headloss (m)'])

#Graph headloss for different pipe diameters
df.index.name = 'Pipe Diameter (ID, inch)'
df.name = 'Total Headloss Through Various Pipe Diameters'
df.plot().set(ylabel="Headloss (m)", title = df.name)
plt.show()
display(df)


I am working on the file "Gracias/ConductionLine.ipynb".
The first task is:
Now the required headloss is less than 1 m, wich will help reduce shared headloss dramatically.
Total Shared (Trunk) Headloss
Now a conservative estimate of the headloss from the main conduction line is added to form the total shared headloss. The remaining head is used in the branch lengths. Using so much head to drive the flow through each branch leads to using smaller pipes and thus smaller gate valves. The following calculations prove that a 6" branch pipe diameter can be used and still achieve the full train flow rate.
Can you write Python code for it?

# set a conservative guess for a pressure break to tee headloss determined above:
headloss_pressure_break_to_tee = 1 * u.m

# headloss in the combined trunk:
headloss_conduction_line = pc.headloss(flow_full, 12*u.inch,
                                         l_total,NU_WATER,PVC_ROUGHNESS,k_conduction_line)

# total shared headloss:
headloss_shared = headloss_conduction_line + headloss_pressure_break_to_tee

# set the headloss available as the difference in height from pressure break to plant entrance:
head_available_total = min(height_pressure_break_4, height_pressure_break_6) - height_plant
head_available_for_trains = head_available_total - headloss_shared

print('The total shared headloss is: ' + str(headloss_shared))
print('The remaining headloss available for each train: ' + str(head_available_for_trains))

# calculate the headloss for various pipe sizes for a singe train branch:
pipe_diameters_nd_branch = [3,4,6,8]*u.inch
pipe_diameters_id_branch = pipe.ID_sch40(pipe_diameters_nd_branch)*u.inch

# calculate minor losses:
k_value_tee_to_plant_entrance = k.k_value_reduction(id_12, pipe_diameters_id_branch, flow_branch)\
    + exp.K_MINOR_90*4 + exp.K_MINOR_GATE_VALVE

# calculate length:
l_branch = 5 * u.m + 5 * u.m + 2 * u.m

headloss_branch = pc.headloss(flow_branch, pipe_diameters_id_branch, l_branch,
                                   exp.NU_WATER, mat.PIPE_ROUGH_PVC, k_value_tee_to_plant_entrance)
pd.DataFrame(np.array([np.array(pipe_diameters_id_branch),np.array(headloss_branch)]),
             columns=pipe_diameters_nd_branch, index=['Pipe Size (inch)', 'Headloss (m)'])


I am working on the file "GradientChecks.ipynb".
The first task is:
Batch Normalization Layer
Layers/BatchnormLayer.py
Can you write Python code for it?

batchnorm = lml.layers.BatchnormLayer(10, 0.9)
test_input = np.random.randn(20, 10)
dout = np.random.randn(20, 10)
_ = batchnorm.forward_train(test_input)
dx_num = lml.utils.numerical_gradient_layer(lambda x : batchnorm.forward_train(x), test_input, dout)
dx = batchnorm.backward(dout)
print('Batchnorm dx error:', np.max(lml.utils.relative_error(dx, dx_num)))


I am working on the file "GradientChecks.ipynb".
The first task is:
Multilayer Fully Connected Network with Augmentations
Can you write Python code for it?

opts = {
    'input_dim' : 10,
    'data_type' : np.float64,
    'init_scheme' : 'xavier'
}
nn = lml.NeuralNetwork(opts)
nn.add_layer('Affine', {'neurons':10})
nn.add_layer('Batchnorm', {'decay':0.9})
nn.add_layer('PReLU', {})
nn.add_layer('Dropout', {'dropout_param':0.85, 'seed':5684})
nn.add_layer('Affine', {'neurons':10})
nn.add_layer('Batchnorm', {'decay':0.7})
nn.add_layer('PReLU', {})
nn.add_layer('Dropout', {'dropout_param':0.90, 'seed':5684})
nn.add_layer('Affine', {'neurons':10})
nn.add_layer('Batchnorm', {'decay':0.8})
nn.add_layer('PReLU', {})
nn.add_layer('Dropout', {'dropout_param':0.95, 'seed':5684})
nn.add_layer('SoftmaxLoss', {})
test_scores = np.random.randn(20, 10)
test_classes = np.random.randint(1, 10, 20)
loss, dx = nn.backward(test_scores, test_classes)

f = lambda _: nn.backward(test_scores, test_classes, reg_param=0.7)[0]
d_b1_num = lml.utils.numerical_gradient(f, nn.layers[0].b, accuracy=1e-8)
d_W1_num = lml.utils.numerical_gradient(f, nn.layers[0].W, accuracy=1e-8)
print('Weight 1 error:', np.max(lml.utils.relative_error(nn.layers[0].dW, d_W1_num)))
print('Bias 1 error:', np.max(lml.utils.relative_error(nn.layers[0].db, d_b1_num)))

d_gamma1_num = lml.utils.numerical_gradient(f, nn.layers[1].gamma, accuracy=1e-8)
d_beta1_num = lml.utils.numerical_gradient(f, nn.layers[1].beta, accuracy=1e-8)
print('Gamma 1 error:', np.max(lml.utils.relative_error(nn.layers[1].dgamma, d_gamma1_num)))
print('Beta 1 error:', np.max(lml.utils.relative_error(nn.layers[1].dbeta, d_beta1_num)))

d_r1_num = lml.utils.numerical_gradient(f, nn.layers[2].W, accuracy=1e-8)
print('Rectifier 1 error:', np.max(lml.utils.relative_error(nn.layers[2].dW, d_r1_num)))

d_b1_num = lml.utils.numerical_gradient(f, nn.layers[4].b, accuracy=1e-8)
d_W1_num = lml.utils.numerical_gradient(f, nn.layers[4].W, accuracy=1e-8)
print('Weight 2 error:', np.max(lml.utils.relative_error(nn.layers[4].dW, d_W1_num)))
print('Bias 2 error:', np.max(lml.utils.relative_error(nn.layers[4].db, d_b1_num)))

d_gamma2_num =...


I am working on the file "GradientChecks.ipynb".
The first task is:
PReLU (Parametric Rectified Linear Unit) Layer
Layers/PReLULayer.py
Can you write Python code for it?

prelu = lml.layers.PReLULayer(10)
test_input = np.random.randn(50, 10)
dout = np.random.randn(50, 10)
_ = prelu.forward(test_input)
dx_num = lml.utils.numerical_gradient_layer(lambda x : prelu.forward(x, prelu.W), test_input, dout)
dW_num = lml.utils.numerical_gradient_layer(lambda w : prelu.forward(test_input, w), prelu.W, dout)
dx = prelu.backward(dout)
print('PReLU dx error:', np.max(lml.utils.relative_error(dx, dx_num)))
print('PReLU dW error:', np.max(lml.utils.relative_error(prelu.dW, dW_num)))


I am working on the file "GradientChecks.ipynb".
The first task is:
ReLU (Rectified Linear Unit) Layer
Layers/ReLULayer.py
Can you write Python code for it?

relu = lml.layers.ReLULayer(10)
test_input = np.random.randn(50, 10)
dout = np.random.randn(50, 10)
_ = relu.forward(test_input)
dx_num = lml.utils.numerical_gradient_layer(lambda x : relu.forward(x), test_input, dout)
dx = relu.backward(dout)
print('ReLU dx error:', np.max(lml.utils.relative_error(dx, dx_num)))


I am working on the file "GradientChecks.ipynb".
The first task is:
Softmax Loss Layer
Layers/SoftmaxLossLayer.py
Can you write Python code for it?

softmax = lml.layers.SoftmaxLossLayer(10)
test_scores = np.random.randn(50, 10)
test_classes = np.random.randint(1, 10, 50)
_, dx = softmax.loss(test_scores, test_classes)
dx_num = lml.utils.numerical_gradient(lambda x : softmax.loss(x, test_classes)[0], test_scores)
print('Softmax backprop error:', np.max(lml.utils.relative_error(dx, dx_num)))


I am working on the file "GradientChecks.ipynb".
The first task is:
Tanh Layer
Layers/TanhLayer.py
tanh = lml.layers.TanhLayer(10)
test_input = np.random.randn(50, 10)
dout = np.random.randn(50, 10)
_ = tanh.forward(test_input)
dx_num = lml.utils.numerical_gradient_layer(lambda x : tanh.forward(x), test_input, dout)
dx = tanh.backward(dout)
print('Tanh dx error:', np.max(lml.utils.relative_error(dx, dx_num)))
Full Model Gradient Checks
Two Layer Network
This is a gradient check for a simple example network with the following architecture:
Affine, ReLU, Affine, Softmax
Can you write Python code for it?

opts = {
    'input_dim' : 10,
    'data_type' : np.float64
}

nn = lml.NeuralNetwork(opts)
nn.add_layer('Affine', {'neurons':10, 'weight_scale':5e-2})
nn.add_layer('ReLU', {})
nn.add_layer('Affine', {'neurons':10, 'weight_scale':5e-2})
nn.add_layer('SoftmaxLoss', {})
test_scores = np.random.randn(20, 10)
test_classes = np.random.randint(1, 10, 20)
loss, dx = nn.backward(test_scores, test_classes)

print('With regularization off:')
f = lambda _: nn.backward(test_scores, test_classes)[0]
d_b1_num = lml.utils.numerical_gradient(f, nn.layers[0].b, accuracy=1e-8)
d_W1_num = lml.utils.numerical_gradient(f, nn.layers[0].W, accuracy=1e-8)
print('Weight 1 error:', np.max(lml.utils.relative_error(nn.layers[0].dW, d_W1_num)))
print('Bias 1 error:', np.max(lml.utils.relative_error(nn.layers[0].db, d_b1_num)))

d_b2_num = lml.utils.numerical_gradient(f, nn.layers[2].b, accuracy=1e-8)
d_W2_num = lml.utils.numerical_gradient(f, nn.layers[2].W, accuracy=1e-8)
print('Weight 2 error:', np.max(lml.utils.relative_error(nn.layers[2].dW, d_W2_num)))
print('Bias 2 error:', np.max(lml.utils.relative_error(nn.layers[2].db, d_b2_num)))

print('With regularization at lambda = 1.0:')
f = lambda _: nn.backward(test_scores, test_classes, reg_param=1.0)[0]
d_b1_num = lml.utils.numerical_gradient(f, nn.layers[0].b, accuracy=1e-8)
d_W1_num = lml.utils.numerical_gradient(f, nn.layers[0].W, accuracy=1e-8)
print('Weight 1 error:', np.max(lml.utils.relative_error(nn.layers[0].dW, d_W1_num)))
print('Bias 1 error:', np.max(lml.utils.relative_error(nn.layers[0].db, d_b1_num)))

d_b2_num = lml.utils.numerical_gradient(f, nn.layers[2].b, accuracy=1e-8)
d_W2_num = lml.utils.numerical_gradient(f, nn.layers[2].W, accuracy=1e-8)
print('Weight 2 error:', np.max(lml.utils.relative_error(nn.layers[2].dW, d_W2_num)))
print('Bias 2 error:', np.max(lml.utils.relative_error(nn.layers[2].db, d_b2_num)))


I am working on the file "GradientDescend.ipynb".
The first task is:
Ground Truth Model
Can you write Python code for it?

n = 5
a = 2.0
b = 3.0
epsilon = 0.02

# Specify our ground truth model
X = torch.randint(10, (n, 1))
ps = torch.ones(n)/2.0
exponents = torch.bernoulli(ps) * epsilon + (torch.ones(n) - epsilon/2)
e = exponents.reshape(5, 1)
T = a * (X.pow(e)) + b

print(X)
print(e)
print(T)

# verify that our implementation is correct.
print( a * X[0][0]**e[0][0] + b)


I am working on the file "GradientDescend.ipynb".
The first task is:
The above output shows that we can do vectorized operations on tensors easily.
You should be able to write down the ground truth model mathematically.
Fitting a Linear Model
$$\hat{\mathbf{y}} = \mathbf{x}\mathbf{w} + \mathbf{b}$$
Can you write Python code for it?

# note that all operations here are vectorized operation.

def forward(x): # dangerous: assuming w and b are global variables.
    return x.mm(w) + b

# Loss function
def SSEloss(y_pred, y):
    return (y_pred - y) * (y_pred - y) /2.0

def loss(x, t):
    y_pred = forward(x)
    return (y_pred - t).pow(2).sum() /2.0

def gradient_w(x, t):
    return  torch.t(forward(x) - t).mm(x)

def gradient_b(x, t):
    return  (forward(x) - t).sum() # need to add sum(); otherwise, each b_i will be different due to different gradients it receives.

# Training loop
MAX_EPOCHES = 500
learning_rate = 0.01 # it is tricky to find a good learning rate.

w = torch.randn(1, 1)
b = torch.ones(n, 1)

for epoch in range(MAX_EPOCHES):
    grad_w = gradient_w(X, T)
    w = w - learning_rate * grad_w
    grad_b = gradient_b(X, T)
    b = b - learning_rate * grad_b

    print(".. grad: ", w.item(), b[0].item())
    l = loss(X, T)
    print("epoch = ", epoch, ", loss=", l.item(), "learning rate = ", learning_rate)

    # Adaptive learning rate on: 500 epoches => loss = 0.04446
    #                       off: 500 epoches => loss = 0.04179
    #learning_rate = 1.0 / (100 + epoch)

# After training
print(forward(X))
print(T)

print(w)
print(b)


I am working on the file "GradientDescend.ipynb".
The first task is:
Vectorizes Operation
Can you write Python code for it?

u = torch.randint(10, (5, 1))
v = torch.randint(10, (5, 1))
print(u)
print(v)

u.size()

u * v

u.pow(v)


I am working on the file "GradientDescent.ipynb".
The first task is:
Advanced Material
Why does momentum work? by Goh. Interactive articles
https://distill.pub/2017/momentum/
Beyond vanilla Gradient Descent, Sebastian Ruder blog
http://ruder.io/optimizing-gradient-descent/
Blog post by M. Hardt
http://blog.mrtz.org/2013/09/07/the-zen-of-gradient-descent.html
A great talk by Ben Recht:
https://simons.berkeley.edu/talks/ben-recht-2013-09-04
Backpropagation, by blog post by S. Arora
http://www.offconvex.org/2016/12/20/backprop/
Analysis of convergence of Gradient descent for a quadratic function
Recall that the error function we minimize is
$$
E(w) = \frac{1}{2} (y-Aw)^T(y-Aw) = \frac{1}{2}(y^\top y - 2 y^\top A w + w^\top A^\top A w)
$$
The gradient at the point $w$ will be denoted as $\nabla E(w) = g(w)$ where
$$g(w) = -A^\top (y - Aw) = A^\top A w - A^\top y$$
Moreover, the gradient at the minimum will vanish:
$$
g(w_\star) = 0
$$
Indeed, we can solve
$$0 = A^\top (Aw_\star - y)$$
as
$$w_\star =  (A^\top A)^{-1}A^\top y  $$
but this is not our point.
For a constant learning rate $\eta$, gradient descent executes the following iteration
$$
w_t = w_{t-1} - \eta g(w_{t-1}) = w_{t-1} - \eta A^\top (Aw_{t-1} - y)
$$
$$
w_t  =  (I - \eta A^\top A) w_{t-1} + \eta A^\top y
$$
This is a fixed point equation of form
$$
w_t  =  T(w_{t-1})
$$
where $T$ is an affine transformation.
We will assume that $T$ is a contraction, i.e. for any two different
parameters $w$ and $w'$ in the domain we have
$$
\| T(w) - T(w') \| \leq L_\eta \|w-w' \|
$$
where $L_\eta < 1$, then the distance shrinks. Hence the mapping converges to a fixed point (this is a consequence of a deeper result in analysis called the Brouwer fixed-point theorem (https://en.0wikipedia.org/wiki/Brouwer_fixed-point_theorem))
We will consider in particular the distance between the optimum and the current point $w(t)$
$$
\| T(w_t) - T(w_\star) \| \leq L_\eta \|w_t - w_\star \|
$$
But we have
$T(w_\star) = w_\star$ and $w_t = T(w_{t-1})$ so
$\|w_t - w_\star \| = \|T(w_{t-1}) - T(w_\star) \|$.
\begin{align}
\| T(w_t) - T(w_\star)...
Can you write Python code for it?

left = 0.0000
right = 0.015

N = 1000
ETA = np.linspace(left,right,N)

def compute_largest_eig(ETA, A):

    LAM = np.zeros(N)
    D = A.shape[1]
    n = A.shape[0]

    for i,eta in enumerate(ETA):
        #print(eta)
        lam,v = np.linalg.eig(np.eye(D) - 2*eta*A.T.dot(A)/n)
        LAM[i] = np.max(np.abs(lam))

    return LAM

# This number is L_\eta
LAM = compute_largest_eig(ETA, A)

plt.plot(ETA, LAM)
#plt.plot(ETA, np.ones((N,1)))
#plt.gca().set_ylim([0.98, 1.02])
plt.ylim([0.997,1.01])
plt.xlabel('eta')
plt.ylabel('absolute value of the largest eigenvalue')
plt.show()

plt.semilogy(ETA,LAM)
plt.ylim([0.997,1])
plt.show()


I am working on the file "GradientDescent.ipynb".
The first task is:
Analysis of Momentum
\begin{align}
p(\tau) & =  \nabla E(w(\tau-1)) + \beta p(\tau-1) \
w(\tau) & =  w(\tau-1) - \alpha p(\tau) \
w(\tau-1) & =  w(\tau-2) - \alpha p(\tau-1) \
\end{align}
\begin{align}
\left(\begin{array}{c}
w(\tau) \
w(\tau-1)
\end{array}
\right)
& = &\left(\begin{array}{cc}
\cdot & \cdot \
\cdot & \cdot
\end{array}
\right)
\left(\begin{array}{c}
w(\tau-1) \
w(\tau-2)
\end{array}
\right)
\end{align}
\begin{align}
\frac{1}{\alpha}(w(\tau-1) - w(\tau))  & =  p(\tau) =  \nabla E(w(\tau-1)) + \beta \frac{1}{\alpha}(w(\tau-2) - w(\tau-1)) = \
\frac{1}{\alpha}(w(\tau-2) - w(\tau-1))  & =  p(\tau-1) \
\end{align}
\begin{align}
\frac{1}{\alpha}(w(\tau-1) - w(\tau))  & = \nabla E(w(\tau-1)) + \beta \frac{1}{\alpha}(w(\tau-2) - w(\tau-1)) \
  w(\tau)  & = -\alpha \nabla E(w(\tau-1)) - \beta w(\tau-2) + (\beta+1)  w(\tau-1)
\end{align}

Note that GD is sensetive to scaling of data
For example, if we would not have shifted the $x$ axis our original data, GD might not have worked. The maximum eigenvalue is very close to $1$ for all $\eta$ upto numerical precision
Can you write Python code for it?

w = 7
alpha = 0.7
EPOCH = 100
W = []
for tau in range(EPOCH):
    W.append(w)
    w = w - alpha*(2*w - 4)

plt.plot(W)
plt.show()

w = 7
alpha = 0.1
beta = 0.95
p = 0
EPOCH = 1000
W = []
for tau in range(EPOCH):
    W.append(w)
    p = (2*w - 4) + beta*p
    w = w - alpha*p

plt.plot(W)
plt.show()


I am working on the file "GradientDescent.ipynb".
The first task is:
If $E$ is twice differentiable, contractivity means that $E$ is convex.
For $t>0$
\begin{align}
\|T(x + t \Delta x) - T(x) \| & \leq \rho \|t \Delta x\| \
\frac{1}{t} \|T(x + t \Delta x) - T(x) \| &\leq \rho \|\Delta x\|
\end{align}
If we can show that $\rho< 1$, then $T$ is a contraction.
By definitions
$$
T(x)  = x - \alpha \nabla E(x)
$$
$$
T(x + t \Delta x)  = x + t \Delta x - \alpha \nabla E(x + t \Delta x)
$$
\begin{align}
\frac{1}{t} \|T(x + t \Delta x) - T(x) \| & = \frac{1}{t} \|x + t \Delta x - \alpha \nabla E(x + t \Delta x) - x + \alpha \nabla E(x) \| \
& = \| \Delta x - \frac{\alpha}{t}  (\nabla E(x + t \Delta x) - \nabla E(x) ) \|  \
\end{align}
As this relation holds for all $t$, we take the limit when $t\rightarrow 0^+$
\begin{align}
\| \Delta x - \alpha \nabla^2 E(x) \Delta x  \| & =  \| (I - \alpha \nabla^2 E(x)) \Delta x  \| \
& \leq \| I - \alpha \nabla^2 E(x) \| \| \Delta x  \|
\end{align}
If we can choose $\alpha$ for all $\xi$ in the domain such that
$$
\| I - \alpha \nabla^2 E(\xi) \| \leq \rho < 1
$$
is satisfied, we have a sufficient condition for a contraction.
Lemma:
Assume that for $0 \leq \rho < 1$, $\alpha> 0$ and $U(\xi)$ is a symmetric matrix valued function for all $\xi \in \mathcal{D}$ and we have
$$
\| I - \alpha U(\xi) \| \leq \rho
$$
then $U = U(\xi)$ is positive semidefinite with $$\frac{1 - \rho}{\alpha} I \preceq U $$ for every $\xi$.
Proof:
$$
\|I - \alpha U \| = \sup_{x\neq 0} \frac{x^\top(I - \alpha U )x }{x^\top x} \leq \rho
$$
$$
x^\top(I - \alpha U )x \leq \rho x^\top x
$$
$$
(1- \rho) x^\top x  \leq \alpha x^\top U x
$$
This implies that for all $x$ we have
$$
0 \leq x^\top (U - \frac{1 - \rho}{\alpha} I) x
$$
In other words, the matrix $U - \frac{1 - \rho}{\alpha} I$ is positive semidefinite, or:
$$
\frac{1 - \rho}{\alpha} I \preceq U
$$
We now see that $\rho<1$ we have the guarantee that $U$ is positive semidefinite.
$$
T(x) = M x + b
$$
$$
\|T(x) - T(x_\star) \| = \|Mx + b - M x_\star + b \| = \| M(x-x_\star) \|
$$
By Schwarz inequality
$$
\|T(x) -...
Can you write Python code for it?

# Try to fit with GD to the original data
BaseYear2 = 0
x2 = np.matrix(df_arac.Year[31:]).T-BaseYear2

# Setup the vandermonde matrix
N = len(x2)
A = np.hstack((np.ones((N,1)), x2))

left = -8
right = -7.55
N = 100
ETA = np.logspace(left,right,N)

LAM = compute_largest_eig(ETA, A)

plt.plot(ETA, LAM)
plt.plot(ETA, np.ones((N,1)))
plt.gca().set_ylim([0.98, 1.02])
plt.xlabel('eta')
plt.ylabel('absolute value of the largest eigenvalue')
plt.show()


I am working on the file "GradientDescent.ipynb".
The first task is:
The following cell demonstrates interactively the progress of plain gradient descent
and how its solution differs from the optimum found by solving the corresponding least squares problem.
Can you write Python code for it?

fig = plt.figure(figsize=(5,5))

left = -1.5
right = 1.5
xx = np.linspace(left,right,50)
AA = np.hstack((np.power(xx.reshape(len(xx),1),i) for i in range(degree+1)))

# Find best
A_orth, R = np.linalg.qr(A)
w_orth, res, rank, s = np.linalg.lstsq(A_orth, y)
w_star = np.linalg.solve(R, w_orth)
yy = AA.dot(w_star)

#ax.set_xlim((2,15))

#dots = plt.Line2D(x,y, linestyle='', markerfacecolor='b',marker='o', alpha=0.5, markersize=5)
#ax.add_line(dots)
plt.plot(x,y, linestyle='', markerfacecolor='b',marker='o', alpha=0.5, markersize=5)
plt.plot(xx, yy, linestyle=':', color='k', alpha=0.3)
ln = plt.Line2D(xdata=[], ydata=[], linestyle='-',linewidth=2)

ax = fig.gca()
ax.add_line(ln)
plt.close(fig)

ax.set_xlim((left,right))
ax.set_ylim((5,9))

def plot_gd(iteration=0):
    w = W[iteration]
    f = AA.dot(w)
    #print(w)
    ln.set_ydata(f)
    ln.set_xdata(xx)

    ax.set_title('$E = '+str(Err[iteration])+'$')

    display(fig)

res = interact(plot_gd, iteration=(0,len(W)-1))


I am working on the file "GradientDescentExamples.ipynb".
The first task is:
Here we can see that we took a pretty big step towards the minimum, just as we'd like.
Let's take another step:
Can you write Python code for it?

another_x = next_x - alpha*f_prime(next_x)

# take a look at the combination of the two steps we've taken
plt.plot(x, f(x), c='black')
plt.plot([x_start, next_x], [f(x_start), f(next_x)], c='red')
plt.scatter([x_start, next_x], [f(x_start), f(next_x)], c='red')
plt.plot([x_start, next_x, another_x], [f(x_start), f(next_x), f(another_x)], c='red')
plt.scatter([x_start, next_x, another_x], [f(x_start), f(next_x), f(another_x)], c='red')
plt.xlim((-4, 4))
plt.ylim((0, 16))
sns.despine();


I am working on the file "GradientDescentExamples.ipynb".
The first task is:
In this algorithm, alpha is known as the "learning rate", and all it does is keep us from taking steps that are too aggressive, where we could shoot past the minimum.
Can you write Python code for it?

# let's take our first step!
step1 = alpha*f_prime(x_start)

# our new value of x is just the previous value, minus the step
next_x = x_start - step1

# take a look at the step we took, with respect to the curve
plt.plot(x, f(x), c='black')
plt.scatter([x_start, next_x], [f(x_start), f(next_x)], c='red')
plt.plot([x_start, next_x], [f(x_start), f(next_x)], c='red')
plt.xlim((-4, 4))
plt.ylim((0, 16))
sns.despine();


I am working on the file "GradientDescentExamples.ipynb".
The first task is:
Let's assume we start at the top of the curve, at x = -4, and want to get down to x=0.
Can you write Python code for it?

# starting position on the curve
x_start = -4.0

# looking at the values of the derivative, for each value of x.
# we see the greatest change at the tops of the curve, namely 4 and -4
plt.plot(x, f_prime(x), c='black');

# learning rate
alpha = 0.2


I am working on the file "GradientDescentExamples.ipynb".
The first task is:
Start with our simple example
Let's start with $f(x) = x^2$:
Can you write Python code for it?

# make our x array
x = np.linspace(-4, 4, 801)

# f(x) = x^2
def f(x):
    return x**2

# derivative of x^2 is 2x
def f_prime(x):
    return 2*x

# take a look at the curve
plt.plot(x, f(x), c='black')
sns.despine();


I am working on the file "Gradient_Checking.ipynb".
The first task is:
Compute diffs between theta's gradient as mathematically defined and the gradient as defined by our function above.
Can you write Python code for it?

diffs = np.empty_like(theta)
for i in range(theta_plus.shape[0]):
    gradient_def = (
        (cost_function(theta_plus[i], train_X, train_y) - cost_function(theta_minus[i], train_X, train_y)) /
        (2 * epsilon)
    )
    gradient_lin_reg = gradient(theta, train_X, train_y)[i]
    diffs[i] = np.absolute(gradient_def - gradient_lin_reg)

diffs


I am working on the file "Gradient_Checking.ipynb".
The first task is:
Define some necessary functions.
Can you write Python code for it?

def cost_function(theta, X, y):
    squared_errors = (X.dot(theta) - y) ** 2
    J = 0.5 * squared_errors.sum()
    return J

def gradient(theta, X, y):
    errors = X.dot(theta) - y
    return errors.dot(X)


I am working on the file "Gradient_Checking.ipynb".
The first task is:
Gradient Checking
Define "step size" (don't set it too low to avoid numerical precision issues).
Can you write Python code for it?


I am working on the file "Gradient_Checking.ipynb".
The first task is:
Lookin' good! The smaller the values, the better.<br>
(Any value significantly larger than 1e-4 indicates a problem.)
Can you write Python code for it?


I am working on the file "Gradient_Checking.ipynb".
The first task is:
Prepare theta step values (making use of numpy broadcasting).
Can you write Python code for it?

mask = np.identity(theta.size)
theta_plus = theta + epsilon * mask
theta_minus = theta - epsilon * mask


I am working on the file "Gradients.ipynb".
The first task is:
And write them to an excel file
Can you write Python code for it?

data = df_phen.loc[df_phen["selec"] == 1]
data['filenamelow'] = data['filename'].str.lower()
data = data.sort(['filenamelow'])

output = data.merge(data_grdnt, left_on='filename',right_on='file',how='outer')
output.to_csv('Combined.csv', sep='\t')


I am working on the file "Gradients.ipynb".
The first task is:
Check all files to see if any have missing nodal information and create a selection list based on the ones that are 100% complete.
Can you write Python code for it?

# check to see which files contains nodes with missing information
missingarray = []
for i in onlyfiles:
# load timeseries
    filename = i
    ts_raw = np.loadtxt(filename)

# check zero columns
    missingn = np.where(~ts_raw.any(axis=0))[0]
    missingarray.append(missingn)

# select the ones that don't have missing data
ids = np.where([len(i) == 0 for i in missingarray])[0]
selected = [onlyfiles[i] for i in ids]
# could be useful to have one without pathnames later one
selected2 = [os.path.basename(onlyfiles[i]) for i in ids]
print(len(selected))


I am working on the file "Gradients.ipynb".
The first task is:
Check all individual images
Can you write Python code for it?

nims = rebuild_nii_individ(0)

!fslview resCaltech_0051474_rois_cc400.1D.npy.nii


I am working on the file "Gradients.ipynb".
The first task is:
Output everything to an excel file
Can you write Python code for it?

import pandas as pd
# read in csv
df_phen = pd.read_csv('Phenotypic_V1_0b_preprocessed1.csv')
# add a column that matches the filename
for i in df_phen:
    df_phen['filename'] = join(df_phen['FILE_ID']+"_rois_cc400.1D")
    df_phen['filenamelpy'] = join(df_phen['FILE_ID']+"_rois_cc400.1D.npy")

df_phen['selec'] = np.where(df_phen['filename'].isin((selected2)), 1, 0)


I am working on the file "Gradients.ipynb".
The first task is:
Plot some stuff
Can you write Python code for it?

## numpy is used for creating fake data
%matplotlib inline
import numpy as np
import matplotlib as mpl

## agg backend is used to create plot as a .png file
mpl.use('agg')

import matplotlib.pyplot as plt
df = pd.DataFrame(output, columns = ['DX_GROUP', 0,1,2,3,4,5,6,7,8,9])
ASC = df['DX_GROUP'] == 2
NT = df['DX_GROUP'] == 1
G1 = df[ASC]
G2 = df[NT]

# some plotting options
fs = 10  # fontsize
flierprops = dict(marker='o', markerfacecolor='green', markersize=12,
                  linestyle='none')

## combine the groups collections into a list
Grd0 = [G1[0], G2[0]]
Grd1 = [G1[1], G2[1]]
Grd2 = [G1[2], G2[2]]
Grd3 = [G1[3], G2[3]]
Grd4 = [G1[4], G2[4]]
Grd5 = [G1[5], G2[5]]
Grd6 = [G1[6], G2[6]]
Grd7 = [G1[7], G2[7]]
Grd8 = [G1[8], G2[8]]
Grd9 = [G1[9], G2[9]]

fig, axes = plt.subplots(nrows=2, ncols=5, figsize=(6, 6), sharey=True)

axes[0, 0].boxplot(Grd0, patch_artist=True)
axes[0, 0].set_title('G0', fontsize=fs)

axes[0, 1].boxplot(Grd1, patch_artist=True)
axes[0, 1].set_title('G1', fontsize=fs)

axes[0, 2].boxplot(Grd2, patch_artist=True)
axes[0, 2].set_title('G2', fontsize=fs)

axes[0, 3].boxplot(Grd3, patch_artist=True)
axes[0, 3].set_title('G3', fontsize=fs)

axes[0, 4].boxplot(Grd4, patch_artist=True)
axes[0, 4].set_title('G4', fontsize=fs)

axes[1, 0].boxplot(Grd5, patch_artist=True)
axes[1, 0].set_title('G5', fontsize=fs)

axes[1, 1].boxplot(Grd6, patch_artist=True)
axes[1, 1].set_title('G6', fontsize=fs)

axes[1, 2].boxplot(Grd7, patch_artist=True)
axes[1, 2].set_title('G7', fontsize=fs)

axes[1, 3].boxplot(Grd8, patch_artist=True)
axes[1, 3].set_title('G8', fontsize=fs)

axes[1, 4].boxplot(Grd9, patch_artist=True)
axes[1, 4].set_title('G9', fontsize=fs)

fig.suptitle("Gradient Slopes")
fig.subplots_adjust(hspace=0.4)


I am working on the file "Gradients.ipynb".
The first task is:
Run Statis to back-project the grouped embeddings
Can you write Python code for it?

%%capture
from pySTATIS import statis

#load vectors
names = list(xrange(392))
X = [np.load("./data/Outputs/Embs/"+ os.path.basename(filename)+"_embedding_dense_res_veconly.npy") for filename in selected2]
out = statis.statis(X, names, fname='statis_results.npy')
statis.project_back(X, out['Q'], path = "./data/Outputs/Regs/",fnames = selected2)
np.save("Mean_Vec.npy",out['F'])

# saving everything in one dump
import pickle
with open('output.pickle' ,'w') as f:
    pickle.dump([selected, out],f)


I am working on the file "Gradients.ipynb".
The first task is:
Some quality control
Can you write Python code for it?

%matplotlib inline
# this cell in only necessary for plotting below
import matplotlib.pylab as plt
import nilearn
import nilearn.plotting

import numpy as np
import nibabel as nib
from os import listdir
from os.path import isfile, join

def rebuild_nii(num):

    data = np.load('Mean_Vec.npy')
    a = data[:,num].copy()
    nim = nib.load('cc400_roi_atlas.nii')
    imdat=nim.get_data()
    imdat_new = imdat.copy()

    for n, i in enumerate(np.unique(imdat)):
        if i != 0:
            imdat_new[imdat == i] = a[n-1] * 100000 # scaling factor. Could also try to get float values in nifti...

    nim_out = nib.Nifti1Image(imdat_new, nim.get_affine(), nim.get_header())
    nim_out.set_data_dtype('float32')
    # to save:
    # nim_out.to_filename('res.nii')

    nilearn.plotting.plot_epi(nim_out)

def rebuild_nii_individ(num):

    onlyfiles = [f for f in listdir_nohidden('./data/Outputs/Regs/') if isfile(join('./data/Outputs/Regs/', f))]
    for index in range(178):

        sub = onlyfiles[index]
        print(sub)
        data = np.load('./data/Outputs/Regs/%s' % sub)
        a = data[:,num].astype('float32')
        nim = nib.load('cc400_roi_atlas.nii')
        imdat = nim.get_data().astype('float32')

        #print(np.unique(a))
        for i in np.unique(imdat):
            #a[a>0.1] = 0
            #a[a<-0.1] = 0
            if i != 0 and i < 392:
                imdat[imdat == i] = a[int(i)-1] # scaling factor. Could also try to get float values in nifti...
            elif i >= 392:
                imdat[imdat == i] = np.nan

        nim_out = nib.Nifti1Image(imdat, nim.get_affine(), nim.get_header())
        nim_out.set_data_dtype('float32')
        # to save:
        nim_out.to_filename(os.getcwd() + '/data/Outputs/individual/' + 'res' + sub + str(num) + '.nii')
        print(os.getcwd())
        # nilearn.plotting.plot_epi(nim_out)


I am working on the file "Gradients.ipynb".
The first task is:
This example shows very nicely that altought we are close at finding the minimum we still have trouble to converge. This is due to the fixed learning rate. Hence, it is sensible to reduce the learning rate with time, e.g. by using exponetial decay $\mu = \mu_0*e^{-kt}$ (see for others).
Can you write Python code for it?

def decay_exp(mu_0, t, k):
    return mu_0 * np.exp(-k*t)

def optimize_simple(f, x, mu, mudecay = decay_exp, k = 1, maxiter = 1000, eps = 0.001):
    y = f(x)
    i = 1
    yn = np.inf

    xhist = [x]
    yhist = [y]
    gradhist = [np.inf]

    mu_act = mudecay(mu, 0, k)
    while (not np.isclose(y, yn)) and (i < maxiter):
        y = yn
        ftick_x = numerical_derivative(x, f)
        x = x - mu_act * ftick_x
        yn = f(x)

        xhist.append(x)
        yhist.append(yn)
        gradhist.append(ftick_x)

        mu_act = mudecay(mu, i, k)
        i += 1

    return xhist, yhist, gradhist

plt.plot(x,y)
xhist, yhist, gradhist = optimize_simple(any_function, 2, 0.2)
print len(xhist)
plt.plot(xhist, yhist, '.r')

plt.show()

@interact(x_in = (-4, 4, 0.1), mu = (0.01, 1, 0.01), k = (0.01, 5, 0.01))
def interactive_optim(x_in, mu, k):
    xhist, yhist, gradhist = optimize_simple(any_function, x_in, mu, k=k)
    xx = np.arange(min(xhist), max(xhist), 0.01)
    yy = any_function(xx)
    print len(xhist)
    plt.plot(xx,yy)
    plt.plot(xhist, yhist, '.r')
    plt.show()


I am working on the file "Gradients.ipynb".
The first task is:
plotting
plot to surface for inspection
this cell in only necessary for plotting below
Can you write Python code for it?

%matplotlib inline
import matplotlib.pylab as plt
import nilearn
import nilearn.plotting

import numpy as np
import nibabel as nib

def rebuild_nii(num):

    data = np.load('Mean_Vec.npy')
    a = data[:,num].copy()
    nim = nib.load('cc400_roi_atlas.nii')
    imdat=nim.get_data()
    imdat_new = imdat.copy()

    for n, i in enumerate(np.unique(imdat)):
        if i != 0:
            imdat_new[imdat == i] = a[n-1] * 10000 # scaling factor. Could also try to get float values in nifti...

    nim_out = nib.Nifti1Image(imdat_new, nim.get_affine(), nim.get_header())
    nim_out.set_data_dtype('float32')
    # to save:
    nim_out.to_filename('Gradient_'+ str(num) +'_res.nii')

    nilearn.plotting.plot_epi(nim_out)
    return(nim_out)

for i in range(10):
    nims = rebuild_nii(i)


I am working on the file "Gradients.ipynb".
The first task is:
run the diffusion embedding
Can you write Python code for it?

# run the diffusion embedding
from mapalign import embed

for i in selected:
    # load timeseries
    #print i
    filename = i
    ts = np.loadtxt(filename)
    # create correlation matrix
    dcon = np.corrcoef(ts.T)
    dcon[np.isnan(dcon)] = 0

    # Get number of nodes
    N = dcon.shape[0]

    # threshold
    perc = np.array([np.percentile(x, 90) for x in dcon])

    for ii in range(dcon.shape[0]):
        #print "Row %d" % ii
        dcon[ii, dcon[ii,:] < perc[ii]] = 0

    # If there are any left then set them to zero
    dcon[dcon < 0] = 0

    # compute the pairwise correctionlation distances
    aff = 1 - pairwise_distances(dcon, metric = 'cosine')

    # start saving
    savename = os.path.basename(filename)
    np.save("./data/Outputs/Affn/"+savename+"_cosine_affinity.npy", aff)
    # get the diffusion maps
    emb, res = embed.compute_diffusion_map(aff, alpha = 0.5)
    # Save results
    np.save("./data/Outputs/Embs/"+savename+"_embedding_dense_emb.npy", emb)
    np.save("./data/Outputs/Embs/"+savename+"_embedding_dense_res.npy", res)

    X = res['vectors']
    X = (X.T/X[:,0]).T[:,1:]

    np.save("./data/Outputs/Embs/"+savename+"_embedding_dense_res_veconly.npy", X) #store vectors only


I am working on the file "Graficas 2 y 3d y gráficas de nivel con matplotlib.ipynb".
The first task is:
Campos vectoriales
Can you write Python code for it?

fig = plt.figure()
ax = fig.gca(projection='3d')

# Make the grid
x, y, z = np.meshgrid(np.arange(-0.8, 1, 0.2),
                      np.arange(-0.8, 1, 0.2),
                      np.arange(-0.8, 1, 0.8))

# Make the direction data for the arrows
u = np.sin(np.pi * x) * np.cos(np.pi * y) * np.cos(np.pi * z)
v = -np.cos(np.pi * x) * np.sin(np.pi * y) * np.cos(np.pi * z)
w = (np.sqrt(2.0 / 3.0) * np.cos(np.pi * x) * np.cos(np.pi * y) *
     np.sin(np.pi * z))

ax.quiver(x, y, z, u, v, w, length=0.1, normalize=True)

plt.show()

fig = plt.figure(figsize=(8,8))
ax = fig.gca(projection='3d')

# Make the grid
theta, phi = np.meshgrid(np.arange(0, 3.1416, 0.1), np.arange(-3.1415, 3.1416, 0.2))
theta2, phi2 = np.meshgrid(np.arange(0, 3.1416, 0.5), np.arange(-3.1415, 3.1416, 0.5))
xs = np.sin(theta)*np.cos(phi)
ys = np.sin(theta)*np.sin(phi)
zs = np.cos(theta)

xs2 = np.sin(theta2)*np.cos(phi2)
ys2 = np.sin(theta2)*np.sin(phi2)
zs2 = np.cos(theta2)

ax.plot_surface(xs,ys,zs)
size = 3
ax.quiver(xs2, ys2, zs2, size*xs2, size*ys2, size*zs2, length=0.1, normalize=False,color="red")

plt.show()


I am working on the file "Graficas 2 y 3d y gráficas de nivel con matplotlib.ipynb".
The first task is:
Gráficos de curvas y superficies paramétricas
Can you write Python code for it?

# Graficar una curva paramétrica en dos dimensiones

t = np.linspace(0,10*np.pi,100)

x = t * np.cos(t)
y = t * np.sin(t)
plt.plot(x,y)

# Graficar una curva paramétrica en tres dimensiones

# Genera una lista de valores para el parámetro
t = np.linspace(0,10*np.pi,100)

# Genera la curva
x = t * np.cos(t)
y = t * np.sin(t)
z = t

# Construye la figura y el sistema de ejes
fig = plt.figure()
ax = fig.gca(projection='3d')

# Grafica la curva paramétrica
ax.plot(x, y, z, label='parametric curve 1')
ax.plot(x, z, y, label='parametric curve 2')
ax.legend()

# Graficar una superficie paramétrica en tres dimensiones

u = np.linspace(0,np.pi,100)
v = np.linspace(0,2*np.pi,100)

# Construye a partir del par de parámetros, una grilla con los valores
U, V =np.meshgrid(u, v)

# A partir de la grilla, construye el conjunto de puntos. En este caso, una esfera.
x = np.sin(U)*np.cos(V)
y = np.sin(U)*np.sin(V)
z = np.cos(U)

# Construye la figura y los ejes
fig = plt.figure()
ax = fig.gca(projection='3d')

# Plotea una superficie (llena)
ax.plot_surface(x, y, z, color="orange")

# Acá construimos un hiperboloide

x = np.sinh(.5*U)*np.cos(V)
y = np.sinh(.5*U)*np.sin(V)
z = np.cosh(.5*U)

# Ploteamos una gráfica de alambres (sin relleno)
ax.plot_wireframe(x, y, z, label='hiperboloide',color="red")

ax.legend()


I am working on the file "GraphStructure.ipynb".
The first task is:
Structural properties of a graph
degree(), betweeness(), edge_betweeness(),
Some examples#
Can you write Python code for it?

# Styling graph
from igraph import *
import numpy as np

# Create the graph
vertices = ["one", "two", "three"]
edges = [(0,2),(2,1),(0,1)]

g = Graph(vertex_attrs={"label": vertices}, edges=edges, directed=True)

visual_style = {}

# Scale vertices based on degree
outdegree = g.outdegree()
visual_style["vertex_size"] = [x/max(outdegree)*50+110 for x in outdegree]

# Set bbox and margin
visual_style["bbox"] = (800,800)
visual_style["margin"] = 100

# Define colors used for outdegree visualization
colours = ['#fecc5c', '#a31a1c']

# Order vertices in bins based on outdegree
bins = np.linspace(0, max(outdegree), len(colours))
digitized_degrees =  np.digitize(outdegree, bins)

# Set colors according to bins
g.vs["color"] = [colours[x-1] for x in digitized_degrees]

# Also color the edges
for ind, color in enumerate(g.vs["color"]):
        edges = g.es.select(_source=ind)
        edges["color"] = [color]

# Don't curve the edges
visual_style["edge_curved"] = False
visual_style["bbox"] = (500, 500)
# Plot the graph
plot(g, **visual_style)

# Community detection
# https://stackoverflow.com/questions/25254151/using-igraph-in-python-for-community-detection-and-writing-community-number-for

from igraph import *
import numpy as np

# Create the graph
vertices = [i for i in range(7)]
edges = [(0,2),(0,1),(0,3),(1,0),(1,2),(1,3),(2,0),(2,1),(2,3),(3,0),(3,1),(3,2),(2,4),(4,5),(4,6),(5,4),(5,6),(6,4),(6,5)]

g = Graph(vertex_attrs={"label":vertices}, edges=edges, directed=True)

visual_style = {}

# Scale vertices based on degree
outdegree = g.outdegree()
visual_style["vertex_size"] = [x/max(outdegree)*25+50 for x in outdegree]

# Set bbox and margin
visual_style["bbox"] = (600,600)
visual_style["margin"] = 100

# Define colors used for outdegree visualization
colours = ['#fecc5c', '#a31a1c']

# Order vertices in bins based on outdegree
bins = np.linspace(0, max(outdegree), len(colours))
digitized_degrees =  np.digitize(outdegree, bins)

# Set colors according to bins
g.vs["color"] = [colours[x-1] for x in...


I am working on the file "GraphStructure.ipynb".
The first task is:
Various layouts are available:
- circle, drl, lgl, fr, grid_fr (https://github.com/gephi/gephi/wiki/Fruchterman-Reingold), kk (https://pdfs.semanticscholar.org/b8d3/bca50ccc573c5cb99f7d201e8acce6618f04.pdf), random, tree (rt, http://hci.stanford.edu/courses/cs448b/f09/lectures/CS448B-20091021-GraphsAndTrees.pdf),
- fr3d, kk3d, random3d, sphere not plottable in python: https://stackoverflow.com/questions/16907564/how-can-i-plot-a-3d-graph-in-python-with-igraph
- grid_fr listed in the documentation, but not implemented
Can you write Python code for it?


I am working on the file "GraphStructure.ipynb".
The first task is:
betweenness(vertices=None, directed=True, cutoff=None, weights=None, nobigint=True)
    source code
Calculates or estimates the betweenness of vertices in a graph.
Can you write Python code for it?


I am working on the file "GrapheneWithQrays.ipynb".
The first task is:

Note these are the expected numbers for consecutive rings.
Now that we have our database, it's time to generate some graphical output.  As with the FCC, I'll use POV-Ray's scene description language and then render in POV-Ray.  We just want to look at the edges and carbon atom vertexes.
Can you write Python code for it?

sph = """sphere { %s 0.1 texture { pigment { color rgb <1,0,0> } } }"""
cyl = """cylinder { %s %s 0.05 texture { pigment { color rgb <1.0, 0.65, 0.0> } } }"""

def make_graphene(fname="../c6xty/graphene.pov", append=True):
    """
    Scan through carbons, edges, converting to XYZ and embedding
    in POV-Ray Scene Description Language
    """
    if append:
        pov = open(fname, "a")
    else:
        pov = open(fname, "w")

    # graphene will be included as a single object in the
    # parent povray script, where lighting, camera position,
    # and background are defined

    print("#declare graphene = union{", file=pov)
    for atom in carbons:
        v = Qvector(atom).xyz()
        s = sph % "<{0.x}, {0.y}, {0.z}>".format(v)
        print(s, file=pov)
    for bond in edges:
        v0, v1 = bond
        v0 = Qvector(v0).xyz()
        v1 = Qvector(v1).xyz()
        c = cyl % ("<{0.x}, {0.y}, {0.z}>".format(v0), "<{0.x}, {0.y}, {0.z}>".format(v1))
        print(c, file=pov)
    print("}\n", file=pov)

make_graphene(append=False)


I am working on the file "GrapheneWithQrays.ipynb".
The first task is:
I have elsewhere used this fact to algorithmically generate consecutive shells of 12, 42, 92, 162... spheres (balls) respectively; a growing cuboctahedron of $10 S^{2} + 2$ balls per shell S = 1,2,3... (1 when S=0).

However suppose we don't want to grow the grid omni-directionally, but only in a plane.  Each ball will be surrounded by six neighbors meaning at the center of a hexagon.
The cuboctahedron supplies four such hexagons i.e. its 24 edges comprise four hexagons orbiting the center.  We may use any one of them.
The Algorithm
The algorithm begins with a planar subset of the vectors {2, 1, 1, 0} used to compute the six vertexes surrounding (0,0,0,0).  We may call these six vertexes "carbons".
Then hop to neighboring hexagon centers (where no carbon is located) using an additional set of vectors.
From these new centers, compute the six surrounding carbons again, some of which will have already been found, as neighbors share fences, with three faces (centers) sharing each fence post (carbon).
Using the Python set object, the algorithm filters to keep only unique carbons.
Keep track of hexagon centers, a dual mesh, in a separate set.
(0,0,0,0) will be the first center (ring0).
If qrays r, s are 60 degrees apart on the same hexagon, pointing to neighboring carbons, then r + s will be the "hop" vector over the fence (edge) to the neighboring "yard" (face), or center.
Once we have six vertex vectors from a center, computing the six hop vectors (for jumping over the fences) will be a matter of summing pairs of adjacent (60 degree separated) vectors.  We only keep new centers i.e. those of the next ring (see below).
What about edges?
As we go around a hexagon in 60 degree increments, say in a clockwise direction, we will be finding edges in terms of adjacent ball pairs.
To avoid redundancy, (ball_a, ball_b) -- any edge -- will be sorted.  Any two quadrays may be ordered as 4-tuples e.g. (2, 1, 1, 0) is "greater than" (2, 1, 0, 1).
With unique representations of any edge, in the form of sorted...
Can you write Python code for it?

from qrays import Qvector, IVM

A, B, C, D = Qvector((1,0,0,0)), Qvector((0,1,0,0)), Qvector((0,0,1,0)), Qvector((0,0,0,1))
E,F,G,H = B+C+D, A+C+D, A+B+D, A+B+C
I,J,K,L,M,N = A+B, A+C, A+D, B+C, B+D, C+D
O,P,Q,R,S,T = I+J, I+K, I+L, I+M, N+J, N+K; U,V,W,X,Y,Z = N+L, N+M, J+L, L+M, M+K, K+J

# two "beacons" of six spokes
hexrays = [T, Z, O, Q, X, V]              # to surrounding carbon atoms
hoprays = [T+Z, Z+O, O+Q, Q+X, X+V, V+T]  # to neighboring (vacant) hex centers

(T.angle(Z), Z.angle(O), O.angle(Q), Q.angle(X), X.angle(V), V.angle(T))


I am working on the file "GrapheneWithQrays.ipynb".
The first task is:
I will use this as a check as the algorithm generates multiple rings.
Can you write Python code for it?

centers = {IVM(0,0,0,0)}  # center face
edges   = set()           # no duplicates permitted
carbons = set()

ring0 = [Qvector((0,0,0,0))]

def next_ring(ring):
    """
    Use only the most recently added hexagonal ring
    of face centers to compute the next ring, moving
    outward: 1, 6, 12, 18, 24...
    """
    new_faces = []
    for face in ring:
        verts = []

        # CARBONS
        for spoke in hexrays:
            v = face + spoke
            carbons.add(v.coords) # just the namedtuple is added to the set
            verts.append(v)

        # EDGES
        for bond in zip(verts, verts[1:] + [verts[0]]):
            # adding carbon-to-carbon bonds if not already in the set
            edge = tuple(sorted([bond[0].coords, bond[1].coords]))
            edges.add(edge)

        # CENTERS
        for jump in hoprays:
            neighbor = face + jump
            previous = len(centers)
            centers.add(neighbor.coords)
            if len(centers) > previous:  # if True, face is new
                new_faces.append(neighbor)

    return new_faces

def rings(n):
    prev = ring0
    for ring in range(n):
        print("Ring: {:3}  Number: {:4}".format(ring, len(prev)))
        nxt = next_ring(prev)
        prev = nxt

rings(12)


I am working on the file "GrapheneWithQrays.ipynb".
The first task is:
Looks like we're in business!
As with the growing cuboctahedron and the CCP packing, it makes sense to think in terms of consecutive rings.
The hexagonal coordination sequence is generated by:
Can you write Python code for it?

def A008458(n):
    # OEIS number
    if n == 0:
        return 1
    return 6 * n

[A008458(x) for x in range(10)]


I am working on the file "Graphical Analysis.ipynb".
The first task is:
Error bars graph (raw scores)
In this graph, you will find the generations on the x-axis and the raw scores on the y-axis. The green vertical bars represents the maximum and the minimum raw scores of the current population at generation indicated in the x-axis. The blue line between them is the average raw score of the population.
Can you write Python code for it?


I am working on the file "Graphical Analysis.ipynb".
The first task is:
Min/max difference graph, raw and fitness scores
In this graph, we have two subplots, the first is the difference between the best individual raw score and the worst individual raw score. The second graph shows the difference between the best individual fitness score and the worst individual fitness score. Both subplots show the generation on the x-axis and the score difference in the y-axis.
Can you write Python code for it?


I am working on the file "Graphical Analysis.ipynb".
The first task is:
When you run your GA, all the statistics will be dumped to this database. When you use the graph tool, it will read the statistics from this database file.
Let's evolve the example. Now, instead of evolving step by step, we will set a number of generations for completing the evolution with a single call to <tt>ga.evolve</tt>.
Can you write Python code for it?

ga = GSimpleGA.GSimpleGA(genome)
ga.setDBAdapter(sqlite_adapter)
ga.setGenerations(20)
ga.evolve(freq_stats=5)
print("Generation: %d" % ga.currentGeneration)
best = ga.bestIndividual()
print('\tBest individual: %s' % str(best.genomeList))
print('\tBest score: %.0f' % best.score)


I am working on the file "Graphing Equations.ipynb".
The first task is:
Here we're starting to introduce how data may be selected by numeric indexes, yes, but also by labels.
Can you write Python code for it?

df.loc[:,"3rd Power"].head()

df.loc[:10,["Input", "3rd Power"]]  # rows 0-10 inclusive, two columns


I am working on the file "Graphing Equations.ipynb".
The first task is:
Now that we've plotted some data, lets organize the data into a data table, or "data frame" to be more precise.  Pandas is all about the DataFrame object.
Our domain is a 1-D ndarray lets remember.  As such, we may turn it into a Series, which is the one dimensional equivalent in pandas.  DataFrames are 2-dimensional and above.
Can you write Python code for it?

domain.shape

col0 = pd.Series(domain)
col1 = pd.Series(np.power(domain,2))
col2 = pd.Series(x3)
col3 = Poly(domain)

datadict = {"Input":col0, "Parabola":col1, "3rd Power":col2, "Polynomial":col3}
df = pd.DataFrame(datadict, columns = ["Input", "Parabola", "3rd Power", "Polynomial"])


I am working on the file "Graphing Equations.ipynb".
The first task is:
Without the columns argument, there's no guarantee that datadict will gives us the left-to-right column order we desire.
Can you write Python code for it?


I am working on the file "Greenberg-Hastings examples.ipynb".
The first task is:
Again we can still use spacetime_animation, this time if we want to watch that specific evolution again. Since the .animate() method animates on the fly, it is typically slower.
Can you write Python code for it?

spacetime_animation(spirals.get_spacetime(), colors = plt.cm.rainbow)


I am working on the file "Greenberg-Hastings examples.ipynb".
The first task is:
Example code for using Greenberg-Hastings CA module
See documenation in Greenberg_Hastings.py for details on the Greenberg-Hastings model.
Classic spiral wave behavior
Can you write Python code for it?

#Set model parameters

#square neighborhood radius
R = 3
#number of states / colors
K = 8
#excitation threshhold
T = 6

#initial conditions on 300x300 lattice
initial = random_state_v2((300,300), K, 2.0/9)

#initialize CA object with chosen parameters and initial condition
spirals = greenberg_hastings(initial, R, K, T)
#evolve the CA for 300 time steps
spirals.evolve(300)


I am working on the file "Greenberg-Hastings examples.ipynb".
The first task is:
Global oscillations induced by stochastic driving
Start with very low driving, just see random sprinkling of excitations, no oscillations
Driving strength: 0.02
Can you write Python code for it?

R = 1
K = 10
T = 2
initial = random_state_v1((200, 200), 0.5)
example2 = greenberg_hastings(initial, R, K, T)

example2.drive(200, 0.02)

spacetime_animation(example2.get_spacetime(), colors = plt.cm.rainbow)


I am working on the file "Greenberg-Hastings examples.ipynb".
The first task is:
Some phenomonolgy: exploring parameter space and adding stochastic driving
Some fun travelling particles and collisions
Can you write Python code for it?

R = 1
K = 4
T = 2
initial = random_state_v1((250, 250), 0.2)
example1 = greenberg_hastings(initial, R, K, T)

example1.evolve(300)

spacetime_animation(example1.get_spacetime(), colors = plt.cm.rainbow)


I am working on the file "Greenberg-Hastings examples.ipynb".
The first task is:
You can also animate on the fly using the .animate() method. We run the same code again, but use .animate(duration = 300) instead of .evolve(300). Both methods produce a 300x300x300 spacetime field that can be visualized with spacetime_animation.
Can you write Python code for it?

#Set model parameters

#square neighborhood radius
R = 3
#number of states / colors
K = 8
#excitation threshhold
T = 6

#initial conditions on 300x300 lattice
initial = random_state_v2((300,300), K, 2.0/9)

#initialize CA object with chosen parameters and initial condition
spirals = greenberg_hastings(initial, R, K, T)
#evolve the CA for 300 time steps
spirals.animate(time = 300)


I am working on the file "Grid Searches for Hyper Parameters.ipynb".
The first task is:
A GridSearchCV object behaves just like a normal classifier.
Can you write Python code for it?

grid_search.fit(X_train, y_train)

grid_search.predict(X_test)

grid_search.score(X_test, y_test)

grid_search.best_params_

# We extract just the scores

scores = [x.mean_validation_score for x in grid_search.grid_scores_]
scores = np.array(scores).reshape(6, 5)

plt.matshow(scores)
plt.xlabel('gamma')
plt.ylabel('C')
plt.colorbar()
plt.xticks(np.arange(5), param_grid['gamma'])
plt.yticks(np.arange(6), param_grid['C']);


grid_search.fit(X_train, y_train)

grid_search.predict(X_test)

grid_search.score(X_test, y_test)

grid_search.best_params_

# We extract just the scores
%matplotlib notebook
import matplotlib.pyplot as plt

scores = [x[1] for x in grid_search.grid_scores_]
scores = np.array(scores).reshape(6, 5)

plt.matshow(scores)
plt.xlabel('gamma')
plt.ylabel('C')
plt.colorbar()
plt.xticks(np.arange(5), param_grid['gamma'])
plt.yticks(np.arange(6), param_grid['C']);


I am working on the file "Grid Searches for Hyper Parameters.ipynb".
The first task is:
Define parameter grid:
Can you write Python code for it?

import numpy as np

param_grid = {'C': 10. ** np.arange(-3, 3),
              'gamma' : 10. ** np.arange(-5, 0)}

np.set_printoptions(suppress=True)
print(param_grid)

grid_search = GridSearchCV(SVC(), param_grid, verbose=3)


import numpy as np

param_grid = {'C': 10. ** np.arange(-3, 3),
              'gamma' : 10. ** np.arange(-5, 0)}

np.set_printoptions(suppress=True)
print(param_grid)

grid_search = GridSearchCV(SVC(), param_grid, verbose=3, cv=5)


I am working on the file "Group classification/Group classification.ipynb".
The first task is:
From the figures above, it is difficult to determine an optimal number of clusters. The silhouette score clearly shows that we need more than 5 clusters. 6 looks like a good number, but one of the clusters shows odd behavior when plotting generation change vs net demand change (looks like 2 solid regression lines in the figure).
Can you write Python code for it?

cluster_labels = cluster.export_cluster_id(k=16)
cluster_labels[:5]

labeled_plants = cluster.label_and_export(k=16)

labeled_plants.to_clipboard()

export_df = fossil_with_ramp.loc[:,['year', 'plant_id', 'cluster_id_6']]

export_df.to_csv('Cluster labels.csv', index = False)


I am working on the file "Group classification/Group classification.ipynb".
The first task is:
Start classifying only with all nan values dropped
Can you write Python code for it?

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_samples, silhouette_score, calinski_harabaz_score
from sklearn import preprocessing

fossil_with_ramp.columns

sns.pairplot(fossil_with_ramp, hue='fuel_type', vars=[u'capacity', u'capacity_factor', u'efficiency',
       u'ramp_rate'])


I am working on the file "Group classification/Group classification.ipynb".
The first task is:
The code below begins by scaling the X-vector data to mean 0 and a standard variance. It then loops through a wide range of k values for the number of clusters (3 to 14), calculating the labels for each plant, the Calinski Harabaz score, and the Silhouette score for each value of k.
Can you write Python code for it?

cluster_data = pd.DataFrame(index=range(3,15), columns=['n_clusters', 'score', 'silhouette'])
cluster_labels = {}

X = fossil_with_ramp[['capacity', 'capacity_factor', 'efficiency', 'ramp_rate']]
X_scaled = preprocessing.StandardScaler().fit_transform(X)

for idx, n_clusters in enumerate(range(3,15)):
    cluster_data.loc[n_clusters, 'n_clusters'] = n_clusters
    clusterer = KMeans(n_clusters, random_state=42)

    # fit_clusters = clusterer.fit(X)
#     cluster_labels = clusterer.fit_predict(X)
    cluster_labels[n_clusters] = clusterer.fit_predict(X_scaled)

    # http://scikit-learn.org/stable/modules/clustering.html#calinski-harabaz-index
    # The score is higher when clusters are dense and well separated
#     score[idx] = metrics.calinski_harabaz_score(X, cluster_labels)
    cluster_data.loc[n_clusters, 'score'] = calinski_harabaz_score(X_scaled, cluster_labels[n_clusters])

#     silhouette[idx] = silhouette_score(X, cluster_labels)
    cluster_data.loc[n_clusters, 'silhouette'] = silhouette_score(X_scaled, cluster_labels[n_clusters])

#     print 'For ', n_clusters, ' clusters, the average silhouette score is :', silhouette[idx], \
#         ' and the score is :', score[idx]


I am working on the file "Group classification/Group classification.ipynb".
The first task is:
The two score values are plotted below. They don't agree exactly on the optimal number of centers, but it looks like 6 or 8 is probably best.
Can you write Python code for it?

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(8,3))
cluster_data.plot(y='score', ax=ax1)
# ax1.plot(range(3,15), cluster_data['score'])
ax1.set_title('Calinski Harabaz score\nHigher is better')
cluster_data.plot(y='silhouette', ax=ax2)
# ax2.plot(range(3,15), silhouette)
ax2.set_title('Silhouette score\nLower is better')


I am working on the file "Group classification/Group classification.ipynb".
The first task is:
missing '1-hr ramp rate'
Can you write Python code for it?

cluster_df[cluster_df.plant_id==127]

cluster_df[cluster_df.plant_id==3466]

cluster_df[cluster_df.plant_id==3584]

cluster_df['fuel_type'].unique()


I am working on the file "GroupGensimAndTextblob.ipynb".
The first task is:
See the data
Can you write Python code for it?

all_ranked_dfs = []
all_similarity_dfs = []
for ranked_sentences, similarity_mtrx in zip(all_ranked_sentences, all_matrxs):
    df_ranked_sentences = pd.DataFrame.from_dict(ranked_sentences,
                                                 orient='index')
    df_similarity_mtrx = pd.DataFrame(similarity_mtrx)
    all_ranked_dfs.append(df_ranked_sentences)
    all_similarity_dfs.append(df_similarity_mtrx)

all_similarity_dfs[0]


I am working on the file "GroupGensimAndTextblob.ipynb".
The first task is:
Skip the clustering for grouping, use comments and the article as groups
Can you write Python code for it?

from gensim import corpora, models, similarities
from nltk.tokenize import TweetTokenizer
import nltk.stem

def preprocess_docs(documents):
    tokenizer = TweetTokenizer()
    english_stemmer = nltk.stem.SnowballStemmer('english')

    texts = [tokenizer.tokenize(d) for d in documents]

    stemmed_texts = []
    for text in texts:
        stemmed_text = [english_stemmer.stem(t) for t in text]
        stemmed_texts.append(stemmed_text)
    return stemmed_texts

def query_all(lsi_queries, index):
    total_pairs = 0
    selected_pairs = 0
    strong_sentences = 0
    similarity_dict = {}
    for i, query in enumerate(lsi_queries):
        sims = index[query]
        strong_sims = [s for s in list(enumerate(sims)) if s[1] > 0.99]
        pairs = len(strong_sims)
        if pairs > 1 and pairs < 8:
#             print "\ns{0} similarities with comments".format(i+comment_start_index)
#             print "{0} strong similarity pairs".format(len(strong_sims))
#             print all_comments_sentences[0][i]
            strong_sentences += 1
            selected_pairs += pairs
            similarity_dict[i] = strong_sims

        total_pairs += pairs
    print "\n{0} strong sentences".format(strong_sentences)
    print "{0} total sentence-group pairs".format(selected_pairs)
    print total_pairs/float(len(lsi_queries))
    return similarity_dict

def find_links_between_in(documents, comments_sentences,
                          article_sentences):
    stemmed_texts = preprocess_docs(documents)
    dictionary = corpora.Dictionary(stemmed_texts)
    dictionary.filter_extremes(no_below=1, no_above=0.8)
    corpus = [dictionary.doc2bow(text) for text in stemmed_texts]

    lsi = models.LsiModel(corpus, id2word=dictionary, num_topics=2)
    index = similarities.MatrixSimilarity(lsi[corpus])

#     comment_start_index = len(article_sentences)
    stemmed_queries = preprocess_docs(comments_sentences)
    query_dict = corpora.Dictionary(stemmed_queries)
    lsi_queries = [lsi[query_dict.doc2bow(text)] for text in...


I am working on the file "GroupGensimAndTextblob.ipynb".
The first task is:
Vectorize sentences just to check if it is better if I do this myself other than lexrank
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer
import nltk.stem

english_stemmer = nltk.stem.SnowballStemmer('english')

class StemmedTfidfVectorizer(TfidfVectorizer):
    def build_analyzer(self):
        analyzer=super(StemmedTfidfVectorizer,self).build_analyzer()
        return lambda doc:(english_stemmer.stem(w) for w in analyzer(doc))

vectorizer = StemmedTfidfVectorizer(min_df=1, stop_words='english')

article_s =  all_article_sentences
comment_s = all_comments_sentences
for i, (article, comments) in enumerate(zip(article_s, comment_s)):
    X_article = vectorizer.fit_transform(article)
    X_comments = vectorizer.fit_transform(comments)
    db_a = DBSCAN(eps=0.3, min_samples=2).fit(X_article)
    db_c = DBSCAN(eps=0.3, min_samples=2).fit(X_comments)

    print "\nARTICLE {0}".format(i)
    print_results(db_a)
    print "\nCOMMENTS {0}".format(i)
    print_results(db_c)


I am working on the file "GroupHW_1_Exposure_ForwardBond.ipynb".
The first task is:
Create Libor and portfolioScheduleOfCF.  This datelist contains all dates
to be used in any calculation of the portfolio positions.
BondCoupon class has to have a method getScheduleComplete, which return
fullSet on [0] and datelist on [1], calculated by BondCoupon as:
def getScheduleComplete(self):
 self.datelist=self.myScheduler.getSchedule(start=self.start,end=self.maturity,freq=self.freq,referencedate=self.referencedate)
    self.ntimes = len(self.datelist)
    fullset = sorted(set(self.datelist)
                               .union([self.referencedate])
                               .union([self.start])
                               .union([self.maturity])
                               )
    return fullset,self.datelist
portfolioScheduleOfCF is the concatenation of all fullsets. It defines the set of all dates for which Libor should be known.
Can you write Python code for it?

portfolioScheduleOfCF = set(ReferenceDateList)
for i in range(len(myPortfolio)):
    portfolioScheduleOfCF=portfolioScheduleOfCF.union(myPortfolio[i].getScheduleComplete()[0]
)
portfolioScheduleOfCF = sorted(portfolioScheduleOfCF.union(ReferenceDateList))
OIS = myVasicek.getSmallLibor(datelist=portfolioScheduleOfCF)
# at this point OIS contains all dates for which the discount curve should be known.
# If the OIS doesn't contain that date, it would not be able to discount the cashflows and the calcualtion would faill.

print(OIS)

pvs={}
for t in portfolioScheduleOfCF:
    pvs[t] = np.zeros([1,simNumber])
    for i in range(len(myPortfolio)):
        myPortfolio[i].setLibor(OIS)
        pvs[t] = pvs[t] + myPortfolio[i].getExposure(referencedate=t).values

#print(portfolioScheduleOfCF)
#print(pvs)
pvsPlot = pd.DataFrame.from_dict(list(pvs.items()))
pvsPlot.index= list(pvs.keys())
pvs1={}
for i,t in zip(pvsPlot.values,pvsPlot.index):
    pvs1[t]=i[1][0]
pvs = pd.DataFrame.from_dict(data=pvs1,orient="index")
ax=pvs.plot(legend=False)
ax.set_xlabel("Year")
ax.set_ylabel("Coupon Bond Exposure")


I am working on the file "GroupHW_1_Exposure_ForwardBond.ipynb".
The first task is:
Create forward bond future PV (Exposure) time profile
Setting up parameters
Can you write Python code for it?

t_step = 1.0 / 365.0
simNumber = 10
trim_start = date(2005,3,10)
trim_end = date(2010,12,31)  # Last Date of the Portfolio
start = date(2005, 3, 10)
referenceDate = date(2005, 5, 10)


t_step = 1.0 / 365.0
simNumber = 10
trim_start = date(2005,3,10)
trim_end = date(2010,12,31)  # Last Date of the Portfolio
start = date(2005, 3, 10)
referenceDate = date(2005, 3, 10)


I am working on the file "GroupMeeting_11_16.ipynb".
The first task is:
Results
Can you write Python code for it?

from pandas import read_csv

res = read_csv('data3.csv')
tru = read_csv('true3.csv')

start = min([res[res[c] > 0][c].min() for c in res.columns[1:-1]])
stop = res.max().max()

base = 10
start = log(start, base)
end = log(stop, base)
res_logspace = np.logspace(start, end, num=10, base=base)

plt.rcParams['figure.figsize'] = (20, 12)

for i,val in enumerate(tru.columns[1:]):
    plt.subplot(int('91' + str(i+1)))
    x = res[val][res[val] > 0]
    weights = np.exp(res['log-likelihood'][res[val] > 0])
    t = tru[val].loc[0]
    plt.hist(x, bins=res_logspace, alpha=0.4, normed=True, label='prior')
    plt.hist(x, bins=res_logspace, weights=weights, alpha=0.4, normed=True, label='posterior')
    plt.axvline(x=t, color='red', label='truth', linewidth=1)
    plt.xscale('log')
    plt.legend()
    plt.ylabel('PDF')
    plt.xlabel('Halo Mass (log-scale)')
    plt.title('Halo ID ' + val)

plt.show()

res.columns

res[['112009306000027', 'log-likelihood']].sort('log-likelihood')


I am working on the file "Gráfico simple.ipynb".
The first task is:
La función std() calcula la desviación estándar de cada columna
Can you write Python code for it?


I am working on the file "Gráfico simple.ipynb".
The first task is:
Luego, importar los datos
Can you write Python code for it?

df = pd.read_csv("https://raw.githubusercontent.com/plotly/datasets/master/school_earnings.csv")

table = ff.create_table(df)

py.iplot(table, filename='table1')

schools = df.School

print(schools[0],";" ,schools[1])


I am working on the file "Gráfico simple.ipynb".
The first task is:
Luego, se puede generar, por ejemplo, un gráfico de barra
Can you write Python code for it?

data = [Bar(x=df.School,
            y=df.gap)]

py.iplot(data, filename='basic_bar')

trace_women = Bar(x=df.School,
                  y=df.Women,
                  name='Women',
                  marker=dict(color='#ffcdd2'))

trace_men = Bar(x=df.School,
                y=df.Men,
                name='Men',
                marker=dict(color='#A2D5F2'))

trace_gap = Bar(x=df.School,
                y=df.gap,
                name='Gap',
                marker=dict(color='#59606D'))

data = [trace_women, trace_men, trace_gap]
layout = Layout(title="Average Earnings for Graduates",
                xaxis=dict(title='School'),
                yaxis=dict(title='Salary (in thousands)'))
fig = Figure(data=data, layout=layout)

py.iplot(fig, filename='simple_styled_bar')


I am working on the file "Guerilla Archiving RadNet data.ipynb".
The first task is:
Before we go further, we'll want to know the size of each table:
Can you write Python code for it?

def table_count(tablename):
    url= "https://iaspub.epa.gov/enviro/efservice/{tablename}/COUNT/JSON".format_map(locals())
    print(url)
    return requests.get(url).json()[0]['TOTALQUERYRESULTS']

table_count('erm_project')

table_count('t_design_for_environment')


I am working on the file "Guerilla Archiving RadNet data.ipynb".
The first task is:
Example REST query:
Can you write Python code for it?


I am working on the file "Guerilla Archiving RadNet data.ipynb".
The first task is:
I can use this to extract the table data:
Can you write Python code for it?

def makeurl(tablename,start,end):
    return "https://iaspub.epa.gov/enviro/efservice/{tablename}/JSON/rows/{start}:{end}".format_map(locals())

url=makeurl( 't_design_for_environment',1,19)
out=requests.get(url)

pandas.DataFrame(out.json())


I am working on the file "Guerilla Archiving RadNet data.ipynb".
The first task is:
Let's compare this with our previous method
Can you write Python code for it?

erm_analyte_=table_data('ERM_ANALYTE')

erm_analyte.head()

erm_analyte_.head()


I am working on the file "Guerilla Archiving RadNet data.ipynb".
The first task is:
Performance Profiling.
Are there limits to how much data we can pull per request?  How fast we can retrieve it?
Let's try a reasonably big table: ERM_ANALYSIS
Can you write Python code for it?

table_count('ERM_ANALYSIS')

%%time

erm_analysis = table_data('ERM_ANALYSIS',0,10)
print(len(erm_analysis),"rows retrieved")

erm_analysis.head()

%%time

erm_analysis = table_data('ERM_ANALYSIS',0,100)
print(len(erm_analysis),"rows retrieved")

%%time

erm_analysis = table_data('ERM_ANALYSIS',0,200)
print(len(erm_analysis),"rows retrieved")

%%time

erm_analysis = table_data('ERM_ANALYSIS',0,400)
print(len(erm_analysis),"rows retrieved")

%%time

erm_analysis = table_data('ERM_ANALYSIS',0,800)
print(len(erm_analysis),"rows retrieved")

%%time

erm_analysis = table_data('ERM_ANALYSIS',0,1000)
print(len(erm_analysis),"rows retrieved")

%%time

erm_analysis = table_data('ERM_ANALYSIS',0,500)
print(len(erm_analysis),"rows retrieved")

%%time

erm_analysis = table_data('ERM_ANALYSIS',0,600)
print(len(erm_analysis),"rows retrieved")


I am working on the file "Guerilla Archiving RadNet data.ipynb".
The first task is:
So retrieval time seems to go up significantly once we go past 500 rows.
So let's write an iterative retrieval function
Can you write Python code for it?

def collect(tablename, rowcount=100, limit = 1000):
    '''
        The API understands start/end to be INCLUSIVE
    '''
    count   =table_count(tablename)
    def inner():
        start   =0
        end     = rowcount-1
        while start <=count:
            end=min(end,limit-1)
            url = makeurl(tablename,start,end)
            print(url)
            yield requests.get(url).json()

            start+= rowcount
            end  += rowcount
            if start>limit-1:return
    return pandas.DataFrame(list(chain.from_iterable(inner())))

erm_analyte=collect('ERM_ANALYTE',rowcount=20,limit=35)

len(erm_analyte)


I am working on the file "Guided+example+and+Challenge.ipynb".
The first task is:
Gradient boost guided example
Having walked through gradient boost by hand, now let's try it with SKlearn.  We'll still use the European Social Survey Data, but now with a categorical outcome: Whether or not someone lives with a partner.
Can you write Python code for it?

df = pd.read_csv((
    "https://raw.githubusercontent.com/Thinkful-Ed/data-201-resources/"
    "master/ESS_practice_data/ESSdata_Thinkful.csv")).dropna()

# Define outcome and predictors.
# Set our outcome to 0 and 1.
y = df['partner'] - 1
X = df.loc[:, ~df.columns.isin(['partner', 'cntry', 'idno'])]

# Make the categorical variable 'country' into dummies.
X = pd.concat([X, pd.get_dummies(df['cntry'])], axis=1)
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.10)
# As originally designed, this dataset trains on non-Estonians and tests on Estonians in an imbalanced way.
# We may have better results doing a cross-validation.
# Create training and test sets.
#offset = int(X.shape[0] * 0.9)

# Put 90% of the data in the training set.
#X_train, y_train = X[:offset], y[:offset]

# And put 10% in the test set.
#X_test, y_test = X[offset:], y[offset:]