I am working on the file "Jupyter/2017-09-06-another-keras-tutorial-for-neural-network-beginners.ipynb".
The first task is:
You may also want to visualise how the model performed on the test set over time. Remember that we saved the model weights to a file at each epoch.  This means we can reproduce each intermediate model by loading the weights into an appropriately constructed (i.e. same number of layers/neurons) neural network.
Can you write Python code for it?

temp_test_model = Sequential() # create model
temp_test_model.add(Dense(5, input_dim=8, activation='relu')) # hidden layer
temp_test_model.add(Dense(1, activation='sigmoid')) # output layer
temp_test_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
test_over_time = []
for i in range(len(my_first_nn_fitted.history['acc'])):
    temp_test_model.load_weights("nn_weights-%02d.hdf5" % i)
    scores = temp_test_model.evaluate(X_test, Y_test, verbose=0)
    # 0 is loss; 1 is accuracy
    test_over_time.append(scores)

fig, (ax1, ax2) = plt.subplots(2, 1)
ax1.plot(range(len(my_first_nn_fitted.history['loss'])), my_first_nn_fitted.history['loss'],linestyle='-', color='blue',label='Training', lw=2)
ax1.plot(range(len(np.array(test_over_time)[:,0])), np.array(test_over_time)[:,0], linestyle='-', color='green',label='Test', lw=2)
ax2.plot(range(len(my_first_nn_fitted.history['acc'])), my_first_nn_fitted.history['acc'],linestyle='-', color='blue',label='Training', lw=2)
ax2.plot(range(len(np.array(test_over_time)[:,1])), np.array(test_over_time)[:,1], linestyle='-', color='green',label='Test', lw=2)
leg = ax1.legend(bbox_to_anchor=(0.7, 0.9), loc=2, borderaxespad=0.,fontsize=13)
ax1.set_xticklabels('')
#ax1.set_yscale('log')
ax2.set_xlabel('# Epochs',fontsize=14)
ax1.set_ylabel('Loss',fontsize=14)
ax2.set_ylabel('Accuracy',fontsize=14)
plt.show()


I am working on the file "Jupyter/2017-09-06-another-keras-tutorial-for-neural-network-beginners.ipynb".
The first task is:
You'll notice a gradual reduction in the training error, without any corresponding improvement in the test prediction (it actually appears to get worse). This represents textbook overfitting. Our neural network model has fine-tuned its parameters to the training dataset, so that it performs poorly on any unseen data. In other words, the model is highly specialised to the training set and doesn't generalise well beyond the training data. It's analogous to including more complex interaction terms in a simple regression task (also known as the bias-variance tradeoff). A good analogy is a model that's trained to detect a specific species of bird. The more it trains, its ability to detect other bird species worsens.
You may also notice that we didn't achieve our target of 95 % accuracy on the training set. It's possible that we would eventually hit this target if the model was allowed more time to improve itself. As the model has 9000 parameters (8x1000+ 1x1000), it can take many epochs to find every marginal gain. We could also decide to play around with the learning rate. But the model possesses an inherent flaw. The data was not scaled and/or normalised. Scaling/normalising the data traditionally involves linear transformation to force all values within a specific interval (e.g. between 0 and 1), while standardising typically transforms the inputs to have the mean centred on zero with unit variance. Either way, no input feature dominates the model and this forces consistent weights. For example, 5000 x 0.0002 = 2 x 0.5, so that weights differ by several orders of magnitude, which could inhibit backpropagation.
Scaling/normalising is easy; we can just employ some tools from scikit-learn. Note that you should scale/normalise the training and test sets separately. Otherwise, you'd be using information from your test set to inform your model training (if in doubt, think what would Eminem do?. We'll opt for the unit variance with the mean centred on 0 approach.
Can you write Python code for it?

scaler = StandardScaler()
nn_output_unscaled = runNN(X_train, Y_train, X_test, Y_test, 1000, 1000)
nn_output_scaled = runNN(scaler.fit_transform(X_train), Y_train,
                         scaler.fit_transform(X_test), Y_test, 1000, 1000)

fig, (ax1, ax2) = plt.subplots(2, 1)
ax1.plot(range(len(nn_output_unscaled['loss'])), nn_output_unscaled['loss'], linestyle='-', color='blue',label='Training', lw=2)
ax1.plot(range(len(nn_output_unscaled['test_loss'])), nn_output_unscaled['test_loss'], linestyle='-', color='green',label='Test', lw=2)
ax1.plot(range(len(nn_output_scaled['loss'])), nn_output_scaled['loss'], linestyle='-', color='deepskyblue',label='Training (Standardised)', lw=2)
ax1.plot(range(len(nn_output_scaled['test_loss'])), nn_output_scaled['test_loss'], linestyle='-', color='lightgreen',label='Test (Standardised)', lw=2)
ax2.plot(range(len(nn_output_unscaled['acc'])), nn_output_unscaled['acc'], linestyle='-', color='blue',label='Training', lw=2)
ax2.plot(range(len(nn_output_unscaled['test_acc'])), nn_output_unscaled['test_acc'], linestyle='-', color='green',label='Test', lw=2)
ax2.plot(range(len(nn_output_scaled['acc'])), nn_output_scaled['acc'], linestyle='-', color='deepskyblue',label='Training (Standardised)', lw=2)
ax2.plot(range(len(nn_output_scaled['test_acc'])), nn_output_scaled['test_acc'], linestyle='-', color='lightgreen',label='Test (Standardised)', lw=2)
leg = ax1.legend(bbox_to_anchor=(0.6, 0.9), loc=2, borderaxespad=0.,fontsize=13)
ax1.set_xticklabels('')
ax2.set_xlabel('# Epochs',fontsize=14)
ax1.set_ylabel('Loss',fontsize=14)
ax2.set_ylabel('Accuracy',fontsize=14)
ax2.annotate('Overfitting starts', xy=(80, 0.8), xytext=(100, 0.5),fontsize=14,
            arrowprops=dict(facecolor='black', shrink=0.0,headwidth=10))
plt.tight_layout()
plt.show()

nn_output_scaled['acc'][-1]


I am working on the file "Jupyter/2017-12-19-charting-the-rise-of-song-collaborations-with-scrapy-and-pandas.ipynb".
The first task is:
Briefly explaining what happened there: We create a class called ChartSpider, essentially our customised spider (called ukChartSpider). We specify the page we want to scrape (start_urls). The spider then selects specific CSS elements (response.css()) within the page that contain the information we want (e.g. #main .artist a represents the artist's name). These tags may seem complicated, but they're actually quite easy to retrieve with a tool like Selector Gadget. Isolate the elements you want to extract and copy the css elements highlighted with the tool (see image below).

Finally, we'll opt to write the spider output to a json file called uk_charts.json. Scrapy accepts numerous file formats (including CSV), but I went with JSON as it's easier to append to this file type, which may be useful if your spider unexpectedly terminates.  We're now ready to launch ukChartSpider. Note that the process for the US Billboard chart is very similar. That code can be found in the accompanying Jupyter notebook.
Can you write Python code for it?

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
'FEED_FORMAT': 'json',
'FEED_URI': 'uk_charts.json'
})

# minimising the information presented on the scrapy log
logging.getLogger('scrapy').setLevel(logging.WARNING)
process.crawl(ChartSpider)
process.start()


I am working on the file "Jupyter/2017-12-19-charting-the-rise-of-song-collaborations-with-scrapy-and-pandas.ipynb".
The first task is:
In the 1960s, 70s and 80s, colloborations were relatively rare (~5% of charted singles) and generally took the form of duets. Things changed in the mid 90s, when the number of colloborations increases significantly, with duets dying off and featured artists taking over. I blame rap music. Comparing the two charts, the UK and US prefer 'ft' and 'featuring', repsectively (two nations divided by a common language). The Billboard chart doesn't seem to like the '/' notation, while the UK is generally much more eclectic.
Finally, we can plot the proportion of songs that were collobarations (satisfied any of these conditions).
Can you write Python code for it?

fig, ax1 = plt.subplots(1,1)
uk_charts.groupby(['chart_week'])['collab'].mean().plot(ax=ax1, color='#F38181')
us_charts.groupby(['chart_week'])['collab'].mean().plot(ax=ax1, color='#756C83')
ax1.set_xlabel('')
ax1.set_ylabel('Collaborative Songs (%)')
ax1.set_yticklabels(['{:3.0f}'.format(x*100) for x in ax1.get_yticks()])
ax1.set_xlim([datetime.date(1952,10,1), datetime.date(2018,1,1)])
fig.set_size_inches(9, 3.5)
ax1.legend(["UK Singles Chart", "Billboard Hot 100"],
           bbox_to_anchor=(0.07, 1), loc=2, borderaxespad=0., prop={'size': 12})
fig.tight_layout()
plt.show()


I am working on the file "Jupyter/2017-12-19-charting-the-rise-of-song-collaborations-with-scrapy-and-pandas.ipynb".
The first task is:
Okay, we've identified various terms that denote collaborations of some form. Not too bad. We just need to count the number of instances where the artist name includes one of these terms. Right? Maybe not.
Can you write Python code for it?

pd.concat((uk_charts[uk_charts['artist'].str.contains('AC/DC')].tail(1),
uk_charts[uk_charts['artist'].str.contains('BOB MARLEY AND')].tail(1),
uk_charts[uk_charts['artist'].str.contains('BOB MARLEY &')].tail(1)))


I am working on the file "Jupyter/2017-12-19-charting-the-rise-of-song-collaborations-with-scrapy-and-pandas.ipynb".
The first task is:
That table shows the top 5 singles in the UK for week starting 8st December 2017. I think I recognise two of those songs. As we're interested in collaborations, you'll notice that we have a few in this top 5 alone, which are marked with an 'FT' in the artist name. Unfortunately, there's no consistent nomenclature to denote collaborations on the UK singles chart (the Billboard chart isn't as bad).
Can you write Python code for it?

pd.concat((uk_charts[uk_charts['artist'].str.contains(' FEAT\\.')][0:1],
        uk_charts[uk_charts['artist'].str.contains(' FEATURING ')][0:1],
        uk_charts[uk_charts['artist'].str.contains(' FEAT ')][0:1],
        uk_charts[uk_charts['artist'].str.contains('/')][0:1],
        uk_charts[uk_charts['artist'].str.contains(' AND ')][0:1],
        uk_charts[uk_charts['artist'].str.contains(' & ')][0:1],
        uk_charts[uk_charts['artist'].str.contains(' WITH ')][0:1],
        uk_charts[uk_charts['artist'].str.contains(' VS ')][0:1],
        uk_charts[uk_charts['artist'].str.contains(' VS. ')][0:1]))


I am working on the file "Jupyter/2018-01-10-home-advantage-in-football-leagues-around-the-world.ipynb".
The first task is:
Honestly, there's not much going on there. With the poissble exception of the Spanish La Liga since 2010, the home field advantage enjoyed by the teams in each league is broadly similar (and that's before we bring in the idea of confidence intervals and hypothesis testing).
Home Advantage Around the World
To find more interesting contrasts, we must venture to crappier and more corrupt leagues. My hunch is that home advantage would be negligible in countries where the overall quality (team, infastructure, etc.) is very low. And by low, I mean leagues worse than the Irish Premier Division (yes, they exist). Unfortunately, the historical results for such leagues are not available on football-data.co.uk. Instead, we'll scrape the data off betexplorer. I'm extremely impressed by the breadth of this site. You can even retrieve past results for the French overseas department of Réunion. Fun fact: Dimtri Payet spent the 2004 season at AS Excelsior of the Réunion Premier League.
We'll use Scrapy to pull the appropriate information off the website. If you've never used Scrapy before, then you should check out this post. I won't spend too long on this part, but you can find the full code on here.
Can you write Python code for it?

import scrapy
import re # for text parsing
import logging
import pandas as pd

class FootballSpider(scrapy.Spider):
    name = 'footballSpider'
    # page to scrape
    start_urls = pd.read_csv(
        'https://raw.githubusercontent.com/dashee87/blogScripts/master/files/league_links.csv')['link'].tolist() + \
    pd.read_csv("league_links_.csv")['link2'].dropna(axis=0, how='all').tolist()
    # if you want to impose a delay between sucessive scrapes
#   download_delay = 1.0

    def parse(self, response):
        self.logger.info('Scraping page: %s', response.url)
        country_league = response.css('.list-breadcrumb__item__in::text').extract()

        for num, (hometeam, awayteam, match_result, date) in \
            enumerate(zip(response.css('.in-match span:nth-child(1)'),
                                 response.css('.in-match span:nth-child(2)'),
                                 response.css('td.h-text-center'),
                                 response.css('.h-text-no-wrap::text').extract())):
            yield {'country':country_league[2], 'league': country_league[3],
                   'HomeTeam': hometeam.css('::text').extract_first(),
                   'AwayTeam':awayteam.css('::text').extract_first(),
                   'FTHG':  re.sub(':.*', '', match_result.css('::text').extract_first()),
                   'FTAG':  re.sub('.*:', '', match_result.css('::text').extract_first()),
                   'awarded': ' ' in match_result.css('::text').extract_first() or
                   'AWA.' in match_result.css('::text').extract_first(),
                   'date':date}

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
'FEED_FORMAT': 'json',
'FEED_URI': 'all_league_goals_.json'
})

# minimising the information presented on the scrapy log
logging.getLogger('scrapy').setLevel(logging.WARNING)
process.crawl(FootballSpider)
process.start()


I am working on the file "Jupyter/2018-01-10-home-advantage-in-football-leagues-around-the-world.ipynb".
The first task is:
I've essentially combined various parts of the previous post into one convenient function. If it looks a little strange, then I suggest you consult the original post. Okay, we're ready to start calculating some home advantage scores.
Can you write Python code for it?

# home field advantage for EPL 2016/17 season
get_home_team_advantage(pd.read_csv("http://www.football-data.co.uk/mmz4281/1617/E0.csv"))


I am working on the file "Jupyter/2018-01-10-home-advantage-in-football-leagues-around-the-world.ipynb".
The first task is:
It's as easy as that. Feed a url from football-data.co.uk into the function and it'll quickly tell you the statistical advantage enjoyed by home teams in that league. Note that the latter two values repesent the left and right limit of the 95% confidence interval around the mean value. The first value in the array is actually just the log of the number of goals scored by the home team divided by the total number of away goals.
Can you write Python code for it?


I am working on the file "Jupyter/2018-01-10-home-advantage-in-football-leagues-around-the-world.ipynb".
The first task is:
Shapefiles and TopoJSON
Generating the world map required a little bit of command line, python and a whole lot of JavaScript (specifically D3). The command line was used to convert the shapefiles into geojson files (see ogr2ogr and finally into the topojson format. The main reason for the last step is that it drastically reduces the file size, which should improve its onsite loading (though it could also affect the quality of the map).
My particular map was complicated by the fact that some sovereign states are composed of several countries that organise their own national competitions. If that sounds weird, think of the United Kingdom. It's a member of the UN and a sovereign state in its own right (despite what Brexiteers may say). But there's no UK (or British) Premier League; there's the English/Welsh/Scottish/Northern Irish Premier League/Premiership. Similarly, Reunion is part of France but has its own football league. Then again, the Basque country is recognised as a nation within Spain, but has no internationally recognised national league. In summary, it's complicated.
Political realities aside, we need to get the geojson file for all of the countries in the world (see all.geojson available here). We must remove the United Kingdom, France and a few others. To reduce the file size, I also removed some country information that wasn't relevant for my purposes (population, GDP, etc.). The geojson files for England, Scotland, Reunion, etc. were a little harder to track down. The shapefile containing those country subdivisions can be downloaded here, which can be converted into geojson files with ogr2ogr. Unfortunately, that file contains various subdivisions that don't correspond to actual football leagues (e.g. Belgium is split into the Flemish and Walloon regions). That means we need append the subdivisions we do want to higher level geojson file, which I did my manipulating the two json files in Python.
Can you write Python code for it?

import json

all_nations = json.load(open('all.geojson'))
# command line: ogr2ogr -f GEOJson -where "ADM0_A3 IN ('GBR', 'FRA','NLD', 'USA')" \
# football_subdivisions.json ne_10m_admin_0_map_units.shp
subdivisions = json.load(open('football_subdivisions.json'))

all_nations_tidy = {}
all_nations_tidy['type']= all_nations['type']
all_nations_tidy['crs'] = all_nations['crs']
all_nations_tidy['features'] = []

for country_features in all_nations['features']:
    # skip UK, France, US, Netherlands and Antartica
    if (country_features['properties']['ADM0_A3'] in ['GBR', 'FRA', 'USA', 'NLD', 'ATA'] or
    # skip minor islands and territories with populations less than 1000
    country_features['properties']['POP_EST']<1000) and \
    # don't want to exclude Western Sahara
    country_features['properties']['NAME_LONG'] != 'Western Sahara':
        continue
    if True:
        all_nations_tidy['features'].append({'properties': {'country': country_features['properties']['NAME_LONG'],
                                   'countryCode': country_features['properties']['ADM0_A3']},
                                   'geometry': country_features['geometry']})

for subdiv_features in subdivisions['features']:
    all_nations_tidy['features'].append({'properties': {'country': subdiv_features['properties']['NAME_LONG'],
                               'countryCode': subdiv_features['properties']['BRK_A3']},
                               'geometry': subdiv_features['geometry']})

with open('countries.json', 'w') as outfile:
    json.dump(all_nations_tidy, outfile)


I am working on the file "Jupyter/2018-01-10-home-advantage-in-football-leagues-around-the-world.ipynb".
The first task is:
We're ready to put it all together. I'll omit the code (though it can be found here), but we'll loop through each country and league combination (just in case you decide to include multiple leagues from the same country) and calculate the home advantage score, plus its confidence limits as well as some other information for each league (number of teams, average number of goals in each match). I've converted the pandas output to a datatables table that you can interactively filter and sort.
Can you write Python code for it?

home_advantage_country = pd.DataFrame(all_league_goals.assign(match_goals = all_league_goals['FTHG'] +
                                      all_league_goals['FTHG']).groupby(['country','league']).agg(
        {'HomeTeam':['size','nunique'], 'match_goals':'mean'}).to_records())
home_advantage_country.columns = ['country', 'league', 'num_games', 'num_teams', 'avg_goals']
temp_set = []
for i in range(home_advantage_country.shape[0]):
    temp_set.append(get_home_team_advantage(all_league_goals[(
                    all_league_goals['country']==home_advantage_country['country'][i]) & (
                all_league_goals['league']==home_advantage_country['league'][i])]))
temp_set = pd.DataFrame(temp_set,columns= ['home_advantage_score', 'left_tail', 'right_tail'])
home_advantage_country = pd.concat([home_advantage_country, temp_set], axis=1).sort_values('home_advantage_score',
                                            ascending=False).reset_index(drop=True)
home_advantage_country.index = home_advantage_country.index + 1
# if you want display more/less rows than the default option
pd.options.display.max_rows = 40
home_advantage_country.assign(avg_goals= pd.Series.round(home_advantage_country['avg_goals'], 3),
                             home_advantage_score= pd.Series.round(home_advantage_country['home_advantage_score'], 3),
                             left_tail= pd.Series.round(home_advantage_country['left_tail'], 3),
                             right_tail= pd.Series.round(home_advantage_country['right_tail'], 3))


I am working on the file "Jupyter/2018-09-13-predicting-football-results-with-statistical-modelling-dixon-coles-and-time-weighting.ipynb".
The first task is:
First published by Maher in 1982, the BP model still serves a good starting point from which you can add features that more closely reflect the reality. That brings us onto the Dixon Coles (DC) model.
Dixon-Coles Model
In their 1997 paper, Mark Dixon and Stuart Coles proposed two specific improvements to the BP model:
* Introduce an interaction term to correct underestimated frequency of low scoring matches
* Apply time decay component so that recent matches are weighted more strongly
The authors claim that low score results (0-0, 1-0, 0-1 and 1-1) are inherently under-reported by the BP model. In the paper, they provide some analysis that supports their case- though I wouldn't call their approach particularly rigorous. The matrix below shows the average difference between actual and model predicted scorelines for the 2005/06 season all the way up to the 2017/18 season. Green cells imply the model underestimated those scorelines, while red cells suggest overestimation- the colour strength indicates the level of disagreement.
Can you write Python code for it?

def poiss_actual_diff(football_url, max_goals):
    epl_1718 = pd.read_csv(football_url)
    epl_1718 = epl_1718[['HomeTeam','AwayTeam','FTHG','FTAG']]
    epl_1718 = epl_1718.rename(columns={'FTHG': 'HomeGoals', 'FTAG': 'AwayGoals'})
    team_pred = [[poisson.pmf(i, team_avg) for i in range(0, max_goals)] \
                 for team_avg in [epl_1718['HomeGoals'].mean(), epl_1718['AwayGoals'].mean()]]
    return np.outer(np.array(team_pred[0]), np.array(team_pred[1])) - \
        np.array([sum((epl_1718['HomeGoals']==i) & (epl_1718['AwayGoals']==j))
                  for i in range(max_goals) for j in range(max_goals)]).reshape((6,6))/len(epl_1718)

year_arrays = []
for year in range(2005,2018):
    year_arrays.append(poiss_actual_diff("http://www.football-data.co.uk/mmz4281/{}{}/E0.csv".format(
        str(year)[-2:], str(year+1)[-2:]),6))

cmap = sns.diverging_palette(10, 133, as_cmap=True)

fig, ax = plt.subplots(figsize=(5,5))
with sns.axes_style("white"):
    ax = sns.heatmap(np.mean(year_arrays, axis=0), annot=True, fmt='.4f', cmap=cmap, vmin=-0.013, vmax=.013, center=0.00,
                square=True, linewidths=.5, annot_kws={"size": 11}, cbar_kws={"shrink": .8})
    ax.tick_params(axis=u'both', which=u'both',length=0)
    ax.grid(which='major', axis='both', linestyle='')
    ax.set_xlabel('Away Team Goals', fontsize=13)
    ax.set_ylabel('Home Team Goals', fontsize=13)
    ax.xaxis.set_label_position('top')
    ax.xaxis.set_ticks_position('top')
    plt.figtext(0.45, 0.1, 'Actual Proportion - Model Probability', horizontalalignment='center',
                fontsize=14, multialignment='left', fontweight='bold')
plt.tight_layout()
plt.show()


I am working on the file "Jupyter/2018-09-13-predicting-football-results-with-statistical-modelling-dixon-coles-and-time-weighting.ipynb".
The first task is:
In one way, this is a good plot. The correction was only intended to have an effect on 4 specific match results (0-0, 1-0, 0-1 and 1-1) and that's what has happened. On the other hand, that was alot of hard work to essentially tweak the existing model. And that's without even considering whether it was a beneficial adjustment. Without exploring that point any further, I'm going to discuss the second advancement introduced by the DC model.
Dixon-Coles Time Decay Model
Crystal Palace famously (!) lost their opening seven fixtures of the 2017/18 EPL season, conceding 17 times and scoring zero goals. During his short reign, Ronald De Boer had disastrously tried to transform Palace into a more attractive, possession based side. Under their new manager, Roy Hodgson, they returned to their traditional counter attacking style. They recovered well from their poor start to end the season in a respectable 11th place.
Intuitively, if you were trying to predict a Crystal Palace match in January 2018, you would want the model to somewhat discount those losses in August and September 2017. That's the rationale behind adding a time component to the adjusted Poisson model outlined above. How exactly to down-weight those earlier games is the tricky part. Two weighting options offered in the Dixon-Coles paper are illustrated below.
Can you write Python code for it?

fig,(ax1,ax2) = plt.subplots(2, 1, figsize=(10,5))

ax1.plot(range(1000), [0 if y >600 else 1 for y in range(1000)], label='Component 1', color='#38003c', marker='')
ax2.plot(range(1000), np.exp([y*-0.005 for y in range(1000)]), label='Component 1', color='#07F2F2', marker='')
ax2.plot(range(1000), np.exp([y*-0.003 for y in range(1000)]), label='Component 1', color='#05F26C', marker='')
ax2.plot(range(1000), np.exp([y*-0.001 for y in range(1000)]), label='Component 1', color='#e90052', marker='')

ax1.set_ylim([-0.05,1.05])
ax2.set_ylim([-0.05,1.05])
ax1.set_xlim([-0.5,1000])
ax2.set_xlim([-0.5,1000])
ax1.set_xticklabels([])
ax2.xaxis.set_tick_params(labelsize=12)
ax1.yaxis.set_tick_params(labelsize=12)
ax2.yaxis.set_tick_params(labelsize=12)
ax1.set_title("Time Decay Weighting Functions",size=14,fontweight='bold')
ax2.set_xlabel("Number of Days Ago",size=13)
ax1.set_ylabel("ϕ(t)",size=13)
ax2.set_ylabel("ϕ(t)",size=13)
ax1.text(830, 0.5, '1     $t \leq \mathregular{t_0}$\n0     $t > \mathregular{t_0}$',
        verticalalignment='bottom', horizontalalignment='left',
        color='black', fontsize=15)
ax1.text(800, 0.5, '{',
        verticalalignment='bottom', horizontalalignment='left',
        color='black', fontsize=44)
ax1.text(730, 0.62, 'ϕ(t)  = ',
        verticalalignment='bottom', horizontalalignment='left',
        color='black', fontsize=15)
ax2.text(730, 0.62, 'ϕ(t)  =   exp(−ξt)',
        verticalalignment='bottom', horizontalalignment='left',
        color='black', fontsize=15)
ax2.text(250, 0.8, 'ξ = 0.001',
        verticalalignment='bottom', horizontalalignment='left',
        color='#e90052', fontsize=15)
ax2.text(250, 0.5, 'ξ = 0.003',
        verticalalignment='bottom', horizontalalignment='left',
        color='#05F26C', fontsize=15)
ax2.text(250, 0.0, 'ξ = 0.005',
        verticalalignment='bottom', horizontalalignment='left',
        color='#07F2F2', fontsize=15)
plt.tight_layout()
plt.show()


I am working on the file "Jupyter/2018-09-13-predicting-football-results-with-statistical-modelling-dixon-coles-and-time-weighting.ipynb".
The first task is:
The first option forces the model to only consider matches within some predefined period (e.g. since the start of the season), while the negative exponential downweights match results more strongly going further into the past. The refined model can be written in these mathematical terms:
$$
L(\alpha_i, \beta_i, \rho, \gamma, i=1,\dots,n) = \prod_{k \in A_t}{\tau_{\lambda_k,\mu_k}(x_k, y_k) \frac{e^{-\lambda} \lambda^{x_k} }{x_k!} \frac{e^{-\mu} \mu^{y_k} }{y_k!}}^{\phi(t-t_k)}
$$
where $t_k$ represents the time that match $k$ was played,  $A_t = {k: t_k < t}$ (i.e. set of matches played before time $t$), $\alpha$, $\beta$, $\gamma$ and $\tau$ are defined as before. $\phi$ represents the non-increasing weighting function. Copying the original Dixon Coles paper, we'll set $\phi(t)$ to be a negative exponential with rate $\xi$ (called xi). As before, we need to determine the parameters that maximise this likelihood function. We can't just feed this equation into a minimisation algorithm for various reasons (e.g. we can trivially maximise this function by increasing $\xi$). Instead, we'll fix $\xi$ and determine the remaining parameters the same way as before. We can thus write the corresponding log-likelihood function in the following Python code (recall $\log(a^b) = \log(a) \log(b)$). Note how $\xi$=0 equates to the standard non-time weighted log-likelihood function.
Can you write Python code for it?

def dc_log_like_decay(x, y, alpha_x, beta_x, alpha_y, beta_y, rho, gamma, t, xi=0):
    lambda_x, mu_y = np.exp(alpha_x + beta_y + gamma), np.exp(alpha_y + beta_x)
    return  np.exp(-xi*t) * (np.log(rho_correction(x, y, lambda_x, mu_y, rho)) +
                              np.log(poisson.pmf(x, lambda_x)) + np.log(poisson.pmf(y, mu_y)))


I am working on the file "Jupyter/2018-09-13-predicting-football-results-with-statistical-modelling-dixon-coles-and-time-weighting.ipynb".
The first task is:
There does seem to be an issue around low scoring draws, though it is less apparent with 1-0 and 0-1 results. The Dixon-Coles (DC) model applies a correction to the BP model. It can be written in these mathematical terms:
$$
P\left(X_{i,j} = x, Y_{j,i} = y \right) = \tau_{\lambda, \mu}(x) \frac{e^{-\lambda} \lambda^x }{x!} \frac{e^{-\mu} \mu^y }{y!}
\ \text{where } \quad \lambda = \alpha_i \beta_j \gamma \quad \mu = \alpha_j \beta_i
\
\tau_{\lambda, \mu}(x, y) =
  \begin{cases}
                                   1 - \lambda \mu \rho & \text{if $x = y = 0$} \
                                   1 - \lambda \rho & \text{if $x=0$, $y=1$} \
                                   1 + \mu \rho & \text{if $x=0$, $y=1$} \
                                   1 - \rho & \text{if $x = y = 1$} \
                                   1 & \text{otherwise} \
  \end{cases}
$$
The key difference over the BP model is the addition of the $\tau$ (tau) function. It is highly dependent on the $\rho$ (rho) parameter, which controls the strength of the correction (note: setting $\rho$=0 equates to the standard BP model). We can easily convert $\tau_{\lambda, \mu}(x, y)$ to Python code.
Can you write Python code for it?

def rho_correction(x, y, lambda_x, mu_y, rho):
    if x==0 and y==0:
        return 1- (lambda_x * mu_y * rho)
    elif x==0 and y==1:
        return 1 + (lambda_x * rho)
    elif x==1 and y==0:
        return 1 + (mu_y * rho)
    elif x==1 and y==1:
        return 1 - rho
    else:
        return 1.0


I am working on the file "Jupyter/2018-09-13-predicting-football-results-with-statistical-modelling-dixon-coles-and-time-weighting.ipynb".
The first task is:
To determine the optimal value of $\xi$, we'll select the model that makes the best predictions. Repeating the process in the Dixon-Coles paper, rather working on match score predictions, the models will be assessed on match result predictions. Essentially, the model that predicted the actual match results with the highest probability will be deemed the winner. An obvious flaw here is that only one match result is considered. For example, if the result was a home win, then the draw and away win probabilities are ignored. Alternative approaches could utilise Ranked Probability Scores or betting probabilities. But we'll keep things simple and replicate the Dixon Coles paper. We can redefine the objective in mathematical terms; we wish to find $\xi$ that maximises $S(\xi)$:
$$
S(\xi) = \sum^{N}{k=1} (\delta^{H}{k} \log p^{H}{k} + \delta^{A}{k} \log p^{A}{k} + \delta^{D}{k} \log p^{D}_{k})
$$
This looks more complicated than it really is. The $\delta$ terms just captures the match result e.g. $\delta^{H}{k}$ = 1 if match $k$ ended in a home win, while the $p$ terms are simply the match result probabilities. For example, we can rewrite $p^{H}{k}$ (probability of home win):
$$
p^{H}{k} = \sum{l,m \in B_H} P(X_k = l, Y_k = m), \text{ where } B_H = {(l,m): l>m}
$$
Each of these $p$ terms translates to the matrix operations outlined previously. One part that will change is the log likelihood function.
To assess the predictive accuracy of the model, we'll utilise an approach analogous to the validation set in machine learning. Let's say we're trying to predict the fixtures occurring on the 13th January 2018. With $\xi$ fixed to a specific value, we use all of the previous results in that season to build a model. We determine how that model predicted the actual results of those matches with the above equations. We move onto the next set of fixtures (say 20th January) and build the model again- this time including the 13th January games- and assess how well it predicted the results of those matches. We repeat this...
Can you write Python code for it?

epl_1718 = pd.read_csv("http://www.football-data.co.uk/mmz4281/1718/E0.csv")
epl_1718['Date'] = pd.to_datetime(epl_1718['Date'],  format='%d/%m/%y')
epl_1718['time_diff'] = (max(epl_1718['Date']) - epl_1718['Date']).dt.days
epl_1718 = epl_1718[['HomeTeam','AwayTeam','FTHG','FTAG', 'FTR', 'time_diff']]
epl_1718 = epl_1718.rename(columns={'FTHG': 'HomeGoals', 'FTAG': 'AwayGoals'})
epl_1718.head()


I am working on the file "Jupyter/2018-09-13-predicting-football-results-with-statistical-modelling-dixon-coles-and-time-weighting.ipynb".
The first task is:
Unfortunately, you can't just update your match score matrix with this function; you need to recalculate the various coefficients that go into the model. And unfortunately again, you can't just implement an off the shelf generalised linear model, as we did before. We have to construct the likelihood function and find the coefficients that maximise it- a technique known as Maximum Likelihood Estimation. With matches indexed $k=1,\dots,N$ and corresponding scores ($x_k$, $y_k$), this is the likelihood function that we seek to maximise:
$$
L(\alpha_i, \beta_i, \rho, \gamma, i=1,\dots,n) = \prod_{k=1}^{N} \tau_{\lambda_k,\mu_k}(x_k, y_k) \ \frac{e^{-\lambda} \lambda^{x_k} }{x_k!} \frac{e^{-\mu} \mu^{y_k} }{y_k!}
\ \text{where } \quad \lambda_k = \alpha_{i(k)} \beta_{j(k)} \gamma \quad \mu_k = \alpha_{j(k)} \beta_{i(k)}
$$
In this equation, $i(k)$ and $j(k)$ respectively denote the indices of the home and away teams in match $k$. For a few different reasons (numerical precision, practicality, etc.), we'll actually maximise the log-likelihood function. As the logarithm is a strictly increasing function (i.e. $\log(b) > \log(a) \ \forall \ b > a$), both likelihood functions are maximised at the same point. Also, recall that $\log(a \ b) = \log(a) + \log(b)$. We can thus write the log-likelihood function in Python code.
Can you write Python code for it?

def dc_log_like(x, y, alpha_x, beta_x, alpha_y, beta_y, rho, gamma):
    lambda_x, mu_y = np.exp(alpha_x + beta_y + gamma), np.exp(alpha_y + beta_x)
    return (np.log(rho_correction(x, y, lambda_x, mu_y, rho)) +
            np.log(poisson.pmf(x, lambda_x)) + np.log(poisson.pmf(y, mu_y)))


I am working on the file "Jupyter/2018-09-13-predicting-football-results-with-statistical-modelling-dixon-coles-and-time-weighting.ipynb".
The first task is:
We can formulate the model in mathematical terms:
$$
P\left(X_{i,j} = x, Y_{j,i} = y \right) = \frac{e^{-\lambda} \lambda^x }{x!} \frac{e^{-\mu} \mu^y }{y!}
\ \text{where } \quad \lambda = \alpha_i \beta_j \gamma \quad \mu = \alpha_j \beta_i
$$
In this equation, $i$ and $j$ refer to the home and away teams, respectively; $\alpha$ and $\beta$ denote each team's attack and defensive strength, respectively, while $\gamma$ represents the home advantage factor. So, we need to calculate $\alpha$ and $\beta$ for each team, as well as $\gamma$ (the home field advantage term- it's the same value for every team). As this was explained in the previous post, I'll just show the model output.
Can you write Python code for it?

# importing the tools required for the Poisson regression model
import statsmodels.api as sm
import statsmodels.formula.api as smf

goal_model_data = pd.concat([epl_1718[['HomeTeam','AwayTeam','HomeGoals']].assign(home=1).rename(
            columns={'HomeTeam':'team', 'AwayTeam':'opponent','HomeGoals':'goals'}),
           epl_1718[['AwayTeam','HomeTeam','AwayGoals']].assign(home=0).rename(
            columns={'AwayTeam':'team', 'HomeTeam':'opponent','AwayGoals':'goals'})])

poisson_model = smf.glm(formula="goals ~ home + team + opponent", data=goal_model_data,
                        family=sm.families.Poisson()).fit()

print(poisson_model.summary())

poisson_model.predict(pd.DataFrame(data={'team': 'Arsenal', 'opponent': 'Southampton',
                                       'home':1},index=[1]))

poisson_model.predict(pd.DataFrame(data={'team': 'Southampton', 'opponent': 'Arsenal',
                                       'home':0},index=[1]))


I am working on the file "Jupyter/2018-09-13-predicting-football-results-with-statistical-modelling-dixon-coles-and-time-weighting.ipynb".
The first task is:
You may have noticed that dc_log_like included a transformation of $\lambda$ and $\mu$, where  $\lambda = \exp(\alpha_i + \beta_j + \gamma)$ and $\mu = \exp(\alpha_j + \beta_i)$, so that we're essentially trying to calculate expected log goals. This is equivalent to the log link function in the previous BP glm implementation. It shouldn't really affect model accuracy, it just means that convergence of the maximisation algorithm should be easier as $\lambda, \mu > 0 \ \forall \ \alpha, \beta, \gamma$. Non-positive lambdas are not compatible with a Poisson distribution, so this would return warnings and/or errors during implementation.
We're now ready to find the parameters that maximise the log likelihood function. Basically, you design a function that takes a set of model parameters as an argument. You set some initial values and potentially include some constraints and select the appropriate optimisation algorithm. I've opted for scipy's minimise function (a possible alternative is fmin- note: the functions  seek to minimise the negative log-likelihood). It employs a process analogous to gradient descent, so that the algorithm iteratively converges to the optimal parameter set. The computation can be quite slow as it's forced to approximate the derivatives. If you're not as lazy as me, you could potentially speed it up by manually constructing the partial derivatives.
In line with the original Dixon Coles paper and the opisthokonta blog, I've added the constraint that $\frac{1}{n}\sum_{i} \alpha_{i}=1$ (i.e. the average attack strength value is 1). This step isn't strictly necessary, but it means that it should return a unique solution (otherwise, the model would suffer from overparamterisation and each execution would return different coefficients).
Okay, we're ready to find the coefficients that maximise the log-likelihood function for the 2017/18 EPL season.
Can you write Python code for it?

def solve_parameters(dataset, debug = False, init_vals=None, options={'disp': True, 'maxiter':100},
                     constraints = [{'type':'eq', 'fun': lambda x: sum(x[:20])-20}] , **kwargs):
    teams = np.sort(dataset['HomeTeam'].unique())
    # check for no weirdness in dataset
    away_teams = np.sort(dataset['AwayTeam'].unique())
    if not np.array_equal(teams, away_teams):
        raise ValueError("Something's not right")
    n_teams = len(teams)
    if init_vals is None:
        # random initialisation of model parameters
        init_vals = np.concatenate((np.random.uniform(0,1,(n_teams)), # attack strength
                                      np.random.uniform(0,-1,(n_teams)), # defence strength
                                      np.array([0, 1.0]) # rho (score correction), gamma (home advantage)
                                     ))
    def dc_log_like(x, y, alpha_x, beta_x, alpha_y, beta_y, rho, gamma):
        lambda_x, mu_y = np.exp(alpha_x + beta_y + gamma), np.exp(alpha_y + beta_x)
        return (np.log(rho_correction(x, y, lambda_x, mu_y, rho)) +
                np.log(poisson.pmf(x, lambda_x)) + np.log(poisson.pmf(y, mu_y)))

    def estimate_paramters(params):
        score_coefs = dict(zip(teams, params[:n_teams]))
        defend_coefs = dict(zip(teams, params[n_teams:(2*n_teams)]))
        rho, gamma = params[-2:]
        log_like = [dc_log_like(row.HomeGoals, row.AwayGoals, score_coefs[row.HomeTeam], defend_coefs[row.HomeTeam],
                     score_coefs[row.AwayTeam], defend_coefs[row.AwayTeam], rho, gamma) for row in dataset.itertuples()]
        return -sum(log_like)
    opt_output = minimize(estimate_paramters, init_vals, options=options, constraints = constraints, **kwargs)
    if debug:
        # sort of hacky way to investigate the output of the optimisation process
        return opt_output
    else:
        return dict(zip(["attack_"+team for team in teams] +
                        ["defence_"+team for team in teams] +
                        ['rho',...


I am working on the file "Jupyter/BasicAlgorithmAnalysisTemplate.ipynb".
The first task is:
Read out the orders of a specific backtest
Can you write Python code for it?


I am working on the file "Jupyter/BasicQuantBookTemplate.ipynb".
The first task is:
Historical Options Data Requests

Select the option data
Sets the filter, otherwise the default will be used SetFilter(-1, 1, timedelta(0), timedelta(35))
Get the OptionHistory, an object that has information about the historical options data
Can you write Python code for it?

goog = qb.AddOption("GOOG")
goog.SetFilter(-2, 2, timedelta(0), timedelta(180))

option_history = qb.GetOptionHistory(goog.Symbol, datetime(2017, 1, 4))
print option_history.GetStrikes()
print option_history.GetExpiryDates()
h6 = option_history.GetAllData()


I am working on the file "Jupyter/BasicQuantBookTemplate.ipynb".
The first task is:
Indicators
We can easily get the indicator of a given symbol with QuantBook.
For all indicators, please checkout QuantConnect Indicators Reference Table
Can you write Python code for it?

# Example with BB, it is a datapoint indicator
# Define the indicator
bb = BollingerBands(30, 2)

# Gets historical data of indicator
bbdf = qb.Indicator(bb, "SPY", 360, Resolution.Daily)

# drop undesired fields
bbdf = bbdf.drop('standarddeviation', 1)

# Plot
bbdf.plot()

# For EURUSD
bbdf = qb.Indicator(bb, "EURUSD", 360, Resolution.Daily)
bbdf = bbdf.drop('standarddeviation', 1)
bbdf.plot()

# Example with ADX, it is a bar indicator
adx = AverageDirectionalIndex("adx", 14)
adxdf = qb.Indicator(adx, "SPY", 360, Resolution.Daily)
adxdf.plot()

# For EURUSD
adxdf = qb.Indicator(adx, "EURUSD", 360, Resolution.Daily)
adxdf.plot()

# Example with ADO, it is a tradebar indicator (requires volume in its calculation)
ado = AccumulationDistributionOscillator("ado", 5, 30)
adodf = qb.Indicator(ado, "SPY", 360, Resolution.Daily)
adodf.plot()

# For EURUSD.
# Uncomment to check that this SHOULD fail, since Forex is data type is not TradeBar.
# adodf = qb.Indicator(ado, "EURUSD", 360, Resolution.Daily)
# adodf.plot()

# SMA cross:
symbol = "EURUSD"
# Get History
hist = qb.History[QuoteBar](symbol, 500, Resolution.Daily)
# Get the fast moving average
fast = qb.Indicator(SimpleMovingAverage(50), symbol, 500, Resolution.Daily)
# Get the fast moving average
slow = qb.Indicator(SimpleMovingAverage(200), symbol, 500, Resolution.Daily)

# Remove undesired columns and rename others
fast = fast.drop('rollingsum', 1).rename(columns={'simplemovingaverage': 'fast'})
slow = slow.drop('rollingsum', 1).rename(columns={'simplemovingaverage': 'slow'})

# Concatenate the information and plot
df = pd.concat([hist.loc[symbol]["close"], fast, slow], axis=1).dropna(axis=0)
df.plot()

# Get indicator defining a lookback period in terms of timedelta
ema1 = qb.Indicator(ExponentialMovingAverage(50), "SPY", timedelta(100), Resolution.Daily)
# Get indicator defining a start and end date
ema2 = qb.Indicator(ExponentialMovingAverage(50), "SPY", datetime(2016,1,1), datetime(2016,10,1), Resolution.Daily)

ema = pd.concat([ema1, ema2],...


I am working on the file "Jupyter/Calibration Check.ipynb".
The first task is:
Differences
Below are statistics on the differences between the measured and calibrated CO₂ values.
Can you write Python code for it?

data['Difference'] = data['Calibrated'] - data['Measured']

# Time series of differences
diff_timeseries = figure(plot_width=900, plot_height=600, x_axis_type='datetime', x_axis_label='Time', y_axis_label='Calibrated - Measured')
diff_timeseries.circle(data['Timestamp'], data['Difference'], color='black', size=5, legend_label='Calibrated - Measured')
show(diff_timeseries)

hist, edges = np.histogram(data['Difference'], density=True, bins=100)

print(f'Difference range {min(data["Difference"])} to {max(data["Difference"])}')
print(f'Mean difference {statistics.mean(data["Difference"])}')
print(f'Median difference {statistics.median(data["Difference"])}')

p = figure(plot_width=900, plot_height=600, x_axis_label='Difference', y_axis_label='Proprotion')
p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], fill_color="navy", line_color="white", alpha=0.5)
show(p)


I am working on the file "Jupyter/Calibration Check.ipynb".
The first task is:
Load and prepare data
Can you write Python code for it?

in_data = pd.read_csv(os.path.join(FILE_DIR, chosen_file))
data = in_data[['Date/Time', 'CO2 Mole Fraction [umol mol-1]', 'xCO2 In Water - Calibrated In Dry Air [umol mol-1]']]
data.rename(columns = {'Date/Time':'Timestamp'}, inplace = True)
data.rename(columns = {'CO2 Mole Fraction [umol mol-1]':'Measured'}, inplace = True)
data.rename(columns = {'xCO2 In Water - Calibrated In Dry Air [umol mol-1]':'Calibrated'}, inplace = True)
data['Timestamp'] = data['Timestamp'].apply(pd.to_datetime)
data = data[pd.to_numeric(data['Calibrated'], errors='coerce').notnull()]


I am working on the file "Jupyter/Calibration Check.ipynb".
The first task is:
Time Series
The cell below builds a simple plot of the time series, showing the measured vs calibrated CO₂ for water measurements
Can you write Python code for it?

timeseries = figure(plot_width=900, plot_height=600, x_axis_type='datetime', x_axis_label='Time', y_axis_label='CO₂')
timeseries.circle(data['Timestamp'], data['Measured'], color='black', size=5, legend_label='Measured')
timeseries.circle(data['Timestamp'], data['Calibrated'], color='blue', size=5, legend_label='Calibrated')
show(timeseries)


I am working on the file "Jupyter/ClarityViz Tutorial.ipynb".
The first task is:
2) local histogram equilization

local histogram equilization is the process of enhancing low contrast images, and makes the pixel values easier to process
after you apply localeq, load the nii file generated from it
Can you write Python code for it?


I am working on the file "Jupyter/ClarityViz Tutorial.ipynb".
The first task is:
Sample output from generate_plotly_html():
Fear199:
https://neurodatadesign.github.io/seelviz/reveal/html/clarityvizhtmls/Fear199plotly.html
Cocaine174:
https://neurodatadesign.github.io/seelviz/reveal/html/Cocaine174localeq.html
Control181:
https://neurodatadesign.github.io/seelviz/reveal/html/Control181localeq.html
After you finish generating all the files from claritybase, you have two options for more advanced analysis: the densitygraph module and the atlasregiongraph module
II) Using densitygraph
Once the graphml file is generated with graphmlconvert(), you can use the densitygraph module.
The density graph module is used to visualize the density of nodes in the graph, i.e. the density of neurons, in a colored fashion.
Can you write Python code for it?

from clarityviz import densitygraph

# Uses the same token before, must be in the same directory as before.
dg = densitygraph(token)

# generates a 3d plotly with color representations of density
dg.generate_density_graph()

# generates a heat map, essentially a legend, telling how many edges a certain color represents,
# with number of edges representing how dense a certain node clustering may be.
dg.generate_heat_map()


I am working on the file "Jupyter/ClarityVizPipelineResultsAccuracy.ipynb".
The first task is:
plots
Fear199
confirmed plot:
https://neurodatadesign.github.io/seelviz/reveal/html/Fear199localeq.html
clarityviz package plot:
https://neurodatadesign.github.io/seelviz/reveal/html/clarityvizhtmls/Fear199plotly.html
Cocaine174
confirmed plot:
https://neurodatadesign.github.io/seelviz/reveal/html/Cocaine174localeq.html
clarityviz package plot:
https://neurodatadesign.github.io/seelviz/reveal/html/clarityvizhtmls/Cocaine174plotly.html
Control181
confirmed plot:
https://neurodatadesign.github.io/seelviz/reveal/html/Control181localeq.html
clarityviz package plot:
https://neurodatadesign.github.io/seelviz/reveal/html/clarityvizhtmls/Control181plotly.html
Density graphs
Can you write Python code for it?

from clarityviz import densitygraph

dg = densitygraph('Fear199')
dg.generate_density_graph()
dg.generate_heat_map()


I am working on the file "Jupyter/Components/4_the_folk.ipynb".
The first task is:
Stacked Bars
Can you write Python code for it?

"""
http://bokeh.pydata.org/en/latest/docs/reference/charts.html

"""

from collections import OrderedDict
from bokeh.charts import Bar, output_file, show

# (dict, OrderedDict, lists, arrays and DataFrames are valid inputs)
xyvalues = OrderedDict()
xyvalues['python']=[-2, 5]
xyvalues['pypy']=[12, 40]
xyvalues['jython']=[22, 30]

cat = ['1st', '2nd']

bar = Bar(xyvalues, cat, title="Stacked bars",
        xlabel="category", ylabel="language")

output_file("stacked_bar.html")
show(bar)

from collections import OrderedDict

import pandas as pd

from bokeh._legacy_charts import Bar, output_file, show
from bokeh.sampledata.olympics2014 import data

df = pd.io.json.json_normalize(data['data'])

# filter by countries with at least one medal and sort
df = df[df['medals.total'] > 0]
df = df.sort("medals.total", ascending=False)

# get the countries and we group the data by medal type
countries = df.abbr.values.tolist()
gold = df['medals.gold'].astype(float).values
silver = df['medals.silver'].astype(float).values
bronze = df['medals.bronze'].astype(float).values

# build a dict containing the grouped data
medals = OrderedDict(bronze=bronze, silver=silver, gold=gold)

# any of the following commented are also alid Bar inputs
#medals = pd.DataFrame(medals)
#medals = list(medals.values())

output_file("stacked_bar.html")

bar = Bar(medals, countries, title="Stacked bars", stacked=True)

show(bar)


I am working on the file "Jupyter/Components/more_work_5_the_senate.ipynb".
The first task is:
Average age of Men and Women
Can you write Python code for it?


I am working on the file "Jupyter/Components/more_work_5_the_senate.ipynb".
The first task is:
Now we have read the data from the csv files
Can you write Python code for it?

ls_labels_men = []
ls_values_men = []

for i in range(1,df_men.shape[0]):
    ls_labels_men.append(str(df_men.iat[i,0]))
    ls_values_men.append(float(df_men.iat[i,1]))

import numpy as np
import matplotlib.pyplot as plt

N = len(ls_labels_men)

ind = np.arange(N)  # the x locations for the groups
width = 0.35       # the width of the bars

fig, ax = plt.subplots()

rects1 = ax.bar(ind, ls_values_men, width, color = 'red', alpha = 0.6)

#rects2 = ax.bar(ind + width, ls_values_men, width )

# add some text for labels, title and axes ticks
ax.set_ylabel('Scores')

ax.set_title('Scores by group and gender')
ax.set_xticks(ind + width)
ax.set_xticklabels(ls_labels_men)

#ax.legend((rects1[0], rects2[0]), ('Men', 'Women'))

plt.show()


I am working on the file "Jupyter/Components/more_work_5_the_senate.ipynb".
The first task is:
Sample Pie Plot
Can you write Python code for it?

import matplotlib.pyplot as plt
import pandas as pd

# The slices will be ordered and plotted counter-clockwise.

labels = 'Frogs', 'Hogs', 'Dogs', 'Logs'
sizes = [15, 30, 45, 10]
colors = ['yellowgreen', 'gold', 'lightskyblue', 'lightcoral']
explode = (0.1, 0.1, 0.1, 0.1)

plt.pie(sizes, explode=explode, labels=labels, colors=colors,
        autopct='%1.1f%%', shadow=True, startangle=90)
# Set aspect ratio to be equal so that pie is drawn as a circle.
plt.axis('equal')

plt.show()


I am working on the file "Jupyter/DeleteCvJobs.ipynb".
The first task is:
Insert your admin password
Can you write Python code for it?


I am working on the file "Jupyter/Filter_and_Plotly_Luke.ipynb".
The first task is:
Filter Image Again
Don't do this
Clean out noise (Filter Image)
(values, bins) = np.histogram(filterImg, bins=100, range=(0,500))
plt.plot(bins[:-1], values)
counts = np.bincount(values)
maximum = np.argmax(counts)
lowerThreshold = maximum
upperThreshold = filterImg.max()+1
filterX2Img = sitk.Threshold(inImg,lowerThreshold,upperThreshold,lowerThreshold) - lowerThreshold
newImg = sitk.GetArrayFromImage(filterX2Img)  ##convert to simpleITK image to normal numpy ndarray
Can you write Python code for it?

## Histogram Equalization
## Cut from generateHistogram from clarityviz
import cv2
im = filterImg
img = im[:,:,:]
shape = im.shape
#affine = im.get_affine()
x_value = shape[0]
y_value = shape[1]
z_value = shape[2]
#####################################################
imgflat = img.reshape(-1)
#img_grey = np.array(imgflat * 255, dtype = np.uint8)
#img_eq = exposure.equalize_hist(img_grey)#new_img = img_eq.reshape(x_value, y_value, z_value)
#globaleq = nib.Nifti1Image(new_img, np.eye(4))

#nb.save(globaleq, '/home/albert/Thumbo/AutAglobaleq.nii')
######################################################
#clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
img_grey = np.array(imgflat * 255, dtype = np.uint8)
#threshed = cv2.adaptiveThreshold(img_grey, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY, 3, 0)
cl1 = clahe.apply(img_grey)
#cv2.imwrite('clahe_2.jpg',cl1)
#cv2.startWindowThread()
#cv2.namedWindow("adaptive")
#cv2.imshow("adaptive", cl1)
#cv2.imshow("adaptive", threshed)
#plt.imshow(threshed)
localimgflat = cl1 #cl1.reshape(-1)
newer_img = localimgflat.reshape(x_value, y_value, z_value)  ##this is the numpy.ndarray version
localeq = nb.Nifti1Image(newer_img, np.eye(4))

print newer_img.shape


I am working on the file "Jupyter/Filter_and_Plotly_Luke.ipynb".
The first task is:
Plotting post filtering/HistEq
Can you write Python code for it?

x_histeq = newer_img[:,0,0]
y_histeq = newer_img[0,:,0]
z_histeq = newer_img[0,0,:]

## Randomly sample 100k points after filtering

xdimensions = len(x)
ydimensions = len(y)
zdimensions = len(z)
index = random.sample(xrange(0,xdimensions*ydimensions*zdimensions),100000)  #66473400 is multiplying xshape by yshape by zshape

X_val = []
Y_val = []
Z_val = []

for i in index:
    z_val = int(i/(xdimensions*ydimensions))
    z_rem = i%(xdimensions*ydimensions)
    y_val = int(z_rem/xdimensions)
    x_val = int(z_rem/ydimensions)

    X_val.append(x_val)
    Y_val.append(y_val)
    Z_val.append(z_val)

xlist = []
ylist = []
zlist = []

for i in range(100000):
    xlist.append(random.sample(xrange(0,xdimensions), 1)[0])
    ylist.append(random.sample(xrange(0,ydimensions), 1)[0])
    zlist.append(random.sample(xrange(0,zdimensions), 1)[0])

for i in index:
    z_val = int(i/(xdimensions*ydimensions))
    z_rem = i%(xdimensions*ydimensions)
    y_val = int(z_rem/xdimensions)
    x_val = int(z_rem/ydimensions)

    X_val.append(x_val)
    Y_val.append(y_val)
    Z_val.append(z_val)

trace1 = go.Scatter3d(
    x = x,
    y = y,
    z = z,
    mode='markers',
    marker=dict(
        size=1.2,
        color='purple',                # set color to an array/list of desired values
        colorscale='Viridis',          # choose a colorscale
        opacity=0.15
    )
)
data = [trace1]
layout = go.Layout(
    margin=dict(
        l=0,
        r=0,
        b=0,
        t=0
    )
)

fig = go.Figure(data=data, layout=layout)
print('Aut1367postfilterHistEq' + "plotly")
plotly.offline.plot(fig, filename= 'Aut1367postfilterHistEq' + "plotly.html")

print "done"


I am working on the file "Jupyter/Filter_and_Plotly_Luke.ipynb".
The first task is:
Randomly sample 100k points
Can you write Python code for it?

filterImg = sitk.GetArrayFromImage(filteredImg)  ##convert to simpleITK image to normal numpy ndarray

print filterImg[0][0]

## Randomly sample 100k points after filtering
x = filterImg[:,0,0]
y = filterImg[0,:,0]
z = filterImg[0,0,:]

# mod by x to get x, mod by y to get y, mod by z to get z

xdimensions = len(x)
ydimensions = len(y)
zdimensions = len(z)
index = random.sample(xrange(0,xdimensions*ydimensions*zdimensions),100000)  #66473400 is multiplying xshape by yshape by zshape

X_val = []
Y_val = []
Z_val = []

for i in index:
    z_val = int(i/(xdimensions*ydimensions))
    z_rem = i%(xdimensions*ydimensions)
    y_val = int(z_rem/xdimensions)
    x_val = int(z_rem/ydimensions)

    X_val.append(x_val)
    Y_val.append(y_val)
    Z_val.append(z_val)

xlist = []
ylist = []
zlist = []
for i in range(100000):
    xlist.append(random.sample(xrange(0,xdimensions), 1)[0])
    ylist.append(random.sample(xrange(0,ydimensions), 1)[0])
    zlist.append(random.sample(xrange(0,zdimensions), 1)[0])
print xlist

print X_val
print Y_val
print Z_val

print len(X_val)
print len(Y_val)
print len(Z_val)


I am working on the file "Jupyter/Filter_and_Plotly_Luke.ipynb".
The first task is:
Rawest Plot (100k sampling)
Can you write Python code for it?

rawestImg = sitk.GetArrayFromImage(inImg)  ##convert to simpleITK image to normal numpy ndarray

## Randomly sample 100k points
import random
x = rawestImg[:,0,0]
y = rawestImg[0,:,0]
z = rawestImg[0,0,:]

# mod by x to get x, mod by y to get y, mod by z to get z

xdimensions = len(x)
ydimensions = len(y)
zdimensions = len(z)
# index = random.sample(xrange(0,xdimensions*ydimensions*zdimensions),100000)  #66473400 is multiplying xshape by yshape by zshape

####   random sampling of values > 250 ######
# X_val = []
# Y_val = []
# Z_val = []

# for i in index:
#     z_val = int(i/(xdimensions*ydimensions))
#     z_rem = i%(xdimensions*ydimensions)
#     y_val = int(z_rem/xdimensions)
#     x_val = int(z_rem/ydimensions)

#     X_val.append(x_val)
#     Y_val.append(y_val)
#     Z_val.append(z_val)

# xlist = []
# ylist = []
# zlist = []
xyz = []
# for i in range(100000):
#     value = 0
#     while(value == 0):
#         xval = random.sample(xrange(0,xdimensions), 1)[0]
#         yval = random.sample(xrange(0,ydimensions), 1)[0]
#         zval = random.sample(xrange(0,zdimensions), 1)[0]
#         value = rawestImg[xval,yval,zval]
#         if [xval, yval, zval] not in xyz and value > 250:
#             xyz.append([xval, yval, zval])
#         else:
#             value = 0
# print xyz

####  obtaining 100000 brightest points #######
total = 100000
bright = np.max(rawestImg)
allpoints = []
brightpoints = []
for (x,y,z),value in np.ndenumerate(rawestImg):
#     print (x,y,z)
#     print (x,z,z)[0]
#     print value
    if value == bright:
        brightpoints.append([(x,y,z)])
    else:
        allpoints.append([(x,y,z),value])

total -= len(brightpoints)
bright -= 1
if total < 0:
    index = random.sample(xrange(0, len(brightpoints)), total)
    for ind in index:
        xyz.append(brightpoints[ind])
else:
    xyz = xyz + brightpoints
#     for item in brightpoints:
#         outfile.write(",".join(item) + "\n")

while(total > 0):
    del brightpoints[:]
    for item in allpoints:
        if item[3] ==...


I am working on the file "Jupyter/Ilastik and Membrane Detection.ipynb".
The first task is:
I then got the file from the server onto my local computer.  I then wrote some code (see this other Jupyter notebook I made here: https://neurodatadesign.github.io/seelviz/Jupyter/ilastik/NIFTIAndIlastikMembraneDetection.html) that converts the .nii file into an np array and saves each individual plane as a TIFF.
From documentation, 3D grayscale is organized such that we effectively have (pln, row, col),where pln is plane, row is row and column is column.  Elements at the top left have the lowest brightness value, while elements in the bottom right have the highest brightness values.
My script converts each plane into a TIFF output, and saves the output TIFFs accordingly.  Now, I'm going to repeat the process I ran with the screencapped file and then apply this classification to each layer of the NIFTI image.
Since the 100x downsampled versions were simply too small (1kb), I reran the script on Fear197 downsampled only 10x. (Changing the spacing to 10x instead of 100x).  Thus, the .nii was around 11 MB
From there, I repeated the entire process from earlier using outfileFear197ds10.nii12 (layer12) as the training file.
Below is the training process on plane 12.
The raw image for plane 12 is shown below:

First, draw preliminary labels.

Now, see live results.  When satisfied, export as TIFF probability mapping.

We can now open the results and compare them with our original.
Original:

Post Ilastik:

From there, we can run our headless commands as discussed earlier.  As I discussed with Greg, the main follow up from here is to:
1)  Ascertain how reasonable these results are
     * We can do this either by taking these images, finding the bright points therein, and then comparing these       bright points to our histogram equalization based point cloud representation.
2)  Automate progress further, such that I can load images much faster.
As for all the pseudocode, it can be found here: https://www.overleaf.com/6610849ssmpyy#/22404584/
Miscellania:
1)  Could not get token Fear200 from ndreg -- throws error (see...
Can you write Python code for it?

## Running on inToken Fear200 error (so used Fear199 and Fear197 instead)

[u'1', u'0', u'3', u'2', u'5', u'4']
Traceback (most recent call last):
  File "tony.py", line 10, in <module>
    inImg = imgDownload(inToken, resolution=5)
  File "/root/.local/lib/python2.7/site-packages/ndreg-0.0.1-py2.7.egg/ndreg/ndreg.py", line 152, in imgDownload
    array = nd.get_cutout(token, channel, offset[0], size[0], offset[1], size[1], offset[2], size[2], resolution)
  File "build/bdist.linux-x86_64/egg/ndio/remote/neurodata.py", line 97, in wrapped
  File "build/bdist.linux-x86_64/egg/ndio/remote/neurodata.py", line 572, in get_cutout
  File "build/bdist.linux-x86_64/egg/ndio/remote/neurodata.py", line 633, in _get_cutout_blosc_no_chunking
IOError: Bad server response for http://openconnecto.me/ocp/ca/Fear200/Grayscale/blosc/5/0,557/0,721/16,32/: 404: (
2003, "Can't connect to MySQL server on 'dsp063.pha.jhu.edu' (111)")
from ndreg import *


I am working on the file "Jupyter/Ilastik and Membrane Detection.ipynb".
The first task is:
I then modified a script Albert wrote last week that downsamples these .nii files.  I used that script and used 1/100th of the spacing generated from above.  Thus, the spacing was (0.0032, 0.0032, 0.0001)
Can you write Python code for it?

## Script used to downsample nii to manageable size
from ndreg import *
import matplotlib
import ndio.remote.neurodata as neurodata
import nibabel as nb
inToken = "Fear197"
nd = neurodata()
print(nd.get_metadata(inToken)['dataset']['voxelres'].keys())
inImg = imgDownload(inToken, resolution=5)
inImg = imgResample(inImg, spacing=(0.0032, 0.0032, 0.0001))
imgWrite(inImg, "./Fear199ds100.nii")


I am working on the file "Jupyter/Introduction to Jupyter Notebook.ipynb".
The first task is:
Equations
You can use LaTex within Jupyter.  Check out the LaTex cheatsheet!
One alternative to LaTeX is prettyPy or sympy.  These are all great ways to write out equations, although the pros and cons of each need to be weighed before using.  For example prettyPy does not evaluate expressions but you also don't have to initialize variables.
Can you write Python code for it?

from IPython.display import display, Math, Latex
display(Math(r'F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx'))

from IPython.display import Latex
Latex(r"""\begin{eqnarray}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{eqnarray}""")


I am working on the file "Jupyter/Introduction to Jupyter Notebook.ipynb".
The first task is:
For this next example, we will import the pandas library.  Most python scripts use some sort of library or module that contains global variables, source files, and functions.  These libraries or modules can be called and executed with your code.
Can you write Python code for it?

import pandas as pd

df = pd.DataFrame()
a=range(11)
b=range(10,21)
c=range(20,31)
df['a']=a
df['b']=b
df['c']=c

df


I am working on the file "Jupyter/Introduction to Jupyter Notebook.ipynb".
The first task is:
Magic commands- Single percent sign means the cells arguments come from the same line, 2 mean the entire cell is used for the entire argument.  lsmagic command can be used to list all other commands.
Can you write Python code for it?


I am working on the file "Jupyter/Introduction to Jupyter Notebook.ipynb".
The first task is:
Matplotlib examples can be found here.
Can you write Python code for it?

import numpy as np
import matplotlib
import matplotlib.pyplot as plt

# Fixing random state for reproducibility
np.random.seed(19680801)

matplotlib.rcParams['axes.unicode_minus'] = False
fig, ax = plt.subplots()
ax.plot(10*np.random.randn(100), 10*np.random.randn(100), 'o')
ax.set_title('Using hyphen instead of Unicode minus')
plt.show()

import matplotlib.pyplot as plt
from matplotlib import collections, colors, transforms
import numpy as np

nverts = 50
npts = 100

# Make some spirals
r = np.arange(nverts)
theta = np.linspace(0, 2*np.pi, nverts)
xx = r * np.sin(theta)
yy = r * np.cos(theta)
spiral = list(zip(xx, yy))

# Make some offsets
# Fixing random state for reproducibility
rs = np.random.RandomState(19680801)

xo = rs.randn(npts)
yo = rs.randn(npts)
xyo = list(zip(xo, yo))

# Make a list of colors cycling through the default series.
colors = [colors.to_rgba(c)
          for c in plt.rcParams['axes.prop_cycle'].by_key()['color']]

fig, axes = plt.subplots(2, 2)
fig.subplots_adjust(top=0.92, left=0.07, right=0.97,
                    hspace=0.3, wspace=0.3)
((ax1, ax2), (ax3, ax4)) = axes  # unpack the axes

col = collections.LineCollection([spiral], offsets=xyo,
                                 transOffset=ax1.transData)
trans = fig.dpi_scale_trans + transforms.Affine2D().scale(1.0/72.0)
col.set_transform(trans)  # the points to pixels transform
# Note: the first argument to the collection initializer
# must be a list of sequences of x,y tuples; we have only
# one sequence, but we still have to put it in a list.
ax1.add_collection(col, autolim=True)
# autolim=True enables autoscaling.  For collections with
# offsets like this, it is neither efficient nor accurate,
# but it is good enough to generate a plot that you can use
# as a starting point.  If you know beforehand the range of
# x and y that you want to show, it is better to set them
# explicitly, leave out the autolim kwarg (or set it to False),
# and omit the 'ax1.autoscale_view()' call below.

# Make a transform for the line segments such...


I am working on the file "Jupyter/Introduction to Jupyter Notebook.ipynb".
The first task is:
The math functions can be called by saying "import math" before your code.  This way jupyter, and any python script for that matter, knows to use those functions in the following code.
Can you write Python code for it?

import math

# Input list.
values = [0.9999999, 1, 2, 3]

# Sum values in list.
r = sum(values)
print(r)

# Sum values with fsum.
r = math.fsum(values)
print(r)

import math

value1 = 9
value2 = 16
value3 = 100

# Use sqrt method.
print(math.sqrt(value1))
print(math.sqrt(value2))
print(math.sqrt(value3))


I am working on the file "Jupyter/Jupyter_Notebook_Demo.ipynb".
The first task is:
Other useful IPython cell magic
IPython magics don't only let you use other language interpreters.
Can you write Python code for it?

# change the current working directory
%cd jupyterhub/

# list the variables currently available to the kernel
%who

# list the variables and their string representation
%whos

%%time
for i in range(10):
    !sleep 1

%%timeit
np.random.normal(0,1,1000).sum()

# to capture plot output and display it inline:
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns

df['s2'].hist();
plt.show()

# using the question mark will bring up any help documentation
?pd.DataFrame


I am working on the file "Jupyter/Jupyter_Notebook_Demo.ipynb".
The first task is:
Running code in different languages
The code in this notebook is executed by the designated "kernel" loaded at creation. In this case, the IPython kernel was loaded. All code entered will therefore be interpreted by this kernel and run as python code. However, when the kernel is IPython, you have access to "cell magic" (using the % syntax), where it is possible to have cells run by a different interpreter.

Using a single % will run the magic on that line only.
Starting a cell with %% will run the magic on the entire cell.
Can you write Python code for it?

%%bash
# this cell is run in a bash shell created specially for the following code.
echo "Hello, world"

# it is also possible to invoke bash commands using the ```!``` syntax:
!ls -al | head -n 8 | tail -n 2

%%html

This is an html interpreted header

This is an html link


I am working on the file "Jupyter/Jupyter_Notebook_Demo.ipynb".
The first task is:
Sharing variables between languages
It is even possible to capture the variables from each cell/interpreter/language, and pass them into others:
Can you write Python code for it?

# capturing the output of the bash ls command:
directory_contents = !ls -la
directory_contents

%%bash -s "$a"
# The above line puts the variable a into the bash shell as a positional parameter.
# Be aware of any characters (eg. quotation marks) in the python variable -
# these will need to be escaped before being passed to the bash cell.
echo $1

# an alternative to send variables into bash:
!echo {a * 2}

# R requires a few extra steps to access
# rpy2 provides access to R from within Python
# (you can read more here: http://rpy2.readthedocs.io)
# after installing rpy2 - we load the extension into the kernel:
%load_ext rpy2.ipython

# now we can access the installed version of R
iris_dataset = %R iris

iris_dataset.describe()

%%R -i df
# the above line sets R as the interpreter for this cell,
# and imports the variable df (it will be referenced in this cell using the same name)

# Now we can manipulate and graph the dataframe using R functions:
require(ggplot2)
ggplot(data=df) + geom_point(aes(x=s1, y=s2))


I am working on the file "Jupyter/Jupyter_Notebook_Demo.ipynb".
The first task is:
This is a text cell
It is formatted using markdown syntax.
(to edit a markdown cell, just double click on the text. Don't forget to 'execute' the cell afterwards to implement the formatting)
Markdown cells within a notebook have a number of advantages:
1. Easy to type
2. Easy to read
3. Great for discussion of code:
  * Choice of analysis
  * Choice of parameters
  * Implications of results
  * Introduction/methods/conclusions/references...
Cells are switched between code and markdown by using the menu
Cell > Cell Type > Markdown
Or by using the  dropdown box in the icon bar.
You can even create links!
Can you write Python code for it?

import numpy as np
import pandas as pd
# notice that this cell doesn't execute when you press enter.
# Only by pressing shift-enter or alt-enter, or clicking on the 'run' icon.

# this cell does not generate any output to stdout or stderr,
# so nothing is shown after executing the cell.
s1 = np.random.normal(0,1,1000)  # generate a random sample with normal distribution (mean 0, sd 1, 1000 samples)
s2 = np.random.normal(2,4,1000)
df = pd.DataFrame({"s1":s1, "s2":s2})

# this cell outputs to stdout,
# which is printed immediately following the cell:
df.info()

# table output is formatted to make it easy to view:
df.T.head()


I am working on the file "Jupyter/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Historical Data Requests
We can use the QuantConnect API to make Historical Data Requests. The data will be presented as multi-index pandas.DataFrame where the first index is the Symbol.
For more information, please follow the link.
Can you write Python code for it?

# Gets historical data from the subscribed assets, the last 360 datapoints with daily resolution
h1 = qb.History(qb.Securities.Keys, 360, Resolution.Daily)

# Plot closing prices from "SPY"
h1.loc["SPY"]["close"].plot()

# Gets historical data from the subscribed assets, from the last 30 days with daily resolution
h2 = qb.History(qb.Securities.Keys, timedelta(360), Resolution.Daily)

# Plot high prices from "EURUSD"
h2.loc["EURUSD"]["high"].plot()

# Gets historical data from the subscribed assets, between two dates with daily resolution
h3 = qb.History([btc.Symbol], datetime(2014,1,1), datetime.now(), Resolution.Daily)

# Plot closing prices from "BTCUSD"
h3.loc["BTCUSD"]["close"].plot()

# Only fetchs historical data from a desired symbol
h4 = qb.History([spy.Symbol], 360, Resolution.Daily)
# or qb.History(["SPY"], 360, Resolution.Daily)

# Only fetchs historical data from a desired symbol
h5 = qb.History([eur.Symbol], timedelta(360), Resolution.Daily)
# or qb.History(["EURUSD"], timedelta(30), Resolution.Daily)

# Fetchs custom data
h6 = qb.History([fxv.Symbol], timedelta(360))
h6.loc[fxv.Symbol.Value]["volume"].plot()


I am working on the file "Jupyter/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Historical Future Data Requests

Select the future data
Sets the filter, otherwise the default will be used SetFilter(timedelta(0), timedelta(35))
Get the FutureHistory, an object that has information about the historical future data
Can you write Python code for it?

es = qb.AddFuture("ES")
es.SetFilter(timedelta(0), timedelta(180))

future_history = qb.GetFutureHistory(es.Symbol, datetime(2017, 1, 4))
print (future_history.GetExpiryDates())
h7 = future_history.GetAllData()


I am working on the file "Jupyter/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Historical Options Data Requests

Select the option data
Sets the filter, otherwise the default will be used SetFilter(-1, 1, timedelta(0), timedelta(35))
Get the OptionHistory, an object that has information about the historical options data
Can you write Python code for it?

goog = qb.AddOption("GOOG")
goog.SetFilter(-2, 2, timedelta(0), timedelta(180))

option_history = qb.GetOptionHistory(goog.Symbol, datetime(2017, 1, 4))
print (option_history.GetStrikes())
print (option_history.GetExpiryDates())
h7 = option_history.GetAllData()


I am working on the file "Jupyter/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Selecting Asset Data
Checkout the QuantConnect docs to learn how to select asset data.
Can you write Python code for it?

spy = qb.AddEquity("SPY")
eur = qb.AddForex("EURUSD")
btc = qb.AddCrypto("BTCUSD")
fxv = qb.AddData[FxcmVolume]("EURUSD_Vol", Resolution.Hour)


I am working on the file "Jupyter/Labs/02_VectorAdd/HYB_CUDA_CSHARP.ipynb".
The first task is:
Introduce Parallelism
As we can see in the solution, a plain scalar iterative approach only uses one thread, while modern CPUs have typically 4 cores and 8 threads.
Fortunately, .Net and C# provide an intuitive construct to leverage parallelism : Parallel.For.
Modify 01-vector-add.cs to distribute the work among multiple threads.
If you get stuck, you can refer to the solution.
Can you write Python code for it?

!hybridizer-cuda ./01-vector-add/01-vector-add.cs -o ./01-vector-add/parallel-vectoradd.exe -run


I am working on the file "Jupyter/Labs/03_Streams/HYB_CUDA_Csharp_Stream.ipynb".
The first task is:
Run Code on the GPU
Using Hybridizer to run the above code on a GPU is quite straightforward. We need to
- Decorate methods we want to run on the GPU
This is done by adding [EntryPoint] attribute on methods of interest.
- "Wrap" current object into a dynamic object able to dispatch code on the GPU
This is done by the following boilerplate code:
csharp
dynamic wrapped = HybRunner.Cuda().Wrap(new Program());
wrapped.mymethod(...);
wrapped object has the same methods signatures (static or instance) as the current object, but dispatches calls to GPU.
Modify the 02-gpu-add.cs so the Add method runs on a GPU.
If you get stuck, you can refer to the solution.
Can you write Python code for it?

!hybridizer-cuda ./02-gpu-add/02-gpu-add.cs -o ./02-gpu-add/gpu-add.exe -run


I am working on the file "Jupyter/Labs/03_Streams/HYB_CUDA_Csharp_Stream.ipynb".
The first task is:
STREAM
the purpose of this example is to allow you to use streams with the Hybridizer, on one very big vector without cut it. We will use 8 streams for this example.

You can create a stream with the object cudaStream_t and  cuda.StreamCreate(out yourStream).
To set a stream on a kernel you have to use the SetStream(stream) function on wrapped.
csharp
wrapped.SetStream(stream).mymethod(...);
You have the possibility to make an asynchronous cudaMemCpy when you copy data
csharp
cuda.MemcpyAsync(IntPtr dst, IntPtr src, size_t size, cudaMemcpyKind kindOfCopy, cudaStream_t stream =0);
You can block until the stream finish to compute with cuda.StreamSynchronize(stream).
Finally destroy your stream with cuda.StreamDestroy(stream).

Modify the 04-stream-add.cs so you can create and use multiple streams.
If you get stuck, you can refer to the solution.
Can you write Python code for it?

!hybridizer-cuda ./04-stream-add/04-stream-add.cs -o ./04-stream-add/stream-add.exe -run


I am working on the file "Jupyter/Labs/04_MedianFilter/IP_MEDIAN_FILTER.ipynb".
The first task is:
Feeding the Beast
A modern GPU is made of thousands of CUDA-cores, for which operations need to be stacked to hide latency. In our image processing example, the distribution of work is done on lines, that is a couple of thousand for the whole image. Then, each CUDA-thread will process a complete line, as illustrated in the following image:
<img title="SlicingWork" src="./images/work-stripes.png"/>
Using CUDA API, we may query the number of CUDA cores multiprocessors, and the number of CUDA cores. HybRunner has a default execution configuration that is suited to most use-cases.
Run the following code to query information on the GPU and execution configuration - see also cudaGetDeviceProperties.
Can you write Python code for it?

!hybridizer-cuda ./05-dice-gpu/01-query-config.cs -o ./05-dice-gpu/01-query-config.exe -run


I am working on the file "Jupyter/Labs/04_MedianFilter/IP_MEDIAN_FILTER.ipynb".
The first task is:
Local load and store transactions should have been reduced to zero.
The display of kernel time actually includes some memory transfer and synchronization. The real execution time is returned by the profiler when run in summary mode.
Can you write Python code for it?

!cd 06-regsort-gpu/hybrid ; nvprof -s --profile-child-processes  ./run.sh


I am working on the file "Jupyter/Labs/04_MedianFilter/IP_MEDIAN_FILTER.ipynb".
The first task is:
Profiling application
Profiling the application may be done with nvprof. The following execution box provides the command line. We are querying the following metrics:
 - local_{load,store}transactions : to get the memory transactions on local memory
 - {gld_gst}_transactions : to get the memory transactions to global memory - cache misses
 - l2{read,write}_transactions : to get the memory transactions on the L2 cache
Can you write Python code for it?

!cd 05-dice-gpu/hybrid ; nvprof --profile-child-processes --metrics local_load_transactions,local_store_transactions,gld_transactions,gst_transactions,l2_read_transactions,l2_write_transactions ./run.sh


I am working on the file "Jupyter/Labs/04_MedianFilter/IP_MEDIAN_FILTER.ipynb".
The first task is:
Running on the GPU
In order to run the filter on the GPU, three operations are needed:
 - Mark the method with an EntryPoint attribute to indicate it should run on GPU
 - Launch the kernel using the HybRunner, generated with static method HybRunner.Cuda()
 - Array.Sort is not a builtin mapped to an existing code, the sort will be changed (this is out of the scope of this lab and will be already done in the input file)
Creating an instance of HybRunner and wrapping an object is done as follow:
csharp
HybRunner runner = HybRunner.Cuda();
dynamic wrapper = runner.Wrap(new Program());
Note that the result of the Wrap method is a dynamic type generated on the fly by the runner. It exposes the same methods as the wrapped type, with the same signature. Hence, launching the kernel is simply done by calling the method using the wrapper instance instead of the base instance (or no instance for static methods).
We will start from the Parallel.For version of the code. This expression of parallelism is interpreted by Hybridizer and transformed into a grid stride loop on threads and blocks. Hence adding the EntryPoint attribute should suffice.
Modify 01-naive-gpu.cs to make sure the method runs on GPU.
Should you need, refer to the solution.
Can you write Python code for it?

import platform
if platform.system() == "Windows" : # create directory on Windows
    !mkdir output-03-naive-gpu
if platform.system() == "Linux" : # create directory on Linux
    !mkdir -p ./output-03-naive-gpu

!hybridizer-cuda ./03-naive-gpu/01-naive-gpu.cs graybitmap.cs -intrinsics bitonicsort.cuh=./ -o ./03-naive-gpu/01-naive-gpu.exe -run

# convert bmp to png to have interactive display
from PIL import Image
img = Image.open('./output-03-naive-gpu/denoised.bmp')
img.save('./output-03-naive-gpu/denoised.png', 'png')
from IPython.display import Image
Image(filename="./output-03-naive-gpu/denoised.png", width=384, height=384)


I am working on the file "Jupyter/Labs/04_MedianFilter/IP_MEDIAN_FILTER.ipynb".
The first task is:
Static Sort
We can see that the number of local transactions are many: the local stack buffer has not been placed in registers. Indeed, when sorting the buffer, the size of the buffer is not known statically. We may change this with an alternate sort implementation: a static sort.
We can use a static sort template class with an intrinsic type:
csharp
public class StaticSort
{
    [IntrinsicFunction("::hybridizer::StaticSort<49>::sort<uint16_t>")]
    public static void Sort(ushort[] data)
    {
        Array.Sort(data);
    }
}
In the above code, when calling method Sort running C# with the dot net virtual machine, the Array.Sort method gets called. When Hybridizer processes this method, it replaces the call to StaticSort.Sort by a call to the intrinsic function with same parameters, here: ::hybridizer::StaticSort<49>::sort<uint16_t>, which is a template method of a template type.
Modify 01-regsort-gpu.cs to make use of static sort instead of the bitonic sort.
Should you need, refer to the solution.
Can you write Python code for it?

import platform
if platform.system() == "Windows" : # create directory on Windows
    !mkdir output-06-regsort-gpu
if platform.system() == "Linux" : # create directory on Linux
    !mkdir -p ./output-06-regsort-gpu

!hybridizer-cuda ./06-regsort-gpu/01-regsort-gpu.cs graybitmap.cs -o ./06-regsort-gpu/01-regsort-gpu.exe -run

# convert bmp to png to have interactive display
from PIL import Image
img = Image.open('./output-06-regsort-gpu/denoised.bmp')
img.save('./output-06-regsort-gpu/denoised.png', 'png')
from IPython.display import Image
Image(filename="./output-06-regsort-gpu/denoised.png", width=384, height=384)

!cd 06-regsort-gpu/hybrid ; nvprof --profile-child-processes --metrics local_load_transactions,local_store_transactions,gld_transactions,gst_transactions,l2_read_transactions,l2_write_transactions ./run.sh


I am working on the file "Jupyter/Labs/04_MedianFilter/IP_MEDIAN_FILTER.ipynb".
The first task is:
The number of gld_transactions and l2_read_transactions should significantly reduce.
Can you write Python code for it?

!cd 07-cache-aware-gpu/hybrid ; nvprof -s --profile-child-processes  ./run.sh


I am working on the file "Jupyter/Labs/07_ConjugateGradient/HYB_CUDA_Csharp_ConjugateGradient.ipynb".
The first task is:
Parallelism
Now we can distributing the work on each CPU Thread, for this make use of Parallel.For and ParallelEnumerable with the method Range(Int32, Int32) and Sum(Funct<Int32>) like : ParallelEnumerable.Range(begin, end).Sum(i => compute) when you have to use an atomic add.
Modify 01-parallelism.cs to make the use of parallelism.
If you stuck, have a look at the solution
Can you write Python code for it?

!hybridizer-cuda ./02-Parallel/01-parallelism.cs ./Common_Files/SparseMatrixNaive.cs -o ./02-Parallel/01-parallelism.exe -run


I am working on the file "Jupyter/LoadData.ipynb".
The first task is:
2) Importar scidbpy
pip install git+http://github.com/paradigm4/scidb-py.git@devel
Can you write Python code for it?

import scidbpy
scidbpy.__version__

from scidbpy import connect


I am working on the file "Jupyter/LoadData.ipynb".
The first task is:
Imprime el contenido del arreglo
Can you write Python code for it?

import matplotlib.pyplot as plt
simpleArray = myArray["x"];
simpleArray.plot()
plt.show()

simpleArray


I am working on the file "Jupyter/LoadData.ipynb".
The first task is:
Lee el contenido del archivo donde se encuentran las ondas ya leidas
Can you write Python code for it?

file = open("readedWaves.txt","r")
arrays = dir(sdb.arrays)
filelines = file.readlines()
arrays.extend([w.replace('\n','') for w in filelines])
#type(arrays)
arrays


I am working on the file "Jupyter/LoadData.ipynb".
The first task is:
Normaliza la señal y le quita los valores en null
Can you write Python code for it?

array = wfdb.processing.normalize(x=sig[:, signalII], lb=-2, ub=2)
arrayNun = array[~numpy.isnan(array)]
arrayNun = numpy.trim_zeros(arrayNun)
arrayNun


I am working on the file "Jupyter/LoadData.ipynb".
The first task is:
Pandas (optional)
tested with version 0.15. (0.20.3) Required only for importing/exporting SciDB arrays as Pandas Dataframe objects.
Can you write Python code for it?


I am working on the file "Jupyter/LoadData.ipynb".
The first task is:
Requests
tested with version 2.7 (2.18.1) Required for using the Shim interface to SciDB.
Can you write Python code for it?


I am working on the file "Jupyter/LoadData.ipynb".
The first task is:
SciPy (optional)
tested with versions 0.10-0.12. (0.19.0) Required only for importing/exporting SciDB arrays as SciPy sparse matrices.
Can you write Python code for it?


I am working on the file "Jupyter/LoadData.ipynb".
The first task is:
conectarse al servidor de Base de datos
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
2) Importar scidbpy
pip install git+http://github.com/paradigm4/scidb-py.git@devel
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
3) Leer archivo con cada una de las ondas
Can you write Python code for it?

import urllib.request  # urllib2 in python2 the lib that handles the url stuff
target_url = "https://www.physionet.org/physiobank/database/mimic2wdb/matched/RECORDS-waveforms"
data = urllib.request.urlopen(target_url) # it's a file like object and works just like a file

lines = data.readlines();
line = str(lines[100])


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
4) Importar WFDB para conectarse a physionet
Can you write Python code for it?

import wfdb

sig, fields = wfdb.srdsamp(onda,pbdir='mimic2wdb/matched/'+carpeta) #, sampfrom=11000

print(sig)
print("signame: " + str(fields['signame']))
print("units: " + str(fields['units']))
print("fs: " + str(fields['fs']))
print("comments: " + str(fields['comments']))
print("fields: " + str(fields))


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
Busca la ubicacion de la señal tipo II
Can you write Python code for it?

signalII = None
try:
    signalII = fields['signame'].index("II")
except ValueError:
    print("List does not contain value")
if(signalII!=None):
    print("List contain value")


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
Normaliza la señal y le quita los valores en null
Can you write Python code for it?

array = wfdb.processing.normalize(x=sig[:, signalII], lb=-2, ub=2)
arrayNun = array[~np.isnan(array)]
arrayNun = np.trim_zeros(arrayNun)
arrayNun


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
NumPy
tested with version 1.9 (1.13.1)
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
Quitarle caracteres especiales
Can you write Python code for it?

carpeta,onda = line.replace('b\'','').replace('\'','').replace('\\n','').split("/")
onda


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
Requests
tested with version 2.7 (2.18.1) Required for using the Shim interface to SciDB.
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-II.ipynb".
The first task is:
SciPy (optional)
tested with versions 0.10-0.12. (0.19.0) Required only for importing/exporting SciDB arrays as SciPy sparse matrices.
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
3) Leer archivo con cada una de las ondas
Can you write Python code for it?

import urllib.request  # urllib2 in python2 the lib that handles the url stuff
target_url = "https://physionet.org/physiobank/database/mimic3wdb/matched/RECORDS-waveforms"
data = urllib.request.urlopen(target_url) # it's a file like object and works just like a file

lines = data.readlines();
line = str(lines[2])
line


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
4) Importar WFDB para conectarse a physionet
Can you write Python code for it?

import wfdb

carpeta = "p05/p050140"
onda = "p050140-2188-07-26-05-51"

sig, fields = wfdb.srdsamp(onda,pbdir='mimic3wdb/matched/'+carpeta, sampfrom=10000)

print(sig)
print("signame: " + str(fields['signame']))
print("units: " + str(fields['units']))
print("fs: " + str(fields['fs']))
print("comments: " + str(fields['comments']))
print("fields: " + str(fields))


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
Busca la ubicacion de la señal tipo II
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
Cambiar los guiones "-" por raya al piso "_" porque por algun motivo SciDB tiene problemas con estos caracteres
Si el arreglo sin valores nulos no queda vacio lo sube al SciDB
Can you write Python code for it?

ondaName = onda.replace("-", "_")

if arrayNun.size>0 :
    sdb.input(upload_data=array).store(ondaName,gc=False)
#    sdb.iquery("store(input(<x:int64>[i], '{fn}', 0, '{fmt}'), "+ondaName+")", upload_data=array)


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
NumPy
tested with version 1.9 (1.13.1)
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
Pandas (optional)
tested with version 0.15. (0.20.3) Required only for importing/exporting SciDB arrays as Pandas Dataframe objects.
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
Quitarle caracteres especiales
Can you write Python code for it?

line = line.replace('b\'','').replace('\'','').replace('\\n','')
splited = line.split("/")
splited

carpeta,subCarpeta,onda = line.split("/")
carpeta = carpeta+"/"+subCarpeta
onda


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
Requests
tested with version 2.7 (2.18.1) Required for using the Shim interface to SciDB.
Can you write Python code for it?


I am working on the file "Jupyter/LoadDataMimic-III.ipynb".
The first task is:
conectarse al servidor de Base de datos
Can you write Python code for it?


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
73% accuracy seems good, but what's the null error rate?
Can you write Python code for it?


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
An increase in age, yrs_married, and children appears to correlate with a declining marriage rating.
Data Visualization
Can you write Python code for it?

# show plots in the notebook
%matplotlib inline


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
As you can see, the classifier is predicting a 1 (having an affair) any time the probability in the second column is greater than 0.5.
Now let's generate some evaluation metrics.
Can you write Python code for it?

# generate evaluation metrics
print(metrics.accuracy_score(y_test, predicted))
print(metrics.roc_auc_score(y_test, probs[:, 1]))


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
Data Pre-Processing
First, let's load the dataset and add a binary affair column.
Can you write Python code for it?

# load dataset
dta = sm.datasets.fair.load_pandas().data

# add "affair" column: 1 represents having affairs, 0 represents not
dta['affair'] = (dta.affairs > 0).astype(int)


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
Let's take a look at the distribution of marriage ratings for those having affairs versus those not having affairs.
Can you write Python code for it?

# barplot of marriage rating grouped by affair (True or False)
pd.crosstab(dta.rate_marriage, dta.affair.astype(bool)).plot(kind='bar')
plt.title('Marriage Rating Distribution by Affair Status')
plt.xlabel('Marriage Rating')
plt.ylabel('Frequency')


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
Let's use a stacked barplot to look at the percentage of women having affairs by number of years of marriage.
Can you write Python code for it?

affair_yrs_married = pd.crosstab(dta.yrs_married, dta.affair.astype(bool))
affair_yrs_married.div(affair_yrs_married.sum(1).astype(float), axis=0).plot(kind='bar', stacked=True)
plt.title('Affair Percentage by Years Married')
plt.xlabel('Years Married')
plt.ylabel('Percentage')


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
Looks good. It's still performing at 73% accuracy.
Predicting the Probability of an Affair
Just for fun, let's predict the probability of an affair for a random woman not present in the dataset. She's a 25-year-old teacher who graduated college, has been married for 3 years, has 1 child, rates herself as strongly religious, rates her marriage as fair, and her husband is a farmer.
Can you write Python code for it?


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
Model Evaluation Using Cross-Validation
Now let's try 10-fold cross-validation, to see if the accuracy holds up more rigorously.
Can you write Python code for it?

# evaluate the model using 10-fold cross-validation
scores = cross_val_score(LogisticRegression(), X, y, scoring='accuracy', cv=10)
print(scores)
print(scores.mean())


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
Only 32% of the women had affairs, which means that you could obtain 68% accuracy by always predicting "no". So we're doing better than the null error rate, but not by much.
Let's examine the coefficients to see what we learn.
Can you write Python code for it?

# examine the coefficients
trans = np.transpose(model.coef_)
zipi = list(zip(X.columns, trans))
pd.DataFrame(zipi)


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
Prepare Data for Logistic Regression
To prepare the data, I want to add an intercept column as well as dummy variables for occupation and occupation_husb, since I'm treating them as categorial variables. The dmatrices function from the patsy module can do that using formula language.
Can you write Python code for it?

# create dataframes with an intercept column and dummy variables for
# occupation and occupation_husb
y, X = dmatrices('affair ~ rate_marriage + age + yrs_married + children + \
                  religious + educ + C(occupation) + C(occupation_husb)',
                  dta, return_type="dataframe")
print(X.columns)


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
The column names for the dummy variables are ugly, so let's rename those.
Can you write Python code for it?

# fix column names of X
X = X.rename(columns = {'C(occupation)[T.2.0]':'occ_2',
                        'C(occupation)[T.3.0]':'occ_3',
                        'C(occupation)[T.4.0]':'occ_4',
                        'C(occupation)[T.5.0]':'occ_5',
                        'C(occupation)[T.6.0]':'occ_6',
                        'C(occupation_husb)[T.2.0]':'occ_husb_2',
                        'C(occupation_husb)[T.3.0]':'occ_husb_3',
                        'C(occupation_husb)[T.4.0]':'occ_husb_4',
                        'C(occupation_husb)[T.5.0]':'occ_husb_5',
                        'C(occupation_husb)[T.6.0]':'occ_husb_6'})


I am working on the file "Jupyter/Logistic.ipynb".
The first task is:
We also need to flatten y into a 1-D array, so that scikit-learn will properly understand it as the response variable.
Can you write Python code for it?

# flatten y into a 1-D array
y = np.ravel(y)


I am working on the file "Jupyter/MimicII/0a Fill Database WaveForm Headers.ipynb".
The first task is:
3) Limpiamos los caracteres extraños y Dividimos la cadena donde pXXNNNN-YYYY-MM-DD-hh-mm donde XXNNNN es el identificador unico del paciente SUBJECT_ID y YYYY-MM-DD-hh-mm es la fecha de la estadia del paciente'
Can you write Python code for it?

line = line.replace('b\'','').replace('\'','').replace('\\n','')
splited = line.split("/")
carpeta,onda = line.split("/")
subject_id = carpeta.replace('s','')
recordDate = onda.replace(carpeta+"-","")
print("subject_id: ",subject_id)
print("recordDate: ",recordDate)
print("onda: ",onda)
print("carpeta: ",carpeta)


I am working on the file "Jupyter/MimicII/0a Fill Database WaveForm Headers.ipynb".
The first task is:
4) Leemos el encabezado del waveform, para obtener la información del paciente que almacenaremos
Can you write Python code for it?

try:
    sig, fields = wfdb.srdsamp(onda,pbdir='mimic2wdb/matched/'+carpeta, sampto=1)
    print(fields)
except Exception as inst:
    print("onda vacia")


I am working on the file "Jupyter/MimicII/0a Fill Database WaveForm Headers.ipynb".
The first task is:
Creamos la tabla donde quedaran almacenados los datos
Can you write Python code for it?

table = "waveformFields"
#cur.execute("DROP TABLE "+table)
cur.execute('''CREATE TABLE IF NOT EXISTS waveformFields
             (id serial PRIMARY KEY,
            comments character varying(255)[],
            fs integer, signame character varying(255)[],
            units character varying(255)[],
            subject_id integer,
            recordDate character varying(255),
            database character varying(50))''')


I am working on the file "Jupyter/MimicII/0a Fill Database WaveForm Headers.ipynb".
The first task is:
Hacemos commit
Can you write Python code for it?