I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Dans ce manuel, j'utiliserai l'interface qui me semblera approprié.
Limites sur les axes
Matplotlib gère automatiquement les limites des axes pour présenter toutes les données. Mais vous désirerez peut-être ne présenter qu'une partie de vos données.
Can you write Python code for it?

import datetime # module pour gérer des dates

plt.plot(sherb.time, sherb.temp)
plt.xlabel('Time')
plt.ylabel('Temperature')
plt.xlim([datetime.date(2017, 8, 2), datetime.date(2017, 8, 3)]) # sous forme de liste [min, max] ou de tuple (min, max)
plt.ylim([10, 30])
plt.gcf().autofmt_xdate()


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
De la même manière, une information peut être utilisée pour segmenter la régression dans une grille de graphique (on parle alors de trellis ou de facet). La segmentation des graphiques en lignes se fera avec l'argument row, et celle par colonne avec l'argument col (les deux peuvent être utilisées de paire).
Can you write Python code for it?

sns.lmplot(x='petal_width', y='petal_length', col='species', data=iris)


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Enregistrer un graphique
Certains graphiques sont destinés à rester dans un notebook, d'autres devront être exportés pour être intégrer dans d'autres médias. Vous pouvez exporter les figures avec un clic droit, puis enregistrer l'image..., mais la fonction savefig() permettra davantage d'options, comme exporter un graphique avec une haute résolution avec l'argument dpi (dots per inch).
Can you write Python code for it?

plt.plot(iris.sepal_length, iris.sepal_width, 'o')
plt.savefig('images/ma_figure.png', dpi=300)


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Il faut d'abord invoquer FacetGrid, identifier la manière de segmenter (hue pour segmenter en différentes couleurs, row et col pour segmenter en lignes et en colonnes, respectivement), puis d'identifier le graphique à cartographier dans la segmentation, ici un distplot. Au lieu d'utiliser figsize pour contrôler la dimension du graphique, avec FacetGrid l'argument size définit la dimension du graphique et l'argument aspect défini le ratio largeur / longueur.
Distributions multivariées
La plupart des données écologiques sont multivariées, c'est-à-dire qu'ils contiennent plusieurs variables. Dans le cas de visualisations bivariées, il est commun de présenter un graphique présentant la relation entre deux variables, ainsi que les distributions marginales. Nous parlons alors de joint plots.
Can you write Python code for it?

sns.jointplot(data=iris, x='petal_width', y='petal_length') # exercice: ajouter l'argument kind='kde'


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
J'ai utilisé le format d'image png, utile pour les images de type graphique, avec des changements de couleurs drastiques. Pour les photos, vous préférerez le format jpg. Des éditeurs demandront peut-être des formats vectoriels comme pdf ou eps. Si vous désirez éditer vos graphiques (bouger des étiquettes ou des légendes, ajouter des éléments graphiques), vous pouvez exporter voter graphique en format svg, puis l'éditer dans le logiciel libre Inkscape. Quoi qu'il en soit, voici les format d'exportation disponibles.
Can you write Python code for it?


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
La bande de part et d'autre de la droite est une région de confiance de la régression. Nous verrons comment interpréter cette région dans le prochain chapitre.
Le paradoxe de Simpson stipule que la tendance d'un phénomène peut être radicalement différente lorsqu'on y ajoute une information catégorielle. En effet, pour les données d'iris, la relation entre les variables de dimensions des pétales devrait être différente selon l'espèce. C'est ici qu'entre en jeu l'aspect déclaratif de Seaborn. En mode impératif, nous générerions une ligne par espèce. Mais en mode déclaratif, nous utilisons l'information d'une colonne pour segmenter le graphique en différents éléments. Par exemple, nous utilisons l'information de l'espèce pour segmenter la régression de lmplot selon  différentes couleurs, spécifiées par l'argument hue.
Can you write Python code for it?

sns.lmplot(x='petal_width', y='petal_length', hue='species', data=iris)


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Lançons le graphique avec plt.plot, qui est une fonction générique. Les commandes sont expliquées en commentaire.
Can you write Python code for it?

plt.plot(sherb.time, sherb.temp) # température
plt.plot(sherb.time, sherb.temp_dew) # point de rosée
plt.xlabel('Time') # Titre de l'axe des x
plt.ylabel('Temperature / Dew point') # Titre de l'axe des y
plt.gcf().autofmt_xdate() # format automatique de date pour l'axe des x


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Le graphique affiche la corrélation de Pearson ainsi que la p-value, des statistiques qui seront couvertes au prochain chapitre. Le diagramme en paires (pairs plot) est une généralisation du joint plot. Les colonnes non-numériques sont ignorées.
Can you write Python code for it?

sns.pairplot(iris, hue="species") # essayez d'ajouter l'argument diag_kind="kde"


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Le style seaborn-dark-palette me semble visuellement plaisant.
Can you write Python code for it?


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Légendes
Une légende permet d'associer une décoration graphique (forme / couleur) à une catégorie. Prenons par exemple des données de dimension de sépales de différentes variétés d'iris, données que l'on retrouve dans le module Seaborn, que nous couvrirons plus tard.
Can you write Python code for it?

import seaborn as sns
iris = sns.load_dataset("iris")
iris.sample(3)

plt.plot(iris.loc[iris.species == 'setosa', 'sepal_length'],
         iris.loc[iris.species == 'setosa', 'sepal_width'],
         'o', # seulement un point
         label='Setosa') # étiquette pour la légende
plt.plot(iris.loc[iris.species == 'versicolor', 'sepal_length'],
         iris.loc[iris.species == 'versicolor', 'sepal_width'],
        'o',
         label='Versicolor')
plt.plot(iris.loc[iris.species == 'virginica', 'sepal_length'],
         iris.loc[iris.species == 'virginica', 'sepal_width'],
        'o',
         label='Virginica')
plt.axis('equal') # les distances des unités en x et y sont égales
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')
plt.legend(loc='lower right'); # afficher la légende, selon les labels, à la position donnée.
                               # truc: Pour une position automatique, inscrivez loc='best'


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Matplotlib dans un notebook
À moins de travailler dans une console Python ou IPython, vous n'aurez pas besoin de lancer plt.show() dans un notebook pour afficher un graphique. Pour les graphiques soient affichés dans un notebook, la commande %matplotlib inline doit être préalablement lancée, peu importe si vous utilisez matplotlib, seaborn, pandas ou tout module graphique basé sur Matplotlib. La commande alterntive %matplotlib notebook peut être utilisée pour des graphiques plus intéractifs, mais en ce qui me concerne, le résultat n'est pas très convivial.
Can you write Python code for it?


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Matplotlib: les principes
Disons que vous désiriez créer un graphique de la température et du point de rosée en fonction du temps. Le fichier weather_sherbrooke.csv contient ces variables pour la première semaine du mois d'août 2017 à Sherbrooke, QC.
Can you write Python code for it?

sherb = pd.read_csv('data/weather_sherbrooke.csv', parse_dates=['time'])


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Projetons d'abord les coordonnées x et y sur un graphique. J'utilise FacetGrid ici, sachant que ce sera utile pour l'exploration.
Can you write Python code for it?

sns.FacetGrid(data=datasaurus, size=5).\
    map(plt.plot, 'x', 'y', marker='o', linestyle='None', alpha=0.5)


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Seaborn
Les statistiques forment un pilier de l'écologie mathématique. La visualisation statistique permet d'interpréter et de communiquer vos résultats. Seaborn automatise la prodution des graphiques Matplotlib: quelques instructions suffisent pour créer des graphiques visuellement attrayants, prêts à être exportés pour la production de rapports et de publications scientifiques. Seaborn fonctionne de paire avec Pandas, le module de gestion de données que nous avons couvert au chapitre précédent. Seaborn est conventionnellement importé avec l'alias sns (au cas où vous voudriez savoir pourquoi).
Can you write Python code for it?


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Styliser Matplotlib
De nombreux styles ont été développés pour modifier le style par défaut de Matplotlib.
Can you write Python code for it?


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Voilà! Fait intéressant, ni les statistiques, ni les algorithmes de regroupement ne nous auraient été utiles pour différencier les groupes.
Autres modules
plotnine
J'ai présenté un aperçu de Seaborn, étant donné que ce module est largement utilisé en Python. Mais lorsque vient le temps de créer des graphiques, je préfère plotnine (étant à l'aise avec ggplot2 sur R).
Can you write Python code for it?

from plotnine import *
ggplot(data=iris, mapping=aes(x='petal_width', y='petal_length')) +\
    facet_grid('species ~ .') +\
    stat_smooth(aes(colour='species'), method='lm') +\
    geom_point(aes(colour='species'))


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Vous trouverez peut-être fastidieux d'avoir à créer une commande pour chaque catégorie que vous devez préalablement extraire, alors que vous désirez un seul nuage de point, dont la couleur est déterminée par la colonne species. Vous avez raison. C'est le désavantage de travailler en mode impératif. Nous verrons plus loin comment travailler en mode déclaratif.
Annoter une figure
Si vous avez à annoter une figure, vous utiliserez la fonction plt.annotate.
Can you write Python code for it?

plt.plot(iris.sepal_length, iris.sepal_width, 'o')
plt.annotate('Setosa', #texte
             xytext=[4.5, 4.3], # s'il y a une flèche, emplacement du départ de la flèche et du texte
             xy=[5.0, 3.5], # endroit où pointe la flèche, ou s'il n'y a pas de flèche emplacement du texte
             arrowprops={'color': 'gray'}) # propriétés de la flèche


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
À venir

Séries temporelles
Cartes
Barres

<div class="alert alert-block alert-info">**Exporter un graphique**. Pour envoyer un graphique à un.e collègue ou pour l'insérer dans un document, vous voudrez sans doute libérer votre graphique de votre carnet de calcul. Rapidement, avec un clic-droit, vous pouvez copier le graphique dans le presse-papier de votre environnement de travail, puis le coller où vous voulez. *altair* vous offre d'exporter le graphique en format PNG, qui est un format bitmap (pixels). Mais ces deux options vous donneront des graphiques de basse résolution. Pour une résolution supérieure, choisissez "Open in Vega Editor". En haut à droite, sélectionnez le format SVG, puis téléchargez. Ce fichier est un format vectoriel (formes) qui peut être importé dans un logiciel de dessin (je vous suggère le logiciel libre [Inkscape](https://inkscape.org)), puis exporté en bitmap à haute résolution ou en format vectoriel EPS ou PDF. Par ailleurs, un bon logiciel de dessin vectoriel, comme Inkscape, vous permettra d'éditer le graphique (par exemple, déplacer la légende) ou de l'annoter.
</div>

Les graphiques comme outil d'exploration des données
La plupart des graphiques que vous créerez ne seront pas destinés à être publiés, mais serviront d'outil d'exploration des données. Le jeu de données datasaurus, présenté en début de section, permet de saisir l'importance des outils graphiques pour bien comprendre les données.
Can you write Python code for it?

datasaurus = pd.read_csv('data/DatasaurusDozen.tsv', sep='\t')
datasaurus.sample(5)


I am working on the file "04_analysis.ipynb".
The first task is:
Answer: [Write your answer text here]
Task 4
Copy the scoring function score_ntn from Task 4 of mini-assignment 3. Rename it to score_ntn_pagerank and change its code to incorporate a paper's PageRank score in it's final score, in addition to tf-idf. In other words, the new function should return a single value that is calculated based on both scores (PageRank and tf-idf). Explain your decision on how to combine the two scores.
Can you write Python code for it?


I am working on the file "04_analysis.ipynb".
The first task is:
Answer: [Write your answer text here]
Task 5
Copy the query function query_ntn from Task 4 of mini-assignment 3. Rename it to query_ntn_pagerank and change the code to use our new scoring function score_ntn_pagerank from task 4 above. Demonstrate these functions with an example query that returns our paper 10399593 from above as the top result.
Can you write Python code for it?


I am working on the file "04_analysis.ipynb".
The first task is:
As this graph was generated from citations only, we need to add all isolated nodes (nodes that are not cited and do not cite other papers) as well:
Can you write Python code for it?


G.add_nodes_from(isolated)

print(nx.info(G))
print(nx.is_directed(G))
print(nx.density(G))


I am working on the file "04_analysis.ipynb".
The first task is:
Assignments
Your name: ...

Plot the in-degree distribution (the distribution of the number of incoming links) for the citation network. What can you tell about the shape of this distribution, and what does this tell us about the network?
Can you write Python code for it?


I am working on the file "04_analysis.ipynb".
The first task is:
Citations network
Next, we can look at the citation network. We'll start by expanding the our data about citations into two mappings:

papers_citing[id]: papers citing a given paper
cited_by[id]: papers cited by a given paper (in other words: its list of references)

papers_citing will give us the list of a node's incoming links, whereas cited_by will give us the list of its outgoing links.
Can you write Python code for it?

papers_citing = Citations  # no changes needed, this is what we are storing already in the Citations dataset

cited_by = defaultdict(list)

for ref, papers_citing_ref in papers_citing.items():
    for id in papers_citing_ref:
        cited_by[ id ].append( ref )

display_summary(24130474)


I am working on the file "04_analysis.ipynb".
The first task is:
Citations network
We'll start by expanding the Citations dataset into two mappings:

papers_citing[id]: papers citing a given paper;
cited_by[id]: papers cited by a given paper (in other words, its list of references).

If we see the Citations dataset as a directed graph where papers are nodes, and citations links between then, then papers_citing gives you the list of a node's incoming links, whereas cited_by gives you the list of its outgoing links.
The dataset was assembled by querying for papers citing a given paper. As a result, the data mapped to in cited_by (its values) is necessarily limited to ids of papers that are part of the dataset.
Can you write Python code for it?

papers_citing = Citations  # no changes needed, this is what we are storing already in the Citations dataset

cited_by = defaultdict(list)

for ref, papers_citing_ref in papers_citing.items():
    for id in papers_citing_ref:
        cited_by[ id ].append( ref )


I am working on the file "04_analysis.ipynb".
The first task is:
Co-authorship network
Summaries maps paper ids to paper summaries. Let us now create here mappings by different criteria.
We'll start by building a mapping from authors to the set of ids of papers they authored.
We'll be using Python's sets again for that purpose.
Can you write Python code for it?

papers_of_author = defaultdict(set)

for id,p in Summaries.items():
    for a in p.authors:
        papers_of_author[a].add( id )

papers_of_author['Vine AK']

for id in papers_of_author['Vine AK']:
    display_summary(id)


I am working on the file "04_analysis.ipynb".
The first task is:
Copy your search engine from mini-assignment 3, and create a version that incorporates a paper's PageRank score in it's final score, in addition to tf-idf. Show the result of an example query, and explain your decision on how to combine the two scores (PageRank and tf-idf).
Can you write Python code for it?


I am working on the file "04_analysis.ipynb".
The first task is:
If we lookup the same paper in papers_citing, we now see that some of the cited papers are themselves in our dataset, but others are not (denote here by '??'):
Can you write Python code for it?

{ id : Summaries.get(id,['??'])[0]  for id in papers_citing[paper_id] }


I am working on the file "04_analysis.ipynb".
The first task is:
Let's try it out:
Can you write Python code for it?

papers_of_author['Clauset A']

for id in papers_of_author['Clauset A']:
    display_summary(id)


I am working on the file "04_analysis.ipynb".
The first task is:
Link Analysis for Search Engines
In order to use the citation network, we need to be able to perform some complex graph algorithms on it. To make our lives easier, we will load the data into the python package NetworkX, a package for the creation, manipulation, and study of the structure, dynamics, and function of complex networks, which provides a number of these graph algorithms (such as HITS and PageRank) out of the box.
You probably have to install the NetworkX package first.
Can you write Python code for it?

import networkx as nx

G = nx.DiGraph(cited_by)


I am working on the file "04_analysis.ipynb".
The first task is:
Link Analysis for Search Engines
In order to use the citation network, we need to be able to perform some complex graph algorithms on it. To make our lives easier, we will use NetworkX, a Python package for dealing with complex networks. You might have to install the NetworkX package first.
Can you write Python code for it?


I am working on the file "04_analysis.ipynb".
The first task is:
Now that we have a better understanding about the data we're dealing with, let us obtain again some basic statistics about our graph.
Can you write Python code for it?

n = len(Ids)
print('Number of papers in our subset: %d (%.2f %%)' % (n, 100.0) )

with_citation = [ id for id in Ids if papers_citing[id] != [] ]
with_citation_rel = 100. * len(with_citation) / n
print('Number of papers cited at least once: %d (%.2f %%)' % (len(with_citation), with_citation_rel) )

isolated = set( id for id in Ids if papers_citing[id] == [] and id not in cited_by )
isolated_rel = 100. * len(isolated) / n
print('Number of isolated nodes: %d (%.2f %%)' % (len(isolated), isolated_rel) )

id_set = set( Ids )
citing_set = set( cited_by.keys() )

outsiders = citing_set - id_set   # set difference
nodes = citing_set | id_set   # set union
non_isolated = nodes - isolated   # set difference

print('Overall number of nodes: %d (%.2f %%)' % (len(nodes), 100.0) )

non_isolated_rel = 100. * len(non_isolated) / len(nodes)
print('Number of non-isolated nodes: %d (%.2f %%)' % (len(non_isolated), non_isolated_rel) )

outsiders_rel = 100. * len(outsiders) / len(nodes)
print('Number of nodes outside our subset: %d (%.2f %%)' % ( len(outsiders), outsiders_rel ) )

all_citations = [ c for citing in papers_citing.values() for c in citing ]
outsider_citations = [ c for citing in papers_citing.values() for c in citing if c in outsiders ]

print('Overal number of links (citations): %d (%.2f %%)' % (len(all_citations), 100.0) )

outsider_citations_rel = 100. * len(outsider_citations) / len(all_citations)
print('Citations from outside the subset: %d (%.2f %%)' % (len(outsider_citations), outsider_citations_rel) )


print('Number of core ids %d (100.00 %%)' % len(Ids))

with_cit = [ id for id in Ids if papers_citing[id]!=[] ]
print('Number of papers cited at least once: %d (%.2f %%)' % (len(with_cit), 100.*len(with_cit)/len(Ids)))

isolated = set( id for id in Ids if papers_citing[id]==[] and id not in cited_by )
print('Number of isolated nodes: %d (%.2f %%)\n\t'   \
      '(papers that are not cited by any others, nor do themselves cite any in the dataset)'% (
    len(isolated), 100.*len(isolated)/len(Ids) ))

noCit_withRefs = [ id for id in Ids if papers_citing[id]==[] and id in cited_by ]
print('Number of dataset ids with no citations, but known references: %d (%.2f %%)' % (
    len(noCit_withRefs), 100.*len(noCit_withRefs)/len(Ids)))

print('(percentages calculated with respect to just the core ids (members of `Ids`) -- exclude outsider ids)\n')

Ids_set    = set( Ids )
citing_Ids = set( cited_by.keys() ) # == set( c for citing in papers_citing.itervalues() for c in citing )

outsiders = citing_Ids - Ids_set    # set difference: removes from `citing_Ids` all the ids that occur in `Ids_set`
nodes     = citing_Ids | Ids_set - isolated     # set union, followed by set difference

print('Number of (non-isolated) nodes in the graph: %d\n\t(papers with at least 1 known citation, or 1 known reference)' % len(nodes))
print(len( citing_Ids ), 'distinct ids are citing papers in our dataset.')
print('Of those, %d (%.2f %%) are ids from outside the dataset.\n' % ( len(outsiders), 100.*len(outsiders)/len(citing_Ids) ))

all_cits      = [ c for citing in papers_citing.values() for c in citing ]
outsider_cits = [ c for citing in papers_citing.values() for c in citing if c in outsiders ]

print('Number of links (citations) in the graph:', len(all_cits))
print('A total of %d citations are logged in the dataset.' % len(all_cits))
print('Citations by ids from outside the dataset comprise %d (%.2f %%) of that total.\n' % (
    len(outsider_cits),
    100.*len(outsider_cits)/len(all_cits) ))


I am working on the file "04_analysis.ipynb".
The first task is:
Now we can have a look at some basic statistics about our graph:
Can you write Python code for it?

print('Number of nodes: %8d (node = author)' % len(coauthors))
print('Number of links: %8d (link = collaboration between the two linked authors on at least one paper)'  \
        % sum( len(cas) for cas in coauthors.values() ))


print('Number of nodes (authors): ', len(coauthors))

coauthor_rel_count = sum( len(c) for c in coauthors.values() )
print('Number of links (co-authorship relations): ', coauthor_rel_count)


I am working on the file "04_analysis.ipynb".
The first task is:
Paper 17696886, for example, is not in our dataset and we do not have any direct information about it, but its repeated occurrence in other papers' citation lists does allow us to reconstruct a good portion of its references. Below is the list of papers in our dataset cited by that paper:
Can you write Python code for it?

paper_id2 = 17696886
refs2 = { id : Summaries[id].title for id in cited_by[paper_id2] }
print(len(refs2), 'references identified for the paper with id', paper_id2)
refs2


I am working on the file "04_analysis.ipynb".
The first task is:
Task 3
Why do the two papers above have such different PageRank values? Write code below to investigate and show the cause of this, and then explain the cause of this difference based on the results generated by your code.
Can you write Python code for it?


I am working on the file "04_analysis.ipynb".
The first task is:
We can now build a co-authorship network, that is a graph linking authors to the set of co-authors they have published with:
Can you write Python code for it?

coauthors = defaultdict(set)

for p in Summaries.values():
    for a in p.authors:
        coauthors[a].update(p.authors)

# The code above results in each author being listed as having co-authored with himself/herself.
# We remove these self-references here:
for (a, ca) in coauthors.items():
    ca.remove(a)


I am working on the file "04_analysis.ipynb".
The first task is:
We now have a NetworkX Directed Graph stored in G, where a node represents a paper, and an edge represents a citation. This means we can now apply the algorithms and functions of NetworkX to our graph:
Can you write Python code for it?

print(nx.info(G))
print('Directed graph:', nx.is_directed(G))
print('Density of graph:', nx.density(G))


I am working on the file "04_analysis.ipynb".
The first task is:
With this data in hand, we can plot the degree distribution by showing the number of collaborators a scientist has published with:
Can you write Python code for it?

plt.hist( x=[ len(ca) for ca in coauthors.values() ], bins=range(55), histtype='bar', align='left', normed=True )
plt.xlabel('number of collaborators')
plt.ylabel('fraction of scientists')
plt.xlim(0,50);


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Create blend
Can you write Python code for it?

# create XGBoost blend
pred_blender('../data',
            ['submission_Thu_Jun_15_15_58_31_2017.csv',
             'submission_Thu_Jun_15_16_01_59_2017.csv',
             'submission_Thu_Jun_15_16_27_42_2017.csv',
             'submission_Thu_Jun_15_17_07_26_2017.csv'])
# 0.14705 on public leaderboard


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Define ranked predictions plot function
Can you write Python code for it?

def ranked_preds_plot(y, valid, preds):

    """ Generates ranked prediction plot.

    :param y: Name of target variable.
    :param valid: Name of validation H2OFrame.
    :param preds: Column vector of predictions to plot.

    """

    # plot top frame values
    preds.columns = ['predict']
    yhat_frame = valid.cbind(preds)
    print(yhat_frame[0:10, [y, 'predict']])

    # plot sorted predictions
    yhat_frame_df = yhat_frame[[y, 'predict']].as_data_frame()
    yhat_frame_df.sort_values(by='predict', inplace=True)
    yhat_frame_df.reset_index(inplace=True, drop=True)
    _ = yhat_frame_df.plot(title='Ranked Predictions Plot')


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Determine data types
Can you write Python code for it?

def get_type_lists(frame=train, rejects=['Id', 'SalePrice']):

    """Creates lists of numeric and categorical variables.

    :param frame: The frame from which to determine types.
    :param rejects: Variable names not to be included in returned lists.
    :return: Tuple of lists for numeric and categorical variables in the frame.

    """

    nums, cats = [], []
    for key, val in frame.types.items():
        if key not in rejects:
            if val == 'enum':
                cats.append(key)
            else:
                nums.append(key)

    print('Numeric =', nums)
    print()
    print('Categorical =', cats)

    return nums, cats

original_nums, cats = get_type_lists()


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Execute encoding
Can you write Python code for it?

total = len(cats)
for i, var in enumerate(cats):

    tr_enc, _ = target_encoder(train, test, var, 'SalePrice')
    v_enc, ts_enc = target_encoder(valid, test, var, 'SalePrice')

    print('Encoding: ' + var + ' (' + str(i+1) + '/' + str(total) + ') ...')

    train = train.cbind(tr_enc)
    valid = valid.cbind(v_enc)
    test = test.cbind(ts_enc)

print('Done.')


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Function to generate submission file
Can you write Python code for it?

import re
import time

def gen_submission(preds, test=test):

    """ Generates submission file for Kaggle House Prices contest.

    :param preds: Column vector of predictions.
    :param test: Test data.

    """

    # create time stamp
    time_stamp = re.sub('[: ]', '_', time.asctime())

    # create predictions column
    sub = test['Id'].cbind(preds.exp())
    sub.columns = ['Id', 'SalePrice']

    # save file for submission
    sub_fname = '../data/submission_' + str(time_stamp) + '.csv'
    h2o.download_csv(sub, sub_fname)


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
H2O GBM model
Can you write Python code for it?

# initialize H2O GBM
h2o_gbm_model = H2OGradientBoostingEstimator(
    ntrees = 10000,
    learn_rate = 0.005,
    sample_rate = 0.1,
    col_sample_rate = 0.8,
    max_depth = 5,
    nfolds = 3,
    keep_cross_validation_predictions=True,
    stopping_rounds = 10,
    seed = 12345)

# execute training
h2o_gbm_model.train(x=encoded_combined_nums,
                    y='SalePrice',
                    training_frame=train,
                    validation_frame=valid)

# print model information/create submission
print(h2o_gbm_model)
h2o_gbm_preds1_val = h2o_gbm_model.predict(valid)
ranked_preds_plot('SalePrice', valid, h2o_gbm_preds1_val) # better validation error
h2o_gbm_preds1_test = h2o_gbm_model.predict(test)
gen_submission(h2o_gbm_preds1_test) # 0.15062 public leaderboard


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Import data
Can you write Python code for it?

train = h2o.import_file('../../03_regression/data/train.csv')
test = h2o.import_file('../../03_regression/data/test.csv')

# bug fix - from Keston
dummy_col = np.random.rand(test.shape[0])
test = test.cbind(h2o.H2OFrame(dummy_col))
cols = test.columns
cols[-1] = 'SalePrice'
test.columns = cols
print(train.shape)
print(test.shape)


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Impute numeric missing
Can you write Python code for it?

# median usually better than mean
# (_ signifies temporary throw-away variable, used to suppress output)
_ = train[['MasVnrArea', 'GarageYrBlt', 'LotFrontage']].impute(method='median')
_ = valid[['MasVnrArea', 'GarageYrBlt', 'LotFrontage']].impute(method='median')
_ = test[['BsmtHalfBath', 'BsmtFinSF1', 'BsmtFullBath', 'BsmtFinSF2', 'BsmtUnfSF', 'MasVnrArea',
          'GarageYrBlt', 'LotFrontage', 'GarageCars', 'TotalBsmtSF', 'GarageArea']].impute(method='median')


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Random forest model - typically not tuned as much as GBM
Can you write Python code for it?

# initialize rf model
rf_model1 = H2ORandomForestEstimator(
    ntrees=10000,
    max_depth=10,
    col_sample_rate_per_tree=0.1,
    sample_rate=0.8,
    stopping_rounds=50,
    score_each_iteration=True,
    nfolds=3,
    keep_cross_validation_predictions=True,
    seed=12345)

# train rf model
rf_model1.train(
    x=encoded_combined_nums,
    y='SalePrice',
    training_frame=train,
    validation_frame=valid)

# print model information
print(rf_model1)

rf_preds1_val = rf_model1.predict(valid)
ranked_preds_plot('SalePrice', valid, rf_preds1_val) # valid RMSE not so hot ...
rf_preds1_test = rf_model1.predict(test)
gen_submission(rf_preds1_test) # 0.14574 public leaderboard


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Redefine numerics and explore
Can you write Python code for it?

encoded_nums, cats = get_type_lists(frame=train)

print('Imputed and encoded numeric training data:')
train[encoded_nums].describe() #79 numeric columns w/ no missing
print('--------------------------------------------------------------------------------')
print('Imputed and encoded numeric validation data:')
valid[encoded_nums].describe() #79 numeric columns w/ no missing
print('--------------------------------------------------------------------------------')
print('Imputed and encoded numeric test data:')
test[encoded_nums].describe() #79 numeric columns w/ no missing

# Check Neighborhood_Tencode

print(test[0:5, ['Neighborhood', 'Neighborhood_Tencode']])
_, _ = target_encoder(valid, test, 'Neighborhood', 'SalePrice', test=True)
del _

# NAmes   152080
# NAmes   152080
# Gilbert 191808
# Gilbert 191808
# StoneBr 288359


encoded_combined_nums, cats = get_type_lists(frame=train)

# check number of created variables is correct
# 1 id column, 1 target column, 79 original + encoded numeric columns, 43 original categorical variables
# sum(range(1, 79)) combined variables
print(train.shape == (1001, sum(range(1, 79), (79 + 43 + 1 + 1))))
print(test.shape == (1459, sum(range(1, 79), (79 + 43 + 1 + 1))))

# check multiplication for a random column
ridx = np.random.choice(sum(range(1, 79)))
combined_only = [name for name in encoded_combined_nums if name not in encoded_nums]
combined_check_vars = combined_only[ridx].split('|')
combined_check_vars.append(combined_only[ridx])

print(train[736, combined_check_vars])
print(test[637, combined_check_vars])

print(train[736, combined_check_vars[0]]*train[736, combined_check_vars[1]])
print(test[637, combined_check_vars[0]]*test[637, combined_check_vars[1]])


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Shutdown H2O
Can you write Python code for it?

# Shutdown H2O - this will erase all your unsaved frames and models in H2O
h2o.cluster().shutdown(prompt=True)


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Simple prediction blending function
Can you write Python code for it?

import os

def pred_blender(dir_, files):

    """ Performs simple blending of prediction files.

    :param dir_: Directory in which files to be read are stored.
    :param files: List of prediction files to be blended.

    """

    # read predictions in files list and cbind
    for i, file in enumerate(files):
        if i == 0:
            df = pd.read_csv(dir_ + os.sep + file).drop('SalePrice', axis=1)
        col = pd.read_csv(dir_ + os.sep + file).drop('Id', axis=1)
        col.columns = ['SalePrice' + str(i)]
        df = pd.concat([df, col], axis=1)

    # create mean prediction
    df['mean'] = df.iloc[:, 1:].mean(axis=1)
    print(df.head())

    # create time stamp
    time_stamp = re.sub('[: ]', '_', time.asctime())

    # write new submission file
    df = df[['Id', 'mean']]
    df.columns = ['Id', 'SalePrice']

    # save file for submission
    sub_fname = '../data/submission_' + str(time_stamp) + '.csv'
    df.to_csv(sub_fname, index=False)


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Split into to train and validation (before doing data prep!!!)
Can you write Python code for it?

train, valid = train.split_frame([0.7], seed=12345)
print(train.shape)
print(valid.shape)


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Train H2O stacked ensemble
Can you write Python code for it?

stack = H2OStackedEnsembleEstimator(training_frame=train,
                                    validation_frame=valid,
                                    base_models=[rf_model1, ert_model1,
                                                 h2o_gbm_model])

stack.train(x=encoded_combined_nums,
            y='SalePrice',
            training_frame=train,
            validation_frame=valid)

# print model information/create submission
print(stack)
stack_preds1_val = stack.predict(valid)
ranked_preds_plot('SalePrice', valid, stack_preds1_val)
stack_preds1_test = stack.predict(test)
gen_submission(stack_preds1_test)
# 0.14630 on public leaderboard


I am working on the file "04_decision_trees/src/py_part_4_kaggle_xgboost.ipynb".
The first task is:
Train models
Can you write Python code for it?

h2o.show_progress() # turn on progress bars

# Check log transform - looks good
%matplotlib inline
train['SalePrice'].log().as_data_frame().hist()

# Execute log transform
train['SalePrice'] = train['SalePrice'].log()
valid['SalePrice'] = valid['SalePrice'].log()
print(train[0:3, 'SalePrice'])


I am working on the file "04_detect_segment/04a_retinanet_anthropods_download.ipynb".
The first task is:
Create API Key on Kaggle

Please go https://www.kaggle.com/ and go to your Account profile page
Click on "Create New API Token"
Upload the kaggle.json to this directory.
Can you write Python code for it?

!ls ./kaggle.json

import os
current_dir=!pwd
current_dir=current_dir[0]
os.environ['KAGGLE_CONFIG_DIR']=current_dir

!${HOME}/.local/bin/kaggle datasets download mistag/arthropod-taxonomy-orders-object-detection-dataset

!unzip -q *dataset.zip


I am working on the file "04_detect_segment/04ab_retinanet_arthropods_train.ipynb".
The first task is:
Configuration
Can you write Python code for it?

TRAIN_DATA_PATH_PATTERN = 'gs://practical-ml-vision-book/arthropod_detection_tfr/size_w1024px/*.train.tfrec'
VALID_DATA_PATH_PATTERN = 'gs://practical-ml-vision-book/arthropod_detection_tfr/size_w1024px/*.test.tfrec'
SPINET_MOBILE_CHECKPOINT = 'gs://practical-ml-vision-book/arthropod_detection_tfr/spinenet49mobile_checkpoint'

# The spinet mobile 49 checkpoint is published here:
# https://github.com/tensorflow/models/blob/master/official/vision/MODEL_GARDEN.md#mobile-size-retinanet-trained-from-scratch

BATCH_SIZE = 32 * strategy.num_replicas_in_sync

EPOCHS = 80

RAW_CLASSES = ['Lepidoptera', 'Hymenoptera', 'Hemiptera', 'Odonata', 'Diptera', 'Araneae', 'Coleoptera',
               '_truncated', '_blurred', '_occluded', ]
CLASSES = [klass for klass in RAW_CLASSES if klass not in ['_truncated', '_blurred', '_occluded']]

# Lepidoptera = butterfies and moths
# Hymenoptera = wasps, bees and ants
# Hemiptera = true bugs (cicadas, aphids, shield bugs, ...)
# Odonata = dragonflies
# Diptera = fies
# Araneae = spiders
# Coleoptera = beetles

# NOT IN DATASET
# Orthoptera = grasshoppers

print("Model dir:", MODEL_DIR)


I am working on the file "04_detect_segment/04ab_retinanet_arthropods_train.ipynb".
The first task is:
Create the model
Can you write Python code for it?

task = tfm.core.task_factory.get_task(params.task, logging_dir=MODEL_DIR)

# this works too:
#task = official.vision.beta.tasks.retinanet.RetinaNetTask(params.task)

# this returns a RetinaNetModel
#task.build_model()
# note: none of the expected model functionalities work: model.fit(), model.predict(), model.save()

# this returns the training dataset
#train_dataset = task.build_inputs(train_data_cfg)
# note: the dataset already includes FPN level and anchor pairing and is therefore not very readable

# this returns the validation dataset
#valid_dataset = task.build_inputs(valid_data_cfg)
# note: the dataset already includes FPN level and anchor pairing and is therefore not very readable

# this code allows you to see if the TFRecord fields are read correctly
#ds = tf.data.TFRecordDataset(tf.io.gfile.glob(TRAIN_DATA_PATH_PATTERN))
#dec = official.vision.beta.dataloaders.tf_example_decoder.TfExampleDecoder()
#ds = ds.map(dec.decode)

# training and validatoin data parsing happens in:
# official.vision.beta.dataloaders.retinanet_input.Parser._parse_train_data
# official.vision.beta.dataloaders.retinanet_input.Parser._parse_eval_data
# official.vision.beta.dataloaders.Parser.parse() # dispatches between _parse_train_data and _parse_eval_data


I am working on the file "04_detect_segment/04ab_retinanet_arthropods_train.ipynb".
The first task is:
Export the model
To test the exported model, please use the notebook "04ac_retinanet_arthropods_predict.ipynb"
Can you write Python code for it?

tfm.vision.serving.export_saved_model_lib.export_inference_graph(
      input_type='image_tensor',
      batch_size=4,
      input_image_size=IMAGE_SIZE,
      params=params,
      checkpoint_path=MODEL_DIR,
      export_dir=MODEL_DIR,
      export_checkpoint_subdir='saved_chkpt',
      export_saved_model_subdir='saved_model')


I am working on the file "04_detect_segment/04ab_retinanet_arthropods_train.ipynb".
The first task is:
GCS bucket
This bucket will receive:
 - Tensorboard summaries that allow you to follow the training
 - checkpoints
 - the saved model after training
Can you write Python code for it?

# Use your own GCS bucket here. GCS is required if training on TPU.
# On GPU, a local folder will work.
MODEL_ARTIFACT_BUCKET = 'gs://ml1-demo-martin/arthropod_jobs/'
MODEL_DIR = MODEL_ARTIFACT_BUCKET + str(int(time.time()))

# If you are running on Colaboratory, you must authenticate
# for Colab to have write access to the bucket.

IS_COLAB_BACKEND = 'COLAB_GPU' in os.environ  # this is always set on Colab, the value is 0 or 1 depending on GPU presence
if IS_COLAB_BACKEND:
    from google.colab import auth
    auth.authenticate_user()


I am working on the file "04_detect_segment/04ab_retinanet_arthropods_train.ipynb".
The first task is:
TPU / GPU detection
Can you write Python code for it?

try: # detect TPUs
    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect()
    strategy = tf.distribute.TPUStrategy(tpu)
except ValueError: # detect GPUs or multi-GPU machines
    strategy = tf.distribute.MirroredStrategy()

print("REPLICAS: ", strategy.num_replicas_in_sync)


I am working on the file "04_detect_segment/04ab_retinanet_arthropods_train.ipynb".
The first task is:
Train the model
Training takes approximately 30min on a TPUv3-8, 40min on a TPUv2-8 on Colab
Can you write Python code for it?

print(MODEL_DIR)
model,_ = tfm.core.train_lib.run_experiment(
    distribution_strategy=strategy,
    task=task,
    mode="train_and_eval", # 'train', 'eval', 'train_and_eval' or 'continuous_eval'
    params=params,
    model_dir=MODEL_DIR)


I am working on the file "04_detect_segment/04b_unet_segmentation.ipynb".
The first task is:
Helper functions to see model output as it is training
Can you write Python code for it?

# display helper functions
def display(display_list):
  plt.figure(figsize=(15, 15))

  title = ['Input Image', 'True Mask', 'Predicted Mask']

  for i in range(len(display_list)):
    plt.subplot(1, len(display_list), i+1)
    plt.title(title[i])
    plt.imshow(tf.keras.preprocessing.image.array_to_img(display_list[i]))
    plt.axis('off')
  plt.show()

def show_predictions(dataset, num):
  for image, mask in dataset.take(num):
    pred_mask = model.predict(image)
    display([image[0], mask[0], create_mask(pred_mask)])

# before the model is trained
show_predictions(train.batch(1), 1)


I am working on the file "04_detect_segment/04b_unet_segmentation.ipynb".
The first task is:
Modified U-Net
The model being used here is a modified U-Net.
A U-Net typically consists of an encoder
which downsamples an image to an encoding,
and a mirrored decoder which upsamples the encoding
back to the desired mask. The decoder blocks have a number of skip connections
that directly connect the encoder blocks to the decoder.
Doing this from scratch requires a lot of trainable parameters and is better done using Generative
Adversarial Networks (GANs) which are covered in
Chapter 12.
In this notebook, we will use a pretrained MobileNetV2 to
create the encoding and a set of upsampling layers to
get back to the desired mask.
When doing so, however, we will pull out layers with the
desired sizes so that the upsampling skip layers use
corresponding weights from the pretrained models.
To make the dataset go a little bit farther,
we also augment the training dataset to flip the images.
(data augmentation is covered in Chapter 6)
Note that the output consists of 3 channels. This is because we have 3 possible labels for each pixel (background, outline, interior). Think of this as multi-classification problem with three possible classes.
Can you write Python code for it?

BATCH_SIZE = 64
BUFFER_SIZE = 1000

def augment(img, mask):
  if tf.random.uniform(()) > 0.5:
    img = tf.image.flip_left_right(img)
    mask = tf.image.flip_left_right(mask)
  return img, mask

train_dataset = train.cache().map(augment).shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()
train_dataset = train_dataset.prefetch(buffer_size=tf.data.AUTOTUNE)

test_dataset = test.batch(BATCH_SIZE)

TRAIN_LENGTH = info.splits['train'].num_examples
STEPS_PER_EPOCH = TRAIN_LENGTH // BATCH_SIZE
OUTPUT_CHANNELS = 3

base_model = tf.keras.applications.MobileNetV2(input_shape=[128, 128, 3], include_top=False)

# Use the activations of these layers to form skip connections
layer_names = [
    'block_1_expand_relu',   # 64x64
    'block_3_expand_relu',   # 32x32
    'block_6_expand_relu',   # 16x16
    'block_13_expand_relu',  # 8x8
    'block_16_project',      # 4x4
]
base_model_outputs = [base_model.get_layer(name).output for name in layer_names]

# Create the feature extraction model
down_stack = tf.keras.Model(inputs=base_model.input, outputs=base_model_outputs,
                            name='pretrained_mobilenet')

down_stack.trainable = False


I am working on the file "04_detect_segment/04b_unet_segmentation.ipynb".
The first task is:
Read Oxford-IIIT Pets dataset
The dataset is part of TensorFlow datasets.
Version 3 and higher of the dataset has ground truth
segmentation masks.
The dataset already contains test and train splits.
Can you write Python code for it?

import tensorflow_datasets as tfds
dataset, info = tfds.load('oxford_iiit_pet:3.*.*', with_info=True)


I am working on the file "04_detect_segment/04b_unet_segmentation.ipynb".
The first task is:
The decoder/upsampler is a series of upsample blocks.
Conv2DTranspose is what does the upsampling.
We also add batchnorm and relu.
Can you write Python code for it?

def upsample(filters, size, name):
  return tf.keras.Sequential([
     tf.keras.layers.Conv2DTranspose(filters, size, strides=2, padding='same'),
     tf.keras.layers.BatchNormalization(),
     tf.keras.layers.ReLU()
  ], name=name)

up_stack = [
    upsample(512, 3, 'upsample_4x4_to_8x8'),
    upsample(256, 3, 'upsample_8x8_to_16x16'),
    upsample(128, 3, 'upsample_16x16_to_32x32'),
    upsample(64, 3,  'upsample_32x32_to_64x64')
]

import re

def unet_model(output_channels):
  inputs = tf.keras.layers.Input(shape=[128, 128, 3], name='input_image')

  # Downsampling through the model
  skips = down_stack(inputs)
  x = skips[-1]
  skips = reversed(skips[:-1])

  # Upsampling and establishing the skip connections
  for idx, (up, skip) in enumerate(zip(up_stack, skips)):
    x = up(x)
    concat = tf.keras.layers.Concatenate(name='expand_{}'.format(idx))
    x = concat([x, skip])

  # This is the last layer of the model
  last = tf.keras.layers.Conv2DTranspose(
      output_channels, 3, strides=2,
      padding='same')  #64x64 -> 128x128

  x = last(x)

  return tf.keras.Model(inputs=inputs, outputs=x)

model = unet_model(OUTPUT_CHANNELS)
tf.keras.utils.plot_model(model, show_shapes=True)


I am working on the file "04_detect_segment/04b_unet_segmentation.ipynb".
The first task is:
To observe how the model improves while it is training we define a callback function to show the predictions when the epoch finishes.
Can you write Python code for it?

# from IPython.display import clear_output
class DisplayCallback(tf.keras.callbacks.Callback):
  def on_epoch_end(self, epoch, logs=None):
    if epoch%5 == 0:
      # clear_output(wait=True) # if you want replace the images each time, uncomment this
      show_predictions(train.batch(1), 1)
      print ('\nSample Prediction after epoch {}\n'.format(epoch+1))

EPOCHS = 20
VAL_SUBSPLITS = 5
VALIDATION_STEPS = info.splits['test'].num_examples//BATCH_SIZE//VAL_SUBSPLITS

model_history = model.fit(train_dataset, epochs=EPOCHS,
                          steps_per_epoch=STEPS_PER_EPOCH,
                          validation_steps=VALIDATION_STEPS,
                          validation_data=test_dataset,
                          callbacks=[DisplayCallback()])

loss = model_history.history['loss']
val_loss = model_history.history['val_loss']

epochs = range(EPOCHS)

plt.figure()
plt.plot(epochs, loss, 'r', label='Training loss')
plt.plot(epochs, val_loss, 'bo', label='Validation loss')
plt.title('Training and Validation Loss')
plt.xlabel('Epoch')
plt.ylabel('Loss Value')
plt.ylim([0, 1])
plt.legend()
plt.show()


I am working on the file "04_detect_segment/04b_unet_segmentation.ipynb".
The first task is:
Train the model
We treat this as a multi-class prediction problem.
So, the loss function used is SparseCategoricalCrossentropy
and the metric is accuracy.
Can you write Python code for it?

model = unet_model(OUTPUT_CHANNELS)
model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

# Assign to the pixel the label with the highest probability
def create_mask(pred_mask):
  pred_mask = tf.argmax(pred_mask, axis=-1)
  pred_mask = pred_mask[..., tf.newaxis]
  return pred_mask[0]


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
And the plot for the delta of the second risk factor.
Can you write Python code for it?

plot_greeks_3d([a_1, a_2, delta_2], ['gbm1', 'gbm2', 'delta gbm2'])


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Correlations are stored as well and the resulting corrleation and Cholesky matrices are generated.
Can you write Python code for it?

vc.correlations

vc.correlation_matrix

vc.val_env.get_list('cholesky_matrix')


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Delta Surfaces
The same exercise as before for the two delta surfaces.
Can you write Python code for it?

delta_1 = np.zeros_like(a_1)
delta_2 = np.zeros_like(a_1)

%%time
for i in range(np.shape(delta_1)[0]):
    for j in range(np.shape(delta_1)[1]):
        vca.update('gbm1', initial_value=a_1[i, j])
        vca.update('gbm2', initial_value=a_2[i, j])
        delta_1[i, j] = vca.delta('gbm1')
        delta_2[i, j] = vca.delta('gbm2')

plot_greeks_3d([a_1, a_2, delta_1], ['gbm1', 'gbm2', 'delta gbm1'])

plot_greeks_3d([a_1, a_2, delta_2], ['gbm1', 'gbm2', 'delta gbm2'])


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Delta and vega estimates are generated in exactly the same fashion as in the two risk factor case.
Can you write Python code for it?

vc.delta('jd', interval=0.1)

vc.delta('sv')

vc.vega('jd')

vc.vega('sv')


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Delta for Jump Diffusion and Stochastic Vol Process
Of course, we cannot visualize Greek surfaces dependent on initial values for all four risk factors but still for two. In what follows we generate the delta surfaces with respect to the jump diffusion- and stochastic volatility-based risk factors.
Can you write Python code for it?

delta_1 = np.zeros_like(a_1)
delta_2 = np.zeros_like(a_1)

%%time
for i in range(np.shape(delta_1)[0]):
    for j in range(np.shape(delta_1)[1]):
        vc.update('jd', initial_value=a_1[i, j])
        vc.update('sv', initial_value=a_2[i, j])
        delta_1[i, j] = vc.delta('jd')
        delta_2[i, j] = vc.delta('sv')

plot_greeks_3d([a_1, a_2, delta_1], ['jump diffusion', 'stochastic vol', 'delta jd'])

plot_greeks_3d([a_1, a_2, delta_2], ['jump diffusion', 'stochastic vol', 'delta sv'])


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Example Output and Calculations
The following just displays some example output and the results from certain calculations.
Can you write Python code for it?

vc.risk_factors

vc.underlying_objects

vc.present_value()


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
However, another illustration that even such a complex instrument can be handled as elegantly as the most simple one (i.e. European option on single risk factor). Let us compare the present value estimates for both the European and American maximum basket options.
Can you write Python code for it?

# restore initial values
vc.update('jd', initial_value=36., volatility=0.1)
vc.update('sv', initial_value=36., volatility=0.1)
%time vc.present_value()

%time vca.present_value()

%time vca.delta('gbm1')

%time vca.delta('gbm2')

%time vca.vega('jd')

%time vca.vega('sv')

print "Duration for whole notebook %.2f in min" % ((time.time() - t0) / 60)


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
However, the instantiation of the valuation classe remains the same.
Can you write Python code for it?

vc = valuation_mcs_european_multi(
            name='European maximum call',
            val_env=val_env,
            risk_factors=risk_factors,
            correlations=correlations,
            payoff_func=payoff)


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
In this case, we need to specify three correlation values.
Can you write Python code for it?


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Let us reset the values to the original parameters.
Can you write Python code for it?

vc.update('gbm1', initial_value=36., volatility=0.1)
vc.update('gbm2', initial_value=36., volatility=0.5)


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
More than Two Risk Factors
The principles of working with multi-risk valuation classes can be illustrated quite well in the two risk factor case. However, there is---in theory---no limitation on the number of risk factors used for derivatives modeling.
Four Asset Basket Option
Consider a maximum basket option on four different risk factors. We add a jump diffusion as well as a stochastic volatility model to the mix
Can you write Python code for it?

me3 = market_environment('me3', dt.datetime(2015, 1, 1))
me4 = market_environment('me4', dt.datetime(2015, 1, 1))

me3.add_environment(me1)
me4.add_environment(me1)

# for jump-diffusion
me3.add_constant('lambda', 0.5)
me3.add_constant('mu', -0.6)
me3.add_constant('delta', 0.1)
me3.add_constant('model', 'jd')

# for stoch volatility model
me4.add_constant('kappa', 2.0)
me4.add_constant('theta', 0.3)
me4.add_constant('vol_vol', 0.2)
me4.add_constant('rho', -0.75)
me4.add_constant('model', 'sv')

val_env.add_constant('paths', 10000)
val_env.add_constant('frequency', 'W')
val_env.add_curve('time_grid', None)


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Present value estimations are generated by a call of the present_value method.
Can you write Python code for it?


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Sensitivities Positive Correlation
Almos in complete analogy to the single-risk valuation classes, sensitivities can be estimated for the multi-risk valuation classes.
Sensitivities Risk Factor 1
Consider first the case from before with positive correlation between the two risk factors. The following estimates and plots the sensitivities for the first risk factor gbm1.
Can you write Python code for it?

%%time
s_list = np.arange(28., 46.1, 2.)
pv = []; de = []; ve = []
for s in s_list:
    vc.update('gbm1', initial_value=s)
    pv.append(vc.present_value())
    de.append(vc.delta('gbm1', .5))
    ve.append(vc.vega('gbm1', 0.2))
vc.update('gbm1', initial_value=36.)

%matplotlib inline

plot_option_stats(s_list, pv, de, ve)


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Sensitivities Risk Factor 2
Now the sensitivities for the second risk factor.
Can you write Python code for it?

%%time
s_list = np.arange(28., 46.1, 2.)
pv = []; de = []; ve = []
for s in s_list:
    vc.update('gbm2', initial_value=s)
    pv.append(vc.present_value())
    de.append(vc.delta('gbm2', .5))
    ve.append(vc.vega('gbm2', 0.2))

plot_option_stats(s_list, pv, de, ve)


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Surfaces for Positive Correlation Case
Let us return to the case of positive correlation between the two relevant risk factors.
Can you write Python code for it?

correlations = [['gbm1', 'gbm2', 0.5]]

# European maximum call option
payoff_func = "np.maximum(np.maximum(maturity_value['gbm1'], maturity_value['gbm2']) - 38, 0)"
vc = valuation_mcs_european_multi(
            name='European maximum call',
            val_env=val_env,
            risk_factors=risk_factors,
            correlations=correlations,
            payoff_func=payoff_func)


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
The payoff function in this case gets a bit more complex.
Can you write Python code for it?

# European maximum call payoff
payoff_1 = "np.maximum(np.maximum(np.maximum(maturity_value['gbm1'], maturity_value['gbm2']),"
payoff_2 = " np.maximum(maturity_value['jd'], maturity_value['sv'])) - 40, 0)"
payoff = payoff_1 + payoff_2

payoff


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
The payoff of a European option is a one-dimensional ndarray object.
Can you write Python code for it?


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
The plot for the delta surface of the first risk factor.
Can you write Python code for it?

plot_greeks_3d([a_1, a_2, delta_1], ['gbm1', 'gbm2', 'delta gbm1'])


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
The present value surface is generated in the same way as before for the European option on the two risk factors. The computational burden is of course much higher for the American option, which are valued by the use of the Least-Squares Monte Carlo approach (LSM) according to Longstaff-Schwartz (2001).
Can you write Python code for it?

asset_1 = np.arange(28., 44.1, 4.)
asset_2 = asset_1
a_1, a_2 = np.meshgrid(asset_1, asset_2)
value = np.zeros_like(a_1)

%%time
for i in range(np.shape(value)[0]):
    for j in range(np.shape(value)[1]):
        vca.update('gbm1', initial_value=a_1[i, j])
        vca.update('gbm2', initial_value=a_2[i, j])
        value[i, j] = vca.present_value()

plot_greeks_3d([a_1, a_2, value], ['gbm1', 'gbm2', 'present value'])


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
The resulting plot then looks as follows. Here, a helper plot function of DX Analytics is used.
Can you write Python code for it?

plot_greeks_3d([a_1, a_2, value], ['gbm1', 'gbm2', 'present value'])


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
The surface for the first risk factor.
Can you write Python code for it?

plot_greeks_3d([a_1, a_2, vega_1], ['gbm1', 'gbm2', 'vega gbm1'])


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
The update method allows updating of certain parameters.
Can you write Python code for it?

vc.update('gbm1', initial_value=50.)

vc.present_value()

vc.update('gbm2', volatility=0.6)

vc.present_value()


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Value Surface
We are now interested in the value surface of the derivative instrument for both different initial values of the first and second risk factor.
Can you write Python code for it?

asset_1 = np.arange(28., 46.1, 4.)  # range of initial values
asset_2 = asset_1
a_1, a_2 = np.meshgrid(asset_1, asset_2)
  # two-dimensional grids out of the value vectors
value = np.zeros_like(a_1)


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
Vega Surfaces
The same approach can of course be applied to generate vega surfaces.
Can you write Python code for it?

vega_1 = np.zeros_like(a_1)
vega_2 = np.zeros_like(a_1)

for i in range(np.shape(vega_1)[0]):
    for j in range(np.shape(vega_1)[1]):
        vc.update('gbm1', initial_value=a_1[i, j])
        vc.update('gbm2', initial_value=a_2[i, j])
        vega_1[i, j] = vc.vega('gbm1')
        vega_2[i, j] = vc.vega('gbm2')


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
We assum a positive correlation between the two risk factors.
Can you write Python code for it?

risk_factors = {'gbm1' : me1, 'gbm2' : me2}
correlations = [['gbm1', 'gbm2', 0.5]]


I am working on the file "04_dx_valuation_multi_risk.ipynb".
The first task is:
When calculating Greeks the risk factor now has to be specified by providing its name.
Can you write Python code for it?

vc.delta('gbm2', interval=0.5)

vc.vega('gbm1')


I am working on the file "04_hacking_training_loop/distribution_strategies.ipynb".
The first task is:
Build the model and set up distribution strategy
Next, we'll combine the components of the model above to build the DNN model.
Here is also where we'll define the distribution strategy. To do that, we'll place the building of the model inside the scope of the distribution strategy. Notice the output after excuting the cell below. We'll see
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3')
This indicates that we're using the MirroredStrategy on 4 GPUs. That is because my machine has 4 GPUs. Your output may look different depending on how many GPUs you have on your device.
Can you write Python code for it?

def build_dnn_model():
    """Builds simple DNN using Keras Functional API.

    Returns:
        `tf.keras.models.Model` object.
    """
    # Create input layer
    inputs = create_input_layers()

    # Create feature columns
    feature_columns = create_feature_columns()

    # The constructor for DenseFeatures takes a list of numeric columns
    # The Functional API in Keras requires: LayerConstructor()(inputs)
    dnn_inputs = layers.DenseFeatures(
        feature_columns=feature_columns.values())(inputs)

    # Get output of model given inputs
    output = get_model_outputs(dnn_inputs)

    # Build model and compile it all together
    model = tf.keras.models.Model(inputs=inputs, outputs=output)
    model.compile(optimizer="adam", loss="mse", metrics=[rmse, "mse"])

    return model

# Create the distribution strategy
mirrored_strategy = tf.distribute.MirroredStrategy()
with mirrored_strategy.scope():
    model = build_dnn_model()

print("Here is our DNN architecture so far:\n")
print(model.summary())


I am working on the file "04_hacking_training_loop/hyperparameter_tuning.ipynb".
The first task is:
Below, we're copying our model training code to a model.py file in our trainer package directory. This code runs training and after training completes, reports the model's final loss to Cloud HyperTune.
Can you write Python code for it?

%%writefile trainer/model.py
import argparse
import hypertune
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim

from sklearn.utils import shuffle
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import normalize

def get_args():
    """Argument parser.
    Returns:
        Dictionary of arguments.
    """
    parser = argparse.ArgumentParser(description='PyTorch MNIST')
    parser.add_argument('--job-dir',  # handled automatically by AI Platform
                        help='GCS location to write checkpoints and export ' \
                             'models')
    parser.add_argument('--lr',  # Specified in the config file
                        type=float,
                        default=0.01,
                        help='learning rate (default: 0.01)')
    parser.add_argument('--momentum',  # Specified in the config file
                        type=float,
                        default=0.5,
                        help='SGD momentum (default: 0.5)')
    parser.add_argument('--hidden-layer-size',  # Specified in the config file
                        type=int,
                        default=8,
                        help='hidden layer size')
    args = parser.parse_args()
    return args

def train_model(args):
    # Get the data
    natality = pd.read_csv('https://storage.googleapis.com/ml-design-patterns/natality.csv')
    natality = natality.dropna()
    natality = shuffle(natality, random_state = 2)
    natality.head()

    natality_labels = natality['weight_pounds']
    natality = natality.drop(columns=['weight_pounds'])

    train_size = int(len(natality) * 0.8)
    traindata_natality = natality[:train_size]
    trainlabels_natality = natality_labels[:train_size]

    testdata_natality = natality[train_size:]
    testlabels_natality = natality_labels[train_size:]

    # Normalize and convert to PT tensors
    normalized_train = normalize(np.array(traindata_natality.values), axis=0)
    normalized_test =...


I am working on the file "04_hacking_training_loop/hyperparameter_tuning.ipynb".
The first task is:
Hyperparameter tuning on Cloud AI Platform
In this section we'll show you how to scale your hyperparameter optimization by running it on Google Cloud's AI Platform. You'll need a Cloud account with AI Platform Training enabled to run this section.
We'll be using PyTorch to build a regression model in this section. To train the model we'll be the BigQuery natality dataset. We've made a subset of this data available in a public Cloud Storage bucket, which we'll download from within the training job.
Can you write Python code for it?

from google.colab import auth
auth.authenticate_user()


I am working on the file "04_hacking_training_loop/hyperparameter_tuning.ipynb".
The first task is:
In the cells below, replcae your-project-id with the ID of your Cloud project, and your-gcs-bucket with the name of your Cloud Storage bucket.
Can you write Python code for it?

!gcloud config set project your-project-id

BUCKET_URL = 'gs://your-gcs-bucket'


I am working on the file "04_hacking_training_loop/hyperparameter_tuning.ipynb".
The first task is:
Let's see which hyperparameters resulted in the best accuracy.
Can you write Python code for it?


I am working on the file "04_hacking_training_loop/hyperparameter_tuning.ipynb".
The first task is:
Next, we'll build our Scikit-learn model and define the hyperparameters we want to optimize using grid serach.
Can you write Python code for it?

model = RandomForestClassifier()

grid_vals = {
  'max_depth': [5, 10, 100],
  'n_estimators': [100, 150, 200]
}

grid_search = GridSearchCV(model, param_grid=grid_vals, scoring='accuracy')

# Train the model while running hyperparameter trials
grid_search.fit(train_data.values, train_labels.values)


I am working on the file "04_hacking_training_loop/hyperparameter_tuning.ipynb".
The first task is:
To keep things simple, we'll first convert the label column to numeric and then
use pd.get_dummies() to covert the data to numeric.
Can you write Python code for it?

# 1 = edible, 0 = poisonous
mushroom_data.loc[mushroom_data['class'] == 'p', 'class'] = 0
mushroom_data.loc[mushroom_data['class'] == 'e', 'class'] = 1

labels = mushroom_data.pop('class')

dummy_data = pd.get_dummies(mushroom_data)

# Split the data
train_size = int(len(mushroom_data) * .8)

train_data = dummy_data[:train_size]
test_data = dummy_data[train_size:]

train_labels = labels[:train_size].astype(int)
test_labels = labels[train_size:].astype(int)


I am working on the file "04_hacking_training_loop/overfitting.ipynb".
The first task is:
Create the input pipeline.
Can you write Python code for it?

# train/validation dataset
trainds = tf.data.Dataset.from_tensor_slices((X_train.to_dict('list'), y_train.values))
evalds = tf.data.Dataset.from_tensor_slices((X_valid.to_dict('list'), y_valid.values))

# Examine the datasets.
for dict_slice in trainds.take(1):
        print("{}\n".format(dict_slice))


I am working on the file "04_hacking_training_loop/overfitting.ipynb".
The first task is:
Lastly, we'll plot the validation and train loss curves.
Can you write Python code for it?

def plot_curves(history, metrics):
    nrows = 1
    ncols = 2
    fig = plt.figure(figsize=(10, 5))

    for idx, key in enumerate(metrics):
        ax = fig.add_subplot(nrows, ncols, idx+1)
        plt.plot(history.history[key])
        plt.plot(history.history['val_{}'.format(key)])
        plt.title('model {}'.format(key))
        plt.ylabel(key)
        plt.xlabel('epoch')
        plt.legend(['train', 'validation'], loc='upper left');

plot_curves(history, ['loss'])


I am working on the file "04_hacking_training_loop/overfitting.ipynb".
The first task is:
Set up the features for the model
Can you write Python code for it?

# prepare inputs
df.is_male = df.is_male.astype(str)

df.mother_race.fillna(0, inplace = True)
df.mother_race = df.mother_race.astype(str)

FEATURES = ['is_male', 'mother_age', 'plurality', 'gestation_weeks', 'mother_race']
LABEL = ['weight_pounds']

N_TRAIN = int(df.shape[0] * 0.80)

X_train = df[FEATURES][:N_TRAIN]
X_valid = df[FEATURES][N_TRAIN:]

y_train = df[LABEL][:N_TRAIN]
y_valid = df[LABEL][N_TRAIN:]


I am working on the file "04_hacking_training_loop/overfitting.ipynb".
The first task is:
Then we train the model on the single batch. Note we must use .repeat() since we'll quickly run out of data training on just a single batch. Also, we've set the number of training examples to BATCH_SIZE.
Can you write Python code for it?

# train the classifcation model
tf.random.set_seed(33)

NUM_TRAINING_EXAMPLES = BATCH_SIZE  # total number of training examples
NUM_VALID_EXAMPLES = X_valid.shape[0]
NUM_EPOCHS = 100
TOTAL_TRAINING_EXAMPLES = int(NUM_EPOCHS * NUM_TRAINING_EXAMPLES)

steps_per_epoch = (TOTAL_TRAINING_EXAMPLES // (BATCH_SIZE * NUM_EPOCHS))

evalds = evalds.batch(X_valid.shape[0]).take(int(NUM_VALID_EXAMPLES//BATCH_SIZE))

# train the model
history = model.fit(single_batch.repeat(),
                    validation_data=evalds,
                    epochs=NUM_EPOCHS,
                    steps_per_epoch=steps_per_epoch,
                    verbose=0
                   )


I am working on the file "04_hacking_training_loop/transfer_learning.ipynb".
The first task is:
Building a medical imaging classification model with Keras and VGG
We'll use a colorectal histology dataset that comes pre-installed with TF Datasets. First we'll extract the data as a TF Dataset. Then we'll load a VGG model without the top classification layers. Finally, we'll add layers specific to our classification problem so that the final model outputs a softmax classification corresponding with 1 of the 8 classes in our dataset.
Can you write Python code for it?

# These images will be (150,150,3)
(train, validation, test), info = tfds.load(
    'colorectal_histology',
    split=['train[:80%]', 'train[80%:90%]', 'train[90%:]'],
    shuffle_files=True,
    as_supervised=True,
    with_info=True
)

# Some info on the original dataset
info

# Utility for what each label corresponds with
label_map = ['tumor','stroma','complex','lympho','debris','mucosa', 'adipose']

# Preview 2 examples from our dataset
get_label_name = info.features['label'].int2str
for image, label in train.take(2):
  plt.figure()
  plt.imshow(np.array(image))
  plt.title(get_label_name(label))


I am working on the file "04_hacking_training_loop/transfer_learning.ipynb".
The first task is:
Here we'll import a TF Hub module for text classification. Because we're using TF Hub, we can feed the data to our model directly as strings. We don't need to worry about preprocessing since TF Hub will handle converting the text to embeddings for us.
Can you write Python code for it?

hub_layer = hub.KerasLayer("https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1",
                           input_shape=[], dtype=tf.string, trainable=True)


I am working on the file "04_hacking_training_loop/transfer_learning.ipynb".
The first task is:
To see what the TF Hub layer is doing, let's see the embedding for an example sentence.
Can you write Python code for it?

test_embedding = hub_layer(["I'm excited to try out transfer learning with TF Hub"])
print(test_embedding)

model = keras.Sequential([
  hub_layer,
  keras.layers.Dense(32, activation='relu'),
  keras.layers.Dense(1, activation='sigmoid')
])

model.summary()

model.compile(optimizer='adam',
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              metrics=['accuracy'])

# Note: you can train for more epochs to get higher accuracy
model.fit(
    reviews_train.shuffle(10000).batch(512),
    validation_data=reviews_validate.batch(512),
    epochs=15
)

# Evaluate the model
results = model.evaluate(reviews_test.batch(512))


I am working on the file "04_intro-to-tensorflow/intro_to_tensorflow.ipynb".
The first task is:
<img src="image/Learn_Rate_Tune_Image.png" style="height: 70%;width: 70%">
Problem 3
Below are 2 parameter configurations for training the neural network. In each configuration, one of the parameters has multiple options. For each configuration, choose the option that gives the best acccuracy.
Parameter configurations:
Configuration 1
* Epochs: 1
* Learning Rate:
  * 0.8
  * 0.5
  * 0.1
  * 0.05
  * 0.01
Configuration 2
* Epochs:
  * 1
  * 2
  * 3
  * 4
  * 5
* Learning Rate: 0.2
The code will print out a Loss and Accuracy graph, so you can see how well the neural network performed.
If you're having trouble solving problem 3, you can view the solution here.
Can you write Python code for it?

results

# Change if you have memory restrictions
batch_size = 128

# Find the best parameters for each configuration

#results = []
#for epochs in [1,2,3,4,5]:
#    for learning_rate in [0.8, 0.5, 0.1, 0.05, 0.01]:

epochs = 4
learning_rate = 0.1

### DON'T MODIFY ANYTHING BELOW ###
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

# The accuracy measured against the validation set
validation_accuracy = 0.0

# Measurements use for graphing loss and accuracy
log_batch_step = 50
batches = []
loss_batch = []
train_acc_batch = []
valid_acc_batch = []

with tf.Session() as session:
    session.run(init)
    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        # Progress bar
        batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(epoch_i+1, epochs), unit='batches')

        # The training cycle
        for batch_i in batches_pbar:
            # Get a batch of training features and labels
            batch_start = batch_i*batch_size
            batch_features = train_features[batch_start:batch_start + batch_size]
            batch_labels = train_labels[batch_start:batch_start + batch_size]

            # Run optimizer and get loss
            _, l = session.run(
                [optimizer, loss],
                feed_dict={features: batch_features, labels: batch_labels})

            # Log every 50 batches
            if not batch_i % log_batch_step:
                # Calculate Training and Validation accuracy
                training_accuracy = session.run(accuracy, feed_dict=train_feed_dict)
                validation_accuracy = session.run(accuracy, feed_dict=valid_feed_dict)

                # Log batches
                previous_batch = batches[-1] if batches else 0
                batches.append(log_batch_step + previous_batch)
                loss_batch.append(l)
                train_acc_batch.append(training_accuracy)
                valid_acc_batch.append(validation_accuracy)

       ...


I am working on the file "04_intro-to-tensorflow/intro_to_tensorflow.ipynb".
The first task is:
<img src="image/Mean_Variance_Image.png" style="height: 75%;width: 75%; position: relative; right: 5%">
Problem 1
The first problem involves normalizing the features for your training and test data.
Implement Min-Max scaling in the normalize_grayscale() function to a range of a=0.1 and b=0.9. After scaling, the values of the pixels in the input data should range from 0.1 to 0.9.
Since the raw notMNIST image data is in grayscale, the current values range from a min of 0 to a max of 255.
Min-Max Scaling:
$
X'=a+{\frac {\left(X-X_{\min }\right)\left(b-a\right)}{X_{\max }-X_{\min }}}
$
If you're having trouble solving problem 1, you can view the solution here.
Can you write Python code for it?

# Problem 1 - Implement Min-Max scaling for grayscale image data
def normalize_grayscale(image_data):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
    arr = np.array(image_data)
    arr = (arr - arr.min())/(arr.max() - arr.min())
    arr = arr * 0.8 + 0.1
    return arr.tolist()

### DON'T MODIFY ANYTHING BELOW ###
# Test Cases
np.testing.assert_array_almost_equal(
    normalize_grayscale(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 255])),
    [0.1, 0.103137254902, 0.106274509804, 0.109411764706, 0.112549019608, 0.11568627451, 0.118823529412, 0.121960784314,
     0.125098039216, 0.128235294118, 0.13137254902, 0.9],
    decimal=3)
np.testing.assert_array_almost_equal(
    normalize_grayscale(np.array([0, 1, 10, 20, 30, 40, 233, 244, 254,255])),
    [0.1, 0.103137254902, 0.13137254902, 0.162745098039, 0.194117647059, 0.225490196078, 0.830980392157, 0.865490196078,
     0.896862745098, 0.9])
print(train_features)
if not is_features_normal:
    train_features = normalize_grayscale(train_features)
    test_features = normalize_grayscale(test_features)
    is_features_normal = True

print('Tests Passed!')

if not is_labels_encod:
    # Turn labels into numbers and apply One-Hot Encoding
    encoder = LabelBinarizer()
    encoder.fit(train_labels)
    train_labels = encoder.transform(train_labels)
    test_labels = encoder.transform(test_labels)

    # Change to float32, so it can be multiplied against the features in TensorFlow, which are float32
    train_labels = train_labels.astype(np.float32)
    test_labels = test_labels.astype(np.float32)
    is_labels_encod = True

print('Labels One-Hot Encoded')

assert is_features_normal, 'You skipped the step to normalize the features'
assert is_labels_encod, 'You skipped the step to One-Hot Encode the labels'

# Get randomized datasets for training and validation
train_features, valid_features, train_labels, valid_labels =...


I am working on the file "04_intro-to-tensorflow/intro_to_tensorflow.ipynb".
The first task is:
Checkpoint
All your progress is now saved to the pickle file.  If you need to leave and comeback to this lab, you no longer have to start from the beginning.  Just run the code block below and it will load all the data and modules required to proceed.
Can you write Python code for it?

%matplotlib inline

# Load the modules
import pickle
import math

import numpy as np
import tensorflow as tf
from tqdm import tqdm
import matplotlib.pyplot as plt

# Reload the data
pickle_file = 'notMNIST.pickle'
with open(pickle_file, 'rb') as f:
  pickle_data = pickle.load(f)
  train_features = pickle_data['train_dataset']
  train_labels = pickle_data['train_labels']
  valid_features = pickle_data['valid_dataset']
  valid_labels = pickle_data['valid_labels']
  test_features = pickle_data['test_dataset']
  test_labels = pickle_data['test_labels']
  del pickle_data  # Free up memory

print('Data and modules loaded.')


I am working on the file "04_intro-to-tensorflow/intro_to_tensorflow.ipynb".
The first task is:
Problem 2
Now it's time to build a simple neural network using TensorFlow. Here, your network will be just an input layer and an output layer.

For the input here the images have been flattened into a vector of $28 \times 28 = 784$ features. Then, we're trying to predict the image digit so there are 10 output units, one for each label. Of course, feel free to add hidden layers if you want, but this notebook is built to guide you through a single layer network.
For the neural network to train on your data, you need the following float32 tensors:
 - features
  - Placeholder tensor for feature data (train_features/valid_features/test_features)
 - labels
  - Placeholder tensor for label data (train_labels/valid_labels/test_labels)
 - weights
  - Variable Tensor with random numbers from a truncated normal distribution.
    - See tf.truncated_normal() documentation for help.
 - biases
  - Variable Tensor with all zeros.
    - See  tf.zeros() documentation for help.
If you're having trouble solving problem 2, review "TensorFlow Linear Function" section of the class.  If that doesn't help, the solution for this problem is available here.
Can you write Python code for it?

# All the pixels in the image (28 * 28 = 784)
features_count = 784
# All the labels
labels_count = 10

features = tf.placeholder(tf.float32, [None, features_count])
labels = tf.placeholder(tf.float32, [None, labels_count])

weights = tf.Variable(tf.truncated_normal([features_count, labels_count]))
biases = tf.Variable(tf.zeros([labels_count]))

### DON'T MODIFY ANYTHING BELOW ###

#Test Cases
from tensorflow.python.ops.variables import Variable

assert features._op.name.startswith('Placeholder'), 'features must be a placeholder'
assert labels._op.name.startswith('Placeholder'), 'labels must be a placeholder'
assert isinstance(weights, Variable), 'weights must be a TensorFlow variable'
assert isinstance(biases, Variable), 'biases must be a TensorFlow variable'

assert features._shape == None or (\
    features._shape.dims[0].value is None and\
    features._shape.dims[1].value in [None, 784]), 'The shape of features is incorrect'
assert labels._shape  == None or (\
    labels._shape.dims[0].value is None and\
    labels._shape.dims[1].value in [None, 10]), 'The shape of labels is incorrect'
assert weights._variable._shape == (784, 10), 'The shape of weights is incorrect'
assert biases._variable._shape == (10), 'The shape of biases is incorrect'

assert features._dtype == tf.float32, 'features must be type float32'
assert labels._dtype == tf.float32, 'labels must be type float32'

# Feed dicts for training, validation, and test session
train_feed_dict = {features: train_features, labels: train_labels}
valid_feed_dict = {features: valid_features, labels: valid_labels}
test_feed_dict = {features: test_features, labels: test_labels}

# Linear Function WX + b
logits = tf.matmul(features, weights) + biases

prediction = tf.nn.softmax(logits)

# Cross entropy
cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)

# Training loss
loss = tf.reduce_mean(cross_entropy)

# Create an operation that initializes all variables
init = tf.global_variables_initializer()

# Test Cases
with...


I am working on the file "04_intro-to-tensorflow/intro_to_tensorflow.ipynb".
The first task is:
Test
You're going to test your model against your hold out dataset/testing data.  This will give you a good indicator of how well the model will do in the real world.  You should have a test accuracy of at least 80%.
Can you write Python code for it?

### DON'T MODIFY ANYTHING BELOW ###
# The accuracy measured against the test set
test_accuracy = 0.0

with tf.Session() as session:

    session.run(init)
    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        # Progress bar
        batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(epoch_i+1, epochs), unit='batches')

        # The training cycle
        for batch_i in batches_pbar:
            # Get a batch of training features and labels
            batch_start = batch_i*batch_size
            batch_features = train_features[batch_start:batch_start + batch_size]
            batch_labels = train_labels[batch_start:batch_start + batch_size]

            # Run optimizer
            _ = session.run(optimizer, feed_dict={features: batch_features, labels: batch_labels})

        # Check accuracy against Test data
        test_accuracy = session.run(accuracy, feed_dict=test_feed_dict)

assert test_accuracy >= 0.80, 'Test accuracy at {}, should be equal to or greater than 0.80'.format(test_accuracy)
print('Nice Job! Test Accuracy is {}'.format(test_accuracy))


I am working on the file "04_intro-to-tensorflow/intro_to_tensorflow.ipynb".
The first task is:
The notMNIST dataset is too large for many computers to handle.  It contains 500,000 images for just training.  You'll be using a subset of this data, 15,000 images for each label (A-J).
Can you write Python code for it?

def download(url, file):
    """
    Download file from <url>
    :param url: URL to file
    :param file: Local file path
    """
    if not os.path.isfile(file):
        print('Downloading ' + file + '...')
        urlretrieve(url, file)
        print('Download Finished')

# Download the training and test dataset.
download('https://s3.amazonaws.com/udacity-sdc/notMNIST_train.zip', 'notMNIST_train.zip')
download('https://s3.amazonaws.com/udacity-sdc/notMNIST_test.zip', 'notMNIST_test.zip')

# Make sure the files aren't corrupted
assert hashlib.md5(open('notMNIST_train.zip', 'rb').read()).hexdigest() == 'c8673b3f28f489e9cdf3a3d74e2ac8fa',\
        'notMNIST_train.zip file is corrupted.  Remove the file and try again.'
assert hashlib.md5(open('notMNIST_test.zip', 'rb').read()).hexdigest() == '5d3c7e653e63471c88df796156a9dfa9',\
        'notMNIST_test.zip file is corrupted.  Remove the file and try again.'

# Wait until you see that all files have been downloaded.
print('All files downloaded.')

def uncompress_features_labels(file):
    """
    Uncompress features and labels from a zip file
    :param file: The zip file to extract the data from
    """
    features = []
    labels = []

    with ZipFile(file) as zipf:
        # Progress Bar
        filenames_pbar = tqdm(zipf.namelist(), unit='files')

        # Get features and labels from all files
        for filename in filenames_pbar:
            # Check if the file is a directory
            if not filename.endswith('/'):
                with zipf.open(filename) as image_file:
                    image = Image.open(image_file)
                    image.load()
                    # Load image data as 1 dimensional array
                    # We're using float32 to save on memory space
                    feature = np.array(image, dtype=np.float32).flatten()

                # Get the the letter from the filename.  This is the letter of the image.
                label = os.path.split(filename)[1][0]

                features.append(feature)
        ...


I am working on the file "04_jit/04_actividad.ipynb".
The first task is:
En esta actividad implementaremos una conocida métrica para medir disimilitud entre conjuntos: La métrica de Hausdorff. Esta corresponde a un métrica o distancia ocupada para medir cuán disímiles son dos subconjuntos dados.
Esta tiene muchas aplicaciones, en particular para comparar el parecido entre imágenes. En el caso en donde los conjuntos son arreglos bidimensionales, la definición es la siguiente:
Sean $X \in \mathbb{R}^{m \times 3}$ e  $Y \in \mathbb{R}^{n \times 3}$ dos matrices, la métrica/distancia de Hausdorff sobre sobre estas como:
$$
d_H(X,Y) = \max \left(\ \max_{i\leq m} \min_{j \leq n} d(X[i],Y[j]), \ \max_{j\leq n} \min_{i \leq m} d(Y[j],X[i]) \ \right)
$$
donde $d$ es la distancia Euclideana clásica. ($X[i]$ indíca la i-ésima fila de X).
Ilustración unidimensional: Distancia entre funciones.
<img src='data/hausdorff.png' style="width: 600px;">

Implemente la métrica de Hausdorff en Python clásico.
Implemente la métrica de Hausdorff usando Numba (Forzando el modo nopython y definiendo explícitamente las signatures de las funciones).
Cree 10 arreglos $X,Y$ aleatorios, con cantidad creciente de filas, y realice análsis de tiempos de ejecuciones de las funciones anteriores sobre estos arreglos.
Concluya.

Paso 1.
Can you write Python code for it?

def metric_python(x, y):
    """
    standard Euclidean distance
    """
    ret = x-y
    ret *= ret
    return np.sqrt(ret).sum()

def inf_dist_python(x, Y):
    """
    inf distance between row x and array Y
    """
    m = Y.shape[0]
    inf = np.inf

    for i in range(m):
        dist = metric_python(x, Y[i])
        if dist < inf:
            inf = dist
    return inf

def hausdorff_python(X, Y):
    """
    Hausdorff distance between arrays X and Y
    """
    m = X.shape[0]
    n = Y.shape[0]
    sup1 = -1.
    sup2 = -1.

    for i in range(m):
        inf1 = inf_dist_python(X[i], Y)
        if inf1 > sup1:
            sup1 = inf1
    for i in range(n):
        inf2 = inf_dist_python(Y[i], X)
        if inf2 > sup2:
            sup2 = inf2

    return max(sup1, sup2)


I am working on the file "04_jit/04_actividad.ipynb".
The first task is:
Paso 2.
Notar que los cambios son mínimos: Decoradores + nombres de funciones.
Can you write Python code for it?

@numba.jit('float64 (float64[:], float64[:])')
def metric_numba(x, y):
    """
    standard Euclidean distance
    """
    ret = x-y
    ret *= ret
    return np.sqrt(ret).sum()

@numba.jit('float64 (float64[:], float64[:,:])', nopython=True)
def inf_dist_numba(x, Y):
    """
    inf distance between row x and array Y
    """
    m = Y.shape[0]
    inf = np.inf

    for i in range(m):
        dist = metric_numba(x, Y[i])
        if dist < inf:
            inf = dist
    return inf

@numba.jit('float64 (float64[:,:], float64[:,:])', nopython=True)
def hausdorff_numba(X, Y):
    """
    Hausdorff distance between arrays X and Y
    """
    m = X.shape[0]
    n = Y.shape[0]
    sup1 = -1.
    sup2 = -1.

    for i in range(m):
        inf1 = inf_dist_numba(X[i], Y)
        if inf1 > sup1:
            sup1 = inf1
    for i in range(n):
        inf2 = inf_dist_numba(Y[i], X)
        if inf2 > sup2:
            sup2 = inf2

    return max(sup1, sup2)


I am working on the file "04_jit/04_actividad.ipynb".
The first task is:
Paso 3.
Can you write Python code for it?

#nrows = [10**n for n in range(10)]
nrows = np.linspace(100,5000,10).astype(int)

for nrow in nrows:
    X = np.random.random((nrow,3))
    Y = np.random.random((nrow,3))
    tp = %timeit -o hausdorff_python(X,Y)
    tn = %timeit -o hausdorff_numba(X,Y)
    print("Number of rows: {0}".format(nrow))
    print("Best time in Python: {0}".format(tp.best))
    print("Best time in Numba: {0} \n".format(tn.best))
    del X,Y


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
<div id='numexpr' />
4.- NumExpr
The problem...
As mention in previous lectures, NumPy is good (fast and efficient) at doing vector operations. Moreover, it has some problems when trying to evaluate to complex expressions:
Can you write Python code for it?

def test_func(a,b,c):
    """
    Consider that a, b and c are 1D ndarrys
    """
    return np.sin(a**2 + np.exp(b)) + np.cos(b**2 + np.exp(c)) + np.tan(a**2+b**2+c**2)

n = 1000000
a = np.random.random(n)
b = np.random.random(n)
c = np.random.random(n)

%timeit test_func(a,b,c)


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Additionally we can explicitly specify the number of threads that NumExpr can use to evaluate the expression with the set_num_threads() function:
Can you write Python code for it?

n_threads = 4
for i in range(1, n_threads+1):
    ne.set_num_threads(i)
    %timeit ne.evaluate('sin(a**2 + exp(b)) + cos(b**2 + exp(c)) + tan(a**2+b**2+c**2)')


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
For a full reference of the signature types supported by Numba see the documentation.
Now that we've run sum1 and sum2 once, they are  now compiled and we can check out what's happened behind the scenes. Use the inspect_types method to see how Numba translated the functions.
Can you write Python code for it?


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Let's create now a Numba function that performs the same operations but iteratively:
Can you write Python code for it?

@numba.jit('float64[:] (float64[:], float64[:], float64[:])', nopython=True)
def test_func_numba(a,b,c):
    n = len(a)
    res = np.empty(n)
    for i in range(n):
        res[i] = np.sin(a[i]**2 + np.exp(b[i])) + np.cos(b[i]**2 + np.exp(c[i])) + np.tan(a[i]**2+b[i]**2+c[i]**2)
    return res

%timeit test_func_numba(a,b,c)


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Lets create a silly example where Numba fails at nopython mode. Prepare for a long error...
Can you write Python code for it?

@numba.jit('float64[:] (float64[:,:], float64[:])', nopython=True)
def dot_numba2(A,b):
    m,n = A.shape
    c = dict()
    for i in range(m):
        c[i] =  np.dot(A[i],b)
    return np.array(c.values)


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Mandelbrot fractal
Now we will create a Mandelbrot fractal (which is a task that cannot be vectorized) using native Python and Numba...
It basically consist in the next iteration, with starting point $z_0 = 0$:
$$ z_{i+1} = z_{i}^2 + c  $$
for which the values of the sequence remains bounded.
Can you write Python code for it?

size = 200
iterations = 100

def mandelbrot_python(m, size, iterations):
    for i in range(size):
        for j in range(size):
            c = -2 + 3./size*j + 1j*(1.5-3./size*i)
            z= 0
            for n in range(iterations):
                if np.abs(z) <= 10:
                    z = z*z + c
                    m[i, j] = n
                else:
                    break

m = np.zeros((size,size))
mandelbrot_python(m, size, iterations)

plt.figure(figsize=(7,7))
plt.imshow(np.log(m), cmap=plt.cm.hot)
plt.axis('off')


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Next, we add the numba.jit decorator and let Numba infer the types of all the variables (Lazzy Compilation):
Can you write Python code for it?

@numba.jit
def mandelbrot_numba(m, size, iterations):
    for i in range(size):
        for j in range(size):
            c = -2 + 3./size*j + 1j*(1.5-3./size*i)
            z= 0
            for n in range(iterations):
                if np.abs(z) <= 10:
                    z = z*z + c
                    m[i, j] = n
                else:
                    break

%%timeit m = np.zeros((size,size))
mandelbrot_python(m,size,iterations)


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Now we create a random walk, plot it and %timeit its execution:
Can you write Python code for it?

n = 100000
x = walk(n)

plt.figure(figsize=(8,8))
plt.plot(x)
plt.show()

python_t = %timeit -o walk(n)


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Now, let's JIT-compile this function with Numba:
Can you write Python code for it?

@numba.jit(nopython=True)
def step_numba():
    if random.random()>.5: return 1.
    else: return -1.

@numba.jit(nopython=True)
def walk_numba(n):
    x = np.zeros(n)
    dx = 1./n
    for i in range(n-1):
        x_new = x[i] + dx * step_numba()
        if abs(x_new) > 5e-3:
            x[i+1] = 0.
        else:
            x[i+1] = x_new
    return x

numba_t = %timeit -o walk_numba(n)

print("Improvement: {0} times faster".format(python_t.best/numba_t.best))


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Some important notes:
* The first time we invoke a JITted function, it is translated to native machine code.
* The very first time you run a numba compiled function, there will be a little bit of overhead for the compilation step to take place.
* As an optimizing compiler, Numba needs to decide on the type of each variable to generate efficient machine code.
* When no argument are passed to the numba.jit decorator, this will try to automatically detect the types of input, output and intermediate variables.
* Additionally we can explicitily pass the signature of the function to the decorator, to make the work easier for Numba :).
Can you write Python code for it?

#single signature
@numba.jit('float64[:] (float64[:], float64[:])')
def sum1(a,b):
    return a+b

a = np.arange(10, dtype=np.float64)
b = np.arange(10, dtype=np.float64)
print(sum1(a,b))

#multiple signatures (polymorphism)
signatures = ['int32[:] (int32[:], int32[:])', 'int64[:] (int64[:], int64[:])', \
              'float32[:] (float32[:], float32[:])', 'float64[:] (float64[:], float64[:])']
@numba.jit(signatures)
def sum2(a,b):
    return a+b

a = np.arange(10, dtype=np.int64)
b = np.arange(10, dtype=np.int64)
#print(sum1(a,b))
print(sum2(a,b))


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
Table of Contents

1.- Just in Time Compilation
2.- Numba
3.- Applications
4.- NumExp

1.- Just in Time Compilation
A JIT compiler runs after the program has started and compiles the code (usually bytecode) on the fly (or just-in-time, as it's called) into a form that's usually faster, typically the host CPU's native instruction set. A JIT has access to dynamic runtime information whereas a standard compiler doesn't and can make better optimizations like inlining functions that are used frequently.
This is in contrast to a traditional compiler (AOT, Ahead Of Time compilation) that compiles all the code to machine language before the program is first run.

2.- Numba

Numba takes pure Python code and translates it automatically (just-in-time) into optimized machine code, thanks to the LLVM (Low Level Virtual Machine) compiler architecture.
We can write a non-vectorized function in pure Python, using for loops, and have this function vectorized automatically by using a single decorator.
Performance speedups when compared to pure Python code can reach several orders of magnitude and may even outmatch manually-vectorized NumPy code.

Why NumPy is not sufficient?

Sometimes it is difficult to visualize a vectorized implementation of an algorithm (difficult to understand also),
And sometimes it is just imposible to implement a vectorized solution (Typical case: When we need the result from from one iteration to perform the next).

(Some kind of) Rule of thumb: When a loop in an algorithm needs the result from one iteration to perform the next iteration, then it is not possible to implement it in a vectorized way.
Lets introduce its usage with a naive example:
Array sum
The function below is a naive sum function that sums all the elements of a given array.
Can you write Python code for it?

def sum_array(inp):
    I,J = inp.shape
    mysum = 0
    for i in range(I):
        for j in range(J):
            mysum += inp[i, j]
    return mysum

arr = np.random.random((500,500))

sum_array(arr)

naive = %timeit -o sum_array(arr)

#lazzy compilation
@numba.jit
def sum_array_numba(inp):
    I,J = inp.shape
    mysum = 0
    for i in range(I):
        for j in range(J):
            mysum += inp[i, j]
    return mysum

sum_array_numba(arr)

jitted = %timeit -o sum_array_numba(arr)

print("Improvement: {0} times faster".format(naive.best/jitted.best))

%timeit np.sum(arr)


I am working on the file "04_jit/04_jit.ipynb".
The first task is:
and create the simulation in pure Python, where the function walk() takes a number of steps as input:
Can you write Python code for it?

def walk(n):
    x = np.zeros(n)
    dx = 1./n
    for i in range(n-1):
        x_new = x[i] + dx * step()
        if abs(x_new) > 5e-3:
            x[i+1] = 0.
        else:
            x[i+1] = x_new
    return x