I am working on the file "ORIG_GetRadLexTerms.ipynb".
The first task is:
tmp = [r for r in query_results["pathophysiology_metaclass"] \
                                if "short bowel" in r[1]]
tmp[0][2].toPython()
Import functional form of pyConTextNLP
Can you write Python code for it?

import pyConTextNLP.functional.itemData as ID
import pyConTextNLP.functional.conTextItem as CI
import pyConTextNLP.functional.ConTextMarkup as CM

def create_sentence_conTextMarkup(s, targets ):
    markup = CM.create_ConTextMarkup()
    markup = CM.setRawText(markup, s)
    markup = CM.cleanText(markup)
    markup = CM.mark_items_in_text(markup, targets, mode="target")
    markup = CM.pruneMarks(markup)
    markup = CM.dropMarks(markup, category='Exclusion')
    # apply modifiers to any targets within the modifiers scope
    return markup


I am working on the file "OSB2015_intro_data.ipynb".
The first task is:
2. built-in documentation
Can you write Python code for it?


I am working on the file "OSB2015_intro_data.ipynb".
The first task is:
5. multimedia
Can you write Python code for it?


I am working on the file "OSB2015_intro_data.ipynb".
The first task is:
6. Notebooks are now viewable directly on github, and also exportable as PDF or HTML.
pandas
Input
Source: IRS.gov
Can you write Python code for it?

?pd.read_csv()

# read in a CSV
# specify that zipcode should be treated as a string rather than an int!
AGI = pd.read_csv('12zpallagi.csv',dtype={'zipcode': str})

AGI.info()


I am working on the file "OSB2015_intro_data.ipynb".
The first task is:
A WILD ZIP CODE APPEARS!
<img src="http://ecdn.funzypics.com/grumpycatmemes/pics/16/OH-No--The-Cute-Kitty-Cats-Are-Attacking-Like-Aliens----The-Hunt-Is-On-Wow-Quest-Guide-.jpg" align="left">
Delete or render null? You decide!
Can you write Python code for it?

AGI_grouped['population'].mean()

#this can also be done using the na_values param upon being read in
null_zips = AGI_grouped['zipcode'] == '00000'
AGI_grouped.loc[null_zips, 'zipcode'] = np.nan

AGI_grouped.head()

AGI_notnull = AGI_grouped.dropna()

AGI_notnull['population'].mean()

AGI_grouped.dropna(inplace=True)

# make a new column with the real amount, not in thousands
AGI_grouped['actual_amount'] = AGI_grouped['amount'] * 1000

# make a mean!
AGI_grouped['weighted_mean_AGI'] = AGI_grouped['actual_amount']/AGI_grouped['population']

#use anonymous functions to change every value in a column!
AGI_grouped['weighted_mean_AGI']= AGI_grouped['weighted_mean_AGI'].apply(lambda x: round(x, 0))

AGI_grouped.head()

# drop columns you won't need
AGI_grouped.drop(['AGI_STUB','amount','actual_amount'],axis=1,inplace=True)

AGI_grouped.head()


I am working on the file "OSB2015_intro_data.ipynb".
The first task is:
Merging! Better than in traffic!
Can you write Python code for it?

# also look into pandas.Series.unique
AGI_subset_geo = AGI[['zipcode','STATE']].drop_duplicates()

AGI_subset_geo

#merge rather than join if you want to use a common column other than the index
AGI_final = pd.merge(AGI_grouped, AGI_subset_geo, how='left', on='zipcode')

AGI_final.head()

# this gives you the greated weighted_mean_AGI first
AGI_final.sort('weighted_mean_AGI',ascending=False).head()

# chain methods!
AGI_final.groupby('STATE').mean().sort('weighted_mean_AGI',ascending=False)

AGI_final.sort('weighted_mean_AGI').head().plot(kind='bar')


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Across runs and experiments, TTQ is:

relatively consistent.
a small part of TTC.
Can you write Python code for it?

ttc_stats = {}

for exp in sessions['experiment'].sort_values().unique():
    for nun in sessions['nunits'].sort_values().unique():
        tag = exp+'_'+str(int(nun))
        ttc_stats[tag] = sessions[ (sessions['experiment'] == exp) &
                                   (sessions['nunits'] == nun) ]['TT_PILOT_LRMS_QUEUING'].describe()

ttc_compare = pd.DataFrame(ttc_stats)

sort_cols = ['exp1_8' , 'exp2_8' , 'exp1_16', 'exp2_16',
             'exp1_32', 'exp2_32', 'exp1_64', 'exp2_64']
ttc_compare = ttc_compare.reindex_axis(sort_cols, axis=1)

ttc_compare.round(2)


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Analysis
We measure a set of durations. Each duration has two and only two
timestamps, the first always preceding in time the second. Each
timestamp represents an event, in this case of a state transition.
Our choice of the durations depends on the design of the experiment for
which we are collecting data. In this case, we want to measure the
overall time to completion (TTC) of the run and isolate two of its
components:

TTQ and Tq: The amount of time spent in the queue waiting for the pilots;
TTX and Tx: The amount of time spent to execute each unit.

We use TTQ and Tq to understand whether queue time has the same
dominance on TTC as we measured on XSEDE. This comparison is
relevant when evaluating the performance of OSG and the
distribution of tasks to OSG, XSEDE, or across both.
TTX and Tx contribute to this understanding by showing how the
supposed heterogeneity of OSG resources affects compute performance.
The experiment is designed to use homogeneous CUs: every CU has the
same compute requirements. Data requirements are not emulated so the
differences in execution time across CUs can be related mainly to
core performance.
Consistency
The first step in analyzing the experiments' dataset is to verify the consistenty of the data. Without such a verification, we cannot trust any of the results our analysis will produce. Feeling a nice chill down the spine thinking about the results you have already published?! ;)
Here we check for the consistency of the state model and of the timestamps. As documented in the RADICAL-Analytics API, a third test mode is available for the event models. Currently, event models are not implemented.
Can you write Python code for it?

os.environ['RADICAL_ANALYTICS_VERBOSE']='ERROR'

for s in sessions['session']:
    s.consistency(['state_model','timestamps'])


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
As expected, TTR is largely equivalent to TTC-TTQ. This tells us that we will have to investigate the time spent describing, binding, scheduling, and executing CUs, measuring whether pilots TTR is spent effectively running CUs or managing them. Also, we will have to measure how much time is spent staging data in and out to and from the resources. In these experiments, data staging was not performed so we will limit our analysis to the execution time of CUs.
In the diagram above, we confirm that the differences previously observed between the TTC of Experiment 1 and 2 (particularly evident when comparing runs with 8 and 64 units). As they depend on running pilots, we investigate whether every run uses the same amount of (active) pilots to execute CUs. The rationale is that when using fewer pilots, running the same amount of CUs will take more time (sequential execution on a single core).
We add a column to the session DataFrame with the number of pilots for each session.
Can you write Python code for it?

# Temporary: workaround for bug ticket \#15. Calculates
# the number of active pilots by looking into the
# length of the list returned by timestamp on the
# PMGR_ACTIVE state.

for sid in sessions.index:
    sessions.ix[sid, 'npilot_active'] = len(sessions.ix[sid, 'session'].filter(etype='pilot', inplace=False).timestamps(state='PMGR_ACTIVE'))

display(sessions[['npilot_active']].head(3))
display(sessions[['npilot_active']].tail(3))


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Entities
We need to define the start and end event for TTQ, TTX, Tq, Tx durations.
As such, we need to choose the RADICAL-Pilot entity or entities that are
relevant to our measurements. We look at what entities were recorded by
the experimental runs.
Can you write Python code for it?

expment  = None
pexpment = None

for sid in sessions.index:
    etypes  = sessions.ix[sid, 'session'].list(['etype'])
    expment = sessions.ix[sid, 'experiment']
    if expment != pexpment:
        print '%s|%s|%s' % (expment, sid, etypes)
        pexpment = expment


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Even with an average of just 4 runs for each workload size, TTQ variance is relatively small with a couple of exceptions, as shown by STD/mean
Can you write Python code for it?

std_mean = (ttc_compare.loc['std']/ttc_compare.loc['mean'])*100
std_mean.round(2)


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Experiment 2 shows better concurrency than Experiment 1. This might be due to the evolution of the RADICAL Pilot code or more performant resources used by the runs of Experiment 2. These data shows at least three elements that need further development:

We lack information on what resource each pilot has been running. Without this information we cannot make statistical inferences about some of the durations we measure.
RADICAL Pilot overheads affect experimental measures at both global (see state models) and local level (see concurrency overheads). Their characterization and, in case, normalization is required by every experiment and need to be considered a first citizen of analytics.
Experiments are potentially affected by small differences in the RADICAL Pilot code base. This would require the use of a stable RADICAL Pilot code base across all the experiment computational campaign.

Time execution (Tx)
Overall, our analysis confirms that the heterogeneity of OSG resources affects TTX and that TTQ has a marginal impact. The following step is trying to characterize statistically the effect of resource heterogeneity on TTX. We begin this characterization by looking at the time that each unit takes to execute on a OSG resource (Tx).
Each CU has the same computing requirements. Therefore, we can aggregate their analysis across experiments. Note that each unit executed on only and only one pilot so the number of pilot that becomes active have no bearing on the units' Tx.
Can you write Python code for it?

# Model of unit durations.
udm = {'UNIT_UMGR_SCHEDULING'   : ['NEW'                         , 'UMGR_SCHEDULING_PENDING'],
       'UNIT_UMGR_BINDING'      : ['UMGR_SCHEDULING_PENDING'     , 'UMGR_SCHEDULING'],
       'IF_UMGR_SCHEDULING'     : ['UMGR_SCHEDULING'             , 'UMGR_STAGING_INPUT_PENDING'],
       'IF_UMGR_QUEING'         : ['UMGR_STAGING_INPUT_PENDING'  , 'UMGR_STAGING_INPUT'],
       #'IF_AGENT_SCHEDULING'    : ['UMGR_STAGING_INPUT'          , 'AGENT_STAGING_INPUT_PENDING'],
       'IF_AGENT_QUEUING'       : ['AGENT_STAGING_INPUT_PENDING' , 'AGENT_STAGING_INPUT'],
       'IF_AGENT_TRANSFERRING'  : ['AGENT_STAGING_INPUT'         , 'AGENT_SCHEDULING_PENDING'],
       'UNIT_AGENT_QUEUING'     : ['AGENT_SCHEDULING_PENDING'    , 'AGENT_SCHEDULING'],
       'UNIT_AGENT_SCHEDULING'  : ['AGENT_SCHEDULING'            , 'AGENT_EXECUTING_PENDING'],
       'UNIT_AGENT_QUEUING_EXEC': ['AGENT_EXECUTING_PENDING'     , 'AGENT_EXECUTING'],
       'UNIT_AGENT_EXECUTING'   : ['AGENT_EXECUTING'             , 'AGENT_STAGING_OUTPUT_PENDING'],
       #'OF_AGENT_QUEUING'       : ['AGENT_STAGING_OUTPUT_PENDING', 'AGENT_STAGING_OUTPUT'],
       #'OF_UMGR_SCHEDULING'     : ['AGENT_STAGING_OUTPUT'        , 'UMGR_STAGING_OUTPUT_PENDING'],
       'OF_UMGR_QUEUING'        : ['UMGR_STAGING_OUTPUT_PENDING' , 'UMGR_STAGING_OUTPUT'],
       'OF_UMGR_TRANSFERRING'   : ['UMGR_STAGING_OUTPUT'         , 'DONE']}

# DataFrame structure for pilot durations.
uds = { 'pid': [],
        'sid': [],
        'experiment' : [],
        'UNIT_UMGR_SCHEDULING'   : [],
        'UNIT_UMGR_BINDING'      : [],
        'IF_UMGR_SCHEDULING'     : [],
        'IF_UMGR_QUEING'         : [],
        'IF_AGENT_SCHEDULING'    : [],
        'IF_AGENT_QUEUING'       : [],
        'IF_AGENT_TRANSFERRING'  : [],
        'UNIT_AGENT_QUEUING'     : [],
        'UNIT_AGENT_SCHEDULING'  : [],
        'UNIT_AGENT_QUEUING_EXEC': [],
        'UNIT_AGENT_EXECUTING'   : [],
        'OF_AGENT_QUEUING'       : [],
        'OF_UMGR_SCHEDULING'     :...


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Experiment Settings
RADICAL-Pilot save runtime data in two types of file: profiles and databases. Profile files are written by the agent module on the work nodes of the remote resources; database files are stored in the MongoDB instance used for the workload execution. Both types of file need to be retrieved from the remote resource and from the MongoDB instance. Currently, ...
The data used in this notebook are collected in two compressed bzip2 tar archives, one for each experiment dataset. The two archives need to be decompressed into radical.analytics/use_cases/rp_on_osg/data. Once unzipped, we acquire the datasets by constructing a ra.session object for each experimental run.
We use a helper function to construct the session objects in bulk while keeping track of the experiment to which each run belong. We save both to a Pandas DataFrame. This helps to elaborate the datasets furthers offering methods specifically aimed at data analysis and plotting.
Can you write Python code for it?

def load_data(rdir):
    sessions = {}
    experiments = {}
    start = rdir.rfind(os.sep)+1
    for path, dirs, files in os.walk(rdir):
        folders = path[start:].split(os.sep)
        if len(path[start:].split(os.sep)) == 2:
            sid = os.path.basename(glob.glob('%s/*.json' % path)[0])[:-5]
            if sid not in sessions.keys():
                sessions[sid] = {}
            sessions[sid] = ra.Session(sid, 'radical.pilot', src=path)
            experiments[sid] = folders[0]
    return sessions, experiments

# Load experiments' dataset into ra.session objects
# stored in a DataFrame.
rdir = 'data/'
sessions, experiments = load_data(rdir)
sessions = pd.DataFrame({'session': sessions,
                         'experiment': experiments})

# Check the first/last 3 rows
display(sessions.head(3))
display(sessions.tail(3))


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
For 8 and 16 CUs, the TTC of the second experiment shows a mean twice as large as those of the first experiment. Less pronounced is the difference for mean of the TTC of 32 CUs. The mean of the TTC of the first experiment is 25% smaller that the one of the second experiment for 64 CUs.
The standard deviation among runs of the same experiment and number of CUs, varies between the two experiments. We describe this variation by calculating and comparing STD/mean for each experiment and workload size.
Can you write Python code for it?


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Load all the Python modules we will use for the analysis. Note that both RADICAL Utils and RADICAL Pilot need to be loaded alongside RADICAL Analytics.
Can you write Python code for it?

import os
import sys
import glob
import pprint

import numpy as np
import scipy as sp
import pandas as pd
import scipy.stats as sps
import statsmodels.api as sm

import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab
import matplotlib.ticker as ticker
import matplotlib.gridspec as gridspec

import radical.utils as ru
import radical.pilot as rp
import radical.analytics as ra

from IPython.display import display


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Skewness and Kurtosis
Can you write Python code for it?

Tx_measures['skew'] = sps.skew(Txs, bias=True)
Tx_measures['kurt'] = sps.kurtosis(Txs)

u_skew_test = sps.skewtest(Txs)
u_kurt_test = sps.kurtosistest(Txs)

print Tx_measures['skew']
print Tx_measures['kurt']

print u_skew_test
print u_kurt_test

metric      = 'T_x'
description = 'Histogram of $%s$' % metric
task        = 'Gromacs emulation'
repetition  = '$%s$ repetitions' % len(Txs)
resource    = 'XSEDE OSG Virtual Cluster'
stats       = '$\mu$=%.3f,\ \sigma=%.3f,\ SE_\mu=%.3f$' % (Tx_measures['mu'], Tx_measures['std'], Tx_measures['sem'])
title       = '%s.\n%s; %s;\n%s.' % (description, repetition, resource, stats)

ax = fig_setup()

n, bins, patches = ax.hist(Txs, bins='fd',
        normed=1,
        histtype='stepfilled',
        label="$T_x$",
        linewidth=0.75,
        edgecolor=tableau20[8],
        color=tableau20[9])

# ax.xaxis.set_major_locator(ticker.MultipleLocator(50))
# ax.set_xlim(150, 700)
ax.set_ylim(0.0, 0.005)

plt.ylabel('$P(Tx)$')
plt.xlabel('$T_x$ (s)')
plt.title(title)#, y=1.05)
plt.legend(loc='upper right')

plt.savefig('figures/osg_cu_spread_hist.pdf', dpi=600, bbox_inches='tight')

Tx_measures_exp1['skew'] = sps.skew(Txs, bias=True)
Tx_measures_exp1['kurt'] = sps.kurtosis(Txs)

u_skew_test = sps.skewtest(Txs)
u_kurt_test = sps.kurtosistest(Txs)

print Tx_measures_exp1['skew']
print Tx_measures_exp1['kurt']

print u_skew_test
print u_kurt_test

metric      = 'T_x_exp1'
description = 'Histogram of $%s$' % metric
task        = 'Gromacs emulation'
repetition  = '$%s$ repetitions' % len(Txs_exp1)
resource    = 'XSEDE OSG Virtual Cluster'
stats       = '$\mu$=%.3f,\ \sigma=%.3f,\ SE_\mu=%.3f$' % (Tx_measures_exp1['mu'], Tx_measures_exp1['std'], Tx_measures_exp1['sem'])
title       = '%s.\n%s; %s;\n%s.' % (description, repetition, resource, stats)

ax = fig_setup()

n, bins, patches = ax.hist(Txs_exp1, bins='fd',
        normed=1,
        histtype='stepfilled',
        label="$T_x$",
        linewidth=0.75,
        edgecolor=tableau20[0],
        color=tableau20[1])

#...


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
The diagram confirms the similarity between the size of TTR and TTX. More analytically:
Can you write Python code for it?


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Total Time eXecuting (TTX)
We now calculate the total amount of TTC spent executing CUs. This will tell us how much of TTR is indeed spent executing CUs.
Can you write Python code for it?

# Add number of unique resorces per session.

for sid in sessions.index:
    sessions.ix[sid, 'n_unique_host'] = len(pilots[pilots['sid'] == sid]['parsed_hostID'].unique())

fig = plt.figure(figsize=(13,14))

title = 'XSEDE OSG Virtual Cluster'
subtitle = 'TTQ, TTR, TTX and TTC with Number of Active Pilots (black) and Number of Unique Resources (red)'
fig.suptitle('%s:\n%s.' % (title, subtitle), fontsize=16)

defs = {'ttq': 'TTQ = Total Time Queuing pilots',
        'ttr': 'TTR = Total Time Running pilots',
        'ttx': 'TTR = Total Time Executing compute units',
        'ttc': 'TTC = Total Time Completing experiment'}
defslist = '%s;\n%s;\n%s;\n%s.' % (defs['ttq'], defs['ttr'], defs['ttx'], defs['ttc'])
plt.figtext(.38,.89, defslist, fontsize=14, ha='left')

gs = []
grid  = gridspec.GridSpec(2, 2)
grid.update(wspace=0.4, hspace=0.4, top=0.825)
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[0]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[1]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[2]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 5, subplot_spec=grid[3]))

ttq_subplots = []
for exp in sessions['experiment'].sort_values().unique():
    for nun in sessions['nunits'].sort_values().unique():
        if not sessions[ (sessions['experiment'] == exp) &
                 (sessions['nunits'] == nun) ].empty:
            ttq_subplots.append(sessions[ (sessions['experiment'] == exp) &
                                          (sessions['nunits'] == nun) ].sort_values('TTC'))

colors = {'exp1': [tableau20[0] ,tableau20[18],tableau20[1] ,tableau20[19]],
          'exp2': [tableau20[2] ,tableau20[6] ,tableau20[3] ,tableau20[7] ],
          'exp3': [tableau20[8] ,tableau20[12],tableau20[9] ,tableau20[13]],
          'exp4': [tableau20[4] ,tableau20[16],tableau20[5] ,tableau20[17]],
          'exp5': [tableau20[10],tableau20[14],tableau20[11],tableau20[15]]}

nun_exp = []
nun_exp.append(len(sessions[sessions['experiment'] ==...


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Unit Durations
We define 15 durations to measure the aggreted time spent by all the units of a session in each state:
| Duration                           | Start timestamp              | End time Stamp               | Description |
|------------------------------------|------------------------------|------------------------------|-------------|
| TT_UNIT_UMGR_SCHEDULING            | NEW                          | UMGR_SCHEDULING_PENDING      | total time spent by a unit being scheduled to a UMGR |
| TT_UNIT_UMGR_BINDING               | UMGR_SCHEDULING_PENDING      | UMGR_SCHEDULING              | total time spent by a unit being bound to a pilot by a UMGR |
| TT_IF_UMGR_SCHEDULING              | UMGR_SCHEDULING              | UMGR_STAGING_INPUT_PENDING   | total time spent by input file(s) being scheduled to a UMGR |
| TT_IF_UMGR_QUEING                  | UMGR_STAGING_INPUT_PENDING   | UMGR_STAGING_INPUT           | total time spent by input file(s) queuing in a UMGR |
| TT_IF_AGENT_SCHEDULING             | UMGR_STAGING_INPUT           | AGENT_STAGING_INPUT_PENDING  | total time spent by input file(s) being scheduled to an agent's MongoDB queue |
| TT_IF_AGENT_QUEUING                | AGENT_STAGING_INPUT_PENDING  | AGENT_STAGING_INPUT          | total time spent by input file(s) queuing in an agent's MongoDB queue |
| TT_IF_AGENT_TRANSFERRING           | AGENT_STAGING_INPUT          | AGENT_SCHEDULING_PENDING     | total time spent by input file(s)' payload to be transferred from where the UMGR is being executed (e.g., the user's workstation) to the resource on which the agent is executing |
| TT_UNIT_AGENT_QUEUING              | AGENT_SCHEDULING_PENDING     | AGENT_SCHEDULING             | total time spent by a unit in the agent's scheduling queue |
| TT_UNIT_AGENT_SCHEDULING           | AGENT_SCHEDULING             | AGENT_EXECUTING_PENDING      | total time spent by a unit to be scheduled to the agent's executing queue |
| TT_UNIT_AGENT_QUEUING_EXECUTION    | AGENT_EXECUTING_PENDING      |...
Can you write Python code for it?

# Model of unit durations.
udm = {'TT_UNIT_UMGR_SCHEDULING'   : ['NEW'                         , 'UMGR_SCHEDULING_PENDING'],
       'TT_UNIT_UMGR_BINDING'      : ['UMGR_SCHEDULING_PENDING'     , 'UMGR_SCHEDULING'],
       'TT_IF_UMGR_SCHEDULING'     : ['UMGR_SCHEDULING'             , 'UMGR_STAGING_INPUT_PENDING'],
       'TT_IF_UMGR_QUEING'         : ['UMGR_STAGING_INPUT_PENDING'  , 'UMGR_STAGING_INPUT'],
       'TT_IF_AGENT_SCHEDULING'    : ['UMGR_STAGING_INPUT'          , 'AGENT_STAGING_INPUT_PENDING'],
       'TT_IF_AGENT_QUEUING'       : ['AGENT_STAGING_INPUT_PENDING' , 'AGENT_STAGING_INPUT'],
       'TT_IF_AGENT_TRANSFERRING'  : ['AGENT_STAGING_INPUT'         , 'AGENT_SCHEDULING_PENDING'],
       'TT_UNIT_AGENT_QUEUING'     : ['AGENT_SCHEDULING_PENDING'    , 'AGENT_SCHEDULING'],
       'TT_UNIT_AGENT_SCHEDULING'  : ['AGENT_SCHEDULING'            , 'AGENT_EXECUTING_PENDING'],
       'TT_UNIT_AGENT_QUEUING_EXEC': ['AGENT_EXECUTING_PENDING'     , 'AGENT_EXECUTING'],
       'TT_UNIT_AGENT_EXECUTING'   : ['AGENT_EXECUTING'             , 'AGENT_STAGING_OUTPUT_PENDING'],
       'TT_OF_AGENT_QUEUING'       : ['AGENT_STAGING_OUTPUT_PENDING', 'AGENT_STAGING_OUTPUT'],
       'TT_OF_UMGR_SCHEDULING'     : ['AGENT_STAGING_OUTPUT'        , 'UMGR_STAGING_OUTPUT_PENDING'],
       'TT_OF_UMGR_QUEUING'        : ['UMGR_STAGING_OUTPUT_PENDING' , 'UMGR_STAGING_OUTPUT'],
       'TT_OF_UMGR_TRANSFERRING'   : ['UMGR_STAGING_OUTPUT'         , 'DONE']}

# Calculate total unit durations for each session.
for sid in sessions.index:
    s = sessions.ix[sid, 'session'].filter(etype='unit', inplace=False)
    for d in udm.keys():
        sessions.ix[sid, d] = s.duration(udm[d])

# Print the new columns of the session DF with total unit durations.
display(sessions[['TT_UNIT_UMGR_SCHEDULING'   , 'TT_UNIT_UMGR_BINDING'   , 'TT_IF_UMGR_SCHEDULING'   ,
                  'TT_IF_UMGR_QUEING'         , 'TT_IF_AGENT_SCHEDULING' , 'TT_IF_AGENT_QUEUING'     ,
                  'TT_IF_AGENT_TRANSFERRING'  , 'TT_UNIT_AGENT_QUEUING'  ,...


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
Variation within Experiment 1 is between 7 and 18% of mean, almost proportionally increasing with the increasing of the number of unit executed. Variation in Experiment 2 is more pronounced, ranging from 25 to 78% of the mean. Clearly, these values are not indicative.
While our analysis showed that this difference is due to the varying number of active pilots in Experiment 2, the relative uniformity of Experiment 1 may be also a byproduct of a reduced functionality of RP that limited the number of resources in which the experiment was able to run successfully. Experiment 1 failed when any of the pilots failed while in Experiment 2, failures where limited to pilots, without affecting the overall execution.
Concurrency
Finally, another element that should be considered is the impact of RP overheads on the execution of CUs. While the comparison between TTR and TTX excluded major overheads outside TTX, the averarages between the two parameters show that TTR grown bigger than TTX proportionally to the increase of the number of CU executed. This can be measured by looking at the degree of concurrency of CU executions across pilots.
Here we plot this degree only for 64 CUs comparing both exp1 and exp2 runs.
Can you write Python code for it?

from collections import OrderedDict

cTTXs = OrderedDict()
ncu = 64

ss = sessions[sessions['nunits'] == ncu].sort_values(['experiment','TTC'])['session']

for s in ss:
    cTTXs[s._sid] = s.filter(etype='unit').concurrency(state=['AGENT_EXECUTING','AGENT_STAGING_OUTPUT_PENDING'],
                                                       sampling=1)

for sid, cTTX in cTTXs.iteritems():
    title = 'Degree of Concurrent Execution of %s CUs - XSEDE OSG Virtual Cluster\nSession %s' % (ncu, sid)
    x = [x[0] for x in cTTX]
    y = [y[1] for y in cTTX]

    color = tableau20[2]
    if 'ming' in sid:
        color = tableau20[0]

    fig, ax = fig_setup()
    fig.suptitle(title, fontsize=14)
    ax.set_xlabel('Time (s)')
    ax.set_ylabel('Number of CU')
    display(ax.plot(x, y, marker='.',  linestyle='', color=color))


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
We add the numbers of pilot that become active in each run to the plot we used above to show TTQ, TTR, and TTC.
Can you write Python code for it?

fig = plt.figure(figsize=(13,14))

title = 'XSEDE OSG Virtual Cluster'
subtitle = 'TTQ, TTR and TTC with Number of Active Pilots'
defs = {'ttq': 'TTQ = Total Time Queuing pilots',
        'ttr': 'TTR = Total Time Running pilots',
        'ttc': 'TTC = Total Time Completing experiment'}

fig.suptitle('%s:\n%s.\n%s;\n%s;\n%s.' % (title,
                                      subtitle,
                                      defs['ttq'],
                                      defs['ttr'],
                                      defs['ttc']), fontsize=14)

gs = []
grid  = gridspec.GridSpec(2, 2)
grid.update(wspace=0.4, top=0.85)
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[0]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[1]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[2]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 5, subplot_spec=grid[3]))

ttq_subplots = []
for exp in sessions['experiment'].sort_values().unique():
    for nun in sessions['nunits'].sort_values().unique():
        if not sessions[ (sessions['experiment'] == exp) &
                 (sessions['nunits'] == nun) ].empty:
            ttq_subplots.append(sessions[ (sessions['experiment'] == exp) &
                                          (sessions['nunits'] == nun) ].sort_values('TTC'))

colors = {'exp1': [tableau20[0] ,tableau20[18],tableau20[19]],
          'exp2': [tableau20[2] ,tableau20[6] ,tableau20[7] ],
          'exp3': [tableau20[8] ,tableau20[12],tableau20[13]],
          'exp4': [tableau20[4] ,tableau20[16],tableau20[17]],
          'exp5': [tableau20[10],tableau20[14],tableau20[15]]}

nun_exp = []
nun_exp.append(len(sessions[sessions['experiment'] == 'exp1']['nunits'].sort_values().unique()))
nun_exp.append(len(sessions[sessions['experiment'] == 'exp2']['nunits'].sort_values().unique()))
nun_exp.append(len(sessions[sessions['experiment'] == 'exp3']['nunits'].sort_values().unique()))
nun_exp.append(len(sessions[sessions['experiment'] ==...


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
We also add a column to the session dataframe with the number of units for each session.
Can you write Python code for it?

for sid in sessions.index:
    sessions.ix[sid, 'nunits'] = len(sessions.ix[sid, 'session'].filter(etype='unit', inplace=False).get())

display(sessions[['nunits']].head(3))
display(sessions[['nunits']].tail(3))


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
We calculate add the name of the resource (hostID) on which the pilot (agent) have become active to the pilots DataFrame. Often, the hostID recoreded by RADICAL-Pilot is not the public name of the resource on whic the pilot becomes active but, instead, the name of a working/compute node/unit of that resource. We use an heuristic to isolate the portion of the hostID string that is common to all the nodes/units of the same resource. It should be noted though, that in some cases this is not possible.
Can you write Python code for it?

def parse_osg_hostid(hostid):
    '''
    Heuristic: eliminate node-specific information from hostID.
    '''
    domain = None

    # Split domain name from IP.
    host = hostid.split(':')

    # Split domain name into words.
    words = host[0].split('.')

    # Get the words in the domain name that do not contain
    # numbers. Most hostnames have no number but there are
    # exceptions.
    literals = [l for l in words if not any((number in set('0123456789')) for number in l)]

    # Check for exceptions:
    # a. every word of the domain name has a number
    if len(literals) == 0:

        # Some hostname use '-' instead of '.' as word separator.
        # The parser would have returned a single word and the
        # any of that word may have a number.
        if '-' in host[0]:
            words = host[0].split('-')
            literals = [l for l in words if not any((number in set('0123456789')) for number in l)]

            # FIXME: We do not check the size of literals.
            domain = '.'.join(literals)

        # Some hostnames may have only the name of the node. We
        # have to keep the IP to decide later on whether two nodes
        # are likely to belong to the same cluster.
        elif 'n' in host[0] or 'nod' in host[0]:
            domain = '.'.join(host)

        # The hostname is identified by an alphanumeric string
        else:
            domain = '.'.join(host)

    # Some hostnames DO have numbers in their name.
    elif len(literals) == 1:
        domain = '.'.join(words[1:])

    # Some hostname are just simple to parse.
    else:
        domain = '.'.join(literals)

    return domain


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
We notice lage variations both within and across experiments. Specifically:
Can you write Python code for it?

ttx_stats = {}

for exp in sessions['experiment'].sort_values().unique():
    for nun in sessions['nunits'].sort_values().unique():
        tag = exp+'_'+str(int(nun))
        ttx_stats[tag] = sessions[ (sessions['experiment'] == exp) &
                                   (sessions['nunits'] == nun) ]['TT_UNIT_AGENT_EXECUTING'].describe()

ttx_compare = pd.DataFrame(ttx_stats)

sort_cols_runs = ['exp1_8' , 'exp2_8' , 'exp1_16', 'exp2_16',
                  'exp1_32', 'exp2_32', 'exp1_64', 'exp2_64']
sort_cols_exp  = ['exp1_8' , 'exp1_16', 'exp1_32', 'exp1_64',
                  'exp2_8',  'exp2_16', 'exp2_32', 'exp2_64']

ttx_compare_runs = ttx_compare.reindex_axis(sort_cols_runs, axis=1)
ttx_compare_exp  = ttx_compare.reindex_axis(sort_cols_exp,  axis=1)

ttx_compare_exp.round(2)

std_mean = (ttx_compare_exp.loc['std']/ttx_compare_exp.loc['mean'])*100
std_mean.round(2)


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
We notice that STD/mean among repetitions of the same run in Experiment 1 goes from 6.55% to 19.77%, increasing with the increase of the number of CUs. In Experiment 2, STD/mean goes from 20.63% up to 56.18%, independently from the amount of CUs executed by the repeated run. This shows:

More measurements are needed to lower STD;
TTC of experiment 2 may be influenced by one or more factors not present or relevant in Experiment 1.

In order to clarify this discrepancy between Experiments, we need to measure the components of TTC to understand what determines the differences in TTC within the same experiment and between the two. These measurements requires to:

define the state model and transitions of the pilot and CU entities;
verify that this model has been consistently implemented by each experimental run;
define and measure the duration of each state and compare them across experimental runs.

Pilot State Model
From the RADICAL-Pilot documentation and state model description, we know that:
<img src="images/global_state_model_rp_paper.png" width="600">
The states of the pilots are therefore as follow:
* pilot described, state NEW;
* pilot being queued in a pilot manager (PMGR), state PMGR_LAUNCHING_PENDING;
* pilot being queued in a local resource management system (LRMS), state PMGR_LAUNCHING;
* pilot having a bootstrapping agent, state PMGR_ACTIVE_PENDING;
* pilot having an active agent, state PMGR_ACTIVE;
* pilot marked as done by the PMGR, state DONE.
We verify whether our run match this model and we test whether the state model of each pilot of every session of all our experiments is the same. In this way we will know:

whether our data are consistent with the authoritative RADICAL-Pilot state model; and
what states of the pilots we can compare in our analysis given our dataset.
Can you write Python code for it?

last_sv = None
last_id = None

for s in sessions['session']:
    sv = s.describe('state_values', etype=['pilot']).values()[0].values()[0]

    if last_sv and last_sv != sv:
        print "Different state models:\n%s = %s\n%s = %s" % (last_id, last_sv, sid, sv)

    last_sv = sv
    last_id = s._sid

pprint.pprint(last_sv)


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
We plot the frequency of Tq for both experiments as histogram. Ideally, this should be the first step towards the definition of the characteristic distribution of Tq. Part of this characterization will be to study how stable this distribution is across time, i.e., between two experiments executed at different point in time. As we know that the pool or resources of OSG Is both heterogeneous and dynamic, we expect this distribution not to be stable across time due to the potentially different pool of resources available at any point in time.
Can you write Python code for it?

fig, ax = fig_setup()
title='XSEDE OSG Virtual Cluster\nDensity of Pilot Tq'

tq_exp1 = pilots[pilots['experiment'].str.contains('exp1')]['LRMS_QUEUING'].dropna().reset_index(drop=True)
tq_exp2 = pilots[pilots['experiment'].str.contains('exp2')]['LRMS_QUEUING'].dropna().reset_index(drop=True)
tq_exp3 = pilots[pilots['experiment'].str.contains('exp3')]['LRMS_QUEUING'].dropna().reset_index(drop=True)
tq_exp4 = pilots[pilots['experiment'].str.contains('exp4')]['LRMS_QUEUING'].dropna().reset_index(drop=True)

plots = pd.DataFrame({'exp1': tq_exp1, 'exp2': tq_exp2,  'exp3': tq_exp3, 'exp4': tq_exp4})

#plots.plot.hist(ax=ax, color=[tableau20[19],tableau20[7],tableau20[13],tableau20[17]], title=title)
plots.plot.density(ax=ax, color=[tableau20[0],tableau20[2],tableau20[8],tableau20[4]], title=title)

ax.set_xlabel('Time (s)')
ax.legend(labels=['Tq Experiment 1','Tq Experiment 2','Tq Experiment 3','Tq Experiment 4'])

plt.savefig('figures/osg_tq_frequency.pdf', dpi=600, bbox_inches='tight')


I am working on the file "OSG/analysis/osg_analysis.ipynb".
The first task is:
We see:

A large variation among the TTC of the runs of both experiments.
Variations are similar between the two experiments.

We still do not know:

How the observed differences in TTC varies depending on the size of the workload.
What and how the entities of RADICAL-Pilot contribute to TTC and its variations.
What and how resource properties contribute to TTC and its variations.

First, we subdivide each plot into four plots, one for each workload size: 8, 16, 32, 64 CUs.
Can you write Python code for it?

fig = plt.figure(figsize=(13,14))

title = 'XSEDE OSG Virtual Cluster'
subtitle = 'TTC'
defs = {'ttq': 'TTQ = Total Time Queuing pilots',
        'ttr': 'TTR = Total Time Running pilots',
        'ttc': 'TTC = Total Time Completing experiment'}

fig.suptitle('%s:\n%s.\n%s.' % (title,
                                subtitle,
                                defs['ttc']), fontsize=14)

gs = []
grid  = gridspec.GridSpec(2, 2)
grid.update(wspace=0.4, top=0.85)
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[0]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[1]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 4, subplot_spec=grid[2]))
gs.append(gridspec.GridSpecFromSubplotSpec(1, 5, subplot_spec=grid[3]))

ttc_subplots = []
for exp in sessions['experiment'].sort_values().unique():
    for nun in sessions['nunits'].sort_values().unique():
        if not sessions[ (sessions['experiment'] == exp) &
                 (sessions['nunits'] == nun) ].empty:
            ttc_subplots.append(sessions[ (sessions['experiment'] == exp) &
                                          (sessions['nunits'] == nun) ].sort_values('TTC'))

colors = {'exp1': [tableau20[19]],
          'exp2': [tableau20[7] ],
          'exp3': [tableau20[13]],
          'exp4': [tableau20[17]],
          'exp5': [tableau20[15]]}

nun_exp = []
nun_exp.append(len(sessions[sessions['experiment'] == 'exp1']['nunits'].sort_values().unique()))
nun_exp.append(len(sessions[sessions['experiment'] == 'exp2']['nunits'].sort_values().unique()))
nun_exp.append(len(sessions[sessions['experiment'] == 'exp3']['nunits'].sort_values().unique()))
nun_exp.append(len(sessions[sessions['experiment'] == 'exp4']['nunits'].sort_values().unique()))

ax = []
i  = 0
while(i < len(ttc_subplots)):
    for gn in range(4):
        for gc in range(nun_exp[gn]):
            session = ttc_subplots.pop(0)
            experiment = session['experiment'].unique()[0]
            ntasks = int(session['nunits'].unique()[0])
            repetitions =...


I am working on the file "Oanda v1 REST-oandapy/01.02 Understanding the Documentations.ipynb".
The first task is:
Once we have done the above, we can start making API call to extract information.
Can you write Python code for it?

response = oanda.get_prices(instruments="USD_JPY")
print(response)


I am working on the file "Oanda v1 REST-oandapy/01.02 Understanding the Documentations.ipynb".
The first task is:
We need to provide Python with the Account ID and API Key. We store these two under two variables called account_id and api_key. We will show in later lesson how we can segregate this from our main code body thus protecting our account.
Can you write Python code for it?

account_id = 0123456
api_key = "this-is-not-the-actual-api-key-as-we-will-see-how-we-can-protect-"


I am working on the file "Oanda v1 REST-oandapy/01.02 Understanding the Documentations.ipynb".
The first task is:
We then instantiate as follow the next step shows. You can use other variable name instead of oanda here.
Can you write Python code for it?

oanda = oandapy.API(environment="practice",
                    access_token=api_key)


I am working on the file "Oanda v1 REST-oandapy/02.00 Rates Information.ipynb".
The first task is:
Obtaining a List of Trabable Instruments
get_instruments(self, account_id, **params)
Can you write Python code for it?

response = oanda.get_instruments(account_id)
pd.DataFrame(response['instruments']).head()

pd.DataFrame(response['instruments']).tail()


I am working on the file "Oanda v1 REST-oandapy/02.00 Rates Information.ipynb".
The first task is:
The above value is in the format of Python lists object. Python lists object can be identified by the squared brackets [].
To access the ask price for the first lists object:
Can you write Python code for it?


I am working on the file "Oanda v1 REST-oandapy/02.00 Rates Information.ipynb".
The first task is:
The oandapy wraps the Oanda API in a format that make our codes cleaner and the task of extracting information from the API easier.
The configparser is an in-built Python library where it allows us to read configuration files easily. The documentation can be found via this link. It is a good practice to segregate your Python code from your API keys.
Using the configparser, we extract the relevant Account ID and API Key information.
We shall come back to the pandas library later.
Can you write Python code for it?


I am working on the file "Oanda v1 REST-oandapy/02.00 Rates Information.ipynb".
The first task is:
Using the instance we created earlier with the name oanda, we make an API call using get_prices method whilst passing the argument instruments="AUD_USD,NZD_USD,EUR_USD" to the API. The argument provided tells the Oanda API to return information only on these instruments. All the data is to be stored in the response variable.
We then print the variable to screen to observe what information has been returned.
The returned data is in the format of Python dictionary object. We can tell that it is a Python dictionary object by the curly braces {}.
'prices' here is the key. to access the dictionary elements, we use square brackets along with the key to obtain its value. Example:
Can you write Python code for it?


I am working on the file "Oanda v1 REST-oandapy/02.00 Rates Information.ipynb".
The first task is:
We can also use the pandas DataFrame to display the value in a familiar and easier to read table format. The pandas library is a powerful library created for the sole purpose of data analysis
Can you write Python code for it?


I am working on the file "Oanda v1 REST-oandapy/02.00 Rates Information.ipynb".
The first task is:
We can store individual value for future use
Can you write Python code for it?

data = response['prices']
time_stamp = data[0]['time']
instrument = data[0]['instrument']
bid_price = data[0]['bid']
ask_price = data[0]['ask']
print("[{}] {} bid={} ask={}".format(time_stamp,
                                     instrument,
                                     bid_price,
                                     ask_price))


I am working on the file "Oanda v1 REST-oandapy/03.00 Account Information.ipynb".
The first task is:
For a detailed explanation of the above, please refer to Rates Information.
Can you write Python code for it?

response = oanda.get_accounts()
print(response)

pd.DataFrame(response['accounts'])


I am working on the file "Oanda v1 REST-oandapy/04.00 Order Management.ipynb".
The first task is:
Close Order
close_order(self, account_id, order_id, **params)
Can you write Python code for it?

response = oanda.get_orders(account_id)
id = response['orders'][0]['id']

oanda.close_order(account_id, order_id=id)


I am working on the file "Oanda v1 REST-oandapy/04.00 Order Management.ipynb".
The first task is:
Getting Specific Order Information
get_order(self, account_id, order_id, **params)
Can you write Python code for it?

response = oanda.get_orders(account_id)
id = response['orders'][0]['id']

oanda.get_order(account_id, order_id=id)


I am working on the file "Oanda v1 REST-oandapy/04.00 Order Management.ipynb".
The first task is:
Now when we check the orders. The above order has been closed and removed without being filled. There is only one outstanding order now.
Can you write Python code for it?


I am working on the file "Oanda v1 REST-oandapy/05.00 Trade Management.ipynb".
The first task is:
Close An Open Trade
close_trade(self, account_id, trade_id, **params)
Can you write Python code for it?

response = oanda.close_trade(account_id, instrument='EUR_USD',
                             trade_id=trade_id)
print(response)

response = oanda.get_trades(account_id)
pd.DataFrame(response['trades'])


I am working on the file "Oanda v1 REST-oandapy/05.00 Trade Management.ipynb".
The first task is:
Get Specific Trade Information
get_trade(self, account_id, trade_id, **params)
Can you write Python code for it?

response = oanda.get_trade(account_id,trade_id=trade_id)
print(response)


I am working on the file "Oanda v1 REST-oandapy/06.00 Position Management.ipynb".
The first task is:
Close an Existing Position
close_position(self, account_id, instrument, **params)
Can you write Python code for it?

response = oanda.close_position(account_id,
                                instrument="USD_JPY")
print(response)

response = oanda.get_positions(account_id)
pd.DataFrame(response['positions'])


I am working on the file "Oanda v1 REST-oandapy/06.00 Position Management.ipynb".
The first task is:
Get the Position for An Instrument
get_position(self, account_id, instrument, **params)
Can you write Python code for it?

response = oanda.get_position(account_id,
                              instrument="USD_JPY")
print(response)


I am working on the file "Oanda v1 REST-oandapy/09.00 Bonus Materials I.ipynb".
The first task is:
Auto Chartist
Can you write Python code for it?

response = oanda.get_autochartist()

pd.DataFrame(response['signals'][:5])


I am working on the file "Oanda v1 REST-oandapy/09.00 Bonus Materials I.ipynb".
The first task is:
Get Historical Spread
Can you write Python code for it?

response = oanda.get_historical_spreads(instrument='GBP_USD', period=600)
print(response)


I am working on the file "Oanda v1 REST-oandapy/09.00 Bonus Materials I.ipynb".
The first task is:
Historical Position Ratios
Can you write Python code for it?

response = oanda.get_historical_position_ratios(instrument='USD_JPY',
                                                period=60)
print(response['data']['USD_JPY']['data'])

df = pd.DataFrame(response['data']['USD_JPY']['data'])
df.columns = ["Timestamp", "Long_pos_ratio", "exchange_rate"]
df["Timestamp"] = pd.to_datetime(df['Timestamp'], unit='s')
df

df = df.set_index("Timestamp")

df['Long_pos_ratio'].plot();


I am working on the file "Oanda v1 REST-oandapy/09.00 Bonus Materials I.ipynb".
The first task is:
Obtaining Historical Economic Calendar
Can you write Python code for it?

calendar = oanda.get_eco_calendar(instrument="EUR_USD", period=86400)

print(calendar)

df = pd.DataFrame(calendar)
df['timestamp'] = pd.to_datetime(df['timestamp'], unit='s')
df.tail(10)


I am working on the file "Oanda v1 REST-oandapy/09.00 Bonus Materials I.ipynb".
The first task is:
Orderbook
Can you write Python code for it?

response = oanda.get_orderbook(instrument="EUR_USD", period=1)

pd.DataFrame(response)

pd.DataFrame(response['1485820801']['price_points'])


I am working on the file "Oanda v1 REST-oandapy/10.00 Bonus Materials II.ipynb".
The first task is:
Experiment With the Training Data Set
Can you write Python code for it?

k = range(1,13)
h = 1

for oo in k:
    dtrain['signal'] = np.sign(dtrain['rtns'].rolling(oo).sum())
    dtrain['strategy_rtn'] =  dtrain['signal'].shift(1) * dtrain['rtns']
    res = dtrain['strategy_rtn'].dropna().sum()
    print('{0:3} {1:>8.4f}'.format(oo, res))


I am working on the file "Oanda v1 REST-oandapy/10.00 Bonus Materials II.ipynb".
The first task is:
Vectorized Backtesting With the Test Set - Momentum
Note: we use the k=11,h=1 combination as it has the highest returns in the training set
Can you write Python code for it?

dtest['signal'] = np.sign(dtest['rtns'].rolling(11).sum())
dtest['result'] = dtest['signal'].shift(1) * dtest['rtns']
dtest['result'].dropna().cumsum().plot(figsize=(10,6));


I am working on the file "Oanda v1 REST-oandapy/10.00 Bonus Materials II.ipynb".
The first task is:
Vectorized Backtesting With the Test Set - Reversion
Note: we use the k=3,h=1 combination as it has the highest returns in the training set for a reversion strategy
Can you write Python code for it?

dtest['signal'] = - np.sign(dtest['rtns'].rolling(3).sum())
dtest['result'] = dtest['signal'].shift(1) * dtest['rtns']
dtest['result'].dropna().cumsum().plot(figsize=(10,6));


I am working on the file "Oanda v20 REST-oandapyV20/01.02 Understanding the Documentations.ipynb".
The first task is:
Below is an extract from the oandapyv20 documentation. Note that you only need to provide accountID and params.

The corresponding Oanda API for pricing information:

We then instantiate as the next step shows. You can use other variable name aside from api.
Can you write Python code for it?

api = API(access_token)                                     # step 1

params ={"instruments": "EUR_USD"}                          # step 2

r = pricing.PricingInfo(accountID=accountID, params=params) # step 3

api.request(r);                                             # step 4


I am working on the file "Oanda v20 REST-oandapyV20/03.00 Account Information.ipynb".
The first task is:
Account Instruments
Can you write Python code for it?

r = accounts.AccountInstruments(accountID=accountID, params = "EUR_USD")

client.request(r)

pd.DataFrame(r.response['instruments'])


I am working on the file "Oanda v20 REST-oandapyV20/03.00 Account Information.ipynb".
The first task is:
Account List
Can you write Python code for it?

r = accounts.AccountList()

client.request(r)

print(r.response)


I am working on the file "Oanda v20 REST-oandapyV20/04.00 Order Management.ipynb".
The first task is:
Get a List of Orders for an Account
Can you write Python code for it?

r = orders.OrderList(accountID)

client.request(r)

print(r.response)

pd.Series(r.response['orders'][0])


I am working on the file "Oanda v20 REST-oandapyV20/04.00 Order Management.ipynb".
The first task is:
List all Pending Orders in an Account
Can you write Python code for it?

r = orders.OrdersPending(accountID)

client.request(r)

print(r.response)

res = r.response['orders']

print(res)

last_order_id = res[0]['id']

pd.Series(r.response['orders'][0])


I am working on the file "Oanda v20 REST-oandapyV20/04.00 Order Management.ipynb".
The first task is:
Replace an Order in an Account by simultaneously cancelling it and createing a replacement Order.
Can you write Python code for it?

data = {
  "order": {
    "units": "-500000",
    "instrument": "EUR_USD",
    "price": "1.25000",
    "type": "LIMIT"
  }
}

r = orders.OrderReplace(accountID=accountID, orderID=last_order_id, data=data)

client.request(r)

print(r.response)

req_id = r.response['lastTransactionID']


I am working on the file "Oanda v20 REST-oandapyV20/05.00 Trade Management.ipynb".
The first task is:
Get a list of trades for an Account.
Can you write Python code for it?

client = oandapyV20.API(access_token=access_token)

params ={
          "instrument": "DE30_EUR,EUR_USD"
        }

r = trades.TradesList(accountID=accountID, params=params)

client.request(r)

print(r.response)


I am working on the file "Oanda v20 REST-oandapyV20/06.00 Position Management.ipynb".
The first task is:
Closeout the open Position regarding instrument in an Account.
Can you write Python code for it?

data = {
  "longUnits": "ALL"
}

r = positions.PositionClose(accountID=accountID,
                            instrument=instrument,
                            data=data)

client.request(r)


I am working on the file "Oanda v20 REST-oandapyV20/06.00 Position Management.ipynb".
The first task is:
List all Positions for an Account.
Can you write Python code for it?

r = positions.PositionList(accountID=accountID)

client.request(r)

print(r.response)


I am working on the file "Oanda v20 REST-oandapyV20/06.00 Position Management.ipynb".
The first task is:
List all open Positions for an Account.
Can you write Python code for it?

r = positions.OpenPositions(accountID=accountID)

client.request(r)


I am working on the file "Oanda v20 REST-oandapyV20/07.00 Transaction History.ipynb".
The first task is:
Get the details of a single Account Transaction.
Can you write Python code for it?

r = trans.TransactionDetails(accountID=accountID, transactionID=49)

client.request(r)


I am working on the file "Obfuscated_code_or_piece_of_art.ipynb".
The first task is:
Bitcoin address & private key generator
This is the most concise (and the most sexy!) implementation of the Bitcoin protocol to generate a new address and private key!
Can you write Python code for it?


I am working on the file "Obfuscated_code_or_piece_of_art.ipynb".
The first task is:
Penrose patterns
This second nice little code will write a visualization of a Penrose tiling (infinite pattern) to a PNG image, of resolution $2000 \times 2000$.
Can you write Python code for it?


I am working on the file "Object Oriented Programming/Object-Oriented Programming.ipynb".
The first task is:
If you do not already know, the word "instantiation" means to create a version of an object. Here is how we would instantiate a bike.
Can you write Python code for it?

# An 'instance' of a bike
my_bike = Bike()
type(my_bike)


I am working on the file "Object Oriented Programming/Object-Oriented Programming.ipynb".
The first task is:
Now, my_bike is an object reference to "Bike". This means that the variable doesn't actually hold the object in memory, but simply points to it.
Attribute
To give objects attributes (i.e. a bike's wheel size, speed, weight), you need to create the
__init__(attributes...)

function. This function is called whenever an object is instantiated. It's what assigns the values you want on an object. Let's see its use below.
Can you write Python code for it?

class Bike:
    def __init__(self, speed, wheel, weight):
        self.speed = speed
        self.wheel = wheel
        self.weight = weight


I am working on the file "Object Oriented Programming/Object-Oriented Programming.ipynb".
The first task is:
The instantiation checks out. Here's what happened:
self.speed = 2
self.wheel = 4
self.weight = 5

This allows us to use dot notation to access the properties. How do we get the wheel size of the bike? We use the following notation:
object.attr

It's that simple.
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Copy1.ipynb".
The first task is:
Can you sort a dictionary? Why or why not?
No, dictionaries aren't indexed.
Tuples
What is the major difference between tuples and lists?
mutability: lists are, tuples aren't
How do you create a tuple?
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Copy1.ipynb".
The first task is:
Sets
What is unique about a set?
non-repeating elements
Use a set to find the unique values of the list below:
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Copy1.ipynb".
The first task is:
Sort the list below:
Can you write Python code for it?

l = [3,4,5,5,6]
result = l.sort()
print(result)


I am working on the file "Objects and Data Structures Assessment Test-Copy1.ipynb".
The first task is:
Strings
Given the string 'hello' give an index command that returns 'e'. Use the code below:
Can you write Python code for it?

s = 'hello'
# Print out 'e' using indexing

# Code here
print(s[1])


I am working on the file "Objects and Data Structures Assessment Test-Copy1.ipynb".
The first task is:
What is the type of the result of the expression 3 + 1.5 + 4?
Floating point number, 8.5
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Copy1.ipynb".
The first task is:
What would you use to find a number’s square root, as well as its square?
Can you write Python code for it?

#x**y for square, x**0.5 for square root
print(2**2)
print(4**0.5)


I am working on the file "Objects and Data Structures Assessment Test-Solution.ipynb".
The first task is:
Can you sort a dictionary? Why or why not?
Answer: No! Because normal dictionaries are mappings not a sequence.
Tuples
What is the major difference between tuples and lists?
Tuples are immutable!
How do you create a tuple?
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Solution.ipynb".
The first task is:
Explain what the cell below will produce and why. Can you change it so the answer is correct?
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Solution.ipynb".
The first task is:
Lists
Build this list [0,0,0] two separate ways.
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Solution.ipynb".
The first task is:
Reassign 'hello' in this nested list to say 'goodbye' item in this list:
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Solution.ipynb".
The first task is:
Sets
What is unique about a set?
Answer: They don't allow for duplicate items!
Use a set to find the unique values of the list below:
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test-Solution.ipynb".
The first task is:
Sort the list below:
Can you write Python code for it?

l = [5,3,4,6,1]

#Method 1
sorted(l)

#Method 2
l.sort()
l


I am working on the file "Objects and Data Structures Assessment Test-Solution.ipynb".
The first task is:
What is the type of the result of the expression 3 + 1.5 + 4?
Answer: Floating Point Number
What would you use to find a number’s square root, as well as its square?
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test.ipynb".
The first task is:
Answer these 3 questions without typing code. Then type code to check your answer.
What is the value of the expression 4 * (6 + 5)
    44

What is the value of the expression 4 * 6 + 5
    29

What is the value of the expression 4 + 6 * 5
    34
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test.ipynb".
The first task is:
Can you sort a dictionary? Why or why not?
    you cant sort a dictionary because it is keys and values
Tuples
What is the major difference between tuples and lists?
    tuples are immutable and lists are mutable
How do you create a tuple?
    with parentheses and a comma delimited list
    (1,2.3)
Sets
What is unique about a set?
    a set contains only unique items and is immutable
Use a set to find the unique values of the list below
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test.ipynb".
The first task is:
Dictionaries
Using keys and indexing, grab the 'hello' from the following dictionaries:
Can you write Python code for it?

d = {'simple_key':'hello'}
# Grab 'hello'
print(d['simple_key'])

d = {'k1':{'k2':'hello'}}
# Grab 'hello'
print(d['k1']['k2'])

# Getting a little tricker
d = {'k1':[{'nest_key':['this is deep',['hello']]}]}

#Grab hello
print(d['k1'][0]['nest_key'][1][0])

# This will be hard and annoying!
d = {'k1':[1,2,{'k2':['this is tricky',{'tough':[1,2,['hello']]}]}]}

d = {'k1':[1,2,{'k2':['this is tricky',{'tough':[1,2,['hello']]}]}]}# print it all
print(d)
# get the set at 'k1'
my_set = d['k1']
print('my_set: ' + str(my_set))
# get the dictionary at index 2 of the set
my_dict = my_set[2]
print('my_dict: ' + str(my_dict))
my_set1 = my_dict['k2']
print ('my_set1: ' + str(my_set1))
my_dict2 = my_set1[1]
print ('my_dict2: ' + str(my_dict2))
my_set2 = my_dict2['tough']
print ('my_set2: ' + str(my_set2))
my_set3 = my_set2[2]
print('my_set3: ' + str(my_set3))
indx = my_set3.index('hello')
print('indx = ' + str(indx))
print(my_set3[indx])

print (d['k1'][2]['k2'][1]['tough'][2][0])


I am working on the file "Objects and Data Structures Assessment Test.ipynb".
The first task is:
Lists
Build this list [0,0,0] two separate ways.
Can you write Python code for it?

l = [0,0,0]
print (l)
l = [0]*3
print(l)
l1 = list()
l1.append(0)
l1.append(0)
l1.append(0)
print(l1)


I am working on the file "Objects and Data Structures Assessment Test.ipynb".
The first task is:
Reverse the string 'hello' using indexing:
Can you write Python code for it?

s ='hello'

# Reverse the string using indexing
print (s[::-1])
# Code here


I am working on the file "Objects and Data Structures Assessment Test.ipynb".
The first task is:
Sort the list below:
Can you write Python code for it?


I am working on the file "Objects and Data Structures Assessment Test.ipynb".
The first task is:
Test your knowledge.
 Answer the following questions
Write a brief description of all the following Object Types and Data Structures we've learned about:
Numbers:
Strings:
Lists:
Tuples:
Dictionaries:
Numbers
Write an equation that uses multiplication, division, an exponent, addition, and subtraction that is equal to 100.25.
Hint: This is just to test your memory of the basic arithmetic commands, work backwards from 100.25
Can you write Python code for it?


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/List Comprehensions-checkpoint.ipynb".
The first task is:
Generators
We can do more complex things than just the basic list comprehensions. We can create more complex, succint comprehensions by introducing generators in our very statements. We haev the following data to work with:
rows = 'ABC
cols = '123'

In general, we want to access both the rows in cols at the same time.... How did we do that?
Can you write Python code for it?

# Simply accessing rows and cols in a comprehensions [A1, A2, A3, B1, B2, B3, C1, C2, C3]

# Non-Pythonic
lst = []
for r in rows:
    for c in cols:
        lst.append(r+c)

# Pythonic
lst = [r+c for r in rows for c in cols]
print(lst)


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/List Comprehensions-checkpoint.ipynb".
The first task is:
Let's take a look at some values and see how we can produce certain outputs.
Can you write Python code for it?

rows = 'ABC'
cols = '123'
vowels = ('a', 'e', 'i', 'o', 'u')
sentence = 'cogito ergo sum'
words = sentence.split()

# Produce [A3, B2, C1]
number_letter_lst = [rows[element]+cols[2-element] for element in range(3)]
print(number_letter_lst)

# Produce [A1, B1, C1, A2, B2, C2, A3, B3, C3]
letter_number_lst = [r+c for c in cols for r in rows]
print(letter_number_lst)

x = [s1 + ' x ' + s2
            for s1 in (rows[i]+cols[i] for i in range(3))
            for s2 in (rows[2-j]+cols[j] for j in range(3))]
print(x)


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/List Comprehensions-checkpoint.ipynb".
The first task is:
Let's try Creating:
['A1 x C1', 'A1 x B2', 'A1 x A3', 'B2 x C1', 'B2 x B2', 'B2 x A3', 'C3 x C1', 'C3 x B2', 'C3 x A3']
Thought process, we know that rows will change every other 3, meaning we keep it constant...
Can you write Python code for it?

# let's figure this list out with normal syntax
lst = []
for r in (rows[i]+cols[i] for i in range(3)):
    for c in (rows[2-i]+cols[i] for i in range(3)):
        lst.append(r + 'x' + c)

print(lst)

# shortened
crossed_list = [x + " x " + y for x in (rows[i]+cols[i] for i in range(3)) for y in (rows[2-i]+cols[i] for i in range(3))]

print(crossed_list)

x = sorted(words, key=lambda x: len(x))

print(x)


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/Tuples-checkpoint.ipynb".
The first task is:
All structures that can use slicing (sequences... i.e. strings, tuples, lists) have skips. Recall the general notation for slicing.
my_sequence[start:end:skip]

where "start" is inclusive, end is exclusive, and skip is the sequence traversal factor.
Can you write Python code for it?

# Getting every other object in a Tuple
print(day_tuple[::2])

# Getting the tuple objects backward
print(day_tuple[::-1])


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/Tuples-checkpoint.ipynb".
The first task is:
Here is an example of a heterogenous tuple. Recall that are definable types in Python are objects, and thus even data structures like lists and dictionaries can be the items of a tuple. On the flipside, tuples can act as a key and a value in a dictionary! This is because all keys in a dictionary must be hashable, and tuples are. Tuples can also be objects in lists.
Can you write Python code for it?

# Initializing a heterogenous Tuple
my_tuple = ("red", 40, 9.0, [1, 4, 5], "hey", {"Micah":9})


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/Tuples-checkpoint.ipynb".
The first task is:
No form of modification on an already existing tuple object will work, whether is appending, prepending, etc. Only reassignment.
Tuple Functions and Methods
There are a few functions and methods associated with tuples. We explore them below.
Tuple Functions
As always, we know that the len() function works on tuples.
Can you write Python code for it?


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/Tuples-checkpoint.ipynb".
The first task is:
Now that we've seen accessing by normal bracket notation and slicing, let's discuss modification. Tuples are immutable sequences, meaning that they cannot be changed in place. What does this mean exactly? Like strings, tuples can physically change what they are referring to but they cannot change the obejcts that are in them in place.
Can you write Python code for it?

# Intializing a Tuple
day_tuple = ("MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY")
# Attempt to change an item via access bracket notation
day_tuple[1] = "SUNDAY"


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/Tuples-checkpoint.ipynb".
The first task is:
The index(object) methods finds the first occurence of the explicit object, and returns the indice.
Can you write Python code for it?

# Initializing a Tuple (with ony one WEDNESDAY string literal)
day_tuple = ("MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY")
# Return the index of Wednesday
day_tuple.index("WEDNESDAY")
# Reassign the tuple to another object
day_tuple = ("WEDNESDAY", "TUESDAY", "WEDNESDAY", "FRIDAY", "WEDNESAY")
# What index will it return now that are 3 "WEDNESDAY" string literals
day_tuple.index("WEDNESDAY")


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/Tuples-checkpoint.ipynb".
The first task is:
Tuple Access, Slicing, and Modification
You can access any item in a tuple with bracket notation.
Can you write Python code for it?

# Initializing a Tuple
day_tuple = ("MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY")
# Access Thursday
print(day_tuple[3])


I am working on the file "Objects and Data Structures/.ipynb_checkpoints/Tuples-checkpoint.ipynb".
The first task is:
Tuple Methods
The two primary methods we should be aware of that tuples posses are count() and index().
Calling the count(object) method on a tuple returns the number of occurences of the explicit object in the tuple.
Can you write Python code for it?

# Initializing the Tuple
day_tuple = ("MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY")
# Counting how many Mondays there are
day_tuple.count("MONDAY")
# Reassigning the object to have more Mondays
day_tuple = ("MONDAY", "WEDNESDAY", "MONDAY", "THURSDAY", "FRIDAY")
# Recounting the number of Mondays
day_tuple.count("MONDAY")


I am working on the file "Objects and Data Structures/Booleans, and None.ipynb".
The first task is:
The general format for if-else control flow in Python is the following:
if something_that_evals_to_True:
    foo()
elif something_else_:
    woo()
.
.
else:
    falsework()

None
Often, you will not always be sure as to what an object should hold yet. You may have a variable in mind to store an incoming object, but you need a placeholder. Or, perhaps, you want to check if the object you received is nothing. In this case, we use the "None" object.
Can you write Python code for it?

# Example of assigning None, and changing it.
some_obj = None

if 2 < 3:
    some_obj = True


I am working on the file "Objects and Data Structures/Booleans, and None.ipynb".
The first task is:
To stay practical, it is important to understand that you won't be assigning True and False values to variables as much as you will be receiving them. We talked in the "Comparison Operators" series about said operators, and what they return. How do we capture True and False from an expression?
Can you write Python code for it?

# Capturing True from an expression
x = 2 < 3

# Capturing False from an expression
y = 5 > 9


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
As we can see, Michigan Ave. is no longer in the dictionary.
Adding a key: To add a key is to simply use the same notation as reassigning a key to a new value. If the dictionary detects that the key you put in does not exist, it will take that key and assign it to the value you assigned. The notation remains the same:
my_dict[key] = new_value

As long as key is not already in the dictionary, it will just create it and map it to the value you sent in anyways.
Can you write Python code for it?

# Initializing a Dictionary
pothole_dict = {"Morgan St.":4, "Tulsen Blvd.":0, "Michigan Ave.":8}
# Add a new street (Key)
pothole_dict["Wallace St."] = 12
# Verify we added Wallace St.
print(pothole_dict)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Can we map dictionary to other dictionaries? Yes. Note below that "Ruth" and "Barbara" are keys of final_dict, and that "John" and "Leslie" are keys of the dictionary that Ruth is mapped to.
Can you write Python code for it?

final_dict = {"Ruth":{"John":[1, 2, 5], "Leslie":6 }, "Barbara":4}


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Can we map two integers? Yes, we can. Keys must be hashable types, and integers are hashable.
Can you write Python code for it?

# Initializing a dictionary with integer-integer pairs
another_dict = {4:1, 8:2, 9:4, 2:6}


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Can we vary the key type across mappings in the dictionary? Yes. The example below creates a string, integer, and float key, and maps them all to integers.
Can you write Python code for it?

# Initializing a Dictionary with varying key types
varying_dict = {"John":4, 9:5, 4.32:8}


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Can we vary the value type? Yes. Also, in most of our examples, we will end up mapping strings to numbers or data structures. Below, we've mapped out string keys to integers and lists.
Can you write Python code for it?

# Initializing a Dictionary that maps strings to either to an integer or list
last_dict = {"Micah":[1, 2, 5], "Rose":4, "John":9, "Gwen":[5, 3]}


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Dictionary Functions and Methods
There are built-in functions that can be used with dictionaries, and methods that belong inherently to the dictionary library class in Python.
Dictionary Functions
We will discuss the dict() function. The dict function is documented the following way:
dict(**kwarg)
dict(mappings, **kwarg)
dict(iterable, **kwarg)

This can look daunting at first, but it is not too difficult to understand. In essense, the dict() function creates a dictionary. We can supply it with mappings, an iterable, keywords, or simply an argument that evaluates to either a mapping or iterable.
Let's first look at recreating the following dictionary with the 3 types of notations:
Can you write Python code for it?

# "Normal" Initialization
bball_wins = {"Lakers":20, "Heat":24, "Bulls":26}
print(bball_wins)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
First, we showcase the first notation. Here, the equal sign denotes a "name=value" relationship. To make it simple, this notation treats the left variable as the key, and the right as the value.
Can you write Python code for it?

# Using keywords... dict(**kwargs)
bball_wins2 = dict(Lakers=20, Heat=24, Bulls=26)
print(bball_wins2)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
How do we get Ada's last name? Use the same notation from before.
Can you write Python code for it?

last_name = computer_scientists_dict["Ada"]
print(last_name)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
How do we get Mariel Somer's shares in the company?
my_dict[key][subkey]....

The same notation access as a list, except now with keys.
Can you write Python code for it?

# Nested dictionary access
print(my_dict["Mariel"]["Somers"])


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
How do we get the number of shares Michael has in the company? Using the above notation, we write the following.
Can you write Python code for it?

michael_shares = share_holdings_dict["michael"]
print(michael_shares)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Instead of mutating by reassignment, we can also mutate by using methods or operators on dictionary values. Below, we will make a dictionary and show a variety of ways to put this into action.
Can you write Python code for it?

# Initialize a Dictionary that maps Strings to numbers
temp_dict = {"Tulsa":79, "Anchorage":10, "Chicago":65}
# Add 20 to the current temperature, and then reassign
temp_dict["Anchorage"] = temp_dict["Anchorage"] + 20
# Print result
print(temp_dict["Anchorage"])


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Let's change Morgan St. and Mich Ave, too.
Can you write Python code for it?

# Change Michigan and Morgan
pothole_dict["Michigan Ave."] = 30
pothole_dict["Morgan St."] = 2


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Multiple values cannot be assigned to a key in a dictionary. However, as we experimented above, we can map a key to a data structure that holds multiple values by nature.
Accessing and Mutating Dictionaries
Like any other data structure, dictionaries have certain methods of access and mutation.
Accessing Dictionaries
The common notation for access into a dictionary is by specifying the key, which returns the associated value. Consider the notation:
my_dict[key_here]

This notation will return the value(s) that the explicit key is mapped to in the dictionary. Let's try accessing values from dictionaries below.
First, let us create a dictionary.
Can you write Python code for it?

# Initializing a Dictionary that maps strings to integers
share_holdings_dict = {"john":5, "michael":4, "rutherford":19}


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Now the second notation. Here, we've sent in a mapping as the argument (a dictionary itself) into the dict function. It may look odd at first, but it is another way to write the dictionary.
Can you write Python code for it?

# Using mapping... dict(mapping, **kwargs)
bball_wins3 = dict({"Lakers":20, "Heat":24, "Bulls":26})
print(bball_wins3)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
So, what happened? Well, the iterable we sent in happened to have 3 items in itself, which also all happened to be iterables themselves (lists). However, each internal iterable must have 2 objects in it. This is what Python uses to map the iterable:
dict = {}
for k, v in iterable:
    d[k] = v

While this line(s) of code may seem puzzling, we will discuss it again when we reach for-statements. However, it can still be understoof reasonably if you understand the idea of unpacking, which you can learn about right now if you open the "Unpacking.ipynb" iPython notebook (you should right now, before you move on).
You should know it because sometimes dict() will be used with the built-in function zip(). Zip() is fully covered in the built-in functions lecture (which you should also review now before moving on). Consider the code below:
Can you write Python code for it?

bball_wins5 = dict(zip(["Lakers", "Heat", "Bulls"], (20, 24, 26)))
print(bball_wins5)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
Sure enough, we get the number of shares Michael has: 4. This notation will allow you to extract a key of any type. Let's try this example again. This time, let's map first names to last names.
Can you write Python code for it?

# Initializing a Dictionary that maps strings to strings
computer_scientists_dict = {"Peter":"Norvig", "Donald":"Knuth", "Ada":"Lovelace", "Grace":"Hopper"}


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
The answer, as you can see, is to simply use bracket notation for this list. The expression
conversion_rate["Moe's Pizza"]

gives us [5, 13, 4]. If we call
[5, 13, 4][2]

we would land 4. This is why we can use the bracket notation after getting the list to begin with.
Mutating Dictionaries
You can mutate a dictionary multiple ways. We could consider:
Change value?
Delete key?
Add key?
Change key?
We won't consider deleting a value or adding a value because neither follow the natural state of a dictionary mapping: a key is never value-less.
To change the value of a key we could simple reassign the value, or perform some operation on it. Use the notation following to reassign the value:
my_dict[key] = new_value

This is the same notation as retrieving the value, but now with an equal sign to specify reassignment.
Can you write Python code for it?

# Initializing a Dictionary
pothole_dict = {"Morgan St.":4, "Tulsen Blvd.":0, "Michigan Ave.":8}


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
We already know how to retrive values from keys - we just accessed the conversion rate list for Moe's Pizza. So, how do we access the the "things" inside of a value if it is a data structure (in this case, it is a list). In example: let us assume that the elements in the list represent the conversation rate from the 1st year, 2nd year, and 3rd year of opening business. How we do access Moe's Pizza's third year conversion rates. Here is the shorthand way to do it:
Can you write Python code for it?

third_year_conv = conversion_rate["Moe's Pizza"][2]
print(third_year_conv)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
We can also change a key. Imagine we accidentally mapped the wrong name to a set of values. How would we do this? While there isn't a single-liner notation for this, we can think methodically. To assign a new key, is to save the values of the old key into a new variable, deleting the old key afterwards (so that we don't lose the values to begin with). Here's the notation:
my_dict[new_key] = my_dict[old_key]
del my_dict[old_key]

We create a new key (a key that's not already in the dictionary), and then assign it to the value that is mapped to the old_key (my_dict[old_key]). Afterwards, we no longer need the mapping of the old_key to its values, so we use the del operator to delete it and its values. The new_key and its values remain unscathed.
Let's change Morgan St. to Hollywood Blvd.
Can you write Python code for it?

# Initializing a Dictionary
pothole_dict = {"Morgan St.":4, "Tulsen Blvd.":0, "Michigan Ave.":8}
# Store the values with new key
pothole_dict["Hollywood Blvd."] = pothole_dict["Morgan St."]
# Delete the old key, and the values it was mapped to
del pothole_dict["Morgan St."]
# Verify that Hollywood Blvd. replaced Morgan St.
print(pothole_dict)


I am working on the file "Objects and Data Structures/Dictionaries.ipynb".
The first task is:
What is zip doing? Well, the documentation describes it in this way:
"Make an iterator that aggregates elements from each of the iterables."

It makes an iterator. Well, what does that mean? In short, it takes two iterable objects... these could be two lists, two tuples, a list and a tuple, etc. It takes an element from the first iterable, and the same positioned element in the second iterable, and throws them together in a tuple. Here's a diagram:
zip(["Lakers", "Heat", "Bulls"], (20, 24, 26)]
---> First take first elements of both iterables, and put them together... ("Lakers", 20)
    ---> Now return the tuple... the statement looks like dict([("Lakers", 20)])
        ---> They get mapped together using the dict(mapping, **kwarg) notation
---> Now take seocnd elements of both iterables, and tuple them together... ("Heat", 24)
    ---> refer above
        --> refer above
---> Lastly, put the last elements into a tuple... ("Bulls", 26)
    ---> refer above
        --> refer above

By the end, all of the lists that were created get added as mappings in the dictionary. The result is the dictionary we expected.
Nested Dictionaries
We can nest dictionaries. The access notation gets longer everytime we nest.
Below, we've made a dictionary that maps a first name to a dictionary of possible last names and their shareholdings in a company.
Can you write Python code for it?

my_dict = {"John":{"Doe":9, "Hansen":13}, "Mariel":{"Stevenson":11, "Somers":2}, "Rocky":9}


I am working on the file "Objects and Data Structures/List Comprehensions.ipynb".
The first task is:
Generators
We can do more complex things than just the basic list comprehensions. We can create more complex, succint comprehensions by introducing generators in our very statements. We haev the following data to work with:
rows = 'ABC
cols = '123'

In general, we want to access both the rows in cols at the same time.... How did we do that?
Can you write Python code for it?

# Simply accessing rows and cols in a comprehensions [A1, A2, A3, B1, B2, B3, C1, C2, C3]

# Non-Pythonic
lst = []
for r in rows:
    for c in cols:
        lst.append(r+c)

# Pythonic
lst = [r+c for r in rows for c in cols]
print(lst)


I am working on the file "Objects and Data Structures/List Comprehensions.ipynb".
The first task is:
Let's take a look at some values and see how we can produce certain outputs.
Can you write Python code for it?

rows = 'ABC'
cols = '123'
vowels = ('a', 'e', 'i', 'o', 'u')
sentence = 'cogito ergo sum'
words = sentence.split()

# Produce [A3, B2, C1]
number_letter_lst = [rows[element]+cols[2-element] for element in range(3)]
print(number_letter_lst)

row_col_lst = [rows[i]+cols[2-i] for i in range(3)]

# Produce [A1, B1, C1, A2, B2, C2, A3, B3, C3]
my_lst = [r+c for c in cols for r in rows]
print(my_lst)


I am working on the file "Objects and Data Structures/List Comprehensions.ipynb".
The first task is:
Let's try Creating:
['A1 x C1', 'A1 x B2', 'A1 x A3', 'B2 x C1', 'B2 x B2', 'B2 x A3', 'C3 x C1', 'C3 x B2', 'C3 x A3']
Thought process, we know that rows will change every other 3, meaning we keep it constant...
Can you write Python code for it?

# let's figure this list out with normal syntax
lst = []
for r in (rows[i]+cols[i] for i in range(3)):
    for c in (rows[2-i]+cols[i] for i in range(3)):
        lst.append(r + 'x' + c)

print(lst)

# shortened
crossed_list = [x + " x " + y for x in (rows[i]+cols[i] for i in range(3)) for y in (rows[2-i]+cols[i] for i in range(3))]

print(crossed_list)

x = sorted(words, key=lambda x: len(x))

print(x)


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
A few more examples.
Can you write Python code for it?

# Access beginning to 2nd character to end
print(my_str[1:])

# Access every other character (skipping)
print(my_str[::2])

# Access backwards
print(my_str[::-1])


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
Adding two lists is what you would expect. The list is appended (added) to the end. It could be a list of a single object - it does not matter.
Can you write Python code for it?

# Adding two lists
my_list = my_list + my_list
print(my_list)


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
How can we access the number object 3 in the list? We need to specify a deeper level of access.
Can you write Python code for it?

# Accessing the object of a list's list
my_list[1][1]


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
How would be access the list object (literal) [2, 3, 4]? Like any other object, of course.
Can you write Python code for it?


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
Indexing Lists
Lists are contiguous, monolithic data structures. When we talk about objects in memory, we mean that every object has some dedicated space in memory. In a list, objects are "next" to eachother in memory. This means that accessing the next list object is as simple as finding the index of that object in the list, since it can only be some constant defined in memory. Consider this list:
lst = [1, "hello", 2.3]

Here, the number 1 comes before "hello" in memory, and the number 2.3 comes after "hello" in memory. They are seperated by some given space, and are thus contiguous in memory. Why does contiguousness matter? It means we can index lists and get their objects out. Recall indexing a string:
Can you write Python code for it?

# Declaring a string
my_str = "hello"

# Writing a print function with comma seperation (each object is printed with a space next to the previous)
print(my_str[0], my_str[1], my_str[2], my_str[3], my_str[4])


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
Let's breakdown the usage of a list comprehension.
lst_here = [object_to_add for loop_control_variable in range(start, end)]

where "lst_here" is the new list, "object_to_add" is what is appended to the list after running the for-loop, where "loop_control_variable" assumes the values given in the range start to end, where end is exclusive.  We will discuss Pythonic Code in the form of list comprehensions and other constructs much, much more in detail (a whole notebook) later in the course. Just keep it in mind, as we will be using it very often. Below are a few more examples of the list comprehesion - see if you can understand them!
Can you write Python code for it?

# Storing the squares of 0-9 in a list with a list comprehension
my_list = [x**2 for x in range(0,10)]
print(my_list)

# Non-Pythonically
my_list = []
for name in ["miranda", "mike", "john"]:
    my_list.append(name + " is cool")
print("Non-pythonic:", my_list)

# Pythonically
my_list = [name+" is cool" for name in ["miranda", "mike", "john"]]
print("Pythonic:",my_list)


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
List Functions and Methods
In the Python Standard Library, each built-in object has its own set of functions and methods. The mathematical operators used with numbers +, -, *, and / actually correspond to the "add", "sub", "mul" and "div" functions. Lists are no different. They also have their own functions.
List Functions
The function len() is one you will use very often. Recall that functions are not associated with specific objects.
Can you write Python code for it?

# Length of my list
my_list = [1, 2, "baby", "just"]

# Get the length
print(len(my_list))


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
List Methods
There are many important list methods. Recall that methods are functions that have an association with an object.
We'll cover most of the methods you can use on a list:
1. append(object) - returns and appends given item to a list
2. clear() - clears (emptys) a list
3. copy() - returns a replicated list object of implicit list object
4. count(object) - returns count of object occurences in list
5. extend(iterable object) - returns a list object with objects appended from explicit iterable
6. index(object) - returns index of the first occurence of object in list
7. insert(inidice, object) - object added at given position
8. pop - returns and removes the last object in a list
9. remove - returns a list with object removed
10. reverse - returns a reversed list
11. sort - returns sorted list
Can you write Python code for it?

# Appending an item to a list
my_list = ["charlotte", "winnipeg", "sydney"]
print("before:", my_list)
my_list.append("chicago")
print("after: ", my_list)

# clearing a list
my_list = ["charlotte", "winnipeg", "sydney"]
print("before:", my_list)
my_list.clear()
print("after:", my_list)

# Copying a list
my_list = ["charlotte", "winnipeg", "sydney"]
print("before:", my_list)
my_new_list = my_list.copy()
print("after:", my_list)
print("my_new_list:", my_new_list)

# counting a list
my_list = ["charlotte", "winnipeg", "sydney"]
print("before:", my_list.count("winipeg"))
my_list.append("winipeg")
print("after:", my_list.count("winipeg"))

# extending a list
my_list = ["charlotte", "winipeg", "sydney"]
print("before:", my_list)
my_list.extend(["chicago", "houston"])
print("after:", my_list)

# what would this do? Look at the markdown cell a few below.
# my_list.extend("chicago")

# indexing a list
my_list = ["charlotte", "winipeg", "sydney"]
print("before:", my_list.index("sydney"))
my_list.append("sydney")
print("before:", my_list.index("sydney"))

# inserting into a list
my_list = ["charlotte", "winipeg", "sydney"]
print("before:", my_list)
my_list.insert(1, "chicago")
print("after:", my_list)

# popping from a list
my_list = ["charlotte", "winnipeg", "sydney"]
print("before:", my_list)
my_list.pop()
print("after:", my_list)

# removing from a list
my_list = ["charlotte", "winnipeg", "sydney"]
print("before:", my_list)
my_list.remove("sydney")
print("after:", my_list, '\n')

# another example of removing
my_list = ["charlotte", "winnipeg", "sydney"]
print("before:", my_list)
my_list.insert(2, "charlotte")
print("after inserting:", my_list)
my_list.remove("charlotte")
print("after removing:", my_list)

# which "charlotte" got removed? (Hint: first occurences, only!)

# reversing a list
my_list = ["charlotte", "winnipeg", "sydney"]
print("before:", my_list)
my_list.reverse()
print("after:", my_list)

# sorting a list
my_list = ["charlotte", "winipeg", "sydney"]
print("before:",...


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
Lists work no differently. Slicing lists is shown below.
Can you write Python code for it?

# A list with strings, integer, and float
my_list = ["a", "collection", "of", 4, "version", 5.0, "glasses"]

# To end
print(my_list[0:])

# middle 3
print(my_list[2:5])

# backwards
print(my_list[::-1])


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
Multiplication on a list can be done with a constant number. You cannot multiply two lists together (only a constant to replicate the list at hand).
Can you write Python code for it?

my_list = ["wow", 1, 2]
my_list = my_list * 3
print(my_list)


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
Or, we can just append an item. (Note that we are chaning my_list permanently as we go to each cell)
Can you write Python code for it?

# Appending the string "work" to our list
my_list = my_list + ["work"]
print(my_list)


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
Other Pythonic Specifications
There are many Pythonic constructs, and we will use almost all of them. There will be a whole notebook covering Pythonic code after we cover the rest of the object and data structure and control flow lectures. For now, refer to this guide: http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html
Nesting Lists
Lists can be nested into one another. A nested lists acts like any other object in a list, except that a list object inside a list has a "deeper" level of access. Firstly, you can access the list object itself, and then access the objects in the list object. Consider the following example.
We declare a list the same way, seperating our internal list object with a commma, too.
Can you write Python code for it?

# Declaring a nested list
my_list = [1, [2, 3, 4], 5]


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
Recall that we mentioned about lists having the ability to store multiple types of objects. If you are familiar with other programming languages, this is not always the case. Python, however, has allowed us to do so.
Can you write Python code for it?

# Creating a list with an integer, string, and float.
my_new_list = [3, "Houston", 4.5]


I am working on the file "Objects and Data Structures/Lists.ipynb".
The first task is:
That look a total of 3 lines of code to do. We can reduce this 1 line with list comprehensions.
Can you write Python code for it?

# Now, using a list comprehension
my_list = ["hello" for count in range(0,10)]
print(my_list)


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
Arithmetic Operators
Before discussing numbers, let's briefly look at the simple concept of using arithmetic operators. Arithmetic operators what you would expect, +, -, *, / and //. Division is a special case, which we'll look at.
Can you write Python code for it?

# Addition
3 + 15

#Subtraction
9 - 3

#Multiplication
3 * 4


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
As you can see, the foo that is to the right of the "=" operator has the value of 9 currently (as we defined before). We added 9 to it, and then updated it by setting it to "foo =". Note that the previous "foo + 5" we did did NOT store foo again, and so the value of foo never changed.
Additionally, it is important to understand the naming convention of variables in Python. Stick to these conventions, and all will sail well:

All characters in a variable name should be lowercase.
No spaces in variable names, use the underscore "_", instead.
Don't use traditional keyboard symbols in your names.
Can you write Python code for it?

# Correct variable names
building_height = 100
water = 4

result = building_height + water


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
Conversion Functions
You can convert numbers back and forth from integer to floating-point using the conversion functions int() and float().
Can you write Python code for it?

# Converts the argument (3) which is an integer, and spits out a float (3.0) in the REPL.
float(3)

# Coverts the argument (5.9) which is a float, and spits out an integer (5) in the REPL.
# Note: truncation, not rounding.
int(5.9)


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
Division has two forms in Python 3 (and 2): divison and integer division. In Python, to get the full decimal number of a division operation, simply specify a single '/'. To specifiy integer division (drop the remainer, don't round), alternatively specify a '//'.
Can you write Python code for it?

# Division
15 / 3

# Integer Division
15 // 3

# Division
9 / 4

# Integer Division
9 // 4

# Modulo
3 % 2


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
Floating-point Inaccuracy
Just remember: floating-point types are never used to for any type of calculation that requires accuracy. Why is this? Take a look at the example below.
Can you write Python code for it?

# Let's add 0.1 and 0.2, to get 0.3.
result = 0.1 + 0.2
result

# This operation should return 16.3
7.6 + 8.7


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
Jupyter Notebooks are know to be REPL environments. This simply means that the notebooks serve as "interactive" programming environments, where calculations or other processes can be done impromptu. REPL stands for Read-Evaluate-Print-Loop.
Can you write Python code for it?

# Try running this block!
foo + 5

# Evaluate the variable itself
foo


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
The result was not what we expected. Why can't the interpreter get this right? Simply put, it is because computers have their number systems represented in base-2, while we typically use base 10. There are some numbers that are easy to represent in base-10 that are impossible to accurately represent in base-2, and vice-versa.
What does this mean to you, as the programmer? Whenever you are doing critical calculations that rely on accuracy, use the Decimal module. Below, I've imported the decimal module (more on modules later) and then I've checked what the current precision is. Right now, it's not at what we want. We change the precision, and then follow through wil the calculation
Can you write Python code for it?

import decimal

# We use the 'decimal' word as the module name, and use the "getcontext" function that is a part of the module, and
# set it's .prec value (precision) to 3 decimal places.
decimal.getcontext().prec=3

# We use what's called a print() function to literally print our result to the screen (similar to the REPL behavior)
print(decimal.Decimal(0.1) + decimal.Decimal(0.2))


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
We can use declaration constructs that contain a variable name twice. This allows a programmer to update variables as they place. When the variable that appears twice is on the right of the "=" operator, it is using whatever value it held from the last assignment. If there are other operators, like the "+" addition operator, then the calculation will be done first, then stored/updated into the variable.
Can you write Python code for it?

# Here, foo has the value of 9. Let's add 9 to it.
foo = foo + 9
foo


I am working on the file "Objects and Data Structures/Numbers.ipynb".
The first task is:
Whenever you add two integers together, you will end up with an integer. Same applies to floats. However, you if you add a float and an integer together, it will always become a float. This process is called automatic type conversion.
Can you write Python code for it?

# Adding two integers
2 + 3

# Adding two doubles
9.1 + 3.0

# Adding an integer and float (automatic type conversion)
4 + 11.3

2.1 + .9


I am working on the file "Objects and Data Structures/Print Formatting.ipynb".
The first task is:
Formatting with the format() Method
Traditionally, string formatting has included using the percent notation with %s, %f, and %d. We've seen the syntax before in the String iPython notebook. Here we can see it again.
my_str.format(var1, var2)
Can you write Python code for it?

my_str = "No one said becoming a {x} entrepreneur was as easy as {y} ({z})".format(x="tech", y=3.14, z="pie")

print(my_str)