I am working on the file "Explore.ipynb".
The first task is:
For women, it seems pretty clear-cut: The women with nobility titles survived (as did women on the whole). The men with titles (all except Rev.) seem to do better on average, but it's highly variable. Since the gender-based model where all women live and men die attains over a 76% accuracy, the hard part of our model seems to be picking out the few male survivors.
Family Size
Here we'll work with the SibSp and Parch features, which involve family size. To look for lone travelers, we'll first look at the distribution of the features added together, separated by gender:
Can you write Python code for it?

train['FamSize'] = train['SibSp'] + train['Parch']
test['FamSize'] = test['SibSp'] + test['Parch']

plt.figure(5, figsize=(8, 4))
plt.subplot(121)
sns.countplot(train.loc[train['Sex'] == 'female', 'FamSize'])
plt.subplot(122)
sns.countplot(train.loc[train['Sex'] == 'male', 'FamSize'])
plt.show()

train['FamSize'] = train['SibSp'] + train['Parch']
test['FamSize'] = test['SibSp'] + test['Parch']

plt.figure(6, figsize=(12, 8))
plt.subplot(231)
sns.countplot(train.loc[(train['Sex'] == 'female') & (train['Pclass'] == 1), 'FamSize'])
plt.subplot(234)
sns.countplot(train.loc[(train['Sex'] == 'male') & (train['Pclass'] == 1), 'FamSize'])
plt.subplot(232)
sns.countplot(train.loc[(train['Sex'] == 'female') & (train['Pclass'] == 2), 'FamSize'])
plt.subplot(235)
sns.countplot(train.loc[(train['Sex'] == 'male') & (train['Pclass'] == 2), 'FamSize'])
plt.subplot(233)
sns.countplot(train.loc[(train['Sex'] == 'female') & (train['Pclass'] == 3), 'FamSize'])
plt.subplot(236)
sns.countplot(train.loc[(train['Sex'] == 'male') & (train['Pclass'] == 3), 'FamSize'])
plt.show()


I am working on the file "Explore.ipynb".
The first task is:
How did this impact survival?
Can you write Python code for it?

plt.figure(7, figsize=(8, 4))
plt.subplot(121)
sns.barplot(x='FamSize', y='Survived', data=train[train['Sex'] == 'female'])
plt.subplot(122)
sns.barplot(x='FamSize', y='Survived', data=train[train['Sex'] == 'male'])
plt.show()

plt.figure(9, figsize=(12, 8))
plt.subplot(231)
sns.barplot(x='FamSize', y='Survived', data=train[(train['Sex'] == 'female') & (train['Pclass'] == 1)])
plt.subplot(234)
sns.barplot(x='FamSize', y='Survived', data=train[(train['Sex'] == 'male') & (train['Pclass'] == 1)])
plt.subplot(232)
sns.barplot(x='FamSize', y='Survived', data=train[(train['Sex'] == 'female') & (train['Pclass'] == 2)])
plt.subplot(235)
sns.barplot(x='FamSize', y='Survived', data=train[(train['Sex'] == 'male') & (train['Pclass'] == 2)])
plt.subplot(233)
sns.barplot(x='FamSize', y='Survived', data=train[(train['Sex'] == 'female') & (train['Pclass'] == 3)])
plt.subplot(236)
sns.barplot(x='FamSize', y='Survived', data=train[(train['Sex'] == 'male') & (train['Pclass'] == 3)])
plt.show()


I am working on the file "Explore.ipynb".
The first task is:
Initial Exploration and Plotting
Can you write Python code for it?

train.head()

train.info()

test.info()

plt.figure(1, figsize=(6, 6))
sns.barplot(x='Sex', y='Survived', data=train)
plt.show()

s_ages = train.loc[train['Survived'] == 1, 'Age'].dropna()
d_ages = train.loc[train['Survived'] == 0, 'Age'].dropna()
s_fares = train.loc[train['Survived'] == 1, 'Fare'].add(1).apply(np.log).dropna()
d_fares = train.loc[train['Survived'] == 0, 'Fare'].add(1).apply(np.log).dropna()

plt.figure(2, figsize=(12, 8))
plt.subplot(231)
sns.barplot(x='Pclass', y='Survived', data=train)
plt.subplot(234)
sns.barplot(x='Embarked', y='Survived', data=train)
plt.subplot(233)
sns.barplot(x='SibSp', y='Survived', data=train)
plt.subplot(236)
sns.barplot(x='Parch', y='Survived', data=train)
plt.subplot(232)
sns.distplot(d_ages, color='C0')
sns.distplot(s_ages, color='C1')
plt.subplot(235)
sns.distplot(d_fares, color='C0')
sns.distplot(s_fares, color='C1')
plt.show()


I am working on the file "Explore.ipynb".
The first task is:
It seems like most of them did. Let's take a look at the fares:
Can you write Python code for it?


I am working on the file "Explore.ipynb".
The first task is:
Let's also search for duplicate cabins, since that may indicate party size and help impute missing values.
Can you write Python code for it?

train[(train['Cabin'].duplicated(keep=False)) & (train['Cabin'].notnull())].set_index('Cabin', append=True).swaplevel(0, 1).sort_index()


I am working on the file "Explore.ipynb".
The first task is:
Let's assume that it is linear. We'll divide by the ticket size, and look at the skew for each class:
Can you write Python code for it?

train['AdjFare'] = train['Fare'].div(train['TicketSize'])
g = sns.FacetGrid(train, col='Pclass')
g = g.map(plt.hist, 'AdjFare')
plt.show()
train.groupby('Pclass')['AdjFare'].apply(st.skew)


I am working on the file "Explore.ipynb".
The first task is:
Let's ignore (for the moment) possible effects from passenger class. We can then draw the following conclusions:

Most men traveled alone. Those with families were generally in smaller ones. A huge amount of men traveled alone in third class; they had very low survival chances.
Many women traveled alone, but not as many as men. Larger groups consisted of mostly women.
In first and second class:
Women seem to have roughly the same survival chance, independent of family size.
Men with larger family sizes seem to have relatively higher chances of survival.

In third class:
Women and men seem to have relatively higher chances of survival, up to a family size of 3.
Women and men with a family size of 4 or higher had drastically lower odds of survival.

Tickets
One thing we can do with ticket numbers is scan for duplicates:
Can you write Python code for it?

ticket_dupes = train[(train['Ticket'].duplicated(keep=False))].set_index('Ticket', append=True).swaplevel(0, 1).sort_index()
ticket_dupes


I am working on the file "Explore.ipynb".
The first task is:
Let's look at the outliers values that are above 200...
Can you write Python code for it?


I am working on the file "Explore.ipynb".
The first task is:
Let's see how these titles did:
Can you write Python code for it?

plt.figure(4, figsize=(8, 4))
plt.subplot(121)
sns.barplot(x='AdjTitle', y='Survived', data=train[train['Sex'] == 'female'])
plt.subplot(122)
sns.barplot(x='AdjTitle', y='Survived', data=train[train['Sex'] == 'male'])
plt.show()


I am working on the file "Explore.ipynb".
The first task is:
Missing Ages
Something we observed is that there are no missing titles. One possibility to impute missing ages would be to check how ages are distributed among titles:
Can you write Python code for it?

train.loc[train['AdjTitle'] == 'Master', 'Age'].describe()

train.loc[train['AdjTitle'] == 'Mr', 'Age'].describe()


I am working on the file "Explore.ipynb".
The first task is:
Only one pair of fares that are different (and not by much). For all we know, this could be an entry error, but let's ignore this for now. Let's look at embark locations:
Can you write Python code for it?


I am working on the file "Explore.ipynb".
The first task is:
Since she is relatively young and traveling alone, we'll throw her in with the "Miss" group.
Can you write Python code for it?

title_map = {'Mr': 'Mr',
             'Mrs': 'Mrs',
             'Miss': 'Miss',
             'Master': 'Master',
             'Dr': 'Dr',
             'Rev': 'Rev',
             'Don': 'mnoble',
             'Sir': 'mnoble',
             'Jonkheer': 'mnoble',
             'Lady': 'fnoble',
             'the Countess': 'fnoble',
             'Dona': 'fnoble',
             'Col': 'mil',
             'Capt': 'mil',
             'Major': 'mil',
             'Mme': 'Mrs',
             'Mlle': 'Miss',
             'Ms': 'Miss'}

train['AdjTitle'] = train['Title'].map(title_map)
test['AdjTitle'] = test['Title'].map(title_map)


I am working on the file "Explore.ipynb".
The first task is:
So any male with the title 'Master' is no more than 12! This puts him in the (luckier) basket of male children, increasing his surival odds. Similarly, most (but not all) males who are 'Mr' are above 12, making them less likely to be lucky.
Can you write Python code for it?


I am working on the file "Explore.ipynb".
The first task is:
The low number of most of the titles suggest grouping up the more esoteric ones. We'll do so as follows (there are no hard rules, so we'll use some judgment):

Merge Mme. into Mrs. and Mlle. into Miss.
Merge Lady, the Countess, and Dona (from the test set) into a category of noblewomen.
Merge Don, Sir, and Jonkheer into a category of noblemen.
Merge Col, Capt, and Major into a category of military.

For 'Ms.', we'll look at the woman's age, and also check her party.
Can you write Python code for it?


I am working on the file "Explore.ipynb".
The first task is:
The names look very consistently formatted, in the form of (last), (title). (first) (middle)
Since there are only a handful of distinct titles (versus the largely unique names), we'll extract this information:
Can you write Python code for it?

train['Title'] = train['Name'].str.extract('\,\s(.*?)[.]', expand=False)
print(train['Title'].unique())

test['Title'] = test['Name'].str.extract('\,\s(.*?)[.]', expand=False)
print(test['Title'].unique())


I am working on the file "Explore.ipynb".
The first task is:
There's obviously a dichomotomy in both graphs: We see that teenaged or older males had a very poor survival rate compared with younger males. It seems back in the day, teenage boys were not considered "children."
For females, age seems to matter much less. There is a cutoff with about 50/50 survival rate (very young children dependent on others?) somewhere around 11 to 15.
To find a good cutoff point for "child" versus "adult," we can zoom our data in around ages 10-15.
Can you write Python code for it?

train.loc[(train['Age'] < 15) & (train['Age'] > 10), ['Age', 'Survived', 'Sex']].sort_values(['Sex', 'Age'])


I am working on the file "Explore.ipynb".
The first task is:
This is still somewhat right skewed. If we want, we can later use a square root transform; however, for now, we will leave the Fare as is.
Cabins
Only a fraction of the passengers have known cabin information. We'll create a feature called CabinKnown that indicates if the cabin is given. Let's see if a known cabin is related to survival:
Can you write Python code for it?

train['CabinKnown'] = train['Cabin'].notnull()
pd.crosstab(train['CabinKnown'], train['Survived'])

plt.figure(2)
sns.barplot(x='CabinKnown', y='Survived', data=train)
plt.show()


I am working on the file "Explore.ipynb".
The first task is:
To start with, let's get an idea of how many passengers are holding each title.
Can you write Python code for it?

plt.figure(3, figsize=(14, 4))
plt.subplot(121)
sns.countplot(train.loc[train['Sex'] == 'female', 'Title'])
plt.subplot(122)
sns.countplot(train.loc[train['Sex'] == 'male', 'Title'])
plt.show()


I am working on the file "Explore.ipynb".
The first task is:
We can set the cutoff at 12 or below to be considered a "child," and 13 or above to be considered an "adult." This will capture the border cases of two 13 year old girls surviving, and a 12 year old boy surviving.
Can you write Python code for it?


I am working on the file "Explore.ipynb".
The first task is:
We have many more duplicate values here; it's plausible different families could split tickets, or bring servants/maids. Also, for family size, it's worth remembering that non-married partners do not count toward SibSp.
Fares
We see that Fare is a highly right-skewed variable.
Can you write Python code for it?

plt.figure(10)
sns.distplot(train['Fare'])
plt.show()
st.skew(train['Fare'])


I am working on the file "Explore.ipynb".
The first task is:
We see that almost all of the fares have shared cabins and shared tickets. Let's test the theory that 'Fare' refers to a group fare of all tickets with the same number, rather than fare per ticket:
Can you write Python code for it?

train['TicketSize'] = train['Ticket'].value_counts()[train['Ticket']].values
test['TicketSize'] = test['Ticket'].value_counts()[test['Ticket']].values

plt.figure(11, figsize=(12, 4))
plt.subplot(131)
sns.regplot(x='TicketSize', y='Fare', data=train[train['Pclass'] == 1])
plt.subplot(132)
sns.regplot(x='TicketSize', y='Fare', data=train[train['Pclass'] == 2])
plt.subplot(133)
sns.regplot(x='TicketSize', y='Fare', data=train[train['Pclass'] == 3])
plt.show()


I am working on the file "Exploring Heating Problems in Manhattan.ipynb".
The first task is:
How does the number of complaints vary by Zip code? Let's remove any data points where a zip code hasn't been provided and filter to those that are of type 'HEAT/HOT WATER'.
Note: If just exploring the data, where you do not intend to re-use the resulting dataframe, you can just use Spark SQL with the function "show" without assigning it to a variable.
Can you write Python code for it?

spark.sql("select `Incident Zip` as Zip, count(*) as ZipHeatingCnt "
          "from nyc311ct "
          "where `Complaint Type` = 'HEAT/HOT WATER' and `Incident Zip` <> '' group by `Incident Zip`").show()


I am working on the file "Exploring Heating Problems in Manhattan.ipynb".
The first task is:
Let's get a visual representation of the data within nyc311Agr_df. This next cell contains statements to create a custom pandas data frame which will be used to create a bubble chart in the following cell, where the size of the bubble represents the number of complaints. The complaint type is assigned a color, and, if large enough, the bubble is labeled, else the type displayed when hovered over.
Can you write Python code for it?

custom_frame = nyc311Agr_df.toPandas()
custom_frame.head(4)

%brunel data('custom_frame') bubble size(Complaint_Count) color(Complaint_Type) label(Complaint_Type) legends(none) tooltip(Complaint_Type)


I am working on the file "Exploring Heating Problems in Manhattan.ipynb".
The first task is:
Let's see which date and zip codes had the most complaints. The 'Created Date' field includes a time, therefore use the "split" function to just use the date, and also limit the data to only heat/hot water complaints, and for the year of 2016.
Can you write Python code for it?

spark.sql("select split(`Created Date`, ' ')[0] as Incident_Date, `Incident Zip` as Incident_Zip, "
          "count(`Unique Key`) as HeatingComplaintCount "
          "from nyc311ct where `Complaint Type` = 'HEAT/HOT WATER' and `Incident Zip` <> '' "
          "and split(split(`Created Date`, ' ')[0], '/')[2] = '16' "
          "group by split(`Created Date`, ' ')[0], `Incident Zip` order by HeatingComplaintCount desc limit 50").show()


I am working on the file "Exploring Heating Problems in Manhattan.ipynb".
The first task is:
Read the Data
The data was previously  saved to an Object Store; so the following code is an example of the appropriate credentials to be able to access and read the data. Use the Insert the Code function to insert your credentials. This statement explicitly tells Spark that the data is in CSV format, has a header, and that the document schema should be inferred. This will be saved as a DataFrame named nyc311DF
Can you write Python code for it?

import ibmos2spark

credentials = {
    'endpoint': 'https://s3-api.us-geo.objectstorage.service.networklayer.com',
    'api_key': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    'service_id': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
    'iam_service_endpoint': 'https://iam.ng.bluemix.net/oidc/token'}

configuration_name = 'os_xxxxxxxxxxxxxxxxxxxxxxxxxx_configs'
cos = ibmos2spark.CloudObjectStorage(sc, credentials, configuration_name, 'bluemix_cos')

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
nyc311DF = spark.read\
  .format('org.apache.spark.sql.execution.datasources.csv.CSVFileFormat')\
  .option('header', 'true')\
  .load(cos.url('IAE_examples_data_311NYC.csv', 'trafficanalysisxxxxxxxxxxxxxxxxxxxxxxx'))
nyc311DF.take(5)


I am working on the file "Exploring Heating Problems in Manhattan.ipynb".
The first task is:
Spark SQL Exploration
SparkSQL is a powerful tool allowing users a (often) familiar and (relatively) intuitive way to explore the data. In order to refer to the data within an SQL query, it needs to be stored as a view. The statement below creates a temporary view named nyc311ct.
Can you write Python code for it?

nyc311DF.createOrReplaceTempView("nyc311ct")

spark.sql("select distinct Borough from nyc311ct").show()


I am working on the file "Exploring Public Datasets/nyc_restaurants/scripts/ingestRestaurantData.ipynb".
The first task is:
Create a dictionary of unique Addresses. We do this to avoid calling the Google geocoding api multiple times for the same address
Can you write Python code for it?

addDict = t[['Address','Zipcode']].copy(deep=True)
addDict = addDict.drop_duplicates()
addDict['Coord'] = [None]* len(addDict)


I am working on the file "Exploring Public Datasets/nyc_restaurants/scripts/ingestRestaurantData.ipynb".
The first task is:
Get address for the geolocation for each address. This step can take a while because it's calling the Google geocoding API for each unique address.
Can you write Python code for it?

for item_id, item in addDict.iterrows():
    if item_id % 100 == 0:
        print(item_id)
    if addDict['Coord'][item_id] == None:
        addDict['Coord'][item_id] = getLatLon(item)
    #print(addDict.loc[item_id]['Coord'])

# Save address dictionary to CSV
#addDict.to_csv('./dict_final.csv')

# Merge coordinates into original table
t1 = t.merge(addDict[['Address', 'Coord']])

# Keep only 1 value of score and grade per inspection
t2=t1.copy(deep=True)
t2['raw_num'] = t2.index
t2['RI'] = t2.apply(combineCT, axis=1)
yy = t2.groupby('RI').first().reset_index()['raw_num']

t2['Unique_Score'] = None
t2['Unique_Score'].loc[yy.values] = t2['Score'].loc[yy.values]
t2['Unique_Grade'] = None
t2['Unique_Grade'].loc[yy.values] = t2['Grade'].loc[yy.values]

del(t2['RI'])
del(t2['raw_num'])
del(t2['Grade'])
del(t2['Score'])

t2.rename(columns={'Unique_Grade' : 'Grade','Unique_Score':'Score'}, inplace=True)
t2['Grade'].fillna('', inplace=True)

t2.iloc[1]


I am working on the file "Exploring Public Datasets/nyc_restaurants/scripts/ingestRestaurantData.ipynb".
The first task is:
Import Data
Import restaurant inspection data into a Pandas dataframe
Can you write Python code for it?

t = pd.read_csv('https://data.cityofnewyork.us/api/views/43nn-pn8j/rows.csv?accessType=DOWNLOAD', header=0, sep=',', dtype={'PHONE':str, 'INSPECTION DATE':str});

## Helper Functions
from datetime import datetime
def str_to_iso(text):
    if text != '':
        for fmt in (['%m/%d/%Y']):
            try:
                #print(fmt)
                #print(datetime.strptime(text, fmt))
                return datetime.isoformat(datetime.strptime(text, fmt))

            except ValueError:
                #print(text)
                pass
                #raise ValueError('Changing date')
    else:

        return None

def getLatLon(row):
    if row['Address'] != '':
        location = geolocator.geocode(row['Address'], timeout=10000, sensor=False)
        if location != None:
            lat = location.latitude
            lon = location.longitude
            #print(lat,lon)
            return [lon, lat]

    elif row['Zipcode'] !='' or location != None:
        location = geolocator.geocode(row['Zipcode'], timeout=10000, sensor=False)

        if location != None:
            lat = location.latitude
            lon = location.longitude
            #print(lat,lon)
            return [lon, lat]
    else:
        return None

def getAddress(row):
    if row['Building'] != '' and row['Street'] != '' and row['Boro'] != '':
        x = row['Building']+' '+row['Street']+' '+row['Boro']+',NY'
        x = re.sub(' +',' ',x)
        return x
    else:
        return ''

def combineCT(x):

    return str(x['Inspection_Date'][0][0:10])+'_'+str(x['Camis'])


I am working on the file "Exploring Public Datasets/nyc_restaurants/scripts/ingestRestaurantData.ipynb".
The first task is:
In this example, we use the Google geocoding API to translate addresses into geo-coordinates. Google imposes usages limits on the API. If you are using this script to index data, you many need to sign up for an API key to overcome limits.
Can you write Python code for it?

from geopy.geocoders import GoogleV3
geolocator = GoogleV3()
# geolocator = GoogleV3(api_key=<your_google_api_key>)


I am working on the file "Exploring Public Datasets/nyc_restaurants/scripts/ingestRestaurantData.ipynb".
The first task is:
Index Data
Can you write Python code for it?

### Create and configure Elasticsearch index

# Name of index and document type
index_name = 'nyc_restaurants';
doc_name = 'inspection'

# Delete donorschoose index if one does exist
if es.indices.exists(index_name):
    es.indices.delete(index_name)

# Create donorschoose index
es.indices.create(index_name)

# Add mapping
with open('./inspection_mapping.json') as json_mapping:
    d = json.load(json_mapping)

es.indices.put_mapping(index=index_name, doc_type=doc_name, body=d)

# Index data
for item_id, item in t2.iterrows():
    if item_id % 1000 == 0:
        print(item_id)
    thisItem = item.to_dict()
    #thisItem['Coord'] = getLatLon(thisItem)
    thisDoc = json.dumps(thisItem);
    #pprint.pprint(thisItem)

    # write to elasticsearch
    es.index(index=index_name, doc_type=doc_name, id=item_id, body=thisDoc)


I am working on the file "Exploring netCDF Datasets Using xarray.ipynb".
The first task is:
Dataset attributes are metadata.
They tell us about the dataset as a whole:
how, when, and by whom it was created, how it has been modified, etc.
The meanings of the various attributes and the conventions for them that we use
in the Salish Sea MEOPAR project are documented elsewhere.
Variables also have attributes :
Can you write Python code for it?


I am working on the file "Exploring netCDF Datasets Using xarray.ipynb".
The first task is:
Dataset variables are xarray.DataArray objects.
In addition to their attributes,
they carry a bunch of other useful properties and methods that you can read about in the
xarray docs.
Perhaps most importantly the data associated with the variables
are stored as NumPy arrays.
So, we can use NumPy indexing and slicing to access the data values.
For instance, to get the latitudes and longitudes of the 4 corners of the domain:
Can you write Python code for it?

ds.latitude.shape

print('Latitudes and longitudes of domain corners:')
pt = (0, 0)
print('  0, 0:        ', ds.latitude.values[pt], ds.longitude.values[pt])
pt = (0, ds.latitude.shape[1] - 1)
print('  0, x-max:    ', ds.latitude.values[pt], ds.longitude.values[pt])
pt = (ds.latitude.shape[0] - 1, 0)
print('  y-max, 0:    ', ds.latitude.values[pt], ds.longitude.values[pt])
pt = (ds.latitude.shape[0] - 1, ds.longitude.shape[1] - 1)
print('  y-max, x-max:', ds.latitude.values[pt], ds.longitude.values[pt])


I am working on the file "Exploring netCDF Datasets Using xarray.ipynb".
The first task is:
Printing the string respresentation of the ds data structure that open_dataset()
returns gives us lots of information about the dataset and its metadata:
Can you write Python code for it?


I am working on the file "Exploring netCDF Datasets Using xarray.ipynb".
The first task is:
See the Exploring netCDF Datasets from ERDDAP
notebook for more information about ERDDAP dataset URLs.
We could have opened the same dataset from a local file with:
ds = xr.open_dataset('../../NEMO-forcing/grid/bathy_meter_SalishSea2.nc')
Can you write Python code for it?


I am working on the file "Exploring netCDF Datasets Using xarray.ipynb".
The first task is:
So, we have a dataset that has 2 dimensions called gridX and gridY
of size 398 and 898, respectively,
3 variables called longitude, latitude, and bathymetry,
and 2 coordinates with the same names as the dimensions,
gridX and gridY.
The xarray docs have a
good explanation and a diagram
about the distinction between coordinates and data variables.
If you are already familiar with working with netCDF datasets via the netCDF4-python package,
you will note that the dims and data_vars attributes provide similar information to that
produced by functions in the
SalishSeaTools.nc_tools module.
xarray provides a higher level Python interface to datasets.
We'll see how the dimensions and variables are related,
and how to work with the data in the variables in a moment,
but first,
let's look at the dataset attributes:
Can you write Python code for it?


I am working on the file "Exploring netCDF Datasets Using xarray.ipynb".
The first task is:
You can also access the entire variable data array,
or subsets of it using slicing.
Can you write Python code for it?

ds.latitude.values

ds.longitude.values[42:45, 128:135]

ds.longitude.values[:2, :2], ds.latitude.values[-2:, -2:]


I am working on the file "Exploring netCDF Datasets Using xarray.ipynb".
The first task is:
open_dataset() returns an xarray.Dataset object
that is xarray’s multi-dimensional equivalent of a pandas.DataFrame.
It is a dict-like container of labeled arrays (DataArray objects) with aligned dimensions.
It is designed as an in-memory representation of the data model from the netCDF file format.
Dataset objects have four key properties:

dims: a dictionary mapping from dimension names to the fixed length of each dimension
(e.g., {'x': 6, 'y': 6, 'time': 8})
data_vars: a dict-like container of DataArrays corresponding to variables
coords: another dict-like container of DataArrays intended to label points used in data_vars
(e.g., arrays of numbers, datetime objects or strings)
attrs: an OrderedDict to hold arbitrary metadata

Let's look at them one at a time:
Can you write Python code for it?


I am working on the file "Exploring the Bitcoin Cryptocurrency Market/notebook.ipynb".
The first task is:
3. How big is Bitcoin compared with the rest of the cryptocurrencies?

At the time of writing, Bitcoin is under serious competition from other projects, but it is still dominant in market capitalization. Let's plot the market capitalization for the top 10 coins as a barplot to better visualize this.

Can you write Python code for it?

#Declaring these now for later use in the plots
TOP_CAP_TITLE = 'Top 10 market capitalization'
TOP_CAP_YLABEL = '% of total cap'

# Selecting the first 10 rows and setting the index
cap10 = cap.head(10).set_index(cap.id[:10])

# Calculating market_cap_perc
cap10 = cap10.assign(market_cap_perc = lambda x: (x.market_cap_usd / cap.market_cap_usd.sum()) * 100)

# Plotting the barplot with the title defined above
ax = cap10.plot.bar(x='id',y='market_cap_perc',title=TOP_CAP_TITLE)

# Annotating the y axis with the label defined above
ax.set_ylabel(TOP_CAP_YLABEL)


I am working on the file "Exploring the Bitcoin Cryptocurrency Market/notebook.ipynb".
The first task is:
4. Making the plot easier to read and more informative

While the plot above is informative enough, it can be improved. Bitcoin is too big, and the other coins are hard to distinguish because of this. Instead of the percentage, let's use a log10 scale of the "raw" capitalization. Plus, let's use color to group similar coins and make the plot more informative1.

For the colors rationale: bitcoin-cash and bitcoin-gold are forks of the bitcoin blockchain2. Ethereum and Cardano both offer Turing Complete smart contracts. Iota and Ripple are not minable. Dash, Litecoin, and Monero get their own color.

1 This coloring is a simplification. There are more differences and similarities that are not being represented here.

2 The bitcoin forks are actually very different, but it is out of scope to talk about them here. Please see the warning above and do your own research.

Can you write Python code for it?

# Colors for the bar plot
COLORS = ['orange', 'green', 'orange', 'cyan', 'cyan', 'blue', 'silver', 'orange', 'red', 'green']

# Plotting market_cap_usd as before but adding the colors and scaling the y-axis
ax = cap10.plot.bar(x='id',y='market_cap_perc',title=TOP_CAP_TITLE, color=COLORS, log=True)

# Annotating the y axis with 'USD'
ax.set_ylabel("USD")

# Final touch! Removing the xlabel as it is not very informative
ax.set_xlabel('')


I am working on the file "Exploring the Bitcoin Cryptocurrency Market/notebook.ipynb".
The first task is:
5. What is going on?! Volatility in cryptocurrencies

The cryptocurrencies market has been spectacularly volatile since the first exchange opened. This notebook didn't start with a big, bold warning for nothing. Let's explore this volatility a bit more! We will begin by selecting and plotting the 24 hours and 7 days percentage change, which we already have available.

Can you write Python code for it?

# Selecting the id, percent_change_24h and percent_change_7d columns
volatility = dec6[['id', 'percent_change_24h', 'percent_change_7d']]

# Setting the index to 'id' and dropping all NaN rows
volatility = volatility.set_index('id').dropna()

# Sorting the DataFrame by percent_change_24h in ascending order
volatility = volatility.sort_values('percent_change_24h', ascending=True)

# Checking the first few rows
volatility.head()


I am working on the file "Exploring the Bitcoin Cryptocurrency Market/notebook.ipynb".
The first task is:
6. Well, we can already see that things are a bit crazy

It seems you can lose a lot of money quickly on cryptocurrencies. Let's plot the top 10 biggest gainers and top 10 losers in market capitalization.

Can you write Python code for it?

#Defining a function with 2 parameters, the series to plot and the title
def top10_subplot(volatility_series, title):
    # Making the subplot and the figure for two side by side plots
    fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(10, 6))

    # Plotting with pandas the barchart for the top 10 losers
    ax = (volatility_series[:10].plot.bar(color='darkred', ax=axes[0]))

    # Setting the figure's main title to the text passed as parameter
    fig.suptitle(title)

    # Setting the ylabel to '% change'
    ax.set_ylabel('% change')

    # Same as above, but for the top 10 winners
    ax = (volatility_series[-10:].plot.bar(color='darkblue', ax=axes[1]))

    # Returning this for good practice, might use later
    return fig, ax

DTITLE = "24 hours top losers and winners"

# Calling the function above with the 24 hours period series and title DTITLE
fig, ax = top10_subplot(volatility.percent_change_24h,DTITLE)


I am working on the file "Exploring the Bitcoin Cryptocurrency Market/notebook.ipynb".
The first task is:
7. Ok, those are... interesting. Let's check the weekly Series too.

800% daily increase?! Why are we doing this tutorial and not buying random coins?1

After calming down, let's reuse the function defined above to see what is going weekly instead of daily.

1 Please take a moment to understand the implications of the red plots on how much value some cryptocurrencies lose in such short periods of time

Can you write Python code for it?

# Sorting in ascending order
volatility7d = volatility.sort_values('percent_change_7d', ascending=True)

WTITLE = "Weekly top losers and winners"

# Calling the top10_subplot function
fig, ax = top10_subplot(volatility7d.percent_change_7d, WTITLE)


I am working on the file "Exploring the Bitcoin Cryptocurrency Market/notebook.ipynb".
The first task is:
9. Most coins are tiny

Note that many coins are not comparable to large companies in market cap, so let's divert from the original Investopedia definition by merging categories.

This is all for now. Thanks for completing this project!

Can you write Python code for it?

# Making a nice function for counting different marketcaps from the
# "cap" DataFrame. Returns an int.
# INSTRUCTORS NOTE: Since you made it to the end, consider it a gift :D
def capcount(query_string):
    return cap.query(query_string).count().id

# Labels for the plot
LABELS = ["biggish", "micro", "nano"]

# Using capcount count the biggish cryptos
biggish = ...

# Same as above for micro ...
micro = ...

# ... and for nano
nano =  ...

# Making a list with the 3 counts
values = ...

# Plotting them with matplotlib
# ... YOUR CODE FOR TASK 10 ...


I am working on the file "Exploring_Domestic_Flights.ipynb".
The first task is:
If a flight is more than 10 minutes late for leaving or arriving I have marked it with a boolean variable below.
Can you write Python code for it?

conditionsArr = [(df['ArrDelay'] > 10)]
choicesArr = [1]
df['isDelArr'] = np.select(conditionsArr, choicesArr, default = 0)

conditionsDep = [(df['DepDelay'] > 10)]
choicesDep = [1]
df['isDelDep'] = np.select(conditionsDep, choicesDep, default = 0)

ts = df[['isDelArr', 'isDelDep']].sum().to_frame()
ts.columns = ['Count']

sns.barplot(x = ts.index , y = ts.Count);


I am working on the file "Exploring_Domestic_Flights.ipynb".
The first task is:
Next thing we would like to do is remove cancelled flights. ALthough they can be interesting for other analysis for ours they are not needed. As a side note only 0.02% of flights were cancelled
Can you write Python code for it?

df = df_flights_2008
df = df[df.Cancelled != 1]


I am working on the file "Exploring_Domestic_Flights.ipynb".
The first task is:
The test and training errors are very similar this means the model is most likely bias. For this model since 30% of flights are normally delayed the accuracy is really only 4% better than random guessing. That is not very good. We need to play around a bit. We will try some transformations next
Can you write Python code for it?

X_dum[['DepTime_2', 'ArrTime_2', 'ActualElapsedTime_2','Distance_2']] = np.log(X_dum[['DepTime', 'ArrTime', 'ActualElapsedTime','Distance']])

X_train, X_test, y_train, y_test = train_test_split(X_dum, y, test_size=0.3)

#model = LogisticRegression()
#model = model.fit(X_train, y_train)

# check the accuracy on the training set
#print(model.score(X_train, y_train),
#model.score(X_test, y_test))

clf = svm.SVC(kernel='linear', C=1, random_state=0)
clf = clf.fit(X_train.head(n=10000), y_train.head(n=10000))
clf.score(X_test, y_test)


I am working on the file "Exploring_Domestic_Flights.ipynb".
The first task is:
This notebook will be describing an analysis of US domestic flight data. We will first take a dive into a single year, 2008 and do some basic analysis. Then we may expand this into multiple years to try and see trends. Finally we will use a single year to build a predictive model to predict weather or not a flight will be delayed or not.
Can you write Python code for it?

df_flights_2008 = pd.read_csv('/home/hakim/Documents/Airline_delays-/flight_data_historical/2008.csv')
print(df_flights_2008.tail())

df_flights_2008.shape

df_flights_2008.isnull().sum(axis=0)


I am working on the file "Exploring_Domestic_Flights.ipynb".
The first task is:
We will also remove diverted flights. For this analysis we would like to only focus on delays.
Can you write Python code for it?

df = df[df.Diverted != 1]
df.isnull().sum(axis=0)


I am working on the file "Exploring_Domestic_Flights.ipynb".
The first task is:
What we have left are all non-cancelled diverted flights. Therefore we can remove the columns related to that. Also as we can see by looking at the numbers about 76% percent of flights were on time. That is not great.
Can you write Python code for it?

df = df.drop(['Cancelled', 'Diverted', 'CancellationCode'], axis = 1)


I am working on the file "Exploring_decompaction_with_Python.ipynb".
The first task is:
Let's use this function to decompact a simple stratigraphic column that consists of 5 alternating layers of sand and mud.
Can you write Python code for it?

tops = np.array([1.0,1.1,1.15,1.3,1.5,2.0])
lith = np.array([0,1,0,1,0]) # lithology labels: 0 = mud, 1 = sand
phi_sand_0 = 0.49 # surface porosity for sand
phi_mud_0 = 0.63 # surface porosity for mud
c_sand = 0.27 # porosity-depth coefficient for sand (km-1)
c_mud = 0.57 # porosity-depth coefficent for mud (km-1)
tops_new = decompact(tops,lith,0.0,phi_sand_0,phi_mud_0,c_sand,c_mud) # compute new tops


I am working on the file "Exploring_decompaction_with_Python.ipynb".
The first task is:
Now let's see what happens if we use the 90% mud and 90% sand curves from Komminz et al. (2011).
Can you write Python code for it?

tops = np.array([1.0,1.1,1.15,1.3,1.5,2.0])
lith = np.array([0,1,0,1,0]) # lithology labels: 0 = mud, 1 = sand
c_sand = 1000/18605.0 # Kominz et al. 2011 >90% sand curve
c_mud = 1000/1671.0 # Kominz et al. 2011 >90% mud curve
phi_sand_0 = 0.407 # Kominz et al. 2011 >90% sand curve
phi_mud_0 = 0.614 # Kominz et al. 2011 >90% mud curve
tops_new = decompact(tops,lith,0.0,phi_sand_0,phi_mud_0,c_sand,c_mud) # compute new tops

plot_decompaction(tops,tops_new)


I am working on the file "Exploring_decompaction_with_Python.ipynb".
The first task is:
Plot the results:
Can you write Python code for it?

def plot_decompaction(tops,tops_new):
    for i in range(len(tops)-1):
        x = [0,1,1,0]
        y = [tops[i], tops[i], tops[i+1], tops[i+1]]
        if lith[i] == 0:
            color = 'xkcd:umber'
        if lith[i] == 1:
            color = 'xkcd:yellowish'
        plt.fill(x,y,color=color)
        x = np.array([2,3,3,2])
        y = np.array([tops_new[i], tops_new[i], tops_new[i+1], tops_new[i+1]])
        if lith[i] == 0:
            color = 'xkcd:umber'
        if lith[i] == 1:
            color = 'xkcd:yellowish'
        plt.fill(x,y,color=color)
    plt.gca().invert_yaxis()
    plt.tick_params(axis='x',which='both',bottom='off',top='off',labelbottom='off')
    plt.ylabel('depth (km)');

plot_decompaction(tops,tops_new)


I am working on the file "Exploring_decompaction_with_Python.ipynb".
The first task is:
Posing the problem
Given a sediment column of a certain lithology with its top at $y_1$ and its base at $y_2$, we are trying to find the thickness and average porosity of the same sediment column at a different depth (see figure below). We are going to set the new top $y_1'$ and work towards finding the new base $y_2'$.
Can you write Python code for it?


I am working on the file "Exploring_decompaction_with_Python.ipynb".
The first task is:
While the compaction trends for mud happen to be fairly similar in the plot above, the ones for sandy lithologies are very different. This highlights that porosity-depth curves vary significantly from one basin to another, and are strongly affected by overpressures and exhumation. Using local data and geological information is critical. As Giles et al. (1998) have put it, "The use of default compaction curves can introduce significant errors into thermal history and pore- fluid pressure calculations, particularly where little well data are available to calibrate the model." To see how widely - and wildly - variable compaction trends can be, check out the review paper by Giles et al. (1998).
Deriving the general decompaction equation
Compacting or decompacting a column of sediment means that we have to move it along the curves in the figure above. Let's consider the volume of water in a small segment of the sediment column (over which porosity does not vary a lot):
$$ dV_w = \phi dV_t $$
As we have seen before, porosity at depth $y$ is
$$ \phi(y) = \phi_0 e^{-cy} $$
The first equation then becomes
$$ dV_w = \phi_0 e^{-cy} dV_t $$
But
$$ dV_w = A dy_w $$
and
$$ dV_t = A dy_t $$
where $y_w$ and $y_t$ are the thicknesses that the water and total volumes occupy respectively, and $A$ is the area of the column we are looking at. So the relationship is equivalent to
$$ dy_w = \phi_0 e^{-cy} dy_t $$
If we integrate this over the interval $y_1$ to $y_2$ we get
$$ y_w = \int_{y1}^{y2} \phi_0 e^{-cy} dy_t $$
Integrating this yields
$$ y_w = \phi_0 \Bigg(\frac{1}{-c}e^{-cy_2} - \frac{1}{-c}e^{-cy_1}\Bigg) = \frac{\phi_0}{c} \big(e^{-cy_1}-e^{-cy_2}\big) $$
As the total thickness equals the sediment thickness plus the water "thickness", we get
$$ y_s = y_t - y_w = y_2 - y_1 - y_w = y_2 - y_1 - \frac{\phi_0}{c} \big(e^{-cy_1}-e^{-cy_2}\big) $$
The decompacted value of $y_w$ is
$$ y_w' = \frac{\phi_0}{c} \big(e^{-cy_1'}-e^{-cy_2'}\big) $$
Now we can write the general decompaction equation:
$$ y_2'-y_1' = y_s+y_w'  $$
That...
Can you write Python code for it?

# compaction function - the unknown variable is y2a
def comp_func(y2a,y1,y2,y1a,phi,c):
    # left hand side of decompaction equation:
    LHS = y2a - y1a
    # right hand side of decompaction equation:
    RHS = y2 - y1 - (phi/c)*(np.exp(-c*y1)-np.exp(-c*y2)) + (phi/c)*(np.exp(-c*y1a)-np.exp(-c*y2a))
    return LHS - RHS


I am working on the file "Exploring_decompaction_with_Python.ipynb".
The first task is:
Write code to (de)compact stratigraphic column with multiple layers
Next we write a function that does the depth calculation for more than one layer in a sedimentary column:
Can you write Python code for it?

def decompact(tops,lith,new_top,phi_sand,phi_mud,c_sand,c_mud):
    tops_new = [] # list for decompacted tops
    tops_new.append(new_top) # starting value
    for i in range(len(tops)-1):
        if lith[i] == 0:
            phi = phi_mud; c = c_mud
        if lith[i] == 1:
            phi = phi_sand; c = c_sand
        comp_func_1 = functools.partial(comp_func,y1=tops[i],y2=tops[i+1],y1a=tops_new[-1],phi=phi,c=c)
        base_new_a = tops_new[-1]+tops[i+1]-tops[i]
        base_new = bisect(comp_func_1, base_new_a, 4*base_new_a) # bisection
        tops_new.append(base_new)
    return tops_new


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
Checking some inequalities
Let's check that $\forall x\in\mathbb{R}, \mathrm{Ei}(x) \leq \mathrm{e}^x$:
Can you write Python code for it?


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
Limits
We can check that $\mathrm{Ei}(x)\to0$ for $x\to-\infty$ and $\mathrm{Ei}(x)\to+\infty$ for $x\to\infty$:
Can you write Python code for it?

for x in -np.linspace(1, 1000, 10):
    print(f"For x = {x:>6.3g}, Ei(x) = {Ei(x):>10.3g} : it goes to 0 quite fast!")

for x in np.linspace(1, 800, 9):
    print(f"For x = {x:>6.3g}, Ei(x) = {Ei(x):>10.3g} : it goes to +oo quite fast!")


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
Other plots
Can you write Python code for it?

X = np.linspace(1e-3, 2*e, 1000)  # 1000 points
Y = Ei(X)
plt.plot(X, Y, 'b')
plt.title("The function $Ei(x)$ on $[0, e^2]$")
plt.xlabel("$x$")
plt.ylabel("$y$")
plt.show()


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
Plots
And we can plot the $Ei(x)$ function, from $-1$ to $1$.
Can you write Python code for it?

plt.plot(X, Y, 'b')
plt.title("The function $Ei(x)$ on $[-1,1]$")
plt.xlabel("$x$")
plt.ylabel("$y$")
plt.show()


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
Special values
We can compute some special values, like $\mathrm{Ei}(1)$ and solving (numerically) $\mathrm{Ei}(x)=0$.
Can you write Python code for it?

Ei(1)

from scipy.optimize import root

res = root(Ei, x0=1)
res

print(f"The approximate solution to Ei(x)=0 is x0 = {res.x[0]} (for which Ei(x)={res.fun})...")


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
The difference is not too large:
Can you write Python code for it?

np.max(np.abs(Y - Y_3))

assert np.allclose(Y, Y_3, rtol=1e-6, atol=1e-6)
print(f"Our version of Ei(x) is the same as the one in scipy.special.expi ({len(X)} values).")


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
They both take about the same time, but the second implementation seems (slightly) faster.
Comparison with scipy.special.expi
The $\mathrm{Ei}$ function is also implemented as scipy.special.expi:
Can you write Python code for it?

from scipy.special import expi

Y_3 = expi(X)

np.allclose(Y, Y_3)


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
We can check that $\mathrm{Ei}(x)\to-\infty$ for $x\to0^-$ and $x\to0^+$:
Can you write Python code for it?

for x in -1/np.logspace(1, 20, 10):
    print(f"For x = {x:>10.3g} --> 0^-, Ei(x) = {Ei(x):>5.3g} : it doesn't go to -oo numerically!")

for x in 1/np.logspace(1, 20, 10):
    print(f"For x = {x:>8.3g} --> 0^+, Ei(x) = {Ei(x):>5.3g} : it doesn't go to -oo numerically!")


I am working on the file "Exponential_Integral_Python.ipynb".
The first task is:
We can compare which is fastest to evaluate:
Can you write Python code for it?


I am working on the file "Exposin Project.ipynb".
The first task is:
Earth-Apophis Transfer Example
This example reimplements the transfer example given in [2]. The resulting graph appears to match the results in [2].
Can you write Python code for it?

'''
This code will produce a porkchop plot of the J function over a range of launch times and flight times
for a transfer between Earth and Apophis.
'''
import math
import numpy
import PyKEP as pk
import matplotlib.pyplot as plt
import Vector

p1 = pk.planet.jpl_lp('earth')
p2 = pk.planet.mpcorb('99942   19.2   0.15 K107N 202.49545  126.41859  204.43202    3.33173  0.1911104  1.11267324   0.9223398  1 MPO164109  1397   2 2004-2008 0.40 M-v 3Eh MPCAPO     C802  (99942) Apophis            20080109')
k2 = 0.6
n = 0
isp_chem = 350
isp_lt = 4000

t0_range = [4000, 10000]
tof_range = [100, 900]

@numpy.vectorize
def J(t0, tof):
    ep1 = pk.epoch(t0)
    ep2 = pk.epoch(t0 + tof)
    r1, v1 = p1.eph(ep1)
    r2, v2 = p2.eph(ep2)
    resJ = []
    for longway in [True, False]:
        prob = pk.lambert_exposin(r1, r2, tof * pk.DAY2SEC, pk.MU_SUN, longway, n, k2)
        for i in range(prob.num_solutions()):
            exps = prob.get_exposins()[i]
            dv1 = Vector.mag(Vector.sub(prob.get_v1()[i], v1))
            dv2 = Vector.mag(Vector.sub(prob.get_v2()[i], v2))
            dvlt = exps.get_delta_v(pk.MU_SUN)
            resJ.append(1.0 - math.exp(-(dv1 + dv2) / 9.81 / isp_chem - dvlt / 9.81 / isp_lt))
    if len(resJ) == 0:
        return numpy.nan
    else:
        return numpy.nanmin(resJ)

t0 = numpy.linspace(t0_range[0], t0_range[1], 400)
tof = numpy.linspace(tof_range[0], tof_range[1], 200)
X, Y = numpy.meshgrid(t0, tof)
Z = J(X, Y)
mZ = numpy.ma.array(Z, mask=numpy.isnan(Z))
plt.figure()
plt.pcolor(X, Y, mZ, vmin=numpy.nanmin(Z), vmax=numpy.nanmax(Z))
plt.colorbar()
plt.title('J for Earth-Apophis direct transfer')
plt.show()


I am working on the file "Exposin Project.ipynb".
The first task is:
In all, the Python bindings contain:
lambert_exposin object
* get_v1()
* get_v2()
* get_r1()
* get_r2()
* get_tof()
* get_mu()
* get_transfer_angle()
* get_k2()
* has_solutions()
* num_solutions()
* min_revs()
* max_revs()
* get_exposins()
exposin object (no constructor)
* get_tof(psi,mu)
* get_theta(tof,mu)
* get_state(theta,mu)
* get_psi()
* get_revs()
* get_transfer_angle()
* get_final_mass(mu,isp,mass)
* get_maximum_thrust(mu,isp,mass)
* get_delta_v(mu)
A graphing utility in orbit_plots
C++ backend

lambert_exposin, class_exposin, and exposin classes are fully implemented

lambert_exposin provides the solutions to a Lambert BVP.

class_exposin handles the relevant exposin space for lambert_exposin and handles TOF searching - I do not think it is useful alone, so I did not create bindings in Python.

exposin represents a single exponential sinusoid trajectory.

A few tuning parameters are set at the top of each file (mostly tolerances and iteration limits).
Speed in comparison to conic lambert solver
Can you write Python code for it?

setup = '''
import PyKEP as pk
r1 = [149.0e9, 0.0, 0.0]
r2 = [300.0e9, -100.0e9, 0.0]
mu = 1.32e20
tof = 86400 * 550.0
lw = True
N = 0
k2 = 0.2
'''
import timeit
t_exp = timeit.repeat("pk.lambert_exposin(r1,r2,tof,mu,lw,N,k2)", setup, number=5000)
t_lam = timeit.repeat("pk.lambert_problem(r1,r2,tof,mu,lw,N)", setup, number=5000)

print 'Time ratio: %.2fx' % (max(t_exp)/min(t_lam))


I am working on the file "Exposin Project.ipynb".
The first task is:
REPL functionality and some other readout functions are implemented:
Can you write Python code for it?

print prob
exps = prob.get_exposins()
isp = 5000
mass = 4000
print 'Exposin details:'
for ex in exps:
    print '-'
    print 'Mass: %.1f kg -> %.1f kg'            % (mass, ex.get_final_mass(PyKEP.MU_SUN, isp, mass))
    print 'Maximum thrust needed: %.3f N'       % ex.get_maximum_thrust(PyKEP.MU_SUN, isp, mass)
    print 'Low-thrust delta-v: %.3f m/s'        % ex.get_delta_v(PyKEP.MU_SUN)
    print 'Revolutions: %i'                     % ex.get_revs()
    print 'Total angle traversed: %.1f degrees' % (ex.get_psi() * PyKEP.RAD2DEG)


I am working on the file "ExposureEntireNetwork.ipynb".
The first task is:
Choose between a directed or undirected graph (if the direction of the edges is taken into account or not)
Can you write Python code for it?

import networkx as nx
#Directed graph:
G = nx.DiGraph()
#Undirected graph:
#G = nx.Graph()

(plotN,G,pos,pngLimits) = utilsNetwork.draw_network(nodes, edges, weights,G)


I am working on the file "ExposureEntireNetwork.ipynb".
The first task is:
Choose between option A or B:
A: Provide a vector GIS file containing the location of the bridges (bridges_shp)
OR
B: Provide a .csv file containing the assets coordinates (one node per asset) (brNodes).
In this option, if you have the exact coordinates of the assets in the network, use find_br_rows_exact_coord (faster).
Else, if you want to search the closest node to the given coordinates, use find_br_rows_closest_coord
Can you write Python code for it?

#Option A
bridgesShp = '/home/openquake/GEM/Lifelines/Building_NetworkNew/Input_files/EntireNetwork/mo_br.shp'
brAdj = utilsNetwork.shp_adj_br(bridgesShp,maxDistBr,resultsFolder)
brRows = utilsNetwork.one_node_per_bridge(brAdj,shpAdjMaxDist,resultsFolder)

##### OR #####

#Option B
#brNodes = '/media/sf_Shared_Folder/testFilesExposure/br-simple.csv'
#brRows = utilsNetwork.find_br_rows_closest_coord(shpAdjMaxDist, brNodes)
#brRows = utilsNetwork.find_br_rows_exact_coord(shpAdjMaxDist,brNodes)


I am working on the file "ExposureEntireNetwork.ipynb".
The first task is:
If the fragility functions depend on the zone where the asset is located, provide files with the corresponding zones below:
Can you write Python code for it?

from time import gmtime, strftime
print "Task started at "+strftime("%Y-%m-%d %H:%M:%S", gmtime())

zoneA = './Input_files/ShpZones/ZonaA-buffer.shp'
zoneB = './Input_files/ShpZones/ZonaB-buffer.shp'
zoneC = './Input_files/ShpZones/ZonaC-buffer.shp'
zoneD = './Input_files/ShpZones/ZonaD-buffer.shp'

pathNodes = '/media/sf_Shared_Folder/testFilesExposure/network-ff.txt'

utilsNetwork.RSA_zones(zoneA,zoneB,zoneC,zoneD,pathNodes,resultsFolder)

print "Task ended at "+strftime("%Y-%m-%d %H:%M:%S", gmtime())


I am working on the file "ExposureEntireNetwork.ipynb".
The first task is:
In this case we assume four typologies of assets: three typologies of bridges and one of viaducts
FF1: Single span bridges (length < 40m)
FF2: Multi-span medium length (40m < length < 150m)
FF3: Multi-span major bridges (length > 150m)
FF4: Viaducts
Can you write Python code for it?

limit_length1 = 40.
limit_length2 = 150.
adj = utilsNetwork.sim_adj(shpAdjMaxDist,brRows,brPerpRows,maxDist,limit_length1,limit_length2)

(nodes, edges, weights) = utilsNetwork.save_files_networkx(adj,maxNumConn,resultsFolder)


I am working on the file "ExposureEntireNetwork.ipynb".
The first task is:
Option A: if the exposure model will only include point assets (the assets previously specified)
OR
Option B: if the exposure model will include point and linear assets
Can you write Python code for it?

#nodes = '/media/sf_Shared_Folder/testFilesExposure/network-nodes.txt'

#Option A: Select only the nodes that are bridges
utilsNetwork.exposure_only_br(nodes,adj,resultsFolder)

##### OR #####

#Option B: Select all the nodes (bridges and pavement)
#utilsNetwork.exposure_br_pav(nodes,adj,resultsFolder)


I am working on the file "ExposureEntireNetwork.ipynb".
The first task is:
Specify maxDist, the maximum distance between 2 points in the network (in km). If an edge is bigger than maxDist, it will be divided in two, creating a new point.
Note: only verifies if 1st or 2nd relation node distances are > maxDist
Can you write Python code for it?

maxDist = 20
maxDistBr = 20

shpAdjMaxDist = utilsNetwork.divide_edges(shpAdj,maxNumConn,maxDist)


I am working on the file "ExposureEntireNetwork.ipynb".
The first task is:
Specify the location of the vector GIS file containing the network
Can you write Python code for it?

networkShp = '/home/openquake/GEM/Lifelines/Building_NetworkNew/Input_files/EntireNetwork/mo_FINAL.shp'

(shpAdj,maxNumConn) = utilsNetwork.shp_adj(networkShp)


I am working on the file "ExposureShortestPath.ipynb".
The first task is:
Choose between a directed or undirected graph (if the direction of the edges is taken into account or not)
Can you write Python code for it?

import networkx as nx
#Directed graph:
#G = nx.DiGraph()
#Undirected graph:
G = nx.Graph()
(plotN,G,pos,pngLimits) = utilsNetwork.draw_network(nodes, edges, weights, G)
utilsNetwork.draw_nodes(source,destination,G,pos,pngLimits)

saveImage = False
filename = './pathLx.png'
utilsNetwork.shortest_path(G,source,destination,pos,filename,saveImage,resultsFolder,pngLimits)


I am working on the file "ExposureShortestPath.ipynb".
The first task is:
Choose between option A or B:
A: Provide a vector GIS file containing the location of the bridges (bridges_shp)
OR
B: Provide a .csv file containing the assets coordinates (one node per asset) (brNodes).
In this option, if you have the exact coordinates of the assets in the network, use find_br_rows_exact_coord (faster).
Else, if you want to search the closest node to the given coordinates, use find_br_rows_closest_coord
Can you write Python code for it?

#Option A
bridgesShp = '/media/sf_Shared_Folder/Paper_Scenarios/testFilesExposure/highways-partBr.shp'
bridgesShp = '/home/openquake/GEM/Lifelines/Building_NetworkNew/Input_files/ShortestPathSpain/mo_br.shp'

#brAdj = utilsNetwork.shp_adj_br(bridgesShp,maxDistBr,resultsFolder)
#brRows = utilsNetwork.one_node_per_bridge(brAdj,shpAdjMaxDist,resultsFolder)

##### OR #####

#Option B
brNodes = '../Input_files/railwayPathSines/bridges.csv'
brRows = utilsNetwork.find_br_rows_closest_coord(shpAdjMaxDist, brNodes)
#brRows = utilsNetwork.find_br_rows_exact_coord(shpAdjMaxDist,brNodes)


I am working on the file "ExposureShortestPath.ipynb".
The first task is:
In this case we assume four typologies of assets: three typologies of bridges and one of viaducts
FF1: Single span bridges (length < 40m)
FF2: Multi-span medium length (40m < length < 150m)
FF3: Multi-span major bridges (length > 150m)
FF4: Viaducts
Can you write Python code for it?


I am working on the file "ExposureShortestPath.ipynb".
The first task is:
Option A: If assets (viaducts) that cross the network are to be considered, provide a .csv file with the coordinates of these assets (brPerpNodes)
Option B: If these assets are not considered
Can you write Python code for it?

#Option A
#brPerpNodes = '/media/sf_Shared_Folder/testFilesExposure/br_links_nodes_mo.csv'
brPerpNodes = '/home/openquake/GEM/Lifelines/Building_NetworkNew/Input_files/ShortestPathSpain/br_nodes+links_nodes_mo.csv'
brPerpNodes = '/home/openquake/GEM/Lifelines/Building_NetworkNew/Input_files/railwayPathSines/viaducts.csv'

#brPerpRows = utilsNetwork.find_br_rows_exact_coord(shpAdjMaxDist,brPerpNodes)
brPerpRows = utilsNetwork.find_br_rows_closest_coord(shpAdjMaxDist,brPerpNodes)

#Option B
#brPerpRows = 0


I am working on the file "ExposureShortestPath.ipynb".
The first task is:
Shortest path
Specify the coordinates of the source and destination, to calculate the shortest path between these locations.
Can you write Python code for it?

#Source coordinates - Lisbon
#lon1 = -9.142685
#lat1 = 38.736946
#Source coordinates - Spain
#lon1 = -7.035
#lat1 = 38.883
#Source coordinates - Sines Port
lon1 = -8.836
lat1 = 37.939
#Destination coordinates - Factory
lon2 = -7.979
lat2 = 37.575

(source, destination) = utilsNetwork.node_in_out(adj,lon1,lat1,lon2,lat2)


I am working on the file "ExposureShortestPath.ipynb".
The first task is:
Specify maxDist, the maximum distance between 2 points in the network (in km). If an edge is bigger than maxDist, it will be divided in two, creating a new point.
Note: only verifies if 1st or 2nd relation node distances are > maxDist
Can you write Python code for it?


I am working on the file "ExposureShortestPath.ipynb".
The first task is:
Specify the location of the vector GIS file containing the network:
Can you write Python code for it?

networkShp = '/home/openquake/GEM/Lifelines/Building_NetworkNew/Input_files/ShortestPathSpain/mo_FINAL.shp'
networkShp = '/home/openquake/GEM/Lifelines/Building_NetworkNew/Input_files/railwayPathSines/railway.shp'
#networkShp = '/media/sf_Shared_Folder/Paper_Scenarios/testFilesExposure/highways-part.shp'

(shpAdj,maxNumConn) = utilsNetwork.shp_adj(networkShp)


I am working on the file "ExposureShortestPath.ipynb".
The first task is:
To create the a .csv file with the coordinates of the assets, specify the file created in the previous step.
Can you write Python code for it?

exposureTxt = '/media/sf_Shared_Folder/Paper_Scenarios/2016-09-Revision/path1-exposure.txt'

utilsNetwork.exposure_1path(exposureTxt,resultsFolder)


I am working on the file "Extended Precision.ipynb".
The first task is:
<img src="https://sciencenode.org/img/img_2012/stamp.JPG" alt="Ramanujan Postage Stamp"
style="width: 50%; height: 50%"/>
The fancier LaTeX below renders a famous equation by Ramanujan, which has been shown to converge to 1/π and therefore π very quickly, relative to many other algorithms.  I don't think anyone understands how some random guy could think up such a miraculaous equation.
Can you write Python code for it?


I am working on the file "Extended Precision.ipynb".
The first task is:
In the context below, the value of n starts at 10 and then gets two more zeros every time around the loop.
The yield keyword is similar to return in handing back an object, however a generator function then pauses to pick up where it left off when nudged by next(), (which triggers __next__ internally).
Generator functions do not forget their internal state as they advance through next values.
Note that when a Decimal type object operates with an integer, that integer is coerced (cast) as a Decimal object.
Can you write Python code for it?

with decimal.localcontext() as ctx:  # context manager
    ctx.prec = 1000
    def converge():  # generator function
        n = decimal.Decimal('10')
        while True:
            yield (1 + 1/n) ** n
            n = n * 100 # two more zeros
    f = converge()
    for _ in range(9):
        next(f) # f.__next__() <--- not quite like Python 2.x (f.next())

r = next(f)
r

str(r)[:20] == e_1000_places[:20]


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
Adding this unified display function
Can you write Python code for it?

modes = [ "str", "matrix", "call", "calls", "obj", "tree", "lol", "list", "tree" ]

def unified_lolviz(obj=None, mode:modes=None, **kwargs):
    """ Unified function to display `obj` with lolviz, in Jupyter notebook only."""
    try:
        if mode == "str" or isinstance(obj, str):
            return lolviz.strviz(obj, **kwargs)
        elif "<class 'numpy.ndarray'>" == str(type(obj)):
            # can't use isinstance(obj, np.ndarray) without import numpy!
            return lolviz.matrixviz(obj, **kwargs)
        elif mode == "matrix":
            # Experimental transparent use of np.array()
            try:
                from numpy import array as _
                return lolviz.matrixviz(_(obj), **kwargs)
                del _
            except:
                return lolviz.matrixviz(obj, **kwargs)
        elif mode == "call":
            from sys import _getframe
            return lolviz.callviz(frame=_getframe(1), **kwargs)
            del _getframe
        elif mode == "calls":
            return lolviz.callsviz( **kwargs)
        elif mode == "lol" or isinstance(obj, (tuple, list)) and obj and isinstance(obj[0], (tuple, list)):
            # obj is a list, is non empty, and obj[0] is a list!
            return lolviz.lolviz(obj, **kwargs)
        elif mode == "list" or isinstance(obj, (tuple, list)):
            return lolviz.listviz(obj, **kwargs)
        elif mode == "tree" or isinstance(obj, dict):
            return lolviz.treeviz(obj, **kwargs)  # default
        else:
            return lolviz.objviz(obj, **kwargs)  # default
    except TypeError:
        # unable to use lolviz, let's just return the object,
        # it will be nicely displayed by IPython
        return obj

viz = unified_lolviz


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
And the different modes, except for "call" and "calls" which only work inside functions:
Can you write Python code for it?

partial_modes = ['str', 'matrix', 'obj', 'tree', 'lol', 'list']

@interact(
    obj = objs,
    mode = partial_modes,
)
def showcase_unified_viz(obj, mode):
    print(f"\nObject of type = {type(obj)}, for mode = {mode}")
    pprint(obj)
    return viz(obj, mode=mode)
    return lolviz.strviz(obj)


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
And this works without having to manually import numpy or convert the list-of-list to a numpy array!
Can you write Python code for it?


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
But it's simpler to use the automatic function!
Can you write Python code for it?


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
If we want, we can force the lolviz.matrixviz mode:
Can you write Python code for it?


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
It works out of the box for dictionaries!
Let's check another example:
Can you write Python code for it?

anakin_rec = {
    "name": "Anakin Skywalker",
}

luke_rec = {
    "name": "Luke Skywalker",
    "father": anakin_rec,
}
leia_rec = {
    "name": "Leia Skywalker",
    "father": anakin_rec,
},

anakin_rec.update({
    "son": luke_rec,
    "daughter": leia_rec,
})

from pprint import pprint
pprint(anakin_rec)


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
It works, even if it is not as pretty.
Tree
Only for binary trees, apparently. Let's try with a dictionary that looks like a binary tree:
Can you write Python code for it?

anakin = {
    "name": "Anakin Skywalker",
    "son": {
        "name": "Luke Skywalker",
    },
    "daughter": {
        "name": "Leia Skywalker",
    },
}

from pprint import pprint
pprint(anakin)


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
List of lists of lists???
Can you write Python code for it?

n, m, o = 2, 3, 4
example_3D_matrix = [[[
    1 if i < j < k else 0
    for i in range(n)]
    for j in range(m)]
    for k in range(o)]

example_3D_matrix

viz(example_3D_matrix)


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
More experiments on a "triple" recursive function
Just to complicate things more, let's write a "fibonnaci" like recursive function that calls three sub-terms:
Can you write Python code for it?

cg = rcviz.CallGraph()

@rcviz.viz(cg)
def fibo_three(n):
    if   n <= 0: return 0
    elif n == 1: return 1
    elif n == 2: return 1
    else: return fibo_three(n-1) + fibo_three(n-2) + fibo_three(n-3)

n = 5
print(f"{n}! = {fibo_three(n)}")

cg.render()


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
Objects
Can you write Python code for it?

viz(anakin)

viz(anakin.values())

viz(anakin.items())


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
Okay, by using the optional frame= argument of lolviz.callviz, and some sys._getframe black magic, we can do this!

See https://code.activestate.com/recipes/579105-how-a-python-function-can-find-the-name-of-its-cal/

Can we write a @show_recursion function decorator, to automatically do this visualization of recursive calls for us?
Can you write Python code for it?

def show_recursion(varnames=None):
    def showing_recursion_with_varnames(function):
        nonlocal varnames
        # https://stackoverflow.com/questions/582056/getting-list-of-parameter-names-inside-python-function#4051447
        if varnames is None:
            varnames = function.__code__.co_varnames
        import lolviz
        from IPython.display import display
        from functools import wraps
        @wraps(function)
        def wrapped_function(*args, **kwargs):
            nonlocal display, wraps, varnames, lolviz
            ##print(f"args = {args}, of type = {type(args)}")
            ##print(f"kwargs = {kwargs}, of type = {type(kwargs)}")
            #print(f"varnames = {varnames}")
            display(lolviz.callsviz(varnames=varnames))
            return function(*args, **kwargs)
        return wrapped_function
    return showing_recursion_with_varnames

@show_recursion()
def factorial4(n, f0=0, f1=1):
    # this is automatically done: display(viz(varnames=["n"], mode="call"))
    if n < 0: return f0
    elif n == 0: return f1
    else: return n * factorial4(n - 1, f0=f0, f1=f1)

n = 4
print(f"{n}! = {factorial4(n)}")


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
Testing naively
Can you write Python code for it?

data = ['hi', 'mom', {3, 4}, {"parrt": "user"}]
viz(data)


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
Then by remembering the correction function, we can do:
Can you write Python code for it?


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
This is what lolviz.callviz shows:
Can you write Python code for it?

def factorial2(n):
    display(lolviz.callviz(varnames=["n"]))
    if n < 0: return 0
    elif n == 0: return 1
    else: return n * factorial2(n - 1)

n = 4
print(f"{n:>2}! = {factorial2(n):>10}")


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
Trying on all the objects used above:
Can you write Python code for it?

objs = [
    string.ascii_lowercase,
    lolviz,
    data,
    squares,
    example_matrix,
    example_3D_matrix,
    anakin,
    anakin_rec,
    leia_rec,
    luke_rec,
    root,
    z,
]


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
Under the hood of this unified function viz, it was calling lolvzi.listviz:
Can you write Python code for it?


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
Using this non-memoized function leads to already 13 nodes for just $n=5$!
Can you write Python code for it?

cg = rcviz.CallGraph()

@rcviz.viz(cg)
@memoize(maxsize=None)
def fibo_three_memo(n):
    if   n <= 0: return 0
    elif n == 1: return 1
    elif n == 2: return 1
    else: return fibo_three_memo(n-1) + fibo_three_memo(n-2) + fibo_three_memo(n-3)

n = 5
print(f"{n}! = {fibo_three_memo(n)}")

cg.render()


I am working on the file "Extending_lolviz_with_a_unified_display_function_that_chooses_the_best_one_based_on_the_datatype.ipynb".
The first task is:
We really see the "call stack" as the system keeps track of the nested calls. I like that! 👌
But again, I'm lazy, I want to just use the viz function:
Can you write Python code for it?

def factorial3(n):
    display(viz(varnames=["n"], mode="call"))
    if n < 0: return 0
    elif n == 0: return 1
    else: return n * factorial3(n - 1)

n = 4
print(f"{n}! = {factorial3(n)}")


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Below is a small set of gspread examples. Additional examples are shown on the gspread Github page.
Creating a new sheet with data from Python
Can you write Python code for it?


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Drive REST API
The first step is to authenticate.
Can you write Python code for it?

from google.colab import auth
auth.authenticate_user()


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Google Cloud Storage (GCS)
We'll start by authenticating to GCS and creating the service client.
Can you write Python code for it?


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Google Sheets
Our examples below will use the existing open-source gspread library for interacting with Sheets.
First, we'll install the package using pip.
Can you write Python code for it?


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Next, we'll import the library, authenticate, and create the interface to sheets.
Can you write Python code for it?

from google.colab import auth
auth.authenticate_user()

import gspread
from oauth2client.client import GoogleCredentials

gc = gspread.authorize(GoogleCredentials.get_application_default())


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Now we can construct a Drive API client.
Can you write Python code for it?

from googleapiclient.discovery import build
drive_service = build('drive', 'v3')


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Once the upload has finished, the data will appear in the cloud console storage browser for your project:
https://console.cloud.google.com/storage/browser?project=YOUR_PROJECT_ID_HERE
Downloading a file from GCS to Python
Next, we'll download the file we just uploaded in the example above. It's as simple as reversing the order in the gsutil cp command.
Can you write Python code for it?

# Download the file.
!gsutil cp gs://{bucket_name}/to_upload.txt /tmp/gsutil_download.txt

# Print the result to make sure the transfer worked.
!cat /tmp/gsutil_download.txt


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
PyDrive
The example below shows 1) authentication, 2) file upload, and 3) file download. More examples are available in the PyDrive documentation
Can you write Python code for it?

!pip install -U -q PyDrive

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

# 1. Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

# PyDrive reference:
# https://gsuitedevs.github.io/PyDrive/docs/build/html/index.html

# 2. Create & upload a file text file.
uploaded = drive.CreateFile({'title': 'Sample upload.txt'})
uploaded.SetContentString('Sample upload file content')
uploaded.Upload()
print('Uploaded file with ID {}'.format(uploaded.get('id')))

# 3. Load a file by ID and print its contents.
downloaded = drive.CreateFile({'id': uploaded.get('id')})
print('Downloaded content "{}"'.format(downloaded.GetContentString()))


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
The cell below uploads the file to our newly created bucket.
Can you write Python code for it?

from googleapiclient.http import MediaFileUpload

media = MediaFileUpload('/tmp/to_upload.txt',
                        mimetype='text/plain',
                        resumable=True)

request = gcs_service.objects().insert(bucket=bucket_name,
                                       name='to_upload.txt',
                                       media_body=media)

response = None
while response is None:
  # _ is a placeholder for a progress object that we ignore.
  # (Our file is small, so we skip reporting progress.)
  _, response = request.next_chunk()

print('Upload complete')


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Upload a file from Python to a GCS bucket
We'll start by creating the sample file to be uploaded.
Can you write Python code for it?

# Create a local file to upload.
with open('/tmp/to_upload.txt', 'w') as f:
  f.write('my sample file')

print('/tmp/to_upload.txt contains:')
!cat /tmp/to_upload.txt


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Using Python
This section demonstrates how to upload files using the native Python API rather than gsutil.
This snippet is based on a larger example with additional uses of the API.
Can you write Python code for it?

# The first step is to create a bucket in your cloud project.
#
# Replace the assignment below with your cloud project ID.
#
# For details on cloud projects, see:
# https://cloud.google.com/resource-manager/docs/creating-managing-projects
project_id = 'Your_project_ID_here'

# Authenticate to GCS.
from google.colab import auth
auth.authenticate_user()

# Create the service client.
from googleapiclient.discovery import build
gcs_service = build('storage', 'v1')

# Generate a random bucket name to which we'll upload the file.
import uuid
bucket_name = 'colab-sample-bucket' + str(uuid.uuid1())

body = {
  'name': bucket_name,
  # For a full list of locations, see:
  # https://cloud.google.com/storage/docs/bucket-locations
  'location': 'us',
}
gcs_service.buckets().insert(project=project_id, body=body).execute()
print('Done')


I am working on the file "External_data_Drive,_Sheets,_and_Cloud_Storage.ipynb".
The first task is:
Using Python
We repeat the download example above using the native Python API.
Can you write Python code for it?

# Authenticate to GCS.
from google.colab import auth
auth.authenticate_user()

# Create the service client.
from googleapiclient.discovery import build
gcs_service = build('storage', 'v1')

from apiclient.http import MediaIoBaseDownload

with open('/tmp/downloaded_from_gcs.txt', 'wb') as f:
  request = gcs_service.objects().get_media(bucket=bucket_name,
                                            object='to_upload.txt')
  media = MediaIoBaseDownload(f, request)

  done = False
  while not done:
    # _ is a placeholder for a progress object that we ignore.
    # (Our file is small, so we skip reporting progress.)
    _, done = media.next_chunk()

print('Download complete')

# Inspect the file we downloaded to /tmp
!cat /tmp/downloaded_from_gcs.txt


I am working on the file "Extra 1 - Comparing ML algorithms.ipynb".
The first task is:
Charting the relative performance
In this section, we plot the measurements (accuracy and time) collected above to compare the relative difference in the performance of the tested against that of Decision Tree classifiers.
Since we have normalised the measurements agains those of Decision Tree classifiers, the measurements of Decision Tree classifiers average at 1.0 (for both accuracy and time). Therefore, to simplify the charts, we remove them from the plotted data.
Can you write Python code for it?

# Dropping measurements of DecisionTreeClassifier
performance = performance[~(performance.classifier == 'DecisionTreeClassifier')]

%matplotlib inline
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("poster")


I am working on the file "Extra 1 - Comparing ML algorithms.ipynb".
The first task is:
CollabMap Datasets
There are three datasets from CollabMap: buildings, routes, and route sets. See Application 2 - CollabMap Data Quality.ipynb for more details.
Can you write Python code for it?

df = pd.read_csv("collabmap/depgraphs.csv", index_col='id')
trust_threshold = 0.75
df['label'] = df.apply(lambda row: 'Trusted' if row.trust_value >= trust_threshold else 'Uncertain', axis=1)
df.drop('trust_value', axis=1, inplace=True)

datasets.append(('CollabMap/Buildings', df.filter(like="Building", axis=0)))
datasets.append(('CollabMap/Routes', df.filter(regex="^Route\d", axis=0)))
datasets.append(('CollabMap/Routesets', df.filter(like="RouteSet", axis=0)))


I am working on the file "Extra 1 - Comparing ML algorithms.ipynb".
The first task is:
Loading and preparing data
Can you write Python code for it?

datasets = []  # list of the datasets to be tested with the above classifiers


I am working on the file "Extra 1 - Comparing ML algorithms.ipynb".
The first task is:
Next, we list the classifiers to be trained and tested in our experiment in classifier_classes.
Can you write Python code for it?

from sklearn.model_selection import StratifiedKFold
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.linear_model import SGDClassifier
from sklearn.neural_network import MLPClassifier

classifier_classes = [
    DecisionTreeClassifier,
    SVC,
    KNeighborsClassifier,
    RandomForestClassifier,
    AdaBoostClassifier,
    GradientBoostingClassifier,
    GaussianNB,
    QuadraticDiscriminantAnalysis,
    SGDClassifier,
    MLPClassifier
]


I am working on the file "Extra 1 - Comparing ML algorithms.ipynb".
The first task is:
Radiation Response Game Dataset
Application 3 - RRG Messages.ipynb
Can you write Python code for it?

filepath = lambda k: "rrg/depgraphs-%d.csv" % k
label = lambda l: 'other' if l != 'instruction' else l
df = pd.read_csv(filepath(11), index_col=0)
df.label = df.label.apply(label).astype('category')
datasets.append(('RRG/k=11', df))


I am working on the file "Extra 1 - Comparing ML algorithms.ipynb".
The first task is:
Results: The chart above shows that, with all the 5 datasets, all other classifiers yield the same level of accuracy as that of Decision Tree classifiers (1.0) or worse, with the only exception of the performance of Random Forest classifiers on the CollabMap/Routes and CollabMap/Routesets datasets. Having said that, the improvement in accuracy is only about 1% as shown below.
Some algorithms have better mean accuracy on the RRG/k=11 dataset, but the 95% confidence intervals are too broad for us to say conclusively that they are actually better than the decision tree classifier.
Can you write Python code for it?

performance[
    (performance.classifier == 'RandomForestClassifier') & (performance.dataset == 'CollabMap/Routes')
].mean()

performance[
    (performance.classifier == 'RandomForestClassifier') & (performance.dataset == 'CollabMap/Routesets')
].mean()


I am working on the file "Extra 1 - Comparing ML algorithms.ipynb".
The first task is:
Since we want to compare the other classifier against Decision Tree classifiers, we use the performance of the Decision Tree classifiers as the base line. The normalisation function below divides the accuracy and computing time of all classifiers by the mean accuracy and mean computing time of Decision Tree classifiers, respectively.
Can you write Python code for it?

def normalise_accuracy_time(performance, baseline_classifier="DecisionTreeClassifier"):
    baseline_accuracy, baseline_time = performance[performance.classifier == baseline_classifier].mean()
    performance.accuracy = performance.accuracy / baseline_accuracy
    performance.time = performance.time / baseline_time


I am working on the file "Extra 1 - Comparing ML algorithms.ipynb".
The first task is:
Testing and collecting measurements
First, we define the Timer class (below, taken from this recipe) to measure the computing time of training and classification. Note that this class allows us to disable garbage collection between measurements for more consistent results.
Can you write Python code for it?

import gc
import timeit

class Timer:
    def __init__(self, timer=None, disable_gc=False, verbose=True):
        if timer is None:
            timer = timeit.default_timer
        self.timer = timer
        self.disable_gc = disable_gc
        self.gc_state = None
        self.verbose = verbose
        self.start = self.end = self.interval = None

    def __enter__(self):
        if self.disable_gc:
            self.gc_state = gc.isenabled()
            gc.disable()
        self.start = self.timer()
        return self

    def __exit__(self, *args):
        self.end = self.timer()
        if self.disable_gc and self.gc_state:
            gc.enable()
            self.gc_state = None
        self.interval = self.end - self.start
        if self.verbose:
            print('time taken: %f seconds' % self.interval)


I am working on the file "Extra 2.1 - Unbalanced Data - Application 1.ipynb".
The first task is:
Balancing the data
With an unbalanced like the above, the resulted trained classifier will typically be skewed towards the majority labels. In order to mitigate this, we balance the dataset using the SMOTE Oversampling Method.
Can you write Python code for it?

df = balance_smote(df)

results_bal, importances_bal = test_classification(df)


I am working on the file "Extra 2.1 - Unbalanced Data - Application 1.ipynb".
The first task is:
Classification on unbalanced (original) data
Can you write Python code for it?

from analytics import test_classification


I am working on the file "Extra 2.1 - Unbalanced Data - Application 1.ipynb".
The first task is:
Cross Validation tests: We now run the cross validation tests on the dataset (df) using all the features (combined), only the generic network metrics (generic), and only the provenance-specific network metrics (provenance). Please refer to Cross Validation Code.ipynb for the detailed description of the cross validation code.
Can you write Python code for it?

results, importances = test_classification(df)


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
## Classification on balanced data
We repeat the same experiements but now with balanced datasets
Balancing Data
This section explore the balance of each of the three datasets and balance them using the SMOTE Oversampling Method.
Can you write Python code for it?


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Balancing the route set dataset:
Can you write Python code for it?


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Building Classification
We test the classification of buildings, collect individual accuracy scores results and the importance of every feature in each test in importances (both are Pandas Dataframes). These two tables will also be used to collect data from testing the classification of routes and route sets later.
Can you write Python code for it?

# Cross validation test on building classification
res, imps = test_classification(df_buildings)

# adding the Data Type column
res['Data Type'] = 'Building'
imps['Data Type'] = 'Building'

# storing the results and importance of features
results_bal = res
importances_bal = imps


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Charting the accuracy scores
Can you write Python code for it?

%matplotlib inline
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("talk")


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Classification on unbalanced (original) data
We now run the cross validation tests on the three unbalanced datasets (df_buildings, df_routes, and df_routesets) using all the features (combined), only the generic network metrics (generic), and only the provenance-specific network metrics (provenance). Please refer to Cross Validation Code.ipynb for the detailed description of the cross validation code.
Can you write Python code for it?


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Combining the results
Can you write Python code for it?

# Merging the two result sets
results_unb['Balanced'] = False
results_bal['Balanced'] = True
results = results_unb.append(results_bal, ignore_index=True)


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Converting the accuracy score from [0, 1] to percentage, i.e [0, 100]:
Can you write Python code for it?

results.Accuracy = results.Accuracy * 100

pal = sns.light_palette("seagreen", n_colors=3, reverse=True)
g = sns.factorplot(data=results, x='Data Type', y='Accuracy', hue='Metrics', col='Balanced',
                   kind='bar', palette=pal, aspect=1.2, errwidth=1, capsize=0.04)
g.set(ylim=(88, 98))


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Filtering data
We split the dataset into three: buildings, routes, and route sets.
Can you write Python code for it?

df_buildings = df.filter(like="Building", axis=0)
df_routes = df.filter(regex="^Route\d", axis=0)
df_routesets = df.filter(like="RouteSet", axis=0)
df_buildings.shape, df_routes.shape, df_routesets.shape  # The number of data points in each dataset


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Having used the trust valuue to label all the data entities, we remove the trust_value column from the data frame.
Can you write Python code for it?

# We will not use trust value from now on
df.drop('trust_value', axis=1, inplace=True)
df.shape  # the dataframe now have 23 columns (22 metrics + label)


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Labelling data
Based on its trust value, we categorise the data entity into two sets: trusted and uncertain. Here, the threshold for the trust value, whose range is [0, 1], is chosen to be 0.75.
Can you write Python code for it?

trust_threshold = 0.75
df['label'] = df.apply(lambda row: 'Trusted' if row.trust_value >= trust_threshold else 'Uncertain', axis=1)
df.head()  # The new label column is the last column below


I am working on the file "Extra 2.2 - Unbalanced Data - Application 2.ipynb".
The first task is:
Route Classification
Can you write Python code for it?

# Cross validation test on route classification
res, imps = test_classification(df_routes)

# adding the Data Type column
res['Data Type'] = 'Route'
imps['Data Type'] = 'Route'

# storing the results and importance of features
results_unb = results_unb.append(res, ignore_index=True)
importances_unb = importances_unb.append(imps, ignore_index=True)


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Balancing the building dataset:
Can you write Python code for it?


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Balancing the route dataset:
Can you write Python code for it?


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Balancing the route set dataset:
Can you write Python code for it?


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Charting the accuracy scores
Can you write Python code for it?

%matplotlib inline
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("paper", font_scale=1.4)


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Converting the accuracy score from [0, 1] to percentage, i.e [0, 100]:
Can you write Python code for it?

results.Accuracy = results.Accuracy * 100
results.head()

from matplotlib.font_manager import FontProperties
fontP = FontProperties()
fontP.set_size(12)

pal = sns.light_palette("seagreen", n_colors=3, reverse=True)
plot = sns.barplot(x="Data Type", y="Accuracy", hue='Metrics', palette=pal, errwidth=1, capsize=0.02, data=results)
plot.set_ylim(40, 90)
plot.legend(loc='upper center', bbox_to_anchor=(0.5, 1.0), ncol=3)
plot.set_ylabel('Accuracy (%)')


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Cross Validation
We now run the cross validation tests on the three balanaced datasets (df_buildings, df_routes, and df_routesets) using all the features (combined), only the generic network metrics (generic), and only the provenance-specific network metrics (provenance). Please refer to Cross Validation Code.ipynb for the detailed description of the cross validation code.
Can you write Python code for it?


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Filtering data
We split the dataset into three: buildings, routes, and route sets.
Can you write Python code for it?


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Labelling data
Based on its trust value, we categorise the data entity into two sets: trusted and uncertain. Here, the threshold for the trust value, whose range is [0, 1], is chosen to be 0.75.
Can you write Python code for it?


I am working on the file "Extra 3.1 - Historical Provenance - Application 2.ipynb".
The first task is:
Routes
Can you write Python code for it?


I am working on the file "Extra 3.2 - Historical Provenance - Application 3.ipynb".
The first task is:
Balancing data
This section explore the balance of the RRG datasets.
Can you write Python code for it?

# Examine the balance of the dataset
df.label.value_counts()


I am working on the file "Extra Resources.ipynb".
The first task is:
As you can see, this fails because we don't have sufficient privileges. On your own computer you could use sudo. We can get around this by installing to our user account with the --user flag:
Can you write Python code for it?


I am working on the file "Extra Resources.ipynb".
The first task is:
Dependencies
So you've looked through the projects that are available and cloned one you think looks interesting and you want to start coding. But, you start up the notebook and this happens:
Can you write Python code for it?


I am working on the file "Extra Resources.ipynb".
The first task is:
Even more resources
Here are some other things that may be useful:

Miscellaneous Python resources:
pycrumbs: a wiki page populated with a massive number of useful links (e.g. if you wanted to program something for Google Glass, it's there).
pythonidae: an index of tools for scientific programming in Python for practically any task.
awesome-python: an index of Python tools for everything.

Dealing with Data in Python
Python is a popular language for Science and data science, and many of the most popular notebooks are data science using Python. Therefore, there are also some very good introductions to data science available as IPython notebooks.
The following cell will populate a repository python-data-science with some of the best resources for learning data science techniques in Python:

Learn Data Science
An Introduction to Pandas
Holoviews Tutorials
A Tutorial on Model Reliability
Statistics and data analysis in Python
Python-for-Data-Analysis
Simple Machine Learning with Scikit-Learn
Parallel Machine Learning with Scikit-Learn
Can you write Python code for it?

%%bash
mkdir python-data-science
cd python-data-science
git clone https://github.com/nborwankar/LearnDataScience.git Learn-Data-Science
curl https://raw.githubusercontent.com/mwaskom/Psych216/master/week6_tutorial.ipynb > A-Tutorial-on-Model-Reliability.ipynb
mkdir Holoviews-tutorials
git clone https://github.com/ioam/holoviews.git
mv holoviews/doc/Tutorials/* Holoviews-tutorials/
rm -rf holoviews
git clone https://bitbucket.org/hrojas/learn-pandas.git An-Introduction-To-Pandas
git clone https://github.com/amueller/tutorial_ml_gkbionics.git Simple-Machine-Learning-with-Scikit-Learn
git clone https://github.com/mwaskom/Psych216.git Statistics-and-Data-Analysis-in-Python
git clone https://github.com/ogrisel/parallel_ml_tutorial.git Paralel-Machine-Learning-with-Scikit-Learn
git clone https://github.com/ResearchComputing/Meetup-Fall-2013.git
mkdir Python-for-Data-Analysis
mv Meetup-Fall-2013/python/* Python-for-Data-Analysis/
rm -rf Meetup-Fall-2013


I am working on the file "Extra Resources.ipynb".
The first task is:
However, if we add the above path to Python's path, we will be able to import the package:
Can you write Python code for it?

sys.path.append("/home/gngdb/.local/lib/python3.4/site-packages/")

import pymc


I am working on the file "Extra Resources.ipynb".
The first task is:
So what's happened is we've tried to import the package PyMC, but we haven't installed it yet so we get an ImportError. Normally, if you were on you're own computer you could just run the following:
Can you write Python code for it?


I am working on the file "Extra Resources.ipynb".
The first task is:
Unfortunately, importing will still fail:
Can you write Python code for it?


I am working on the file "Extra Resources.ipynb".
The first task is:
When Python tries to install a package, it looks in a few places, and it turns out that when you install something with the --user flag it is installed to your user account at ~/.local/:
Can you write Python code for it?


I am working on the file "Extra_Material/Examples/Separate_Files/Using separate files.ipynb".
The first task is:
But the result of this may be that you get a very cluttered main directory. In order to prevent this, we can make a separate folder with all the scripts. I've put an example below. (Also look in the actual folder!)
IMPORTANT: there needs to be a file called __init__.py in the scripts folder. Without it, Python doesn't know it can import scripts from there.
Can you write Python code for it?

from scripts import adele

adele.hello()

from scripts.adele import hello

hello()


I am working on the file "Extract_GCprof_station_run-example.ipynb".
The first task is:
2. Run

Set the command line to be executed
Can you write Python code for it?

import sys

cmd = "{executable} {script} input.yaml --loglevel={loglevel} --nengines={ncpu}"

cmd = cmd.format(
    # path to executable (same python interpreter than the one used to run the notebook server)
    executable=sys.executable,
    # path to the script
    script=os.path.join(this_notebook_dir, 'run_scripts', 'extract_gcprof_station.py'),
    # number of CPU to use
    ncpu=4,
    # loglevel ('CRITICAL', 'ERROR', 'WARNING', 'INFO' or 'DEBUG')
    loglevel='INFO',
)

print("Command to execute: " + cmd)


I am working on the file "Extract_GCprof_station_run-example.ipynb".
The first task is:
3. Monitor

Check the status of the process
Can you write Python code for it?

import sys

try:
    if process.poll() is None:
        print("process is running")
        status = 'running'

    elif process.poll() == 0:
        print("process has terminated succesfully")
        status = 'success'

    else:
        sys.stderr.write("process has terminated with errors\n")
        status = 'error'

except NameError:
    print("no process is running! "
          "(or connection with the process loosed due "
          "to a kernel issue or kernel shutdown/restart)")
    status = None


I am working on the file "Extract_GCprof_station_run-example.ipynb".
The first task is:
4. Clean / Inspect outputs

Check if output files were created by listing them (with size, last modified date...)
Can you write Python code for it?


I am working on the file "Extract_GCprof_station_run-example.ipynb".
The first task is:
Display the log file
Can you write Python code for it?

if status is not None:
    %cat process.log


I am working on the file "Extract_GCprof_station_run-example.ipynb".
The first task is:
Execute the command in a new process in the background (only if no process is already running)
Can you write Python code for it?

import subprocess
import os
import sys
import shlex

# prevent running a new process if a process is already running.
try:
    if process.poll() is None:
        raise RuntimeError('A process is already running')
except NameError:
    pass

# split the command into a sequence
cmd = shlex.split(cmd)
# comment the line above and use the command string instead of sequence if shell is True
# http://stackoverflow.com/questions/16840427/python-on-linux-subprocess-popen-works-weird-with-shell-true

with open('process.log', 'w') as log:
    process = subprocess.Popen(cmd, shell=False, stdout=log, stderr=log)

print("New process started. PID: {}".format(process.pid))


I am working on the file "Extract_GCprof_station_run-example.ipynb".
The first task is:
If using the IPython parallel cluster, display ouputs (stdout, stderr) of all engines as they are printed out (debug)

open a terminal session in the server
activate the virtual environment
while the process is running, run the script iopubwatcher.py in the run_scripts directory

$ python iopubwatcher.py

(WARNING) Use the cell below to terminate the process if needed
Can you write Python code for it?

import signal

if status == 'running':
    process.send_signal(signal.SIGINT)


I am working on the file "Extract_GCprof_station_run-example.ipynb".
The first task is:
Set up the working directories.
Locate the directory of this notebook
Create a new directory for this run, which will have the name of this notebook and which will be located in \$HOME/IPYRuns (plus the same relative path from \$HOME/IPYNotebooks to this notebook).
'cd' into this run directory. The process will be run from within this directory. Unless the absolute path is specified explicitly, any file (input, output, log file) will be saved into this directory.
Can you write Python code for it?

import os

notebook_rel_path = os.path.dirname(notebook_rel_path)
notebook_rel_path = notebook_rel_path.replace('IPYNotebooks/', '')
this_notebook_dir = os.path.join(os.path.expanduser('~'), 'IPYNotebooks',
                                 notebook_rel_path)
notebook_runs_dir = os.path.join(os.path.expanduser('~'), 'IPYRuns')
this_run_dir = os.path.join(notebook_runs_dir, notebook_rel_path, notebook_name)

if not os.path.exists(this_run_dir):
    os.makedirs(this_run_dir)
os.chdir(this_run_dir)

print("run directory is: " + this_run_dir)