I am working on the file "Baseline.ipynb".
The first task is:
Cleaning and preparing the datasets
Can you write Python code for it?

X_finite = X_hr[np.isfinite(X_hr["X1"])]
X_finite = X_finite[np.isfinite(X_finite["X2"])]
X_finite = X_finite[np.isfinite(X_finite["X3"])]
X_finite = X_finite[np.isfinite(X_finite["X4"])]
X_finite = X_finite[np.isfinite(X_finite["Y"])]
X_finite.head()

Xn = X_finite
Xn.describe()

y = Xn["Y"]
X = Xn[["X1", "X2", "X3", "X4"]]

plt.figure(figsize=(9, 5))
plt.hist(y, bins=30)
plt.xlabel('Production Value',fontsize=15)
plt.ylabel('Occurences',fontsize=15)
plt.title('Distribution of the Rice Production Values',fontsize=18)

Xplot = Xn[["X1", "X2", "X3", "X4","Y"]]

var_name = "X1"
plt.figure(figsize=(10,6))
sns.regplot(x=var_name, y='Y', data=Xplot, scatter_kws={'alpha':0.6, 's':20})
plt.xlabel(var_name + " (Crop Produce of Last Year)", fontsize=15)
plt.ylabel('Y', fontsize=15)
plt.title("Distribution of y variable with feature "+var_name, fontsize=18)
plt.show()

var_name = "X2"
plt.figure(figsize=(10,6))
sns.regplot(x=var_name, y='Y', data=Xplot, scatter_kws={'alpha':0.6, 's':20})
plt.xlabel(var_name + " (Crop Produce of Last to Last Year)", fontsize=15)
plt.ylabel('Y', fontsize=15)
plt.title("Distribution of y variable with feature "+var_name, fontsize=18)
plt.show()

var_name = "X3"
plt.figure(figsize=(10,6))
sns.regplot(x=var_name, y='Y', data=Xplot, scatter_kws={'alpha':0.6, 's':20})
plt.xlabel(var_name + " (Rainfall of Present Year)", fontsize=15)
plt.ylabel('Y', fontsize=15)
plt.title("Distribution of y variable with feature "+var_name, fontsize=18)
plt.show()

var_name = "X4"
plt.figure(figsize=(10,6))
sns.regplot(x=var_name, y='Y', data=Xplot, scatter_kws={'alpha':0.6, 's':20})
plt.xlabel(var_name + " (Rainfall of Last Year)", fontsize=15)
plt.ylabel('Y', fontsize=15)
plt.title("Distribution of y variable with feature "+var_name, fontsize=18)
plt.show()

# Z-Score Normalization
cols = list(X.columns)

for col in cols:
    col_zscore = col + '_zscore'
    X[col_zscore] = (X[col] - X[col].mean())/X[col].std(ddof=0)
X = X[["X1_zscore", "X2_zscore", "X3_zscore", "X4_zscore"]]

X.head()


I am working on the file "Baseline.ipynb".
The first task is:
Constructing the features X1 and X2, the production for the last 2 years.
Can you write Python code for it?

l = rice.shape[0]
for row in range(0,l):
    if row-1<0 or rice.iloc[row,1] != rice.iloc[row-1,1]:
        continue
    else:
        rice.iloc[row,8] = rice.iloc[row-1,6]
        if row-2<0 or rice.iloc[row,1] != rice.iloc[row-2,1]:
            continue
        else:
            rice.iloc[row,9] = rice.iloc[row-2,6]

rice.head()

def func(s):
    x = s.strip()
    return x.lower()

rice['ind_district'] = rice['ind_district'].apply(func)
rice['Season'] = rice['Season'].apply(func)
rain1['ind_district'] = rain1['ind_district'].apply(func)
rain2['ind_district'] = rain2['ind_district'].apply(func)
rice.head()

rain1.head()

# can reduce the time by searching only one variable for some cases atleast
rice = rice[rice['Season'] == 'kharif']
l = rice.shape[0]

for row in range(0,l):

    dt = rice.iloc[row,1]
    yr = rice.iloc[row,2]

    if yr<=2002:

        # rainfall for the same year
        r = rain1.loc[(rain1.ind_district == dt) & (rain1.Year == yr)]
        if r.shape[0] == 1:
            rice.iloc[row,10] = r.iloc[0,3]

        # rainfall for the previous year
        r = rain1.loc[(rain1.ind_district == dt) & (rain1.Year == yr-1)]
        if r.shape[0] == 1:
            rice.iloc[row,11] = r.iloc[0,3]

    if yr>2004:

        # rainfall for the same year
        r = rain2.loc[(rain2.ind_district == dt) & (rain2.Year == yr)]
        if r.shape[0] == 1:
            rice.iloc[row,10] = r.iloc[0,3]

        # rainfall for the previous year
        r = rain2.loc[(rain2.ind_district == dt) & (rain2.Year == yr-1)]
        if r.shape[0] == 1:
            rice.iloc[row,11] = r.iloc[0,3]

# X1 = prod-1
# X2 = prod-2
# X3 = rain
# X4 = rain-1
rice.describe()

ricex = rice[np.isfinite(rice["Production"])]
ricex = ricex[np.isfinite(ricex["X1"])]
ricex = ricex[np.isfinite(ricex["X2"])]
ricex = ricex[np.isfinite(ricex["X3"])]
ricex = ricex[np.isfinite(ricex["X4"])]
ricex.head()

X = ricex[["X1","X2","X3","X4"]]
y =...


I am working on the file "Baseline.ipynb".
The first task is:
Good enough results for the Haryana State
Can you write Python code for it?


I am working on the file "Baseline.ipynb".
The first task is:
Importing the required datasets
Can you write Python code for it?

rice = pd.read_csv("/Users/macbook/Documents/BTP/Notebook/Rice.csv")
rice.head()

rice_haryana = rice[rice["State_Name"]=="Haryana"]
rice_haryana.head()

rainfall = pd.read_csv("/Users/macbook/Documents/BTP/Notebook/rainfall.csv")
rainfall.head()

rain_haryana = rainfall[rainfall["State"]=="Haryana"]
print(rain_haryana.head())
print(rain_haryana.describe())

X_hr = pd.read_csv("/Users/macbook/Documents/BTP/Notebook/haryana.csv")
X_hr.head()


I am working on the file "Baseline.ipynb".
The first task is:
Lets calculate the average RMSE (Cross Validation, 5-Fold)
Can you write Python code for it?

clf = LinearRegression()
scores = cross_val_score(clf, X, y, cv=5, scoring='neg_mean_squared_error')

for i in range(0,5):
    scores[i] = sqrt(-1*scores[i])

print(scores)
avg_rmse_phos = scores.mean()
print("\n\nAvg RMSE is ",scores.mean())

yt = yt/1000
yp = yp/1000
yt = yt.round(decimals=1)
yp = yp.round(decimals=1)

yo = np.concatenate((yp,yt),axis=1)

p = pd.DataFrame(data=yo,columns=['Predicted','Actual'])
p.describe()

p

rmse/1000


I am working on the file "Baseline.ipynb".
The first task is:
Result
This time the predictions are not as good as we got for just one state.
Can you write Python code for it?


I am working on the file "Baseline.ipynb".
The first task is:
The range of the values is (0,1362) and rmse is 70.6
We can say the results are decent but not good enough as we got with just one state.
Can you write Python code for it?


I am working on the file "Baselines.ipynb".
The first task is:
IMDB
A large movie review dataset with 50K full-length reviews Maas et al., 2011.
Can you write Python code for it?

imdb_df = pd.read_csv('/home/data/sentiment-analysis-and-text-classification/baselines-and-bigrams/aclImdb/data-frame.csv')

imdb_X_train = imdb_df.loc[:25000, 'review'].values
imdb_y_train = imdb_df.loc[:25000, 'sentiment'].values
imdb_X_test = imdb_df.loc[25000:, 'review'].values
imdb_y_test = imdb_df.loc[25000:, 'sentiment'].values


I am working on the file "Baselines.ipynb".
The first task is:
Multinomial Naive Bayes (MNB)
Can you write Python code for it?

import numpy as np
from sklearn import metrics
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

imdb_clf = Pipeline([
    ('vect', CountVectorizer(ngram_range=(1,2))),
    ('tfidf', TfidfTransformer()),
    ('clf', MultinomialNB())
])
_ = imdb_clf.fit(imdb_X_train, imdb_y_train)

predicted = imdb_clf.predict(imdb_X_test)
np.mean(predicted == imdb_y_test)

print(metrics.classification_report(y_test, predicted, target_names=['neg', 'pos']))
metrics.confusion_matrix(imdb_y_test, predicted)


I am working on the file "Baselines.ipynb".
The first task is:
NBSVM
There are several implementations of NBSVM available; for example:

Sida Wang's original implementation in Matlab.
A Python version by Grégoire Mesnil.
Daniel Pressel's version in Java.

I'll follow the elegant implementation in scikit-learn by Joshua Chin.
Can you write Python code for it?

from scipy.sparse import spmatrix, coo_matrix
from sklearn.base import BaseEstimator
from sklearn.linear_model.base import LinearClassifierMixin, SparseCoefMixin
from sklearn.svm import LinearSVC

class NBSVM(BaseEstimator, LinearClassifierMixin, SparseCoefMixin):

    def __init__(self, alpha=1, C=1, beta=0.25, fit_intercept=False):
        self.alpha = alpha
        self.C = C
        self.beta = beta
        self.fit_intercept = fit_intercept

    def fit(self, X, y):
        self.classes_ = np.unique(y)
        if len(self.classes_) == 2:
            coef_, intercept_ = self._fit_binary(X, y)
            self.coef_ = coef_
            self.intercept_ = intercept_
        else:
            coef_, intercept_ = zip(*[
                self._fit_binary(X, y == class_)
                for class_ in self.classes_
            ])
            self.coef_ = np.concatenate(coef_)
            self.intercept_ = np.array(intercept_).flatten()
        return self

    def _fit_binary(self, X, y):
        p = np.asarray(self.alpha + X[y == 1].sum(axis=0)).flatten()
        q = np.asarray(self.alpha + X[y == 0].sum(axis=0)).flatten()
        r = np.log(p/np.abs(p).sum()) - np.log(q/np.abs(q).sum())
        b = np.log((y == 1).sum()) - np.log((y == 0).sum())

        if isinstance(X, spmatrix):
            indices = np.arange(len(r))
            r_sparse = coo_matrix(
                (r, (indices, indices)),
                shape=(len(r), len(r))
            )
            X_scaled = X * r_sparse
        else:
            X_scaled = X * r

        lsvc = LinearSVC(
            C=self.C,
            fit_intercept=self.fit_intercept,
            max_iter=10000
        ).fit(X_scaled, y)

        mean_mag =  np.abs(lsvc.coef_).mean()
        coef_ = (1 - self.beta) * mean_mag * r + self.beta * (r * lsvc.coef_)
        intercept_ = (1 - self.beta) * mean_mag * b + self.beta * lsvc.intercept_

        return coef_, intercept_

imdb_nbsvm = Pipeline([
    ('vect', CountVectorizer(),
    ('clf', NBSVM()
])
_ =...


I am working on the file "Basemap-final.ipynb".
The first task is:
Add to the function to allow amplitude to be varied and aadd in an additional slider to vary both f and a

may want to limit ylim
Can you write Python code for it?


I am working on the file "Basemap-final.ipynb".
The first task is:
This is great but one cool enhancement would be to make the size of the point represent the magnitude of the earthquake.
Here's one way to do it:
Read the magnitudes into a list along with their respective lat and long
Loop through the list, plotting one point at a time
As the magnitudes start at 1.0, you can just use the magnitude directly as the scale factor
To get the marker size, multiply the magnitude by the smallest dot you want on the map.
Add an extra enhancement of colour:
make small earthquakes
See if you can get similar data, perhaps for Whale sightings, and plot those on a map.
You might even have some of your own data to plot..
Can you write Python code for it?


I am working on the file "Basemap.ipynb".
The first task is:
Create the figure
Can you write Python code for it?

fig = plt.figure(figsize=(8, 6))

# Create basemap, 870 km east-west, 659 km north-south,
# intermediate resolution, Transverse Mercator projection,
# centred around lon/lat 1°/58.5°
m = Basemap(width=870000, height=659000,
            resolution='i', projection='tmerc',
            lon_0=1, lat_0=58.5)

# Draw coast line
m.drawcoastlines(color='k')

# Draw continents and lakes
m.fillcontinents(lake_color='b', color='none')

# Draw a think border around the whole map
m.drawmapboundary(linewidth=3)

# Convert etopo1 coordinates lon/lat in ° to x/y in m
# (From the basemap help: Calling a Basemap class instance with the arguments
# lon, lat will convert lon/lat (in degrees) to x/y map projection coordinates
# (in meters).)
rlons, rlats = m(*np.meshgrid(lons,lats))

# Draw etopo1, first for land and then for the ocean, with different colormaps
llevels = np.arange(-500,2251,100) # check etopo.ravel().max()
lcs = m.contourf(rlons, rlats, etopo, llevels, cmap=cm.terrain)
olevels = np.arange(-3500,1,100) # check etopo.ravel().min()
cso = m.contourf(rlons, rlats, etopo, olevels, cmap=cm.ocean)

# Draw parallels and meridians
m.drawparallels(np.arange(-56,63.,2.), color='.2', labels=[1,0,0,0])
m.drawparallels(np.arange(-55,63.,2.), color='.2', labels=[0,0,0,0])
m.drawmeridians(np.arange(-6.,12.,2.), color='.2', labels=[0,0,0,1])
m.drawmeridians(np.arange(-7.,12.,2.), color='.2', labels=[0,0,0,0])

# Draw Block 9 boundaries
m.plot([1, 2, 2, 1, 1], [59, 59, 60, 60, 59], 'b', linewidth=2, latlon=True)
plt.annotate('9', m(1.1, 59.7), color='b')

# Draw maritime boundaries
m.readshapefile('data/basemap/DECC_OFF_Median_Line', 'medline', linewidth=2)

# Add Harding, Edinburgh, Bergen
# 1. Convert coordinates
EDIx, EDIy = m(-3.188889, 55.953056)
BERx, BERy = m(5.33, 60.389444)
HARx, HARy = m(1.5, 59.29)
# 2. Plot symbol
plt.plot(HARx, HARy, mfc='r', mec='k', marker='s', markersize=10)
plt.plot(EDIx, EDIy, mfc='r', mec='k', marker='o', markersize=10)
plt.plot(BERx, BERy, mfc='r', mec='k', marker='o', markersize=10)
# 3. Plot...


I am working on the file "Basemap.ipynb".
The first task is:
Load data
Can you write Python code for it?

# Load the topo file to get header information
etopo1name = 'data/basemap/etopo1_bedrock.asc'
topo_file = open(etopo1name, 'r')

# Read header (number of columns and rows, cell-size, and lower left coordinates)
ncols = int(topo_file.readline().split()[1])
nrows = int(topo_file.readline().split()[1])
xllcorner = float(topo_file.readline().split()[1])
yllcorner = float(topo_file.readline().split()[1])
cellsize = float(topo_file.readline().split()[1])
topo_file.close()

# Read in topography as a whole, disregarding first five rows (header)
etopo = np.loadtxt(etopo1name, skiprows=5)

# Data resolution is quite high. I decrease the data resolution
# to decrease the size of the final figure
dres = 2

# Swap the rows
etopo[:nrows+1, :] = etopo[nrows+1::-1, :]
etopo = etopo[::dres, ::dres]

# Create longitude and latitude vectors for etopo
lons = np.arange(xllcorner, xllcorner+cellsize*ncols, cellsize)[::dres]
lats = np.arange(yllcorner, yllcorner+cellsize*nrows, cellsize)[::dres]


I am working on the file "Bases_de_Python.ipynb".
The first task is:
L'encodage des caractères
Can you write Python code for it?

print("\N{GREEK CAPITAL LETTER DELTA}")
print("\u0394")


I am working on the file "Bases_de_Python.ipynb".
The first task is:
Les graphiques avec matplotlib
Affichage de courbes avec matplotlib.pyplot
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
def graphiques(f = lambda x : x**2):
    """tracés des courbes"""
    X=np.linspace(-2,1.5,100)#100 valeurs de -2 à 1.5 dans un ndarray de numpy
    Y=f(X)
    plt.plot(X,Y,'r-')#- -- ^ s -. : b g r c m y k w , linewidth=0.8, marker="+", label="Trajet 2"
    plt.ylabel('$C_f$')
    plt.title("f en fonction de x")
    plt.show()
graphiques()


I am working on the file "Bases_de_Python.ipynb".
The first task is:
Liste de nombres pseudo-aléatoires, mais périodiques... (principe des tiroirs de Dirichlet)
Can you write Python code for it?

import matplotlib.pyplot as plt
def pseudo_alea(n=100):
    """liste de nombres pseudo-aléatoires"""
    liste=[13]
    for I in range(n):
        liste.append((16805*liste[I] + 1) % 32768)
    plt.subplot(1,2,1)
    plt.plot(liste,'.')
    plt.title('nombres')
    pileFace=[bin(nb)[-1] for nb in liste]
    plt.subplot(1,2,2 )
    plt.plot(pileFace,'.')
    plt.title('bits de poids faibles')
    plt.show()
pseudo_alea()


I am working on the file "Bases_de_Python.ipynb".
The first task is:
Liste en compréhension avec des entrées.
Can you write Python code for it?

def si_en_ligne():
    """liste en compréhention"""
    return sum([float(input('jour '+str(I)+' : ')) for I in range(7)])/7
si_en_ligne()


I am working on the file "Bases_de_Python.ipynb".
The first task is:
Pour tester une égalité, l'astuce est de voir si les deux nombres sont \og vraiment \fg{} proches.
Can you write Python code for it?

from math import *
def egalite(a=0.3,b=0.1+0.1+0.1):
    """retourne le test d'égalité stricte et le test de proximité"""
    return (a==b, isclose(a,b))
egalite()


I am working on the file "Bases_de_Python.ipynb".
The first task is:
Récursivité
Can you write Python code for it?

def puissance_de_2(n=10):
    """retourne la puissance n-ième de 2"""
    return 1 if n==0 else 2*puissance_de_2(n-1)
puissance_de_2()

def PGCD(a,b):
    """retourne le PGCD à partir de l'algorithme d'Euclide"""
    return b if a%b==0 else PGCD(b,a%b)
PGCD(30,42)

def base_10_vers_n_recursif(nb=6007,n=5):
    """retourne l'écriture de nb en base n"""
    return str(nb) if nb//n==0 else base_10_vers_n_recursif(nb//n,n)+str(nb%n)
base_10_vers_n_recursif(1020,2)


I am working on the file "Bases_de_Python.ipynb".
The first task is:
avec les listes différence entre le tri en place a.sort() et sorted(a) qui produit une autre liste
Can you write Python code for it?

def deux_tris(liste=[2,1,8,6,5,7,5,6]):
    """illusrte le tri en place et le tri dans une autre liste"""
    print(sorted(liste))#renvoie une autre liste
    print(liste)
    liste.sort()#en place
    print(liste)
deux_tris()


I am working on the file "Bases_de_Python.ipynb".
The first task is:
histogramme et diagramme en boite
Can you write Python code for it?

from random import *
import matplotlib.pyplot as plt
def graphiques(n=1000):
    """simule n fois la somme de 5 dés"""
    X1=[randint(1,6)+randint(1,6)+randint(1,6)+randint(1,6)+randint(1,6) for I in range(n)]
    X2=[randint(1,6)+randint(1,6)+randint(1,6)+randint(1,6)+randint(1,6)for I in range(n)]

    plt.subplot(1,2,1)
    plt.boxplot([X1,X2])

    plt.subplot(1,2,2)
    plt.hist(X1,bins=[0,10,15,18,20,22,25,31],color=(1,0,0,1))
    plt.hist(X2,bins=[0,10,15,18,20,22,25,31],color=(0,0,1,0.5))
    plt.grid()

    plt.savefig('boite et histogramme.png')
    plt.show()
graphiques()


I am working on the file "Bash-Tutorial.ipynb".
The first task is:
Blocking on Results
We call analyze with the list of data futures as inputs. This will block until all the simulate runs have completed and the data futures have 'resolved'. Finally, we print the result when it is ready.
Can you write Python code for it?

results = analyze(inputs=simulation_outputs,
                  stdout='analyze.out',
                  stderr='analyze.err')
results.result()
with open('analyze.out', 'r') as f:
    print(f.read())


I am working on the file "Bash-Tutorial.ipynb".
The first task is:
Defining Bash Apps
To demonstrate how to run apps written as Bash scripts, we use two mock science applications: simulate.sh and stats.sh. The simulation.sh script serves as a trivial proxy for any more complex scientific simulation application. It generates and prints a set of one or more random integers in the range [0-2^62) as controlled by its command line arguments. The stats.sh script serves as a trivial model of an "analysis" program. It reads N files each containing M integers and prints the average of all those numbers to stdout. Like simulate.sh it logs environmental information to stderr.
The following cell show how apps can be composed from arbitrary Bash scripts. The simulate signature shows how variables can be passed to the Bash script (e.g., "sim_steps") as well as how standard Parsl parameters are managed (e.g., "stdout").
Can you write Python code for it?

@App('bash', dfk)
def simulate(sim_steps=1, sim_range=100, sim_values=5, outputs=[], stdout=None, stderr=None):
    # The bash app function requires that the bash script is returned from the function as a
    # string. Positional and Keyword args to the fn() are formatted into the cmd_line string
    # All arguments to the app function are made available at the time of string formatting a
    # string assigned to cmd_line.

    # Here we compose the command-line call to simulate.sh with keyword arguments to simulate()
    # and redirect stdout to the first file listed in the outputs list.
    return '''echo "sim_steps: {sim_steps}\nsim_range: {sim_range}\nsim_values: {sim_values}"
    echo "Starting run at $(date)"
    $PWD/bin/simulate.sh --timesteps {sim_steps} --range {sim_range} --nvalues {sim_values} > {outputs[0]}
    echo "Done at $(date)"
    ls
    '''


I am working on the file "Bash-Tutorial.ipynb".
The first task is:
Retrieving Results
Each of the Apps return one DataFuture. Here we put all of these (data futures of file outputs) together into a list (simulation_outputs). This is done by iterating over each of the AppFutures and taking the first and only DataFuture in it's outputs list.
Can you write Python code for it?

# Grab just the data futures for the output files from each simulation
simulation_outputs = [i.outputs[0] for i in simulated_results]


I am working on the file "Bash-Tutorial.ipynb".
The first task is:
Running Bash Apps
Now that we've defined an app, let's run 10 parallel instances of it using a for loop. Each run will write 100 random numbers, each between 0 and 99, to the output file.
In order to track files created by Bash apps, a list of data futures (one for each file in the outputs[] list) is made available as an attribute of the AppFuture returned upon calling the decorated app fn.
<App_Future> = App_Function(... , outputs=['x.txt', 'y.txt'...])
[<DataFuture> ... ] = <App_Future>.outputs
Can you write Python code for it?

simulated_results = []
# Launch 10 parallel runs of simulate() and put the futures in a list
for sim_index in range(10):
    sim_fut = simulate(sim_steps=1,
                       sim_range=100,
                       sim_values=100,
                       outputs = ['stdout.{0}.txt'.format(sim_index)],
                       stderr='stderr.{0}.txt'.format(sim_index))
    simulated_results.extend([sim_fut])


I am working on the file "Basic Algorithms.ipynb".
The first task is:
Convert a 10-integer phone number to every possible alphabetical combinations on a telephone keypad.
Do not replace number 0 and 1 with any alphabet.
Can you write Python code for it?

def phone_keypad(num):
    result = ['0', '1', 'ABC', 'DEF', 'GHI', 'JKL', 'MNO', 'PRS', 'TUV', 'WXY']
    return result[num]

def alphabets(nums):
    if len(nums) == 0:
        return set()

    retval = set()
    all_endings = alphabets(nums[1:])
    for char in phone_keypad(int(nums[0])):
        if len(all_endings) == 0:
            retval.add(char)
        for ending in all_endings:
            retval.add(char + ending)
    return retval

print str(alphabets('5106493114'))


I am working on the file "Basic Algorithms.ipynb".
The first task is:
Convert a base-10 RGB value in ([0-255], [0-255], [0-255]) to a hex code
Verify with http://www.rgbtohex.net/
Can you write Python code for it?

def hexcode(a,b,c):
    result = hex(a).upper()
    result += hex(b)[2:].upper()
    result += hex(c)[2:].upper()
    return result

print(hexcode(255, 32, 222))
print(hexcode(31, 156, 222))
print(hexcode(255, 255, 255))


I am working on the file "Basic Algorithms.ipynb".
The first task is:
Find the first non-repeating character in a string
Can you write Python code for it?

from collections import OrderedDict

def first_nonrepeating(string):
    d =  OrderedDict()
    for char in string:
        if char in d:
            d[char] += 1
        else:
            d[char] = 1
    while len(d) > 0:
        char, count = d.popitem(last=False)
        if count == 1:
            return char
    return None

print(first_nonrepeating('sterling cooper advertising agency'))


I am working on the file "Basic Algorithms.ipynb".
The first task is:
Print every 120 possible permutation of "A,B,C,D,E"
Can you write Python code for it?

def permuteA(s):
    all =[]
    permutation = []
    if(len(s) == 2):
        permutation=[s[0]+s[1],s[1]+s[0]]
        return permutation
    for i in range(0,len(s)):
        permutation=permuteA(s.translate(None,s[i]))
        for word in permutation:
            all.append(s[i] + word)
    return all

def permuteB(s):
    permutation=[]
    for c in s:
        words=permuteB(s.translate(None,c))
        if(len(words) ==0):
            return c
        for word in words:
            permutation.append(c + word)
    return permutation

def permuteCHelper(s,permutations):
    for c in s:
        words=permuteCHelper(s.translate(None,c),permutations)
        if(len(s) ==0):
            permutations.append()
            return ""
        for word in words:
            permutations.append(c + word)
            print c+word
    return permutations

def permuteC(s):
    permutations = []
    return permuteCHelper(s,permutations)

def main():
    word="ABCDE"
    permutations = permuteC(word)
    print permutations
    print len(permutations)

    permutations = permuteA(word)
    print permutations

if __name__ == '__main__':
    main()


I am working on the file "Basic Algorithms.ipynb".
The first task is:
Reverse a string
Can you write Python code for it?

def reverse_1(string):
    return string[::-1]

def reverse_2(string):
    rev = ''
    for i in range(len(string)):
        index = len(string) - i - 1
        rev += string[index]
    return rev

print reverse_1('sterling cooper advertising agency')
print reverse_2('sterling cooper advertising agency')


I am working on the file "Basic Algorithms.ipynb".
The first task is:
Sort all integers in the set in increasing order
Can you write Python code for it?


I am working on the file "Basic Array Review and Transformations.ipynb".
The first task is:
Going back to the source code to perform histogram equalization
Can you write Python code for it?

#

b = np.random.randint(0,10,(2,3,4))
print b

print b[0]
print b[1]
print "---------"
print b[0,0]
print b[0,1]

bb = b.reshape(-1)
print bb
print bb.sum()

plt.hist(bb)
plt.show()

bhistnorm = bb*1.0/sum(bb)
print bhistnorm

plt.hist(ahistnorm, bins = 10)
plt.show

## Histogram looks different than above due to the relatively large histogram sum number


I am working on the file "Basic Array Review and Transformations.ipynb".
The first task is:
Let's pretend Z is a very simple image
Can you write Python code for it?

import math

## Reminder of Z's values
print z

# Number of pixels
pixels = len(z) * len(z[0])
print "The number of pixels is {}".format(pixels)
## The output should be 4 since there are 4 numbers in this array

##Flatten method doesn't work because of tuple conversion later on
'''# Now we flatten Z
zz = z.reshape(-1)'''

# Initialize histogram and CDF
hist = {}
cdf = {}
norm_cdf = {}
## The range value should be be adjusted to the bin number
### In our case the range and bin number is obviously 10
### since we generated numbers 1-10
BINS = 10

for i in range(BINS):
    hist[i] = 0
    cdf[i] = 0
    norm_cdf[i] = 0

# Create histogram

for row in z:
    for val in row:
        hist[val] += 1

'''
for val in zz:
    hist[val] += 1
'''

# Create cdf
for i in range(BINS):
    for j in range(i+1):
        cdf[i] += hist[j]
    norm_cdf[i] = int(math.floor(float(cdf[i]-1)/63*BINS))

print "The histogram values are {}".format(hist)
print "The cdf values are {}".format(cdf)
print "The normalized cdf values are {}".format(norm_cdf)


I am working on the file "Basic Array Review and Transformations.ipynb".
The first task is:
This matches our constructed histogram so we can proceed to create a new histogram and construct the equalized one
Can you write Python code for it?

newimg = np.empty(z.shape)
## This should be the same exact dimensions as the original array
print newimg
print z.shape

## set x_length to the first number, y_length to the second
x_length = z.shape[0]
y_length = z.shape[1]

print x_length, y_length

for i in range(x_length):
    for j in range(y_length):
        newimg[i][j] = norm_cdf[ z[i][j] ]

print newimg


I am working on the file "Basic Array Review and Transformations.ipynb".
The first task is:
This probably didn't work because of the tiny size of the array, let's try something more established
Can you write Python code for it?

img = [
    [52, 55, 61,  66,  70,  61, 64, 73],
    [63, 59, 55,  90, 109,  85, 69, 72],
    [62, 59, 68, 113, 144, 104, 66, 73],
    [63, 58, 71, 122, 154, 106, 70, 69],
    [67, 61, 68, 104, 126,  88, 68, 70],
    [79, 65, 60,  70,  77,  68, 58, 75],
    [85, 71, 64,  59,  55,  61, 65, 83],
    [87, 79, 69,  68,  65,  76, 78, 94]
]

img = np.asarray(img)
print img

print " "

print img[0]
print img[1]

print " "

imgflat = img.reshape(-1)
print imgflat
print imgflat.sum()

print " "
fig = plt.hist(imgflat, bins='auto')
plt.title('Histogram')
plt.show()

print " "

imgnorm = imgflat*1.0/sum(imgflat)
print imgnorm

fig = plt.hist(imgnorm, bins = 'auto')
plt.title('Normalized Histogram')
plt.show

import math

## Reminder of Z's values
print img

# Number of pixels
pixels = len(img) * len(img[0])
print "The number of pixels is {}".format(pixels)
## The output should be 4 since there are 4 numbers in this array

##Flatten method doesn't work because of tuple conversion later on
'''# Now we flatten Z
zz = z.reshape(-1)'''

# Initialize histogram and CDF
hist = {}
cdf = {}
norm_cdf = {}
## The range value should be be adjusted to the bin number
### In our case the range and bin number is obviously 10
### since we generated numbers 1-10
BINS = 255

for i in range(BINS):
    hist[i] = 0
    cdf[i] = 0
    norm_cdf[i] = 0

# Create histogram

for row in img:
    for val in row:
        hist[val] += 1

'''
for val in zz:
    hist[val] += 1
'''

# Create cdf
for i in range(BINS):
    for j in range(i+1):
        cdf[i] += hist[j]
    norm_cdf[i] = int(math.floor(float(cdf[i]-1)/63*BINS))

print "The histogram values are {}".format(hist)
print "The cdf values are {}".format(cdf)
print "The normalized cdf values are {}".format(norm_cdf)

newimg = np.empty(img.shape)
## This should be the same exact dimensions as the original array
print newimg
print img.shape

## set x_length to the first number, y_length to the second
x_length = img.shape[0]
y_length = img.shape[1]

print x_length, y_length

for i in...


I am working on the file "Basic Linear Regression.ipynb".
The first task is:
But you really don't need to write that on your own as Scikit provides that for you already.
Can you write Python code for it?

x = np.random.randint(100, size=100000000)
y = x * 0.10

x = x[:,None]

now = time.time()

clf = SGDRegressor()
clf.fit(x, y)

later = time.time()
difference = int(later - now)
print("Time spent for SGDRegressor is %d seconds" % difference)
print("slope is %f and intercept is %s" % (clf.coef_, clf.intercept_[0]))

clf.predict(70) # How much tip


I am working on the file "Basic Linear Regression.ipynb".
The first task is:
Generate Data
We are going to generate a 1000 samples of random number between \$0 - \$100. And let's say that each meal the customer tips 10% of the amount
Can you write Python code for it?

total_bills = np.random.randint(100, size=1000)
tips = total_bills * 0.10


I am working on the file "Basic Linear Regression.ipynb".
The first task is:
Let's say if the customer spent $70 how much the customer will tip
Can you write Python code for it?

predicted_tips = (slope * 70) + intercept

print('The customer will leave the tip of $%f' % predicted_tips)


I am working on the file "Basic Linear Regression.ipynb".
The first task is:
Let's see after 1000 interations how close are we. Pretty close I think
Can you write Python code for it?

gradient_predicted_tips = (m * 70) + b
gradient_predicted_tips


I am working on the file "Basic Notes on Tensor Flow.ipynb".
The first task is:
Place Holder: 一般放实时进入的数据，以用来train model
Can you write Python code for it?

a = tf.placeholder(tf.float32)
b = tf.placeholder(tf.float32)
adder_node = a + b  # + provides a shortcut for tf.add(a, b)


I am working on the file "Basic Notes on Tensor Flow.ipynb".
The first task is:
Variable: 一般是要被train的东西，初始化后就等着被程序更新
Can you write Python code for it?

W = tf.Variable([.3], dtype=tf.float32)
b = tf.Variable([-.3], dtype=tf.float32)
x = tf.placeholder(tf.float32)
linear_model = W * x + b


I am working on the file "Basic Notes on Tensor Flow.ipynb".
The first task is:
add Node:
Can you write Python code for it?

node3 = tf.add(node1, node2)
node4 = a + b


I am working on the file "Basic Notes on Tensor Flow.ipynb".
The first task is:
对于定义的variable，需要进行初始化操作
Can you write Python code for it?

init = tf.global_variables_initializer()
sess.run(init)

print(sess.run(linear_model, {x: [1, 2, 3, 4]}))


I am working on the file "Basic Presentation.ipynb".
The first task is:
If we want to look at covariates, we need a new approach.
We'll use Cox proprtional hazards, a very popular regression model.
To fit in python we use the module lifelines:
http://lifelines.readthedocs.io/en/latest/
Can you write Python code for it?


I am working on the file "Basic Presentation.ipynb".
The first task is:
Once we've fit the data, we need to do something useful with it.  Try to do the following things:
1 - Plot the baseline survival function

2 - Predict the functions for a particular set of features

3 - Plot the survival function for two different set of features

4 - For your results in part 3 caculate how much more likely a death event is for one than the other for a given period of time
Can you write Python code for it?

#### Plot baseline hazard function

#### Predict

#### Plot survival functions for different covariates

#### Plot some odds


I am working on the file "Basic Presentation.ipynb".
The first task is:
The first step in any data analysis is acquiring and munging the data
Our starting data set can be found here:
   http://jakecoltman.com in the pyData post
It is designed to be roughly similar to the output from DCM's path to conversion
Download the file and transform it into something with the columns:
id,lifetime,age,male,event,search,brand
where lifetime is the total time that we observed someone not convert for and event should be 1 if we see a conversion and 0 if we don't.  Note that all values should be converted into ints
It is useful to note that end_date = datetime.datetime(2016, 5, 3, 20, 36, 8, 92165)
Can you write Python code for it?

####Data munging here

###Parametric Bayes
#Shout out to Cam Davidson-Pilon

## Example fully worked model using toy data
## Adapted from http://blog.yhat.com/posts/estimating-user-lifetimes-with-pymc.html
## Note that we've made some corrections

N = 2500

##Generate some random data
lifetime = pm.rweibull( 2, 5, size = N )
birth = pm.runiform(0, 10, N)
censor = ((birth + lifetime) >= 10)
lifetime_ = lifetime.copy()
lifetime_[censor] = 10 - birth[censor]

alpha = pm.Uniform('alpha', 0, 20)
beta = pm.Uniform('beta', 0, 20)

@pm.observed
def survival(value=lifetime_, alpha = alpha, beta = beta ):
    return sum( (1-censor)*(log( alpha/beta) + (alpha-1)*log(value/beta)) - (value/beta)**(alpha))

mcmc = pm.MCMC([alpha, beta, survival ] )
mcmc.sample(50000, 30000)

pm.Matplot.plot(mcmc)
mcmc.trace("alpha")[:]


I am working on the file "Basic Regression with neon.ipynb".
The first task is:
Cost function
How "close" is the model's prediction is to the true value? For the case of regression we'll just define the sum of the squared error between the model's prediction and the true value. Other types of models may require different cost functions.
Can you write Python code for it?

# Loss function is the squared difference
cost = GeneralizedCost(costfunc=SumSquared())


I am working on the file "Basic Regression with neon.ipynb".
The first task is:
Gradient descent
All of our models will use gradient descent. We will iteratively update the model weights and biases in order to minimize the cost of the model.
Can you write Python code for it?

optimizer = GradientDescentMomentum(0.1, momentum_coef=0.9)


I am working on the file "Basic Regression with neon.ipynb".
The first task is:
Initialize the weights and bias variables
We'll use numbers from the Gaussian distribution ($\mu=0, \sigma=1$) to initialize the weights and bias terms for our regression model.
Can you write Python code for it?


I am working on the file "Basic Regression with neon.ipynb".
The first task is:
Print the results
How close are we to the true line?
Play around with the noiseScale, m, and b parameters to convince yourself that neon is properly fitting the model.
Can you write Python code for it?

# print weights
slope = model.get_description(True)['model']['config']['layers'][0]['params']['W'][0][0]
print ("calculated slope = {:.3f}, true slope = {:.3f}".format(slope, m))

bias_weight = model.get_description(True)['model']['config']['layers'][1]['params']['W'][0][0]
print ("calculated bias = {:.3f}, true bias = {:.3f}".format(bias_weight, b))

plt.figure(figsize=(10,7))
plt.plot(X, slope*X+bias_weight, alpha=0.5, color='b', marker='^')
plt.scatter(X, y, alpha=0.7, color='g')
plt.plot(X, trueLine, '--', alpha=0.5, color='r')

plt.title('How close is our predicted model?', fontsize=18);
plt.grid('on');
plt.legend(['Predicted Line', 'True line', 'Raw Data'], fontsize=18);


I am working on the file "Basic Shapes.ipynb".
The first task is:
Another way to obtain the Klein bottle is by a quotient map of the torus.  For example, for the torus $[0, 1] \times [0, 1]$, make the identification $(x, y) \sim (x + 0.5, 1-y)$.
Exercise

Create a function below that computes the distance from one point set on the Klein bottle to another point set on the Klein bottle.
<b>Hint:</b> invoke your function to compute distances on the torus, then apply the quotient map
Can you write Python code for it?

def get_flat_klein_dists(x1, y1, x2, y2):
    """
    Compute all pairwise distances between all points (x1, y1) and points (x2, y2)
    on the flat torus [0, 1] x [0, 1]

    Parameters:
    x1 : ndarray (M)
        An M-length list of x coordinates of each point in the first point set
    y1 : ndarray (M)
        An M-length list of y coordinates of each point in the first point set
    x2 : ndarray (N)
        An N-length list of x coordinates of each point in the second point set
    y2 : ndarray (N)
        An N-length list of y coordinates of each point in the second point set

    Returns:
    D : ndarray (M, N)
        A distance matrix whose ijth entry is the distance along the flat Klein bottle
        between (x1[i], y1[i]) and (x2[j], y2[j]), where the flat Klein bottle is
        defined to be a quotient map over the torus $[0, 1] \times [0, 1]$
    """

    D = get_flat_torus_dists(x1, y1, x2, y2)
    # TODO: Finish this; apply quotients on the torus to turn
    # it into a Klein bottle

    return D

res = 15 #Number of points along each dimension
#Sample slightly differently so that the two persistence dots won't be the same
gridx = np.linspace(0, 1, res+1)[0:res]
gridy = gridx*0.99 #Only need to sample half in the y direction since there's a double cover
x, y = np.meshgrid(gridx, gridy)
x = x.flatten()
y = y.flatten()

# Compute the distance matrix, which is the distance between
# all points and themselves
D = get_flat_klein_dists(x, y, x, y)
np.fill_diagonal(D, 0)

plt.figure(figsize=(8, 4))
dgmsz2 = ripser(D, distance_matrix=True, maxdim=2)['dgms']
dgmsz3 = ripser(D, distance_matrix=True, maxdim=2, coeff=3)['dgms']
plt.subplot(121)
plot_diagrams(dgmsz2)
plt.title("$\mathbb{Z} / 2 \mathbb{Z}$")
plt.subplot(122)
plot_diagrams(dgmsz3)
plt.title("$\mathbb{Z} / 3 \mathbb{Z}$")
plt.show()


I am working on the file "Basic Shapes.ipynb".
The first task is:
Example 1: Circles
As a first example, we will make a circle.  Once you understand this example, starting from the code below, see if you can modify the point cloud or come up with a new point cloud that does the following
Exercises / Questions

Increase the birth time of the H1 dot without changing the radius of the circle
Decrease the birth time of the H1 dot without changing the radius of the circle
What happens to H1 when you add a point that's at the center of the circle at position $(0, 0)$?
Change the point cloud so that there are two 1D dots with large (>1) but different persistences
Can you write Python code for it?

# Create 100 points on the unit circle
N = 100
t = np.linspace(0, 2*np.pi, N+1)[0:N]
X = np.zeros((N, 2))
X[:, 0] = np.cos(t)
X[:, 1] = np.sin(t)

# Compute the persistence diagram of this point cloud
dgms = ripser(X)['dgms']

plt.figure(figsize=(8, 4))
plt.subplot(121)
plt.scatter(X[:, 0], X[:, 1])
plt.axis('equal')
plt.title("Point Cloud")
plt.subplot(122)
plot_diagrams(dgms)
plt.show()


I am working on the file "Basic Shapes.ipynb".
The first task is:
Example 2: Sphere And Projective Plane
We will now randomly sample some points on the two dimensional sphere and compute persistent homology up to dimension 2, using the intrinsic metric, which is the arc length between points.  A point $X = (x, y, z)$ is randomly drawn from a 3D gaussian distribution
$(x, y, z) \sim \left(\frac{1}{2 \pi}\right)^{3/2} \exp{\left( - (x^2+y^2+z^2)/2 \right) }$
and normalized to have a length of 1 (why does this evenly sample on the sphere?)
Note that the arc length along a unit circle is equal to the angle subtending that circle in radians. Thus, given two unit vectors on the sphere $x_1$ and $x_2$, the arc length between them is $\cos^{-1}(x_1 \cdot x_2)$.  We will implement this in code below to compute all pairwwise arc lengths between our sampled points on the sphere.  We will then compute a rips filtration up to dimension 2.
Questions

How many essential classes are in H1 and H2?  Does this match the homology of a sphere?
Can you write Python code for it?

# Step 1: Setup the point cloud
N = 200 #Number of points
X = np.random.randn(N, 3) #Draw from 3D Gaussian
X = X/np.sqrt(np.sum(X**2, 1))[:, None] #Normalize each point to unit length

# Step 2: Compute all pairwise arc lengths between sampled points
dotProds = X.dot(X.T) #Entry ij in this matrix holds the dot product between point i and j, or cos(theta)
#The dot products should be in [-1, 1], but may leave this range due to numerical roundoff
dotProds[dotProds < -1] = -1
dotProds[dotProds > 1] = 1
DSphere = np.arccos(dotProds) #The arc length is the inverse cosine of the dot products of unit vectors
np.fill_diagonal(DSphere, 0) #Be sure distance of points to themselves is zero

tic = time.time()
dgms = ripser(DSphere, distance_matrix=True, maxdim=2)['dgms']
print("Elapsed Time: ", time.time()-tic)
fig = plt.figure(figsize=(8, 4))
ax = plt.subplot(121, projection='3d')
ax.scatter(X[:, 0], X[:, 1], X[:, 2])
plt.title("Sphere Samples")
plt.subplot(122)
plot_diagrams(dgms)
plt.title("Persistence Diagrams")


I am working on the file "Basic control structures.ipynb".
The first task is:
"type" lets you examine what class an object is. Don't worry about that __main__ just for the moment.
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
...and get an alphabetical list of tasks. Note that sorted doesn't update the original object, so if you re-run without sorted, it's back to the previous order.
To do this, you use a slightly different technique or you just assign: things_to_do = sorted(things_to_do).
sorted guesses the ordering you want, based on the type of data. It doesn't all have to be strings, you could have mixture of strings and integers, say. sorted can take optional arguments, letting you force it to do things the way you want, or even provide your own ordering function.
So how do we get things out of a list?
Lists are ordered, so you can say, "I'd like the fifth item, please". You put it in brackets after the list name:
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
...but it's fixed, so you can't do this...
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
...but not...
Because it doesn't exist.
Extending lists and dicts
Extending lists is a little unwieldy, but dicts are more intuitive:
Can you write Python code for it?

things_to_do.append("Find a nice retirement village in the Galapagos Islands")
my_meetup_dot_com_profile["Interests"] = ["Python2", "Python3", "Scientific Python", "Pottery"]
print("TODO:", things_to_do, "\n\nMEETUP:", my_meetup_dot_com_profile)


I am working on the file "Basic control structures.ipynb".
The first task is:
...if you wonder why things_to_do[4] (the fifth item) didn't appear, note that that number after the colon means 'up-to-but-not-including'...
Can you write Python code for it?

things_to_do[:2]

things_to_do[3:]

things_to_do[-2:]


I am working on the file "Basic control structures.ipynb".
The first task is:
...in other words, give math an alias...
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
...we've seen that...
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
...which saves some typing if you're sure you don't want to use any other variable called e...
Pythonic
Walks like a Python...
This is a fundamental principle of Python and, unusually for something for fundamental, it isn't part of the code. Python is a bit like Ultimate Frisbee - in case you haven't had the pleasure, referees are not required, even in international competitions. This is on the basis that, if you're playing the sport, then you've decided you're there with the ethos, and the players can sort out those decisions amongst themselves and get on with the game. If you're trying to take advantage of that, then the fundamental question is, why play Ultimate Frisbee?
Good Python practice frequently diverges from what is commended in other languages. Succinctness or efficiency is not necessarily good - clarity comes before cleverness. However, using a pattern from previous experience when Python has a neater, more beautiful, more Pythonic way - that's something to try to avoid. The upshot of all this is that, when you come to what the community considers "good code", it is simple, elegant and clear.
Not-so-Pythonic way:
Can you write Python code for it?

for i in range(len(things_to_do)):
    print(things_to_do[i])


I am working on the file "Basic control structures.ipynb".
The first task is:
Aside: A hashable type can be used as an index in a dict - tuples are, lists aren't (because they can change, aka are mutable)
PS: an introduction to objects...
Objects
Totally class
If you haven't done object oriented coding before, I am afraid I am not going to give you the magic bullet now. It is an important concept to get to grips with, but for the moment, we will keep it fairly functional. Do take time to look through it later - Python is an excellent language for playing around with it and getting an intuition for what an object is. A number of you will be very familiar with this so please bear with me and the very rough introductory description I will give - for those who are not so familiar, this is worth getting the basics down early.
In real life, everything has properties, and many things have things they can do. In Python these properties are called attributes and the things an object can do are called methods. Together these are called members and, in Python, are accessed by putting a dot after the object and the attribute or method name. Everything else is just like any other variable or function, respectively.
For instance, my dog Freddie has properties - he has a colour, which is black (technically, he is imaginary, but in my head he is definitely black). In Python terms, this would be:
Attribute: freddie.colour = black
I can tell Freddie to roll-over. When I do so, I am, in some sense, calling Freddie's method:
Method: freddie.roll_over()
I have another dog, Nitwit. Nitwit can also roll over...
Method: nitwit.roll_over()
Both dogs can shake hands, but I need to tell them which paw...
Method: freddie.shake_paw('left')
At this point, you're probably wondering what exactly is the set of attributes and methods that my dogs have? This template, showing what attributes and methods a dog of mine can be expected to have, is called a class.
A simple class
Can you write Python code for it?

class PhilsDog:
    name = ""
    colour = ""
    def shake_paw(self, side):
        print("My name is", self.name, "and I am shaking my", side, "paw like a good dog")


I am working on the file "Basic control structures.ipynb".
The first task is:
But double-check Etherpad...
Remember the post-its? So, everybody stick your star post-it somewhere I can see it...
Your task is to write a Jupyter cell to give you the $x^{th}$ digit of $pi$ and write the digit at the start of that Etherpad line, the one with your name on it. In particular, try and do this, just using x, without typing the number in directly. Then you can experiment to see if other numbers work too.
There are several ways to do this... I've given you enough tools above to find at least one. If, when you get it working, no-one has described your approach on Etherpad, write a short description at the bottom of the notepad.
When done, swap your star for an arrow.
Can you write Python code for it?

# Overwrite this cell with your code and run it


I am working on the file "Basic control structures.ipynb".
The first task is:
Emm...we tried to make a directory with the same name as a key system file and are blithely assuming that every failure in doing so is simply because it is an existing directory. Not so good.
Can you write Python code for it?

import errno

dirname = "/etc/passwd"

try:
    os.makedirs(dirname)
except OSError as e:
    if e.errno != errno.EEXIST:
        raise
    # Great, someone has already created that directory
    # We can carry on!
# lalalalala...
print("just mucking about with my friend", dirname)


I am working on the file "Basic control structures.ipynb".
The first task is:
Essentially, for sees what we see - a sequence of five numbers.
Bear in mind that you can cast like this to various types. str will make something a string...
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
First note pass. This is required because every block must have at least one non-comment line - if nothing else is there, we can use pass - it is Python's no-op if that helps.
Now ask yourself, "did we ever actually define dirname?" No? Then as soon as we use it after our supposed check, we will get an unhandled NameError exception.
We can specify what type of exceptions we catch... dir-already-exists have type OSError.
Can you write Python code for it?

try:
    os.makedirs(dirname)
except OSError:
    # Great, someone has already created that directory
    # We can carry on!
    pass
# lalalalala...
print("just mucking about with my friend dirname")


I am working on the file "Basic control structures.ipynb".
The first task is:
Good dogs. Here we see the shake_paw method being called for each dog and with a different side parameter. To remind you, the body of the method was:
python
    def shake_paw(self, side):
        print("My name is", self.name, "and I am shaking my", side, "paw like a good dog")
You can see how the self.name matches the name I gave each dog on the previous slide.
If that was familiar to you, then that was probably a very unexciting few minutes - if not, it's probably come and gone very quickly. Thankfully, we only need to know that, if we have an object, we can get its attributes and methods by adding a dot and the attribute/method name.
Now why is that important...
In Python
Everything is an object (nearly)
Can you write Python code for it?

"just a normal string".upper()

(3 + 2j).imag

"Another string".islower()


I am working on the file "Basic control structures.ipynb".
The first task is:
Here we create a class called PhilsDog that all of my dogs implement (that is, they are of that type). It indicates that they will have a colour and that they have a method called shake_paw. Think of this from the perspective of the dog - the first argument, self, is a little Python magic that refers to the dog itself. This lets the dog use it's name and colour (and any other methods) in the shake_paw method. The second argument is which paw I told my dog to shake. In response, any dog of mine says "Shaking left/right paw like a good dog". That's quite impressive, but I'd rather they just learned to zip it and actually shake their paws instead.
Now, that's just the template for one of my dogs, so how do I use it?
Can you write Python code for it?

freddie = PhilsDog()
freddie.name = "Freddie"
freddie.colour = "black"

nitwit = PhilsDog()
nitwit.name = "Nitwit"
nitwit.colour = "brown"

print("Freddie is", freddie.colour, "while Nitwit is", nitwit.colour)


I am working on the file "Basic control structures.ipynb".
The first task is:
Here we have added one or two minor surprises. One is the horrendously misleading use of len (hint: you are unlikely to get a Nobel Prize by 2020), which in reality calculates the number of items in any "iterable". This is the general term for something sort-of-list-like. According to my script, tasks take as long as the number of items in their "list", i.e. characters in the string.
Another aspect is the double slash. This is an important difference between Python3 and Python2 (and many other languages). Dividing non-divisible integers with one slash, as normal, gives a float in Python3. If you want to get an integer (that is, whatever the float answer is with the decimal chopped off), you can use double-slash. Try removing the second slash and re-running to see the exact non-integer answer. In Python2, using one slash always gives an integer, unless you cast the bottom or top to a float (as in C, say).
Modules
Gotta catch em all
Introduction to modules

Where Python gets its power

Wealth of modules providing everything from full GUI toolkits to astrophysics simulations. Mature and reliable numerics libraries, and an evolving ecosystem that grows day by day

Some modules come with Python, but you can install many more

Python has tools so you can just specify the name and it will get it from the appropriate online repository. On Windows, this can be tricky, but a recent Python distribution, Anaconda, makes it easy, and includes thousands of packages out of the box. This seems your best bet if on Windows or Mac and I have USB sticks here for you to get going at the end of the day.

Some are part of Python, lots are third-party

It is good to have an idea of which is which, as you can get (normally free) help, often at pretty short notice, by heading to the project's forums - remember, always be polite and, bear in mind, it isn't a commercial service, so an answer isn't guaranteed. For reference, when someone is really desperate they can offer a bounty for a solution or a bit of code on certain websites, not...
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
Hopefully, you should notice that x is indeed what you input. This should all be pretty familiar, unless you come from statically-typed languages (C/C++/Java) where you would have to declare a variable. Python lets you effectively declare it, and its type, by assigning to it.
We can refer to this output as Out[2], using it like any variable, without having to re-run everything so far.
Do the same trick in the next In [ ] (edit and Ctrl+Return)
Can you write Python code for it?

if Out[2] / 2 == x:
    print("Thankfully, maths still works")


I am working on the file "Basic control structures.ipynb".
The first task is:
However, note that...
PYTHON IS ZERO-INDEXED
Like C, but unlike MATLAB or FORTRAN. The first item in a list is "things_to_do[0]"
By the way, did you notice that In[] and Out[] are lists?
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
I call the class like a function. This creates a new PhilsDog object - in computing terminology, I instantiate the class, creating a new instance of PhilsDog. Freddie is an instance of PhilsDog and so is Nitwit. But PhilsDog is just a template - as you can see, both Freddie and Nitwit have their own colour and name. I can update these just as any variable, and Freddie's doesn't affect Nitwit's, and I can read both back out again.
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
If you had errors that don't make sense, try the chat window, try someone beside you (my normal approach in life) or ask me. While you are doing that, I am going to wrap up...
What about all the other structures??
Well, they are not so hard once you get this far:
but you'll come across them throughout today

Loops can be while [SOME BOOLEAN]:, as well as for

We will explore some actual modules after lunch

Generators put loops inside lists

I'll let that boggle your mind for a moment. Our future sessions are not going to be so closely led, partially because it cannot be as interesting experiment with my code as it is getting your hands dirty with your own. Now you've got plenty of tools at your disposal.
Now, I'm going to do a quick tour of scripting applications of Python, a few ideas you might want to ask me about over lunch, to give you an idea of what Python can do, so sit back relax and let me fiddle with projector technology at the front for a couple of minutes...
Finally...
After a whistle-stop tour, we pull into a Python siding...

Please write one great thing about this morning on your arrow and leave it beside the door
Please write one dire (or just not so good) thing about this morning on your star and leave it beside the door

PS: Tuples
The Ice-List
Tuples are basically frozen lists...
Can you write Python code for it?

target_coordinates = (56, -5)
print(target_coordinates[0], "N", target_coordinates[1], "E")


I am working on the file "Basic control structures.ipynb".
The first task is:
It has functions and variables...
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
Lists and dicts
Conveyor belts and storage units
Like any language, often need to be able to group things in Python.
We can create a list, which is like a one-dimensional C or FORTRAN array, as follows (don't forget to execute it!):
Can you write Python code for it?

things_to_do = ['Learn Python', 'Finish PhD', 'Publish research',
                'Accept Nobel prize', 'Inspire a new generation']


I am working on the file "Basic control structures.ipynb".
The first task is:
Lots of pretty colours! What do they mean?
In general, the most immediately useful information is on the very last line: SyntaxError. Above it, is the last "frame" called - that is, level of function. Say you call a function, and it calls another, and so forth until the innermost function hits an error. Here, it will list all the functions in that sequence, so you can track exactly how you got to the dodgy line.
In Jupyter, the filename is not very informative - but you can see the input index (ipython-input-4 ⇆ In [4]).
Bonus mark if you spotted the error (not that we actually have marks, but hey) - have a look and see if you can fix it and re-run the command, the same way you ran it the first time. Feel free to discuss, use Etherpad chat, whichever. Note that the In [4] changes to In [5], and so forth, each time you run that "cell".
In general, "a string" and 'a string' are equivalent - Python does not mind whether single or double quotes are used, as long as they match at either end.
[PTW: keep slide on screen to view syntax error]

Check the error at the bottom
Check the caret (^) showing where Python sees a problem
Look at the syntax highlighting in the cell

Functions
Painting the Black Box
Like every language, Python has functions... they are defined like so:
Can you write Python code for it?

def tell_me_if_im_old(name, age):
    print("Is", name, "old?")
    if age < 30:
        return "No"
    else:
        return "No"


I am working on the file "Basic control structures.ipynb".
The first task is:
Note that # is the comment symbol in Python - everything after it is ignored.
This just set the variable x (globally for this notebook) to 1
Click to edit the next In [ ] and press Ctrl+Return (without changing anything this time)
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
Note that anything can be an item in a dictionary or list - in this case, a list is an item in our dict. Also note that, as with other languages, \n signifies a newline.
Join
The most useful method in the str object
Different languages have different ways of collapsing a string with a separator - it is something you will need to do again and again. In Python, the approach is a little surprising - we have already seen that strings are objects (of type str), but it turns out Python uses this to provide a way of joining iterables (like lists or dicts):
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
Now indent everything beneath the def line by four spaces and run it. You have just defined a function! If so, this should give you a 1:
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
Now we get the same exception we would have got the last time, but the raise keyword has kept it moving on past. This is quite useful, as Python exceptions tend to have lots of juicy info we wouldn't want to lose by ending our except block with a boring print('oh noes') and program exit.
Better yet
So, at the moment, we catch every possible exception. This is probably a very bad idea, as, usually when we catch an exception, it is because we expect a particular thing to go wrong. As a case in point:
Can you write Python code for it?

try:
    os.makedirs(dirname)
except:
    # Great, someone has already created that directory
    # We can carry on!
    pass
# lalalalala...
print("just mucking about with my friend dirname")


I am working on the file "Basic control structures.ipynb".
The first task is:
Same output, but one feels more like pseudocode, more like how you would verbally convey the idea if somebody asked.
If you want to get what Python is about, try this (literally):
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
So it doesn't work with a list - try it now with a tuple - swap the square brackets on the second line for parens () and re-run
Can you write Python code for it?

print("At", coordinates, "we have a", battleships[coordinates])


I am working on the file "Basic control structures.ipynb".
The first task is:
The last command tells your script to exit, with code 1. I don't really want to try that right now, as whatever happens won't be good.
There are four key variants of importing
You will see all four, so check back here
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
There are a couple of things here to dissect. Many languages need braces {}, brackets [] or parentheses () to draw boundaries around the argument of the if statement, or the body. In Python, symbols are reduced in favour of words and spaces, for readability. To achieve the same effect, Python separates the condition from the body with the colon : and the indentation (four spaces, by convention)
Indentation is Python's Marmite for other programmers. As they are keen to point out, most whitespace does not matter: e.g. I could write
Out[2] / 2           ==    x  :
and it's the same as
Out[2] / 2 = x:
. However, indentation (whitespace at the start of a line) does. It is an extremely visually clear way to show where blocks of code, like the body of this if statement, start and end. For scientists, it forces a very good programming practice - basic code style - your code doesn't run without if it isn't visually clear (well... neatly indented). One of the reasons I got into Python was that my PhD C++ code was so unreadable, dodgily indented and unweildy, it was easier to rewrite in Python than try and extend it - once I had modification was easy, I could see how everything was laid out, where functions started and ended, before even reading a character.
Another thing to note is that you can print with print. Here we are running Python3, the latest version. If you've played with Python2, you will notice this syntax is a bit different - it has parentheses (aka. parens) around the argument. A whole raft of changes came in with Python3 and, like XP, Python2 will be around for some time to come through sheer momentum. However by 2016, almost all important libraries are now Python3-ready and I, for one, develop new software in Python3 only, where possible. Good porting tools exist for moving old code.
To summarize those key points:

Indentation separates bodies of functions, conditionals, loops from the outside code
Basic boolean operators are ==, or, and, not, >, <, etc.
This is Python3 syntax (very slightly...
Can you write Python code for it?

if (x < 5) or (x > 100):
    print('Do I know you?')


I am working on the file "Basic control structures.ipynb".
The first task is:
This brings in a whole set of tools for dealing with basic mathematics. (so execute it!)
We reach into this with a dot...
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
This emphasises the fact that subclasses of Exception often have additional contextually-relevant properties, which you should use. It also points out that if statements still have their place in error-handling!
When raise has no argument, it re-raises whichever exception it was that got us into this mess.
A few important Exceptions
That you might want to catch

KeyError - box_of_tricks["not here"]
TypeError - 1 + "banana"
IOError - open('/etc/passwd', 'w')

Exceptions are often more pythonic than pre-checking. This Python2 doc is a good start on that road. CRUCIAL READING!
We are near the end of the walk-through
Feel free to go back and forward through this. I am going to give you one final task to get you started on this topic...
Remember your line number from Etherpad? If not, it should be in variable x...
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
This is completely unsorted. However, you can get any element back by its name (key):
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
This is in fact using a submodule, os.path, we can reach in twice to get functions inside that.
Also sys...
Can you write Python code for it?

import sys
print(sys.path)
# sys.exit(1)  # Exit with error code 1


I am working on the file "Basic control structures.ipynb".
The first task is:
This is one of the hardest things to get when switching to Python, particularly from compiled languages, but it is one of Python's greatest rewards. Nothing much beats opening up a script you hacked together two years ago to see zen-like transparency - no confusion, beautifully professional, and thanks to Python's design, no extra time spent writing it, just a feel for what Pythonic is.
A couple of resources...

If there is one link from this course to click: What is Pythonic

PEP (Python Enhancement Proposal) 8 - code style

These are publicly accessible design documents and nearly everything new of importance passes through here. PEP8 is, in fact, a style guide; unlike many languages, there is more or less a definitive recommended style, and this spells it out. Since I discovered flake8, I have my IDE set to highlight style errors and it has reaped dividends to the moon and back - I strongly recommend you do so too, especially if we ever have to code together.
Exceptions
Are more than ordinary
In the words of one of early computing's most idiosyncratic legends, Rear Admiral Dr Grace Murray Hopper, "It is often easier to ask for forgiveness than to ask for permission". A distinguishing feature of Python is that it is built from the ground up with this maxim in mind, known as EAFP. In practice, this means preferring exceptions over tests, so using a try-except block (or, elsewhere, try-catch, etc.) instead of an if statement when checking whether you can perform an action.
Simple examples where you are more likely to use an exception in Python than another language:

When making a directory

Can you see why this might be useful? If you use an if to check for non-existence, there is a potential race condition: by the time you reach the body, it could be made. If you catch the exception, you know it existed exactly when you tried to make it. (Although, the recommended routine, os.makedirs, has an optional don't-complain-if-dir-exists argument, which is probably even better)

When testing file...
Can you write Python code for it?

try:
    something_stupid()
except:
    print('Doh!')


I am working on the file "Basic control structures.ipynb".
The first task is:
This is the most succint as it just dumps everything in the math module into the global namespace, which saves much typing, but it can make debugging a headache, especially if there are some strange functions or variables in there that happen to have the same name as something else you later decide to use (this does happen from time to time)
Can you write Python code for it?


I am working on the file "Basic control structures.ipynb".
The first task is:
Well, the first thing is that we didn't define something_stupid. So we get an exception and our extremely generic except provides no useful information.
An improvement
Can you write Python code for it?

try:
    something_stupid()
except Exception as e:
    print('Doh! You forgot that', e)


I am working on the file "Basic control structures.ipynb".
The first task is:
Yeah. Great.
Useful tool just slotted in there: multi-line strings. If you start a string with three quotes, you can keep going on and on until you hit another three.
One final example to show how, for this particular case, you really can be a little more specific.
Can you write Python code for it?

dirname = "/etc/passwd"
try:
    os.makedirs(dirname)
except OSError:
    # Great, someone has already created that directory
    # We can carry on!
    pass
# lalalalala...
print("just mucking about with my friend", dirname)


I am working on the file "Basic control structures.ipynb".
The first task is:
Yes, yes it can: check out the 'for c in "YMCA"'. We also snuck in a nested loop - you just keep indenting each time you nest - no more complicated than that.
And, admittedly, a new operator has appeared: "+=". Familiar to most languages it is equivalent to d = d + blah. Note that we therefore have to have d defined before the first +=, as "d = d + anything" doesn't make sense if d does not already exist. This is the reason for our first line: "d = """.
Finally, note that we have looped twice using range, just as above, but our loop variable i is never used - it is effectively a placeholder. All we want from that line is to have the body below run twice.
Sidenote: some people use an underscore "_" as a loop variable in this case, to indicate to someone reading that the variable is nothing more than that and never used - as far as Python is concerned, "_" is as good a name for a variable as any other, so this is entirely about readability.
Just to prove that "for" will iterate over any list, numeric, string or otherwise, we can try it with our todo list:
Can you write Python code for it?

total_months = 0
for task in things_to_do:
    total_months += len(task)
    print(task, ":")
    print ("  using Python, this task will take", len(task), "months to complete")
print("You cannot retire for at least", total_months // 12, "years")


I am working on the file "Basic control structures.ipynb".
The first task is:
You can try len as well!
Slicing
Sounds cool, is cool
Suppose I want a bit of a list, like the middle 3 items, or the first 2, or last 2...
Can you write Python code for it?


I am working on the file "Basic-Usage.ipynb".
The first task is:
From above, it is possible to inspect basic properties such as whether or not the ImageController's image is running in local or Spark mode, its type and shape. (3, 2048, 2048) indicates that our image is 2048 x 2048 with 3 channels. Let's look at some of the other information housed by the ImageController:
Can you write Python code for it?

imcontroller.channels
imcontroller
print('Numerical Labels that index image: ')
print(imcontroller.image_obj.labels)
print('Channels: ')
print(imcontroller.channels)
print('Image types: ' + str(imcontroller.feat_data.keys()))
print ('Features: ')
print(imcontroller.feat_data['IF'].keys())
print('ER features: ')
print(imcontroller.feat_data['IF']['intensity']['ER'].keys())
xy = imcontroller.feat_data['IF']['xy']
x = xy[:,0]
a = ((xy[:, 0] > 100) & (xy[:, 1] < 400))
print sum(a)
# print np.where([xy[:, 0] > 100], [xy[:, 1] < 300])
# print xy[:,0]
b = [True, False, True]

def generate_patches():
    num_patches = 5
    patches = np.zeros((num_patches, 3, 256, 256))
    for i in xrange(num_patches):
        a = np.ones((3, 256, 256))
        patches[i] = a

    return patches, np.ones(num_patches)

patches = []
labels = []

for i in xrange(5):
    pats, labs =generate_patches()
    patches.append(pats)
    labels.append(labs)

print np.vstack(patches).shape
print np.hstack(labels).shape

# X = np.vstack(zip(*patches_labels)[0])

# a = np.ones((3, 256, 256))
# b = np.ones((3, 256, 256))
# aug_patches = []
# aug_patches2 = []

# aug_patches.append(a)
# aug_patches.append(b)

# print type(aug_patches)

# aug_patches2.append(a)
# aug_patches2.append(b)

# aug_patches = np.stack(aug_patches, axis=0)
# aug_patches2 = np.stack(aug_patches2, axis=0)
# print aug_patches.shape
# print aug_patches2.shape

# c = []
# c.append((aug_patches,1))
# c.append((aug_patches2,2))
# cz = zip(*c)
# print np.vstack(cz[0]).shape

# d = [(1, 3),(2, 4)]

# e =  zip(*d)
# print e[0]
# f = np.stack(e[0], axis=0)
# print f
# print f.shape


I am working on the file "Basic-Usage.ipynb".
The first task is:
Let's see the ImageController in action:
Since we don't have a database up and running, we will pass the ImageController the information it needs manually. As soon as the database is set up, the Provider will make queries to the database and provide individual ImageControllers the information they need.
Loading images and creating ImageControl
Can you write Python code for it?

imcontroller = ImageController(demo.image_info)
demo.image_info.items()


I am working on the file "Basic-Usage.ipynb".
The first task is:
Patches:
Can you write Python code for it?

patch_shape = (3, 256, 256)
imcontroller.patchify(patch_shape=patch_shape, overlap=64)
print imcontroller.patches.shape

patches = imcontroller.patches[0].reshape((-1,) + patch_shape)
print patches.shape
print np.stack(patches, axis=0).shape
class_num = 2
labels = np.ones(60) * int(class_num)
labels2 = np.ones(60) * int(class_num)
l = [labels, labels2]
print np.hstack(l)
print

plt.imshow(imcontroller.patches[0,1,0,0,:, :])
# plt.show()

plt.imshow(patches[8, 0,:, :])
# plt.show()

for k in xrange(64):
    for i in xrange(8):
        for j in xrange(8):
            a = sum(sum(patches[k,0,:, :] == imcontroller.patches[0,i,j,0,:, :]))
            if a > 60000:
                print (i, j, k)
                print'(%d, %d, %d)' % ((k / imcontroller.patches.shape[1]), (k % imcontroller.patches.shape[2]), k)
                print 'found match'
print 'done'
p1 = patches
p2 = patches
p3 = patches
p = [p1, p2, p3]

# flash.show_subset_patches(imcontroller.patches, (8,8), 0)

imcontroller.patchify(patch_shape=(512, 512, 3),overlap=200)
flash.show_subset_patches(imcontroller.patches, (4,4), 0)


I am working on the file "Basic-Usage.ipynb".
The first task is:
Showing a pseudo-colored image:
Can you write Python code for it?


I am working on the file "Basic/BinaryTree.ipynb".
The first task is:
C++
``` c++
struct TreeNode
{
    int val;
    TreeNode left;
    TreeNode right;
    TreeNode(int x) : val(x), left(NULL), right(NULL){}
};
```
Tree traversal
Python
Can you write Python code for it?

class TreeNode:
    def __init__(self, val):
        self.val = val
        self.left, self.right = None, None

class Traversal(object):
    def __init__(self):
        self.traverse_path = list()

    def preorder(self, root):
        if root:
            self.traverse_path.append(root.val)
            self.preorder(root.left)
            self.preorder(root.right)

    def inorder(self, root):
        if root:
            self.inorder(root.left)
            self.traverse_path.append(root.val)
            self.inorder(root.right)

    def postorder(self, root):
        if root:
            self.postorder(root.left)
            self.postorder(root.right)
            self.traverse_path.append(root.val)

if __name__ == '__main__':
    root = TreeNode(1)
    root.left = TreeNode(2)
    root.left.left = TreeNode(3)
    root.left.right = TreeNode(4)
    root.right = TreeNode(5)
    root.right.left = TreeNode(6)

    traver_pre = Traversal()
    traver_pre.preorder(root)
    print traver_pre.traverse_path

    traver_in = Traversal()
    traver_in.inorder(root)
    print traver_in.traverse_path

    traver_pos = Traversal()
    traver_pos.postorder(root)
    print traver_pos.traverse_path


I am working on the file "Basic/LinkedList.ipynb".
The first task is:
C++
``` c++
inlcude <iostream>
struct ListNode
{
    int val;
    ListNode next;
    ListNode(int val, ListNode next=NULL):val(val), next(next){};
};
ListNode Reverseist(ListNode head)
{
    ListNode pre = NULL, tmp;
    while(head)
    {
        tmp = head->next;
        head->next = pre;
        pre = head;
        head = tmp;
    }
    return pre;
}
```
Dual Linked List
Python
Can you write Python code for it?

class DListNode:
    def __init__(self, val):
        self.val = val
        self.prev = self.next = None

    def reverse(self, head):
        curt = None
        while head:
            curt = head
            head = curt.next
            curt.next = curt.prev
            curt.prev = head
        return curt


I am working on the file "BasicLinearRegression.ipynb".
The first task is:
Bivariate distribution
If we look now at the distribution of the two variables in a scatter plot. We can see that our singular countries for the two variables of study result in outlier points. Nothing special was done about them as there are no managment error on them and no reason to discard them.
Moreover, the distribution of the data along the regression line shows that the homoscedasticity hypothesis is not well met as the spread of the data along the line is bigger for higher values of income.
Can you write Python code for it?

sns.regplot(x='income', y='electricity', data=subdata2)
plt.xlabel('Income per person (2000 US$)')
plt.ylabel('Residential electricity consumption (kWh)')
plt.title('Scatterplot for the association between the income and the residential electricity consumption');


I am working on the file "BasicLinearRegression.ipynb".
The first task is:
From the distribution graph, we can see that the distribution is skewed-right and bimodal. There is also a singular case with high income per capita. That country is :
Can you write Python code for it?

subdata2.loc[subdata2['income'] > 45000]

subdata2['income'].describe()


I am working on the file "BasicLinearRegression.ipynb".
The first task is:
General information on the Gapminder data
Can you write Python code for it?

display(Markdown("Number of countries: {}".format(len(data))))
display(Markdown("Number of variables: {}".format(len(data.columns))))


I am working on the file "BasicLinearRegression.ipynb".
The first task is:
Income per person
Can you write Python code for it?

sns.distplot(subdata2['income'], kde=False)
plt.xlabel('Income per person (2000 US$)');


I am working on the file "BasicLinearRegression.ipynb".
The first task is:
The residential electricity consumption is also skewed-right. And there are also a couple of countries presenting unusual higher values. Those countries are :
Can you write Python code for it?


I am working on the file "BasicLinearRegression.ipynb".
The first task is:
Variables distribution
Before computing the linear regression between the icome per person and the residential electricity consumption, let's have a look at the distributions of the variables.
Can you write Python code for it?

subdata2 = (data[['incomeperperson', 'relectricperperson']]
                .assign(income=lambda x: pd.to_numeric(data['incomeperperson'], errors='coerce'),
                        electricity=lambda x: pd.to_numeric(data['relectricperperson'], errors='coerce'))
                .dropna())


I am working on the file "BasicMathReview.ipynb".
The first task is:
1.1.1 Elementwise Operations
Can you write Python code for it?


I am working on the file "BasicMathReview.ipynb".
The first task is:
1.1.3 Hadamard product
This is element wise multiplication which results to another vector.
Can you write Python code for it?


I am working on the file "BasicMathReview.ipynb".
The first task is:
2. Matrices
Matrix is a rectangular array of scalars. Primarily, an n × m matrix A is used to describe a linear
transformation from m to n dimensions, where the matrix is an operator. We describe the dimensions of a matrix in terms of rows by columns
\begin{split}\begin{bmatrix}
2 & 4 \
5 & -7 \
12 & 5 \
\end{bmatrix}
\begin{bmatrix}
a² & 2a & 8\
18 & 7a-4 & 10\
\end{bmatrix}\end{split}
The first has dimensions (3,2). The second (2,3).
Can you write Python code for it?


I am working on the file "BasicMathReview.ipynb".
The first task is:
2.2 Elementwise operations
In order to add, subtract, or divide two matrices they must have equal dimensions. We combine corresponding values in an elementwise fashion to produce a new matrix.
\begin{split}\begin{bmatrix}
a & b \
c & d \
\end{bmatrix}
+
\begin{bmatrix}
1 & 2\
3 & 4 \
\end{bmatrix}
=
\begin{bmatrix}
a+1 & b+2\
c+3 & d+4 \
\end{bmatrix}\end{split}
Can you write Python code for it?


I am working on the file "BasicMathReview.ipynb".
The first task is:
2.4 Matrix transpose
Neural networks frequently process weights and inputs of different sizes where the dimensions do not meet the requirements of matrix multiplication. Matrix transpose provides a way to “rotate” one of the matrices so that the operation complies with multiplication requirements and can continue. There are two steps to transpose a matrix:

Rotate the matrix right 90°
Reverse the order of elements in each row (e.g. [a b c] becomes [c b a])

As an example, transpose matrix M into T:
\begin{split}\begin{bmatrix}
a & b \
c & d \
e & f \
\end{bmatrix}
\quad \Rightarrow \quad
\begin{bmatrix}
a & c & e \
b & d & f \
\end{bmatrix}\end{split}
3. Calculus
You need to know some basic calculus in order to understand how functions change over time (derivatives), and to calculate the total amount of a quantity that accumulates over a time period (integrals). The language of calculus will allow you to speak precisely about the properties of functions and better understand their behaviour.
3.1 Derivertives
it is an instantaneous rate of change
Can you write Python code for it?

def get_derivative(func, x):
    """Compute the derivative of `func` at the location `x`."""
    h = 0.0001                          # step size
    return (func(x+h) - func(x)) / h    # rise-over-run

def f(x): return x**2                   # some test function f(x)=x^2
x = 3                                   # the location of interest
computed = get_derivative(f, x)
actual = 2*x

computed, actual   # = 6.0001, 6        # pretty close if you ask me...


I am working on the file "BasicSentenceMarkup.ipynb".
The first task is:
Apply modifiers
We now call the applyModifiers method of the ConTextMarkup object to identify any relationships between the nodes.
Can you write Python code for it?


I am working on the file "BasicSentenceMarkup.ipynb".
The first task is:
Clean the text
Prior to processing we do some basic cleaning of the text, sucha s replacing multiple white spaces with a single space. You'll notice this in the spacing between the colon and "no" in the raw and clean versions of the text.
Can you write Python code for it?


I am working on the file "BasicSentenceMarkup.ipynb".
The first task is:
Identify concepts in the sentence
The markItems method takes a list of itemData and uses the regular expressions to identify any instances of the itemData in the sentence. With the mode keyword we specify whether these itemData are targets or modifiers. This value will be stored as a data attribute of the node that is created in the graph for any identified concepts.
Can you write Python code for it?

markup.markItems(modifiers, mode="modifier")
print(markup.nodes(data=True))
print(type(markup.nodes()[0]))


I am working on the file "BasicSentenceMarkup.ipynb".
The first task is:
We're going to start with our simplest of sentences
Can you write Python code for it?


I am working on the file "BasicSentenceMarkup.ipynb".
The first task is:
What does our initial markup look like?

We've identified one concept in the sentence: no
We've created a tagObject for this concept which keeps track of the actual phrase identified by the regular expression, what the category of the itemData was (definite_negated_existence), this is a list because there can be multiple categories. There is also an absurdly long identifier for the node. Note that our mode modifier has been stored as a data element of the node. In NetworkX each node (or edge) has a dictionary for data.

Now let's markup the targets
Can you write Python code for it?

markup.markItems(targets, mode="target")

print(markup.nodes(data=True))


I am working on the file "BasicSentenceMarkup.ipynb".
The first task is:
What does our markup look like now?
We've added another node to the graph. This time the target pneumothorax.
Prune Marks
After identifying concepts, we prune concepts that are a subset of another identified concept. This results in no changes here, but the importance will be shown later with a different sentence.
Can you write Python code for it?


I am working on the file "BasicSentenceMarkup.ipynb".
The first task is:
marking up a sentence
We start by creating an instance of the ConTextMarkup class. This is a subclass of a NetworkX DiGraph. Information will be stored in the nodes and edges.
Can you write Python code for it?

markup = pyConText.ConTextMarkup()

isinstance(markup,nx.DiGraph)

#### Set the text to be processed

markup.setRawText(reports[3].lower())
print(markup)
print(len(markup.getRawText()))


I am working on the file "BasicSentenceMarkup.ipynb".
The first task is:
pyConTextGraph contains the bulk of the pyConTextNLP functionality, including basic class definitions such as the ConTextMarkup class that represents the markup of a sentence.
itemData contains a class definition for an itemData and functions for reading itemData definitions which are assumed to be in a tab seperated file that is specified as either a local file or a remote resource. In this example we will read definitions straight from the GitHub repository.
An itemData in its most basic form is a four-tuple consisting of
A literal (e.g. "pulmonary embolism", "no definite evidence of")
A category (e.g. "CRITICAL_FINDING", "PROBABLE_EXISTENCE")
A regular expression that defines how to identify the literal concept. If no regular expression is specified, a regular expression will be built directly from the literal by wrapping it with word boundaries (e.g. r"""\bpulmonary embolism\b""")
A rule that defines how the concept works in the sentence (e.g. a negation term that looks forward in the sentence).

Sentences
These example reports are taken from (with modification) the MIMIC2 demo data set that is a publically available database of de-identified medical records for deceased individuals.
Can you write Python code for it?

reports = [
    """IMPRESSION: Evaluation limited by lack of IV contrast; however, no evidence of
      bowel obstruction or mass identified within the abdomen or pelvis. Non-specific interstitial opacities and bronchiectasis seen at the right
     base, suggestive of post-inflammatory changes.""",
    """IMPRESSION: Evidence of early pulmonary vascular congestion and interstitial edema. Probable scarring at the medial aspect of the right lung base, with no
     definite consolidation."""
    ,
    """IMPRESSION:

     1.  2.0 cm cyst of the right renal lower pole.  Otherwise, normal appearance
     of the right kidney with patent vasculature and no sonographic evidence of
     renal artery stenosis.
     2.  Surgically absent left kidney.""",
    """IMPRESSION:  No pneumothorax.""",
    """IMPRESSION: No definite pneumothorax"""
    """IMPRESSION:  New opacity at the left lower lobe consistent with pneumonia."""
]


I am working on the file "BasicSentenceMarkupPart2.ipynb".
The first task is:
Identify concepts in the sentence
Can you write Python code for it?

markup.markItems(modifiers, mode="modifier")
markup.markItems(targets, mode="target")
print(markup.nodes(data=True))


I am working on the file "BasicSentenceMarkupPart2.ipynb".
The first task is:
Read the itemData definitions
Can you write Python code for it?

modifiers = itemData.instantiateFromCSVtoitemData(
    "https://raw.githubusercontent.com/chapmanbe/pyConTextNLP/master/KB/lexical_kb_05042016.tsv")
targets = itemData.instantiateFromCSVtoitemData(
    "https://raw.githubusercontent.com/chapmanbe/pyConTextNLP/master/KB/utah_crit.tsv")


I am working on the file "BasicSentenceMarkupPart2.ipynb".
The first task is:
Sentences
These example reports are taken from (with modification) the MIMIC2 demo data set that is a publically available database of de-identified medical records for deceased individuals.
Can you write Python code for it?

reports = [
    """IMPRESSION: Evaluation limited by lack of IV contrast; however, no evidence of
      bowel obstruction or mass identified within the abdomen or pelvis. Non-specific interstitial opacities and bronchiectasis seen at the right
     base, suggestive of post-inflammatory changes.""",
    """IMPRESSION: Evidence of early pulmonary vascular congestion and interstitial edema. Probable scarring at the medial aspect of the right lung base, with no
     definite consolidation."""
    ,
    """IMPRESSION:

     1.  2.0 cm cyst of the right renal lower pole.  Otherwise, normal appearance
     of the right kidney with patent vasculature and no sonographic evidence of
     renal artery stenosis.
     2.  Surgically absent left kidney.""",
    """IMPRESSION:  No pneumothorax.""",
    """IMPRESSION: No definite pneumothorax""",
    """IMPRESSION:  New opacity at the left lower lobe consistent with pneumonia."""
]


I am working on the file "BasicSentenceMarkupPart2.ipynb".
The first task is:
We're going to start with our simplest of sentences
Can you write Python code for it?


I am working on the file "BasicSentenceMarkupPart2.ipynb".
The first task is:
What does our initial markup look like?

We've identified three concepts in the sentence:
"no"
"no definite"
"pneumothorax"

Here "no" is not a true concept in the sentence; it is a subset of the concept "no definite"

Prune Marks
After identifying concepts, we prune concepts that are a subset of another identified concept.
Can you write Python code for it?


I am working on the file "BasicSentenceMarkupPart2.ipynb".
The first task is:
What is the effect of pruneMarks
We've correctly dropped no as an identified concept.
Apply modifiers
We now call the applyModifiers method of the ConTextMarkup object to identify any relationships between the nodes.
Can you write Python code for it?


I am working on the file "BasicStats2.ipynb".
The first task is:
Did you expect that answer for the pdf?  Why?  What would the pdf be if you changed the width to 4?
Gaussian Distribution
We have already seen that the Gaussian distribution is given by
$$p(x|\mu,\sigma) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(\frac{-(x-\mu)^2}{2\sigma^2}\right).$$
It is also called the normal distribution and can be noted by $\mathscr{N}(\mu,\sigma)$.
Can you write Python code for it?

# Execute this cell
%run code/fig_gaussian_distribution.py

# Complete and execute this cell
from scipy import stats
dist = # Normal distribution with mean = 0, stdev = 1
r = # 10 random draws
p = # pdf evaluated at x=0
print p,r

# Uncomment the next line and run
# I just want you to know that this magic function exists.
#%load code/fig_gaussian_distribution.py


I am working on the file "BasicStats2.ipynb".
The first task is:
In addition to the "average", we'd like to know something about deviations from the average.  The simplest thing to compute is $$d_i = x_i - \mu.$$  However, the average deviation is zero by definition of the mean.  The next simplest thing to do is to compute the mean absolute deviation:
$$\frac{1}{N}\sum|x_i-\mu|,$$
but the absolute values can hide the true scatter of the distribution in some cases.  So the next simplest thing to do is to square the differences $$\sigma^2 = \frac{1}{N}\sum(x_i-\mu)^2,$$ which we call the variance.
Indeed the variance is just expectation value of $(x-\mu)^2$
$$\sigma^2 = V = \int_{-\infty}^{\infty}  (x-\mu)^2 h(x) dx,$$
where, again,  the integral gets replaced by a sum for discrete distributions.
Can you write Python code for it?


I am working on the file "BasicStats2.ipynb".
The first task is:
Log Normal
Note that if $x$ is Gaussian distributed with $\mathscr{N}(\mu,\sigma)$, then $y=\exp(x)$ will have a log-normal distribution, where the mean of y is $\exp(\mu + \sigma^2/2)$.  Try it.
Can you write Python code for it?

# Execute this cell
x = stats.norm(0,1) # mean = 0, stdev = 1
y = np.exp(x)
print y.mean()
print x


I am working on the file "BasicStats2.ipynb".
The first task is:
Now instead of averaging 2 draws, average 3.  Then do it for 10.  Then for 100.  Each time for 1,000,000 samples.
Make sure that you adjust the "expected" Gaussian based on the number of draws.
Can you write Python code for it?

# Copy your code from above and edit accordingly (or just edit your code from above)


I am working on the file "BasicStats2.ipynb".
The first task is:
Now let's average those two draws and plot the result (in the same panel).  Do it as a histogram for 1,000,000 samples (of 2 each).  Use a stepfilled histogram that is normalized with 50% transparency and 100 bins.
Can you write Python code for it?

# Copy your code from above

# Add a histogram that is the 2-sample mean of 1,000,000 draws

yy = # Complete

plt.hist(yy, #Complete


I am working on the file "BasicStats2.ipynb".
The first task is:
Sample vs. Population Statistics
Statistics estimated from the data are called sample statistics as compared to population statistics which come from knowing the functional form of the pdf.  Up to now we have been computing population statistics.
Specifically, $\mu$ is the population average, i.e., it is the expecation value of $x$ for $h(x)$.  But we don't know $h(x)$.  So the sample mean, $\overline{x}$, is an estimator of $\mu$, defined as
$$\overline{x} \equiv \frac{1}{N}\sum_{i=1}^N x_i,$$
which we determine from the data itself.
Then instead of $\sigma^2$, which is the population variance, we have the sample variance, $s^2$, where
$$s^2 = \frac{1}{N-1}\sum_{i=1}^N(x_i-\overline{x})^2$$
Where it is $N-1$ instead of $N$ since we had to determine $\overline{x}$ from the data instead of using a known $\mu$.  Ideally one tries to work in a regime where $N$ is large enough that we can be lazy and ignore this.
So the mean and variance of a distribution are $\mu$ and $\sigma^2$.  The estimators of the distribution are $\overline{x}$ (or $\hat{x}$) and $s^2$.
Bias
If there is a difference between the estimator and the population values, we say that the estimator is biased (perhaps not quite the usage of the word that you are used to).  Again, more on this later.
Uncertainty
We would also like to know the uncertainty of our estimates $\overline{x}$ and $s$.  Note that $s$ is NOT the uncertainty of $\overline{x}$.  Rather the uncertainty of $\overline{x}$, $\sigma_{\overline{x}}$ is
$$ \sigma_{\overline{x}} = \frac{s}{\sqrt{N}},$$
which we call the standard error of the mean.
The uncertainty of $s$ itself is
$$\sigma_s = \frac{s}{\sqrt{2(N-1)}} = \frac{1}{\sqrt{2}}\sqrt{\frac{N}{N-1}}\sigma_{\overline{x}}.$$
Note that for large $N$, $\sigma_{\overline{x}} \sim \sqrt{2}\sigma_s$ and for small $N$, $\sigma_s$ is not much smaller than $s$.
Distributions
If we are attempting to characterize our data in a way that is parameterized, then we need a functional form or a distribution.  There are many naturally occurring...
Can you write Python code for it?

# Execute this cell
%matplotlib inline
%run code/fig_uniform_distribution.py


I am working on the file "BasicStats2.ipynb".
The first task is:
Student's $t$ Distribution
Another distribution that we'll see later is the Student's $t$ Distribution.
If you have a sample of $N$ measurements, ${x_i}$, drawn from a Gaussian distribution, $\mathscr{N}(\mu,\sigma)$, and you apply the transform
$$t = \frac{\overline{x}-\mu}{s/\sqrt{N}},$$
then $t$ will be distributed according to Student's $t$ with the following pdf (for $k$ degrees of freedom):
$$p(x|k) = \frac{\Gamma(\frac{k+1}{2})}{\sqrt{\pi k} \Gamma(\frac{k}{2})} \left(1+\frac{x^2}{k}\right)^{-\frac{k+1}{2}}$$
As with a Gaussian, Student's $t$ is bell shaped, but has "heavier" tails.
Can you write Python code for it?

# Execute this cell
%run code/fig_student_t_distribution.py


I am working on the file "BasicStats2.ipynb".
The first task is:
The arithmetic mean (or Expectation value) is
$$\mu = E(x) = \int_{-\infty}^{\infty} x h(x) dx,$$
where $h(x)$ must be properly normalized and the integral gets replaced by a sum for discrete distributions.
Specifically, this is the expecation value of $x$.  If you want the expectation value of something else--say $x^2$ or $(x-\mu)^2$, you replace $x$ with that.
Can you write Python code for it?

# Execute this cell
mean = np.mean(data)
print mean


I am working on the file "BasicStats2.ipynb".
The first task is:
The catch here is that stats.norm(0,1) returns an object and not something that we can just do math on in the expected manner.  What can you do with it?  Try dir(x) to get a list of all the methods and properties.
Can you write Python code for it?

# Complete and execute this cell
dist = stats.norm(0,1) # mean = 0, stdev = 1
x = # Complete
y = # Complete
print x.mean(),np.log(y.mean())


I am working on the file "BasicStats2.ipynb".
The first task is:
The mode is the most probable value, determined from the peak of the distribution, which is the value where the derivative is 0:
$$ \left(\frac{dh(x)}{dx}\right)_{x_m} = 0$$
Can you write Python code for it?

# Execute this cell
mode = scipy.stats.mode(data)


I am working on the file "BasicStats2.ipynb".
The first task is:
We can implement uniform in scipy as follows.  Use the methods listed at the bottom of the link to complete the cell.
Can you write Python code for it?

# Complete and execute this cell
from scipy import stats
dist = # Complete for left edge = 0, width = 2
r = dist # Complete for 10 random draws
print r
p = dist # Complete for pdf evaluated at x=1
print p