I am working on the file "Paper_Implementation.ipynb".
The first task is:
We swap the test data to the beginning of the transition probability matrix.
Can you write Python code for it?

def front_swap(M, inds, dim = 1):
    M_new = np.copy(M)
    swap_inds = [np.arange(inds.shape[0]) for i in range(dim)]
    orig_inds = [inds for i in range(dim)]
    temp = M_new[swap_inds]
    M_new[swap_inds] = M_new[orig_inds]
    M_new[orig_inds] = temp
    return M_new

P_new = front_swap(P, ts, dim = 2)
P_new.shape


I am working on the file "Paper_Implementation.ipynb".
The first task is:
We then calculate the transition probability from a user $u_i$ to any other user $u_k$.
Can you write Python code for it?

def p(u_i, R = Rnan, P_uo = P_uo):
    """ Transition probability from user u_i to each user. """
    return np.nansum(P_uo[u_i] * P_ou(u_i, R = R), axis = 1)
p(0)


I am working on the file "Paradosis.ipynb".
The first task is:
After this segmentation, our corpus contains 103 consecutive, equal-sized samples:
Can you write Python code for it?


I am working on the file "Paradosis.ipynb".
The first task is:
Here, a threefold clustering seems to emerge, distinghuising between three main groups:
- green samples (begin and and of the text?)
- the curious red part with roughly samples 10-120 (cf. PCA results)
- the remainder of the text in blue, with a divergent clade for samples 65-72 (in purple)
A slightly fancier visualization method is to run seaborn's clustermap on our distance matrix, which runs largely parallel:
Can you write Python code for it?

from pystyl.visualization import clustermap
clustermap(corpus, distance_matrix=dm, fontsize=4, color_leafs=True,\
           outputfile='/Users/mike/Desktop/cm.pdf',
          show=True, save=True, return_svg=False)


I am working on the file "Paradosis.ipynb".
The first task is:
On this distance matrix, we can run a standard clustering algorithm with Ward's linkage method:
Can you write Python code for it?

from pystyl.analysis import hierarchical_clustering
cluster_tree = hierarchical_clustering(dm, linkage='ward')


I am working on the file "Paradosis.ipynb".
The first task is:
The corpus is empty right now:
Can you write Python code for it?


I am working on the file "Paradosis.ipynb".
The first task is:
The previous block gives some errors, but yields the correct image in the end, which seems to show a similar structure. Finally, we can now apply the VNC algorithm, which will cluster the data while taking into account (and leave unchanged) the original order of the samples in the text. The setup is again similar:
Can you write Python code for it?

from pystyl.analysis import vnc_clustering
vnc_tree = vnc_clustering(dm, linkage='ward')
scipy_dendrogram(corpus=corpus, tree=vnc_tree,
                 outputfile='~/Desktop/vnc.pdf',\
                 fontsize=5, save=True, return_svg=False)


I am working on the file "Paradosis.ipynb".
The first task is:
This step is more crucial: we segment our corpus (a single text in this case) into consecutive windows or slices of length 2000 (cf. segment_size, expressed in words). The step_size controls to which extent these samples are allowed to overlap:
Can you write Python code for it?


I am working on the file "Paradosis.ipynb".
The first task is:
We are now ready to vectorize our corpus, meaning that we will represent as a two-dimensional matrix, with a row for each sample, and a column for each stylistic feature (in this case word frequencies). With the following settings, we extract information word unigram features (ngram_size, ngram_type) for the 30 most frequent items (MFI) in the samples. We scale the original relative frequencies in our matrix, using their standard deviation. This sort of scaling is common in stylometry (cf. Burrows's Delta). The parametrization min_df=0.50 indicates that a feature should be present in at least 50% of the samples analyzed.
Can you write Python code for it?

mfi = corpus.vectorize(mfi=30,
                       ngram_type='word',
                       ngram_size=1,
                       vector_space='tf_std',
                       min_df=0.50)


I am working on the file "Paradosis.ipynb".
The first task is:
We can add our single text under the data directory using a single command:
Can you write Python code for it?


I am working on the file "Paradosis.ipynb".
The first task is:
We can now turn to some exploratory analysis. PCA is a common dimension reduction method in stylometry. It is especially useful because it allows us to plot word loadings. We first import the PCA functionality and apply it to our corpus, asking to retain only the top-2 components:
Can you write Python code for it?

from pystyl.analysis import pca
pca_coor, pca_loadings = pca(corpus, nb_dimensions=2)


I am working on the file "Paradosis.ipynb".
The first task is:
We can now visualize and export the resulting dendrogram, as we did with the PCA scatterplot:
Can you write Python code for it?

from pystyl.visualization import scipy_dendrogram
scipy_dendrogram(corpus=corpus, tree=cluster_tree,
                 outputfile='~/Desktop/scipy_dendrogram.pdf',
                 fontsize=5, save=True, return_svg=False)


I am working on the file "Paradosis.ipynb".
The first task is:
We can now visualize the samples in the two PCs:
Can you write Python code for it?

from pystyl.visualization import scatterplot
scatterplot(corpus, coor=pca_coor,
            nb_clusters=3, loadings=pca_loadings,
            save=True, outputfile="/Users/mike/Desktop/pca.pdf")


I am working on the file "Paradosis.ipynb".
The first task is:
We tokenize the corpus (the default is to split along whitespace):
Can you write Python code for it?


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
A LoadBalancedView is an interface to the engines that provides dynamic load
balancing at the expense of not knowing which engine will execute the code.
Can you write Python code for it?


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Assemble the result into a structured NumPy array.
Can you write Python code for it?

prices = np.empty(n_strikes*n_sigmas,
    dtype=[('ecall',float),('eput',float),('acall',float),('aput',float)]
)

for i, price in enumerate(results):
    prices[i] = tuple(price)

prices.shape = (n_strikes, n_sigmas)


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Here are the basic parameters for our computation.
Can you write Python code for it?

price = 100.0  # Initial price
rate = 0.05  # Interest rate
days = 260  # Days to expiration
paths = 10000  # Number of MC paths
n_strikes = 6  # Number of strike values
min_strike = 90.0  # Min strike price
max_strike = 110.0  # Max strike price
n_sigmas = 5  # Number of volatility values
min_sigma = 0.1  # Min volatility
max_sigma = 0.4  # Max volatility

strike_vals = np.linspace(min_strike, max_strike, n_strikes)
sigma_vals = np.linspace(min_sigma, max_sigma, n_sigmas)

print "Strike prices: ", strike_vals
print "Volatilities: ", sigma_vals


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Monte-Carlo option pricing function
The following function computes the price of a single option.  It returns the call and put prices for both European and Asian style options.
Can you write Python code for it?

def price_option(S=100.0, K=100.0, sigma=0.25, r=0.05, days=260, paths=10000):
    """
    Price European and Asian options using a Monte Carlo method.

    Parameters
    ----------
    S : float
        The initial price of the stock.
    K : float
        The strike price of the option.
    sigma : float
        The volatility of the stock.
    r : float
        The risk free interest rate.
    days : int
        The number of days until the option expires.
    paths : int
        The number of Monte Carlo paths used to price the option.

    Returns
    -------
    A tuple of (E. call, E. put, A. call, A. put) option prices.
    """
    import numpy as np
    from math import exp,sqrt

    h = 1.0/days
    const1 = exp((r-0.5*sigma**2)*h)
    const2 = sigma*sqrt(h)
    stock_price = S*np.ones(paths, dtype='float64')
    stock_price_sum = np.zeros(paths, dtype='float64')
    for j in range(days):
        growth_factor = const1*np.exp(const2*np.random.standard_normal(paths))
        stock_price = stock_price*growth_factor
        stock_price_sum = stock_price_sum + stock_price
    stock_price_avg = stock_price_sum/days
    zeros = np.zeros(paths, dtype='float64')
    r_factor = exp(-r*h*days)
    euro_put = r_factor*np.mean(np.maximum(zeros, K-stock_price))
    asian_put = r_factor*np.mean(np.maximum(zeros, K-stock_price_avg))
    euro_call = r_factor*np.mean(np.maximum(zeros, stock_price-K))
    asian_call = r_factor*np.mean(np.maximum(zeros, stock_price_avg-K))
    return (euro_call, euro_put, asian_call, asian_put)


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Parallel computation across strike prices and volatilities
The Client is used to setup the calculation and works with all engines.
Can you write Python code for it?


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Plot the value of the Asian call in (volatility, strike) space.
Can you write Python code for it?

plt.figure()
plt.contourf(sigma_vals, strike_vals, prices['acall'])
plt.axis('tight')
plt.colorbar()
plt.title("Asian Call")
plt.xlabel("Volatility")
plt.ylabel("Strike Price")


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Plot the value of the Asian put in (volatility, strike) space.
Can you write Python code for it?

plt.figure()
plt.contourf(sigma_vals, strike_vals, prices['aput'])
plt.axis('tight')
plt.colorbar()
plt.title("Asian Put")
plt.xlabel("Volatility")
plt.ylabel("Strike Price")


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Plot the value of the European call in (volatility, strike) space.
Can you write Python code for it?

plt.figure()
plt.contourf(sigma_vals, strike_vals, prices['ecall'])
plt.axis('tight')
plt.colorbar()
plt.title('European Call')
plt.xlabel("Volatility")
plt.ylabel("Strike Price")


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Plot the value of the European put in (volatility, strike) space.
Can you write Python code for it?

plt.figure()
plt.contourf(sigma_vals, strike_vals, prices['eput'])
plt.axis('tight')
plt.colorbar()
plt.title("European Put")
plt.xlabel("Volatility")
plt.ylabel("Strike Price")


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
Process and visualize results
Retrieve the results using the get method:
Can you write Python code for it?

results = [ar.get() for ar in async_results]


I am working on the file "Parallel Monte Carlo Option Pricing.ipynb".
The first task is:
We can time a single call of this function using the %timeit magic:
Can you write Python code for it?


I am working on the file "Parallel Testing.ipynb".
The first task is:
Also suppose that there are 16 processors, each operating on a single chunk.
Can you write Python code for it?

numProc = 16

def get_chunk_lengths(vec, numProc):
    """Get the length of each chunk"""
    numDivFloor = len(vec)//numProc
    lengths = [numDivFloor]*numProc
    remainder = len(vec) - numDivFloor*numProc
    if remainder > len(vec):
        print('Error: remainder greater than vector length')
        return
    else:
        for i in range(remainder):
            lengths[i] += 1
        return lengths

splitlengths = get_chunk_lengths(pairs, numProc)

splitlengths

divpos = np.cumsum([0] + splitlengths)

divpos

len(divpos)

def chunks(divpos, vec):
    for i in range(len(divpos)-1):
        yield vec[divpos[i]:divpos[i+1]]

tuple(chunks(divpos, pairs))


I am working on the file "Parallel Testing.ipynb".
The first task is:
Suppose the iterable is a tuple of all possible pairs from ids:
Can you write Python code for it?

pairs = tuple(itertools.combinations(ids, 2))

len(pairs)


I am working on the file "Parallel Testing.ipynb".
The first task is:
Try it on an iterable that is the range of integers from 10 to 74, with number of processors equal to 7:
Can you write Python code for it?

numProc = 7
v = tuple(range(10,75))
splitLengths = get_chunk_lengths(v, numProc)
divpos = np.cumsum([0] + splitLengths)
tuple(chunks(divpos, v))


I am working on the file "Parallel-Profile.ipynb".
The first task is:
Process:
note how the output is returned
Can you write Python code for it?

import multiprocessing as mp
from multiprocessing import Pool
import time
import numpy as np

npoints = 10
input_arr = np.arange(0,npoints)

nproc=int(np.floor(npoints/2.))

output = mp.Queue()

def makedistance(i,output):
    output_arr = i*i
    output.put(output_arr)

t1 = time.time()

procs = mp.Process(target=makedistance, args=(input_arr,output))

procs.start()
procs.join()
results = [output.get()]
print results

t2 = time.time()
print t2 - t1


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Bonferroni adjusted thresholds
Can you write Python code for it?

g_hiv_b = (r4.HIV_LR.ix[probe_idx].p * len(r4)) < .01
g_age_b = (p_vals.in_set_s1.ix[probe_idx] * len(p_vals)) < .01


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Combinations
Can you write Python code for it?

combo = combine(g_age, g_hiv) == 'both'
combo_b = combine(g_hiv_b, g_age_b) == 'both'
combo_r = combine(g_age_r, g_hiv_r) == 'both'

features = {'HIV (BH)': g_hiv & (g_age_nom == False),
            'HIV (adj)': g_hiv_r & (age_nom_r == False),
            'HIV (Bonf)': g_hiv_b & (g_age_nom == False),
            'Age (BH)': g_age_2 & (g_hiv_nom == False),
            'Age (adj)': g_age_r & (hiv_nom_r == False),
            'Age (Bonf)': g_age_b & (g_hiv_nom == False),
            'HIV + Age (BH)': combo,
            'HIV + Age (adj)': combo_r,
            'HIV + Age (Bonf)': combo_b}

c1 = pd.DataFrame({i:v.value_counts() for i,v in features.iteritems()}).T
c1


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Feature sets from rolling statistics
Can you write Python code for it?

g_hiv_r = hiv_rolled.rank(ascending=True).dropna() < g_hiv.sum()
g_age_r = age_rolled.rank(ascending=True).dropna() < g_age.sum()


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Form Feature Sets
Can you write Python code for it?

g_age = bhCorrection(p_vals.in_set_s1.ix[probe_idx]) < .01
g_age_2 = pd.Series(True, r2).ix[probe_idx].fillna(False)
g_age = g_age_2
g_hiv = bhCorrection(r4.HIV_LR.p.ix[probe_idx]) < .01
combo = combine(g_age, g_hiv) == 'both'

#Do not import
fisher_exact_test(g_age, g_hiv)

combo.value_counts()


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Nested Screen for Age Associated Probes

First pass is Hannum et al. dataset
Next we look in the probes passing stage 1 for probes in the EPIC dataset
The two tiers is for power, we only need to correct in stage II for the number of probes passing stage I
I'm using a Benjamini-Hochberg adusted p-value cutoff of 0.01
Can you write Python code for it?

rr = ti(bhCorrection(p_vals.in_set_s1) < .01)
len(rr)

v = p_vals.in_set_s3.ix[rr]
r2 = ti(bhCorrection(v) < .01)
len(r2)

rr = bhCorrection(p_vals.in_set_s1) < .01
rr = rr + pd.Series(1, index=r2).ix[rr.index].fillna(0)
rr.value_counts()


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Nominal significance thresholds
Can you write Python code for it?

g_hiv_nom = r4.HIV_LR.p < .05
g_age_nom = p_vals.in_set_s1 < .05


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Nominal significance thresholds: rolled
Can you write Python code for it?

hiv_nom_r = hiv_rolled.rank(ascending=True).dropna() < g_hiv_nom.sum()
age_nom_r = age_rolled.rank(ascending=True).dropna() < g_age_nom.sum()


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Read in Age Models

Here we went and ran this notebook
Now we need to unpack the data and combine it together
Can you write Python code for it?

outdir = '/cellar/users/agross/Data/tmp'
tables = ['in_set_s1','in_set_s2','in_set_s3']

res = {}
for tp in tables:
    p = outdir + '/' + tp + '/'
    res[tp] = pd.concat([pd.read_csv(p + f, index_col=0, header=[0,1]) for
                    f in os.listdir(p) if f.endswith('age_out.csv')])
res = pd.concat(res, axis=1)

p_vals = res.xs('age_LR', axis=1, level=1).xs('p', axis=1, level=1)
p_vals = p_vals.ix[probe_idx].dropna()
len(p_vals)

def disorder_frac(direction, beta, test_set):
    ct = pd.crosstab(beta < .5, direction.ix[test_set]>0)
    return (1.*ct[1][1] + ct[0][0]) / ct.sum().sum()


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Read in HIV Case Control Models
Can you write Python code for it?

outdir = '/cellar/users/agross/Data/tmp'
p = outdir + '/hiv_consented/'
def read_models(suffix):
    df = pd.concat([pd.read_csv(p + f, index_col=0, header=[0,1]) for
                    f in os.listdir(p) if f.endswith('{}.csv'.format(suffix))])
    return df

r4 = read_models('hiv_age_cc')

gg = bhCorrection(r4.HIV_LR.p.ix[probe_idx]) < .01
gg.value_counts()


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Second pass to simplify things...
Can you write Python code for it?

features = {'HIV (BH)': g_hiv & (g_age_2 == False),
            'HIV (adj)': g_hiv_r & (age_nom_r == False),
            'HIV (Bonf)': g_hiv_b & (g_age_b == False),
            'Age (BH)': g_age_2 & (g_hiv == False),
            'Age (adj)': g_age_r & (g_hiv == False),
            'Age (Bonf)': g_age_b & (g_hiv_b == False),
            'HIV + Age (BH)': combo,
            'HIV + Age (adj)': combo_r,
            'HIV + Age (Bonf)': combo_b}

c1 = pd.DataFrame({i:v.value_counts() for i,v in features.iteritems()}).T
c1


I am working on the file "Parallel/Age_HIV_Features.ipynb".
The first task is:
Size Normalize Datasets
Can you write Python code for it?

mm = df_hiv.ix[:, ti(hiv == 'HIV-')].mean(1)

hiv_rolled = size_normalize(r4.HIV_LR.p, mm)
age_rolled = size_normalize(p_vals.in_set_s1, mm)


I am working on the file "Parallel/Init_Parallel.ipynb".
The first task is:
Logit Transform and Normalize Methylation Data
Can you write Python code for it?

k = ti((age < 68) & (age > 25))

dd = logit_adj(df_meth.ix[:, k])
m = dd.mean(1)
s = dd.std(1)
df_norm = dd.subtract(m, axis=0).divide(s, axis=0)
df_norm = df_norm.clip(-7,7)


I am working on the file "Parallel/Init_Parallel.ipynb".
The first task is:
Prepare Data for Association Tests

The association tests take a while to run in serial so we do them in a map-reduce type format
The idea is we break the data into 100 chunks, run the tests in parallel, and then combine the results
This is not entirely necissary but drops run-time from ~15 min to about 15 seconds
Can you write Python code for it?

def chunkify_df(df, store, table_name, N=100):
    df = df.dropna(1)
    for i in range(N):
        g = df.index[i::N]
        dd = df.ix[g]
        dd.to_hdf(store, '{}/chunk_{}'.format(table_name, i))

gender.value_counts()

labels.ix[k.intersection(df_meth.columns)].value_counts()

store = '/cellar/users/agross/Data/tmp/for_parallel.h5'
store = pd.HDFStore(store)

store['labels'] = labels
store['bio_age'] = mc_adj_c
store['cell_counts'] = cell_counts
store['age'] = age
store['gender'] = gender == 'M'
#store['bio_age'] = age_adv.append(age_adv0)

labels.ix[k.intersection(df_meth.columns)].value_counts()

chunkify_df(df_norm.ix[:, ti(labels == 's1')], store.filename, 'in_set_s1')
chunkify_df(df_norm.ix[:, ti(labels == 's2')], store.filename, 'in_set_s2')
chunkify_df(df_norm.ix[:, ti(labels == 's3')], store.filename, 'in_set_s3')

store.close()
store.open()


I am working on the file "Parallel/Init_Parallel_HIV.ipynb".
The first task is:
Logit Transform and Normalize Methylation Data
Can you write Python code for it?

df = df_hiv.ix[:, pred_c.index]

dd = logit_adj(df)
m = dd.ix[:, ti(duration == 'Control')].mean(1)
s = dd.ix[:, ti(duration == 'Control')].std(1)
df_norm = dd.subtract(m, axis=0).divide(s, axis=0)
df_norm = df_norm.clip(-7,7)

df_norm.shape


I am working on the file "Parallel/Init_Parallel_HIV.ipynb".
The first task is:
Prepare Data for Association Tests

The association tests take a while to run in serial so we do them in a map-reduce type format
The idea is we break the data into 100 chunks, run the tests in parallel, and then combine the results
This is not entirely necissary but drops run-time from ~15 min to about 15 seconds
Can you write Python code for it?

def chunkify_df(df, store, table_name, N=100):
    df = df.dropna(1)
    for i in range(N):
        g = df.index[i::N]
        dd = df.ix[g]
        dd.to_hdf(store, '{}/chunk_{}'.format(table_name, i))

duration.ix[df_norm.columns].value_counts()

hiv.value_counts()

store = '/cellar/users/agross/Data/tmp/for_parallel.h5'
store = pd.HDFStore(store)
(hiv == 'HIV+').ix[pred_c.index].to_hdf(store, 'HIV')
#store['bio_age'] = mc_adj_c
#store['cell_counts'] = cell_counts
#store['age'] = age
#store['gender'] = gender == 'M'
#store['bio_age'] = age_adv.append(age_adv0)

chunkify_df(df_norm, store.filename, 'hiv_consented')

store.close()
store.open()


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
However there are a few annoyances with this approach: We have to add another for loop for each dimensions in the parameter space; and we have to handle storing the results ourselves. Here, especially, the assignment to input parameters in only implicit.
With psyrun, we can define a parameter space in a more natural way:
Can you write Python code for it?

from psyrun import Param

pspace = Param(D=np.arange(2, 5)) * Param(N=[10, 50, 100])
print(pspace)


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
If at least one argument is a sequence, non-sequence arguments will generate lists of the required length:
Can you write Python code for it?


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
Parallelizaton
It would be nice to parallelize all these simulations to make the best use of multicore processors (though, your BLAS library might be parallelized already). This is easy with psyrun as we just have to use a different map function. The only problem is that the function we want to apply needs to be pickleable which requires us to put it into an importable Python module. (It is also possible to switch the parallelization backend from multiprocessing to threading, but that can introduce other issues.)
Can you write Python code for it?

%%sh
cat << EOF > model.py
from ctn_benchmark.nengo.communication import CommunicationChannel
def evaluate(**kwargs):
    return CommunicationChannel().run(**kwargs)
EOF

from model import evaluate
result2 = psyrun.map_pspace_parallel(evaluate, pspace)

pprint(result2)


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
Parameter space exploration
Let us try to find out how the RMSE of the communication channel varies with the number of dimensions and number of neurons. Usually you would write a bunch of nested for loops like this:
Can you write Python code for it?

rmses = []
for D in np.arange(2, 5):
    for N in [10, 50, 100]:
        rmses.append(CommunicationChannel().run(D=D, N=N)['rmse'])

pprint(rmses)


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
Per default NumPy .npz files are used to store this data. However, this format has a few disadvantages. For example, to append to such a file (which happens in the merge step), the whole file has to be loaded into memory. Thus, it is possible to switch to HDF5 which is better in this regard. Just add the line store = psyrun.H5Store() to your task file.
Note that this will require pytables to be installed and that the Sharcnet pytables seems to be broken at the moment.
Speaking of Sharcnet: Currently psy-doit runs the jobs sequantially and immediatly. To actually get serial farming we have to tell psyrun to invoke the right job scheduler. For Sharcnet this would be sqsub. Here is the updated task file:
Can you write Python code for it?

# task_cc2.py

import platform

from ctn_benchmark.nengo.communication import CommunicationChannel
import numpy as np
from psyrun import Param, Sqsub

pspace = Param(D=np.arange(2, 5)) * Param(N=[10, 50, 100])
min_items = 1
max_jobs = 6

sharcnet_nodes = ['narwhal', 'bul', 'kraken', 'saw']
if any(platform.node().startswith(x) for x in sharcnet_nodes):
    workdir = '/work/jgosmann/tcm'
    scheduler = Sqsub(workdir)
    scheduler_args = {
        'timelimit': '10m',
        'memory': '512M'
    }

def execute(**kwargs):
    return CommunicationChannel().run(**kwargs)


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
Serial farming
After doing some initial experiments, you might want to run a larger number of simulations. Because all the simulations are independent from one another, they could run one after another on your computer, but that would take a very long time. It would be faster to distribute the individual simulations to a high performance cluster (like Sharcnet) with many CPU cores that can run the simulations in parallel. This is called serial farming.
To do serial farming, first make sure that it is a Python module that can be installed, is installed, and can be imported. (It is possible to change sys.path if you don't want to install it as a Python module, but this approach is more complicated and error prone.) Then you need to define a "task" for what you intend to run. To define a task you have to create a file task_<name>.py in a directory called psy-tasks. Here is an example task for our communication channel:
Can you write Python code for it?

# task_cc1.py

from ctn_benchmark.nengo.communication import CommunicationChannel
import numpy as np
import psyrun
from psyrun import Param

pspace = Param(D=np.arange(2, 5)) * Param(N=[10, 50, 100])
min_items = 1
max_jobs = 6

def execute(**kwargs):
    return CommunicationChannel().run(**kwargs)


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
Such parameters definitions support basic operations like a cartesian product (as shown above), concatenation with +, and two subtraction like operations with - and the psyrun.pspace.missing function.
Once we defined a parameter space we want to explore, it is easy to apply a function (i.e. run our model) to each set of parameters.
Can you write Python code for it?

import psyrun

result = psyrun.map_pspace(CommunicationChannel().run, pspace)

pprint(result)


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
The map_pspace function conveniently returns a dictionary with all the input parameters and output values. This makes it easy to convert it to other data structurs like a Pandas data frame for further analysis:
Can you write Python code for it?

import pandas as pd
pd.DataFrame(result)


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
There are a number of variables with a special meaning in these kind of task files. pspace defines the parameter space to explore which should be familar by now. execute is the function that gets called for each assignment of parameters and returns a dictionary with the results. max_jobs says that a maximum of 6 processing jobs should be created (a few other jobs not doing the core processing might be created in addition).
When running this task (we will get to that in a moment), the parameter space will be split into parts of the same size and assigned to the processing jobs to be processed. At the end all the results from the processing jobs will be merged into a single file. min_items defines how many parameter assignments should at least be processd by each job.
To run tasks defined in this way, we use the psy-doit command. As the name suggests, this tool is based on the excellent doit automation tool. Be sure that your working directory contains the psy-tasks directory when invoking this command.
First we can list the available tasks:
Can you write Python code for it?


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
This immediatly runs the first (and only the first) parameter assignment as a way to check that nothing crashes. Now let us run the full task.
Can you write Python code for it?


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
This produces standard doit output with dots (.) indicating that a task was executed. As you can see the first task is splitting the parameter space and then several processing jobs are started and in the end the results are merged.
All intermediary files for this will be written to the psy-work/<task name> directory by default. This is also where we find the result file. psyrun provides some helper functions to load this file:
Can you write Python code for it?

data = psyrun.NpzStore().load('psy-work/cc1/result.npz')
pprint(data)


I am working on the file "Parameter exploration - psyrun.ipynb".
The first task is:
We use the Param class to define sets of parameters with keyword arguments to the constructor. As long as all sequences have the same length it is possible to give multiple arguments to the constructor:
Can you write Python code for it?


I am working on the file "ParameterEstimation/Gaussian.ipynb".
The first task is:
4.3 Repeated measurement of IQ
The model:
$$ \mu_{i} \sim \text{Uniform}(0, 300)  $$
$$ \sigma \sim \text{Uniform} (0, 100)  $$
$$ x_{ij} \sim \text{Gaussian} (\mu_{i}, \frac{1}{\sigma^2})  $$
Data Come From Gaussians With Different Means But Common Precision
Can you write Python code for it?

# Data
y = np.array([[90,95,100],[105,110,115],[150,155,160]])
ntest = 3
nsbj = 3

import sys
eps = sys.float_info.epsilon

with pm.Model() as model3:
    # mu_i ~ Uniform(0, 300)
    mui = pm.Uniform('mui', 0, 300, shape=(nsbj,1))

    # sg ~ Uniform(0, 100)
    # sg = pm.Uniform('sg', .0, 100)

    # It is more stable to use a Gamma prior
    lambda1 = pm.Gamma('lambda1', alpha=.01, beta=.01)
    sg = pm.Deterministic('sg',1 / np.sqrt(lambda1))

    # y ~ Normal(mu_i, sg)
    yd = pm.Normal('y', mu=mui, sd=sg, observed=y)

    trace3 = pm.sample(5e3, njobs=2)

burnin = 500
pm.traceplot(trace3[burnin:]);

mu = trace3['mui'][burnin:]
sigma = trace3['sg'][burnin:]

print('The mu estimation is: ', np.mean(mu, axis=0))
print('The sigma estimation is: ',sigma.mean())


I am working on the file "ParameterEstimation/Gaussian.ipynb".
The first task is:
Chapter 4 - Inferences with Gaussians
4.1 Inferring a mean and standard deviation
Inferring the mean and variance of a Gaussian distribution.
$$ \mu \sim \text{Gaussian}(0, .001)  $$
$$ \sigma \sim \text{Uniform} (0, 10)  $$
$$ x_{i} \sim \text{Gaussian} (\mu, \frac{1}{\sigma^2})  $$
Can you write Python code for it?

# Data
x = np.array([1.1, 1.9, 2.3, 1.8])
n = len(x)

with pm.Model() as model1:
    # prior
    mu = pm.Normal('mu', mu=0, tau=.001)
    sigma = pm.Uniform('sigma', lower=0, upper=10)
    # observed
    xi = pm.Normal('xi',mu=mu, tau=1/(sigma**2), observed=x)
    # inference
    trace = pm.sample(1e3, njobs=2)

pm.traceplot(trace[50:]);

from matplotlib.ticker import NullFormatter
nullfmt = NullFormatter()         # no labels
y = trace['mu'][50:]
x = trace['sigma'][50:]

# definitions for the axes
left, width = 0.1, 0.65
bottom, height = 0.1, 0.65
bottom_h = left_h = left + width + 0.02

rect_scatter = [left, bottom, width, height]
rect_histx = [left, bottom_h, width, 0.2]
rect_histy = [left_h, bottom, 0.2, height]

# start with a rectangular Figure
plt.figure(1, figsize=(8, 8))

axScatter = plt.axes(rect_scatter)
axHistx = plt.axes(rect_histx)
axHisty = plt.axes(rect_histy)

# no labels
axHistx.xaxis.set_major_formatter(nullfmt)
axHisty.yaxis.set_major_formatter(nullfmt)

# the scatter plot:
axScatter.scatter(x, y, c=[1, 1, 1], alpha=.5)

# now determine nice limits by hand:
binwidth1 = 0.25
axScatter.set_xlim((-.01, 10.5))
axScatter.set_ylim((-0, 5))

bins1 = np.linspace(-.01, 10.5, 20)
axHistx.hist(x, bins=bins1)
bins2 = np.linspace(-0, 5, 20)
axHisty.hist(y, bins=bins2, orientation='horizontal')

axHistx.set_xlim(axScatter.get_xlim())
axHisty.set_ylim(axScatter.get_ylim());

print('The mu estimation is: ', y.mean())
print('The sigma estimation is: ', x.mean())


I am working on the file "ParameterEstimation/Gaussian.ipynb".
The first task is:
Note from Junpeng Lao
There are might be divergence warning (Uniform prior on sigma is not a good idea in general), which you can further visualize below
Can you write Python code for it?

# display the total number and percentage of divergent
divergent = trace['diverging']
print('Number of Divergent %d' % divergent.nonzero()[0].size)
divperc = divergent.nonzero()[0].size/len(trace)
print('Percentage of Divergent %.5f' % divperc)

# scatter plot for the identifcation of the problematic neighborhoods in parameter space
plt.figure(figsize=(6, 6))
y = trace['mu']
x = trace['sigma']
plt.scatter(x[divergent == 0], y[divergent == 0], color='r', alpha=.05)
plt.scatter(x[divergent == 1], y[divergent == 1], color='g', alpha=.5);


I am working on the file "Parameter_Tuning_XGBoost_with_Example/Data Hackathon 3.x XGBoost models.ipynb".
The first task is:
Load Data:
The data has gone through following pre-processing:
1. City variable dropped because of too many categories
2. DOB converted to Age | DOB dropped
3. EMI_Loan_Submitted_Missing created which is 1 if EMI_Loan_Submitted was missing else 0 | EMI_Loan_Submitted dropped
4. EmployerName dropped because of too many categories
5. Existing_EMI imputed with 0 (median) - 111 values were missing
6. Interest_Rate_Missing created which is 1 if Interest_Rate was missing else 0 | Interest_Rate dropped
7. Lead_Creation_Date dropped because made little intuitive impact on outcome
8. Loan_Amount_Applied, Loan_Tenure_Applied imputed with missing
9. Loan_Amount_Submitted_Missing created which is 1 if Loan_Amount_Submitted was missing else 0 | Loan_Amount_Submitted dropped
10. Loan_Tenure_Submitted_Missing created which is 1 if Loan_Tenure_Submitted was missing else 0 | Loan_Tenure_Submitted dropped
11. LoggedIn, Salary_Account removed
12. Processing_Fee_Missing created which is 1 if Processing_Fee was missing else 0 | Processing_Fee dropped
13. Source - top 2 kept as is and all others combined into different category
14. Numerical and One-Hot-Coding performed
Can you write Python code for it?

data_path = '/Users/wy/notebook/dataset/Parameter_Tuning_XGBoost_with_Example/'
train = pd.read_csv(data_path+'train_modified.csv')
test = pd.read_csv(data_path+'test_modified.csv')

train.shape, test.shape

target='Disbursed'
IDcol = 'ID'

train['Disbursed'].value_counts()


I am working on the file "Parameter_Tuning_XGBoost_with_Example/Data Hackathon 3.x XGBoost models.ipynb".
The first task is:
Step 1- Find the number of estimators for a high learning rate
Can you write Python code for it?

#Choose all predictors except target & IDcols
predictors = [x for x in train.columns if x not in [target, IDcol]]
xgb1 = XGBClassifier(
 learning_rate =0.1,
 n_estimators=1000,
 max_depth=5,
 min_child_weight=1,
 gamma=0,
 subsample=0.8,
 colsample_bytree=0.8,
 objective= 'binary:logistic',
 nthread=4,
 scale_pos_weight=1,
 seed=27)
modelfit(xgb1, train, predictors)


I am working on the file "Parameter_Tuning_XGBoost_with_Example/Data Hackathon 3.x XGBoost models.ipynb".
The first task is:
Step 3: Tune gamma
Can you write Python code for it?

param_test3 = {
 'gamma':[i/10.0 for i in range(0,5)]
}
gsearch3 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.1, n_estimators=140, max_depth=4,
 min_child_weight=6, gamma=0, subsample=0.8, colsample_bytree=0.8,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27),
 param_grid = param_test3, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch3.fit(train[predictors],train[target])
gsearch3.grid_scores_, gsearch3.best_params_, gsearch3.best_score_

xgb2 = XGBClassifier(
 learning_rate =0.1,
 n_estimators=1000,
 max_depth=4,
 min_child_weight=6,
 gamma=0,
 subsample=0.8,
 colsample_bytree=0.8,
 objective= 'binary:logistic',
 nthread=4,
 scale_pos_weight=1,
 seed=27)
modelfit(xgb2, train, predictors)


I am working on the file "Parameter_Tuning_XGBoost_with_Example/Data Hackathon 3.x XGBoost models.ipynb".
The first task is:
Step 4: Tune subsample and colsample_bytree
Can you write Python code for it?

param_test4 = {
 'subsample':[i/10.0 for i in range(6,10)],
 'colsample_bytree':[i/10.0 for i in range(6,10)]
}
gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.1, n_estimators=177, max_depth=4,
 min_child_weight=6, gamma=0, subsample=0.8, colsample_bytree=0.8,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27),
 param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch4.fit(train[predictors],train[target])
gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_

param_test5 = {
 'subsample':[i/100.0 for i in range(75,90,5)],
 'colsample_bytree':[i/100.0 for i in range(75,90,5)]
}
gsearch5 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.1, n_estimators=177, max_depth=4,
 min_child_weight=6, gamma=0, subsample=0.8, colsample_bytree=0.8,
 objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27),
 param_grid = param_test5, scoring='roc_auc',n_jobs=4,iid=False, cv=5)
gsearch5.fit(train[predictors],train[target])


I am working on the file "Parameter_Tuning_XGBoost_with_Example/Data Hackathon 3.x XGBoost models.ipynb".
The first task is:
Step 6: Reducing Learning Rate
Can you write Python code for it?

xgb4 = XGBClassifier(
 learning_rate =0.01,
 n_estimators=5000,
 max_depth=4,
 min_child_weight=6,
 gamma=0,
 subsample=0.8,
 colsample_bytree=0.8,
 reg_alpha=0.005,
 objective= 'binary:logistic',
 nthread=4,
 scale_pos_weight=1,
 seed=27)
modelfit(xgb4, train, predictors)


I am working on the file "Parameteric and Bilevel Polynomial Optimization Problems.ipynb".
The first task is:
Example 3
Note that this is Example 4 in the paper. The set $\mathbf{X}=[0,1]$  remains the same, whereas $\mathbf{K}={(x,y): xy_1^2+y_2^2-x= 0,  y_1^2+xy_2^2-x= 0}$, and $f(x,y) = (1-2x)(y_1+y_2)$. The optimal $J(x)$ is $-2|1-2x|\sqrt{x/(1+x)}$. We enter the equalities as pairs of inequalities.
Can you write Python code for it?

def J(x):
    return -2*abs(1-2*x)*sqrt(x/(1+x))

x = generate_variables('x')[0]
y = generate_variables('y', 2)
f = (1-2*x)*(y[0] + y[1])

gamma = [integrate(x**i, (x, 0, 1)) for i in range(1, 2*level+1)]
marginals = flatten([[x**i-N(gamma[i-1]), N(gamma[i-1])-x**i] for i in range(1, 2*level+1)])

inequalities = [x*y[0]**2 + y[1]**2 - x,  - x*y[0]**2 - y[1]**2 + x,
                y[0]**2 + x*y[1]**2 - x,  - y[0]**2 - x*y[1]**2 + x,
                1-x, x]
sdp = SdpRelaxation(flatten([x, y]))
sdp.get_relaxation(level, objective=f, momentinequalities=marginals,
                   inequalities=inequalities)
sdp.solve(solver="mosek")
print(sdp.primal, sdp.dual, sdp.status)
coeffs = [sdp.extract_dual_value(0, range(len(inequalities)+1))]
coeffs += [sdp.y_mat[len(inequalities)+1+2*i][0][0] - sdp.y_mat[len(inequalities)+1+2*i+1][0][0]
           for i in range(len(marginals)//2)]
plt.plot(x_domain, [J(xi) for xi in x_domain], linewidth=2.5)
plt.plot(x_domain, [Jk(xi, coeffs) for xi in x_domain], linewidth=2.5)
plt.show()


I am working on the file "Parameteric and Bilevel Polynomial Optimization Problems.ipynb".
The first task is:
Finally we define the objective function and the constraints that define the semialgebraic sets, and we generate and solve the relaxation.
Can you write Python code for it?

f = -x*y**2
inequalities = [1.0-x**2-y**2, 1-x, x, 1-y, y]

sdp = SdpRelaxation([x, y], verbose=0)
sdp.get_relaxation(level, objective=f, momentinequalities=marginals,
                   inequalities=inequalities)
sdp.solve()
print(sdp.primal, sdp.dual, sdp.status)
coeffs = [sdp.extract_dual_value(0, range(len(inequalities)+1))]
coeffs += [sdp.y_mat[len(inequalities)+1+2*i][0][0] - sdp.y_mat[len(inequalities)+1+2*i+1][0][0]
           for i in range(len(marginals)//2)]


I am working on the file "Parameteric and Bilevel Polynomial Optimization Problems.ipynb".
The first task is:
Moment constraints play a crucial role in the joint+marginal approach of the SDP relaxation of polynomial optimization problems, and hence also indirectly in the bilevel polynomial optimization problems.
Joint+marginal approach
In a parametric polynomial optimization problem, we can separate two sets of variables, and one set acts as a parameter to the problem. More formally, we would like to find the following function:
$$ J(x) = \inf_{y\in\mathbb{R}^m}{f(x,y): h_j(y)\geq 0, j=1,\ldots,r},$$
where $x\in\mathbf{X}={x\in \mathbb{R}^n: h_k(x)\geq 0, k=r+1,\ldots,t}$. This can be relaxed as an SDP, and we can extract an approximation $J_k(x)$ at level-$k$ from the dual solution. The primal form reads as
$$ \mathrm{inf}z L_z(f)$$
such that
$$ M_k(z)\succeq 0,\
M{k-v_j}(h_j z)\succeq 0, j=1,\ldots,t\
L_z(x^\beta) = \gamma_\beta, \forall\beta\in\mathbb{N}_k^n.$$
Notice that the localizing matrices also address the polynomial constraints that define the semialgebraic set $\mathbf{X}$. If the positivity constraints are fulfilled, then we have a finite Borel representing measure $\mu$ on $\mathbf{K}={h_j(y)\geq 0, j=1,\ldots,r}$ such that $z_{\alpha\beta}=\int_\mathbf{K} x^\alpha y^\beta\mathrm{d}\mu$.
The part that is different from the regular Lasserre hierachy is the last line, where $\gamma_\beta=\int_\mathbf{X} x^\beta\mathrm{d}\varphi(x)$. This establishes a connection between the moments of $x$ on $\mathbf{K}$ in measure $\mu$ and the moments of $x$ on $\mathbf{X}$ in measure $\varphi$. This $\varphi$ measure is a Borel probability measure on $\mathbf{X}$ with a positive density with respect to the Lebesgue measure on $\mathbb{R}^n$. In other words, the marginal of one measure must match the other on $\mathbf{X}$.
The dual of the primal form of the SDP with these moment constraints is
$$ \mathrm{sup}{p, \sigma_i} \int\mathbf{X} p \mathrm{d}\varphi$$
such that
$$ f - p = \sigma_0 + \sum_{j=1}^t \sigma_j h_j\
p\in\mathbb{R}[x], \sigma_j\in\Sigma[x, y], j=0,\ldots,t\
\mathrm{deg} p\leq 2k, \mathrm{deg}...
Can you write Python code for it?

def J(x):
    return -(1-x**2)*x

def Jk(x, coeffs):
    return sum(ci*x**i for i, ci in enumerate(coeffs))

x = generate_variables('x')[0]
y = generate_variables('y')[0]


I am working on the file "Parameteric and Bilevel Polynomial Optimization Problems.ipynb".
The first task is:
Next, we define the level of the relaxation and the moment constraints:
Can you write Python code for it?

level = 4
gamma = [integrate(x**i, (x, 0, 1)) for i in range(1, 2*level+1)]
marginals = flatten([[x**i-N(gamma[i-1]), N(gamma[i-1])-x**i] for i in range(1, 2*level+1)])


I am working on the file "Parameteric and Bilevel Polynomial Optimization Problems.ipynb".
The first task is:
Notice that even in this formulation, there is an implicit constraint on a moment: the top left element of the moment matrix is 1. Given a representing measure, this means that $\int_\mathbf{K} \mathrm{d}\mu=1$. It is actually because of this that a $\lambda$ dual variable appears in the dual formulation:
$$max_{\lambda, \sigma_0} \lambda$$
such that
$$2x^2 - \lambda = \sigma_0\
\sigma_0\in \Sigma{[x]}, \mathrm{deg}\sigma_0\leq 2.$$
In fact, we can move $\lambda$ to the right-hand side, where the sum-of-squares (SOS) decomposition is, $\lambda$ being a trivial SOS multiplied by the constraint $\int_\mathbf{K} \mathrm{d}\mu$, that is, by 1.
We normally think of adding some $g_i(x)$ polynomial constraints that define a semialgebraic set, and then constructing matching localizing matrices. We can, however, impose more constraints on the moments. For instance, we can add a constraint that $\int_\mathbf{K} x\mathrm{d}\mu = 1$. All of these constraints will have a constant instead of an SOS polynomial in the dual. To ensure the moments are not substituted out while generating the SDP, we enter them as pairs of moment inequalities. Solving this problem gives the correct result again:
Can you write Python code for it?

moments = [x-1, 1-x]
sdp = SdpRelaxation([x])
sdp.get_relaxation(1, objective=x**2, momentinequalities=moments)
sdp.solve()
print(sdp.primal, sdp.dual)


I am working on the file "Parameteric and Bilevel Polynomial Optimization Problems.ipynb".
The first task is:
The dual changed, slightly. Let $\gamma_\beta=\int_\mathbf{K} x^\beta\mathrm{d}\mu$ for $\beta=0, 1$. Then the dual reads as
$$max_{\lambda_\beta, \sigma_0} \sum_{\beta=0}^1\lambda_\beta \gamma_\beta$$
such that
$$2x^2 - \sum_{\beta=0}^1\lambda_\beta x^\beta = \sigma_0\
\sigma_0\in \Sigma{[x]}, \mathrm{deg}\sigma_0\leq 2.$$
Indeed, if we extract the coefficients, we will see that $x$ gets a $\lambda_1=2$ (note that equalities are replaced by pairs of inequalities):
Can you write Python code for it?

coeffs = [-sdp.extract_dual_value(0, range(1))]
coeffs += [sdp.y_mat[2*i+1][0][0] - sdp.y_mat[2*i+2][0][0]
           for i in range(len(moments)//2)]
sigma_i = sdp.get_sos_decomposition()
print(coeffs, [sdp.dual, sigma_i[1]-sigma_i[2]])


I am working on the file "Parameteric and Bilevel Polynomial Optimization Problems.ipynb".
The first task is:
To check the correctness of the approximation, we plot the optimal and the approximated functions over the domain.
Can you write Python code for it?

x_domain = [i/100. for i in range(100)]
plt.plot(x_domain, [J(xi) for xi in x_domain], linewidth=2.5)
plt.plot(x_domain, [Jk(xi, coeffs) for xi in x_domain], linewidth=2.5)
plt.show()


I am working on the file "Parameteric and Bilevel Polynomial Optimization Problems.ipynb".
The first task is:
We define the relaxation of the parametric polynomial optimization problem that returns an approximation of $J(x)$ from the dual:
Can you write Python code for it?

def lower_level(k, G, h, M):
    gamma = [integrate(x**i, (x, -M, M))/(2*M) for i in range(1, 2*k+1)]
    marginals = flatten([[x**i-N(gamma[i-1]), N(gamma[i-1])-x**i] for i in range(1, 2*k+1)])
    inequalities = h + [x**2 <= M**2]
    lowRelaxation = SdpRelaxation([x, y])
    lowRelaxation.get_relaxation(k, objective=G,
                                 momentinequalities=marginals,
                                 inequalities=inequalities)
    lowRelaxation.solve()
    print("Low-level:", lowRelaxation.primal, lowRelaxation.dual, lowRelaxation.status)
    coeffs = []
    for i in range(len(marginals)//2):
        coeffs.append(lowRelaxation.y_mat[len(inequalities)+1+2*i][0][0] -
                      lowRelaxation.y_mat[len(inequalities)+1+2*i+1][0][0])
    blocks = [i for i in range(len(inequalities)+1)]
    constant = lowRelaxation.extract_dual_value(0, blocks)
    return constant + sum(ci*x**(i+1) for i, ci in enumerate(coeffs))


I am working on the file "Parcial_1SimMatDiegoP.ipynb".
The first task is:
Actividad 1.1
- Modificar el programa anterior para que el pédulo describa una trayectoria completamente circular. (10 puntos)
- Indicar en el caso anterior con código de colores, los momentos en que el péndulo se encuetra por arriba del cero en $y$ y cuando se encuentra por debajo. (10 puntos)
- Considere un conjunto de péndulos, con las siguientes longitudes:
$$ l = [0.5, 1, 2, 4, 8, 16]. $$
Realice una representación similar a la anterior, pero ahora para todos los péndulos representados en la misma gráfica y cada uno con diferente color. Discutir resultados.(10 puntos)
Actividad 1.2
- Modificar el programa anterior para que el pédulo describa una trayectoria semi-circular. (10 puntos)
- Indicar en el caso anterior con código de colores, los momentos en que el péndulo se encuetra tienes coordenas en $x$ negativas y cuando se encuentra en positivas.  (10 puntos)
- Considere un conjunto de péndulos con la misma longitudes, pero con diferentes gravedades:
$$ g = [0.5, 1, 2, 4, 8, 16]. $$
Realice una representación similar a la anterior, pero ahora para todos los péndulos representados en la misma gráfica y cada uno con diferente color. Discutir resultados. (10 puntos)
Can you write Python code for it?

# Respuesta
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def theta_t(theta_0, theta_0_dot, g, l, t):
    omega_0 = np.sqrt(g/l)
    return theta_0 * np.cos(omega_0 * t) + theta_0_dot * np.sin(omega_0 * t)

t = np.linspace(0,10)

fig = plt.figure(figsize = (7,5))
ax = fig.add_subplot(1, 1, 1)

x = 2 * np.sin(theta_t(1.57, .6, 9.8, 2, t))
y =  - 2 * np.cos(theta_t(1.57, .6, 9.8, 2, t))

ax.plot(x, y, 'bd', ms = 5)
ax.plot([0], [0], 'gD')

ax.set_xlim(xmin = -2.2, xmax = 2.2)
ax.set_ylim(ymin = -2.2, ymax = .2)

plt.show()

# Respuesta
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def theta_t(theta_0, theta_0_dot, g, l, t):
    omega_0 = np.sqrt(g/l)
    return theta_0 * np.cos(omega_0 * t) + theta_0_dot * np.sin(omega_0 * t)

t = np.linspace(0,10)

fig = plt.figure(figsize = (7,5))
ax = fig.add_subplot(1, 1, 1)
x = 2 * np.sin(theta_t(1.57, 0, 9.8, 2, t))
y =  - 2 * np.cos(theta_t(1.57, 0, 9.8, 2, t))
ax.plot(x, y, 'bo', ms = 3)
ax.plot([0], [0], 'gD')
ax.set_xlim(xmin = -3, xmax = 3)
ax.set_ylim(ymin = -3, ymax = .3)
plt.show()

z=x[x<0]
z

len(x[x<0])

np.where(x<0)

x1=x[np.where(x<0)]

x2=x[np.where(x>0)]

y2=y[np.where(x>0)]

y1=y[np.where(x<0)]

fig = plt.figure(figsize = (6,3))

ax = fig.add_subplot(1, 1, 1)

ax.plot(x1, y1, 'm*')
ax.plot(x2, y2, 'cd')

ax.set_xlim(xmin = -3, xmax = 3)
ax.set_ylim(ymin = -3, ymax = .3)

plt.show()

gravedad = np.array ([.5, 1, 2, 4, 8, 16])

def theta_t(theta_0, theta_0_dot, g, l, t):
    omega_0 = np.sqrt(g/l)
    return theta_0 * np.cos(omega_0 * t) + theta_0_dot * np.sin(omega_0 * t)
for indx, g in enumerate (gravedad):
    x = 2 * np.sin(theta_t(.4, .6, g, 2, t))
    y =  - 2 * np.cos(theta_t(.4, .6, g, 2, t))
(x)
(y)

cmap= ['red','blue','green','black','orange',]

fig = plt.figure(figsize = (15,10))
for indx, g in enumerate (gravedad):
    ax = fig.add_subplot(1, 1, 1)
    x = 2 * np.sin(theta_t(1.57, 0, g, 2, t))
    y =  - 2 * np.cos(theta_t(1.57, 0, g, 2, t))
    plt.scatter(x,y, cmap = 'inferno', s= 35,...


I am working on the file "Parcial_1SimMatDiegoP.ipynb".
The first task is:
Actividad 2.1
¿Cuánto esperar para tomar el café?
Primero calentamos agua a  80°C
 . Posteriormente agregamos café al vaso con el agua caliente. Después realizamos la medición de la temperatura ambiente, la cual fue de  24°C.

Simula el sistema en un tiempo de  0
  a  120 unidades de tiempo con una constante de proporcionalidad  $k=−0.0565$. (10 puntos)
Supoga que cada unidad de tiempo corresponde a un minuto. ¿En que tiempo aproximadamente la temperatura es menor a  30°C
 ? (10 puntos)
Busca una constante de proporcionalidad  $k$
  en un rango ( de  −0.2
  a  0.2
  con incremento de 0.01), para la cual el café tiene una temperatura menor de  30°C
  en un tiempo a 20 minutos. (10 puntos)
Can you write Python code for it?

def temperatura(T,t):
    k=-.0565
    Tm=34
    return k*(T-Tm)
t = np.linspace(0,120)
T0=80
t = np.linspace(0,120)
respuesta = odeint(temperatura,T0,t)
plt.plot(t,respuesta)
plt.xlabel('$t$', fontsize = 40)
plt.ylabel('$T$', fontsize = 40)
plt.show()


I am working on the file "Parcial_1SimMatDiegoP.ipynb".
The first task is:
Ahora si gráficamos al péndulo en las coordenas $x,y$, se obtiene algo similar a lo siguiente:
Can you write Python code for it?

t = np.linspace(0,10)

fig = plt.figure(figsize = (5,5))
ax = fig.add_subplot(1, 1, 1)

x = 2 * np.sin(theta_t(.4, .6, 9.8, 2, t))
y =  - 2 * np.cos(theta_t(.4, .6, 9.8, 2, t))
ax.plot(x, y, 'ko', ms = 1.5)
ax.plot([0], [0], 'rD')
ax.set_xlim(xmin = -2.2, xmax = 2.2)
ax.set_ylim(ymin = -2.2, ymax = .2)
plt.show()


I am working on the file "ParmiSomigliAdUnSempliceEsempio.ipynb".
The first task is:
Ben presto ci accorgiamo di quanta pazienza devono avere le scimmie, dopo 100000 tentativi (saremo pur pigri ma anche molto tenaci) abbiamo azzeccato appena 8 lettere su 28:
“vfgoyflcmorpiisd untmsonqcji”
“parmi somigli ad una donnola”
Un risultato non proprio stellare.
Abbiamo 28 lettere da azzeccare, per ognuna abbiamo 27 scelte ('abcdefghijklmnopqrstuvwxyz '), il che vuol dire che la nostra probabilità di azzeccare tirando a caso è una su $27^{28}$ cioè circa:
0.00000000000000000000000000000000000000008<br>
Mentre proponiamo la 1000001-esima frase pensiamo tra noi che sarebbe bello non dover ricominciare da capo ogni volta, che magari ora che abbiamo una frase con 8 lettere corrette sarebbe bello poter migliorarla invece di buttarla via e ricominciare. Se solo il nostro insegnante fosse così gentile da dirci quali sono le lettere che abbiamo azzeccato oltre che quante, finiremmo in un attimo.
Vabbè, cerchiamo comunque di provarci, invece di buttare via la nostra frase ogni volta proviamo a tenerla e cambiare una lettera cercando di ottenere risultati sempre migliori:
Can you write Python code for it?

def altera(vecchia_frase):
    posizione_da_cambiare = random.choice(range(0,len(vecchia_frase)))
    lettera_da_cambiare = vecchia_frase[posizione_da_cambiare]
    alternative = (string.ascii_lowercase + ' ').replace(lettera_da_cambiare,'')
    nuova_frase = list(vecchia_frase)
    nuova_frase[posizione_da_cambiare] = random.choice(alternative)
    return nuova_frase

i=0
miglior_frase = [random_char() for n in range(0,len(amleto))]
miglior_risultato = valuta(miglior_frase)
while(miglior_risultato < len(amleto)):
    frase = altera(miglior_frase)
    risposta = valuta(frase)
    i = i+1
    if risposta > miglior_risultato:
        miglior_risultato = risposta
        miglior_frase = frase
        print(str(i)+':\t"'+''.join(miglior_frase)+'"\t'+str(miglior_risultato))


I am working on the file "ParmiSomigliAdUnSempliceEsempio.ipynb".
The first task is:
Con questa semplice modifica siamo in grado di imparare la frase corretta in appena qualche migliaio di tentativi.
Circa tremila tentativi è decisamente un risultato molto migliore di prima! Certo che però provarle una alla volta è un po’ noioso, perché non proviamo invece a proporne di più allo stesso tempo? Proviamo a proporne 100 in un colpo solo:
Can you write Python code for it?

def migliore(candidati):
    ordinati = sorted(candidati,key=lambda tup: tup[1], reverse=True)
    return ordinati[0]

def genera_candidati(num_candidati):
    candidati = []
    for i in range(0,num_candidati):
        tmp_frase = genera_frase()
        tmp_risposta = valuta(tmp_frase)
        candidati.append((tmp_frase,tmp_risposta))
    return candidati

candidati = genera_candidati(100)
i=0
miglior_frase, miglior_risultato = migliore(candidati)
while(miglior_risultato < len(amleto)):
    i = i+1
    for n in range(0,len(candidati)):
        frase,risposta = candidati[n]
        nuova_frase = altera(frase)
        nuova_risposta = valuta(nuova_frase)
        if nuova_risposta > risposta:
            candidati[n] = (nuova_frase,nuova_risposta)
        if nuova_risposta > miglior_risultato:
            miglior_risultato = nuova_risposta
            miglior_frase = nuova_frase
            print(str(i)+':\t"'+''.join(miglior_frase)+'"\t'+str(miglior_risultato))


I am working on the file "ParmiSomigliAdUnSempliceEsempio.ipynb".
The first task is:
Diamine! Sì, è vero che ci metto circa un terzo delle iterazioni adesso ma quel poveretto del mio insegnante deve valutare uno sproposito di tentativi in più! circa 130000! Mentre prima quando gli proponevo una frase alla volta ne bastavano circa 3000. E dal momento che devo aspettare che lui abbia finito prima di  poter provare nuove frasi mi sto tirando la zappa sui piedi!
Inoltre cos’è questo macello nei miei tentativi inziali?
Come possono cambiare così tanto l’uno dall’altro? Io modifico solo una lettera per volta! Aspetta… ah già, dal momento che ho molte frasi contemporaneamente può essere che alcune di loro abbiano trovato alcune lettere corrette ma che siano state battute in velocità da altre alcune iterazioni dopo. Alla fine mi curo solo di chi è il miglior candidato, se per caso qualche suo rivale riesce a trovare anche solo una lettera in più il mio povero campione viene gettato nel dimenticatoio. Ma perchè sul finale le cose sembrano molto più “uniformi” come me le aspettavo?
Mmmm… potrebbe essere che all’inizio fosse facile migliorare il proprio risultato dato che la maggior parte delle lettere erano sbagliate. Sul finale invece solo una o due lettere sono sbagliate quindi è difficile progredire, bisogna sia avere la fortuna di azzeccare la posizione giusta che la lettera mancante!
Forse è meglio se do un occhio a cosa succede alle mie frasi oltre che al campione:
Can you write Python code for it?

import pprint
pp = pprint.PrettyPrinter()
def stampa_candidati(candidati):
    # candidati -> array di char, li trasformo in stringhe con ''.join(...)
    # [' ', 'x', 'p', 'l', 'f', … ,'d', 'z', 'h', 'f'] -> ' xplfrvvjjvnmzkovohltroudzhf'
    stringhe_e_valori = list(map(lambda x : (''.join(x[0]),x[1]), candidati))
    # per comodità ordino le stringhe in base al numero di lettere corrette, decrescente
    stringhe_ordinate = sorted(stringhe_e_valori,key=lambda tup: tup[1], reverse=True)
    pp.pprint(stringhe_ordinate)
stampa_candidati(genera_candidati(10))


I am working on the file "ParmiSomigliAdUnSempliceEsempio.ipynb".
The first task is:
E poi proporre la nostra frase all’insegnante il quale si limita a:
Can you write Python code for it?

def valuta( candidato ):
        azzeccate = 0
        for (lettera1, lettera2) in zip(candidato, amleto):
            if lettera1 == lettera2:
                azzeccate = azzeccate + 1
        return azzeccate

risposta = valuta(frase)
print(risposta)


I am working on the file "ParmiSomigliAdUnSempliceEsempio.ipynb".
The first task is:
Eh sì è come pensavo, quello che era il mio campione ha fallito a trovare un miglioramento e quindi un suo rivale è passato in vantaggio!
Ma perchè devono competere tra di loro! Io voglio solo trovare la frase che vuole l’insegnante, se collaborassero sarebbe molto più semplice...
Posso far si che condividano le parti corrette che ognuno di loro ha trovato? Diamine se solo sapessi quali sono! Maledetto insegnante che mi dici solo quante lettere azzecco.
Mmmm e se facessi mescolare tra di loro le varie frasi sperando che mettano insieme le parti corrette che hanno trovato? Potrei prendere due frasi a caso e costruirne una nuova prendendo lettere da una o l’altra. Sì, sembra una buona idea, alla fine se entrambe le frasi hanno trovato una lettera giusta non importa da chi pesco, la nuova frase avrà senz’altro quella lettera, almeno posso star tranquillo che non rovinerò le soluzioni che trovano!
Can you write Python code for it?

def mescola(frase1, frase2):
    nuova_frase = []
    for i in range(0,len(frase1[0])):
        if random.random() > 0.5:
            nuova_frase.append(frase1[0][i])
        else:
            nuova_frase.append(frase2[0][i])
    return (nuova_frase,valuta(nuova_frase))

test_frase1 , test_frase2 = genera_frase(), genera_frase()
print('frase1: "'+''.join(test_frase1)+'"')
print('frase2: "'+''.join(test_frase2)+'"')
print('mix:    "'+''.join(mescola((test_frase1,1),(test_frase2,1))[0])+'"')


I am working on the file "ParmiSomigliAdUnSempliceEsempio.ipynb".
The first task is:
Ok, ma chi mescolo tra loro? Argh…
Uff,riflettiamo, senz’altro voglio la frase migliore, alla fine è quella con più lettere giuste, ma con chi potrei mescolarla? La seconda migliore? E se frasi meno “buone” avessero comunque trovato parti che alla migliore mancano? Facciamo così le scelgo a caso, ma do priorità a quelle con valori più alti. Si, mi sembra sensato, ma come faccio a farlo? T_T
Mmmm.... è come se volessi girare una roulette, dove chi ha un fitness più elevato ha più “spicchi” proviamo a fare uno schizzo:
<img src="roulette.png">
Mmm una frase che ha 24 di valore, merita 24 spicchi, mentre una che ha 20 ne riceverà solo 20, quindi il totale degli spicchi è la somma di tutti i valori.
Can you write Python code for it?

def genera_ruota(candidati):
    totale = 0
    ruota = []
    for frase,valore in candidati:
        totale = totale + valore
        ruota.append((totale,frase,valore))
    return ruota

def gira_ruota(wheel):
    totale = wheel[-1][0]
    pick = totale * random.random()
    for (parziale,candidato,valore) in wheel:
        if parziale >= pick:
            return (candidato,valore)
    return wheel[-1][1:]

candidati = genera_candidati(10)
wheel = genera_ruota(candidati)
pretty_wheel = list(map(lambda x:(x[0],''.join(x[1]),x[2]),wheel))
pp.pprint(pretty_wheel)
print("migliore='"+''.join(migliore(candidati)[0])+"'")

def prova_piu_frasi_e_mescola(num_frasi):
    global valutazioni
    valutazioni = 0
    i=0
    candidati = genera_candidati(num_frasi)
    miglior_frase, miglior_risultato = migliore(candidati)
    while(miglior_risultato < len(amleto)):
        i = i+1
        ruota = genera_ruota(candidati)
        nuovi_candidati = []
        for n in range(0,len(candidati)):
            minitorneo = [gira_ruota(ruota),gira_ruota(ruota)]
            nuova_frase = altera(mescola(minitorneo[0],minitorneo[1])[0])
            nuova_risposta = valuta(nuova_frase)
            minitorneo.append((nuova_frase,nuova_risposta))
            vincitore,valore_vincitore = migliore(minitorneo)
            nuovi_candidati.append((vincitore,valore_vincitore))
            if valore_vincitore > miglior_risultato:
                miglior_risultato = valore_vincitore
                miglior_frase = vincitore
                print(str(i)+':\t"'+''.join(miglior_frase)+'"\t'+str(miglior_risultato))
                stampa_candidati(candidati)
        candidati = nuovi_candidati
    print('valutazioni: '+str(valutazioni))

prova_piu_frasi_e_mescola(10)


I am working on the file "ParmiSomigliAdUnSempliceEsempio.ipynb".
The first task is:
Wow ora sono riuscito a scendere attorno al migliaio, ma ho davvero guadagnato qualcosa in termini di numero di tentativi necessari? Alla fine ora faccio 100 tentativi alla volta, perchè non ci metto 100 volte meno? In effetti è sleale contare il numero di volte che passo i miei tentativi all’insegnante, dovrei contare invece quante volte l’insegnante valuta i miei tentativi!
Vediamo un po’:
Can you write Python code for it?

def valuta( candidato ):
        global valutazioni
        valutazioni = valutazioni + 1
        azzeccate = 0
        for (lettera1, lettera2) in zip(candidato, amleto):
            if lettera1 == lettera2:
                azzeccate = azzeccate + 1
        return azzeccate

def prova_piu_frasi_insieme(num_frasi):
    global valutazioni
    valutazioni = 0
    i=0
    candidati = genera_candidati(num_frasi)
    miglior_frase, miglior_risultato = migliore(candidati)
    while(miglior_risultato < len(amleto)):
        i = i+1
        for n in range(0,len(candidati)):
            frase,risposta = candidati[n]
            nuova_frase = altera(frase)
            nuova_risposta = valuta(nuova_frase)
            if nuova_risposta > risposta:
                candidati[n] = (nuova_frase,nuova_risposta)
            if nuova_risposta > miglior_risultato:
                miglior_risultato = nuova_risposta
                miglior_frase = nuova_frase
                print(str(i)+':\t"'+''.join(miglior_frase)+'"\t'+str(miglior_risultato))
    print('Valutazioni totali: '+str(valutazioni))

prova_piu_frasi_insieme(100)


I am working on the file "Parsers/SecLists/Index-Parse.ipynb".
The first task is:
Finally, let's put this data into csv format.
Can you write Python code for it?

import csv
import sys

output = csv.writer(sys.stdout)
output.writerow(['id', 'title', 'date', 'author', 'parent'])
for x in messages:
    output.writerow([x['id'],
                     x['title'],
                     x['date'],
                     x['author'],
                     x['parent']])


I am working on the file "Parsers/SecLists/Index-Parse.ipynb".
The first task is:
The generated code from seclists.org contains an unterminated anchor tag, so to make things easier for BeautifulSoup's parser, we'll just replace this manually. This particular tag is a good tag to use as a locator for the messages section, so it's good to make sure that it's valid html.
Our soup "query" basically consists of:
1. Find me the tag with name = "begin"
2. Find me the first ul after this tag
Can you write Python code for it?

raw = raw.replace('', '')
soup = BeautifulSoup(raw, 'html5lib')

begin = soup.find(attrs={'name':'begin'}) #beginning of msg links
items = begin.find_next('ul').find_all('li', recursive=False)

pp.pprint(items)


I am working on the file "Parsers/SecLists/Index-Parse.ipynb".
The first task is:
The index file summarizes the reply's details, but we would like the full author with email, and a complete timestamp. To do this, we'll need to delve into the actual raw.html. For this notebook, we'll download the file again, but in actual usage, we would just open the existing file.
The full author and timestamp are contained between the X-Head-of-Message comments.
Can you write Python code for it?

import pendulum

message = messages[4]
reply_url = url + '/' + message['index']
r = requests.get(reply_url)
reply = r.text

start = reply.index('<!--X-Head-of-Message-->') + 24
end = reply.index('<!--X-Head-of-Message-End-->')

head = reply[start:end]
soup = BeautifulSoup(head, 'html5lib')
ems = soup.find_all('em')

for em in ems:
    if em.text == 'From':
        author = em.next_sibling
        #list obfuscates email by replacing @ with ' () ' and removing periods from domain name
        if author.startswith(': '):
            author = author[2:]
        author = author.replace(' () ', '@')
        at = author.find('@')
        author = author[:at] + author[at:].replace(' ', '.')
        message['author'] = author
    elif em.text == 'Date':
        date = em.next_sibling
        if date.startswith(': '):
            date = date[2:]
        message['date'] = str(pendulum.parse(date).in_timezone('UTC'))

print(message)


I am working on the file "Parsers/SecLists/Reply-Parse.ipynb".
The first task is:
At least for 2017_Jan_0, it is pretty effective. 2017_Jan_45 was not successful at all. Now, we'll try the machine learning style, to compare.
Can you write Python code for it?

talon.init()
from talon import signature
reply_ml, sig_ml = signature.extract(raw, sender="dawid@legalhackers.com")
print(sig_ml)
#reply_ml


I am working on the file "Parsers/SecLists/Reply-Parse.ipynb".
The first task is:
Each message in the FD list is wrapped in seclists.org code, including navigation, ads, and trackers, all irrelevant to us. The body of the reply is contained between two comments, <!--X-Body-of-Message--> and <!--X-Body-of-Message-End-->.
BeautifulSoup isn't great at handling comments, so we first use simple indexing to extract the relevant chars. We'll then send it through BeautifulSoup so we can use its .text property to strip out the html tags. BS4 automatically adds tags to create valid html, so remember to parse using the generated <body> tags.
What we end up with is a plaintext version of the message's body.
Can you write Python code for it?

start = content.index('<!--X-Body-of-Message-->') + 24
end = content.index('<!--X-Body-of-Message-End-->')
body = content[start:end]

soup = BeautifulSoup(body, 'html5lib')
bodyhtml = soup.find('body')
raw = bodyhtml.text
Pretty(raw)


I am working on the file "Parsers/SecLists/Reply-Parse.ipynb".
The first task is:
Extract link domains
We'll record what domains are linked to in each message. We use BeautifulSoup to pull out all <a> tags, then urlparse to determine the domain within.
Can you write Python code for it?

from urllib.parse import urlparse

sites = {}

atags = bodyhtml.find_all('a')
hrefs = [link.get('href') for link in atags]

for link in hrefs:
    parsedurl = urlparse(link)
    site = parsedurl.netloc
    if site in sites:
        sites[site] = sites[site] + 1
    else:
        sites[site] = 1

sites


I am working on the file "Parsers/SecLists/Reply-Parse.ipynb".
The first task is:
PGP messages
As can be expected, many messages offer a PGP signature validation. This isn't useful to our processing, so we'll take it out. First, we define get_raw_message with code we've used previously. We then create strip_pgp, looking for the PGP signature. We can just use simple text searches again, with an exception of using RE for the Hash, which can change.
http://seclists.org/fulldisclosure/2017/Oct/11 is a message that includes a PGP signature, so we'll use that to test.
Can you write Python code for it?


I am working on the file "Parsers/SecLists/Reply-Parse.ipynb".
The first task is:
This doesn't seem to output anything. I'm unclear whether or not this library is already trained; documentation states that it was trained on the authors' personal email and an ENRON set. There is an open issue on github https://github.com/mailgun/talon/issues/143 from July asking about the same thing. We will stick with the "brute force" method for now, and continue to look for more libraries.
Extract HTML tags
We'll use a fairly simple regex to extract any tags from the reply.
<([^\s>]+)(\s|/>)+
  * [^\s>]+ one or more non-whitespace characters, followed by:
  * \s|/ either a whitespace character, or a slash (/) for self-closing tags.
We then use a dictionary to count the instances of each unique tag.
Can you write Python code for it?

rx = re.compile('<([^\s>]+)(\s|/>)+')
tags = {}
for tag in rx.findall(str(bodyhtml)):
    tagtype = tag[0]
    if not tagtype.startswith('/'):
        if tagtype in tags:
            tags[tagtype] = tags[tagtype] + 1
        else:
            tags[tagtype] = 1
print(tags)


I am working on the file "Parsers/SecLists/Reply-Parse.ipynb".
The first task is:
We'll gather the contents of a single message. 2017_Jan_0 is one that includes a personal signature, as well as the standard Full Disclosure footer.
2017_Jan_45 is a message that includes a PGP signature.
Can you write Python code for it?

year = '2005'
month = 'Jan'
id = '0'
url = 'http://seclists.org/fulldisclosure/' + year + '/' + month + '/' + id

r = requests.get(url)
content = r.text
from IPython.display import Pretty
Pretty(content)


I am working on the file "Part 05 - Preprocessing and Pipelines.ipynb".
The first task is:
Cross-validated pipelines including scaling, we need to estimate mean and standard deviation separately for each fold.
To do that, we build a pipeline.
Can you write Python code for it?

from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler

standard_scaler = StandardScaler()
standard_scaler.fit(X_train)
X_train_scaled = standard_scaler.transform(X_train)
svm = SVC().fit(X_train_scale, y_train)

#pipeline = Pipeline([("scaler", StandardScaler()),
#                     ("svm", SVC())])
# in new versions:
pipeline = make_pipeline(StandardScaler(), SVC())

pipeline.fit(X_train, y_train)

pipeline.score(X_test, y_test)

pipeline.predict(X_test)


I am working on the file "Part 05 - Preprocessing and Pipelines.ipynb".
The first task is:
Grid Search with a pipeline
Can you write Python code for it?

import numpy as np
from sklearn.grid_search import GridSearchCV

param_grid = {'svc__C': 10. ** np.arange(-3, 3),
              'svc__gamma' : 10. ** np.arange(-3, 3),
              'standardscalar__with_mean' : [True, False],
             }

grid_pipeline = GridSearchCV(pipeline, param_grid=param_grid, n_jobs=-1)

grid_pipeline.fit(X_train, y_train)

grid_pipeline.score(X_test, y_test)


I am working on the file "Part 1- Data Collection - Crawling Flight Crash Data.ipynb".
The first task is:
Connecting to the Mongo DB client running on the same machine.

Must change if the Mongo DB is running on a separate machine. Check MongoDB docs
Can you write Python code for it?

# Connecting to Mongo instance
client = MongoClient()
# specify the name of the db in  brackets
db = client['aircrashdb']
# specify the name of the collection in brackets
collection = db['crawled_data']


I am working on the file "Part 1- Data Collection - Crawling Flight Crash Data.ipynb".
The first task is:
Crawler- The Core

MAIN IDEA: Leveraging the pattern in the url of the website.

The hostname of the url remains the same for all the years - i.e.  http://<-hostname-> .
The path for each year comes after the hostname, i.e. http://<-hostname->/<-year->, where year is a 4 digit year from 1920 to 2016.
The sub path that actually points us to the record page is http://hostname/ <-year>/<-year>-<-record_number-> , where record_number is a number between 1 and the number of crashes that took place in the corresponding year.
http://www.planecrashinfo.com/<-year->/<-year->-<-record_number->.htm

We will iterate through all the years specified at the beginning of this notebook, and send an appropriate HTTP request by building a url, leveraging the url pattern described above.

The code can be parallelized by using IPython.parallel library, not done for the sake of simplicity
Can you write Python code for it?

program_start_time = datetime.datetime.utcnow() # you could uncomment this line if you wish to time the runtime of blocks from here onwards

for i in year_range:
    year_start = datetime.datetime.utcnow()
    # appending the path (year) to the url hostname
    newurl = rooturl + "/" + str(i) + "/" + str(i) + ".htm"
    soup = makeBeautifulSoupObject(newurl)
    tables = soup.find_all('table')
    print (newurl)

    for table in tables:
        #finding the no. of records for the given year
        number_of_rows = len(table.findAll(lambda tag: tag.name == 'tr' and tag.findParent('table') == table))
        row_range = range(1,number_of_rows,1)

        for j in row_range:
            # appending the row number to sub-path of the url, and building the final url that will be used for sending http request
            accident_url = newurl.replace(".htm","") + "-" + str(j) + ".htm"
            web_record = makeBeautifulSoupObject(accident_url)
            # removing all the boilerplate html code except the data table
            table_ = web_record.find_all('table')
            push_record_to_mongo(table_)

    print("Time to crawl year " + str(i) + "-" + str(datetime.datetime.utcnow()-year_start))

program_end_time = datetime.datetime.utcnow()
print ("_____________________________________")
print ("Total program time - " + str(program_end_time-program_start_time))


I am working on the file "Part 1- Data Collection - Crawling Flight Crash Data.ipynb".
The first task is:
Declaring the important 'Global Variables'.

Basically the configuration of our crawl. The start_year and the end year of the crawl could be changed to suit your needs
Can you write Python code for it?

# The URL
rooturl = "http://www.planecrashinfo.com"
url = "http://www.planecrashinfo.com/database.htm"
#change start_year to 1920 to crawl the entire dataset
start_year = 2014
end_year = 2016
year_range = range(start_year,end_year+1,1)
newurl=''


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Alternately, we can use while loops to get things done.
Can you write Python code for it?

i = 0
while i < len(dogs):
    print(i, ':', dogs[i])
    i += 1


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Conditionals
Python's conditional statements are pretty straightforrward.
Can you write Python code for it?

x = -15

if x == 0:
    print(x, "is zero")
elif x > 0:
    print(x, "is positive")
elif x < 0:
    print(x, "is negative")
else:
    print(x, "is unlike anything I've ever seen...")


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Good Coding Practices
Finally, now that our code is becoming slightly longer, let's take a step back and talk about good coding practices. Some of these are coded into Python itself.
Can you write Python code for it?


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Hello World
Let's quickly go through some basics of variable assignment.
Can you write Python code for it?


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
In addition, there's also the PEP8 style guide.
Review
Starting here.
Exercises
Courtesy of Luke Kelley.
References:
- pyformat.info

docs.python.org/3/library/string.html

Starting with the following variables:
Can you write Python code for it?

title = "Banneker + Aztlan Initiative"
month = 6
day = 11
year = 2018
width = 30


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Let's see this in action below.
Can you write Python code for it?

# Addition.
x = a + b
print(x, type(x))

# Subtraction.
x = c - a
print(x, type(x))

# Multiplication.
x = b * a
print(x, type(x))

# Division.
x = a / b
print(x, type(x))

# Exponentiation.
x = c ** b
print(x, type(x))

# ???
x = b // a
print(x, type(x))

x = c // a
print(x, type(x))


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Lists
We can keep track of many objects simultaneously by using lists.
Can you write Python code for it?

dogs = ['border collie', 'australian cattle dog', 'labrador retriever']


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Loops
Python is able to work with any type of iterable, not just integers. This gives loops a lot more flexibility.
Can you write Python code for it?

# Traditional loop.
for i in range(3):
    print('I like ' + dogs[i] + 's.')

# "Pythonic" loop.
for dog in dogs:
    print('I like ' + dog + 's.')


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Math
Magnitudes are a pretty common unit in astronomy and are defined in terms of $m \equiv -2.5 \log(F)$ where $F$ is in units of flux density (energy/area/time/frequency).
Compute $m$ given $F = b(a-1)^2 + c(a-1) + 5$. Note that not all math operations are included in Python by default.
Can you write Python code for it?

# Compute magnitude given flux.
F = ...
m = ...
print(m)


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Note that dictionaries do not maintain any sense of order for the input parameters. This is by design, since this lack of ordering allows dictionaries to be implemented very efficiently.
Sets
Sets contain unordered collections of unique items only. It is really fast to perform bulk quick checks but can be hard to search. They are most useful for union, intersection, difference, symmetric difference math operations, and others. Python's sets have all of these operations built-in, via methods or operators.
Can you write Python code for it?

primes = {2, 3, 5, 7}
odds = {1, 3, 5, 7, 9}

# Union: items appearing in either
print(primes.union(odds)) # equivalently with a method

# Intersction
{1, 2, 3, 5, 7, 9}
print(primes.intersection(odds)) # equivalently with a method

# difference: Items in primes but not in odds
print(primes.difference(odds)) # equivalently with a method


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Note that everything in Python is at some level an object. Our variables are no exception.
Can you write Python code for it?


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Now let's print out an element of the list and capitalize it. We can figure out how to do this within the notebook by taking advantage of tab completion.
Can you write Python code for it?

# Access element.
dog = dogs[0]

# Print result.
print(dog.)


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Numbers
While you can define some variables explicitly, Python defines many of them implicitly and can change them when doing math. This mostly matters when working with integers and floats.
Can you write Python code for it?

# Define two variables.
a, b, c = 3, 2, 5.

print('a = {0}'.format(a))
print('b =', b)
print('c', '=', c)


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
See if you can figure out how to find where 'australian cattle dog' is located using just standard Python operators and built-in list functions.
Can you write Python code for it?


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
This flexibility makes loops in Python incredibly powerful. As an example, see if you can get the loop below to run using some combination of (1) range, (2) zip, and (3) enumerate.
Can you write Python code for it?

for ... in ...:
    print(counter, ':', dog)


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
This type of distinction matters because roundoff errors can be an issue. These are a humongous pain but can be dealt with once you know what to watch out for.
Can you write Python code for it?

f1, f2 = 0.3, 0.1 + 0.2
f1, f2, f1 == f2  # leaving variables for auto-output


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Tuples
Like lists, tuples can keep track of many items at once. However, unlike lists, they're static and can't be changed after initialization.
Can you write Python code for it?

date = (6, 11, 2018)

# Check tomorrow's date.


I am working on the file "Part 1/Introductory Exercises (Part 1).ipynb".
The first task is:
Variable (Re-)Assignment
Beware (re-)assigning things without warning! Python by default only "protects" a certain subset of objects by automatically copying them.
Can you write Python code for it?

# Example 1.
josh = quote  # reassign
josh += " He got #rekt."  # ???
print(quote)
print(josh)

# Example 2.
quotes = quote.split()  # reassign
josh = quotes
josh += " He got #rekt.".split()  # ???
print(quotes)
print(josh)

# Example 3.
quotes = quote.split()  # reassign
josh = quotes[:]
josh += " He got #rekt.".split()  # ???
print(quotes)
print(josh)

# Example 4.
d = b
d += 3.
print(b)
print(d)


I am working on the file "Part 2- Data Wrangling- Adding Latitudes and Longitudes using Google Maps Geocoding API.ipynb".
The first task is:
Connecting to the Mongo DB client running on the same machine.

Must change if the Mongo DB is running on a separate machine. Check MongoDB docs
Can you write Python code for it?

# Connecting to Mongo instance
client = MongoClient()
# specify the name of the db in  brackets
db = client['aircrashdb']
# specify the name of the collection in brackets
collection = db['crawled_data']


I am working on the file "Part 2- Data Wrangling- Adding Latitudes and Longitudes using Google Maps Geocoding API.ipynb".
The first task is:
Declaring the important helper functions and global variables

You have to set up your Geocoding API Key before you proceed.

client_key: insert your Google Maps Geocoding API client_key as specified below . Check Geocoding API docs for help.

_URL_: The Google Maps Geocoding API url constant. Must not be changed.
Can you write Python code for it?

# Global Config Variables
client_key = '&key=<insert_your_39_character_api_key_here>'
_URL_ = 'https://maps.googleapis.com/maps/api/geocode/xml?address='
count = 0

# same helper function as the Flight Crash Data Crawler
def makeBeautifulSoupObject(url):
    # Use a `Session` instance to customize how `requests` handles making HTTP requests.
    session = requests.Session()
    # `mount` a custom adapter that retries failed connections for HTTP and HTTPS requests, in this case- 5 times
    session.mount("http://", requests.adapters.HTTPAdapter(max_retries=5))
    session.mount("https://", requests.adapters.HTTPAdapter(max_retries=5))
    source_code = session.get(url=url)
    plain_text = source_code.text.encode('utf8')
    soup = BeautifulSoup(plain_text, "lxml")
    return soup


I am working on the file "Part 2- Data Wrangling- Adding Latitudes and Longitudes using Google Maps Geocoding API.ipynb".
The first task is:
Extracting latitude and longitude data of the aircrash location, and appending to the MongoDB collection

There are a lot of try and except blocks as the location strings do not follow a nice format.
Some special cases have to be handled,
Example: if 'Off the coast of Peru' is sent the the geocoding api, it will return an error. Instead, the Peru should be sent. We won't get the exact location, but the best possible approximation. You could explore the dataset and find out why this is a problem.

Open to change. Please email me at shivam6294@gmail.com if you can think of a more elegant solution to handle special cases with the geocoding api

After running the code block below, the following fields should've be added to MongoDB Collection:
* geolat: the latitude of the aircrash site
* geolong: the longitude of the aircrash site
Can you write Python code for it?

# for all the records in the collection
cursor = collection.find()
for cur in cursor:
    print(cur["loc"])
    if not cur["loc"] =='NULL':
        # if the latitude and logitude of aircrash location do not exist
        if not "geolat" in cur or not "geolong" in cur:
            try:
                if not cur['loc'] == 'NULL':
                    address_field = '+'.join(cur['loc'].split(' '))
                    Request_and_append(address_field)
                    count = count + 1
                else:
                    print ("NULL- No Route Field")
            except:
                print ("COULD NOT PROCESS " + cur['loc'].encode('utf-8'))
                new_attempt1 = cur['loc'].encode('utf-8').rpartition(',')[-1]
                print ('trying : ' +  new_attempt1)
                try:
                    address_field = '+'.join(new_attempt1.encode('utf-8').strip().split(' '))
                    Request_and_append(address_field)
                except:
                    print ('New attempt has failed as well')
                    new_attempt2 = cur['loc'].encode('utf-8')
                    new_attempt2 = re.sub('[^0-9a-zA-Z ]+', '', new_attempt2)
                    arr = new_attempt2.split()
                    try:
                        i=0
                        for s in arr:
                            if (s.lower() == 'coast'):
                                new_attempt_final = (arr [i-1] + ' ' + arr[i]).encode('utf-8')
                                address_field = '+'.join(new_attempt_final.encode('utf-8').strip().split(' '))
                                Request_and_append(address_field)
                                break
                            elif (s.lower() == 'ocean'):
                                new_attempt_final = (arr [i-1] + ' ' + arr[i]).encode('utf-8')
                                address_field = '+'.join(new_attempt_final.encode('utf-8').strip().split(' '))
                                Request_and_append(address_field)
                              ...


I am working on the file "Part 2- Data Wrangling- Adding Latitudes and Longitudes using Google Maps Geocoding API.ipynb".
The first task is:
Extracting latitude and longitude data of the source and destination, and appending to the MongoDB collection

There are a few try and except blocks as the location strings do not follow a nice format.

Some special cases have to be handled,

Open to change. Please email me at shivam6294@gmail.com if you can think of a more elegant solution to handle special cases with the geocoding api

After running the code block below, the following fields should've be added to MongoDB Collection:
* src_lat: the latitude of the starting point (source) of the aircraft
* src_long: the longitude of the starting point (source) of the aircraft
* dest_lat: the latitude of the crash location ("destination") of the aircraft
* dest_long: the longitude of the crash location ("destination") of the aircraft
Can you write Python code for it?

counter = 0
for cur in cursor:
    print(cur["route"])
    if not cur["route"]=='NULL':
        if not "srclat" in cur and not "srclong" in cur or not "deslat" in cur and not "deslong" in cur:
            try:
                if not cur['route'] == 'NULL':
                    source_dest = cur["route"].split('-')
                    source_dest[0] = source_dest[0].strip()
                    source_dest[-1] = source_dest[-1].strip()
                    address_field1 = ' '.join(source_dest[0].split(' '))
                    print (address_field1)
                    address_field2 = ' '.join(source_dest[-1].split(' '))
                    print (address_field2)
                    print ('\n')
                    finalurl1 = url + address_field1 + client_key_sohail
                    finalurl2 = url + address_field2 + client_key_sohail
                    soup1 = makeBeautifulSoupObject(finalurl1)
                    soup2 = makeBeautifulSoupObject(finalurl2)
                    srclat = soup1.find_all('lat')
                    srclong = soup1.findAll('lng')
                    deslat = soup2.find_all('lat')
                    deslong = soup2.find_all('lng')
                    collection.find_one_and_update({'_id':cur["_id"]},{'$set':{'srclat':srclat[0].string}})
                    collection.find_one_and_update({'_id':cur["_id"]},{'$set':{'srclong':srclong[0].string}})
                    collection.find_one_and_update({'_id':cur["_id"]},{'$set':{'deslat':deslat[0].string}})
                    collection.find_one_and_update({'_id':cur["_id"]},{'$set':{'deslong':deslong[0].string}})
                    print (srclat[0].string)
                    print (srclong[0].string)
                    print (deslat[0].string)
                    print (deslong[0].string)
                    counter = counter +1
                else:
                    print ("NULL- No Route Field")
            except:
                print ("COULD NOT PROCESS " + cur['route'].encode('utf-8'))
        else:
            print...


I am working on the file "Part 2- Data Wrangling- Adding Latitudes and Longitudes using Google Maps Geocoding API.ipynb".
The first task is:
Generating XML File with the following Schema:
-root- <br />
    * -element-
            * -date-      -/date-
            * -lat-       -/lat-
            * -long-      -/long-
            * -fatal-     -/fatal-
    * -/element-
-/root-
Can you write Python code for it?

root = Element('root')
tree = ElementTree(root)
for cur in cursor:
    if "geolat" in cur and "geolong" in cur:
        element = Element('element')
        root.append(element)
        date = Element('date')
        date.text= str(cur['date'])
        element.append(date)
        lat = Element('lat')
        lat.text= cur['geolat']
        element.append(lat)
        long = Element('long')
        long.text= cur['geolong']
        element.append(long)
        fatal = Element('fatal')
        if not cur['fatalities_total'] == 'NULL' and not cur['ground'] == 'NULL':
            total_fatalities = int(cur['fatalities_total']) + int(cur['ground'])
            fatal.text= str(total_fatalities)
        elif cur['fatalities_total'] == 'NULL':
            fatal.text= cur['ground']
        elif cur['ground'] == 'NULL':
            fatal.text= cur['fatalities_total']
        else:
            fatal.text= cur['fatalities_total']

        element.append(fatal)

xml = xml.dom.minidom.parseString(etree.tostring(root))
pretty_xml_as_string = xml.toprettyxml()
print (pretty_xml_as_string)
with open(r'C:\Users\admin\Desktop\GE2324\crash_location_data_with_total_fatal.xml', "wb") as f:
    f.write(pretty_xml_as_string.encode('utf-8'))


I am working on the file "Part 2- Data Wrangling- Adding Latitudes and Longitudes using Google Maps Geocoding API.ipynb".
The first task is:
This is what the XML should look like:
*  here, element: represents each aircrash
* fatal: is the number of fatalities due to each aircraft
* others: self explanatory

When visualised on our website, this is what it looks like:

Generating XML File with the following Schema:
-root-
    * -element-
            * -srclat-      -/srclat-
            * -srclong-       -/srclong-
            * -deslat-      -/deslat-
            * -deslong-     -/deslong-
    * -/element-
-/root-
Can you write Python code for it?

cursor = collection.find()

root = Element('root')
tree = ElementTree(root)
for cur in cursor:
    if "srclat" in cur and "srclong" in cur and "deslat" in cur and "deslong" in cur:
        element = Element('element')
        root.append(element)
        srclat = Element('srclat')
        srclat.text= cur['srclat']
        element.append(srclat)
        srclong = Element('srclong')
        srclong.text= cur['srclong']
        element.append(srclong)
        deslat = Element('deslat')
        deslat.text= cur['deslat']
        element.append(deslat)
        deslong = Element('deslong')
        deslong.text= cur['deslong']
        element.append(deslong)
xml = xml.dom.minidom.parseString(etree.tostring(root))
pretty_xml_as_string = xml.toprettyxml()
print (pretty_xml_as_string)
with open('route_data.xml', "wb") as f:
    f.write(pretty_xml_as_string.encode('utf-8'))


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Finally, let's plot everything together: the galaxy SED, the expected photometry, and the observed filter set. A bare-bones example is shown below, where I've input lots of values by hand using trial and error.
Can you write Python code for it?

# plot our results
plt.figure(figsize=(16, 5))
plt.plot(wave, fgal_nu / np.median(fgal_nu) * 0.8, color='red', alpha=0.4)
for i in range(Nfilt):
    plt.plot(fw[i], ft[i], ls='-', lw=2, color='blue')
plt.xlim([2e3, 2e4])
plt.ylim([0, 1.5])
plt.errorbar(filt_cent, phot / max(phot) * 1.2, xerr=[xe_low, xe_high],
             yerr = phot * phot_ferr / max(phot) * 1.2,
             marker='o', markersize=10, linestyle='none', color='red')
plt.xlabel('Wavelength [A]')
plt.ylabel('Flux Density [per Hz]', fontsize=26)
plt.title('UGCA 166 Photometry')


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
I always find the default label and axes markers to be too small to easily read (especially when showing people plots). Luckily, it's pretty straightforward to change the plotting defaults for matplotlib to make things easier to read. We can override the defaults whenever we plot something (which we'll get to in a bit) or we can just update them all at once (as below).
Can you write Python code for it?

# re-defining plotting defaults
rcParams.update({'xtick.major.pad': '7.0'})
rcParams.update({'xtick.major.size': '7.5'})
rcParams.update({'xtick.major.width': '1.5'})
rcParams.update({'xtick.minor.pad': '7.0'})
rcParams.update({'xtick.minor.size': '3.5'})
rcParams.update({'xtick.minor.width': '1.0'})
rcParams.update({'ytick.major.pad': '7.0'})
rcParams.update({'ytick.major.size': '7.5'})
rcParams.update({'ytick.major.width': '1.5'})
rcParams.update({'ytick.minor.pad': '7.0'})
rcParams.update({'ytick.minor.size': '3.5'})
rcParams.update({'ytick.minor.width': '1.0'})
rcParams.update({'xtick.color': 'k'})
rcParams.update({'ytick.color': 'k'})
rcParams.update({'font.size': 30})


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
In addition to the effective wavelength, we also want some basic metric of how "wide" our filter is. There a lot of ways to define this, just as there are a lot of ways to define an "effective" wavelength. Here, we will use the 95% interval where most of the transmission is contained. In other words, our "width" will be determined by the wavelengths where there's a total of 2.5% transmission remaining on the left/right edges. This is computed below.
Can you write Python code for it?

# initializes a Nfilt x 2 array of "empty" values
filt_bounds = np.empty((Nfilt, 2))

# fraction of total flux from filter (confidence interval)
fbound = 0.95
for i in range(Nfilt):
    cdf = np.cumsum(ft[i])  # compute the cumulative sum over the transmission curve
    cdf /= cdf[-1]  # normalize the transmission curve to sum to 1.
    fremain = (1 - fbound) / 2.  # amount remaining (fraction) on either end

    # compute left bound
    temp = np.abs(cdf - fremain)  # absolute value
    idx_left = np.argmin(temp)  # find the **index** of the minimum position

    # compute right bound
    temp = np.abs(cdf - (1. - fremain))  # absolute value
    idx_right = np.argmin(temp)  # find the **index** of the minimum position

    # assign bounds to our
    filt_bounds[i] = fw[i][idx_left], fw[i][idx_right]  # lower/upper bound [A]

# compute the "width" as the difference between the upper and lower 95% bounds
filt_width = filt_bounds[:,1] - filt_bounds[:,0] # filter width [A]

print(filt_width)


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Let's make some quick plots to see how things look using some fake data. We will use numpy to help us with this.
Can you write Python code for it?


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Let's now plot the data to see what our galaxy looks like. Build on the bare-bones example below using the skills you've learned to make a better-looking plot. (At the minimum, please label the axes!)
Can you write Python code for it?

# flux density (per wavelength) vs wavelength (angstroms) for UGCA 166
#plt.plot(wave, fgal_wave)
#plt.semilogx(wave, fgal_wave)
#plt.semilogy(wave, fgal_wave)
#plt.loglog(wave, fgal_wave)


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Let's quickly generate some data. We'll start with a "grid" of points $\mathbf{x}$ and compute the corresponding output $\mathbf{y}$.
Can you write Python code for it?

# define a relationship: y = ax + b
a, b = 1., 0.  # the trailing decimal guarantees this is a "float" rather than "int"

# initialize our data
n = 1000  # number of data points
x = np.linspace(0., 100., n)  # our grid of `n` data points from 0. to 100.
y = a * x + b  # our output y


I am working on the file "Part 2/galaxy_phot.ipynb".
The first task is:
Let's see how our results look.
Can you write Python code for it?

# plot our results
plt.plot(x, yobs)
plt.plot(x, y)
plt.xlabel('x')
plt.ylabel('y')
plt.title('A (Noisy) Line')