I am working on the file "Notebook_02_asymptotics/asymptotic_analysis_code.ipynb".
The first task is:
Compare tail bounds for different actions with the Rademacher bound
Can you write Python code for it?

varepsilon = .05
aa = .99

sample_size = np.arange(5, 1000, 10)
actions = [.4, .9, .95, 1]
truths = [.01, .6, .95]

store_tailbound = np.zeros((len(sample_size), len(actions)))
store_truths = np.zeros((len(sample_size), len(truths)))

for i, n in enumerate(sample_size):
    for j, a in enumerate(actions):
        store_tailbound[i, j] = tail_figure(n, theta0, a, varepsilon)

for i, n in enumerate(sample_size):
    for j, t in enumerate(truths):
        store_truths[i, j] = tail_figure(n, t, aa, varepsilon)

RC_bound = 2*RC + np.sqrt(np.log(2/varepsilon)/(2*samples))
ERC_bound = 2*np.asarray(er) + np.sqrt(4*np.log(2/varepsilon)/(samples))

colors = sns.color_palette("Blues")
colors_g = sns.color_palette("BuGn_r")

fig, ax = plt.subplots(1, 2, figsize = (12, 4), sharey = True)

ax[0].set_title('Uniform and action specific bounds using the true model\n' +
             r"$\varepsilon = {:1.2f}$,  $\theta_0 = {:1.2f}$".format(varepsilon, theta0), fontsize = 13)
ax[0].plot(samples, RC_bound, lw = 3, color = sns.color_palette()[2], label = 'Uniform bound')
for i, a in enumerate(actions):
    ax[0].plot(sample_size, store_tailbound[:, i], label = r'$\delta_n$' + '({:1.2f})'.format(a), color = colors[2+i])
ax[0].legend(loc = 'best', fontsize = 12)
ax[0].set_xlabel('Sample size', fontsize = 12)
ax[0].set_ylabel('Absolute difference', fontsize = 12)
ax[0].set_ylim([0, .5])

ax[1].set_title(r'Uniform and distribution specific bounds using $\mathsf{R}\left(\mathcal{L}_{\mathcal{A}}(z^n)\right)$' + '\n' +
             r"$\varepsilon = {:1.2f}$, $a = {:1.2f}$".format(varepsilon, aa), fontsize = 13)

ax[1].plot(samples, RC_bound, lw = 3, color = sns.color_palette()[2], label = 'Uniform bound')
for i, t in enumerate(truths):
    ax[1].plot(sample_size, store_truths[:, i], label = r'$\theta=$' + '{:1.2f}'.format(t), color = colors_g[i])
ax[1].legend(loc = 'best', fontsize = 12)
ax[1].set_xlabel('Sample size', fontsize = 12)
plt.tight_layout()

plt.savefig('rademacher_tail_bound.png', format = 'png', dpi=800...


I am working on the file "Notebook_02_asymptotics/asymptotic_analysis_code.ipynb".
The first task is:
Uniform LLN -- coin-tossing tail probabilities
Can you write Python code for it?

def loss_cdf(l_value, aa, true_dist, upper_part):
    """
    Uses the formula for the change of discrete random variable. It
    takes care of the fact that the quadratic loss is non-monotone.

    L(theta0, a) = theta0*(1-a)**2 + (1-theta0)*a**2

    Emprical loss = l_value implies the following empirical dist
        p_n = (l - a**2)/((1 - a)**2 - a**2)
    """

    n, theta0 = true_dist.args

    # p_n corresponding to l_value and aa is
    p_n = (l_value - aa**2)/((1 - aa)**2 - aa**2)
    y_n = int(p_n * n)

    # plot p_n(a), it's a hyperbola with break at a=.5
    if aa < .5:            # for a given aa, emp loss is increasing in p_n
        if upper_part:                         # prob(emp loss > l_value)
            return 1 - true_dist.cdf(y_n)
        else:                                  # prob(emp loss < l_value)
            return true_dist.cdf(y_n)
    else:                  # for a given a, emp loss is decreasing in p_n

        if upper_part:                         # prob(emp loss > l_value)
            return true_dist.cdf(y_n)
        else:                                  # prob(emp loss < l_value)
            return 1 - true_dist.cdf(y_n)

def prob(delta, a, true_dist):
    """
    We want the probability
        |L_n - L| > delta

    The random variable is L_n and the intervals of interest
        L_n < L-delta < L+delta < L_n

    Prob = CDF(L-delta) + (1 - CDF(L+delta))
    """
    n, theta0 = true_dist.args
    L = theta0 * (1 - a)**2 + (1 - theta0) * a**2

    upper_part = loss_cdf(L + delta, a, true_dist, upper_part = True)
    lower_part = loss_cdf(L - delta, a, true_dist, upper_part = False)

    return lower_part + upper_part

sample_size = np.arange(5, 1000, 2)
action_grid1 = [.01, .3, .4, .6]
action_grid2 = [.99, .95, .9, .8]
delta = .02
epsilon = .18

fig, ax = plt.subplots(1, 2, figsize = (12, 4))
fig.suptitle('Tail probabilities for the MLE estimator (evaluated with quadratic loss) in the coin tossing example...


I am working on the file "Notebook_anatomy.ipynb".
The first task is:
In this figure are a few labels of notebook parts I will refer to

OK, change this cell to markdown to see some examples (you'll recognize this if you speak markdown)
This will be Heading1

first thing
second thing
third thing

A horizontal rule:

Indented text

Code snippet:
python
import numpy as np
a2d = np.random.randn(100).reshape(10, 10)
LaTeX inline equation:
$\Delta =\sum_{i=1}^N w_i (x_i - \bar{x})^2$
LaTeX table:
First Header  | Second Header
------------- | -------------
Content Cell  | Content Cell
Content Cell  | Content Cell
HTML:

As you can see on your jupyter homepage, you can open up any notebook
NB: You can return to the homepage by clicking the Jupyter icon in the very upper left corner at any time
You can also Upload a notebook (button on upper right)

As well as start a new notebook with a specific kernel (button to the right of Upload)

So, what's that number after In or Out?  That's the order of running this cell relative to other cells (useful for keeping track of what order cells have been run).  When you save this notebook that number along with any output shown will also be saved.  To reset a notebook go to Cell -> All Output -> Clear and then Save it.

You can do something like this to render a publicly available notebook on github statically (this I do as a backup for presentations and course stuff):
http://nbviewer.jupyter.org/github/<username>/<repo name>/blob/master/<notebook name>.ipynb
like:

http://nbviewer.jupyter.org/github/michhar/rpy2_sample_notebooks/blob/master/TestingRpy2.ipynb

Also, you can upload or start a new interactive, free notebook by going here:

https://tmpnb.org

The nifty thing about Jupyter notebooks (and the .ipynb files which you can download and upload) is that you can share these.  They are just written in JSON language.  I put them up in places like GitHub and...
Can you write Python code for it?

import json

# hit Tab at end of this to see all methods
json.

# hit Shift-Tab within parenthesis of method to see full docstring
json.loads()

?sum()

import json
?json


I am working on the file "Notebooks/.ipynb_checkpoints/AppendMicrosoftAIData-checkpoint.ipynb".
The first task is:
Rank list of images by share rates with Microsoft Image Tagging API output
Block of code for building rank list of images shared in the descending order of their share rates
Appended with Microsoft Image Tagging API results
The output is a rank list of all the images by their share rates along with the tags against every image.
There is a capability to display the actual images as well alongside the rank-list.
Known issue - The '<' and '>' characters in the HTML tags in URL are often intepreted as is.
Future - make sure to add escape logic for these characters in HTML tags. There are opportunities to convert some of these code blocks into methods.
Can you write Python code for it?

header,rnkFlLst = DS.genlstTupFrmCsv("../FinalResults/rankListImages_expt2.csv")
rnkListDf = pd.DataFrame(rnkFlLst,columns=header)
rnkListDf['Proportion'] = rnkListDf['Proportion'].astype('float')
rnkListDf.sort_values(by="Proportion",ascending=False,inplace=True)

# create an overall giant csv
gidFtrs = ImageMap.genMSAIDataHighConfidenceTags("../data/GZC_data_tagged.json",0.5)

gidFtrsLst = DS.cnvrtDictToLstTup(gidFtrs)
df = pd.DataFrame(gidFtrsLst,columns=['GID','tags'])

shrPropsTags = pd.merge(rnkListDf,df,left_on='GID',right_on='GID')

shrPropsTags.to_csv("../FinalResults/resultsExpt2RankList_Tags.csv",index=False)
shrPropsTags['URL'] = '<img src = "https://socialmediabias.blob.core.windows.net/wildlifephotos/All_Zebra_Count_Images/' + shrPropsTags['GID'] + '.jpeg" width = "350">'

shrPropsTags.sort_values(by=['Proportion','GID'],ascending=False,inplace=True)
fullFl = HT.html(HT.body(HT.HTML(shrPropsTags.to_html(bold_rows = False,index=False))))

fullFl
outputFile = open("../FinalResults/resultsExpt2RankList_Tags.html","w")
outputFile.write(fullFl)
outputFile.close()


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-checkpoint.ipynb".
The first task is:
Change True to 1, False to 0
Can you write Python code for it?

diabetes_map = {True : 1, False : 0}

df['diabetes'] = df['diabetes'].map(diabetes_map)

df.head(5)


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-checkpoint.ipynb".
The first task is:
Check true/false ratio
Can you write Python code for it?

num_true = len(df.loc[df['diabetes'] == True])
num_false = len(df.loc[df['diabetes'] == False])
print("Number of True cases:  {0} ({1:2.2f}%)".format(num_true, (num_true/ (num_true + num_false)) * 100))
print("Number of False cases: {0} ({1:2.2f}%)".format(num_false, (num_false/ (num_true + num_false)) * 100))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-checkpoint.ipynb".
The first task is:
Definition of features
From the metadata on the data source we have the following definition of the features.
| Feature  | Description | Comments |
|--------------|-------------|--------|
| num_preg     | number of pregnancies         |
| glucose_conc | Plasma glucose concentration a 2 hours in an oral glucose tolerance test         |
| diastolic_bp | Diastolic blood pressure (mm Hg) |
| thickness | Triceps skin fold thickness (mm) |
|insulin | 2-Hour serum insulin (mu U/ml) |
| bmi |  Body mass index (weight in kg/(height in m)^2) |
| diab_pred |  Diabetes pedigree function |
| Age (years) | Age (years)|
| skin | ???? | What is this? |
| diabetes | Class variable (1=True, 0=False) |  Why is our data boolean (True/False)? |
Handle null values
Pandas makes it easy to see if there are any null values in the data frame.
The isnull() method will check each value in the data frame for null values,  and then .any() will return if any nulls are found.
Can you write Python code for it?

df.isnull().values.any()

def plot_corr(df, size=10):
    """
    Function plots a graphical correlation matrix for each pair of columns in the dataframe.

    Input:
        df: pandas DataFrame
        size: vertical and horizontal size of the plot

    Displays:
        matrix of correlation between columns.  Blue-cyan-yellow-red-darkred => less to more correlated
                                                0 ------------------>  1
                                                Expect a darkred line running from top left to bottom right
    """

    corr = df.corr()    # data frame correlation function
    fig, ax = plt.subplots(figsize=(size, size))
    ax.matshow(corr)   # color code the rectangles by correlation value
    plt.xticks(range(len(corr.columns)), corr.columns)  # draw x tick marks
    plt.yticks(range(len(corr.columns)), corr.columns)  # draw y tick marks

plot_corr(df)

df.corr()

df.head()

del df['skin']

df.head()

plot_corr(df)


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-checkpoint.ipynb".
The first task is:
Load and review data
Can you write Python code for it?

df = pd.read_csv("./data/pima-data.csv")      # load Pima data.  Adjust path as necessary

df.shape

df.head(5)

df.tail(5)


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Are these 0 values possible?
How many rows have have unexpected 0 values?
Can you write Python code for it?

print("# rows in dataframe {0}".format(len(df)))
print("# rows missing glucose_conc: {0}".format(len(df.loc[df['glucose_conc'] == 0])))
print("# rows missing diastolic_bp: {0}".format(len(df.loc[df['diastolic_bp'] == 0])))
print("# rows missing thickness: {0}".format(len(df.loc[df['thickness'] == 0])))
print("# rows missing insulin: {0}".format(len(df.loc[df['insulin'] == 0])))
print("# rows missing bmi: {0}".format(len(df.loc[df['bmi'] == 0])))
print("# rows missing diab_pred: {0}".format(len(df.loc[df['diab_pred'] == 0])))
print("# rows missing age: {0}".format(len(df.loc[df['age'] == 0])))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
At this point our data is ready to be used for prediction.
Predict diabetes with the prediction data.  Returns 1 if True, 0 if false
Can you write Python code for it?


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Change diabetes from boolean to integer, True=1, False=0
Can you write Python code for it?

diabetes_map = {True : 1, False : 0}
df['diabetes'] = df['diabetes'].map(diabetes_map)


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Correlated Feature Check
Helper function that displays correlation by color.  Red is most correlated, Blue least.
Can you write Python code for it?

def plot_corr(df, size=11):
    """
    Function plots a graphical correlation matrix for each pair of columns in the dataframe.

    Input:
        df: pandas DataFrame
        size: vertical and horizontal size of the plot

    Displays:
        matrix of correlation between columns.  Blue-cyan-yellow-red-darkred => less to more correlated
                                                0 ------------------>  1
                                                Expect a darkred line running from top left to bottom right
    """

    corr = df.corr()    # data frame correlation function
    fig, ax = plt.subplots(figsize=(size, size))
    ax.matshow(corr)   # color code the rectangles by correlation value
    plt.xticks(range(len(corr.columns)), corr.columns)  # draw x tick marks
    plt.yticks(range(len(corr.columns)), corr.columns)  # draw y tick marks

plot_corr(df)

df.corr()

df.head(5)


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Data has 0 in places it should not.
Just like test or test datasets we will use imputation to fix this.
Can you write Python code for it?

#Impute with mean all 0 readings
fill_0 = Imputer(missing_values=0, strategy="mean", axis=0)
X_predict = fill_0.fit_transform(X_predict)


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Definition of features
From the metadata on the data source we have the following definition of the features.
| Feature  | Description | Comments |
|--------------|-------------|--------|
| num_preg     | number of pregnancies         |
| glucose_conc | Plasma glucose concentration a 2 hours in an oral glucose tolerance test         |
| diastolic_bp | Diastolic blood pressure (mm Hg) |
| thickness | Triceps skin fold thickness (mm) |
|insulin | 2-Hour serum insulin (mu U/ml) |
| bmi |  Body mass index (weight in kg/(height in m)^2) |
| diab_pred |  Diabetes pedigree function |
| Age (years) | Age (years)|
| skin | ???? | What is this? |
| diabetes | Class variable (1=True, 0=False) |  Why is our data boolean (True/False)? |
Check for null values
Can you write Python code for it?


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Good distribution of true and false cases.  No special work needed.
Spliting the data
70% for training, 30% for testing
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

feature_col_names = ['num_preg', 'glucose_conc', 'diastolic_bp', 'thickness', 'insulin', 'bmi', 'diab_pred', 'age']
predicted_class_names = ['diabetes']

X = df[feature_col_names].values     # predictor feature columns (8 X m)
y = df[predicted_class_names].values # predicted class (1=true, 0=false) column (1 X m)
split_test_size = 0.30

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=split_test_size, random_state=42)
                            # test_size = 0.3 is 30%, 42 is the answer to everything


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Loading and Reviewing the Data
Can you write Python code for it?

df = pd.read_csv("./data/pima-data.csv")

df.shape

df.head(5)

df.tail(5)


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Logisitic regression with class_weight='balanced'
Can you write Python code for it?

C_start = 0.1
C_end = 5
C_inc = 0.1

C_values, recall_scores = [], []

C_val = C_start
best_recall_score = 0
while (C_val < C_end):
    C_values.append(C_val)
    lr_model_loop = LogisticRegression(C=C_val, class_weight="balanced", random_state=42)
    lr_model_loop.fit(X_train, y_train.ravel())
    lr_predict_loop_test = lr_model_loop.predict(X_test)
    recall_score = metrics.recall_score(y_test, lr_predict_loop_test)
    recall_scores.append(recall_score)
    if (recall_score > best_recall_score):
        best_recall_score = recall_score
        best_lr_predict_test = lr_predict_loop_test

    C_val = C_val + C_inc

best_score_C_val = C_values[recall_scores.index(best_recall_score)]
print("1st max value of {0:.3f} occured at C={1:.3f}".format(best_recall_score, best_score_C_val))

%matplotlib inline
plt.plot(C_values, recall_scores, "-")
plt.xlabel("C value")
plt.ylabel("recall score")

from sklearn.linear_model import LogisticRegression
lr_model =LogisticRegression( class_weight="balanced", C=best_score_C_val, random_state=42)
lr_model.fit(X_train, y_train.ravel())
lr_predict_test = lr_model.predict(X_test)

# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_test, lr_predict_test)))
print(metrics.confusion_matrix(y_test, lr_predict_test) )
print("")
print("Classification Report")
print(metrics.classification_report(y_test, lr_predict_test))
print(metrics.recall_score(y_test, lr_predict_test))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

lr_model =LogisticRegression(C=0.7, random_state=42)
lr_model.fit(X_train, y_train.ravel())
lr_predict_test = lr_model.predict(X_test)

# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_test, lr_predict_test)))
print(metrics.confusion_matrix(y_test, lr_predict_test) )
print("")
print("Classification Report")
print(metrics.classification_report(y_test, lr_predict_test))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
LogisticRegressionCV
Can you write Python code for it?

from sklearn.linear_model import LogisticRegressionCV
lr_cv_model = LogisticRegressionCV(n_jobs=-1, random_state=42, Cs=3, cv=10, refit=False, class_weight="balanced")  # set number of jobs to -1 which uses all cores to parallelize
lr_cv_model.fit(X_train, y_train.ravel())


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Metrics
Can you write Python code for it?

print("Confusion Matrix")
print("{0}".format(metrics.confusion_matrix(y_test, nb_predict_test)))
print("")

print("Classification Report")
print(metrics.classification_report(y_test, nb_predict_test))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
No obvious null values.
Check class distribution
Rare events are hard to predict
Can you write Python code for it?

num_obs = len(df)
num_true = len(df.loc[df['diabetes'] == 1])
num_false = len(df.loc[df['diabetes'] == 0])
print("Number of True cases:  {0} ({1:2.2f}%)".format(num_true, (num_true/num_obs) * 100))
print("Number of False cases: {0} ({1:2.2f}%)".format(num_false, (num_false/num_obs) * 100))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Predict Test Data
Can you write Python code for it?

rf_predict_test = rf_model.predict(X_test)

# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_test, rf_predict_test)))

print(metrics.confusion_matrix(y_test, rf_predict_test) )
print("")
print("Classification Report")
print(metrics.classification_report(y_test, rf_predict_test))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Predict Training Data
Can you write Python code for it?

rf_predict_train = rf_model.predict(X_train)
# training metrics
print("Accuracy: {0:.4f}".format(metrics.accuracy_score(y_train, rf_predict_train)))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Random Forest
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
rf_model = RandomForestClassifier(random_state=42)      # Create random forest object
rf_model.fit(X_train, y_train.ravel())


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Test Prediction on data
Once the model is loaded we can use it to predict on some data.  In this case the data file contains a few rows from the original Pima CSV file.
Can you write Python code for it?

# get data from truncated pima data file
df_predict = pd.read_csv("./data/pima-data-trunc.csv")
print(df_predict.shape)

df_predict


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
The correlations look good.  There appear to be no coorelated columns.
Mold Data
Data Types
Inspect data types to see if there are any issues.  Data should be numeric.
Can you write Python code for it?


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
The skin and thickness columns are correlated 1 to 1.  Dropping the skin column
Can you write Python code for it?


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Using your trained Model
Save trained model to file
Can you write Python code for it?

from sklearn.externals import joblib
joblib.dump(lr_cv_model, "./data/pima-trained-model.pkl")


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
Verifying predicted value was split correctly
Can you write Python code for it?

print("Original True  : {0} ({1:0.2f}%)".format(len(df.loc[df['diabetes'] == 1]), (len(df.loc[df['diabetes'] == 1])/len(df.index)) * 100.0))
print("Original False : {0} ({1:0.2f}%)".format(len(df.loc[df['diabetes'] == 0]), (len(df.loc[df['diabetes'] == 0])/len(df.index)) * 100.0))
print("")
print("Training True  : {0} ({1:0.2f}%)".format(len(y_train[y_train[:] == 1]), (len(y_train[y_train[:] == 1])/len(y_train) * 100.0)))
print("Training False : {0} ({1:0.2f}%)".format(len(y_train[y_train[:] == 0]), (len(y_train[y_train[:] == 0])/len(y_train) * 100.0)))
print("")
print("Test True      : {0} ({1:0.2f}%)".format(len(y_test[y_test[:] == 1]), (len(y_test[y_test[:] == 1])/len(y_test) * 100.0)))
print("Test False     : {0} ({1:0.2f}%)".format(len(y_test[y_test[:] == 0]), (len(y_test[y_test[:] == 0])/len(y_test) * 100.0)))


I am working on the file "Notebooks/.ipynb_checkpoints/Pima-Prediction-with-reload-checkpoint.ipynb".
The first task is:
We check to ensure we have the the desired 70% train, 30% test split of the data
Can you write Python code for it?

print("{0:0.2f}% in training set".format((len(X_train)/len(df.index)) * 100))
print("{0:0.2f}% in test set".format((len(X_test)/len(df.index)) * 100))


I am working on the file "Notebooks/.ipynb_checkpoints/Python BootCamp 2017 - Your code pretty and healthy-checkpoint.ipynb".
The first task is:
Do a proper name
Look the following method definition:

Muti-word variables and methods should be lower-case and separated by underscore. Ex:
number_of_states = 5

def sum_integers(x, y):
    return (x, y)

Classes can follow CamelCase notation:
class PhaseMap:
    def __init__(self, **kwargs):
        self.input_filename = kwargs['filename']

Re-using files
I already had the terrible experience of getting to codes where it is simply impossible to track down where the variables come from. Ok, not impossible. But rearly there. At least considering the time that I had to look into those codes.
I ended up not using them simply because the way that the author was importing files was giving me no clue about where I could find the definition of functions, classes, etc. Here is an example. Consider you have a file called file_1.py
Can you write Python code for it?


I am working on the file "Notebooks/.ipynb_checkpoints/Python BootCamp 2017 - Your code pretty and healthy-checkpoint.ipynb".
The first task is:
There are different documentation standards that you or your group may choose to use. The important thing here is to make sure that Sphinx can read it and process it so you are saving your time. Really.
Here are two examples of documentation standards that I already used:
PEP252 DocString Standard
Can you write Python code for it?

def say_hello(name):
    """
    This method simply says hello to the person
    whose name is given as parameter.

    param name: the name of the person that will be greeted.
    ptype name: string
    """
    print "Hello, {0}!".format(name)


I am working on the file "Notebooks/.ipynb_checkpoints/Python BootCamp 2017 - Your code pretty and healthy-checkpoint.ipynb".
The first task is:
This documentation standard is called PEP 252 and can be found here.
Google DocString Standard
Can you write Python code for it?

def say_hello(name):
    """
    This method simply says hello to the person
    whose name is given as parameter.

    Parameters
    ----------
        name: string
        The name of the person that will be greeted.
    """
    print "Hello, {0}!".format(name)


I am working on the file "Notebooks/.ipynb_checkpoints/Python BootCamp 2017 - Your code pretty and healthy-checkpoint.ipynb".
The first task is:
This is almost like poetry but it is how we should be writing our codes. And you know what? It is EASY! This notebook will have a quick overview of how to add docstrings and comments to your code and how to write a clear code.
Do the docstring for your life
It is absurdely simple to add docstrings to your python code. That can be done in the beggining of a file or after the definition of a method/function or class. Let me show you some examples.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
My Sample Code

This is such a simple documentation! The only thing I have to do
is to add three single quotes (') or double quotes (") together to
start a docstring block! Then just write down in a few words what
you want to do in your file.

by J. Bond - Feb 31, 202X
"""

And there you go! You added documentation! Yay!
You can do that also if you have the definition of a method.
Can you write Python code for it?

def say_hello(name):
    """
    This method simply says hello to the person whose name is given
    as parameter.
    """
    print "Hello, {0}!".format(name)

say_hello("Bruno")


I am working on the file "Notebooks/.ipynb_checkpoints/Python BootCamp 2017 - Your code pretty and healthy-checkpoint.ipynb".
The first task is:
This is how Google has been used to document their codes and it is also read by Sphinx. You can find more about this style here. The final choice is yours to do but you want to make sure you are the clearest possible.
Why write if others can’t read?
People that work with Python are worried about code readability. That is so true that a document called PEP-8 was written in order to help mortals to get their code clearer and easier to read. The full document is quite short and easy to read and it can be found here. Here are some examples.
Language
It may be very tempting to write comments and documentation in your mother language. But if you are not 10000000% sure that your code will be used by people that speak it, write them in English.
Identation
According to PEP-8, one should always use space instead of usig tabs. The recommended identation size is 4 spaces. This avoid errors when messing up with both identation methods. Just search over the Internet how to setup your favorite editor to keep the identation method always the same (hopefully 4 spaces).
Can you write Python code for it?

for i in range(10):
	x = i * 2
    print x


I am working on the file "Notebooks/.ipynb_checkpoints/Python BootCamp 2017 - Your code pretty and healthy-checkpoint.ipynb".
The first task is:
Three lines. Two files. And we are already screwed. Please, be a nice person and just do a proper import so the reader know what is going on.
Can you write Python code for it?

# file_2.py
import file_1
import file_2
print file_1.x + file_2.a + file_1.y


I am working on the file "Notebooks/.ipynb_checkpoints/Unsaturated_Zone_Validation-checkpoint.ipynb".
The first task is:
2. EMPTYING AND FILLING THE UNSATURATED ZONE
In this example we investigate the filling and emptying of the unsaturated zone. The unsaturated zone has certain physical boundaries, as it cannot have negative values for $S_r$ not values larger than $S_{rmax}$.
$P = 0.002$ m/d for:  $0 < t < 183$
$P = 0.00$ m/d for:  $183 < t $
We model the situation where the unsaturated zone is full and where it is empty, to validate that the the amount of water cannot excess the soil capacity $S_{cap}$ and cannot go below zero. We therefore first consider half a year with very wet conditions, followed by half a year with no rain.
Can you write Python code for it?

n = 365
t = np.arange(0,n,1)
P = np.zeros(n)
P[:int(n/2.0)] = 2.0 / 1000 # All forcings must be provided in meters
E = 0.001 * np.ones(n)

# Provide the model parameters for the soil module
S_rmax = 0.01 # is equal to ±100 milimeters
K_p = 0.001 # is equal to 0.01 m/d
Gamma = 1.0
Imax = 0.0015

R, S = Unsat_Zone.perc(t, P, E, S_rmax, K_p, Gamma, Imax, dt, solver=0)
R1, S1 = Unsat_Zone.perc(t, P, E, S_rmax, K_p, Gamma, Imax, dt, solver=1)

plt.figure(figsize=(15,4))
plt.subplot(121)
plt.plot(t,S)
plt.plot(t,S1)
plt.ylabel('S [meter]')
plt.xlabel('Time [days]')
plt.axhline(S_rmax, color='r')
plt.legend(['explicit', 'implicit', r'$S_rmax$'], loc='best')
plt.title('Soil State over Time')
plt.ylim(0, S_rmax+0.001)

plt.subplot(122)
plt.plot(R)
plt.plot(R1)
plt.title('recharge over time')
plt.ylabel('Recharge [m/d]')
plt.xlabel('Time [days]')
plt.legend(['explicit', 'implicit', r'$S_rmax$'], loc='best')
plt.ylim(0, max(R)+0.0001)


I am working on the file "Notebooks/.ipynb_checkpoints/Unsaturated_Zone_Validation-checkpoint.ipynb".
The first task is:
Compare performance of Cythonized and Python Module
(Non-functioning 29/11/2015)
The computations in this script are computationally demanding and slows down the calibration of a non-linear time series model significantly. Therefore, the python script for the unsaturated zone was ported to the programming C-language using Cython. The Python-script to be cythonized (or in other words: compiled) is Unsat_Zone.pyx using the
python script setup_unsat_zone.py. Compiling the script on Mac goes as follows:

Open Terminal
Move to the directory where the setup_unsat_zone.py and the Unsat_Zone.pyx files are located
In your terminal, type: 'python setup_unsat_zone.py build_ext --inplace' and press enter

The file will now be compiled to an Unsat_Zone.c and an Unsat_Zone.so file (Shared Object) that can be imported as usual in python. To show the speed-up that is gained by Cythonizing our script, some performance tests are done below. It is shown that the Cythonized file of the python script is approximatly 20-25 times faster than the python script, a significant time saving.
Can you write Python code for it?

#Create an variable to store the percolation functions as the wrapper does not work with function?
X1 = Unsat_Zone.perc
X2 = Unsat_Zone_Python.perc

# Write a wrapper for the Cython file and time it
def wrapper(X1, t, P, E, S_cap, K_sat, Beta, D, dt, solver=1):
    def wrapped():
        return X1(t, P, E, S_cap, K_sat, Beta, D, dt, solver=1)
    return wrapped

wrapped = wrapper(X1,t, P, E, S_cap, K_sat, Beta, D, dt, solver=1)

Cython = timeit.timeit(wrapped, number=1000)

print 'Time taken is', Cython

# Write a wrapper for the Python file and time it
def wrapper(X2, t, P, E, S_cap, K_sat, Beta, D, dt):
    def wrapped():
        return X2(t, P, E, S_cap, K_sat, Beta, D, dt)
    return wrapped

wrapped = wrapper(X2,t, P, E, S_cap, K_sat, Beta, D, dt)

Python = timeit.timeit(wrapped, number=1000)

print 'using Cython File is', (Python/Cython), 'times faster than python to solve the soil module'


I am working on the file "Notebooks/0. Gather data.ipynb".
The first task is:
1 . Recoger tweets a partir de un id
Can you write Python code for it?

tweet = twitter_api.statuses.show(_id='628949369883000832')

tweet['text']


I am working on the file "Notebooks/0. Gather data.ipynb".
The first task is:
2. Recoger tweets de una usuaria
Can you write Python code for it?

femfreq_tweet_search = twitter_api.statuses.user_timeline(screen_name="femfreq", count=100)

femfreq_tweet_search[0]['user']['description']

femfreq_tweet_search[-1]['text']


I am working on the file "Notebooks/0. Gather data.ipynb".
The first task is:
Configurar las credenciales para acceder al API de Twitter
Can you write Python code for it?

config = ConfigParser()
config.read(join(pardir,'src','credentials.ini'))

APP_KEY = config['twitter']['app_key']
APP_SECRET = config['twitter']['app_secret']
OAUTH_TOKEN =  config['twitter']['oauth_token']
OAUTH_TOKEN_SECRET =  config['twitter']['oauth_token_secret']

from twitter import oauth, Twitter, TwitterHTTPError


I am working on the file "Notebooks/0. Gather data.ipynb".
The first task is:
Esta es la molona librería que vamos a utilizar: https://github.com/sixohsix/twitter/tree/master
Can you write Python code for it?

auth = oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET,
                   APP_KEY, APP_SECRET)

twitter_api = Twitter(auth=auth)
twitter_api.retry = True


I am working on the file "Notebooks/00-Hi-C_quality_check.ipynb".
The first task is:
Count the number of lines in the file (4 times the number of reads)
Can you write Python code for it?

! wc -l /media/storage/FASTQs/K562_HindIII_1.fastq


I am working on the file "Notebooks/00-Hi-C_quality_check.ipynb".
The first task is:
There are 40 M lines in the file, which means 10 M reads in total.
Quality check before mapping
Can you write Python code for it?

from pytadbit.utils.fastq_utils import quality_plot

for r_enz in ['HindIII', 'MboI']:
    quality_plot('/media/storage/FASTQs/K562_{0}_1.fastq'.format(r_enz), r_enz=r_enz,
                 nreads=1000000, paired=False)


I am working on the file "Notebooks/01-Mapping-Copy1.ipynb".
The first task is:
And for the second side of the read:
Can you write Python code for it?

# for the second side of the reads
full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',
             out_map_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r2/'.format(r_enz),
             fastq_path='/media/storage/FASTQs/K562_%s_2.fastq' % (r_enz),
             r_enz=r_enz, frag_map=False,  clean=True, nthreads=20,
             windows=((1,25),(1,30),(1,35),(1,40),(1,45),(1,50),(1,55),(1,60),(1,65),(1,70),(1,75)),
             temp_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r2_tmp/'.format(r_enz))


I am working on the file "Notebooks/01-Mapping-Copy1.ipynb".
The first task is:
The full mapping function can be used to perform either iterative or fragment-based mapping, or a combination of both.
Iterative mapping
Here an example of use as iterative mapping:
Can you write Python code for it?

r_enz = 'HindIII'

! mkdir -p results/iterativ/$r_enz
! mkdir -p results/iterativ/$r_enz/01_mapping

# for the first side of the reads
full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',
             out_map_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r1/'.format(r_enz),
             fastq_path='/media/storage/FASTQs/K562_%s_1.fastq' % (r_enz),
             r_enz='hindIII', frag_map=False, clean=True, nthreads=20,
             windows=((1,25),(1,30),(1,35),(1,40),(1,45),(1,50),(1,55),(1,60),(1,65),(1,70),(1,75)),
             temp_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r1_tmp/'.format(r_enz))


I am working on the file "Notebooks/01-Mapping.ipynb".
The first task is:
Fragment-based mapping
With fragment based mapping it would be:
Can you write Python code for it?

! mkdir -p results/fragment/$r_enz
! mkdir -p results/fragment/$r_enz/01_mapping

# for the first side of the reads
full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',
             out_map_dir='results/fragment/{0}/01_mapping/mapped_{0}_r1/'.format(r_enz),
             fastq_path='/media/storage/FASTQs/K562_%s_1.fastq' % (r_enz),
             r_enz=r_enz, frag_map=True, clean=True, nthreads=20,
             temp_dir='results/fragment/{0}/01_mapping/mapped_{0}_r1_tmp/'.format(r_enz))

# for the second side of the reads
full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',
             out_map_dir='results/fragment/{0}/01_mapping/mapped_{0}_r2/'.format(r_enz),
             fastq_path='/media/storage/FASTQs/K562_%s_2.fastq' % (r_enz),
             r_enz=r_enz, frag_map=True, clean=True, nthreads=20,
             temp_dir='results/fragment/{0}/01_mapping/mapped_{0}_r2_tmp/'.format(r_enz))


I am working on the file "Notebooks/01-Mapping.ipynb".
The first task is:
The full mapping function can be used to perform either iterative or fragment-based mapping, or a combination of both.
Iterative mapping
Here an example of use as iterative mapping:
Can you write Python code for it?

r_enz = 'MboI'

! mkdir -p results/iterativ/$r_enz
! mkdir -p results/iterativ/$r_enz/01_mapping

# for the first side of the reads
full_mapping(gem_index_path='/media/storage/db/reference_genome/Homo_sapiens/hg38/hg38.gem',
             out_map_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r1/'.format(r_enz),
             fastq_path='/media/storage/FASTQs/K562_%s_1.fastq' % (r_enz),
             r_enz='hindIII', frag_map=False, clean=True, nthreads=20,
             windows=((1,25),(1,30),(1,35),(1,40),(1,45),(1,50),(1,55),(1,60),(1,65),(1,70),(1,75)),
             temp_dir='results/iterativ/{0}/01_mapping/mapped_{0}_r1_tmp/'.format(r_enz))


I am working on the file "Notebooks/01_Introduction.ipynb".
The first task is:
Basic Axioms of Linear Algebra
Taken from the Ikkipedia
Vector spaces
The main structures of linear algebra are vector spaces. A vector space over a field F (often the field of the real numbers) is a set V equipped with two binary operations satisfying the following axioms. Elements of V are called vectors, and elements of F are called scalars. The first operation, vector addition, takes any two vectors v and w and outputs a third vector v + w. The second operation, scalar multiplication, takes any scalar a and any vector v and outputs a new vector av. The operations of addition and multiplication in a vector space must satisfy the following axioms.[15] In the list below, let u, v and w be arbitrary vectors in V, and a and b scalars in F.
Axiom   Signification

Associativity of addition u + (v + w) = (u + v) + w
Commutativity of addition u + v = v + u
Identity element of addition  There exists an element 0 ∈ V, called the zero vector, such that v + 0 = v for all v ∈ V.
Inverse elements of addition  For every v ∈ V, there exists an element −v ∈ V, called the additive inverse of v, such that v + (−v) = 0
Distributivity of scalar multiplication with respect to vector addition   a(u + v) = au + av
Distributivity of scalar multiplication with respect to field addition    (a + b)v = av + bv
Compatibility of scalar multiplication with field multiplication  a(bv) = (ab)v
Identity element of scalar multiplication 1v = v, where 1 denotes the multiplicative identity in F.

The first four axioms are those of V being an abelian group under vector addition. Elements of a vector space may have various nature; for example, they can be sequences, functions, polynomials or matrices. Linear algebra is concerned with properties common to all vector spaces.
Lets see how these look in numpy
Can you write Python code for it?

# Fix vectors u, v, w
# Fix scalars a, b

u = np.array([2,3,4]).reshape(3,1)
v = np.array([-2,3,4.5]).reshape(3,1)
w = np.array([-3.2, 5, 12]).reshape(3,1)
zero = np.array([0,0,0]).reshape(3,1)

a = 3.2
b = 4

print("Check associativity:\n {} \n== \n{}\n True!".format(u + (v + w),(u + v) + w))
print("Check commutativity:\n {}\n ==\n {}".format(u + v, v + u))
print("Check addition has an identity element:\n {}\n+\n{}\n==\n{}".format(v, -v, zero))
print("You're getting the picture...")


I am working on the file "Notebooks/01_Introduction.ipynb".
The first task is:
Creating and shaping numpy arrays.
Can you write Python code for it?

# Create numpy arrays like this.

print("A 1 row by 4 column numpy matrix\n{}".format(np.array(range(4))))

# Create a 2 by 2 matrix like this.

A = np.array(range(4)).reshape(2,2) # Reshape into an array of two rows and 2 columns.
print("A 2 row by 2 column numpy matrix\n{}".format(A))

# Create a 3 by 2 matrix.

A = np.array(range(6)).reshape(3,2)
print("A 3 row by 2 column numpy matrix\n{}".format(A))

# Note the ordering of how it reshapes.  It places items item by item until the row is filled.

# Be careful to match the number of elements to the shape you give to reshape!
np.array(range(6)).reshape(2,4) # Size of the inputs must equal the product of the dimensions.


I am working on the file "Notebooks/01_Introduction.ipynb".
The first task is:
This is analagous to scalar multiplication with matrices.
2 * [[2,3],[4,5]] = [[4,6],[8,10]]
Can you write Python code for it?

# Numpy dynamically casts to floats, etc.
0.5 * A

# Careful to multiply matrices, vectors, etc with the np.matmul operation.
A = np.array(range(4)).reshape(2,2)
b = np.array([2,5]).reshape(2,1)

element_wise_multiplication = A * b
matrix_multiplication = np.matmul(A, b)

print("Element-wise multiplication:\nA .* b = \n{}\n".format(element_wise_multiplication))
print("Matrix-Multiplication:\nA * b = \n{}".format(matrix_multiplication))


I am working on the file "Notebooks/01_Introduction.ipynb".
The first task is:
Transpositions of Matrices.
Can you write Python code for it?

# Take a Matrix and use it's .transpose method.
print(A.transpose())


I am working on the file "Notebooks/01_Introduction.ipynb".
The first task is:
We will frequently write vectors as $[1,2]^T$ so we can write them in line.
Can you write Python code for it?

a = np.array([1,2]).reshape(2,1) # Or declare them like this in numpy.
print(a)


I am working on the file "Notebooks/01_Mnist-Test.ipynb".
The first task is:
Data load
Can you write Python code for it?

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

u.plot_images(X_train[0:9], y_train[0:9])

if K.image_dim_ordering() == 'th':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)


I am working on the file "Notebooks/01_Mnist-Test.ipynb".
The first task is:
Definitions
Can you write Python code for it?

batch_size = 512
nb_classes = 10
nb_epoch = 25
# path of the model graph
model_image_path = 'images/model_01_MNIST.png'

# input image dimensions
img_rows, img_cols = 28, 28
# number of convolutional filters to use
nb_filters = 32
# size of pooling area for max pooling
pool_size = (2, 2)
# convolution kernel size
kernel_size = (3, 3)
# dense layer size
dense_layer_size = 128


I am working on the file "Notebooks/01_Mnist-Test.ipynb".
The first task is:
Training and evaluation
Can you write Python code for it?

history = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=1, validation_data=(X_test, Y_test))

score = model.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

u.plot_history(history)
u.plot_history(history, metric='loss', loc='upper left')


I am working on the file "Notebooks/02-Parsing_mapped_reads-HindIII.ipynb".
The first task is:
Extract uniquely mapped reads
Can you write Python code for it?

# replicate and enzyme useds
r_enz = 'HindIII'


I am working on the file "Notebooks/02-Parsing_mapped_reads-HindIII.ipynb".
The first task is:
Load all reads, check if they are uniquely mapped. Result is stored in two separate file tab-separated files that will contain the essential information of each read
Can you write Python code for it?

from pytadbit.parsers.map_parser import parse_map

! mkdir -p results/iterativ/$r_enz/02_parsing

parse_map(maps1, maps2,
          'results/iterativ/{0}/02_parsing/reads1_{0}.tsv'.format(r_enz),
          'results/iterativ/{0}/02_parsing/reads2_{0}.tsv'.format(r_enz),
          genome_seq=genome_seq, re_name=r_enz verbose=True)

from pytadbit.mapping.analyze import plot_iterative_mapping

total_reads = 10000000

lengths = plot_iterative_mapping(
    'results/iterativ/{0}/02_parsing/reads1_{0}.tsv'.format(r_enz),
    'results/iterativ/{0}/02_parsing/reads2_{0}.tsv'.format(r_enz),
    total_reads)


I am working on the file "Notebooks/02-Parsing_mapped_reads-HindIII.ipynb".
The first task is:
Then, we search for restriction enzyme sites in this sequence.
Can you write Python code for it?

from pytadbit.mapping.restriction_enzymes import map_re_sites

frags_HindIII = map_re_sites('HindIII', genome_seq, verbose=True)
frags_MboI    = map_re_sites('MboI'   , genome_seq, verbose=True)

c0 = frags_HindIII.keys()[0]
v0 = 0
dists_HindIII = []
for c1, v1 in [(c, v) for c in frags_HindIII for p in frags_HindIII[c]
               for v in frags_HindIII[c][p]]:
    if c1 == c0:
        diff = abs(v1 - v0)
        if diff < 30000:  # larger fragments are probably poorly assembled regions
            dists_HindIII.append(diff)
    v0 = v1
c0 = frags_MboI.keys()[0]
v0 = 0
dists_MboI = []
for c1, v1 in [(c, v) for c in frags_MboI for p in frags_MboI[c]
               for v in frags_MboI[c][p]]:
    if c1 == c0:
        diff = abs(v1 - v0)
        if diff < 30000:  # larger fragments are probably poorly assembled regions
            dists_MboI.append(diff)
    v0 = v1

plt.figure(figsize=(12, 6))
_ = plt.hist([dists_HindIII, dists_MboI], bins=50, color=['red', 'green'],
             alpha=0.5, label=['HindIII', 'MboI'], normed=True, range=(0, 10000))
y0, y1 = plt.ylim()
plt.vlines(np.median(dists_HindIII), y0, y1 * 0.7, color='red', linestyle='--',
          label='median HindIII')
plt.vlines(np.median(dists_MboI)   , y0, y1 * 0.75, color='green', linestyle='--',
          label='median MboI')
plt.text(np.median(dists_HindIII), y1 * 0.7, '%.0f' % np.median(dists_HindIII), va='bottom',
         ha='left')
plt.text(np.median(dists_MboI)   , y1 * 0.75,'%.0f' % np.median(dists_MboI)   , va='bottom',
         ha='left')
plt.title('Restriction fragment size distribution')
plt.legend()
plt.xlim(0, 10000)
plt.grid()


I am working on the file "Notebooks/02-Parsing_mapped_reads-HindIII.ipynb".
The first task is:
We are working with a cell line with a very aberrant karyotype (https://commons.wikimedia.org/wiki/File:Karyotype_of_the_T47D_breast_cancer_cell_line.svg).
First Hi-C map
Can you write Python code for it?

from pytadbit.mapping.analyze import hic_map

hic_map(reads, resolution=1000000, show=True)


I am working on the file "Notebooks/02-Parsing_mapped_reads-HindIII.ipynb".
The first task is:
We have loaded the full human genome:
Can you write Python code for it?

plt.bar(range(len(genome_seq)), [len(genome_seq[c]) for c in genome_seq],
        color='k', alpha=0.7)
_ = plt.xticks([v + 0.5 for v in range(len(genome_seq))], genome_seq.keys(), rotation=90)
plt.grid()


I am working on the file "Notebooks/02_Inverse_Matrix_Theorem.ipynb".
The first task is:
Hmm... Interesting.
The Determinant of a matrix is just the product of it's eigenvalues.
Can you write Python code for it?

vals, vecs = la.eig(A)
print(vecs)

# Lets define a distance function for vectors in a matrix.
def l2_distance_cols(matrix):
    norms = []
    for row in matrix.transpose():
        dist = 0
        for val in row:
            dist += val ** 2

        norms.append(dist)

    return np.array(norms)

l2_distance_cols(vecs)


I am working on the file "Notebooks/02_Inverse_Matrix_Theorem.ipynb".
The first task is:
Hmm.... is this related to anything??
Can you write Python code for it?

# Let's check it's eigenvalues.
print("The Matrix A has eigenvalues: {}".format([x for x in la.eigvals(A)]))
print("And their product is: {}".format(np.product(la.eigvals(A))))
determinant_A - np.product(la.eigvals(A)) < 10**-5

# Lets define an epsilon and check if they are the same.
eps = 10 ** -5
def check_floats_equal(float_1, float_2, eps=eps):
    return float_1 - float_2 < eps

check_floats_equal(np.product(la.eigvals(A)), la.det(A))


I am working on the file "Notebooks/02_Inverse_Matrix_Theorem.ipynb".
The first task is:
It looks like the eigenvectors are normalized to length 1.
Let's look at another matrix and look at its eigenvalues and eigenvectors.  This one should be a bit simpler.
Can you write Python code for it?

A = np.array([0, -1, 1, 0]).reshape(2, 2)

A

np.linalg.det(A)

vals, vecs = np.linalg.eig(A)
print("Values:\n{}".format(vals))
print("Vectors:\n{}".format(vecs))


I am working on the file "Notebooks/02_Inverse_Matrix_Theorem.ipynb".
The first task is:
This looks like a twist to me.  It's sending
$$ [1, 0]^T \mapsto [0,1]^T$$
$$ [0, 1]^T \mapsto [-1,0]^T$$
If it's a twist... how can it have a direction it stretches something in a straight line in... we'd have to imagine really hard...
Oh.
That's why it has imaginary eigenvectors -- they don't really exist.  But they do symbolically and mathematically.
Well... according to the inverse matrix theorem, the fact that $Det(A) = 1 \neq 0$ implies that it has an inverse matrix.  Lets check that out.
Can you write Python code for it?

A_inv = np.linalg.inv(A)
print("Here is A^-1\n{}".format(A_inv))

np.matmul(A_inv, list_to_vector([1,0]))

np.matmul(A_inv, list_to_vector([0,1]))


I am working on the file "Notebooks/02_Inverse_Matrix_Theorem.ipynb".
The first task is:
We just get the identity matrix back -- just like we wanted!!
Lets take another example.
Fix a new matrix $A = 2 * I_2$
Can you write Python code for it?

A = np.eye(2) * 2
print(A)

vals, vecs = np.linalg.eig(A)
print("Vals:\n{}".format(vals))
print("Vecs:\n{}".format(vecs))


I am working on the file "Notebooks/02_Mnist.ipynb".
The first task is:
Confusion matrix (relu)
Can you write Python code for it?


I am working on the file "Notebooks/02_Mnist.ipynb".
The first task is:
Data load
Can you write Python code for it?


I am working on the file "Notebooks/02_Mnist.ipynb".
The first task is:
Examples of correct predictions (relu)
Can you write Python code for it?

u.plot_images(X_test[correct_indices_relu[:9]], y_test[correct_indices_relu[:9]],
              predicted_classes_relu[correct_indices_relu[:9]])


I am working on the file "Notebooks/02_Mnist.ipynb".
The first task is:
Examples of incorrect predictions (tanh)
Can you write Python code for it?

u.plot_images(X_test[incorrect_indices_tanh[:9]], y_test[incorrect_indices_tanh[:9]],
              predicted_classes_tanh[incorrect_indices_tanh[:9]])


I am working on the file "Notebooks/02_Mnist.ipynb".
The first task is:
Inspecting the result
Can you write Python code for it?

# The predict_classes function outputs the highest probability class
# according to the trained classifier for each input example.
predicted_classes_tanh = model_tanh.predict_classes(X_test)
predicted_classes_relu = model_relu.predict_classes(X_test)

# Check which items we got right / wrong
correct_indices_tanh = np.nonzero(predicted_classes_tanh == y_test)[0]
incorrect_indices_tanh = np.nonzero(predicted_classes_tanh != y_test)[0]

correct_indices_relu = np.nonzero(predicted_classes_relu == y_test)[0]
incorrect_indices_relu = np.nonzero(predicted_classes_relu != y_test)[0]


I am working on the file "Notebooks/02_Mnist.ipynb".
The first task is:
Model definition
The model is structurally similar to the previous one, with 2 Convolutional layers and 1 Fully conneted layers.
However there are major difference in values and sizes, and also there is one more intermediate max pooling layer and the activation function is a scaled hyperbolic tangent, as described in the paper. However, since Rectified Linear Units started spreading after 2015, we are going to compare two different CNN, one using tanh (as in the paper) and the other one using relu.
1x29x29-20C4-MP2-40C5-MP3-150N-10N DNN.
<img src="images/cvpr2012.PNG" alt="1x29x29-20C4-MP2-40C5-MP3-150N-10N DNN" style="width: 400px;"/>
The paper doesn't seem to use any dropout layer to avoid overfitting, so we're going to use a dropout of 0.15, way lower then we did before.
It is also worth mentioning that the authors of the paper have their methods to avoid overfitting, like dataset expansion by adding translations, rotations and deformations to the images of the training set.
Can you write Python code for it?

model_tanh = Sequential()
model_relu = Sequential()

def initialize_network_with_activation_function(model, activation, checkpoints_filepath):
    model.add(Convolution2D(nb_filters1, kernel_size1[0], kernel_size1[1],
                            border_mode='valid',
                            input_shape=input_shape, name='covolution_1_' + str(nb_filters1) + '_filters'))
    model.add(Activation(activation, name='activation_1_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size1, name='max_pooling_1_' + str(pool_size1) + '_pool_size'))
    model.add(Convolution2D(nb_filters2, kernel_size2[0], kernel_size2[1]))
    model.add(Activation(activation, name='activation_2_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size2, name='max_pooling_1_' + str(pool_size2) + '_pool_size'))
    model.add(Dropout(dropout))

    model.add(Flatten())
    model.add(Dense(dense_layer_size1, name='fully_connected_1_' + str(dense_layer_size1) + '_neurons'))
    model.add(Activation(activation, name='activation_3_' + activation))
    model.add(Dropout(dropout))
    model.add(Dense(nb_classes, name='output_' + str(nb_classes) + '_neurons'))
    model.add(Activation('softmax', name='softmax'))

    model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy', 'precision', 'recall', 'mean_absolute_error'])

    # loading weights from checkpoints
    if os.path.exists(checkpoints_filepath):
        model.load_weights(checkpoints_filepath)

initialize_network_with_activation_function(model_tanh, 'tanh', checkpoints_filepath_tanh)
initialize_network_with_activation_function(model_relu, 'relu', checkpoints_filepath_relu)

Image(u.maybe_save_network(model_relu, model_image_path), width=300)


I am working on the file "Notebooks/02_Mnist.ipynb".
The first task is:
Training and evaluation
Using non verbose output for training, since we already get some informations from the callback.
Can you write Python code for it?

# checkpoint
checkpoint_tanh = ModelCheckpoint(checkpoints_filepath_tanh, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list_tanh = [checkpoint_tanh]

# training
print('training tanh model')
history_tanh = model_tanh.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=0, validation_data=(X_test, Y_test), callbacks=callbacks_list_tanh)
# evaluation
print('evaluating tanh model')
score = model_tanh.evaluate(X_test, Y_test, verbose=1)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')

u.plot_history(history_tanh)
u.plot_history(history_tanh, metric='loss', loc='upper left')

# checkpoint
checkpoint_relu = ModelCheckpoint(checkpoints_filepath_relu, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list_relu = [checkpoint_relu]

# training
print('training relu model')
history_relu = model_relu.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=0, validation_data=(X_test, Y_test), callbacks=callbacks_list_relu)

# evaluation
print('evaluating relu model')
score = model_relu.evaluate(X_test, Y_test, verbose=1)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')

u.plot_history(history_relu)
u.plot_history(history_relu, metric='loss', loc='upper left')


I am working on the file "Notebooks/02_Mnist.ipynb".
The first task is:
We are using TensorFlow-GPU 0.12.1 on Python 3.5.2, running on Windows 10 with Cuda 8.0.
We have 3 machines with the same environment and 3 different GPUs, respectively with 384, 1024 and 1664 Cuda cores.
Imports
Can you write Python code for it?

import os.path
from IPython.display import Image

from util import Util
u = Util()

import numpy as np
# Explicit random seed for reproducibility
np.random.seed(1337)

from keras.callbacks import ModelCheckpoint
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras import backend as K

from keras.datasets import mnist


I am working on the file "Notebooks/03-Filtering_mapped_reads.ipynb".
The first task is:
Evaluate changes
Can you write Python code for it?

from pytadbit.mapping.analyze import hic_map

r_enz = 'HindIII'
hic_map('results/fragment/{0}/03_filtering/valid_reads12_{0}.tsv'.format(r_enz),
        resolution=1000000, show=True)

r_enz = 'MboI'
hic_map('results/fragment/{0}/03_filtering/valid_reads12_{0}.tsv'.format(r_enz),
        resolution=1000000, show=True)


I am working on the file "Notebooks/03-Filtering_mapped_reads.ipynb".
The first task is:
The max_molecule_length parameter used to filter-out pseudo-dangling-ends can be extracted from the previous section in the computation of insert size.
The min_distance_to_re, that affects the detection of random breaks, should be a bit larger in order to contain almost all the fragments.
Can you write Python code for it?

! mkdir -p results/HindIII/03_filtering
! mkdir -p results/MboI/03_filtering

r_enz = 'HindIII'

# this will last ~10 minutes
masked1 = filter_reads(
    'results/{0}/03_filtering/reads12_{0}.tsv'.format(r_enz),
    max_molecule_length=750, over_represented=0.005, max_frag_size=100000,
    min_frag_size=100, re_proximity=5, min_dist_to_re=1000)

r_enz = 'MboI'

# this will last ~10 minutes
masked2 = filter_reads(
    'results/{0}/03_filtering/reads12_{0}.tsv'.format(r_enz),
    max_molecule_length=750, over_represented=0.005, max_frag_size=100000,
    min_frag_size=100, re_proximity=5, min_dist_to_re=1000)


I am working on the file "Notebooks/03-Filtering_mapped_reads.ipynb".
The first task is:
This generates a dictionary with the different filters and the reads affected by each.
The filters are:
1. self-circle: reads are coming from a single RE fragment and point to the outside (—-<===—===>—)
2. dangling-end: reads are coming from a single RE fragment and point to the inside (—-===>—<===—)
3. error: reads are coming from a single RE fragment and point in the same direction
4. extra dangling-end: reads are coming from different RE fragment but are close enough (< Maximum molecule length) and point to the inside. Maximum molecule length parameter depends on the size of the sequenced fragments.
5. too close from REs: semi-dangling-end filter, start position of one of the read is too close (5 bp by default) from RE cutting site (with 4 base-pair-cutter enzyme it can be set to 4 nt). This filter is in general not taken into account in in-situ Hi-C experiments, and with 4bp cutters as the ligation may happen only one side of the DNA fragments.
6. too short: remove reads coming from small restriction less than 100 bp (default) because they are comparable to the read length, and are thus probably artifacts.
7. too large: remove reads coming from large restriction fragments (default: 100 Kb, P < 10-5 to occur in a randomized genome) as they likely represent poorly assembled or repeated regions
8. over-represented: reads coming from the top 0.5% most frequently detected restriction fragments, they may be prone to PCR artifacts or represent fragile regions of the genome or genome assembly errors
9. duplicated: the combination of the start positions (and direction) of the reads is repeated -> PCR artifact (only keep one copy)
10. random breaks: start position of one of the read is too far (more than Minimum distance to RE site) from RE cutting site. Non-canonical enzyme activity or random physical breakage of the chromatin.
Apply filters on the data
Can you write Python code for it?

from pytadbit.mapping.filter import apply_filter

r_enz = 'HindIII'
apply_filter('results/fragment/{0}/03_filtering/reads12_{0}.tsv'.format(r_enz),
             'results/fragment/{0}/03_filtering/valid_reads12_{0}.tsv'.format(r_enz), masked1,
             filters=[1, 2, 3, 4, 9, 10])

r_enz = 'MboI'
apply_filter('results/fragment/{0}/03_filtering/reads12_{0}.tsv'.format(r_enz),
             'results/fragment/{0}/03_filtering/valid_reads12_{0}.tsv'.format(r_enz), masked2,
             filters=[1, 2, 3, 4, 9, 10])


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Confusion matrix (56)
Can you write Python code for it?


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Confusion matrix (800)
Can you write Python code for it?


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Data load
Can you write Python code for it?

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

u.plot_images(X_train[0:9], y_train[0:9])

if K.image_dim_ordering() == 'th':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Examples of correct predictions (new)
Can you write Python code for it?

u.plot_images(X_test[correct_indices_new[:9]], y_test[correct_indices_new[:9]],
              predicted_classes_new[correct_indices_new[:9]])


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Examples of incorrect predictions (56)
Can you write Python code for it?

u.plot_images(X_test[incorrect_indices_56[:9]], y_test[incorrect_indices_56[:9]],
              predicted_classes_56[incorrect_indices_56[:9]])


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Examples of incorrect predictions (800)
Can you write Python code for it?

u.plot_images(X_test[incorrect_indices_800[:9]], y_test[incorrect_indices_800[:9]],
              predicted_classes_800[incorrect_indices_800[:9]])


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Examples of incorrect predictions (new)
Can you write Python code for it?

u.plot_images(X_test[incorrect_indices_new[:9]], y_test[incorrect_indices_new[:9]],
              predicted_classes_new[incorrect_indices_new[:9]])


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Image preprocessing
As said in the introduction, we're going to apply random transformations: rotation with a window of 40 degrees and both vertical and horizontal shifts, zoom and scale with a range of 10% (so about 3 pixels more or 3 pixels less). We avoid flips and rotations that would basically alter the meaning of the symbol.
Can you write Python code for it?

datagen = ImageDataGenerator(
    rotation_range=30,
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.1,
    horizontal_flip=False)

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(X_train)


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Model definition
Can you write Python code for it?

model_800 = Sequential()
model_56 = Sequential()

model_new = Sequential()

def initialize_network(model, checkpoints_filepath, dropout1=dropout, dropout2=dropout, dense_layer_size1=dense_layer_size1):
    model.add(Convolution2D(nb_filters1, kernel_size1[0], kernel_size1[1],
                            border_mode='valid',
                            input_shape=input_shape, name='covolution_1_' + str(nb_filters1) + '_filters'))
    model.add(Activation(activation, name='activation_1_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size1, name='max_pooling_1_' + str(pool_size1) + '_pool_size'))
    model.add(Convolution2D(nb_filters2, kernel_size2[0], kernel_size2[1]))
    model.add(Activation(activation, name='activation_2_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size2, name='max_pooling_1_' + str(pool_size2) + '_pool_size'))
    model.add(Dropout(dropout))

    model.add(Flatten())
    model.add(Dense(dense_layer_size1, name='fully_connected_1_' + str(dense_layer_size1) + '_neurons'))
    model.add(Activation(activation, name='activation_3_' + activation))
    model.add(Dropout(dropout))
    model.add(Dense(nb_classes, name='output_' + str(nb_classes) + '_neurons'))
    model.add(Activation('softmax', name='softmax'))

    model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy', 'precision', 'recall', 'mean_absolute_error'])

    # loading weights from checkpoints
    if os.path.exists(checkpoints_filepath):
        model.load_weights(checkpoints_filepath)
    else:
        print('Warning: ' + checkpoints_filepath + ' could not be loaded')

initialize_network(model_800, checkpoints_filepath_800)
initialize_network(model_56, checkpoints_filepath_56)

initialize_network(model_new, checkpoints_filepath_new, dense_layer_size1_new)


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
This part trains the new network (the one using image pre-processing) and then we output the scores.
We are going to use 800 epochs divided between the pre-processed images and the original ones.
Can you write Python code for it?

# checkpoint
checkpoint_new = ModelCheckpoint(checkpoints_filepath_new, monitor='val_precision', verbose=1, save_best_only=True, mode='max')
callbacks_list_new = [checkpoint_new]

# fits the model on batches with real-time data augmentation, for nb_epoch-100 epochs
history_new = model_new.fit_generator(datagen.flow(X_train, Y_train,
                                                   batch_size=batch_size,
                                                  # save_to_dir='distorted_data',
                                                  # save_format='png'
                                                   seed=1337),
                        samples_per_epoch=len(X_train), nb_epoch=nb_epoch-25, verbose=0,
                        validation_data=(X_test, Y_test), callbacks=callbacks_list_new)

# ensuring best val_precision reached during training
model_new.load_weights(checkpoints_filepath_new)


I am working on the file "Notebooks/03_Mnist-Dataset-Expansion.ipynb".
The first task is:
Training and evaluation
First the evaluations for the network of the previous notebook, with 800 and 56 epochs of training.
Can you write Python code for it?

# evaluation
print('evaluating 800 epochs model')
score = model_800.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')

print('evaluating 56 epochs model')
score = model_56.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])
print('Test error:', (1-score[2])*100, '%')


I am working on the file "Notebooks/03_Regression_and_PCA.ipynb".
The first task is:
Discussion
It seems like we can translate them between the two different representations without a problem.
Thats good enough for me!
We can consider the non-square matrix $A$ of shape $(5, 10)$ to have 5 eigenvalues.
WE ONLY NEED A NUMBER OF EIGENVALUES EQUAL TO THE NUMBER OF FEATURES!!!!
We actually do not have to summarize our data as much as it seems like we might.
From here on out we need to define a few statistical Matrices.  These will be left as challenges
Define a mean vector $\vec \mu$ derived from the original A, which is a vector of the mean of each of the original features.
Define a matrix B (m, n), such that each element of B is A - $\vec \mu$ applied to each sample.
Define the covariance matrix $S \colon = \frac{1}{n-1} B \times B^T$.
Can you write Python code for it?

# Why should the covariance matrix be a square matrix in the number of features?


I am working on the file "Notebooks/03_Regression_and_PCA.ipynb".
The first task is:
PCA
Can you write Python code for it?


I am working on the file "Notebooks/03_Regression_and_PCA.ipynb".
The first task is:
PCA Algorithm Basics
The PCA Algorithm relies heavily on the Spectral (eigenvalue) related properties of a matrix.
Dumb question $ -$ what are the eigenvalues of a non-square matrix?
Recall that we can think of $m$ features of $n$ samples as being represented by a $(m, n)$ matrix.
Can you write Python code for it?

# Let us see what this would look like in numpy.

# First make choose m and n such that m != n
m = 5
n = 10

# Make the matrix A
A = np.random.rand(m, n)
print(A)

# Now compute its eigenvalues.
try:
    vals, vecs = np.linalg.eig(A)
    print(vals)
except:
    print("Uh Oh we caused a linear algebra error!")
    print("The last two dimensions must be square!")
    print("This means we can't compute the eigenvalues of the matrix.")


I am working on the file "Notebooks/03_Regression_and_PCA.ipynb".
The first task is:
The Trace of a Matrix
$$Tr(A) = \Sigma \lambda_i$$
The trace is the sum of the eigenvalues of a matrix.
It can be alternatively stated as the sum of the values on the diagonal, but this is not obvious!!
In our case, we know the values on the Diagonal are the Variance for the feature in that column/row.
Therefore the $Tr(S)$ is just the total variance in the data set!
The eigenvectors $\vec v_i $ are the directions of maximum variance.
The eigenvalues are the amount of variance in that direction.
Can you write Python code for it?

def gen_noisy_line(n_samples=50):
    '''
    This function generates a noisy line of slope 1 and returns the
    matrix associated with these n_samples, with noise +- 1 from a
    straight line.

    This matrix follows the convention that
    rows are features, and columns are samples.
    '''

    return matrix_A

def make_B_from_A(matrix_A):
    '''
    This function generates the B matrix from the sample matrix A.
    '''

    return matrix_B

def make_S_from_B(matrix_B):
    '''
    This function generates the matrix S from B.
    '''

    return matrix_S


I am working on the file "Notebooks/03_Regression_and_PCA.ipynb".
The first task is:
This.... actually makes sense.  Lets compare the other multiplication just to see whats going on.
Can you write Python code for it?

AT_A = np.matmul(A_T, A)
vals, vecs = np.linalg.eig(AT_A)
print(vals)


I am working on the file "Notebooks/04-Bin-filtering_and_normalization-1Mb.ipynb".
The first task is:
Dryhic normalization
We can launch the external script to use dryhic normalization
Can you write Python code for it?

! dryhic3.r results/HindIII/03_filtering/valid_reads12_HindIII.tsv \
       results/HindIII/04_normalizing/bad_columns_1000000_HindIII.pick \
       HindIII hg38 1000000 results/HindIII/04_normalizing/biases_dryhic_1000000_HindIII.pick

! dryhic3.r results/MboI/03_filtering/valid_reads12_MboI.tsv \
       results/MboI/04_normalizing/bad_columns_1000000_MboI.pick \
       MboI hg38 1000000 results/MboI/04_normalizing/biases_dryhic_1000000_MboI.pick

bias_dry_path = 'results/{1}/04_normalizing/biases_dryhic_{0}_{1}.tsv'

hic_data_1.bias = dict([(int(l.split()[0]), float(l.split()[1])) for l in open(bias_dry_path.format(reso, 'HindIII'))])
hic_data_2.bias = dict([(int(l.split()[0]), float(l.split()[1])) for l in open(bias_dry_path.format(reso, 'MboI'))])

hic_map(hic_data_1, normalized=True, focus='chr18', show=True)
hic_map(hic_data_2, normalized=True, focus='chr18', show=True)


I am working on the file "Notebooks/04-Bin-filtering_and_normalization-1Mb.ipynb".
The first task is:
If the resolution is not to low, we could also save genomic matrices:
Can you write Python code for it?

if reso >= 300000:
    hic_map(hic_data_1, by_chrom=False, normalized=False,
            savedata='results/{1}/04_normalizing/{0}_raw.mat'.format(reso, r_enz_1))

    hic_map(hic_data_1, by_chrom=False, normalized=True,
            savedata='results/{1}/04_normalizing/{0}_norm.mat'.format(reso, r_enz_1))
    hic_map(hic_data_2, by_chrom=False, normalized=False,
            savedata='results/{1}/04_normalizing/{0}_raw.mat'.format(reso, r_enz_2))

    hic_map(hic_data_2, by_chrom=False, normalized=True,
            savedata='results/{1}/04_normalizing/{0}_norm.mat'.format(reso, r_enz_2))


I am working on the file "Notebooks/04-Bin-filtering_and_normalization-1Mb.ipynb".
The first task is:
Normalization
As normalization we use either the ICE normalization (Imakaev et al., 2012) with a "full" normalization until the sum of all columns of the matrix are equal, or something more similar to the vanilla normalization used in (Rao et al., 2014) which is exactly like running the ICE normalization without iterations.
Note: if columns with a lot of zeroes are present the ICE normalization will last very long to converge, and these low-coverage columns will present, at the end of the normalization, few cells with very high values of interaction
Can you write Python code for it?

hic_data_1.normalize_hic(iterations=10, max_dev=0.1)
hic_data_2.normalize_hic(iterations=10, max_dev=0.1)


I am working on the file "Notebooks/04-Bin-filtering_and_normalization-1Mb.ipynb".
The first task is:
Other normalizations
ICE normalization is widely used however other, more convaluted, normalizations (Hu et al., 2012) (Yaffe and Tanay, 2011) can be used outside TADbit and then added as normalized matrices.
Save raw and normalized matrices
Save biases and bin filtering
Can you write Python code for it?

from cPickle import dump

! mkdir -p results/$r_enz_1/04_normalizing
! mkdir -p results/$r_enz_2/04_normalizing


I am working on the file "Notebooks/04-Bin-filtering_and_normalization-1Mb.ipynb".
The first task is:
Save normalized chromosome/genome matrices
This time we do not need to save appart the normalization biases and the list of columns with poor signal.
Can you write Python code for it?

hic_map(hic_data_1, by_chrom='intra', normalized=False,
       savedata='results/{1}/04_normalizing/{0}_raw'.format(reso, r_enz_1))
hic_map(hic_data_2, by_chrom='intra', normalized=False,
       savedata='results/{1}/04_normalizing/{0}_raw'.format(reso, r_enz_2))

hic_map(hic_data_1, by_chrom='intra', normalized=True,
       savedata='results/{1}/04_normalizing/{0}_norm'.format(reso, r_enz_1))
hic_map(hic_data_2, by_chrom='intra', normalized=True,
       savedata='results/{1}/04_normalizing/{0}_norm'.format(reso, r_enz_2))


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
At 100 kb itmakes no sense to view the full matrix (your matrix will have more cells than your screen have pixels).
Try plotting a region of the genome only, with the focus parameter.
Can you write Python code for it?

from pytadbit.mapping.analyze import hic_map

hic_map(hic_data, normalized=True, focus='chr18', show=True)


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
Filter columns with too few interaction count
For this, very sparse, example dataset we are going to ask for very few interactions per bin.
This can be done either by setting min_perc asking for each bin to contain a minimum percentage of cells with interaction data. Or but setting min_count asking that the number of cells with interaction data of each bin is above the defined cutoff.
Can you write Python code for it?

hic_data.filter_columns(draw_hist=True, min_count=10, by_mean=True)

print len(hic_data)
print len(hic_data.bads)


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
If the resolution is not to low, we could also save genomic matrices:
Can you write Python code for it?

if reso >= 300000:
    hic_map(hic_data, by_chrom=False, normalized=False,
            savedata='results/fragment/{1}/04_normalizing/{0}_raw.mat'.format(reso, r_enz))

    hic_map(hic_data, by_chrom=False, normalized=True,
            savedata='results/fragment/{1}/04_normalizing/{0}_norm.mat'.format(reso, r_enz))


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
Normalization
As normalization we use either the ICE normalization (Imakaev et al., 2012) with a "full" normalization until the sum of all columns of the matrix are equal, or something more similar to the vanilla normalization used in (Rao et al., 2014) which is exactly like running the ICE normalization without iterations.
Note: if columns with a lot of zeroes are present the ICE normalization will last very long to converge, and these low-coverage columns will present, at the end of the normalization, few cells with very high values of interaction
Can you write Python code for it?


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
Other normalizations
ICE normalization is widely used however other, more convaluted, normalizations (Hu et al., 2012) (Yaffe and Tanay, 2011) can be used outside TADbit and then added as normalized matrices.
Save raw and normalized matrices
Save biases and bin filtering
Can you write Python code for it?

from cPickle import dump

! mkdir -p results/fragment/$r_enz/04_normalizing


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
Save "bad" columns to an other file
Can you write Python code for it?

out = open('results/fragment/{1}/04_normalizing/bad_columns_{0}_{1}.pick'.format(reso, r_enz), 'w')
dump(hic_data.bads, out)
out.close()


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
Save biases to separate file.
Can you write Python code for it?

out = open('results/fragment/{1}/04_normalizing/biases_{0}_{1}.pick'.format(reso, r_enz), 'w')
dump(hic_data.bias, out)
out.close()


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
Save normalized chromosome/genome matrices
This time we do not need to save appart the normalization biases and the list of columns with poor signal.
Can you write Python code for it?

hic_map(hic_data, by_chrom='intra', normalized=False,
       savedata='results/fragment/{1}/04_normalizing/{0}_raw'.format(reso, r_enz))

hic_map(hic_data, by_chrom='intra', normalized=True,
       savedata='results/fragment/{1}/04_normalizing/{0}_norm'.format(reso, r_enz))


I am working on the file "Notebooks/04-Bin-filtering_and_normalization.ipynb".
The first task is:
Save the sum of bins for Quique
Can you write Python code for it?

bins = []
for i in range(len(hic_data)):
    bins.append(sum(hic_data[i, j] for j in range(len(hic_data))))

out = open('results/fragment/{1}/04_normalizing/biases_{0}_{1}.tsv'.format(reso, r_enz), 'w')
for val in bins:
    out.write(str(val) + '\n')
out.close()


I am working on the file "Notebooks/04_Mnist-Ensemble.ipynb".
The first task is:
Data load
Can you write Python code for it?

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

u.plot_images(X_train[0:9], y_train[0:9])

if K.image_dim_ordering() == 'th':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)


I am working on the file "Notebooks/04_Mnist-Ensemble.ipynb".
The first task is:
Evaluation - Multi column
Can you write Python code for it?

print('Evaluating ensemble')
score = merged_model.evaluate([np.asarray(X_test)] * number_of_models,
                              Y_test,
                              verbose=0)
print('Test accuracy:', score[1]*100, '%')
print('Test error:', (1-score[2])*100, '%')


I am working on the file "Notebooks/04_Mnist-Ensemble.ipynb".
The first task is:
Image preprocessing
Can you write Python code for it?

datagen = ImageDataGenerator(
    rotation_range=30,
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.1,
    horizontal_flip=False)

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(X_train)


I am working on the file "Notebooks/04_Mnist-Ensemble.ipynb".
The first task is:
Just by the different seeds, error changes from 0.5% to 0.42% (our best result so far with a single column). The training took 12 hours.
Model definition - Multi column
The MCDNN is obtained by creating a new model that only has 1 layer, Merge, that does the average of the outputs of the models in the given list. No training is required since we're only doing the average.
Can you write Python code for it?

merged_model = Sequential()
merged_model.add(Merge(models, mode='ave'))

merged_model.compile(loss='categorical_crossentropy',
                     optimizer='adadelta',
                     metrics=['accuracy', 'precision', 'recall'])


I am working on the file "Notebooks/04_Mnist-Ensemble.ipynb".
The first task is:
Model definition - Single column
This time we are going to define a helper functions to initialize the model, since we're going to use it on a list of models.
Can you write Python code for it?

def initialize_network(model, dropout1=dropout, dropout2=dropout):
    model.add(Convolution2D(nb_filters1, kernel_size1[0], kernel_size1[1],
                            border_mode='valid',
                            input_shape=input_shape, name='covolution_1_' + str(nb_filters1) + '_filters'))
    model.add(Activation(activation, name='activation_1_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size1, name='max_pooling_1_' + str(pool_size1) + '_pool_size'))
    model.add(Convolution2D(nb_filters2, kernel_size2[0], kernel_size2[1]))
    model.add(Activation(activation, name='activation_2_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size2, name='max_pooling_1_' + str(pool_size2) + '_pool_size'))
    model.add(Dropout(dropout))

    model.add(Flatten())
    model.add(Dense(dense_layer_size1, name='fully_connected_1_' + str(dense_layer_size1) + '_neurons'))
    model.add(Activation(activation, name='activation_3_' + activation))
    model.add(Dropout(dropout))
    model.add(Dense(nb_classes, name='output_' + str(nb_classes) + '_neurons'))
    model.add(Activation('softmax', name='softmax'))

    model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy', 'precision', 'recall'])

# pseudo random generation of seeds
seeds = np.random.randint(10000, size=number_of_models)

# initializing all the models
models = [None] * number_of_models

for i in range(number_of_models):
    models[i] = Sequential()
    initialize_network(models[i])


I am working on the file "Notebooks/04_Mnist-Ensemble.ipynb".
The first task is:
The error improved from 0.42% with the best network of the ensemble, to 0.4%, that is out best result so far.
Can you write Python code for it?

# The predict_classes function outputs the highest probability class
# according to the trained classifier for each input example.
predicted_classes = merged_model.predict_classes([np.asarray(X_test)] * number_of_models)

# Check which items we got right / wrong
correct_indices = np.nonzero(predicted_classes == y_test)[0]
incorrect_indices = np.nonzero(predicted_classes != y_test)[0]

u.plot_images(X_test[correct_indices[:9]], y_test[correct_indices[:9]],
              predicted_classes[correct_indices[:9]])

u.plot_images(X_test[incorrect_indices[:9]], y_test[incorrect_indices[:9]],
              predicted_classes[incorrect_indices[:9]])

u.plot_confusion_matrix(y_test, nb_classes, predicted_classes)


I am working on the file "Notebooks/04_Mnist-Ensemble.ipynb".
The first task is:
Training and evaluation - Single column
Again we are going to define a helper functions to train the model, since we're going to use them on a list.
Can you write Python code for it?

def try_load_checkpoints(model, checkpoints_filepath, warn=False):
    # loading weights from checkpoints
    if os.path.exists(checkpoints_filepath):
        model.load_weights(checkpoints_filepath)
    elif warn:
        print('Warning: ' + checkpoints_filepath + ' could not be loaded')

def fit(model, checkpoints_name='test', seed=1337, initial_epoch=0,
        verbose=1, window_size=(-1), plot_history=False, evaluation=True):

    if window_size == (-1):
        window = 1 + np.random.randint(14)
    else:
        window = window_size
    if window >= nb_epoch:
        window = nb_epoch - 1

    print("Not pre-processing " + str(window) + " epoch(s)")

    checkpoints_filepath = os.path.join(checkpoints_dir, '04_MNIST_weights.best_' + checkpoints_name + '.hdf5')

    try_load_checkpoints(model, checkpoints_filepath, True)

    # checkpoint
    checkpoint = ModelCheckpoint(checkpoints_filepath, monitor='val_precision', verbose=verbose, save_best_only=True, mode='max')
    callbacks_list = [checkpoint]

    # fits the model on batches with real-time data augmentation, for nb_epoch-100 epochs
    history = model.fit_generator(datagen.flow(X_train, Y_train,
                                               batch_size=batch_size,
                                               # save_to_dir='distorted_data',
                                               # save_format='png'
                                               seed=1337),
                            samples_per_epoch=len(X_train), nb_epoch=(nb_epoch-window), verbose=0,
                            validation_data=(X_test, Y_test), callbacks=callbacks_list)

    # ensuring best val_precision reached during training
    try_load_checkpoints(model, checkpoints_filepath)

    # fits the model on clear training set, for nb_epoch-700 epochs
    history_cont = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=window,
                                    verbose=0, validation_data=(X_test, Y_test), callbacks=callbacks_list)

    #...


I am working on the file "Notebooks/05_ICR-Test.ipynb".
The first task is:
Confusion matrix
Can you write Python code for it?


I am working on the file "Notebooks/05_ICR-Test.ipynb".
The first task is:
Data load
Can you write Python code for it?

(X_train, y_train, X_test, y_test) = dataset.generate_half_labeled_half_height(char, char_type)

u.plot_images(X_train[0:9], y_train[0:9], img_shape=(56,34))

if K.image_dim_ordering() == 'th':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)


I am working on the file "Notebooks/05_ICR-Test.ipynb".
The first task is:
Definitions
Can you write Python code for it?

batch_size = 512
nb_classes = 2
nb_epoch = 500
# checkpoint path
checkpoints_filepath = "checkpoints/05_ICR_A_no-dist_weights.best.hdf5"

# input image dimensions
img_rows, img_cols = 34, 56
# number of convolutional filters to use
nb_filters1 = 30
nb_filters2 = 50
# size of pooling area for max pooling
pool_size1 = (2, 2)
pool_size2 = (3, 3)
# convolution kernel size
kernel_size1 = (4, 4)
kernel_size2 = (5, 5)
# dense layer size
dense_layer_size1 = 250
# dropout rate
dropout = 0.15
# activation
activation = 'relu'


I am working on the file "Notebooks/05_ICR-Test.ipynb".
The first task is:
Examples of incorrect predictions
Can you write Python code for it?

u.plot_images(X_test[incorrect_indices[:9]], y_test[incorrect_indices[:9]],
              predicted_classes[incorrect_indices[:9]], img_shape=(56,34))


I am working on the file "Notebooks/05_ICR-Test.ipynb".
The first task is:
Inspecting the result
Can you write Python code for it?

# The predict_classes function outputs the highest probability class
# according to the trained classifier for each input example.
predicted_classes = model.predict_classes(X_test)

# Check which items we got right / wrong
correct_indices = np.nonzero(predicted_classes == y_test)[0]
incorrect_indices = np.nonzero(predicted_classes != y_test)[0]


I am working on the file "Notebooks/05_ICR-Test.ipynb".
The first task is:
Training and evaluation
Using non verbose output for training, since we already get some informations from the callback.
Can you write Python code for it?

# checkpoint
checkpoint = ModelCheckpoint(checkpoints_filepath, monitor='val_precision', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

# training
print('training model')
history = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=0, validation_data=(X_test, Y_test), callbacks=callbacks_list)

# ensures that the weights with the best evaluation are loaded
if os.path.exists(checkpoints_filepath):
    model.load_weights(checkpoints_filepath)

# evaluation
print('evaluating model')
score = model.evaluate(X_test, Y_test, verbose=1)
print('Test score:', score[0])
print('Test accuracy:', score[1]*100, '%')
print('Test error:', (1-score[2])*100, '%')

u.plot_history(history)
u.plot_history(history, metric='loss', loc='upper left')


I am working on the file "Notebooks/06a-Modeling_-_parameter_optimization.ipynb".
The first task is:
Check that the data is there:
Can you write Python code for it?

crm.visualize([('HindIII', 'MboI')], normalized=True, paint_tads=True)


I am working on the file "Notebooks/06a-Modeling_-_parameter_optimization.ipynb".
The first task is:
Decide which region to model.
Can you write Python code for it?

crm.visualize([('MboI', 'HindIII')], normalized=True, paint_tads=True,
              focus=(670, 770))


I am working on the file "Notebooks/06a-Modeling_-_parameter_optimization.ipynb".
The first task is:
For the other replicate, we can reduce the space of search:
Can you write Python code for it?

opt_mboi = mboi.optimal_imp_parameters(start=685, end=750, n_models=40, n_keep=20, n_cpus=8,
                                     upfreq_range=(0, 0.6, 0.3),
                                     lowfreq_range=(-0.9, -0.3, 0.3),
                                     maxdist_range=(1500, 2000, 500),
                                     dcutoff_range=[1.5, 2])

opt_mboi.plot_2d(show_best=5)

opt_mboi.get_best_parameters_dict()


I am working on the file "Notebooks/06a-Modeling_-_parameter_optimization.ipynb".
The first task is:
Load raw data matrices, and normalized matrices
Can you write Python code for it?


I am working on the file "Notebooks/06a-Modeling_-_parameter_optimization.ipynb".
The first task is:
Optimization of parameters
We are going to optimize these parameters for both experiments:
 - maxdit: maximum distance associated to the minimum amount of interactions
 - upfreq: minimum amount of interactions that defines attraction
 - lowfreq: maximum number of interactions that defines repulsion
 - dcutoff: the definition of "contact" in units of bead diameter. Value of 2 means that a contact will occur when 2 beads are closer than 2 times their diameter. This will be used to compare 3D models with Hi-C interaction maps.
Can you write Python code for it?

opt_hindiii = hindiii.optimal_imp_parameters(
    start=685, end=750,
    n_models=40, n_keep=20, n_cpus=8,
                                   upfreq_range=(0, 0.6, 0.3),
                                   lowfreq_range=(-0.9, 0, 0.3),
                                   maxdist_range=(1000, 2000, 500),
                                   dcutoff_range=[1.5, 2, 2.5])

opt_hindiii.plot_2d(show_best=5)


I am working on the file "Notebooks/06a-Modeling_-_parameter_optimization.ipynb".
The first task is:
Refine optimization in a small region:
Can you write Python code for it?

opt_hindiii.run_grid_search(upfreq_range=(0, 0.3, 0.3), lowfreq_range=(-0.6, -0.3, 0.3),
                       maxdist_range=[1750],
                       dcutoff_range=[1.5, 2],
                       n_cpus=8)

opt_hindiii.plot_2d(show_best=5)

opt_hindiii.run_grid_search(upfreq_range=(0, 0.3, 0.3), lowfreq_range=(-0.3, 0, 0.1),
                       maxdist_range=[2000, 2250],
                       dcutoff_range=[1.5, 2],
                       n_cpus=8)

opt_hindiii.plot_2d(show_best=5)

opt_hindiii.run_grid_search(upfreq_range=(0, 0.3, 0.1), lowfreq_range=(-0.3, 0, 0.1),
                       n_cpus=8,
                       maxdist_range=[2000, 2250],
                       dcutoff_range=[1.5, 2])

opt_hindiii.plot_2d(show_best=5)

opt_hindiii.get_best_parameters_dict()


I am working on the file "Notebooks/07092017-PRA-Home-and-Ciriculum-Notebook-001.ipynb".
The first task is:
Dataframes
Working with Pandas and DataFrames.  Importing necessary packages.
Can you write Python code for it?

import pandas as pd
import numpy as np

df = pd.DataFrame(data=np.array([[1,2,3,4], [5,6,7,8]], dtype=int), columns=['Pacific','Mountain','Central','Eastern'])
plt.plot(df)
df


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
As you can see above what we are doing is printing every filename in the img_file_list variable. There is another function called enumerate() which counts the number of items in your object and returns both the count of the objects and the objects themselves. We use it below to print both the number of the file and the file name itself.
Can you write Python code for it?

for file_num, file_name in enumerate(img_file_list):
    print(file_num)
    print(file_name)


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
As you can see, python interprets a*b as "repeat a b number of times. What happens if you try to add a and b? Can you imagine why?
As variables go, a and b are terrible names for variables, because if you read them tomorrow you will forget what they were for. So as a rule you should always give sensible names where possible.
Suppose we have the following problem: The average squirrel weighs 0.5 kg whats the weight of a thousand squirrels?
We should assign sensible names for the calculations as below:
Can you write Python code for it?

# Average Weight of Squirrels
avg_squirl_wt = 0.5
# Number of Squirrels
no_of_squirl = 1000
# Calculate and print total weight
tot_weight = avg_squirl_wt*no_of_squirl
print(tot_weight)


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
Boo! lean
The computer is also able to make comparisons and tell you if something is bigger, smaller, same. These type of operations are called boolean operations. Lets try out some
Can you write Python code for it?

big_number = 100
small_number = 2
print(big_number<small_number)
print(big_number>small_number)
print(big_number==small_number)
print(big_number>=small_number)


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
Congrats! you have written your first little program. So what are you doing you ask?
well the print() part of that statement is a function which tells the computer to print something on the screen. The something is enclosed in the quotation marks. Try printing something else, like your name maybe?
Can you write Python code for it?

print("Captain, Captain Jack Sparrow!")

print("Print this)


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
Great! now we know that root_root has two directories where our images are stored, but thats not the full path to the images is it? Lets see how we can get that for images in the Type1 directory.
Note If you got an error, make sure that your 'root_root' variable has a forward slash at the end. This signifies it as a directory.
Can you write Python code for it?

img_file_list = os.listdir(root_root+dir_of_root[0])
img_file_list