I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
Now we will load our sample data and create a 'Alcabuse' variable from 'Dalc' and 'Walc' (weekday and weekend alcohol consumption level, respectively) and make it a binary variable by calling it alcohol abuse if a student scores more than 8 on drinking alcohol in a week. The dataset was retrieved from here but slightly modified to better illustrate our purpose.
Can you write Python code for it?

# Load the data
df = pd.read_csv('../Data/student-mat.csv')
df['Alcabuse']=0
df.loc[df['Dalc']+df['Walc'] >=8,'Alcabuse']=1 # Let's call it abuse if more than 8 alcohol consumption per week


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
Question: Why do you think the accuracy is lower for the test data compared to the training?
Accuracy is one number to represent the overall model performance, but doesn't differentiate between different types of classification errors such as false positives and false negatives.  Sensitivity and Specificity are useful metrics to assess these aspects of the model's performance.
| Predicted Positive | Predicted Negative
--- | --- | ---
Actual Positive | 192 | 21
Actual Negative | 8 | 16
Sensitivity $ \frac{TP}{TP+FN} $ : measures the proportion of positives that are correctly identified (e.g. we correctly identify the alcohol abusers)
Specificity $ \frac{TN}{TN+FP} $ : measures the proportion of negatives that are correctly identified (e.g. we correctly identify the non-alcohol abusers)
Can you write Python code for it?

# Sensitivity and false negatives
print('Training')
display(accs_train)
print('')
print('sensitivity : %.2f ' %(float(accs_train[1][1])/(accs_train[1][1]+accs_train[1][0])))
print('false negative : %.2f ' %(float(accs_train[1][0])/(accs_train[1][1]+accs_train[1][0])))
print('specificity : %.2f ' %(float(accs_train[0][0])/(accs_train[0][0]+accs_train[0][1])))
print('false positives : %.2f ' %(float(accs_train[0][1])/(accs_train[0][0]+accs_train[0][1])))

# Specificity and false positives
print('\n')
print('Test')
display(accs_test)
print('\n')
print('sensitivity : %.2f ' %(float(accs_test[1][1])/(accs_test[1][1]+accs_test[1][0])))
print('false negative : %.2f ' %(float(accs_test[1][0])/(accs_test[1][1]+accs_test[1][0])))
print('specificity : %.2f ' %(float(accs_test[0][0])/(accs_test[0][0]+accs_test[0][1])))
print('false positives : %.2f ' %(float(accs_test[0][1])/(accs_test[0][0]+accs_test[0][1])))


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
Regularization
Regularization is a method to help deal with multicollinearity and also avoid overfitting. Overfitting occurs when you have an overly complex model such as having more features(Xs) than observations(Y). For instance, if you try to fit 10 observations with 10 features, each coefficient can be adjusted for a perfect fit but it wouldn't generalize well. In other cases, you might face the problem of feature selection. If you have numerous variables, it is time consuming to try every single combination of features in your model to see what yields the best result.
<img src="http://i.stack.imgur.com/0NbOY.png">
Regularization attempts to solve this problem by introducting a loss function that penalizes the model for each additional features added to the model. The AIC / BIC values we mentioned in the first session is a form of regularization in comparing models.
In this section we will focus on feature selection regularization methods Lasso and Ridge regressions.
Lasso regression
In short, Lasso is a feature selection method that reduces the number of features to use in a regression.
This is useful if you have a lot of variables that are correlated or you have more variables than observations.
Here is a study from Dartmouth that used Lasso on the Student Life Dataset.
In our example, we will try to find which variables can predict the level of alcohol consumption 'AlcSum'.
Can you write Python code for it?

df['AlcSum']=df['Dalc']+df['Walc']

# We have a lot of features to choose from
df.keys()

# Make a model with all the feature variables
formula = 'AlcSum ~ school+sex+age+address+famsize+Pstatus+Medu+\
       Fedu+Mjob+Fjob+reason+guardian+traveltime+\
       studytime+failures+schoolsup+famsup+paid+activities+nursery+higher+\
       internet+romantic+famrel+freetime+goout+health+absences+G1+G2+G3'

# Generate a design matrix with the formula
# This automatically generates separate regressors for categorical variables.
from patsy import dmatrices
y, X = dmatrices(formula_like=formula, data=df, return_type='dataframe')
# Ridge if L1_wt = 0 , Lasso if L1_wt = 1
penalty = .036
regm = smf.OLS(y,X).fit_regularized(method='coord_descent', maxiter=1000, alpha=penalty, L1_wt=1.0)
print regm.summary()


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
Ridge regression
The goal of the ridge function is to choose a penalty λ for which the coefficients are not rapidly changing and have “sensible” signs. It is especially useful when data suffers from multicollinearity, that is some of your predictor variables are highly correlated. Unlike LASSO, ridge does not produce a sparse solution, but rather shrinks variables that are highly collinear towards zero.
Can you write Python code for it?

from sklearn import linear_model
y, X = dmatrices(formula_like=formula, data=df, return_type='dataframe')
n_alphas = 200
alphas = np.logspace(-2, 6, n_alphas)

# Get optimal lambda
clf = linear_model.RidgeCV(alphas=alphas)
clf.fit(X,y)
aestim = clf.alpha_ # Estimated regularization param

# Get paths
clf = linear_model.Ridge(fit_intercept=False)
coefs = []
for a in alphas:
    clf.set_params(alpha=a)
    clf.fit(X, y)
    coefs.append(clf.coef_)

###############################################################################
# Display results
plt.figure(figsize=(15,8))

ax = plt.gca()
ax.plot(alphas, np.squeeze(coefs))
ax.set_xscale('log')
plt.axvline(aestim, linestyle='--', color='k', label='alpha: CV estimate')
plt.legend()
plt.xlabel('Lambda',fontsize=16)
plt.ylabel('Weights',fontsize=16)
plt.title('Ridge coefficients as a function of the regularization',fontsize=18)
plt.axis('tight')


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
The points on the left vertical axis (the left ends of the lines) are the ordinary least squares regression values.
These occur for k equal zero. As k is increased, the values of the regression estimates change, often wildly at first.
At some point, the coefficients seem to settle down and then gradually drift towards zero.
The task of the ridge regression analyst is to determine at what value of k these coefficients are at their stable
values. A vertical line is drawn at the value selected for reporting purposes. It is anticipated that you would run
the program several times until an appropriate value of k is determined. In our case, the selected alpha value is indicated by a vertical line at which a = 65.93. The smaller the value of k, the smaller the amount of bias that is included in the estimates.
Can you write Python code for it?

clf = linear_model.Ridge(fit_intercept=False)
a = aestim
clf.set_params(alpha=a)
clf.fit(X, y)
np.round(clf.coef_,3)

# Ridge if L1_wt = 0 , Lasso if L1_wt = 1
penalty = aestim
regm = smf.OLS(y,X).fit_regularized(method='coord_descent', maxiter=1000, alpha=penalty, L1_wt=0)
print regm.summary()


I am working on the file "Notebooks/6_Introduction_to_SocialNetworkAnalysis.ipynb".
The first task is:
A similar approach is to calculate clustering coefficient for each node and for the graph.
Clustering is the measure of how likely it is if A has edges to B and C, that B and C related?
For instance, Charlie form a triangle with Luke and Jin, but Sam and Heidi don't form a single triangle.
We can calculate this for each node and then get an average value overall to get the average clustering coefficient.
Can you write Python code for it?

d = nx.clustering(G)
print_network(d,'Clustering Coefficients per node')
print
print 'Average clustering coefficient : ' + str(nx.average_clustering(G))


I am working on the file "Notebooks/6_Introduction_to_SocialNetworkAnalysis.ipynb".
The first task is:
Average Shortest Path
The average shortest path length is the average of calculating the shortest number of edges it takes for every pair of nodes in the network.
This can be a mesure of how efficient/fast the network is in the distribution of information, rumors, disease, etc.
Can you write Python code for it?

print 'Network 1 average shortest path: ' + str(round(nx.average_shortest_path_length(G),2))
print 'Network 2 average shortest path: ' + str(nx.average_shortest_path_length(G2))


I am working on the file "Notebooks/6_Introduction_to_SocialNetworkAnalysis.ipynb".
The first task is:
Clustering coefficient.
Another way to look at how tight-knit a network is to look at how clustered it is.
To estimate clustering, we start from cliques, which can be considered as a subnetwork composed of three nodes forming a triangle.
We can first calculate for each node, how many cliques(triangles) it is a member of.
Then we calculate transitivity, which is simply the number of triangles in the network divided by the number of all possible traingles.
Can you write Python code for it?

d = nx.triangles(G)
print_network(d,'Number of cliques(triangles)')
print
print 'Transitivity : ' + str(nx.transitivity(G))

plot_network(d,'Transitivity')


I am working on the file "Notebooks/6_Introduction_to_SocialNetworkAnalysis.ipynb".
The first task is:
Eigenvector Centrality
The underlying assumption of the Eigenvector centrality is that a node's importance is determined by how important its neighbors are.
For instance, having more influential friends (betweeness centrality) can be more important than just having more number of friends (degree centrality).
Now we can observe that Luke is back to being more important than Eshin (who had higher betweeness centrality), because Luke is friends with Eshin who also has high centrality.
Can you write Python code for it?

d = nx.eigenvector_centrality(G)
title = "Eigenvector Centrality"
print_network(d,title)
plot_network(d,title)


I am working on the file "Notebooks/6_Introduction_to_SocialNetworkAnalysis.ipynb".
The first task is:
Notice there are no connections between the two nodes because we haven't added any edges yet.
To add edges between nodes, for example node1 and node 2, we can use the .add_edge('node1','node2') method on our graph.
Can you write Python code for it?

G = nx.Graph()
G.add_edge('Jin','Luke',weight=1)
G.add_edge('Jin','Eshin',weight=1)
G.add_edge('Luke','Eshin',weight=1)
plt.figure(figsize=(5,3))
pos = nx.spring_layout(G) # One way of specifiying a layout for nodes.
nx.draw(G,pos,with_labels=True,node_size=5000,font_size=20,alpha=.5)


I am working on the file "Notebooks/6_Introduction_to_SocialNetworkAnalysis.ipynb".
The first task is:
Now you can see that we have an edge connecting Jin and Luke.
There are several different types of graphs.

weighted vs unweighted graphs

a graph is said to be unweighted if every connection is either 1 or 0
it is weighted if the edges take on other values indicating the strength of the relationship.

directed vs undirected graphs

a graphs is undirected if the edges are symmetrical
it is directed if the edges are asymmetrical, meaning that each edge can indicate the direction of the relationship

For simplicity, today we will only cover unweighted and undirected graphs.
There are multiple ways to indicate nodes and edges in a graph.
As illustrated above, we can manually add each node and edge.  This approach is fine for small graphs, but won't scale well.  It is also possible to add nodes is to use a python dictionary.
The key is the node and the values indicate what that key(node) connects to.
Can you write Python code for it?

d = {'Jin':['Luke','Eshin','Antonia','Andy','Sophie','Rob','Charlie','Vanessa'],
     'Luke':['Eshin','Antonia','Seth','Andy','Sophie','Rob','Charlie','Vanessa'],
     'Antonia':['Luke','Jin','Eshin','Seth','Andy'],
     'Eshin':['Heidi','Antonia','Jin','Sam','Andy'],
     'Sophie':['Rob','Vanessa'],
     'Rob':['Sophie','Vanessa']}
G = nx.Graph(d)
plt.figure(figsize=(15,8))
np.random.seed(2) # Just to keep things same
pos = pos = nx.fruchterman_reingold_layout(G) # Another way of specifiying a layout for nodes.
nx.draw(G,with_labels=True,node_size=1500,font_size=20,alpha=.3,width=2)


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
Add_hydrogens
Add Hydrogen Atoms. Auto: std changes at pH 7.0. His->Hie. pH: set pH value
    list: Explicit list as [*:]HisXXHid, Interactive[_his]: Prompts for all selectable residues
    Fixes missing side chain atoms unless --no_fix_side is set
    Existing hydrogen atoms are removed before adding new ones unless --keep_h set.
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
Backbone
Detects and fixes several problems with the backbone
use any of
--fix_atoms All|None|Residue List
--fix_chain All|None|Break list
--add_caps All|None|Terms|Breaks|Residue list
--no_recheck
--no_check_clashes
Can you write Python code for it?

st_c.backbone()

st_c.backbone('--fix_atoms All --fix_chain none --add_caps none')


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
Comparing both checks it becomes clear that GLN A42, GLN E498, ASN A103, and ASN A194 are getting new contacts as thay have both changed, ASN E394 is worse as it has now two contacts
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
Fix all amide residues and recheck
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
General help
Can you write Python code for it?

with open(args['commands_help_path']) as help_file:
    print(help_file.read())
#TODO: prepare a specific help method
# print_help(command)


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
Initializing checking engine, loading structure and showing statistics
Can you write Python code for it?

st_c = StructureChecking(base_dir_path, args)


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
Set input (PDB or local file, pdb or mmCif formats allowed) and output (local file, pdb format).
Use pdb:pdbid for downloading structure from PDB (RCSB)
Can you write Python code for it?

base_path = '/home/gelpi/DEVEL/BioPhysics/wdir/'
args['input_structure_path'] = base_path + '6m0j.cif'
args['output_structure_path'] = base_path + '6m0j_fixed.pdb'
args['output_structure_path_charges'] = base_path + '6m0j_fixed.pdbqt'
args['debug'] = False
args['verbose'] = False


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
We need to choose one of the alternative forms for each residue
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
altloc
Checks for the presence of residues with alternative locations. Atoms with alternative coordinates and their occupancy are reported.
MD simulations requires a single position for each atom.
Use altloc('occupancy | alt_ids | list of res:id) to select the alternative
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
amide
Amide terminal atoms in Asn ang Gln residues can be labelled incorrectly.
amide suggests possible fixes by checking the sourrounding environent.
To fix use amide ('All | None | residue_list')
Note that the inversion of amide atoms may trigger additional contacts.
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
chains
Checks for chains (also obtained from print_stats), and allow to select one or more.
MD simulations are usually performed with complete structures. However input structure may contain several copies of the system, or contains additional chains like peptides or nucleic acids that may be removed.
Use chains('X,Y') to select chain(s) X and Y to proceed
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
chiral
Side chains of Thr and Ile are chiral, incorrect atom labelling lead to the wrong chirality.
To fix use chiral('All | None | residue_list')
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
clashes
Detects steric clashes based on distance criteria.
Contacts are classified in:
* Severe: Too close atoms, usually indicating superimposed structures or badly modelled regions. Should be fixed.
* Apolar: Vdw colissions.Usually fixed during the simulation.
* Polar and ionic. Usually indicate wrong side chain conformations. Usually fixed during the simulation
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
ligands
Detects HETATM (excluding Water molecules) to selectively remove them.
To remove use ligands('All | None | Residue List (by id, by num)')
Can you write Python code for it?

st_c.ligands()

st_c.ligands('All')

st_c.ligands()


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
metals
Detects HETATM being metal ions allow to selectively remove them.
To remove use metals (' All | None | metal_type list | residue list ')
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
models
Checks for the presence of models in the structure.
MD simulations require a single structure, although some structures (e.g. biounits) may be defined as a series of models, in such case all of them are usually required.
Use models('--select N') to select model num N for further analysis
Can you write Python code for it?


I am working on the file "Notebooks/6m0j_check.ipynb".
The first task is:
water
Detects water molecules and allows to remove them
Crystallographic water molecules may be relevant for keeping the structure, however in most cases only some of them are required. These can be later added using other methods (titration) or manually.
To remove water molecules use water('yes')
Can you write Python code for it?


I am working on the file "Notebooks/7_Introduction_to_Scraping.ipynb".
The first task is:
Getting data using requests
Requests is an excellent library for performing HTTP requests.
In this simple example, we will scrape data from the PBS faculty webpage.
Can you write Python code for it?

page = requests.get("http://pbs.dartmouth.edu/people")
print(page)


I am working on the file "Notebooks/7_Introduction_to_Scraping.ipynb".
The first task is:
Here the response '200' indicates that the get request was successful.  Now let's look at the actual text that was downloaded from the webpage.
Can you write Python code for it?


I am working on the file "Notebooks/7_Introduction_to_Scraping.ipynb".
The first task is:
Here we are going to find the unordered list tagged with the id 'faculty-container'.  We are then going to look for any nested tag that use the 'h4' header tag.  This should give us all of the lines with the faculty names as a list.
Can you write Python code for it?

names_html = soup.find_all('ul',id='faculty-container')[0].find_all('h4')
names = [x.text for x in names_html]
print(names)


I am working on the file "Notebooks/7_Introduction_to_Scraping.ipynb".
The first task is:
One thing we might do with this data is create a dictionary with names and emails of all of the professors in the department.  This could be useful if we wanted to send a bulk email to them.
Can you write Python code for it?

email_dict = dict([(x.split('@')[0],x) for x in email])
print(email_dict)


I am working on the file "Notebooks/7_Introduction_to_Scraping.ipynb".
The first task is:
Parsing string data
What if we wanted to grab the name from the list of email addresses?
Can you write Python code for it?


I am working on the file "Notebooks/7_Introduction_to_Scraping.ipynb".
The first task is:
What if we wanted to get all of the faculty email addresses?
Can you write Python code for it?

email_html = soup.find_all('ul',id='faculty-container')[0].find_all('span',{'class' : 'contact'})
email = [x.text for x in email_html]
print(email)


I am working on the file "Notebooks/7_Introduction_to_Scraping.ipynb".
The first task is:
You can see that every name also includes an initial.  Let's try to just pull out the first and last name.
Can you write Python code for it?

for x in email_dict.keys():
    old = x.split('.')
    email_dict[" ".join([i for i in old if len(i) > 2])] = email_dict[x]
    del email_dict[x]

print(email_dict)


I am working on the file "Notebooks/A0-Preparing_your_computer_for_HiC_data_analysis.ipynb".
The first task is:
And run it with all the default options. The installer will create a miniconda2 folder in your home directory where all the programs that you need will be stored (including python).
Python dependencies
With conda you can install the needed dependencies:
Can you write Python code for it?

! conda install -y scipy # scientific computing in python
! conda install -y numpy # scientific computing in python
! conda install -y matplotlib # to produce plots
! conda install -y jupyter # this notebook :)
! conda install -y -c https://conda.anaconda.org/bcbio pysam # to deal with SAM/BAM files
! conda install -y -c https://conda.anaconda.org/salilab imp # for 3D modeling
! conda install -y pip # yet another python package manager
! conda install -y -c bioconda mcl # for clustering


I am working on the file "Notebooks/A0-Preparing_your_computer_for_HiC_data_analysis.ipynb".
The first task is:
And, if you have root access:
Can you write Python code for it?


I am working on the file "Notebooks/A0-Preparing_your_computer_for_HiC_data_analysis.ipynb".
The first task is:
DSRC FASTQ compressor
DSRC is a FASTQ compressor, it's not needed, but we use it as the size of the files is significantly smaller than using gunzip (>30%), and, more importantly, the access to them can be parallelized, and is much faster than any other alternative.
It can be downloaded from https://github.com/lrog/dsrc
Can you write Python code for it?

! wget http://sun.aei.polsl.pl/dsrc/download/2.0rc/dsrc

! chmod +x dsrc


I am working on the file "Notebooks/A0-Preparing_your_computer_for_HiC_data_analysis.ipynb".
The first task is:
In case you do not have root access, just copy the binaries to some path and add this path to your global PATH:
Can you write Python code for it?

! mkdir ~/bin
! cp GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-mapper ~/bin/
! cp GEM-binaries-Linux-x86_64-core_i3-20121106-022124/gem-indexer* ~/bin/
! echo $PATH=$PATH:"~/bin/" >> ~/.bashrc


I am working on the file "Notebooks/A0-Preparing_your_computer_for_HiC_data_analysis.ipynb".
The first task is:
TADbit
For now TADbit is not available through conda or pip package manager, so to install it we will have to clone the repository, and compile the binaries:
Can you write Python code for it?

! git clone git@github.com:3DGenomes/TADbit.git
! cd TADbit; python setup.py install


I am working on the file "Notebooks/A0-Preparing_your_computer_for_HiC_data_analysis.ipynb".
The first task is:
Uncompress the archive:
Can you write Python code for it?


I am working on the file "Notebooks/A1-Preparation_reference_genome.ipynb".
The first task is:
Concatenate all contigs/chromosomes into a single file
Can you write Python code for it?

contig_file = open('genome/Homo_sapiens_contigs.fa','w')
for molecule in contig:
    for line in open('genome/' + molecule):
        # replace the header of the sequence in the fasta file
        if line == '\n':
            continue
        if line.startswith('>'):
            line = '>' + molecule[3:].replace('.fasta', '') + '\n'
        contig_file.write(line)
contig_file.close()


I am working on the file "Notebooks/A1-Preparation_reference_genome.ipynb".
The first task is:
Download from the NCBI the list of chromosome/contigs
Can you write Python code for it?

! wget -q $sumurl -O chromosome_list.txt

! head chromosome_list.txt

dirname = 'genome/'
! mkdir -p $dirname


I am working on the file "Notebooks/A3-Compare_and_merge_Hi-C_experiments.ipynb".
The first task is:
All in one loop to:
 - filter
 - normalize
 - generate intra-chromosome and genomic matrices
all at diferent resolutions, and for both time points
Can you write Python code for it?

for rep in ['T0', 'T60']:
    print ' -', rep
    for reso in [1000000, 300000, 100000]:
        print '   *', reso
        # load hic_data
        hic_data = load_hic_data_from_reads(
            'results/fragment/both_{0}/03_filtering/valid_reads12_{0}.tsv'.format(rep),
            reso)
        # filter columns
        hic_data.filter_columns(draw_hist=False, min_count=10, by_mean=True)
        # normalize
        hic_data.normalize_hic(iterations=0)
        # save biases to reconstruct normalization
        out = open('results/fragment/both_{1}/04_normalizing/biases_{0}_{1}.pick'.format(reso, rep), 'w')
        dump(hic_data.bias, out)
        out.close()
        # save filtered out columns
        out = open('results/fragment/both_{1}/04_normalizing/bad_columns_{0}_{1}.pick'.format(reso, rep), 'w')
        dump(hic_data.bads, out)
        out.close()
        # save data as raw matrix per chromsome
        hic_map(hic_data, by_chrom='intra', normalized=False,
                savedata='results/fragment/both_{1}/04_normalizing/{0}_raw'.format(reso, rep))
        # save data as normalized matrix per chromosome
        hic_map(hic_data, by_chrom='intra', normalized=True,
                savedata='results/fragment/both_{1}/04_normalizing/{0}_norm'.format(reso, rep))
        # if the resolution is low save the full genomic matrix
        if reso > 500000:
            hic_map(hic_data, by_chrom=False, normalized=False,
                    savefig ='results/fragment/both_{1}/04_normalizing/{0}_raw.png'.format(reso, rep),
                    savedata='results/fragment/both_{1}/04_normalizing/{0}_raw.mat'.format(reso, rep))

            hic_map(hic_data, by_chrom=False, normalized=True,
                    savefig ='results/fragment/both_{1}/04_normalizing/{0}_norm.png'.format(reso, rep) ,
                    savedata='results/fragment/both_{1}/04_normalizing/{0}_norm.mat'.format(reso, rep))


I am working on the file "Notebooks/A3-Compare_and_merge_Hi-C_experiments.ipynb".
The first task is:
Between conditions
HindIII
Can you write Python code for it?

renz1 = 'HindIII'
renz2 = 'MboI'

hic_data1 = my_load_hic_data(renz1, reso)
hic_data2 = my_load_hic_data(renz2, reso)

## this part is to "tune" the plot ##
plt.figure(figsize=(9, 6))
axe = plt.subplot()
axe.grid()
axe.set_xticks(range(0, 55, 5))
axe.set_xticklabels(['%d Mb' % int(i * 0.2) if i else '' for i in range(0, 55, 5)], rotation=-45)
#####################################

_ = correlate_matrices(hic_data1, hic_data2, max_dist=50, show=False, axe=axe, normalized=True)

hic_data1 = my_load_hic_data(renz1, rep1, 1000000)
hic_data2 = my_load_hic_data(renz2, rep2, 1000000)

corrs = eig_correlate_matrices(hic_data1, hic_data2, show=True, aspect='auto', normalized=True)

for cor in corrs:
    print ' '.join(['%5.3f' % (c) for c in cor]) + '\n'


I am working on the file "Notebooks/A3-Compare_and_merge_Hi-C_experiments.ipynb".
The first task is:
Between replicates with different restriction enzymes
T0
Can you write Python code for it?

renz1 = 'HindIII'
renz2 = 'NcoI'
rep1  = 'T0'
rep2  = 'T0'

hic_data1 = my_load_hic_data(renz1, rep1, reso)
hic_data2 = my_load_hic_data(renz2, rep2, reso)

## this part is to "tune" the plot ##
plt.figure(figsize=(9, 6))
axe = plt.subplot()
axe.grid()
axe.set_xticks(range(0, 55, 5))
axe.set_xticklabels(['%d Mb' % int(i * 0.2) if i else '' for i in range(0, 55, 5)], rotation=-45)
#####################################

_ = correlate_matrices(hic_data1, hic_data2, max_dist=50, show=False, axe=axe)

hic_data1 = my_load_hic_data(renz1, rep1, 1000000)
hic_data2 = my_load_hic_data(renz2, rep2, 1000000)

corrs = eig_correlate_matrices(hic_data1, hic_data2, show=True, aspect='auto', normalized=True)

for cor in corrs:
    print ' '.join(['%5.3f' % (c) for c in cor]) + '\n'


I am working on the file "Notebooks/A3-Compare_and_merge_Hi-C_experiments.ipynb".
The first task is:
Merge Hi-C experiments
Once agreed that experiments are similar, they can be merged.
Here is a simple way to merge valid pairs. Arguably we may want to merge unfiltered data but the difference would be minimal specially with non-replicates.
Can you write Python code for it?

from pytadbit.mapping import merge_2d_beds

! mkdir -p results/fragment/both_T0/
! mkdir -p results/fragment/both_T60/
! mkdir -p results/fragment/both_T0/03_filtering/
! mkdir -p results/fragment/both_T60/03_filtering/

renz1 = 'HindIII'
renz2 = 'NcoI'
rep   = 'T60'

hic_data1 = 'results/fragment/{0}_{1}/03_filtering/valid_reads12_{0}_{1}.tsv'.format(renz1, rep)
hic_data2 = 'results/fragment/{0}_{1}/03_filtering/valid_reads12_{0}_{1}.tsv'.format(renz2, rep)
hic_data  = 'results/fragment/both_{0}/03_filtering/valid_reads12_{0}.tsv'.format(rep)

merge_2d_beds(hic_data1, hic_data2, hic_data)

renz1 = 'HindIII'
renz2 = 'NcoI'
rep   = 'T0'

hic_data1 = 'results/fragment/{0}_{1}/03_filtering/valid_reads12_{0}_{1}.tsv'.format(renz1, rep)
hic_data2 = 'results/fragment/{0}_{1}/03_filtering/valid_reads12_{0}_{1}.tsv'.format(renz2, rep)
hic_data  = 'results/fragment/both_{0}/03_filtering/valid_reads12_{0}.tsv'.format(rep)

merge_2d_beds(hic_data1, hic_data2, hic_data)


I am working on the file "Notebooks/A3-Compare_and_merge_Hi-C_experiments.ipynb".
The first task is:
NcoI
Can you write Python code for it?

renz1 = 'NcoI'
renz2 = 'NcoI'
rep1  = 'T0'
rep2  = 'T60'

hic_data1 = my_load_hic_data(renz1, rep1, reso)
hic_data2 = my_load_hic_data(renz2, rep2, reso)

## this part is to "tune" the plot ##
plt.figure(figsize=(9, 6))
axe = plt.subplot()
axe.grid()
axe.set_xticks(range(0, 55, 5))
axe.set_xticklabels(['%d Mb' % int(i * 0.2) if i else '' for i in range(0, 55, 5)], rotation=-45)
#####################################

_ = correlate_matrices(hic_data1, hic_data2, max_dist=50, show=False, axe=axe)

hic_data1 = my_load_hic_data(renz1, rep1, 1000000)
hic_data2 = my_load_hic_data(renz2, rep2, 1000000)

corrs = eig_correlate_matrices(hic_data1, hic_data2, show=True, aspect='auto', normalized=True)

for cor in corrs:
    print ' '.join(['%5.3f' % (c) for c in cor]) + '\n'


I am working on the file "Notebooks/A3-Compare_and_merge_Hi-C_experiments.ipynb".
The first task is:
Normalizing merged data
Can you write Python code for it?

from pytadbit.mapping.analyze import hic_map
from cPickle import dump

! mkdir -p results/fragment/both_T0/04_normalizing
! mkdir -p results/fragment/both_T60/04_normalizing


I am working on the file "Notebooks/A3-Compare_and_merge_Hi-C_experiments.ipynb".
The first task is:
T60
Can you write Python code for it?

renz1 = 'HindIII'
renz2 = 'NcoI'
rep1  = 'T60'
rep2  = 'T60'

hic_data1 = my_load_hic_data(renz1, rep1, reso)
hic_data2 = my_load_hic_data(renz2, rep2, reso)

## this part is to "tune" the plot ##
plt.figure(figsize=(9, 6))
axe = plt.subplot()
axe.grid()
axe.set_xticks(range(0, 55, 5))
axe.set_xticklabels(['%d Mb' % int(i * 0.2) if i else '' for i in range(0, 55, 5)], rotation=-45)
#####################################

_ = correlate_matrices(hic_data1, hic_data2, max_dist=50, show=False, axe=axe)

hic_data1 = my_load_hic_data(renz1, rep1, 1000000)
hic_data2 = my_load_hic_data(renz2, rep2, 1000000)

corrs = eig_correlate_matrices(hic_data1, hic_data2, show=True, aspect='auto', normalized=True)

for cor in corrs:
    print ' '.join(['%5.3f' % (c) for c in cor]) + '\n'


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
All in one loop to:
 - filter
 - normalize
 - generate intra-chromosome and genomic matrices
all at diferent resolutions, and for both time points
Can you write Python code for it?

for reso in [1000000, 300000, 100000]:
    print '   *', reso
    # load hic_data
    hic_data = load_hic_data_from_reads(
        'results/both/03_filtering/valid_reads12.tsv', reso)
    # filter columns
    hic_data.filter_columns(draw_hist=False, min_count=10, by_mean=True)
    # normalize
    hic_data.normalize_hic(iterations=0)
    # save biases to reconstruct normalization
    out = open('results/both/04_normalizing/biases_{0}.pick'.format(reso), 'w')
    dump(hic_data.bias, out)
    out.close()
    # save filtered out columns
    out = open('results/both/04_normalizing/bad_columns_{0}.pick'.format(reso), 'w')
    dump(hic_data.bads, out)
    out.close()
    # save data as raw matrix per chromsome
    hic_map(hic_data, by_chrom='intra', normalized=False,
            savedata='results/both/04_normalizing/{0}_raw'.format(reso))
    # save data as normalized matrix per chromosome
    hic_map(hic_data, by_chrom='intra', normalized=True,
            savedata='results/both/04_normalizing/{0}_norm'.format(reso))
    # if the resolution is low save the full genomic matrix
    if reso > 500000:
        hic_map(hic_data, by_chrom=False, normalized=False,
                savefig ='results/both/04_normalizing/{0}_raw.png'.format(reso),
                savedata='results/both/04_normalizing/{0}_raw.mat'.format(reso))

        hic_map(hic_data, by_chrom=False, normalized=True,
                savefig ='results/both/04_normalizing/{0}_norm.png'.format(reso) ,
                savedata='results/both/04_normalizing/{0}_norm.mat'.format(reso))


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
Compare eigenvectors
ICE
Can you write Python code for it?

corrs = eig_correlate_matrices(hic_data1_ice, hic_data2_ice, show=True, aspect='auto', normalized=True)

for cor in corrs:
    print ' '.join(['%5.3f' % (c) for c in cor]) + '\n'


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
DRY
Can you write Python code for it?

## this part is to "tune" the plot ##
plt.figure(figsize=(9, 6))
axe = plt.subplot()
axe.grid()
axe.set_xticks(range(0, 55, 5))
axe.set_xticklabels(['%d Mb' % int(i * 0.2) if i else '' for i in range(0, 55, 5)], rotation=-45)
#####################################

_ = correlate_matrices(hic_data1_dry, hic_data2_dry, max_dist=50, show=False, axe=axe, normalized=True)


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
DRY normalized
Can you write Python code for it?

hic_data1_dry = my_load_hic_data(renz1, reso, which='dry')
hic_data2_dry = my_load_hic_data(renz2, reso, which='dry')

%matplotlib inline


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
ICE
Can you write Python code for it?

## this part is to "tune" the plot ##
plt.figure(figsize=(9, 6))
axe = plt.subplot()
axe.grid()
axe.set_xticks(range(0, 55, 5))
axe.set_xticklabels(['%d Mb' % int(i * 0.2) if i else '' for i in range(0, 55, 5)], rotation=-45)
#####################################

_ = correlate_matrices(hic_data1_ice, hic_data2_ice, max_dist=50, show=False, axe=axe, normalized=True)


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
Load data
ICE normalized
Can you write Python code for it?

hic_data1_ice = my_load_hic_data(renz1, reso, which='ice')
hic_data2_ice = my_load_hic_data(renz2, reso, which='ice')


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
Merge Hi-C experiments
Once agreed that experiments are similar, they can be merged.
Here is a simple way to merge valid pairs. Arguably we may want to merge unfiltered data but the difference would be minimal specially with non-replicates.
Can you write Python code for it?

from pytadbit.mapping import merge_2d_beds

! mkdir -p results/both
! mkdir -p results/both/
! mkdir -p results/both03_filtering/
! mkdir -p results/both/03_filtering/

renz1 = 'HindIII'
renz2 = 'MboI'

hic_data1 = 'results/{0}/03_filtering/valid_reads12_{0}.tsv'.format(renz1)
hic_data2 = 'results/{0}/03_filtering/valid_reads12_{0}.tsv'.format(renz2)
hic_data  = 'results/both/03_filtering/valid_reads12.tsv'

merge_2d_beds(hic_data1, hic_data2, hic_data)


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
Normalizing merged data
Can you write Python code for it?

from pytadbit.mapping.analyze import hic_map
from cPickle import dump

! mkdir -p results/both/04_normalizing


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
Repeat at 1 Mb resolution
Can you write Python code for it?

reso = 1000000
hic_data1_ice = my_load_hic_data(renz1, reso, which='ice')
hic_data2_ice = my_load_hic_data(renz2, reso, which='ice')
hic_data1_dry = my_load_hic_data(renz1, reso, which='dry')
hic_data2_dry = my_load_hic_data(renz2, reso, which='dry')


I am working on the file "Notebooks/A3_1-Compare_Hi-C_experiments.ipynb".
The first task is:
Write a little function to load HiCData obeject
Can you write Python code for it?

def my_load_hic_data(renz, reso, which='ice'):
    hic_data = load_hic_data_from_reads(base_path.format(renz), resolution=reso)
    if which=='ice':
        hic_data.bias = load(open(bias_ice_path.format(reso, renz)))
    else:
        hic_data.bias = dict([(int(l.split()[0]), float(l.split()[1]))
                              for l in open(bias_dry_path.format(reso, renz))])
    hic_data.bads = load(open(bads_path.format(reso, renz)))
    return hic_data

renz1 = 'HindIII'
renz2 = 'MboI'


I am working on the file "Notebooks/A4-Align_and_compare_TADs.ipynb".
The first task is:
Search for aligned TADs with specific features
Strongly conserved broders
Can you write Python code for it?

ali.get_column(lambda x: x['score']>5, min_num=2)


I am working on the file "Notebooks/A4-Align_and_compare_TADs.ipynb".
The first task is:
Significance
Can you write Python code for it?

ali, stats = crm.align_experiments(['T0', 'T60'], randomize=True)

print ali

stats

print 'Alignment score: %.3f, p-value: %.4f\n  proportion of borders of T0 found in T60: %.3f, of T60 in T0 %.3f' % stats


I am working on the file "Notebooks/A5-Modeling_-_analysis_of_3D_models.ipynb".
The first task is:
Calculate the length of a model, the sum of all distances between consecutive particles:
Can you write Python code for it?


I am working on the file "Notebooks/A5-Modeling_-_analysis_of_3D_models.ipynb".
The first task is:
The minimum cube:
Can you write Python code for it?

print model.cube_side()
print model.cube_volume()


I am working on the file "Notebooks/AAE.ipynb".
The first task is:
Generate new data
Approximate samples from the posterior distribution over the latent variables p(z|x)
Can you write Python code for it?

Zdemo = np.random.normal(size=[128, latentD], loc=0.0, scale=1.).astype(np.float32)

gen_sample = demo_sess.run(Xgen_op, feed_dict={Zph: Zdemo , is_training:False})
vis_square(gen_sample[:121], [11, 11], save_path=work_dir + 'sample.jpg')
Image(filename=work_dir + 'sample.jpg')


I am working on the file "Notebooks/AAE.ipynb".
The first task is:
Training AAE
You can either get the fully trained models from google drive or train your own models using the AAE.py script.
Experiments
Create demo networks and restore weights
Can you write Python code for it?

iter_num = 18018
best_model = work_dir + "Model_Iter_%.3d.ckpt"%iter_num
best_img = work_dir + 'Gen_Iter_%d.jpg'%iter_num
Image(filename=best_img)

latentD = 2 # of the best model trained
batch_size = 128

tf.reset_default_graph()
demo_sess = tf.InteractiveSession()

is_training = tf.placeholder(tf.bool, [], 'is_training')

Zph = tf.placeholder(tf.float32, [None, latentD])
Xph = tf.placeholder(tf.float32, [None, 28, 28, 1])

Z_op = create_encoder(Xph, is_training, latentD, reuse=False, networktype=networktype + '_Enc')
Xrec_op = create_decoder(Z_op, is_training, latentD, reuse=False, networktype=networktype + '_Dec')
Xgen_op = create_decoder(Zph, is_training, latentD, reuse=True, networktype=networktype + '_Dec')

tf.global_variables_initializer().run()

enc_varlist = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope=networktype + '_Enc')
dec_varlist = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope=networktype + '_Dec')
saver = tf.train.Saver(var_list=enc_varlist+dec_varlist)
saver.restore(demo_sess, best_model)

#Get uniform samples over the labels
spl = 800  # sample_per_label
data = input_data.read_data_sets(data_dir, one_hot=False, reshape=False)
Xdemo, Xdemo_labels = get_demo_data(data, spl)

decoded_data = demo_sess.run(Z_op, feed_dict={Xph:Xdemo, is_training:False})
plot_latent_variable(decoded_data, Xdemo_labels)


I am working on the file "Notebooks/ActivityNet-Release1.3.Proposals.ipynb".
The first task is:
Evaluate the uniform random proposals and plot the metric results
Can you write Python code for it?

%%time

uniform_average_nr_proposals_valid, uniform_average_recall_valid, uniform_recall_valid = run_evaluation(
    "../Evaluation/data/activity_net.v1-3.min.json",
    "../Evaluation/data/uniform_random_proposals.json",
    max_avg_nr_proposals=100,
    tiou_thresholds=np.linspace(0.5, 0.95, 10),
    subset='validation')

plot_metric(uniform_average_nr_proposals_valid, uniform_average_recall_valid, uniform_recall_valid)


I am working on the file "Notebooks/ActivityNet-Release1.3.Proposals.ipynb".
The first task is:
Generate uniform random proposal for the validation subset
Can you write Python code for it?

%%time

# seed the random number generator to get consistent results across multiple runs
np.random.seed(42)

with open("../Evaluation/data/activity_net.v1-3.min.json", 'r') as fobj:
    gd_data = json.load(fobj)

subset='validation'
avg_nr_proposals = 100
proposal_data = {'results': {}, 'version': gd_data['version'], 'external_data': {}}

for vid_id, info in gd_data['database'].iteritems():
    if subset != info['subset']:
        continue
    this_vid_proposals = []
    for _ in range(avg_nr_proposals):
        # generate random proposal center, length, and score
        center = info['duration']*np.random.rand(1)[0]
        length = info['duration']*np.random.rand(1)[0]
        proposal = {
                    'score': np.random.rand(1)[0],
                    'segment': [center - length/2., center + length/2.],
                   }
        this_vid_proposals += [proposal]

    proposal_data['results'][vid_id] = this_vid_proposals

with open("../Evaluation/data/uniform_random_proposals.json", 'w') as fobj:
    json.dump(proposal_data, fobj)


I am working on the file "Notebooks/ActivityNet-Release1.3.Proposals.ipynb".
The first task is:
Help functions to evaluate a proposal submission file and plot the metric results
Can you write Python code for it?

def run_evaluation(ground_truth_filename, proposal_filename,
                   max_avg_nr_proposals=100,
                   tiou_thresholds=np.linspace(0.5, 0.95, 10),
                   subset='validation'):

    anet_proposal = ANETproposal(ground_truth_filename, proposal_filename,
                                 tiou_thresholds=tiou_thresholds,
                                 max_avg_nr_proposals=max_avg_nr_proposals,
                                 subset=subset, verbose=True, check_status=True)
    anet_proposal.evaluate()

    recall = anet_proposal.recall
    average_recall = anet_proposal.avg_recall
    average_nr_proposals = anet_proposal.proposals_per_video

    return (average_nr_proposals, average_recall, recall)

def plot_metric(average_nr_proposals, average_recall, recall, tiou_thresholds=np.linspace(0.5, 0.95, 10)):

    fn_size = 14
    plt.figure(num=None, figsize=(6, 5))
    ax = plt.subplot(1,1,1)

    colors = ['C0', 'C1', 'C2', 'C3', 'C4', 'C5', 'C6', 'C7', 'C8', 'C9']

    area_under_curve = np.zeros_like(tiou_thresholds)
    for i in range(recall.shape[0]):
        area_under_curve[i] = np.trapz(recall[i], average_nr_proposals)

    for idx, tiou in enumerate(tiou_thresholds[::2]):
        ax.plot(average_nr_proposals, recall[2*idx,:], color=colors[idx+1],
                label="tiou=[" + str(tiou) + "], area=" + str(int(area_under_curve[2*idx]*100)/100.),
                linewidth=4, linestyle='--', marker=None)

    # Plots Average Recall vs Average number of proposals.
    ax.plot(average_nr_proposals, average_recall, color=colors[0],
            label="tiou = 0.5:0.05:0.95," + " area=" + str(int(np.trapz(average_recall, average_nr_proposals)*100)/100.),
            linewidth=4, linestyle='-', marker=None)

    handles, labels = ax.get_legend_handles_labels()
    ax.legend([handles[-1]] + handles[:-1], [labels[-1]] + labels[:-1], loc='best')

    plt.ylabel('Average Recall', fontsize=fn_size)
    plt.xlabel('Average Number of Proposals per Video',...


I am working on the file "Notebooks/ActivityNet-Temporal-Proposals.ipynb".
The first task is:
Lets compute the Recall at different Temporal IoU thresholds
Can you write Python code for it?

iou_thrs = np.arange(0.1, 0.6, 0.1)
recall = recall_vs_iou_thresholds(proposals_df, ground_truth_df, iou_threshold=iou_thrs)


I am working on the file "Notebooks/ActivityNet-Temporal-Proposals.ipynb".
The first task is:
Load annotations for validation subset in a Pandas Data Frame.
Can you write Python code for it?

# Modify this paths
ANNOTATION_FILE = 'activity_net.v1-3.min.json'
PROPOSALS_FILENAME = 'activitynet_v1-3_proposals.hdf5'
SUBSET = 'validation'

# Read json file containing the annotations
with open(ANNOTATION_FILE, 'r') as fobj:
    data = json.load(fobj)['database']

# Parsing and looking for agnostic segments of an intended subset.
video_id_fmt = 'v_{}'
gt_s_init, gt_s_end, video_id = [], [], []
for vid, vitem in data.iteritems():
    if vitem['subset'] != SUBSET:
        continue
    for ann in vitem['annotations']:
        gt_s_init.append(ann['segment'][0])
        gt_s_end.append(ann['segment'][1])
        video_id.append(video_id_fmt.format(vid))
# Creates ground truth data frame.
ground_truth_df = pd.DataFrame({'s-init': gt_s_init,
                                's-end': gt_s_end,
                                'video-id': video_id})


I am working on the file "Notebooks/ActivityNet-Temporal-Proposals.ipynb".
The first task is:
Load temporal proposals in a Pandas Data Frame.
Can you write Python code for it?

# Looking for videos in the subset
intended_videos = []
for vid, vitem in data.iteritems():
    if vitem['subset'] == SUBSET:
        intended_videos.append('v_{}'.format(vid))

# Reading proposals from HDF5 file.
s_init, s_end, score, video_id = [], [], [], []
fobj = h5py.File(PROPOSALS_FILENAME, 'r')
for vid in fobj.keys():
    if vid not in intended_videos:
        continue
    s_init.extend(fobj[vid]['segment-init'].value.tolist())
    s_end.extend(fobj[vid]['segment-end'].value.tolist())
    score.extend(fobj[vid]['score'].value.tolist())
    video_id.extend(np.repeat(vid, fobj[vid]['segment-init'].value.size).tolist())
fobj.close()
proposals_df = pd.DataFrame({'s-init': s_init,
                             's-end': s_end,
                             'score': score,
                             'video-id': video_id})
print 'Average number of proposals: {}'.format(proposals_df.shape[0] / float(len(intended_videos)))


I am working on the file "Notebooks/ActivityNet-Temporal-Proposals.ipynb".
The first task is:
Subrutines to compute temporal IoU and wrapper for recall
Can you write Python code for it?

def segment_iou(target_segments, test_segments):
    """Compute intersection over union btw segments
    Parameters
    ----------
    target_segments : ndarray
        2-dim array in format [m x 2:=[init, end]]
    test_segments : ndarray
        2-dim array in format [n x 2:=[init, end]]
    Outputs
    -------
    iou : ndarray
        2-dim array [m x n] with IOU ratio.
    Note: It assumes that target-segments are more scarce that test-segments
    """
    if target_segments.ndim != 2 or test_segments.ndim != 2:
        raise ValueError('Dimension of arguments is incorrect')

    m, n = target_segments.shape[0], test_segments.shape[0]
    iou = np.empty((m, n))
    for i in xrange(m):
        tt1 = np.maximum(target_segments[i, 0], test_segments[:, 0])
        tt2 = np.minimum(target_segments[i, 1], test_segments[:, 1])

        # Non-negative overlap score
        intersection = (tt2 - tt1).clip(0)
        union = ((test_segments[:, 1] - test_segments[:, 0]) +
                 (target_segments[i, 1] - target_segments[i, 0]) -
                 intersection)
        # Compute overlap as the ratio of the intersection
        # over union of two segments at the frame level.
        iou[i, :] = intersection / union
    return iou

def recall_vs_iou_thresholds(proposal_df, df, iou_threshold=np.array([0.5])):
    vds = proposal_df['video-id'].unique()
    score_lst = []
    # Compute iou score
    for i, v in enumerate(vds):
        # Proposals
        idx = proposal_df['video-id'] == v
        this_df = proposal_df.loc[idx]
        proposals = np.stack((this_df['s-init'],
                              this_df['s-end']), axis=-1)

        # Sort proposals
        idx = this_df['score'].argsort()[::-1]
        proposals = proposals[idx, :]

        # Annotations
        jdx = df['video-id'] == v
        ann_df = df.loc[jdx]
        annotations = np.stack((ann_df['s-init'],
                                ann_df['s-end']), axis=-1)
        if proposals.ndim == 1:
            proposals = proposals[np.newaxis,...


I am working on the file "Notebooks/Analysis of runs with and without order constraints.ipynb".
The first task is:
Let's look at the age difference between two nodes constrained in an older-younger fashion (the first constraint.
Can you write Python code for it?

diffABMinusCD = list()
for a in allHeights:
    diffABMinusCD.append(a["ab"] - a["cd"])

pd.DataFrame( diffABMinusCD ).describe()


I am working on the file "Notebooks/Analysis of runs with and without order constraints.ipynb".
The first task is:
The difference is not always negative.
Can you write Python code for it?

diffEFMinusGHNC = list()
for a in allHeightsNC:
    diffEFMinusGHNC.append(a["ef"] - a["gh"])

pd.DataFrame( diffEFMinusGHNC ).describe()


I am working on the file "Notebooks/Analysis of runs with and without order constraints.ipynb".
The first task is:
Then I will compare two MCMC runs: one with 2 node order constraints, and one without.
The node order constraints are as follow:
c       d       a       b
g       h       e       f
meaning that the MRCA of cd is older than the MRCA of AB (same thing for the second line), just like in the tree.
First we take a short look at the MCMC run.
Can you write Python code for it?

runConsMet = pd.read_csv("output/testConstraintsMet.log", sep="\t")

runConsMet.describe()

%matplotlib inline
plt.plot(runConsMet['Iteration'], runConsMet['Posterior'], 'b-')
plt.xlabel("Iteration", fontsize=15)
plt.ylabel("Posterior", fontsize=15)

%matplotlib inline
plt.plot(runConsMet['Iteration'], runConsMet['root'], 'b-')
plt.xlabel("Iteration", fontsize=15)
plt.ylabel("Root age", fontsize=15)

def readTreesFromRBOutput (file):
    try:
        f=open(file, 'r')
    except IOError:
        print ("Unknown file: "+file)
        sys.exit()
    line = ""
    treeStrings = list()
    for l in f:
        if "teration" not in l:
            m = re.sub('\[&index=\d+\]', "", l)
            treeStrings.append(m.split()[4])
    trees=list()
    for l in treeStrings:
        trees.append ( Tree( l ) )
    return trees

trees = readTreesFromRBOutput("output/testConstraintsMet.trees")

def getNodeHeights( t ):
    node2Height = dict()
    id2Height = dict()
    for node in t.traverse("postorder"):
        if node not in node2Height:
            node2Height[node] = 0.0
            id2Height[node.name] = 0.0
        if node.up:
            if node.up.name =='':
                leaves = node.up.get_leaves()
                name=""
                for l in leaves:
                    name += l.name
                node.up.name=name
            node2Height[node.up] = node2Height[node] + node.dist
            id2Height[str(node.up.name)] = node2Height[node] + node.dist
      # print node.name + " : " + str(node2Height[node])
    #return node2Height,id2Height
    return id2Height

allHeights = list()
for t in trees:
    allHeights.append(getNodeHeights(t))


I am working on the file "Notebooks/Analysis of runs with and without order constraints.ipynb".
The first task is:
They are all negative, and thus agree with the constraint.
We look at the other constraint.
Can you write Python code for it?

diffEFMinusGH = list()
for a in allHeights:
    diffEFMinusGH.append(a["ef"] - a["gh"])

pd.DataFrame( diffEFMinusGH ).describe()


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Accuracy score by region
Can you write Python code for it?

nerc_labels = nercs.nerc.dropna().unique()

for region in nerc_labels:
    mask = y_state_test == region

    X_state_masked = X_state_test[mask]
    y_state_hat_masked = clf_rf_state.predict(X_state_masked)
    y_state_test_masked = y_state_test[mask]

    accuracy = metrics.accuracy_score(y_state_test_masked, y_state_hat_masked)
    print('{} : {}'.format(region, accuracy))


for region in nerc_labels:
    mask = y_test == region

    X_masked = X_test[mask]
    y_hat_masked = clf_rf.predict(X_masked)
    y_test_masked = y_test[mask]

    accuracy = metrics.accuracy_score(y_test_masked, y_hat_masked)
    print('{} : {}'.format(region, accuracy))


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Append my 2017 SPP and TRE guesses to the full nerc dataframe
Can you write Python code for it?

nercs = pd.concat([nercs, additional_spp, additional_tre])


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Because the 2017 facility dataframe only includes annually reporting facilities I'm going to duplicate the plant id, lat/lon, and state information from 2016.
Can you write Python code for it?

# make a copy of 2016 (or most recent annual data year) and change the year to
plants_2017 = plants.loc[plants['year'] == most_recent_860_year, :].copy()
plants_2017.loc[:, 'year'] += 1

plants = pd.concat([plants.loc[plants['year']<=most_recent_860_year, :], plants_2017])

(set(plants.loc[plants.year==2016, 'plant id']) - set(plants.loc[plants.year==2017, 'plant id']))


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Date string for filenames
This will be inserted into all filenames (reading and writing)
Can you write Python code for it?


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Drop plants that don't have lat/lon data (using just lon to check), and then drop duplicates. If any plants have kept the same plant id but moved over time (maybe a diesel generator?) or switched NERC they will show up twice.
Can you write Python code for it?

cols = ['plant id', 'lat', 'lon', 'nerc', 'state', 'year']
df_slim = (df.loc[:, cols].dropna(subset=['lon'])
             .drop_duplicates(subset=['plant id', 'year', 'nerc']))

df_slim.tail()


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Ensuring that no plants in Alaska or Hawaii are assigned to continental NERCs, or the other way around.
Can you write Python code for it?

print(unknown.loc[unknown.state.isin(['AK', 'HI']), 'nerc'].unique())
print(unknown.loc[unknown.nerc.isin(['HICC', 'ASCC']), 'state'].unique())

Counter(unknown['nerc'])

unknown.head()

unknown_state.head()


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Export plants with lat/lon, state, and nerc
Can you write Python code for it?

nercs.tail()

unknown.head()

unknown_state.tail()

len(unknown_state['plant id'].unique())

df_slim.head()

labeled = pd.concat([df_slim.loc[df_slim.notnull().all(axis=1)],
                     unknown,
                     unknown_state.loc[:, ['plant id', 'nerc', 'state', 'year']]])

labeled.tail()

labeled.loc[labeled.nerc.isnull()]


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
F1 score by region
Can you write Python code for it?

y_hat = clf_rf.predict(X_test)

for region in nerc_labels:
    f1 = metrics.f1_score(y_test, y_hat, labels=[region], average='macro')
    print('{} : {}'.format(region, f1))


y_state_hat = clf_rf_state.predict(X_state_test)

for region in nerc_labels:
    f1 = metrics.f1_score(y_state_test, y_state_hat, labels=[region], average='macro')
    print('{} : {}'.format(region, f1))


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
GridSearch to find the best parameters in a RandomForest Classifier
I previously used k-nearest neighbors with just lat/lon as input features. The problem is that some facilities don't have lat/lon data. They do usually have a state geography label though. Categorical labels don't work well in KNN, but the combination of lat/lon and a state label will work well in a tree model. RandomForest is usually a quite effective tree model and my results are more accurate with this than they were with KNN.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier()
params = dict(
    n_estimators = [5, 10, 25, 50],
    min_samples_split = [2, 5, 10],
    min_samples_leaf = [1, 3, 5],
)

clf_rf = GridSearchCV(rf, params, n_jobs=-1, iid=False, verbose=1)
clf_rf.fit(X_train, y_train)

clf_rf.best_estimator_, clf_rf.best_score_

clf_rf.score(X_test, y_test)

nerc_labels = nercs.nerc.dropna().unique()


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
I want to assign NERC regions for every year. We have data for 2012 onward from the EIA-860 files. For the purpose of this analysis I'll assume that all years from 2001-2011 are the same NERC as 2012.
Also assume that values in 2017 are the same as in 2016. I'll fill in nerc values for plants that were built in 2017 below.
Can you write Python code for it?

for year in range(2001, 2012):
    # the pandas .copy() method is deep by default but I'm not sure in this case
    df = deepcopy(eia_nercs[2012])
    df['year'] = year

    eia_nercs[year] = df

df = deepcopy(eia_nercs[2016])
df['year'] = 2017
eia_nercs[2017] = df

eia_nercs.keys()

eia_nercs[2001].head()

nercs = pd.concat(eia_nercs.values())
nercs.sort_values('year', inplace=True)

nercs.head()

(set(nercs.loc[(nercs.nerc == 'MRO') &
              (nercs.year == 2016), 'plant id'])
 - set(nercs.loc[(nercs.nerc == 'MRO') &
              (nercs.year == 2017), 'plant id']))

nercs.year.unique()


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Load data
This loads facility data that has been assembled from the EIA bulk data file, and EIA-860 excel files. The EIA-860 excel files need to be downloaded manually.
Load EIA facility data
Only need to keep the plant id, year (as a check that plants don't move between years), and lat/lon
Can you write Python code for it?

path = os.path.join(data_path, 'Derived data',
                    'Facility gen fuels and CO2 {}.csv'.format(file_date))
facility_df = pd.read_csv(path)
facility_df['state'] = facility_df['geography'].str[-2:]

plants = facility_df.loc[:, ['plant id', 'year', 'lat', 'lon', 'state']]
plants.drop_duplicates(inplace=True)


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Look for plants listed with different NERC labels
 This may not matter anymore if I use NERC labels for each year
There are 30 plants duplicated. Five of them don't have a NERC label in one of the years. The largest move is from MRO to other regions (12), with most of those to SPP (7). After that, moves from RFC (5) to MRO (3) and SERC (2). There are also some moves from WECC and FRCC to HICC/ASCC - these might be diesel generators that get moved.
The plants that have duplicate NERC region labels represent a small fraction of national generation, but one that is growing over time. By 2016 they consist of 0.15% of national generation.
Can you write Python code for it?

for df_ in list(eia_nercs.values()) + [nercs]:
    print('{} total records'.format(len(df_)))
    print('{} unique plants'.format(len(df_['plant id'].unique())))

dup_plants = nercs.loc[nercs['plant id'].duplicated(keep=False), 'plant id'].unique()
dup_plants

region_list = []
for plant in dup_plants:
    regions = nercs.loc[nercs['plant id'] == plant, 'nerc'].unique()
#     regions = regions.tolist()
    region_list.append(regions)
Counter(tuple(x) for x in region_list)

(facility_df.loc[facility_df['plant id'].isin(dup_plants), :]
            .groupby('year')['generation (MWh)'].sum()
 / facility_df.loc[:, :]
              .groupby('year')['generation (MWh)'].sum())


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Make lists of plant codes for SPP and TRE facilities
Can you write Python code for it?

nercs.head()

# Create additional dataframes with 2017 SPP and TRE plants.
# Use these to fill in values for 2017 plants

m860_spp_plants = (m860.loc[m860['nerc'] == 'SPP', 'plant id']
                       .drop_duplicates()
                       .reset_index(drop=True))

additional_spp = pd.DataFrame(m860_spp_plants.copy())
# additional_spp['plant id'] = m860_spp_plants
additional_spp['nerc'] = 'SPP'
additional_spp['year'] = 2017

m860_tre_plants = (m860.loc[m860['nerc'] == 'TRE', 'plant id']
                       .drop_duplicates()
                       .reset_index(drop=True))

additional_tre = pd.DataFrame(m860_tre_plants)
# additional_tre['plant id'] = m860_tre_plants
additional_tre['nerc'] = 'TRE'
additional_tre['year'] = 2017

additional_spp

additional_tre


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Plants without lat/lon
Use just the state for plants that don't have lat/lon info. Less accurate, especially where NERC regions cross state lines, but better than nothing.
Need to start with the lon column so I can filter to only unknown facilities that also don't have lon
Can you write Python code for it?

cols = ['plant id', 'nerc', 'state', 'year', 'lon']
df_state_slim = (df.loc[:, cols].dropna(subset=['state']).copy())

df_state_slim.head()

len(df_state_slim)


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Separate out the list of plants where we don't have NERC labels from EIA-860.
Can you write Python code for it?

unknown = df_slim.loc[df_slim.nerc.isnull()].copy()

print("{} plants/years don't have NERC labels\n".format(len(unknown)))
print(unknown.head())

unknown.tail()


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Some plants in EIA-860 don't have NERC labels. Drop them now.
This is my training data. All of these plants should still be in my plants dataframe.
Can you write Python code for it?

nan_plants = {}
all_nan = []
years = nercs.year.unique()
for year in years:
    nan_plants[year] = nercs.loc[(nercs.year == year) &
                                 (nercs.isnull().any(axis=1)), 'plant id'].tolist()
    all_nan.extend(nan_plants[year])

# number of plants that don't have a nerc in at least one year
len(all_nan)

# drop all the rows without a nerc value
nercs.dropna(inplace=True)

nan_plants[2017]


I am working on the file "Notebooks/Assign NERC region labels.ipynb".
The first task is:
Use best RandomForest parameters to predict NERC for unknown plants
Can you write Python code for it?

unknown.loc[:, 'nerc'] = clf_rf.predict(unknown.loc[:, ['lat', 'lon', 'year']])
unknown_state.loc[:, 'nerc'] = clf_rf_state.predict(unknown_state.loc[:, ['enc state', 'year']])


I am working on the file "Notebooks/Assignment2.ipynb".
The first task is:
In the above figures, first image represents observed degree distribution of random graph $G(n, p)$ and the binomial pmf with parameters $n$ and $p$. The second image describes observed degree distribution of random graph $G(n, m)$ with $m = \frac{1}{2}pn(n-1)$ and the binomial pmf with parameters $n$ and $p$.
We can see that in both cases, observed degree distribution looks similar to binomial pmf.
Exercise 2
In this exercise, we want to demonstrate the following:
- if $np < 1$, a graph in $G_{n,p}$ will almost surely have no connected components of size larger than $O(\log n)$
- if $np = 1$, a graph in $G_{n,p}$ will almost surely have a largest component whose size is $O(n^\frac{2}{3})$,
- if $np \to c > 1$, where c is a constant, then a graph in $G_{n,p}$ will almost surely have a unique giant component containing a positive fraction of the vertices. No other component will contain more than $O(\log n)$ vertices. Furthermore,
- if $np < (1 − \epsilon)\log n$, a graph in $G_{n,p}$ will almost surely contain isolated vertices, and
- if $np > (1 + \epsilon)\log n$, a graph in $G_{n,p}$ will almost surely be connected.
To do so, we will do the following experiments.
- We will generate random graphs $G(n, p)$ with $n \in [100, 200]$ and $p \in [0.003, 0.04]$.
- For each graph we will compute the size of the largest component
- Use color map to visualize the observed sizes
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import numpy as np
import matplotlib.pyplot as plt

def component_experiment((n_l, n_r), (p_l, p_r), seeds = None, offset = 3):
    # n_l, n_r bounds for n
    # p_l, p_r bounds for probability
    # s sample size
    # generate s graphs from G(n,p)
    s = n_r - n_l # sample size
    if seeds is None:
        seeds = np.random.choice(2000000, s*s).reshape((s, s))
    # compute

    step_p = float(p_r - p_l) / float(s)

    N = np.arange(n_l, n_r, step=1)
    P = np.arange(p_l, p_r, step=step_p)

    N, P = np.meshgrid(N, P)
    CC = np.zeros((s, s))
    # r_graphs = []
    for i in range(s):
        # r_graphs.append([])
        for j in range(s):
            G = er.er_np(int(N[i, j]), P[i, j], seed=seeds[i, j])
            # r_graphs[-1].append(G)
            cc_sizes = G.connected_component()[1]

            # get the largest
            largest_comp_size = 0 if len(cc_sizes) == 0 else max(cc_sizes)
            # print("Largest comp of graph (%d, %f) is %d" % (int(N[i, j]), P[i, j], largest_comp_size))
            CC[i, j] = largest_comp_size

    plot3D(N, P, CC)

def plot3D(X, Y, Z):
    # plt.figure(num=None, figsize=(15, 6), dpi=80, facecolor='w', edgecolor='k')
    fig = plt.figure(figsize=(15, 8))
    mapa = plt.pcolor(X, Y, Z, cmap=cm.coolwarm, antialiased=True)
    fig.colorbar(mapa, shrink=0.5, aspect=5)

    # plot a few lines
    tX = X[1]
    n_times_p = [0.7, 0.8, 0.9, 1.0, 1.3, 1.7, 2.0]
    lines = []
    for prod in n_times_p:
        line, = plt.plot(tX, prod / tX, label="n*p = %f"%prod)
        lines.append(line)

    plt.legend(lines)

    plt.show()

component_experiment((100, 200), (0.003, 0.04))


I am working on the file "Notebooks/Assignment2.ipynb".
The first task is:
Plotting
We will now plot observed degree distribution and binomial distribution.
Note: In the next block of code, we will use python modules scipy, numpy and matplotlib.
Can you write Python code for it?

import scipy.stats as stat
import numpy as np
import matplotlib.pyplot as plt

def plot_test(n, p, degree_sequence, counter, title="Plot"):
    # plot setup
    plt.figure(num=None, figsize=(15, 6), dpi=80, facecolor='white', edgecolor='green')

    # Binomial pmf with parameters n-1, p
    x_axis = np.linspace(0, n-1, n)
    y_axis = np.array([stat.binom.pmf(i, n-1, p) for i in range(n)]) #stat.binom.pmf(x_axis, n-1, p)
    # display degree distribution and binomial pmf
    #print("deg seq")
    #print(degree_sequence)
    my_counter, bins, patches = plt.hist(degree_sequence, x_axis, normed=1.0, edgecolor='k', linewidth=0.1)
    plt.title(title)
    plt.plot(x_axis, y_axis, 'r--', linewidth=1)
    plt.grid(True)
    plt.axis([0, n-1, 0, 0.05])
    plt.show()

# G(n, p) model
plot_test(n, p, g_degree_sequence, num_counter,
          title="Observed degree distribution and binomial distribution: n=%d, p=%f" % (n, p))
print("******************Second test*******************")
num_counterG1[0]=0
plot_test(n, p, g1_degree_sequence, num_counterG1, title="Plot1")


I am working on the file "Notebooks/Audio/filter_caduceus.ipynb".
The first task is:
As of this writing, the L1 h(t) data has a very large dynamic range at low frequencies. We need to remove this before doing anything.
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Agora, para descobrirmos quando o arquivo foi modificado pela última vez, precisamos usar uma função que não retorna a data da última modificação no formato em que estamos acostumados. Ela retorna o tempo, em segundos, decorrido desde 1o de janeiro de 1970 (se você estiver no Unix). Para podermos obter o que queremos, usamos então a função ctime do módulo time.
Can you write Python code for it?

import time
print(os.path.getmtime(os.path.join(diretorio,"file1.txt")))
time.ctime(os.path.getmtime(os.path.join(diretorio,"file1.txt")))

lista = os.listdir(diretorio)
for arquivo in lista:
    print(time.ctime(os.path.getmtime(os.path.join(diretorio,arquivo))))

os.mkdir(os.path.join(diretorio,"arquivos_setembro"))
os.mkdir(os.path.join(diretorio,"arquivos_agosto"))

for arquivo in lista:
    if arquivo != "teste.txt":
        data_modificacao = time.ctime(os.path.getmtime(os.path.join(diretorio,arquivo)))
        if data_modificacao[4:7] == "Sep":
            os.rename(os.path.join(diretorio,arquivo), os.path.join(diretorio,"arquivos_setembro",arquivo))
        elif data_modificacao[4:7] == "Aug":
            os.rename(os.path.join(diretorio,arquivo), os.path.join(diretorio,"arquivos_agosto",arquivo))


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Agora, vamos desfazer o exercício para restaurarmos o diretório à estrutura original:
Can you write Python code for it?

lista = os.listdir(diretorio)
for item in lista:
    if os.path.isdir(os.path.join(diretorio,item)):
        locais = os.listdir(os.path.join(diretorio,item))
        print(locais)
        for arquivo in locais:
            os.rename(os.path.join(diretorio,item,arquivo),os.path.join(diretorio,arquivo))
        os.rmdir(os.path.join(diretorio,item))

print(os.listdir(diretorio))


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Agora, vamos efetuar a seguinte operação: todos os arquivos deste diretório que tiverem a extensão ".txt" terão um novo nome iniciando com letra maiúscula.
Can you write Python code for it?

for arquivo in lista:
    if arquivo[-3:] == "txt":
        os.rename(os.path.join(diretorio,arquivo),os.path.join(diretorio,arquivo.capitalize()))


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Assim, para acessarmos um elemento individual da lista numeros, precisamos utilizar um índice para a lista externa, e outro índice para a lista interna:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Atenção: Escolha um diretório no seu computador para executar esse exemplo, com arquivos que possam ser renomeados.
No meu caso, usarei os arquivos do diretório oceanobiopython/exemplos/exemplo_1
Can you write Python code for it?

diretorio = os.path.join(os.getcwd(), "..","exemplos/exemplo_1")


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Comandos de repetição e condicionais
Repetição: for
Às vezes, desejamos repetir um certo número de vezes a mesma operação.
De fato, uma lista comprehension inclui um comando de repetição: executamos alguma ação com cada item de uma lista; se lermos a expressão da lista comprehension, poderíamos dizer: "retorne n para cada n no conjunto que vai de 0 a 10":
Can you write Python code for it?

numeros = [n for n in range(0,11)]
print(numeros)


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Em seguida, vamos listar todos os arquivos deste diretório.
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Exemplo 2
Organizar os arquivos de um diretório pela data da última modificação.
Can you write Python code for it?

import os
diretorio = os.path.join(os.getcwd(), "..","exemplos/exemplo_2")
print(diretorio)
print(os.listdir(diretorio))


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Exemplo: língua do P
Can you write Python code for it?

linguadope = ["Pe"+palavra for palavra in minhalista]

linguadope


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Fazemos isso explicitamente quando usamos a estrutura for. Observe que, no Python, não precisamos sinalizar o fim de um bloco de código; na verdade, a indentação sinaliza um bloco. Observe a diferença entre as células abaixo:
Can you write Python code for it?

for item in numeros: # Indentação
    print("Numero "+str(item))
print("bla")

for item in numeros: # Indentação
    print("Numero "+str(item))
    print("bla")


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Observe que o slicing também vale para sequências de caracteres (strings):
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Observe que x é uma variável local à list comprehension; ela serve apenas para que possamos fazer referência a cada item da lista sendo percorrida na nova expressão que queremos gerar. Fora da list comprehension, o Python não sabe quem é esse x:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Observe que, com um parâmetro a mais, podemos selecionar os elementos da lista de 2 em 2:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Para que possamos percorrer a lista que contém os nomes de todos os arquivos deste diretório, vamos salvar esta lista de nomes de arquivo:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Para transformarmos aquela lista de listas em uma lista simples, podemos usar o comando abaixo (admito que é um pouco mágico, mas funciona! ;))
Can you write Python code for it?

lista = [item for sublist in numeros for item in sublist]
lista


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Slicing
Podemos selecionar pedaços de uma lista (ou string) facilmente usando o conceito de slicing.
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Também podemos calcular o tamanho de uma lista de números:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_2.ipynb".
The first task is:
Uma explicação para a sintaxe do join: http://www.faqs.org/docs/diveintopython/odbchelper_join.html (basicamente, o método join precisa de duas strings: um resultado e uma "cola". Mas o argumento a ser unido pode ser outra coisa além de uma string (notadamente, iterables)
Exemplo: lista de números
É comum querermos gerar uma lista de números delimitada por dois valores. A função range(a,b) gera uma lista que vai do número a até b-1; porém, o resultado da função range não é a lista diretamente, mas uma estrutura de dados que precisamos transformar em uma lista.
Can you write Python code for it?

numeros = [n for n in range(0,10)]
print(numeros)


I am working on the file "Notebooks/Aula_3.ipynb".
The first task is:
Agora, vamos desfazer o exemplo:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_3.ipynb".
The first task is:
Agora, vamos ler apenas a primeira linha de cada arquivo:
Can you write Python code for it?

for item in lista:
    with open(item,"r") as arquivo:
        print(arquivo.readline())

with open("resumo.txt", "w") as arquivo_saida:
    for item in lista:
        with open(item,"r") as arquivo:
            arquivo_saida.write(arquivo.readline()+"\n")


I am working on the file "Notebooks/Aula_3.ipynb".
The first task is:
Exemplo
Tentar encontrar uma string específica (no nosso caso, "sf") dentro do arquivo file1.txt
Can you write Python code for it?

string = "sf"
b = []
with open(os.path.join(diretorio,"file1.txt"),"r") as arquivo:
    for line in arquivo:
        if string in line:
            b.append(line.rstrip("\n"))


I am working on the file "Notebooks/Aula_3.ipynb".
The first task is:
Exemplo:
Ler a primeira linha de cada arquivo de um diretorio e escrever o resultado em outro arquivo.
Primeiro, usamos uma list comprehension para obtermos uma lista dos arquivos no diretorio em que estamos interessados, mas queremos excluir o arquivo teste.txt e queremos que os arquivos estejam listados com seu caminho completo.
Can you write Python code for it?

print([os.path.join(diretorio,item) for item in os.listdir(diretorio) if item != "teste.txt"])

lista = [os.path.join(diretorio,item) for item in os.listdir(diretorio) if item != "teste.txt"]

lista


I am working on the file "Notebooks/Aula_3.ipynb".
The first task is:
Para alguns casos específicos, pode ser interessante carregar um arquivo completo na memória. Para isso, usamos
Can you write Python code for it?

with open(os.path.join(diretorio,"file1.txt"),"r") as arquivo:
    conteudo = arquivo.read()

print(conteudo)


I am working on the file "Notebooks/Aula_3.ipynb".
The first task is:
Sem argumentos, ele lê a próxima linha do arquivo; isto quer dizer que se ele é executado diversas vezes em sequência, com o arquivo aberto, ele lê a cada vez que é executado uma das linhas do arquivo.
Can you write Python code for it?

with open(os.path.join(diretorio,"file1.txt"), "r") as arquivo:
    for i in range(0,5):
        print(arquivo.readline())


I am working on the file "Notebooks/Aula_3.ipynb".
The first task is:
Também podemos usar o comando readline:
Can you write Python code for it?

with open(os.path.join(diretorio,"file1.txt"), "r") as arquivo:
    print(arquivo.readline())


I am working on the file "Notebooks/Aula_4.ipynb".
The first task is:
Na verdade, dados é um tipo diferente de estrutura:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_4.ipynb".
The first task is:
Pandas
A biblioteca mais apropriada para lidarmos com grandes arquivos de texto, estilo csv ou txt, é a Pandas
http://pandas.pydata.org
Observe que o arquivo abaixo tem uma visualização confusa justamente por seu tamanho:
Can you write Python code for it?

with open('../exemplos/exemplo_4/table_w_tax.txt', 'r') as arquivo:
    for linha in arquivo:
        print(linha)


I am working on the file "Notebooks/Aula_4.ipynb".
The first task is:
Podemos acessar uma coluna do DataFrame acima (por exemplo, a coluna cujo rótulo é "#OTU ID") usando a seguinte sintaxe:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_4.ipynb".
The first task is:
Por fim, para eliminarmos a primeira linha do arquivo já que está não faz parte do cabeçalho em que estamos interessados, modificamos mais uma vez o comando:
Can you write Python code for it?

dados = pd.read_csv('../exemplos/exemplo_4/table_w_tax.txt', delimiter='\t', header=1)
dados


I am working on the file "Notebooks/Aula_4.ipynb".
The first task is:
Se quisermos desfazer o exemplo acima...
Can you write Python code for it?


I am working on the file "Notebooks/Aula_4.ipynb".
The first task is:
Um DataFrame é uma estrutura criada pela biblioteca pandas que pode ser considerada como uma tabela, organizada com nomes de colunas e índices de linhas.
No entanto, para o arquivo anterior, ainda temos problemas de leitura. De fato, o arquivo tem colunas separadas pelo caracter TAB. Assim, podemos modificar o comando para obtermos
Can you write Python code for it?

dados = pd.read_csv('../exemplos/exemplo_4/table_w_tax.txt', delimiter='\t')

dados


I am working on the file "Notebooks/Aula_4.ipynb".
The first task is:
Uma das vantagens desta formulação é que podemos acessar as colunas de dados de maneira bastante eficiente. Por exemplo, suponha que queremos imprimir apenas os valores referentes às colunas "Estação" e "Prof. (m)" da nossa tabela.
Can you write Python code for it?

import csv
with open('../exemplos/exemplo_3/MCT II Rio Chui Final.csv', 'r') as arquivo:
    reader = csv.DictReader(arquivo)
    for row in reader:
        print(row['Estação'], row['Prof. (m)'])


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
(O r antes da string do título é importante pois significa que o Python não deve tentar interpretar o que se segue)
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

fig, ax = plt.subplots()

t = np.arange(0.0, 5.0, 0.01)
s = np.cos(2*np.pi*t)
line, = ax.plot(t, s, lw=2)

ax.annotate('máximo local', xy=(2, 1), xytext=(3, 1.5),
            arrowprops=dict(facecolor='black', shrink=0.05),
            )

plt.ylim(-2,2)


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Adicionando anotações
O comando text() pode ser utilizado para adicionarmos texto em posições arbitrárias, enquanto que xlabel(), ylabel() e title() são utilizados para adicionar texto em posições pré-definidas:
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

mu, sigma = 100, 15
x = mu + sigma * np.random.randn(10000)

fig = plt.figure(1)
n, bins, patches = plt.hist(x, 50, normed=1, facecolor='g', alpha=0.75)

plt.xlabel('Domínio')
plt.ylabel('Probabilidade')
plt.title('Histograma')
plt.text(60, .025, r'$\mu=100,\ \sigma=15$')
plt.axis([40, 160, 0, 0.03])
plt.grid(True)
plt.show()

import numpy as np
import matplotlib.pyplot as plt

mu, sigma = 100, 15
x = mu + sigma * np.random.randn(10000)

fig = plt.figure(1)
n, bins, patches = plt.hist(x, 50, normed=1, facecolor='g', alpha=0.75)

plt.xlabel('Domínio')
plt.ylabel('Probabilidade')
plt.title('Histograma')
plt.text(60, .025, r'$\mu=100,\ \sigma=15$')
plt.axis([40, 160, 0, 0.03])
plt.grid(True)

t = plt.xlabel('Dados', fontsize=14, color='red')

plt.show()


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Agora, vamos criar uma janela com dois subplots:
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

def f(t):
    return np.exp(-t) * np.cos(2*np.pi*t)

t1 = np.arange(0.0, 5.0, 0.1)
t2 = np.arange(0.0, 5.0, 0.02)

plt.figure(1)
plt.subplot(211)
plt.plot(t1, f(t1), 'bo', t2, f(t2), 'k')

plt.subplot(212)
plt.plot(t2, np.cos(2*np.pi*t2), 'r--')
plt.show()


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Alguns conceitos são um pouco diferentes no numpy (em relação ao MATLAB). Cuidado:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Atenção Como vimos em aula, em algumas instalações a opção notebook não funciona corretamente; aconselho sempre que estiverem usando notebooks, usarem
%matplotlib inline
Can you write Python code for it?


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Atenção: Note que numpy.array não é o mesmo que a classe padrão do Python array.array, que lida apenas com vetores unidimensionais e oferece muito menos funcionalidades.
Can you write Python code for it?

A = np.matrix([[10,3],[1,2],[3,1]])

print(A)

lista = [1,2,3,4]
np.asarray(lista)

dir(A)

print("Matriz: ",A)
print("Diagonal: ", A.diagonal())


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Como queremos que os gráficos fiquem embutidos no notebook, podemos usar a linha
Can you write Python code for it?


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Múltiplos gráficos na mesma janela
No pyplot, assim como no MATLAB, temos um conceito de figura e axes (eixos). Todos os comandos de gráfico se aplicam ao eixo atual. A função gca() retorna uma referência ao eixo atual e gcf() retorna a figura atual.
Por exemplo, podemos usar
Can you write Python code for it?

fig, axes = plt.subplots()
pontos = [1,3,2,7,5,9,4]
axes.plot(pontos, 'y*', markersize=20)
axes.plot(pontos, 'b', linestyle="dotted")
axes.axis([-1,8,0,10])


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
O comando axis permite ajustarmos os limites horizontais e verticais do gráfico:
Can you write Python code for it?


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
O comando subplot() especifica (linha, coluna, figura) em que figura vai de 1 até linhascolunas (As vírgulas são opcionais se linhascolunas<10; então subplot(211) é o mesmo que subplot(2, 1, 1).
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.figure(1)                # primeira figura
plt.subplot(211)             # primeiro subplot na primeira figura
plt.plot([1, 2, 3])
plt.subplot(212)             # segundo subplot na primeira figura
plt.plot([4, 5, 6])

plt.figure(2)                # segunda figura
plt.plot([4, 5, 6])          # cria um subplot(111) automaticamente

plt.figure(1)                # na figura 1, o subplot(212) ainda é o mais atual
plt.subplot(211)             # selecionar o subplot(211) como atual
plt.title('Easy as 1, 2, 3') # atualizar o título do subplot(211)


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Para saber mais:
Quickstart: https://docs.scipy.org/doc/numpy-dev/user/quickstart.html
Diferenças entre Numpy e MATLAB: https://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html
Observação: namespaces
Talvez você já tenha percebido que ao executarmos comandos, nos referimos ao módulo em que o comando está definido. Por exemplo, para usar a função transpose do módulo numpy, que importamos como np, usamos o comando
np.transpose(A)
isso significa que a função transpose que estamos utilizando "mora" dentro do módulo numpy.
Exemplo:
Can you write Python code for it?

import math
import numpy
import scipy

print(math.pi, "do módulo math")
print(numpy.pi, "do módulo numpy")
print(scipy.pi, "do módulo scipy")

import math as matematica

matematica.pi

import math as biblioteca

biblioteca.pi


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Podemos selecionar a escala do gráfico usando
plt.xscale(‘log’)

por exemplo.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

# inventar dados no intervalo ]0, 1[
y = np.random.normal(loc=0.5, scale=0.4, size=1000)
y = y[(y > 0) & (y < 1)]
y.sort()
x = np.arange(len(y))

# criar gráficos com diversas escalas
plt.figure()

# linear
plt.subplot(221)
plt.plot(x, y)
plt.yscale('linear')
plt.title('linear')
plt.grid(True)

# log
plt.subplot(222)
plt.plot(x, y)
plt.yscale('log')
plt.title('log')
plt.grid(True)

# log simétrico
plt.subplot(223)
plt.plot(x, y - y.mean())
plt.yscale('symlog', linthreshy=0.05)
plt.title('symlog')
plt.grid(True)

# logit
plt.subplot(224)
plt.plot(x, y)
plt.yscale('logit')
plt.title('logit')
plt.grid(True)


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
Também podemos selecionar o tipo de traço e a cor, usando exatamente a mesma sintaxe que o MATLAB:
Can you write Python code for it?

fig = plt.figure()
plt.plot([1,2,3,4], [1,4,9,16], 'ro')
plt.plot([1,2,3,4],[1,4,9,16], 'g', linewidth=2)
plt.plot([1,3,5,7,9], linestyle="dashed", linewidth=4)

import numpy as np
import matplotlib.pyplot as plt

# t vai de 0 a 5, com subintervalos de 0.2 unidades de comprimento
t = np.arange(0., 5., 0.2)

fig = plt.figure()
# em tracejado vermelho faremos o gráfico de t;
# em quadrados azuis faremos o gráfico de t**2;
# em triângulos verdes faremos o gráfico de t**3
plt.plot(t, t, 'r--', t, t**2, 'bs', t, t**3, 'g^')
plt.show()

fig = plt.figure()
plt.plot([1,3,2,7,5,9,4], 'y*', markersize=20)


I am working on the file "Notebooks/Aula_5.ipynb".
The first task is:
ou
Can you write Python code for it?


I am working on the file "Notebooks/Bessel-Ceros.ipynb".
The first task is:
Ceros de las funciones de Bessel de orden entero $J_\nu(\alpha_{\nu,n})=0$:
En el módulo scipy.special las raíces de las funciones de Bessel están implementadas en la función jn_zeros. En particular jn_zeros(nu,N) entrega un arreglo con las primeras $N$ raíces de la función $J_\nu$.
Can you write Python code for it?

for nu in range(5):
    print('nu = '+str(nu)+': '+str(jn_zeros(nu,5)))


I am working on the file "Notebooks/Bessel-Ceros.ipynb".
The first task is:
En este caso, el gráfico de las diferencias es también similar al caso anterior:
Can you write Python code for it?

figure()
nmax = 20
numax = 10
for nu in range(numax):
    plot(range(nmax-1),diff(jnp_zeros(nu,nmax)), marker='o', markersize=5, lw=1, label='$\\nu = %d$'%nu)
xlim(-1,nmax-1)
ylim(3,)
xlabel('$n$', fontsize=15)
ylabel('$\\beta_{\\nu,n+1}-\\beta_{\\nu,n}$', fontsize=15)
xticks(range(nmax))
legend()
grid()


I am working on the file "Notebooks/Bessel-Ceros.ipynb".
The first task is:
Podemos graficar estas raíces, de la forma siguiente:
Can you write Python code for it?

figure()
nmax = 20
numax = 10
for nu in range(numax):
    scatter(nu*ones(nmax),jn_zeros(nu,nmax))
ylim(0,50)
xlim(-1,numax)
xlabel('$\\nu$', fontsize=15)
ylabel(r'$\alpha_{\nu,n}$', fontsize=15)
xticks(range(numax))
grid()
savefig('../figs/fig-Bessel-ceros-01.pdf')


I am working on the file "Notebooks/Brazilian_epiweek.ipynb".
The first task is:
Example 1: 2016-10-31
Function episem can take 1 to 3 inputs:
episem(x, sep='W', out='YW')
'''
@param x: date in format string YYYY-MM-DD or as datetime.datetime
@param sep: separator character
@param out: returned info. Y returns epiyear alone, W returns only epiweek and YW returns
epiyear and epiweek separated by sep

returns str
'''

Passing string
Can you write Python code for it?


I am working on the file "Notebooks/Brazilian_epiweek.ipynb".
The first task is:
Example 2: 2016-01-01
2016-10-01 was a Friday, as can be seen by isoweekday function, which returns 1 for Monday and 7 for Sunday:
Can you write Python code for it?

dt2 = datetime.datetime.strptime('2016-01-01', '%Y-%m-%d')
dt2.isoweekday()


I am working on the file "Notebooks/Brazilian_epiweek.ipynb".
The first task is:
Passing datetime.datetime
Can you write Python code for it?

import datetime

datetime.datetime.strptime(d, '%Y-%m-%d')

dt = datetime.datetime.strptime(d, '%Y-%m-%d')
episem(dt)


I am working on the file "Notebooks/Brazilian_epiweek.ipynb".
The first task is:
Therefore, according to Brazilian epiweek system, it should fall on the last epiweek of year 2105:
Can you write Python code for it?


I am working on the file "Notebooks/CAPEC/Fielder_Parser/CAPEC Field Parser.ipynb".
The first task is:
0. Preparing Data
Before digging into the parser notebook, the version of the CAPEC xml file within this notebook is v2.11, which can be downloaded from this link. Here we loaded CAPEC v21 xml file. Therefore, if there is any new version of XML raw file, please make change for the following code. If the order of attack pattern table is changed, please change the code for function <b>extract_target_field_elements</b> in section 2.1.
Can you write Python code for it?


I am working on the file "Notebooks/CAPEC/Fielder_Parser/CAPEC Field Parser.ipynb".
The first task is:
2.2 XML Node Field Parser
Once the node is provided by the former function, its XML structure is consistent for that field, and CAPEC version, but it is inconsistent for different versions. For example, a table represented in XML is different than a paragraph in XML. However, even for the same expected structure such as a table, the XML may have different tags used within it.
The associated parser function then is tested to cover all possible cases of this field in the XML, and also interpreted against its .html format to understand its purpose. We again refer to the introductory notebook Sections 4 and 5 which convey respectively the potential usage of the field for PERCEIVE and the overall format of the structure when compared to others.
The purpose is then documented as part of the functions documentation (sub-sections of this Section), conveying the rationale behind what was deemed potentially useful in the content to be kept, as well as how certain tags were mapped to a data structure while being removed.
The parser function outputs one of the known data structures (i.e. memory representation) that is shared among the different fields of what is deemed relevant. For example, while 5 field nodes may have their relevant information in different ways, they may all be at the end of the day tables, named lists, or single blocks of texts. Sharing a common representation in this stage decouples the 3rd step of the pipeline from understanding the many ways the same information is stored in the XML.
Because different CAPEC versions may organize a .html table, bullet list etc in different ways even for the same field, this organization also decouples the previous and following section functions from being modified on every new version if necessary.
The following fields have parser functions as of today:
|Field Name| Function Name|
|:---:|:----:|
|Solutions_and_Mitigations|parse_list_format_field|
|Attack_Prerequisites|parse_list_format_field|
2.2.1 Parse the fields in list format
Both Solutions and Mitigations...
Can you write Python code for it?

def parse_list_format_field(field_node):
    '''
    The parser function concern is abstracting how Solutions_and_Mitigations field and Attack_Prerequisites field are stored in XML,
    and provide it in a common and simpler data structure

    Args:
        - field_node: the node that has the Solutions_and_Mitigations tag or Attack_Prerequisites tag, such as the above image

    Outcomes:
        - A dictionary that pairs capec_id as key and the content list as value.
          In the dictionary, the content list will be a list of strings that contain bullet or unbullet content
          More details can be found in the following example for CAPEC-10.
    '''

    # extract capec_id from the attribute of its parent node
    capec_id=field_node.getparent().attrib.get('ID')
    capec_id='CAPEC_'+capec_id
    field_tag=field_node.tag
    field_content_list=[]
    # for each field entry node under the target field node
    for field_entry in list(field_node):
        # traverse all entry_element nodes under each field entry
        for entry_text in list(field_entry):
            # generate tag and content of each entry_element
            entry_text_content=entry_text.text
            # if there is not content, then move to next entry element
            if type(entry_text_content)==type(None):
                    continue
            #add each point to the content list
            field_content_list.append(entry_text_content)
    # pair the capec_id with the content list
    field_dict=dict()
    field_dict[capec_id]=field_content_list

    return field_dict


I am working on the file "Notebooks/CAPEC/Fielder_Parser/CAPEC Field Parser.ipynb".
The first task is:
4. Main Execution
After developing the 3 steps parsing pipeline, this section will combine these 3 steps and produce the output file for different fields. As introduced in Section 2, although the parsing procedure keeps same for all fields, each field will have own parsing function, while the same format of fields may share a same exporting function. As a result, the main execution varies for each field.
4.1 Main execution for Solutions and Mitigations
The main execution will combine the above 3 steps parsing pipeline for Solutions_and_Mitigations. After developing function <b>export_data</b>, the following code should produce the output file that contains the parsed content of Solutions_and_Mitigations for all CAPEC_id.
Can you write Python code for it?

if __name__ == "__main__":
    # extract the nodes, whose tag is Solutions and Mitigation, from capec_xml_file
    solutions_and_mitigations_nodes=extract_target_field_elements('Solutions_and_Mitigations',capec_xml_file)
    # read each Solutions and Mitigations node
    for solutions_and_mitigations_node in solutions_and_mitigations_nodes:
        # parse the content for each solutions_and_mitigation_node
        solutions_and_mitigations_info=parse_list_format_field(solutions_and_mitigations_node)
        # export the parsed content  TO-DO
        export_data(solutions_and_mitigations_info)


I am working on the file "Notebooks/CAPEC/Fielder_Parser/CAPEC Field Parser.ipynb".
The first task is:
4.2 Main execution for Attack Prerequisites
The main execution will combine the above 3 steps parsing pipeline for Attack_Prerequisites. After developing function <b>export_data</b>, the following code should produce the output file that contains the parsed content of Attack_Prerequisites for all CAPEC_id.
Can you write Python code for it?

if __name__ == "__main__":
    # extract the nodes, whose tag is Attack Prerequisites, from capec_xml_file
    attack_prerequisites_nodes=extract_target_field_elements('Attack_Prerequisites',capec_xml_file)
    # read each Attack_Prerequisites node
    for attack_prerequisites_node in attack_prerequisites_nodes:
        # parse the content for each Attack_Prerequisites node
        attack_prerequisites_info=parse_list_format_field(attack_prerequisites_node)
        # export the parsed content  TO-DO
        export_data(attack_prerequisites_info)


I am working on the file "Notebooks/CAPEC/Fielder_Parser/CAPEC Field Parser.ipynb".
The first task is:
Through Function  parse_list_format_field, the above Solutions_and_Mitigations node for CAPEC-10 will be parsed into the following data format in memory.

 How content represent in memory for Solutions_and_Mitigations field (CAPEC-10)

Through Function  parse_list_format_field, the above Attack_Prerequisites node for CAPEC-10 will be parsed into the following data format in memory.

 How content represent in memory for Attack_Prerequisites field (CAPEC-10)

3. Export Data Structure
At the point this notebook is being created, it is still an open end question on how will tables, bullet lists and other potential structures in CAPEC will be used for topic modeling. For example, should rows in a table be concatenated into a paragraph and used as a document? What if the same word is repeated as to indicate a lifecycle?
In order to keep this notebook future-proof, this section abstracts how we will handle each field representation (e.g. table, bullet list, etc) from the memory data structure. It also keeps it flexible for multi-purpose: A table may be parsed for content for topic modeling, but also for extracting graph relationships (e.g. the Related Attack Pattern and Related Weaknesses fields contain hyperlinks to other CAPEC entries which could be reshaped as a graph).
Can you write Python code for it?

def export_data(target_field_node):
    '''This section code will be done in the future.'''
    pass


I am working on the file "Notebooks/CDAE.ipynb".
The first task is:
Network definitions
Can you write Python code for it?

from CDAE import create_encoder, create_decoder, create_cdae_trainer