I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
1.2 Préparation des données
Un travail important de nettoyage des données doit être réalisé pour supprimer les données manquantes, éviter des erreurs de codage, regrouper des modalités bien trop nombreuses, éliminer des variables redondantes. Cette étape de préparation et d'exploration est réalisée dans la saison 2- exploration. Seuls sont reproduits ci-dessous les codes permmant la rpéapration du fichier.
Can you write Python code for it?

def create_categorical_data(df, column_name):
    cat_columns = pd.Categorical(df[column_name], ordered=False)
    return cat_columns

print(np.sort(adult["mariStat"].unique()))

# mariStat
cat_name_dic = {" Never-married": "Never-Married", " Married-AF-spouse": "Married",
                " Married-civ-spouse": "Married", " Married-spouse-absent": "Not-Married",
                " Separated": "Not-Married", " Divorced": "Not-Married", " Widowed": "Widowed"}
adult['mariStat'] = adult.mariStat.map(cat_name_dic)

print(np.sort(adult["nativCountry"].unique()))

# nativCountry
cat_country = {" Cambodia": "SE-Asia", " Canada": "British-Commonwealth", " China": "China", " Columbia": "South-America",
               " Cuba": "Other", " Dominican-Republic": "Latin-America", " Ecuador": "South-America",
               " El-Salvador": "South-America", " England": "British-Commonwealth", " France": "Euro_1",
               " Germany": "Euro_1", " Greece": "Euro_2", " Guatemala": "Latin-America", " Haiti": "Latin-America",
               " Holand-Netherlands": "Euro_1", " Honduras": "Latin-America", " Hong": "China", " Hungary": "Euro_2",
               " India": "British-Commonwealth", " Iran": "Other", " Ireland": "British-Commonwealth", " Italy": "Euro_1",
               " Jamaica": "Latin-America", " Japan": "Other", " Laos": "SE-Asia", " Mexico": "Latin-America",
               " Nicaragua": "Latin-America", " Outlying-US(Guam-USVI-etc)": "Latin-America", " Peru": "South-America",
               " Philippines": "SE-Asia", " Poland": "Euro_2", " Portugal": "Euro_2", " Puerto-Rico": "Latin-America",
               " Scotland": "British-Commonwealth", " South": "Euro_2", " Taiwan": "China", " Thailand": "SE-Asia",
               " Trinadad&Tobago": "Latin-America", " Vietnam": "SE-Asia", " United-States": "United-States",
               " Yugoslavia": "Euro_2"}
adult["nativCountry"] = adult.nativCountry.map(cat_country)

print(np.sort(adult["education"].unique()))

# education
cat_educ = {" 10th": "Dropout", " 11th":...


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
1.3 Description multivariée
Une afcm serait bienvenue à ce niveau de l'étude; elle est exécutée dans le calepin en R avec des graphiques plus élégants.
1.4 Variables indicatrices
Q Pourquoi l’introduction de dummy variables? Pour quelles méthodes cela serait-il aussi nécessaire en R?
Can you write Python code for it?

adultDum=pd.get_dummies(adult[["workClass","education","mariStat",
    "occup","origEthn","sex","capitalGain","capitalLoss","ageQ","hoursWeekQ","child"]])
adultDum.head()

adultJoin = adult[["age","educNum","hoursWeek","LcapitalGain","LcapitalLoss","income"]].join(adultDum)


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
1.3 Description élémentaire
Can you write Python code for it?


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
2 Méthodes de modélisation
2.1  Extraction des échantillons
Q Quels sont les objectifs des cellules suivantes ? Justifier la nécessité de ce traitement.
Can you write Python code for it?

ind_ech = np.random.choice(adultJoin.index.values, 20000, replace=False)
adultEch=adultJoin.loc[ind_ech]
# Variable cible
Y=adultEch["income"]
# Variables prédictives
X=adultEch.drop(["income"],axis=1)

from sklearn.model_selection import train_test_split
X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=5000,random_state=11)


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
2.2 Arbre binaire de discrimination
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier
tps0=time.perf_counter()
# Optimisation de la profondeur de l'arbre
param=[{"max_depth":list(range(2,10))}]
tree= GridSearchCV(DecisionTreeClassifier(),param,cv=10,n_jobs=-1)
treeOpt=tree.fit(X_train, Y_train)
# paramètre optimal
tps1=(time.perf_counter()-tps0)
print("Temps arbre = %f, Meilleur taux = %f, Meilleur paramètre = %s" % (tps1,
                             1. - treeOpt.best_score_,treeOpt.best_params_))

# Estimation de l'erreur de prévision
1-treeOpt.score(X_test,Y_test)

# prévision de l'échantillon test
y_chap = treeOpt.predict(X_test)
# matrice de confusion
table=pd.crosstab(y_chap,Y_test)
print(table)

treeOpt.best_params_['max_depth']

from sklearn.tree import export_graphviz
from sklearn.externals.six import StringIO
import pydotplus
treeG=DecisionTreeClassifier(max_depth=treeOpt.best_params_['max_depth'])
treeG.fit(X_train,Y_train)
dot_data = StringIO()
export_graphviz(treeG, out_file=dot_data)
graph=pydotplus.graph_from_dot_data(dot_data.getvalue())
graph.write_png("treeOpt.png")

from IPython.display import Image
Image(filename='treeOpt.png')


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
2.2 Régression logistique
Q Commenter  les options de la commande  GridSearchCV. A quoi sert param ?
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV
import time
tps0=time.perf_counter()
# Optimisation du paramètre de pénalisation
# grille de valeurs
param=[{"C":[0.8,0.9,1,1.1,1.2]}]
logit = GridSearchCV(LogisticRegression(penalty="l1",solver="liblinear"), param,cv=10,n_jobs=-1)
logitOpt=logit.fit(X_train, Y_train)  # GridSearchCV est lui même un estimateur
# paramètre optimal
logitOpt.best_params_["C"]
tps1=(time.perf_counter()-tps0)
print("Temps logit = %f, Meilleur taux = %f, Meilleur paramètre = %s" % (tps1,
                              1.-logitOpt.best_score_,logitOpt.best_params_))

# erreur sur l'échantillon test
1-logitOpt.score(X_test, Y_test)

# Prévision
y_chap = logitOpt.predict(X_test)
# matrice de confusion
table=pd.crosstab(y_chap,Y_test)
print(table)


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
2.4 Forêts aléatoires
Q Commenter les choix de tous les paramètres.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
# définition des paramètres
forest = RandomForestClassifier(n_estimators=500,
   criterion='gini', max_depth=None,
   min_samples_split=2, min_samples_leaf=1,
   max_features='auto', max_leaf_nodes=None,
   bootstrap=True, oob_score=True)
# apprentissage
rfFit = forest.fit(X_train,Y_train)
print(1-rfFit.oob_score_)

# erreur de prévision sur le test
1-rfFit.score(X_test,Y_test)

# optimisation du paramètre
tps0=time.perf_counter()
param=[{"max_features":list(range(2,10,1))}]
rf= GridSearchCV(RandomForestClassifier(n_estimators=100),param,cv=10,n_jobs=-1)
rfOpt=rf.fit(X_train, Y_train)
# paramètre optimal
tps1=(time.perf_counter()-tps0)
print("Temps r forest = %f, Meilleur taux = %f, Meilleur paramètre = %s" % (tps1,
                                    1. - rfOpt.best_score_,rfOpt.best_params_))

# erreur de prévision sur le test
1-rfOpt.score(X_test,Y_test)

# prévision
y_chap = rfFit.predict(X_test)
# matrice de confusion
table=pd.crosstab(y_chap,Y_test)
print(table)

rf= RandomForestClassifier(n_estimators=100,max_features=6)
rfFit=rf.fit(X_train, Y_train)
# Importance décroissante des variables
importances = rfFit.feature_importances_
indices = np.argsort(importances)[::-1]
for f in range(20):
    print(X_train.columns[indices[f]], importances[indices[f]])

# Graphe des importances
plt.figure()
plt.title("Importances des variables")
plt.bar(range(X_train.shape[1]), importances[indices])
plt.xticks(range(X_train.shape[1]), indices)
plt.xlim([-1, X_train.shape[1]])
plt.show()


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
3 Comparaison des méthodes
3.1 Courbes ROC
Q En cohérence avec les résultats précédents, quelle est la courbe la plus au dessus des autres? Commenter ce graphique, que signifie AUC ?
Can you write Python code for it?

from sklearn.metrics import roc_curve
listMethod=[["GBM",gbmOpt],["RF",rfOpt],["NN",nnetOpt],["Tree",treeOpt],["Logit",logitOpt]]

# Courbes ROC des précédents modèles optimaux
for method in enumerate(listMethod):
    probas_ = method[1][1].fit(Xnet_train, Y_train).predict_proba(Xnet_test)
    fpr, tpr, thresholds = roc_curve(Y_test,probas_[:,1], pos_label="incLow")
    plt.plot(fpr, tpr, lw=1,label="%s"%method[1][0]),
plt.xlabel('Taux de faux positifs')
plt.ylabel('Taux de vrais positifs')
plt.legend(loc="best")
plt.show()


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
Q Comment est obtenu le graphique ? Quelle importance ? Comment interpréter ces résultats ?
2.5 Gradient boosting
Q Pourquoi pas de paramètre njobs=-1?
Q En plus de celui optimiser, quels sont les 2 principaux paramètres cet algorithme laissés par défaut ?
Can you write Python code for it?

from sklearn.ensemble import GradientBoostingClassifier
tps0=time.perf_counter()
param=[{"n_estimators":[200, 250, 300]}]
gbm= GridSearchCV(GradientBoostingClassifier(),param,cv=10)
gbmOpt=gbm.fit(X_train, Y_train)
# paramètre optimal
tps1=(time.perf_counter()-tps0)
print("Temps boosting = %f, Meilleur taux = %f, Meilleur paramètre = %s" % (tps1,
                        1. - gbmOpt.best_score_,gbmOpt.best_params_))

# erreur de prévision sur le test
1-gbmOpt.score(X_test,Y_test)

# prévision de l'échantillon test
y_chap = gbmOpt.predict(X_test)
# matrice de confusion
table=pd.crosstab(y_chap,Y_test)
print(table)


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
Q La matrice de confusion n’est pas symétrique. Quelle pourrait en être la raison ?
Q Quels algorithmes pourraient être exécutés en R pour la régression logistique? Quelles informations complémentaires en tirer?
Can you write Python code for it?

# Coefficients
LogisticRegression(penalty="l1",C=logitOpt.best_params_['C'],
                   solver='liblinear').fit(X_train, Y_train).coef_


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
Q Que dire des liaisons : age x hoursWeek, age x income, sex x income ?
Can you write Python code for it?

adult.plot(kind="scatter",x="hoursWeek",y="age")
plt.show()

adult.boxplot(column="age",by="income")
plt.show()


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
Q Quelle est l’insuffisance de l’implémentation des arbres de décision dans Scikit-learn  par rapport à celle de rpart de R ? Que dire de l’arbre?
2.3 Perceptron
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import StandardScaler
# L'algorithme ds réseaux de neurones nécessite éventuellement une normalisation
# des variables explicatives avec les commandes ci-dessous
scaler = StandardScaler()
scaler.fit(X_train.astype(float))
Xnet_train = scaler.transform(X_train.astype(float))
# Meme transformation sur le test
Xnet_test = scaler.transform(X_test.astype(float))

tps0=time.perf_counter()
param_grid=[{"hidden_layer_sizes":list([(4,),(5,),(6,)])}]
nnet= GridSearchCV(MLPClassifier(max_iter=500),param_grid,cv=10,n_jobs=-1)
nnetOpt=nnet.fit(Xnet_train, Y_train)
# paramètre optimal
tps1=(time.perf_counter()-tps0)
print("Temps perceptron = %f, Meilleur taux = %f, Meilleur paramètre = %s" % (tps1,
                                    1. - nnetOpt.best_score_,nnetOpt.best_params_))


I am working on the file "Adult-Census/Apprent-Python-AdultCensus.ipynb".
The first task is:
Q Quelle stratégie d’optimisation est adoptée ? Quelle autre pourrait l’être? Quel réseau pourrait également être pris en compte? Quelles sont les fonctions d’activation des neurones?
Can you write Python code for it?

# Estimation de l'erreur de prévision sur le test
1-nnetOpt.score(Xnet_test,Y_test)

# prévision de l'échantillon test
y_chap = nnetOpt.predict(Xnet_test)
# matrice de confusion
table=pd.crosstab(y_chap,Y_test)
print(table)


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
Columnar: Top 10 Products Sold
Can you write Python code for it?

sqlin = \
'''
WITH
  SALES(product, quantity) AS (
      SELECT PRODUCT_NAME, PRODUCT_QTY
      from TRANSACTIONS
      )
SELECT product, sum(quantity) from SALES
GROUP BY product
ORDER BY 2 DESC
FETCH FIRST 10 ROWS ONLY
WITH UR
'''
plotSQL(sqlin,
    "Products Sold",
    "Product",
    "Quantity",
    True,
    m2)


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
Columnar: Top 10 States by Sales
Can you write Python code for it?

sqlin = \
'''
WITH
SALES(state, amount) AS
   (
   SELECT STATE, SUM(PRODUCT_QTY * PRODUCT_COST)
   from TRANSACTIONS
   GROUP BY STATE
   )
SELECT state, sum(amount) from SALES
GROUP BY state
ORDER BY 2 DESC FETCH FIRST 10 ROWS ONLY
WITH UR
'''
plotSQL(sqlin,
    "Sales per State",
    "State",
    "Revenue",
    False,
    m1)


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
Converting JSON into Relational Tables
As you can see from the previous examples, it is possible to analytics against the JSON data. However, it requires some sophisticated techniques for manipulating (and converting) the JSON data. One approach is to use the JSON functions and move the data into a traditional relational table for analysis. The following SQL will create a "flat" relational table that will allow for further analysis without having to do joins. In addition, we are going to create the table using column organization which will give us the ability to do queries without having to worry about what indexes are needed.
Can you write Python code for it?

%%sql
DROP TABLE TRANSACTIONS;
CREATE TABLE TRANSACTIONS
  (
  TX_DATE DATE,
  CITY VARCHAR(32),
  STATE CHAR(2),
  ZIPCODE INT,
  PRODUCT_NAME VARCHAR(32),
  PRODUCT_QTY INT,
  PRODUCT_COST DEC(7,2),
  CCARD CHAR(4)
  )
ORGANIZE BY COLUMN;


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
Db2 JSON Macro Extensions
The Db2 notebook on JSON features described how to retrieve individual fields from a record by using the JSON_VAL function. The following code retrieves the first and last name of customer number 100000.
Can you write Python code for it?

%%sql
SELECT JSON_VAL(DETAILS,'identity.firstname','s:32') AS FIRSTNAME,
       JSON_VAL(DETAILS,'identity.lastname','s:32') AS LASTNAME
FROM CUSTOMERS
WHERE CUSTNO = 100000


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
In order to make JSON retrieval easier in Jupyter notebooks, the Db2 Jupyter %sql command allows for the creation of macros. Macros are used to expand text in a SQL statement. Macros do not execute any code, but are used specifically to insert text into a SQL statement. An example of a macro is found below.
Can you write Python code for it?

%%sql macro helloworld
echo Everything you said is here: {1}
exit

%sql %helloworld(Hello World)


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
Most Popular Product by State
Here we need to get the product counts by state, and then pick the maximum quantity found in the answer set. The steps we need to perform in the SQL are:

Get all products purchased
Sum the sales of products by state
Compute the max for each state

The next set of SQL statements will get us the results. Note that it is possible to have more than one product being the top seller in a state. Note: This SQL may take a while to execute.
Can you write Python code for it?

%%sql -a
WITH
  %jsonarray(customers,custno,details,purchases),
  SALES(product, state, quantity) AS (
      SELECT %js(j.item,product,32), %js(c.details,contact.state,2), SUM(%ji(j.item,quantity))
      from jsonarray j, customers c
      where j.pk = c.custno
      GROUP BY %js(j.item,product,32), %js(c.details,contact.state,2)
      ),
  MAXSALES(state, total) AS (
      SELECT STATE, MAX(quantity)
      FROM SALES
      GROUP BY STATE
  )
SELECT s.state, s.product, m.total
  FROM SALES s, MAXSALES m
WHERE
  s.state = m.state and
  s.quantity = m.total
ORDER BY s.product desc


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
Orders of a Product
This calculation gives us the number of times a product was ordered. This doesn't tell us the quantity since we need to look at the individual purchases to see how many of the product the customer actually ordered. The SQL command will use the -a flag so that the output will be displayed in a TABLE so that you can find it easier to scroll the results.
Can you write Python code for it?

%%sql -a
WITH
  %jsonarray(customers,custno,details,purchases),
  SALES(product, quantity) AS (
      SELECT %js(j.item,product,32),count(*)
      from jsonarray j
      GROUP BY %js(j.item,product,32)
      )
SELECT product, quantity from SALES
ORDER BY 2 DESC


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
The next piece of code will import the data into Db2. The Python code will read the customer.js file, parse it and then insert in into the CUSTOMERS table. Note that the entire JSON record is stored into the DETAILS column, while the CUSTNO column is set to the customerid field that is found in the JSON record.
Can you write Python code for it?

import io
import json
print("Starting Load")
start_time = time.time()
%sql autocommit off
x = %sql prepare INSERT INTO CUSTOMERS_BSON VALUES (?,SYSTOOLS.JSON2BSON(?))
if (x != False):
    i = 0
    with open("customers.js","r") as records:
        for record in records:
            rec = json.loads(record)
            custno = rec['customerid']
            i += 1
            rc = %sql execute :x using :custno, :record
            if (rc == False): break
            if ((i % 5000) == 0):
                print(str(i)+" rows read.")
                %sql commit hold

    %sql commit work
%sql autocommit on
end_time = time.time()
print('Total load time for {:d} records is {:.2f} seconds'.format(i,end_time-start_time))
customer_bson = end_time - start_time


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
The table does not have an index currently defined on it. We could have created the table with a primary key but chose not to in order to speed up the load step. The next statement will create the index for our table.
Can you write Python code for it?

%sql CREATE UNIQUE INDEX IX_CUST_BSON ON CUSTOMERS_BSON(CUSTNO)


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
This code can be simplified by using the %jsonarray macro instead.
Can you write Python code for it?

%%sql -e
WITH
  %jsonarray(customers,custno,details,purchases,where="custno = 100000")
SELECT
  %ji(item,product_id),%js(item,product,32),%ji(item,quantity),%jd(item,item_cost,'7,2') from jsonarray


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
This section will work with a larger table that will give us some idea of the performance of using JSON within a Db2 database. This next statement will generate 25,000 customer records that have a structure similar to this:
json
{
    "customerid": 100000,
    "identity": {
        "firstname": "Jacob",
        "lastname": "Hines",
        "birthdate": "1982-09-18"
    },
    "contact": {
        "street": "Main Street North",
        "city": "Amherst",
        "state": "OH",
        "zipcode": "44001",
        "email": "Ja.Hines@yahii.com",
        "phone": "813-689-8309"
    },
    "payment": {
        "card_type": "MCCD",
        "card_no": "4742-3005-2829-9227"
    },
    "purchases": [
        {
            "tx_date": "2018-02-14",
            "tx_no": 157972,
            "product_id": 1860,
            "product": "Ugliest Snow Blower",
            "quantity": 1,
            "item_cost": 51.86
        },...
    ]
}
The JSON record contains four distinct pieces of information:

Customerid - Primary key
Identity - Information on the customer including name and birthdate
Contact - Address, email, and phone number information
Payment - Current payment card that is used
Purchase - The purchase that the customer has made

The purchase structure contains information on the customer purchases. For each purchased items there is the following information:

tx_date - Date of the transaction
tx_no - Transaction number
product_id - Id for the product
product - Name of the product
quantity - Quantity of products purchased
item_cost - Cost of one product

If this was a relational database you would probably split these fields up into different tables and use join techniques to bring the information back together. In a JSON document we are able to keep all of this information in one record, which makes retrieval of an individual customer purchases easier.
Connect to the Db2 Database
You may need to add connection information here if you have not yet connected to the database.
Can you write Python code for it?


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
Top Products Sales by State
Can you write Python code for it?

import seaborn as sns
cm = sns.light_palette("green", as_cmap=True)
sqlin = \
"""
WITH
  SALES(product, state, quantity) AS (
      SELECT PRODUCT_NAME, STATE, SUM(PRODUCT_QTY)
      from TRANSACTIONS
      GROUP BY PRODUCT_NAME, STATE
      ),
  MAXSALES(state, total) AS (
      SELECT STATE, MAX(quantity)
      FROM SALES
      GROUP BY STATE
  )
SELECT s.state, s.product, m.total
  FROM SALES s, MAXSALES m
WHERE
  s.state = m.state and
  s.quantity = m.total
ORDER BY m.total desc
"""
result = %sql {sqlin}
result.style.background_gradient(cmap=cm)


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
Using the -e flag is useful when debugging any SQL that uses JSON functions. In addition it gives you the raw SQL that is generated so that you can cut and paste it into your own application. The examples with strings use the following format:
TRIM(JSON_VAL(...'s:12'))
If you use a specification that is too small to retrieve the field (s:12), then the JSON_VAL will return a NULL value rather than a truncated string. For this reason, you may want to use a larger size and the value will be trimmed down to a smaller size.
The final macro that is included in the list is the #jsonarray() macro. The macro call has the following format:
jsonarray(table_name, pk, json_column, array, [optional where clause])
The fields are:
- table_name - table that we are accessing
- pk - primary key of the table
- json_column - column that contains the json data
- array - the field that contains an array of items (which must be a structure and not individual values)
- where clause - An optional expression to limit the result set
The #jsonarray macro is used in situations where you want to retrieve the contains of an array within a JSON record. The macro will create a WITH clause specification where the name of the resulting table is called JSONARRAY with two columns: PK for the primary key to the array record and ITEM for the items returned. The following example shows how you would retrieve all of the purchases that a customer made using the JSON_TABLE function.
Can you write Python code for it?

%%sql
with purchases(items) as
   (select systools.json2bson(items.value)
    from customers,
    table( systools.json_table(customers.details,'purchases','s:2048')) as items
    where custno = 100000
   )
select %ji(items,product_id),%js(items,product,32),%ji(items,quantity),%jd(items,item_cost,'7,2') from purchases


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
We need to create JSON records for loading into the table. This next command will run the db2json.ipynb file which contains code to generate JSON records. If you want to change the number of records created or the data used, edit the file to modify the defaults. The program will display its progress as it runs.
Can you write Python code for it?


I am working on the file "Advanced Db2 JSON Techniques.ipynb".
The first task is:
You will notice that a green box is placed around the generated code and the SQL has been modified by the macro to include the full JSON_VAL function. This makes it easier to create the SQL (and hopefully less error prone) when dealing with JSON objects. There are 5 macros that have been predefined for JSON queries.

-e - This flag turns on the display of the generated SQL. This is useful if you want to cut and paste the final SQL into an application. If you do not use the -e flag, the code will not be displayed.
#js - Return a string from a JSON field
#ji - Return an integer from a JSON field
#jd - Return a number (decimal) from a JSON field
#jdate - Return a date from a JSON field

The format for the ji (Integer) and jdate (Date) macro is:
```
ji(json_column,json_field)
jdate(json_column,json_field)
```
Note that you do not need to enclose quotes around the column or field names unless they have special characters in the name. The quotes are stripped out of the parameters but the macros will add them back into the SQL function if necessary. Also, field names are case sensitive in JSON so make sure you use the proper spelling for the field you are trying to retrieve. The column name that contains JSON data is not case sensitive.
The #jd macro works similar to the #ji macro except that it provides for an additional decimal formatting operator:
```
jd(json_column, json_field, 'digits,decimals')
```
A decimal number if formatted as x,y where x is the number of digits in total, and y represents the number of digits after the decimal point. So a value of 5,2 would mean there are 5 digits in total with 2 of them after the decimal point.
The #js (JSON string) macro the following format:
```
js(column,field,length)
```
The macro needs which JSON column to use, the field that is being searched for, and the length that needs to be returned. Note that you do not use the JSON_VAL specification in this field ('s:255'). Instead you are supplying the length of the return string as an integer.
The next SQL command...
Can you write Python code for it?

%%sql -e
SELECT
  %js(details,identity.firstname,12) AS FIRSTNAME,
  %js(details,identity.lastname,15) AS LASTNAME,
  %js(details,contact.street,30) AS STREET,
  %js(details,contact.city,20) AS CITY,
  %js(details,contact.state,2) AS STATE,
  %ji(details,contact.zipcode) AS ZIPCODE,
  %jdate(details,identity.birthdate) AS BIRTHDATE,
  %js(details,contact.phone,12) AS PHONE
FROM CUSTOMERS
WHERE CUSTNO = 100000


I am working on the file "Advanced Dictionaries.ipynb".
The first task is:
One of the reasons it is not as common is the difficulty in structuring the key names that are not based off the values.
Iteration over keys,values, and items
Dictionaries can be iterated over using the iter methods available in a dictionary. For example:
Can you write Python code for it?

for k in d.iterkeys():
    print k

for v in d.itervalues():
    print v

for item in d.iteritems():
    print item


I am working on the file "Advanced Functions Test.ipynb".
The first task is:
Problem 3
Use filter to return the words from a list of words which start with a target letter.
Can you write Python code for it?

def filter_words(word_list, letter):
    return filter(lambda x: x[0] is letter, word_list)

l = ['hello','are','cat','dog','ham','hi','go','to','heart']
filter_words(l,'h')


I am working on the file "Advanced Functions Test.ipynb".
The first task is:
Problem 5
Use enumerate and other skills to return a dictionary which has the values of the list as keys and the index as the value. You may assume that a value will only appear once in the given list.
Can you write Python code for it?

def d_list(L):
    return_dic = {}
    for i,v in enumerate(L):
        return_dic[v] = i
    return return_dic

def d_list_one_zip(L):
    # the trick is that you can create a dict from a list of tuple
    return dict(zip(L, range(0,len(L))))

def d_list_one(L):
    return dict([(v,i) for i,v in enumerate(L)])

d_list_one(['a','b','c']) # expected output: {'a':0, 'b':1, 'c':2}


I am working on the file "Advanced Functions Test.ipynb".
The first task is:
Problem 6
Use enumerate and other skills from above to return the count of the number of items in the list whose value equals its index.
Can you write Python code for it?

def count_match_index(L):
    print len([vals for vals in enumerate(L) if vals[0] == vals[1]])

count_match_index([0,2,2,1,5,5,6,10]) # should output 4


I am working on the file "Advanced Modules/Collections Module.ipynb".
The first task is:
defaultdict
The whole point is that it will always return a value even if you query for a key that doesnt exist. That value you  set ahead of time is called a factory object. That key also gets turned into a new key/value pair with the factory object
Can you write Python code for it?

from collections import defaultdict

d = {'k1':1}

d["k1"]

d["k2"] # this will get an error because the k2 key doesnt exist

d = defaultdict(object)

d['one'] # this doesn't exist, but in calling for it it will create a new element {'one' : object}

d['two'] # same, this will add {'two' : object}

for k, v in d.items():
    print "key:", k, "item:", v

e = defaultdict(lambda: 0) # lambda just returns 0 here

e['four']
e['twelve']

def error():
    return 'error'

f = defaultdict(error) #returned item must be callable or None

f['new']

f.items()


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Advanced exercise Jack's dilemma
(If time permits.)
email
Date: Wed, 16 Jul 2008 16:45:37 -0500
From: Jack Cook
To: <numpy-discussion@scipy.org>
Subject: Numpy Advanced Indexing Question
Greetings,
I have an I,J,K 3D volume of amplitude values at regularly sampled
time intervals. I have an I,J 2D slice which contains a time (K)
value at each I, J location. What I would like to do is extract a
subvolume at a constant +/- K window around the slice. Is there an
easy way to do this using advanced indexing or some other method?
Thanks in advanced for your help.
-- Jack
Can you write Python code for it?

# "data"

ni, nj, nk = (10, 15, 20)
amplitude = np.random.rand(ni, nj, nk)
horizon = np.random.randint(5, 15, size=(ni, nj))


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Advanced operations: axis-wise evaluation
Can you write Python code for it?

expr = np.load('expr.npy')

print_info(expr)


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Even more advanced: NumPy Array Interface
An author of a foreign package (included with the exercizes as
problems/mutable_str.py) provides a string class that
allocates its own memory:
ipython
In [1]: from mutable_str import MutableString
In [2]: s = MutableString('abcde')
In [3]: print s
abcde
You'd like to view these mutable (mutable means the ability to modify in place)
strings as ndarrays, in order to manipulate the underlying memory.
Add an array_interface dictionary attribute to s, then convert s to an
ndarray. Numerically add "2" to the array (use the in-place operator +=).
Then print the original string to ensure that its value was modified.

Hint: Documentation for NumPy's __array_interface__
  may be found in the online docs.

Here's a skeleton outline:
Can you write Python code for it?

import numpy as np
from mutable_str import MutableString

s = MutableString('abcde')

# --- EDIT THIS SECTION ---

# Create an array interface to this foreign object
s.__array_interface__ = {'data' : (XXX, False), # (ptr, is read_only?)
                         'shape' : XXX,
                         'typestr' : '|u1', # typecode unsigned character
                         }

# --- EDIT THIS SECTION ---

print 'String before converting to array:', s
sa = np.asarray(s)

print 'String after converting to array:', sa

sa += 2
print 'String after adding "2" to array:', s


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Example: skipping rows and columns with slicing
Can you write Python code for it?

print_info(a.T)

print_info(a.T[::2])

print_info(a.T[::2, ::2])


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Example: take the transpose of a
Can you write Python code for it?


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Exercise: 3D broadcasting
Below, produce the array containing the sum of every element in x with every element in y
Can you write Python code for it?

x = np.random.rand(3, 5)
y = np.random.randint(10, size=8)
z = x # FIX THIS


x = np.random.rand(3, 5)
y = np.random.randint(10, size=8)
z = x[..., newaxis] + y#[:,newaxis, newaxis]

print(x.shape)
print(y.shape)

# To use broadcasting efficiently, the dimensions need to either match or be one.
# This is the explicit way.
x[newaxis,:,:] + y[:,newaxis, newaxis]


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Exercise: broadcast expr_lib and gene_len together to produce RPKM
Can you write Python code for it?

rpkm = expr_lib  # FIX THIS

from matplotlib import pyplot as plt
from scipy import stats

def plot_col_density(data, xlim=None, *args, **kwargs):
    # Use gaussian smoothing to estimate the density
    density_per_col = [stats.kde.gaussian_kde(col) for col in data.T]
    if xlim is not None:
        m, M = xlim
    else:
        m, M = np.min(data), np.max(data)
    x = np.linspace(m, M, 100)

    plt.figure()
    for density in density_per_col:
        plt.plot(x, density(x), *args, **kwargs)
    plt.xlabel('log-counts')
    plt.ylabel('frequency')
    if xlim is not None:
        plt.xlim(xlim)
    plt.show()

%matplotlib inline

plt.style.use('ggplot')

plot_col_density(np.log(expr+1))

plot_col_density(np.log(rpkm + 1), xlim=(0, 250))


gene_len.shape
lib_size.shape

#rpkm[i,j] = 10**11 * expr[i,j] / (lib_size[j]*gene_size[i])
# option 1: Use np.newaxis
%timeit 10**3 * expr_lib / gene_len[:, np.newaxis]

# probably better
rpkm = 10**3 * expr_lib / gene_len[:, np.newaxis]

# have the expr_lib matrix dancing around untill it works
rpkm2 = (10**3* expr_lib.T / gene_len).T

rpkm1 - rpkm2

%timeit (10**11* (expr_lib / lib_size ).T / gene_len).T

from matplotlib import pyplot as plt
from scipy import stats

def plot_col_density(data, xlim=None, *args, **kwargs):
    # Use gaussian smoothing to estimate the density
    density_per_col = [stats.kde.gaussian_kde(col) for col in data.T]
    if xlim is not None:
        m, M = xlim
    else:
        m, M = np.min(data), np.max(data)
    x = np.linspace(m, M, 100)

    plt.figure()
    for density in density_per_col:
        plt.plot(x, density(x), *args, **kwargs)
    plt.xlabel('log-counts')
    plt.ylabel('frequency')
    if xlim is not None:
        plt.xlim(xlim)
    plt.show()

%matplotlib inline

plt.style.use('ggplot')

plot_col_density(np.log(expr+1))

plot_col_density(np.log(rpkm + 1), xlim=(0, 20))


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Exercise: np.lib.stride_tricks.as_strided
Use as_strided to produce a sliding-window view of a 1D array.
Can you write Python code for it?

def sliding_window(arr, size=2):
    """Produce an array of sliding window views of `arr`

    Parameters
    ----------
    arr : 1D array, shape (N,)
        The input array.
    size : int, optional
        The size of the sliding window.

    Returns
    -------
    arr_slide : 2D array, shape (N - size - 1, size)
        The sliding windows of size `size` of `arr`.

    Examples
    --------
    >>> a = np.array([0, 1, 2, 3])
    >>> sliding_window(a, 2)
    array([[0, 1],
           [1, 2],
           [2, 3]])
    """
    return arr  # fix this

# test your code here
sliding_window(np.arange(8), 3)


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Exercise: quantile normalization
Quantile Normalization(https://en.wikipedia.org/wiki/Quantile_normalization) is a method to align distributions. Implement it using NumPy axis-wise operations and fancy indexing.
Hint: look for documentation for scipy.mstats.rankdata, np.sort, and np.argsort.
Can you write Python code for it?

np.mgrid[0:4,0:3][1]

repeat(np.arange(0,3),4)

testlist = np.array([1.1,2.1, 2.1, 2.1, 3.1, 6.1, 4.1, 5.1])

indices = np.argsort(testlist)
unique_items,number_of_occurences = np.unique(testlist, return_counts=True)

indices, unique_items, number_of_occurences

index = 0
while index < len(testlist):
    if number_of_occurences[i] == 1:
        pass
    else:
        indices[index+1:index+number_of_occurences[i]] -= np.arange(number_of_occurences[i]-1)
        indices[index+number_of_occurences[i]:] -= number_of_occurences[i]-1
    index += 1
indices

index

np.unique(testdata)

for r in testdata.T:
    print (np.unique(r, return_counts=True))

np.argsort?

testdata = np.loadtxt('./testdata.dat', delimiter=',')
print(testdata)
# convert into an array with rank values according to column
rank_idx = np.argsort(np.argsort(testdata,axis=0), axis=0)

#np.argsort(rank_idx, axis=0)

# TODO SOLVE FOR 2 values of 4

# sort the input data
sorted_testdata = np.sort(testdata, axis=0)
# determine the ranks by computing the mean for each row
ranks = np.mean(sorted_testdata, axis=1)
ranks[rank_idx]
#print(rank_idx)
#testdata[rank_idx, np.mgrid[0:4,0:3][1]]

np.sort(testdata, axis=0)

def qnorm(x):
    """Quantile normalize an input matrix.

    Parameters
    ----------
    x : 2D array of float, shape (M, N)
        The input data, with each column being a
        distribution to normalize.

    Returns
    -------
    xn : 2D array of float, shape (M, N)
        The normalized data.
    """
    # convert into an array with rank values according to column
    rank_idx = np.argsort(np.argsort(x,axis=0), axis=0)
    # sort the input data
    sorted_x = np.sort(x, axis=0)
    # determine the ranks by computing the mean for each row
    ranks = np.mean(sorted_x, axis=1)
    xn = ranks[rank_idx]
    return xn

logexpr = np.log(expr + 1)
logrpkm = np.log(rpkm + 1)

logexprn = qnorm(logexpr)
logrpkmn = qnorm(logrpkm)

plot_col_density(logexprn)

plot_col_density(logrpkmn, xlim=(0, 0.25))


def qnorm(x):
    """Quantile normalize an input matrix.

    Parameters
    ----------
    x : 2D array of float, shape (M, N)
        The input data, with each column being a
        distribution to normalize.

    Returns
    -------
    xn : 2D array of float, shape (M, N)
        The normalized data.
    """
    xn = np.copy(x) # replace this by normalizing code
    return xn

logexpr = np.log(expr + 1)
logrpkm = np.log(rpkm + 1)

logexprn = qnorm(logexpr)
logrpkmn = qnorm(logrpkm)

plot_col_density(logexprn)

plot_col_density(logrpkmn, xlim=(0, 0.25))


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Fancy indexing
You can index arrays with slicing, but also with boolean arrays (including broadcasting!), integer arrays, and individual indices along multiple dimensions.
Can you write Python code for it?

values = np.array([0, 5, 99])
selector = np.random.randint(0, 3, size=(3, 4))
print(selector)
print(values[selector])


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Finally, longer genes are more likely to produce reads. So we normalize by the gene length (in kb) to produce a measure of expression called Reads Per Kilobase per Million reads (RPKM).
Can you write Python code for it?

gene_len = np.load('gene-lens.npy')
print(gene_len.shape)


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
Getting a copy
Can you write Python code for it?


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
However, NumPy will automatically prepend singleton dimensions until the array shapes match or there is an error:
Can you write Python code for it?

np.all(expr / lib_size ==
       expr / lib_size[np.newaxis, :])

expr_lib = expr / lib_size


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
We also multiply by $10^6$ in order to keep the numbers on a readable scale (reads per million reads).
Can you write Python code for it?


I am working on the file "Advanced NumPy Patterns.ipynb".
The first task is:
We are going to:

Obtain an RPKM expression matrix
Quantile normalize the data

using the awesome power of NumPy
Inside a numpy ndarray
Can you write Python code for it?

def print_info(a):
    print('number of elements:', a.size)
    print('number of dimensions:', a.ndim)
    print('shape:', a.shape)
    print('data type:', a.dtype)
    print('strides:', a.strides)
    print('flags:')
    print(a.flags)

print_info(a)

print(a.data)

abytes = a.ravel().view(dtype=np.uint8)

print_info(abytes)

print(abytes[:24])


I am working on the file "Advanced Python Objects - Test.ipynb".
The first task is:
Advanced Dictionaries
Problem 7: Create this dictionary:
{0: 0, 1: 1, 2: 8, 3: 27, 4: 64}
 using dictionary comprehension.
Can you write Python code for it?


I am working on the file "Advanced Python Objects - Test.ipynb".
The first task is:
Advanced Lists
Problem 8: Reverse the list below:
Can you write Python code for it?

l = [1,2,3,4]
l.reverse() # reverses in place, call the list again to check
l


I am working on the file "Advanced Python Objects - Test.ipynb".
The first task is:
Advanced Sets
Problem 5: Find the elements in set1 that are not in set2:
Can you write Python code for it?

set1 = {2,3,1,5,6,8}
set2 = {3,1,7,5,6,8}

print set1.difference(set2) # in set 1 but not set 2
print set2.difference(set1) # in set 2 but not set 1


I am working on the file "Advanced Python Objects - Test.ipynb".
The first task is:
Advanced Strings
Problem 3: Check if every letter in the string s is lower case
Can you write Python code for it?

s = 'hello how are you Mary, are you feeling okay?'
print 'Yup' if s.islower() else 'Nope'


I am working on the file "Advanced Python Objects - Test.ipynb".
The first task is:
Problem 2: Round 5.23222 to two decimal places
Can you write Python code for it?


I am working on the file "Advanced Python Objects - Test.ipynb".
The first task is:
Problem 4: How many times does the letter 'w' show up in the string below?
Can you write Python code for it?

s = 'twywywtwywbwhsjhwuwshshwuwwwjdjdid'
print s.count('w')


I am working on the file "Advanced Python Objects - Test.ipynb".
The first task is:
Problem 9: Sort the list below
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Balance Analysis.ipynb".
The first task is:
Exercise

Try a few different objectives.
Assess the effects of gene deletions.
Hint: a gene can be knocked out by running model.genes.get_by_id('<geneID>').knock_out() where <geneID> needs to be replace with an actual gene identifier.

Solutions
Can you write Python code for it?

gene_essentiality = {}
for gene in model.genes:
    mutant = model.copy()
    mutant.genes.get_by_id(gene.id).knock_out()
    try:
        print(gene, fba(mutant).objective_value)
        gene_essentiality[gene] = fba(mutant).objective_value
    except Infeasible:
        print(gene, 0)
        gene_essentiality[gene] = 0


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Exercises
Exercise 1
Explore how relaxing the constraint on the growth rate affects the solution space:
1. Modify the code to explore flux ranges for $\mu \gt 0.7 \ h^{-1}$
1. Plot the sum of flux ranges over a range of percentages.
Exercise 2
Using FVA, determine all blocked reactions ($v = 0$) in the model.
Solutions
Solution 1
Can you write Python code for it?

percentage = (0.7 / model.solve().f) * 100
percentage

result_80perc_max_obj = flux_variability_analysis(model, fraction_of_optimum=percentage/100, remove_cycles=True)

result_80perc_max_obj.plot(index=result_80perc_max_obj.data_frame.index, height=1200)


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
First we load a model from the BiGG database (and make a copy of it).
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Get an overview of a few key statistics of the resulting flux ranges.
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Inspect the result.
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Run flux variability analysis for optimally growing E. coli
(Optimal) Flux Balance Analysis solutions are not necessariliy unique. Flux Variablity Analysis is a good tool for estimating the space of alternative optimal solutions.
Can you write Python code for it?

fba(model)

model_optimal = model.copy()

model_optimal.reactions.BIOMASS_Ecoli_core_w_GAM.lower_bound = 0.8739215069684299

result_max_obj = flux_variability_analysis(model_optimal, remove_cycles=True)

result_max_obj.plot(index=result_max_obj.data_frame.index, height=1200)


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Solution 3
Can you write Python code for it?

result = flux_variability_analysis(model, remove_cycles=True)

result.data_frame[(result.data_frame.lower_bound == 0) & (result.data_frame.upper_bound == 0)]


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Those reactions showing up in red are futile cyles.
Can you write Python code for it?

result.data_frame[result.data_frame.upper_bound > 500]

result_no_cyles = flux_variability_analysis(model, remove_cycles=True)

abs_flux_ranges = abs(result_no_cyles.data_frame.lower_bound - result_no_cyles.data_frame.upper_bound).to_dict()
escher.Builder('e_coli_core.Core metabolism', reaction_data=abs_flux_ranges).display_in_notebook()


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Turns out that in this small core metabolic model, the optimal solution is actually unique!
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Visualize the flux ranges on a pathway map of E. coli's central carbon metabolism.
Can you write Python code for it?

abs_flux_ranges = abs(result.data_frame.lower_bound - result.data_frame.upper_bound).to_dict()
escher.Builder('e_coli_core.Core metabolism', reaction_data=abs_flux_ranges).display_in_notebook()


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Flux Variability Analysis.ipynb".
The first task is:
Visualize the flux ranges.
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Genome-scale metabolic reconstruction.ipynb".
The first task is:
Exercises
This is how the the stoichiometry matrix S looks like when visualized as a matrix plot.
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.spy(S, precision=0.01, markersize=.1)


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Genome-scale metabolic reconstruction.ipynb".
The first task is:
The stoichiometric matrix S
Can you write Python code for it?

S = model.to_array_based_model().S.todense()
S


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Genome-scale metabolic reconstruction.ipynb".
The first task is:
The stoichiometry matrix S has the following dimensions.
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Pathway visualization with Escher.ipynb".
The first task is:
Display a map E. coli central carbon metabolism.
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Pathway visualization with Escher.ipynb".
The first task is:
Visualize reaction-centric and/or metabolite-centric data.
Can you write Python code for it?

escher.Builder('e_coli_core.Core metabolism', reaction_data={'PGK': 100}, metabolite_data={'ATP': 20}).display_in_notebook()


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Production vs. Growth.ipynb".
The first task is:
Look at the result in a tabular view.
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Production vs. Growth.ipynb".
The first task is:
So let's quickly write a custom plot (you can use your mouse to rotate the graphic).
Can you write Python code for it?

%matplotlib notebook

import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure(figsize=(12, 8))
ax = fig.add_subplot(111, projection='3d')
ax.plot_trisurf(
    result_3D.data_frame.EX_ac_e.values,
    result_3D.data_frame.EX_o2_e.values,
    result_3D.data_frame.objective_upper_bound.values,
    cmap=cm.coolwarm, linewidth=0, antialiased=True)
ax.set_xlabel('Acetate [mmol gDW^-1 h^-1]')
ax.set_ylabel('O2 [mmol gDW^-1 h^-1]')
ax.set_zlabel('Biomass [h^-1]')
ax.mouse_init()
# interact(f(fig, ax), angle=(0, 360), elevation=(-100, 100))


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Production vs. Growth.ipynb".
The first task is:
Unfortunately, 3D plots are not yet supported out of the box.
Can you write Python code for it?


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Vanillin Production.ipynb".
The first task is:
And now we can plug the pathway to the model.
Can you write Python code for it?

vanillin_pathway.plug_model(model)

from cameo import phenotypic_phase_plane


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Vanillin Production.ipynb".
The first task is:
Constraints can be set in the model according to data found in the literature. The defined conditions allow the simulation of phenotypes very close to the experimental results.
<img src=http://www.biomedcentral.com/content/figures/1752-0509-7-36-2.jpg/>
Model validation by comparing in silico prediction of the specific growth rate with experimental data. Growth phenotypes were collected from literature and compared to simulated values for chemostat cultivations at four different conditions, nitrogen limited aerobic (green) and anaerobic (red), carbon limited aerobic (blue) and anaerobic (white).
Österlund, T., Nookaew, I., Bordel, S., & Nielsen, J. (2013). Mapping condition-dependent regulation of metabolism in yeast through genome-scale modeling. BMC Systems Biology, 7, 36. doi:10.1186/1752-0509-7-36
Can you write Python code for it?

model.reactions.EX_glc__D_e.lower_bound = -13 #glucose exchange
model.reactions.EX_o2_e.lower_bound = -3  #oxygen exchange

model.medium

model.objective = model.reactions.BIOMASS_SC5_notrace #growth
model.optimize().f


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Vanillin Production.ipynb".
The first task is:
Heterologous pathway
Vanillin is not produced by S. cervisiae. In their work an heterolgous pathway is inserted to allow generate a vanillin production strain. The pathway is described as:
<img src=http://static-content.springer.com/image/art%3A10.1186%2F1475-2859-9-84/MediaObjects/12934_2010_Article_474_Fig1_HTML.jpg>
Schematic representation of the de novo VG biosynthetic pathway in S. Cerevisisae (as designed by Hansen et al [5]). Metabolites are shown in black, enzymes are shown in black and in italic, cofactors and additional precursors are shown in red. Reactions catalyzed by heterologously introduced enzymes are shown in red. Reactions converting glucose to aromatic amino acids are represented by dashed black arrows. Metabolite secretion is represented by solid black arrows where relative thickness corresponds to relative extracellular accumulation. 3-DSH stands for 3-dedhydroshikimate, PAC stands for protocathechuic acid, PAL stands for protocatechuic aldehyde, SAM stands for S-adenosylmethionine. 3DSD stands for 3-dedhydroshikimate dehydratase, ACAR stands for aryl carboxylic acid reductase, PPTase stands for phosphopantetheine transferase, hsOMT stands for O-methyltransferase, and UGT stands for UDP-glycosyltransferase. Adapted from Hansen et al. [5].
Brochado et al. Microbial Cell Factories 2010 9:84   doi:10.1186/1475-2859-9-84
Using cameo, is very easy to generate a pathway and add it to a model.
Can you write Python code for it?

from cameo.strain_design.pathway_prediction import PathwayPredictor

predictor = PathwayPredictor(model)

pathways = predictor.run('vanillin', max_predictions=3)

vanillin_pathway = pathways.pathways[0]

from cameo.core.pathway import Pathway

vanillin_pathway = Pathway.from_file("data/vanillin_pathway.tsv")
vanillin_pathway.data_frame


I am working on the file "Advanced-SynBio-for-Cell-Factories-Course/Vanillin Production.ipynb".
The first task is:
To find gene knockout targets, we use cameo.strain_design.heuristic package which implements the OptGene strategy.
The authors used the biomass-product coupled yield (bpcy) for optimization which is the equivalent of running OptGene in non-robust mode. All simulations were computed using MOMA but because cameo does not implement MOMA we use it's equivalent linear version (it minimizes the absolute distance instead of the quadratic distance). The linear MOMA version is faster than the original MOMA formulation.
By default, our OptGene implementation will run 20'000 evaluations.
Can you write Python code for it?

from cameo.strain_design.heuristic.evolutionary_based import OptGene
from cameo.flux_analysis.simulation import lmoma

optgene = OptGene(model)

results = optgene.run(target="EX_vnl_b_glu_c",
                      biomass="BIOMASS_SC5_notrace",
                      substrate="EX_glc__D_e",
                      simulation_method=lmoma)

results


I am working on the file "AdvancedSlice.ipynb".
The first task is:
Open a chest located on a remote globus endpoint and load a remote json configuration file.
Can you write Python code for it?

c = Chest(path      = "{:s}-results".format(config.name),
          open      = partial(glopen,      endpoint=config.arch_end),
          open_many = partial(glopen_many, endpoint=config.arch_end))
sc = CachedSlict(c)
with glopen(
            "{:s}.json".format(config.name), mode='r',
            endpoint = config.arch_end,
            ) as f:
    params = json.load(f)


I am working on the file "AdvancedSlice.ipynb".
The first task is:
Plot the bubble height, the 'H' keys, vs. time.
Use a spline to compute the derivative.
Can you write Python code for it?

fig = plt.figure()

nx = frame['t_yz'].shape[0]
ny = frame['t_yz'].shape[1]

plt.imshow(frame['fz_yz'][:,int(ny*config.lower):int(ny*config.upper)].transpose(),
           vmin = -params['atwood']*params['g']*2,
           vmax =  params['atwood']*params['g']*2,
           origin='lower')
plt.colorbar();
plt.contour(frame['fz_yz'][:,int(ny*config.lower):int(ny*config.upper)].transpose(),
            colors='k',
            levels=[-params['atwood']*params['g'], params['atwood']*params['g']])
#axs[1].imshow(frame['fz_yz'].transpose(), origin='lower')

def laplace(grid):
    from numpy.fft import fftn, ifftn, fftfreq
    kx = fftfreq(grid.shape[0])
    ky = fftfreq(grid.shape[1])
    kgrid = fftn(grid)
    for i in range(kgrid.shape[0]):
        for j in range(kgrid.shape[1]):
            if kx[i] != 0 or ky[j] != 0:
                kgrid[i,j] = kgrid[i,j] / (kx[i]**2 + ky[j]**2)
    kgrid[0,0] = 0.
    rgrid = np.array(ifftn(kgrid), dtype=float)
    return rgrid

fig = plt.figure()

nx = frame['t_yz'].shape[0]
ny = frame['t_yz'].shape[1]
full = np.concatenate((frame['vorticity_yz'], -np.flipud(frame['vorticity_yz'])), axis=0)
plt.imshow(full[:,int(ny*config.lower):int(ny*config.upper)].transpose(),
           origin='lower')
#axs[1].imshow(frame['fz_yz'].transpose(), origin='lower')
plt.colorbar();

full = np.concatenate((frame['vorticity_yz'], -np.flipud(frame['vorticity_yz'])), axis=0)
streamfunction = laplace(full[:,:]);

fig = plt.figure()

nx = frame['t_yz'].shape[0]
ny = frame['t_yz'].shape[1]
print(np.max(streamfunction), np.min(streamfunction))
full = np.concatenate((frame['t_yz'], np.flipud(frame['t_yz'])), axis=0)

#background = np.tile(frame['t_yz'][:,int(ny*config.lower):int(ny*config.upper)].transpose(), (1,4))
background = np.tile(streamfunction[:,int(ny*config.lower):int(ny*config.upper)].transpose(), (1,4))
foreground = np.tile(streamfunction[:,int(ny*config.lower):int(ny*config.upper)].transpose(), (1,4))

plt.imshow(background,
           origin='lower',...


I am working on the file "Advection.ipynb".
The first task is:
For more complicated hyperbolic problems, we may have multiple sets of characteristics, they may not be parallel, and the solution may not be constant along them.  But it will still be the case that information is propagated along characteristics.  The idea that information propagates at finite speed is an essential property of hyperbolic PDEs.
The Riemann problem
The Riemann problem consists of a hyperbolic PDE, such as (\ref{adv:advection}), together with piecewise constant initial data consisting of two states.  For convenience, we place the interface (or jump) at $x=0$ and refer to the left state (for $x<0$) as $q_\ell$ and the right state (for $x>0$) as $q_r$.  We thus have
\begin{align}
    q_0(x) & = \begin{cases} q_\ell & x<0 \ q_r & x>0. \end{cases}
\end{align}
For the advection equation, the solution to the Riemann problem is immediately obvious; it is simply a special case of (\ref{adv:solution}).  The discontinuity initially at $x=0$ moves at speed $a$.  We have
\begin{align}
    q(x,t) & = \begin{cases} q_\ell & x-at<0 \ q_r & x-at>0. \end{cases}
\end{align}
Can you write Python code for it?


I am working on the file "Advection.ipynb".
The first task is:
Notice how the initial discontinuity follows the characteristic coming from $x=0$ at $t=0$.
In the live notebook, you can adjust the advection speed $a$ and see how this changes the solution.
Here's another way to look at the Riemann solution that will be very useful for more complicated equations.
Can you write Python code for it?

q_l = 1.
q_r = 0.
advection.plot_riemann_solution(q_l, q_r, a=1.);


I am working on the file "Advection.ipynb".
The first task is:
We can think of the initial values $q_0(x)$ being transmitted along these lines; we sometimes say that information is transmitted along characteristics.
The next figure shows how an initial hump propagates to the right along characteristics.
Can you write Python code for it?


I am working on the file "Aki_Rich_sol/double_coupled.ipynb".
The first task is:
3. COMPUTE AKI & RICHARDS SOLUTION
Can you write Python code for it?

#%% Initialization of setup
# -----------------------------------------------------------------------------
x = 4000   # x receiver coordinate
y = 4000   # y receiver coodinate
z = 4000   # z receiver coodinate

rho = 2500                # Density kg/m^3
beta = 3000               # S-wave velocity
alpha = sqrt(3)*beta      # p-wave velocity

stf = 'gauss'             # Set the desired source time function 'heaviside' , 'gauss'
Trise = 0.25              # Rise time used in the source time function
Mo = 4*10E16              # Scalar Moment

r, th, phi = cart2sph(x, y, z)     # spherical receiver coordinates

tmin = r/alpha - 2*Trise           # Minimum observation time
tmax = r/beta + Trise + 2*Trise    # Maximum observation time

# SOURCE TIME FUNCTION
# -----------------------------------------------------------------------------
if stf == 'heaviside':
    M0 = lambda t: 0.5*Mo*0.5*(sign(t) + 1)
if stf == 'gauss':
    M0 = lambda t: Mo*(1 + erf(t/Trise))

#******************************************************************************
# COMPUTE AKI & RICHARDS SOLUTION
#******************************************************************************
# Scalar factors int the AKI & RICHARDS solution
# -----------------------------------------------------------------------------
CN  = (1/(4 * pi * rho))
CIP = (1/(4 * pi * rho * alpha**2))
CIS = (1/(4 * pi * rho * beta**2))
CFP = (1/(4 * pi * rho * alpha**3))
CFS = (1/(4 * pi * rho * beta**3))

# Radiation patterns: near(AN), intermedia(AIP,AIS), and far(AFP,AFS) fields
# -----------------------------------------------------------------------------
def AN(th, phi):
    AN = [[9*sin(2*th)*cos(phi), -6*cos(2*th)*cos(phi), 6*cos(th)*sin(phi)]]
    return asarray(AN)

def AIP(th, phi):
    AIP = [[4*sin(2*th)*cos(phi), -2*cos(2*th)*cos(phi), 2*cos(th)*sin(phi)]]
    return asarray(AIP)

def AIS(th, phi):
    AIS = [-3*sin(2*th)*cos(phi), 3*cos(2*th)*cos(phi), -3*cos(th)*sin(phi)]
    return asarray(AIS)

def AFP(th, phi):
   ...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Calculate the total and average time of all recordings
Can you write Python code for it?

total_recording_time = timedelta(0)
for recording in recordings:
    total_recording_time += recording_duration[recording]

total_recording_time

mean_recording_time = total_recording_time / len(recordings)
mean_recording_time


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Check which are the longest alarms
Can you write Python code for it?

# How many alarm events are longer than 1 hour?

over_1_hour =  sorted([al for al in all_durations if al > 3600])
len(over_1_hour)

# Which alarms were longer than one hour?

alarms_over_1_hour = []
for recording in recordings:
    for alarm in alarm_list[recording]:
        for event in alarm_events[recording][alarm]['duration_seconds']:
            if event > 3600:
                alarms_over_1_hour.append((recording, alarm, event))

alarms_over_1_hour = DataFrame(sorted(alarms_over_1_hour, key = lambda x: x[2], reverse = True),
                               columns = ['recording', 'alarm', 'duration (seconds)'])

alarms_over_1_hour

alarms_over_1_hour.groupby('alarm').count()


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Check which are the most frequent alarms
Can you write Python code for it?

# Identify the most frequent alarm events

frequent_alarms = alarm_stats_cum_al.loc['count'].sort_values(inplace = False, ascending = False)

# The eight most frequent alarms

frequent_alarms[:8]

# How many percent of all alarms were these 8 frequent alarms?

round(frequent_alarms[:8].sum() / frequent_alarms.sum(), 3) * 100

# Write frequent alarm in an Excel file

writer = pd.ExcelWriter('%s/%s' % (DIR_WRITE, 'frequent_alarms.xlsx'))
DataFrame(frequent_alarms[:8]).to_excel(writer, 'frequent_alarms')
writer.save()

# Number of alarms where the user sets the limits

user_set_alarms = (frequent_alarms['Minute volume < low limit'] + frequent_alarms['Minute volume > high limit'] +
                frequent_alarms['Respiratory rate > high limit'])

int(user_set_alarms)

# What proportion of all alarms were these 3 user-set alarms?

print('%.3f' % (user_set_alarms / frequent_alarms.sum()))

# Frequent alarms related to VT not achieved

other_frequent_alarms = (frequent_alarms['Tidal volume < low Limit'] + frequent_alarms['Volume not constant'] +
                frequent_alarms['Tube obstructed'])

int(other_frequent_alarms)

# What proportion of all alarms were alarms related to VT not achieved?

print('%.3f' % (other_frequent_alarms / frequent_alarms.sum()))


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Create the tables and figures of the paper
Table 1
Can you write Python code for it?

clinical_details_for_paper = clinical_details[['Gestation', 'Birth weight', 'Current weight', 'Main diagnoses']]
clinical_details_for_paper = clinical_details_for_paper.loc[recordings]
# clinical_details_for_paper

vent_modes_all = {}
for recording in recordings:
    vent_modes_all[recording] = vent_modes_selected[recording].Text.unique()
    vent_modes_all[recording] = [mode[5:] for mode in vent_modes_all[recording] if mode.startswith(' Mode')]

vent_modes_all = DataFrame([vent_modes_all]).T
vent_modes_all.columns = ['Ventilation modes']
vent_modes_all = vent_modes_all.loc[recordings]
# vent_modes_all

recording_duration_hours_all = DataFrame([recording_duration_hours]).T
recording_duration_hours_all.columns = ['Recording duration (hours)']

Table_1 = recording_duration_hours_all.join([clinical_details_for_paper, vent_modes_all])

Table_1

writer = pd.ExcelWriter('%s/%s' % (DIR_WRITE, 'Table_1.xlsx'))
Table_1.to_excel(writer)
writer.save()


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Example plots
Can you write Python code for it?


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Figure 1
Can you write Python code for it?

rec = 'DG032_2'
filetype = 'jpg'
dpi = 300

alarm_state = alarm_states[rec]
numbered = Series(np.zeros(len(alarm_state)), index = alarm_state.index)
for i in range(1, len(alarm_state)):
    if alarm_state.iloc[i]['State New'] == 'Active':
        numbered[i] = alarm_list[rec].index(alarm_state.iloc[i]['Id']) + 1

fig = plt.figure()
fig.set_size_inches(10, 4)
fig.subplots_adjust(left=0.4, bottom=None, right=None, top=None, wspace=None, hspace=None)

ax1 = fig.add_subplot(1, 1, 1);
ax1.plot(alarm_state.index,  numbered, '|', color = 'red', markersize = 14, markeredgewidth = 0.5 )
plt.xlabel("Time", fontsize  = 14)
plt.title(rec)
plt.yticks([i+1  for i, _ in enumerate(alarm_list[rec])], alarm_list[rec], fontsize = 14);
plt.xticks(fontsize = 8)

plt.ylim(0.5, len(alarm_list[rec]) + 0.5)

fig.savefig('%s/%s.jpg' % (DIR_WRITE, 'Figure_1a'), dpi=dpi, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format= filetype,
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

rec = 'DG032_2'
filetype = 'jpg'
dpi = 300

fig = plt.figure()
fig.set_size_inches(8, 4)
fig.subplots_adjust(left=0.5, bottom=None, right=None, top=None, wspace=None, hspace= None)
ax1 = fig.add_subplot(1, 1, 1)
xs = [i + 0.1 for i, _ in enumerate(alarm_list[rec])]
stats = []
for alarm in alarm_list[rec]:
    stats.append(alarm_stats[rec][alarm]['percentage of recording length (%)'])
stats_all = pd.concat(stats)
plt.barh(xs, stats_all, color = 'red')
plt.xlabel("% of total recording time", fontsize  = 14)
plt.title(rec)
plt.yticks([i + 0.5 for i, _ in enumerate(alarm_list[rec])], alarm_list[rec], fontsize = 14)
plt.xticks(fontsize = 14);

fig.savefig('%s/%s.jpg' % (DIR_WRITE, 'Figure_1b'), dpi=dpi, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format= filetype,
        transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

rec = 'DG032_2'
filetype = 'tiff'
dpi = 300

alarm_state =...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Figure 2
Can you write Python code for it?

rec = 'DG003'
filetype = 'jpg'
dpi = 300

ymax = slow_measurements[rec]['MV_high_weight'].max() + 0.3

fig = plt.figure()
fig.set_size_inches(8, 6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None,
    wspace=None, hspace=None)
ax1 = fig.add_subplot(1, 1, 1);
slow_measurements[rec]['MV_kg'].plot(ax = ax1, color = 'blue', ylim = [0, ymax] );
slow_measurements[rec]['MV_low_weight'].plot(ax = ax1, color = 'green', linewidth = 3, ylim = [0, ymax] );
slow_measurements[rec]['MV_high_weight'].plot(ax = ax1, color = 'red', linewidth = 3,  ylim = [0, ymax] );
ax1.set_title(rec, size = 14, color = 'black')
ax1.set_xlabel('Time', size = 14, color = 'black')
ax1.set_ylabel('L/min/kg', size = 14, color = 'black')
ax1.tick_params(which = 'both', labelsize=12)
ax1.grid('on', linestyle='-', linewidth=0.5, color = 'gray')
ax1.legend(['MV_kg', 'alarm_low', 'alarm_high']);

fig.savefig('%s/%s.jpg' % (DIR_WRITE, 'Figure_2a_color'), dpi=dpi, facecolor='w', edgecolor='w',
    orientation='portrait', papertype=None, format= filetype,
    transparent=False, bbox_inches=None, pad_inches=0.1, frameon=True)

rec = 'DG003'
filetype = 'jpg'
dpi = 300

ymax = slow_measurements[rec]['MV_high_weight'].max() + 0.3

fig = plt.figure()
fig.set_size_inches(8, 6)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None,
    wspace=None, hspace=None)
ax1 = fig.add_subplot(1, 1, 1);

slow_measurements[rec]['MV_kg'].plot(ax = ax1, color = 'black', alpha = 0.6, ylim = [0, ymax] );
slow_measurements[rec]['MV_low_weight'].plot(ax = ax1, color = 'black', linewidth = 3, ylim = [0, ymax] );
slow_measurements[rec]['MV_high_weight'].plot(ax = ax1, color = 'black', linewidth = 3,  ylim = [0, ymax] );

ax1.set_title(rec, size = 14, color = 'black')
ax1.set_xlabel('Time', size = 14, color = 'black')
ax1.set_ylabel('L/min/kg', size = 14, color = 'black')
ax1.tick_params(which = 'both', labelsize=12)
ax1.grid('on', linestyle='-', linewidth=0.5, color = 'gray')
ax1.legend(['MV_kg', 'alarm_low',...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
For each alarm what was the mean, median, sd, mad, min, 25pc, 75pc, max of alarm durations
Can you write Python code for it?

# libraries containing various descriptive statistics for each recording

mean_alarm_duration_alarms = {}
median_alarm_duration_alarms = {}
sd_alarm_duration_alarms = {}
mad_alarm_duration_alarms = {}
min_alarm_duration_alarms = {}
pc25_alarm_duration_alarms = {}
pc75_alarm_duration_alarms = {}
max_alarm_duration_alarms = {}

for alarm in total_alarm_list:
    mean_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].mean(), 4)
    median_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].median(), 4)
    sd_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].std(), 4)
    mad_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].mad(), 4)
    min_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].min(), 4)
    pc25_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].quantile(0.25), 4)
    pc75_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].quantile(0.75), 4)
    max_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].max(), 4)

# Create DataFrame containing cumulative alarm statistics for each alarm

alarm_stats_cum_al = DataFrame([total_alarm_number_alarms,
                                 total_alarm_number_alarms_24H,
                                 cum_alarm_duration_alarms,
                                 cum_alarm_duration_alarms_24H,
                                 mean_alarm_duration_alarms,
                                 median_alarm_duration_alarms,
                                 sd_alarm_duration_alarms,
                                 mad_alarm_duration_alarms,
                                 min_alarm_duration_alarms,
                                 pc25_alarm_duration_alarms,
                                 pc75_alarm_duration_alarms,
                                 max_alarm_duration_alarms],
    index = ['count', 'count per 24h', 'total alarm duration (sec)', 'total alarm duration per 24 hours (sec)',
             'mean duration (sec)', 'median duration...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
For each alarm, what was number of alarm events across all recordings and the number of events normalized per 24 hour recording time
Can you write Python code for it?

total_alarm_number_alarms = {} # dictionary containing the number of alarm events in all recordings for the
                               # various alarm categories
for alarm in total_alarm_list:
    total = 0
    for recording in recordings:
        if alarm in alarm_list[recording]:
            total += len(alarm_events[recording][alarm].index)
    total_alarm_number_alarms[alarm] = total

# Write alarm list to Excel file

writer = pd.ExcelWriter('%s/%s' % (DIR_WRITE, 'total_alarm_list_numbers.xlsx'))
DataFrame([total_alarm_number_alarms]).T.to_excel(writer, 'total_alarm_list')
writer.save()

total_alarm_number_alarms_24H = {} # dictionary containing the number of alarm events in all recordings for the
                                   # various alarm categories normalized for 24 hour recording periods
for alarm in total_alarm_list:
    total_alarm_number_alarms_24H[alarm] = round(((total_alarm_number_alarms[alarm] /
                                                    (total_recording_time.total_seconds() / 86400))), 4)


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
For each alarm, what was the total duration of alarm events across all recordings and normalized per 24 hour recording time
Can you write Python code for it?

alarm_durations_alarms = {} # a dictionary of Series. Each Series contains all durations of a particular alarm
                            # in all recordings

for alarm in total_alarm_list:
    durations = []
    for recording in recordings:
        if alarm in alarm_list[recording]:
            durations.append(alarm_events[recording][alarm]['duration_seconds'])
    durations = pd.concat(durations)
    alarm_durations_alarms[alarm] = durations

cum_alarm_duration_alarms = {} # dictionary containing the total duration of alarms in all recordings for the
                               # various alarm categories

for alarm in total_alarm_list:
    cum_alarm_duration_alarms[alarm] = round(alarm_durations_alarms[alarm].sum(), 4)

cum_alarm_duration_alarms_24H = {} # dictionary containing the total duration of alarms in all recordings for the
                                   # various alarm categories normalized for 24 hour recording periods
for alarm in total_alarm_list:
    cum_alarm_duration_alarms_24H[alarm] = round(((cum_alarm_duration_alarms[alarm] /
                                                    (total_recording_time.total_seconds() / 86400))), 4)


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Generate alarm events from alarm states
Can you write Python code for it?

# Define function to retrieve alarm events from alarm timing data
def alarm_events_calculator(dframe, al):

    '''
    DataFrame, str -> DataFrame

    dframe: DataFrame containing alarm states
    al:  alarm category (string)

    Returns a pd.DataFrame object with the time stamps when the alarm went off and the duration (in seconds)
    of the alarm for alarm 'al' in recording 'rec'
    '''

    alarms = dframe
    alarm = alarms[alarms.Name == al]
    length = len(alarm)
    delta = np.array([(alarm.Date_Time[i] - alarm.Date_Time[i-1]).total_seconds()
            for i in range(1, length) if alarm['State New'][i] == 'NotActive' and alarm['State New'][i-1] == 'Active'])
    stamp = np.array([alarm.index[i-1]
            for i in range(1, length) if alarm['State New'][i] == 'NotActive' and alarm['State New'][i-1] == 'Active'])
    data = {'duration_seconds': delta, 'time_went_off': stamp,}
    alarm_t = DataFrame(data, columns = ['time_went_off', 'duration_seconds'])
    return alarm_t


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Generate cumulative statistics of each alarm in all recordings combined
Can you write Python code for it?

# Create a list of all alarms occurring in any recording

total_alarm_list = set()
for recording in recordings:
    total_alarm_list.update(alarm_list[recording])
total_alarm_list = sorted(total_alarm_list)

# A list of all alarms occurring during the service evaluation

total_alarm_list

# Write alarm list to Excel file

writer = pd.ExcelWriter('%s/%s' % (DIR_WRITE, 'total_alarm_list.xlsx'))
DataFrame(total_alarm_list, columns = ['alarm categories']).to_excel(writer, 'total_alarm_list')
writer.save()


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
How many alarm events are longer than 10 minutes but shorter than 1 hour?¶
Can you write Python code for it?

over_10_minutes =  sorted([al for al in all_durations if al > 600 and al <= 3600])
len(over_10_minutes)

alarms_over_10_min = []
# which alarms were longer than 10 minutes but shorter than 1 hour
for recording in recordings:
    for alarm in alarm_list[recording]:
        for event in alarm_events[recording][alarm]['duration_seconds']:
            if event > 600 and event <= 3600:
                alarms_over_10_min.append((recording, alarm, event))

alarms_over_10_min = DataFrame(sorted(alarms_over_10_min, key = lambda x: x[2], reverse = True),
                               columns = ['recording', 'alarm', 'duration (seconds)'])

alarms_over_10_min.groupby('alarm').count()


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Import alarm settings
Can you write Python code for it?

alarm_settings = {}

for recording in recordings:
    flist = os.listdir('%s/%s' % (DIR_READ, recording))
    flist = [file for file in flist if not file.startswith('.')] # There are some hidden
    # files on the hard drive starting with '.'; this step is necessary to ignore them
    files = alarm_setting_finder(flist)
    # print('Loading recording %s' % recording)
    # print(files)
    fnames = ['%s/%s/%s' % (DIR_READ, recording, filename) for filename in files]
    alarm_settings[recording] =  data_loader(fnames)

# Remove etCO2 limits which were not used

alarm_settings_selected = {}

for recording in recordings:
    alarm_settings_selected[recording] = alarm_setting_cleaner(alarm_settings[recording])

# Create a another dictionary of Dataframes with some of the alarm settings

lsts = [(['MV_high_weight'], ['MVe_HL']), (['MV_low_weight'], ['MVe_LL']),
        (['PIP_high'], ['PIP_HL']), (['RR_high'], ['RR_HL'])]
alarm_settings_2 = {}

for recording in recordings:
    frmes = []
    for name, pars in lsts:
        if pars in [['MVe_HL'], ['MVe_LL']]:
            ind = []
            val = []
            for index, row in alarm_settings_selected[recording].iterrows():
                if row['Id'] in pars:
                    ind.append(index)
                    val.append(row['Value New'] / current_weights[recording])
                frmes.append(DataFrame(val, index = ind, columns = name))
        else:
            ind = []
            val = []
            for index, row in alarm_settings_selected[recording].iterrows():
                if row['Id'] in pars:
                    ind.append(index)
                    val.append(row['Value New'])
                frmes.append(DataFrame(val, index = ind, columns = name))
    alarm_settings_2[recording] = pd.concat(frmes)
    alarm_settings_2[recording].drop_duplicates(inplace = True)

# Write DataFrames containing alarm settings to a...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Import alarm states
Can you write Python code for it?

alarm_states = {}

for recording in recordings:
    flist = os.listdir('%s/%s' % (DIR_READ, recording))
    flist = [file for file in flist if not file.startswith('.')] # There are some hidden
    # files on the hard drive starting with '.'; this step is necessary to ignore them
    files = alarm_state_finder(flist)
    # print('Loading recording %s' % recording)
    # print(files)
    fnames = ['%s/%s/%s' % (DIR_READ, recording, filename) for filename in files]
    alarm_states[recording] =  data_loader(fnames)

# Write DataFrames containing alarm states to a multisheet Excel file

writer = pd.ExcelWriter('%s/%s' % (DIR_WRITE, 'alarm_states.xlsx'))
for recording in recordings:
    alarm_states[recording].to_excel(writer,'%s' % recording)
writer.save()


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Import clinical details
Can you write Python code for it?

clinical_details = pd.read_excel('%s/data_grabber_patient_data_combined.xlsx' % CWD)
clinical_details.index = clinical_details['Recording']

clinical_details.info()

current_weights = {}
for recording in recordings:
    current_weights[recording] = clinical_details.loc[recording, 'Current weight' ] / 1000


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Import modules containing own functions
Can you write Python code for it?

from gb_loader import *
from gb_stats import *
from gb_transform import *


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Import ventilator modes and settings
Import ventilation settings
Can you write Python code for it?

vent_settings = {}

for recording in recordings:
    flist = os.listdir('%s/%s' % (DIR_READ, recording))
    flist = [file for file in flist if not file.startswith('.')] # There are some hidden
    # files on the hard drive starting with '.'; this step is necessary to ignore them
    files = slow_setting_finder(flist)
    # print('Loading recording %s' % recording)
    # print(files)
    fnames = ['%s/%s/%s' % (DIR_READ, recording, filename) for filename in files]
    vent_settings[recording] =  data_loader(fnames)

# remove less important ventilator settings to simplify the table

vent_settings_selected = {}
for recording in recordings:
    vent_settings_selected[recording] = vent_settings_cleaner(vent_settings[recording])

# Create a another dictionary of Dataframes wit some of the ventilation settings (set VT, set RR, set Pmax)

lsts = [(['VT_weight'], ['VTi', 'VThf']), (['RR_set'], ['RR']), (['Pmax'], ['Pmax', 'Ampl hf max'])]
vent_settings_2 = {}

for recording in recordings:
    frmes = []
    for name, pars in lsts:
        if pars in [['VTi', 'VThf']]:
            ind = []
            val = []
            for index, row in vent_settings_selected[recording].iterrows():
                if row['Id'] in pars:
                    ind.append(index)
                    val.append(row['Value New'] / current_weights[recording])
                frmes.append(DataFrame(val, index = ind, columns = name))
        else:
            ind = []
            val = []
            for index, row in vent_settings_selected[recording].iterrows():
                if row['Id'] in pars:
                    ind.append(index)
                    val.append(row['Value New'])
                frmes.append(DataFrame(val, index = ind, columns = name))
    vent_settings_2[recording] = pd.concat(frmes)
    vent_settings_2[recording].drop_duplicates(inplace = True)


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
In each recording, what was the mean, median, sd, mad, min, 25pc, 75pc, max of alarm durations
Can you write Python code for it?

alarm_durations_recordings = {} # a dictionary of Series. Each series contains all the alarm durations of a recording
for recording in recordings:
    durations = []
    for alarm in alarm_list[recording]:
        durations.append(alarm_events[recording][alarm]['duration_seconds'])
    durations = pd.concat(durations)
    alarm_durations_recordings[recording] = durations

# Dictionaries containing various descriptive statistics for each recording

mean_alarm_duration_recordings = {}
median_alarm_duration_recordings = {}
sd_alarm_duration_recordings = {}
mad_alarm_duration_recordings = {}
min_alarm_duration_recordings = {}
pc25_alarm_duration_recordings = {}
pc75_alarm_duration_recordings = {}
max_alarm_duration_recordings = {}

for recording in recordings:
    mean_alarm_duration_recordings[recording] = round(alarm_durations_recordings[recording].mean(), 4)
    median_alarm_duration_recordings[recording] = round(alarm_durations_recordings[recording].median(), 4)
    sd_alarm_duration_recordings[recording] = round(alarm_durations_recordings[recording].std(), 4)
    mad_alarm_duration_recordings[recording] = round(alarm_durations_recordings[recording].mad(), 4)
    min_alarm_duration_recordings[recording] = round(alarm_durations_recordings[recording].min(), 4)
    pc25_alarm_duration_recordings[recording] = round(alarm_durations_recordings[recording].quantile(0.25), 4)
    pc75_alarm_duration_recordings[recording] = round(alarm_durations_recordings[recording].quantile(0.75), 4)
    max_alarm_duration_recordings[recording] = round(alarm_durations_recordings[recording].max(), 4)

# Create DataFrame containing cumulative alarm statistics for each recording

alarm_stats_cum_rec = DataFrame([total_alarm_number_recordings,
                                 total_alarm_number_recordings_24H,
                                 mean_alarm_duration_recordings,
                                 median_alarm_duration_recordings,
                                 sd_alarm_duration_recordings,
                                ...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Investigate the relationship of MV and RR parameter readings, ventilation settings and alarm settings
Can you write Python code for it?

for recording in recordings:
    slow_measurements[recording] = pd.concat([slow_measurements[recording],
            vent_settings_2[recording], alarm_settings_2[recording]], axis = 0, join = 'outer')
    slow_measurements[recording].sort_index(inplace = True)

for recording in recordings:
    slow_measurements[recording] = slow_measurements[recording].fillna(method = 'pad')

def minute_volume_plotter(rec, ylim = False):
    '''
    Plots the total minute volumme (using the data obtained with 1/sec sampling rate)
    together with the "MV low" and "MV high" alarm limits
    Displays the plot

    '''
    if ylim:
        ymax =  ylim
    else:
        ymax = slow_measurements[rec]['MV_high_weight'].max() + 0.3

    fig = plt.figure()
    fig.set_size_inches(12, 8)
    fig.subplots_adjust(left=None, bottom=None, right=None, top=None,
    wspace=None, hspace=0.7)
    ax1 = fig.add_subplot(1, 1, 1);
    slow_measurements[rec]['MV_kg'].plot(ax = ax1, color = 'blue', ylim = [0, ymax] );
    slow_measurements[rec]['MV_low_weight'].plot(ax = ax1, color = 'green', linewidth = 3, ylim = [0, ymax] );
    slow_measurements[rec]['MV_high_weight'].plot(ax = ax1, color = 'red', linewidth = 3, ylim = [0, ymax] );
    ax1.set_title('Minute volume - %s' % rec, size = 22, color = 'black')
    ax1.set_xlabel('Time', size = 22, color = 'black')
    ax1.set_ylabel('L/kg/min', size = 22, color = 'black')
    ax1.grid('on', linestyle='-', linewidth=0.5, color = 'gray')
    ax1.legend(['MV_kg', 'alarm_low', 'alarm_high']);

minute_volume_plotter('DG003')

def minute_volume_plotter_2(rec, ylim = False, version = ''):
    '''
    Plots the total minute volumme (using the data obtained with 1/sec sampling rate)
    together with the "MV low" and "MV high" alarm limits
    Writes the plot to file (does not display the plot)

    '''
    if ylim:
        ymax =  ylim
    else:
        ymax = slow_measurements[rec]['alarm_MV_high_weight'].max() + 0.3

    fig = plt.figure()
    fig.set_size_inches(12, 8)
   ...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
List of the  recordings
Can you write Python code for it?


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Resampling to remove half-empty rows
Can you write Python code for it?

# 1/sec data are retrieved in two parts which need to be joined
# This resampling steps combines the two parts

for recording in recordings:
    slow_measurements[recording] = slow_measurements[recording].resample('1S').mean()

# Example
slow_measurements['DG003'].head();


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Using the files containing the alarm states, for each alarm category in each recording create a DataFrame with the timestamps the alarm went off and the duration of the alarm and store them in a dictionary of dictionaries
Can you write Python code for it?

# Create a list of alarms occurring during each recording

alarm_list = {}

for recording in recordings:
    alarm_list[recording] = sorted(set(alarm_states[recording].Name))

alarm_events = {}

for recording in recordings:
    alarm_events[recording] = {}
    for alarm in alarm_list[recording]:
        alarm_events[recording][alarm] = alarm_events_calculator(alarm_states[recording], alarm)

# Write Dataframes containing the alarm events in Excel files,
# one Excel file for each recording

for recording in recordings:
    writer = pd.ExcelWriter('%s/%s%s' % (DIR_WRITE, recording, '_alarm_events.xlsx'))
    for alarm in alarm_list[recording]:
        alarm_events[recording][alarm].to_excel(writer, alarm[:20])

    writer.save()


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Visualise alarm statistics for the individual alarms in the individual recording
Can you write Python code for it?

# Generates a plot with the cumulative times (in seconds) of the various alarm occurring during recording (rec).
# Displays the plot

def alarm_plot_1(rec):
    fig = plt.figure()
    fig.set_size_inches(25, 8)
    fig.subplots_adjust(left=0.4, bottom=None, right=None, top=None,
    wspace=None, hspace=0.7)
    ax1 = fig.add_subplot(1, 1, 1)
    xs = [i + 0.1 for i, _ in enumerate(alarm_list[rec])]
    stats = []
    for alarm in alarm_list[rec]:
        stats.append(alarm_stats[rec][alarm]['cumulative duration (s)'])
    stats_all = pd.concat(stats)
    plt.barh(xs, stats_all, color = 'red')
    plt.xlabel("seconds", fontsize  = 24)
    plt.title("Recording %s : How long was the alarm active over the %d seconds of recording?" % (rec,
                recording_duration_seconds[rec]), fontsize = 22)
    plt.yticks([i + 0.5 for i, _ in enumerate(alarm_list[rec])], alarm_list[rec], fontsize = 22)
    plt.xticks(fontsize = 20)

# Generates a plot with the cumulative times (in seconds) of the various alarm occurring during recording (rec).
# Does not displays the plot but write it into a jpg file.
# NB: the resolution of the image is only 100 dpi - for publication quality higher is needed

def alarm_plot_1_write(rec):
    fig = plt.figure()
    fig.set_size_inches(25, 8)
    fig.subplots_adjust(left=0.4, bottom=None, right=None, top=None,
    wspace=None, hspace=0.7)
    ax1 = fig.add_subplot(1, 1, 1)
    xs = [i + 0.1 for i, _ in enumerate(alarm_list[rec])]
    stats = []
    for alarm in alarm_list[rec]:
        stats.append(alarm_stats[rec][alarm]['cumulative duration (s)'])
    stats_all = pd.concat(stats)
    plt.barh(xs, stats_all, color = 'red')
    plt.xlabel("seconds", fontsize  = 24)
    plt.title("Recording %s : How long was the alarm active over the %d seconds of recording?" % (rec,
                recording_duration_seconds[rec]), fontsize = 22)
    plt.yticks([i + 0.5 for i, _ in enumerate(alarm_list[rec])], alarm_list[rec], fontsize = 22)
    plt.xticks(fontsize = 20)

   ...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Visualise the duration of all alarm events as histogram
Can you write Python code for it?

# Histogram showing the number of alarms which were shorter than 1 minute

fig = plt.figure()
fig.set_size_inches(12, 6)
fig.subplots_adjust(left=0.4, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
ax1 = fig.add_subplot(1, 1, 1)
n, bins, patches = plt.hist(all_durations, bins = range(0, 60))
plt.grid(True)
plt.xlabel('Alarm duration (seconds)', fontsize = 20)
plt.ylabel('Number of events', fontsize = 20)
plt.xticks(range(0,60,2), fontsize = 10)
plt.yticks(fontsize = 10)
plt.title('Histogram of alarm durations', fontsize = 20)

fig.savefig('%s/%s' % (DIR_WRITE, 'alarm_duration_hist_1.jpg'), dpi=300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format='jpg',
        transparent=False, bbox_inches=None, pad_inches=0.1,
        frameon=True)

# Histogram showing the number of alarms which were shorter than 10 minutes

fig = plt.figure()
fig.set_size_inches(12, 6)
fig.subplots_adjust(left=0.4, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
ax1 = fig.add_subplot(1, 1, 1)
n, bins, patches = plt.hist(all_durations, bins = range(0, 600))
plt.grid(True)
plt.xlabel('Alarm duration (seconds)', fontsize = 20)
plt.ylabel('Number of events', fontsize = 20)
plt.xticks(range(0, 600, 60), fontsize = 10)
plt.yticks(fontsize = 10)
plt.title('Histogram of alarm durations', fontsize = 20)

fig.savefig('%s/%s' % (DIR_WRITE, 'alarm_duration_hist_2.jpg'), dpi=300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format='jpg',
        transparent=False, bbox_inches=None, pad_inches=0.1,
        frameon=True)

# Histogram showing all data with a bin size of 1minutes and log X axis

fig = plt.figure()
fig.set_size_inches(12, 6)
fig.subplots_adjust(left=0.4, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
ax1 = fig.add_subplot(1, 1, 1)
n, bins, patches = plt.hist(all_durations, bins = range(0, 50000, 60))
plt.grid(True)
plt.xlabel('Alarm duration (seconds)', fontsize = 20)
plt.ylabel('Number of events', fontsize = 20)
plt.xticks(range(0,...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Visualising cumulative statistics of alarms
Can you write Python code for it?

# Reduce a too long alarm name
total_alarm_list[0] = 'A setting, alarm limit or vent...'

# Total number of alarm events in all recordings

fig = plt.figure()
fig.set_size_inches(17, 12)
fig.subplots_adjust(left=0.4, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
ax1 = fig.add_subplot(1, 1, 1);
ax1.barh(list(range(1, len(total_alarm_list)+1)), alarm_stats_cum_al.loc['count', :], color = 'blue')
plt.ylabel("Alarms", fontsize  = 22)
plt.xlabel("", fontsize  = 22)
plt.title("Number of alarm events" , fontsize = 26)
ax1.tick_params(which = 'both', labelsize=14)
ax1.tick_params(which = 'both', labelsize=14)
plt.yticks([i+1.5  for i, _ in enumerate(total_alarm_list)], total_alarm_list, rotation = 'horizontal')
plt.grid()

fig.savefig('%s/%s' % (DIR_WRITE, 'number_events_al.jpg'), dpi=300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format='jpg',
        transparent=False, bbox_inches=None, pad_inches=0.1,
        frameon=True)

# Total number of alarm events in all recordings normalized for 24 hour periods

fig = plt.figure()
fig.set_size_inches(17, 12)
fig.subplots_adjust(left=0.4, bottom=None, right=None, top=None, wspace=None, hspace=0.7)
ax1 = fig.add_subplot(1, 1, 1);
ax1.barh(list(range(1, len(total_alarm_list)+1)), alarm_stats_cum_al.loc['count per 24h', :], color = 'blue')
plt.ylabel("Alarms", fontsize  = 22)
plt.xlabel("", fontsize  = 22)
plt.title("Number of alarm events per 24 hour" , fontsize = 26)
ax1.tick_params(which = 'both', labelsize=14)
plt.yticks([i+1.5  for i, _ in enumerate(total_alarm_list)], total_alarm_list, rotation = 'horizontal')
plt.grid()

fig.savefig('%s/%s' % (DIR_WRITE, 'number_events_24H_al.jpg'), dpi=300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format='jpg',
        transparent=False, bbox_inches=None, pad_inches=0.1,
        frameon=True)

# Median duration of alarm events in all recordings

fig = plt.figure()
fig.set_size_inches(17, 12)
fig.subplots_adjust(left=0.4, bottom=None, right=None,...


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Visualising recording durations
Can you write Python code for it?

v = list(range(1, len(recordings)+1))
w = [value for key, value in sorted(recording_duration_hours.items()) if key in recordings]

fig = plt.figure()
fig.set_size_inches(20, 10)
fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=None)
ax1 = fig.add_subplot(1, 1, 1);
ax1.bar(v, w, color = 'blue')
plt.xlabel("Recordings", fontsize  = 22)
plt.ylabel("Hours", fontsize  = 22)
plt.title("Recording periods" , fontsize = 22)
plt.yticks(fontsize = 22)
plt.xticks([i+1.5  for i, _ in enumerate(recordings)], recordings, fontsize = 22, rotation = 'vertical');
plt.grid()

fig.savefig('%s/%s' % (DIR_WRITE, 'recording_durations.jpg'), dpi=300, facecolor='w', edgecolor='w',
        orientation='portrait', papertype=None, format='jpg',
        transparent=False, bbox_inches=None, pad_inches=0.1,
        frameon=True)


I am working on the file "Alarms_GitHub.ipynb".
The first task is:
Write recording times out into files
Can you write Python code for it?

recording_times_frame = DataFrame([recording_duration, recording_duration_hours, recording_duration_seconds],
                            index = ['days', 'hours', 'seconds'])

recording_times_frame

writer = pd.ExcelWriter('%s/%s' % (DIR_WRITE, 'recording_periods.xlsx'))
recording_times_frame.to_excel(writer,'rec_periods')
writer.save()


I am working on the file "Alex/.ipynb_checkpoints/Cancer-checkpoint.ipynb".
The first task is:
Se realiza un grafico de distribuciones de los 5 genes que fueron los que mas reads presentaron
Can you write Python code for it?

for i in range(len(gen_list)):
    name=gen_list[i]
    gen1=df[df['Hybridization REF']== name].ix[:,'TCGA-05-4244-01A-01R-1107-07':]
    sns.distplot(gen1)

genes=df.loc[df.ix[:,1] > 100000 ]

genes

g = sns.factorplot(x='Hybridization REF', data=genes, kind="count",
                   palette="BuPu", size=6, aspect=1.5)
g.set_xticklabels(step=2)
genes

maximo = maximo.set_index('Hybridization REF')

maximo1 = maximo.transpose()
maximo1.head()

sns.violinplot(maximo1[['ACTB|60']])

import matplotlib.pyplot as plt
maximo1=maximo.transpose()
gen_list=maximo1.ix[1:].as_matrix()
plt.boxplot(gen_list,1)
gen_list

print(maximo.mean(1))
print (maximo.std(1))

gen2=df[df['Hybridization REF']=='SFTPB|6439']
gen2=gen2.drop('Hybridization REF',1)
gen2=gen2.values.tolist()
plt.boxplot(gen2)


I am working on the file "Alex/.ipynb_checkpoints/Cancer-checkpoint.ipynb".
The first task is:
Se seleccionan los genes que posean mas de cien mil reads
Can you write Python code for it?

maximo=df.loc[df.ix[:,1] > 100000 ]
gen_list=maximo['Hybridization REF'].values.tolist()
gen_list
#maximo

sns.boxplot(x="Hybridization REF", y= 'TCGA-05-4389-01A-01R-1206-07', data=maximo, palette="PRGn") #mirar cuantos pacientes presentan un gen mas expresado que otro


I am working on the file "Alex/.ipynb_checkpoints/Cancer-checkpoint.ipynb".
The first task is:
Trabajo de programación con Carcinoma de Pulmón
Se importan las librerias necesarias para trabajar
Can you write Python code for it?

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
sns.set()
%matplotlib inline