I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
If you connected to the SAMPLE database, you will have the EMPLOYEE and DEPARTMENT tables available to you. However, if you are connecting to a different database, you will need to execute the next command to populate the tables for you. Note, if you run this command and the two tables already exist, the tables will not be replaced. So don't worry if you execute this command by mistake.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
If you want an unlimited number of rows returned, set maxrows to -1.
Can you write Python code for it?

%odata set maxrows -1
%odata select * from employee


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Now that we have a working DB2 connection, we will need to set up an OData service to talk to DB2.
Back to Top

Connecting through OData
Connecting through OData requires a different approach than a DB2 client. We still need to ask a bunch of questions on how we connect to the database, but this doesn't create a connection from the client. Instead what we end up creating is a service URL. This URL gives us access to DB2 through the OData gateway server.
The OData Server take the URL request and maps it to a DB2 resource, which could be one or more tables. The RESTful API needs this URL to communicate with DB2 but nothing else (userids, passwords, etc...) are sent with the request.
The following %odata command will prompt you for the connection parameters, similar to what happened with the DB2 connect. There are a few differences however. The connection requires the userid and password of the user connecting to the database, and the userid and password of a user with administration (DBABM) privileges.
The administrative user creates the service connection that will be used to communicate through the OData gateway and DB2. The regular userid and password is for the actual user that will connect to the database to manipulate the tables. Finally we need to have the schema (or owner) of the tables that will be accessed. From a DB2 perspective, this is similar to connecting to a DATABASE (SAMPLE) as userid FRED. The EMPLOYEE table was created under the userid DB2INST1, so to access the table we need to use DB2INST1.EMPLOYEE. If we didn't include the schema (DB2INST1), the query would fail since FRED was not the owner of the table.
The %odata PROMPT command will request all of the connection parameters and explain what the various fields are. Note: If you have DBADM privileges (and you created the sample tables yourself), you can leave the USERID/PASSWORD/SCHEMA values blank and they will default to the administrative user values.
Back to Top

Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Now that we have it working, we can try exactly the same thing but with OData. Our first transaction will update the search key to SERVICE.
Can you write Python code for it?

%odata UPDATE PATTERN SET SEARCH = '%SERVICE%' WHERE PATTERN_NUMBER = 1


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Now we create a view that access this PATTERN table to do the actual search. Note that values that are inserted into the PATTERN table must have the SQL special characters like % to make sure patterns can be anywhere in the string.
Can you write Python code for it?

%odata RESET TABLE EMPDEPT
%odata RESET TABLE PATTERN

%%sql
CREATE OR REPLACE VIEW EMPDEPT AS
  (
  SELECT
    COUNT(*) AS COUNT
  FROM
    EMPLOYEE E, DEPARTMENT D
  WHERE
    E.WORKDEPT = D.DEPTNO
    AND D.DEPTNAME LIKE
        (
        SELECT SEARCH FROM PATTERN WHERE PATTERN_NUMBER=1
        )
  );


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Now you can try the SQL statement again.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
OData (and DB2) will return an error message about our missing SALARY column which requires a value.
Can you write Python code for it?

%odata -e INSERT INTO TESTODATA(EMPNO, LASTNAME, BONUS) VALUES (2,'Wilma',50000)


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
One of the recommendations would be not to use the COUNT(*) function to determine the amount of rows that will be retrieved, especially if you expect there to a large of number rows. To minimize the data returned, you can use the form COUNT(column) which will modify the OData request to return the count and ONLY that column in the result set. This is a compromise in terms of the amount of data returned. This example using the -r (raw) flag which results in all of the JSON headers and data to be displayed. The JSON flag (-j) will not display any records.
Can you write Python code for it?

s = %odata -e -r SELECT COUNT(EMPNO) FROM EMPLOYEE LIMIT 5


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
One of the unusual behaviors of the COUNT(*) function is that will actually return the entire answer set under the covers. The %odata command strips the count out from the results and doesn't display the rows returned. That is probably not would you expect from this syntax! The COUNT function is better described as the count of physical rows returned. Here is the same example with 5 rows returned and the JSON records.
Can you write Python code for it?

s = %odata -e -r SELECT COUNT(*) FROM EMPLOYEE LIMIT 5


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Requesting data from DB2 using the standard %sql (ibm_db) interface is relatively straight-forward. We just need to place the SQL in the command and execute it to get the results.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Retrying the SELECT statement will result in 43 rows with no columns returned!
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
The COUNT(*) function is available as part of a SELECT list and it cannot include any other column names. If you do include other column names they will be ignored.
Can you write Python code for it?

s = %odata -e SELECT COUNT(*) FROM EMPLOYEE LIMIT 1


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
The datatypes are not the same as what one expect from a relational database. You get generic information on the character columns (String), and the numbers (Int16, Decimal). The Decimal specification actually contains the number of digits and decimal places but that isn't returned when using the table display.
| Data Type  |   Contents
|:-----------|:---------------
| Binary     |   Binary data
| Boolean    |   Binary-valued logic
| Byte       |   Unsigned  8-bit integer
| Date       |   Date without a time-zone offset
| Decimal    |   Numeric values with fixed precision and scale
| Double     |   IEEE 754 binary64 floating-point number (15-17 decimal digits)
| Duration   |   Signed duration in days, hours, minutes, and (sub)seconds
| Guid       |   16-byte (128-bit) unique identifier
| Int16      |   Signed 16-bit integer
| Int32      |   Signed 32-bit integer
| Int64      |   Signed 64-bit integer
| SByte      |   Signed 8-bit integer
| Single     |   IEEE 754 binary32 floating-point number (6-9 decimal digits)
| String     |   Sequence of UTF-8 characters
| TimeOfDay  |   Clock time 00:00-23:59:59.999999999999
Back to Top

WHERE Clause
The WHERE clause is used to filter out the rows that you want to retrieve from the table. The WHERE clause allows the following operators:

>, =>, <, <=, =, !=, <>, LIKE
AND, OR
Parenthesis to override order () of operators

The WHERE clause does not allow for mathematical operators at this time (*, -, +, /) or the unary NOT or "-" operators.
The LIKE clause can contain the special % character, but the equivalent OData syntax always searches the entire string and does not anchor at the beginning of the string. What this means is that the LIKE clause will turn into a search of the entire string whether you use the % character in your search string or not.
Example: Single comparison
The following select statement will search for employees who have a salary less than 40000.
Can you write Python code for it?

s = %odata -e SELECT EMPNO, WORKDEPT, SALARY FROM EMPLOYEE WHERE SALARY < 40000


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
The initial INSERT will populate the table with valid data. The echo option will show the json document that is sent via the POST command to OData to insert the row.
Can you write Python code for it?

%odata -e INSERT INTO TESTODATA(EMPNO, LASTNAME, SALARY, BONUS) VALUES (1,'Fred',10000,1000)


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
The next OData statement should select the count of employees working in service departments.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
The results will show the URL service command used (http:// followed by details of the host location and service ID) and the OData command. In this case the command should be /EMPLOYEES. This may seem like a spelling mistake, but the OData service creates a mapping from the database table (EMPLOYEE) to a service request. To give the service request a unique name, the letter "S" is appended to the table name. Do not confuse the service
name with the table name. That can sometimes lead to coding errors!
If we tried to request a table that didn't exist in the database, we would get an error message instead.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
There are two departments with the name SYSTEMS in them, but there is no easy way to create a view for every possible combination of searches that you may want. Instead what we do is create a table that contains the pattern we want to look for and create the view so that it references this table.
The first step is to create our PATTERN table. Note we make sure it has a primary key so our OData update calls can change it!
Can you write Python code for it?

%%sql -q
DROP TABLE PATTERN;
CREATE TABLE PATTERN
  (
  PATTERN_NUMBER INT NOT NULL PRIMARY KEY,
  SEARCH VARCHAR(16)
  );


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
This actually can cause some issues if you try to reuse the connection information that was created with the UNKNOWN_TBL. Since the service could not determine the structure of the table, the service will not return any column information with a select statement. The next SQL statement will create the UNKNOWN_TBL.
Can you write Python code for it?

%sql CREATE TABLE UNKNOWN_TBL AS (SELECT * FROM EMPLOYEE) WITH DATA


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
To correct this situation, you need to DROP the connect that the %odata program is using and reissue the SELECT statement.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Trying the DELETE this time will work.
Can you write Python code for it?

%odata -e DELETE FROM TESTODATA WHERE EMPNO=1


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
We can try this on the DB2 side as well to get the details of the error.
Can you write Python code for it?

%sql INSERT INTO TESTODATA(EMPNO, LASTNAME, BONUS) VALUES (2,'Wilma',50000)


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
We do a describe on the table and this will force another service URL to be generated for us.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
We doublecheck the results to make sure we got it right!
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
When a URL is generated, we need to append the \$format=json tag at the end to tell the OData service and the browser how to handle the results. When we run OData and RESTful calls from a programming language (like Python), we are able to send information in the header which tells the API how to handle the results and parameters. All of the RESTful calls to the OData gateway use the following header information:

  {
  "Content-Type":"application/json",
  "Accept":"application/json"
  }

When we send the URL to the OData gateway, it needs to be told how to return the information. We need to append the $format=json flag at the end of our query when sending the request via a browser. Note that the ampersand must be appended to the end of the existing URL since we already have one parameter in it.
Back to Top

JSON Display in Firefox
Depending on what version of Firefox you have, you may not get the JSON to be displayed very nicely. To use the built-in JSON formatter, issue the following commands in a separate browser window:

about:config
Search for devtools.jsonview.enabled

Right click on the jsonview setting and enable it. This will result in the JSON being easier to view.
Back to Top

SQL Command Syntax
The %odata command has been designed to translate the SQL syntax for INSERT, DELETE, UPDATE, and SELECT into an equivalent OData format. There are very specific ways of requesting data from OData, so this ends up placing some limitations on what SQL you can use. This section will cover the four major SQL commands and how they can be used with OData. If you need the syntax for a particular SQL command, just enter the command name by itself on the %odata line and it will give you a brief summary of the syntax. Here is the DELETE help.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
You can also refer to these values by using the settings['name'] variable. So the DROP statement just took the current DATABASE and SCHEMA settings and deleted the definition for the EMPLOYEE table. You could have done this directly with:

RESET DATABASE SAMPLE SCHEMA DB2INST1 TABLE EMPLOYEE

The list of settings and their variable names are listed below.
| Setting       | Variable name
|:--------------|:---------------------
| DATABASE      | odata_settings['database']
| SCHEMA        | odata_settings['schema']
| ADMIN         | odata_settings['admin']
| A_PWD         | odata_settings['a_pwd']
| USER          | odata_settings['userid']
| U_PWD         | odata_settings['u_pwd']
| HOST          | odata_settings['host']
| PORT          | odata_settings['port']
| MAXROWS       | odata_settings['maxrows']
Back to Top

Variables in %OData Statements
To use local Jupyter/Python variables in a notebook, all you need to do is place braces {} around the name of the variable. Before we illustrate this, we need to create another connection (since we just dropped it in the last example). Fortunately, none of the settings have been removed, so we still have the connection information (DATABASE, SCHEMA, ...) available.
In the event you have closed the notebook and started up from scratch, there is no need to do a full connect command (or prompt). The settings are automatically written to disk and then restored when you start up another session. If you want to connect to another database then you will need to use the following SET statement.
Can you write Python code for it?

%odata set DATABASE {odata_settings['database']} SCHEMA {odata_settings['schema']}


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
You can use this URL to directly access the results through a browser, or any application that can read the results returned by the OData gateway. The print statement below will display the URL as an active link. Click on that to see the results in another browser window.
Can you write Python code for it?


I am working on the file "Notebooks/DCGAN.ipynb".
The first task is:
2) Sample single number from the generator
Can you write Python code for it?

num = 3
Z1 = np.random.uniform(size=[1, 100], low=-1., high=1.).astype(np.float32)
L1 = OneHot(np.repeat([[num]],1,axis=1))
Gz_sample = demo_sess.run(Gz, feed_dict={inZ: Z1, inL: L1, is_training:False})
img = vis_square(Gz_sample, [1, 1],save_path=work_dir + 'test.jpg')
Image(filename=work_dir + 'test.jpg')


I am working on the file "Notebooks/DCGAN.ipynb".
The first task is:
3) Class Sweep
With the same z slowly move the class labels to smoothly generate different numbers
Can you write Python code for it?

class_sweep_dir = work_dir+'class_sweep/'
if not os.path.exists(class_sweep_dir): os.makedirs(class_sweep_dir)

batch_size = 100

Z_test = np.random.uniform(size=[batch_size, 100], low=-1., high=1.).astype(np.float32)
labels_test = OneHot(np.repeat([[0,0,0,0,0,0,0,0,0,0]],batch_size//10,axis=0))

images = []
for num in range(9):
    for p in np.linspace(1.,0.,30):
        count=len(images)
        fname = class_sweep_dir+ '%.2d_%d_%.2f.jpg'%(count,num,p)

        for i in range(batch_size):
            labels_test[i,num] = p; labels_test[i,num+1] = 1-p

        Gz_sample = demo_sess.run(Gz, feed_dict={inZ: Z_test, inL: labels_test, is_training:False})
        img = vis_square(Gz_sample, [batch_size//10, 10],save_path= fname)
        images.append(imageio.imread(fname))
        try: os.remove(fname)
        except: pass

imageio.mimsave(class_sweep_dir+'class_sweep.gif', images)
#display(Image(url=class_sweep_dir+'class_sweep.gif'))
Image(url='../common/images/cgan_class_sweep.gif')


I am working on the file "Notebooks/DCGAN.ipynb".
The first task is:
3) Z-space Sweep
Vary a coefficient alpha that determines how much of two different Z values are used to sample from the generator.
This is also called interpolation in z space in the original paper.
Can you write Python code for it?

zspace_sweep_dir = work_dir+'zspace_sweep/'
if not os.path.exists(zspace_sweep_dir): os.makedirs(zspace_sweep_dir)
batch_size = 100

Z1 = np.random.uniform(size=[batch_size, 100], low=-1., high=1.).astype(np.float32)
Z2 = np.random.uniform(size=[batch_size, 100], low=-1., high=1.).astype(np.float32)

images = []
for alpha in np.linspace(1.,0.,50):
    count=len(images)
    fname = zspace_sweep_dir+'%.2d.jpg'%(count)

    Z_test = alpha * Z1 + (1-alpha)*Z2
    labels_test = OneHot(np.repeat([[0,1,2,3,4,5,6,7,8,9]],batch_size//10,axis=0))

    Gz_sample = demo_sess.run(Gz, feed_dict={inZ: Z_test, inL: labels_test, is_training:False})
    img = vis_square(Gz_sample, [batch_size//10, 10],save_path=fname)
    images.append(imageio.imread(fname))
    try: os.remove(fname)
    except: pass

imageio.mimsave(zspace_sweep_dir+'zspace_sweep.gif', images)
#display(Image(url=zspace_sweep_dir+'zspace_sweep.gif'))
Image(url='../common/images/cgan_zspace_sweep.gif')


I am working on the file "Notebooks/DCGAN.ipynb".
The first task is:
Adversarial Training
You can either get the trained models from my google drive or train your own models using the DCGAN.py script.
I will later look into what if we make our noise gaussian (and make some samples more frquent)?
Can you write Python code for it?

best_iter = 4280#visually selected
best_img = work_dir + 'Iter_%d.jpg' %best_iter
best_ganMNIST_model = work_dir + "%.3d_model.ckpt" % best_iter
Image(filename=best_img)


I am working on the file "Notebooks/DCGAN.ipynb".
The first task is:
Experiments
Create demo networks and restore weights
Can you write Python code for it?

tf.reset_default_graph()
demo_sess = tf.InteractiveSession()

is_training = tf.placeholder(tf.bool, [], 'is_training')

inZ = tf.placeholder(tf.float32, [None, 100])
inL = tf.placeholder(tf.float32, [None, 10])
inX = tf.placeholder(tf.float32, [None, 28, 28, 1])

Gz = create_gan_G(inZ, inL, is_training, Cout=1, trainable=True, reuse=False, networktype=networktype + '_G')

DGz = create_gan_D(Gz, inL, is_training, trainable=True, reuse=False, networktype=networktype + '_D')

tf.global_variables_initializer().run()

ganG_var_list = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope=networktype + '_G')
saver = tf.train.Saver(var_list=ganG_var_list)
saver.restore(demo_sess, best_ganMNIST_model)


I am working on the file "Notebooks/DCGAN.ipynb".
The first task is:
Network definitions
Can you write Python code for it?

# define networks
from tools_networks import deconv, conv, dense, clipped_crossentropy, dropout

def concat_labels(X, labels):
    if X.get_shape().ndims == 4:
        X_shape = tf.shape(X)
        labels_reshaped = tf.reshape(labels, [-1, 1, 1, 10])
        a = tf.ones([X_shape[0], X_shape[1], X_shape[2], 10])
        X = tf.concat([X, labels_reshaped * a], axis=3)
    return X

def create_gan_G(z, labels, is_training, Cout=1, trainable=True, reuse=False, networktype='ganG'):
    '''input : batchsize * 100 and labels to make the generator conditional
        output: batchsize * 28 * 28 * 1'''
    with tf.variable_scope(networktype, reuse=reuse):
        z = tf.concat(axis=-1, values=[z, labels])
        Gz = dense(z, is_training, Cout=4 * 4 * 256, act='reLu', norm='batchnorm', name='dense2')
        Gz = tf.reshape(Gz, shape=[-1, 4, 4, 256])  # 4
        Gz = deconv(Gz, is_training, kernel_w=5, stride=2, Cout=256, trainable=trainable, act='reLu', norm='batchnorm', name='deconv1')  # 11
        Gz = deconv(Gz, is_training, kernel_w=5, stride=2, Cout=128, trainable=trainable, act='reLu', norm='batchnorm', name='deconv2')  # 25
        Gz = deconv(Gz, is_training, kernel_w=4, stride=1, Cout=1, act=None, norm=None, name='deconv3')  # 28
        Gz = tf.nn.sigmoid(Gz)
    return Gz

def create_gan_D(xz, labels, is_training, trainable=True, reuse=False, networktype='ganD'):
    with tf.variable_scope(networktype, reuse=reuse):
        xz = concat_labels(xz, labels)
        Dxz = conv(xz, is_training, kernel_w=5, stride=2, Cout=128, trainable=trainable, act='lrelu', norm=None, name='conv1')  # 12
        Dxz = conv(Dxz, is_training, kernel_w=5, stride=2, Cout=256, trainable=trainable, act='lrelu', norm='batchnorm', name='conv2')  # 4
        Dxz = conv(Dxz, is_training, kernel_w=2, stride=2, Cout=256, trainable=trainable, act='lrelu', norm='batchnorm', name='conv3')  # 2
        Dxz = conv(Dxz, is_training, kernel_w=2, stride=2, Cout=1, trainable=trainable, act='lrelu', norm='batchnorm', name='conv4')  # 2
   ...


I am working on the file "Notebooks/DX_screen.ipynb".
The first task is:
Plot disributions of fraction upregulated statistic
We are going to want to reuse this plot so here I'm wrapping it in a function.
Can you write Python code for it?

def fig_1e(ax):
    draw_dist(dx_meth.frac, ax=ax, lw=2.5)
    draw_dist(dx_rna.frac, ax=ax, lw=2.5, bins=200)
    draw_dist(dx_mir.frac, ax=ax, lw=2.5, bins=100)

    ax.set_yticks([])
    ax.set_xticks([0,.5,1])
    ax.set_ylabel('Density')
    ax.set_xlabel('Fraction')
    ax.legend(('Methylation','mRNA','miRNA'), frameon=False)
    prettify_ax(ax)
    return ax

#Do not import
fig, ax = subplots(1,1, figsize=(5,3))
fig_1e(ax);


I am working on the file "Notebooks/DX_screen.ipynb".
The first task is:
Read in matched Gene and miRNA expression data.
Can you write Python code for it?

matched_rna = matched_tn
matched_mir = pd.read_hdf(MIRNA_STORE, 'matched_tn')


I am working on the file "Notebooks/DX_screen.ipynb".
The first task is:
Run a simple screen for DX genes

Here we take the matched data and run a basic screen
fc = 1 means that we have no foldchange buffer for a gene to be considered over or underexpressed in a patient
If there are ties or missing data, I omit these from the test.  This can cause underpowered tests which have extreme test statistics but weak p-values.  For this reason I filter all gene/probes/markers with a sample size of less than 300 patients.
Can you write Python code for it?

dx_rna = binomial_test_screen(matched_rna, fc=1.)
dx_rna = dx_rna[dx_rna.num_dx > 300]

dx_rna.sort('p').head(10)

dx_rna.ix['FOXM1']


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Calculando uma previsão
Can you write Python code for it?

from infodenguepredict.models import sarimax,GAS,GASX
import statsmodels.api as sm

fig, axes = plt.subplots(1, 2, figsize=(15, 4))

fig = sm.graphics.tsa.plot_acf(Full.ix[1:, 'casos'], lags=52, ax=axes[0])
fig = sm.graphics.tsa.plot_pacf(Full.ix[1:, 'casos'], lags=52, ax=axes[1])

# Short.casos = Short.casos.apply(pd.np.log)
model_1 = sarimax.build_model(Full, 'casos', [])

fit_1 = model_1.fit()

fit_1.summary()

def plot_pred(fit):
    plt.Figure(figsize=(10,7))
    predict = fit.get_prediction(start='2017-01-01', dynamic=False)
    predict_ci = predict.conf_int()
    Full.casos.plot(style='o',label='obs')
    predict.predicted_mean.plot(style='r--', label='In sample')
    plt.fill_between(predict_ci.index, predict_ci.ix[:, 0], predict_ci.ix[:, 1], color='r', alpha=0.1)
    forecast = fit.get_prediction(start='2017-03-05', end='2017-06-21', dynamic=False)
    forecast_ci = forecast.conf_int()
    forecast.predicted_mean.plot(style='b--', label='Out of Sample')
    plt.fill_between(forecast_ci.index, forecast_ci.ix[:, 0], forecast_ci.ix[:, 1], color='b', alpha=0.1)
    plt.legend(loc=0)
plot_pred(fit_1)

model_2 = GAS.build_model(Full, ar=2, sc=6, target='casos')
fit_2 = model_2.fit()

fit_2.summary()

model_2.plot_fit()
plt.savefig('GAS_in_sample.png')
Full.casos.plot(style='ko')
model_2.plot_predict(h=10, past_values=52)

model_2.plot_z(figsize=(15,5))


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Converting dataframe from long format to wide format
The dataframe currently have all cities stacked on top of each other. In order to use this data in a predictive model, we need this table in wide format, that is, have only time along the rows and have cities variable listed as columns.
Can you write Python code for it?

riopiv = rio.pivot(index=rio.index, columns='municipio_geocodigo')

riopiv.head()

riopiv['SE'].head()


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Let's keep only the columns we want to use
Can you write Python code for it?

for col in ['casos_est_min', 'casos_est_max', 'Localidade_id', 'versao_modelo', 'municipio_nome']:
    del rio[col]

rio.head()


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Let's look at the tables
Can you write Python code for it?

A.head()

T = T[~T.index.duplicated()]
T.to_csv('temperature_rio.csv', header=True, sep=',')
T.head()

Tw = Tw[~Tw.index.duplicated()]
Tw.head()


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Let's try to join the tables by date. To align them, we must downsample each one to a weekly time frame
Can you write Python code for it?

T.resample('W-SUN').mean().tail()

Full = A.join(T.resample('W-SUN').mean()).join(Tw.resample('W-SUN').sum())
Full.head()


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Loading The Data
for our exploration let's pick the city of Rio de Janeiro.
Can you write Python code for it?


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Looking at state-wide data
Can you write Python code for it?

rio  = get_alerta_table(state='RJ')

rio.head()


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Note que as datas para as datas mais antigas os dados faltantes de Temperatura e Tweets, foram substituídos por NaN. Podemos remover estas datas, ficando com uma tabela sem dados faltantes. Mas perde-se mais de dois anos de dados.
Can you write Python code for it?

Short = Full.dropna()
Short.head()

Short[['casos_est', 'temp_min', 'umid_min', 'numero']].plot(subplots=True, figsize=(15,10),grid=True);


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Now we have a multi-level column index. It may be preferable to flatten it.
Can you write Python code for it?

riopiv.columns = ['{}_{}'.format(*col).strip() for col in riopiv.columns.values]
riopiv.head()

riopiv.shape


I am working on the file "Notebooks/Data Exploration.ipynb".
The first task is:
Splitting the dataset for out-of-sample prediction validation
Can you write Python code for it?

plt.figure()
ax = plt.gca()
train = Full.loc[Full.index<'2015-01-01']
model_3 = GAS.build_model(train, ar=2, sc=6, target='casos')
fit_3 = model_3.fit()
Full.casos.plot(style='ko', ax=ax, figsize=(15,10))
plt.hold(True)
model_3.plot_predict(h=10, past_values=20, ax=ax, intervals=True, figsize=(15,10))

model_4 = GASX.build_model(Full.dropna(), ar=4, sc=6, formula='casos~1+temp_min')

fit_4 = model_4.fit()

fit_4.summary()

model_4.plot_fit()

model_4.plot_predict(h=10, past_values=15)


I am working on the file "Notebooks/Dataset_Comparision/dataset_comparision.ipynb".
The first task is:
Motivation
The functions below are reused from the nvd_introduction notebook and cve_mitre_introdcution notebook. These functions define fileloading functionality of csv and xml files. There are seperate function defenitions for both as they differ in method. While running this notebook please create a seperate folder (called data) to hold specifically data files. You may add or delete files from this folder depending on what is required for the results at that point. To count the number of vulnerabilities(in this case CVE ID), the count functions will tackle respective files.
Can you write Python code for it?

#function to load a csv file
#accepts folderpath and headerlist as parameter to load the data files
def file_csv(folderpath,addheader,headerlist):
    #this reads all files under that folder
    filepaths = glob.glob(folderpath+"data/*.csv")
    #we prepare a list, that will contain all the tables that exist in these file paths
    dataframe = []
    for filepath in filepaths:
        #if header is required to be added: as in NVD
        if addheader is True:
            dataframe.append(pd.read_csv(filepath,names=headerlist))
        else:
            dataframe.append(pd.read_csv(filepath))
    return pd.concat(dataframe);

#function to load a xml file
#accepts folderpath as parameter to load the data files
def file_xml(folderpath):
    filepaths = glob.glob(folderpath+"data/*.xml")
    count = 0;
    #uses ElementTree to parse the tree structure
    for filepath in filepaths:
        CVE_tree = ElementTree()
        CVE_tree.parse(filepath)
        CVE_root= CVE_tree.getroot()
        count = count+ countrow_xml(CVE_root,'{http://www.icasi.org/CVRF/schema/vuln/1.1}')
    return count


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
As expected, the conditioning results are the same. Labels will also be saved to and read from files when the save_model() and read_model() functions are used.
Below we plot the original Gaussian components in the $x-y$ plane, a blue line showing the condition $y=1.5$, and a red line showing the new conditional distribution for $x$ given $y=1.5\pm0.05$.
Can you write Python code for it?


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Based on the results of the above test, the model with 3 components has the lowest BIC score and is the optimal choice. However, on different runs with the same dataset, the 5 and 6 component models have sometimes shown a lower BIC, and some testing indicates that the 6 component model provides a superior fit to the data. Thus, for the rest of the demo we will use 6 components in the model.

Fitting a Model
Once we know how many components to use in the model, we can fit a model using that number of components using the fit_model function. Before fitting for this demo, we are going to split our SN dataset 65-35, with the larger subsample being used to fit the model and the smaller subsample providing a test sample that we can use to predict supernovae and compare with our predicted sample.
Can you write Python code for it?

# Split the dataset 65/35:
X_train, X_test, Xerr_train, Xerr_test, R_param_train, R_param_test = \
    train_test_split(X, Xerr, R_params, test_size=0.35, random_state=42)

# Fit the model:
#empiricist.fit_model(X_train, Xerr_train, filename = 'demo_model.fit', n_components=7)
empiricist.read_model('demo_model.fit')


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Component Number Selection
We could fit this data directly, but one problem is that we don't know the optimal number of components (Gaussians) to use in the fit. Knowing the optimal number of components to fit allows us to obtain a good fit in the smallest amount of time without overfitting the data.
BIC
One way this can be determined is by fitting the model with different numbers of components and calculating the Bayesian information criterion (BIC) for each model. The BIC incorporates the number of components in the model, the sample size, and the likelihood of the data under the model, and the model with the lowest score is the optimal model to use.
We can test for the model with the lowest BIC score using the bic_test function, which will compute the BIC for a given dataset and range of n_components and return an array containing all the BIC scores as well as the optimal number of components and corresponding BIC value. The code below will use this to test the performance of the model with n_components ranging from 1 to 10 and produce a figure to display the results.
Note that due to the multiple fits, a BIC test may take a while to run, depending on your system.
Can you write Python code for it?

# Instantiate an XDGMM model:
xdgmm = XDGMM()

# Define the range of component numbers, and get ready to compute the BIC for each one:
param_range = np.array([1,2,3,4,5,6,7,8,9,10])

# Loop over component numbers, fitting XDGMM model and computing the BIC:
bic, optimal_n_comp, lowest_bic = xdgmm.bic_test(X, Xerr, param_range)

plot_bic(param_range, bic, optimal_n_comp)


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Component Test
We could fit our supernova/host dataset directly, but one problem is that we don't know the optimal number of components (Gaussians) to use in the fit. Knowing the optimal number of components to fit allows us to obtain a good fit in the smallest amount of time without overfitting the data.
BIC
One way to determine the optimal number of Gaussian components to use in the model is by fitting the model with different numbers of components and calculating the Bayesian information criterion (BIC) for each model. The BIC incorporates the number of components in the model, the sample size, and the likelihood of the data under the model, and the model with the lowest score is the optimal model to use.
We can test for the model with the lowest BIC score for a given dataset using the component_test function, which will compute the BIC for a given dataset and range of n_components and return an array containing all the BIC scores as well as the optimal number of components and corresponding BIC value. The code below will read in all the SN and host parameters we want to use from our data files (using the get_data function) and use these data to test the performance of the model with n_components ranging from 1 to 8. (Larger numbers of components tend to run into errors occurring because too few observations map to a given Gaussian component. With a bigger dataset, this range could be increased.)
Note that due to the multiple fits and the large dataset, the BIC test will likely take a while to run, depending on your system.
Can you write Python code for it?

# Instantiate an empiriciSN worker object:
empiricist = empiriciSN.Empiricist()

# Define the range of component numbers and read in the dataset:
component_range = np.array([1,2,3,4,5,6,7,8])
flist = (['data_files/sdss_master.csv','data_files/snls_master.csv'])
X, Xerr, R_params = get_demo_data(flist)

# Loop over component numbers, fitting XDGMM model and computing the BIC.
# This takes 30 minutes or so. Uncomment this line to run the test, but
# to save time you can just accept the values below.

#bics, optimal_n_comp, lowest_bic = empiricist.component_test(X, Xerr, component_range)

bics = np.array([-7586.49881994, -19276.8161054, -24393.2132252, -24178.7063611,
                 -23154.5882926, -23740.5529757, -22697.0279733, -20953.6879974])
lowest_bic = -24393.2132252
optimal_n_comp = 3

plot_bic(component_range, bics, optimal_n_comp)


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Conditional Prediction
As stated before, one potential use of a conditioned model is to create a "prediction engine" that can predict some parameters using an XDGMM model conditioned on known values of other parameters.
To demonstrate this, let's sample 1000 data points from our joint model to create a dataset to be compared with our predictions. Now, at the same $y$ values as this "observed" dataset, we'll condition the model on each $y$ value in turn, and draw a single predicted $x$ value. These predicted $x$'s should follow the same distribution as the observed $x$'s.
Can you write Python code for it?

# Simulate a dataset:
true_sample = xdgmm.sample(1000)
true_x = true_sample[:,0]
y = true_sample[:,1]

# Predict x values given y values:
predicted_x = np.array([])
for this_y in y:
    # Specify y-conditioning to apply to P(x,y):
    on_this = np.array([np.nan,this_y])
    # Compute conditional PDF P(x|y):
    cond_gmm = xdgmm.condition(on_this)
    # Draw a sample x value from this PDF, and add it to the growing list
    predicted_x = np.append(predicted_x, cond_gmm.sample())

# Plot the two datasets, to compare the true x and the predicted x:
plot_conditional_predictions(y, true_x, predicted_x)


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Conditioning the Model
One of the key advantages of our implementation of the XDGMM class is that it contains a method for producing a model for the conditional PDF $P(x | y)$ from the original GMM model for the joint PDF $P(x, y)$. The conditional PDF is also a GMM with the same number of components, but with adjusted weights, means and variance. This is useful for predicting new $x$ values given test $y$ values: we simply sample from the conditioned model.
For our example dataset, let's assume that we have measured the $y$ value to be $1.5\pm0.05$. We can produce a new model for $x$ that is conditioned on $y=1.5\pm0.05$, and use that to sample values of $x$. When condition() is called, we set all parameters that we do not want included in the conditioning to NaN. The model can be conditioned on values either with uncertainties or without; to condition without uncertainty either do not pass an error array, or set all the uncertainties equal to $0$. If provided, the uncertainties are interpreted as being Gaussian and uncorrelated, and thus providing a simple $P(y)$ to be combined with the joint PDF.
Can you write Python code for it?

cond_X = np.array([np.nan, 1.5])
cond_Xerr = np.array([0.0,0.05])
cond_xdgmm = xdgmm.condition(X_input = cond_X,Xerr_input = cond_Xerr)

# Compare the conditioned model to the original:
print xdgmm.weights
print cond_xdgmm.weights
print "\n"
print xdgmm.mu
print cond_xdgmm.mu


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Cross-Validation
Another way to determine the number of components is to perform a cross-validation test using different numbers of components. Since this XDGMM class implements all the methods necessary to use scikit-learn cross-validation tools, we can use them to test the performance of the model with different numbers of components.
Here we do a cross-validation test using the same numbers of Gaussian components as in the BIC test above. As with the BIC test, this can be a slow process depending on your system due to the repeated fitting.

One important thing to note is the trick used to pass errors to the scikit-learn methods. Normally for unsupervised learning you would only pass an X array, and not a "target" y array, to the validation_curve method. However, we need to pass an error array to our xdgmm.fit method, and validation_curve simply uses the y parameter as the second argument for the fit function. So, by treating the Xerr array as the "target" array, we can pass it to our fit function so that it can be used in fitting the data.
Can you write Python code for it?

param_range = np.array([1,2,3,4,5,6,7,8,9,10])
shuffle_split = ShuffleSplit(3, test_size=0.3,random_state=0)

train_scores,test_scores = validation_curve(xdgmm, X=X, y=Xerr,
                                            param_name="n_components",
                                            param_range=param_range,
                                            n_jobs=3,
                                            cv=shuffle_split,
                                            verbose=1)

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)

plot_val_curve(param_range, train_scores_mean, train_scores_std, test_scores_mean, test_scores_std)


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
If we sample 1000 points from this conditional distribution, we would get something like this:
Can you write Python code for it?

cond_sample = cond_xdgmm.sample(1000)
y = np.ones(1000)*1.5
plot_cond_sample(cond_sample,y)


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Next we will use our host properties to predict a supernova for each host, then plot those with the test sample. Ideally, these distributions should be the same as those from the test sample.
Can you write Python code for it?

# Set up an array to contain the new supernova properties:
n_Hosts = len(sample_logR)
sample_SNe = np.zeros([n_Hosts,3])

np.random.seed(10)

# Loop over host galaxies, conditioning on each one's properties and drawing one supernova:
for i in range(n_Hosts):

    x = np.append(X_test[i][3],sample_logR[i])
    x = np.append(x,X_test[i][5:15])
    x = np.append(x,local_SB[i])
    x = np.append(np.array([np.nan,np.nan,np.nan]),x)

    xerr = np.append(np.array([0.0,0.0,0.0]),Xerr_test[i,3,3])
    xerr = np.append(xerr,0.0)
    for j in range(5,15):
        xerr = np.append(xerr,Xerr_test[i,j,j])
    xerr = np.append(xerr,local_SB_err[i])

    sample_SNe[i] = empiricist.get_SN(x, Xerr=xerr, n_SN=1)

# Plot the SNe parameters:
corner.corner(sample_SNe[:,0:3], labels=['$x_0$','$x_1$','c'], color='red',
                                 fig=figure, range=[(-0.0001,0.0003),(-4.7,5.0),(-0.5,0.5)])


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Next, download the latest version of the SDSS and SNLS SN/host datasets that were used to build this class so we have some data files to use.
Can you write Python code for it?

#Due to changes in Dropbox's file structure, the former method of downloading the data files no longer works.
#There hasn't been an update to the files in a few months, so we are changing this box to use the files already
#In the data_files directory. If there are updates to the data files in the future, we will update this to find
#a better solution.

targdir = 'data_files'
if not os.path.isdir(targdir):
    os.mkdir(targdir)

filenames = ('sdss_master.csv','snls_master.csv')

remotedir = 'https://www.dropbox.com/s/k960gsg4b5ni6ap/'

for filename in filenames:
    path = os.path.join(targdir, filename)
    url = os.path.join(remotedir, filename)

    #Always download the files, in case something has been updated:
    #urllib.urlretrieve(url, path)

    !wc -l $path


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Next, generate some data to use for our fitting and plotting. This generates the same dataset as in the AstroML demo.
Can you write Python code for it?

N = 2000
np.random.seed(0)

# generate the true data
x_true = (1.4 + 2 * np.random.random(N)) ** 2
y_true = 0.1 * x_true ** 2

# add scatter to "true" distribution
dx = 0.1 + 4. / x_true ** 2
dy = 0.1 + 10. / x_true ** 2

x_true += np.random.normal(0, dx, N)
y_true += np.random.normal(0, dy, N)

# add noise to get the "observed" distribution
dx = 0.2 + 0.5 * np.random.random(N)
dy = 0.2 + 0.5 * np.random.random(N)

x = x_true + np.random.normal(0, dx)
y = y_true + np.random.normal(0, dy)

# stack the results for computation
X = np.vstack([x, y]).T
Xerr = np.zeros(X.shape + X.shape[-1:])
diag = np.arange(X.shape[-1])
Xerr[:, diag, diag] = np.vstack([dx ** 2, dy ** 2]).T


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Not too bad overall. Given a larger test sample, this would likely look a bit better, but we are dealing with fairly small sample sizes.
Finally, let's take a look at the distribution of properties for a single radius in a single host galaxy. By sampling 1000 different supernovae at that position, we can see how the model has been conditioned to that host. We should see a significantly different distribution of supernova parameters from the previous plot, if supernova properties depend on either their positions within their hosts, or their host's properties, or both.
Can you write Python code for it?

# Choose one (host galaxy, radial position) combination:
i = 10

np.random.seed(10)

# Set up the conditioning data:
x = np.append(X_test[i][3], sample_logR[i])
x = np.append(x, X_test[i][5:15])
x = np.append(x, local_SB[i])
x = np.append(np.array([np.nan, np.nan, np.nan]), x)

xerr = np.append(np.array([0.0,0.0,0.0]), Xerr_test[i,3,3])
xerr = np.append(xerr, 0.0)
for j in range(5,15):
    xerr = np.append(xerr, Xerr_test[i,j,j])
xerr = np.append(xerr, local_SB_err[i])

# Draw a thousand SNe:
sample_SNe = empiricist.get_SN(x, Xerr=xerr, n_SN=1000)

# Plot the SNe parameters:
corner.corner(sample_SNe[:,0:3], labels=['$x_0$','$x_1$','c'], color='blue',
                                 fig=figure, range=[(-0.0001,0.0003),(-4.7,5.0),(-0.5,0.5)])


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Note how the fit_model function also saves the fit to a file for later re-use. If a model has already been fit, it can be read into an existing Empiricist worker, or a new Empiricist can be instantiated using the model, like this:
Can you write Python code for it?

alternative = empiriciSN.Empiricist(model_file='demo_model.fit')

# Print the weights array for each object---they should be the same...
print empiricist.XDGMM.weights
print alternative.XDGMM.weights


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Note how the number of components in the conditioned model is the same as in the original joint model, but that the weights of the components have changed, and the mu array is now 1-dimensional (since $y$ has been conditioned out).
A list of labels for each parameter in the dataset can also be stored in the XDGMM object, and then a dictionary object can be used to condition the model using the labels. This removes the need for the user to remember which index corresponds to which parameter. When using a dictionary, it is only necessary to pass values for the labels that you want to condition on. If floats are given with each label, then condition() assumes that these are data values and that the uncertainty is 0. If tuples are given, the function assumes that the tuples contain (data, uncertainty) pairs. The new, conditioned XDGMM object will retain the labels of any parameters that were not conditioned out. This functionality is demonstrated below.
If the data used to fit the XDGMM model are in the form of a Pandas DataFrame and the columns have been labelled, the XDGMM object will automatically save the column labels in the labels array for future use.
Can you write Python code for it?

# First, set the labels in the XDGMM object
xdgmm.labels = np.array(['x','y'])

# The dictionary can pass either floats or tuples
cond_dict = {'y':(1.5,0.05)}
cond_xdgmm2 = xdgmm.condition(X_dict = cond_dict)

# Print the weights and means of the new model.
print cond_xdgmm2.weights
print cond_xdgmm2.mu
print cond_xdgmm2.labels


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Now we have a set of test separations in units of log(R/Re) and a set of separations sampled from the model. These should have the same distribution when plotted.
Can you write Python code for it?


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Once the model is saved, it can be read into an XDGMM object using the read_model() function, or a new XDGMM object can be initialized directly from the saved model file. Note that if both a filename and model parameters are passed to the constructor, the parameters saved in the file will override those passed by the user.
Can you write Python code for it?

# Read model into an existing XDGMM object
xdgmm.read_model('demo_model.fit')

# Initialize a new XDGMM object using the model
xdgmm2 = XDGMM(filename='demo_model.fit')

# Comparison --- the arrays should be the same.
print xdgmm.weights
print xdgmm2.weights


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Predicting Supernova Parameters
Now that we have the local surface brightnesses and separations, we can generate supernova parameters using our full range of host parameters (redshift, separation, colors, and local surface brightnesses).
First, let's plot the distributions of supernova parameters in our test sample.
Can you write Python code for it?


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Predicting a Likely Supernova
For the rest of this demo we will show how to use a model that has been fit to predict realistic supernovae from known host quantities.
Since some of the quantities used to fit the model require knowledge of the supernova location (e.g., local surface brightness), this is a multi-step process.

First, we will use a subset of the host properties to select a likely location.

Then we will compute the local surface brightness, and use the full set of host parameters to condition the XDGMM model and sample a realistic supernova.

The "test" sample generated above gives us a set of 482 host properties that we can use to fit supernovae, and a set of supernova properties to compare with our model fits.
Selecting the SN location within the host galaxy
<!-- One desired function of this tool is to provide a realistic location for a supernova within a host, along with realistic supernova properties. However, our overall model is trained with both overall host properties, like color and redshift, and the local surface brightness at the location of the supernova. In order for our tool to allow the user to fit a supernova based on host photometry and a surface brightness profile, we have provided a way to select a location for the supernova based on a subset of host properties so that local surface brightness can be calculated, and then the full set of host properties can be used to condition the model to provide supernova properties. -->

The get_logR function:

takes as input an array of host properties, an array of corresponding indices in the dataset that is being fit, and the index of the $\log{R/R_e}$ quantity in the dataset;

conditions the model based on the input host properties;

and returns a value for $\log{R/R_e}$ that is sampled from the conditioned model.

We built the function to accept any number of host parameters and any indices from the input data for its conditioning set in order to make it as general as possible. However, for this demo we will use the redshift...
Can you write Python code for it?

# Get actual separations from dataset, for comparison:
logR_test = X_test[:,4]

# Predict a radial separation for each host:
np.random.seed(0)
cond_indices = np.array([3,5,6,7,8,9,10,11,12,13,14])
sample_logR = np.array([])

for x in X_test:
    r = empiricist.get_logR(cond_indices,4,x[cond_indices])
    sample_logR = np.append(sample_logR,r)


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Sampling from the Model
Now that we have a model that has been fit to the data, we can use it to sample data points. Here we sample 2000 points from the model and plot the "true" distribution of the data, the "observed" noisy data, the data generated by the model, and the 1-sigma Gaussian components in the model.
Can you write Python code for it?

sample = xdgmm.sample(N)

plot_sample(x_true, y_true, x, y, sample, xdgmm)


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
Saving to and Reading from a File
Since fitting the model, even one with only a few components, can be a lengthy process, it can save a lot of time to save the model after fitting so that the same model can be used in the future without the need to re-fit the data. This can be done by calling the save_model() function.
Can you write Python code for it?


I am working on the file "Notebooks/Demo.ipynb".
The first task is:
We can see here that the cross-validation test prefers the maximum number of components (10) that we allowed for the model (and in fact, the score continues to rise as more components are added beyond 10). This is a result of the particular dataset being fit: the likelihood of the data being fit increases with more components, and there is enough structure to the data that even with a large number of Gaussians, the trained model continues to be a good predictor of new data.
However, increasing the number of components in the model rapidly causes the fit algorithm to become computationally expensive, especially for the astroML algorithm. While a model with a large number of components may be mathematically superior, in most cases the BIC seems to provide a way to find a model that is "good enough" to fit the data well, while also keeping the number of components at a value that keeps the computation of new fits reasonable.
Based on the BIC results, let's choose 5 components to model these data.
Model Fitting
Now that we know the best number of components to use, we can fit the data. First set the number of components, then perform the fit.
Can you write Python code for it?

xdgmm.n_components = optimal_n_comp
xdgmm = xdgmm.fit(X, Xerr)


I am working on the file "Notebooks/Density Profile.ipynb".
The first task is:
$$
\rho_{BE} (r)  = \frac{C_s^2}{4 \pi G r^2} =\frac{L_0^2 \rho _0 t_0}{t_0^2  L_0^2}
$$
Can you write Python code for it?

xs=HCG['X'][mid:]
ysP=HCG['PRS'][mid,mid:,ind]
xscgs=xs*10.
xau=xscgs*206265.
ysPcgs=ysP*f.PRS0
ax=plt.subplot(111)
ax.plot(xscgs,ysPcgs)
ax.set_yscale('log')
ax.set_ylabel('$ P\, (\si{dyne.cm^{-2}})$')
ax.set_xlabel('$\si{pc}$')
ax.set_xlim(None,10)


I am working on the file "Notebooks/Density Profile.ipynb".
The first task is:
Data from H2Cooling with Gravity
Can you write Python code for it?

#HCG=np.load('../Data/H2CoolingG512.npz')
HCG=np.load('../Data/TabulatedG.npz')
f.quadruple(HCG,np.log10(HCG['RHO']),rows=2,nn=0,tlim=26,Save_Figure='H2CoolGRquad')
f.pprofile(HCG,'RHO',steps=4,itlim=26,tdk='Myrs',Save_Figure='H2CoolGRHOprofile',sc2='log',xlim=[0,20],yprop=256)
f.pprofile(HCG,'Temp',steps=4,itlim=26,tdk='Myrs',yl='Temperature (K)',Save_Figure='H2CoolGTempprofile',sc2='log',xlim=[0,20],yprop=256)
f.pprofile(HCG,'PRS',steps=4,itlim=26,tdk='Myrs',yl='Pressure $(\si{dyne.cm^{-2}})$',y0=f.PRS0,Save_Figure='H2CoolGPRSprofile',yprop=256,sc2='log',xlim=[0,15])

(HCG['T'][26]-HCG['T'][25])/1e6

NCG['RHO'][128,128,:60].argmax()

plt.figure(figsize=(12,6))
plt.plot(NCG['T'][:38]/1e6,NCG['PRS'][128,128,:][:38]*f.Temp0/NCG['RHO'][128,128,:][:38],label='No Cooling Simulation Data')
plt.plot(HCG['T']/1e6,HCG['PRS'][256,256,:]*f.Temp0/HCG['RHO'][256,256,:],label='Cooling Simulation Data')
plt.yscale('log')
plt.ylabel('Temperature (K)')
plt.xlabel('$\si{Myrs}$')
plt.legend()
datafolder='../Document/DataImages/'
plt.savefig(datafolder+'GRcenterTimeTemp.png',bbox_inches='tight',format='png', dpi=100)

plt.figure(figsize=(12,6))
plt.plot(NCG['T'][:38]/1e6,NCG['RHO'][128,128,:][:38]*f.RHO0,label='No Cooling Simulation Data')
plt.plot(HCG['T']/1e6,HCG['RHO'][256,256,:]*f.RHO0,label='Cooling Simulation Data')
plt.yscale('log')
plt.ylabel('$ \\rho\, (\si{g.cm^{-3}})$')
plt.xlabel('$\si{Myrs}$')
plt.legend()
datafolder='../Document/DataImages/'
plt.savefig(datafolder+'GRcenterTimeRHO.png',bbox_inches='tight',format='png', dpi=100)

mid=256
ind= HCG['RHO'][mid,mid,:].argmax()
print 'Maximum Density: {:.2e} g/cm3 ({:.2e} cu) at time {:.2f} Myrs (index:...


I am working on the file "Notebooks/Density Profile.ipynb".
The first task is:
Simulate Profile
Can you write Python code for it?

Radius = 1.0
Density1 = 10.
P0=1e-8
T0=10891304347826.088
def rho(r,a=2.3,rho1=10.,R=1.,rc=0.002): return np.piecewise(r, [r < R , r >= R], [lambda r: rho1/(rc+r**a), 1.])
def mass(r,sm=False):
    return 4.*np.pi*r**2 *rho(r)*24.73 if sm else 4.*np.pi*r**2 *rho(r)
#def massmo(r,a,rc): return mass(r,a=-2.3,rho0=1e5,rc=0.005,R=10.) *24.73 #(10pc)^3 * hydrogen_mass /cm^3 = 24.73 Mo

A=10.
B=0.002
a=2.3

A/B,B**(1/a)

r=np.linspace(-2,3,500)
plt.ylabel('Number Density $(\si{cm^{-3}})$')
plt.xlabel('Radius $(\si{pc})$')
plt.yscale('log')
plt.plot(r*10.,rho(r),label=-2.3)
#plt.plot(r*10.,10./r**(2.3),label=-2.3)
#plt.vlines(0.07,1e2,1e5,linewidth=0.5)
plt.legend()

r=np.linspace(0,3,500)
plt.ylabel('$ \\rho\, (\si{g.cm^{-3}})$')
plt.xlabel('Radius $(\si{pc})$')
plt.yscale('log')
plt.plot(r*10.,rho(r)*f.RHO0,label='{} Density Profile'.format(-2.3))
plt.legend()
plt.xlim(0,16)
datafolder='../Document/DataImages/'
plt.savefig(datafolder+'SimRHOProfile.png',bbox_inches='tight',format='png', dpi=100)

plt.ylabel('Temperature$(\si{K})$')
plt.xlabel('Radius $(\SI{10}{pc})$')
plt.yscale('log')
plt.plot(r*10,P0*T0/rho(r),label=-2.3)
#plt.legend()
plt.xlim(0,16)
datafolder='../Document/DataImages/'
plt.savefig(datafolder+'SimTMPProfile.png',bbox_inches='tight',format='png', dpi=100)

(P0*T0/rho(r))[0]

plt.plot(r*10,mass(r,sm=True),label=-2.3)
#plt.plot(r,4.*np.pi*r**2 *rho(r),label=-2.3)
plt.ylabel('Mass $(\si{M_\odot})$')
plt.xlabel('Radius $(\si{pc})$')
plt.legend()

plt.plot(r,mass(r),label=-2.3)
plt.ylabel('Mass (cu)')
plt.xlabel('Radius $(\si{pc})$')
plt.legend()

TotalMass=quad(mass,0,1.,args=True)[0]
print u'Total Mass: {:.3e} M☉'.format(TotalMass)
TotalMass=quad(mass,0,1.)[0]
print u'Total Mass: {:e} cu'.format(TotalMass)


I am working on the file "Notebooks/DetectorExample.ipynb".
The first task is:
First, we load the image and show it.
Can you write Python code for it?

#Load the image
path_to_image = 'images/graffiti.jpg'
img = cv2.imread(path_to_image)
sr.show_image(img)


I am working on the file "Notebooks/DetectorExample.ipynb".
The first task is:
Now we create a SalientDetector object, with some parameters.
Can you write Python code for it?

det = sr.SalientDetector(SE_size_factor=0.20,
                                lam_factor=4)


I am working on the file "Notebooks/DetectorExample.ipynb".
The first task is:
To load the saved features from file, use the loading funciton:
Can you write Python code for it?

import sys, os
sys.path.insert(0, os.path.abspath('..'))
import salientregions as sr
total_num_regions, num_regions, features = sr.load_ellipse_features_from_file('poly_features.txt')
print("total_num_regions: ", total_num_regions)
print("number of features per saliency type: ", num_regions)
#print "features: ", features


I am working on the file "Notebooks/DetectorExample.ipynb".
The first task is:
We can also save the elliptic parameters in text files. Below is an example of saving the polynomial coefficients of all regions represented as ellipses.
Can you write Python code for it?

total_num_regions = sr.save_ellipse_features2file(num_regions, features_poly, 'poly_features.txt')
print("total_num_regions", total_num_regions)


I am working on the file "Notebooks/Download EPA emissions data.ipynb".
The first task is:
Correct column names and export all files (1 file per year)
Also convert the OP_DATE column to a datetime object
Using joblib for this.
Joblib on windows requires the if name == 'main': statement. And in a Jupyter notebook the function needs to be imported from an external script. I probably should have done the parallel part at a higher level - the longest part is saving the csv files. Could use this method - disable a check - to speed up the process.
Joblib has to be at least version 10.0, which is only available through pip - got some errors when using the version installed by conda.
Create a dictionary mapping column names. Any values on the left (keys) should be replaced by values on the right (values).
Can you write Python code for it?

col_name_map = {'CO2_MASS' : 'CO2_MASS (tons)',
                'CO2_RATE' : 'CO2_RATE (tons/mmBtu)',
                'GLOAD' : 'GLOAD (MW)',
                'HEAT_INPUT' : 'HEAT_INPUT (mmBtu)',
                'NOX_MASS' : 'NOX_MASS (lbs)',
                'NOX_RATE' : 'NOX_RATE (lbs/mmBtu)',
                'SLOAD' : 'SLOAD (1000lb/hr)',
                'SLOAD (1000 lbs)' : 'SLOAD (1000lb/hr)',
                'SO2_MASS' : 'SO2_MASS (lbs)',
                'SO2_RATE' : 'SO2_RATE (lbs/mmBtu)'
                }


I am working on the file "Notebooks/Download EPA emissions data.ipynb".
The first task is:
Export file names and update timestamp
This is just a sample file with 2017 filenames
Can you write Python code for it?

name_time_df = pd.DataFrame(name_time_list, columns=['file name', 'last updated'])

name_time_df.head()

path = join(data_path, 'EPA downloads', 'name_time example.csv')
name_time_df.to_csv(path, index=False)


I am working on the file "Notebooks/Download EPA emissions data.ipynb".
The first task is:
Final script
This downloads hourly data from the ftp server over a range of years, and saves all of the file names/last update times in a list. The downloads can take some time depending on how much data is being retrieved.
Some of the code below assumes that we only need to retrieve new or modified files. If you are retrieving this data for the first time, create an empty dataframe named already_downloaded with column names file name and last updated.
Can you write Python code for it?

# Replace the filename with whatever csv stores already downloaded file info
# path = join('EPA downloads', 'name_time 2015-2016.csv')
# already_downloaded = pd.read_csv(path, parse_dates=['last updated'])

# Uncomment the line below to create an empty dataframe
already_downloaded = pd.DataFrame(columns=['file name', 'last updated'])

already_downloaded.head()

# Years of data to download
years = [2017]

# Make the EPA download directory if it doesn't exist
try:
    os.mkdir(join(data_path, 'EPA downloads'))
except:
    pass

# Timestamp
start_time = timeit.default_timer()
name_time_list = []
# Open ftp connection and navigate to the correct folder
print('Opening ftp connection')
ftp = FTP('newftp.epa.gov')
ftp.login()
ftp.cwd('/dmdnload/emissions/hourly/monthly')

for year in years:
    print(year)

    year_str = str(year)
    print('Change directory to {}'.format(year_str))
    try:
        ftp.cwd(year_str)
    except ftplib.all_errors as e:
        print(e)
        break

    # Use ftplib to get the list of filenames
    print('Fetch filenames')
    fnames = ftp.nlst()

    # Create new directory path if it doesn't exist
    new_path = join(data_path, 'EPA downloads', year_str)
    try:
        os.mkdir(new_path)
        print('New local folder created for {}'.format(year_str))
    except:
        pass

    # Look for files without _HLD in the name
    name_list = []
    time_list = []
    print('Find filenames without _HLD and time last updated')
    for name in fnames:
        if '_HLD' not in name:
            try:
                # The ftp command "MDTM" asks what time a file was last modified
                # It returns a code and the date/time
                # If the file name isn't already downloaded, or the time isn't the same
                tm = ftp.sendcmd('MDTM '+ name).split()[-1]
                dt = pd.to_datetime(tm[:8], format='%Y%m%d')
                if name not in already_downloaded['file name'].values:

                    time_list.append(dt)
...


I am working on the file "Notebooks/Download EPA emissions data.ipynb".
The first task is:
From the table below, recent years always have the units after an emission name. Before 2009 some files have the units and some don't. UNITID is consistent through all years, but UNIT_ID was added in after 2008 (not the same thing).
Can you write Python code for it?

pd.DataFrame(col_names)

pd.DataFrame(col_names).index

pd.DataFrame(num_cols)


I am working on the file "Notebooks/Download EPA emissions data.ipynb".
The first task is:
This takes ~100 ms per file.
Can you write Python code for it?

base_path = join(data_path, 'EPA downloads')
num_cols = {}
col_names = {}
for year in range(2001, 2018):

    n_cols_list = []
    col_name_list = []
    path = join(base_path, str(year))
    fnames = os.listdir(path)

    for name in fnames:
        fullpath = os.path.join(path, name)
        columns = pd.read_csv(fullpath, nrows=5).columns

        # Add the column names to the large list
        col_name_list.extend(columns)
        # Add the number of columns to the list
        n_cols_list.append(len(columns))

    col_names[year] = Counter(col_name_list)
    num_cols[year] = Counter(n_cols_list)


I am working on the file "Notebooks/Download_From_Firehose.ipynb".
The first task is:
Clean Up Downloads
No going back from here, so I would check your data to make sure everything got downloaded correctly.
Can you write Python code for it?

!rm -f tmp*
!rm script.sh

!rm fh_get.zip
!rm firehose_get

if not os.path.isdir(OUT_PATH):
    os.makedirs(OUT_PATH)

analyses_folder = 'analyses__' + RUN_DATE
!mv $analyses_folder {OUT_PATH + '/' + analyses_folder}

stddata_folder = 'stddata__' + RUN_DATE
!mv $stddata_folder {OUT_PATH + '/' + stddata_folder}


I am working on the file "Notebooks/Download_From_Firehose.ipynb".
The first task is:
Download Data
Can you write Python code for it?

d = 'http://gdac.broadinstitute.org/runs/analyses__{}/ingested_data.tsv'.format(RUN_DATE)
tab = pd.read_table(d, sep='\n', header=None)
skip = tab[0].dropna().apply(lambda s: s.startswith('#'))
skip = list(skip[skip==True].index)
tab = pd.read_table(d, skiprows=skip, index_col=0).dropna()

cancers = tab[tab.Clinical>0].index[:-1]
cancer_string = ' '.join(cancers)
cancer_string


I am working on the file "Notebooks/Download_From_Firehose.ipynb".
The first task is:
Get rid of all of the downloaded zip files that we processed
Can you write Python code for it?

!rm -rf {OUT_PATH + '/' + stddata_folder}
!rm -rf {OUT_PATH + '/' + analyses_folder}


I am working on the file "Notebooks/Download_From_Firehose.ipynb".
The first task is:
Initialization
Can you write Python code for it?

!curl http://gdac.broadinstitute.org/runs/code/firehose_get_latest.zip -o fh_get.zip
!unzip fh_get.zip


I am working on the file "Notebooks/Downsampling.ipynb".
The first task is:
Correlation of t-statistic and fraction overexpressed
Can you write Python code for it?

fig, ax = subplots()
(1 - pcc).sort_index().plot()
plt.yscale('log')


I am working on the file "Notebooks/EC50_Results/EC50 Prelim Results.ipynb".
The first task is:
ION_ACTIVITY [ACTIVE] Exemplar molecules that were added to the training set when moving from EC50_200 to EC50_500
Can you write Python code for it?

ion_active = ec50_500_added.loc[train_results["ION_ACTIVITY"] == "ACTIVE"].sort_values("EC50")
mols, labels = mol_label_list(ion_active)
Draw.MolsToGridImage(mols,legends=labels,molsPerRow=4)


I am working on the file "Notebooks/EC50_Results/EC50 Prelim Results.ipynb".
The first task is:
The results of the EC50_500 classification of the test molecules.
Can you write Python code for it?

sorted = test_results.sort_values("M5_500_250", ascending=False)
mols, labels = mol_label_list(sorted)
Draw.MolsToGridImage(mols,legends=labels,molsPerRow=4)


I am working on the file "Notebooks/EC50_Results/EC50 Prelim Results.ipynb".
The first task is:
We visualize the training set probability maps by normalizing them to the unit interval [0, 1] and sorting them in descending order.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from pylab import *
from sklearn.preprocessing import minmax_scale

def sort_map(column):
    array = minmax_scale(train_results[column])
    return array[np.argsort(-array)]

scale = 1.0
fig = plt.figure(num=None, figsize=(8 * scale, 6 * scale), dpi=80, facecolor='w', edgecolor='k')
for map in all_active: plt.plot(sort_map(map), label=map)
xlabel("molecules")
ylabel("normalized probability")
title(" Training Set [ACTIVE] Probability Maps")
legend(loc=1); # upper right corner


I am working on the file "Notebooks/EC50_Results/EC50 Prelim Results.ipynb".
The first task is:
Where the ML MODELs are as follows:

LOGC - A Logistic classifier from SKLearn.
DNN - A Deep Neural Network classifier [2048, 2048, 512, 64, 3] from the Keras toolkit. Cross-entropy loss function.

The molecular descriptors are as follows:

M2048_n - A 2048 bit, mol radius = n, Morgan fingerprint from RDKIT
DRAGON - A vector of 1666 pharmacophore molecular fields scaled to be on the interval [0,1].

Modelling.
To run these models, download (follow the readme setup) the software on gitHub here: https://github.com/kellerberrin/OSM-QSAR
For the LOGC SKLearn model (--help for flag descriptions) the following cmd was used:
$python OSM_QSAR.py --classify logc --depend EC50_500 --indep MORGAN2048_5
For the DNN with MORGAN fingerprints:
$python OSM_QSAR.py --classify bin_m --depend EC50_500 --indep MORGAN2048_5 --train 300 --check 10
For the DNN with the DRAGON descriptor:
$python OSM_QSAR.py --classify bin_d --depend EC50_500 --indep DRAGON --train 300 --check 10
Convergence of the DNN was prompt in all cases and occurred within 300 epochs.
Commentary.

Morgan fingerprints out-perform DRAGON in all cases.

LOGC and DNN have very similar performance with Morgan fingerprints.

A big jump in AUC from 0.82 training on EC50_200 (active at 200nM) to 0.93 training on EC50_500 (active at 500nM)

The LOGC classifier got significantly lower AUC on DRAGON than the DNN. This reinforces our observation in the previous notebook that the "off-the-shelf" classifiers of SKLearn have trouble interpreting the unstructured data of DRAGON.

We now take a closer look at the results.
The following fields are available in the results files.

ID
EC50 - in uMol
ION_ACTIVITY - molecule PfATP4 ion activity [ACTIVE, INACTIVE, PARTIAL, X_MISSING] (X_MISSING is unknown ion activity)
ACTUAL_200 - [ACTIVE, INACTIVE] at 200 nMol
ACTUAL_500 - [ACTIVE, INACTIVE] at 500 nMol
ACTUAL_1000 - [ACTIVE, INACTIVE] at 1000 nMol
M2_200_90 - The probability map of the DNN trained 90 epochs with --depend EC50_200 --indep MORGAN2048_2 (AUC...
Can you write Python code for it?

# Import the probability maps
import pandas as pd
test_results = pd.read_csv("./test_results.csv")
train_results = pd.read_csv("./train_results.csv")
EC50_500_active = ["M5_500_40", "M5_500_130", "M5_500_250", "LOGC_500"]
all_active = EC50_500_active + ["M5_1000_210", "M2_200_90"]

# Train results Pearson correlation
corr = train_results[EC50_500_active].corr(method="pearson")
corr

# Test results Pearson correlation
corr = test_results[EC50_500_active].corr(method="pearson")
corr


I am working on the file "Notebooks/EDA/.ipynb_checkpoints/EDA Worksheet - Python Version - Answers-checkpoint.ipynb".
The first task is:
Exercise 1: Which industry sees the most Ramnit infections? Least?
Count the number of infected days for "Ramnit" in each industry industry.
How:
1.  First filter the data to remove all the infections we don't care about
2.  Aggregate the data on the column of interest. HINT:  You might want to use the groupby() function
3.  Add up the results
Can you write Python code for it?

grouped_df = data[data.botfam == "Ramnit"].groupby(['industry'])
grouped_df.sum()


I am working on the file "Notebooks/EDA/.ipynb_checkpoints/EDA Worksheet - Python Version - Answers-checkpoint.ipynb".
The first task is:
Exercise 3: Which date had the total most bot infections and how many infections on that day?
In this exercise you are asked to aggregate and sum the number of infections (hosts) by date.  Once you've done that, the next step is to sort in descending order.
Can you write Python code for it?

df3 = data[['date','hosts']].groupby('date').agg(['count'])
df3.sort_values(by=[('hosts', 'count')], ascending=False).head(10)


I am working on the file "Notebooks/EDA/.ipynb_checkpoints/EDA Worksheet - Python Version - Answers-checkpoint.ipynb".
The first task is:
Exercise 4: Plot the daily infected hosts for Necurs, Ramnit and PushDo
In this exercise you're going to plot the daily infected hosts for three infection types.  In order to do this, you'll need to do the following steps:
1.  Filter the data to remove the botfamilies we don't care about.
2.  Use groupby() to aggregate the data by date and family, then sum up the hosts in each group
3.  Plot the data.  Hint:  You might want to use the unstack() function to prepare the data for plotting.
Can you write Python code for it?

filteredData = data[ data['botfam'].isin(['Necurs', 'Ramnit', 'PushDo']) ][['date', 'botfam', 'hosts']]
groupedFilteredData = filteredData.groupby( ['date', 'botfam']).sum()
groupedFilteredData.unstack(level=1).plot(kind='line', subplots=False)


I am working on the file "Notebooks/EDA/.ipynb_checkpoints/EDA Worksheet - Python Version - Answers-checkpoint.ipynb".
The first task is:
Exercise 5: What are the distribution of infected hosts for each day-of-week across all bot families?
Hint: try a box plot and/or violin plot.  In order to do this, there are two steps:
1.  First create a day column where the day of the week is represented as an integer.  You'll need to convert the date column to an actual date/time object.  See here: http://pandas.pydata.org/pandas-docs/stable/timeseries.html
2.  Next, use the .boxplot() method to plot the data. This has grouping built in, so you don't have to group by first.
Can you write Python code for it?

data.date = data.date  = pd.to_datetime( data.date )
data['day'] = data.date.dt.weekday
data[['hosts', 'day']].boxplot( by='day')

grouped = data.groupby('day')
grouped.boxplot('hosts')


I am working on the file "Notebooks/EDA/EDA Worksheet - Python Version - Answers.ipynb".
The first task is:
Exercise 1: Which industry sees the most Ramnit infections? Least?
Count the number of infected days for "Ramnit" in each industry industry.
How:
1.  First filter the data to remove all the infections we don't care about
2.  Aggregate the data on the column of interest. HINT:  You might want to use the groupby() function
3.  Add up the results
Can you write Python code for it?


I am working on the file "Notebooks/EDA/EDA Worksheet - Python Version - Answers.ipynb".
The first task is:
Exercise 2: Calculate the min, max, median and mean infected orgs for each bot family, sort by median
In this exercise, you are asked to calculate the min, max, median and mean of infected orgs for each bot family sorted by median.  HINT:
1.  Using the groupby() function, create a grouped data frame
2.  You can do this one metric at a time OR you can use the .agg() function. You might want to refer to the documentation here:  http://pandas.pydata.org/pandas-docs/stable/groupby.html#applying-multiple-functions-at-once
3.  Sort the values (HINT HINT) by the median column
Can you write Python code for it?

group2 = data[['botfam','orgs']].groupby( ['botfam'])
summary = group2.agg([np.min, np.max, np.mean, np.median, np.std])
summary.sort_values( [('orgs', 'median')], ascending=False)


I am working on the file "Notebooks/EDA/EDA Worksheet - Python Version - Answers.ipynb".
The first task is:
Exercise 3: Which date had the total most bot infections and how many infections on that day?
In this exercise you are asked to aggregate and sum the number of infections (hosts) by date.  Once you've done that, the next step is to sort in descending order.
Can you write Python code for it?

df3 = data[['date','hosts']].groupby('date').sum()
df3.sort_values(by='hosts', ascending=False).head(10)


I am working on the file "Notebooks/EDA/EDA Worksheet - Python Version - Answers.ipynb".
The first task is:
Exercise 4: Plot the daily infected hosts for Necurs, Ramnit and PushDo
In this exercise you're going to plot the daily infected hosts for three infection types.  In order to do this, you'll need to do the following steps:
1.  Filter the data to remove the botfamilies we don't care about.
2.  Use groupby() to aggregate the data by date and family, then sum up the hosts in each group
3.  Plot the data.  Hint:  You might want to use the unstack() function to prepare the data for plotting.
Can you write Python code for it?


I am working on the file "Notebooks/EDA/EDA Worksheet - Python Version - Answers.ipynb".
The first task is:
Exercise 5: What are the distribution of infected hosts for each day-of-week across all bot families?
Hint: try a box plot and/or violin plot.  In order to do this, there are two steps:
1.  First create a day column where the day of the week is represented as an integer.  You'll need to convert the date column to an actual date/time object.  See here: http://pandas.pydata.org/pandas-docs/stable/timeseries.html
2.  Next, use the .boxplot() method to plot the data. This has grouping built in, so you don't have to group by first.
Can you write Python code for it?

data.date  = pd.to_datetime( data.date )
data['day'] = data.date.dt.weekday
data[['hosts', 'day']].boxplot( by='day')

grouped = data[['hosts', 'day']].groupby('day')
print( grouped.sum() )

grouped.box


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Add datetime and quarter columns
Can you write Python code for it?


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Apply emission factors
Fuel emission factor is kg/mmbtu
Can you write Python code for it?

# Start with 0 emissions in all rows
# For fuels where we have an emission factor, replace the 0 with the calculated value
all_facility_data['all fuel fossil CO2 (kg)'] = 0
all_facility_data['elec fuel fossil CO2 (kg)'] = 0
all_facility_data['all fuel total CO2 (kg)'] = 0
all_facility_data['elec fuel total CO2 (kg)'] = 0
for fuel in total_factors.keys():
    # All fuel CO2 emissions
    all_facility_data.loc[all_facility_data['fuel']==fuel,'all fuel fossil CO2 (kg)'] = \
        all_facility_data.loc[all_facility_data['fuel']==fuel,'total fuel (mmbtu)'] * fossil_factors[fuel]
    all_facility_data.loc[all_facility_data['fuel']==fuel,'all fuel total CO2 (kg)'] = \
        all_facility_data.loc[all_facility_data['fuel']==fuel,'total fuel (mmbtu)'] * total_factors[fuel]

    # Electric fuel CO2 emissions
    all_facility_data.loc[all_facility_data['fuel']==fuel,'elec fuel fossil CO2 (kg)'] = \
        all_facility_data.loc[all_facility_data['fuel']==fuel,'elec fuel (mmbtu)'] * fossil_factors[fuel]
    all_facility_data.loc[all_facility_data['fuel']==fuel,'elec fuel total CO2 (kg)'] = \
        all_facility_data.loc[all_facility_data['fuel']==fuel,'elec fuel (mmbtu)'] * total_factors[fuel]


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Apply factors to facility generation
Can you write Python code for it?

fossil_factors = dict(zip(ef.index, ef['Fossil Factor']))
total_factors = dict(zip(ef.index, ef['Total Factor']))
fossil_factors, total_factors


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Combine generation into one large dataframe
Can you write Python code for it?

if __name__ == '__main__':
    exception_list = []
    facility_gen = pd.concat(Parallel(n_jobs=-1)(delayed(facility_line_to_df)(json.loads(row)) for row in gen_rows))
    facility_gen.reset_index(drop=True, inplace=True)
    facility_gen.rename({'value':'generation (MWh)'}, axis=1, inplace=True)

facility_gen.loc[:,'lat'] = facility_gen.loc[:,'lat'].astype(float)
facility_gen.loc[:,'lon'] = facility_gen.loc[:,'lon'].astype(float)
facility_gen.loc[:, 'plant id'] = facility_gen.loc[:, 'plant id'].astype(int)

#drop
facility_gen.tail()


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Combine total fuel use for electricity into one large dataframe
Can you write Python code for it?

if __name__ == '__main__':
    exception_list = []
    facility_eg_fuel = pd.concat(Parallel(n_jobs=-1)(delayed(facility_line_to_df)(json.loads(row)) for row in eg_fuel_rows))
    facility_eg_fuel.reset_index(drop=True, inplace=True)
    facility_eg_fuel.rename({'value':'elec fuel (mmbtu)'}, axis=1, inplace=True)

facility_eg_fuel.loc[:,'lat'] = facility_eg_fuel.loc[:,'lat'].astype(float)
facility_eg_fuel.loc[:,'lon'] = facility_eg_fuel.loc[:,'lon'].astype(float)
facility_eg_fuel.loc[:,'plant id'] = facility_eg_fuel.loc[:,'plant id'].astype(int)


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Combine total fuel use into one large dataframe
Can you write Python code for it?

if __name__ == '__main__':
    exception_list = []
    facility_all_fuel = pd.concat(Parallel(n_jobs=-1)(delayed(facility_line_to_df)(json.loads(row)) for row in total_fuel_rows))
    facility_all_fuel.reset_index(drop=True, inplace=True)
    facility_all_fuel.rename({'value':'total fuel (mmbtu)'}, axis=1, inplace=True)

facility_all_fuel.loc[:,'lat'] = facility_all_fuel.loc[:,'lat'].astype(float)
facility_all_fuel.loc[:,'lon'] = facility_all_fuel.loc[:,'lon'].astype(float)
facility_all_fuel.loc[:,'plant id'] = facility_all_fuel.loc[:,'plant id'].astype(int)


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Date string for filenames
This will be inserted into all filenames (reading and writing)
Can you write Python code for it?

file_date = '2018-03-06'

%load_ext watermark

%watermark -iv -v

# Load the "autoreload" extension
%load_ext autoreload

# always reload modules marked with "%aimport"
%autoreload 1

# add the 'src' directory as one where we can import modules
src_dir = join(os.getcwd(), os.pardir, 'src')
sys.path.append(src_dir)

%aimport Data.data_extraction
from Data.data_extraction import facility_line_to_df

%aimport Analysis.index
from Analysis.index import add_datetime, add_quarter


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Export
Can you write Python code for it?

path = join(data_path, 'Derived data',
            'Facility gen fuels and CO2 {}.csv'.format(file_date))
all_facility_data.to_csv(path, index=False)


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Fill in missing values from the first merge
Can you write Python code for it?

def fill_missing(df):
    cols = [col[:-2] for col in df.columns if '_x' in col]

    # Create new column from the _x version, fill missing values from the _y version
    for col in cols:
        df[col] = df.loc[:, col + '_x']
        df.loc[df[col].isnull(), col] = df.loc[df[col].isnull(), col + '_y']

        df.drop([col+'_x', col+'_y'], axis=1, inplace=True)

fill_missing(gen_total_fuel)

keep_cols = ['fuel', 'elec fuel (mmbtu)', 'month', 'plant id', 'prime mover', 'year',
             'geography', 'lat', 'lon', 'last_updated']
all_facility_data = gen_total_fuel.merge(facility_eg_fuel.loc[:,keep_cols],
                                         how='outer', on=merge_cols)


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Load emission factors
These are mostly EIA emission factors
Can you write Python code for it?

path = join(data_path, 'Final emission factors.csv')
ef = pd.read_csv(path, index_col=0)


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Merge dataframes
Need to be careful here because there are fuel/prime mover combinations that have generation but no fuel use (e.g. the steam cycle of a combined cycle system - but only in some cases).
Can you write Python code for it?

keep_cols = ['fuel', 'generation (MWh)', 'month', 'plant id', 'prime mover', 'year',
             'geography', 'lat', 'lon', 'last_updated']
merge_cols = ['fuel', 'month', 'plant id', 'year']
gen_total_fuel = facility_all_fuel.merge(facility_gen.loc[:,keep_cols],
                                    how='outer', on=merge_cols)


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Read ELEC.txt file
Download the most current file from EIA's bulk download site. Save it to \Data storage\Raw data. I've tried to do this via the requests library, but the data file often gets corrupted.
Can you write Python code for it?

path = join(data_path, 'Raw EIA bulk', '{} ELEC.txt'.format(file_date))
with open(path, 'r') as f:
    raw_txt = f.readlines()


I am working on the file "Notebooks/EIA Bulk Download - extract facility generation.ipynb".
The first task is:
Set nan and negative emissions to 0
When no fuel was used for electricity production, or when negative fuel is somehow reported by EIA, set the emissions to 0. This is implemented by filtering out all values that are greater than or equal to 0.
Can you write Python code for it?

# Fossil CO2
all_facility_data.loc[~(all_facility_data['all fuel fossil CO2 (kg)']>=0),
                      'all fuel fossil CO2 (kg)'] = 0
all_facility_data.loc[~(all_facility_data['elec fuel fossil CO2 (kg)']>=0),
                      'elec fuel fossil CO2 (kg)'] = 0
# Total CO2
all_facility_data.loc[~(all_facility_data['all fuel total CO2 (kg)']>=0),
                      'all fuel total CO2 (kg)'] = 0
all_facility_data.loc[~(all_facility_data['elec fuel total CO2 (kg)']>=0),
                      'elec fuel total CO2 (kg)'] = 0


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Add CO2 emissions
The difficulty here is that EIA combines all types of coal fuel consumption together in the bulk download and API. Fortunately the emission factors for different coal types aren't too far off on an energy basis (BIT is 93.3 kg/mmbtu, SUB is 97.2 kg/mmbtu). I'm going to average the BIT and SUB factors rather than trying to do something more complicated. In 2015 BIT represented 45% of coal energy for electricity and SUB represented 48%.
Same issue with petroleum liquids. Using the average of DFO and RFO, which were the two largest share of petroleum liquids.
Can you write Python code for it?


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Add datetime and quarter columns
Can you write Python code for it?


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Combine three datasets
Need to estimate fuel use for OOG, because EIA doesn't include any (this is only ~2% of OOG fuel for electricity in 2015).
Can you write Python code for it?

fuel_df = pd.concat([total_fuel_df, eg_fuel_df['elec fuel (mmbtu)']], axis=1)


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Date string for filenames
This will be inserted into all filenames (reading and writing)
Can you write Python code for it?

file_date = '2018-03-06'

%load_ext watermark

%watermark -iv -v

# Load the "autoreload" extension
%load_ext autoreload

# always reload modules marked with "%aimport"
%autoreload 1

# add the 'src' directory as one where we can import modules
src_dir = join(os.getcwd(), os.pardir, 'src')
sys.path.append(src_dir)

%aimport Analysis.index
from Analysis.index import add_datetime, add_quarter


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Export data
State-level
Can you write Python code for it?

path = join(data_path, 'EIA state-level gen fuel CO2 2018-03-06.csv')
gen_fuel_df.to_csv(path)


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Filter lines to only include total generation and fuel use
Only want monthly US data for all sectors
- US-99.M
- ELEC.GEN, ELEC.CONS_TOT_BTU, ELEC.CONS_EG_BTU
- not ALL
Fuel codes:
- WWW, wood and wood derived fuels
- WND, wind
- STH, solar thermal
- WAS, other biomass
- TSN, all solar
- SUN, utility-scale solar
- NUC, nuclear
- NG, natural gas
- PEL, petroleum liquids
- SPV, utility-scale solar photovoltaic
- PC, petroluem coke
- OTH, other
- COW, coal,
- DPV, distributed photovoltaic
- OOG, other gases
- HPS, hydro pumped storage
- HYC, conventional hydroelectric
- GEO, geothermal
- AOR, other renewables (total)
Can you write Python code for it?

def line_to_df(line):
    """
    Takes in a line (dictionary), returns a dataframe
    """
    for key in ['latlon', 'source', 'copyright', 'description',
                'geoset_id', 'iso3166', 'name', 'state']:
        line.pop(key, None)

    # Split the series_id up to extract information
    # Example: ELEC.PLANT.GEN.388-WAT-ALL.M
    series_id = line['series_id']
    series_id_list = series_id.split('.')
    # Use the second to last item in list rather than third
    plant_fuel_mover = series_id_list[-2].split('-')
    line['type'] = plant_fuel_mover[0]
#     line['state'] = plant_fuel_mover[1]
    line['sector'] = plant_fuel_mover[2]
    temp_df = pd.DataFrame(line)

    try:
        temp_df['year'] = temp_df.apply(lambda x: x['data'][0][:4], axis=1).astype(int)
        temp_df['month'] = temp_df.apply(lambda x: x['data'][0][-2:], axis=1).astype(int)
        temp_df['value'] = temp_df.apply(lambda x: x['data'][1], axis=1)
        temp_df.drop('data', axis=1, inplace=True)
        return temp_df
    except:
        exception_list.append(line)
        pass

states = ["AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE",
          "FL", "GA", "HI", "ID", "IL", "IN", "IA", "KS",
          "KY", "LA", "ME", "MD", "MA", "MI", "MN", "MS",
          "MO", "MT", "NE", "NV", "NH", "NJ", "NM", "NY",
          "NC", "ND", "OH", "OK", "OR", "PA", "RI", "SC",
          "SD", "TN", "TX", "UT", "VT", "VA", "WA", "WV", "WI", "WY"]

state_geos = ['USA-{}'.format(state) for state in states]

type(json.loads(raw_txt[0]))

json.loads(raw_txt[0])['geography']

gen_rows = [row for row in raw_txt if 'ELEC.GEN' in row
            and 'series_id' in row
            and '-99.M' in row
            and 'ALL' not in row]

total_fuel_rows = [row for row in raw_txt if 'ELEC.CONS_TOT_BTU' in row
                   and 'series_id' in row
                   and '-99.M' in row
                   and 'ALL' not in row
                   and 'US-99.m' not in row]

eg_fuel_rows = [row for row in raw_txt if 'ELEC.CONS_EG_BTU' in row
      ...


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Multiply generation values by 1,000,000 and change the units to MMBtu
Can you write Python code for it?

eg_fuel_df.loc[:,'value'] *= 1E6
eg_fuel_df.loc[:,'units'] = 'mmbtu'

eg_fuel_df.rename_axis({'value':'elec fuel (mmbtu)'}, axis=1, inplace=True)

eg_fuel_df.set_index(['type', 'year', 'month', 'geography'], inplace=True)

#drop
eg_fuel_df.head()


total_fuel_df.loc[:,'value'] *= 1E6
total_fuel_df.loc[:,'units'] = 'mmbtu'

total_fuel_df.rename_axis({'value':'total fuel (mmbtu)'}, axis=1, inplace=True)

total_fuel_df.set_index(['type', 'year', 'month', 'geography'], inplace=True)


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
No records with positive fuel use but no generation
Can you write Python code for it?

gen_fuel_df['generation (MWh)'].fillna(value=0, inplace=True)

gen_fuel_df.loc['COW',:].head()


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Not sure how this happens in EIA's data, but we do see the negative fuel consumption for electricity generation.
Can you write Python code for it?

#drop
fuel_df.loc[~(fuel_df['elec fuel (mmbtu)']>=0)]

#drop
fuel_df.loc[~(fuel_df['total fuel (mmbtu)']>=0)]


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Read ELECT.txt file
Can you write Python code for it?

cwd = os.getcwd()
path = join(data_path, 'Raw EIA bulk', '{} ELEC.txt'.format(file_date))
with open(path, 'r') as f:
    raw_txt = f.readlines()


I am working on the file "Notebooks/EIA bulk download - non-facility (distributed PV & state-level).ipynb".
The first task is:
Total fuel consumption
Can you write Python code for it?

total_fuel_dict = [json.loads(row) for row in total_fuel_rows]

total_fuel_df = pd.concat([line_to_df(x) for x in total_fuel_dict
                           if x['geography'] in state_geos])


I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
As we can see, data are not ordered chronologically. We must reverse the order of rows so that first row is the oldest, last row is the newest.
Can you write Python code for it?

df = df.sort_index(axis=0, ascending=True)
df = df.iloc[::-1]
df.head(3)

df.tail(3)


I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
Conclusion: This way of finding intersecting points is not acceptable for our case.
Can you write Python code for it?

#
# line segment intersection using vectors
# see Computer Graphics by F.S. Hill
#
# Stolen directly from http://www.cs.mun.ca/~rod/2500/notes/numpy-arrays/numpy-arrays.html
#
from numpy import *
def perp(a):
    b = empty_like(a)
    b[0] = -a[1]
    b[1] = a[0]
    return b

# line segment a given by endpoints a1, a2
# line segment b given by endpoints b1, b2
# return
def seg_intersect(a1, a2, b1, b2) :
    da = a2-a1
    db = b2-b1
    dp = a1-b1
    dap = perp(da)
    denom = dot( dap, db)
    num = dot( dap, dp )
    return (num / denom.astype(float))*db + b1

def get_intersection_point_between_points(a1, a2, b1, b2):
    intersection_point = seg_intersect(a1, a2, b1, b2)
    intersection_point_x = intersection_point[0]

    if intersection_point_x >= max([a1[0], b1[0]]) and intersection_point_x <= min([a2[0], b2[0]]):
        return intersection_point
    else:
        return [nan, nan]

def get_intersection_points(x_values, y_values_1, y_values_2):
    intersection_points = []
    for x in x_values[:-1]:
        point = get_intersection_point_between_points(array([x, y_values_1[x]]),     \
                                                      array([x+1, y_values_1[x+1]]), \
                                                      array([x, y_values_2[x]]),     \
                                                      array([x+1, y_values_2[x+1]]))
        if not math.isnan(point[0]):
            intersection_points.append(point)
    return intersection_points

def get_buy_sell_points(x_values, y_values_1, y_values_2):
    intersection_points = get_intersection_points(x_values, y_values_1, y_values_2)

    buy_points = []
    sell_points = []

    for i in intersection_points:
        original_x = i[0]
        next_business_day = int(original_x) + 1

        # y_values_1 represents values of 'shorter' moving average
        # y_values_2 represents values of 'longer' moving average
        if y_values_1[next_business_day] < y_values_2[next_business_day]:
      ...


I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
Determine exchange rates
INFORMATION: Exchange rates are determined for "a next business day", so it's first available day once we are know that the crossing took place
Can you write Python code for it?

#print("Exact intersection points between SMA-{} and SMA-{}".format(comparison_pair[0], comparison_pair[1]))
#print(intersection_points)
#print()
#print("Intersection points between SMA-{} and SMA-{} changed to indicate next business day".format(comparison_pair[0], \
#                                                                                comparison_pair[1]))
#for i in intersection_points:
#    original_x = i[0]
#    next_business_day = int(original_x) + 1
#    print("{} ==> date: {}, exchange rate: {}".format(next_business_day, \
#                                                      recent_dates[next_business_day], \
#                                                      recent_exchange_rates[next_business_day]))
#print()


I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
Example on how to find points where the curves intersect
Can you write Python code for it?

x = np.arange(0, 1000)
f = np.arange(0, 1000)
g = np.sin(np.arange(0, 10, 0.01)) * 1000

plt.plot(x, f, '-')
plt.plot(x, g, '-')

idx = np.argwhere(np.isclose(f, g, atol=10)).reshape(-1)
plt.plot(x[idx], f[idx], 'ro')
plt.show()


I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
Now we have better algorithm of finding intersection points, so it's time redraw
Can you write Python code for it?

def plot_pair_v2(pair):
    window1 = pair[0]
    window2 = pair[1]
    # get SMA values for interesting windows
    sma1 = df_sma[window1][-last_days_to_observe:-1].values
    sma2 = df_sma[window2][-last_days_to_observe:-1].values

    # make extra calculations that simplify process of finding crossing points
    sma_1_flattened = [item for sublist in sma1 for item in sublist]
    sma_2_flattened = [item for sublist in sma2 for item in sublist]
    x_values = range(0, len(sma_1_flattened), 1)

    # calculate crossing points between both value ranges
    #intersection_points = get_intersection_points(x_values, sma_1_flattened, sma_2_flattened)

    # calculate buy, sell points for both value ranges
    (buy_points, sell_points) = get_buy_sell_points(x_values, sma_1_flattened, sma_2_flattened)

    # plot
    plt.xticks(np.arange(0, last_days_to_observe, 10))

    plt.plot(sma1)
    plt.plot(sma2)
    for buy in buy_points:
        plt.plot(buy[0], buy[1], 'go')

    for sell in sell_points:
        plt.plot(sell[0], sell[1], 'ro')

    # return all found buy, sell points
    return (buy_points, sell_points)

# set up plotting parameters
plt.figure(figsize=(20,7))
plt.grid(True)

plt.plot(recent_exchange_rates)
comparison_pair = (3, 10)
(buy_points, sell_points) = plot_pair_v2(comparison_pair)
plt.legend(['exchange rates', "SMA-{}".format(comparison_pair[0]), "SMA-{}".format(comparison_pair[1])], loc='upper left')