I am working on the file "Notebooks/CDAE.ipynb".
The first task is:
Organization of the data on the latent space
Here we encode all the test set data and plot the corresponding 2D values. The color will repsent respective number.
Can you write Python code for it?

#Get uniform samples over the labels
spl = 800  # sample_per_label
data = input_data.read_data_sets(data_dir, one_hot=False, reshape=False)
Xdemo, Xdemo_labels = get_demo_data(data, spl)
Zdemo = np.random.normal(size=[spl * 10, latentD], loc=0.0, scale=1.).astype(np.float32)

decoded_data = demo_sess.run(Xenc_op, feed_dict={Xph:Xdemo, is_training:False})
plot_latent_variable(decoded_data, Xdemo_labels)


I am working on the file "Notebooks/CVE_Details/cve_details_introduction.ipynb".
The first task is:
A pattern emerges in the construction of the types: For vulnerability types with a higher number of cwe entries, this higher number is led by a single cwe id. This is the case for 3 vulnerability types in the table above: Dir.Trav. being led by cwe_id 22, Exec Code Sql being led by cwe_id 89, and vulnerability type SQL on cwe_id 89.
Additionally, we should remember that the column vulnerability type can be a combination of 1 or more types of the 13 proposed by CVE Details. Specifically, Exec Code Sql actually refers to the type Code Execution and the type SQL.
We can also observe just the distribution of cwe_id's reported for SQL table for 2016.
Can you write Python code for it?

vulnerability_histogram = cved_df.groupby(by=['cwe_id'])['cve_id'].count()
vulnerability_histogram


I am working on the file "Notebooks/CVE_Details/cve_details_introduction.ipynb".
The first task is:
Notice that the column vulnerability_type not only includes sql injection. It may also cite other identified types out of the 13. For instance, row 4 value is SQL XSS, indicating the entry is both of SQL and XSS types.
Important: Entries labeled with multiple types WILL appear, accordingly, on the tables. A combination of tables must be done carefully when collecting textual descriptions, to avoid redundance!
Analysis
To begin the analysis, let's consider how cwe ids, which by definition group vulnerabilities, are distributed across the type for this year and SQL Injection. Notice a cwe_id may or not occur, but the type is always defined. cve_id's, always occur, hence defining the upper bound a vulnerability type could had.
Can you write Python code for it?


I am working on the file "Notebooks/CVE_Details/cve_details_introduction.ipynb".
The first task is:
The histogram below represents vulnerability types as mentioned in the Cve Details database. The vulnerability types are explained below:

Dir. Trav. stands for Directory Traversal
Dir. Trav. Bypass stands for Directory Traversal Bypass
Dir. Trav. File Inclusion stands for Directory Traversal File Inclusion
DoS Sql stands for Denial of Service attack using Structured Query Language
Exec Code Dir. Trav. stands for Execution of arbitrary code causing a Directory Traversal vulnerability
Exec code Sql stands for Execution of arbitrary code causing a Structured Query Language vulnerability
Exec Code Sql +Info stands for Execution of arbitrary code causing a Structured Query Language vulnerability or access to sensitive information
Sql stands for Structured Query language vulnerability
Sql +Info stands for a Structured Query Language vulnerability or access to sensitive information
Sql Bypass +Info stands for a Structured Query Language Bypass vulnerability or access to sensitive information
Sql XSS stands for Cross-site scripting(XSS) vulnerability in SQL
Can you write Python code for it?


I am working on the file "Notebooks/CVE_Details/cve_details_introduction.ipynb".
The first task is:
We can note some combinations of type occur much more frequently than others.
Let's explore further the vulnerability types proposed by CVE Details, by not only counting the number of cwe id's per vulnerability type, but also what are the cwe_id'per type. Out of curiosity, let's also include the number of exploits that were reported.
Can you write Python code for it?

vulnerability_type_histogram = cved_df.groupby(by=['vulnerability_type','cwe_id'])['cve_id','n_exploits'].count()
print(vulnerability_type_histogram)
vulnerability_list = np.unique(cved_df['vulnerability_type'])
vulnerability_by_month = cved_df.groupby(by=['vulnerability_type','month'])['cve_id'].count()


I am working on the file "Notebooks/CVE_Details/cve_details_introduction.ipynb".
The first task is:
We created this histogram to gain insights into the the number of occurances of each of the vulnerability types. On analysis we can see that Exec. code Sql is the most frequent type of attack, followed by Dir. Trav. and Sql types which qualify to the three most frequent vulnerability types.
CWE ID count histogram
Can you write Python code for it?

#creating a histogram for cwe ID types by creating a dictionary
data = {}
data['Entries'] = vulnerability_histogram
#saving in dictionary for sorting and visualising
df_data = pd.DataFrame(data).sort_values(by='Entries', ascending=True)
series = df_data.loc[:,'Entries']

p = figure(width=800, y_range=series.index.tolist(), title="Histogram of CWE IDs from CVE Details")

p.xaxis.axis_label = 'Frequency'
p.xaxis.axis_label_text_font_size = '10pt'
p.xaxis.major_label_text_font_size = '8pt'

p.yaxis.axis_label = 'CWE ID'
p.yaxis.axis_label_text_font_size = '10pt'
p.yaxis.major_label_text_font_size = '8pt'

j = 1
for k,v in series.iteritems():

  #Print fields, values, orders
  #print (k,v,j)
  p.rect(x=v/2, y=j, width=abs(v), height=0.4,
    width_units="data", height_units="data")
  j += 1
show(p)


I am working on the file "Notebooks/CWE/Fielder_Parser/CWE Field Parser.ipynb".
The first task is:
0. Preparing Data
Before digging into the parser notebook, the version of the CWE xml file within this notebook is v3.0, which can be downloaded from this link. Here we loaded CWE v3.0 xml file. Therefore, if there is any new version of XML raw file, please make change for the following code. If the order of weakness table is changed, please change the code for function <b>extract_target_field_elements</b> in section 2.1.
Can you write Python code for it?


I am working on the file "Notebooks/CWE/Fielder_Parser/CWE Field Parser.ipynb".
The first task is:
1. Introduction
The purpose of this notebook is to build the fields parser and extract the contents from various fields in the CWE 3.0 XML file so that the field content can be directly analyzed and stored into database. Guided by CWE Introduction notebook, this notebook will focus on the detail structure under Weakness table and how parser functions work within the weakness table.
To preserve the semantic information and not lose details during the transformation from the representation on website and XML file to the output file, we build a 3-step pipeline to modularize the parser functions for various fields in different semantic format. The 3-step pipeline contains the following steps: searching XML Field node location, XML field node parser, and exporting the data structure to the output file based on the semantic format in Section 4 of CWE Introduction Notebook. More details will be explained in Section 2.
2. Parser Architecture
The overall parser architecture is constituted by the following three procedures: 1) extracting the nodes with the target field tag, 2) parsing the target field node to the representation in memory, and 3) exporting the data structure to the output file.
Section 2.1 explains the way to search XML field nodes with the target field tag. No matter parsing which field, the first step is to use Xpath and then locate all XML field nodes with the field tag we are intended to parse. The function in section 2.1 has been tested for all fields and thus can locate XML nodes with any given field naming. However, the function is inconsistent for different versions, since the order of weakness table might be different. It happens between v2.9 and v3.0.
Section 2.2 explains the way to parse and extract the content of the target field into the representation in memory. Since different fields have various nested structures in xml raw file and the content we will parse varies field by field, the worst situation is that there will be one parser function for each different field. However, from...
Can you write Python code for it?

def extract_target_field_elements(target_field, cwe_xml_file):
    '''
    This function responsibility is to abstract how nodes are found given their field name and should be used together with the histogram.

    Args:
        - target_field: the arg defines which nodes are found given by the field that we are aiming to target
        - cwe_xml_file: the CWE xml file that this function will work and extract the target field nodes
    Outcome:
        - a list of nodes that have the pre-defined target field as the element tag
    '''
    # read xml file and store as the root element in memory
    tree = lxml.etree.parse(cwe_xml_file)
    root = tree.getroot()

    # Remove namespaces from XML.
    for elem in root.getiterator():
        if not hasattr(elem.tag, 'find'): continue  # (1)
        i = elem.tag.find('}') # Counts the number of characters up to the '}' at the end of the XML namespace within the XML tag
        if i >= 0:
            elem.tag = elem.tag[i+1:] # Starts the tag a character after the '}'

    # define the path of target field. Here we select all element nodes that the tag is the target field
    target_field_path='Weakness/./'+target_field
    # extract weakness table in the XML // if the order of weakness table is changed, please make change for the following code
    weakness_table = root[0]

    # generate all elements with the target field name
    target_field_nodes=weakness_table.findall(target_field_path)
    return target_field_nodes


I am working on the file "Notebooks/CWE/Fielder_Parser/CWE Field Parser.ipynb".
The first task is:
2.2 XML Node Field Parser
Once the node is provided by the former function, its XML structure is consistent for that field, and cwe version, but it is inconsistent for different versions. For example, a table represented in XML is different than a paragraph in XML. However, even for the same expected structure such as a table, the XML may have different tags used within it.
The associated parser function then is tested to cover all possible cases of this field in the XML, and also interpreted against its .html format to understand its purpose. We again refer to the introductory notebook Sections 4 and 5 which convey respectively the potential usage of the field for PERCEIVE and the overall format of the structure when compared to others.
The purpose is then documented as part of the functions documentation (sub-sections of this Section), conveying the rationale behind what was deemed potentially useful in the content to be kept, as well as how certain tags were mapped to a data structure while being removed.
The parser function outputs one of the known data structures (i.e. memory representation) that is shared among the different fields of what is deemed relevant. For example, while 5 field nodes may have their relevant information in different ways, they may all be at the end of the day tables, named lists, or single blocks of texts. Sharing a common representation in this stage decouples the 3rd step of the pipeline from understanding the many ways the same information is stored in the XML.
Because different CWE versions may organize a .html table, bullet list etc in different ways even for the same field, this organization also decouples the previous and following section functions from being modified on every new version if necessary.
The following fields have parser functions as of today:
|Field Name| Function Name|
|:---:|:----:|
|Potential_Mitigations|parse_potential_mitigations|
|Common_Consequences|parse_common_consequences|
2.2.1 Parse Potential_Mitigations
Potential_Mitigations field has a...
Can you write Python code for it?

def parse_potential_mitigations(potential_mitigation_node):
    '''
    The parser function concern is abstracting how the Potential_Mitigations field is stored in XML,
    and provide it in a common and simpler data structure

    Args:
        - potential_mitigations_node: the node that has Potential_Mitigations tag, such as the above image

    Outcomes:
        - A dictionary that pairs cwe_id as key and the mitigation list as value.
          In the dictionary, the mitigation list will be a list of dictionaries that each dictionary pairs tag and the corresponding content for each mitigation.
          More details can be found in the following example for cwe-1022
    '''
    # extract cwe_id from the attribute of potential_mitigations element's parent node
    cwe_id=potential_mitigations_node.getparent().attrib.get('ID')
    cwe_id='CWE_'+cwe_id
    # the mitigation list that each element represents an indivudual mitigation element
    mitigation_list=[]
    target_field=potential_mitigations_node.tag

    # for each mitigation node under the potential_mitigations node
    for mitigation in list(potential_mitigations_node):

        # the dictionary that contain the information for each mitigation element
        mitigation_dict=dict()

        # traverse all mitigation_element nodes under each mitigation node
        for mitigation_element in list(mitigation):
            # generate tag and content of each mitigation_element
            mitigation_element_tag=mitigation_element.tag.lower()
            mitigation_element_content=mitigation_element.text

            ## in case there is nested elements under mitigation_element but store the content from a same tag
            # check whether there is an element under mitigation_element
            if mitigation_element_content.isspace():
                entry_element_content=''

                # iterate all child elements below mitigation_element,
                for mitigation_element_child in...


I am working on the file "Notebooks/CWE/Fielder_Parser/CWE Field Parser.ipynb".
The first task is:
4. Main Execution
After developing the 3 steps parsing pipeline, this section will combine these 3 steps and produce the output file for different fields. As introduced in Section 2, although the parsing procedure keeps same for all fields, each field will have own parsing function, while the same format of fields may share a same exporting function. As a result, the main execution varies for each field.
4.1 Main execution for Potential_Mitigations
The main execution will combine the above 3 steps parsing pipeline for Potential_Mitigations. After developing function <b>export_data</b>, the following code should produce the output file that contains the parsed content of Potential_Mitigations for all CWE_id.
Can you write Python code for it?

if __name__ == "__main__":
    # extract the nodes, whose tag is Potential_Mitigations,from cwe_xml_file
    potential_mitigations_nodes=extract_target_field_elements('Potential_Mitigations',cwe_xml_file)
    # read each Potential_Mitigation node
    for potential_mitigations_node in potential_mitigations_nodes:
        # parse the content for each potential_mitigation node
        potential_mitigations_info=parse_potential_mitigations(potential_mitigations_node)
        # export the parsed content TO-DO
        export_data(potential_mitigations_info)


I am working on the file "Notebooks/CWE/Fielder_Parser/CWE Field Parser.ipynb".
The first task is:
Through function  parse_common_consequences, the above Common_Consequences node for cwe_103 will be parsed into the following data format in memory.

 How content represent in memory (cwe-103)

3. Export Data Structure
At the point this notebook is being created, it is still an open end question on how will tables, bullet lists and other potential structures in CWE will be used for topic modeling. For example, should rows in a table be concatenated into a paragraph and used as a document? What if the same word is repeated as to indicate a lifecycle?
In order to keep this notebook future-proof, this section abstracts how we will handle each field representation (e.g. table, bullet list, etc) from the memory data structure. It also keeps it flexible for multi-purpose: A table may be parsed for content for topic modeling, but also for extracting graph relationships (e.g. the Related Attack Pattern and Related Weaknesses fields contain hyperlinks to other CWE entries which could be reshaped as a graph).
Can you write Python code for it?

def export_data(target_field_node):
    '''This section code will be done in the future.'''
    pass


I am working on the file "Notebooks/CWE/Fielder_Parser/CWE Field Parser.ipynb".
The first task is:
Through function  parse_potential_mitigations, the above Potential_Mitigations node for cwe_1022 will be parsed into the following data format in memory.

 How content represent in memory (cwe-1022)

2.2.2 Parse Common_Consequences
Common_Consequences field has a nested structure under the field element.
To understand the nesting structure, here we use the Common_Consequences field in cwe-103 as example. Under Common_Consequences element, there are two field entries named by 'Consequence', which represent two different consequences associated with the weakness. Under each consequence element, three entry elements constitute one weakness consequence, including scope, impact, and note, which have the contents that our parser is intended to to extract.
To preserve the table format of Common_Consequences, we use the dictionary to pair the CWE id and the content we parse from Common_Consequences field. Since there are multiple consequences for one weakness, a list of dictionaries will be used to store the content, where the number of dictionaries is equal to the number of consequences. Since one consequence may have multiple impacts and scopes but only one note, we use tuple to store the content of impact and scope, while directly store the content of note. In summary, the data structure in memory can be represented as the following format: {CWE_id: [{'Scope':scope tuple, 'Impact':impact tuple, 'Note': Text}, {'Scope':scope tuple, 'Impact':impact tuple, 'Note': Text}...]}. More details can be found in the example of cwe-103.
Since the number of element varies depending on the field, here is the cardinality of these fields:
|Tag|Cardinality|
|:---:|:---:|
|Scope|1 or more|
|Impact|1 or more|
|Note|0 or 1|

How content represent on the website (cwe-103)

How content represent in the xml file (cwe-103)
Can you write Python code for it?

def parse_common_consequences(common_consequences_node):
    '''
    The parser function concern is abstracting how the Common_Consequences field is stored in XML,
    and provide it in a common and simpler data structure

    Args:
        - common_consequences_node: the node that has Common_Consequences tag, such as the above image

    Outcomes:
        - A dictionary that pairs cwe_id as key and the consequence list as value.
          In the dictionary, the consequence list will be a list of dictionaries that each dictionary pairs tag and the corresponding content for each consequence.
          More details can be found in the following example for cwe-103.
    '''
    # extract cwe_id from the attribute of common_consequences element's parent node
    cwe_id=common_consequences_node.getparent().attrib.get('ID')
    cwe_id='CWE_'+cwe_id
    # the consequence list that each element represents an indivudual consequence element
    consequence_list=[]
    target_field=common_consequences_node.tag

    # for each consequence node under the common_consequence node
    for consequence in list(common_consequences_node):

        # the dictionary that contain the information for each consequence element
        consequence_dict=dict()

        # traverse all consequence_element nodes under each consequence node
        for consequence_element in list(consequence):
            # generate tag and content of each consequence_element
            consequence_element_tag=consequence_element.tag.lower()
            consequence_element_content=consequence_element.text.strip()

            # parse the note content directly as the value
            if consequence_element_tag=='note':
                consequence_dict[consequence_element_tag]=consequence_element_content
            # for scope and impact, parse the content for scope and impact as tuple
            else:
                # if the tag is already in the dictionary, add the content to the existing tuple
                if...


I am working on the file "Notebooks/CWE/Fielder_Parser/Legacy CWE Field Parser.ipynb".
The first task is:
After running the above codes, the file named by 'Description.csv' should be created under the same directory as this notebook. For parsing other fields, need to change the name of the target field.
Can you write Python code for it?


I am working on the file "Notebooks/CWE/Fielder_Parser/Legacy CWE Field Parser.ipynb".
The first task is:
Format and Field Parser
Although there are various kinds of field, in general, there are only three ways to store the field information in the CWE XML file: 1) fields with no nesting element, 2) fields with nesting element, 3) fields with attribute information.
|Format|CWE Field Example|
|:----:|:---------:|
|Fields with no nesting element|Description, Extended_Description, Likelihood_Of_Exploit, Background_Details|
|Fields with nesting element|Potential_Mitigations, Weakness_Ordinalities, Common_Consequences, Alternate_Terms, Modes_Of_Introduction, Affected_Resources, Observed_Examples, Functional_Areas, Content_History, Detection_Methods|
|Fields with attribute information|Demonstrative_Exampls, Taxonomy_Mappings, Applicable_Platforms, References,Related Attack Pattern|
We will discuss the detail structure and how to parse the first two types of field below.
1.1 Fields with no nesting element
Typically, the fields in this format will keep of the information directly under the field element, without any nesting structure and attribute. For example, Description and Extended_Description are the fields in this format. There is no further nesting structure under the field element  and thus cannot be extended (no plus sign on the left)
However, when parsing Extended_Description in cwe-1007, there are nesting html elements under Extended_Description element. In this case, we will remove the html tag and concatenate the contents under separate html elements
General case:

HTML elements under Extended_Description:

1.2 Parser function for field with no nesting element
Before introducing the parser function, we need a function that can write the dictionary that stores the field content to a CSV file. Function  write_dict_to_csv  will append the given dictionary to the end of the CSV file. If the file does not exist, the function will create a CSV file and take the csv_header as the header of this CSV file.
Can you write Python code for it?

def write_dict_to_csv(output_file,csv_header,dict_data):
    '''
    Create a CSV file with headers and write a dictionary;
    If the file already existes, only append a dictionary.

    Args:
        output_file -- name of the output csv file
        csv_header -- the header of the output csv file.
        dict_data -- the dictionary that will be writen into the CSV file. The number of
                     element in the dictionary should be equal to or lower than the number of
                     headers of the CSV file.

    Outcome:
        a new csv file with headers and one row that includes the information from the dictionary;
        or an existing CSV file with a new row that includes the information from the dictionary
    '''
    # create a file if the file does not exist; if exsits, open the file
    with open(output_file, 'a') as csv_file:
        writer = csv.DictWriter(csv_file, fieldnames=csv_header,lineterminator='\n')

        # check whether the csv file is empty
        if csv_file.tell()==0:
            # if empty, write header and the dictionary
            writer.writeheader()
            writer.writerow(dict_data)
        else:
            # if not empty, only write the dictionary
            writer.writerow(dict_data)


I am working on the file "Notebooks/CWE/Fielder_Parser/Legacy CWE Field Parser.ipynb".
The first task is:
Given the target field, function <b> no_nesting_field_parser </b> will extract the contents within the target field element and write cwe_id and content into a CSV file named by the target field. Each row in the output CSV file will include the following information:
- cwe_id: The CWE identifier
- field: The name of the target field
- (field name)_content:  The text information stored under the target field. The header of this column varies depending on the field. For example, the header will be 'description_content' if parsing 'Description' field
The following fields have been tested successfully: Description, Extended_Description, Likelihood_Of_Exploit, Background_Details.
Can you write Python code for it?

def no_nesting_field_parser(target_field, root):
    '''
    Parse the field with no nesting element from cwec_v3.0.xml file and output the information to a csv file.

    Args:
        target_field -- the target field that will be parsed through this function. The format of this arg should be string.
        root -- the root element of the whole parsed tree.
    Outcome:
        a csv file named by the field name. Each row will include the following information:
            - cwe_id: The CWE identifier
            - field: The name of the target field
            - (field name)_content:  The text information stored under the target field. The header varies depending on field.
                                     For example, the header will be 'description_content' if parsing 'Description' field
    '''
    # define the path of target field. Here we select all element nodes that the tag is the target field
    target_field_path='Weakness/./'+target_field
    # extract weakness table in the XML
    weakness_table = root[0]
    #define the headers
    field_header=target_field.lower()+'_content'
    output_header=['cwe_id','field',field_header]
    #define path of the output file
    output_path=target_field+'.csv'

    # for each target field node
    for field in weakness_table.findall(target_field_path):
        # extract cwe_id from the parent node of the target field node
        cwe_id=field.getparent().attrib.get('ID')
        # extract the content under the target field
        field_entry_content=field.text

        # in case there are nested html tags under the field
        if field_entry_content.isspace()==True:
            for field_entry in field:
                # extract the content under html tags and concatenate
                field_entry_content=field_entry.text
                field_entry_content=field_entry_content+field_entry

        # build the dictionary that is used to write
        field_entry_dict=dict()
        field_entry_dict['cwe_id']=cwe_id
 ...


I am working on the file "Notebooks/CWE/Fielder_Parser/Legacy CWE Field Parser.ipynb".
The first task is:
Introduction
The purpose of this notebook is to build a field parser and extract the contents of various fields in the CWE 3.0 XML file so that the field content can be directly analyzed and stored into database. The raw XML file can be downloaded at http://cwe.mitre.org/data/xml/cwec_v3.0.xml.zip.  Guided by CWE Introduction notebook, this notebook will focus on the detail structure under Weakness table and how parser functions work in order to extract two formats of field: fields with no nesting element and fields with nesting structure.
Although the overall structure of CWE XML file has been documented in CWE Introduction notebook, the Introduction notebook is built on version 2.9. Therefore, the following differences about weakness table between version 2.9 and 3.0 can be observed:

The order of four tables is changed and weakness table in version 3.0 is the first.
Several fields are removed or changed to other names in version 3.0: Time_of_Introduction, Maintenace_Notes, Causal_Nature, Research_Gaps, White_Box_Definitions, Terminology_Notes, Other_Notes, Enabling_Factors_for_Exploitation,  Relevant_Properties
Can you write Python code for it?

tree = lxml.etree.parse('cwec_v3.0.xml')
root = tree.getroot()

# Remove namespaces from XML.
for elem in root.getiterator():
    if not hasattr(elem.tag, 'find'): continue  # (1)
    i = elem.tag.find('}') # Counts the number of characters up to the '}' at the end of the XML namespace within the XML tag
    if i >= 0:
        elem.tag = elem.tag[i+1:] # Starts the tag a character after the '}'

for table in root:
    print (table.tag)


I am working on the file "Notebooks/Calculate NERC results.ipynb".
The first task is:
Combine generation
Can you write Python code for it?

total_gen = gen_fuels_nerc.copy()

total_gen.loc[idx[2016:, :, :, :]] = (total_gen.loc[2016:]
                                     .add(extra_nerc.loc[:, 'generation (mwh)']
                                          , fill_value=0))
total_gen = total_gen.reset_index()
add_datetime(total_gen)

final = group_fuel_cats(total_gen,
                        custom_fuel_cat, 'type',
                        'fuel category', extra_group_cols=['nerc', 'datetime'])
final.set_index(['nerc', 'fuel category', 'datetime'], inplace=True)

total = final.groupby(['nerc', 'datetime']).sum()

nercs = total.index.get_level_values('nerc').unique()


I am working on the file "Notebooks/Calculate NERC results.ipynb".
The first task is:
Load emission factors
Can you write Python code for it?

path = join(cwd, '..', 'Data storage', 'Final emission factors.csv')
ef = pd.read_csv(path, index_col=0)

ef_type = reduce_emission_factors(ef)

ef_type = pd.Series(ef_type, name='type')


I am working on the file "Notebooks/Calculate NERC results.ipynb".
The first task is:
Monthly CO₂ emissions
Can you write Python code for it?

co2 = add_facility_location(co2, facility_labels,
                            labels=['lat', 'lon', 'state', 'nerc', 'year'])

co2_nerc = co2.groupby(['year', 'nerc', 'month'])['final co2 (kg)'].sum()


I am working on the file "Notebooks/Calculate NERC results.ipynb".
The first task is:
Percent generation by fuel
Can you write Python code for it?

df_list = []
for nerc in nercs:
    percent_gen = final.loc[nerc].divide(total.loc[nerc], level='datetime')
    percent_gen['nerc'] = nerc
    percent_gen.set_index('nerc', append=True, inplace=True)
    df_list.append(percent_gen)

percent_gen = pd.concat(df_list)
percent_gen.drop(['year', 'month'], axis=1, inplace=True)
percent_gen.columns = ['% generation']

path = join(cwd, '..', 'Data storage', 'Final NERC data',
            'NERC percent gen {}.csv'.format(file_date))
percent_gen.to_csv(path)

total_monthly_gen = final.groupby(['fuel category', 'year', 'nerc', 'month']).sum()
total_monthly_gen.sort_index(inplace=True)

path = join(cwd, '..', 'Data storage', 'Final NERC data',
            'NERC generation {}.csv'.format(file_date))
total_monthly_gen.to_csv(path)


I am working on the file "Notebooks/Calculate NERC results.ipynb".
The first task is:
Write data to file
Can you write Python code for it?

path = join(cwd, '..', 'Data storage', 'Final NERC data',
            'NERC gen emissions and index {}.csv'.format(file_date))
nerc_index.to_csv(path, index=False)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Add NERC region labels
Can you write Python code for it?

%aimport Analysis.state2nerc
from Analysis.state2nerc import fraction_state2nerc, add_region

cwd = os.getcwd()
path = join(data_path, 'Facility labels',
            'Facility locations_RF.csv')
location_labels = pd.read_csv(path)

nerc_state_path = join(data_path, 'Derived data',
                  'NERC_states.json')

with open(nerc_state_path, 'r') as f:
    nerc_states = json.load(f)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Allocate extra gen from the state-level to regions
I still need to generate state-level total generation and fuel use!
Can you write Python code for it?

idx = pd.IndexSlice

# a dictionary to match column names
nerc_frac_match = {'% generation': 'generation (mwh)',
                   '% total fuel': 'total fuel (mmbtu)',
                   '% elec fuel': 'elec fuel (mmbtu)'}


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Calculate the extra gen/fuel consumption at the state levels
Only worrying about extra generation from 2016 forward.
Can you write Python code for it?

state_extra = (state_total.loc[idx[:, 2016:, :, :], :]
               - eia_fac_type.loc[idx[:, 2016:, :, :], :])
state_extra.dropna(how='all', inplace=True)
state_extra = state_extra.reorder_levels(['year', 'state', 'month', 'type'])
state_extra.sort_index(inplace=True)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Combine the lists of annual plants in 2015 and 2017. This lets us catch facilities that have gone from monthly to annual since 2015, but it also includes plants that were annual in 2015 an may have retired.
There is the possibility of an error in allocation to NERC regions for 2016 state-level generation when using the facilities that changed to annual in 2017. But since the state-level generation is so much smaller in 2016 I don't think this is much of an issue.
Can you write Python code for it?

annual_ids = set(annual_ids_2015.tolist() + annual_ids_2017.tolist())

len(annual_ids)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Create a copy of the nerc_fraction dataframe with repeated values for every month of the year, so that they MultiIndex matches the state_extra MultiIndex
Can you write Python code for it?

df_list = []
for month in range(1, 13):
    df = nerc_fraction.copy()
    df['month'] = month
    df.set_index('month', append=True, inplace=True)
    df_list.append(df)

nerc_frac_monthly = pd.concat(df_list, axis=0)
nerc_frac_monthly.sort_index(inplace=True)
nerc_frac_monthly = (nerc_frac_monthly
                     .reorder_levels(['nerc', 'state', 'month', 'type']))


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Facility state and lat/lon file generation
~~Create a .csv with lat/lon and state code for each facility. Need to manually add the NERC region label.~~ ~~I've done this with a spatial join between the lat/lon and NERC shapefiles in QGIS. It should be possible to do the spatial join in GeoPandas.  Could also add custom region labels (e.g. eGRID subregions, ISO/RTO boundaries, etc).~~
This isn't possible with a spatial join because NERC regions aren't strictly based on geography. Instead, I've taken the NERC codes from EIA-860 and assigned unknown plants (mostly those that retired before 2012 when modern NERCs were mostly defined, and those that were assigned a plant id in 2017) NERC labels using a k-nearest neighbors algorithm and lat/lon information.
See the Assign NERC region labels notebook for the creation of this file.
Fraction of estimated gen/fuels in each NERC region
NOTE EIA changed the reporting requirements for wind and solar facilities in 2017. I'm now going to use the list of annual facilities from 2017 rather than from 2015, but still use the reported generation by those facilities in 2015.
Can you write Python code for it?

%aimport Data.make_data
from Data.make_data import get_annual_plants

annual_ids_2015 = get_annual_plants(2015)

annual_ids_2017 = get_annual_plants(2017)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Group the facility data to state fuel categories and add state labels
Can you write Python code for it?

cols = list(nerc_frac_match.values())
eia_fac_type = group_fuel_cats(eia_fac, state_fuel_cat)
eia_fac_type = add_facility_location(eia_fac_type, location_labels, ['state', 'year'])
eia_fac_type = eia_fac_type.groupby(['state', 'year', 'month', 'type'])[cols].sum()


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Load data
Can you write Python code for it?

cwd = os.getcwd()
path = join(data_path, 'Derived data',
            'Facility gen fuels and CO2 {}.csv'.format(file_date))
eia_fac = pd.read_csv(path)

rename_cols(eia_fac)

path = join(data_path, 'Derived data',
            'Monthly EPA emissions {}.csv'.format(file_date))
epa = pd.read_csv(path)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
National Index and gen by fuels
Total national generation (all fuels)
Add generation/fuel consumption reported by facilities (with state-level fuel codes) and extra generation/fuel consumption (which also uses state-level fuel codes).
Can you write Python code for it?

gen_fuels_state.groupby(['type', 'year', 'month']).sum().head()

national_gen = (gen_fuels_state
                .groupby(['type', 'year', 'month'])['generation (mwh)'].sum()
                .add(extra_gen_fuel['generation (mwh)'], fill_value=0))


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Percent generation by custom fuel type
Can you write Python code for it?

df_list = []
for fuel in national_gen.index.get_level_values('fuel category').unique():
    percent_gen = national_gen.loc[fuel].divide(total_gen, fill_value=0)
    percent_gen['fuel category'] = fuel
    percent_gen.set_index('fuel category', inplace=True, append=True)
    df_list.append(percent_gen)
percent_gen = pd.concat(df_list)

path = join(data_path, 'National data',
            'National generation {}.csv'.format(file_date))
national_gen.to_csv(path)

path = join(data_path, 'National data',
            'National percent gen {}.csv'.format(file_date))
percent_gen.to_csv(path)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Regroup generation from state codes to my custom fuel codes
Can you write Python code for it?

national_gen = group_fuel_cats(national_gen.reset_index(), custom_fuel_cat,
                               'type', 'fuel category').set_index(['fuel category', 'year', 'month'])

total_gen = national_gen.groupby(['year', 'month']).sum()


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Simplify the dataframe
Can you write Python code for it?

cols = list(nerc_frac_match.values())
state_total = state_total.groupby(['state', 'year', 'month', 'type'])[cols].sum()

# list of NERC regions
nercs = nerc_fraction.index.get_level_values('nerc').unique()


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Sort the index of each dataframe to make sure they can be easily combined.
Can you write Python code for it?


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
This is 2015 data on annual reporting facilities (from both 2015 and 2017)
Can you write Python code for it?

# Get a list of all state abbreviations

all_states = []
for value in nerc_states.values():
    all_states.extend(value)
all_states = set(all_states)

df_list = []

for state in all_states:
    try:
        df_list.append(fraction_state2nerc(eia_2015_annual_nerc,
                       state, region_col='nerc', fuel_col='type'))
    except:
        print(state)
        pass

nerc_fraction = pd.concat(df_list)
nerc_fraction.set_index(['state', 'nerc', 'type'], inplace=True)
nerc_fraction.sort_index(inplace=True)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Total CO₂ (national)
Combine adjusted CO₂ at facilities and CO₂ from fuel consumption that is not captured by facility data.
Can you write Python code for it?

facility_co2 = co2.groupby(['year', 'month']).sum()

national_co2 = (facility_co2.loc[:, 'final co2 (kg)']
                + extra_co2.loc[:, 'elec fuel co2 (kg)']
                           .groupby(['year', 'month']).sum())
national_co2.name = 'final co2 (kg)'


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
Total EIA generation/fuel consumption and emission factors
Can you write Python code for it?

cwd = os.getcwd()
path = join(data_path, 'Derived data',
            'EIA country-wide gen fuel CO2 {}.csv'.format(file_date))
eia_total = pd.read_csv(path)

path = join(data_path,
            'Final emission factors.csv')
ef = pd.read_csv(path, index_col=0)


I am working on the file "Notebooks/Calculate national and NERC gen and emissions.ipynb".
The first task is:
With the values below I can allocate extra state-level generation and fuel use to each of the NERC regions!
Making sure that no values are greater than 1 (within tolerance)
Can you write Python code for it?

(nerc_fraction.groupby(['state', 'type']).sum() > 1.0001).any()


I am working on the file "Notebooks/Capacity.ipynb".
The first task is:
Clean up column names and only keep desired columns
Can you write Python code for it?

op.columns = op.columns.str.strip()
ret.columns = ret.columns.str.strip()

op_cols = [
    'Plant ID', 'Nameplate Capacity (MW)', 'Net Summer Capacity (MW)',
    'Energy Source Code', 'Prime Mover Code', 'op datetime'
]

ret_cols = [
    'Plant ID', 'Nameplate Capacity (MW)', 'Net Summer Capacity (MW)',
    'Energy Source Code', 'Prime Mover Code', 'Retirement Month',
    'Retirement Year', 'Operating Month', 'Operating Year',
    'op datetime', 'ret datetime'
]

op = op.loc[:, op_cols]
ret = ret.loc[:, ret_cols]

op.columns = op.columns.str.lower()
ret.columns = ret.columns.str.lower()

op.head()


I am working on the file "Notebooks/Capacity.ipynb".
The first task is:
Determine operable capacity in every month
Can you write Python code for it?

# Define iterables to loop over
years = range(2001,2018)
months = range(1,13)

fuels = list(custom_cats.keys())

# capacity_type is defined at the top of this notebook
op_df_capacity = monthly_capacity_all(op=op, ret=ret, years=years,
                                      nerc_plant_list=nerc_dict, fuels=fuels,
                                      cap_type=capacity_type, n_jobs=-1,
                                      print_year=False)

op_df_capacity.tail()

# Write data to file
out_path = data_path / 'Derived data' / 'Plant capacity' / 'monthly capacity by fuel.csv'
op_df_capacity.to_csv(out_path)


I am working on the file "Notebooks/Capacity.ipynb".
The first task is:
Load monthly EIA-860 data
Can you write Python code for it?

file_path = data_path / 'EIA downloads' / 'december_generator2017.xlsx'
op = pd.read_excel(file_path, sheet_name='Operating', skiprows=1, skip_footer=1,
                   parse_dates={'op datetime': [14, 15]},
                   na_values=' ')

# need to make some helper functions for the retired generators sheet
def bad_month_values(month):
    'Change value to 1 if outside 1-12'

    if month > 12 or month < 1:
        new_month = 1
    else:
        new_month = month
    return new_month

def make_dt_col(df, month_col, year_col):
    months = df[month_col].astype(str)
    years = df[year_col].astype(str)
    dt_string = years + '-' + months + '-' + '01'
    dt = pd.to_datetime(dt_string)
    return dt

ret = pd.read_excel(file_path, sheet_name='Retired', skiprows=1, skip_footer=1,
                    converters={'Operating Month': bad_month_values},
                    # parse_dates={'op datetime': [16, 17],
                    #              'ret datetime': [14, 15]},
                    na_values=' ')

ret['op datetime'] = make_dt_col(ret, 'Operating Month', 'Operating Year')
ret['ret datetime'] = make_dt_col(ret, 'Retirement Month', 'Retirement Year')


I am working on the file "Notebooks/Capacity.ipynb".
The first task is:
Need to make this into a dictionary of dictionaries of lists (year, nerc, plant ids)
Can you write Python code for it?

nerc_dict = {}
for year in facility_nerc.index.get_level_values('year').unique():
    nerc_dict[year] = {}

    for nerc in facility_nerc.index.get_level_values('nerc').unique():
        nerc_dict[year][nerc] = facility_nerc.loc[idx[nerc, year], 'plant id'].tolist()

# Make sure there aren't lots of plants in 2016 that disappear in 2017
set(nerc_dict[2016]['MRO']) - set(nerc_dict[2017]['MRO'])


I am working on the file "Notebooks/Capacity.ipynb".
The first task is:
Type of capacity
The default is net summer capacity, which is what EIA uses for their capacity factor calculations. eGRID uses nameplate capacity. Valid parameter values are:
- nameplate capacity (mw)
- net summer capacity (mw)
- net winter capacity (mw)
Can you write Python code for it?

capacity_type = 'net summer capacity (mw)'
# capacity_type = 'nameplate capacity (mw)'

%load_ext watermark
%watermark -iv -v

# Load the "autoreload" extension
%load_ext autoreload

# always reload modules marked with "%aimport"
%autoreload 1

# add the 'src' directory as one where we can import modules
cwd = os.getcwd()
src_dir = join(cwd, os.pardir, 'src')
sys.path.append(src_dir)

data_path = Path(cwd, '..', 'Data storage')

%aimport Analysis.index
from Analysis.index import group_fuel_cats

%aimport Analysis.capacity
from Analysis.capacity import monthly_capacity_all, monthly_capacity_year
from Analysis.capacity import monthly_ng_type_all, monthly_ng_type_year


I am working on the file "Notebooks/Chapter 12.ipynb".
The first task is:
Data
Using R, I executed lines 18-63 from the script OneOddGroupModelComp2E.R to generate the exact same data used in the book. The script can be downloaded from the book's website. After executing the lines, the List object dataList in R contains five elements:
 1. nCond: A scalar value (4) representing the number of conditions (background music types).
 2. nSubj: A scalar value (80) representing the number of subjects.
 3. CondOfSubj: A vector representing the condition (1, 2, 3 or 4) of a subject during a test.
 4. nTrlOfSubj: A vector with the number of trials/words per subject (20 for all subjects).
 5. nCorrOfSubj: A vector with number of correct recalls per subject.
I exported the last three elements of dataList to a csv file using the following command in R:
write.csv(data.frame(dataList[c(3:5)]), file='background_music.csv', row.names=FALSE)
Can you write Python code for it?

df = pd.read_csv('data/background_music.csv', dtype={'CondOfSubj':'category'})

# Mapping the condition descriptions to the condition codes. Just for illustrative purposes.
bgmusic = {0:'Das Kruschke', 1:'Mozart', 2:'Bach', 3:'Beethoven'}
df['CondText'] = df.CondOfSubj.cat.codes.map(bgmusic)

cond_idx = df.CondOfSubj.cat.codes.values
cond_codes = df.CondOfSubj.cat.categories
nCond = cond_codes.size

nSubj = df.index.size

df.info()

df.groupby('CondOfSubj').head(3)


I am working on the file "Notebooks/Chapter 12.ipynb".
The first task is:
Figure 12.5
Can you write Python code for it?

fig = plt.figure(figsize=(12,8))

# Define gridspec
gs = gridspec.GridSpec(3, 3)
ax1 = plt.subplot(gs[0,0])
ax2 = plt.subplot(gs[0,1])
ax3 = plt.subplot(gs[0,2])
ax4 = plt.subplot(gs[1,0])
ax5 = plt.subplot(gs[1,1])
ax6 = plt.subplot(gs[1,2])
ax7 = plt.subplot(gs[2,:])

# Group the first six axes in a list for easier access in loop below
axes = [ax1, ax2, ax3, ax4, ax5, ax6]
# Differences of posteriors to be displayed: omega x - omega y
x = [0,0,0,1,1,2]
y = [1,2,3,2,3,3]

# Plot histograms
for ax, a, b in zip(axes, x, y):
    diff = trace1['omega'][:,a]-trace1['omega'][:,b]
    pm.plot_posterior(diff, ref_val=0, point_estimate='mode', color=color, ax=ax)
    ax.set_xlabel('$\omega_{}$ - $\omega_{}$'.format(a,b), fontdict={'size':18})
    ax.xaxis.set_ticks([-.2, -.1, 0.0, 0.1, 0.2])

# Plot trace values of model index (0, 1)
ax7.plot(np.arange(1, len(trace1['m_idx'])+1),trace1['m_idx'], color=color, linewidth=4)
ax7.set_xlabel('Step in Markov chain', fontdict={'size':14})
ax7.set_ylabel('Model Index (0, 1)', fontdict={'size':14})
ax7.set_ylim(-0.05,1.05)

fig.tight_layout()


I am working on the file "Notebooks/Chapter 12.ipynb".
The first task is:
Note: in contrast to the R output in the book, the parameters in PyMC3 (like $\omega$ and model index) are indexed starting with 0.
Model 0 = condition specific $\omega_c$
Model 1 = same $\omega$ for all conditions
Can you write Python code for it?

with pm.Model() as model_1:
    # constants
    aP, bP = 1., 1.

    # Pseudo- and true priors for model 1.
    a0 = tt.as_tensor([.48*500, aP])
    b0 = tt.as_tensor([(1-.48)*500, bP])

    # True and pseudopriors for model 0
    a = tt.as_tensor(np.c_[np.tile(aP, 4), [(.40*125), (.50*125), (.51*125), (.52*125)]])
    b = tt.as_tensor(np.c_[np.tile(bP, 4), [(1-.40)*125, (1-.50)*125, (1-.51)*125, (1-.52)*125]])

    # Prior on model index [0,1]
    m_idx = pm.Categorical('m_idx', np.asarray([.5, .5]))

    # Priors on concentration parameters
    kappa_minus2 = pm.Gamma('kappa_minus2', 2.618, 0.0809, shape=nCond)
    kappa = pm.Deterministic('kappa', kappa_minus2 +2)

    # omega0
    omega0 = pm.Beta('omega0', a0[m_idx], b0[m_idx])

    # omega (condition specific)
    omega = pm.Beta('omega', a[:,m_idx], b[:,m_idx], shape=nCond)

    # Use condition specific omega when m_idx = 0, else omega0
    aBeta = pm.math.switch(pm.math.eq(m_idx, 0), omega * (kappa-2)+1, omega0 * (kappa-2)+1)
    bBeta = pm.math.switch(pm.math.eq(m_idx, 0), (1-omega) * (kappa-2)+1, (1-omega0) * (kappa-2)+1)

    # Theta
    theta = pm.Beta('theta', aBeta[cond_idx], bBeta[cond_idx], shape=nSubj)

    # Likelihood
    y = pm.Binomial('y', n=df.nTrlOfSubj.values, p=theta, observed=df.nCorrOfSubj)

pm.model_to_graphviz(model_1)

with model_1:
    trace1 = pm.sample(5000, target_accept=.95)

pm.traceplot(trace1);


I am working on the file "Notebooks/Chapter 16.ipynb".
The first task is:
16.1 - Estimating the mean and standard deviation of a normal distribution
Model (Kruschke, 2015)
Can you write Python code for it?

Image('images/fig16_2.png', width=300)

with pm.Model() as model:
    mu = pm.Normal('mu', y.mean(), sd=y.std())
    sigma = pm.Uniform('sigma', y.std()/1000, y.std()*1000)

    # PyMC's Normal likelihood can take either precision or standard deviation as an argument.
    likelihood = pm.Normal('likelihood', mu, sd=sigma, observed=y)

pm.model_to_graphviz(model)

with model:
    trace = pm.sample(2000, cores=4,  nuts_kwargs={'target_accept': 0.95})

pm.traceplot(trace);


I am working on the file "Notebooks/Chapter 16.ipynb".
The first task is:
16.2 - Outliers and robust estimation: the t distribution
Model
Can you write Python code for it?

with pm.Model() as model2:
    mu = pm.Normal('mu', y.mean(), sd=y.std())
    sigma = pm.Uniform('sigma', y.std()/1000, y.std()*1000)
    nu_minus1 = pm.Exponential('nu_minus1', 1/29)
    nu = pm.Deterministic('nu', nu_minus1+1)

    likelihood = pm.StudentT('likelihood', nu,  mu, sd=sigma, observed=y)

pm.model_to_graphviz(model2)

with model2:
    trace2 = pm.sample(5000, cores=4, nuts_kwargs={'target_accept': 0.95})

pm.traceplot(trace2);


I am working on the file "Notebooks/Chapter 16.ipynb".
The first task is:
16.2 - Two Groups
Model (Kruschke, 2015)
Can you write Python code for it?

Image('images/fig16_11.png', width=400)

grp_idx = df.Group.cat.codes.values
grp_codes = df.Group.cat.categories
n_grps = grp_codes.size

with pm.Model() as model3:
    mu = pm.Normal('mu', df.Score.mean(), sd=df.Score.std(), shape=n_grps)
    sigma = pm.Uniform('sigma', df.Score.std()/1000, df.Score.std()*1000, shape=n_grps)
    nu_minus1 = pm.Exponential('nu_minus1', 1/29)
    nu = pm.Deterministic('nu', nu_minus1+1)

    likelihood = pm.StudentT('likelihood', nu,  mu[grp_idx], sd=sigma[grp_idx], observed=df.Score)

pm.model_to_graphviz(model3)

with model3:
    trace3 = pm.sample(5000, cores=4, nuts_kwargs={'target_accept': 0.95})

pm.traceplot(trace3);


I am working on the file "Notebooks/Chapter 16.ipynb".
The first task is:
Figure 16.3
Can you write Python code for it?

fig, [(ax1, ax2), (ax3, ax4)] = plt.subplots(2,2, figsize=(10,6))

font_d = {'size':16}

# Upper left
pm.plot_posterior(trace['mu'], point_estimate='mode', ref_val=100, ax=ax1, color=color)
ax1.set_xlabel('$\mu$', fontdict=font_d)
ax1.set_title('Mean', fontdict=font_d)

# Upper right
tr_len = len(trace)
# Plot only 20 posterior prediction curves.
n_curves = 20
# Create an index of length 20 with which we step through the trace.
stepIdxVec = np.arange(0, tr_len, tr_len//n_curves)
x_range = np.arange(y.min(), y.max())
x = np.tile(x_range.reshape(-1,1), (1,20))

ax2.hist(y, bins=25, density=True, color='steelblue')
ax2.plot(x, norm.pdf(x, trace['mu'][stepIdxVec], trace['sigma'][stepIdxVec]), c=color)
ax2.set_xlabel('y', fontdict=font_d)
ax2.set_title('Data w. Post. Pred.\nN=63')
[ax2.spines[spine].set_visible(False) for spine in ['left', 'right', 'top']]
ax2.yaxis.set_visible(False)

# Lower left
pm.plot_posterior(trace['sigma'], point_estimate='mode', ref_val=15, ax=ax3, color=color)
ax3.set_xlabel('$\sigma$', fontdict=font_d)
ax3.set_title('Std. Dev.', fontdict=font_d)

# Lower right
pm.plot_posterior((trace['mu']-100)/trace['sigma'], point_estimate='mode', ref_val=0,
                  ax=ax4, color=color)
ax4.set_title('Effect Size', fontdict=font_d)
ax4.set_xlabel('$(\mu - 100)/\sigma$', fontdict=font_d)

plt.tight_layout();


I am working on the file "Notebooks/Chapter 9.ipynb".
The first task is:
9.2.4 - Example: Therapeutic touch
Can you write Python code for it?

df = pd.read_csv('data/TherapeuticTouchData.csv', dtype={'s':'category'})
df.info()

df.head()


I am working on the file "Notebooks/Chapter 9.ipynb".
The first task is:
Figure 9.10 - Marginal posterior distributions
Can you write Python code for it?

plt.figure(figsize=(10,12))

# Define gridspec
gs = gridspec.GridSpec(4, 6)
ax1 = plt.subplot(gs[0,:3])
ax2 = plt.subplot(gs[0,3:])
ax3 = plt.subplot(gs[1,:2])
ax4 = plt.subplot(gs[1,2:4])
ax5 = plt.subplot(gs[1,4:6])
ax6 = plt.subplot(gs[2,:2])
ax7 = plt.subplot(gs[2,2:4])
ax8 = plt.subplot(gs[2,4:6])
ax9 = plt.subplot(gs[3,:2])
ax10 = plt.subplot(gs[3,2:4])
ax11 = plt.subplot(gs[3,4:6])

# thetas and theta pairs to plot
thetas = (0, 13, 27)
theta_pairs = ((0,13),(0,27),(13,27))

font_d = {'size':14}

# kappa & omega posterior plots
for var, ax in zip(['kappa', 'omega'], [ax1, ax2]):
    pm.plot_posterior(trace[var], point_estimate='mode', ax=ax, color=color, round_to=2)
    ax.set_xlabel('$\{}$'.format(var), fontdict={'size':20, 'weight':'bold'})
ax1.set(xlim=(0,500))

# theta posterior plots
for var, ax in zip(thetas,[ax3, ax7, ax11]):
    pm.plot_posterior(trace['theta'][:,var], point_estimate='mode', ax=ax, color=color)
    ax.set_xlabel('theta[{}]'.format(var), fontdict=font_d)

# theta scatter plots
for var, ax in zip(theta_pairs,[ax6, ax9, ax10]):
    ax.scatter(trace['theta'][::10,var[0]], trace['theta'][::10,var[1]], alpha=0.75, color=color, facecolor='none')
    ax.plot([0, 1], [0, 1], ':k', transform=ax.transAxes, alpha=0.5)
    ax.set_xlabel('theta[{}]'.format(var[0]), fontdict=font_d)
    ax.set_ylabel('theta[{}]'.format(var[1]), fontdict=font_d)
    ax.set(xlim=(0,1), ylim=(0,1), aspect='equal')

# theta posterior differences plots
for var, ax in zip(theta_pairs,[ax4, ax5, ax8]):
    pm.plot_posterior(trace['theta'][:,var[0]]-trace['theta'][:,var[1]], point_estimate='mode', ax=ax, color=color)
    ax.set_xlabel('theta[{}] - theta[{}]'.format(*var), fontdict=font_d)

plt.tight_layout()


I am working on the file "Notebooks/Chapter 9.ipynb".
The first task is:
Figure 9.17
Posterior distribution of hyper parameter omega after sampling.
Can you write Python code for it?

pm.plot_posterior(trace2['omega'], point_estimate='mode', color=color)
plt.title('Overall', fontdict={'fontsize':16, 'fontweight':'bold'})
plt.xlabel('omega', fontdict={'fontsize':14});


I am working on the file "Notebooks/Chapter 9.ipynb".
The first task is:
Here we concatenate the trace results (thetas) from both models into a dataframe. Next we shape the data into a format that we can use with Seaborn's pointplot.
Can you write Python code for it?

df_shrinkage = (pd.concat([pm.summary(unpooled_trace).iloc[:,0],
                           pm.summary(trace).iloc[3:,0]],
                          axis=1)
                .reset_index())
df_shrinkage.columns = ['theta', 'unpooled', 'hierarchical']
df_shrinkage = pd.melt(df_shrinkage, 'theta', ['unpooled', 'hierarchical'], var_name='Model')
df_shrinkage.head()


I am working on the file "Notebooks/Chapter 9.ipynb".
The first task is:
Model (Kruschke, 2015)
Can you write Python code for it?

Image('images/fig9_7.png', width=200)

practitioner_idx = df.s.cat.codes.values
practitioner_codes = df.s.cat.categories
n_practitioners = practitioner_codes.size

with pm.Model() as hierarchical_model:
    omega = pm.Beta('omega', 1., 1.)
    kappa_minus2 = pm.Gamma('kappa_minus2', 0.01, 0.01)
    kappa = pm.Deterministic('kappa', kappa_minus2 + 2)

    theta = pm.Beta('theta', alpha=omega*(kappa-2)+1, beta=(1-omega)*(kappa-2)+1, shape=n_practitioners)

    y = pm.Bernoulli('y', theta[practitioner_idx], observed=df.y)

pm.model_to_graphviz(hierarchical_model)

with hierarchical_model:
    trace = pm.sample(5000, cores=4, nuts_kwargs={'target_accept': 0.95})

pm.traceplot(trace, ['omega','kappa', 'theta']);

pm.summary(trace)
# Note that theta is indexed starting with 0 and not 1, as is the case in Kruschke (2015).


I am working on the file "Notebooks/Chapter 9.ipynb".
The first task is:
Posterior distributions of the omega_c parameters after sampling.
Can you write Python code for it?

fig, axes = plt.subplots(3,3, figsize=(14,8))

for i, ax in enumerate(axes.T.flatten()):
    pm.plot_posterior(trace2['omega_c'][:,i], ax=ax, point_estimate='mode', color=color)
    ax.set_title(pripos_codes[i], fontdict={'fontsize':16, 'fontweight':'bold'})
    ax.set_xlabel('omega_c__{}'.format(i), fontdict={'fontsize':14})
    ax.set_xlim(0.10,0.30)

plt.tight_layout(h_pad=3)


I am working on the file "Notebooks/Chapter 9.ipynb".
The first task is:
Shrinkage
Let's create a model with just the theta estimations per practitioner, without the influence of a higher level distribution. Then we can compare the theta values with the hierarchical model above.
Can you write Python code for it?

with pm.Model() as unpooled_model:

    theta = pm.Beta('theta', 1, 1, shape=n_practitioners)

    y = pm.Bernoulli('y', theta[practitioner_idx], observed=df.y)

pm.model_to_graphviz(unpooled_model)

with unpooled_model:
    unpooled_trace = pm.sample(5000, cores=4)


I am working on the file "Notebooks/Chapter01/Exercise-1.5-py.ipynb".
The first task is:
However, at this point we are at an impass analytically, as the integral of a Gaussian function over a finite range has no analytical solution, and must be evaluated numerically.
So using erfc,
Can you write Python code for it?

pvalues = [0.10, 0.05, 0.01]

for p in pvalues:
    print("{} standard deviations corresponds to a p-value of {}".format(math.sqrt(2.)*special.erfcinv(p),p))


I am working on the file "Notebooks/Chapter01/Exercise-1.5-py.ipynb".
The first task is:
Two-tailed $p$-value
As for the two-tailed Gaussian,
$\displaystyle p(x) = P(\left|X\right| \geq x) = 1-\text{erf}\left(\frac{x}{\sqrt{2}\sigma}\right) \equiv \text{erfc}\left(\frac{x}{\sqrt{2}\sigma}\right)$,
it is seen that for $x=n \sigma$, then
$\displaystyle p(n \sigma) = P(\left|X\right| \geq n \sigma) = 1-\text{erf}\left(\frac{n}{\sqrt{2}}\right)$,
thus,
$\displaystyle \text{erf}\left(\frac{n}{\sqrt{2}}\right) = 1 - p(n \sigma)$.
Can you write Python code for it?

mean = 0
sigma = 1
nsigma = 1
x = np.linspace(-5,5,100)
plt.plot(x,mlab.normpdf(x,mean,sigma),color='black')

xlTail = np.linspace(-5,-nsigma)
xrTail = np.linspace(nsigma,5)
plt.fill_between(xlTail,0,mlab.normpdf(xlTail,mean,sigma),facecolor='red')
plt.fill_between(xrTail,0,mlab.normpdf(xrTail,mean,sigma),facecolor='red')

plt.show()


I am working on the file "Notebooks/Chapter01/Exercise-1.5-py.ipynb".
The first task is:
the same output is found (as required by the defintion of the functions).
One-tailed $p$-value
A one-sided p-value considers the probability for the data to have produced a value as extreme or grearer than the observed value on only one side of the distribution. For example, the p-value for the right tail of a Gaussian is $p(x) = \displaystyle P\left(X \geq x\right) = 1-\Phi(x)$, and the p-value for the left tail of a Gaussian is $p(-x) = \displaystyle P\left(X \leq -x\right) = \Phi(-x)$.
It is seen by symmetry $p(x) = p(-x)$ and that for a normalized Gaussian a one-tailed p-vaule is 1/2 that of a two-tailed p-value.
\begin{split}
    p(x) = P\left(X \geq \left|x\right|\right)&= 1 - \frac{1}{\sqrt{2\pi}}\int\limits_{-\infty}^{x} e^{-t^2/2}\,dt = 1 - \frac{1}{2}\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)\
    &= 1-\Phi(x)\
    &= \frac{1}{2}\left(1-\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right) = \frac{1}{2}\text{erfc}\left(\frac{x}{\sqrt{2}}\right)
\end{split}
Can you write Python code for it?


I am working on the file "Notebooks/Code examples for exam - optimized.ipynb".
The first task is:
Optimization - Step 2: NumPy array operations
Can you write Python code for it?

# Implementation of Eq. (1) in the exam set
def doublegyre(x, y, t, A, e, w):
    a = e * np.sin(w*t)
    b = 1 - 2*e*np.sin(w*t)
    f = a*x**2 + b*x
    return np.array([
            -np.pi*A*np.sin(np.pi*f) * np.cos(np.pi*y),              # x component of velocity
             np.pi*A*np.cos(np.pi*f) * np.sin(np.pi*y) * (2*a*x + b) # y component of velocity
        ])

# Wrapper function to pass to integrator
# X0 is a two-component vector [x, y]
def f(X, t):
    # Parameters of the velocity field
    A = 0.1
    e = 0.25 # epsilon
    w = 1    # omega
    return doublegyre(X[0,:], X[1,:], t, A, e, w)

# 4th order Runge-Kutta integrator
# X0 is a two-component vector [x, y]
def rk4(X, t, dt, f):
    k1 = f(X,           t)
    k2 = f(X + k1*dt/2, t + dt/2)
    k3 = f(X + k2*dt/2, t + dt/2)
    k4 = f(X + k3*dt,   t + dt)
    return X + dt*(k1 + 2*k2 + 2*k3 + k4) / 6

# Function to calculate a trajectory from an
# initial position X0 at t = 0, moving forward
# until t = tmax, using the given timestep and
# integrator
def trajectory(X0, tmax, dt, integrator, f):
    t    = 0
    # Number of timesteps
    Nt = int(tmax / dt)
    # Array to hold the entire trajectory
    PX = np.zeros((*X0.shape, Nt+1))
    # Initial position
    PX[:,:,0] = X0
    # Loop over all timesteps
    for i in range(1, Nt+1):
        PX[:,:,i] = integrator(PX[:,:,i-1], t, dt, f)
        t += dt
    # Return entire trajectory
    return PX

N  = 10000
X0 = grid_of_particles(N, w = 0.1)
# Array to hold all grid points after transport
X1 = np.zeros((2, N))

# Transport parameters
tmax = 5.0
dt   = 0.5

# Loop over grid and update all positions
# This is where parallelisation would happen, since
# each position is independent of all the others
tic = time()
# Keep only the last position, not the entire trajectory
X1 = trajectory(X0, tmax, dt, rk4, f)[:,:,-1]
toc = time()
print('Transport took %.3f seconds' % (toc - tic))

# Make scatter plot to show all grid points
fig = plt.figure(figsize = (12,6))
plt.scatter(X1[0,:], X1[1,:], lw = 0,...


I am working on the file "Notebooks/Code examples for exam - optimized.ipynb".
The first task is:
Transport a collection of particles
Can you write Python code for it?

def grid_of_particles(N, w):
    # Create a grid of N evenly spaced particles
    # covering a square patch of width and height w
    # centered on the region 0 < x < 2, 0 < y < 1
    x  = np.linspace(1.0-w/2, 1.0+w/2, int(np.sqrt(N)))
    y  = np.linspace(0.5-w/2, 0.5+w/2, int(np.sqrt(N)))
    x, y = np.meshgrid(x, y)
    return np.array([x.flatten(), y.flatten()])

X = grid_of_particles(50, 0.1)
# Make a plot to confirm that this works as expected
fig = plt.figure(figsize  = (12,6))
plt.scatter(X[0,:], X[1,:], lw = 0, marker = '.', s = 1)
plt.xlim(0, 2)
plt.ylim(0, 1)

X


I am working on the file "Notebooks/Comparing branch lengths across species trees.ipynb".
The first task is:
And we plot the correlations between branch lengths
Can you write Python code for it?

def plotAndComputeCorrelation(x,y,namex, namey, logyn, logxn):
    print("Pearson correlation coefficient and p-value: "+ str(scipy.stats.pearsonr(x, y)))
    #Plotting:
    plt.plot(x, y, 'bo')
    plt.xlabel(namex, fontsize=15)
    plt.ylabel(namey, fontsize=15)
#plt.legend(['data'], loc='upper left')
    if logyn:
        plt.yscale('log')
    if logxn:
        plt.xscale('log')


I am working on the file "Notebooks/Comparing branch lengths across species trees.ipynb".
The first task is:
Computing an ultrametricity index
Can you write Python code for it?

# We compute all the leaf to root distances, and we compute the normalized variance across leaves.
def computeUltrametricityIndex(t):
    root = t.get_tree_root()
    leaves = root.get_leaves()
    distances=list()
    for leaf in leaves:
        distances.append(root.get_distance(leaf))
    meanDist = np.mean(distances)
    for i in range(len(distances)):
        distances[i] = distances[i] / meanDist
    return np.var(distances)

print ("Deviance from ultrametricity of the substitution tree: " + str(computeUltrametricityIndex(substitutionTree)))
print ("Deviance from ultrametricity of the duplication tree: " + str(computeUltrametricityIndex(duplicationTree)))
print ("Deviance from ultrametricity of the transfer tree: " + str(computeUltrametricityIndex(transferTree)))
print ("Deviance from ultrametricity of the loss tree: " + str(computeUltrametricityIndex(lossTree)))


I am working on the file "Notebooks/Comparing branch lengths across species trees.ipynb".
The first task is:
Fifth, transfer vs losses.
Can you write Python code for it?

%matplotlib inline
plotAndComputeCorrelation(transferTreeBls,lossTreeBls,'Transfers', 'Losses', True, True)


I am working on the file "Notebooks/Comparing branch lengths across species trees.ipynb".
The first task is:
Fourth, Duplications vs transfers.
Can you write Python code for it?

%matplotlib inline
plotAndComputeCorrelation(duplicationTreeBls,transferTreeBls,'Duplications', 'Transfers', True, True)


I am working on the file "Notebooks/Comparing branch lengths across species trees.ipynb".
The first task is:
Second, substitutions vs transfers.
Can you write Python code for it?

%matplotlib inline

plotAndComputeCorrelation(substitutionTreeBls,transferTreeBls,'Substitutions', 'Transfers', True, True)


I am working on the file "Notebooks/Compile population data.ipynb".
The first task is:
Combine data
Can you write Python code for it?

pop_total = pd.concat([pop1_tidy, pop2_tidy])


I am working on the file "Notebooks/Compile population data.ipynb".
The first task is:
Import 2000-2010 data
The sex, origin, race, and age columns are ALL when they have values of 0
Not clear if these are beginning or end of year values.
https://www2.census.gov/programs-surveys/popest/datasets/2000-2010/intercensal/state/
Can you write Python code for it?

path = os.path.join(data_directory, 'Population data',
                    'st-est00int-alldata.csv')
pop1 = pd.read_csv(path)

pop1.head()

pop1 = pop1.loc[(pop1['SEX'] == 0) &
                (pop1['ORIGIN'] == 0) &
                (pop1['RACE'] == 0) &
                (pop1['AGEGRP'] == 0), :]

# Column names for population estimate
est_cols = ['POPESTIMATE{}'.format(x) for x in range(2000, 2011)]

# Melt the wide-form data into a tidy dataframe
pop1_tidy = pd.melt(pop1, id_vars='NAME',
                    value_vars=est_cols, var_name='Year',
                    value_name='Population')

pop1_tidy.head()

def map_year(x):
    'Return last 4 characters (the year)'
    year = x[-4:]
    return int(year)

pop1_tidy['Year'] = pop1_tidy['Year'].map(map_year)


I am working on the file "Notebooks/Compile population data.ipynb".
The first task is:
Import 2010-2016 data
https://www.census.gov/data/tables/2016/demo/popest/state-total.html
Can you write Python code for it?

path = os.path.join(data_directory, 'Population data', 'nst-est2016-01.xlsx')
pop2 = pd.read_excel(path, header=3, parse_cols='A, D:J', skip_footer=7)

pop2.head()

pop2.tail()

drop_rows = ['Northeast', 'Midwest', 'South', 'West']
pop2.drop(drop_rows, inplace=True)

pop2.index = pop2.index.str.strip('.')

pop2.head()

pop2.columns

pop2_tidy = pd.melt(pop2.reset_index(), id_vars='index',
                    value_vars=range(2010, 2017), value_name='Population',
                    var_name='Year')
pop2_tidy.columns = ['State', 'Year', 'Population']


I am working on the file "Notebooks/Compile population data.ipynb".
The first task is:
The overlapping 2010 values are different, but just barely. I'm going to re-combine the datasets and keep values from the second dataset.
Can you write Python code for it?

pop_total.loc[pop_total['Year']==2010].sort_values('State')

pop_total = pd.concat([pop1_tidy.loc[pop1_tidy['Year'] < 2010], pop2_tidy])

pop_total.head()

pop_total.tail()

path = os.path.join('Data storage', 'Derived data', 'State population.csv')
pop_total.to_csv(path, index=False)


I am working on the file "Notebooks/Compile population data.ipynb".
The first task is:
The values shown below are ever slightly different than those listed in the later dataset.
Can you write Python code for it?

pop1_tidy.loc[pop1_tidy['Year'] == 2010].head()

pop1_tidy.head()

pop1_tidy.tail()

pop1_tidy.columns = ['State', 'Year', 'Population']


I am working on the file "Notebooks/Compile state index & gen files.ipynb".
The first task is:
Calculate state-level monthly CO₂ intensity and generation by fuel category
Can you write Python code for it?

index_list = []
gen_list = []
for state in states:
    eia_fac_state = facility_df.loc[facility_df.state == state].copy()
    eia_totals_state = eia_totals.loc[eia_totals.state == state].copy()
    epa_state = epa_df.loc[epa_df.state == state].copy()

    co2, gen_fuels_state = facility_emission_gen(eia_facility=eia_fac_state,
                                                 epa=epa_state,
                                                 state_fuel_cat=state_fuel_cat,
                                                 custom_fuel_cat=custom_fuel_cat,
                                                 export_state_cats=True,
                                                 print_status=False)

    extra_co2, extra_gen = extra_emissions_gen(gen_fuels_state,
                                               eia_totals_state, ef)

    # Combine facility and extra co2, name the series
    co2_monthly = co2.groupby(['year', 'month']).sum()
    total_co2 = (co2_monthly.loc[:, 'final co2 (kg)']
                 + extra_co2.loc[:, 'elec fuel co2 (kg)']
                            .groupby(['year', 'month']).sum())
    total_co2.name = 'final co2 (kg)'

    # Total gen, and the co2 intensity
    total_gen = (eia_totals_state
                 .groupby(['year', 'month'])['generation (mwh)'].sum())

    state_index = pd.concat([total_co2, total_gen], axis=1)
    state_index['index (g/kwh)'] = (state_index['final co2 (kg)']
                                    / state_index['generation (mwh)'])
    state_index['state'] = state
    state_index.set_index('state', append=True, inplace=True)

    # Generation by fuel category
    gen_category = group_fuel_cats(eia_totals_state, custom_fuel_cat,
                                   fuel_col='type', new_col='fuel category')

    keep_cols = ['fuel category', 'generation (mwh)', 'total fuel (mmbtu)',
                 'elec fuel (mmbtu)', 'all fuel co2 (kg)',
                 'elec fuel co2 (kg)', 'year', 'month']
    gen_category = gen_category[keep_cols]
   ...


I am working on the file "Notebooks/Compile state index & gen files.ipynb".
The first task is:
EIA facility data and EPA monthly emissions
Can you write Python code for it?

facility_path = join(data_path, 'Derived data',
                     'Facility gen fuels and CO2 {}.csv'.format(file_date))
facility_df = pd.read_csv(facility_path)
facility_df['state'] = facility_df.geography.str[-2:]
rename_cols(facility_df)

epa_path = join(data_path, 'Derived data',
                'Monthly EPA emissions {}.csv'.format(file_date))
epa_df = pd.read_csv(epa_path)
rename_cols(epa_df)
facility_locations = pd.read_csv(join(data_path, 'Facility labels',
                                      'Facility locations.csv'))
# Add state labels to the EPA facilities
epa_df = add_facility_location(epa_df, facility_locations, labels=['state'])


I am working on the file "Notebooks/Compile state index & gen files.ipynb".
The first task is:
EIA total monthly gen and fuel consumption
Can you write Python code for it?

path = join(data_path, 'Derived data',
            'EIA state-level gen fuel CO2 {}.csv'.format(file_date))
eia_totals = pd.read_csv(path, parse_dates=['datetime'])
rename_cols(eia_totals)
eia_totals['state'] = eia_totals.geography.str[-2:]

# Remove fuel categories that are duplicated with other categories
eia_totals = eia_totals.loc[~eia_totals.type.isin(['SPV', 'AOR', 'TSN'])]


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Adding the -j option to the %sql (or %%sql) command will format the first column of a return set to better
display the structure of the document. Note that if your answer set has additional columns associated with it, they will not be displayed in this format.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
And finally a line chart.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Connections to DB2
Before any SQL commands can be issued, a connection needs to be made to the DB2 database that you will be using. The connection can be done manually (through the use of the CONNECT command), or automatically when the first %sql command is issued.
The DB2 magic command tracks whether or not a connection has occured in the past and saves this information between notebooks and sessions. When you start up a notebook and issue a command, the program will reconnect to the database using your credentials from the last session. In the event that you have not connected before, the system will prompt you for all the information it needs to connect. This information includes:

Database name (SAMPLE)
Hostname - localhost (enter an IP address if you need to connect to a remote server)
PORT - 50000 (this is the default but it could be different)
Userid - DB2INST1
Password - No password is provided so you have to enter a value
Maximum Rows - 10 lines of output are displayed when a result set is returned

There will be default values presented in the panels that you can accept, or enter your own values. All of the information will be stored in the directory that the notebooks are stored on. Once you have entered the information, the system will attempt to connect to the database for you and then you can run all of the SQL scripts. More details on the CONNECT syntax will be found in a section below.
The next statement will force a CONNECT to occur with the default values. If you have not connected before, it will prompt you for the information.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
DB2 CONNECT Statement
As mentioned at the beginning of this notebook, connecting to DB2 is automatically done when you issue your first
%sql statement. Usually the program will prompt you with what options you want when connecting to a database. The
other option is to use the CONNECT statement directly. The CONNECT statement is similar to the native DB2
CONNECT command, but includes some options that allow you to connect to databases that has not been
catalogued locally.
The CONNECT command has the following format:
<pre>
%sql CONNECT TO <database> USER <userid> USING <password | ?> HOST <ip address> PORT <port number>
</pre>
If you use a "?" for the password field, the system will prompt you for a password. This avoids typing the
password as clear text on the screen. If a connection is not successful, the system will print the error
message associated with the connect request.
If the connection is successful, the parameters are saved on your system and will be used the next time you
run a SQL statement, or when you issue the %sql CONNECT command with no parameters.
If you want to force the program to connect to a different database (with prompting), use the CONNECT RESET command. The next time you run a SQL statement, the program will prompt you for the  the connection
and will force the program to reconnect the next time a SQL statement is executed.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Finally, for a line chart, the x contains the labels and the y values are used.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
For a pie chart, the first column is used to label the slices, while the data comes from the second column.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
If you retrieve two columns of information, the first column is used for the labels (X axis or pie slices) and
the second column contains the data.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
JSON Formatting
DB2 supports querying JSON that is stored in a column within a table. Standard output would just display the
JSON as a string. For instance, the following statement would just return a large string of output.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Line versus Cell Command
The DB2 extension is made up of one magic command that works either at the LINE level (%sql) or at the CELL level (%%sql). If you only want to execute a SQL command on one line in your script, use the %sql form of the command. If you want to run a larger block of SQL, then use the %%sql form. Note that when you use the %%sql form of the command, the entire contents of the cell is considered part of the command, so you cannot mix other commands in the cell.
The following is an example of a line command:
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Plotting
Sometimes it would be useful to display a result set as either a bar, pie, or line chart. The first one or two
columns of a result set need to contain the values need to plot the information.
The three possible plot options are:

-pb - bar chart (x,y)
-pp - pie chart (y)
-pl - line chart (x,y)

The following data will be used to demonstrate the different charting options.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Result Set
By default, any %sql block will return the contents of a result set as a table that is displayed in the notebook. If you want to capture the results from the SQL into a variable, you would use the -r option:
<pre>
var = %sql -r select * from employee
</pre>
The result is a list of rows. Each row is a list itself. The rows and columns all start at zero (0), so
to access the first column of the first row, you would use var[0][0] to access it.
Can you write Python code for it?

rows = %sql -r select * from employee
print(rows[0][0])


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
SQL and Command Mode
The %sql looks at the beginning of your SQL to determine whether or not to run it as a SELECT statement, or
as a command. There are three statements that trigger SELECT mode: SELECT, WITH, and VALUES. Aside from
CONNECT, all other statements will be considered to be an SQL statement that does not return a result set. In
most cases this will work fine. However, it is possible that you may have a statement that is actually a command and does not return a result set. However, because of the simplistic nature of determining the statement
type, the statement will not be executed properly. To treat all statements as commands, use the -n flag (no output) and the -s flag for output.
Timing SQL Statements
Sometimes you want to see how the execution of a statement changes with the addition of indexes or other
optimization changes. The -t option will run the statement on the LINE or one SQL statement in the CELL for
exactly one second. The results will be displayed and optionally placed into a variable. The syntax of the
command is:
<pre>
sql_time = %sql -t SELECT * FROM EMPLOYEE
</pre>
For instance, the following SQL will time the VALUES clause.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
SQL output will not be suppressed, so the following command will still show the results.
Can you write Python code for it?

%%sql -q
DROP TABLE TABLE_NOT_FOUND;
DROP TABLE TABLE_SPELLED_WRONG;
VALUES 1,2,3;


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
Since the data may be returned in different formats (like integers), you should use the str() function
to convert the values to strings. Otherwise, the concatenation function used in the above example will fail. For
instance, the 6th field is a birthdate field. If you retrieve it as an individual value and try and concatenate a string to it, you get the following error.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
The following SQL will plot the number of employees per department.
Can you write Python code for it?

%%sql -pb
SELECT WORKDEPT, COUNT(*)
  FROM EMPLOYEE
GROUP BY WORKDEPT


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
The number of rows in the result set can be determined by using the length function.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
The script will generate messages and output as it executes. Each SQL statement that generates results will have a table displayed with the result set. If a command is executed, the results of the execution get listed as well. The script you just ran probably generated an error on the DROP table command.
Options
Both forms of the %sql command have options that can be used to change the behavior of the code. For both forms of the command (%sql, %%sql), the options must be on the same line as the command:

%sql -t ...
%%sql -t

The only difference is that the %sql command can have SQL following the parameters, while the %%sql requires the
SQL to be placed on subsequent lines.
There are a number of parameters that you can specify as part of the %sql statement.

-d - Use alternative delimiter
-t - Time the statement execution
-n - Run all statements as commands (no answer sets)
-s - Run all statements as SQL
-q - Suppress messages
-qq - Suppress messages and any SQL output
-j - JSON formatting of a column
-a - Show all output
-pb - Bar chart of results
-pp - Pie chart of results
-pl - Line chart of results
-sampledata Load the database with the sample EMPLOYEE and DEPARTMENT tables
-r - Return the results into a variable (list of rows)

Multiple parameters are allowed on a command line. Each option should be separated by a space:

%sql -a -j ...

The sections below will explain the options in more detail.
Delimiters
The default delimiter for all SQL statements is the semicolon. However, this becomes a problem when you try to create a trigger, function, or procedure that uses SQLPL (or PL/SQL). Use the -d option to turn the SQL delimiter into the at (@) sign. The semi-colon is then ignored as a delimiter.
For example, the following SQL will use the @ sign as the delimiter.
Can you write Python code for it?

%%sql -d
DROP TABLE STUFF
@
CREATE TABLE STUFF (A INT)
@
INSERT INTO STUFF VALUES
  1,2,3
@
SELECT * FROM STUFF
@


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
To change the default value of rows displayed, you can either do a CONNECT RESET (discussed later) or set the
db2 control variable db2_max to a different value. A value of -1 will display all rows.
Can you write Python code for it?

# Save previous version of maximum rows
last_max = db2_max
db2_max = 5
%sql values 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
# Set the maximum back
db2_max = last_max


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
To set the messages back to being formatted, set db2_error_highlight to True.
Can you write Python code for it?

db2_error_highlight = True
%sql DROP TABLE TABLE_NOT_FOUND;


I am working on the file "Notebooks/DB2 Jupyter Extensions Tutorial.ipynb".
The first task is:
You can fix this problem by adding the str function to convert the date.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:

Table of Contents

A Brief Introduction to Odata

* [DB2 and OData Connection Requirements](#connect)
* [Connecting through OData](#connectodata)
* [Set Command Syntax](#set)

* [A Quick Introduction](#quick)
  * [Selecting Data from a Table](#sampleselect)
  * [Displaying the OData Syntax](#viewodata)
  * [Limiting Output Results](#limit)
  * [Persistent Connection Information](#persistant)
  * [Variables in OData Statements](#variables)
  * [Retrieving URL, OData Command, and Parameters](#retrieveurl)
  * [JSON Display in Firefox](#firefox)

* [SQL Command Syntax](#sqlsyntax)
  * [SELECT Statements](#select)
    * [Selecting Columns to Display](#columns)
    * [FROM  Clause](#from)
    * [Describing the Table Structure](#describe)
    * [WHERE Clause](#where)
    * [LIMIT Clause](#limitclause)
  * [INSERT Statement](#insert)
  * [DELETE Statement](#delete)
  * [UPDATE Statement](#update)
  * [VIEWS](#views)

* [Summary](#summary)

[Back to Top](#top)

## An Brief Introduction to OData
Rather than paraphrase what OData does, here is the official statement from the OData home page:

http://www.odata.org/

>OData (Open Data Protocol) is an ISO/IEC approved, OASIS standard that defines a set of best practices for building and consuming RESTful APIs. OData helps you focus on your business logic while building RESTful APIs without having to worry about the various approaches to define request and response headers, status codes, HTTP methods, URL conventions, media types, payload formats, query options, etc. OData also provides
guidance for tracking changes, defining functions/actions for reusable procedures, and sending asynchronous/batch requests.

### Why OData for DB2?
Customers have a wealth of data in their databases (not just DB2) and publishing it to different devices is often fraught with many challenges. DB2 requires the use of client code to communicate between the application and the database itself. Many of APIs that are used are well known: JDBC,...
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
A couple of things about the table design. The salary is NOT NULL, while the BONUS allows for nulls. Unfortunately, the DESCRIBE command only tells us about the columns in the table and their OData data type, and no indication of whether table.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
A primary key is required to issue a DELETE command. You also need to make sure that the primary key column does not contain NULLs because a primary key must always contain a value. The following SQL tries to fix the primary key issue.
Can you write Python code for it?

%sql ALTER TABLE TESTODATA ADD CONSTRAINT PKTD PRIMARY KEY (EMPNO)


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Adding a primary key after the fact won't help because the service URL would have already recorded the information about the table (and the fact it didn't have a primary key at the time). We need to drop our SERVICE URL and generate another one.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
And now we can test our view by selecting from it.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
At this point we can update their salary.
Can you write Python code for it?

%odata -e UPDATE TESTODATA SET BONUS=2000 WHERE EMPNO=1


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Back to Top

DELETE Command
The DELETE command only takes one parameter and that is the key value for the record that we want to delete from the table. The format of the command is:

DELETE FROM <table> WHERE KEY=VALUE

Key refers to the column that is the primary key in the table we are deleting from. Unless you have a primary key, the DELETE command will not work.
Can you write Python code for it?

%odata -e DELETE FROM TESTODATA WHERE EMPNO=1


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Back to Top

FROM Clause
The FROM clause is mandatory in any SELECT statement. If an OData service has already been established, there will be no service request sent to OData. Instead, the URL information stored on disk will be used to establish the connection.
If a service has not been established, the %odata command will create the service and then build the OData select statement. If you want to see the command to establish the service as well as the SELECT command, use the -e flag to echo the results.
If the table does not exist in the database you will receive an error message.
Can you write Python code for it?

%sql -q DROP TABLE UNKNOWN_TBL
%odata RESET TABLE UNKNOWN_TBL
s = %odata -e SELECT * FROM UNKNOWN_TBL


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Back to Top

INSERT Command
OData allows you to insert data into a table through the use of the RESTful POST command and a JSON document that contains the field names and contents of those fields.
The format of the INSERT command is:

INSERT INTO <table>(col1, col2, ...) VALUES (val1, val2, ...)

The TABLE must be defined before you can issue this statement. There is no requirement to have a primary key on the table, but this will prevent you from updating it with the OData interface because filtering (WHERE) is not allowed on UPDATEs or DELETEs. The column list and value list must match (i.e. there must be a value for every column name). If you do not supply the list of all columns in the table, the missing columns will have null values assigned to them. The insert will fail if any of these missing columns requires a value (NOT NULL).
Example: Insert into a table
In this example we will insert a single row into a table. We start by defining the table within DB2 and then doing a DESCRIBE to get the column definitions back with OData.
Can you write Python code for it?

%%sql -q
DROP TABLE TESTODATA;
CREATE TABLE TESTODATA
  (
  EMPNO INT NOT NULL,
  LASTNAME VARCHAR(10) NOT NULL,
  SALARY INT NOT NULL,
  BONUS INT
  );

%sql select * from testodata

%odata -e select * from testodata


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Back to Top

Limiting Output Results
The results contain 43 rows. If you want to reduce the amount of rows being returned we can use the LIMIT clause on the SELECT statement. In addition, we can use the -j flag to return the data as JSON records.
Can you write Python code for it?

s = %odata -e -j SELECT * FROM EMPLOYEE LIMIT 1


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Back to Top

Retrieving URL, OData Command, and Parameters
The %odata command will return the URL command for a select statement as part of the command:

<url> = %odata -e select * from employee limit 1

The variable "url" will contain the full URL required to retrieve data from the OData service. The next command illustrates how this works. You must use the echo (-e) option to get the URL returned. Note that you cannot use this syntax with the %%odata version of the command.
Can you write Python code for it?

url = %odata -e select * from employee limit 1


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Back to Top

SELECT Statements
The SELECT statement is the most complicated of the four statements that are allowed in OData. There are generally two forms that can be used when accessing a record. The first method uses the primary key of the table and it requires no arguments. Note that the examples will not show the URL that points to the OData service.

/EMPLOYEES('000010')

The second method is to use the \$filter query option. \$filter allows us to compare any column against a value. The equivalent OData statement for retrieving an individual employee is:

/EMPLOYEES?$filter=EMPNO eq '000010'

The generated SELECT statements will always use this format, rather than relying on a primary key. This becomes more important when we deal with Views.
SELECT Syntax
The SELECT command will return data from one table. There is no ability to join tables with the current implementation of OData. If you do want to join tables, you may want to create a VIEW on the DB2 system and then use that as the TABLE. This will allow for SELECT, but no INSERT/DELETE/UPDATE.
You do not need to use the primary key in the WHERE clause to use this statement. By default, any results will be displayed in a table. If you want to retrieve the results as JSON records, use the -j option on the %odata command.

SELECT \[col1, col2, ... | count(\*)\] FROM <table> \[ WHERE logic\] \[ LIMIT rows \]

The column list can contain as many values as you want, or just COUNT(*). COUNT(*) will return the count of rows found. If you use the -r or -j flags to display everything in JSON format, you will also get the entire answer set along with the row count. This is the behavior of using count in OData.
The FROM clause must contain the name of the table you want to access.
The WHERE clause is optional, as is the LIMIT clause. The WHERE clause can contain comparisons between columns and constants (EMPNO='000010'), logic (AND, OR) as well as LIKE clauses (COLUMN LIKE 'xxx'). The current version...
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Back to Top

UPDATE Command
The update command requires both a primary key to update and the name of the field that you want changed. Note that you can only change one field at a time. There is no ability to specify multiple fields at this time.
The format of the UDPATE command is:

UPDATE <table> SET column=value WHERE key=keyvalue

You must have a primary key on the table if you want an update to work. The filtering (WHERE) is allowed only to specify the primary key for the row and no filtering is allowed. The primary can be changed in the statement, but the update will fail if the key already exists in another record.
THe other restriction is that no calculations can be done as part of the SET clause. You can only pass atomic values to the UPDATE statement.
Example: Update a BONUS value of employee
This SQL will update employee number 1 bonus to 2000. The first step is to put the employee back into the table.
Can you write Python code for it?

%odata -e \
INSERT INTO TESTODATA(EMPNO, LASTNAME, SALARY, BONUS) \
VALUES (1,'Fred',10000,1000)


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Back to Top

Displaying the OData Syntax
Under the covers a number of things happened when running this command. The SELECT * FROM EMPLOYEE is not what is sent to OData. The syntax is converted to something that the RESTful API understands. To view the actual OData syntax, the -e option is used to echo back the commands.
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Converting to OData will mean that the search will look across the entire string, not just the beginning.
Can you write Python code for it?

s = %odata -e SELECT LASTNAME FROM EMPLOYEE WHERE LASTNAME LIKE '%ON'


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
DB2 Extensions
Since we are connecting to a DB2 database, the following command will load the DB2 Jupyter notebook extension (%sql). The DB2 extension allows you to fully interact with the DB2 database, including the ability to drop and create objects. The OData gateway provides INSERT, UPDATE, DELETE, and SELECT capability to the database, but it doesn't have the ability to create or drop actual objects. The other option would be to use DB2 directly on the database server using utilities like CLP (Command Line Processor) or DSM (Data Server Manager).
Can you write Python code for it?


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Deleting the record again still gives you a successful return code. The call always returns a successful status even if the record doesn't exist.
Can you write Python code for it?

%odata -e DELETE FROM TESTODATA WHERE EMPNO=2


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Example: Beginning of string search
In SQL, you can search for a name that ends with the letters ON by using LIKE '%ON'
Can you write Python code for it?

%sql SELECT LASTNAME FROM EMPLOYEE WHERE LASTNAME LIKE '%ON'


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Example: Overriding the order of comparisons
You can override the order of comparisons in the WHERE clause by using parenthesis. Here we are asking for employees in department E21 or E11 and have a salary less than 40000.
Can you write Python code for it?

s = %odata -e \
SELECT EMPNO, WORKDEPT, SALARY \
       FROM EMPLOYEE \
WHERE SALARY < 40000 AND (WORKDEPT = 'E21' OR WORKDEPT = 'E11')


I am working on the file "Notebooks/DB2 OData Gateway Tutorial.ipynb".
The first task is:
Example: Select statement limiting output to 5 rows
This SELECT statement will limit output to 5 rows. If MAXROWS was set to a smaller value, it would still read all rows before displaying them.
Can you write Python code for it?

s = %odata -e SELECT * FROM EMPLOYEE LIMIT 5