I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
I only lost 4270 sites with several PMID and several tissue. Some sites where demultiplexed and I now have 290018 sites with 1 PMID and 1 tissue
Convert DARNED coordinates from hg19 to hg39 with Crossmap
Coordinate conversion using CrossMap and a hg19 tp hg38 chain file in BASH
Can you write Python code for it?

# Conversion to hg38 with Crossmap/liftover
lifover_chainfile = "../LiftOver_chain_files/hg19ToHg38.over.chain.gz"
input_bed = "./PTM_Original_Datasets/DARNED_human_hg19_inosine_cleaned.bed"
temp_bed = "./PTM_Original_Datasets/DARNED_human_hg38_inosine_temp.bed"

cmd = "CrossMap.py bed {} {} {}".format(lifover_chainfile, input_bed, temp_bed)
bash(cmd)

# Rewriting and updating of the header removed by Crossmap
final_bed = "./PTM_Original_Datasets/DARNED_human_hg38_inosine_cleaned.bed"

header = "# DARNED database Human all sites hg38 coordinates\n"
header+= "# Data cleaned, filtered for Inosine editing, standardized, converted to BED6 format and updated to hg38 coordinates\n"
header+= "# Adrien Leger (aleg@ebi.ac.uk) - {}\n".format(str (datetime.datetime.today()))
header+= "# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

with open (temp_bed, "r") as infile, open (final_bed, "w") as outfile:
    outfile.write (header)
    for line in infile:
        outfile.write (line)

file_summary(final_bed)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
I tried to see if the PMID and the tissue field always had the same lengh, so I could de multiplex the fused positions. The answer is no, the number of PMID and tissues could be different. However 272551 positions have only 1 tissue and 1 PMID. These are maybe not the more relialable positions but they might be more easy to interprete. The sites with only 1 PMID but several tissues can also be used. Same think for several PMID, 1 tissue. I will demultiplex them so as to have only 1 PMID and 1 tissue by site. Concerning the site with more than 1 PMID and 1 tissue, I will extract then in another backup file.
Can you write Python code for it?

infile = "./PTM_Original_Datasets/DARNED_human_hg19_inosine.bed"
outclean = "./PTM_Original_Datasets/DARNED_human_hg19_inosine_cleaned.bed"
outunclean = "./PTM_Original_Datasets/DARNED_human_hg19_inosine_unclean.bed"

with open(infile, "r") as inf, open(outclean, "w") as outf_clean, open(outunclean, "w") as outf_unclean:

    init_sites = uniq = several_tissue = several_pmid = several_all = final_sites = 0

    for line in inf:
        if line [0] == "#":
            outf_clean.write(line)
            outf_unclean.write(line)
        else:
            init_sites += 1
            ls = supersplit(line, ["\t","|"])
            tissue_list = ls[4].split(",")
            PMID_list = ls[6].split(",")
            n_tissue = len(tissue_list)
            n_PMID = len(PMID_list)

            if n_tissue == 1:

                # 1 PMID, 1 tissue = no problem
                if n_PMID == 1:
                    uniq += 1
                    final_sites += 1
                    outf_clean.write(line)

                # Several PMID, 1 tissue = demultiplex PMID lines
                else:
                    several_pmid += 1
                    for PMID in PMID_list:
                        final_sites += 1
                        outf_clean.write("{0}\t{1}\t{2}\t{3}|{4}|{5}|{6}|{7}\t{8}\t{9}".format(
                            ls[0],ls[1],ls[2],ls[3],ls[4],ls[5],PMID.strip(),ls[7],ls[8],ls[9]))
            else:

                # 1 PMID, several tissues = demultiplex tissues lines
                if n_PMID == 1:
                    several_tissue += 1
                    for tissue in tissue_list:
                        final_sites += 1
                        outf_clean.write("{0}\t{1}\t{2}\t{3}|{4}|{5}|{6}|{7}\t{8}\t{9}".format(
                            ls[0],ls[1],ls[2],ls[3],tissue.strip().strip("_").strip("."),ls[5],ls[6],ls[7],ls[8],ls[9]))

                # Several PMID, several tissues = extract the line in uncleanable...


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
It is working ok > looping over all the cleaned PTM files
Can you write Python code for it?

# Annotation vs gencodev23 lncRNA genes

annotation_file = '/home/aleg/Data/Reference_Annotation/gencode_v24_lncRNAs.gff3'

for peak_file in sorted(glob("./PTM_Clean_Datasets/*.bed")):
    outdir = "./PTM_Annotated_Datasets"
    intersect_extract_genecodeID (annotation_file, peak_file, outdir)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
No problem with this dataset, Since there is no gene loci, I just filed the field with a dash to indicate that it is empty
m5C_Khoddami_hg38
Can you write Python code for it?

infile="./PTM_Original_Datasets/m5C_Khoddami_hg38.bed"
PMID = "23604283"
cell = "MEF"
modification = "m5C"
method = "AzaIP"
author = "Khoddami"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

init_template=[0,"\t",1,"\t",2,"\t",3,"|",4,"\t",5,"\t",6]
final_template=[0,"\t",1,"\t",2,"\t",modification,"|",cell,"|",method,"|",PMID,"|",4,"\t",5,"\t",6]

reformat_table(
    input_file=infile,
    output_file=outfile,
    init_template=init_template,
    final_template=final_template,
    keep_original_header = False,
    header = generate_header(PMID, cell, modification, method),
    replace_internal_space='_',
    replace_null_val="-")

file_summary(outfile)

distrib_peak_len(outfile)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
No problem with this dataset. It seems to be focussed on tRNA gene that are clearly over-represented in the gene list

m5C_Squires_hg38
Can you write Python code for it?

infile="./PTM_Original_Datasets/m5C_Squires_hg38.bed"
PMID = "22344696"
cell = "HeLa"
modification = "m5C"
method = "bisulfite_seq"
author = "Squires"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

init_template=[0,"\t",1,"\t",2,"\t",3,"|",4,"\t",5,"\t",6]
final_template=[0,"\t",1,"\t",2,"\t",modification,"|",cell,"|",method,"|",PMID,"|-\t",5,"\t",6]

reformat_table(
    input_file=infile,
    output_file=outfile,
    init_template=init_template,
    final_template=final_template,
    keep_original_header = False,
    header = generate_header(PMID, cell, modification, method),
    replace_internal_space='_',
    replace_null_val="-")

file_summary(outfile)

distrib_peak_len(outfile)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
Read the original file, reformat the field and write a new file BED6 compliant.
chrom   chromstart  chromend    name    score   orient
chr4    774138  774138  A>I|LOC100129917|LUNG:LYMPHOBLASTOID_CELL_LINE|15342557:15258596:22327324   0   -
Can you write Python code for it?

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

def reformat_RADAR (main_file, secondary_file, outfile, errfile, header):

    # Read and structure the 2 database files
    print("Parse the main database file")
    main = parse_RADAR_main(main_file)

    print("Parse the secondary database file")
    secondary = parse_RADAR_secondary (secondary_file)

    print("Combine the data together in a new bed formated file")
    with open (outfile, "w+") as csvout, open (errfile, "w+") as errout:

        # rewrite header
        csvout.write(header)

        fail = success = 0
        for total, site in enumerate(secondary):

            try:
                line = "{0}\t{1}\t{1}\t{2}|{3}|{4}|{5}|{6}\t{7}\t{8}\n".format(
                    site.chromosome,
                    site.position,
                    "A>I",
                    site.tissue,
                    "-",
                    site.PMID,
                    main[site.chromosome][site.position]["gene"],
                    site.editing_level,
                    main[site.chromosome][site.position]["strand"],
                )
                csvout.write(line)
                success += 1

            except KeyError as E:
                line = "{0}\t{1}\t{2}\t{3}\t{4}\n".format(
                    site.chromosome,
                    site.position,
                    site.tissue,
                    site.PMID,
                    site.editing_level
                )
                errout.write(line)
                fail += 1

    print ("{} Sites processed\t{} Sites pass\t{} Sites fail".format(total, success, fail))

header = "# RADAR database Human v2 all sites hg19 coordinates\n"
header+= "# Data cleaned, standardized and converted to BED6 format\n"
header+= "# Adrien Leger (aleg@ebi.ac.uk) - {}\n".format(str (datetime.datetime.today()))
header+= "# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

reformat_RADAR(
    main_file =...


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
The conversion resulted in the lost of 16 sites, which is neglectable compared with the 290002 sites in the database

RADAR
Reformat RADAR Database
The file dowloaded from DARNED is not a bed file. I need to modify it to comply with the standard format. The format of the mane file Human_AG_all_hg19_v2.txt is the following:
Can you write Python code for it?


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
The filtering based on the peak length size remove a lot of peaks = nearly 90 % of the dataset
Can you write Python code for it?

# Conversion to hg38 with Crossmap/liftover
lifover_chainfile = "../LiftOver_chain_files/hg19ToHg38.over.chain.gz"
input_bed = "./PTM_Clean_Datasets/Dominissini_m6A_HepG2_hg19_cleaned.bed"
temp_bed = "./PTM_Clean_Datasets/Dominissini_m6A_HepG2_hg38_temp.bed"

cmd = "CrossMap.py bed {} {} {}".format(lifover_chainfile, input_bed, temp_bed)
bash(cmd)

# Rewriting and updating of the header removed by Crossmap
final_bed = "./PTM_Clean_Datasets/Dominissini_m6A_HepG2_hg38_cleaned.bed"
header = generate_header(PMID, cell, modification, method)

with open (temp_bed, "r") as infile, open (final_bed, "w") as outfile:
    outfile.write (header)
    for line in infile:
        outfile.write (line)

file_summary(final_bed)

distrib_peak_len(final_bed, normed=False, bins=200)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
The name field is unusable since it contains a random number of fieds... I cannot parse it easily. That is not a big problem since I will reannotate the data based on the last gencode annotation realease, I did not save any of the informations contained in the original name field. With miCLIP data the peak are 1 nt wide

MeRIPseq_m1A_Dominissini2016_hg38
Can you write Python code for it?

infile="./PTM_Original_Datasets/MeRIPseq_m1A_Dominissini2016_hg38.bed"
PMID = "26863196"
cell = "HeLa:HEK293:HepG2"
modification = "m1A"
method = "M1A_seq"
author = "Dominissini"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
The operation is quite complex since I will have to fuse the 2 files and extract only specific values. I need to code a specific parser. The main RADAR file will be parsed and organised as a simple embed dict. The secondary file will be the more important since I will start parsing from it to find the complementary information in the main database file. Each site will be added to a list of Site objects, that will be subsequently iterated to combine with the main database before writing in a new Bed formated file.
Can you write Python code for it?

# Create a structured dict of dict to parse the main database file
from collections import OrderedDict

def parse_RADAR_main (file):

    # Define the top level access dict
    radar_dict = OrderedDict()

    for line in open (file, "r"):
        if line[0] != "#":

            sl = line.split("\t")

            assert len(sl) == 11
            chromosome, position, gene, strand = sl[0].strip(), int(sl[1].strip()), sl[2].strip(), sl[3].strip()

            if chromosome not in radar_dict:
                radar_dict[chromosome] = OrderedDict()

            # There should be only one line per position
            assert position not in radar_dict[chromosome]
            radar_dict[chromosome][position] = {"gene":gene,"strand":strand}

    return radar_dict

# Create a class to store a line of the additional file.

from collections import OrderedDict

class Site (object):

    #~~~~~~~CLASS FIELDS~~~~~~~#
    # Table of correspondance reference => PMID
    TITLE_TO_PMID = {
        "Peng et al 2012":"22327324",
        "Bahn et al 2012":"21960545",
        "Ramaswami et al 2012":"22484847",
        "Ramaswami et al 2013":"23291724",
        }

    # Table of correspondance reference => PMID
    TISSUE_TO_SAMPLE = {
        "Brain":"Brain",
        "Illumina Bodymap":"Illumina_Bodymap",
        "Lymphoblastoid cell line":"YH",
        "U87 cell line":"U87MG"
        }

    #~~~~~~~FONDAMENTAL METHODS~~~~~~~#
    # Parse a line of the aditional information file
    def __init__(self, line, ):
        sl = line.strip().split("\t")

        self.chromosome = sl[0].split(":")[0].strip()
        self.position = int(sl[0].split(":")[1].strip())
        self.PMID = self.TITLE_TO_PMID[sl[1].strip()]
        self.tissue = self.TISSUE_TO_SAMPLE[sl[2].strip()]
        self.coverage = sl[3].strip()
        self.editing_level = sl[4].strip()

    # Fundamental class methods str and repr
    def __repr__(self):
        msg = "SITE CLASS\n"
        # list all values in...


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
The original dataset was OK and clean. Contrary to the previous dataset the width of the peaks are between 100 and 220 pb wich is clearly better. It is however interesting to notice that the peak lengths are not randomly distributed

miCLIP_m6A_Linder2015_hg38
Can you write Python code for it?

infile="./PTM_Original_Datasets/miCLIP_m6A_Linder2015_hg38.bed"
PMID = "26121403"
cell = "HEK293"
modification = "m6A:m6Am"
method = "miCLIP"
author = "Linder"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

init_template=[0,"\t",1,"\t",2,"\t",3,"\t",4,"\t",5]
final_template=[0,"\t",1,"\t",2,"\t",modification,"|",cell,"|",method,"|",PMID,"|-\t",4,"\t",5]

reformat_table(
    input_file=infile,
    output_file=outfile,
    init_template=init_template,
    final_template=final_template,
    keep_original_header = False,
    header = generate_header(PMID, cell, modification, method),
    replace_internal_space='_',
    replace_null_val="-")

file_summary(outfile)

distrib_peak_len(outfile)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
The same problem is also found in the original data, though the values only goes up to 2 500 000... I think that long peaks were improperly called... Looking at the data in detail we can see that most of the peaks are found in the 1 to 130 range. There is also a second smaller peak around 1000. Looking at the original article the mapping was done on the human transcriptome and not the genome. Apparently the coordinates were converted to the genome after and that might explain this decrepancy. I have 2 options => Starting from scratch with a recent genome build, but the dataset seems quite tricky and I am not sure I could do it as well as the original authors. The second option is to be retrictive and keep only the small peaks ie > 1000 pb. I think that I will start by this alternative and go back to the data again if needed. To be sure of the quality of the data I will start from the original data and do the liftover conversion myself.
Can you write Python code for it?

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"
infile="./PTM_Original_Datasets/m6A_Dominissini_hg19_original_table.csv"
outfile = "./PTM_Clean_Datasets/Dominissini_m6A_HepG2_hg19_cleaned.bed"

init_template=[0,"\t",1,"\t",2,"\t",3,"\t",4]
final_template=[0,"\t",1,"\t",2,"\t",modification,"|",cell,"|",method,"|",PMID,"|",4,"\t-\t",3]

# Predicate function to filter out large peaks
predicate = lambda val_list: abs(int(val_list[1])-int(val_list[2])) <= 1000

reformat_table(
    input_file=infile,
    output_file=outfile,
    init_template=init_template,
    final_template=final_template,
    keep_original_header = False,
    header = generate_header(PMID, cell, modification, method),
    replace_internal_space='_',
    replace_null_val="-",
    predicate = predicate)

file_summary(outfile)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
editing_Peng_hg38
Can you write Python code for it?

infile="./PTM_Original_Datasets/editing_Peng_hg38.bed"
PMID = "22327324"
cell = "YH"
modification = "A>I"
method = "A_to_I_editing"
author = "Peng"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)

print(colsum(infile, colrange=[9], header=False, ignore_hashtag_line=True, separator=["\t", "|"], max_items=20, ret_type="report"))


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
pseudoU_Carlile_hg38
Can you write Python code for it?

infile="./PTM_Original_Datasets/pseudoU_Carlile_hg38.bed"
PMID = "25192136"
cell = "HeLa"
modification = "Y"
method = "Pseudo_seq"
author = "Carlile"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

init_template=[0,"\t",1,"\t",2,"\t",3,"|",4,"\t",5,"\t",6]
final_template=[0,"\t",1,"\t",2,"\t",modification,"|",cell,"|",method,"|",PMID,"|",4,"\t",5,"\t",6]

reformat_table(
    input_file=infile,
    output_file=outfile,
    init_template=init_template,
    final_template=final_template,
    keep_original_header = False,
    header = generate_header(PMID, cell, modification, method),
    replace_internal_space='_',
    replace_null_val="-")

file_summary(outfile)

distrib_peak_len(outfile)


I am working on the file "Notebooks/2016_18_08_Epi2_pilot_datasets_analysis_protein_interaction.ipynb".
The first task is:
Convert coordinates from hg19 to hg39 with Crossmap and remap on last gencode human genes...
Coordinate conversion using CrossMap and a hg19 tp hg38 chain file in BASH
Can you write Python code for it?

def liftover (input_file, output_file, unmap_file, liftover_chainfile):
    # Conversion of coordinates with Crossmap/liftover

    with open (output_file, "w") as output, open(unmap_file, "w") as unmap:
        # Build the command line
        cmd = "CrossMap.py bed {} {}".format(liftover_chainfile, input_file)
        # Write Liftovers coordinates
        for line in pycl.bash(cmd, ret_stdout=True).split("\n"):
            ls = line.split("\t")
            try:
                output.write("{}\t{}\t{}\t{}\t{}\t{}\n".format(ls[7], ls[8], ls[9], ls[10], ls[11], ls[12]))
            except IndexError as E:
                unmap.write(line+"\n")

# Test with one file
pycl.mkdir("./Protein_interaction/Li_ClipSeq/test")
liftover_chainfile = "../../LiftOver_chain_files/hg19ToHg38.over.chain.gz"
input_file = "./Protein_interaction/Li_ClipSeq/Human/HHCD1.bed"
output_file = "./Protein_interaction/Li_ClipSeq/test/HHCD1_converted.bed"
unmap_file = "./Protein_interaction/Li_ClipSeq/test/HHCD1_unmap.bed"
liftover(input_file, output_file, unmap_file, liftover_chainfile)
pycl.head(input_file, 3)
pycl.head(output_file, 3)
pycl.head(unmap_file, 3)


I am working on the file "Notebooks/2016_18_08_Epi2_pilot_datasets_analysis_protein_interaction.ipynb".
The first task is:
Hendrickson = fRIP with 24 chromatin related protein
Hendrickson D G, DR Kelley, D Tenen, B Bernstein, JL Rinn. Widespread RNA binding by chromatin-associated proteins.. Genome Biol 17, 28 (2016).

Formaldehyde RNA immunoprecipitation of 24 RPB and chromatin related proteins
Really detaillled and extensive datasets to have transcriptome wide lncRNA interacting with chromatin.
Contain all the list of lncRNA gene significantly enriched in captured factions
Problem = K562 (myelogenous leukemia line) with highly rearranged genomes...

=> The data provided will be hard to analyse and cannnot simply be lifovered
=> I would have to re-analign the fastq files at transcript level, compare bound and unbound fractions, call the peaks...
I am not sure that this is worth to do it

Li  / Starbase =  CLIP-Seq data
JH Li, S Liu, LL Zheng, J Wu, WJ Sun, ZL Wang, H Zhou, LH Qu, JH Yang. Discovery of Protein-lncRNA Interactions by Integrating Large-Scale CLIP-Seq and RNA-Seq Datasets.. Front Bioeng Biotechnol 2, 88 (2014).
JH Li, S Liu, H Zhou, LH Qu, JH Yang. starBase v2.0: decoding miRNA-ceRNA, miRNA-ncRNA and protein-RNA interaction networks from large-scale CLIP-Seq data.. Nucleic Acids Res 42, D92-7 (2014).

Comprehensive collection of lncRNA protein interaction from
Database dates from 2014 => contain datasets up to 2013 = not up to date.
No specific chromatin protein => have to be crossmatched with protein localization => Done from uniprot = few nuclei pure proteins

Cross-match Protein location Uniprot with data protein/lncRNA database from Starbase
Selection of a limited number of candidate CLIP-Seq datasets based on their localization and the experimental conditions:
- Nuclear candidates: ALKBH5, ZC3H7B, FUS, TAF15, hnRNPC, DGCR8, FUS
- Cytoplasmic candidates: AGO1, AGO3, AGO4, FXR1, LIN28B, PUM2
Can you write Python code for it?

df = pd.read_csv("./Protein_interaction/Li_ClipSeq/Protocol_info/Selected_datasets.csv", sep="\t", index_col=0)
df


I am working on the file "Notebooks/2016_18_08_Epi2_pilot_datasets_analysis_protein_interaction.ipynb".
The first task is:
List the uniq gene ids for each samples
for experiment with several samples, the function only retain those in common in ALL the replicates
Can you write Python code for it?

def unique_genes_list (input_file_list, output_file):

    gene_dict = {}

    # iterate on replicates of the same experiment
    for input_file in input_file_list:
        df = pd.read_csv(input_file, sep="[\t|]", engine='python')

        # iterate on lines of a replicate bed file
        for gene_id, gene_df in df.groupby("gene_id"):
            if gene_id not in gene_dict:
                gene_dict[gene_id] = {
                    "gene_name":gene_df["gene_name"].unique()[0],
                    "gene_type":gene_df["gene_type"].unique()[0],
                    "all_count":0, "rep_count":0}
            gene_dict[gene_id]["all_count"]+=len(gene_df)
            gene_dict[gene_id]["rep_count"]+=1

    # Write the valid gene entries in a file
    with open (output_file, "w") as output:
        output.write("gene_id\tgene_name\tgene_type\tall_count\trep_count\n")
        for gene_id, val in gene_dict.items():
            if val["rep_count"] == len(input_file_list):
                output.write("{}\t{}\t{}\t{}\t{}\n".format(gene_id, val["gene_name"], val["gene_type"], val["all_count"], val["rep_count"]))

# Test with one file
input_file = ["./Protein_interaction/Li_ClipSeq/test/HCTA1_annotated.bed"]
output_file = "./Protein_interaction/Li_ClipSeq/test/HCTA1_uniq_gene.tsv"
unique_genes_list(input_file, output_file)
pycl.linerange(output_file)

# Test with a list of files
input_file = [ "./Protein_interaction/Li_ClipSeq/test/HCTA1_annotated.bed", "./Protein_interaction/Li_ClipSeq/test/HHCD1_annotated.bed"]
output_file = "./Protein_interaction/Li_ClipSeq/test/HHMF1-3_uniq_gene.tsv"
unique_genes_list(input_file, output_file)
pycl.linerange(output_file)


I am working on the file "Notebooks/2016_18_08_Epi2_pilot_datasets_analysis_protein_interaction.ipynb".
The first task is:
Merge the replicates of the same experiment
Iterate on all the files listed in the sample list and apply hg38 coordinate conversion, gencodev24 annotation and listing of unique genes
Can you write Python code for it?

liftover_chainfile = "../../LiftOver_chain_files/hg19ToHg38.over.chain.gz"
annotation_file = "../../Reference_Annotation/gencode_v24.gff3"
intermediate_folder = "./Protein_interaction/Li_ClipSeq/selected_datasets/"
final_folder = "./Protein_interaction/Li_ClipSeq/final_datasets/"
pycl.mkdir(intermediate_folder)
pycl.mkdir(final_folder)

# df containing the sample list details
df = pd.read_csv("./Protein_interaction/Li_ClipSeq/Protocol_info/Selected_datasets.csv", sep="\t", index_col=0)

# dict to count the file of the different dataset and experiments
dataset_dict = {}
exp_dict = {}

# Iterate by all sample of the same esperiment (include replicates if there are some replicates)
for (prot,exp, cl, loc, aut), exp_df in df.groupby(["protein","experiment", "cell_line", "localization", "author"]):
    exp_id = "{}_{}_{}_{}_{}".format(prot,exp, cl, loc, aut)
    print("Analysis experiment ", exp_id)
    intersect_file_list=[]

    # Iterate on the replicates of the same experiment
    for dataset_id, line in exp_df.iterrows():

        print("\tAnalysis dataset {}".format(dataset_id))
        input_file = line["filepath"]

        print ("\t\tConverting coordinates with liftover")
        converted_file = "{}{}_coord_hg38.bed".format(intermediate_folder, dataset_id)
        unmap_file = "{}{}_unmap.bed".format(intermediate_folder, dataset_id)
        liftover(input_file, converted_file, unmap_file, liftover_chainfile)

        print ("\t\tIntersect with annotation file")
        intersect_file = "{}{}_coord_hg38_annot_gencode24.bed".format(intermediate_folder, dataset_id)
        intersect_file_list.append (intersect_file)
        intersect_bed (annotation_file, converted_file, intersect_file)

        # Fill the counter
        dataset_dict[dataset_id] = {
            "Init_sites":pycl.simplecount(input_file),
            "Converted_coord_sites": pycl.simplecount(converted_file),
            "Unmap_sites": pycl.simplecount(unmap_file),
            "Annotated_sites":...


I am working on the file "Notebooks/2016_18_08_Epi2_pilot_datasets_analysis_protein_interaction.ipynb".
The first task is:
Meta Analysis of the Datasets for the Epi² pilot project

lncRNA interaction with proteins

PYTHON 3 / R Notebook
Adrien Leger / EMBL EBI
Starting date 18/08/2016

Import general package and definition of specific functions
Can you write Python code for it?

# Standard lib imports
from glob import glob
from os import remove

# Local lib import
import pycl
import pyBioPlot as pbl

# Third party import
import pandas as pd
import numpy as np
import pylab as pl
import seaborn as sns
import scipy.stats as stats

# Pyplot tweaking
%matplotlib inline
pl.rcParams['figure.figsize'] = 30, 10  # that's default image size for this interactive session

# Larger display
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:75% !important; }</style>"))

# Allow to use R directly
%load_ext rpy2.ipython

# Simplify warning reporting to lighten the notebook style
import warnings
warnings.formatwarning=(lambda message, *args: "{}\n".format(message))


I am working on the file "Notebooks/2016_18_08_Epi2_pilot_datasets_analysis_protein_interaction.ipynb".
The first task is:
Remap on the human transcriptome with pyBedTools
Can you write Python code for it?

import pybedtools

def intersect_bed (annotation_file, peak_file, output_file):

    # Parse file with pybedtools and intersect the coordinate ranges
    peak = pybedtools.BedTool(peak_file)
    annotation = pybedtools.BedTool(annotation_file)
    intersection = peak.intersect(annotation, wo=True, s=True)

    # Reformat the file generated by pybedtools with the merged intervals to a simple Bed format
    init_template=["{chrom}","\t","{start}","\t","{end}","\t","{info}","\t","{score}","\t","{strand}","\t","{chrom2}","\t","{source}","\t","{feature}","\t","{start2}",
                   "\t","{end2}","\t","{score2}","\t","{strand2}","\t","{frame}","\tID=","{ID}",";gene_id=","{gene_id}",";gene_type=","{gene_type}",
                   ";gene_status=","{gene_status}",";gene_name=","{gene_name}",";level=","{level}",";havana_gene=","{havana_gene}","\t","{len}"]

    final_template=["{chrom}","\t","{start}","\t","{end}","\t","{gene_id}","|","{gene_name}","|","{gene_type}","|","{len}","\t","{score}","\t","{strand}"]

    pycl.reformat_table(
        input_file=intersection.fn, output_file=output_file, init_template=init_template, final_template=final_template,
        replace_internal_space='_', replace_null_val="-", keep_original_header=False, predicate=lambda v: v["feature"] == "gene", header_from_final_template=True)

# Test with one file
annotation_file = "../../Reference_Annotation/gencode_v24.gff3"
peak_file = "./Protein_interaction/Li_ClipSeq/test/HHCD1_converted.bed"
output_file = "./Protein_interaction/Li_ClipSeq/test/HHCD1_annotated.bed"
intersect_bed (annotation_file, peak_file, output_file)
pycl.linerange(peak_file)
pycl.linerange(output_file)


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Boolean or logical indexing is useful if you need to sort the data based on some True or False value.
For instance, who are the people with salaries greater than 90K but lower than 100K ?
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Combining datasets : pd.concat
Recall that we sliced the dataframes into male and female dataframe in 2.3 Indexing and Slicing. Now we will learn how to put dataframes together which is done by the pd.concat method. Note how the index of this output retains the old index.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Create multiple plots
Can you write Python code for it?

f,axs = plt.subplots(1,2,figsize=(15,5)) # create a plot figure, specify the size and number of figures.
axs[0].scatter(df.age,df.salary,color='k',marker='o')
axs[0].set_ylim([40000,120000])
axs[0].set_xlim([20,70])
axs[0].set_yticklabels([str(int(tick)/1000)+'K' for tick in axs[0].get_yticks()])
axs[0].set_ylabel('salary')
axs[0].set_xlabel('age')
axs[0].set_title('Scatterplot of age and salary')

axs[1].scatter(df.publications,df.salary,color='k',marker='o')
axs[1].set_ylim([40000,120000])
axs[1].set_xlim([20,70])
axs[1].set_yticklabels([str(int(tick)/1000)+'K' for tick in axs[1].get_yticks()])

axs[1].set_ylabel('salary')
axs[1].set_xlabel('publications')
axs[1].set_title('Scatterplot of publication and salary')

f.suptitle('Scatterplots of salary and other factors')

plt.show()


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Dealing with missing values
It is easy to quickly count the number of missing values for each column in the dataset using the isnull() method.  One thing that is  nice about Python is that you can chain commands, which means that the output of one method can be the input into the next method.  This allows us to write intuitive and concise code.  Notice how we take the sum() of all of the null cases.
The isnull() method will return a dataframe with True/False values on whether a datapoint is null or not a number (nan).
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Describing the data
We can use the .describe() method to get a quick summary of the continuous values of the data frame. We will .transpose() the output to make it slightly easier to read.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Heatmap plots
Heatmap plots allow you to visualize matrices such as correlation matrices that show relationships across multiple variables
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Here we can create a new column pubperyear, which is the ratio of the number of papers published per year
Can you write Python code for it?

df['pubperyear'] = df['publications']/df['years']

df.head()


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Indexing and slicing
Indexing in Pandas can be tricky. There are four ways to index: loc, iloc, ix, and explicit indexing(useful for booleans).
First, we will try using .loc.  This method references the explicit index. it works for both index names and also column names.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Let's make a new data frame with just Males and another for just Females. Notice, how we added the .reset_index(drop=True) method?   This is because assigning a new dataframe based on indexing another dataframe will retain the original index.  We need to explicitly tell pandas to reset the index if we want it to start from zero.
Can you write Python code for it?

maledf = df[df.gender==0].reset_index(drop=True)
femaledf = df[df.gender==1].reset_index(drop=True)


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Manipulating data in Groups
One manipulation we often do is look at variables in groups.
One way to do this is to usethe .groupby(key) method.
The key is a column that is used to group the variables together.
For instance, if we want to group the data by gender and get group means, we perform the following.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Matplotlib
Learn other matplotlib tutorials here
create a basic lineplot
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Modify different aspects of the plot
Can you write Python code for it?

# plt.subplots allows you to control different aspects of multiple plots
f,ax = plt.subplots(1,1,figsize=(4,2))
ax.scatter(df.salary,df.age,color='k',marker='o')
# Setting limits on axes
ax.set_xlim([40000,120000])
ax.set_ylim([20,70])
# Changing tick labels
ax.set_xticklabels([str(int(tick)/1000)+'K' for tick in ax.get_xticks()])
# changing label names
ax.set_xlabel('salary')
ax.set_ylabel('age')
# changing the title
ax.set_title('Scatterplot of age and salary')
plt.show()

# save figure
f.savefig('MyFirstPlot.png')


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Next we wil try .iloc.  This method references the implicit python index (starting from 0, exclusive of last number).  You can think of this like row by column indexing using integers.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Now we have a corrected dataframe!
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Now we have salary_in_departm column showing standardized salary per department.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
On the top row, you have column names, that can be called like a dictionary (a dataframe can be essentially thought of as a dictionary with column names as the keys). The left most column (0,1,2,3,4...) is called the index of the dataframe. The default index is sequential integers, but it can be set to anything as long as each row is unique (e.g., subject IDs)
Can you write Python code for it?

print("Indexes")
print(df.index)
print("Columns")
print(df.columns)
print("Columns are like keys of a dictionary")
print(df.keys())


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Plotting Categorical Variables. Replacing variables with .map
If we want to plot department on the x-axis, Pandas plotting functions won't know what to do
because they don't know where to put bio or chem on a numerical x-axis.
Therefore one needs to change them to numerical variable to plot them with basic functionalities (we will later see how Seaborn sovles this).
Can you write Python code for it?

# create a new numericalSeries called dept_num for visualization.
df['dept_num'] = 0
df.loc[:,['dept_num']] = df.departm.map({'bio':0, 'chem':1,'geol':2,'neuro':3,'stat':4,'physics':5,'math':6})
df.tail()

## Now plot all four categories
f, axs = plt.subplots(1, 4, sharey=True)
f.suptitle('Salary in relation to other variables')
df.plot(kind='scatter', x='gender', y='salary', ax=axs[0], figsize=(15, 4))
df.plot(kind='scatter', x='dept_num', y='salary', ax=axs[1])
df.plot(kind='scatter', x='years', y='salary', ax=axs[2])
df.plot(kind='scatter', x='age', y='salary', ax=axs[3])
# The problem is that it treats department as a continuous variable.


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Plotting in pandas
Before we move into Matplotlib, here are a few plotting methods already implemented in Pandas.
Boxplot
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Seaborn
Seaborn is a plotting library built on Matplotlib that has many pre-configured plots that are often used for visualization.
Other great tutorials about seaborn are here
Can you write Python code for it?

ax = sns.regplot(df.age,df.salary)
ax.set_title('Salary and age')
plt.show()

sns.jointplot("age", "salary", data=df, kind='reg');


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
Single bracket returns a Series
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
There is also an older method called .ix, which will likely eventually be phased out of pandas.  It can be useful to combine explicit and implicit indexing.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
We can also get quick summary of a pandas series, or specific column of a pandas dataframe.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
We can chain the .null() and .sum() methods to see how many null values are added up.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
You can also call a column like an attribute if the column name is a string
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
You can create new columns to fit your needs.
For instance you can set initialize a new column with zeros.
Can you write Python code for it?


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
You can use the boolean indexing once again to see the datapoints that have missing values. We chained the method .any() which will check if there are any True values for a given axis.  Axis=0 indicates rows, while Axis=1 indicates columns.  So here we are creating a boolean index for row where any column has a missing value.
Can you write Python code for it?

df[df.isnull().any(axis=1)]

# you may look at where the values are not null
# Note that indexes 18, and 24 are missing.
df[~df.isnull().any(axis=1)]


I am working on the file "Notebooks/2_Introduction_to_Dataframes_&_Plotting.ipynb".
The first task is:
replace original dataframe without the miscoded data
Can you write Python code for it?


I am working on the file "Notebooks/2a. Supervised Learning - Regression Analysis.ipynb".
The first task is:
1. Import dataset
Can you write Python code for it?

#Reading the dataset in a dataframe using Pandas
df = pd.read_csv('../data/master.csv')

#Print first observations
df.head()


I am working on the file "Notebooks/2a. Supervised Learning - Regression Analysis.ipynb".
The first task is:
I was expecting a much lower score for this model compared to the previous one. Population isn't really a good indicator of restaurants' health inspection scores, since the more populous Zip Codes (for instance: 78704) correspond to really diverse neighborhoods with a large variety of establishments of all kinds. This will end up being an important result of this entire project: Austin's areas are really diverse.
Of course, the arguments from above against much faith in this model still stand: we have 35 pairs of data points and we are overfitting.
Can you write Python code for it?

# Visualization using Seaborn:

sns.lmplot(x="Population", y="Score", data=average_scores);

print "The Pearson Correlation coefficient between Population and Average Score is {0}".\
format(np.corrcoef(average_scores['Population'].values, average_scores['Score'].values)[1][0])


I am working on the file "Notebooks/2a. Supervised Learning - Regression Analysis.ipynb".
The first task is:
It is of paramount importance to remember that we are NOT supposed to be extracting any useful information from this regression model we are about to apply. After all, we only have a tiny number of rows, each one corresponding to an entire Zip Code. However we might still be able to create a model that will yield some strong trends which would help us make, if not a prediction, an educated guess about the average score of all restaurants in a Zip Code with given demographic data.
Let's take it one by one:
Predict Score using zip code's Median Income
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
lm = LinearRegression()

X = average_scores['Med_Income'].values
y = average_scores['Score'].values

# Check the shapes of the X and y vectors:

print X.shape
print y.shape

# Reshape to get them to work when we fit the model:

X = X.reshape(34,1);
y = y.reshape(34,1);

# Fit linear regression model:
lm.fit(X, y)

# print intercept and coefficients
print lm.intercept_
print lm.coef_

# And score it:

lm.score(X,y)


I am working on the file "Notebooks/2a. Supervised Learning - Regression Analysis.ipynb".
The first task is:
It is quite interesting that the correlation between the two quantities is negative. Of course this could just mean that the really "small" Zip Codes correspond to rather affluent (and hence, sparsely populated) neighborhoods with very few restaurants that probably score really well in the inspection.
Another thing to take into account is that the slope of this "best fit" line is really minuscule. This means that if our model had any merit, in order for us to predict a failing score of 69, the population of an imaginary Zip Code would have to be almost... 450,000 people!
Can you write Python code for it?

print "For a predicted score: {0} (just below the cutoff), the population would have to be {1}".\
format(lm.predict(450000)[0][0], 450000)


I am working on the file "Notebooks/2a. Supervised Learning - Regression Analysis.ipynb".
The first task is:
Predict Score using zip code's Home Ownership
As above:
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
lm = LinearRegression()

X = average_scores['Home_Ownership'].values
y = average_scores['Score'].values

# Reshape the X and y vectors to get them to work when we fit the model:

X = X.reshape(34,1);
y = y.reshape(34,1);

# Fit linear regression model:
lm.fit(X, y)

# print intercept and coefficients
print lm.intercept_
print lm.coef_

# And score it:

lm.score(X,y)


I am working on the file "Notebooks/2a. Supervised Learning - Regression Analysis.ipynb".
The first task is:
Throughout the Machine Learning part of this project we will be using scikit-learn, an open source machine learning library for the Python programming language.
Linear Regression
a. Multiple Linear Regression in ALL data - bad idea
We are starting with an absurd idea: let us try to create a simple linear regression model to predict a restaurant's score given our numerics as features: the median income, population and home ownership percentage:
Can you write Python code for it?

# There are some NaN values in our numerics (UT Campus and ABIA)
# Let us remove rows from those zip codes from the DataFrame:

df = df[np.isfinite(df['Population'])]

# create X and y
feature_cols = ['Med_Income', 'Population', 'Home_Ownership']
X = df[feature_cols]
y = df.Score

# follow the usual sklearn pattern: import, instantiate, fit
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(X, y)

# print intercept and coefficients
print lm.intercept_
print lm.coef_

# pair the feature names with the coefficients
zip(feature_cols, lm.coef_)

# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % lm.score(X,y))


I am working on the file "Notebooks/2a. Supervised Learning - Regression Analysis.ipynb".
The first task is:
We are explaining 1% of the variance with our model. Definitely nothing to write home about... This was to be expected anyway, since the numerical data we have added pertain to a whole zip code, not each individual restaurant.
Can you write Python code for it?

# Let us check how many rows we are left with after excluding the UT Campus and ABIA areas:

len(df)


I am working on the file "Notebooks/2a. Supervised Learning - Regression Analysis.ipynb".
The first task is:
b. Linear Regression on averaged data
As we have mentioned, the reason why the above model was so terrible is quite straightforward: even though the numerical data we have in our DataFrame (Median Income, Population, Home Ownership Percentage) pertain to a whole Zip Code, we attached them to each individual row - every restaurant was supposed to carry those features. This, of course, is absurd and wrong. However we can still try a weak prediction if instead of using all 15,957 rows we dramatically shrink our dataset to... 34 rows - i.e. one for each Zip Code. So, we will have an average restaurant score which we will be trying to predict, or, how do restaurants in a certain area do on average, given the demographics of its inhabitants.
Can you write Python code for it?

average_scores = df.groupby('Zip_Code').mean()
len(average_scores)

average_scores.head()


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:

Can you write Python code for it?

# Even a random forest comprising 250 trees yields the exact same classifier score:

from sklearn.ensemble import RandomForestClassifier

## DecisionTreeClassifier
dtree = tree.DecisionTreeClassifier(max_depth=None, min_samples_split=1,
       random_state=0)
scores = cross_validation.cross_val_score(dtree, X_test, y_test)
print "DecisionTreeClassifier:", scores.mean()

## RandomForestClassifier
forest = RandomForestClassifier(n_estimators=250,
     max_features=4, random_state=0, max_depth=None)
scores = cross_validation.cross_val_score(forest, X_test, y_test)
print "RandomForestClassifier:", scores.mean()


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
1. Import dataset
Can you write Python code for it?

#Reading the dataset in a dataframe using Pandas
df = pd.read_csv("../data/master.csv")

#Print first observations
df.head()


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
A score of 91% on 35 rows of data isn't exciting; it is overfitting. Even though we only have few points, we can attempt train/test split and cross validation:
Can you write Python code for it?

# Cross-validation:

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, train_size=0.8)

clf.fit(X_train, y_train)
print clf.score(X_test, y_test)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
According to our Random Forest model, the most important feature is a restaurant's location in Northeast Austin. However, given the terrible performance of the classifier in actually finding any passing restaurants, we can safely ignore this result.
Support Vector Classifier
Our final attempt before we give up trying to classify restaurants given the (ad hoc) area of town they are located in is a support vector machine classifier:
Can you write Python code for it?

from sklearn import svm

svc = svm.SVC(kernel='linear')

X = data[ feature_cols ].values
y = data['Pristine'].values

svc.fit(X, y)

from matplotlib.colors import ListedColormap
# Create color maps for 2-class classification problem
cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])

def plot_estimator(estimator, X, y):
    estimator.fit(X, y)
    x_min, x_max = X[:, 0].min() - .1, X[:, 0].max() + .1
    y_min, y_max = X[:, 1].min() - .1, X[:, 1].max() + .1
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 100),
                         np.linspace(y_min, y_max, 100))
    Z = estimator.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.figure()
    plt.pcolormesh(xx, yy, Z, cmap=cmap_light)

    # Plot also the training points
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=cmap_bold)
    plt.axis('tight')
    plt.axis('off')
    plt.tight_layout()

X_ = data[['Area_NW Austin', 'Area_SE Austin']].values # picking two "important" features
y_ = data['Score'].values
plot_estimator(svc, X_, y_)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
Decision Trees and Random Forests give us some interesting information concerning the relative importance of their estimators, and we can visualize those importances:
Can you write Python code for it?

importances = forest.fit(X, y).feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(X.shape[1]):
    print("{0}. feature {1} ({2})".format(f + 1, feature_cols[f], importances[indices[f]]))

# Plot the feature importances of the forest

import matplotlib.pyplot as plt
%matplotlib inline

plt.figure()
plt.title("Feature importances")
plt.bar(range(X.shape[1]), importances[indices],
       color="r", yerr=std[indices], align="center")
plt.xticks(range(X.shape[1]), indices)
plt.xlim([-1, X.shape[1]])
plt.show()


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
Even though our goal has changed (we are seeking to predict whether a restaurant will ace the health inspection) the results of our model are still disappointing. It is becoming more and more evident that Area isn't a good predictor.
Our final attempt will be to "strong-arm" our model by manually changing the logistic regression classification threshold from 0.5 to something much higher like 0.8
Hopefully this will eradicate some of the misclassifications. In order to manipulate our model like that, we will use statsmodels instead of scikit-learn:
Can you write Python code for it?

import statsmodels.api as sm

logit = sm.Logit(data['Pristine'], data[ feature_cols ])

# fit the model

model_result = logit.fit()

predicted = model_result.predict(data[ feature_cols ])

thresholds = [0.5, 0.6, 0.7, 0.8, 0.9]

for i in thresholds:
    predicted_choice = (predicted > i).astype(int)
    print "The number of restaurants predicted to pass inspection (grade A) with logistic threshold = {0} is {1}".\
    format(i, sum(predicted_choice) )


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
Let us check those DataFrames before we merge them into df:
Can you write Python code for it?

area_dummies.head()

outcome_dummies.head()

letter_dummies.head()


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
Raising the logistic regression classifier threshold to anything higher than 60% results in zero restaurants predicted to pass inspection with a grade "A", given the area.
Can you write Python code for it?


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
The 0.506 ROC score is fairly low. It is telling us that our model is doing slightly better in predicting a restaurant's inspection outcome (pass/fail) than a coin toss (which would have a 0.5 probability)
Can you write Python code for it?

# Confusion Matrix and Classification Report:

# Confusion Matrix

cm = metrics.confusion_matrix(y_test, logm.predict(X_test), labels=[1,0]) #labels = [1,0] will change the order: T,F

cm

print metrics.confusion_matrix(y_test, predicted, labels=[1,0])
print metrics.classification_report(y_test, predicted,labels=[1,0])

# Visualize Confusion Matrix:

def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Blues):
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

plot_confusion_matrix(cm)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
The coefficients above aren't necessarily significant but they show associations with increased/decreased likelihood of passing the health inspection set. As a matter of fact, a restaurant being located in Northwest Austin is more likely to pass the test whereas a restaurant in Southeast Austin is less likely to do so.
The real test of a good model is to train the model on a training set and then test it with data that it has not fitted. Here’s where the rubber meets the road.
Can you write Python code for it?

from sklearn import cross_validation

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, train_size=0.8)

logm.fit(X_train,y_train)

# predict class labels for the test set
predicted = logm.predict(X_test)

print predicted

# generate class probabilities
probs = logm.predict_proba(X_test)

print probs

logm.score(X_test, y_test)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
The results are no better than before, and possibly more disappointing. It seems that the part of town a restaurant is located in is a terrible predictor of its Health Inspection broad outcome. (Pass/Fail)
From now on I am changing my approach to something more drastic. Instead of aiming to classify health inspection outcomes as "Passes" or "Fails", I will implement a different kind of classification: A resulting letter grade of "A" (which implies a Health Inspection score of 90 or higher) vs. everything else (Scores 89 and below)
Before throwing my hands up in the air with the use of Area as input features for my models, I will try once more:
Can you write Python code for it?

num_A = data[data["Letter_Grade"]=='A'].count()
num_other = data[data["Letter_Grade"]!='A'].count()

print "Pristine Restaurants (90 or above) = {0}".format(num_A[0])
print "Restaurants scoring 89 or below = {0}".format(num_other[0])


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
There is a much nicer distribution between the classes now! Let us construct two new dummy variables, one for "Pristine" restaurants (the same ones obtaining a grade "A", which has been the redundant dummy variable in my previous treatment) and one for everything else:
Can you write Python code for it?

# I need the "Grade_A" column:

def pristine(x):
    if x >= 90:
        return 1
    else:
        return 0

data['Pristine'] = data['Score'].apply(pristine)
data.head(2)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
There really isn't any reason for us to delve further into that model because the misclassification rate is really substantial and as we mentioned it's not really doing significantly better than a coin toss.
Let us now quickly approach a different strategy to mitigate the results of class imbalance: oversampling the minority class (failing restaurants)
Can you write Python code for it?

#Grab DataFramea rows where column has certain values

value_list_f = ['Fail']
failures = data[data.Status.isin(value_list_f)]

value_list_p = ['Pass']
passes = data[data.Status.isin(value_list_p)]

# Let us pick two random samples of 100 rows each from the "failures" and "passes" dataframes:

rows_f = np.random.choice(failures.index.values, 100)
rows_p = np.random.choice(passes.index.values, 100)

# Convert to dataframes:

sampled_f = failures.ix[rows_f]
sampled_p = passes.ix[rows_p]

# Concatenate into "training" dataframe:

training_df = pd.concat([sampled_p, sampled_f], axis=0)
print "The training DataFrame comprises {0} rows".format(len(training_df))

# Train our logistic regression model on this data:

logm = linear_model.LogisticRegression() # We don't need balanced class weights anymore

feature_cols = ['Area_NE Austin', 'Area_NW Austin', 'Area_SE Austin', 'Area_SW Austin']

X_tr = training_df[ feature_cols ].values
y_tr = training_df['Status_Pass'].values

logm.fit(X_tr, y_tr)

print "The model's score on the training data is {0}%".format(logm.score(X_tr, y_tr)*100)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
This doesn't sound very promising. Regardless, let us construct the testing DataFrame (comprising all rows that aren't already seen by the model)
Can you write Python code for it?

# Create a DataFrame with all rows that are in data but NOT in training_df:

df1 = data
df2 = training_df

common_cols = list(set(df1.columns) & set(df2.columns))

testing_df = pd.merge(df1,df2,on=common_cols,how="outer",indicator=True)

testing_df = testing_df.loc[testing_df['_merge'] == 'left_only']

X_ts = testing_df[ feature_cols ].values
y_ts = testing_df['Status_Pass'].values

# Model Validation:

# predict class labels for the test set
predicted = logm.predict(X_ts)

# generate class probabilities
probs = logm.predict_proba(X_ts)

# Evaluation Metrics:

print metrics.accuracy_score(y_ts, predicted)
print metrics.roc_auc_score(y_ts, probs[:, 1])


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
This tells us how a 1 unit increase or decrease in a variable affects the odds of scoring an "A". For example, we can expect the odds of scoring an "A" to increase by about 212% if the restaurant is located in Northwest Austin. However we should be really cautious and only perceive those odds ratios relative to each other.
Logistic Regression outcome:
Regardless of the model used, the overall prediction accuracy for the city is close to 60%, which is almost equal to random guessing and therefore implies that the area is probably not a useful factor.
The poor performance of this factor is likely due to the fact that each area is very large and consists of many diverse neighborhoods (zip codes), which means there is too much variation within each area.
k - Nearest Neighbors Classification on average scores
The greatest plight of our DataFrame and our attempts to apply supervised learning techniques to predict the inspection score is the absence of continuous (or categorical) features specific to each restaurant. Let us attempt a different binary classifier, k Nearest Neighbors, for the case where our DataFrame comprises the averaged scores of each Zip Code. Again, we need to be overly cautious because we only have 35 rows now:
Can you write Python code for it?

average_scores = data.groupby('Zip_Code').mean()
len(average_scores)

average_scores.head()


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
Visualizing the decision boundaries of our SVC yield a terrible picture, and I assume it is because of the categorical nature of both the features and the output. Let us score it instead:
Can you write Python code for it?

print X.shape
print y.shape

svc.score(X,y)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
We get a great overview of the coefficients of the model, how well those coefficients fit, the overall fit quality, and several other statistical measures.
The result object also lets us to isolate and inspect parts of the model output. The confidence interval gives an idea for how robust the coefficients of the model are. Once again, we might not be getting anything really important from this classifier, but we can definitely see the "trend" of restaurants being located in my ad hoc "Northwest Austin" area being more prone to do better.
Can you write Python code for it?

# Odds ratio:

print np.exp(model_result.params)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
We have mislabeled 6022 out of the 16113 points, yet the classifier's score is equal to zero. (?)
Let us attempt a train/test split and cross-validation:
Can you write Python code for it?

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=.25)
clf.set_params(min_samples_leaf=4)
clf.set_params(max_depth=3)
clf.fit(X_train, y_train)

print metrics.confusion_matrix(y_train, clf.predict(X_train))
print metrics.classification_report(y_train, clf.predict(X_train))
print "The score of the Decision Tree classifier on the training data is {0}".format(clf.score(X_train,y_train))
print ""
print metrics.confusion_matrix(y_test, clf.predict(X_test))
print metrics.classification_report(y_test, clf.predict(X_test))
print "The score of the Decision Tree classifier on the test data is {0}".format(clf.score(X_test,y_test))


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
We will need to create another dummy column for "Pristine" since averaging removed the 1-0 dichotomy. We do get some interesting average scores for each zip code though.
Can you write Python code for it?

def dummy_pristine(x):
    if x >= 0.50:
        return 1
    else:
        return 0

average_scores['Pristine_Dummy'] = average_scores['Pristine'].apply(dummy_pristine)
average_scores.head()

from matplotlib.colors import ListedColormap
from sklearn import feature_selection
from sklearn.neighbors import KNeighborsClassifier as KNN

feature_cols = ['Area_NE Austin', 'Area_NW Austin', 'Area_SE Austin', 'Area_SW Austin']

X = average_scores[ feature_cols ].values
y = average_scores['Pristine_Dummy'].values

clf = KNN(3)
clf.fit(X, y)
print clf.score(X, y)


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
With 22 columns, the DataFrame has already become very hard to visualize, therefore we need to be very careful with our operations from now on.
Logistic Regression
We will at first present an ill-fated approach in order to illustrate the workings of the logistic regression algorithm on our dataset as well as to discuss the evident class imbalance problem and attempt some techniques to mitigate its effects. A naive first approach was to use some of the features in order to predict whether a restaurant will pass the Health Inspection (Score > 69) or not.
The issue that makes this problem almost impossible to deal with:
Can you write Python code for it?

num_pass = data[data["Status"]=='Pass'].count()
num_fail = data[data["Status"]=='Fail'].count()

print "Passing Inspections = {0}".format(num_pass[0])
print "Failing Inspections = {0}".format(num_fail[0])


I am working on the file "Notebooks/2b. Supervised Learning - Classification.ipynb".
The first task is:
k-fold Cross Validation has not really helped us improve the score of our classifier.
Even though we are far from creating a foolproof model, we can definitely discern some interesting trends.
Can you write Python code for it?

# Evaluation Metrics

from sklearn import metrics

print metrics.accuracy_score(y_test, predicted)
print metrics.roc_auc_score(y_test, probs[:, 1])


I am working on the file "Notebooks/2c. Supervised Learning - Text Mining for Classification.ipynb".
The first task is:
1. Import dataset
Can you write Python code for it?

#Reading the dataset in a dataframe using Pandas
df = pd.read_csv("../data/data.csv")

#Print first observations
df.head()


I am working on the file "Notebooks/2c. Supervised Learning - Text Mining for Classification.ipynb".
The first task is:
By default the decision threshold is 0.5: if we vary the decision threshold from 0 to 1 we could generate a family of binary classifier models that address all the possible trade offs between false positive and false negative prediction errors.
Let us use a pipeline in order to perform 10-fold cross validation:
Can you write Python code for it?

pipeline = Pipeline((
    ('vec', CountVectorizer(max_df = 0.8, ngram_range = (1, 2))),
    ('clf', MultinomialNB(alpha = 0.01)),
))
_ = pipeline.fit(df['Names'], df['Letter_Grade'])

from sklearn.cross_validation import cross_val_score
from scipy.stats import sem

scores = cross_val_score(pipeline, df['Names'],
                         df['Letter_Grade'], cv=10)
scores.mean(), sem(scores)

vec_name, vec = pipeline.steps[0]
clf_name, clf = pipeline.steps[1]

feature_names = vec.get_feature_names()
target_names = df['Letter_Grade'].unique()

feature_weights = clf.coef_

feature_weights.shape

len(feature_names)

def print_top10(vectorizer, clf, class_labels):
    """Prints features with the highest coefficient values, per class"""
    feature_names = vectorizer.get_feature_names()
    for i, class_label in enumerate(class_labels):
        top10 = np.argsort(clf.coef_[i])[-10:]
        print("%s: %s" % (class_label,
              " ".join(feature_names[j] for j in top10)))

print_top10(vectorizer, classifier, target_names)

from sklearn.metrics import classification_report

predicted = pipeline.predict(df['Restaurant_Name'])

print(classification_report(df['Letter_Grade'], predicted,
                            target_names=df['Letter_Grade'].unique()))

from sklearn.metrics import confusion_matrix

pd.DataFrame(confusion_matrix(df['Letter_Grade'], predicted),
             index = pd.MultiIndex.from_product([['actual'], target_names]),
             columns = pd.MultiIndex.from_product([['predicted'], target_names]))


I am working on the file "Notebooks/2c. Supervised Learning - Text Mining for Classification.ipynb".
The first task is:
It seems our Multinomial Naive Bayes classifier does significantly better on predicting a restaurant's status (whether it has gotten a "pristine" score" or not) given the restaurant's name than what we have seen so far with the area of town division.
Can you write Python code for it?

# Some information about our Bag of Words feature vector:

len(X_train.data)

n_samples, n_features = X_train.shape

n_samples

n_features

# The vocabulary of our vectorizer, i.e. the unique words comprising it:

len(vectorizer.vocabulary_)

vectorizer.get_feature_names()[n_features / 3:n_features / 3 + 10]

target_predicted_proba = classifier.predict_proba(X_test)
percentages = pd.DataFrame(target_predicted_proba, columns=df['Letter_Grade'].unique())

# A table of probabilities for each one of the 3223 restaurants in the test set to be assigned a certain letter grade:

percentages.head()

len(percentages)


I am working on the file "Notebooks/2c. Supervised Learning - Text Mining for Classification.ipynb".
The first task is:
String Manipulation: Restaurant Names
Let us start our manipulation of restaurant names:
Can you write Python code for it?


I am working on the file "Notebooks/2c. Supervised Learning - Text Mining for Classification.ipynb".
The first task is:
We will remove all words that are 3 characters long or smaller:
Can you write Python code for it?

import re
shortword = re.compile(r'\W*\b\w{1,3}\b')

for i in range(len(Names)):
    Names[i] = shortword.sub('', Names[i])

# As an example, "JR's Tacos" is now just " Tacos"

Names[3]

# Add a new column into our DataFrame:

df['Names'] = Names

df['Names'].head(10)

df.columns


I am working on the file "Notebooks/3-Putting it all together.ipynb".
The first task is:
Box-Plots
Box plots can be generated in a similar manner to the hist() function.
Can you write Python code for it?

props_df.boxplot(column="Area", by="CellType")
props_df.boxplot(column="MajAxisLength", by="CellType")


I am working on the file "Notebooks/3-Putting it all together.ipynb".
The first task is:
We can peek into what the dataframe looks like with the head function. The n=5 specifies that the first 5 rows should be shown.
Can you write Python code for it?


I am working on the file "Notebooks/3. Dimensionality Reduction.ipynb".
The first task is:
1. Import dataset
Can you write Python code for it?

#Reading the dataset in a dataframe using Pandas
df = pd.read_csv("data.csv")

#Print first observations
df.head()

df.columns


I am working on the file "Notebooks/3. Dimensionality Reduction.ipynb".
The first task is:
66.2% of the variance is explained through our model. This is quite respectable.
Can you write Python code for it?

plt.figure(figsize=(10, 7))
plt.bar(range(300), svd_sparta.explained_variance_)

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer
from sklearn import cross_validation
from sklearn.naive_bayes import MultinomialNB

# Fit a classifier on the training set

classifier = MultinomialNB().fit(np.absolute(X_train_svd_sparta), y_train)
print("Training score: {0:.1f}%".format(
    classifier.score(X_train_svd_sparta, y_train) * 100))

# Evaluate the classifier on the testing set

print("Testing score: {0:.1f}%".format(
    classifier.score(X_test_svd_sparta, y_test) * 100))


I am working on the file "Notebooks/3. Dimensionality Reduction.ipynb".
The first task is:
Even though we do not have more features (3430) than rows of data (14888), we can still attempt to reduce the feature space by using Truncated SVD:
Truncated Singular Value Decomposition  for Dimensionality Reduction
Once we have extracted a vector representation of the data, it's a good idea to project the data on the first 2D of a Singular Value Decomposition (i.e.. Principal Component Analysis) to get a feel of the data. Note that the TruncatedSVD class can accept scipy.sparse matrices as input (as an alternative to numpy arrays). We will use it to visualize the first two principal components of the vectorized dataset.
Can you write Python code for it?

from sklearn.decomposition import TruncatedSVD

svd_two = TruncatedSVD(n_components=2, random_state=42)

X_train_svd = svd_two.fit_transform(X_train)

pc_df = pd.DataFrame(X_train_svd)  # cast resulting matrix as a data frame
sns.pairplot(pc_df, diag_kind='kde');

# Percentage of variance explained for each component

def pca_summary(pca):
    return pd.DataFrame([np.sqrt(pca.explained_variance_),
              pca.explained_variance_ratio_,
              pca.explained_variance_ratio_.cumsum()],
             index = ["Standard deviation", "Proportion of Variance", "Cumulative Proportion"],
             columns = (map("PC{}".format, range(1, len(pca.components_)+1))))

pca_summary(svd_two)

# Only 3.5% of the variance is explained in the data
svd_two.explained_variance_ratio_.sum()

from itertools import cycle

def plot_PCA_2D(data, target, target_names):
    colors = cycle('rgbcmykw')
    target_ids = range(len(target_names))
    plt.figure()
    for i, c, label in zip(target_ids, colors, target_names):
        plt.scatter(data[target == i, 0], data[target == i, 1],
                   c=c, label=label)
    plt.legend()

plot_PCA_2D(X_train_svd, y_train, target_names)


I am working on the file "Notebooks/3. Dimensionality Reduction.ipynb".
The first task is:
Is it worth it to keep adding dimensions? Recall that we started with a 3430-dimensional feature space which we have already reduced to 100 dimensions, and according to the graph above each dimension over the 100th one will be adding less than 0.5% in our explanation of the variance. Let us try once more with 300 dimensions, to see if we can get something respectably over 50% (so we can be sure we are doing better than a coin toss)
Can you write Python code for it?

svd_sparta = TruncatedSVD(n_components=300, random_state=42)

X_train_svd_sparta = svd_sparta.fit_transform(X_train)

X_test_svd_sparta = svd_sparta.fit_transform(X_test)

svd_sparta.explained_variance_ratio_.sum()


I am working on the file "Notebooks/3. Dimensionality Reduction.ipynb".
The first task is:
Our first collection of feature vectors will come from the Restaurant_Name column. We are still trying to predict whether a restaurant falls under the "pristine" category (Grade A, score greater than 90) or not. We could also try to see whether we could predict a restaurant's grade (A, B, C or F)
2. Dimensionality Reduction Techniques
Restaurant Names as a Bag-of-words model
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer

# Turn the text documents into vectors

vectorizer = CountVectorizer(min_df=1, stop_words="english")

X = vectorizer.fit_transform(df['Restaurant_Name']).toarray()
y = df['Letter_Grade']

target_names = y.unique()

# Train/Test split and cross validation:

from sklearn import cross_validation

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, train_size = 0.8)

X_train.shape


I am working on the file "Notebooks/3. Dimensionality Reduction.ipynb".
The first task is:
This must be the most uninformative plot in the history of plots. Obviously 2 principal components aren't enough. Let's try with 100:
Can you write Python code for it?

# Now, let's try with 100 components to see how much it explains
svd_hundred = TruncatedSVD(n_components=100, random_state=42)
X_train_svd_hundred = svd_hundred.fit_transform(X_train)

# 43.7% of the variance is explained in the data for 100 dimensions
# This is mostly due to the High dimension of data and sparcity of the data
svd_hundred.explained_variance_ratio_.sum()

plt.figure(figsize=(10, 7))
plt.bar(range(100), svd_hundred.explained_variance_)


I am working on the file "Notebooks/3D coords.ipynb".
The first task is:
${}^\mathcal{B}C^{\mathcal{I}} ={}^\mathcal{B}C^{\mathcal{C}}{}^\mathcal{C}C^{\mathcal{A}}{}^\mathcal{A}C^{\mathcal{I}} $
Can you write Python code for it?


I am working on the file "Notebooks/3D coords.ipynb".
The first task is:
3-2-3 $(\psi,\theta,\phi)^{\mathcal{I}}_{\mathcal{B}}$ Euler Angles - $\psi$ rotation about $\mathbf{e}_1$ followed by a $\theta$ rotation about $\mathbf{a}_2$ followed by a $\phi$ rotation about $\mathbf{c}_3$
Can you write Python code for it?


I am working on the file "Notebooks/3D coords.ipynb".
The first task is:
Defining ${}^{\mathcal{I}}\boldsymbol{\omega}^\mathcal{B} =  \omega_1 \mathbf{b}_1 + \omega_2 \mathbf{b}_2 + \omega_3 \mathbf{b}_3$ And solving for $\dot\psi,\dot\theta,\dot\phi$ in terms of $\omega_1,\omega_2\omega_3$:
Can you write Python code for it?


I am working on the file "Notebooks/3D coords.ipynb".
The first task is:
Spherical Frame - $\theta$ rotation about $\mathbf{e}_3$ followed by a $\phi$ rotation about $\mathbf{c}_2$
Can you write Python code for it?


I am working on the file "Notebooks/3_Introduction_to_Regression.ipynb".
The first task is:
Confidence in our Model
Now let's examine the output. To learn more about Statsmodels and how to interpret the output, DataRobot has some decent posts on simple linear regression and multiple linear regression.
Question: Is linear regression a high bias/low variance model, or a low bias/high variance model?
Answer: High bias/low variance. Under repeated sampling, the line will stay roughly in the same place (low variance), but the average of those models won't do a great job capturing the true relationship (high bias). Note that low variance is a useful characteristic when you don't have a lot of training data!
A closely related concept is confidence intervals. Statsmodels calculates 95% confidence intervals for our model coefficients, which are interpreted as follows: If the population from which this sample was drawn was sampled 100 times, approximately 95 of those confidence intervals would contain the "true" coefficient.
Hypothesis Testing and p-values
Closely related to confidence intervals is hypothesis testing. Generally speaking, you start with a null hypothesis and an alternative hypothesis (that is opposite the null). Then, you check whether the data supports rejecting the null hypothesis or failing to reject the null hypothesis.
(Note that "failing to reject" the null is not the same as "accepting" the null hypothesis. The alternative hypothesis may indeed be true, except that you just don't have enough data to show that.)
As it relates to model coefficients, here is the conventional hypothesis test:
- null hypothesis: There is no relationship between gender and salary (and thus $\beta_1$ equals zero)
- alternative hypothesis: There is a relationship between TV ads and Sales (and thus $\beta_1$ is not equal to zero)
How do we test this hypothesis? Intuitively, we reject the null (and thus believe the alternative) if the 95% confidence interval does not include zero. Conversely, the p-value represents the probability that the coefficient is actually zero:
Overall Model Fit
The overall goodness of...
Can you write Python code for it?

lm2 = smf.ols(formula = "salary ~ gender + years",data=df).fit()
print(lm2.summary())


I am working on the file "Notebooks/3_Introduction_to_Regression.ipynb".
The first task is:
Here we see that years also significantly explains additional variance.
This is reflected in the $R^2$, which has now increased from 0.09 to 0.143. It is important to note that $R^2$ will always increase as you add more features to the model, even if they are unrelated to the response. Thus, selecting the model with the highest R-squared is not a reliable approach for choosing the best linear model.
There is alternative to $R^2$ called adjusted $R^2$ that penalizes model complexity (to control for overfitting), but it generally under-penalizes complexity.
Next week we will explore another approach known as Cross-validation. Importantly, cross-validation can be applied to any model, whereas the methods described above only apply to linear models.
Here, we will try and see if the new model significantly explains additional variance controlling for the extra free parameter using a nested model comparison.
Can you write Python code for it?


I am working on the file "Notebooks/3_Introduction_to_Regression.ipynb".
The first task is:
Interpreting Model Coefficients
How do we interpret the estimated coefficients for this model?
- $\beta_0$, or the intercept, reflects the average salary for the reference condition of the dummy code (i.e., when gender = 0).  This means that males make an average of $69108.5
- to get the estimated female salary we need to add $\beta_1$ (-13388.83), which becomes $55719.67
We can also get a summary of all of the model statistics using the summary() method.
Can you write Python code for it?


I am working on the file "Notebooks/3_Introduction_to_Regression.ipynb".
The first task is:
Linear Regression
Now let's move on to running an analysis.
Simple linear regression is an approach for predicting a quantitative response using a single feature (or "predictor" or "input variable"). It takes the following form:
$y = \beta_0 + \beta_1\cdot x $
What does each term represent?
- $y$ is the response
- $x$ is the feature
- $\beta_0$ is the intercept
- $\beta_1$ is the coefficient for x
Together, $\beta_0$ and $\beta_1$ are called the model coefficients. To create your model, you must "estimate" the values of these coefficients. And once we've estimated these parameters, we can use the model to predict things!
Estimating Model Coefficients
Generally speaking, coefficients are estimated using the least squares criterion, which means we are find the line (mathematically) which minimizes the sum of squared residuals (or "sum of squared errors"):

What elements are present in the diagram?
- The black dots are the observed values of x and y.
- The blue line is our least squares line.
- The red lines are the residuals, which are the distances between the observed values and the least squares line.
How do the model coefficients relate to the least squares line?
- $\beta_0$ is the intercept (the value of $y$ when $x$=0)
- $\beta_1$ is the slope (the change in $y$ divided by change in $x$)
Here is a graphical depiction of those calculations:

To run a regression we will be using the statsmodels module that was loaded above.  We will first initalize a model object with the regression equation and the data to use.  The format for specifying the regression equation is similar to R. (y ~ x).
Here we will estimate the effect of gender on salary.  Gender is a "dummy variable", meaning that it is only ones and zeros.
After we have initialized the object instance we can run the fit method (this can be chained so you only need to run one line).  After the parameters have been estimated we can query attributes of the model such as...
Can you write Python code for it?

lm = smf.ols(formula = "salary ~ gender",data=df).fit()

# print the coefficients
print(lm.params)
print(lm.params[1])


I am working on the file "Notebooks/3_Introduction_to_Regression.ipynb".
The first task is:
Notice how when we rotate the 3D plot, we can make projections into 2 dimensional space.  This is the effect of each predictor on salary controlling for the other.
We can also generate additional diagnostic plots.
Can you write Python code for it?

f = sm.graphics.plot_regress_exog(lm2, 'years')
f.set_size_inches(10,5)


I am working on the file "Notebooks/3_Introduction_to_Regression.ipynb".
The first task is:
We see that the addition of years in model2 significantly explains additional variance when compared to model 1. Notice that this is similar to the results provided by the t-tests on the individual parameters. Using model comparison to test incrementally adding a single predictor is a feature selection method known as stepwise regression.  However, you can also perform model comparison on models with many different features as long as one is nested within the other.
When we only had one predictor we could investigate the relationship using a scatterplot.  Now that we have 2 predictors, we need to make a 3 dimensional plot to see the effect of each predictor on salary.
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(18,6))
idx = [131,132,133]
pos = {0:(-35,20),1:(-90,0),2:(0,0)}
for i,ii in enumerate(idx):
    ax = fig.add_subplot(ii, projection='3d')

    # generate a mesh
    x_surf = np.arange(0,11)/10.0
    y_surf = np.arange(0,35,3)
    x_surf, y_surf = np.meshgrid(x_surf, y_surf)

    exog = pd.core.frame.DataFrame({'gender': x_surf.ravel(), 'years': y_surf.ravel()})
    exog['intercept'] = np.ones(exog.shape[0])
    out = lm2.predict(exog = exog,transform=True)

    ax.plot_surface(x_surf, y_surf,
                out.reshape(x_surf.shape),
                rstride=1,
                cstride=1,
                color='None',
                alpha = 0.2)

    ax.scatter(df['gender'], df['years'], df['salary'],
           c='blue',
           marker='o',
           alpha=1)

    ax.set_xlabel('Gender')
    ax.set_ylabel('Years')
    ax.set_zlabel('Salary')
    ax.azim = pos[i][0]
    ax.elev = pos[i][1]

plt.tight_layout()


I am working on the file "Notebooks/3_Introduction_to_Regression.ipynb".
The first task is:
We will load the data which is a comma delimited text file using the read_csv() method from pandas.
The '../Data' indicates that we will move one folder up and then into the Data folder.
Can you write Python code for it?

# Load the data
df = pd.read_csv('../Data/salary.csv')

df = df.dropna()
df = df[df.gender!=2]

print 'There are %i rows and %i columns in this data set' % df.shape
df.isnull().sum()

df[['salary','years']].plot(kind='scatter', x='years', y='salary')

# Oneshot visualization
## create a new numericalSeries called dept_num !! Just for visualization !!
df['dept_num'] = df.departm.map({'bio':0, 'chem':1,'geol':2,'neuro':3,'stat':4,'physics':5,'math':6})
df.tail()

## Now plot all four categories
fig, axs = plt.subplots(1, 4, sharey=True)
df.plot(kind='scatter', x='gender', y='salary', ax=axs[0], figsize=(16, 4))
df.plot(kind='scatter', x='dept_num', y='salary', ax=axs[1])
df.plot(kind='scatter', x='years', y='salary', ax=axs[2])
df.plot(kind='scatter', x='age', y='salary', ax=axs[3])
# The problem is that it treats department as a continuous variable.


I am working on the file "Notebooks/4_Mediation_&_Moderation.ipynb".
The first task is:
2) Test effect of X on M
Can you write Python code for it?

lm2 = smf.OLS(m,X).fit()
print lm2.summary()
ea = lm2.params[1] # Save the effect of X on M, a, to ea
sea = lm2.bse[1]


I am working on the file "Notebooks/4_Mediation_&_Moderation.ipynb".
The first task is:
3) Test effect of  X and M on Y
Can you write Python code for it?

X['M'] = m
lm3 = smf.OLS(y,X).fit()
print lm3.summary()
ecq,eb = lm3.params[1:3]
seb = lm3.bse[2]


I am working on the file "Notebooks/4_Mediation_&_Moderation.ipynb".
The first task is:
Moderation Analysis
In a moderation analysis, the moderator modifies or changes the relationship between two variables, akin to an interaction term. Moderation is slightly different from an interaction due to the additional constraint that there is a causal relationship from X to Y, BUT not from Z to Y. Therefore, a moderation implies an interaction exists but an interaction does not imply a moderation.
Here is a schematic representation of a moderation relationship.
This diagram hypothesize that Stress has a causal relationship to Depression
but the effect of Stress is different for people with high or low Social Support

This can be reprsented by an interaction,

The pictures have been retrieved from here
Here are a few examples of moderations. Can you think of more?
1) The effect of compliments on future grades is moderated by growth mindset (Carol Dweck)
2) Effect of favorability on government behavior is moderated by political affiliation.
3) Effect of pressure on performance is moderated by confidence (choking vs boosting).
For more information look at homepage of Kenny who started all this.
Example
Here we examine whether the effect of buying books (Buy) on enjoyment of reading (Enjoy) is moderated by frequency of reading (Read).
The moderation effect exists if there is an interaction of buying and reading on enjoyment.
Can you write Python code for it?


I am working on the file "Notebooks/4_Mediation_&_Moderation.ipynb".
The first task is:
Show how the effect is broken down to direct and indirect effects
Recall how the overall effect C was decomposed to indirect effect (a*b) and direct effect (c')
$c = a \cdot b + c' $
Can you write Python code for it?

print('c : %.2f') % ec
print('a : %.2f') % ea
print('b : %.2f') % eb
print('c\' : %.2f') % ecq

print('Total effect C: %.2f') % ec
print('is decomposed into the indirect(mediated) effect a*b: %.2f') % (ea*eb)
print('plus the direct effect c\': %.2f') % ecq
print('which adds up to %.2f') % (ea*eb+ecq)


I am working on the file "Notebooks/4_siteCanonicality.ipynb".
The first task is:
At short lengths the percent at non-canonical sites is fairly high - this elvels off at ~200 bp
Poss that <200bp utrons are less likely to be real
Below - outputting lists of utron ids which are known / unknown at various lengths
Can you write Python code for it?

def writeFiles(Length):
    a = spliceInfo[spliceInfo["Length"]<=Length]
    b1 = a[a["unknown"]==1].ix[:,0]
    b2 = a[a["unknown"]!=1].ix[:,0]

    unknownFile = "/shared/sudlab1/General/projects/utrons_project/misc_files/SpliceSite/novelUtrons_unknown_%dbp.txt" % Length
    knownFile = "/shared/sudlab1/General/projects/utrons_project/misc_files/SpliceSite/novelUtrons_known_%dbp.txt" % Length
    b1.to_csv(unknownFile, header=None, sep="\t")
    b2.to_csv(knownFile, header=None, sep="\t")

writeFiles(75)
writeFiles(100)
writeFiles(200)
writeFiles(300)
writeFiles(100000)

length = 100
x = spliceInfo[spliceInfo["Length"]<=length]
f = float(len(x))
print x["U2-GT/AG"].sum() /f , x["U2-nc-GC/AG"].sum() / f, x["U12-AT/AC"].sum()/ f, x["unknown"].sum()/ f


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
2. Fit the model on the training dataset
Now we fit a new model and save it to logmt.
We will try to predict 'Alcabuse' with 'freetime' and 'goout'
Note that we specify what dataset we are using by setting data = train.
Can you write Python code for it?

# We run the model on the train dataset
logmt = smf.logit('Alcabuse ~ freetime + goout',data=train).fit()
print(logmt.summary())


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
3. Calculate predictions on the test dataset
We now use the model with coefficients estimated from the training dataset (logmt) and plug in the predictor variable values from the test dataset by using the .predict method. This method outputs the probabilities for us so we don't have to apply separate sigmoid transform. Therefore the output are the probabilities predicted with our model on the test dataset.
Can you write Python code for it?

ytestpred = logmt.predict(test)
np.round(ytestpred,2)


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
4. Calculate accuracy of the predictions
To calculate the accuracy of the model we need to specify the cutoff for the probabilities to say whether one should be classified as an alcohol abuser or not. For simplicity, let's say that we'll classify the person as an alcohol abuser if the estimated probability is greater than 50%.
We will then calculate the accuracy of the model in classifying alcohol abuse for the training data and the test data separately
Can you write Python code for it?

cutoff = .5

# Calculate accuracy for train
ytrainpred = logmt.predict(train)
ytrainpred[ytrainpred>=cutoff] =1
ytrainpred[ytrainpred<cutoff] =0
accs_train = pd.crosstab(ytrainpred,train["Alcabuse"])

# Calculate accuracy for test
ytestpred = logmt.predict(test)
ytestpred[ytestpred>=cutoff] =1
ytestpred[ytestpred<cutoff] =0
accs_test = pd.crosstab(ytestpred,test["Alcabuse"])

# Overall accuracy
print('Overall Accuracy in Training set : %.2f ' %((accs_train[0][0]+accs_train[1][1])/float(train.shape[0])))
print('overall Accuracy in Testing set  : %.2f ' %((accs_test[0][0]+accs_test[1][1])/float(test.shape[0])))


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
Cross-validation
The best way to compare accuracy of models and see how well your model generalize is to train (or fit) your model on a training dataset then test it on a separate test dataset. If you train and test your model on the same dataset (as we did above) you are likely to get a positvely biased result. The true generalizeability can only be observed when you test your model on a separate hold-out test dataset on which the model was not trained. A simple way to do cross-validation is to split the dataset before you begin your analysis.
There are many ways to split the dataset but for simplicity we will use the train_test_split function in the module sci-kit learn.
We will split the data so that the training dataset consists 60% of the data and the test dataset consists 40% of the data.
1. Split the data into training and testing datasets
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
np.random.seed(12345678) # set the random number generator seed to get consistent results
trainratio = .6
train, test = train_test_split(df, train_size=trainratio, random_state=1)
print('train-test ratio %.2f' %(float(len(train))/(len(train)+len(test))))


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
How do we determine the penalty value?
Essentially, we want to select the regularization parameter by identifying the one from a set of possible values (e.g. grid search) that results in the best fit of the model to the data.  However, it is important to note that it is easy to introduce bias into this process by trying a bunch of alphas and selecting the one that works best.  This can lead to optimistic evaluations of how well your model works.
Cross-validation is an ideal method to deal with this.  We can use cross-validation to select the alpha while adding minimal bias to the overall model prediction.
There are several different metrics of model fit to use as a criterion variable.  Log-likelihood or mean squared error are the two most common depending on how you are estimating the parameters (i.e., maximizing log-likelihood or minimizizing squared error).  The Akaike information criterion (AIC) and the Bayes Information criterion (BIC) are two common metrics that apply different penalties to these metrices for the number of free parameters in your model.
Here we will demonstrate using both to select an optimal value of the regularization parameter alpha of the Lasso estimator from an example provided by scikit-learn. For cross-validation, we use 20-fold with 2 algorithms to compute the Lasso path: coordinate descent, as implemented by the LassoCV class.
Can you write Python code for it?

from sklearn.linear_model import LassoCV, LassoLarsCV, LassoLarsIC
import time

# LassoLarsIC: least angle regression with BIC/AIC criterion
model_bic = LassoLarsIC(criterion='bic')
model_bic.fit(X, y)
alpha_bic_ = model_bic.alpha_

model_aic = LassoLarsIC(criterion='aic')
model_aic.fit(X, y)
alpha_aic_ = model_aic.alpha_

def plot_ic_criterion(model, name, value, color):
    alpha_ = model.alpha_
    alphas_ = model.alphas_
    criterion_ = model.criterion_
    plt.plot(-np.log10(alphas_), criterion_, '--', color=color,
             linewidth=3, label='%s criterion' % name)
    plt.axvline(-np.log10(alpha_), color=color, linewidth=3,
                label='%s estimate alpha: %.3f' %(name,value) )
    plt.xlabel('-log(alpha)',fontsize=16)
    plt.ylabel('Model Fit Criterion',fontsize=16)

plt.figure(figsize=(15,5))
plot_ic_criterion(model_aic, 'AIC', alpha_aic_ , 'b')
plot_ic_criterion(model_bic, 'BIC', alpha_bic_ , 'r')
plt.legend()
plt.title('Information-criterion for model selection',fontsize=18)


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
Interpreting coefficients as odds
In some cases it could be more useful to consider the odds of an event happening.
For example, we might want to be able to make statementes such as "If a person smokes, he is 10 times more likely to get lung cancer compared to a person who doesn't smoke."  The odds of an event happening is defined by
$odds = \frac{P(\text{event})}{P(\text{no event})} = \frac{P(\text{Y})}{P(\text{not Y})}$
while $P(\text{Y}) = \frac{1}{1 + e^{-(\beta_0 + \beta_1x_1)}}$
and $P(\text{not Y}) = 1 - P(\text{Y})$
If we want to look at how much the odds change for a unit increase in free time, we can use the following
$\triangle odds = \frac{\text{odds after a unit change in predictor}}{\text{original odds}} $
So, if we want to calculate the change in odds ratios of being an alcohol abuser when free time increases from 4 to 5, we can calculate the following:
$\triangle odds = \frac{\text{odds(Y) when x = 5}}{\text{odds(Y) when x = 4}}  = \frac{\frac{\text{P(Y) when x = 5}}{\text{1 - (P(Y) when x = 5)}}}{\frac{\text{P(Y) when x = 4}}{\text{1 - (P(Y) when x = 4)}}}$
Can you write Python code for it?

odd1 = sigmoid(logm.params[0] + logm.params[1] * 5) / (1-sigmoid(logm.params[0] + logm.params[1] * 5))
odd2 = sigmoid(logm.params[0] + logm.params[1] * 4) / (1-sigmoid(logm.params[0] + logm.params[1] * 4))
dodds = odd1/odd2
print('A student with 5hrs of free time compared to a student with 4 is %.2f times more likely to be an alcohol abuser' %dodds)


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
Logistic regression analysis
Let's start off by plotting a sigmoid function and add necessary modules.
Can you write Python code for it?

# Plot a sigmoid function
plt.figure(figsize=(5,5))  # open a figure base and determine the size (width, height)

def sigmoid(t):                          # Define the sigmoid function
    return (1/(1 + np.e**(-t)))

plot_range = np.arange(-6, 6, 0.1)       # range of x values to plot
y_values = sigmoid(plot_range)

# Plot curve
plt.plot(plot_range,   # X-axis range
         y_values,          # Predicted values
         color="red",linewidth=3)
plt.title("Example of a sigmoid function",fontsize=18)
plt.ylabel("Probability",fontsize=16)


I am working on the file "Notebooks/5_Prediction_&_Regularization.ipynb".
The first task is:
Now let's create a plot showing how well each alpha value performs across cross-validation folds
Can you write Python code for it?

# LassoCV: coordinate descent
y, X = dmatrices(formula_like=formula, data=df, return_type='dataframe')

# Compute paths
print("Computing regularization path using the coordinate descent lasso...")
t1 = time.time()
model = LassoCV(cv=5).fit(X, y)
t_lasso_cv = time.time() - t1

# Display results
m_log_alphas = -np.log10(model.alphas_)

# Display results
plt.figure(figsize=(15,5))
plt.plot(m_log_alphas, model.mse_path_, ':')
plt.plot(m_log_alphas, model.mse_path_.mean(axis=-1), 'k',
         label='Average across the folds', linewidth=2)
plt.axvline(-np.log10(model.alpha_), linestyle='--', color='k',
            label='CV estimate alpha: %.3f' %model.alpha_ )
plt.legend()
plt.xlabel('-log(alpha)',fontsize=16)
plt.ylabel('Mean square error',fontsize=16)
plt.title('Mean square error on each fold: coordinate descent',fontsize=18)
plt.axis('tight')