I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
This tells us that there are 60 patients (rows), each with 40 days of inflammation measurements (columns).
We can get an idea of the wide variety of methods available on the data object...
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/A New Treatment for Arthritis-checkpoint.ipynb".
The first task is:
This type is numpy's N-dimensional matrix class. This means we get a wide swathe of attributes and methods applicable to matrices, and a whole load of functions inside numpy and other libraries that will happily take it as an argument. One particularly useful property is shape:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Automating Multiple Single-Objective Spatial Optimization Models for Efficiency and Reproducibility

James D. Gaboardi    |     Association of American Geographers  2016

Florida State University     |     Department of Geography

Outline

Background & Information

Models

PMP
PCP
CentDian
PMCP Method

Data & Processing

Solutions

Visualizations

Future Work

COIN-OR

Set Path
Can you write Python code for it?

import IPython.display as IPd

# Local path on user's machine
path = '/Users/jgaboardi/AAG_16/Data/'


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Background & Information
$\Rightarrow$ Automating solutions for p-median, p-center, and p-centdian problems with p={1, 2, ..., n} facilities
$\Rightarrow$ Compare coverage and costs numerically, graphically, and visually
PySAL 1.11.0  $\Rightarrow$ PySAL.Network
Python Spatial Analysis Library
[https://www.pysal.readthedocs.org]
Sergio Rey at Arizona State University leads the PySAL project.
[https://geoplan.asu.edu/people/sergio-j-rey].
PySAL.Network was principally developed by Jay Laura at Arizona State Universty and the United States Geological Suvery.  [https://geoplan.asu.edu/people/jay-laura]
"PySAL is an open source library of spatial analysis functions written in Python intended to support the development of high level applications. PySAL is open source under the BSD License." [https://pysal.readthedocs.org/en/latest/]
Gurobi 6.5.0 $\Rightarrow$ gurobipy
Relatively new company founded by optimization experts formerly at key positions with CPLEX.
[http://www.gurobi.com] [http://www.gurobi.com/company/about-gurobi]
Models
The p-Median Problem  $\Rightarrow$ "minimum total cost" $\Rightarrow$ efficiency
The objective of the p-median problem, also know as the minimsum problem and the PMP, is to minimize the total weighted cost while siting [p] facilities to serve all demand/client nodes.  It was originally proposed by Hakimi (1964) and is well-studied in Geography, Operations Research, Mathematics, etc.  In this particular project the network-based vertice PMP is used meaning the cost will be calculated on a road network and solutions will be determined based on discrete locations.  Cost is generally defined as either travel time or distance and it is the latter in the project.  Population (demand) utilized as a weight at each client node.  The average cost can be calculated by dividing the minimized total cost by the total demand.
For more information refer to references section.
Minimize
         $\displaystyle {Z} = {\sum_{i \in 1}^n\sum_{j\in 1}^m a_i c_{ij}...
Can you write Python code for it?

# Conceptual Model Workflow
workflow = IPd.Image(path+'/AAG_16.png')
workflow


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Bokeh CentDian [p vs. cost] trade off
Can you write Python code for it?

source_centdian = ColumnDataSource(
        data=dict(
            x=range(1, len(SER)+1),
            y=VAL_CentDian,
            obj=VAL_CentDian,
            desc=p_list,
            change=CentDian_Diff))

TOOLS = 'wheel_zoom, pan, reset, crosshair, save'

hover = HoverTool(line_policy="nearest", mode="vline" ,tooltips="""

                @desc

                Center Median Cost
                [@obj]

                Variation
                [@change]

""")

# Instantiate Plot
centdian_plot = figure(plot_width=600, plot_height=600, tools=[TOOLS,hover],
           title="Center Median Distance vs. p-Facilities", y_range=(0,2))

# Create plot points and set source
centdian_plot.circle('x', 'y', size=15, color='blue', source=source_centdian,
                   legend='Center Median')
centdian_plot.line('x', 'y', line_width=2, color='blue', alpha=.5, source=source_centdian,
                   legend='Center Median')

centdian_plot.xaxis.axis_label = '[p = n]'
centdian_plot.yaxis.axis_label = 'Miles'

one_quarter = BoxAnnotation(plot=pcp_plot, top=.35,
                            fill_alpha=0.1, fill_color='green')
half = BoxAnnotation(plot=pcp_plot, bottom=.35, top=.7,
                            fill_alpha=0.1, fill_color='blue')
three_quarter = BoxAnnotation(plot=pcp_plot, bottom=.7, top=1.05,
                            fill_alpha=0.1, fill_color='gray')

centdian_plot.renderers.extend([one_quarter, half, three_quarter])

# Display the figure
show(centdian_plot)


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Bokeh PCP [p vs. cost] trade off
Can you write Python code for it?

source_c = ColumnDataSource(
        data=dict(
            x=range(1, len(SER)+1),
            y=VAL_PCP,
            obj=VAL_PCP,
            desc=p_list,
            change=PCP_Diff))

TOOLS = 'wheel_zoom, pan, reset, crosshair, save'

hover = HoverTool(line_policy="nearest", mode="vline", tooltips="""

                @desc

                Worst Case Cost
                [@obj]

                Variation
                [@change]

""")

# Instantiate Plot
pcp_plot = figure(plot_width=600, plot_height=600, tools=[TOOLS,hover],
           title="Worst Case Distance vs. p-Facilities", y_range=(0,2))

# Create plot points and set source
pcp_plot.circle('x', 'y', size=15, color='green', source=source_c,
                   legend='Minimized Worst Case')
pcp_plot.line('x', 'y', line_width=2, color='green', alpha=.5, source=source_c,
                   legend='Minimized Worst Case')

pcp_plot.xaxis.axis_label = '[p = n]'
pcp_plot.yaxis.axis_label = 'Miles'

one_quarter = BoxAnnotation(plot=pcp_plot, top=.35,
                            fill_alpha=0.1, fill_color='green')
half = BoxAnnotation(plot=pcp_plot, bottom=.35, top=.7,
                            fill_alpha=0.1, fill_color='blue')
three_quarter = BoxAnnotation(plot=pcp_plot, bottom=.7, top=1.05,
                            fill_alpha=0.1, fill_color='gray')

pcp_plot.renderers.extend([one_quarter, half, three_quarter])

# Display the figure
show(pcp_plot)


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Bokeh PMP & PCP [p vs. cost] comparision
Can you write Python code for it?

TOOLS = 'wheel_zoom, pan, reset, crosshair, save, hover'

bokeh_df_PMCP = pd.DataFrame()
bokeh_df_PMCP['p'] = [int(i[2:]) for i in p_dens]
bokeh_df_PMCP['Total Obj. Value'] = [VAL_PMCP[x][0] for x in range(len(VAL_PMCP))]
bokeh_df_PMCP['Avg. Obj. Value'] = [VAL_PMCP[x][1] for x in range(len(VAL_PMCP))]
bokeh_df_PMCP['Worst Case Obj. Value'] = [VAL_PMCP[x][2] for x in range(len(VAL_PMCP))]
bokeh_df_PMCP['CentDian Obj. Value'] = [VAL_PMCP[x][3] for x in range(len(VAL_PMCP))]

plot_PMCP = figure(title="Optimal PMP & PCP Selections without Sacrifice",
                        plot_width=800, plot_height=600, tools=[TOOLS], y_range=(0,2))

plot_PMCP.circle('x', 'y', size=5, color='red', source=source_m, legend='PMP')
plot_PMCP.line('x', 'y',
                 color="#ff4d4d", alpha=0.2, line_width=2, source=source_m, legend='PMP')

plot_PMCP.circle('x', 'y', size=5, color='green', source=source_c, legend='PCP')
plot_PMCP.line('x', 'y',
                 color='#00b300', alpha=0.2, line_width=2, source=source_c, legend='PCP')

plot_PMCP.circle('x', 'y', size=5, color='blue', source=source_centdian, legend='CentDian')
plot_PMCP.line('x', 'y',
                 color='#3385ff', alpha=0.2, line_width=2, source=source_centdian, legend='CentDian')

plot_PMCP.circle_x(bokeh_df_PMCP['p'],
                 bokeh_df_PMCP['Avg. Obj. Value'],
                 legend="Location PMP=PCP for PM+CP",
                 color="#ff4d4d",
                 fill_alpha=0.2,
                 size=15)

plot_PMCP.circle_x(bokeh_df_PMCP['p'],
             bokeh_df_PMCP['Worst Case Obj. Value'],
             legend="Location PCP=PMP for PM+CP",
             color='#00b300',
             fill_alpha=0.2,
             size=15)

plot_PMCP.circle_x(bokeh_df_PMCP['p'],
             bokeh_df_PMCP['CentDian Obj. Value'],
             legend="Location CentDian = PMCP",
             color='#3385ff',
             fill_alpha=0.2,
             size=15)

plot_PMCP.xaxis.axis_label = '[p = n]'
plot_PMCP.yaxis.axis_label = 'Miles'

one_quarter =...


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Bokeh PMP [p vs. cost] trade off
Can you write Python code for it?

source_m = ColumnDataSource(
        data=dict(
            x=range(1, len(SER)+1),
            y=AVG_PMP,
            avg=AVG_PMP,
            desc=p_list,
            change=PMP_Avg_Diff))

TOOLS = 'wheel_zoom, pan, reset, crosshair, save'

hover = HoverTool(line_policy="nearest", mode="hline", tooltips="""

                @desc

                Average Minimized Cost
                [@avg]

                Variation
                [@change]

""")

# Instantiate Plot
pmp_plot = figure(plot_width=600, plot_height=600, tools=[TOOLS,hover],
           title="Average Distance vs. p-Facilities", y_range=(0,2))

# Create plot points and set source
pmp_plot.circle('x', 'y', size=15, color='red',source=source_m,
                legend='Total Minimized Cost / Total Demand')
pmp_plot.line('x', 'y', line_width=2, color='red', alpha=.5, source=source_m,
              legend='Total Minimized Cost / Total Demand')

pmp_plot.xaxis.axis_label = '[p = n]'
pmp_plot.yaxis.axis_label = 'Miles'

one_quarter = BoxAnnotation(plot=pmp_plot, top=.35,
                            fill_alpha=0.1, fill_color='green')
half = BoxAnnotation(plot=pmp_plot, bottom=.35, top=.7,
                            fill_alpha=0.1, fill_color='blue')
three_quarter = BoxAnnotation(plot=pmp_plot, bottom=.7, top=1.05,
                            fill_alpha=0.1, fill_color='gray')

pmp_plot.renderers.extend([one_quarter, half, three_quarter])

# Display the figure
show(pmp_plot)


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
COIN warning that will be debugged in the next release

System Specs
Can you write Python code for it?

import datetime as dt
import os
import platform
import sys
import bokeh
import cylp

names = ['OSX', 'Processor ', 'Machine ', 'Python ','PySAL ','Gurobi ','Pandas ','GeoPandas ',
         'Shapely ', 'NumPy ', 'Bokeh ', 'CyLP', 'Date & Time']
versions = [platform.mac_ver()[0], platform.processor(), platform.machine(), platform.python_version(),
            ps.version, gbp.gurobi.version(), pd.__version__, gpd.__version__,
            str(shapely.__version__), np.__version__,
            bokeh.__version__, '0.7.1', dt.datetime.now()]
specs = pd.DataFrame(index=names, columns=['Version'])
specs.columns.name = 'Platform & Software Specs'
specs['Version'] = versions
specs # Pandas DF of specifications


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Call Client to Service Matrix Function
Can you write Python code for it?

# Call Client to Service Matrix Function
c_s_matrix()


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Convert Service Facilities Back to Longitude/Latitude for Google Maps Plots
Can you write Python code for it?

points = SERVICE
points.to_crs(epsg=32616, inplace=True) # UTM 16N
LonLat_Dict = OrderedDict()
LonLat_List = []

for i,j in points['geometry'].iteritems():
    LonLat_Dict[y_list[i]] = utm.to_latlon(j.xy[0][-1], j.xy[1][-1], 16, 'N')
    LonLat_List.append((utm.to_latlon(j.xy[0][-1], j.xy[1][-1], 16, 'N')))

Service_Lat_List = []
Service_Lon_List = []

for i in LonLat_List:
    Service_Lat_List.append(i[0])
for i in LonLat_List:
    Service_Lon_List.append(i[1])


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Create Lists of Selected Locations for Google Maps Plot
Can you write Python code for it?

# p-Median Selected Sites
list_of_p_MEDIAN = []
for y in range(len(y_list)):
    list_of_p_MEDIAN.append([])
    for p in range(len(p_list)):
        if pydf_M[y_list[y]][p_list[p]] == u'\u2588':
            list_of_p_MEDIAN[y].append([p_list[p]])

# p-Center Selected Sites
list_of_p_CENTER = []
for y in range(len(y_list)):
    list_of_p_CENTER.append([])
    for p in range(len(p_list)):
        if pydf_C[y_list[y]][p_list[p]] == u'\u2588':
            list_of_p_CENTER[y].append([p_list[p]])

# p-CentDian Selected Sites
list_of_p_CentDian = []
for y in range(len(y_list)):
    list_of_p_CentDian.append([])
    for p in range(len(p_list)):
        if pydf_CentDian[y_list[y]][p_list[p]] == u'\u2588':
            list_of_p_CentDian[y].append([p_list[p]])

# PMCP Selected Sites
list_of_PMCP = []
for y in range(len(y_list)):
    list_of_PMCP.append([])
    for p in p_dens:
        if pydf_MC[y_list[y]][p] == u'\u2588':
            list_of_PMCP[y].append(p)


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Data & Processing
Process Imports
Can you write Python code for it?

import pysal as ps
import geopandas as gpd
import numpy as np
import networkx as nx
import shapefile as shp
from shapely.geometry import Point
import shapely
from collections import OrderedDict
import pandas as pd
import qgrid
import gurobipy as gbp
import time
import bokeh
from bokeh.plotting import figure, show, ColumnDataSource
from bokeh.io import output_notebook, output_file, show
from bokeh.models import (HoverTool, BoxAnnotation, GeoJSONDataSource,
                          GMapPlot, GMapOptions, ColumnDataSource, Circle,
                          DataRange1d, PanTool, WheelZoomTool, BoxSelectTool,
                          ResetTool, MultiLine)
import utm
from cylp.cy import CyCbcModel, CyClpSimplex
import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
mpl.rcParams['figure.figsize']=8,8
output_notebook()


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Data preparation
Can you write Python code for it?

import spanoptpy as sp

print dir(sp)

from spanoptpy import AAG16_DataPrep

AAG16_DataPrep.data_prep()


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Draw PMCP figure
Can you write Python code for it?

# Draw Network Actual Roads and Nodes
for e in ntw.edges:
    g.add_edge(*e)
nx.draw(g, ntw.node_coords, node_size=5, alpha=0.25, edge_color='r', width=2)

size = 500
shape = 'sdh^Vp<8>'
counter = -1
for i,j in PMCP_Graphs.iteritems():
    if int(i) <= len(SER)-1:
        counter = counter+1
        pmcp = ps.open(path+'Results/Selected_Locations_PMCP'+str(i)+'.shp')
        points_pmcp = {}
        for idx, coords in enumerate(pmcp):
            PMCP_Graphs[i].add_node(idx)
            points_pmcp[idx] = coords
            PMCP_Graphs[i].node[idx] = coords
        nx.draw(PMCP_Graphs[i],
                    points_pmcp,
                    node_size=size,
                    node_shape=shape[counter],
                    alpha=.5,
                    node_color='k')
    else:
        pass

# Legend (Ordered Dictionary)
LEGEND = OrderedDict()
LEGEND['Network Nodes']=g
LEGEND['Roads']=g
for i in PMCP_Graphs:
    if int(i) <= len(SER)-1:
        LEGEND['PMP/PCP == '+str(i)]=PMCP_Graphs[i]
plt.legend(LEGEND,
       loc='upper left',
       fancybox=True,
       framealpha=0.5,
       scatterpoints=1)

# Title
plt.title('Waverly Hills\n Tallahassee, Florida', family='Times New Roman',
      size=40, color='k', backgroundcolor='w', weight='bold')

# North Arrow and 'N' --> Must be changed for different spatial resolutions, etc.
plt.arrow(624000, 164050, 0.0, 500, width=50, head_width=125,
          head_length=75, fc='k', ec='k',alpha=0.75,)
plt.annotate('N', xy=(623900, 164700), fontstyle='italic', fontsize='xx-large',
            fontweight='heavy', alpha=0.75)


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Google Maps Plot
Can you write Python code for it?

map_options = GMapOptions(lat=30.4855, lng=-84.265, map_type="hybrid", zoom=14)

plot = GMapPlot(
    x_range=DataRange1d(), y_range=DataRange1d(), map_options=map_options, title="Waverly Hills")

hover = HoverTool(tooltips="""

                Site @desc

                 \b

                p-Median:

                @p_select_median

                 \b

                p-Center

                @p_select_center

                 \b

                p-CentDian

                @p_select_centdian

                 \b

            PMCP Method

                @p_select_pmcp

""")

source_1 = ColumnDataSource(
        data=dict(
        lat=Service_Lat_List,
        lon=Service_Lon_List,
        desc=y_list,
        p_select_center=list_of_p_CENTER,
        p_select_median=list_of_p_MEDIAN,
        p_select_centdian= list_of_p_CentDian,
       ...


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Pandas PCP Data Frame
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Pandas PMCP Data Frame
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
Thanks for your time! Please share with your students and local government GIS departments, etc.!

email $\Longrightarrow$ jgaboardi@fsu.edu
GitHub $\Longrightarrow$ https://github.com/jGaboardi/AAG_16
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_16 copy-checkpoint.ipynb".
The first task is:
What we have at this point
Can you write Python code for it?

Buff.plot()
STREETS.plot()
CLIENT.plot()
SERVICE.plot(colormap=True)


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Add only intersecting records to a list
Can you write Python code for it?

# Add records that are points within the buffer
point_in = []
for p in INTER['geometry']:
    if type(p) == shapely.geometry.point.Point:
        point_in.append(p)
point_in[:5]


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Automating Multiple Single-Objective Spatial Optimization Models for Efficiency and Reproducibility

James D. Gaboardi    |     Association of American Geographers  2016

Florida State University     |     Department of Geography

Outline

Background & Information

Models

PMP
PCP
CentDian
PMCP Method

Data & Processing

Solutions

Visualizations

Future Work

COIN-OR

Initial Imports
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Background & Information
$\Rightarrow$ Automating solutions for p-median and p-center problems with p={n(p)} facilities
$\Rightarrow$ Compare coverage and costs numerically and visually
PySAL 1.11.0
Python Spatial Analysis Library
[https://www.pysal.readthedocs.org]
Sergio Rey at Arizona State University leads the PySAL project. [https://geoplan.asu.edu/people/sergio-j-rey]
"PySAL is an open source library of spatial analysis functions written in Python intended to support the development of high level applications. PySAL is open source under the BSD License." [https://pysal.readthedocs.org/en/latest/]
I will be only be demonstrating a portion of the functionality in PySAL.Network, but there are many other classes and functions for statistical spatial analysis within PySAL.
PySAL.Network
PySAL.Network was principally developed by Jay Laura at Arizona State Universty and the United States Geological Suvery.  [https://geoplan.asu.edu/people/jay-laura]
Gurobi 6.5.0
Relatively new company founded by optimization experts formerly at key positions with CPLEX.
[http://www.gurobi.com] [http://www.gurobi.com/company/about-gurobi]
gurobipy
Python wrapper for Gurobi
NumPy 1.10.4
"NumPy is the fundamental package for scientific computing with Python." [http://www.numpy.org]
Shapely 1.5.13
"Python package for manipulation and analysis of geometric objects in the Cartesian plane." [https://github.com/Toblerity/Shapely]
GeoPandas 0.1.1
"GeoPandas is an open source project to make working with geospatial data in python easier." [http://geopandas.org]
Pandas 0.17.1
"pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools..." [http://pandas.pydata.org]
Bokeh 0.11.1
"Bokeh is a Python interactive visualization library that targets modern web browsers for presentation." [http://bokeh.pydata.org/en/latest/]

Models
The p-Median Problem
The objective of the p-median problem, also know as the minimsum problem and the PMP, is to minimize the total weighted...
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Bokeh PCP [p vs. cost] trade off
Can you write Python code for it?

#output_notebook()

source_c = ColumnDataSource(
        data=dict(
            x=range(1, len(SER)+1),
            y=VAL_PCP,
            obj=VAL_PCP,
            desc=p_list,
            change=PCP_Diff))

TOOLS = 'wheel_zoom, pan, reset, crosshair, save'

hover = HoverTool(line_policy="nearest", mode="vline", tooltips="""

                @desc

                Worst Case Cost
                [@obj]

                Variation
                [@change]

""")

# Instantiate Plot
pcp_plot = figure(plot_width=600, plot_height=600, tools=[TOOLS,hover],
           title="Worst Case Distance vs. p-Facilities", y_range=(0,2))

# Create plot points and set source
pcp_plot.circle('x', 'y', size=15, color='green', source=source_c,
                   legend='Minimized Worst Case')
pcp_plot.line('x', 'y', line_width=2, color='green', alpha=.5, source=source_c,
                   legend='Minimized Worst Case')

pcp_plot.xaxis.axis_label = '[p = n]'
pcp_plot.yaxis.axis_label = 'Miles'

one_quarter = BoxAnnotation(plot=pcp_plot, top=.35,
                            fill_alpha=0.1, fill_color='green')
half = BoxAnnotation(plot=pcp_plot, bottom=.35, top=.7,
                            fill_alpha=0.1, fill_color='blue')
three_quarter = BoxAnnotation(plot=pcp_plot, bottom=.7, top=1.05,
                            fill_alpha=0.1, fill_color='gray')

pcp_plot.renderers.extend([one_quarter, half, three_quarter])

# Display the figure
show(pcp_plot)


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Bokeh PMP [p vs. cost] trade off
Can you write Python code for it?

#import bokeh
#from bokeh.charts import Scatter, show
from bokeh.plotting import figure, show, ColumnDataSource
from bokeh.io import output_notebook
from bokeh.models import HoverTool, BoxAnnotation
output_notebook()

source_m = ColumnDataSource(
        data=dict(
            x=range(1, len(SER)+1),
            y=AVG_PMP,
            avg=AVG_PMP,
            desc=p_list,
            change=PMP_Avg_Diff))

TOOLS = 'wheel_zoom, pan, reset, crosshair, save'

hover = HoverTool(line_policy="nearest", mode="hline", tooltips="""

                @desc

                Average Minimized Cost
                [@avg]

                Variation
                [@change]

""")

# Instantiate Plot
pmp_plot = figure(plot_width=600, plot_height=600, tools=[TOOLS,hover],
           title="Average Distance vs. p-Facilities", y_range=(0,2))

# Create plot points and set source
pmp_plot.circle('x', 'y', size=15, color='red',source=source_m,
                legend='Total Minimized Cost / Total Demand')
pmp_plot.line('x', 'y', line_width=2, color='red', alpha=.5, source=source_m,
              legend='Total Minimized Cost / Total Demand')

pmp_plot.xaxis.axis_label = '[p = n]'
pmp_plot.yaxis.axis_label = 'Miles'

one_quarter = BoxAnnotation(plot=pmp_plot, top=.35,
                            fill_alpha=0.1, fill_color='green')
half = BoxAnnotation(plot=pmp_plot, bottom=.35, top=.7,
                            fill_alpha=0.1, fill_color='blue')
three_quarter = BoxAnnotation(plot=pmp_plot, bottom=.7, top=1.05,
                            fill_alpha=0.1,...


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Call Client to Service Matrix Function
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Convert Service Facilities Back to Longitude/Latitude for Google Maps Plots
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Create 1000 random ppoints within the bounds of WAVERLY.shp
Can you write Python code for it?

np.random.seed(352)
x = np.random.uniform(shp_W.bbox[0], shp_W.bbox[2], 1000)
np.random.seed(850)
y = np.random.uniform(shp_W.bbox[1], shp_W.bbox[3], 1000)
coords0= zip(x,y)
coords = [shapely.geometry.Point(i) for i in coords0]
Rand = gpd.GeoDataFrame(coords)
Rand.crs = STREETS.crs
Rand.columns = ['geometry']
Rand[:5]


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Create Lists of Selected Locations for Google Maps Plot
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Create Lists to fill index and columns of GeoPandas Data Frames
Can you write Python code for it?

# PANDAS DATAFRAME OF p/y results
p_list = []
for i in range(1, len(SER)+1):
    p = 'p='+str(i)
    p_list.append(p)
y_list = []
for i in range(1, len(SER)+1):
    y = 'y'+str(i)
    y_list.append(y)


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Create PMP, PCP, and CentDian solution graphs
Can you write Python code for it?

# p-Median
P_Med_Graphs = OrderedDict()
for x in range(1, len(SER)+1):
    P_Med_Graphs["{0}".format(x)] = nx.Graph()

# p-Center
P_Cent_Graphs = OrderedDict()
for x in range(1, len(SER)+1):
    P_Cent_Graphs["{0}".format(x)] = nx.Graph()

# p-CentDian
P_CentDian_Graphs = OrderedDict()
for x in range(1, len(SER)+1):
    P_CentDian_Graphs["{0}".format(x)] = nx.Graph()


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Data Frames adjust
Can you write Python code for it?

# PMP
pydf_M = pydf_M[len(SER):]
pydf_M.reset_index()
pydf_M.index = p_list
pydf_M.columns.name = 'Decision\nVariables'
pydf_M.index.name = 'Facility\nDensity'
pydf_M['Tot. Obj. Value'] = VAL_PMP
pydf_M['Tot. % Change'] = PMP_Tot_Diff
pydf_M['Avg. Obj. Value'] = AVG_PMP
pydf_M['Avg. % Change'] = PMP_Avg_Diff
pydf_M = pydf_M.fillna('')
#pydf_M.to_csv(path+'CSV')  <-- need to change squares to alphanumeric to use

# PCP
pydf_C = pydf_C[len(SER):]
pydf_C.reset_index()
pydf_C.index = p_list
pydf_C.columns.name = 'Decision\nVariables'
pydf_C.index.name = 'Facility\nDensity'
pydf_C['Worst Case Obj. Value'] = VAL_PCP
pydf_C['Worst Case % Change'] = PCP_Diff
pydf_C = pydf_C.fillna('')
#pydf_C.to_csv(path+'CSV')  <-- need to change squares to alphanumeric to use

pydf_CentDian = pydf_CentDian[len(SER):]
pydf_CentDian.reset_index()
pydf_CentDian.index = p_list
pydf_CentDian.columns.name = 'Decision\nVariables'
pydf_CentDian.index.name = 'Facility\nDensity'
pydf_CentDian['CentDian Obj. Value'] = VAL_CentDian
pydf_CentDian['CentDian % Change'] = CentDian_Diff
pydf_CentDian = pydf_CentDian.fillna('')
#pydf_CentDian.to_csv(path+'CSV')  <-- need to change squares to alphanumeric to use

# PMCP
pydf_MC = pydf_MC[len(SER):]
pydf_MC.reset_index()
pydf_MC.index = p_dens
pydf_MC.columns.name = 'D.V.'
pydf_MC.index.name = 'F.D.'
pydf_MC['Min.\nTotal'] = [VAL_PMCP[x][0] for x in range(len(VAL_PMCP))]
pydf_MC['Min.\nTotal\n%\nChange'] = [PMCP_Diff[x][0] for x in range(len(PMCP_Diff))]
pydf_MC['Avg.\nTotal'] = [VAL_PMCP[x][1] for x in range(len(VAL_PMCP))]
pydf_MC['Avg.\nTotal\n%\nChange'] = [PMCP_Diff[x][1] for x in range(len(PMCP_Diff))]
pydf_MC['Worst\nCase'] = [VAL_PMCP[x][2] for x in range(len(VAL_PMCP))]
pydf_MC['Worst\nCase\n%\nChange'] = [PMCP_Diff[x][2] for x in range(len(PMCP_Diff))]
pydf_MC['Center\nMedian'] = [VAL_PMCP[x][3] for x in range(len(VAL_PMCP))]
pydf_MC['Center\nMedian\n%\nChange'] = [PMCP_Diff[x][3] for x in range(len(PMCP_Diff))]
pydf_MC = pydf_MC.fillna('')
#pydf_MC.to_csv(path+'CSV')  <-- need to change...


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Define the function to calculate the cost matrix and convert to miles
Can you write Python code for it?

def c_s_matrix():  # Define Client to Service Matrix Function
    global All_Dist_MILES # in meters
    All_Neigh_Dist = ntw.allneighbordistances(
                        sourcepattern=ntw.pointpatterns['Rand_Points_CLIENT'],
                        destpattern=ntw.pointpatterns['Rand_Points_SERVICE'])
    All_Dist_MILES = All_Neigh_Dist * 0.000621371 # to miles


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Define the function to solve the p-Median + p-Center Problems concurrently
Can you write Python code for it?

def Gurobi_PMCP(sites, Ai, AiSum, All_Dist_Miles):

    # Define Global Variables
    global pydf_M
    global selected_M
    global NEW_Records_PMP
    global VAL_PMP
    global AVG_PMP

    global pydf_C
    global selected_C
    global NEW_Records_PCP
    global VAL_PCP

    global pydf_CentDian
    global selected_CentDian
    global NEW_Records_Pcentdian
    global VAL_CentDian

    global pydf_MC
    global VAL_PMCP
    global p_dens

    for p in range(1, sites+1):

        # DATA
        # [p]        --> sites
        # Demand     --> Ai
        # Demand Sum --> AiSum
        # Travel Costs
        Cij = All_Dist_MILES
        # Weighted Costs
        Sij = Ai * Cij
        # Total Client and Service nodes
        client_nodes = range(len(Sij))
        service_nodes = range(len(Sij[0]))

        ##################################################################
        # PMP
        t1_PMP = time.time()

        #     Create Model, Add Variables, & Update Model
        # Instantiate Model
        mPMP = gbp.Model(' -- p-Median -- ')
        # Turn off Gurobi's output
        mPMP.setParam('OutputFlag',False)

        # Add Client Decision Variables (iXj)
        client_var = []
        for orig in client_nodes:
            client_var.append([])
            for dest in service_nodes:
                client_var[orig].append(mPMP.addVar(vtype=gbp.GRB.BINARY,
                                                lb=0,
                                                ub=1,
                                                obj=Sij[orig][dest],
                                                name='x'+str(orig+1)+'_'+str(dest+1)))

        # Add Service Decision Variables (j)
        serv_var = []
        for dest in service_nodes:
            serv_var.append([])
            serv_var[dest].append(mPMP.addVar(vtype=gbp.GRB.BINARY,
                                          lb=0,
                                          ub=1,
                                         ...


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Draw PMCP figure
Can you write Python code for it?

figsize(10,10)
# Draw Network Actual Roads and Nodes
for e in ntw.edges:
    g.add_edge(*e)
nx.draw(g, ntw.node_coords, node_size=5, alpha=0.25, edge_color='r', width=2)

size = 500
shape = 'sdh^Vp<8>'
counter = -1
for i,j in PMCP_Graphs.iteritems():
    if int(i) <= len(SER)-1:
        counter = counter+1
        pmcp = ps.open(path+'Results/Selected_Locations_PMCP'+str(i)+'.shp')
        points_pmcp = {}
        for idx, coords in enumerate(pmcp):
            PMCP_Graphs[i].add_node(idx)
            points_pmcp[idx] = coords
            PMCP_Graphs[i].node[idx] = coords
        nx.draw(PMCP_Graphs[i],
                    points_pmcp,
                    node_size=size,
                    node_shape=shape[counter],
                    alpha=.5,
                    node_color='k')
    else:
        pass

# Legend (Ordered Dictionary)
LEGEND = OrderedDict()
LEGEND['Network Nodes']=g
LEGEND['Roads']=g
for i in PMCP_Graphs:
    if int(i) <= len(SER)-1:
        LEGEND['PMP/PCP == '+str(i)]=PMCP_Graphs[i]
legend(LEGEND,
       loc='upper left',
       fancybox=True,
       framealpha=0.5,
       scatterpoints=1)

# Title
title('Waverly Hills\n Tallahassee, Florida', family='Times New Roman',
      size=40, color='k', backgroundcolor='w', weight='bold')

# North Arrow and 'N' --> Must be changed for different spatial resolutions, etc.
arrow(624000, 164050, 0.0, 500, width=50, head_width=125,
          head_length=75, fc='k', ec='k',alpha=0.75,)
annotate('N', xy=(623900, 164700), fontstyle='italic', fontsize='xx-large',
            fontweight='heavy', alpha=0.75)


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Future Work & Vision
$\Longrightarrow$ Develop a python library for bring together in one package spatial analysis & spatial optimization [spanoptpy] potentially incorporating:
|QGIS|  PySAL  |  NetworkX  |  Pandas  |  GeoPandas  |  NumPy  |  Shapely  |  Bokeh | CyLP
|----|---------|-------------------------------------------------------------------------
|GIS|network analysis|network analysis|data frames|geo dataframes|scientific computing|geometric objects|visualizations|optimization
$\Longrightarrow$ Need PySAL.Network to be able to handle larger networks
$\Longrightarrow$ Develop functionality within a Linux environment
$\Longrightarrow$  scipy.spatial.cKDTree(dist_matrix)
                $\Longrightarrow$  query_ball_point() for close neighbors of the selected sites
$\Longrightarrow$ Master CyLP from COIN-OR or develop an open-source optimization suite

interface with CLP, CBC, CGL
[ http://mpy.github.io/CyLPdoc/ ]

relatively steep learning curve

Computational Optimization Infrastructure for Operations Research

[ http://www.coin-or.org ]

$\ast$ CyLP example
Minimize
         $ 3x + 2y $
Subject To
         $ x \geq 3$
         $ y \geq 5$
         $ x - y \leq 20$
Can you write Python code for it?

# Gurobi
m = gbp.Model()
m.setParam( 'OutputFlag', False )
x = m.addVar(vtype=gbp.GRB.CONTINUOUS, name='x')
y = m.addVar(vtype=gbp.GRB.CONTINUOUS, name='y')
m.update()
m.setObjective(3*x + 2*y, gbp.GRB.MINIMIZE)
m.addConstr(x >= 3)
m.addConstr(y >= 5)
m.addConstr(x - y <= 20)
m.optimize()
#m.write('path_m.lp')
print m.objVal
print m.getVars()

# CyLP
s = CyClpSimplex()
x = s.addVariable('x', 1)
y = s.addVariable('y', 1)
s += x >= 3
s += y >= 5
s += x - y <= 20
s.objective = 3 * x + 2 * y
s.primal()
#s.writeLp('path_s')
print s.objectiveValue
print s.primalVariableSolution
print 'Gurobi & CLP Objective Values match? --> ', m.objVal == s.objectiveValue


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Instaniate non-solution graphs to be drawn
Can you write Python code for it?

g = nx.Graph() # Roads & Nodes
g1 = nx.MultiGraph() # Edges and Vertices
GRAPH_client = nx.Graph() # Clients
g_client = nx.Graph() # Snapped Clients
GRAPH_service = nx.Graph() # Service
g_service = nx.Graph() # Snapped Service


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Instantiate Client .shp
Can you write Python code for it?

client = shp.Writer(shp.POINT) # Client Shapefile
# Add Random Points
for i,j in CLI:
    client.point(i,j)
# Add Fields
client.field('client_ID')
client.field('Weight')
counter = 0
for i in range(len(CLI)):
    counter = counter + 1
    client.record('client_' + str(counter), Ai[i])
client.save(path+'Simulated/RandomPoints_CLIENT') # Save Shapefile


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Instantiate Network and read in WAVERLY.shp
Can you write Python code for it?

ntw = ps.Network(path+'WAVERLY/WAVERLY.shp')
shp_W = ps.open(path+'WAVERLY/WAVERLY.shp')


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Instantiate Service .shp
Can you write Python code for it?

service = shp.Writer(shp.POINT) #Service Shapefile
# Add Random Points
for i,j in SER:
    service.point(i,j)
# Add Fields
service.field('y_ID')
service.field('x_ID')
counter = 0
for i in range(len(SER)):
    counter = counter + 1
    service.record('y' + str(counter), 'x' + str(counter))
service.save(path+'Simulated/RandomPoints_SERVICE') # Save Shapefile


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Instantiate and fill Client and Service point dictionaries
Can you write Python code for it?

points_client = {}
points_service = {}

CLI = ps.open(path+'CLIENT/CLIENT.shp')
for idx, coords in enumerate(CLI):
    GRAPH_client.add_node(idx)
    points_client[idx] = coords
    GRAPH_client.node[idx] = coords

SER = ps.open(path+'SERVICE/SERVICE.shp')
for idx, coords in enumerate(SER):
    GRAPH_service.add_node(idx)
    points_service[idx] = coords
    GRAPH_service.node[idx] = coords


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Instantiate lists for objective values and average values of all models
Can you write Python code for it?

# PMP
VAL_PMP = []
AVG_PMP = []

# PCP
VAL_PCP = []

# CentDian
VAL_CentDian = []

# PMCP
VAL_PMCP = []
p_dens = [] # when the facilities for the p-median & p-center are the same


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Instantiate snapped Service .shp
Can you write Python code for it?

service_SNAP = shp.Writer(shp.POINT) # Snapped Service Shapefile
# Add Points
for i,j in ntw.pointpatterns['Rand_Points_SERVICE'].snapped_coordinates.iteritems():
    service_SNAP.point(j[0],j[1])
# Add Fields
service_SNAP.field('y_ID')
service_SNAP.field('x_ID')
service_SNAP.field('LAT')
service_SNAP.field('LON')
counter = 0
for i in range(len(ntw.pointpatterns['Rand_Points_SERVICE'].snapped_coordinates)):
    counter = counter + 1
    service_SNAP.record('y' + str(counter), 'x' + str(counter), y_snapped[i], x_snapped[i])
service_SNAP.save(path+'Snapped/SERVICE_Snapped') # Save Shapefile


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Pandas PCP Data Frame
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Pandas PMCP Data Frame
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Pandas PMP Data Frame
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Plot Buffers of Individual Streets
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Plot the 1000 random
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Plot the unary union buffer
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Reproject the street network with GeoPandas
Can you write Python code for it?

STREETS_Orig = gpd.read_file(path+'Waverly_Trim/Waverly.shp')
STREETS = gpd.read_file(path+'Waverly_Trim/Waverly.shp')
STREETS.to_crs(epsg=2779, inplace=True) # NAD83(HARN) / Florida North
STREETS.to_file(path+'WAVERLY/WAVERLY.shp')
STREETS[:5]


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Simulate weights for Client Demand
Can you write Python code for it?

# Client Weights for demand
np.random.seed(850)
Ai = np.random.randint(1, 5, len(CLI))
Ai = Ai.reshape(len(Ai),1)
AiSum = np.sum(Ai) # Sum of Weights (Total Demand)


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
Snap Client and Service points to the network
Can you write Python code for it?

# Snap
Snap_C = ntw.snapobservations(path+'Simulated/RandomPoints_CLIENT.shp',
                     'Rand_Points_CLIENT', attribute=True)
Snap_S = ntw.snapobservations(path+'Simulated/RandomPoints_SERVICE.shp',
                     'Rand_Points_SERVICE', attribute=True)


I am working on the file ".ipynb_checkpoints/AAG_2016-checkpoint.ipynb".
The first task is:
email $\Longrightarrow$ jgaboardi@fsu.edu

GitHub $\Longrightarrow$ https://github.com/jGaboardi/AAG_16
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Advancd Python-checkpoint.ipynb".
The first task is:
You need this code segment becasue it allows you to execute the code within it when you are directly running the file. So whe nyou create modules you need some code but it must not be executed when you import that module in that case this if statement serves the need.
Can you write Python code for it?

# If you want to give arguments to a function as a list
def f(x,y):
    print(x, y)

myList= [1,2]
f(*myList)


I am working on the file ".ipynb_checkpoints/AnalysisNotebook-checkpoint.ipynb".
The first task is:
Feature Selection
One of the most important steps in single cell RNA seq processing, is selecting genes that describe most of the biological variance. However, this is confounded by the high levels of technical noise associated with single cell RNA-seq data.
This jupyter notebook contains 2 functions to enable feature selection:
1. variance - select the top variable genes in the dataset
2. overdispersion - select the top variable genes in the dataset corrected for technical variance
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AnalysisNotebook-checkpoint.ipynb".
The first task is:
Make .Rds file for CellView
And finally, let's summarize this analysis into an .Rds file that we can share with others
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AnalysisNotebook-checkpoint.ipynb".
The first task is:
Marker Identification
Identifying genes that differentiate each of these cell populations is an important aspect of single cell RNA-seq data. There are many different methods to do this type of analysis. Given the size of the dataset ome of these are compute heavy. For the sake of brevity, we will use AUROC classification of differentially expressed genes.
Can you write Python code for it?

aurocScoresAll=pd.DataFrame()
for cluster in range(len(1,8,1)):

    cells1=tsnedata_dbscan[tsnedata_dbscan['dbCluster']==cluster].index
    cells2=tsnedata_dbscan.index.difference(cells1)

    data1=cpt.loc[cpt.index,cells1].apply(expMean,1)
    data2=cpt.loc[cpt.index,cells2].apply(expMean,1)
    totaldiff=(data1-data2)
    genes=totaldiff[totaldiff>1.].index

    aurocScorss=findMarkers(cpt,
                  cells1,
                  cells2,
                  genes
                 )
    aurocScores['Associated.Gene.Name']=featuredata['Associated.Gene.Name']
    aurocScores['dbCluster']=cluster
    aurocScoresAll=aurocScoresAll.append(aurocScores)


I am working on the file ".ipynb_checkpoints/AnalysisNotebook-checkpoint.ipynb".
The first task is:
Normalize data
Since number of genes and transcipts detected is directly dependent on read depth, library size normalization is essential. This function will normalize gene expression based on total transcripts detected in each cell, multiply with a constant and log transform.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AnalysisNotebook-checkpoint.ipynb".
The first task is:
Read Input data
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AnalysisNotebook-checkpoint.ipynb".
The first task is:
Visualization
Let's make a heatmap of all markergenes
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AnalysisNotebook-checkpoint.ipynb".
The first task is:
Visualization
Lets take a look at our dimensionality reduction by plotting cells, but this time color each cell by the cluster assignment as determined by DBSCAN
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/AnalysisNotebook-checkpoint.ipynb".
The first task is:
Visualization
Lets take a look at our dimensionality reduction by plotting cells, but this time color each cell by the cluster assignment as determined by DBSCAN. Remember that our data was reduced to 3 dimensions. So, lets plot all 3 dimensions
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Annulus_Simple_Matplotlib-checkpoint.ipynb".
The first task is:
Commands for plotting
These are used so the the usual "plot" will use matplotlib.
Can you write Python code for it?

# commands for plotting, "plot" works with matplotlib
def mesh2triang(mesh):
    xy = mesh.coordinates()
    return tri.Triangulation(xy[:, 0], xy[:, 1], mesh.cells())

def mplot_cellfunction(cellfn):
    C = cellfn.array()
    tri = mesh2triang(cellfn.mesh())
    return plt.tripcolor(tri, facecolors=C)

def mplot_function(f):
    mesh = f.function_space().mesh()
    if (mesh.geometry().dim() != 2):
        raise AttributeError('Mesh must be 2D')
    # DG0 cellwise function
    if f.vector().size() == mesh.num_cells():
        C = f.vector().array()
        return plt.tripcolor(mesh2triang(mesh), C)
    # Scalar function, interpolated to vertices
    elif f.value_rank() == 0:
        C = f.compute_vertex_values(mesh)
        return plt.tripcolor(mesh2triang(mesh), C, shading='gouraud')
    # Vector function, interpolated to vertices
    elif f.value_rank() == 1:
        w0 = f.compute_vertex_values(mesh)
        if (len(w0) != 2*mesh.num_vertices()):
            raise AttributeError('Vector field must be 2D')
        X = mesh.coordinates()[:, 0]
        Y = mesh.coordinates()[:, 1]
        U = w0[:mesh.num_vertices()]
        V = w0[mesh.num_vertices():]
        return plt.quiver(X,Y,U,V)

# Plot a generic dolfin object (if supported)
def plot(obj):
    plt.gca().set_aspect('equal')
    if isinstance(obj, Function):
        return mplot_function(obj)
    elif isinstance(obj, CellFunctionSizet):
        return mplot_cellfunction(obj)
    elif isinstance(obj, CellFunctionDouble):
        return mplot_cellfunction(obj)
    elif isinstance(obj, CellFunctionInt):
        return mplot_cellfunction(obj)
    elif isinstance(obj, Mesh):
        if (obj.geometry().dim() != 2):
            raise AttributeError('Mesh must be 2D')
        return plt.triplot(mesh2triang(obj), color='#808080')

    raise AttributeError('Failed to plot %s'%type(obj))
# end of commands for plotting


I am working on the file ".ipynb_checkpoints/Annulus_Simple_Matplotlib-checkpoint.ipynb".
The first task is:
If you want to do usual "matplotlib" stuff then you still need "plt." prefix on commands.
Can you write Python code for it?

plt.figure()
plt.subplot(1,2,1)
plot(mesh)
plt.xlabel('x')
plt.ylabel('y')
plt.subplot(1,2,2)
plot(u)
plt.title('annulus solution')


I am working on the file ".ipynb_checkpoints/ArchiPi-checkpoint.ipynb".
The first task is:
Neste programa usamos um módulo (decimal)
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Assignment_1-checkpoint.ipynb".
The first task is:
Clearly, $A^2$ as computed above for the undirected unweighted graph G1 (question 1) encodes the information of the number of paths of length 2 from each node to every other node. For Example, Cosider the pair of nodes 1 and 4. There exists a path of length 2 from node 1 to node 4 via node 3, thereby giving $A^2_{(1, 4)} = 1$ . Similarly, there exists three paths of length 2 from node 8 back to node 8, via node 3, node 5 and node 7. Thus we have $A^2_{(8, 8)} = 3$.
In general, $A^n_{(i, j)}$ gives the number of paths of length $n$ from node $i$ to node $j$ in the undirected unweighted graph.
Question 4
Can you write Python code for it?

## Reading C. Elegans neural network dataset
G_celegans = nx.read_gml('Newman_Datasets/celegansneural/celegansneural.gml')
G_celegans_adj = nx.to_numpy_matrix(G_celegans)
G_celegans_deg = print_degree(G_celegans_adj) # function written in question 2

# Plotting
plt.hist(np.asarray(G_celegans_deg[0])[0], bins='auto')
plt.title("In Degree Distribution for C. Elegans Neural Network")
plt.xlabel("Degree")
plt.ylabel("Frequency")
plt.show()

plt.hist(np.asarray(G_celegans_deg[1])[0], bins='auto')
plt.title("Out Degree Distribution for C. Elegans Neural Network")
plt.xlabel("Degree")
plt.ylabel("Frequency")
plt.show()

## Reading C. Elegans neural network dataset
G_adjnoun = nx.read_gml('Newman_Datasets/adjnoun/adjnoun.gml')
G_adjnoun_adj = nx.to_numpy_matrix(G_adjnoun)
G_adjnoun_deg = print_degree(G_adjnoun_adj) # function written in question 2

# Plotting
plt.hist(np.asarray(G_adjnoun_deg[0])[0], bins='auto')
plt.title("In Degree Distribution for C. Elegans Neural Network")
plt.xlabel("Degree")
plt.ylabel("Frequency")
plt.show()


I am working on the file ".ipynb_checkpoints/Assignment_1-checkpoint.ipynb".
The first task is:
Qeustion 2
Can you write Python code for it?

## Fuction to find degree from adjacency matrix

# function definition
def print_degree(adj):
    """
        This function computes the degree of each node for a given adjacency matrix
        For Uniderected Graphs:
            It prints node wise degree and returns it.
        For Directed Graphs:
            It prnts both in degree and out degree and reurns both of them.
    """
    if np.allclose(adj, adj.T, atol=1e-8):
        print "Undirected Graph"
        deg = np.sum(adj != 0, axis=0)
        print "Degree of each node: " + np.str(deg)
    else:
        print "Directed Graph"
        deg = list()
        deg.append(np.sum(adj != 0, axis=0).astype(np.int))
        deg.append(np.sum(adj != 0, axis=1).astype(np.int).T)
        print "In Degree of each node: " + np.str(deg[0])
        print "Out Degree of each node: " + np.str(deg[1])
    return deg

# loading adjacency matrix which were saved in question 1
f = open('ques1_stired_networks.pickle', 'r')
adj_matrix, edge_list = pickle.load(f)

for key in adj_matrix:
    print key
    print_degree(adj_matrix[key])


I am working on the file ".ipynb_checkpoints/Assignment_1-checkpoint.ipynb".
The first task is:
Question 3
Can you write Python code for it?

# Computing A square for Undirected nnweighted graph G1
A = adjacency_matrix['G1 (UD-UW)']
A = A.astype(np.int)
A_square = A*A

pos=nx.spring_layout(G1) # positions for all nodes
nx.draw(G1, pos)
nx.draw_networkx_labels(G1, pos)
plt.title('Undirected Unwieghted Graph')
plt.show()
print "A_square : \n " + np.str(A_square)


I am working on the file ".ipynb_checkpoints/Assignment_1-checkpoint.ipynb".
The first task is:
Question 5
Can you write Python code for it?

## Random Undirected Unweighted Graph

def create_random_graph(n, e):
    """
        This function creates a random adjacncy matrix for a simple
        undirected unweighted graph with n nodes and e edges.
        It returns the randomly computed adjacency matrix.
    """
    adj = np.zeros((nodes, nodes))
    edges = 0;
    while edges < e:
        i = np.random.randint(0, n-2)
        j = np.random.randint(1, n-1)
        if (adj[i][j] == 1):
            continue
        adj[i][j] = 1
        edges = edges+1

?np.zeros


I am working on the file ".ipynb_checkpoints/AtomicString1-checkpoint.ipynb".
The first task is:
3) All derivatives of AString can be represented via AString itself:
AString'(x) = AString(x/2 + 1/2) - AString(x/2 - 1/2)
4) Combination of Atomic Strings can exactly represent a straight line:
x = AString(x) + Astring(x+1) + Astring(x+2)...
Can you write Python code for it?

x = np.arange(-40.0, 40.0, 0.01)

#pl.plot(x, ABline (x, 1, 0),            label='ABLine 1*x')
pl.plot(x, AString(x, 10.0,-15, 10, -15),   '--',  linewidth=1, label='AString 1')
pl.plot(x, AString(x, 10.0, -5, 10, -5),    '--',  linewidth=1, label='AString 2')
pl.plot(x, AString(x, 10.0, +5, 10, +5),    '--',  linewidth=1, label='AString 3')
pl.plot(x, AString(x, 10.0,+15, 10, +15),   '--',  linewidth=1, label='AString 4')

AS2 = Sum(AString(x, 10.0, -15, 10, -15), AString(x, 10., -5, 10, -5))
AS3 = Sum(AS2, AString(x, 10, +5, 10, +5))
AS4 = Sum(AS3, AString(x, 10,+15, 10, +15))
pl.plot(x, AS4,  label='AStrings Joins', linewidth=2)

pl.title('Atomic Strings Combinations')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file ".ipynb_checkpoints/AtomicString1-checkpoint.ipynb".
The first task is:
3) Atomic Function (AF) is a 'finite' function (like spline) not equal to zero only on section |x|<=1
4) Atomic Function is a non-analytical function (can not be represented by Taylor's series), but with known Fourier Transformation allowing to exactly calculate AF in certain points, with tabular representation provided in script above.
Atomic String is a generalisation of an Atomic Function
1) Astring is an swing-like function - Integral of Atomic Function (AF) which can be expressed via AF itself:
AString(x) = Integral(0,x)(Up(x)) = Up(x/2 - 1/2) - 1/2
2) Atomic Function can be represented via simple superposition of Atomic Strings:
up(x) = AString(x/2 + 1/2) - AString(x/2 - 1/2)
Can you write Python code for it?

#########   Presentation of Atomic Function via Atomic Strings ##########
x = np.arange(-2.0, 2.0, 0.01)

pl.plot(x, AString(x, 1, 0, 1, 0),  '--', linewidth=1,  label='AString(x)')
pl.plot(x, AString(x, 0.5, -0.5, +1, 0), '--', linewidth=2, label='+AString(2x+1)')
pl.plot(x, AString(x, 0.5, +0.5, -1, 0), '--', linewidth=2, label='-AString(2x-1)')
#pl.plot(x, up(x, 1.0, 0, 1, 0),     '--', linewidth=1, label='Atomic Function')
AS2 = Sum(AString(x, 0.5, -0.5, +1, 0), AString(x, 0.5, +0.5, -1, 0))
pl.plot(x, AS2, linewidth=3, label='Up(x) via Strings')
pl.title('Atomic Function as a Combination of AStrings')
pl.legend(loc='center left', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file ".ipynb_checkpoints/AtomicString1-checkpoint.ipynb".
The first task is:
Atomic String as an Integral of Atomic Function (introduced in 2017 by Prof S.Eremenko)
Can you write Python code for it?

############### Atomic String #############
def AString1(x: float) -> float:
    res = 1 * (upulse(x/2.0 - 0.5) - 0.5)
    return res

############### Atomic String Pulse with width, shift and scale #############
def AStringPulse(t: float, a = 1., b = 0., c = 1., d = 0.) -> float:
    x = (t - b)/a
    if (x < -1):
        res = -0.5
    elif (x > 1):
        res = 0.5
    else:
        res = AString1(x)
    res = d + res * c
    return res

###### Atomic String Applied to list with width, shift and scale #############
def AString(x: list, a = 1., b = 0., c = 1., d = 0.) -> list:
    res: list = []
    for i in range(len(x)):
        res.append(AStringPulse(x[i], a, b, c, d))
        #res[i] = AStringPulse(x[i], a, b, c)
    return res

###### Summation of two lists #############
def Sum(x1: list, x2: list) -> list:
    res: list = []
    for i in range(len(x1)):
       res.append(x1[i] + x2[i])
    return res

pl.title('Atomic String')
pl.plot(x, AString(x, 1.0, 0, 1, 0), label='Atomic String')
pl.grid(True)
pl.show()


I am working on the file ".ipynb_checkpoints/AtomicString1-checkpoint.ipynb".
The first task is:
Atomic String, Atomic Function and Atomic Function Derivative plotted together
Can you write Python code for it?

#This Calculates Derivative
dx = x[1] - x[0]
dydx = np.gradient(up(x), dx)
pl.plot(x, up(x), label='Atomic Function')
pl.plot(x, AString(x, 1.0, 0, 1, 0), label='Atomic String')
pl.plot(x, dydx, label='A-Function Derivative')

pl.title('Atomic Function and String')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file ".ipynb_checkpoints/AtomicString1-checkpoint.ipynb".
The first task is:
Schematic model of Gravitation explaining General Relativity effects where spacetime Shape, Density and Curvature are deeply related being expressed via the same AString or Atomic Function
Can you write Python code for it?

x = np.arange(-50.0, 50.0, 0.1)
dx = x[1] - x[0]

CS6 = Sum(up(x, 5, -30, 5, 5), up(x, 15, 0, 15, 5))
CS6 = Sum(CS6, up(x, 10, +30, 10, 5))
pl.plot(x, CS6,  label='Spacetime Density')

IntC6 = np.cumsum(CS6)*dx/50
pl.plot(x, IntC6, label='Spacetime Shape')

DerC6 = np.gradient(CS6, dx)
pl.plot(x, DerC6, label='Spacetime Curvature')

LightTrajectory = -10 -IntC6/5
pl.plot(x, LightTrajectory, label='Light Trajectory')

pl.title('Fabric of Curved Spacetime')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file ".ipynb_checkpoints/Aula 10 - Circuitos RL-checkpoint.ipynb".
The first task is:
Exemplo 7.5
No circuito indicado na Figura 7.19, encontre io, vo e i durante todo o tempo, supondo
que a chave fora aberta por um longo período.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Aula 10 - Circuitos RL-checkpoint.ipynb".
The first task is:
Problema Prático 7.3
Determine i e vx no circuito da Figura 7.15. Façamos i(0) = 12 A.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Automation_Lesson1_Intro_InstructorNotes-checkpoint.ipynb".
The first task is:
Indexing
Python indexing is from 0 to length of list - 1
Example:
list_1 = ['Asia', 'Africa', 'Europe']
Asia index = 0,
Africa index = 1,
Europe index = 2
Can you write Python code for it?

list_1 # this is not a very descriptive and identifiable variable
countries = ['Asia', 'Africa', 'Europe']


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
Each row and column corresponds to a rating number. For example, element (1,1) is the number of 1 star reviews that were correctly classified. Element (1,2) is the number of 1 star reviews that were incorrectly classified as 2 stars.  Therefore, the sum of the diagonal represents the total number of correctly classified reviews. As you can see, the bagged decision tree classifier is classifying many four starred reviews as five starred reviews and vice versa.
This indicates that we can improve our results by using more aggregated categories. For example, we can call all four and five star reviews as "good" and all other review ratings as "bad".
Can you write Python code for it?

for review in reviews_json[reviews_json.keys()[0]]:
    print type(review['date'])
    break

latitude_list = []
longitude_list = []
stars_list = []
count_list = []
state_list = []

for biz in biz_json:
    stars_list.append(biz['stars'])
    latitude_list.append(biz['latitude'])
    longitude_list.append(biz['longitude'])
    count_list.append(biz['review_count'])
    state_list.append(biz['state'])

biz_df = pd.DataFrame({'ratings':stars_list, 'latitude':latitude_list, 'longitude': longitude_list, 'review_count': count_list, 'state':state_list})


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
Let's plot the overall distribution of ratings aggregated across all of the states
Can you write Python code for it?

star_hist = pd.DataFrame({'Ratings':stars})
star_hist.plot.hist()


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
Let's take a look at the following regression (information is correlated with review length):
$Rating = \beta_{neg}neg + \beta_{pos}pos + \beta_{num}\text{Std_NumWords} + \epsilon$
Where:
$neg = \frac{\text{Number of Negative Words}}{\text{Total Number of Words}}$
$pos = \frac{\text{Number of Positive Words}}{\text{Total Number of Words}}$
Can you write Python code for it?

harvard_dict = pd.read_csv('HIV-4.csv')
negative_words = list(harvard_dict.loc[harvard_dict['Negativ'] == 'Negativ']['Entry'])
positive_words = list(harvard_dict.loc[harvard_dict['Positiv'] == 'Positiv']['Entry'])

#Use word dictionary from Hu and Liu (2004)
negative_words = open('negative-words.txt', 'r').read()
negative_words = negative_words.split('\n')
positive_words = open('positive-words.txt', 'r').read()
positive_words = positive_words.split('\n')
total_words = negative_words + positive_words
total_words = list(set(total_words))

review_length = []
negative_percent = []
positive_percent = []
for review in reviews:
    length_words = len(review.split())
    neg_words = [x.lower() for x in review.split() if x in negative_words]
    pos_words = [x.lower() for x in review.split() if x in positive_words]
    negative_percent.append(float(len(neg_words))/float(length_words))
    positive_percent.append(float(len(pos_words))/float(length_words))
    review_length.append(length_words)

regression_df = pd.DataFrame({'stars':stars, 'review_length':review_length, 'neg_percent': negative_percent, 'positive_percent': positive_percent})

#Standardize dependent variables
std_vars = ['neg_percent', 'positive_percent', 'review_length']
for var in std_vars:
    len_std = regression_df[var].std()
    len_mu = regression_df[var].mean()
    regression_df[var] = [(x - len_mu)/len_std for x in regression_df[var]]


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
Let's try using Harvard-IV sentiment categories as dependent variables
Can you write Python code for it?

#The R-Squared from using the Harvard Dictionary is 0.1 but with the Hu & Liu word dictionary
X = np.column_stack((regression_df.review_length,regression_df.neg_percent, regression_df.positive_percent))
y = regression_df.stars
X = sm.add_constant(X)

est = sm.OLS(y, X)
est2 = est.fit()
print(est2.summary())


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
Load in the sample JSON file and view its contents
Can you write Python code for it?

#This is a smaller subset of our overall Yelp data
#I randomly chose 5000 reviews from each state and filed them into the JSON file
#Note that for the overall dataset, we have about 2 million reviews.
#That's why we need to use a data management system like MongoDB in order to hold all our data
#and to more efficiently manipulate it

reviews_json = json.load(open(file_directory+reviews_file))
biz_json = json.load(open(file_directory+biz_file))

for key in reviews_json.keys():
    reviews_json[key] = reviews_json[key][0:5000]

#Let's see how reviews_json is set up
print reviews_json.keys()

#We can see that on the highest level, the dictionary keys are the different states
#Let's look at the first entry under Ohio
print reviews_json['OH'][0]['useful']

#So for each review filed under Ohio, we have many different attributes to choose from
#Let's look at what the review and rating was for the first review filed under Ohio
print reviews_json['OH'][0]['text']
print reviews_json['OH'][0]['stars']


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
NOTE: BLUE Estimator does not require normality of errors
Gauss-Markov Theorem states that the ordinary least squares estimate is the best linear unbiased estimator (BLUE) of the regression coefficients ('Best' meaning optimal in terms of minimizing mean squared error) as long as the errors:
(1) have mean zero
(2) are uncorrelated
(3) have constant variance
Can you write Python code for it?

x = np.array(regression_df.stars)
#beta = [3.3648, -0.3227 , 0.5033]
y = [int(round(i)) for i in list(est2.fittedvalues)]
y = np.array(y)
errors = np.subtract(x,y)
np.sum(errors)

fig, ax = plt.subplots(figsize=(5,5))
ax.plot(x, x, 'b', label="data")
ax.plot(x, y, 'o', label="ols")
#ax.plot(x, est2.fittedvalues, 'r--.', label="OLS")
#ax.plot(x, iv_u, 'r--')
#ax.plot(x, iv_l, 'r--')
ax.legend(loc='best');

#Do a QQ plot of the data
fig = sm.qqplot(errors)
plt.show()


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
Now, let's create two lists for all the reviews in Ohio:

One that holds all the reviews
One that holds all the ratings
Can you write Python code for it?

#We want to split up reviews between text and labels for each state
reviews = []
stars = []
for key in reviews_json.keys():
    for review in reviews_json[key]:
        reviews.append(review['text'])
        stars.append(review['stars'])

#Just for demonstration, let's pick out the same review example as above but from our respective lists
print reviews[0]
print stars[0]


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
SKLearn uses what's known as a pipeline. Instead of having to declare each of these objects on their own and passing them into each other, we can just create one object with all the necessary options specified and then use that to run the algorithm. For each pipeline below, we specify the vector to be the CountVectorizer object we have defined above, set it to use tfidf, and then specify the classifier that we want to use.
Below, we create a separate pipeline for Random Forest, a Bagged Decision Tree, and Multinomial Logistic Regression. We then append the results to the dataframe that we've already created.
Can you write Python code for it?

# random_forest = Pipeline([('vect', vectorizer),
#                         ('tfidf', TfidfTransformer()),
#                         ('clf', RandomForestClassifier())])
# random_forest.set_params(clf__n_estimators=100, clf__criterion='entropy').fit(train_reviews, train_ratings)
# output['random_forest'] = random_forest.predict(test_reviews)

# output['Accurate'] = np.where(output['Rating'] == output['random_forest'], 1, 0)
# accurate_percentage = float(sum(output['Accurate']))/float(len(output))
# print accurate_percentage

# bagged_dt = Pipeline([('vect', vectorizer),
#                         ('tfidf', TfidfTransformer()),
#                         ('clf', BaggingClassifier())])
# bagged_dt.set_params(clf__n_estimators=100, clf__n_jobs=1).fit(train_reviews, train_ratings)
# output['bagged_dt'] = bagged_dt.predict(test_reviews)

# output['Accurate'] = np.where(output['Rating'] == output['bagged_dt'], 1, 0)
# accurate_percentage = float(sum(output['Accurate']))/float(len(output))
# print accurate_percentage

multi_logit = Pipeline([('vect', vectorizer),
                        ('tfidf', TfidfTransformer()),
                        ('clf', MultinomialNB())])
multi_logit.set_params(clf__alpha=1, clf__fit_prior = True, clf__class_prior = None).fit(train_reviews, train_ratings)
output['multi_logit'] = multi_logit.predict(test_reviews)

output['Accurate'] = np.where(output['Rating'] == output['multi_logit'], 1, 0)
accurate_percentage = float(sum(output['Accurate']))/float(len(output))
print accurate_percentage

random_forest = Pipeline([('vect', vectorizer),
                        ('tfidf', TfidfTransformer()),
                        ('clf', RandomForestClassifier())])
random_forest.set_params(clf__n_estimators=100, clf__criterion='entropy').fit(train_reviews, train_ratings)
output['random_forest'] = random_forest.predict(test_reviews)

output['Accurate'] = np.where(output['Rating'] == output['random_forest'], 1, 0)
accurate_percentage = float(sum(output['Accurate']))/float(len(output))
print accurate_percentage


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
Test results using all of the states

0.5383 from Naive TF-IDF Linear SVM
0.4567 from Naive TF-IDF Linear SVM using Harvard-IV dictionary
0.5241 from Naive TF-IDF Bagged DT using 100 estimators
0.496 from Naive TF-IDF Bagged DT using 100 estimators and Harvard-IV dictionary
0.5156 from Naive TF-IDF RandomForest and Harvard-IV dictionary
0.53 from Naive TF-IDF RF
0.458 from Naive TF-IDF SVM

As you can see, none of the above classifiers performs significantly better than a fair coin toss. This is most likely due to the heavily skewed distribution of review ratings. There are many reviews that receive 4 or 5 stars, therefore it is likely that the language associated with each review is being confused with each other. We can confirm this by looking at the "confusion matrix" of our predictions.
Can you write Python code for it?

print metrics.confusion_matrix(test_ratings, bagged_dt.predict(test_reviews), labels = [1, 2, 3, 4, 5])


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
This is a function that we'll use later to plot the results of a linear SVM classifier
Can you write Python code for it?

def plot_coefficients(classifier, feature_names, top_features=20):
    coef = classifier.coef_.ravel()[0:200]
    top_positive_coefficients = np.argsort(coef)[-top_features:]
    top_negative_coefficients = np.argsort(coef)[:top_features]
    top_coefficients = np.hstack([top_negative_coefficients, top_positive_coefficients])
    #create plot
    plt.figure(figsize=(15, 5))
    colors = ['red' if c < 0 else 'blue' for c in coef[top_coefficients]]
    plt.bar(np.arange(2 * top_features), coef[top_coefficients], color=colors)
    feature_names = np.array(feature_names)
    plt.xticks(np.arange(1, 1 + 2 * top_features), feature_names[top_coefficients], rotation=60, ha='right')
    plt.show()

def bayesian_average()

#This is the main folder where all the modules and JSON files are stored on my computer.
#You need to change this to the folder path specific to your computer
file_directory = "/Users/robertsonwang/Desktop/Python/Yelp_class/yelp-classification/"
reviews_file = "cleaned_reviews_states_2010.json"
biz_file = "cleaned_business_data.json"


I am working on the file ".ipynb_checkpoints/Basic_Review_Analysis-checkpoint.ipynb".
The first task is:
We run the following linear regression model for each of the states:
$Rating = \beta_{1} Longitude + \beta_{2} Latitude + \beta_{3} Num of Reviews + \epsilon$
Can you write Python code for it?

for state in states:
    state_df = biz_df[biz_df.state == state]
    state_df_filt = state_df[(np.abs(state_df.longitude-state_df.longitude.mean()) <= 2*state_df.longitude.std()) \
                             & (np.abs(state_df.latitude-state_df.latitude.mean()) <= 2*state_df.latitude.std())]
    state_df_filt['longitude'] = (state_df_filt.longitude - state_df.longitude.mean())/state_df.longitude.std()
    state_df_filt['latitude'] = (state_df_filt.latitude - state_df.latitude.mean())/state_df.latitude.std()
    state_df_filt['review_count'] = (state_df_filt.review_count - state_df.review_count.mean())/state_df.review_count.std()

    X = np.column_stack((state_df_filt.longitude, state_df_filt.latitude, state_df_filt.review_count))
    y = state_df_filt.ratings

    est = sm.OLS(y, X)
    est2 = est.fit()
    print(est2.summary())
    print state


I am working on the file ".ipynb_checkpoints/Bifurcations-checkpoint.ipynb".
The first task is:
Example #1:  Saddle node bifurcation
consider the problem
$$ f(r,x) = r + x^2$$
and we will define $f$ and $\partial f/\partial x$ using inlined python lambda functions
Can you write Python code for it?

f = lambda r,x: r + x*x
f_x = lambda r,x: 2.*x

x = numpy.linspace(-4,4,100)
r = numpy.linspace(-4,4,100)

bifurcation_plot(f,f_x,r,x)


I am working on the file ".ipynb_checkpoints/Bifurcations-checkpoint.ipynb".
The first task is:
Example #2:  Logistic equation with constant harvesting
$$ dx/dt = x(1-x) - h $$
Can you write Python code for it?

f = lambda h,x: x*(1-x) - h
f_x = lambda h,x: 1. - 2.*x

x = numpy.linspace(0,1.,100)
h = numpy.linspace(0,.5,100)

bifurcation_plot(f,f_x,h,x,rlabel='h')


I am working on the file ".ipynb_checkpoints/Bifurcations-checkpoint.ipynb".
The first task is:
Example #3:  transcritical bifurcation
$$f(r,x) = rx - x^2$$
Can you write Python code for it?

f = lambda r,x: r*x -  x*x
f_x = lambda r,x: r - 2.*x

x = numpy.linspace(-1.,1.,100)
r = numpy.linspace(-1.,1.,100)

bifurcation_plot(f,f_x,r,x)


I am working on the file ".ipynb_checkpoints/Bifurcations-checkpoint.ipynb".
The first task is:
Example #4  sub-critical pitchfork bifurcation
$$f(r,x) = rx + x^3$$
Can you write Python code for it?

f = lambda r,x: r*x +  x**3
f_x = lambda r,x: r + 3.*x**2

x = numpy.linspace(-1.,1.,100)
r = numpy.linspace(-1.,1.,100)

bifurcation_plot(f,f_x,r,x)


I am working on the file ".ipynb_checkpoints/Bifurcations-checkpoint.ipynb".
The first task is:
FIXME:  this plot needs to mask out the spurious stable branch which is a plotting error
And now you can play with your own function
Can you write Python code for it?

f = lambda r,x: # < your function here >
f_x = lambda r,x: # < your derivative here >

# Adjust your domain and resolution
x = numpy.linspace(-10.,10.,100)
r = numpy.linspace(-10.,10.,100)

#plot and pray (and watch out for glitches)
bifurcation_plot(f,f_x,r,x)


I am working on the file ".ipynb_checkpoints/Bifurcations-checkpoint.ipynb".
The first task is:
Plotting Bifurcations
GOAL: Find where $f(x,r) = 0$ and label the stable and unstable branches.
A bifurcation diagram provides information on how the fixed points of a dynamical system $f(x,r)=0$ vary as a function of the control parameter $r$
Here we write a snazzy little python function to extract the zero contour and label it with respect to whether it is a stable branch with ($\partial f/\partial x < 0$) or an unstable branch ($\partial f/\partial x > 0 $)
Can you write Python code for it?

def bifurcation_plot(f,f_x,r,x,rlabel='r'):
    """ produce a bifurcation diagram for a function f(r,x) given
        f and its partial derivative f_x(r,x) over a domain given by numpy arrays r and x

        f(r,x)  :  RHS function of autonomous ode dx/dt = f(r,x)
        f_x(r,x):  partial derivative of f with respect to x
        r       :  numpy array giving r coordinates of domain
        x       :  numpy array giving x coordinates of domain
        rlabel  :  string for x axis parameter label
    """
    # set up a mesh grid and extract the 0 level set of f
    R,X = numpy.meshgrid(r,x)
    plt.figure()
    CS = plt.contour(R,X,f(R,X),[0],colors='k')
    plt.clf()

    c0 = CS.collections[0]
    # for each path in the contour extract vertices and mask by the sign of df/dx
    for path in c0.get_paths():
        vertices = path.vertices
        vr = vertices[:,0]
        vx = vertices[:,1]
        mask = numpy.sign(f_x(vr,vx))
        stable = mask < 0.
        unstable = mask > 0.

        # plot the stable and unstable branches for each path
        plt.plot(vr[stable],vx[stable],'b')
        plt.hold(True)
        plt.plot(vr[unstable],vx[unstable],'b--')

    plt.xlabel('parameter {0}'.format(rlabel))
    plt.ylabel('x')
    plt.legend(('stable','unstable'),loc='best')
    plt.xlim(r[0],r[-1])
    plt.ylim(x[0],x[-1])


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
After applying logarithm test to the above data set, it seems that only LSTAT is responding correctly and is getting normally distributed. PTRATIO and INDUS is not having any significant impact in the normalization.
Now let us examine the co-relation between the features and the MEDV feature:
Can you write Python code for it?

fig = plt.figure(figsize=(14,7))

plt.subplot(2,2,1)
x = np.log(boston_data[['indus']])
sns.regplot(x=x, y="medv", data=boston_data, color="b")

plt.subplot(2,2,2)
x2 = boston_data[['rm']]
sns.regplot(x=x2, y="medv", data=boston_data, color="r")

plt.subplot(2,2,3)
x3 = np.log(boston_data[['ptratio']])
sns.regplot(x=x3, y="medv", data=boston_data, color="g")

plt.subplot(2,2,4)
x4 = np.log(boston_data[['lstat']])
sns.regplot(x=x4, y="medv", data=boston_data, color="y")

plt.show()


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Building the data with the selected features
Can you write Python code for it?

boston_data['lstat_log']=np.log(boston_data['lstat'])
boston_data_test['lstat_log_test']=np.log(boston_data_test['lstat'])

#boston_data['ptratio_log']=np.log(boston_data['ptratio'])
#boston_data_test['ptratio_log_test']=np.log(boston_data_test['ptratio'])

#boston_data['indus_log']=np.log(boston_data['indus'])
#boston_data_test['indus_log_test']=np.log(boston_data_test['indus'])

X = boston_data[['rm','lstat_log']]
X_bd_test=boston_data_test[['rm','lstat_log_test']]

y = boston_data[['medv']]


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Finding the importance of feature in the regression algorithm
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
From the above dist plots we can conclude that the distribution of the data in 3 out of 4 features have skewed data distribution. The feature : RM is currently having the distribution in a somewhat normalized fashion.
PTRATIO is negatively skewed;
LSTAT and INDUS are positively skewed
Now we need to normalize these data sets to bring the data into a normal distribution.
Can you write Python code for it?

fig = plt.figure(figsize=(14,7))

plt.subplot(2,2,1)
sns.distplot(np.log(boston_data['indus']), rug=True, color="b")
plt.axvline(np.log(boston_data['indus']).mean(), color="b", linestyle='solid', linewidth=2)
plt.axvline(np.log(boston_data['indus']).median(), color="b", linestyle='dashed', linewidth=2)

plt.subplot(2,2,2)
sns.distplot(boston_data['rm'], rug=True, color="r")
plt.axvline(boston_data['rm'].mean(), color="r", linestyle='solid', linewidth=2)
plt.axvline(boston_data['rm'].median(), color="r", linestyle='dashed', linewidth=2)

plt.subplot(2,2,3)
sns.distplot(np.log(boston_data['ptratio']), rug=True, color="g")
plt.axvline(np.log(boston_data['ptratio']).mean(), color="g", linestyle='solid', linewidth=2)
plt.axvline(np.log(boston_data['ptratio']).median(), color="g", linestyle='dashed', linewidth=2)

plt.subplot(2,2,4)
sns.distplot(np.log(boston_data['lstat']), rug=True, color="y")
plt.axvline(np.log(boston_data['lstat']).mean(), color="y", linestyle='solid', linewidth=2)
plt.axvline(np.log(boston_data['lstat']).median(), color="y", linestyle='dashed', linewidth=2)

plt.show()


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
From the above plot we can understand that the price of the houses ranges in an average price range of: 22.76 (in $1000).
So any ML algorithm with a bad accuracy will end up predicting the mean value.

Understanding the features and its relation with the price of the boston houses "medv"

From the data set, let us consider the following features (only the prime contenders out of 15 features):
1. crim (per capita crime rate by town):
    An area with higher crime rate is expected to have lesser price than with a well established areas.
2. indus (proportion of non-retail business acres per town):
    Non retail business can be an important factor in the prices of house as it provides convienence to the house owners and people. But acres of non-retail business land doesn't give much insight into the prices of the house. Hence we can give this feature low priority, subject to correlation with medv data.
3. nox (nitrogen oxides concentration):
    Nitrogen oxide can be a major factor in the housing prices as the preferences of buying a house with lower pollution would be higher.
4. rm (average number of rooms per dwelling):
    Higher the number of rooms, higher the price.
5. dis (weighted mean of distances to five Boston employment centres):
    Closer the distance to offices, expected to have more interest in the areas as it reduces commute. Not sure about the American way, but closer to offices, higher the house prices.
6. ptratio (pupil-teacher ratio by town):
    Though i am not 100% sure about the relation between pupil and teacher. I am assuming more the number of pupil to teacher ratio, people are expected to send kids there, in turn making the prices higher. We can give this feature low priority as it may not be closely related to house pricing, subject to correlation with medv data.
7. black (the proportion of blacks by town):
    Subject to correlation with the target data
8. lstat (lower status of the population) :
    People who are earning lower wages are not expected to live in a high priced...
Can you write Python code for it?

ax = plt.subplots(figsize = (14,7))
sns.heatmap(boston_data[['crim','indus','nox','rm','dis','rad','tax','ptratio','black','lstat','medv']].corr(),
            linecolor = 'white', square=True, annot=True)
plt.show()

sns.jointplot(x='lstat', y='medv', data=boston_data, color="r", kind="reg")
plt.show()


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Generate the test dataframe as csv output
Can you write Python code for it?

submission = pd.DataFrame({
              "ID": boston_data_test['ID'],
              "medv": bd_predict
        })
submission.to_csv(data_path+'output.csv', index=False)


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Loading the boston dataset - Train and Test
Can you write Python code for it?

data_path = "C:/Users/Rishu/Desktop/dATA/boston/"

boston_data=pd.read_csv(data_path+'train.csv')

boston_data.info()

boston_data.head()

boston_data_test=pd.read_csv(data_path+'test.csv')

boston_data_test.head()


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Model Fitting

Using DecisionTreeRegressor for finding the maximum score
Can you write Python code for it?

from sklearn.tree import DecisionTreeRegressor

max_score = 0
max_depth = 0

def decision_tree(j):
    dtr = DecisionTreeRegressor(random_state=0,max_depth=j)
    return dtr.fit(X_train, y_train)

for i in range(1,11):
    _dtr = decision_tree(i)
    clf_score = _dtr.score(X_cv,y_cv)
    print("Decision Tree Regressor at max_depth:",i," scored: ",clf_score)
    if clf_score>max_score:
        max_score = clf_score
        max_depth = i


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Most co-related features

Based on the above co-relation chart, we would like to take into consider the features which are more closely related to the target value. The features in consideration are:
1. indus
2. rm      : Highest positive correlation with medv (coeff: 0.69)
3. ptratio
4. lstat   : Highly negative correlated feature with coefficient of -0.74
Now let us visualize the distribution of the 4 selected features in a pairplot
Can you write Python code for it?

# Pair plot of the features
sns.pairplot(boston_data[['indus','rm','ptratio','lstat','medv']])
plt.show()


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Plotting the target price value: "medv"
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)

sns.distplot(boston_data['medv'], rug=True, color="b")
plt.axvline(boston_data['medv'].mean(), color="b", linestyle='solid', linewidth=2)
plt.axvline(boston_data['medv'].median(), color="b", linestyle='dashed', linewidth=2)

plt.show()

print ("Mean Price value :",boston_data['medv'].mean())
print ("Standard Deviation:",boston_data['medv'].std())


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Selecting the max depth
Can you write Python code for it?

print("The maximum score is achieved at a depth of : ",max_depth," with score of :",max_score)


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
Understanding the distribution and relationship of the data

Describing the data to understand the mean and std for all of the features
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/BostonHousing-checkpoint.ipynb".
The first task is:
We can conclude that rm and lstat are two of them most important factor in the prices of the house in boston area.

Visualizing the decision made on the dataset
Can you write Python code for it?

from IPython.display import Image
import pydotplus
from sklearn.externals.six import StringIO
from sklearn import tree

dot_data = StringIO()
tree.export_graphviz(dtr_clf, out_file=dot_data,
    feature_names=X_train.columns,
    class_names="medv",
    filled=True, rounded=True,
    special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())


I am working on the file ".ipynb_checkpoints/Breast-cancer-clean-checkpoint.ipynb".
The first task is:
After encoding the class labels (diagnosis) in an array y, the malignant tumors are now represented as class 1, and the benign tumors are represented as class 0, respectively, which we can illustrate by calling the transform method of LabelEncoder on two dummy class labels
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Breast-cancer-clean-checkpoint.ipynb".
The first task is:
Dataset
Breast Cancer Wisconsin dataset, which contains 569 samples of malignant and benign tumor cells.
* The first two columns in the dataset store the unique ID numbers of the samples and the corresponding diagnosis (M=malignant, B=benign), respectively.
* The columns 3-32 contain 30 real-value features that have been computed from digitized images of the cell nuclei, which can be used to build a model to predict whether a tumor is benign or malignant. The Breast Cancer Wisconsin dataset has been deposited on the UCI machine learning repository and more detailed information about this dataset can be found at https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic).
Goal: In tumor diagnosis, classification of breast cancer tumor using machine learning (we are more concerned about the detection of malignant tumors in order to help a patient with the appropriate treatment)
load dataset
Can you write Python code for it?

#load dataset from ucsi website
import pandas as pd
df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data', header=None)
df.head()


I am working on the file ".ipynb_checkpoints/Breast-cancer-clean-checkpoint.ipynb".
The first task is:
Diagnosing bias and variance problems with learning curves
Use the learning curve function from scikit-learn to evaluate the model

Via the train_sizes parameter in the learning_curve function, we can control the absolute or relative number of training samples that are used to generate the learning curves.
Here, we set train_sizes=np.linspace(0.1, 1.0, 10) to use 10 evenly spaced relative intervals for the training set sizes.
The learning_curve function uses stratified k-fold cross-validation to calculate the cross-validation accuracy, and we set Diagnosing bias and variance problems with learning curves via the cv parameter.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve

pipe_lr = Pipeline([('scl', StandardScaler()),
                    ('clf', LogisticRegression(penalty='l2', random_state=0))])
train_sizes, train_scores, test_scores =learning_curve(estimator=pipe_lr,
                      X=X_train,
                      y=y_train,
                      train_sizes=np.linspace(0.1, 1.0, 10),
                      cv=10,
                       n_jobs=1)


I am working on the file ".ipynb_checkpoints/Breast-cancer-clean-checkpoint.ipynb".
The first task is:
Mapping using LabelEncoder class implemented in scikit learn
Next, we assign the 30 features to a NumPy array X. Using LabelEncoder, we transform the class labels from their original string representation (M and B) into integers:
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder
X = df.loc[:, 2:].values
y = df.loc[:, 1].values

le = LabelEncoder()
y = le.fit_transform(y)


I am working on the file ".ipynb_checkpoints/Breast-cancer-clean-checkpoint.ipynb".
The first task is:
Plotting a receiver operating characteristic
Receiver operator characteristic (ROC) graphs are useful tools for selecting models for classification based on their performance with respect to the false positive and true positive rates, which are computed by shifting the decision threshold of the classifier.
The diagonal of an ROC graph can be interpreted as random guessing, and classification models that fall below the diagonal are considered as worse than random guessing. A perfect classifier would fall into the top-left corner of the graph with a true positive rate of 1 and a false positive rate of 0.

Based on the ROC curve, we can then compute the so-called area under the curve (AUC) to characterize the performance of a classification model.

Next plot an ROC curve of a classifier that only uses two features from the Breast Cancer Wisconsin dataset to predict whether a tumor is benign or malignant.
Can you write Python code for it?

pipe_lr = Pipeline([('scl', StandardScaler()),
                    ('pca', PCA(n_components=2)),
                    ('clf', LogisticRegression(penalty='l2',
                                               random_state=0,
                                               C=100.0))])

X_train2 = X_train[:, [4, 14]]
cv = list(StratifiedKFold(n_splits=3, random_state=1).split(X_train, y_train))

fig = plt.figure(figsize=(7, 5))
mean_tpr = 0.0
mean_fpr = np.linspace(0, 1, 100)
all_tpr = []

for i, (train, test) in enumerate(cv):
    probas = pipe_lr.fit(X_train2[train],
                         y_train[train]).predict_proba(X_train2[test])
    fpr, tpr, thresholds = roc_curve(y_train[test],
                                     probas[:, 1],
                                     pos_label=1)
    mean_tpr += interp(mean_fpr, fpr, tpr)
    mean_tpr[0] = 0.0
    roc_auc = auc(fpr, tpr)
    plt.plot(fpr,
             tpr,
             lw=1,
             label='ROC fold %d (area = %0.2f)'
                   % (i + 1, roc_auc))

plt.plot([0, 1],
         [0, 1],
         linestyle='--',
         color=(0.6, 0.6, 0.6),
         label='random guessing')

mean_tpr /= len(cv)
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)
plt.plot(mean_fpr, mean_tpr, 'k--',
         label='mean ROC (area = %0.2f)' % mean_auc, lw=2)
plt.plot([0, 0, 1],
         [0, 1, 1],
         lw=2,
         linestyle=':',
         color='black',
         label='perfect performance')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('false positive rate')
plt.ylabel('true positive rate')
plt.title('Receiver Operator Characteristic')
plt.legend(loc="lower right")

# plt.tight_layout()
# plt.savefig('./figures/roc.png', dpi=300)
plt.show()

pipe_lr = pipe_lr.fit(X_train2, y_train)
y_pred2 = pipe_lr.predict(X_test[:, [4, 14]])

from sklearn.metrics import roc_auc_score
from sklearn.metrics import accuracy_score
print('ROC AUC: %.3f' % roc_auc_score( y_true=y_test, y_score=y_pred2))
print('Accuracy: %.3f' % accuracy_score( y_true=y_test,...


I am working on the file ".ipynb_checkpoints/Breast-cancer-clean-checkpoint.ipynb".
The first task is:
Results
As we can see in the abover learning curve plot, our model performs quite well on the test dataset. However, it may be slightly overfitting the training data indicated by a relatively small, but visible, gap between the training and cross-validation accuracy curves.
Addressing overfitting and underfitting with validation curves
Validation curves are a useful tool for improving the performance of a model by addressing issues such as overfitting or underfitting. Validation curves are related to learning curves, but instead of plotting the training and test accuracies as functions of the sample size, we vary the values of the model parameters, for example, the inverse regularization parameter C in logistic regression. Let's go ahead and see how we create validation curves via sckit-learn:
Can you write Python code for it?

from sklearn.learning_curve import validation_curve

param_range = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]
train_scores, test_scores = validation_curve(
               estimator=pipe_lr,
                 X=X_train,
               y=y_train,
               param_name='clf__C',
              param_range=param_range,
                cv=10)

train_mean = np.mean(train_scores, axis=1)
train_std = np.std(train_scores, axis=1)
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)

plt.plot(param_range, train_mean,
         color='blue', marker='o',
         markersize=5,
         label='training accuracy')
plt.fill_between(param_range, train_mean + train_std,
                  train_mean - train_std, alpha=0.15,
                 color='blue')
plt.plot(param_range, test_mean,
          color='green', linestyle='--',
          marker='s', markersize=5,
         label='validation accuracy')
plt.fill_between(param_range,
                  test_mean + test_std,
                 test_mean - test_std,
                 alpha=0.15, color='green')
plt.grid()
plt.xscale('log')
plt.legend(loc='lower right')
plt.xlabel('Parameter C')
plt.ylabel('Accuracy')
plt.ylim([0.92, 1.0])
plt.show()


I am working on the file ".ipynb_checkpoints/Breast-cancer-clean-checkpoint.ipynb".
The first task is:
Test/train split
Split the dataset into training and testing subsets. The goal is to
 * train their model on a subset of the data, called the training set
 * and evaluate its performance on the rest of the data, the test set.
A general rule of thumb is to use approximately 80% of the data for training & 20% for test. Let's now split our breast cancer data
Can you write Python code for it?

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test =  train_test_split(X, y, test_size=0.20, random_state=1)


I am working on the file ".ipynb_checkpoints/Building the Pipeline part 2 Write Up-checkpoint.ipynb".
The first task is:
Ensemble methods performed the best overall, but there was wide variance. Hypertuning made a big difference in model fit according to AUC-ROC. Of the 213 specifications that worked, Gradient boosting classifer resulted in four of the top 10 and four of the bottom ten models. Gradient Boosting classifiers split on max depth, the top performers used max dapth of 5 while the bottom used max depth of 50.
Random forests performed very strongly and had much lower variance in the results; AUC-ROC ranged from 72 to 83 percent. (Compare with GB with a range from roughly 48 to 83). Random Forests also split on max depth with shorter trees out performing longer ones.
Simpler methods such as Decision Trees and KNN almost matched the ensemble methods under certain specifications. DT specifications with trees of depth 1 captured 65 to 72 percent of the area under the curve, which gives us a baseline of comparison. These models also beat trees with max depth of 20, 50 and 100, which suggest overall that with a limited number of features depths 4 or 5 times greater than the feature set length will overfit. Linear regression performed below the 1-deep trees; AUC-ROC range from 63 to 65 over 10 specifications. Naive bayes hit just above that range. SVD and SGD did not run directly from Rayid's set up.
Can you write Python code for it?

second_grid = r.read_csv("refined_results.csv")
sg = second_grid.sort_values(by="auc-roc", ascending=False)
sg.head(10)
#Discuss speed and how they do with an extra feature.


I am working on the file ".ipynb_checkpoints/Building the Pipeline part 2 Write Up-checkpoint.ipynb".
The first task is:
While I was working on improving my previous homework and determining a method for feature selection, I ran the small grid search of Magic Loops with the specifications Rayid selected. This ran over night, but unfortunately I ran it before I thought to add a timer. From observation the gradient boosting and ADA boosting appeared to be some of the slower models. The results informed the smaller loop I ran on the features selected by Random Forest, which were nearly identical to Rayid's feature set except I included monthly_income.
Can you write Python code for it?

first_grid = r.read_csv("small_loop_result.csv")
fg = first_grid.sort_values(by="auc-roc")
fg.head(10)

#top 10
fg.tail(10).sort_values(by="auc-roc", ascending=False)


I am working on the file ".ipynb_checkpoints/CNTK_model_scoring_on_Spark_walkthrough-checkpoint.ipynb".
The first task is:

Evaluate model accuracy
The trained model assigns a class label (represented by an integer value 0-9) to each image. We now compare the true and predicted class labels to evaluate our model's accuracy.
Can you write Python code for it?

df = pd.DataFrame(results, columns=['true_label', 'predicted_label'])

num_correct = sum(df['true_label'] == df['predicted_label'])
num_total = len(results)
print('Correctly predicted {} of {} images ({:0.2f}%)'.format(num_correct, num_total, 100 * num_correct / num_total))


I am working on the file ".ipynb_checkpoints/CNTK_model_scoring_on_Spark_walkthrough-checkpoint.ipynb".
The first task is:

Define functions to be used by worker nodes
The following functions will be used during scoring to load, preprocess, and score images. A class label (integer in the range 0-9) will be returned for each image, along with its filename.
Can you write Python code for it?

def get_preprocessed_image(my_image, mean_image):
    ''' Reshape and flip RGB order '''
    my_image = my_image.astype(np.float32)
    bgr_image = my_image[:, :, ::-1] # RGB -> BGR
    image_data = np.ascontiguousarray(np.transpose(bgr_image, (2, 0, 1)))
    image_data -= mean_image
    return(image_data)

def run_worker(records):
    ''' Scoring script run by each worker '''
    loaded_model = load_model(SparkFiles.get('./model.dnn'))
    mean_image = mean_image_bc.value

    # Iterate through the records in the RDD.
    # record[0] is the image data
    # record[1] is the true label
    # record[2] is the file name
    for record in records:
        preprocessed_image = get_preprocessed_image(record[0], mean_image)
        dnn_output = loaded_model.eval({loaded_model.arguments[0]: [preprocessed_image]})
        yield record[1], np.argmax(np.squeeze(dnn_output))