I am working on the file "MovieReviewSentimentAnalysis/MovieReveiw/.ipynb_checkpoints/NLP_Movies-checkpoint.ipynb".
The first task is:
Well, it wasn't quite as good as our well-tuned Logistic Regression classifier, but that is a pretty good score considering how little we had to do!
One last classifier to try is the SGD classifier, which comes in handy when you need speed on a really large number of training examples/features.
Which machine learning algorithm it ends up using depends on what you set for the loss function. If we chose loss = 'log', it would essentially be identical to our previous logistic regression model. We want to try something different, but we also want a loss option that includes probabilities. We need those probabilities if we are going to be able to calculate the area under a ROC curve. Looking at the documentation, it seems a 'modified_huber' loss would do the trick! This will be a Support Vector Machine that uses a linear kernel.
Can you write Python code for it?

from sklearn.linear_model import SGDClassifier as SGD

sgd_params = {'alpha': [0.00006, 0.00007, 0.00008, 0.0001, 0.0005]} # Regularization parameter

model_SGD = GridSearchCV(SGD(random_state = 0, shuffle = True, loss = 'modified_huber'),
                        sgd_params, scoring = 'roc_auc', cv = 20) # Find out which regularization parameter works the best.

model_SGD.fit(X, y_train) # Fit the model.


I am working on the file "MovieReviewSentimentAnalysis/MovieReveiw/.ipynb_checkpoints/NLP_Movies-checkpoint.ipynb".
The first task is:
You can investigate which parameters did the best and what scores they received by looking at the model_LR object.
Can you write Python code for it?


I am working on the file "MovieTime.ipynb".
The first task is:
Before we generate our video frames, let's check our work

We'll load our image.
Build the interpolated waypoints list.
Draw the points on the image using our draw_point method.
Plot the results
Can you write Python code for it?

# load the image
merged = load_image3(output_mosaic)
# interpolate the waypoints
interp = interpolate_waypoints(waypoints)
# draw them on our scene
[draw_point(pt[1],pt[0],merged) for pt in interp]
# display the scene
plt.figure(0,figsize=(18,18))
plt.imshow(merged)
plt.title("merged")


I am working on the file "MovieTime.ipynb".
The first task is:
Finally, let's make a movie.

Our friend AVConv, which is like ffmpeg is a handy command line util for transcoding video.
AVConv can also convert a series of images into a video and vice versa.
We'll set up our command and use subprocess to make the call.
Can you write Python code for it?

# avconv -framerate 30 -f image2 -i ./movie/img%06d.png -b 65536k out.mpg
os.system("rm ./movie/*.png")
framerate = 30
output = "out.mpg"
command = ["avconv","-framerate", str(framerate), "-f", "image2", "-i", "./movie/img%06d.png", "-b", "65536k", output]
os.system(" ".join(command))


# avconv -framerate 30 -f image2 -i ./movie/img%06d.png -b 65536k out.mpg;
#os.system("rm ./movie/*.png")
framerate = 30
output = "out.mpg"
command = ["avconv","-framerate", str(framerate), "-f", "image2", "-i", "./movie/img%06d.png", "-b", "65536k", output]
os.system(" ".join(command))


I am working on the file "MovieTime.ipynb".
The first task is:
Here we'll call the functions we wrote.

First we open the scene and get the width and height.
Then from the geotransorm we'll reproject those points to lat and long.
Can you write Python code for it?

# TLDR: pixels => UTM coordiantes => Lat Long
raster=output_mosaic
# Load the GDAL File
ds=gdal.Open(raster)
# get the geotransform
gt=ds.GetGeoTransform()
# get the width and height of our image
cols = ds.RasterXSize
rows = ds.RasterYSize
# Generate the coordinates of our image in utm
ext=GetExtent(gt,cols,rows)
# get the spatial referencec object
src_srs=osr.SpatialReference()
# get the data that will allow us to move from UTM to Lat Lon.
src_srs.ImportFromWkt(ds.GetProjection())
tgt_srs = src_srs.CloneGeogCS()
extents = ReprojectCoords(ext,src_srs,tgt_srs)
print extents


I am working on the file "MovieTime.ipynb".
The first task is:
Just like before we clean up our data and distill it down to just the scenes we want.
Can you write Python code for it?

# now let's clean up the datetime stuff
# make a shapely shape from our aoi
portland = shape(areaAOI)
footprints = []
overlaps = []
# go through the geometry from our api call, convert to a shape and calculate overlap area.
# also save the shape for safe keeping
for footprint in scenes["geometry"].tolist():
    s = shape(footprint)
    footprints.append(s)
    overlap = 100.0*(portland.intersection(s).area / portland.area)
    overlaps.append(overlap)
# take our lists and add them back to our dataframe
scenes['overlap'] = pd.Series(overlaps, index=scenes.index)
scenes['footprint'] = pd.Series(footprints, index=scenes.index)
# now make sure pandas knows about our date/time columns.
scenes["acquired"] = pd.to_datetime(scenes["acquired"])
scenes["published"] = pd.to_datetime(scenes["published"])
scenes["updated"] = pd.to_datetime(scenes["updated"])
scenes.head()

# Now let's get it down to just good, recent, clear scenes
clear = scenes['cloud_cover']<0.4
good = scenes['quality_category']=="standard"
recent = scenes["acquired"] > datetime.date(year=2017,month=1,day=1)
partial_coverage = scenes["overlap"] > 10
good_scenes = scenes[(good&clear&recent&partial_coverage)]
print good_scenes


I am working on the file "MovieTime.ipynb".
The first task is:
Let's check our work

First we'll create a draw point function that just puts a red dot at given pixel.
We'll get our scene, and map all of the lat/long points in our path to pixel values.
Finally we'll load our image, plot the points and show our results
Can you write Python code for it?

def draw_point(x,y,img,t=40):
    h,w,d = img.shape
    y = h-y
    img[(y-t):(y+t),(x-t):(x+t),:] = [255,0,0]
h,w,c = merged.shape
waypoints = [poor_mans_lat_lon_2_pix(point[0],point[1],w,h,extents) for point in pathAOI["coordinates"]]
print waypoints
merged = load_image3(output_mosaic)
[draw_point(pt[1],pt[0],merged) for pt in waypoints]
plt.figure(0,figsize=(18,18))
plt.imshow(merged)
plt.title("merged")


I am working on the file "MovieTime.ipynb".
The first task is:
Let's pull it all together to do something cool.

Let's reuse a lot of our code to make a movie of our travel around San Francisco.
We'll first select a bunch of recent scenes, activate, and download them.
After that we'll create a mosaic, a path, and trace the path through the moasic.
We'll use the path to crop subregions, save them as images, and create a video.
First step is to trace our AOI and a path through it.
Can you write Python code for it?

# Basemap Mosaic (v1 API)
mosaicsSeries = 'global_quarterly_2017q1_mosaic'
# Planet tile server base URL (Planet Explorer Mosaics Tiles)
mosaicsTilesURL_base = 'https://tiles0.planet.com/experimental/mosaics/planet-tiles/' + mosaicsSeries + '/gmap/{z}/{x}/{y}.png'
# Planet tile server url
mosaicsTilesURL = mosaicsTilesURL_base + '?api_key=' + api_keys["PLANET_API_KEY"]
# Map Settings
# Define colors
colors = {'blue': "#009da5"}
# Define initial map center lat/long
center = [37.774929,-122.419416]
# Define initial map zoom level
zoom = 11
# Set Map Tiles URL
planetMapTiles = TileLayer(url= mosaicsTilesURL)
# Create the map
m = Map(
    center=center,
    zoom=zoom,
    default_tiles = planetMapTiles # Uncomment to use Planet.com basemap
)
# Define the draw tool type options
polygon = {'shapeOptions': {'color': colors['blue']}}
rectangle = {'shapeOptions': {'color': colors['blue']}}

# Create the draw controls
# @see https://github.com/ellisonbg/ipyleaflet/blob/master/ipyleaflet/leaflet.py#L293
dc = DrawControl(
    polygon = polygon,
    rectangle = rectangle
)
# Initialize an action counter variable
actionCount = 0
AOIs = {}

# Register the draw controls handler
def handle_draw(self, action, geo_json):
    # Increment the action counter
    global actionCount
    actionCount += 1
    # Remove the `style` property from the GeoJSON
    geo_json['properties'] = {}
    # Convert geo_json output to a string and prettify (indent & replace ' with ")
    geojsonStr = json.dumps(geo_json, indent=2).replace("'", '"')
    AOIs[actionCount] = json.loads(geojsonStr)

# Attach the draw handler to the draw controls `on_draw` event
dc.on_draw(handle_draw)
m.add_control(dc)
m


I am working on the file "MovieTime.ipynb".
The first task is:
Let's take a look at what we got
Can you write Python code for it?

merged = load_image3(output_mosaic)
plt.figure(0,figsize=(18,18))
plt.imshow(merged)
plt.title("merged")


I am working on the file "MovieTime.ipynb".
The first task is:
Now let's re-load the image and run the scene maker.
Can you write Python code for it?

os.system("rm ./movie/*.png")
merged = load_image3(output_mosaic)
build_scenes(merged,interp)


I am working on the file "MovieTime.ipynb".
The first task is:
Now things get interesting....

Our path is just a few waypoint but to make a video we need just about every point between our waypoints.
To get all of the points between our waypoints we'll have to write a little interpolation script.
Interpolation is just a fancy word for nicely space points bewteen or waypoints, we'll call the space between each point as our "velocity."
If we were really slick we could define a heading vector and and build a spline so the camera faces the direction of heading. Our approach is fine as the top of the frame is always North, which makes reckoning easy.
Once we have our interpolation function all we need to do is to crop our large mosaic at each point in our interpolation point list and save it in a sequential file.
Can you write Python code for it?

def interpolate_waypoints(waypoints,velocity=10.0):
    retVal = []
    last_pt = waypoints[0]
    # for each point in our waypoints except the first
    for next_pt in waypoints[1:]:
        # calculate distance between the points
        distance = np.sqrt((last_pt[0]-next_pt[0])**2+(last_pt[1]-next_pt[1])**2)
        # use our velocity to calculate the number steps.
        steps = np.ceil(distance/velocity)
        # linearly space points between the two points on our line
        xs = np.array(np.linspace(last_pt[0],next_pt[0],steps),dtype='int64')
        ys = np.array(np.linspace(last_pt[1],next_pt[1],steps),dtype='int64')
        # zip the points together
        retVal += zip(xs,ys)
        # move to the next point
        last_pt = next_pt
    return retVal

def build_scenes(src,waypoints,window=[640,480],path="./movie/"):
    count = 0
    # Use opencv to change the color space of our image.
    src = cv2.cvtColor(src, cv2.COLOR_BGR2RGB)
    # define half our sampling window.
    w2 = window[0]/2
    h2 = window[1]/2
    # for our source image get the width and height
    h,w,d = src.shape
    for pt in waypoints:
        # for each point crop the area out.
        # the y value of our scene is upside down.
        temp = crop_to_area(src,pt[1],h-pt[0],window[0],window[1])
        # If we happen to hit the border of the scene, just skip
        if temp.shape[0]*temp.shape[1]== 0:
            # if we have an issue, just keep plugging along
            continue
        # Resample the image a bit, this just makes things look nice.
        temp = cv2.resize(temp, (int(window[0]*0.75), int(window[1]*.75)))
        # create a file name
        fname = os.path.abspath(path+"img{num:06d}.png".format(num=count))
        # Save it
        cv2.imwrite(fname,temp)
        count += 1


I am working on the file "MovieTime.ipynb".
The first task is:
Now to figure out how our lat/long values map to pixels.

The next thing we need is a way to map from a lat and long in our slippy map to the pixel position in our image.
We'll use what we know about the lat/long of the corners of our image to do that.
We'll ask GDAL to tell us the extents of our scene and the geotransofrm.
We'll then apply the GeoTransform from GDAL to the coordinates that are the extents of our scene.
Now we have the corners of our scene in Lat/Long
Can you write Python code for it?

# Liberally borrowed from this example
# https://gis.stackexchange.com/questions/57834/how-to-get-raster-corner-coordinates-using-python-gdal-bindings
def GetExtent(gt,cols,rows):
    """
    Get the list of corners in our output image in the format
    [[x,y],[x,y],[x,y]]
    """
    ext=[]
    # for the corners of the image
    xarr=[0,cols]
    yarr=[0,rows]

    for px in xarr:
        for py in yarr:
            # apply the geo coordiante transform
            # using the affine transform we got from GDAL
            x=gt[0]+(px*gt[1])+(py*gt[2])
            y=gt[3]+(px*gt[4])+(py*gt[5])
            ext.append([x,y])
        yarr.reverse()
    return ext

def ReprojectCoords(coords,src_srs,tgt_srs):
    trans_coords=[]
    # create a transform object from the source and target ref system
    transform = osr.CoordinateTransformation( src_srs, tgt_srs)
    for x,y in coords:
        # transform the points
        x,y,z = transform.TransformPoint(x,y)
        # add it to the list.
        trans_coords.append([x,y])
    return trans_coords


I am working on the file "MovieTime.ipynb".
The first task is:
Now we are going to write a quick crop function.

this function takes in a, scene, a center position, and the width and height of a window.
We'll use numpy slice notation to make the crop.
Let's pick a spot and see what we get.
Can you write Python code for it?

def crop_to_area(scene,x_c,y_c,w,h):
    tlx = x_c-(w/2)
    tly = y_c-(h/2)
    brx = x_c+(w/2)
    bry = y_c+(h/2)
    return scene[tly:bry,tlx:brx,:]

plt.figure(0,figsize=(3,4))
plt.imshow(crop_to_area(merged,3000,3000,640,480))
plt.title("merged")


I am working on the file "MovieTime.ipynb".
The first task is:
Now, just like before, we will mosaic those scenes.

It is easier to call out using subprocess and use the command line util.
Just iterate through the files and drop them into a single file sf_mosaic.tif
Can you write Python code for it?

subprocess.call(["rm","sf_mosaic.tif"])
commands = ["gdalwarp", # t
           "-t_srs","EPSG:3857",
           "-cutline",aoi_file,
           "-crop_to_cutline",
           "-tap",
            "-tr", "3", "3"
           "-overwrite"]
output_mosaic = "_mosaic.tif"
for tiff in tiff_files:
    commands.append(tiff)
commands.append(output_mosaic)
print " ".join(commands)
subprocess.call(commands)


I am working on the file "MovieTime.ipynb".
The first task is:
Perform the actual activation ... go get coffee
Can you write Python code for it?

to_get = good_scenes["id"][0:10].tolist()
to_get = sorted(to_get)
activated = []
# for each scene to get
for scene in to_get:
    # get the product
    product_types = get_products(client,scene)
    for p in product_types:
        # if there is a visual productfor p in labels:
        if p == "visual": # p == "basic_analytic_dn"
            print "Activating {0} for scene {1}".format(p,scene)
            # activate the product
            _,product = activate_product(client,scene,product=p)
            activated.append(product)


I am working on the file "MovieTime.ipynb".
The first task is:
Query the API

Now we'll save the geometry for our AOI and the path.
We'll also filter and cleanup our data just like before.
Can you write Python code for it?

print AOIs
areaAOI = AOIs[1]["geometry"]
pathAOI = AOIs[2]["geometry"]

aoi_file ="san_francisco.geojson"
with open(aoi_file,"w") as f:
    f.write(json.dumps(areaAOI))
# build a query using the AOI and
# a cloud_cover filter that excludes 'cloud free' scenes

old = datetime.datetime(year=2017,month=1,day=1)
new = datetime.datetime(year=2017,month=8,day=10)

query = filters.and_filter(
    filters.geom_filter(areaAOI),
    filters.range_filter('cloud_cover', lt=0.05),
    filters.date_range('acquired', gt=old),
    filters.date_range('acquired', lt=new)

)
# build a request for only PlanetScope imagery
request = filters.build_search_request(
    query, item_types=['PSScene3Band']
)

# if you don't have an API key configured, this will raise an exception
result = client.quick_search(request)
scenes = []
planet_map = {}
for item in result.items_iter(limit=500):
    planet_map[item['id']]=item
    props = item['properties']
    props["id"] = item['id']
    props["geometry"] = item["geometry"]
    props["thumbnail"] = item["_links"]["thumbnail"]
    scenes.append(props)
scenes = pd.DataFrame(data=scenes)
display(scenes)
print len(scenes)


I am working on the file "MovieTime.ipynb".
The first task is:
This is from the previous notebook. We are just activating and downloading scenes.
Can you write Python code for it?

def get_products(client, scene_id, asset_type='PSScene3Band'):
    """
    Ask the client to return the available products for a
    given scene and asset type. Returns a list of product
    strings
    """
    out = client.get_assets_by_id(asset_type,scene_id)
    temp = out.get()
    return temp.keys()

def activate_product(client, scene_id, asset_type="PSScene3Band",product="analytic"):
    """
    Activate a product given a scene, an asset type, and a product.

    On success return the return value of the API call and an activation object
    """
    temp = client.get_assets_by_id(asset_type,scene_id)
    products = temp.get()
    if( product in products.keys() ):
        return client.activate(products[product]),products[product]
    else:
        return None

def download_and_save(client,product):
    """
    Given a client and a product activation object download the asset.
    This will save the tiff file in the local directory and return its
    file name.
    """
    out = client.download(product)
    fp = out.get_body()
    fp.write()
    return fp.name

def scenes_are_active(scene_list):
    """
    Check if all of the resources in a given list of
    scene activation objects is read for downloading.
    """
    return True
    retVal = True
    for scene in scene_list:
        if scene["status"] != "active":
            print "{} is not ready.".format(scene)
            return False
    return True
def load_image4(filename):
    """Return a 4D (r, g, b, nir) numpy array with the data in the specified TIFF filename."""
    path = os.path.abspath(os.path.join('./', filename))
    if os.path.exists(path):
        with rasterio.open(path) as src:
            b, g, r, nir = src.read()
            return np.dstack([r, g, b, nir])

def load_image3(filename):
    """Return a 3D (r, g, b) numpy array with the data in the specified TIFF filename."""
    path = os.path.abspath(os.path.join('./', filename))
    if os.path.exists(path):
        with rasterio.open(path) as src:
       ...


I am working on the file "MovieTime.ipynb".
The first task is:
To make sure we are good we'll visually inspect the scenes in our slippy map.
Can you write Python code for it?

# first create a list of colors
colors = ["#ff0000","#00ff00","#0000ff","#ffff00","#ff00ff","#00ffff","#ff0000","#00ff00","#0000ff","#ffff00","#ff00ff","#00ffff"]
# grab our scenes from the geometry/footprint geojson
# Chane this number as needed
footprints = good_scenes[0:10]["geometry"].tolist()
# for each footprint/color combo
for footprint,color in zip(footprints,colors):
    # create the leaflet object
    feat = {'geometry':footprint,"properties":{
            'style':{'color': color,'fillColor': color,'fillOpacity': 0.2,'weight': 1}},
            'type':u"Feature"}
    # convert to geojson
    gjson = GeoJSON(data=feat)
    # add it our map
    m.add_layer(gjson)
# now we will draw our original AOI on top
feat = {'geometry':areaAOI,"properties":{
            'style':{'color': "#FFFFFF",'fillColor': "#FFFFFF",'fillOpacity': 0.5,'weight': 1}},
            'type':u"Feature"}
gjson = GeoJSON(data=feat)
m.add_layer(gjson)
m


I am working on the file "Movie_Rating/.ipynb_checkpoints/Culture_difference_movie_rating-checkpoint.ipynb".
The first task is:
1.  Preliminary data visualization and analysis
Can you write Python code for it?

douban_dat = pd.read_csv("douban_movie_review.csv")
douban_dat.rename(columns = {'movie_title':'d_movie_title','year':'d_year','rating':'douban_score','num_review':'dnum_review','flag':'dflag'},inplace = True)
douban_dat.info()

res_dat = pd.concat([imdb_dat,douban_dat],axis = 1)
res_dat.info()


I am working on the file "Movie_Rating/.ipynb_checkpoints/Culture_difference_movie_rating-checkpoint.ipynb".
The first task is:
1.1 Visulze the gross distribution of rating from IMDB (x-axis) and   Douban (y-axis)
Can you write Python code for it?

# 1. visulize the gross distribution of ratings from imdb(x-axis) and douban(y-axis)
import seaborn as sns
g = sns.jointplot(x = 'imdb_score',y = 'douban_score',data = res_dat)
g.ax_joint.set(xlim=(1, 10), ylim=(1, 10))


I am working on the file "Movie_Rating/Culture_difference_movie_rating.ipynb".
The first task is:
1.  Preliminary data visualization and analysis
Can you write Python code for it?


I am working on the file "Movie_Rating/Culture_difference_movie_rating.ipynb".
The first task is:
1.1 Visulze the gross distribution of rating from IMDB (x-axis) and   Douban (y-axis)
Can you write Python code for it?


I am working on the file "Movie_Rating/Culture_difference_movie_rating.ipynb".
The first task is:
1.2 Is it necessary to recenter(scale)  IMDB score  and   Douban score?
Can you write Python code for it?

# plot distribution and bar graphs(significantly different)
from scipy import stats
nbins = 15
fig,axes = plt.subplots(nrows = 1,ncols = 2, figsize = (10,8))
ax0,ax1 = axes.flatten()
ax0.hist([res_dat.douban_score,res_dat.imdb_score],nbins, histtype = 'bar',label = ["Douban","IMDB"])
ax0.set_title('The distribution of movie ratings')
ax0.set_xlabel('Rating')
ax0.set_ylabel('Count')
ax0.legend()
imdb_score = np.mean(res_dat.imdb_score)
douban_score = np.mean(res_dat.douban_score)
ax1.bar([0,1],[imdb_score, douban_score], yerr = [np.std(res_dat.imdb_score),np.std(res_dat.douban_score)],
        align = 'center',color = ['green','blue'], ecolor = 'black')
ax1.set_xticks([0,1])
ax1.set_xticklabels(['IMDB','Douban'])
ax1.set_ylabel('Score')
_,p = stats.ttest_rel(res_dat['imdb_score'], res_dat['douban_score'],nan_policy = 'omit')
ax1.set_title('A comparison of ratings\n'+'t-test: p = %.4f***'%p)
#fig.tight_layout()
plt.show()
# any significant differences


I am working on the file "Movie_Rating/Culture_difference_movie_rating.ipynb".
The first task is:
1.3 Normalize IMDB and Douban rating scores
Can you write Python code for it?

from sklearn import preprocessing
data = res_dat.dropna()
print " delete null values, the remaining data is",data.shape

data.loc[:,'scaled_imdb'] = preprocessing.scale(data['imdb_score'])
data.loc[:,'scaled_douban'] = preprocessing.scale(data['douban_score'])

#stats.ttest_rel(data['scaled_imdb'], data['scaled_douban'],nan_policy = 'omit')
from scipy.stats import norm, lognorm
import matplotlib.mlab as mlab
fig,axes = plt.subplots(nrows = 1,ncols = 2, figsize = (10,8))
ax0,ax1 = axes.flatten()
ax0.plot(data['scaled_imdb'],data['scaled_douban'],'ro')
ax0.set_title('Normalized Scores')
ax0.set_xlabel('Scaled IMDB score')
ax0.set_ylabel('Scaled Douban score')
data.loc[:,'rating_diff'] = data['scaled_imdb'] - data['scaled_douban']
(mu,sigma) = norm.fit(data['rating_diff'])
_,bins,_ = ax1.hist(data['rating_diff'],60,normed = 1, histtype = 'bar',alpha = 0.75)
ax1.plot(bins, mlab.normpdf(bins,mu,sigma),'r--',linewidth = 2)
ax1.set_xlabel('IMDB_score - Douban_score')
ax1.set_ylabel('percentage')
ax1.set_title('Rating difference Distribution')
fig.tight_layout()
plt.show()


I am working on the file "Movie_Rating/Culture_difference_movie_rating.ipynb".
The first task is:
Data from  the "IMDB5000" database
Can you write Python code for it?

imdb_dat = pd.read_csv("movie_metadata.csv")
imdb_dat.info()


I am working on the file "Movie_Rating/Culture_difference_movie_rating.ipynb".
The first task is:
Scrape data from <font color = "red"> DOUBAN.COM </font>
Can you write Python code for it?

import requests
import re
from bs4 import BeautifulSoup
import time
import string
# return the douban movie rating that matches the movie name and year
# read in the movie name
def doubanRating(name):
    movie_name = name.decode('gbk').encode('utf-8')
    url_head = 'http://movie.douban.com/subject_search'
    pageload = {'search_text': movie_name}
    r = requests.get(url_head,params = pageload)
    soup = BeautifulSoup(r.text,'html.parser')
    first_hit = soup.find_all(class_= 'nbg')
    try:
        r2_link = first_hit[0].get('href')
        # sometime douban returns items like celebrity instead of movies
        if 'subject' not in r2_link:
            r2_link = first_hit[1].get('href')
        r2 = requests.get(r2_link)
        soup2 = BeautifulSoup(r2.text,'html.parser')
        title = soup2.find(property = "v:itemreviewed")
        title = title.get_text() # in unicode
        # remove Chinese characters
        title = ' '.join((title.split(' '))[1:])
        title = filter(lambda x:x in set(string.printable),title)
        flag = True
        if title != name:
            print "Warning: name may not match"
            flag = False
        year = (soup2.find(class_='year')).get_text()# in unicode
        rating = (soup2.find(class_="ll rating_num")).get_text() # in unicode
        num_review = (soup2.find(property="v:votes")).get_text()
        return [title, year, rating,num_review,flag]
    except:
        print "Record not found for: "+name
        return [name, None, None, None, None]

#%%2. Store scrapped data
dataset = pd.read_csv("movie_metadata.csv")
total_length = 5043
#first_query = 2500
res = pd.DataFrame(columns = ('movie_title','year','rating','num_review','flag'))
for i in xrange(1,total_length):
    name = dataset['movie_title'][i].strip().strip('\xc2\xa0')
    res.loc[i] = doubanRating(name)
    print "slowly and finally done %d query"%i
    time.sleep(10)
    if (i%50==0):
        res.to_csv("douban_movie_review.csv")
        print "saved until record: %d"%i


I am working on the file "Movie_Recommendations.ipynb".
The first task is:
Another way to explore the data is by using TSNE to group the movies on a two dimensional grid.
Can you write Python code for it?

model = TSNE(n_components=2, random_state=0)
np.set_printoptions(suppress=True)
xy = model.fit_transform(normalized)
xy


I am working on the file "Movie_Recommendations.ipynb".
The first task is:
Extract the weights for the movie embeddings:
Can you write Python code for it?

src = model.get_layer('src_embedding')
src_weights = src.get_weights()[0]
lens = np.linalg.norm(src_weights, axis=1)
normalized = (src_weights.T / lens).T
np.linalg.norm(normalized[0]), normalized.shape


I am working on the file "Movie_Recommendations.ipynb".
The first task is:
It converges quite quickly too. The achieved loss isn't super impressive - 0.4 or so:
Can you write Python code for it?

model.fit_generator(
    batchify(lambda: data_generator(refs_movies, yield_movie_pairs=True), 2048),
    epochs=25,
    steps_per_epoch=3500,
    verbose=2
)


I am working on the file "Movie_Recommendations.ipynb".
The first task is:
Now that we have all the movies, extract all the outgoing links (refs) and keep the ones that are used more than once.
Can you write Python code for it?

ref_counts = Counter()
for _, _, refs, _ in movies:
    ref_counts.update(refs)
all_refs = [ref for ref, count in ref_counts.items() if count > 1]
ref_to_id = {ref: idx for idx, ref in enumerate(all_refs)}
movie_to_id = {title: idx for idx, (title, _, _, _) in enumerate(movies)}
len(all_refs), len(ref_counts)


I am working on the file "Movie_Recommendations.ipynb".
The first task is:
Scatter Plot the first 300 movies to get an idea of whether what the shape is we are looking at:
Can you write Python code for it?


I am working on the file "Movie_Recommendations.ipynb".
The first task is:
The code here relies on wikipedia having been imported into Postgres using Wiki Import. So we setup a connection to postgres, fetch the 10000 most popular movies and then extract for each the movie name, the image for the movie and the outgoing links from that page
Can you write Python code for it?

postgres_conn = psycopg2.connect('dbname=douwe user=notebook')
postgres_cursor = postgres_conn.cursor(cursor_factory=psycopg2.extras.DictCursor)

print 'Getting top movies...'
postgres_cursor.execute(
    "SELECT wikipedia.*, wikistats.viewcount FROM wikipedia "
    "JOIN wikistats ON wikipedia.title = wikistats.title WHERE wikipedia.infobox = 'film' "
    "ORDER BY wikistats.viewcount DESC limit 10000"
)
print 'done'

movies = []
for movie in postgres_cursor:
    wikicode = mwparserfromhell.parse(movie['wikitext'])
    image = None
    for template in wikicode.filter_templates():
        if template.name.lower().startswith('infobox '):
            for param in template.params:
                key = param.name.strip()
                if key == 'image':
                    image = param.value.strip()
            break
    if image:
        image_name = fetch_image(image)
    movies.append((movie['title'], image_name, [unicode(x.title) for x in wikicode.filter_wikilinks()], movie['viewcount']))

len(movies)


I am working on the file "Movie_Recommendations.ipynb".
The first task is:
The fetch_image function below fetches an image from wikipedia given a wikipedia id for that image:
Can you write Python code for it?

IMAGE_PATH_EN = 'http://upload.wikimedia.org/wikipedia/en/%s/%s/%s'
IMAGE_PATH_COMMONS = 'http://upload.wikimedia.org/wikipedia/commons/%s/%s/%s'
image_cache = 'movie_images'

def fetch_image(image_name):
    if not image_name or image_name.endswith('.tiff'):
        return None
    image_name = image_name.replace(' ', '_')
    if image_name[0].upper() != image_name[0]:
        image_name = image_name.capitalize()
    file_path = os.path.join(image_cache, image_name)
    if os.path.isfile(file_path):
        return image_name
    else:
        m = md5()
        m.update(image_name.encode('utf-8'))
        c = m.hexdigest()
        path = IMAGE_PATH_EN % (c[0], c[0:2], image_name)
        r = requests.get(path)
        if r.status_code == 404:
            path = IMAGE_PATH_COMMONS % (c[0], c[0:2], image_name)
            r = requests.get(path)
            if r.status_code == 404:
                print image_name
                return None
        try:
            image = Image.open(BytesIO(r.content))
        except IOError:
            return None
        except ValueError:
            return None
        image.save(file(file_path, 'w'))
        image.thumbnail((240, 640), Image.ANTIALIAS)
        res = BytesIO()
        if image.mode == 'P':
            image = image.convert('RGB')
        try:
            image.save(res, 'WEBP', quality=15)
        except IOError as err:
            print image_name, err.message
            return None
        return image_name

fetch_image('Suicide Squad (film) Poster.png')


I am working on the file "Moving the Robot.ipynb".
The first task is:
Initialization
Throughout the course, some code is already written for you, and organized in modules called packages. The cell below is an initialization step that must be called at the beginning of the notebook.
Can you write Python code for it?

import packages.initialization
import pioneer3dx as p3dx
p3dx.init()


I am working on the file "Moving the Robot.ipynb".
The first task is:
Motion
Let's move the robot on the simulator!
You are going to use a widget, a Graphical User Interface (GUI) with two sliders for moving the robot in two ways: translation and rotation.
Can you write Python code for it?


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Average leakage
Mean per sample:
Can you write Python code for it?

lk_s = pd.DataFrame(index=['mean', 'std'], columns=leakage_kde.index, dtype=float)

lk_s.loc['mean'] = leakage_kde.mean(1)*100
lk_s.loc['std'] = leakage_kde.std(1)*100
lk_s = lk_s.round(5)
lk_s


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Data files
Can you write Python code for it?

bsearch_str = 'DexDem'
leakage_kde = pd.read_csv(
    'results/Multi-spot - leakage coefficient all values KDE %s.csv' % bsearch_str, index_col=0)
leakage_gauss = pd.read_csv(
    'results/Multi-spot - leakage coefficient all values gauss %s.csv' % bsearch_str, index_col=0)
nbursts = pd.read_csv(
    'results/Multi-spot - leakage coefficient all values nbursts %s.csv' % bsearch_str, index_col=0)
for df in (leakage_kde, leakage_gauss, nbursts):
    df.columns.name = 'Channel'

for dx in (leakage_gauss, leakage_kde, nbursts):
    dx.columns = pd.Index(np.arange(1, 9), name='Spot')

PLOT_DIR = './figure/'


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Figure
Can you write Python code for it?

%config InlineBackend.figure_format='retina'  # for hi-dpi displays


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Mean per channel (weighted on the number of bursts):
Can you write Python code for it?

lk_cw = pd.DataFrame(index=['mean', 'std'], columns=leakage_kde.columns, dtype=float)

lk_cw.loc['mean'] = (nbursts*leakage_kde).sum()/nbursts.sum()*100
lk_cw.loc['std'] = np.sqrt((((leakage_kde*100 - lk_cw.loc['mean'])**2) * nbursts).sum(0) / (nbursts.sum(0) - 1))
#lk_cw['mean'] = lk_cw.mean(1)
lk_cw = lk_cw.round(5)
lk_cw

lk_cwg = pd.DataFrame(index=['mean', 'std'], columns=leakage_gauss.columns)

lk_cwg.loc['mean'] = (nbursts*leakage_gauss).sum()/nbursts.sum()*100
lk_cwg.loc['std'] = np.sqrt((((leakage_kde*100 - lk_cwg.loc['mean'])**2) * nbursts).sum(0) / (nbursts.sum(0) - 1))
#lk_cwg['mean'] = lk_cwg.mean(1)
lk_cwg = lk_cwg.round(5)
lk_cwg

lk_cw_m = pd.concat((lk_cw.loc['mean'], lk_cwg.loc['mean']), axis=1, keys=['KDE', 'Gauss'])
lk_cw_m.T

lk_cw_s = pd.concat((lk_cw.loc['std'], lk_cwg.loc['std']), axis=1, keys=['KDE', 'Gauss'])
lk_cw_s.T

sns.set_palette(colors_dark)

kws = dict(lw=5, marker='o', ms=8, alpha=0.5)
lk_cw.loc['mean'].plot(yerr=lk_cw.loc['std'], **kws)
lk_cwg.ix['mean',:].plot(yerr=lk_cwg.loc['std'],**kws)
plt.ylim(0, 4)
plt.xlim(0.75, 8.25)
sns.despine(trim=True)

lk_cw_m.plot.bar(alpha=0.8)
#sns.despine(trim=True, offset=10)


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Mean per channel:
Can you write Python code for it?

lk_c = pd.DataFrame(index=['mean', 'std'], columns=leakage_kde.columns, dtype=float)

lk_c.loc['mean'] = leakage_kde.mean()*100
lk_c.loc['std'] = leakage_kde.std()*100
#lk_c['mean'] = lk_c.mean(1)
lk_c = lk_c.round(5)
lk_c


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Mean per sample (weighted on the number of bursts):
Number of bursts in D-only population:
Can you write Python code for it?

nbursts

leakage_kde

lk_sw = pd.DataFrame(index=['mean', 'std'], columns=leakage_kde.index, dtype=float)

lk_sw.loc['mean'] = (nbursts*leakage_kde).sum(1)/nbursts.sum(1)*100
lk_sw.loc['std'] = np.sqrt((((leakage_kde.T*100 - lk_sw.loc['mean']).T**2) * nbursts).sum(1) / (nbursts.sum(1) - 1))
#lk_sw['mean'] = (nbursts * lk_sw).sum(1) / nbursts.sum(1).sum()
lk_sw = lk_sw.round(5)
lk_sw

lk_swg = pd.DataFrame(index=['mean', 'std'], columns=leakage_gauss.index, dtype=float)

lk_swg.loc['mean'] = (nbursts*leakage_gauss).sum(1)/nbursts.sum(1)*100
lk_swg.loc['std'] = np.sqrt((((leakage_gauss.T*100 - lk_swg.loc['mean']).T**2) * nbursts).sum(1) / (nbursts.sum(1) - 1))
#lk_sw['mean'] = (nbursts * lk_sw).sum(1) / nbursts.sum(1).sum()
lk_swg = lk_swg.round(5)
lk_swg

lk_sw_m = pd.concat((lk_sw.loc['mean'], lk_swg.loc['mean']), axis=1, keys=['KDE', 'Gauss'])
lk_sw_m

lk_sw_s = pd.concat((lk_sw.loc['std'], lk_swg.loc['std']), axis=1, keys=['KDE', 'Gauss'])
lk_sw_s

sns.set_style('ticks')

lk_sw_m.plot(yerr=lk_sw_s, lw=5, alpha=0.6)
plt.xlim(-0.2, 3.2)
plt.xticks(range(4), lk_sw_s.index)
sns.despine(trim=True, offset=10)

lk_sw_m.plot.bar(yerr=lk_sw_s, alpha=0.8)
sns.despine(trim=True, offset=10)

sns.swarmplot(data=leakage_kde*100, size=8, palette=colors_dark);
plt.ylim(0)
lk_sw_m.loc[:,'KDE'].plot(lw=3, alpha=0.8, color='k')
plt.xlim(-0.2, 3.2)
plt.xticks(range(4), lk_sw_s.index)
sns.despine(trim=True, offset=10)


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
NOTE: There is a per-channel trend that cannot be ascribed to the background
because we performend a D-emission burst search and selection and the leakage vs ch
does not resemble the D-background vs channel curve.
The effect is probably due to slight PDE variations (detectors + optics) that
slightly change $\gamma$ on a per-spot basis.

Weighted mean of the weighted mean
Can you write Python code for it?

leakage_kde_wmean = (leakage_kde*nbursts).sum().sum() / nbursts.sum().sum()
leakage_kde_wmean


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Now I will transform leakage_kde in "tidy form" for
easier plotting.
For info on "data tidying" see:

http://stackoverflow.com/questions/37490771/seaborn-categorical-plot-with-hue-from-dataframe-rows/
https://www.ibm.com/developerworks/community/blogs/jfp/entry/Tidy_Data_In_Python
Can you write Python code for it?

leakage_kde

leakage_kde_t = pd.melt((100*leakage_kde).reset_index(), id_vars=['Sample'],
                        value_name='leakage_kde').apply(pd.to_numeric, errors='ignore')
leakage_kde_t.head()

# leakage_kde_t = pd.melt((100*leakage_kde).T.reset_index(), id_vars=['Spot'],
#                         value_name='leakage_kde').apply(pd.to_numeric, errors='ignore')
# leakage_kde_t.head()

sns.set_palette(colors_dark4)

fig, ax = plt.subplots(1, 2, figsize=(12, 5), sharey=True)
plt.subplots_adjust(wspace=0.1)

sns.swarmplot(x='Sample', y='leakage_kde', data=leakage_kde_t, size=6, ax=ax[0])
lk_sw_m.loc[:,'KDE'].plot(lw=3, alpha=0.8, color='k', ax=ax[0])
ax[0].set_ylim(0)
ax[0].set_xlim(-0.2, 3.2)
plt.xticks(range(4), lk_sw_s.index)
sns.despine(trim=True, offset=10, ax=ax[0])

sns.swarmplot(x='Spot', y='leakage_kde', data=leakage_kde_t, size=6, hue='Sample', ax=ax[1])
_ = lk_cw_m.copy().assign(Spot=range(8)).set_index('Spot')
_.loc[:,'KDE'].plot(lw=3, alpha=0.8, color='k', label='mean')
ax[1].set_ylim(0)
ax[1].set_xlim(-0.25, 7.25)
plt.xticks(np.arange(8));
sns.despine(trim=True, offset=10, ax=ax[1], left=True)
ax[1].yaxis.set_visible(False)
ax[0].set_ylabel('Leakage %')
leg = ax[1].get_legend()
h, l = ax[1].get_legend_handles_labels()
ax[1].legend(h[1:] + h[:1], l[1:] + l[:1], title='Sample', loc='lower right')
fs = 28
ax[0].text(0,0, 'A', fontsize=fs)
ax[1].text(0,0, 'B', fontsize=fs)
savefig('multi-spot leakage KDE 2panels.png')


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Plot style
Can you write Python code for it?

bmap = sns.color_palette("Set1", 9)
colors_dark = np.array(bmap)[(1,0,2,3,4,8), :]
colors_dark4 = np.array(bmap)[(1,0,2,8), :]

bmap = sns.color_palette('Paired', 12)
colors_light = np.array(bmap)[(0,4,2,8,6,10), :]
colors_light4 = np.array(bmap)[(0,4,2,8), :]
colors_light[-1] = colors_light4[-1] = [.8, .8, .8]

colors_paired = np.zeros((colors_dark.shape[0]*2, colors_dark.shape[1]))
colors_paired[::2] = colors_dark
colors_paired[1::2] = colors_light
colors_paired4 = colors_paired[(0, 1, 2, 3, 4, 5, 10, 11), :]
sns.palplot(colors_paired4)

sns.set(style='ticks', font_scale=1.4, palette=colors_paired)

fig, ax = plt.subplots()
ax2 = ax.twinx()

kws = dict(lw=2, marker='o', ms=8)
for i, did in enumerate(('7d', '12d', '17d', 'DO')):
    (100*leakage_kde).loc[did].plot(label='%s KDE' % did, ax=ax, color=colors_light4[i], **kws)
    nbursts.loc[did].plot(ax=ax2, ls='--', lw=2.5, color=colors_dark4[i])

for i, did in enumerate(('7d', '12d', '17d', 'DO')):
    (100*leakage_gauss).loc[did].plot(label='%s Gauss' % did, ax=ax, color=colors_dark4[i], **kws)

handles, lab = ax.get_legend_handles_labels()
h = handles#[1::2] + handles[::2]
l = lab[1::2] + lab[::2]
ax.legend(ncol=2, loc=1, bbox_to_anchor=(1, 0.5), borderaxespad=0.)
ax.set_ylim(0)

ax2.set_ylim(0, 3200)
plt.xlim(0.75, 8.25)
plt.xlabel('Channel')
ax.set_ylabel('Leakage %')
ax2.set_ylabel('# Bursts')
sns.despine(offset=10, trim=True, right=False)
savefig('multi-spot leakage KDE vs Gauss.svg')

# bmap = sns.color_palette("Set1", 9)
# colors = np.array(bmap)[(1,0,2,3,4,8,6,7), :]
# sns.set_palette(colors)
# sns.palplot(colors)

sns.swarmplot(data=leakage_kde);
plt.figure()
sns.swarmplot(data=leakage_kde.T);


I am working on the file "Multi-spot 5-Samples analysis - Leakage coefficient - Summary.ipynb".
The first task is:
Transform table in tidy form:
Can you write Python code for it?

leakage_kde_t = pd.melt(leakage_kde.reset_index(), id_vars=['Sample'],
                        value_name='leakage_kde').apply(pd.to_numeric, errors='ignore')
leakage_kde_t.leakage_kde *= 100
leakage_kde_t.head()

_ = lk_cw_m.copy().assign(Spot=range(1, 9)).set_index('Spot')
_.head()

sns.set_palette(colors_dark4)

sns.swarmplot(x='Spot', y='leakage_kde', data=leakage_kde_t, size=6, hue='Sample');
_ = lk_cw_m.copy().assign(Spot=range(8)).set_index('Spot')
_.loc[:,'KDE'].plot(lw=3, alpha=0.8, color='k')
plt.ylim(0)
plt.xlim(-0.25, 7.25)
sns.despine(trim=True)


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
22bp sample
Can you write Python code for it?

data_id = '22d'
if reload_data:
    d22 = loader.photon_hdf5(files_dict[data_id])
    d22.calc_bg_cache(**bg_kwargs_auto)
if burst_search:
    d22.burst_search(m=10, F=F, dither=dither)
    if delete_ph_times: d22.delete('ph_times_m')

dfs22 = Sel(d22, select_bursts.size, th1=30, gamma=gamma_sel, donor_ref=donor_ref)
dx = dfs22

nbursts.loc[data_id] = dx.num_bursts

fitter = bext.bursts_fitter(dx)
fitter.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter.model = mfit.factory_gaussian()
fitter.fit_histogram()
E_pr_fret.loc[data_id] = fitter.params['center']
display(E_pr_fret.loc[[data_id]])
print_fit_report(E_pr_fret.loc[data_id], gamma=0.42, leakage=leakage, dir_ex_t=dir_ex_t)

dplot(dx, hist_fret, show_model=True,
      show_fit_stats=True, fit_from='center', show_fit_value=True);


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
27bp sample
Can you write Python code for it?

data_id = '27d'
if reload_data:
    d27 = loader.photon_hdf5(files_dict[data_id])
    d27.calc_bg_cache(**bg_kwargs_auto)
if burst_search:
    d27.burst_search(m=10, F=F, dither=dither)#, ph_sel=Ph_sel(Dex='Dem'))
    if delete_ph_times: d27.delete('ph_times_m')

dfs27 = Sel(d27, select_bursts.size, th1=30, gamma=gamma_sel, donor_ref=donor_ref)
dx = dfs27

nbursts.loc[data_id] = dx.num_bursts

fitter = bext.bursts_fitter(dx)
fitter.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter.model = mfit.factory_asym_gaussian()
fitter.fit_histogram()
E_pr_fret.loc[data_id] = fitter.params['center']
display(E_pr_fret.loc[[data_id]])
print_fit_report(E_pr_fret.loc[data_id], gamma=0.42, leakage=leakage, dir_ex_t=dir_ex_t)

dplot(dx, hist_fret, show_model=True,
      show_fit_stats=True, fit_from='center', show_fit_value=True);

fitter = bext.bursts_fitter(dx)
fitter.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter.model = mfit.factory_gaussian()
fitter.fit_histogram()
E_pr_fret.loc[data_id] = fitter.params['center']
display(E_pr_fret.loc[[data_id]])
print_fit_report(E_pr_fret.loc[data_id], gamma=0.42, leakage=leakage, dir_ex_t=dir_ex_t)

dplot(dx, hist_fret, show_model=True,
      show_fit_stats=True, fit_from='center', show_fit_value=True);


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
5-samples processing
Can you write Python code for it?

df = pd.DataFrame(index=['7d', '12d', '17d', '22d', '27d'], columns=range(8), dtype=float)
df.index.name = 'Sample'
df.columns.name = 'Channel'

E_pr_fret = df.copy()
E_pr_fret_sig = df.copy()
nbursts = df.copy()


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
7bp sample
Can you write Python code for it?

data_id = '7d'
if reload_data:
    d7 = loader.photon_hdf5(files_dict[data_id])
    d7.calc_bg(**bg_kwargs_auto)
if burst_search:
    d7.burst_search(m=10, F=F, dither=dither)
    if delete_ph_times: d7.delete('ph_times_m')

gamma_sel, donor_ref

dfs7 = Sel(d7, select_bursts.size, th1=30, gamma=gamma_sel, donor_ref=donor_ref)
dx = dfs7

nbursts.loc[data_id] = dx.num_bursts

fitter = bext.bursts_fitter(dx)
fitter.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
fitter.model = mfit.factory_two_gaussians(add_bridge=True, p2_center=0.8)
fitter.fit_histogram()

E_pr_fret.loc[data_id] = fitter.params['p2_center']
display(E_pr_fret.loc[[data_id]])
print_fit_report(E_pr_fret.loc[data_id], gamma=0.42, leakage=leakage, dir_ex_t=dir_ex_t)

dplot(dx, hist_fret, show_model=True,
      show_fit_stats=True, fit_from='p2_center', show_fit_value=True);

fig, axes = plt.subplots(4, 2, figsize=(14, 12), sharex=True, sharey=True)
fig.subplots_adjust(left=0.08, right=0.96, top=0.93, bottom=0.07,
                    wspace=0.06, hspace=0.15)

for ich, ax in enumerate(axes.ravel()):
    mfit.plot_mfit(fitter, ich=ich, ax=ax)


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
8-spot paper plot style
Can you write Python code for it?

PLOT_DIR = './figure/'

import matplotlib as mpl
from cycler import cycler

bmap = sns.color_palette("Set1", 9)
colors = np.array(bmap)[(1,0,2,3,4,8,6,7), :]
mpl.rcParams['axes.prop_cycle'] = cycler('color', colors)
colors_labels = ['blue', 'red', 'green', 'violet', 'orange', 'gray', 'brown', 'pink', ]
for c, cl in zip(colors, colors_labels):
    locals()[cl] = tuple(c) # assign variables with color names
sns.palplot(colors)


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
Check that the folder exists:
Can you write Python code for it?

data_dir = os.path.abspath(data_dir) + '/'
assert os.path.exists(data_dir), "Path '%s' does not exist." % data_dir


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
Define samples lists
Define list of results and labels:
Can you write Python code for it?

d_samples = [dfs7, dfs12, dfs17, dfs22, dfs27]#, dfso]
d_samples_c = [dfs7c, dfs12c, dfs17c, dfs22c, dfs27c ]
d_labels = ['7d', '12d', '17d', '22d', '27d']#, 'DO']
CH = np.arange(8)
CH_labels = ['CH%d' % i for i in CH]
dist_s_bp = [7, 12, 17, 22, 27]


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
List of data files in data_dir:
Can you write Python code for it?

from glob import glob
file_list = sorted(glob(data_dir + '*.hdf5'))

labels = ['7d', '12d', '17d', '22d', '27d', 'DO']
files_dict = {lab: fname for lab, fname in zip(sorted(labels), file_list)}
files_dict


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
Load the direct excitation coefficient ($d_{dirT}$) from disk (computed in usALEX - Corrections - Direct excitation physical parameter):
Can you write Python code for it?

dir_ex_coeff_fname = 'results/usALEX - direct excitation coefficient dir_ex_t beta.csv'
dir_ex_t = np.loadtxt(dir_ex_coeff_fname, ndmin=1)

print('Direct excitation coefficient (dir_ex_t):', dir_ex_t)


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
Parameters from file
Load the leakage coefficient from disk (computed in Multi-spot 5-Samples analyis - Leakage coefficient fit):
Can you write Python code for it?

leakage_coeff_fname = 'results/Multi-spot - leakage coefficient KDE wmean DexDem.csv'
leakage = np.loadtxt(leakage_coeff_fname, ndmin=1)

print('Leakage coefficient:', leakage)


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
Processing and plot options:
Can you write Python code for it?

# Data load options
reload_data = 1
burst_search = 1
delete_ph_times = 0


I am working on the file "Multi-spot 5-Samples analysis - PR-raw.ipynb".
The first task is:
Utility functions
Can you write Python code for it?

def print_fit_report(E_pr, gamma=1, leakage=0, dir_ex_t=0, math=True):
    """Print fit and standard deviation for both corrected and uncorrected E
    Returns d.E_fit.
    """
    E_corr = fretmath.correct_E_gamma_leak_dir(E_pr, gamma=gamma, leakage=leakage, dir_ex_t=dir_ex_t)

    E_pr_mean = E_pr.mean()*100
    E_pr_delta = (E_pr.max() - E_pr.min())*100

    E_corr_mean = E_corr.mean()*100
    E_corr_delta = (E_corr.max() - E_corr.min())*100
    if math:
        display(Math(r'\text{Pre}\;\gamma\quad\langle{E}_{fit}\rangle = %.1f\%% \qquad'
                     '\Delta E_{fit} = %.2f \%%' % \
                     (E_pr_mean, E_pr_delta)))
        display(Math(r'\text{Post}\;\gamma\quad\langle{E}_{fit}\rangle = %.1f\%% \qquad'
                     '\Delta E_{fit} = %.2f \%%' % \
                     (E_corr_mean, E_corr_delta)))
    else:
        print('Pre-gamma  E (delta, mean):  %.2f  %.2f' % (E_pr_mean, E_pr_delta))
        print('Post-gamma E (delta, mean):  %.2f  %.2f' % (E_corr_mean, E_corr_delta))


I am working on the file "Multi-spot Gamma Fitting.ipynb".
The first task is:
Load Data
Multispot
Load the leakage coefficient from disk (computed in Multi-spot 5-Samples analyis - Leakage coefficient fit):
Can you write Python code for it?

leakage_coeff_fname = 'results/Multi-spot - leakage coefficient KDE wmean DexDem.csv'
leakageM = float(np.loadtxt(leakage_coeff_fname, ndmin=1))

print('Multispot Leakage Coefficient:', leakageM)


I am working on the file "Multi-spot Gamma Fitting.ipynb".
The first task is:
Load the direct excitation coefficient ($d_{dirT}$) from disk (computed in usALEX - Corrections - Direct excitation physical parameter):
Can you write Python code for it?

dir_ex_coeff_fname = 'results/usALEX - direct excitation coefficient dir_ex_t beta.csv'
dir_ex_t = float(np.loadtxt(dir_ex_coeff_fname, ndmin=1))

print('Direct excitation coefficient (dir_ex_t):', dir_ex_t)


I am working on the file "Multi-spot Gamma Fitting.ipynb".
The first task is:
usALEX
Corrected $E$ from μs-ALEX data:
Can you write Python code for it?

data_file = 'results/usALEX-5samples-E-corrected-all-ph.csv'
data_alex = pd.read_csv(data_file).set_index('sample')#[['E_pr_fret_kde']]
data_alex.round(6)

E_alex = data_alex.E_gauss_w
E_alex


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
8-spot paper plot style
Can you write Python code for it?

PLOT_DIR = './figure/'

import matplotlib as mpl
from cycler import cycler

bmap = sns.color_palette("Set1", 9)
colors = np.array(bmap)[(1,0,2,3,4,8,6,7), :]
mpl.rcParams['axes.prop_cycle'] = cycler('color', colors)


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
Check that the folder exists:
Can you write Python code for it?


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
Correction parameters
Multispot
Load the multispot leakage coefficient from disk (computed in Multi-spot 5-Samples analyis - Leakage coefficient fit):
Can you write Python code for it?

_fname = 'results/Multi-spot - leakage coefficient KDE wmean DexDem.csv'
leakageM = np.loadtxt(_fname, ndmin=1)

print('Leakage coefficient:', leakageM)


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
Data files
Data folder:
Can you write Python code for it?


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
List of data files in data_dir:
Can you write Python code for it?

from glob import glob
file_list = sorted(glob(data_dir + '*_?.hdf5'))

labels = ['12d', '7d', '17d', '22d', '27d', 'DO']
files_dict = {lab: fname for lab, fname in zip(sorted(labels), file_list)}
files_dict


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
Load the multispot gamma ($\gamma_M$) coefficient (computed in Multi-spot Gamma Fitting):
Can you write Python code for it?

_fname = 'results/Multi-spot - gamma factor.csv'
gammaM = np.loadtxt(_fname, ndmin=1)

print('Multispot gamma coefficient:', gammaM)


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
Load the usALEX direct-excitation coefficient ($d_{exAA}$) (computed in usALEX - Corrections - Direct excitation fit):
Can you write Python code for it?

_fname = 'results/usALEX - direct excitation coefficient dir_ex_aa.csv'
dir_ex_aa = np.loadtxt(_fname)

print('Direct excitation coefficient (dir_ex_aa):', dir_ex_aa)


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
Multi-spot vs usALEX FRET histogram comparison
Load FRETBursts software
Can you write Python code for it?

from fretbursts import *
sns = init_notebook()

import os
import pandas as pd
from IPython.display import display, Math

import lmfit
print('lmfit version:', lmfit.__version__)

figure_size = (5, 4)
default_figure = lambda: plt.subplots(figsize=figure_size)
save_figures = True

def savefig(filename, **kwargs):
    if not save_figures:
        return
    import os
    dir_ = 'figures/'
    kwargs_ = dict(dpi=300, bbox_inches='tight')
                   #frameon=True, facecolor='white', transparent=False)
    kwargs_.update(kwargs)
    plt.savefig(dir_ + filename, **kwargs_)
    print('Saved: %s' % (dir_ + filename))


I am working on the file "Multi-spot vs usALEX FRET histogram comparison.ipynb".
The first task is:
Multispot analysis
Can you write Python code for it?

d = loader.photon_hdf5(files_dict[data_id])
d.calc_bg(**bg_kwargs_auto)
d.burst_search(m=10, F=F, dither=dither)

d.time_max

ds = Sel(d, select_bursts.size, th1=30, gamma=gammaM, donor_ref=donor_ref)

ds.num_bursts

# fitter = bext.bursts_fitter(ds)
# fitter.histogram(bins=np.r_[-0.2 : 1.2 : bandwidth])
# fitter.model = mfit.factory_two_gaussians(add_bridge=False, p2_center=0.4)
# fitter.fit_histogram()
# display(fitter.params['p2_center'])
# print_fit_report(fitter.params['p2_center'], gamma=gammaM, leakage=leakageM, dir_ex_t=dir_ex_tM)

dplot(ds, hist_fret);
      #show_model=True, show_fit_stats=True, fit_from='p2_center', show_fit_value=True);

d_all = ds.collapse()

d_all_chunk = Sel(d_all, select_bursts.time, time_s2=600/8)

dplot(d_all_chunk, hist_fret)

Eraw = d_all_chunk.E[0]

E = fretmath.correct_E_gamma_leak_dir(Eraw, gamma=gammaM, leakage=leakageM, dir_ex_t=dir_ex_tM)

sns.set_style('whitegrid')
%config InlineBackend.figure_format='retina'  # for hi-dpi displays

plt.hist(E, bins=np.arange(-0.2, 1.2, 0.025) + 0.5*0.025);


I am working on the file "MultiCollinearity.ipynb".
The first task is:
これから、Xの係数 $\hat{\beta}$ のP値が小さく有意水準5%でも有意ですが、Zの係数 $\hat{\gamma}$ のP値が大きく明らかに有意でないことがわかります。
Can you write Python code for it?

# 説明変数設定
X = data['X']
X = sm.add_constant(X)
X

# 説明変数をXのみにして推定
# OLSの実行(Ordinary Least Squares: 最小二乗法)
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())

# 説明変数設定
X = data['Z']
X = sm.add_constant(X)
X

# 説明変数をZのみにして推定
# OLSの実行(Ordinary Least Squares: 最小二乗法)
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())

# 標準偏差
data.std()

# 平均
data.mean()

# 相関係数
data.corr()


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
2-D
Can you write Python code for it?

from mpl_toolkits import mplot3d
ax = plt.axes(projection='3d')
ax.scatter(x[0,:], x[1,:], y[0,:], c=y[0,:], cmap='viridis', linewidth=0.1);


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
As all the weight matrices and bias vectors are defined as Theano's shared variables, they can be updated in functions. We define then a gradient_step function to do so.
The gradient_step actually uses minibatch stochastic gradient descent to provide a better performance.
Can you write Python code for it?

updates = [
    (W1, W1 - alpha * dJdW1),  # Update step. W1 = W1 - alpha * dJdW1
    (b1, b1 - alpha * dJdb1),  # Update step. b1 = b1 - alpha * dJdb1
    (W2, W2 - alpha * dJdW2),  # Update step. W2 = W2 - alpha * dJdW2
    (b2, b2 - alpha * dJdb2),  # Update step. b2 = b2 - alpha * dJdb2
    (W3, W3 - alpha * dJdW3),  # Update step. W3 = W3 - alpha * dJdW3
    (b3, b3 - alpha * dJdb3),  # Update step. b3 = b3 - alpha * dJdb3
]

gradient_step = theano.function(
    inputs=[index],
    outputs=loss,
    updates=updates,
    givens={
        X: ng_X_train[index * batch_size: (index + 1) * batch_size],
        y: ng_y_train[index * batch_size: (index + 1) * batch_size]
    }
)


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
Fetching the 20 Newsgroups dataset and processing it
We continue by fetching and processing the 20 Newsgroup dataset, all the subsets, filter the stopwords and convert each document in a Bag-of-Words with CountVectorizer. For simplicity, we consideration words formed only by lower case letters (with no numbers nor punctuation symbols), and we only take the 10000 most frequent words.
Can you write Python code for it?

def process_newsgroups_document(document):
    # To simplify, we ignore everything thas isn't a word
    document = re.sub(r"[^a-zA-Z]", " ", document)

    # We only make use of lower case words
    words = document.lower().split()

    # We filter out every stopword for the english language
    stopwords = set(corpus.stopwords.words("english"))
    document = " ".join([word for word in words if word not in stopwords])

    return document

newsgroups = fetch_20newsgroups(subset='all')
vectorizer = text.CountVectorizer(analyzer='word', preprocessor=process_newsgroups_document, max_features=5000)
newsgroups_dataset = vectorizer.fit_transform(newsgroups.data).todense().astype(theano.config.floatX)
newsgroups_target = newsgroups.target
ng_X_train, ng_X_test, ng_y_train, ng_y_test = train_test_split(newsgroups_dataset, newsgroups_target, test_size=0.2)

# Convert the data to theano shared variables
ng_X_train = theano.shared(ng_X_train, borrow=True)
ng_y_train = theano.shared(ng_y_train, borrow=True)


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
Neural Network Parameters
We define all the parameters for the neural network.
Can you write Python code for it?

N = newsgroups_dataset.shape[0]  # Number of examples in the dataset.
n_input = newsgroups_dataset.shape[1]  # Number of features of the dataset. Input of the Neural Network.
n_output = len(newsgroups.target_names)  # Number of classes in the dataset. Output of the Neural Network.
n_h1 = 2500  # Size of the first layer
n_h2 = 1000  # Size of the second layer
alpha = 0.01  # Learning rate parameter
lambda_reg = 0.01  # Lambda value for regularization
epochs = 500  # Number of epochs for gradient descent
batch_size = 128  # Size of the minibatches to perform sgd
train_batches = ng_X_train.get_value().shape[0] / batch_size


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
Now that we have our computation graph ready, we only need to define our loss function, for which we use the negative log likelihood, also known as the cross entropy. We also add a regularization term to avoid overfitting of the network.
Can you write Python code for it?

# Regularization term to sum in the loss function
loss_reg = 1./N * lambda_reg/2 * (T.sum(T.sqr(W1)) + T.sum(T.sqr(W2)) + T.sum(T.sqr(W3)))

# Loss function
loss = T.nnet.categorical_crossentropy(y_out, y).mean() + loss_reg


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
Now we have the forward propagation and the loss function all set, we need to train the network in order to better classify the documents. In order to do this, we first need to get the gradients of the matrices and bias vectors. Theano do this for us (instead of having to make use of backpropagation to calculate the gradients).
Can you write Python code for it?

dJdW1 = T.grad(loss, wrt=W1)
dJdb1 = T.grad(loss, wrt=b1)
dJdW2 = T.grad(loss, wrt=W2)
dJdb2 = T.grad(loss, wrt=b2)
dJdW3 = T.grad(loss, wrt=W3)
dJdb3 = T.grad(loss, wrt=b3)


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
Once we are done training the model we print some results
Can you write Python code for it?

predictions = predict(ng_X_test)

print "Accuracy: {:.3f}".format(accuracy_score(ng_y_test, predictions))

print "Classification report"
print classification_report(ng_y_test, predictions, target_names=newsgroups.target_names)


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
Theano's Computation Graph
We have to define the computation graph from Theano.
First we define the symbolic variables for the layers.
Can you write Python code for it?

# Stateless variables to handle the input
index = T.lscalar('index')  # Index of a minibatch
X = T.matrix('X')
y = T.lvector('y')

# First layer weight matrix and bias
W1 = theano.shared(
    value=np.random.uniform(
        low=-np.sqrt(6. / (n_input + n_h1)),
        high=np.sqrt(6. / (n_input + n_h1)),
        size=(n_input, n_h1)
    ).astype(theano.config.floatX),
    name='W1',
    borrow=True
)
b1 = theano.shared(
    value=np.zeros((n_h1,), dtype=theano.config.floatX),
    name='b1',
    borrow=True
)

# Second layer weight matrix and bias
W2 = theano.shared(
    value=np.random.uniform(
        low=-np.sqrt(6. / (n_h1 + n_h2)),
        high=np.sqrt(6. / (n_h1 + n_h2)),
        size=(n_h1, n_h2)
    ).astype(theano.config.floatX),
    name='W2',
    borrow=True
)
b2 = theano.shared(
    value=np.zeros((n_h2,), dtype=theano.config.floatX),
    name='b2',
    borrow=True
)

# Output layer weight matrix and bias
W3 = theano.shared(
    value=np.random.uniform(
        low=-np.sqrt(6. / (n_h2 + n_output)),
        high=np.sqrt(6. / (n_h2 + n_output)),
        size=(n_h2, n_output)
    ).astype(theano.config.floatX),
    name='W3',
    borrow=True
)
b3 = theano.shared(
    value=np.zeros((n_output,), dtype=theano.config.floatX),
    name='b3',
    borrow=True
)


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
This functions are now callable as they have been compiled by Theano. As the weight matrices have been initialized randomly (i.e. we still have to train them), if we try to predict the values of a couple of instances, most likely we will end up with random values.
Can you write Python code for it?

# The probabilities for each class given the first 2 examples of the newsgroup dataset
print forward_propagation(newsgroups_dataset[:2])

# The prediction for each of this 2 examples
print predict(newsgroups_dataset[:2])

# The loss function value for each of this 2 examples (most likely a high one)
print loss_calculation(newsgroups_dataset[:2], newsgroups.target[:2])


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
Training
Optimization method: A minibatch SGD algorithm with constant step size.
Can you write Python code for it?

# Euclidian cost
Error = torch.nn.MSELoss(size_average=True)

eta = 0.0005
MAX_ITER = 50000
BatchSize = min(50, N)
EE = []
for epoch in range(MAX_ITER):
    idx = np.random.choice(N, size=BatchSize, replace=False)
    #idx = [epoch%N]
    f = mlp_fun(torch.DoubleTensor(x[:,idx]), Weight, Bias, Func)

    # Measure the error
    E = Error(f, Variable(torch.DoubleTensor(y[:,idx])))
    EE.append(E.data.numpy())

    # Compute the derivative of the error with respect to Weights and Biases
    E.backward()

    # Take the step and reset weights
    for i in range(D-1):
        Weight[i].data.add_(-eta*Weight[i].grad.data)
        Bias[i].data.add_(-eta*Bias[i].grad.data)
        Weight[i].grad.zero_()
        Bias[i].grad.zero_()

plt.plot(EE)
plt.show()

NN = 200
xx = np.linspace(-1.2,1.2,NN).reshape(NN,1)

f = mlp_fun(torch.DoubleTensor(xx.T), Weight, Bias, Func)
plt.plot(xx, f.data.numpy().T,'k')
#plt.plot(x, f.data.numpy(),'ko')
plt.plot(x[0,:], y[0,:],'r+',alpha=0.5, markersize=10)

plt.show()

xx1,xx2 = np.meshgrid(np.linspace(-1,1,200), np.linspace(-1,1,200))
xx = np.c_[xx1.flatten(), xx2.flatten()].T

f = mlp_fun(torch.DoubleTensor(xx), Weight, Bias, Func)
yy = f.data.numpy()[0,:].reshape(xx1.shape)

yy_true = fun(xx).reshape(xx1.shape)

plt.figure(figsize=(10,10))
ax = plt.axes(projection='3d')
ax.scatter(x[0,:], x[1,:], y[0,:], c='k', cmap='gray', linewidth=0.1);
#ax.contour3D(xx1, xx2, yy, 50, cmap='binary')
ax.plot_surface(xx1, xx2, yy, cmap='jet')
#plt.imshow(yy)
#ax.view_init(30, 20)
plt.show()

plt.figure(figsize=(10,10))
ax = plt.axes(projection='3d')
ax.plot_surface(xx1, xx2, yy_true, cmap='jet')
plt.show()

from IPython.display import display, Math, Latex, HTML
import notes_utilities as nut
from importlib import reload

%matplotlib inline

#Code from https://gist.github.com/craffel/2d727968c3aaebd10359

import matplotlib.pyplot as plt

def draw_neural_net(ax, left, right, bottom, top, layer_sizes, bias=0):
    '''
    Draw a neural network cartoon using matplotilb.

   ...


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
Visualize Data
1-D
Can you write Python code for it?

plt.figure()
plt.plot(x, y, 'k.')
plt.show()


I am working on the file "MultiLayerPerceptron.ipynb".
The first task is:
We have defined all the steps for our forward propagation, however we still need to define the function in Theano that does the forward propagation. As the variables, all the functions in Theano are symbolic, we need to define them by setting up their inputs, their outputs and, if we want, we can also define extra information like updates.
Can you write Python code for it?

# Define the functions
forward_propagation = theano.function([X], y_out)
loss_calculation = theano.function([X, y], loss)
predict = theano.function([X], y_pred)


I am working on the file "MultiSentenceDocuments.ipynb".
The first task is:
Define markup_sentence
We are putting the functionality we went through in the previous two notebooks (BasicSentenceMarkup and BasicSentenceMarkupPart2) into a function markup_sentence. We add one step to the function: dropInactiveModifiers will delete any modifier node that does not get attached to a target node.
Can you write Python code for it?

def markup_sentence(s, modifiers, targets, prune_inactive=True):
    """
    """
    markup = pyConText.ConTextMarkup()
    markup.setRawText(s)
    markup.cleanText()
    markup.markItems(modifiers, mode="modifier")
    markup.markItems(targets, mode="target")
    markup.pruneMarks()
    markup.dropMarks('Exclusion')
    # apply modifiers to any targets within the modifiers scope
    markup.applyModifiers()
    markup.pruneSelfModifyingRelationships()
    if prune_inactive:
        markup.dropInactiveModifiers()
    return markup

report = reports[0]
print(report)


I am working on the file "MultiSentenceDocuments.ipynb".
The first task is:
Displaying pyConTextNLP Markups
The display subpackage contains some functionality for visualizing the markups. Here I use HTML to color-code identified concepts.
Can you write Python code for it?

clrs = {\
    "bowel_obstruction": "blue",
    "inflammation": "blue",
    "definite_negated_existence": "red",
    "probable_negated_existence": "indianred",
    "ambivalent_existence": "orange",
    "probable_existence": "forestgreen",
    "definite_existence": "green",
    "historical": "goldenrod",
    "indication": "pink",
    "acute": "golden"
}

display(HTML(html.mark_document_with_html(context,colors = clrs, default_color="black")))


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
1.2 Declare the text informations provided in the problem
The propositions
We create a text object
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
1.3 final step to construct the problem
A problem is the association of a structure and a text as defined above
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
2.3 Define the behavioral constraint of the solvers
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
2.5 Manually run the solver
2.5.1 Apply a schema

check what are the schemas which can be applied.
Can you write Python code for it?

solver1.updater.updateAppliableSchemas()
l=solver1.updater.appliableSchemaList
print(l)
print (l[0].objects)


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
2.6 Automatic use of solver to generate a solution space
2.6.1 Simple example : only solve
2.6.1.1 solver.SOLVER instruction
We create a new solver from start
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
2.6.2 Other intructions
solver.SCHEMAS and solver.INTERP are two other instructions.
For example if the list of actions is [solver.SCHEM,solver.INTERP,solver.SOLVE], then the automatic solver will start with applying a schema, then applying a interpretation change, and finally will finish by a solving process as seen above.
This list of instruction will find all the paths which has this kind of pattern.
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
And we add all the propositions, which are
Can you write Python code for it?

t1='Au supermarché, le kilo de poisson a augmenté de 5 euros cette année'
t2='Un kilo de poisson coute maintenant 12 euros.'
t3='Au début de l\'année, le kilo de viande coutait le même prix que le kilo de poisson.'
t4='Le kilo de viande a augmenté de 3 euros de moins que le kilo de poisson'
q1=Representation(Quantity("PoissonGAIN","P1"),t1)
q2=Representation(Quantity("PoissonEF","T1"),t2)
q3=Representation(Quantity("PoissonEIminusViandeEI","dEI"),t3)
q4=Representation(Quantity("PoissonGAINminusViandeGAIN","d"),t4)
text.addTextInformation(TextInformation(q1))
text.addTextInformation(TextInformation(q2))
text.addTextInformation(TextInformation(q3))
text.addTextInformation(TextInformation(q4))

for info in text.textInformations:
    q=info.representations[0].quantity
    print(q.object, q.value)


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
And we create the schemas in order to create all the correspondances between the first two schemas
Can you write Python code for it?

struct.addBridgingSchemas(schema1,schema2)

ls=struct.schemas
for schem in ls:
    print(schem.objects)


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
Apply a representation change works broadly the same way as applying a Schema :
Can you write Python code for it?

move2=Move(repChange)
stepInformations2=solver1.updater.applyMove(move2,solver1.constraints)
print solver1.updater.problemState.representations, " is the new representations state"


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
As we used the constraint c2=BehavioralConstraint(breakTheOldOne=True) we can see that the old representation of this text information is erased.
Can you write Python code for it?

oldrep=solver1.updater.problem.text.textInformations[3].representations[0]
print "before : ",(oldrep.quantity.object, oldrep.quantity.value)
print "after : ", solver1.updater.problemState.quantitiesDic.dic[oldrep.quantity.object]


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
Let's check that this new value is now in the quantitiesDic
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
Let's inspect a path
Can you write Python code for it?

autoSolver2.TreePaths.scanTree()
aPath=autoSolver2.TreePaths.pathList[43]
print(aPath.formula)
print(aPath.objectFormula)
print(aPath.interpretationsSummary)
print(aPath.valueFound)
print(aPath.problemSolved)


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
Let's inspect the different alternative representations pertaining to the fourth text proposition
Can you write Python code for it?

textInfo=text.textInformations[3]
for representation in textInfo.representations:
    print(representation.quantity.object,representation.quantity.value)


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
The Goal
Can you write Python code for it?

text.setGoal(TextGoal(Goal('ViandeEF','Combien coute le kilo de viande maintenant?')))


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
This constraints implies that if there is an alternative representation is selected, then the quantity formed by its prevous state is destroyed. In other terms, only one representation of the the same proposition is active at the same time.
2.4 Create the solvers (final step)
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
This constraints implies that no negative values can be associated with object which has 'EI' or 'EF' as substring
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
This is the simplest example, the instruction solver.SOLVER means that schemas will be applied until the solution is found. At each step, the research tree has as many branches as schemas which can be applied.
2.6.1.2 inspect the solution space
We can now check what gives treeOutput and pathList with the solver process
Can you write Python code for it?

autoSolver.TreePaths.scanTree()
print(autoSolver.TreePaths.treeOutput)

print(str(autoSolver.TreePaths.pathsCount)+" different paths have been discovered" )


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
TreePath allows many things :
- Adding step to a treepaths bind the steps to its children
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
TreePaths has a method to generate a tree according the different paths taken
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
Updater has a central role in the program as it :
- Check if moves are possible
- Apply moves (schemas, reinterpretations) and then update its main attribute problemState which is also an important class
2.2 (side note) about the problem state

The quantitiesDic : current object-values association
representations which indicates which representations are currently selected
Can you write Python code for it?

probState=upD.problemState
print "quantity dic: ", probState.quantitiesDic.dic
print "\n representations: ",probState.representations


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
We can check if the problem is solved by the solved attribute
Can you write Python code for it?

print "is the problem solved: ",step2.solved


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
We then create the problem structure and add the schemas
Can you write Python code for it?

struct=ProblemStructure()
struct.addSchema(schema1)
struct.addSchema(schema2)
struct.schemas


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
We then update the field corresponding to set of the object in the problem
Can you write Python code for it?

print(struct.objectSet)# before
struct.updateObjectSet()
print(struct.objectSet) #after

print(struct.objectSet)


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
[1] Create a problem
1.1 Declare the mathematical structure of the problem
Two schema describing the first relationships between the quantities are created
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
[2] Handle solver object
2.1 Init its first problem representation
Can you write Python code for it?

upD=Updater(probleme1) # create the problem state
upD.startAsUnderstood() # init the problem state, we start with expert representations


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
[3] Handle the simulated datas
SimulatedDatas is a class used to store different treepath and eventually reduce some redondancies between the different paths.
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
[Note] The last line gives a summary of what happenned. In this specific case we stopped the solving process without finding the solution. A more complex path would give such string :
T1-d : interpretation -> PoissonEF-(PoissonGAINminusViandeGAIN+PoissonEIminusViandeEI)=ViandeEF
This line is produced by the TreePath method trackBack which traceback the different paths and gives the last line. TrackBack is called in scanTree whenever a path is ended.
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
apply a schema
Can you write Python code for it?

move1=Move(l[0]) # a move is one level more abastract than schema or representation
stepInformations1=solver1.updater.applyMove(move1,solver1.constraints) # we can then use the function applyMove for schemas or representation


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
simulatedDatas can print its datas in a CSV via simulatedDatas.printCSV() or in the console using simulatedDatas.printLines()
Can you write Python code for it?


I am working on the file "MultiStep Word Problems investigation.ipynb".
The first task is:
stepInformations is an instance of InfoStep (in subjectRepresentations.py) and contains many informations related to the concrete application of the move, and can be used to inspect precisely a solution path.
Among others :
Can you write Python code for it?


I am working on the file "Multiclass/multiclass classification.ipynb".
The first task is:
By normalizing the three boundary vectors, discrepancies between intra-boundary angles have lowered.
Can you write Python code for it?


I am working on the file "Multiclass/multiclass classification.ipynb".
The first task is:
Multiclass SVM
Can you write Python code for it?

def zeroOne(y,a) :
    '''
    Computes the zero-one loss.
    @param y: output class
    @param a: predicted class
    @return 1 if different, 0 if same
    '''
    return int(y != a)

def featureMap(X,y,num_classes) :
    '''
    Computes the class-sensitive features.
    @param X: array-like, shape = [n_samples,n_inFeatures] or [n_inFeatures,], input features for input data
    @param y: a target class (in range 0,..,num_classes-1)
    @return array-like, shape = [n_samples,n_outFeatures], the class sensitive features for class y
    '''
    #The following line handles X being a 1d-array or a 2d-array
    num_samples, num_inFeatures = (1,X.shape[0]) if len(X.shape) == 1 else (X.shape[0],X.shape[1])
    num_outFeatures = num_inFeatures*num_classes
    output = np.zeros((num_samples, num_outFeatures))

    if X.ndim == 1:
        output[0, y*num_inFeatures: y*num_inFeatures+2] = X

    else:
        for i in range(num_samples):
            output[i, y*num_inFeatures: y*num_inFeatures +2  ] = X[i]

    return output

def sgd(X, y, num_outFeatures, subgd, lam = 0.01, eta = 0.1, T = 10000):
    '''
    Runs subgradient descent, and outputs resulting parameter vector.
    @param X: array-like, shape = [n_samples,n_features], input training data
    @param y: array-like, shape = [n_samples,], class labels
    @param num_outFeatures: number of class-sensitive features
    @param subgd: function taking x,y and giving subgradient of objective
    @param weight w
    @param eta: learning rate for SGD
    @param T: maximum number of iterations
    @return: vector of weights
    '''
    num_samples = X.shape[0]
    w_store = np.zeros(num_outFeatures)
    w = np.zeros(num_outFeatures)
    #w = np.array([0,0.1,0,0.1,0,0.1])
    for t in range(T):
        #eta /= (t+1)
        #sample_ind = np.random.choice(num_samples, 1)
        for sample_ind in range(num_samples):
            w -= eta*subgd(X[sample_ind], y[sample_ind], w)
            #if sample_ind == t:
             #   print(subgd(X[sample_ind],...


I am working on the file "Multiclass/multiclass classification.ipynb".
The first task is:
Notice the margin vs classification accuracy trade-off by tuning parameter C
Can you write Python code for it?

# create a mesh to plot in
h = .02  # step size in the mesh
x_min, x_max = min(X[:,0])-3,max(X[:,0])+3
y_min, y_max = min(X[:,1])-3,max(X[:,1])+3
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))
mesh_input = np.c_[xx.ravel(), yy.ravel()]

def f(parameter):
    svm_estimator = svm.LinearSVC(loss='hinge', fit_intercept=False, C=parameter)
    clf_onevsall = OneVsAllClassifier(svm_estimator, n_classes=3)
    clf_onevsall.fit(X,y)

    Z = clf_onevsall.predict(mesh_input)
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.2)
    # Plot also the training points
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.coolwarm)

for c in [1, 50, 100, 200, 500, 1000]:
    f(c)


I am working on the file "Multilayer Perceptron Example.ipynb".
The first task is:
Define parameters
In this section, I am defining:

relevant hypter parameters (optimizer and model parameters),
define input placeholders, and
define weights and biases
Can you write Python code for it?

# Hyper parameters
training_epochs = 100
learning_rate = 0.01
batch_size = 256
print_loss_for_each_epoch = 10
test_validation_size = 512 # validation images to use during training - solely for printing purposes

# Network parameters
n_input = 784               # MNIST length of 28 by 28 image when stored as a column vector
n_hidden_layer_1 = 1024     # features in the 1st hidden layer
n_hidden_layer_2 = 1024     # features in the 2nd hidden layer
n_classes = 10              # total label classes (0-9 digits)
dropout_keep_rate = 0.75    # only 25% of the hidden outputs are passed on

# Graph input placeholders
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, n_classes])
keep_prob = tf.placeholder(tf.float32)

# Define weights and biases
weights = {'hl_1': tf.Variable(tf.truncated_normal([n_input,n_hidden_layer_1])),
           'hl_2': tf.Variable(tf.truncated_normal([n_hidden_layer_1,n_hidden_layer_2])),
           'output': tf.Variable(tf.truncated_normal([n_hidden_layer_2,n_classes]))}

biases = {'hl_1': tf.Variable(0.01 * tf.truncated_normal([n_hidden_layer_1])),
          'hl_2': tf.Variable(0.01 * tf.truncated_normal([n_hidden_layer_2])),
          'output': tf.Variable(0.01 * tf.truncated_normal([n_classes]))}


I am working on the file "Multilayer Perceptron Example.ipynb".
The first task is:
Import data
I use the MNIST database of handwritten digits found @ http://yann.lecun.com/exdb/mnist/.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
mnist_data = input_data.read_data_sets("MNIST_data/", one_hot=True)


I am working on the file "Multilayer Perceptron Example.ipynb".
The first task is:
Launch TF Graph
Finally, a simple output to calculate the train, validation and test accuracy. Not bad for a simple MLP net.
Can you write Python code for it?

# Init variables
init = tf.global_variables_initializer()

# Run Graph
with tf.Session() as sess:
    sess.run(init)

    # Training cycle
    for epoch in range(training_epochs):

        for batch in range(mnist_data.train.num_examples//batch_size):

            # Get x and y values for the given batch
            batch_x, batch_y = mnist_data.train.next_batch(batch_size)

            # Compute graph with respect to 'optimizer' and 'cost'
            _, loss, training_accuracy = sess.run([optimizer, cost, accuracy], feed_dict={x: batch_x,
                                                             y: batch_y,
                                                             keep_prob: dropout_keep_rate})

            # Compute graph with respect to validation data
            validation_accuracy = sess.run(accuracy, feed_dict={x: mnist_data.validation.images[:test_validation_size],
                                                                y: mnist_data.validation.labels[:test_validation_size],
                                                                keep_prob: 1.})

        # Display logs per epoch step
        if epoch % print_loss_for_each_epoch == 0:
            print('Epoch {:>2}, Batches {:>3}, Loss: {:>10.4f}, Train Accuracy: {:.4f}, Val Accuracy: {:.4f}'.format(
                epoch + 1, # epoch starts at 0
                batch + 1, # batch starts at 0
                loss,
                training_accuracy,
                validation_accuracy))

    print('Optimization Finished!')

    # Testing cycle
    test_accuracy = sess.run(accuracy, feed_dict={x: mnist_data.test.images,
                                                  y: mnist_data.test.labels,
                                                  keep_prob: 1.})
    print('Test accuracy: {:3f}'.format(test_accuracy))


I am working on the file "Multipartite_entanglement.ipynb".
The first task is:
This is the function we would like to optimize:
Can you write Python code for it?

def funk(settings):
    value = get_solution(variables, sdpRelaxation, psi, settings)
    print(value)
    return value


I am working on the file "Multipartite_entanglement.ipynb".
The first task is:
We try it on a particular setting:
Can you write Python code for it?

N = 3      # Number of parties
M = 2      # Number of measuerment settings
K = 2      # Number of outcomes
level = 2  # Order of relaxation
psi = (tensor(basis(2, 0), basis(2, 0), basis(2, 0)) +
       tensor(basis(2, 1), basis(2, 1), basis(2, 1))).unit()  # GHZ state
settings = [pi/2, -pi/12, pi/2, -pi/12+pi/2, pi/2, -pi/12, pi/2,
            -pi/12+pi/2, pi/2, -pi/12, pi/2, -pi/12+pi/2]
result = minimize(funk, settings)


I am working on the file "MultipleRegression.ipynb".
The first task is:
Data managment
In order to carry out the regression analysis, we need to center the potential explanatory variables.
Can you write Python code for it?

subdata = (data[['incomeperperson', 'relectricperperson', 'employrate', 'oilperperson', 'urbanrate']]
               .assign(income=lambda x: pd.to_numeric(x['incomeperperson'], errors='coerce'),
                       electricity=lambda x: pd.to_numeric(x['relectricperperson'], errors='coerce'),
                       employ=lambda x: pd.to_numeric(x['employrate'], errors='coerce'),
                       oil=lambda x: pd.to_numeric(x['oilperperson'], errors='coerce'),
                       urban=lambda x: pd.to_numeric(x['urbanrate'], errors='coerce'))
               .dropna())

display(Markdown("Number of countries after discarding countries with missing data: {}".format(len(subdata))))

centered_data = (subdata.assign(income_c=lambda x: x['income'] - subdata['income'].mean(),
                                employ_c=lambda x: x['employ'] - subdata['employ'].mean(),
                                oil_c=lambda x: x['oil'] - subdata['oil'].mean(),
                                urban_c=lambda x: x['urban'] - subdata['urban'].mean()))


I am working on the file "MultipleRegression.ipynb".
The first task is:
General information on the Gapminder data
Can you write Python code for it?

display(Markdown("Number of countries: {}".format(len(data))))


I am working on the file "MultipleRegression.ipynb".
The first task is:
Let's check the variables are well centered
Can you write Python code for it?

display(Markdown("Mean of income per person after centering: {:3g}".format(centered_data['income_c'].mean())))
display(Markdown("Mean of employment rate after centering: {:3g}".format(centered_data['employ_c'].mean())))
display(Markdown("Mean of oil consumption per person after centering: {:3g}".format(centered_data['oil_c'].mean())))
display(Markdown("Mean of urban rate after centering: {:3g}".format(centered_data['urban_c'].mean())))


I am working on the file "MultipleRegression.ipynb".
The first task is:
Test a second order polynomial regression model.
Can you write Python code for it?

reg2 = smf.ols('electricity ~ income_c + I(income_c**2)', data=centered_data).fit()
reg2.summary()


I am working on the file "MultipleRegression.ipynb".
The first task is:
The leverage plot above shows that our extreme outlier United Arab Emirates does not have a strong influence on the estimation of the model coefficient. On the contrary Norway, the second border highest residual, has a important influence on the model estimation.
Analysis of trouble case
To conclude this assignment, I would to take the same question but taking the oil consumption as primary explanatory variables.
Let's first see if a second order fits better than the linear regression model.
Can you write Python code for it?

sns.regplot(x='oil_c', y='electricity', data=centered_data);
sns.regplot(x='oil_c', y='electricity', order=2, data=centered_data)
plt.xlabel('Centered oil consumption per person (tonnes)')
plt.ylabel('Residential electricity consumption (kWh)')
plt.title('Scatterplot for the association between the oil and the residential electricity consumption');

reg_oil1 = smf.ols('electricity ~ oil_c', data=centered_data).fit()
reg_oil1.summary()

reg_oil2 = smf.ols('electricity ~ oil_c + I(oil_c**2)', data=centered_data).fit()
reg_oil2.summary()


I am working on the file "MultipleRegression.ipynb".
The first task is:
The partial regression plots above are shown for the sake of the assignement as all variables but the income per person are non-significant in the multiple regression model. They all show that some extreme outliers will be present.
And the partial plot against the urban rate has a horizontal partial regression line. This confirms that urban rate cannot improve the model.
Leverage plot
Can you write Python code for it?


I am working on the file "MultipleRegression.ipynb".
The first task is:
The previous plot highlights only one clear extreme outlier. So this confirm that the model is fine and could be improve.
Partial regression plots
Can you write Python code for it?

fig = plt.figure(figsize=(12,8))
sm.graphics.plot_regress_exog(reg3, 'urban_c', fig);

fig = plt.figure(figsize=(12,8))
sm.graphics.plot_regress_exog(reg3, 'oil_c', fig);

fig = plt.figure(figsize=(12,8))
sm.graphics.plot_regress_exog(reg3, 'employ_c', fig);


I am working on the file "MultipleRegression.ipynb".
The first task is:
The residuals do not follow correctly the line. Especially on the second half of the data. As our model is a single linear regression between residential electricity consumption and income per person, this means that the model is a poor predicator for country having larger income.
Residuals plot
Can you write Python code for it?

stdres = pd.DataFrame(reg1.resid_pearson)
plt.plot(stdres, 'o')
plt.axhline(y=0, color='r')
plt.ylabel('Standardized Residual')
plt.xlabel('Observation Number');


I am working on the file "MultipleRegression.ipynb".
The first task is:
収率データ(y)に圧力(x1)、温度(x2)、酸度(x3)の3個の説明変数で重回帰分析を行う。
Can you write Python code for it?

y = np.asarray(data["y"])
X = np.c_[np.ones(len(data)), np.asarray(data[[' x1', ' x2', ' x3']])]
beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
e = y - X.dot(beta)
R2 = 1 - e.dot(e.T)/sum((y - np.mean(y))**2)
print('y = ' + str(y))
print('X = \n' + str(X))
print('b = \n' + str(beta))
print('R2 = \n' + str(R2))


I am working on the file "MultipleRegression.ipynb".
The first task is:
同様にして、温度(x2)を説明変数とした場合は、
Can you write Python code for it?

y = np.asarray(data["y"])
X = np.c_[np.ones(len(data)), np.asarray(data[' x2'])]
beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
e = y - X.dot(beta)
R2 = 1 - e.dot(e.T)/sum((y - np.mean(y))**2)
print('y = ' + str(y))
print('X = \n' + str(X))
print('b = \n' + str(beta))
print('R2 = \n' + str(R2))


I am working on the file "MultipleRegression.ipynb".
The first task is:
故に、
$\hat y_{i} = 11.8 +0.184x_{2i}$が得られ、寄与率は$R^{2} = 0.028$である。
Can you write Python code for it?

plt.scatter(data[' x2'], data['y'])
plt.plot(data[' x2'], X.dot(beta))
plt.text(91, 31,"$R^{2}=%s$" % round(R2, 3), fontsize=17)


I am working on the file "MultipleRegression.ipynb".
The first task is:
故に、
$\hat y_{i} = 39.1 - 0.645 x_{1i}$
が得られ、寄与率は$R^{2} = 0.29$である。
Can you write Python code for it?

plt.scatter(data[' x1'], data['y'])
plt.plot(data[' x1'], X.dot(beta))
plt.text(18, 31,"$R^{2}=%s$" % round(R2, 3), fontsize=17)


I am working on the file "MultipleRegression.ipynb".
The first task is:
重回帰分析
以下のように変数を設定する。
$
y =
\begin{bmatrix}
y_{1} \
y_{2} \
\vdots \
y_{n}
\end{bmatrix}
,
X =
\begin{bmatrix}
x_{11} & \cdots & x_{1i} & \cdots & x_{1k}\
\vdots & \ddots & & & \vdots \
x_{i1} & & x_{ii} & & x_{ik} \
\vdots & & & \ddots & \vdots \
x_{n1} & \cdots & x_{ni} & \cdots & x_{nk}
\end{bmatrix}
,
\beta =
\begin{bmatrix}
\beta_{1} \
\vdots \
\beta_{n}
\end{bmatrix}
,
\epsilon =
\begin{bmatrix}
\epsilon_{1} \
\vdots \
\epsilon_{n}
\end{bmatrix}
$
このとき、最小自乗法により
$\hat \beta = (X'X)^{-1}X'y$となる。
まず教科書p.28の表2.2を読み込む。
Can you write Python code for it?

data = pd.read_csv("tab22.csv") # http://shimotsu.web.fc2.com/Site/duo_bian_liang_jie_xi.html
data


I am working on the file "Multiplicacion de matrices.ipynb".
The first task is:
Ejecutar y tomar las dos primeras filas para verificar si sí funciona el algoritmo
Can you write Python code for it?

matriz = inicializarMatrizNueva(10000,20,1000)
print(matriz.take(2))

matriz2 = inicializarMatrizNueva(20,10000,1000)
print(matriz.take(2))


I am working on the file "Multiplicacion de matrices.ipynb".
The first task is:
Función para separar y limpiar los números del archivo. Recibe cada línea y la separa por espacios, luego limpia ,, [ y ]
Can you write Python code for it?

def quitar(x):
    y = x.split()
    return map(lambda w: int(w.replace(",","").replace("[","").replace("]","")),y)

matriz2 = matriz2.map(quitar)

print(matriz2.take(2))


I am working on the file "Multiplicacion de matrices.ipynb".
The first task is:
Se guarda el archivo en $HOMEHDFS/matriz5
Can you write Python code for it?


I am working on the file "MultivariableRegression_ChallengeWithCrossValidation.ipynb".
The first task is:
Hmmmm....it seems that outlier result has also heavily weighted the R-squared result and coefficients. The linear model which did not incorporate the outlier is likely to be a better indicator of overall trends and accuracy.
Best Model:
Can you write Python code for it?

regr = linear_model.LinearRegression()
Y = dataCleaned['Property\ncrime'].values.reshape(-1, 1)
X = dataCleaned[['Population', 'Violent\ncrime',
       'Murder', 'Rape\n(legacy\ndefinition)2',
       'Robbery', 'Aggravated\nassault']]
regr.fit(X, Y)

# Inspect the results.
print('\nCoefficients: \n', regr.coef_)
print('\nIntercept: \n', regr.intercept_)
print('\nR-squared:')
print(regr.score(X, Y))


I am working on the file "MultivariableRegression_ChallengeWithCrossValidation.ipynb".
The first task is:
Noice!!
What about performance when the binary Murder feature is used?
Can you write Python code for it?


I am working on the file "MultivariableRegression_ChallengeWithCrossValidation.ipynb".
The first task is:
Perfect accuracy, as expected. However......
Predicting ALL property crimes is a more interesting question.
Building a Model to Predict Property Crimes (without using the Property Crime features)
To start, let's take a look at how each of the non-property crime features interact with property crime.
Can you write Python code for it?

plt.figure(figsize=(15,5))
sns.pairplot(data, vars =['Property\ncrime', 'Population', 'Violent\ncrime',
       'Murder and\nnonnegligent\nmanslaughter',
       'Rape\n(legacy\ndefinition)2',
       'Robbery', 'Aggravated\nassault'])
plt.show()


I am working on the file "MultivariableRegression_ChallengeWithCrossValidation.ipynb".
The first task is:
That single outlier is making the relationships difficult to view. Let's remove the outlier.
Can you write Python code for it?

dataCleaned = data[data["Property\ncrime"] < 20000]

plt.figure(figsize=(15,5))
sns.pairplot(dataCleaned, vars =['Property\ncrime', 'Population', 'Violent\ncrime',
       'Murder and\nnonnegligent\nmanslaughter',
       'Rape\n(legacy\ndefinition)2',
       'Robbery', 'Aggravated\nassault'])
plt.show()

plt.scatter(dataCleaned["Property\ncrime"], dataCleaned["Murder and\nnonnegligent\nmanslaughter"])
plt.title('Raw values')
plt.xlabel("Property Crime")
plt.ylabel("Murder")
plt.show()


I am working on the file "MultivariableRegression_ChallengeWithCrossValidation.ipynb".
The first task is:
There is a large number of 0's for Murder. Perhaps let's use a binary value for murder occurring vs no murder occurring.
Can you write Python code for it?

dataCleaned["Murder"] = dataCleaned['Murder and\nnonnegligent\nmanslaughter'].apply(lambda x: 0 if x == 0 else 1)

plt.scatter(dataCleaned["Property\ncrime"], dataCleaned["Murder"])
plt.title('Raw values')
plt.xlabel("Property Crime")
plt.ylabel("Murder")
plt.show()

dataCleaned.head()

regr = linear_model.LinearRegression()
Y = dataCleaned['Property\ncrime'].values.reshape(-1, 1)
X = dataCleaned[['Population', 'Violent\ncrime',
       'Murder and\nnonnegligent\nmanslaughter',
       'Rape\n(legacy\ndefinition)2',
       'Robbery', 'Aggravated\nassault']]
regr.fit(X, Y)

# Inspect the results.
print('\nCoefficients: \n', regr.coef_)
print('\nIntercept: \n', regr.intercept_)
print('\nR-squared:')
print(regr.score(X, Y))


I am working on the file "MultivariableRegression_ChallengeWithCrossValidation.ipynb".
The first task is:
There is a slight increase of performance when the binary indicator for murder is used.
Leave no man behind!
Reintroduce the outlier to the model.
Can you write Python code for it?

data["Murder"] = data['Murder and\nnonnegligent\nmanslaughter'].apply(lambda x: 0 if x == 0 else 1)

regr = linear_model.LinearRegression()
Y = data['Property\ncrime'].values.reshape(-1, 1)
X = data[['Population', 'Violent\ncrime',
       'Murder', 'Rape\n(legacy\ndefinition)2',
       'Robbery', 'Aggravated\nassault']]
regr.fit(X, Y)

# Inspect the results.
print('\nCoefficients: \n', regr.coef_)
print('\nIntercept: \n', regr.intercept_)
print('\nR-squared:')
print(regr.score(X, Y))