I am working on the file ".ipynb_checkpoints/Exercices_220217-checkpoint.ipynb".
The first task is:
Nous créeons ensuite notre fonction que nous appellerons fromFileToCSV. Cette fonction prend deux arguements : le chemin vers le dossier et le nom du fichier csv. La signature de la fonction est comme suit, fromFileToCSV(folderpath, csvfilename)
Can you write Python code for it?

def fromFileToCSV (folderpath,csvfilename) :
    files = [f for f in listdir(folderpath) if isfile(join(folderpath, f))]
    random.shuffle(files)

    for filepath in files:
        if filepath.endswith(".png"):
            label = re.findall("^(\d+)_",filepath)
            csvLine = filepath+","+str(label[0])
            print csvLine
            with open(folderpath+csvfilename, "a") as myfile:
                myfile.write(csvLine)
                myfile.write("\n")


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
Here are all of the papers whose abstracts contain the word 'arthropod':
Can you write Python code for it?

abstractTerms = corpus.features['abstract']

abstractTerms.papers_containing('arthropod')


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
Here is the impact on the value for 'arthropod' in one document, using the two transformations above.
Can you write Python code for it?

print 'Before: '.ljust(15), corpus.features['abstract'].features['WOS:000324532900018'].value('arthropod')
print 'TF*IDF: '.ljust(15), corpus.features['abstracts_tfidf'].features['WOS:000324532900018'].value('arthropod')
print 'mytransformer: '.ljust(15), corpus.features['abstracts_transformed'].features['WOS:000324532900018'].value('arthropod')


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
Networks of features based on co-occurrence
The features module in the tethne.networks subpackage contains a few functions for generating networks of features based on co-occurrence.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
Once again, we'll use the GraphML writer to generate a visualizable network file.
Can you write Python code for it?

from tethne.writers import graph

mi_outpath = '/Users/erickpeirson/Projects/tethne-notebooks/output/mi_graph.graphml'

graph.to_graphml(MI_graph, mi_outpath)


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
Take a look at the ratio of nodes to edges to get a sense of how to tune the min_weight parameter. If you have an extremely high number of edges for the number of nodes, then you should probably increase min_weight to obtain a more legible network. Depending on your field, you may have some guidance from theory as well.
Can you write Python code for it?

print 'This graph has {0} nodes and {1} edges'.format(MI_graph.order(), MI_graph.size())


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
The mutual_information function in the features module generates a network based on the pointwise mutual information of each pair of features in a featureset.
The first argument is a list of Papers, just like most other network-building functions. The second argument is the featureset that we wish to use.
Can you write Python code for it?

MI_graph = features.mutual_information(corpus, 'abstracts_filtered', min_weight=0.7)


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
We can also use transform() to remove words from our FeatureSet. For example, we can apply the NLTK stoplist and remove too-common or too-rare words:
Can you write Python code for it?

from nltk.corpus import stopwords
stoplist = stopwords.words()

def apply_stoplist(f, v, c, dc):
    if f in stoplist or dc > 50 or dc < 3:
        return 0
    return v

corpus.features['abstracts_filtered'] = corpus.features['abstracts_tfidf'].transform(apply_stoplist)

print 'Before: '.ljust(10), len(corpus.features['abstracts_tfidf'].index)
print 'After: '.ljust(10), len(corpus.features['abstracts_filtered'].index)


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
We can then pass transformer function to transform as the first positional argument.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Feature Co-Occurrence-checkpoint.ipynb".
The first task is:
We can use index_feature() to tokenize the abstract into individual words.
Can you write Python code for it?

corpus.index_feature('abstract', tokenize=lambda x: x.split(' '))


I am working on the file ".ipynb_checkpoints/Feature Engineering for Text Data-checkpoint.ipynb".
The first task is:
Baseline accuracy (what if we classify everything as the majority class)
Percentage of 'funny' reviews:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Feature Engineering for Text Data-checkpoint.ipynb".
The first task is:
Create bag-of-words representation of text
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Feature Engineering for Text Data-checkpoint.ipynb".
The first task is:
Filter rows to remove reviews with no votes
Can you write Python code for it?

reviews['total_votes'] > 0

reviews = reviews[reviews['total_votes'] > 0]
reviews

reviews.num_rows()


I am working on the file ".ipynb_checkpoints/Feature Engineering for Text Data-checkpoint.ipynb".
The first task is:
Munge votes and add a new column
Can you write Python code for it?

reviews['total_votes'] = reviews['funny'] + reviews['cool'] + reviews['useful']
reviews


I am working on the file ".ipynb_checkpoints/Feature Engineering for Text Data-checkpoint.ipynb".
The first task is:
Quick data visualization
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Feature Engineering for Text Data-checkpoint.ipynb".
The first task is:
Represent datetime
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Feature Engineering for Text Data-checkpoint.ipynb".
The first task is:
Unpack to extract structure
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Filmfare Data Analysis-checkpoint.ipynb".
The first task is:
Best Picture Run time
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Filmfare Data Analysis-checkpoint.ipynb".
The first task is:
Best Picture by Genre
Can you write Python code for it?

# Movies by Genre
Category=Counter(datatbl['data']['Genre'])
df1 = pd.DataFrame.from_dict(Category,orient='index')
df1 = df1.sort_values([0],ascending=[False]).head(5)
df1.plot(kind='barh',color=['g','c','m']);

# Vote vs rating
%matplotlib inline
import matplotlib.pyplot as plt
df.plot(x=df.Votes,y=['Rating'],kind='scatter');
# plt.scatter(df.Votes, df.Rating, s=df.Rating)

import numpy as np
df = pd.DataFrame(np.random.randint(100000, size=(10000, 2)),
                  columns=['Votes', 'Rating'])

df.plot(kind='scatter', x='Votes', y='Rating', logx=True, alpha=0.5, color='purple', edgecolor='')
plt.ylabel('IMDB Rating')
plt.xlabel('Number of Votes')
plt.show()


I am working on the file ".ipynb_checkpoints/Filmfare Data Analysis-checkpoint.ipynb".
The first task is:
Clean Data
Can you write Python code for it?

for key,values in result.items():
    values['Votes'] = int(values['Votes'].replace(",",""))
    values['Rating']= float(values['Rating'])
    try:
        values['RunTime'] = int(values['RunTime'])
    except ValueError:
        values['RunTime'] = 0

print(json.dumps(result[0], indent=2))


I am working on the file ".ipynb_checkpoints/Filmfare Data Analysis-checkpoint.ipynb".
The first task is:
Data before Clean Up
Can you write Python code for it?

# Dictionary Result
print(json.dumps(result[59], indent=2))


I am working on the file ".ipynb_checkpoints/Filmfare Data Analysis-checkpoint.ipynb".
The first task is:
Data in Pandas Dataframe
Can you write Python code for it?

# create dataframe
df = pd.DataFrame.from_dict(result,orient='index')
df.sort_values(by='Year',ascending=True,inplace=True)
df = df[['Year', 'Movie Name', 'Rating', 'Votes','RunTime']]
df


I am working on the file ".ipynb_checkpoints/Filmfare Data Analysis-checkpoint.ipynb".
The first task is:
How does data in Dictionary looks like
Can you write Python code for it?

# Dictionary Result
print(json.dumps(result[0], indent=2))


I am working on the file ".ipynb_checkpoints/Filmfare Data Analysis-checkpoint.ipynb".
The first task is:
Movies with Highest Ratings
Can you write Python code for it?

#Highest Rating Movies
df.sort_values('Rating',ascending=[False]).head(5)


I am working on the file ".ipynb_checkpoints/Filmfare Data Analysis-checkpoint.ipynb".
The first task is:
Store Data in a  Python Dictionary
Can you write Python code for it?

# Result in a Python Dictionary
Years=range(2015,1954,-1)
result = defaultdict(dict)
for i in range(0,len(datarepo[0])):
    result[i]['Movie Name']= datarepo[0][i]
    result[i]['Year']= Years[i]
    result[i]['Rating']= datarepo[1][i]
    result[i]['Votes']= datarepo[3][i]
    result[i]['RunTime']= datarepo[2][i]


I am working on the file ".ipynb_checkpoints/FilmfareDataAnalysis-checkpoint.ipynb".
The first task is:
Initial Setup and Launch the browser to open the URL
Can you write Python code for it?

url = "http://www.imdb.com/list/ls061683439/"
with open('./img/filmfare.json',encoding="utf-8") as f:
    datatbl = json.load(f)
driver = webdriver.Chrome(datatbl['data']['chromedriver'])
driver.get(url)


I am working on the file ".ipynb_checkpoints/FilmfareDataAnalysis-checkpoint.ipynb".
The first task is:
Let's extract all the required data like Ratings,Votes,Genre, Year of Release for the Best Movies
Can you write Python code for it?

#Extracting Data from Web
Movies_Votes,Movies_Name,Movies_Ratings,Movies_RunTime=[[] for i in range(4)]
datarepo = [[]]*4
Xpath_list = ['Movies_Name_Xpath','Movies_Rate_Xpath','Movies_Runtime_Xpath','Movies_Votes_Xpath']

for i in range(4):
    if(i==3):
        driver.find_element_by_xpath(datatbl['data']['listview']).click()
    datarepo[i] = ExtractText(Xpath_list[i])

driver.quit()


I am working on the file ".ipynb_checkpoints/FilmfareDataAnalysis-checkpoint.ipynb".
The first task is:
Let's see now how the data in dictionary looks like?
Can you write Python code for it?

result

print(json.dumps(result[58], indent=2))


I am working on the file ".ipynb_checkpoints/FilmfareDataAnalysis-checkpoint.ipynb".
The first task is:
Mean of the Movie Run Time
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/FilmfareDataAnalysis-checkpoint.ipynb".
The first task is:
Movie Ratings Visualization using Bar Graph
Can you write Python code for it?

Rating_Histdic = defaultdict(dict)

Rating_Histdic['Btwn 6&7'] = df[(df['Rating']>=6)&(df['Rating']<7)]['Rating'].count()
Rating_Histdic['GTEQ 8'] = df[(df['Rating']>=8)]['Rating'].count()
Rating_Histdic['Btwn 7 & 8'] = df[(df['Rating']>=7)&(df['Rating']<8)]['Rating'].count()

plt.bar(range(len(Rating_Histdic)), Rating_Histdic.values(), align='center',color='brown',width=0.4)
plt.xticks(range(len(Rating_Histdic)), Rating_Histdic.keys(), rotation=25);


I am working on the file ".ipynb_checkpoints/FilmfareDataAnalysis-checkpoint.ipynb".
The first task is:
Now let's look at the data and see how it looks like
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Because some films do not have a close date, we will have to be careful with the close date!
Can you write Python code for it?

# if we decide it's worth just dumping the movies with no close_date, we can use the code below
# movie_df=movie_df[movie_df.close_date != '-'].reset_index(drop=True)


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Because the valences are read in as a string, we will clean it up and change it to an int so it is ready for manipulation.
Can you write Python code for it?

dictionary['valences'] = dictionary['valences'].map(lambda x: x if x.find('\n')== -1 else x[:x.find('\n')])
dictionary['valences']=dictionary['valences'].astype(int)


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Finally we want to save our dictionary of IMDB_dftouse into a JSON file for storage.
Can you write Python code for it?

IMDB_dftouse_dict = {feature: IMDB_dftouse[feature].values.tolist() for feature in IMDB_dftouse.columns.values}

fp = open("IMDB_dftouse_dict.json","w")
json.dump(IMDB_dftouse_dict, fp)
fp.close()

with open("IMDB_dftouse_dict.json", "r") as fd:
    IMDB_dftouse_dict = json.load(fd)


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Finally we're ready to scrape! <br>
Because IMDB was created in 1990, we will scrape that far back in BOM. So we're scraping the past 26 years (1990 - 2015). Also note that because the HTML was changed starting in 2001, our scraping will be a little different before and after then.
Can you write Python code for it?

%%time
years = [1990 + i for i in range(26)]
for year in years:
    pageText = requests.get("http://www.boxofficemojo.com/yearly/chart/?yr=%(yr)d&p=.htm" % {'yr':year})
    soup = BeautifulSoup(pageText.text, "html.parser")
    movieTable = soup.find("td", attrs={"colspan":"3"})
    movieRows = movieTable.find("table").find_all("tr")[2:102]
    print year
    movie_dicts = [dict(zip(fields, rowInfoGrabber(row))) for row in movieRows]
    year_df = pd.DataFrame(movie_dicts)
    year_df['year'] = year
    movie_df = movie_df.append(year_df, ignore_index=True)
    time.sleep(1)

movie_df.shape

movie_df.head()


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Let's take a look at if we can get the run times for each movie!
Can you write Python code for it?

run_time=[]
for index, row in movie_df.iterrows():
    if row.close_date != None:
        run_time.append(row['close_date']-row['open_date'])
    else:
        run_time.append('N/A')

movie_df.head()


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Let's take a look at the data, now!
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Loading and preparing AFINN dictionary
We wil read in the AFINN-111 dictionary from the tab-delimited .txt file and save it into a pandas dataframe dictionary.
Can you write Python code for it?

words = []
valences = []

with open('AFINN/AFINN-111.txt','rb') as source:
    for line in source:
        fields = line.split('\t')
        words.append(fields[0])
        valences.append(fields[1])
dictionary = pd.DataFrame({'words': words, 'valences':valences})


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Loading and preparing IMDB review dataset
We have cleaned up our IMDB review dataset in the ipython notebook New.ipynb. From that notebook we have been able to save dictionaries of our data, which we will now call.
Can you write Python code for it?

with open("train_df_dict.json", "r") as fd:
    train_df_dict = json.load(fd)
with open("test_df_dict.json", "r") as fd:
    test_df_dict = json.load(fd)

train_df = pd.DataFrame(train_df_dict)
test_df = pd.DataFrame(test_df_dict)


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Looks like the data is ready for us to use! Let's save this data so we're ready to use it next time.
Can you write Python code for it?

! pip install pymongo

# Save the movie Dictionaries corresponding to each row of the BoxOfficeMojo table.
import json # (dong)
import pymongo
from bson import json_util

# Make a dictionary out of the dataset for storage in JSON format.
movieSaved = {feature: movie_df[feature].values.tolist() for feature in movie_df.columns.values}
fp = open("allMovies.json","w")
json.dump(movieSaved, fp, default=json_util.default)
fp.close()


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Looks like the dictionary is ready for us to use! Let's save it so we have it in the event that anything happens.
Can you write Python code for it?

# dictionary.to_csv("dictionary.csv", index=False)
dictionary=pd.read_csv("dictionary.csv")


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Milestone 1: Scrape and prepare data before thanksgiving
For our project we will be using data from 3 different sources

Box Office Mojo (BOM) (http://www.boxofficemojo.com) is a website that aggregates, in a table, a list of all movies released in a year and attributes such as how much it grossed in the opening week, how much it grossed in total and how long it aired for

Large Movie Review Dataset (http://ai.stanford.edu/~amaas/data/sentiment/) is a polarized dataset of movie reviews from IMDB prepared by Maas et al from Stanford. The dataset contains 25,000 entries in the training set and 25,000 entries in the test set.

AFINN-111 Dictionary (http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=60100) is a dictionary list of 2477 english words and phrases rated for valence with an integer from -5 to 5. Originally prepared by Finn Årup Nielsen

In this first milestone, we will get all the data into a format that we can start running analysis on!
Scraping and cleaning Box Office Mojo
First we import the requests and BeautifulSoup libraries to make working with HTTP requests easier, and then easily transfer HTML content to Python data structures.
Can you write Python code for it?

from bs4 import BeautifulSoup
# The "requests" library makes working with HTTP requests easier
# than the built-in urllib libraries.
import requests


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Next, we combine the close_date, open_date and year columns into two columns close_date and open_date that are time series. This will make it easier for us to work with the data in the future.
Can you write Python code for it?

# splitting the close_date and open_date into the respective month and day
movie_df['close_month'] = movie_df['close_date'].map(lambda x: '0' if x=='-' else x[:x.find('/')])
movie_df['close_day'] = movie_df['close_date'].map(lambda x: '0' if x=='-' else x[x.find('/')+1:len(x)])
movie_df['open_month'] = movie_df['open_date'].map(lambda x: x[:x.find('/')])
movie_df['open_day'] = movie_df['open_date'].map(lambda x: x[x.find('/')+1:len(x)])

# dropping the old close_date and open_date
movie_df = movie_df.drop('close_date', 1)
movie_df = movie_df.drop('open_date', 1)

# creating an open_year by turning the year column into a string and getting rid of trailing bits
movie_df['open_year'] = movie_df.year.astype(str)
movie_df['open_year'] = movie_df.open_year.map(lambda x: x[:x.find('.')])

# creating a close_year column, by looking at whether the close month is earlier/later than the open month in the year
close_month = movie_df['close_month'].astype(int)
open_month = movie_df['open_month'].astype(int)
year = movie_df['year'].astype(int)
close_year=[]
for i in range (0, len(year)):
    if close_month[i] >= open_month[i]:
        close_year.append(year[i])
    else:
        close_year.append(year[i]+1)
movie_df['close_year'] = close_year
movie_df['close_year'] = movie_df['close_year'].astype(str)

movie_df.head()

# making close_date and open_date by concatenating the year, month and day
import datetime
close_date = []
for index, row in movie_df.iterrows():
    if row.close_day != '0':
        close_date.append(datetime.datetime(int(row.close_year), int(row.close_month), int(row.close_day)))
    else:
        close_date.append(None)
movie_df['close_date'] = close_date

movie_df['open_date']=movie_df.open_year + '-' + movie_df.open_month + '-' + movie_df.open_day
movie_df['open_date']=movie_df['open_date'].apply(pd.datetools.parse)

# dropping unnecessary columns
movie_df = movie_df.drop('close_day', 1)
movie_df = movie_df.drop('close_month', 1)
movie_df = movie_df.drop('close_year', 1)
movie_df =...


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Now let's create a mask over IMDB_df, the boolean values of which indicate whether or not a movie is in the BOM list.
Can you write Python code for it?

movie_mask = [(movie in BOM_movie_list) for movie in IMDB_df.movie_name]

sum(movie_mask)


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Now we write a function rowInfoGrabber that we will call in a loop over the table on the BOM webpage to grab the attributes and save them into the corresponding columns in movie_df.
Can you write Python code for it?

def rowInfoGrabber(r):
    info = []
    # Ranking
    info.append(int(r.find("font").get_text()))
    # Title
    info.append(r.find("a").get_text())
    # Gross
    info.append(int(r.find("td", attrs={"align":"right"}).find("b").get_text().strip("$").replace(",","")))
    '''
    For the next 3 categories, we need to deal with the 2000 Anomaly "Fantasia" where there are missing numbers.
    In this case I have chosen to replace the missing values 'N/A' with the values from 'Final Destination', which
    if right above it in the movie table and differs in gross income by about $1 million, which is a small
    difference. See the picture below for a snapshot of the anomaly in the movie table from 2000.
    '''
    # Total number of theaters
    if r.find_all("td",attrs={"align":"right"})[1].find("font").get_text().replace(",","") == 'N/A':
        info.append(2587)
    else:
        info.append(int(r.find_all("td",attrs={"align":"right"})[1].find("font").get_text().replace(",","")))
    # Opening Gross
    if r.find_all("td", attrs={"align":"right"})[2].find("font").get_text().strip("$").replace(",","") == 'N/A':
        info.append(10015822)
    else:
        info.append(int(r.find_all("td", attrs={"align":"right"})[2].find("font").get_text().strip("$").replace(",","")))
    # Opening Number of Theaters
    if r.find_all("td", attrs={"align":"right"})[3].find("font").get_text().replace(",","") == 'N/A':
        info.append(2587)
    else:
        info.append(int(r.find_all("td", attrs={"align":"right"})[3].find("font").get_text().replace(",","")))
    # Date of Opening
    info.append(r.find_all("td", attrs={"align":"right"})[4].find("a").get_text())
    # Date of Closing: Before 2002 they didn't have a "closing" date in their tables. We must account for this.
    if (len(r.find_all("td", attrs={"align":"right"})) <= 5):
        info.append('-')
    else:
        info.append(r.find_all("td", attrs={"align":"right"})[5].find("font").get_text())
    return info


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
Secondly, we prepare the data frame movie_df to store the data that we will scrape from BOM. We give this dataframe 9 columns: <br>
    * ranking: the ranking of the movie in its release year by gross
    * title: movie title
    * gross: how much the movie grossed while in theatres
    * Total_theaters: the total number of theaters that showed this movie
    * opening_gross: how much the movie grossed in the opening weekend (Fri-Sun)
    * opening_theaters: the total number of theaters that showed this movie in the opening weekend (Fri-Sun)
    * open_date: date of opening
    * close_date: date of closing
    * year: year of release
Can you write Python code for it?

movie_df = pd.DataFrame(columns=['close_date', 'gross', 'open_date', 'opening_gross', 'opening_theaters','ranking','title','total_theaters','year'])


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
The Stanford group distinguishes between train and test because this was relevant for their project. This may prove to be useful later, so we will keep two separate dataframes. However, for our purposes at the moment, we can combine them since the BOM data will serve as our true data set.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
This is the image: <image src=“Example.png”>
Can you write Python code for it?

fields = ["ranking", "title", "gross", "total_theaters", "opening_gross", "opening_theaters", "open_date", "close_date"]


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
We can now create out final, relevant IMDB data frame, with only those movies that also appear in the BOM tables from 1990 - 2015.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project-checkpoint.ipynb".
The first task is:
We want to figure out which movies from IMDB_df are also present in our BOM DF. So let's get all the movie titles in our BOM table.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
ASK ANDREW WHETHER WE NEED TEST/TRAIN FOR OUR REGRESSION
WORK FLOW 0 GOAL: DESCRIPTIVE STATISTICS <br>
- How many unique movies we have in our IMDB_df
- how many reviews they have each
- which years do they represent
WORK FLOW 1 GOAL: MAKE A VALENCE SCORE THAT CLOSELY MATCHES STARS
1. Create a valence score for each review
    * summation
    * average
    * product
1a. Check how good our valence score is by testing on the stars
STEPHEN's EDITS number 2
WORK FLOW 2 GOAL: FIND A REGRESSION MODEL THAT BEST PREDICTS THE FOLLOWING Y VARIABLES
2. Regress on:
    * opening_gross (controlled for opening weekend)
    * opening_gross/opening_theater
    * total_gross
    * total_gross/total_theateer

Just realised that I was meant to take notes.
- try making branches on github: each person should work on their own branch and not push to the origin until you are sure what you're doing is correct
- share andrew on the github
-
Re: pretty woman question
- useful to consider polarizing reviews actually predict the box office performance
Re: regression
- if at the end of the day we choose to do simply box office gross vs. sentiment this is just a correlation, then it is more on the simper side of things
- the difficulty should be more commensurate with the pset difficulty
- should take it to the next level, a few possibilities
*

measures of dispersion on the sentiment analysis
different types of distributions you have in a given review
how frequently are certain words used
average length of words
average document lengths
ratio between high valence words and low valence words
we can do different sentiment scores based on different sentiment dictionaries
could use a BOW model (have the corpus of words in the reviews) & shrink down for regularization

We can adopt the hypothesis that stars predict box office gross
Or hypothesis that evry polarized responses predict box office gross
It's okay to have a dud hypothesis
Graded on the effort we put in - low unimpressive results should not be equated with not having...
Can you write Python code for it?

#getting inflation data and creating a new dataframe
inflation = pd.read_csv("inf.csv")
print inflation

#Creating a dataframe of Cumulative Inflation
years_90 = range(1990,2015)
infdict = {}

infindex = 0
infvalue = 1

testlist = []
for row in inflation.values:
    currentval = 1 + (row[1]/100)
    cuminf = infvalue*currentval
    infdict[years_90[infindex]] = cuminf
    infindex += 1
    infvalue = cuminf
    testlist.append(cuminf)

inframe = pd.DataFrame(data=testlist, index=range(1990,2015))
#infdict exists in case we need it later

inframe

adjframe =


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Analyzing and Saving Review Attributes Using labMT Happiness Dictionary
Now let's download labMT, a word score list for sentiment analysis containing over 10,000 words. The file contains a "happiness" value, and ranks words by their happiness. It also includes mean and standard deviation, Twitter rank and Google rank.
Can you write Python code for it?

url = 'http://www.plosone.org/article/fetchSingleRepresentation.action?uri=info:doi/10.1371/journal.pone.0026752.s001'
labmt = pd.read_csv(url, skiprows=2, sep='\t', index_col=0)

labmt.head()


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Because some films do not have a close date, we will have to be careful with the close date!
Can you write Python code for it?

# if we decide it's worth just dumping the movies with no close_date, we can use the code below
# movie_df=movie_df[movie_df.close_date != '-'].reset_index(drop=True)


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Finally we want to save our dictionary of IMDB_dftouse into a JSON file for storage.
Can you write Python code for it?

IMDB_dftouse_dict = {feature: IMDB_dftouse[feature].values.tolist() for feature in IMDB_dftouse.columns.values}
fp = open("IMDB_dftouse_dict.json","w")
json.dump(IMDB_dftouse_dict, fp)
fp.close()

# Reopen
with open("IMDB_dftouse_dict.json", "r") as fd:
    IMDB_dftouse_dict = json.load(fd)


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Finally we're ready to scrape! <br>
Because IMDB was created in 1990, we will scrape that far back in BOM. So we're scraping the past 26 years (1990 - 2015). Also note that because the HTML was changed starting in 2001, our scraping will be a little different before and after then.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Great! Let's take a look at all the dataframes we've created
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Let's take a look at what our dataframe looks like now!
Can you write Python code for it?

flattened_df.head()

dftouse = flattened_df[~flattened_df['review_count'].map(np.isnan)]
dftouse.shape

dftouse.head()

plt.scatter(y=dftouse.opening_gross, x=dftouse.star_avg)

plt.scatter(y=dftouse_four.opening_gross, x=dftouse_four.star_avg,s=dftouse_four.review_count)

plt.scatter(y=dftouse_seven.opening_gross, x=dftouse_seven.star_avg,s=dftouse_seven.review_count)

dftouse_four = flattened_df[flattened_df['star_avg'] <= 4]
dftouse_seven = flattened_df[flattened_df['star_avg'] >= 7]

dftouse_four.head()

bobby_ols = ols('opening_gross ~ star_avg',dftouse).fit()
bobby_ols.summary()

bobby_ols = ols('opening_gross ~ star_avg',dftouse_four).fit()
bobby_ols.summary()

bobby_ols = ols('opening_gross ~ star_avg',dftouse_seven).fit()
bobby_ols.summary()


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Looks like the data is ready for us to use! Let's save this data so we're ready to use it next time.
Can you write Python code for it?

! pip install pymongo

# Save the movie Dictionaries corresponding to each row of the BoxOfficeMojo table.
import json # (dong)
# import pymongo
# from bson import json_util

# Make a dictionary out of the dataset for storage in JSON format.
movieSaved = {feature: movie_df[feature].values.tolist() for feature in movie_df.columns.values}
fp = open("allMovies_new.json","w")
json.dump(movieSaved, fp)
fp.close()


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Next, we combine the close_date, open_date and year columns into two columns close_date and open_date that are time series. This will make it easier for us to work with the data in the future.
Can you write Python code for it?

# splitting the close_date and open_date into the respective month and day
movie_df['close_month'] = movie_df['close_date'].map(lambda x: '0' if x=='-' else x[:x.find('/')])
movie_df['close_day'] = movie_df['close_date'].map(lambda x: '0' if x=='-' else x[x.find('/')+1:len(x)])
movie_df['open_month'] = movie_df['open_date'].map(lambda x: x[:x.find('/')])
movie_df['open_day'] = movie_df['open_date'].map(lambda x: x[x.find('/')+1:len(x)])

# dropping the old close_date and open_date
movie_df = movie_df.drop('close_date', 1)
movie_df = movie_df.drop('open_date', 1)

# creating an open_year by turning the year column into a string and getting rid of trailing bits
movie_df['open_year'] = movie_df.year.astype(str)
movie_df['open_year'] = movie_df.open_year.map(lambda x: x[:x.find('.')])

# creating a close_year column, by looking at whether the close month is earlier/later than the open month in the year
close_month = movie_df['close_month'].astype(int)
open_month = movie_df['open_month'].astype(int)
year = movie_df['year'].astype(int)
close_year=[]
for i in range (0, len(year)):
    if close_month[i] >= open_month[i]:
        close_year.append(year[i])
    else:
        close_year.append(year[i]+1)
movie_df['close_year'] = close_year
movie_df['close_year'] = movie_df['close_year'].astype(str)

movie_df.head()


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Now let's collect several attributes from a given review's text body, and save all valuable information into a new data frame. First we define a function that removes stop words (all non important words from a valence perspective) from a text body.
Can you write Python code for it?

from sklearn.feature_extraction import text
stopwords = text.ENGLISH_STOP_WORDS
punctuation = list('.,;:!?()[]{}`''\"@#$%^&*+-|-=~_')

def removeStopWords(text, stopwords = stopwords):
    new_text = ""
    for word in text.split():
        if word not in stopwords:
            while len(word) != 0 and word[-1] in punctuation:
                word = word[:len(word)-1]
            new_text += word + ' '
    return new_text


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Now let's create a new dataframe valence_df with the valence statistics run on our IMDB_df. This code takes a few minutes to run.
Can you write Python code for it?

%%time
valence_df = getAllInfo(IMDB_df, happiness, stopwords)

valence_df.head()

# Convert True/False to 1/0: needed to make valence_df JSON serializable, also better practice
valence_df.positive = 1.0*valence_df.positive

# Save to disc
# fp = open("valence_df_dict.json","w")
# json.dump(valence_df.to_dict(), fp)
# fp.close()

# Reopen
with open("valence_df_dict.json", "r") as fp:
    valence_df_dict = json.load(fp)
valence_df = pd.DataFrame(valence_df_dict)


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Now we write a function rowInfoGrabber that we will call in a loop over the table on the BOM webpage to grab the attributes and save them into the corresponding columns in movie_df.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Now we'll import allMoives_new.json. Remember that we still need to convert some of the columns into the datetime type so we'll do that afterwards.
Can you write Python code for it?

with open("allMovies_new.json", "r") as fd:
    movie_df = json.load(fd)
    movie_df = pd.DataFrame(movie_df)

# making close_date and open_date by concatenating the year, month and day
import datetime
close_date = []
for index, row in movie_df.iterrows():
    if row.close_day != '0':
        close_date.append(datetime.datetime(int(row.close_year), int(row.close_month), int(row.close_day)))
    else:
        close_date.append(None)
movie_df['close_date'] = close_date

movie_df['open_date']=movie_df.open_year + '-' + movie_df.open_month + '-' + movie_df.open_day
movie_df['open_date']=movie_df['open_date'].apply(pd.datetools.parse)

# dropping unnecessary columns
movie_df = movie_df.drop('close_day', 1)
movie_df = movie_df.drop('close_month', 1)
movie_df = movie_df.drop('open_day', 1)
movie_df = movie_df.drop('open_month', 1)


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Now we'll write a function that returns total happiness, average happiness, total scorable words, and percentage of scorable words in a given review text.
Can you write Python code for it?

'''
Name: getValenceInfo()
Inputs: review text, dictionary of happiness
Returns: a 4-tuple of (happiness total, happiness average, total # of scorable words, % of scorable words)
'''
def getValenceInfo(text, valenceDict):
    total_words = len(text.split())
    happiness_total, count_relevant = 0, 0
    for word in text.split():
        if word in valenceDict.keys():
            count_relevant += 1
            happiness_total += valenceDict[word]
    if count_relevant != 0:
        avg_valence = 1.*happiness_total/count_relevant
    else:
        avg_valence = 0
    return happiness_total, avg_valence, total_words, 1.*count_relevant / total_words


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
Secondly, we prepare the data frame movie_df to store the data that we will scrape from BOM. We give this dataframe 9 columns: <br>
    * ranking: the ranking of the movie in its release year by gross
    * title: movie title
    * gross: how much the movie grossed while in theatres
    * Total_theaters: the total number of theaters that showed this movie
    * opening_gross: how much the movie grossed in the opening weekend (Fri-Sun)
    * opening_theaters: the total number of theaters that showed this movie in the opening weekend (Fri-Sun)
    * open_date: date of opening
    * close_date: date of closing
    * year: year of release
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
The Stanford group distinguishes between train and test because this was relevant for their project. This may prove to be useful later, so we will keep two separate dataframes. However, for our purposes at the moment, we can combine them since the BOM data will serve as our true data set.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
This is the image: <image src=“Example.png”>
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Final_Project_vAUS-checkpoint.ipynb".
The first task is:
We want to figure out which movies from IMDB_df are also present in our BOM DF. So let's get all the movie titles in our BOM table.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/FirstNotebook-checkpoint.ipynb".
The first task is:
Prepare filter columns that allow us to select any county/town combo, or county/town/vaccine name combo
Can you write Python code for it?

jud_or = judete + "_"+ orase
jud_or_vac = judete + "_"+ orase + "_"+ vaccinuri

print(np.sum(cantitati[jud_or=="Prahova_Busteni"]))
print(np.sum(cantitati[jud_or_vac=="Bucuresti_Sector 2_Hexacima"]))


I am working on the file ".ipynb_checkpoints/Fitting curve-checkpoint.ipynb".
The first task is:
import data for our model
This is data imported from statline CBS webportal.
Can you write Python code for it?

importfile = 'CBS Statline Gas Usage.xlsx'
df = pd.read_excel(importfile, sheetname='Month', skiprows=1)
df.drop(['Onderwerpen_1', 'Onderwerpen_2', 'Perioden'], axis=1, inplace=True)

df

# transpose
df = df.transpose()

new_header = df.iloc[0]
df = df[1:]
df.rename(columns = new_header, inplace=True)

#df.drop(['nan'], axis=0, inplace=True)
df

x = range(len(df.index))
df['Via regionale netten'].plot(figsize=(18,5))
plt.xticks(x, df.index, rotation='vertical')
plt.show()


I am working on the file ".ipynb_checkpoints/FristExample-checkpoint.ipynb".
The first task is:
Implementing the Prediction Step
$$ x_{k+1} = x_{k} + w_k $$
Note here a simplified Random Walk forecast model is used to substitute $f(x)$. The advantage of using a random walk forecast model is that now we are dealing with a linear instead of nonlinear filtering problem, and the computational cost is much lower as we don't need to evaluate $f(x)$. However, when $dt$ is very large, this random walk forecast model will give poor predictions, and the prediction error cannot be well approximated by $w_k\approx N(0,Q)$, a zero mean Gaussian process noise term. Therefore, the random walk forecast model is only useful when the measuremetns are sampled at a high frequency, and $Q$ has to be seclected to reflect the true model error.
Can you write Python code for it?

from filterpy.common import Q_discrete_white_noise
kf.F = np.diag(np.ones(dim_x))
# kf.Q = Q_discrete_white_noise(dim = dim_x, dt = 0.1, var = 2.35)
kf.Q = 2.5
kf.predict()
print kf.x[:10]


I am working on the file ".ipynb_checkpoints/FristExample-checkpoint.ipynb".
The first task is:
Implementing the Update Step
Can you write Python code for it?

kf.H = CO2.H_mtx
kf.R *= 0.5
z = data[0]
kf.update(z)


I am working on the file ".ipynb_checkpoints/FristExample-checkpoint.ipynb".
The first task is:
Simulating the Sensor Measurement
The sensor measures the travel time of a seismic signal from a source to a receiver.
$$ y = Hx + v $$
$x$ is the grid block value of CO$_2$ slowness, an idicator of how much CO$_2$ in a block. The product $Hx$ simulate the travel time measurements by integrating $x$ along a raypath. $v$ is the measurement noise.
The presence of CO$_2$ slows down the seismic signal and increases its travel time along a ray path. If the ray path does not intercepts the CO$_2$ plume, the travel time remains constant over time (Ray path 1), otherwise it tends to increase once the CO$_2$ plume intercepts the ray path (Ray path 2).
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/FristExample-checkpoint.ipynb".
The first task is:
TODO
- Fig: Estimate at k, Forecast at k+1, Estimate at k+1, True at k+1
- A table showing:
    x: the time CO2 reaches the monitoring well
    y: the time CO2 reaches the ray path
    PREDICT:    x       var      y       UPDATE: x       var       y
- Fig: MSE vs time
- Fig: Data fitting, slope 45 degree indicates a perfect fit
Use HiKF instead of KF
Can you write Python code for it?

from HiKF import HiKF
hikf = HiKF(dim_x, dim_z)
hikf.x


I am working on the file ".ipynb_checkpoints/FristExample-checkpoint.ipynb".
The first task is:
TODO:

Fig: Run animation/image of the ray path (shooting from one source and receiver) on top of a CO$_2$ plume and display the travel time changes over time.

Fig: Show the time-series data (Path 1 and Path 2) at a receiver with and without noise.

optional: run getraypath will give me all the index of the cells and the length of the ray path within each cell, this can help me compute the travel time along this particular ray path

Kalman filtering
Initialization step
Define $x$, $P$. Before injection took place, there was no CO$_2$ in the aquifer.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/FristExample-checkpoint.ipynb".
The first task is:
Tracking a CO$_2$ Plume
CO$_2$ from an industrial site can be compressed and injected into a deep saline aquifer for storage. This technology is called CO$_2$ capture and storage or CCS, proposed in (TODO) to combat global warming. As CO$_2$ is lighter than the saline water, it may leak through a natural fracture and contanimate the drinking water. Therefore, monitoring and predicting the long term fate of CO$_2$ at the deep aquifer level is crucial as it will provide an early warning for the CO$_2$ leakage. The goal is to interprete the time-series data recorded in the seismic sensors into spatial maps of a moving CO$_2$ plume, a problem very similar to CT scanning widely used in medical imaging.
The goal is
* Predict and monitor the location of CO$_2$ plume
*
Simulating the Movement of CO$_2$ Plume
Here is a simulated CO$_2$ plume for $5$ days resulted from injecting $300$ tons of CO$_2$ at a depth of $1657m$.
$$ x_{k+1} = f(x_k) + w $$
run code that displays the simulated moving CO$_2$ plume, stored the plume data in SQL?? (TODO)
Can you write Python code for it?

CO2 = CO2simulation('low')
data = []
x = []
for i in range(10):
  data.append(CO2.move_and_sense())
  x.append(CO2.x)

param = vco2.getImgParam('low')
vco2.plotCO2map(x,param)
plt.show()


I am working on the file ".ipynb_checkpoints/Full done-checkpoint.ipynb".
The first task is:
Model selection
Difficult to do with classic tools (here)
Problem:
1 - Calculate the BMA coefficient values

2 - Try running with different priors
Can you write Python code for it?

from pyBMA import CoxPHFitter
bmaCox = CoxPHFitter.CoxPHFitter()
bmaCox.fit(df, "lifetime", event_col= "event", priors= [0.5]*4)
bmaCox.summary

#Low probability for everything favours parsimonious models
bmaCox = CoxPHFitter.CoxPHFitter()
bmaCox.fit(df, "lifetime", event_col= "event", priors= [0.1]*4)
bmaCox.summary

#Boost probability of brand
bmaCox = CoxPHFitter.CoxPHFitter()
bmaCox.fit(df, "lifetime", event_col= "event", priors= [0.3, 0.9, 0.001, 0.3])
print(bmaCox.summary)


I am working on the file ".ipynb_checkpoints/Full done-checkpoint.ipynb".
The first task is:
Problems:
7 - Try testing whether the median is greater than a different values
Can you write Python code for it?

medians = [weibull_median(x[0], x[1]) for x in zip(mcmc.trace("alpha"), mcmc.trace("beta"))]
testing_value = 14.9
number_of_greater_samples = sum([x >= testing_value for x in medians])
100 * (number_of_greater_samples / len(medians))


I am working on the file ".ipynb_checkpoints/Full done-checkpoint.ipynb".
The first task is:
The first step in any data analysis is acquiring and munging the data
Our starting data set can be found here:
   http://jakecoltman.com in the pyData post
It is designed to be roughly similar to the output from DCM's path to conversion
Download the file and transform it into something with the columns:
id,lifetime,age,male,event,search,brand
where lifetime is the total time that we observed someone not convert for and event should be 1 if we see a conversion and 0 if we don't.  Note that all values should be converted into ints
It is useful to note that end_date = datetime.datetime(2016, 5, 3, 20, 36, 8, 92165)
Can you write Python code for it?

running_id = 0
output = [[0]]
with open("E:/output.txt") as file_open:
    for row in file_open.read().split("\n"):
        cols = row.split(",")
        if cols[0] == output[-1][0]:
            output[-1].append(cols[1])
            output[-1].append(True)
        else:
            output.append(cols)
    output = output[1:]

for row in output:
    if len(row) == 6:
        row += [datetime(2016, 5, 3, 20, 36, 8, 92165), False]
output = output[1:-1]

def convert_to_days(dt):
    day_diff = dt / np.timedelta64(1, 'D')
    if day_diff == 0:
        return 23.0
    else:
        return day_diff

df = pd.DataFrame(output, columns=["id", "advert_time", "male","age","search","brand","conversion_time","event"])
df["lifetime"] = pd.to_datetime(df["conversion_time"]) - pd.to_datetime(df["advert_time"])
df["lifetime"] = df["lifetime"].apply(convert_to_days)
df["male"] = df["male"].astype(int)
df["search"] = df["search"].astype(int)
df["brand"] = df["brand"].astype(int)
df["age"] = df["age"].astype(int)
df["event"] = df["event"].astype(int)
df = df.drop('advert_time', 1)
df = df.drop('conversion_time', 1)
df = df.set_index("id")
df = df.dropna(thresh=2)
df.median()

###Parametric Bayes
#Shout out to Cam Davidson-Pilon

## Example fully worked model using toy data
## Adapted from http://blog.yhat.com/posts/estimating-user-lifetimes-with-pymc.html
## Note that we've made some corrections

N = 2500

##Generate some random data
lifetime = pm.rweibull( 2, 5, size = N )
birth = pm.runiform(0, 10, N)
censor = ((birth + lifetime) >= 10)
lifetime_ = lifetime.copy()
lifetime_[censor] = 10 - birth[censor]

alpha = pm.Uniform('alpha', 0, 20)
beta = pm.Uniform('beta', 0, 20)

@pm.observed
def survival(value=lifetime_, alpha = alpha, beta = beta ):
    return sum( (1-censor)*(log( alpha/beta) + (alpha-1)*log(value/beta)) - (value/beta)**(alpha))

mcmc = pm.MCMC([alpha, beta, survival ] )
mcmc.sample(50000, 30000)

pm.Matplot.plot(mcmc)
mcmc.trace("alpha")[:]


I am working on the file ".ipynb_checkpoints/Full presentation-checkpoint.ipynb".
The first task is:
If we want to look at covariates, we need a new approach.  We'll use Cox proprtional hazards.  More information here.
Can you write Python code for it?

#Fitting solution
cf = lifelines.CoxPHFitter()
cf.fit(df, 'time_to_convert', event_col = 'event')
cf.summary


I am working on the file ".ipynb_checkpoints/Full presentation-checkpoint.ipynb".
The first task is:
Model selection
Difficult to do with classic tools (here)
Problem:
1 - Calculate the BMA coefficient values

2 - Compare these results to past the lifelines results

3 - Try running with different priors
Can you write Python code for it?

##Solution to 1
bmaCox = CoxPHFitter()
bmaCox.fit(df, "time_to_convert", event_col= "event", priors= [0.5]*7)

print(coefficient)

difference = coefficient - cf.summary["coef"]
print(difference)
print("Most very close, some significantly out (because no leaps and jumps)")


I am working on the file ".ipynb_checkpoints/Full presentation-checkpoint.ipynb".
The first task is:
Problems:
1 - Work out the mean observed time to convert
2 - Try to fit your data from section 1
3 - Use the results to plot the distribution of the median
4 - Try adjusting the number of samples, the burn parameter and the amount of thinning to correct get good answers
5 - Try adjusting the prior and see how it affects the estimate
6 - Try to fit a different distribution to the data
7 - Compare answers
Bonus - test the hypothesis that the true median is greater than a certain amount

For question 2, note that the median of a Weibull is:
$$β(log 2)^{1/α}$$
Can you write Python code for it?

#Solution to question 4:
def weibull_median(alpha, beta):
    return beta * ((log(2)) ** ( 1 / alpha))
plt.hist([weibull_median(x[0], x[1]) for x in zip(mcmc.trace("alpha"), mcmc.trace("beta"))])

#Solution to question 4:
### Increasing the burn parameter allows us to discard results before convergence
### Thinning the results removes autocorrelation

mcmc = pm.MCMC([alpha, beta, obs, censorfactor ] )
mcmc.sample(50000, burn = 30000, thin = 20)

pm.Matplot.plot(mcmc)

#Solution to Q5
## Adjusting the priors impacts the overall result
## If we give a looser, less informative prior then we end up with a broader, shorter distribution
## If we give much more informative priors, then we get a tighter, taller distribution

## Note the narrowing of the prior
#alpha = pm.Uniform("alpha", 2.5, 4.5)
#beta = pm.Uniform("beta", 14, 15)

####Uncomment this to see the result of looser priors
alpha = pm.Uniform("alpha", 0, 30)
beta = pm.Uniform("beta", 0, 30)

obs = pm.Weibull( 'obs', alpha, beta, value = df["time_to_convert"], observed = True )

mcmc = pm.MCMC([alpha, beta, obs, censorfactor ] )
mcmc.sample(10000, burn = 100, thin = 20)
pm.Matplot.plot(mcmc)
#plt.hist([weibull_median(x[0], x[1]) for x in zip(mcmc.trace("alpha"), mcmc.trace("beta"))])

#Solution to Q6
## To fit a new distribution to the data, we need to change the obs variable to a new distribution.
## This requires creating new hyper parameters,
## The normal distribution has two parameters, the mean and the stdev (we use 1/stdev in pymc)

import pymc as mc
import numpy.ma as ma

#this begins the model
alpha = pm.Uniform("mean", 0,15)
tau = pm.Uniform("tau", 0, 3)
obs = pm.Normal( 'obs', alpha, tau, value = df["time_to_convert"], observed = True )

mcmc = pm.MCMC([alpha, tau, obs, censorfactor ] )
mcmc.sample(50000, burn = 30000, thin = 20)
#pm.Matplot.plot(mcmc)
plt.hist(mcmc.trace("mean")[:])

## Solution to bonus
## Super easy to do in the Bayesian framework, all we need to do is look at what % of samples
## meet our criteria

testing_value =...


I am working on the file ".ipynb_checkpoints/GIM2DvsNSA-checkpoint.ipynb".
The first task is:
CONCLUSION Both bulge and disk 1/2 light radii are less than 1/2 light radius for entire galaxy.  This surprises me.  I would think that the disk is more extended and flatter, so the disk should have a larger half-light radius.  What am I missing?
Comparing Size Ratios (Re(24)/Re(r)) derived from GIM2D vs NSA
Can you write Python code for it?

plt.figure()
bins=np.arange(0,2.5,.1)
plt.hist(s.SIZE_RATIO_gim2d[s.sampleflag],histtype='step',hatch='///',color='r',bins=bins,label='GIM2D')
plt.hist(s.SIZE_RATIO_DISK[s.sampleflag],histtype='step',color='b',hatch='o',bins=bins,label='GIM2D Disk Only')
plt.hist(s.s.SIZE_RATIO[s.sampleflag],histtype='step',color='k',hatch='\\\\',bins=bins,label='NSA')
plt.legend(loc='upper right')
plt.xlabel('Normalized 24um Size')


I am working on the file ".ipynb_checkpoints/GIM2DvsNSA-checkpoint.ipynb".
The first task is:
Conclusion  two measures of radius are comparable, expect for the NSA galaxies with very large radii.  I think I cut these out of the sample.  If I used GIM2D fits, I could include them!  Right now I have:
self.sizeflag=(self.s.SERSIC_TH50*self.DA > minsize_kpc) & (self.s.SERSIC_TH50 < 20.)

Using GIM2D instead will add an additional 16 galaxies.
Comparing GIM2D bulge and disk fits to overall half-light radius
Can you write Python code for it?

plt.figure()
plt.plot(s.s.Rhlr_1[s.gim2dflag],s.s.Rd[s.gim2dflag],'bo',label='Disk')
plt.plot(s.s.Rhlr_1[s.gim2dflag],s.s.Re[s.gim2dflag],'ro',label='Bulge')

xl=np.linspace(0,20,2)
plt.plot(xl,xl,'k--',label='1:1')
plt.ylabel('GIM2D half light radius')
plt.xlabel('GIM2D galaxy half light radius')
plt.legend(numpoints=1,loc='upper left')


I am working on the file ".ipynb_checkpoints/GIM2DvsNSA-checkpoint.ipynb".
The first task is:
Plotting 1/2 light radius from GIM2D vs NSA
Can you write Python code for it?

plt.figure()
plt.plot(s.s.SERSIC_TH50[s.gim2dflag]*s.DA[s.gim2dflag],s.s.Rhlr_1[s.gim2dflag],'ko')
#plt.plot(s.s.SERSIC_TH50[s.gim2dflag]*s.DA[s.gim2dflag],s.s.Rd[s.gim2dflag],'bo')
xl=np.linspace(0,20,2)
plt.plot(xl,xl,'r-')
plt.xlabel('NSA SERSIC_TH50*DA')
plt.ylabel('GIM2D half light radius')


I am working on the file ".ipynb_checkpoints/GetsDrawnDotCom-checkpoint.ipynb".
The first task is:
I have it creating a meta folder and creating/writing username.meta files. It wrote 'test' in each folder, but now it writes the photo author title of post.. the username/image data. It should be writing more than author title - maybe upvotes/downvotes, subreddit, time published etc.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/GetsDrawnDotCom-checkpoint.ipynb".
The first task is:
Need to fix dir to just have /imgs/15/02/reference/imgnam-reference.jpg
Can you write Python code for it?

monzpath

iwcpath = 'imgs/' + yearz + '/' + monthz + '/' + dayz + '/reference'

#monzpath = (yrzpat + '/' + monthz)
#dayzpath = (monzpath + '/' + dayz)
#rmgzdays = (dayzpath + '/reference')

iwcpath

iwcpath

#for liz in lizmon:
#    if os.path.isdir(liz) == True:
##        print 'its true'
 #   else:
#        print 'its false'
#        os.mkdir(liz)

fullhom = ('/home/wcmckee/getsdrawndotcom/')

#artlist

httpad = ('http://getsdrawn.com/imgs')

#im = Image.new("RGB", (512, 512), "white")
#im.save(file + ".thumbnail", "JPEG")

rmgzdays = (dayzpath + '/reference')
imgzdays = (dayzpath + '/art')
metzdays = (dayzpath + '/meta')

os.chdir(metzdays)

metadict = dict()


I am working on the file ".ipynb_checkpoints/GetsDrawnDotCom-checkpoint.ipynb".
The first task is:
Need to save json object.
Dict is created but it isnt saving. Looping through lisrgc twice, should only require the one loop.
Cycle through lisr and append to dict/concert to json, and also cycle through lisr.author meta folders saving the json that was created.
Can you write Python code for it?

for lisr in lisrgc:
    gtdrndic.update({'title': lisr.title})
    lisauth.append(str(lisr.author))
    for osliz in os.listdir(metzdays):
        with open(str(lisr.author) + '.meta', "w") as f:
            rstrin = lisr.title.encode('ascii', 'ignore').decode('ascii')
            #print matdict
            #metadict = dict()
            #for lisz in lisrgc:
            #    metadict.update({'up': lisz.ups})
            #    metadict.update({'down': lisz.downs})
            #    metadict.update({'title': lisz.title})
            #    metadict.update({'created': lisz.created})
            f.write(rstrin)

#matdict


I am working on the file ".ipynb_checkpoints/GetsDrawnDotCom-checkpoint.ipynb".
The first task is:
Something is wrong with the script and it's no longer creating these dir in the correct folder. How did this break?
Fixed that but problems with it
Getting error:
OSError: [Errno 17] File exists: '/home/wcmckee/getsdrawndotcom/imgs/15/01'
If the file exists it should be skipping over it, thats why it has the os.path.isdir == True:
 print its true
else
 print its false, and make the dir
Can you write Python code for it?

if os.path.isdir(monzpath) == True:
    print 'its true'
else:
    print 'its false'
    #os.mkdir('/home/wcmckee/getsdrawndotcom/' + monzpath)

if os.path.isdir(dayzpath) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(dayzpath)

if os.path.isdir(imgzdays) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(imgzdays)

if os.path.isdir(rmgzdays) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(rmgzdays)

if os.path.isdir(metzdays) == True:
    print 'its true'
else:
    print 'its false'
    os.mkdir(metzdays)


I am working on the file ".ipynb_checkpoints/GetsDrawnDotCom-checkpoint.ipynb".
The first task is:
if i save the data to the file how am i going to get it to update as the post is archieved. Such as up and down votes.
Can you write Python code for it?

for lisz in lisrgc:
    metadict.update({'up': lisz.ups})
    metadict.update({'down': lisz.downs})
    metadict.update({'title': lisz.title})
    metadict.update({'created': lisz.created})
    #metadict.update({'createdutc': lisz.created_utc})
    #print lisz.ups
    #print lisz.downs
    #print lisz.created
    #print lisz.comments

metadict


I am working on the file ".ipynb_checkpoints/Getting simple demo data-checkpoint.ipynb".
The first task is:
Looks like there are plenty of options for these.  For demo purposes I want to pick 12 of each with a reasonable range of variation on the key factors of sugar, fat, sodium, protein, so that I can have one plus up to 11 comparison products.
Can you write Python code for it?

# reminder on column names remaining
df.columns


I am working on the file ".ipynb_checkpoints/Getting simple demo data-checkpoint.ipynb".
The first task is:
Now down to a manageable number of rows and columns.  Going to explore for a few typical items to use as demo data.  Let's take a look at donuts, crackers and cereal -- the three categories used in the paper prototype.
Can you write Python code for it?

df[df["product_name"].str.lower().str.contains("donut", na = False)]

df[df["product_name"].str.lower().str.contains("cracker", na = False)]

df[df["product_name"].str.lower().str.contains("cereal", na = False)]


I am working on the file ".ipynb_checkpoints/Getting_Started-checkpoint.ipynb".
The first task is:
Better, but not good. The overridden repr implementation is supposed to be evalable. Note that we need to import datetime and call repr explicitly. Let's try:
Can you write Python code for it?

import datetime
pp.pprint([(eval(repr(i)), eval(repr(i.build))) for i in rs.build_execs])


I am working on the file ".ipynb_checkpoints/Getting_Started-checkpoint.ipynb".
The first task is:
Much better. We can see the 0hash and icepole implementations succeeded but the 0hash implementation failed. The log is mostly empty since we've rebuilt and thus there's not much makefile output.
We want to know why the aesgcm implementations failed. To do this, we must examine the runs relation.
Can you write Python code for it?

import datetime
for i in rs.build_execs:
    if not i.test_ok:
        for j in i.runs:
            print(j)


I am working on the file ".ipynb_checkpoints/Getting_Started-checkpoint.ipynb".
The first task is:
Not much information here. Let's print out information on the builds associated with the build executions:
Can you write Python code for it?

print([(i, i.build) for i in rs.build_execs])


I am working on the file ".ipynb_checkpoints/Getting_Started-checkpoint.ipynb".
The first task is:
Not very readable. Let's use prettyprint:
Can you write Python code for it?

import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint([(i, i.build) for i in rs.build_execs])


I am working on the file ".ipynb_checkpoints/Getting_Started-checkpoint.ipynb".
The first task is:
Print latest RunSession
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Getting_Started-checkpoint.ipynb".
The first task is:
Too much stuff. Let's clean it up:
Can you write Python code for it?

import datetime
for i in rs.build_execs:
    if not i.test_ok:
        for j in i.runs:
            print('{}: {}'.format(i.build.buildid,j.checksumfail_cause))


I am working on the file ".ipynb_checkpoints/Getting_Started-checkpoint.ipynb".
The first task is:
We have overridden the __repr__ function in the base class for SqlAlchemy tables to print out the type, a dictionary of contents, and a list of relations.
Let's print out all build executions:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Gradients-checkpoint.ipynb".
The first task is:
Check all files to see if any have missing nodal information and create a selection list based on the ones that are 100% complete.
Can you write Python code for it?

# check to see which files contains nodes with missing information
missingarray = []
for i in onlyfiles:
# load timeseries
    filename = i
    ts_raw = np.loadtxt(filename)

# check zero columns
    missingn = np.where(~ts_raw.any(axis=0))[0]
    missingarray.append(missingn)

# select the ones that don't have missing data
ids = np.where([len(i) == 0 for i in missingarray])[0]
selected = [onlyfiles[i] for i in ids]
# could be useful to have one without pathnames later one
selected2 = [os.path.basename(onlyfiles[i]) for i in ids]
print(len(selected))


I am working on the file ".ipynb_checkpoints/Gradients-checkpoint.ipynb".
The first task is:
Compare the slopes across subjects
Can you write Python code for it?

from scipy import stats
grdnt_slope = []
for i in selected2:
    # load gradients
    # print i
    filename = i
    grdnt = np.load("./data/Outputs/Regs/" + filename + ".npy")
    # do we need a specific ordering of the nodes??
    y = list(xrange(392))
    temp = []
    for ii in range(10):
        x = sorted(grdnt[:,ii]) # just sort in ascending order?
        slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
        temp.append(slope)

    grdnt_slope.append(temp)
grdnt_slope = np.array(grdnt_slope)
# make it into a dataframe
data_grdnt = pd.DataFrame(grdnt_slope)
data_grdnt['file'] = selected2


I am working on the file ".ipynb_checkpoints/Gradients-checkpoint.ipynb".
The first task is:
Permutations
Can you write Python code for it?

def exact_mc_perm_test(xs, ys, nmc):
    n, k = len(xs), 0
    diff = np.abs(np.mean(xs) - np.mean(ys))
    zs = np.concatenate([xs, ys])
    for j in range(nmc):
        np.random.shuffle(zs)
        k += diff < np.abs(np.mean(zs[:n]) - np.mean(zs[n:]))
    return k / nmc

print(exact_mc_perm_test(G1[0],G2[0],1000))
print(exact_mc_perm_test(G1[1],G2[1],1000))


I am working on the file ".ipynb_checkpoints/Gradients-checkpoint.ipynb".
The first task is:
Run Statis to back-project the grouped embeddings
Can you write Python code for it?

%%capture
from pySTATIS import statis

#load vectors
names = list(xrange(392))
X = [np.load("./data/Outputs/Embs/"+ os.path.basename(filename)+"_embedding_dense_res_veconly.npy") for filename in selected2]
out = statis.statis(X, names, fname='statis_results.npy')
statis.project_back(X, out['Q'], path = "./data/Outputs/Regs/",fnames = selected2)
np.save("Mean_Vec.npy",out['F'])

# saving everything in one dump
import pickle
with open('output.pickle' ,'w') as f:
    pickle.dump([selected, out],f)


I am working on the file ".ipynb_checkpoints/Gradients-checkpoint.ipynb".
The first task is:
plotting
plot to surface for inspection
this cell in only necessary for plotting below
Can you write Python code for it?

%matplotlib inline
import matplotlib.pylab as plt
import nilearn
import nilearn.plotting

import numpy as np
import nibabel as nib

def rebuild_nii(num):

    data = np.load('Mean_Vec.npy')
    a = data[:,num].copy()
    nim = nib.load('cc400_roi_atlas.nii')
    imdat=nim.get_data()
    imdat_new = imdat.copy()

    for n, i in enumerate(np.unique(imdat)):
        if i != 0:
            imdat_new[imdat == i] = a[n-1] * 10000 # scaling factor. Could also try to get float values in nifti...

    nim_out = nib.Nifti1Image(imdat_new, nim.get_affine(), nim.get_header())
    nim_out.set_data_dtype('float32')
    # to save:
    nim_out.to_filename('Gradient_'+ str(num) +'_res.nii')

    nilearn.plotting.plot_epi(nim_out)
    return(nim_out)

for i in range(10):
    nims = rebuild_nii(i)


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
1.10 Instantiate client and service shapefiles
Can you write Python code for it?

# Client
client = shp.Writer(shp.POINT)
# Add Random Points
for i,j in rand_coords_client:
    client.point(i,j)
# Add Fields
client.field('client_ID')
client.field('Weight')
client.field('LAT')
client.field('LON')
counter = 0
for i in range(len(rand_coords_client)):
    counter = counter + 1
    client.record('client_' + str(counter), Ai[i], lat_client[i], lon_client[i])
# Save Shapefile
client.save('shapefiles/RandomPoints_CLIENT')

#Service
service = shp.Writer(shp.POINT)
# Add Random Points
for i,j in rand_coords_service:
    service.point(i,j)
# Add Fields
service.field('y_ID')
service.field('x_ID')
service.field('LAT')
service.field('LON')
counter = 0
for i in range(len(rand_coords_service)):
    counter = counter + 1
    service.record('y' + str(counter), 'x' + str(counter), lat_service[i], lon_service[i])
# Save Shapefile
service.save('shapefiles/RandomPoints_SERVICE')


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
1.13 Create distance matrix
Can you write Python code for it?

t1 = time.time()
All_Neigh_Dist = ntw.allneighbordistances(sourcepattern=ntw.pointpatterns['Rand_Points_CLIENT'],
                                             destpattern=ntw.pointpatterns['Rand_Points_SERVICE'])
All_Dist_MILES = All_Neigh_Dist * float(10000/90) * 0.6214
seconds = round(time.time()-t1, 4)
print seconds, 'seconds'
print 'Client [i] x Service [j] Matrix Shape --> ', All_Dist_MILES.shape


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
1.3 Create Bounding Box from 'Waverly.shp'
Can you write Python code for it?

shp_W = ps.open('Waverly/Waverly.shp')
shp_W.bbox


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
1.5 Zip the latitude and longitude lists together
Can you write Python code for it?

rand_coords_client = map(list, zip(lat_client, lon_client))
rand_coords_service = map(list, zip(lat_service, lon_service))


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
1.7 Fill dictionaries of random roints
Can you write Python code for it?

# CLIENT
for idx, coords in enumerate(rand_coords_client):
    GRAPH_client.add_node(idx)
    points_client[idx] = coords
    GRAPH_client.node[idx] = coords
# SERVICE
for idx, coords in enumerate(rand_coords_service):
    GRAPH_service.add_node(idx)
    points_service[idx] = coords
    GRAPH_service.node[idx] = coords


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
1.8 Draw network, simplified network, and random client & service nodes
Can you write Python code for it?

print dt.datetime.now()
#Instantiate Figure
figsize(10,10)
# Draw Graph of Actual Nodes and Roads
for e in ntw.edges:
    g.add_edge(*e)
nx.draw(g, ntw.node_coords, node_size=5, alpha=0.25, edge_color='r', width=2)
# Draw only unique edges in graph
for e in ntw.graphedges:
    g1.add_edge(*e)
    # highlights cases where start and end node are the same
    if e[0]==e[1]:
        g1.add_node(e[0])
for node_id in g1.node:
    g1.node[node_id] = ntw.node_coords[node_id]
nx.draw(g1, ntw.node_coords, node_size=10, alpha=0.5)
# Draw Graph of Random Client Points
nx.draw(GRAPH_client, points_client,
    node_size=75, alpha=1, node_color='b')
# Draw Graph of Random Service Points
nx.draw(GRAPH_service, points_service,
    node_size=100, alpha=1, node_color='c')
# Legend (Ordered Dictionary)
LEGEND = OrderedDict()
LEGEND['Network Nodes']=g
LEGEND['Roads']=g
LEGEND['Graph Vertices']=g1
LEGEND['Graph Edges']=g1
LEGEND['Client Nodes']=GRAPH_client
LEGEND['Service Nodes']=GRAPH_service
legend(LEGEND, loc='lower right', fancybox=True, framealpha=0.5)

# Title
title('Waverly Hills\n Tallahassee, Florida', family='Times New Roman',
      size=40, color='k', backgroundcolor='w', weight='bold')
# North Arrow and 'N' --> Must be changed for different spatial resolutions, etc.
arrow(-84.281, 30.507, 0.0, 0.01, width=.0003, head_width=0.0012,
          head_length=0.002, fc='k', ec='k',alpha=0.75,)
annotate('N', xy=(-84.2815, 30.52), fontstyle='italic', fontsize='xx-large',
            fontweight='heavy', alpha=0.75)


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
1.9 Create weights at nodes and sum
Can you write Python code for it?

# Client Weights
Ai = np.random.randint(5, 50, len(rand_coords_client))
Ai = Ai.reshape(len(Ai),1)
# Sum
AiSum = np.sum(Ai)


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
2.2a  Cplex p-Median test [p = 2]
Can you write Python code for it?

t1 = time.time()

# Demand
Ai = Ai
# Demand Sum
AiSum = AiSum
# Travel Costs
Cij = All_Dist_MILES
# Weighted Costs
Sij = Ai * Cij
client_nodes = range(len(Sij))
service_nodes = range(len(Sij[0]))

all_nodes = len(Sij) * len(Sij[0])
ALL_nodes = range(all_nodes)

x = 'x'
cli_var = []
for i in client_nodes:
    for j in service_nodes:
        temp = x + str(i+1) + '_' + str(j+1)
        cli_var.append(temp)
client_var = np.array(cli_var)
client_var = client_var.reshape(len(Sij),len(Sij[0]))

y = 'y'
fac_var = []
for i in service_nodes:
    temp = y + str(i+1)
    fac_var.append(temp)
facility_var = np.array(fac_var)
facility_var = facility_var.reshape(1,len(Sij[0]))

#     2. Create Model and Add Variables
# Create Model
mPMP_CPLEX = cp.Cplex()
# Problem Name
mPMP_CPLEX.set_problem_name('\n -- P-Median -- ')
print mPMP_CPLEX.get_problem_name()

# Problem Type  ==>  Linear Programming
mPMP_CPLEX.set_problem_type(mPMP_CPLEX.problem_type.LP)
# Set MIP Emphasis to '2' --> Optimal
mPMP_CPLEX.parameters.emphasis.mip.set(2)
print mPMP_CPLEX.parameters.get_changed()
print '\nProblem Type\n    ' + str(mPMP_CPLEX.problem_type[mPMP_CPLEX.get_problem_type()])
# Objective Function Sense  ==>  Minimize
mPMP_CPLEX.objective.set_sense(mPMP_CPLEX.objective.sense.minimize)
print 'Objective Sense\n    ' + str(mPMP_CPLEX.objective.sense[mPMP_CPLEX.objective.get_sense()])
# Add Client Decision Variables
mPMP_CPLEX.variables.add(names = [cli_var[i] for i in ALL_nodes],
                        obj = [Sij[i][j] for i in client_nodes for j in service_nodes],
                        lb = [0] * all_nodes,
                        ub = [1] * all_nodes,
                        types = ['B'] * all_nodes)
# Add Service Decision Variable
mPMP_CPLEX.variables.add(names = [fac_var[j] for j in service_nodes],
                        lb = [0] * len(Sij[0]),
                        ub = [1] * len(Sij[0]),
                        types = ['B'] * len(Sij[0]))

#    3. Add Constraints
# Add Assignment Constraints
for orig in client_nodes:
 ...


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
1. Data preparation and creation
1.1 Instantiate a network
Can you write Python code for it?

ntw = ps.Network('Waverly/Waverly.shp')
print dt.datetime.now()
print 'PySAL.Network\n'
print dir(ntw)


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
2. Mathematical Optimization
2.1a  Gurobi p-Median test [p = 2]
Can you write Python code for it?

t1 = time.time()

#     1. Data
# Demand
Ai = Ai
# Demand Sum
AiSum = AiSum
# Travel Costs
Cij = All_Dist_MILES
# Weighted Costs
Sij = Ai * Cij
client_nodes = range(len(Sij))
service_nodes = range(len(Sij[0]))

mPMP_GUROBI = gbp.Model(' -- p-Median -- ')

gbp.setParam('MIPFocus', 2)

# Client IxJ
client_var = []
for orig in client_nodes:
    client_var.append([])
    for dest in service_nodes:
        client_var[orig].append(mPMP_GUROBI.addVar(vtype=gbp.GRB.BINARY,
                                            obj=Sij[orig][dest],
                                            name='x'+str(orig+1)+'_'+str(dest+1)))
#J
serv_var = []
for dest in service_nodes:
    serv_var.append([])
    serv_var[dest].append(mPMP_GUROBI.addVar(vtype=gbp.GRB.BINARY,
                                    name='y'+str(dest+1)))
mPMP_GUROBI.update()
mPMP_GUROBI.setObjective(gbp.quicksum(Sij[orig][dest]*client_var[orig][dest]
                        for orig in client_nodes for dest in service_nodes),
                        gbp.GRB.MINIMIZE)
for orig in client_nodes:
    mPMP_GUROBI.addConstr(gbp.quicksum(client_var[orig][dest]
                        for dest in service_nodes) == 1)
for orig in service_nodes:
    for dest in client_nodes:
        mPMP_GUROBI.addConstr((serv_var[orig] - client_var[dest][orig] >= 0))
mPMP_GUROBI.addConstr(gbp.quicksum(serv_var[dest][0] for dest in service_nodes) == 2)
mPMP_GUROBI.optimize()
t2P = time.time()-t1
mPMP_GUROBI.write('LP_Files/WaverlyPMP_GUROBI.lp')
print '\n*************************************************************************'
selected = []
dbf1 = ps.open('shapefiles/RandomPoints_SERVICE.dbf')
NEW_Records_PMP_GUROBI = []
for v in mPMP_GUROBI.getVars():
    if 'x' in v.VarName:
        pass
    elif v.x > 0:
        var = '%s' % v.VarName
        selected.append(var)
        for i in range(dbf1.n_records):
            if var in dbf1.read_record(i):
                x = dbf1.read_record(i)
                NEW_Records_PMP_GUROBI.append(x)
            else:
                pass
       ...


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Pmedian-checkpoint.ipynb".
The first task is:
3. Selected locations
3.1 Gurobi & Cplex p-Median Selected locations
Can you write Python code for it?

figsize(10,10)
# Draw Network Actual Roads and Nodes
nx.draw(g, ntw.node_coords, node_size=5, alpha=0.25, edge_color='r', width=2)
# Draw Graph
nx.draw(g1, ntw.node_coords, node_size=10, alpha=0.5)

# Gurobi p-Median
P_Med_GUROBI = ps.open('shapefiles/Selected_Locations_Pmedian_GUROBI.shp')
points_median_GUROBI = {}
for idx, coords in enumerate(P_Med_GUROBI):
    GUROBI_median_g.add_node(idx)
    points_median_GUROBI[idx] = coords
    GUROBI_median_g.node[idx] = coords
nx.draw(GUROBI_median_g, points_median_GUROBI,
        node_size=600, alpha=1, node_color='g')

# Cplex p-Median
P_Med_CPLEX = ps.open('shapefiles/Selected_Locations_Pmedian_CPLEX.shp')
points_median_CPLEX = {}
for idx, coords in enumerate(P_Med_CPLEX):
    CPLEX_median_g.add_node(idx)
    points_median_CPLEX[idx] = coords
    CPLEX_median_g.node[idx] = coords
nx.draw(CPLEX_median_g, points_median_CPLEX,
        node_size=300, alpha=1, node_color='r')

# Draw Graph of Random Service
nx.draw(GRAPH_client, points_client,
        node_size=15, alpha=.5, node_color='k')

# Draw Graph of Random Service
nx.draw(GRAPH_service, points_service,
        node_size=50, alpha=1, node_color='k')

# Legend (Ordered Dictionary)
LEGEND = OrderedDict()
LEGEND['Network Nodes']=g
LEGEND['Roads']=g
LEGEND['Graph Vertices']=g1
LEGEND['Graph Edges']=g1
LEGEND['Gurobi Optimal p-Median (p=2)']=GUROBI_median_g
LEGEND['Cplex Optimal p-Median (p=2)']=CPLEX_median_g
LEGEND['Client Nodes']=GRAPH_client
LEGEND['Service Nodes']=GRAPH_service
legend(LEGEND, loc='lower right', fancybox=True, framealpha=0.5)

# Title
title('Waverly Hills\n Tallahassee, Florida', family='Times New Roman',
      size=40, color='k', backgroundcolor='w', weight='bold')
# North Arrow and 'N' --> Must be changed for different spatial resolutions, etc.
arrow(-84.281, 30.507, 0.0, 0.01, width=.0003, head_width=0.0012,
          head_length=0.002, fc='k', ec='k',alpha=0.75,)
annotate('N', xy=(-84.2815, 30.52), fontstyle='italic', fontsize='xx-large',
            fontweight='heavy', alpha=0.75)


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
1.10 Instantiate client and service shapefiles
Can you write Python code for it?

# Client
client = shp.Writer(shp.POINT)
# Add Random Points
for i,j in rand_coords_client:
    client.point(i,j)
# Add Fields
client.field('client_ID')
#client.field('Weight')
client.field('LAT')
client.field('LON')
counter = 0
for i in range(len(rand_coords_client)):
    counter = counter + 1
    client.record('client_' + str(counter), lat_client[i], lon_client[i])
# Save Shapefile
client.save('shapefiles/RandomPoints_CLIENT')

#Service
service = shp.Writer(shp.POINT)
# Add Random Points
for i,j in rand_coords_service:
    service.point(i,j)
# Add Fields
service.field('y_ID')
service.field('x_ID')
service.field('LAT')
service.field('LON')
counter = 0
for i in range(len(rand_coords_service)):
    counter = counter + 1
    service.record('y' + str(counter), 'x' + str(counter), lat_service[i], lon_service[i])
# Save Shapefile
service.save('shapefiles/RandomPoints_SERVICE')


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
1.11 Snap Observations to NTW
Can you write Python code for it?

t1 = time.time()
ntw.snapobservations('shapefiles/RandomPoints_CLIENT.shp',
                     'Rand_Points_CLIENT', attribute=True)
ntw.snapobservations('shapefiles/RandomPoints_SERVICE.shp',
                     'Rand_Points_SERVICE', attribute=True)
print round(time.time()-t1, 4), 'seconds'


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
1.4 Create numpy arrays of random floats within a  bounding box
Can you write Python code for it?

lat_client = np.random.uniform(shp_W.bbox[0], shp_W.bbox[2], 500)
lon_client = np.random.uniform(shp_W.bbox[1], shp_W.bbox[3], 500)
lat_service = np.random.uniform(shp_W.bbox[0], shp_W.bbox[2], 500)
lon_service = np.random.uniform(shp_W.bbox[1], shp_W.bbox[3], 500)


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
1.5 Zip the latitude and longitude lists together
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
1.7 Fill dictionaries of random roints
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
1.9 Create S
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
2.2a  Cplex Set Cover test
Can you write Python code for it?

t1 = time.time()

#     1. Read In Data
# Cost Vector
Cij = All_Dist_MILES
# Create Aij: Determine Aij (nodes within S)
# S --> 1 = served; 0 = unserved
Cij = All_Dist_MILES
# Create Aij: Determine Aij (nodes within S)
# S --> 1 = served; 0 = unserved
Aij = []
for i in np.nditer(Cij):
    if i <= S:
        outtext = 1
    else:
        outtext = 0
    Aij.append(outtext)
Aij = np.array(Aij)
Aij = Aij.reshape(len(Cij),len(Cij[0]))

# Indices & Variable Names
nodes = len(Cij)
Nodes = range(len(Cij))

all_nodes = len(Cij) * len(Cij)
ALL_nodes = range(all_nodes)
x = 'x'
cli_var = []
for i in Nodes:
    for j in Nodes:
        temp = x + str(j+1)
        cli_var.append(temp)
client_var = np.array(cli_var)
results_var = []
for i in Nodes:
    temp = x + str(i+1)
    results_var.append(temp)

#     2. Create Model and Add Variables
# Create Model
mSCLP_CPLEX = cp.Cplex()
# Problem Name
mSCLP_CPLEX.set_problem_name('\n -- Set Cover Location Problem -- ')
print mSCLP_CPLEX.get_problem_name()
# Problem Type  ==>  Linear Programming
mSCLP_CPLEX.set_problem_type(mSCLP_CPLEX.problem_type.LP)
# Set MIP Emphasis to '2' --> Optimal
mSCLP_CPLEX.parameters.emphasis.mip.set(2)
print mSCLP_CPLEX.parameters.get_changed()
print '\nProblem Type\n    ' + str(mSCLP_CPLEX.problem_type[mSCLP_CPLEX.get_problem_type()])
# Objective Function Sense  ==>  Minimize
mSCLP_CPLEX.objective.set_sense(mSCLP_CPLEX.objective.sense.minimize)
print 'Objective Sense\n    ' + str(mSCLP_CPLEX.objective.sense[mSCLP_CPLEX.objective.get_sense()])
# Add Client Decision Variables
mSCLP_CPLEX.variables.add(names = [cli_var[i] for i in Nodes],
                        obj = [1] * nodes,
                        lb = [0] * nodes,
                        ub = [1] * nodes,
                        types = ['B'] * nodes)

#    3. Add Constraints
#Add Coverage Constraints
for orig in Nodes:
    coverage_constraints = cp.SparsePair(ind = [client_var[dest]
                                            for dest in Nodes],
          ...


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
2.2b Instantiate Selected Cplex Set Cover shapefile
Can you write Python code for it?

SHP_SCLP_CPLEX = shp.Writer(shp.POINT)
# Add Points
for idy,idx,x,y in NEW_Records_SCLP_CPLEX:
    SHP_SCLP_CPLEX.point(float(x), float(y))
# Add Fields
SHP_SCLP_CPLEX.field('y_ID')
SHP_SCLP_CPLEX.field('x_ID')
SHP_SCLP_CPLEX.field('LAT')
SHP_SCLP_CPLEX.field('LON')
# Add Records
for idy,idx,x,y in NEW_Records_SCLP_CPLEX:
    SHP_SCLP_CPLEX.record(idy,idx,x,y)
# Save Shapefile
SHP_SCLP_CPLEX.save('shapefiles/Selected_Locations_SCLP_CPLEX')


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
3.3 System Information
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
1. Data preparation and creation
1.1 Instantiate a network
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
2. Mathematical Optimization
2.1a  Gurobi Set Cover test
Can you write Python code for it?

t1 = time.time()

#     1. Read In Data
# Cost Vector
Cij = All_Dist_MILES
# Create Aij: Determine Aij (nodes within S)
# S --> 1 = served; 0 = unserved
Aij = []
for i in np.nditer(Cij):
    if i <= S:
        outtext = 1
    else:
        outtext = 0
    Aij.append(outtext)
rows, cols = Cij.shape
Aij = np.array(Aij)
Aij = Aij.reshape(len(Cij),len(Cij[0]))
client_nodes = range(len(Cij))
service_nodes = range(len(Cij[0]))

#     2. Create Model, Set MIP Focus, Add Variables, & Update Model
mSCLP_GUROBI = gbp.Model(" -- SCLP -- ")
# Set MIP Focus to 2 for optimality
gbp.setParam('MIPFocus', 2)

#     3. Add Service Decision Variables
serv_var = []
for dest in service_nodes:
    serv_var.append(mSCLP_GUROBI.addVar(vtype=gbp.GRB.BINARY,
                                ub = 1,
                                name='x'+str(dest+1)))
# Update Model Variables
mSCLP_GUROBI.update()

#     4. Set Objective Function
mSCLP_GUROBI.setObjective(gbp.quicksum(serv_var[dest]
                            for dest in service_nodes),
                            gbp.GRB.MINIMIZE)

#    5. Add Constraints
#Add Coverage Constraints
for orig in client_nodes:
    mSCLP_GUROBI.addConstr(gbp.quicksum(Aij[orig][dest]*serv_var[dest]
                            for dest in service_nodes) >= 1)

#     6. Optimize and Print Results
mSCLP_GUROBI.optimize()
mSCLP_GUROBI.write('GUROBI_LP.lp')
t2G = time.time()-t1
print '\n*****************************************************************************************'
selected = []
dbf1 = ps.open('shapefiles/RandomPoints_SERVICE.dbf')
NEW_Records_SCLP_GUROBI = []
for v in mSCLP_GUROBI.getVars():
    if v.x > 0:
        var = '%s' % v.VarName
        selected.append(v.x)
        for i in range(dbf1.n_records):
            if var in dbf1.read_record(i):
                x = dbf1.read_record(i)
                NEW_Records_SCLP_GUROBI.append(x)
            else:
                pass
        print '    |                                                         ', var
print '    | Selected...


I am working on the file ".ipynb_checkpoints/Gurobi_v_Cplex__Set_Cover-checkpoint.ipynb".
The first task is:
3. Selected locations
3.1 Gurobi & Cplex Set Cover Selected locations
Can you write Python code for it?

figsize(10,10)
# Draw Network Actual Roads and Nodes
nx.draw(g, ntw.node_coords, node_size=5, alpha=0.25, edge_color='r', width=2)
# Draw Graph
nx.draw(g1, ntw.node_coords, node_size=10, alpha=0.5)

# Gurobi Set Cover
SetCover_GUROBI = ps.open('shapefiles/Selected_Locations_SCLP_GUROBI.shp')
points_SetCover_GUROBI = {}
for idx, coords in enumerate(SetCover_GUROBI):
    GUROBI_setcover_g.add_node(idx)
    points_SetCover_GUROBI[idx] = coords
    GUROBI_setcover_g.node[idx] = coords
nx.draw(GUROBI_setcover_g, points_SetCover_GUROBI,
        node_size=600, alpha=1, node_color='g')

# Cplex Set Cover
SetCover_CPLEX = ps.open('shapefiles/Selected_Locations_SCLP_CPLEX.shp')
points_SetCover_CPLEX = {}
for idx, coords in enumerate(SetCover_CPLEX):
    CPLEX_setcover_g.add_node(idx)
    points_SetCover_CPLEX[idx] = coords
    CPLEX_setcover_g.node[idx] = coords
nx.draw(CPLEX_setcover_g, points_SetCover_CPLEX,
        node_size=300, alpha=1, node_color='r')

# Draw Graph of Random Service
nx.draw(GRAPH_client, points_client,
        node_size=15, alpha=.5, node_color='k')

# Draw Graph of Random Service
nx.draw(GRAPH_service, points_service,
        node_size=50, alpha=1, node_color='k')

# Legend (Ordered Dictionary)
LEGEND = OrderedDict()
LEGEND['Network Nodes']=g
LEGEND['Roads']=g
LEGEND['Graph Vertices']=g1
LEGEND['Graph Edges']=g1
LEGEND['Gurobi Optimal Set Cover']=GUROBI_setcover_g
LEGEND['Cplex Optimal Set Cover']=CPLEX_setcover_g
LEGEND['Client Nodes']=GRAPH_client
LEGEND['Service Nodes']=GRAPH_service
legend(LEGEND, loc='lower right', fancybox=True, framealpha=0.5)

# Title
title('Waverly Hills\n Tallahassee, Florida', family='Times New Roman',
      size=40, color='k', backgroundcolor='w', weight='bold')
# North Arrow and 'N' --> Must be changed for different spatial resolutions, etc.
arrow(-84.281, 30.507, 0.0, 0.01, width=.0003, head_width=0.0012,
          head_length=0.002, fc='k', ec='k',alpha=0.75,)
annotate('N', xy=(-84.2815, 30.52), fontstyle='italic', fontsize='xx-large',
           ...


I am working on the file ".ipynb_checkpoints/Hamiltonian MCMC (HMC)-checkpoint.ipynb".
The first task is:
Banana-shaped target distribution
Can you write Python code for it?

dtarget = lambda x: exp( (-x[0]**2)/200. - 0.5*(x[1]+(0.05*x[0]**2) - 100.*0.05)**2)

x1 = np.linspace(-20, 20, 101)
x2 = np.linspace(-15, 10, 101)
X, Y = np.meshgrid(x1, x2)
Z = np.array(map(dtarget, zip(X.flat, Y.flat))).reshape(101, 101)

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)
plt.show()

start = np.array([[uniform.rvs(loc=-10., scale=15.),
                   uniform.rvs(loc=0., scale=10)]
                  for i in xrange(4)])
chains = HMC(dtarget, start, Eps=0.7, L=12, m=2, N=10000)

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)

plt.plot(chains[0][:, 0], chains[0][:, 1], alpha=0.5)
plt.plot(chains[1][:, 0], chains[1][:, 1], alpha=0.5)
plt.plot(chains[2][:, 0], chains[2][:, 1], alpha=0.5)
plt.plot(chains[3][:, 0], chains[3][:, 1], alpha=0.5)
plt.show()

plt.subplot(211)
plt.title(Gelman(chains)[0])
for i in xrange(chains.shape[0]):
    plt.plot(chains[i,:,0])
plt.ylabel('x1')

plt.subplot(212)
for i in xrange(chains.shape[0]):
    plt.plot(chains[i,:,1])
plt.ylabel('x2')

plt.tight_layout()
plt.show()


I am working on the file ".ipynb_checkpoints/Hamiltonian MCMC (HMC)-checkpoint.ipynb".
The first task is:
NUTS Sampler
Toy implementation of No-U-Turn Sampler, described by Hoffman and Gelman (2011). Algorithm 3, page 14.
Can you write Python code for it?

def Leapfrog(U, theta, r, Eps, m=1.):
    """
    Slightly different update rules, since the negative log of the
    target PDF is not used.
    """
    gradient = lambda U, theta: derivative(U, theta, 0.01)
    r += (Eps/2.)*gradient(U, theta)
    theta += (Eps/m)*r
    r += (Eps/2.)*gradient(U, theta)
    return copy.copy(theta), copy.copy(r)

def BuildTree(U, theta, r, u, v, j, Eps, m=1., delta_max=1000):
    """
    Recursive tree-building.

    TODO: Make this less ugly.
    """
    if j == 0:
        # Take one leapfrog step in the direction v.
        theta_p, r_p = Leapfrog(U, theta, r, v*Eps, m=m)
        n_p = float(u <= exp(U(theta_p) - np.dot(0.5*r_p, r_p)))
        s_p = float(u < exp(delta_max + U(theta_p) - np.dot(0.5*r_p, r_p)))
        return theta_p, r_p, theta_p, r_p, theta_p, n_p, s_p
    else:
        # Recursion -- implicitly build the left and right subtrees.
        rargs = (u, v, j-1., Eps)
        rkwargs = {'m':m}
        theta_n, r_n, theta_f, r_f, theta_p, n_p, s_p = BuildTree(U, theta, r, *rargs, **rkwargs)
        if s_p == 1:
            if v == -1:
                theta_n, r_n, null, null, theta_dp, n_dp, s_dp = BuildTree(U, theta_n, r_n, *rargs, **rkwargs)
            else:
                null, null, theta_f, r_f, theta_dp, n_dp, s_dp = BuildTree(U, theta_f, r_f, *rargs, **rkwargs)
            try:
                if uniform.rvs() <= (n_dp/(n_p + n_dp)):
                    theta_p = copy.copy(theta_dp)
            except ZeroDivisionError:
                pass
            s_p = s_p*s_dp*int(np.dot((theta_f - theta_n), r_n) >= 0)*int( np.dot((theta_f - theta_n), r_f) >= 0)
            n_p += n_dp
        return theta_n, r_n, theta_f, r_f, theta_p, n_p, s_p

def NUTS_one_step(U, theta_last, Eps, m=1.):
    """
    TODO: clean up all the copies -- stop being so paranoid.
    """
    r_not = norm.rvs(0, 1., size=len(theta_last))
    u = uniform.rvs(0, exp(U(theta_last) - np.dot(0.5*r_not, r_not)))

    # Initialize.
    theta_m = copy.copy(theta_last)
    theta_n,...


I am working on the file ".ipynb_checkpoints/Hamiltonian MCMC (HMC)-checkpoint.ipynb".
The first task is:
Testing on the banana
Can you write Python code for it?

start = np.array([[uniform.rvs(loc=-10., scale=15.),
                   uniform.rvs(loc=0., scale=10)]
                  for i in xrange(4)])

chains = np.array([ np.array(NUTS(dtarget, start[i, :], Eps=0.55, m=1.5, num_iters=10000)) for i in xrange(start.shape[0])])

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)

for i in xrange(chains.shape[0]):
    plt.scatter(chains[i, :, 0], chains[i, :, 1], alpha=0.5, s=0.02)
plt.show()

plt.subplot(211)
plt.title(Gelman(chains)[0])
for i in xrange(chains.shape[0]):
    plt.plot(chains[i, :, 0])
plt.ylabel('x1')

plt.subplot(212)
for i in xrange(chains.shape[0]):
    plt.plot(chains[i, :, 1])
plt.ylabel('x2')

plt.tight_layout()
plt.show()

plt.hist(chains[0,:,0])


I am working on the file ".ipynb_checkpoints/Hamiltonian MCMC (HMC)-checkpoint.ipynb".
The first task is:
The surface of interest will be $U(q) = -\log{f(q)}$
$K(p) = \frac{p^T p}{2m}$, where $m$ = mass of the puck.
Position over time is a function of momentum:
$\frac{dq_i}{dt} = \frac{p_i}{m}$
Change in momentum over time is a function of surface gradient:
$\frac{dp_i}{dt} = -\frac{\delta U}{\delta q_i}$
Leap-frog algorithm
$ p_i(t + \frac{\epsilon}{2}) = p_i(t) - \frac{\epsilon}{2} \frac{\delta U}{\delta q_i} U(q(t))$
$ q_i(t + \epsilon) = q_i(t) + \frac{\epsilon}{m}p_i(t+\frac{\epsilon}{2})$
$ p_i(t + \epsilon) = p_i(t + \frac{\epsilon}{2}) - \frac{\epsilon}{2} \frac{\delta U}{\delta q_i}(q(t+\epsilon))$
$\epsilon$ -- step size
Can you write Python code for it?

def HMC_one_step(U, current_q, Eps, L, m=1):
    """
    One step of the Hamiltonian Monte Carlo.

    Parameters
    ----------
    U : callable
        A function that takes a single argument, the position.
    q : array-like
        Current position.
    Eps : float
        The step size, epsilon.
    L : int
        Number of leapfrog stpes.
    m : float
        Mass of the particle.

    Returns
    -------
    q_out : array
        Path from ``q`` to the proposed position.
    """

    q = copy.copy(current_q)
    Nq = len(q)
    p = multivariate_normal.rvs([0. for i in xrange(Nq)])
    current_p = copy.copy(p)

    out = {}

    out['p'] = np.zeros((L, Nq))
    out['p'][0,:] = copy.copy(p)
    out['q'] = np.zeros((L, Nq))
    out['q'][0,:] = copy.copy(q)

    for i in xrange(1, L):
        p -= Eps*derivative(U, q, 0.01)/2.
        q += (Eps/m)*p
        out['q'][i, :] = copy.copy(q)
        p -= Eps*derivative(U, q, 0.01)/2.
        out['p'][i, :] = copy.copy(p)

    current_U = U(current_q)
    current_K = (current_p**2).sum()/2.
    proposed_U = U(q)
    proposed_K = (p**2).sum()/2.

    if uniform.rvs() < exp(current_U - proposed_U + current_K - proposed_K):
        out['value'] = q
    else:
        out['value'] = current_q

    return out

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)
U = lambda x: -1.*np.log(dtarget(x))
chain = HMC_one_step(U, np.array([4., 10.]), Eps=0.2, L=10, m=2)['q']
plt.plot(chain[:, 0], chain[:, 1], 'ro')
plt.plot(chain[:, 0], chain[:, 1], 'r-')
plt.plot(chain[0, 0], chain[0,1], 'bo')
plt.xlim(0, 6)
plt.ylim(7, 13)
plt.xlabel('x1')
plt.ylabel('x2')
plt.show()

def HMC(dtarget, start, Eps=0.2, L=10, m=2, N=1000, num_chains=4):
    """
    Perform an HMC simulation.

    Parameters
    ----------
    dtarget : callable
        Target PDF.

    """

    # Invert the target PDF into a concave surface.
    neg_log_dtarget = lambda x: -1.*np.log(dtarget(x))

    # If only one starting position is provided, use it for all...


I am working on the file ".ipynb_checkpoints/Hamiltonian MCMC (HMC)-checkpoint.ipynb".
The first task is:
Tuning parameters: step size, number of steps, and "mass"
HMC does not work discrete parameters. STAN is all HMC.
Gelman metric still applies -- we just have a better way of proposing values.
Can you write Python code for it?

def Gelman(chains):
    if len(chains.shape) == 3:
        N_p = chains.shape[2]
    else:
        N_p = 1
    generate = lambda ptn: np.array([np.array([np.array([ptn(p, i, c)
                                                         for p in xrange(N_p)
                                               for i in xrange(chains.shape[1])])
                                     for c in xrange(chains.shape[0])])])
    params = generate(lambda p, i, c: 'x{0}'.format(p))
    iters = generate(lambda p, i, c: i)
    labels = generate(lambda p, i, c: c)

    data = zip(chains.flat, params.flat, iters.flat, labels.flat)
    dataframe = pd.DataFrame(data, columns=('Value', 'Parameter', 'Iteration', 'Chain'))

    xbar = dataframe.groupby('Parameter').Value.mean()
    m = chains.shape[0]
    xbar_i = dataframe.groupby(('Parameter', 'Chain')).Value.mean()
    s2_i = dataframe.groupby(('Parameter', 'Chain')).Value.var()
    n = dataframe.groupby(('Parameter', 'Chain')).Value.count().mean()

    W = s2_i.mean()
    B = (n/(m-1.)) * ((xbar_i - xbar)**2).sum()
    sigma2_hat = W*(n-1.)/n + B/n
    R_hat = np.sqrt(sigma2_hat/W)
    n_eff = m*n*sigma2_hat/B    # I missed what this was for.

    return R_hat, n_eff

chains = HMC(dtarget, array([4., 10.]), Eps=0.2, L=5, N=1000)

plt.figure(figsize=(10,7))
plt.contour(X, Y, Z)
plt.plot(chains[0][:, 0], chains[0][:, 1], alpha=0.5)
plt.plot(chains[1][:, 0], chains[1][:, 1], alpha=0.5)
plt.plot(chains[2][:, 0], chains[2][:, 1], alpha=0.5)
plt.plot(chains[3][:, 0], chains[3][:, 1], alpha=0.5)
plt.xlim(0, 6)
plt.ylim(7, 13)
plt.show()

plt.subplot(211)
for i in xrange(chains.shape[0]):
    plt.plot(chains[i,:,0])
plt.ylabel('x1')

plt.subplot(212)
for i in xrange(chains.shape[0]):
    plt.plot(chains[i,:,1])
plt.ylabel('x2')

Gelman(chains)


I am working on the file ".ipynb_checkpoints/Homeowrk_3-checkpoint.ipynb".
The first task is:
Great! I hope you're having fun. In the cell below, write an expression or series of statements that uses the widgets list created in the cell above to calculate the total number of widgets that the factory has in its warehouse.
Expected output: 7928
Can you write Python code for it?

total_widget_list= []
for item in widgets:
    total_widget_list.append(item['quantity'])
sum(total_widget_list)


I am working on the file ".ipynb_checkpoints/Homeowrk_3-checkpoint.ipynb".
The first task is:
If our task was to create a dictionary that maps the name of the cheese to the description that follows in the <p> tag directly afterward, we'd be out of luck. Fortunately, Beautiful Soup has a .find_next_sibling() method, which allows us to search for the next tag that is a sibling of the tag you're calling it on (i.e., the two tags share a parent), that also matches particular criteria. So, for example, to accomplish the task outlined above:
Can you write Python code for it?

example_doc = BeautifulSoup(example_html, "html.parser")
cheese_dict = {}
for h2_tag in example_doc.find_all('h2'):
    cheese_name = h2_tag.string
    cheese_desc_tag = h2_tag.find_next_sibling('p')
    cheese_dict[cheese_name] = cheese_desc_tag.string

cheese_dict


I am working on the file ".ipynb_checkpoints/Homeowrk_3-checkpoint.ipynb".
The first task is:
In the cell below, use Beautiful Soup to write some code that prints the names of all the widgets on the page. After your code has executed, widget_names should evaluate to a list that looks like this (though not necessarily in this order):
Skinner Widget
Widget For Furtiveness
Widget For Strawman
Jittery Widget
Silver Widget
Divided Widget
Manicurist Widget
Infinite Widget
Yellow-Tipped Widget
Unshakable Widget
Self-Knowledge Widget
Widget For Cinema
Can you write Python code for it?

w_tag = document.find_all('td', {'class': 'wname'})
#print(w_tag)
for item in w_tag:
    print(item.string)


I am working on the file ".ipynb_checkpoints/Homeowrk_3-checkpoint.ipynb".
The first task is:
In the cell below, write some Python code that prints the names of widgets whose price is above $9.30.
Expected output:
Widget For Furtiveness
Jittery Widget
Silver Widget
Infinite Widget
Widget For Cinema
Can you write Python code for it?

for item in widgets:
    if item['price'] > 9.30:
        print(item['wname'])