I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Now we can plot histgram with log-bin.
Can you write Python code for it?

plt.hist(movie_df['Votes'], bins=bins)
plt.xscale('log')

# TODO: correct the plot
plt.hist(movie_df['Votes'], bins=bins, normed=True)
plt.xscale('log')
plt.yscale('log')


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Or using seaborn:
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Plot on the log scale.
Can you write Python code for it?

plt.scatter( np.arange(len(ratio)), ratio, s=100 )
plt.yscale('log')
plt.ylim( (0.0001,10000) ) # set the scope the y axis
plt.plot( [0,len(ratio)], [1,1], color='k', linestyle='--', linewidth=.5 )


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Ratio and logarithm
If you use linear scale to visualize ratios, it can be very misleading.
Let's first create some ratios.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
Regression
Remember Anscombe's quartet? Let's plot the four datasets and do linear regression, which can be done with scipy's linregress() function.
TODO: display the fitted equations using the text() function.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
We can combine scatter plot with 1D histogram using seaborn's jointplot() function.
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
We can draw a scatter plot of movie votes and ratings using the scatter() function.
Can you write Python code for it?

plt.scatter(subset['Votes'], subset['Rating'])
plt.xlabel('Votes')
plt.ylabel('Rating')


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
We can plot histogram and KDE using pandas:
Can you write Python code for it?


I am working on the file "Data Visualization/Lab 8/w08_lab_Vipul_Munot.ipynb".
The first task is:
What do you see from the two plots? Why do we need to use log scale to visualize ratios?
Let's practice this using random numbers. Generate 10 random numbers between [0,1], calculate the ratios between two consecutive numbers (the second number divides by the first, and so on), and plot the ratios on the linear and log scale.
Can you write Python code for it?

# TODO: generate random numbers and calculate ratios between two consecutive numbers
x = np.random.rand(10)
print(x)
ratio = [ i/j for i,j in zip(x[1:],x[:-1]) ]
print(ratio)

# TODO: plot the ratios on the linear scale
plt.scatter( np.arange(len(ratio)), ratio, s=100 )
plt.plot( [0,len(ratio)], [1,1], color='k', linestyle='--', linewidth=.5 )

# TODO: plot the ratios on the log scale
plt.scatter( np.arange(len(ratio)), ratio, s=100 )
plt.yscale('log')
plt.plot( [0,len(ratio)], [1,1], color='k', linestyle='--', linewidth=.5 )


I am working on the file "Data Visualization/Project/predictive modal - xgboost/kobe_sim_xgboost.ipynb".
The first task is:
Loading data
deleting irrelevant features
Can you write Python code for it?

kobe = pd.read_csv('data.csv', sep=',')
kobe= kobe[np.isfinite(kobe['shot_made_flag'])]
del kobe['lat']
del kobe['lon']
del kobe['game_id']
del kobe['team_id']
del kobe['team_name']

kobe_2 = pd.read_csv('data.csv', sep=',')
kobe_2= kobe_2[np.isfinite(kobe_2['shot_made_flag'])]
del kobe_2['lat']
del kobe_2['lon']
del kobe_2['game_id']
del kobe_2['team_id']
del kobe_2['team_name']


I am working on the file "Data Visualization/Project/predictive modal - xgboost/kobe_sim_xgboost.ipynb".
The first task is:
seperating features and class in both test and train sets
Can you write Python code for it?

columns = kobe.columns.tolist()
columns = [c for c in columns if c not in ["shot_made_flag","team_id","team_name"]]
kobe_train_x =train[columns]
kobe_test_x =test[columns]
kobe_train_y=train['shot_made_flag']
kobe_test_y=test['shot_made_flag']
print(kobe_train_x.shape)
print(kobe_test_x.shape)
print(kobe_train_y.shape)
print(kobe_test_y.shape)


I am working on the file "Data Visualization/Project/predictive modal - xgboost/kobe_sim_xgboost.ipynb".
The first task is:
splitting data into test and train
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
# Generate the training set.  Set random_state to be able to replicate results.
train = kobe.sample(frac=0.6, random_state=1)
train_2 = kobe_2.sample(frac=0.6, random_state=1)
# Select anything not in the training set and put it in the testing set.
test = kobe.loc[~kobe.index.isin(train.index)]
test_2 = kobe_2.loc[~kobe_2.index.isin(train_2.index)]


I am working on the file "Data Visualization/Seaborn/Seaborn Practice - Titanic Data.ipynb".
The first task is:
A heatmap showing the correlations for the entire dataset
Can you write Python code for it?


I am working on the file "Data Visualization/Seaborn/Seaborn Practice - Titanic Data.ipynb".
The first task is:
Displaying passenger and age over a boxplot
Can you write Python code for it?


I am working on the file "Data Visualization/Seaborn/Seaborn Practice - Titanic Data.ipynb".
The first task is:
Plot the fare column as distribution
Can you write Python code for it?


I am working on the file "Data Visualization/Seaborn/Seaborn Practice - Titanic Data.ipynb".
The first task is:
Two histograms ploted using FacetGrid based on age and  sex
Can you write Python code for it?

g = sns.FacetGrid(data=titanic,col='sex')
g.map(plt.hist,'age')


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:

Back to top

4. Conclusion
When checking the output of the data as it was being cleaned, I found that a lot of the earlier data was already cleaned fairly well. This makes sense since these are weekly extracts of data so newer data would not have had time to be cleaned by anyone.
It was also amazing to me how easily multiple cities could be aggregated, and how quickly new cleaning functions could be added into different sub-stages of cleaning.
It doesn't seem that it would be too hard to flesh out more robust cleaning algorithms, and run a script semi-regularly to clean the data automatically.
Also, I did not get a chance to use MongoDB's geospatial querying, but there is a fair amount of lat/lon data that might be able to use this type of querying.
These two cities are fairly small when compared to other more active cities in OpenStreetMap, so API pulls from other databases like google maps might be able to better populate points of interest for the less active areas.

Back to top

Can you write Python code for it?

from IPython import utils
from IPython.core.display import HTML
import os
def css_styling():
    """Load default custom.css file from ipython profile"""
    base = utils.path.get_ipython_dir()
    styles = "<style>\n%s\n</style>" % (open(os.path.join(base,'profile_custom1/static/custom/custom.css'),'r').read())
    return HTML(styles)
css_styling()


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:

Back to top

3. Additional Data Exploration using MongoDB
Since we are reusing the make_city_pipeline() function defined earlier when examining zip codes, we can just show the pipeline that is passed in for our new MongoDB pymongo queries. Everything else would remain the same.

Parking is by far the most reported amenity in either city. It seem that it is most important to find places to park our automobiles, even on small islands.
Can you write Python code for it?

pipeline = [{"$match":{"amenity":{"$exists":1}, "city_name":city}},
            {"$group": {"_id": {"City":"$city_name", "Amenity":"$amenity"},
                        "count": {"$sum": 1}}},
            {"$project": {'_id':0,"City":"$_id.City",
                          "Amenity":"$_id.Amenity", "Count":"$count"}},
            {"$sort": {"Count": -1}},
            {"$limit" : 5 }]

u'result': [{u'Amenity': u'parking', u'City': u'honolulu', u'Count': 280},
            {u'Amenity': u'restaurant', u'City': u'honolulu', u'Count': 123},..}
u'result': [{u'Amenity': u'parking', u'City': u'albuquerque',u'Count': 1270},
             {u'Amenity': u'school', u'City': u'albuquerque', u'Count': 258},..}


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
<hr>

Auto repair shops are more popular in the expansive southwest.
This makes sense since many more miles are put onto automobiles in the much larger state of New Mexico.
Can you write Python code for it?

    pipeline = [{"$match":{"shop":{"$exists":1}, "city_name":city}},
                {"$group":{"_id":{"City":"$city_name", "Shop":"$shop"},
                           "count":{"$sum":1}}},
                {"$project": {'_id':0, "City":"$_id.City",
                              "Shop":"$_id.Shop", "Count":"$count"}},
                {"$sort":{"Count":-1}},
                {"$limit":10}]

 u'result': [{u'City': u'honolulu', u'Count': 50, u'Shop': u'supermarket'},
             {u'City': u'honolulu', u'Count': 24, u'Shop': u'convenience'},
             {u'City': u'honolulu', u'Count': 18, u'Shop': u'clothes'},...}
 u'result': [{u'City': u'albuquerque', u'Count': 66, u'Shop': u'convenience'},
             {u'City': u'albuquerque', u'Count': 46, u'Shop': u'supermarket'},
             {u'City': u'albuquerque', u'Count': 22, u'Shop': u'car_repair'},...}


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:

Combining multiple cities
Two cities were chosen for this project. One city, Albuquerque, was my birth city, and the other, Honolulu, is where I had lived for the last 7 years. It would be interesting to compare the different distributions present in each city. This also posed a small problem of modifying the initial MongoDB loading script to accommodate multiple cities.
The solution was to add a 'city_name' tag name to be passed in with the json data when loading. We could then easily add this key-value pair to each record. We can also make use of projections to clean up what's returned since we are having to match cities first, then group by multiple conditions.

Back to top

2. Data Overview
First, we can see some statistics for our dataset through MongoDB pymongo queries.
File Sizes
Can you write Python code for it?

albuquerque_new-mexico.osm - 53.8 MB
albuquerque_new-mexico.osm.json - 65.7 MBm
honolulu_hawaii.osm - 43.8 MB
honolulu_hawaii.osm.json - 54.4 MB


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
<hr>

Far-away cities have similar types of fast food entrenchment, until the third choice. Next, let's see who these chains are..
Can you write Python code for it?

pipeline = [{"$match":{"amenity":{"$exists":1},
                       "amenity":"fast_food", "city_name":city}},
            {"$group":{"_id":{"City":"$city_name", "Food":"$cuisine"},
                       "count":{"$sum":1}}},
            {"$project":{"_id":0, "City":"$_id.City",
                         "Food":"$_id.Food",
                         "Count":"$count"}},
            {"$sort":{"Count":-1}},
            {"$limit":6}]

 u'result': [{u'City': u'honolulu', u'Count': 13, u'Food': u'burger'},
             {u'City': u'honolulu', u'Count': 2, u'Food': u'sandwich'},
             {u'City': u'honolulu', u'Count': 2, u'Food': u'sushi'},...}

{u'ok': 1.0,
 u'result': [{u'City': u'albuquerque', u'Count': 31, u'Food': u'burger'},
             {u'City': u'albuquerque', u'Count': 16, u'Food': u'sandwich'},
             {u'City': u'albuquerque', u'Count': 6, u'Food': u'pizza'},...}


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
<hr>

For restaurants, pizza is most popular in the isles; regional mexican in the SW.
Can you write Python code for it?

pipeline = [{"$match":{"amenity":{"$exists":1},
                       "amenity":"restaurant", "city_name":city}},
            {"$group":{"_id":{"City":"$city_name","Food":"$cuisine"},
                       "count":{"$sum":1}}},
            {"$project":{"_id":0, "City":"$_id.City",
                         "Food":"$_id.Food", "Count":"$count"}},
            {"$sort":{"Count":-1}},
            {"$limit":6}]

 u'result': [{u'City': u'honolulu', u'Count': 7, u'Food': u'pizza'},
             {u'City': u'honolulu', u'Count': 3, u'Food': u'regional'},
             {u'City': u'honolulu', u'Count': 3, u'Food': u'japanese'}...}
 u'result': [{u'City': u'albuquerque', u'Count': 16, u'Food': u'mexican'},
             {u'City': u'albuquerque', u'Count': 10, u'Food': u'pizza'},
             {u'City': u'albuquerque', u'Count': 6, u'Food': u'american'}...}


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
<hr>

The most popular convenience stores are regional.
Can you write Python code for it?

pipeline = [{"$match":{"shop":{"$exists":1},
                       "city_name":city, "shop":"convenience"}},
            {"$group":{"_id":{"City":"$city_name", "Name":"$name"},
                       "count":{"$sum":1}}},
            {"$project": {'_id':0, "City":"$_id.City",
                          "Name":"$_id.Name", "Count":"$count"}},
            {"$sort":{"Count":-1}},
            {"$limit":5}]

u'result': [{u'City': u'honolulu', u'Count': 7, u'Name': u'ABC Store'},...}
u'result': [{u'City': u'albuquerque', u'Count': 14, u'Name': u'Circle K'}...}


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
<hr>

The top contributor for each city made up the lion's share of the edits. Interestingly, woodpeck_fixbot showed up in both cites taking the #3 spot in Honolulu and #5 spot in Albuquerque.
Can you write Python code for it?

pipeline = [{"$match":{"created.user":{"$exists":1},
                       "city_name":city}},
            {"$group": {"_id": {"City":"$city_name", "User":"$created.user"},
                        "count": {"$sum": 1}}},
            {"$project": {'_id':0, "City":"$_id.City",
                          "User":"$_id.User", "Count":"$count"}},
            {"$sort": {"Count": -1}},
            {"$limit" : 5 }]

 u'result': [{u'City': u'honolulu', u'Count': 98401, u'User': u'Tom_Holland'},
             {u'City': u'honolulu', u'Count': 13051, u'User': u'ikiya'},...}
 u'result': [{u'City': u'albuquerque', u'Count': 88041, u'User': u'EdHillsman'},
             {u'City': u'albuquerque', u'Count': 37604, u'User': u'anjbe'},...}


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
<hr>

Ubiquitous McDonald's and Subway have a far reach.
Can you write Python code for it?

pipeline = [{"$match":{"amenity":{"$exists":1},
                       "amenity":"fast_food","city_name":city}},
            {"$group":{"_id":{"City":"$city_name",
                                        "Name":"$name"},
                                 "count":{"$sum":1}}},
            {"$project":{"_id":0,
                                  "City":"$_id.City",
                                  "Name":"$_id.Name",
                                  "Count":"$count"}},
            {"$sort":{"Count":-1}},
            {"$limit":6}]

 u'result': [{u'City': u'honolulu', u'Count': 8, u'Name': u"McDonald's"},
             {u'City': u'honolulu', u'Count': 4, u'Name': u'Subway'},
             {u'City': u'honolulu', u'Count': 3, u'Name': u'Burger King'},..}
 u'result': [{u'City': u'albuquerque', u'Count': 23, u'Name': u'Subway'},
             {u'City': u'albuquerque', u'Count': 12, u'Name': u"Blake's Lotaburger"},
             {u'City': u'albuquerque', u'Count': 11, u'Name': u"McDonald's"},...}


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
Number of documents
Can you write Python code for it?

print "Both Cities:", db.cities.find().count()
print "Honolulu:", db.cities.find({'city_name':'honolulu'}).count()
print "Albuquerque:", db.cities.find({'city_name':'albuquerque'}).count()

Both Cities: 489274
Honolulu: 227752
Albuquerque: 261522


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
Number of node nodes.
Can you write Python code for it?

print "Both Cities:", db.cities.find({'type':'node'}).count()
print "Honolulu:", db.cities.find({'type':'node', 'city_name':'honolulu'}).count()
print "Albuquerque:", db.cities.find({'type':'node','city_name':'albuquerque'}).count()

Both Cities: 432587
Honolulu: 206262
Albuquerque: 226325


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
Number of way nodes.
Can you write Python code for it?

print "Both Cities:", db.cities.find({'type':'way'}).count()
print "Honolulu:", db.cities.find({'type':'way', 'city_name':'honolulu'}).count()
print "Albuquerque:", db.cities.find({'type':'way', 'city_name':'albuquerque'}).count()

Both Cities: 56648
Honolulu: 21459
Albuquerque: 35189


I am working on the file "Data Wrangling with MongoDB/Project/Data Wrangling with MongoDB - Write-up.ipynb".
The first task is:
Total Number of contributors.
Can you write Python code for it?

print "Contributors:", len(db.cities.distinct("created.user"))

Contributors: 611


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Bonus Round: Questions we can ask of this data
Now, you can start asking questions of this data. What are some interesting questions that this data might answer for us, and how would do that?

Which congressional districts get the most funding?
Which agencies get the most funding in my state?
[With the help of additional data] Which congressional committees do congress folk sit on and is there a coorelation between the committees they sit on and the agencies or contractors that are getting contracts?
How removed is the location of the contractor vs the place where the work is being performed?
What interesting things might there be in the walshhealyact, servicecontractact, davisbaconact, and clingercohenact columns?
What is the distribution of size of vendor to number of contracts and also to the number of contracting dollars? Is any vendor an outlier?
What are the reasons for non-competed contracts?
What percentage of contracts are in each of these categories, and are the numbers reflective of the population as a whole:
issbacertifiedsmalldisadvantagedbusiness
womenownedflag
veteranownedflag
minorityownedbusinessflag
tribalgovernmentflag
ishispanicservicinginstitution
iswomenownedsmallbusiness
isecondisadvwomenownedsmallbusiness
isjointventurewomenownedsmallbusiness
isjointventureecondisadvwomenownedsmallbusiness

What other kinds of questions can we ask?
Can you write Python code for it?


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Filtering rows of data
The basic steps we'll take to filter our list of data won't be much different from what we did above, to compute the total of the 'dollarsobligated' column. We create a new list to store our results, loop through the current list, and when we find a row that matches our criteria, we add it to the new list:
Can you write Python code for it?

filtered_list = []

for row in csv_data:
    if row['womenownedflag'] == 'Y':
        filtered_list.append(row)

print 'Found %d rows that match our criteria, out of all %d rows' % (len(filtered_list), len(csv_data))


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Geocoding addresses
We'll be using the excellent library geopy (docs; run pip install geopy if you don't have it on your machine), which provides a common, simple interface to a variety of different geocoding services. It's worth noting that not all geocoding services work equally well, and they often have limits on how many requests you can make in a short amount of time. So if you're going to geocode a large number of addresses, you'll need to figure out which service is best for you.
To create an instance of a geocoder using a particular service, first import the appropriate class:
from geopy.geocoders import Nominatim

Then create the instance:
geocoder = Nominatim()

Once we have the geocoder instance created, using it is as simple as passing a string containing the address we're interested in:
location = geocoder.geocode("1701 California Street, Denver, CO")

And from there:
print location.latitude, location.longitude

Returns 39.7472023 -104.9904179
For instance
Let's create an instance of the Google geocoder, and use it to find the latitude and longitude of a couple of the vendors in our dataset. (Heads up: most geocoding services restrict heavy usage via IP addresses, so this classroom might get temporarily blocked and the examples may not work).
Can you write Python code for it?

from geopy.geocoders import GoogleV3

geocoder = GoogleV3()

for row in table.limit(10).rows:
    address = ', '.join([
            row['streetaddress'],
            row['city'],
            row['state'],
            str(row['zipcode'])[0:5]])
    coords = geocoder.geocode(address)

    print 'Before', address
    print 'After', coords.address, coords.latitude, coords.longitude
    print '------'


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
If you don't feel comfortable with list comprehensions, don't worry - they're a useful shorthand, but they'll feel more intuitive once you've written enough Python loops to make your fingers bleed. They're not really doing anything different from our first example, and while they're cute and compact, they can be tricky once you need to do anything more than very basic stuff, like filtering based on a single criterion.
Here's how you would accomplish the same filtering task in agate:
Can you write Python code for it?

agate_filtered_list = table.where(lambda row: row['womenownedflag'])

print 'Found %d rows that match our criteria, out of all %d rows' % (len(agate_filtered_list.rows), len(table.rows))


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
It's not much harder to do without agate, however. We can just initialize a counter, loop over each row, and update the counter:
Can you write Python code for it?

counter = 0

for row in csv_data:
    counter += float(row['dollarsobligated'])

print 'Total is $%.2f' % counter


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Loading data from another sources
XML, Excel, and other data formats
There are mant different data formats that exist in the world. We are not going to cover them. Just know that there is a library almost all of them. For example to load XML, you might use Element Tree or lxml, and to load Excel data, you would use openpyxl or xlrd.
Sometimes libraries are clunky and horrible to use. For example, let's say you didn't like how the xlrd library to import Excel data worked. If you didn't have a lot of sheets in your Excel workbook to export, a way around this is to export is manually to CSV first. Besides the manual route, there are a host of tools online and on the commandline that convert data formats from one to another.
Loading from an API
API stands for Application Programming Interface, but what does that mean? It is basically a end point available over the web for you to pick data up from. Before you try to gather data from an API by writing your own code, you should search for a library that does it for you. Larger, more well know APIs such as Twitter's API has multiple libraries in Python. Somtimes these are written by the provider and sometimes by an outside party.
If nothing exists, then use the Requests library to hit the API end point to gather the data. The data will most likely be in one the formats already mentioned (i.e. JSON or CSV). Then you will save that locally and continue the loading process.
API Keys and tokens. Some APIs have keys and also sometimes tokens to limit how much the user can pull data from the API and possibly track their usage. While this isn't covered, you will need to add a few steps to account for this. See the example below to see how this is done.
token = #Your Token
headers = {'Authorization':'token %s' % token}
r = requests.get(url, params=params, headers=headers)
USA Spending API. For the work that we have done so far, we have manually downloaded the data using this form. However, USAspending.gov offers can API to retrieve the data. This is great when you want to...
Can you write Python code for it?

import requests

# Example URL from API documentation page
url = 'https://www.usaspending.gov/fpds/fpds.php?detail=b&fiscal_year=2015&stateCode=TX&max_records=10'
response = requests.get(url)
print response

print(response.content)

import xml.etree.ElementTree as ET
root = ET.fromstring(response.content)

for child in root:
    print child.tag, '-', child.attrib

results = root[1]
for record in results:
    print record

# Let's look at the first record
for record in results[:1]:
    # Let's iterate over the colummns for the record and pull out the data
    for column in record:
        print column.tag, '---', column.text


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Of course this would be shorter in agate. But there's a shorter way in vanilla Python as well, using a very useful shorthand technique called list comprehensions:
Can you write Python code for it?

lc_filtered_list = [row for row in csv_data if row['womenownedflag'] == 'Y']

print 'Found %d rows that match our criteria, out of all %d rows' % (len(lc_filtered_list), len(csv_data))


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Preview the data
Now, let's preview a few of the records to see what the data that we are going to be working with looks like. For this by iterating over the rows in the table. To access the rows we interate over table.rows. Since this is a big file, we are going to return the first few rows by slicing (ie [:3]) table.rows.
To get all the columns headers, we use row.keys() for the each row. Then we iterate over each of those and output the column header, and then the corresponding value through a lookup (i.e. row[column]).
Finally, the last line of dashes (--------------------) is a visual output for us to easily identify where one record ends and the next one begins.
Can you write Python code for it?


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Saving data as JSON
The json module makes it as easy to write JSON as it is to read it. However, since JSON is a very verbose format, it can make for very large files. The contracts_data.csv file, above, is 1.4 MB (enough to fit on a floppy!) - but the exact same data, stored in JSON, is over three times larger - 4.6 MB.
So it's worth considering for a moment whether storing data in JSON is what you really want to do. It's great for Javascript web apps, for instance, because it's highly flexible and self-documenting, and therefore easy for programs to read it and work with it. But, particularly if your data has a large number of columns, the size of your JSON files can get very large very quickly (because every single row will repeat the name of every single column, plus 4 " characters and a : character).
Another thing to keep in mind is that some datatypes can't be represented in JSON. For instance, datetime objects need to be converted to strings first; you'll get an error if you try to save a datetime in JSON directly.
The dumps method of the json module is the exact opposite of the loads method we used above, to load JSON data. Think of dumps as meaning, "DUMP to String" and loads as meaning, "LOAD from String". Here, we just want to write one gigantic string to a file,
Can you write Python code for it?

import json

with open('data/contracts_data.json', 'w+') as fh:
    fh.write(json.dumps(csv_data))


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Sorting rows of data
The sorted function takes a list and returns a sorted version. It expects an iterable, like a list, and by default will just compare every element to every other in order to alphabetically sort them.
You can tell sorted how to compare two elements in the iterable, which allows us to get a version of csv_data sorted by vendor name in just one line:
Can you write Python code for it?

for row in sorted(csv_data, key=lambda row: row['vendorname']):
    print row['vendorname']


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
Storing data
Saving data as a CSV
This will vary somewhat depending on what type of data you have, and in what form. Generally, I find that CSVs are best thought of as lists of dictionaries - each line in the file is an item in the list, and each line contains a dictionary's worth of data. For that reason, it's a good idea to coerce your data into a list of dictionaries (all with the same set of keys) before writing them to a file.
We'll start with the non-agate version first, so you can see each step as it's happening. We'll use the DictWriter class, much as we used the DictReader class to read data in from a CSV.
Can you write Python code for it?

from csv import DictWriter

# First, open the file. Using the mode 'w+' means create the file if it doesn't exist,
# and if it does exist, delete the file first.
with open('data/new_file.csv', 'w+') as fh:

    # Next, create a DictWriter object, passing it two parameters: the file you've opened, and the column names to use
    writer = DictWriter(fh, csv_data[0].keys())

    # Now, make sure to include a header row at the beginning of the file, so we can work with it later
    writer.writeheader()

    # Finally, let's write every line in our data to the file
    writer.writerows(csv_data)


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
This is because, when Python is comparing strings to each other (and both of the above dates, despite looking date-like, are just strings of text) it defaults to comparing them alphabetically. Does the first letter of string A come before the first letter of string B? If so, A < B.
So, we need to tell Python how to convert our string of arbitrary text into a datetime object. Once we do that, we get all kinds of superpowers - we can add and subtract time from a date, compare dates to each other, adjust the timezone of our date, pull out just the month or year, determine what day of the week it was, and on and on.
The datetime module provides several types of date-related classes we can use (in particular, date, time and datetime, which combines the first two) but for now we'll just rely on datetime. Annoyingly, datetime is both the name of a module, and the name of a class within that module, so we have to do dumb stuff like this:
from datetime import datetime

Or
import datetime
datetime.datetime.now()

I like the first one, myself. If we just wanted, say, date then we'd do:
from datetime import date

Or
import datetime
datetime.date.today()

Then we need to determine how to understand the date objects we're working with in our data (and this is where strftime.org is really useful). We do this by creating a format string, which tells datetime how our dates are structured.
Take older, above. It's date is "5-13-1989": "month hyphen day hyphen 4-digit year". In the format string language that datetime uses, that translates to "%m (month) hyphen %d (day) hyphen %Y (4-digit year)". datetime expects that the format string will also tell it about any non-date characters, so we also have to include the hyphens in our format string. The end result will look like this:
format_string = '%m-%d-%Y'

We then use the strptime function to create a datetime object from a string. We have to pass it both the string we'd like to convert, and the format string that tells us how to do so:
dt = datetime.strptime('5-13-1989',...
Can you write Python code for it?

from datetime import datetime

newer = '2010-06-17'
print datetime.strptime(newer, '%Y-%m-%d')

two_digit_year = '1/20/00'
print datetime.strptime(two_digit_year, '%m/%d/%y')
# Now you see why Y2K seemed like a big deal (does anyone even remember Y2K?) Why does it pick this date to convert to?

crazy_text_having_variable = '2013 in June on day 12'
print datetime.strptime(crazy_text_having_variable, '%Y in %B on day %d')


I am working on the file "Data Wrangling with Python.ipynb".
The first task is:
You'll notice that we didn't simply do:
counter += row['dollarsobligated']

This is because, in contrast to agate - which can automatically detect the column type based on the data inside it - DictReader assumes that every column in the csv file is a string. Adding two strings together, even strings only containing numbers, merely combines them:
Can you write Python code for it?


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I apply multiple filter criteria to a pandas DataFrame?
Can you write Python code for it?

movies = pd.read_csv('http://bit.ly/imdbratings')

movies.head()

movies[movies.duration >= 200]

movies[(movies.duration >= 200) & (movies.genre == 'Drama')]


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I avoid a SettingWithCopyWarning in pandas?
Can you write Python code for it?

movies = pd.read_csv('http://bit.ly/imdbratings')
movies.head()

movies.content_rating.isnull().sum()

movies[movies.content_rating.isnull()]

movies.content_rating.value_counts()

movies.loc[movies.content_rating == 'NOT RATED', 'content_rating'] = np.nan

movies.content_rating.isnull().sum()

top_movies = movies.loc[movies.star_rating >= 9, :]
top_movies

top_movies.loc[0, 'duration'] = 150

top_movies

top_movies = movies.loc[movies.star_rating >= 9, :].copy()
top_movies

top_movies.loc[0, 'duration'] = 150
top_movies


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I change display options in pandas?
Can you write Python code for it?

drinks = pd.read_csv('http://bit.ly/drinksbycountry')
drinks

pd.get_option('display.max_rows')

pd.set_option('display.max_rows', None)

drinks

pd.reset_option('display.max_rows')
pd.get_option('display.max_rows')

train = pd.read_csv('http://bit.ly/kaggletrain')
train

pd.get_option('display.max_colwidth')

pd.set_option('display.max_colwidth', 1000)

pd.get_option('display.precision')

pd.set_option('display.precision', 2)

drinks.head()

drinks['x'] = drinks.wine_servings * 1000
drinks['y'] = drinks.total_litres_of_pure_alcohol * 1000
drinks.head()

pd.set_option('display.float_format', '{:,}'.format)

pd.describe_option('rows')

pd.reset_option('all')


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I change the data type of a pandas Series?
Can you write Python code for it?

drinks = pd.read_csv('http://bit.ly/drinksbycountry')

drinks.dtypes

drinks.beer_servings = drinks.beer_servings.astype(float)

drinks.dtypes

drinks = pd.read_csv('http://bit.ly/drinksbycountry', dtype={'beer_servings': float})

drinks.dtypes

orders = pd.read_table('http://bit.ly/chiporders')

orders.head()

orders.dtypes

orders.item_price = orders.item_price.str.replace('$', '').astype(float)

orders.head()

orders.item_name.str.contains('Chicken').astype(int).head()


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I create a pandas DataFrame from another object?
Can you write Python code for it?

df = pd.DataFrame({'id': [100, 101, 102], 'color': ['red', 'blue', 'red']}, columns=['id', 'color'], index=['A', 'B', 'C'])

pd.DataFrame([[100, 'red'], [101, 'blue'], [102, 'red']], columns=['id', 'color'])

import numpy as np
arr = np.random.rand(4, 2)
arr

pd.DataFrame(arr, columns=['one', 'two'])

pd.DataFrame({'student': np.arange(100, 110, 1), 'test': np.random.randint(60, 101, 10)}).set_index('student')

s = pd.Series(['round', 'square'], index=['C', 'B'], name='shape')
s

df

pd.concat([df, s], axis=1)


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I create dummy variables in pandas?
Can you write Python code for it?

train = pd.read_csv('http://bit.ly/kaggletrain')
train

train['Sex_male'] = train.Sex.map({'female': 0, 'male': 1})

train.head()

pd.get_dummies(train.Sex, prefix='Sex').iloc[:, 1:]

train.Embarked.value_counts()

embarked_dummies = pd.get_dummies(train.Embarked, prefix='Embarked').iloc[:, 1:]
train = pd.concat([train, embarked_dummies], axis=1)
train.head()

pd.get_dummies(train, columns=['Sex', 'Embarked'], drop_first=True)


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I explore a pandas Series?
Can you write Python code for it?

movies = pd.read_csv('http://bit.ly/imdbratings')

movies.head()

movies.dtypes

movies.genre.describe()

movies.genre.value_counts()

movies.genre.value_counts(normalize=True)

movies.genre.unique()

movies.genre.nunique()

pd.crosstab(movies.genre, movies.content_rating)

movies.duration.describe()

movies.duration.mean()


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I filter rows of a pandas DataFrame by column value?
Can you write Python code for it?

movies = pd.read_csv('http://bit.ly/imdbratings')

movies.head()

movies.shape

movies[movies.duration >= 200]


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I find and remove duplicate rows in pandas?
Can you write Python code for it?

user_cols = ['user_id', 'age', 'gender', 'occupation', 'zip_code']
users = pd.read_table('http://bit.ly/movieusers', delimiter='|', header=None, names=user_cols, index_col='user_id')
users.head()

users.shape

users.zip_code.duplicated().sum()

users.duplicated().sum()

users.loc[users.duplicated(keep=False), :]

users.drop_duplicates(keep='first').shape

users.duplicated(subset=['age', 'zip_code']).sum()


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I handle missing values in pandas?
Can you write Python code for it?

ufo = pd.read_csv('http://bit.ly/uforeports')

ufo.tail()

ufo.isnull().tail()

ufo.notnull().tail()

ufo.isnull().sum()

ufo[ufo.City.isnull()]

ufo.shape

ufo.dropna(how='any').shape

ufo.dropna(how='all').shape

ufo.dropna(subset=['City', 'Shape Reported'], how='any').shape

ufo['Shape Reported'].value_counts(dropna=False)

ufo['Shape Reported'].fillna(value='VARIOUS', inplace=True)

ufo['Shape Reported'].value_counts()


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I remove columns from a pandas DataFrame?
Can you write Python code for it?

ufo = pd.read_csv('http://bit.ly/uforeports')

ufo.head()

ufo.shape

ufo.drop('Colors Reported', axis=1, inplace=True)

ufo.head()

ufo.drop(labels=['City', 'State'], axis=1, inplace=True)

ufo.head()


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
How do I use pandas with scikit-learn to create Kaggle submissions?
Can you write Python code for it?

train = pd.read_csv('http://bit.ly/kaggletrain')

train.head()

feature_cols = ['Pclass', 'Parch']
X = train.loc[:, feature_cols]

y = train.Survived

from sklearn import linear_model
logreg = linear_model.LogisticRegression()
logreg.fit(X, y)

test = pd.read_csv('http://bit.ly/kaggletest')
X_new = test.loc[:, feature_cols]

y_predict = logreg.predict(X_new)

y_predict.shape

pd.DataFrame({'PassengerId': test.PassengerId, 'Survived': y_predict}).set_index('PassengerId').to_csv('sub.csv')

train.to_pickle('train.pkl')

pd.read_pickle('train.pkl')


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
Tip: Create a new Series in a DataFrame
Can you write Python code for it?

ufo['location'] = ufo.City + ', ' + ufo.State

ufo.head()


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
Tip: Filtered DataFrame is also a DataFrame
Can you write Python code for it?

movies.loc[movies.duration >= 200, 'genre']


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
Tip: Hit "Shift+Tab" inside a method parentheses to get list of arguments

How to rename columns in pandas DataFrame?
Can you write Python code for it?

ufo.head()

ufo.columns

ufo.rename(columns={'Colors Reported': 'Colors_Reported', 'Shape Reported': 'Shape_Reported'}, inplace=True)

ufo.columns

ufo_cols = ['city', 'colors reported', 'state reported', 'state', 'time', 'location']
ufo.columns = ufo_cols

ufo.head()

ufo_cols = ['City', 'Colors Reported', 'State Reported', 'State', 'Time']
ufo = pd.read_csv('http://bit.ly/uforeports', names=ufo_cols, header=0)

ufo.head()


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
Tip: Sort by multiple columns
Can you write Python code for it?


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
Tip: To remove rows instead of columns, choose axis=0
Can you write Python code for it?

ufo.drop([0, 1], axis=0, inplace=True)

ufo.shape


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
Tip: Use str.replace method to drop the space from columns names
Can you write Python code for it?

ufo.columns = ufo.columns.str.replace(' ', '_')

ufo.columns


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
What do I need to know about the pandas index?
Can you write Python code for it?

drinks = pd.read_csv('http://bit.ly/drinksbycountry')
drinks.head()

drinks.index

drinks.loc[23, :]

drinks.set_index('country', inplace=True)

drinks.head()

drinks.loc['Brazil', 'beer_servings']

drinks.index.name = None

drinks.head()

drinks.reset_index(inplace=True)

drinks.head()

drinks = pd.read_csv('http://bit.ly/drinksbycountry')
drinks.head()

drinks.set_index('country', inplace=True)
drinks.head()

drinks.continent.head()

drinks.continent.value_counts()['Africa']

drinks.continent.value_counts().sort_index().head()

people = pd.Series([3000000, 85000], index=['Albania', 'Andorra'], name='population')
people

drinks.beer_servings * people

pd.concat([drinks, people], axis=1).head()


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
When should I use "inplace" parameter in pandas?
Can you write Python code for it?

ufo = pd.read_csv('http://bit.ly/uforeports')
ufo.head()

ufo.drop('City', axis=1, inplace=True)

ufo.head()

ufo.dropna(how='any').shape

ufo.shape


I am working on the file "Data analysis in Python with pandas.ipynb".
The first task is:
Why do some pandas commands end with parentheses, and other commands don't?
Can you write Python code for it?

movies = pd.read_csv('http://bit.ly/imdbratings')

movies.head()

movies.describe()

movies.shape

movies.dtypes

type(movies)

movies.describe(include=['object'])


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
A list of all the columns in the dataset
Can you write Python code for it?


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Calculating the % of missing values
Can you write Python code for it?

df_na= ( df.isnull().sum() / len(df) ) * 100
df_na= df_na.drop(df_na[df_na == 0].index).sort_values(ascending= False)

f, ax= plt.subplots(figsize=(12, 8))
plt.xticks(rotation='90')
sns.barplot(x=df_na.index, y=df_na.values)
ax.set(title='Missing Values Plot', ylabel='% Missing')


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Genre and total sales on platform. Nintendo looks to be selling alot of sports oriented games, especially on the
Wii. However the Wii U is struggling in sales.
Can you write Python code for it?

nintendo_sales= nintendo.groupby(['Genre', 'Platform'])['Global_Sales'].sum()
nintendo_sales.unstack().plot(kind='bar',stacked=True, colormap= 'Set1',  grid=False)


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Global sales of the Xbox consoles globally 1994-2016
Can you write Python code for it?

sales_xb= xb.groupby(['Year_of_Release', 'Platform'])['Global_Sales'].sum()
sales_xb.unstack().plot(kind='bar',stacked=True, colormap= 'Vega20',  grid=False)


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Last generation sales from Japan
Can you write Python code for it?

current_gen_sales= current_gen.groupby(['Year_of_Release', 'Platform'])['JP_Sales'].sum()
current_gen_sales.unstack().plot(kind='bar',stacked=True, colormap= 'Blues',  grid=False)


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Let us start by importing some helper libraries and the dataset as well
Can you write Python code for it?

import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline
from brewer2mpl import qualitative

df= pd.read_csv("../input/Video_Games_Sales_as_at_22_Dec_2016.csv")
df.shape


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Now that we know the shape of the dataset, let's have a peek at the data and try to find if they are any missing values.
Can you write Python code for it?


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Platform and total global sales from 1994-2016
Can you write Python code for it?

nintendo_sales= nintendo.groupby(['Year_of_Release', 'Platform'])['Global_Sales'].sum()
nintendo_sales.unstack().plot(kind='bar',stacked=True, colormap= 'Set1',  grid=False)


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Playststation Global 1994-2016
Can you write Python code for it?

sales_Play= play.groupby(['Year_of_Release', 'Platform'])['Global_Sales'].sum()
sales_Play.unstack().plot(kind='bar',stacked=True, colormap= 'Oranges',  grid=False)
ax.set(title='Playststation Global over the year', ylabel='Cumulative count')


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Plot of the user score v the critic score of games. It appears the users and the critics agree on
games with score greater than 8.
Can you write Python code for it?

sns.jointplot(x='User_Score', y='Critic_Score', data=df_clean, kind='hex', cmap='coolwarm', size=7)


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Rating and total global sales. Nintendo sold majorly in the category of E (everyone)
Can you write Python code for it?

nintendo_sales= nintendo.groupby(['Rating', 'Platform'])['Global_Sales'].sum()
nintendo_sales.unstack().plot(kind='bar',stacked=True, colormap= 'Set1',  grid=False)

current_gen= df_clean[(df_clean['Platform']== 'Wii') | (df_clean['Platform']== 'X360') |
                                                        (df_clean['Platform']== 'PS3')]
current_gen.shape


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Taking a closer look at Xbox
Can you write Python code for it?

xb= df_clean[(df_clean['Platform']== 'X360') | (df_clean['Platform']== 'XOne')
              | (df_clean['Platform']== 'XB')]
xb.shape


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Taking a closer look at nintendo
Can you write Python code for it?

nintendo= df_clean[(df_clean['Platform']== 'DS') | (df_clean['Platform']== 'Wii')
              | (df_clean['Platform']== 'GC')| (df_clean['Platform']== 'GBA')
                  |(df_clean['Platform']== '3DS') | (df_clean['Platform']== 'WiiU')]
nintendo.shape


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
Taking a look at Playstation
Can you write Python code for it?

play= df_clean[(df_clean['Platform']== 'PS2') | (df_clean['Platform']== 'PS3')
              | (df_clean['Platform']== 'PS')| (df_clean['Platform']== 'PS4')]
play.shape


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
The cell below shows the column name and the number of empty rows.
Can you write Python code for it?


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
The cell below shows the data type of the columns
Can you write Python code for it?


I am working on the file "Data analysis/Ign_dataset_Analysis.ipynb".
The first task is:
dropping all NA values
Can you write Python code for it?

df_clean= df.dropna(axis=0)
df_clean.shape

ssc = df_clean.Platform.value_counts()
f, ax= plt.subplots(figsize=(12, 8))
plt.xticks(rotation='90')
sns.barplot(x=ssc.values, y=ssc.index, orient='h')
ax.set(title='Consoles by count after dropping NAs', ylabel='Count')
f.tight_layout()

#df['User_Score']= df['User_Score'].convert_objects(convert_numeric=True)
df_clean.User_Score= df_clean.User_Score.astype('float')
#df.User_Score.dtype
#df['User_Score'] = video['User_Score'].convert_objects(convert_numeric= True)


I am working on the file "Data analysis/NLP_reddit_worldnews_headlines.ipynb".
The first task is:
A glimpse of the dataset rows and columns
Can you write Python code for it?


I am working on the file "Data analysis/NLP_reddit_worldnews_headlines.ipynb".
The first task is:
And for fun let's look at some headlines related to Apple Inc
Can you write Python code for it?


I am working on the file "Data analysis/NLP_reddit_worldnews_headlines.ipynb".
The first task is:
Columns in the dataset
Can you write Python code for it?


I am working on the file "Data analysis/NLP_reddit_worldnews_headlines.ipynb".
The first task is:
Have a glimpse of articles about Obama, Trump and Hilary
Can you write Python code for it?

tell_me_about("Obama")['title'].values.tolist()[:10]

tell_me_about("Hilary")['title'].values.tolist()[:10]

tell_me_about("Donald")['title'].values.tolist()[:10]

tell_me_about("Trump")['title'].values.tolist()[:10]


I am working on the file "Data analysis/NLP_reddit_worldnews_headlines.ipynb".
The first task is:
Let us find out how many times Obama, Hillary and Trump were mentioned from 2008 - 2016
Can you write Python code for it?

cand= ['Obama', 'Hillary', 'Trump']
for sole in cand:
   print( data.title.str.contains(sole).value_counts(),sole )

#data.title.str.contains('Trump').value_counts()

from nltk import word_tokenize
tokens = data.title.map(word_tokenize)

def tell_me_about(x):
    x_l = x.lower()
    x_t = x.title()
    return data.loc[tokens.map(lambda sent: x_l in sent or x_t in sent).values]


I am working on the file "Data analysis/NLP_reddit_worldnews_headlines.ipynb".
The first task is:
Let us look at the mean number of posts over a rolling window of 120 days
Can you write Python code for it?

import seaborn as sns
%matplotlib inline
sns.set_style("dark")

data.groupby('date_created')['up_votes'].mean().plot()
data.groupby('date_created')['up_votes'].mean().rolling(window=120).mean().plot(figsize= (12, 4))


I am working on the file "Data analysis/NLP_reddit_worldnews_headlines.ipynb".
The first task is:
The columns and data type
Can you write Python code for it?


I am working on the file "Data analysis/NLP_reddit_worldnews_headlines.ipynb".
The first task is:
we can see from above that they are no posts with downvotes
Let us check for most upoted headline
Can you write Python code for it?

[title for title in data.sort_values('up_votes', ascending=False)['title'][:10]]


I am working on the file "Data collection and analysis.ipynb".
The first task is:
Bay Area Bike Share System
Preliminary Analysis
data is obtained from:
http://www.bayareabikeshare.com/open-data
Can you write Python code for it?

# read bike trip data

trp = pd.read_csv('data/trip.csv')
weather = pd.read_csv('data/weather.csv')
stations = pd.read_csv('data/station.csv')

# calculate total trip count for each station

start_counts['station_id'] = start_counts.index
start_counts.columns = ['start counts', 'station_id']
end_counts['station_id'] = start_counts.index
end_counts.columns = ['end counts', 'station_id']

stations = stations.merge(start_counts)
stations = stations.merge(end_counts)
stations['total counts'] = stations['start counts'] + stations['end counts']

print 'total number of docks:', stations.dockcount.sum()
print 'total number of bikes:', trp['Bike #'].value_counts().shape[0]
# print trp['Subscription Type'].value_counts()


I am working on the file "Data collection and analysis.ipynb".
The first task is:
Explore the daily trips
Can you write Python code for it?

# plot bike trips for each day

trp['Start Date'] = pd.to_datetime(trp['Start Date'])
trp['End Date'] = pd.to_datetime(trp['End Date'])

nday = trp['Start Date'].dt.date.nunique()
stat_days = trp['Start Date'].dt.date.value_counts()

fig, ax = plt.subplots(1, 1, figsize=(16, 8), sharex=True, sharey=True)
stat_days.plot()

# remove spurious trips that are too long,
normal_trip = trp[trp.Duration < 10000]
normal_trip['minutes'] = normal_trip['Duration'] / 60 # in minutes


I am working on the file "Data collection and analysis.ipynb".
The first task is:
collect public suggestion data
Can you write Python code for it?

# save all jsons to disk
url = 'http://suggest.bayareabikeshare.com/api/places'
pages = range(1,48)

for page in pages:
    payload = {'page': page, 'include_submissions': 'false'}
    response = requests.get(url,payload)
    st_json = response.json()
    with open('data/suggestion-json/page'+str(page),'w') as f:
        json.dump(st_json, f)

# parse out infomation from json, return a namedtuple
def parseJson(st_json):
    stationBasic = namedtuple('stationBasic', ['stid','lat','lng','weight'])
    stationBasics = []
    for i in range(len(st_json['features'])):
        stid = st_json['features'][i]['id']
        lat = st_json['features'][i]['geometry']['coordinates'][1]
        lng = st_json['features'][i]['geometry']['coordinates'][0]
        if st_json['features'][i]['properties']['submission_sets'].has_key('support') == True:
            weight = st_json['features'][i]['properties']['submission_sets']['support']['length']
        else:
            weight = 0
        stationBasics.append( stationBasic(stid = stid, lat = lat, lng = lng, weight = weight) )
    return stationBasics

# test = st_json
# testvalue = parseJson(test)
# test if has valid supports
# df = pd.DataFrame(result)
# df[3].value_counts()
# testvalue

# read suggestions in jsons and parse out the needed data
pages = range(1,48)

parsed = []
for page in pages:
    filename = 'data/suggestion-json/page' + str(page)
    with open(filename, 'r') as f:
        st_json = json.load(f)
#         print len(parseJson(st_json))
        parsed.extend( parseJson(st_json) )

columns = ['stid','lat','lng','weight']
df  = pd.DataFrame(parsed, columns=['stid','lat','lng','weight'])
df['weight'].sum()
df['lng'].describe()

df.to_csv('data/suggestions.csv',index = False)


I am working on the file "Data extraction from Nengo.ipynb".
The first task is:
Step 2: Run it in normal Nengo
Can you write Python code for it?

sim = nengo.Simulator(model)
sim.run(2)

pylab.plot(data)


I am working on the file "Data extraction from Nengo.ipynb".
The first task is:
Step 3: Extract data about groups of neurons
Can you write Python code for it?

for ens in model.all_ensembles:
    print('Ensemble %d' % id(ens))
    print('    number of neurons: %d' % ens.n_neurons)
    print('    tau_rc: %g' % ens.neuron_type.tau_rc)
    print('    tau_ref: %g' % ens.neuron_type.tau_ref)
    print('    bias: %s' % sim.data[ens].bias)


I am working on the file "Data extraction from Nengo.ipynb".
The first task is:
Step 5: Extract data about outputs
Can you write Python code for it?

for node in model.all_nodes:
    if node.size_in > 0:
        print('Output node %d' % id(node))
        print('Function to call: %s' % node.output)


I am working on the file "Data extraction from Nengo.ipynb".
The first task is:
Step 6: Extract data about connections between Ensembles
Can you write Python code for it?

for conn in model.all_connections:
    if isinstance(conn.pre_obj, nengo.Ensemble) and isinstance(conn.post_obj, nengo.Ensemble):
        print('Connection from %d to %d' % (id(conn.pre_obj), id(conn.post_obj)))
        print('    synapse time constant: %g' % conn.synapse.tau)
        decoder = sim.data[conn].weights
        transform = nengo.utils.builder.full_transform(conn, allow_scalars=False)
        encoder = sim.data[conn.post_obj].scaled_encoders
        print('    decoder: %s' % decoder)
        print('    transform: %s' % transform)
        print('    encoder: %s' % encoder)


I am working on the file "Data extraction from Nengo.ipynb".
The first task is:
Step 7: Extract data about inputs
Can you write Python code for it?

for conn in model.all_connections:
    if isinstance(conn.pre_obj, nengo.Node) and isinstance(conn.post_obj, nengo.Ensemble):
        print('Connection from input %d to %d' % (id(conn.pre_obj), id(conn.post_obj)))
        print('    synapse time constant: %g' % conn.synapse.tau)
        transform = nengo.utils.builder.full_transform(conn, allow_scalars=False)
        encoder = sim.data[conn.post_obj].scaled_encoders
        print('    transform: %s' % transform)
        print('    encoder: %s' % encoder)


I am working on the file "Data extraction from Nengo.ipynb".
The first task is:
Step 8: Extract data about outputs:
Can you write Python code for it?

for conn in model.all_connections:
    if isinstance(conn.pre_obj, nengo.Ensemble) and isinstance(conn.post_obj, nengo.Node):
        print('Connection from %d to output %d' % (id(conn.pre_obj), id(conn.post_obj)))
        print('    synapse time constant: %g' % conn.synapse.tau)
        decoder = sim.data[conn].weights
        transform = nengo.utils.builder.full_transform(conn, allow_scalars=False)
        print('    decoder: %s' % decoder)
        print('    transform: %s' % transform)


I am working on the file "Data set preparation.ipynb".
The first task is:
A sample file looks like -
Can you write Python code for it?

img = Image.open(train_folder_labels[0][0])
print(img.size)
image_size = img.size[0]
img


I am working on the file "Data set preparation.ipynb".
The first task is:
As we can see labels are not shuffled so lets shuffle labels accordingly
Can you write Python code for it?

def randomize(dataset, labels):
  permutation = np.random.permutation(labels.shape[0])
  shuffled_dataset = dataset[permutation,:,:]
  shuffled_labels = labels[permutation]
  return shuffled_dataset, shuffled_labels
train_dataset, train_labels = randomize(train_dataset, train_labels)
test_dataset, test_labels = randomize(test_dataset, test_labels)
valid_dataset, valid_labels = randomize(valid_dataset, valid_labels)


I am working on the file "Data set preparation.ipynb".
The first task is:
Finally lets save data for further reuse -
Can you write Python code for it?

pickle_file = os.path.join(os.getcwd(), 'final_dataset.pickle')

try:
  f = open(pickle_file, 'wb')
  save = {
    'train_dataset': train_dataset,
    'train_labels': train_labels,
    'valid_dataset': valid_dataset,
    'valid_labels': valid_labels,
    'test_dataset': test_dataset,
    'test_labels': test_labels,
    }
  pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)
  f.close()
except Exception as e:
  print('Unable to save data to', pickle_file, ':', e)
  raise

statinfo = os.stat(pickle_file)
print('Compressed pickle size:', statinfo.st_size)


I am working on the file "Data set preparation.ipynb".
The first task is:
Spot check to see if the file still looks good
Can you write Python code for it?

import matplotlib.pyplot as plt

pickle_file = train_dataset[0]
with open(pickle_file, 'rb') as file:
    tank_set = pickle.load(file)
    sample_image = tank_set[5,:,:]
    plt.figure()
    plt.imshow(sample_image)
    plt.show()

# we need to merge datasets for training in order to train well
def make_arrays(nb_rows, img_size):
  if nb_rows:
    dataset = np.ndarray((nb_rows, img_size, img_size), dtype=np.float32)
    labels = np.ndarray(nb_rows, dtype=np.int32)
  else:
    dataset, labels = None, None
  return dataset, labels

def merge_datasets(pickle_files, train_size, valid_size=0):
  num_classes = len(pickle_files)
  valid_dataset, valid_labels = make_arrays(valid_size, image_size)
  train_dataset, train_labels = make_arrays(train_size, image_size)
  vsize_per_class = valid_size // num_classes
  tsize_per_class = train_size // num_classes

  start_v, start_t = 0, 0
  end_v, end_t = vsize_per_class, tsize_per_class
  end_l = vsize_per_class+tsize_per_class
  for label, pickle_file in enumerate(pickle_files):
    try:
      with open(pickle_file, 'rb') as f:
        tank_set = pickle.load(f)
        # let's shuffle the tank_type to have random validation and training set
        np.random.shuffle(tank_set)
        if valid_dataset is not None:
          valid_tank = tank_set[:vsize_per_class, :, :]
          valid_dataset[start_v:end_v, :, :] = valid_tank
          valid_labels[start_v:end_v] = label
          start_v += vsize_per_class
          end_v += vsize_per_class

        train_tank = tank_set[vsize_per_class:end_l, :, :]
        train_dataset[start_t:end_t, :, :] = train_tank
        train_labels[start_t:end_t] = label
        start_t += tsize_per_class
        end_t += tsize_per_class
    except Exception as e:
      print('Unable to process data from', pickle_file, ':', e)
      raise

  return valid_dataset, valid_labels, train_dataset, train_labels

# pass test data first and then I will pass train data
# break test into validation and test data set
import...


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
Data cleaning importing an munging
The data is a set of csv files that have been split into a training set and test set already. To support easy read in to a pandas DF we preprocess the files by adding a header that we get from the datasets metadata and adding an index column
Can you write Python code for it?

import csv

def indexAndAnnotateDataSet(filename):
    outname = filename.replace('.csv', '_clean.csv')
    with open(outname, 'wb') as csvfile:
        f = open(filename,'r')
        filewriter = csv.writer(csvfile, delimiter=',')
        columns = ['', 'age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'target']
        filewriter.writerow(columns)
        idx = 0
        for line in f:
            lst = []
            lst.append(str(idx))
            for item in line.strip('.').split(','):
                lst.append(item.strip().strip('.'))

            filewriter = csv.writer(csvfile, delimiter=',')
            filewriter.writerow(lst)
            idx +=1
        f.close()
    return outname


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
Feeding the learning algorithms
As we are dealing with a very dense dataset containing mostly categoricals we might assume that tree classifiers should do well on this problem, so we will look into training a simple tree and a random forest.
But before that we need to split our data frame into training features and the target variables:
Can you write Python code for it?

train_feature_df = numerical_data_df.drop('target',1)
train_target_df = numerical_data_df.target


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
First we are interested how the naive tree is doing in learning our training set. To this end we record the learning curve of our dataset
Can you write Python code for it?

def plotLearningCurve(classifier, feature_df, target_df, train_set_sizes):
    train_sizes, train_scores, test_scores = learning_curve(classifier,
                                                            feature_df,
                                                            target_df,
                                                            train_sizes=train_set_sizes)
    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)

    plt.fill_between(train_sizes, train_scores_mean - train_scores_std,
                     train_scores_mean + train_scores_std, alpha=0.1,
                     color="r")
    plt.fill_between(train_sizes, test_scores_mean - test_scores_std,
                     test_scores_mean + test_scores_std, alpha=0.1, color="g")
    plt.plot(train_sizes, train_scores_mean, 'o-', color="r",
             label="Training score")
    plt.plot(train_sizes, test_scores_mean, 'o-', color="g",
             label="Cross-validation score")

    plt.legend(loc="best")

train_sizes=np.linspace(.1, 1.0, 15)
plotLearningCurve(tree_clf, train_feature_df, train_target_df, train_sizes)


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
Indeed we see a sweet spot at around depth 12, after which the cross-validation score starts to drop. We will fix the classifier at depth 12 and learn the final classifier
Can you write Python code for it?

tree_clf.max_depth = 12
tree_clf.fit(train_feature_df, train_target_df)


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
Learning a tree
Let's learn a simple tree using scikit-learn's tree classifier
Can you write Python code for it?

import numpy as np
from sklearn import tree
from sklearn.learning_curve import learning_curve
from sklearn.learning_curve import validation_curve

tree_clf = tree.DecisionTreeClassifier()


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
Looking at the 'education' and 'education-num' column we might suspect duplicate information. Let's check for that!
Can you write Python code for it?


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
Much better! We see that the curves are starting to converge and haven't saturated yet. We hence reduced our bias and it seems that increasing the data-set size might even improve our classifier.
Finally let's learn the final RF classifier
Can you write Python code for it?

rf_clf.max_depth = 14
rf_clf.n_estimators = 9
rf_clf.fit(train_feature_df, train_target_df)


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
Training a Random Forest
RFs have the advantage that their inherent randomness counter-acts the learning of train-set intrisic patterns which don't generalize. They tend not to overfit and are stable against outliers.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

rf_clf = RandomForestClassifier()


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
We face the same problems as with the single tree, hence we need to dig into the hyper-parameter space. We explore the maximal depth and the number of trees in the forest.
Can you write Python code for it?

#Validation curve to find optimal max_depth of trees
param_range = range(2,17)
plotValidationAccuracyCurve(rf_clf, train_feature_df, train_target_df, 'max_depth', param_range)


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
We find the optimal value at around depth 14. Let's look for the optimal forest size at that maximal depth (Note: we might only find a local optimum or non at all. To find an actual optimal point we would need a grid search in hyper-parameter space)
Can you write Python code for it?

#Validation curve to find optimal number of trees
param_range = range(2,17)
rf_clf.max_depth = 14
plotValidationAccuracyCurve(rf_clf, train_feature_df, train_target_df, 'n_estimators', param_range)


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
We immediately see that the tree as well as the RF share the three most important features 'capital-gain', 'education-num' and 'marital-status' to be indicative of whether or not the income is larger than 50K. However the single tree places slightly more importance on those features than the RF which might be connected to some overfitting due to missing randomness that the RF has. Hover the 4th feature is interesting! RF indicates that 'relationship' is a strong indicator of large income, whereas the single tree completely dismisses this feature as unimportant.
Let's look at this feature in more detail!
Can you write Python code for it?

marital_status_list = list_of_catList[3]
df_to_plot = numerical_data_df[['relationship','target']]

df_to_plot[df_to_plot.target ==1].hist()
df_to_plot[df_to_plot.target ==0].hist()
print marital_status_list


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
We see that around a number of 9 trees the scores start to saturate, so we can choose this value as a sweet spot. Let's see how our learning curve looks at this point in hyper-parameter space
Can you write Python code for it?

train_sizes=np.linspace(.1, 1.0, 15)
rf_clf.max_depth = 14
rf_clf.n_estimators = 9
plotLearningCurve(rf_clf, train_feature_df, train_target_df, train_sizes)


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
We see that the single tree is slighlty more accurate, but that both classifiers perform close to 90% accurate. Looking at the confusion matrix it becomes obvious that the RF is more careful in classifying positives as the total number of false positives is smaller. Let's look at some more metrics to draw a final conclusion
Can you write Python code for it?

print "Tree precision: %0.3f" % precision_score(numerical_test_df.target, tree_prediction)
print "RF precision: %0.3f" % precision_score(numerical_test_df.target, rf_prediction)
print "\n"
print "Tree recall: %0.3f" % recall_score(numerical_test_df.target, tree_prediction)
print "RF recall: %0.3f" % recall_score(numerical_test_df.target, rf_prediction)


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
We see that this is indeed a feature! Not being in the relationship status 'Husband' significantly decreases your chance of making more than 50K a year! Husband's themselves are relatively safe since their status appears mostly uncorrelated with the money they earn. Intuitively it is clear that RF could pick this up since every tree in the forest might split on a different value of the 'relationship' category and the final voting will outweight the insignificant 'husband' value.
Summary
Though the analysis done here is fairly trivial it turns out that using two simple classifiers we are already able to gain insight into a interesting feature. Namley that not being in a relationship status 'Husband' significantly reduces the chances of making more than 50K of income a year. For husbands themselves however, the status is non-decisive. THe feature was detected using a Random Forest classifier but could not be picked up using a single Decision tree.
I hope you find this notebook useful and it motivates you to play some more with it. Contributions are always welcome!
Miscellenea
References
are mostly given throughout the text. But importantly

[1] scikit-learn

Stylesheet
Can you write Python code for it?

from IPython.core.display import HTML

def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Data-Analytics/Adult census data.ipynb".
The first task is:
We will use scikit-learn's built-in cross-validation methodes. Hence, we join the training and test data into a larger data frame to increase the size of the dataset. Moreover, we will drop rows that contain na feature and the column called 'fnlwgt' as this is a control value introduced by the original authors of the dataset.
Inhaling the csv files and appending the data frames we get:
Can you write Python code for it?

raw_data_df = pd.DataFrame.from_csv(clean_DS_1).dropna().drop('fnlwgt',1)
raw_test_df = pd.DataFrame.from_csv(clean_DS_2).dropna().drop('fnlwgt',1)
raw_data_df


I am working on the file "Data-Analytics/PCA_Pitfalls.ipynb".
The first task is:
Explained variance ratios of PCA and the variance in original space
Unscaled Dataset
Can you write Python code for it?

import matplotlib.pyplot as plt
import sklearn.datasets as ds
from sklearn.decomposition import PCA

data = ds.load_breast_cancer()['data'];
pca_trafo = PCA().fit(data);

plt.figure(figsize = (10,6.5));
plt.semilogy(pca_trafo.explained_variance_ratio_, '--o');
plt.xlabel('principal component', fontsize = 20);
plt.ylabel('explained variance', fontsize = 20);
plt.tick_params(axis='both', which='major', labelsize=18);
plt.tick_params(axis='both', which='minor', labelsize=12);
plt.xlim([0, 29]);

plt.figure(figsize = (10,6.5));
plt.semilogy(np.square(data.std(axis=0)) / np.square(data.std(axis=0)).sum(), '--o', label = 'variance ratio');
plt.semilogy(data.mean(axis=0) / np.square(data.mean(axis=0)).sum(), '--o', label = 'mean ratio');
plt.xlabel('original feature', fontsize = 20);
plt.ylabel('variance', fontsize = 20);
plt.tick_params(axis='both', which='major', labelsize=18);
plt.tick_params(axis='both', which='minor', labelsize=12);
plt.xlim([0, 29]);
plt.legend(loc='lower left', fontsize=18);


I am working on the file "Data-Analytics/PCA_Pitfalls.ipynb".
The first task is:
Scaled Dataset
Can you write Python code for it?

import matplotlib.pyplot as plt
import sklearn.datasets as ds
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

z_scaler = StandardScaler()
data = ds.load_breast_cancer()['data'];

z_data = z_scaler.fit_transform(data)
pca_trafo = PCA().fit(z_data);

fig, ax1 = plt.subplots(figsize = (10,6.5))
ax1.semilogy(pca_trafo.explained_variance_ratio_, '--o', label = 'explained variance ratio');
color =  ax1.lines[0].get_color()
ax1.set_xlabel('principal component', fontsize = 20);
for tl in ax1.get_yticklabels():
    tl.set_color(plot1[0].get_color())

plt.legend(loc=(0.01, 0.075) ,fontsize = 18);

ax2 = ax1.twinx()
ax2.semilogy(pca_trafo.explained_variance_ratio_.cumsum(), '--go', label = 'cumulative explained variance ratio');
for tl in ax2.get_yticklabels():
    tl.set_color('g')

ax1.tick_params(axis='both', which='major', labelsize=18);
ax1.tick_params(axis='both', which='minor', labelsize=12);
ax2.tick_params(axis='both', which='major', labelsize=18);
ax2.tick_params(axis='both', which='minor', labelsize=12);
plt.xlim([0, 29]);
plt.legend(loc=(0.01, 0),fontsize = 18);


I am working on the file "Data-Analytics/Sentiment analysis of weather related tweets.ipynb".
The first task is:
For completeness we will also show a snippet from the test set to see what kind of features we are allowed to use.
Can you write Python code for it?

test_df = pd.DataFrame.from_csv('./datasets/test.csv')
test_df.head(2)


I am working on the file "Data-Analytics/Sentiment analysis of weather related tweets.ipynb".
The first task is:
Label creation for Sentiment category
We start out by trying to model the sentiment category of a tweet. To this end we start by replacing the maximum vlaue of the sentiment fields s1-s5 with the corresponding category
Can you write Python code for it?

def create_label(row):
    # row is of time pandas.Series need to cast to a list.
    lst = row.tolist()
    return lst.index(max(lst))+1

# apply defaults to work on columns rather than rows. Use axis = 1 to cahnge to row application.
label_df = raw_df[['s1','s2','s3','s4','s5']].apply(create_label, axis=1)
label_df.head(5)


I am working on the file "Data-Analytics/Sentiment analysis of weather related tweets.ipynb".
The first task is:
Splitting the set and training the model
As the training data and the labels are now spearate and of different type it is annoying to split the training and test-sets manually. Again sklearn has a tool for us that we can use: the cross_validation API.
Can you write Python code for it?

from sklearn import cross_validation

df_train, df_test, y_train, y_test = cross_validation.train_test_split(raw_df, label_df, test_size = 0.3)


I am working on the file "Data-Analytics/Sentiment analysis of weather related tweets.ipynb".
The first task is:
The simple counter already indicates a problem. There are a lot of very common words that have obiously no signal, such as 'the', 'at', 'of', etc. We need to remove those stopwords. Another problem can be seem by comparing '@mention' and '@mention:' which should clearly be identified as the same word, meaning we need to remove the punctuation. Finally we might want to identify 'storm' and 'stormy' as the same and thus require stemming techniques.
The following code snippet provides a tokenizer that does all of the above
Can you write Python code for it?

from nltk.corpus import stopwords
from nltk.stem.snowball import EnglishStemmer
from nltk import word_tokenize
from nltk.util import ngrams
import string

def custom_tokenizer(document_string):

    # define the stop word vocabulary
    stops = [unicode(word) for word in stopwords.words('english')] \
        + ["''", "``", 're:', 'fwd:', '-', '@mention', '@mention:', 'mention', 'link', ':', 'f.', '&']

    # create a default stemmer
    stemmer = EnglishStemmer()

    # return the stemmed list of words
    tokens = [stemmer.stem(unicode(word)) for word in word_tokenize(document_string.lower()) \
            if not (unicode(word.lower()) in stops or unicode(word.lower()) in list(string.punctuation))]

    return tokens + list(ngrams(tokens, 2))


I am working on the file "Data-Analytics/Sentiment analysis of weather related tweets.ipynb".
The first task is:
This looks already much better and we can identify weather realated terms such as 'storm', 'sunni', etc.
This will be a good starting point for the rest of the model.
Model generation
One step that is missing is how to use the custom_tokenizer to actually create feature vectors. Luckily enough sklearn provides use with the right functionality to do just that.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(min_df = 100, max_df = 10000, tokenizer = custom_tokenizer)


I am working on the file "Data-Analytics/Sentiment analysis of weather related tweets.ipynb".
The first task is:
To ensure we have always the same vectorization options we need to "fit" the vectorizer to the set so we can vectorize any new examples according to this. We do it here on the full set from which we will also extract the cross-validation data and we should be aware of this bias, as it necessarily eliminates the possibility of encountering previously unseen terms when we run our test set predicition.
Can you write Python code for it?

# fit the vectorizer to the full set
X = vectorizer.fit_transform(raw_df.tweet.tolist())

X.shape


I am working on the file "Data-Analytics/Sentiment analysis of weather related tweets.ipynb".
The first task is:
Tokenizing, stemming and term counting
In order to use the tweets as input for a machine learning algorithm we need to convert them into numerical features. One way to do this is to chop up the tweets and count the number of words within a tweet and turn them into a large sparse vector whose length is size of the vocabulary of all tweets.
We start with a very simple way of counting the terms in the total of all tweets and count the number of terms.
Can you write Python code for it?

from collections import Counter

full_tweet_string = raw_df.tweet.apply(lambda t: t.lower() + " ").sum()
Counter(full_tweet_string.split()).most_common()[:25]


I am working on the file "Data-Analytics/Sentiment analysis of weather related tweets.ipynb".
The first task is:
We can now start to train the model. As this is a highly sparse problem it lends itself to be tackled using a Naive Bayes classifier.
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB

multi_nb_clf = MultinomialNB()
multi_nb_clf.fit(X_train, y_train)

multi_nb_clf.score(X_test, y_test)


I am working on the file "Data-Capstone-Projects/.ipynb_checkpoints/SF Salaries Exercise- Solutions-checkpoint.ipynb".
The first task is:
Check the head of the DataFrame.
Can you write Python code for it?


I am working on the file "Data-Capstone-Projects/.ipynb_checkpoints/SF Salaries Exercise- Solutions-checkpoint.ipynb".
The first task is:
How many people have the word Chief in their job title? (This is pretty tricky)
Can you write Python code for it?

def chief_string(title):
    if 'chief' in title.lower().split():
        return True
    else:
        return False

sum(sal['JobTitle'].apply(lambda x: chief_string(x)))


I am working on the file "Data-Capstone-Projects/.ipynb_checkpoints/SF Salaries Exercise- Solutions-checkpoint.ipynb".
The first task is:
How many unique job titles are there?
Can you write Python code for it?


I am working on the file "Data-Capstone-Projects/.ipynb_checkpoints/SF Salaries Exercise- Solutions-checkpoint.ipynb".
The first task is:
What is the average BasePay ?
Can you write Python code for it?


I am working on the file "Data-Capstone-Projects/.ipynb_checkpoints/SF Salaries Exercise- Solutions-checkpoint.ipynb".
The first task is:
What is the highest amount of OvertimePay in the dataset ?
Can you write Python code for it?


I am working on the file "Data-Capstone-Projects/.ipynb_checkpoints/SF Salaries Exercise- Solutions-checkpoint.ipynb".
The first task is:
What is the name of highest paid person (including benefits)?
Can you write Python code for it?

sal[sal['TotalPayBenefits']== sal['TotalPayBenefits'].max()] #['EmployeeName']
# or
# sal.loc[sal['TotalPayBenefits'].idxmax()]


I am working on the file "Data-Capstone-Projects/.ipynb_checkpoints/SF Salaries Exercise- Solutions-checkpoint.ipynb".
The first task is:
What is the name of lowest paid person (including benefits)?
Can you write Python code for it?

sal[sal['TotalPayBenefits']== sal['TotalPayBenefits'].min()] #['EmployeeName']
# or
# sal.loc[sal['TotalPayBenefits'].idxmax()]['EmployeeName']


I am working on the file "Data-Capstone-Projects/.ipynb_checkpoints/SF Salaries Exercise- Solutions-checkpoint.ipynb".
The first task is:
What was the average (mean) BasePay of all employees per year? (2011-2014) ?
Can you write Python code for it?


I am working on the file "Data-Visualizations/Lending Club Loan Data - Loan Book Distribution ("Bokeh" Viz) [Python notebook, kaggle].ipynb".
The first task is:
2.2 Loan book Volume by U.S. States
Can you write Python code for it?

# PROVIDE THE CHOROPLETH OF "LOAN BOOK VOLUME PER STATE"
palette2.reverse()
color_mapper = LogColorMapper(palette=palette2,
                              low=grouped_agg_df['loanbook_vol_per_state'].min(),
                              high=grouped_agg_df['loanbook_vol_per_state'].max())

# Define the figure "Tools" we want to make available
TOOLS = "pan, wheel_zoom, reset, hover, save"

# Plot the figure
# Define the figure dimensions and its general details
p = figure(title="Loan Book Volume by U.S. States", tools=TOOLS,
           plot_width=960, plot_height=500,
           x_range=(0, 960), y_range=(500, 0),
           x_axis_location=None, y_axis_location=None)

# Render the "Bokeh" patches in Glyph
p.patches('xs', 'ys', source=geo_source,
          fill_color={'field': "loanbook_vol_per_state" ,'transform': color_mapper},
          fill_alpha=0.7, line_color="white", line_width=0.5)

# Add a Hover Tools over the U.S. States
hover = p.select_one(HoverTool)
hover.point_policy = "follow_mouse"
hover.tooltips = [
    ("State", "@state"),
    ("Loan Book Volume", "@loanbook_vol_per_state{,}"),
    ("(Long, Lat)", "($x, $y)"),
]

# Add a ColorBar Legend
color_bar = ColorBar(color_mapper=color_mapper, ticker=LogTicker(),
                     background_fill_alpha=0.7,
                     label_standoff=5,
                     major_label_text_color='black',
                     major_tick_line_color='black', major_tick_line_width=1.3, major_tick_out=5,
                     border_line_color=None, location=(0,0),
                     orientation='horizontal', width=500)
p.add_layout(color_bar, 'above')

show(p)


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Automatically detect column type
Can you write Python code for it?

continuous_vars, rest = get_var_type(df)


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Categorical variable

'term',
'grade',
'sub_grade',
'emp_length',
'home_ownership',
'is_inc_v',
'loan_status',
'pymnt_plan',
'purpose',
'addr_city',
'addr_state',
'revol_util',
'initial_list_status',

Dates

'accept_d',
'exp_d',
'list_d',
'issue_d',
'earliest_cr_line'
'last_pymnt_d',
'last_credit_pull_d'
Can you write Python code for it?


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Data Preprocessing
Can you write Python code for it?

get_item_description('revol_util')

df['int_rate'] = df['int_rate'].str.extract('(\d+.\d+)')
df['int_rate'] = df['int_rate'].map(lambda x: float(x))

df['revol_util'] = df['revol_util'].str.extract('(\d+.\d+)')
df['revol_util'] = df['revol_util'].map(lambda x: float(x))
df['revol_util'] = df['revol_util'].fillna(-1) # that is 25% missing, so filling with -1 to indicate missing

df.shape

df = df.dropna()
df.shape

categorical_var = ['term','grade','sub_grade',
 'emp_length','home_ownership','is_inc_v','pymnt_plan','purpose',
 'addr_city','addr_state','revol_util','initial_list_status']

date_var = ['accept_d','list_d','exp_d','issue_d',
'earliest_cr_line','last_pymnt_d','last_credit_pull_d']


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Deal with categorical variables
Can you write Python code for it?

display(df[categorical_var[0:10]].head(1))
display(df[categorical_var[10:]].head(1))

from sklearn.preprocessing import LabelEncoder

number = LabelEncoder()
df['term_old'] = df['term']
df['term'] = number.fit_transform(df['term'].astype(str))

df['addr_city_old'] = df['addr_city']
df['addr_city'] = number.fit_transform(df['addr_city'].astype(str))

df['addr_state_old'] = df['addr_state']
df['addr_state'] = number.fit_transform(df['addr_state'].astype(str))

df['home_ownership_old'] = df['home_ownership']
df['home_ownership'] = number.fit_transform(df['home_ownership'].astype(str))

df['purpose_old'] = df['purpose']
df['purpose'] = number.fit_transform(df['purpose'].astype(str))

df['pymnt_plan_old'] = df['pymnt_plan']
df['pymnt_plan'] = number.fit_transform(df['pymnt_plan'].astype(str))

df['is_inc_v_old'] = df['is_inc_v']
df['is_inc_v'] = number.fit_transform(df['is_inc_v'].astype(str))

df['initial_list_status_old'] = df['initial_list_status']
df['initial_list_status'] = number.fit_transform(df['initial_list_status'].astype(str))

df['emp_length_old'] = df['emp_length']
df['emp_length'] = number.fit_transform(df['emp_length'].astype(str))

df['grade_old'] = df['grade']
df['grade'] = number.fit_transform(df['grade'].astype(str))

df['sub_grade_old'] = df['sub_grade']
df['sub_grade'] = number.fit_transform(df['sub_grade'].astype(str))

new_grade_var = [x for x in df.columns if 'grade' in x]
display(df[new_grade_var].head(1))


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Deal with dates
Can you write Python code for it?

for cols in date_var:
    print (cols)
    df[cols+'_months'] = convert_date_months(df[cols])
    df[cols+'_days'] = convert_date_days(df[cols])


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Derive Features

Extract day of the week from dates
Extract week of the year from dates
Can you write Python code for it?

for cols in date_var:
    print (cols)
    df[cols+'_num_day'] = df[cols].apply(lambda x: convert_num_day_of_week(x))
    df[cols+'_day'] = df[cols].apply(lambda x: convert_day_of_week(x))
    df[cols+'_week_of_year'] = df[cols].apply(lambda x: convert_week_of_year(x))


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Derive Features

loan_rank: 1 for bad loans, 0 for good loans
Good loans are the ones where the status is
current
fully paid

Bad loans are the ones where the status is
default
in grace period
late
charged off

Extract day of the week from dates
Extract week of the year from dates

Derive Features

Loan Rank
Can you write Python code for it?

data = df['loan_status']
data.dropna(inplace = True)

df['loan_status'].value_counts()

index_ = df['loan_status'][df['loan_status'].str.contains('(Does not meet the credit*)')].index


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Load data dictionary
Can you write Python code for it?

xls_file = pd.ExcelFile(dataPath+'/LCDataDictionary.xlsx')
xls_file.sheet_names

df_data_dictionary = xls_file.parse('LoanStats')
df_data_dictionary.columns = [u'Item', u'Description']
df_ = xls_file.parse('browseNotes')
df_.columns = df_data_dictionary.columns
df_data_dictionary = df_data_dictionary.append(df_)

df_data_dictionary.head()

df.head()


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
Save Dataset to file
Can you write Python code for it?


I am working on the file "Data/data_EDA_1_preprocessData.ipynb".
The first task is:
To make my analysis tractable given the time in which this analysis must be completed, I am going to restrict myself to features for which less than 20% are missing.
Am dropping url, index, id, member_id, emp_title, title
Can you write Python code for it?

drop_list = ['url', 'index', 'id', 'member_id', 'emp_title', 'title']
for col, val in missing_df:
    if val > 0.2:
        drop_list.append(col)

df.drop(drop_list, axis=1, inplace=True)

df.head()


I am working on the file "Data/data_EDA_2_explorativeDataAnalysis.ipynb".
The first task is:
Categorical variables
Can you write Python code for it?

data_[cat_vars].describe()

pyplt.rcParams['figure.figsize'] = (12, 6)

sns.stripplot(x='sub_grade', y='loan_amnt', hue='loan_rank', data=data_);
pyplt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.);
pyplt.title('Bad Loans By Amount and Sub Grade', fontsize=14)
pyplt.savefig('report/figures/bad_loan_sub_grade.png',bbox_inches='tight', format='png', dpi=200)
pyplt.close();

# reset figure size
pyplt.rcParams['figure.figsize'] = (6, 4)

pyplt.rcParams['figure.figsize'] = (10, 4)

sns.countplot(x='sub_grade', data=data, hue='grade');
pyplt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.);
pyplt.title('Loan Sub Grades', fontsize=14)
pyplt.savefig('report/figures/sub_grade.png',bbox_inches='tight', format='png', dpi=200)
pyplt.close();

# reset figure size
pyplt.rcParams['figure.figsize'] = (6, 4)

sns.factorplot(x='grade', y='annual_inc', data=data_, hue='loan_rank');
pyplt.title('Annual income by Grade', fontsize=14)
pyplt.savefig('report/figures/inc_grade.png', format='png', dpi=200)
pyplt.close();

g = sns.factorplot(x='grade', y='loan_amnt', data=data_, hue='loan_rank',
               col='purpose_old', col_wrap=4, kind='box');
g.fig.subplots_adjust(top=0.9)
g.fig.suptitle('Loans By Grade and Purpose', fontsize=20)
pyplt.savefig('report/figures/loan_grade_purpose.png', bbox_inches='tight', format='png', dpi=200)

pyplt.close();

g = sns.factorplot(x='grade', y='loan_amnt', data=data_, hue='loan_rank',
               col='home_ownership_old', col_wrap=4, kind='box');
g.fig.subplots_adjust(top=0.85)
g.fig.suptitle('Loans By Home Ownership', fontsize=15)
pyplt.savefig('report/figures/loan_home.png', format='png', dpi=200)
pyplt.close();

pyplt.rcParams['figure.figsize'] = (12, 8)
sns.stripplot(x='addr_state_old', y='loan_amnt', hue='loan_rank', data=data_, size=4, jitter=True);

# reset figure size
pyplt.rcParams['figure.figsize'] = (6, 4)
pyplt.close();


I am working on the file "Data/data_EDA_2_explorativeDataAnalysis.ipynb".
The first task is:
Exploratory Data Analysis
Can you write Python code for it?

tmp = ['delinq_amnt', 'acc_now_delinq', 'chargeoff_within_12_mths', 'collections_12_mths_ex_med',
 'tax_liens','policy_code']
data[tmp].describe()

data['delinq_amnt'].plot(kind='kde')
pyplt.title('Density Estimation of Amount Delinquent')
pyplt.legend(loc='upper right', shadow=True, fontsize='medium')
pyplt.savefig('report/figures/delinq_amnt.png', dpi=200)
pyplt.close();

# drop these features because they are not informative
data.drop(tmp, axis=1, inplace=True)
data.shape


I am working on the file "Data/data_EDA_2_explorativeDataAnalysis.ipynb".
The first task is:
Income
Hypothesis: Clients with lower incomes are more apt to default their loans.
Based on my hypothesis, I am starting off my exploration with income. To start off, I am going to check this feature to see if there are any outliers in the data. When I plot the raw features, I see that there are some because the distribution is extremely skewed with standard deviation over \$500k.

I normalize the income data using Tukey's method; this gave a better representation of the underlying data; this distribution is closer to a uniform distribution. The majority of the clients in this dataset make an annual income of around \$65k+.

I ultimately decided to change my outliers skim by only removing folks in the top 1% of incomes from the data. I am making this choice because I believe there might be some signal from the folks in the upper ten percentile of the distribution. As a result of doing this, the mean moved to around \$70k+.

Can you write Python code for it?

pyplt.rcParams['figure.figsize'] = (8, 8)

sns.distplot(data['annual_inc'])
pyplt.ylabel('percent per unit')
pyplt.xlabel('Annual income in dollars')
pyplt.xticks(rotation='vertical')
pyplt.title('Annual Income')
pyplt.savefig('report/figures/annual_income_raw.png', format='png', dpi=200)
pyplt.close();

#reset figure
pyplt.rcParams['figure.figsize'] = (6, 4)

data['annual_inc'].describe()

# Clean income for outliers
mask = get_outliers('annual_inc')
df_inc = pd.DataFrame(index=data.index)
df_inc['income'] = data.annual_inc
df_inc.drop(df_inc.ix[mask].index, inplace = True)

pyplt.rcParams['figure.figsize'] = (8, 8)

sns.distplot(df_inc);
pyplt.ylabel('percent per unit')
pyplt.xlabel('Annual income in dollars')
pyplt.xticks(rotation='vertical')
pyplt.title('Annual Income: Outliers removed (Tukey)')
pyplt.savefig('report/figures/annual_income_cleaned.png', format='png', dpi=200)
pyplt.close();

#reset figure
pyplt.rcParams['figure.figsize'] = (6, 4)

df_inc.describe()

#Am dropping folks whose salary is beyond the 99 percentile

top = data['annual_inc'].quantile(.99)
mask = data.loc[data['annual_inc'] > top].index
data.drop(mask, inplace=True)

pyplt.rcParams['figure.figsize'] = (8, 8)

sns.distplot(data['annual_inc'])
pyplt.ylabel('percent per unit')
pyplt.xlabel('Annual income in dollars')
pyplt.xticks(rotation='vertical')
pyplt.title('Annual Income: Top 1% removed')
pyplt.savefig('report/figures/annual_income.png', format='png', dpi=200)
pyplt.close();

#reset figure
pyplt.rcParams['figure.figsize'] = (6, 4)

data['annual_inc'].describe()


I am working on the file "Data/data_EDA_2_explorativeDataAnalysis.ipynb".
The first task is:
Lending Club Data Explorations
Datastory
My goal in this data challenge is to understand the underlying characteristics that are present in clients that tend to default on their loans. This knowledge will help guide me in building a classifier that predicts the appropriate interest rate of a loan based on client profile.
Can you write Python code for it?

%pylab inline

# Import libraries
from __future__ import absolute_import, division, print_function

# Ignore warnings
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import pandas as pd
from sklearn.externals import joblib

# Graphing Libraries
import matplotlib.pyplot as pyplt
import seaborn as sns
sns.set_style('whitegrid')

def plot_features_by_target(df, features, target):
    """Creates a pair plot of the features by the target variable

    Attributes
    ------------
    df: pandas dataframe
    features: list of strings (cols of df)
    target: list of strings (single col of df)
    """

    sns.pairplot(df, x_vars=features, y_vars=target);

def get_outliers(feature):
    """Get the indices of outliers for a feature if it exist

    Attributes
    ----------
    feature: pandas col

    return outlier indices
    """
    outlier  = []
    outlier_indices = []

    # For feature find the data points with extreme high or low values

    Q1 = data[feature].quantile(.25)
    Q3 = data[feature].quantile(.75)

    step = 1.5 * (Q3 - Q1)

    # Display the outliers
    temp_data = data[~((data[feature] >= Q1 - step) & (data[feature] <= Q3 + step))]
    outlier.append(dict(feature=feature, indices=temp_data.index.tolist()))
    outlier_indices += temp_data.index.tolist()

    return outlier_indices

def sample_data(num_sample, df, with_replacement=False):
    """
    Create a random sample from a table

    Attributes
    ---------
    num_sample: int
    df: dataframe
    with_replacement: boolean

    Returns a random subset of table index
    """
    df_index = []
    lst = np.arange(0, len(df), 1)

    for i in np.arange(0, num_sample, 1):

        # pick randomly from the whole table
        sample_index = np.random.choice(lst)

        if with_replacement:
            # store index
            df_index.append(sample_index)
        else:
            # remove the choice that was selected
            lst = np.setdiff1d(lst,[sample_index])
           ...