I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
According to this plot, which classes seem more difficult to distinguish?
4. Regression example
Now that we know how to load some data and visualize them, we will try to solve a simple regression task.
Our objective in this example is to predict the crime rates in different areas of the US using some socio-demographic data.
This dataset has 127 socioeconomic variables of different nature: categorical, integer, real, and for some of them there are also missing data (check wikipedia). This is usually a problem when training machine learning models, but we will ignore that problem and take only a small number of variables that we think can be useful for regression and which have no missing values.

population: population for community
householdsize: mean people per household
medIncome: median household income

The objective in the regresion problem is another real value that contains the total number of violent crimes per 100K population.
4.1 Loading the data
First of all, load the data from file communities.csv in a new array. This array should have 1994 rows (instances) and 128 columns.
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
Another important array manipulation method is array concatenation or stacking. It is useful to always state explicitly in which direction we want to stack the arrays. For example in the following example we are stacking the arrays vertically.
Can you write Python code for it?

print(my_array)
print(my_array2)
print(np.concatenate( (my_array, my_array2) , axis=1)) # columnwise concatenation


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
Check again the shape and dimension of my_array3
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
Define a new 3x2 array named my_array2 with  [1, 2, 3] in the first row and [4,5,6] in the second.
Check the dimension of the array.
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
EXERCISE: Concatenate the first column of my_array and the second column of my_array2
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
If you have correctly computed it you will see that my_array3 is one dimensional. Sometimes this can be a problem when you are working with 2D matrixes (and vectors can be considered as 2D matrixes with one of the sizes equal to 1). To solve this, numpy provides the newaxis constant.
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
In the previous code we have saved the features in matrix X and the class labels in the vector labels. Both are 2D numpy arrays.
We are also printing the shapes of each variable (see that we can also use array_name.shape to get the shape, appart from function shape()). Checking the shape of matrices is a convenient way to prevent mistakes in your code.
3.2 Visualizing the data
Extract the 2 first features of the data (sepal length and width) and plot the first versus the second in a figure, use a different color for the data corresponding to different classes.
First of all you probably want to split the data according to each class label.
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
It is possible to extract a single row or column from a 2D numpy array so that the result is still 2D, without explictly recurring to np.newaxis. Compare the outputs of the following print commands.
Can you write Python code for it?

print(my_array[:,1])
print(my_array[:,1].shape)
print(my_array[:,1:2])
print(my_array[:,1:2].shape)


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
Plot each variable in X_com versus y_com to have a first (partial) view of the data.
Can you write Python code for it?


I am working on the file "P3.Python_datos/Old/Data_python_student.ipynb".
The first task is:
Take the columns (5,6,17) of the data and save them in a matrix X_com. This will be our input data. Convert this array into a float array. The shape should be (1994,3)
Get the last column of the data and save it in an array called y_com. Convert this matrix into a float array.
Check that the shape is (1994,1) .
Can you write Python code for it?


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Answering question
What is the most popular type of shop in Plaisir?
Can you write Python code for it?

pipeline_shop = make_group_pipeline('$types.shop')
result_shop = aggregate(db.plaisir_osm, pipeline_shop)
pp.pprint(result_shop)


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Audit of the file
Can you write Python code for it?

tags = {}
for event, elem in ET.iterparse("sample.osm"):
    if elem.tag not in tags:
        tags[elem.tag]= 1
    else:
        tags[elem.tag] += 1
print tags


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Audit postcodes
Can you write Python code for it?

# Create a set to store weird postcodes
postcodes = set()
# Create a regular expression to isolate weird postcodes
postcode_re = re.compile(r'\d\d\d\d\d', re.IGNORECASE)

def audit_postcode(postcodes,postcode):
    """
    This function aims to check if the postcode matchs the expected format.
    If it doesn't, the postcode is added to a dictionary.
    Arg1: postcodes >> the dictionary where to store the unexpected data.
    Arg2: postcode >> the postcode to be audited.
    """
    m = postcode_re.search(postcode)
    if m == None:
        postcodes.add(postcode)

def audit_postcode_map(file_in, postcodes, pretty = False):
    """
    This function aims to audit the file by isolating one by one the postcode tags key/value pair.
    The value of each pair is then audited thanks to the audit_postcode function.
    """
    for _, element in ET.iterparse(file_in):
        if element.tag == "way" or element.tag == "node":
            for tag in element.iter("tag"):
                if tag.attrib['k'] == "addr:postcode":
                    audit_postcode(postcodes,tag.attrib["v"])
    pp.pprint(postcodes)

audit_postcode_map("plaisir.osm",postcodes, True)


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Bakery is the most popular shop in Plaisir. No kidding.. Plaisir is in France :)
What is the sport with the most facilities in Plaisir?
Can you write Python code for it?

pipeline_sport = make_group_pipeline('$types.sport')
result_sport = aggregate(db.plaisir_osm, pipeline_sport)
pp.pprint(result_sport)


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
One street type needs to be cleaned ('AVE'). We will clean that street type when shapping our data structure. Please find below the function we'll use to clean street types.
Can you write Python code for it?

#check addr: with :
lower_colon = re.compile(r'^([a-z]|_)*:([a-z]|_)*$')

def clean_street_type(address_tag,address_dict):
    """
    This function aims to transform address tags and add them as a key / value pair of a dedicated address dictionary.
    The function will take the address tag and the address dictionary as arguments:
        address_tag: every sub tag of a "node" or "way" top level tag.
        address_dict: a dictionnary where we store key / value pair of adress elements (postcode, street, etc..)
    The function first creates a cleaned key based on the current key of the tag by removing the "addr:" part of the string.
    If after removing "addr:" the function still detects a colon, the tag isn't added to the dictionary because the data entry was not clean enought.
    Else, if the value of the tag starts with "AVE" as first word, "AVE" is replaced by "Avenue".
    Finally, the key / value is added to the address dictionary.
    """
    key = re.sub('addr:', '', address_tag['k']).strip()
    if lower_colon.match(key):
        return None
    else:
        if address_tag['v'].startswith("AVE"):
            address_dict[key] = re.sub(r'^AVE', 'Avenue', address_tag['v'])
        else:
            address_dict[key] = address_tag['v']

"""
Please find below a test of the function which presents how the function works.
"""

tree = ET.parse(OSM_FILE)
root = tree.getroot()
test_tags = root.findall("./node/tag")

test_address_tags_dict = {}

for tag in test_tags:
    tag_dict = {}
    if tag.attrib['k'].startswith("addr:"):
        clean_street_type(tag.attrib,tag_dict)
    if tag_dict:
        test_address_tags_dict[tag.attrib['k'] + tag.attrib['v']] = tag_dict

def take(n, iterable):
    "Return first n items of the iterable as a list"
    return list(islice(iterable, n))

pp.pprint(take(20, test_address_tags_dict.iteritems()))


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Phone numbers need to be cleaned to match the following pattern (+33 X XX XX XX XX). We will clean that street type when shapping our data structure.</br>
Here are the different case we want to treat:
-'XX XX XX XX XX'
-'XX.XX.XX.XX.XX'
-'XXXXXXXXXX'
Please find below the function we'll use to clean phone numbers.
Can you write Python code for it?

#classic phone number format in France (ie: 01 30 55 84 22)
classic_france = re.compile(r'\d\d\s\d\d\s\d\d\s\d\d\s\d\d')
#classic phone number format with dots in France (ie: 01.30.55.84.22)
classic_france_dot = re.compile(r'\d\d\.\d\d\.\d\d\.\d\d\.\d\d')
#compressed phone number format in France (ie: 0130558422)
classic_france_compiled = re.compile(r'\d\d\d\d\d\d\d\d\d\d')
#wrong format

def clean_phone_numbers(phone_tag,main_dict):
    """
    This function aims to clean phone tags and add them as a key / value pair of our node dictionary.
    The function will take the phone tag and the main dictionary as arguments:
        phone_tag: every phone sub tag of a "node" or "way" top level tag.
        main_dict: a dictionnary where we store key / value pair of each elements of our map.
    The function first identifies if the phone number follow one of the wrong pattern we identified during our audit.
    Using regex, we try to match every pattern and apply the necessary modifications when it's relevant and then store the phone number in the the dictionary.
    Else, we store directly the phone number in the dictionary.
    """
    if classic_france.match(phone_tag['v']):
        value = re.sub(r'^\d', '+33 ', phone_tag['v'])
        main_dict[phone_tag['k']] = value
    elif classic_france_dot.match(phone_tag['v']):
        value = re.sub(r'^\d', '+33 ', phone_tag['v'])
        value = re.sub(r'\.', ' ', value)
        main_dict[phone_tag['k']] = value
    elif classic_france_compiled.match(phone_tag['v']):
        value = " ".join(phone_tag['v'][i:i+2] for i in range(0, len(phone_tag['v']), 2))
        value = re.sub(r'^\d', '+33 ', value)
        main_dict[phone_tag['k']] = value
    else:
        main_dict[phone_tag['k']] = phone_tag['v']

"""
Please find below a test of the function which presents how the function works.
"""

test_phone_tags_dict = {}

for tag in test_tags:
    tag_dict = {}
    if tag.attrib['k'] == "phone":
        clean_phone_numbers(tag.attrib,tag_dict)
    if tag_dict:
       ...


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Postcode are already clean.
Audit phone numbers
Can you write Python code for it?

# Create a list to store weird phone numbers
phone_numbers = []
# Create a regular expression to isolate
phone_re = re.compile(r'\+\d\d\s\d\s\d\d\s\d\d\s\d\d\s\d\d', re.IGNORECASE)

def audit_phone(phone_numbers,phone_number):
    """
    This function aims to check if the phone number matchs the expected format.
    If it doesn't, the phone number is added to a dictionary.
    Arg1: phone_numbers >> the dictionary where to store the unexpected data.
    Arg2: phone_number >> the phone number to be audited.
    """
    m = phone_re.search(phone_number)
    if m == None:
        phone_numbers.append(phone_number)

def audit_phone_map(file_in, phone_numbers, pretty = False):
    """
    This function aims to audit the file by isolating one by one the phone tags key/value pair.
    The value of each pair is then audited thanks to the audit_phone function.
    """
    for _, element in ET.iterparse(file_in):
        if element.tag == "way" or element.tag == "node":
            for tag in element.iter("tag"):
                if tag.attrib['k'] == "phone":
                    audit_phone(phone_numbers,tag.attrib["v"])
    return phone_numbers

audit_phone_map("plaisir.osm",phone_numbers, True)


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Querying the MongoDb
Can you write Python code for it?

def make_group_pipeline(type_node):
    pipeline = [{'$group':{'_id':type_node,'count':{'$sum':1}}},
                {'$sort':{'count':-1}},
                {'$limit' : 5 }
    ]
    return pipeline

def aggregate(db, pipeline):
    return [doc for doc in db.aggregate(pipeline)]

pipeline = make_group_pipeline('$types.type')
result = aggregate(db.plaisir_osm, pipeline)
pp.pprint(result)


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Shape data
Can you write Python code for it?

CREATED = [ "version", "changeset", "timestamp", "user", "uid"]
POS = ["lon","lat"]
BUILDING_TYPES = ["amenity","shop","sport","place","service","building","highway"]

def shape_element(element):
    """
    This function aims to shape every element of our XML file.
    Each top level node of the XML file is reviewed by this function which creates a dedicated dictionary called "node".
    Each node dictionary is then added to the data list. This list will be the file inserted into MongoDb.
    """
    node = {}
    pos = []
    node_refs = []
    created = {}
    address = {}
    types = {}
    if element.tag == "node" or element.tag == "way" :
        types['type'] = element.tag
        if 'lat' in element.attrib.keys() and 'lon' in element.attrib.keys():
            try:
                lat = float(element.attrib['lat'])
                lon = float(element.attrib['lon'])
                pos.insert(0,lat)
                pos.insert(1,lon)
            except:
                pass
        for k, m in element.attrib.items():
            if k not in POS:
                if k in CREATED:
                    created[k] = m
                else:
                    node[k] = m

        for child in element:
            if child.tag == "nd":
                node_refs.append(child.attrib['ref'])
            elif child.tag == "tag":
                if child.attrib['k'].startswith("addr:"):
                    clean_street_type(child.attrib,address)
                elif child.attrib['k'] == 'phone':
                    clean_phone_numbers(child.attrib,node)
                elif child.attrib['k'] in BUILDING_TYPES:
                    types[child.attrib['k']] = child.attrib['v']
        if types:
            node['types'] = types
        if created:
            node['created'] = created
        if pos:
            node['pos'] = pos
        if address:
            node['address'] = address
        if node_refs:
            node['node_refs'] = node_refs
        return node
    else:
        return None

def...


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
Tennis is the sport with the most facilities in Plaisir.
Are there more restaurant or Fastfood in Plaisir?
Can you write Python code for it?

pipeline_restaurant = [{'$match': {"$or" : [{"types.amenity": "restaurant"},{"types.amenity":"fast_food"}]}},
                       {'$group':{'_id':'$types.amenity','count':{'$sum':1}}},
                       {'$sort':{'count':-1}}]
result_restaurant = aggregate(db.plaisir_osm, pipeline_restaurant)
pp.pprint(result_restaurant)


I am working on the file "P3/OSMProject - Plaisir.ipynb".
The first task is:
There are more restaurant than fast_food in Plaisir.. Good news.....
Is there a correlation between the number of amenity and the number of bus stop in a given area?
Can you write Python code for it?

#check of the geospacial index - the few lines below are here to check that the 2d index is properly working.
for doc in db.plaisir_osm.find({'pos': {'$near' : [48.5,1.95]}}):
    pp.pprint(doc)
    break

#Get max and min latitude and longitude

for doc in db.plaisir_osm.aggregate([
    { "$unwind": "$pos" },
    { "$group": {
        "_id": "$_id",
        "lat": { "$first": "$pos" },
        "lon": { "$last": "$pos" }
    }},
    { "$group": {
        "_id": "null",
        "minLat": { "$min": "$lat" },
        "minLon": { "$min": "$lon" },
        "maxLat": { "$max": "$lat" },
        "maxLon": { "$max": "$lon" }
    }}
]):
    pp.pprint(doc)


I am working on the file "P3/P3.ipynb".
The first task is:
3. A Quick Peak into Database
Now, we have everything inside our Mongo database, we are ready to do data audit and cleaning.
let get some basic statistic about our database collection.
Can you write Python code for it?

client = MongoClient()
db = client.osm
sydney = db.sydney


I am working on the file "P3/P3.ipynb".
The first task is:
4. Data Audit and Cleaning
After some exploration of the datasets, I found three problems associated with the data.
* Invalid and Inconsistent Postcode
* Invalid Street Number
* Inconsistent (non-standard) highway attribute

Problem #1 - Invalid and Inconsistent Postcode
Sydney is the capital city of New South Wales(NSW). According to Wikipedia, NSW's postcodes falls in the ranges of,
* 2000—2599
* 2620—2899
* 2921—2999
We will define a function that returns the entries in the database whose postcode, if exists, does not follow the rules described earlier. We call the function, and shows the first five results.
Can you write Python code for it?

def entries_with_unexpected_postcode():
    postcodes = sydney.find(
        {'$and': [
            {'properties.addr:postcode': {'$not': re.compile('^2[0-5][0-9][0-9]$') }}, # 2000—2599
            {'properties.addr:postcode': {'$not': re.compile('^2[6-8][2-9][0-9]$') }}, # 2620—2899
            {'properties.addr:postcode': {'$not': re.compile('^29[2-9][1-9]$') }}, # 2921—2999
            {'properties.addr:postcode': {'$exists': 1}}
        ]},
        { 'properties.addr:postcode': 1 }
    )
    return list(postcodes)

pprint(entries_with_unexpected_postcode()[0:5])


I am working on the file "P3/P3.ipynb".
The first task is:
First, let's take a look at the data in its GeoJSON format.
Can you write Python code for it?

pprint(sydney.find_one(
    { 'id': { '$regex': 'way*' } },
    { 'geometry': 0 } # hide geometry for better display.
))


I am working on the file "P3/P3.ipynb".
The first task is:
I will convert the XML file into GeoJSON format then import it to a mongo database. It's should be much easier to do various data wrangling tasks inside mongo as we can leverage on various its advanced tools.
For converting to GeoJSON, I will be using the command-line inference of osmtogeojson(https://github.com/tyrasd/osmtogeojson) library by tysasd.
Challenge #1. V8 memory limit
According to osmtogeojson's documentation, I can convert any OSM XML file to GeoJson format by running the following in the command line.

osmtogeojson sydney_australia.osm > sydney_australia.geojson

Here, I encounter the first problem. When I run the above command, I got a Error on NodeJS memory limit. After some googling, I found that, by default, the V8 engine can't utilise all spare memory on the a machine. The engine limits the memory usage to somewhere between 1GB to 1.76GB (didn't spent too much time to dig up the exact number from offical source).
Lucky, the solution to the problem is documentated in the osmtogeojson's documentation. You can increase this memory limit by running

node --max_old_space_size=4096 ./node_modules/osmtogeojson/osmtogeojson sydney_australia.osm > sydney_australia.geojson

After running this, I successful converted the .osm file to geojson.
Challenge #2. MongoDB document size limit
Now I got my geojson, it's time to import it into MongoDB. After reading the MongoDB documentation, I know I can do this by using the mongoimport utilities by running this command:

mongoimport --db osm --collection sydney --file "sydney_australia.geojson"

This time I got an Error on hitting MongoDB's document size limit. Again, I turn to google for help and this blog post pointed to the right direction. So, the problem is, in Mongo, a single document can't be larger than 16MB. And GeoJSON wraps everything inside { "type": "FeatureCollection","features": [...] }. And because of this, Mongo is treating the entire geojson as one big document.
After some trial and error, I found the solution is to extract the...
Can you write Python code for it?

# convert xml to geojson
cmd_convert = [
    'node', '--max_old_space_size=4096', './node_modules/osmtogeojson/osmtogeojson', './data/sydney_australia.osm'
]

geojson = subprocess.check_output(cmd_convert, shell=True)

# extract features
with open('./data/sydney_australia.geojson', 'r+') as f:
    geojson = json.loads(geojson)
    json.dump(geojson['features'], f)

# reset mongodb
cmd_resetDB = ['mongo', 'osm', '--eval', 'db.dropDatabase()']
subprocess.call(cmd_resetDB)

# import geojson to mongodb
cmd_import = [
    'mongoimport', '--db', 'osm', '--collection', 'sydney', '--file',
    './data/sydney_australia.geojson', '--jsonArray'
]
subprocess.call(cmd_import)


I am working on the file "P3/P3.ipynb".
The first task is:
Looks like, this is an footpath inside a building
Can you write Python code for it?


I am working on the file "P3/P3.ipynb".
The first task is:
Number of nodes
Can you write Python code for it?

number_of_nodes = sydney.find({'id': {'$regex': 'node*'} }).count()
print number_of_nodes


I am working on the file "P3/P3.ipynb".
The first task is:
Number of ways
Can you write Python code for it?

number_of_ways = sydney.find({ 'id': {'$regex': 'way*'} }).count()
print number_of_ways


I am working on the file "P3/P3.ipynb".
The first task is:
Problem #2: Invalid Street Number
There are some also invalid street numbers in the database. For the vast majority entries, the street number, if exists, is either a number or a range of number. However, invalid street number does exist, as show in the query result below.
Can you write Python code for it?

invalid_street_number = list(sydney.find(
    {'$and': [
        {'properties.addr:housenumber': { '$not': re.compile('[0-9]+(-[0-9]+)?') } },
        {'properties.addr:housenumber': { '$exists': 1 } }
    ]},
    {'properties.addr:housenumber': 1 }
))
pprint(invalid_street_number)


I am working on the file "P3/P3.ipynb".
The first task is:
The above set includes all of the non standard highway types. Let take a closer look at the first one, Corridor.
Can you write Python code for it?

corridors = gpd.GeoDataFrame(list(sydney.find({'properties.highway': "corridor"})))
corridors['geometry'] = corridors['geometry'].apply(lambda x: shape(x))
corridors


I am working on the file "P3/P3.ipynb".
The first task is:
The rest of the postcodes are just invalid entries. Let's remove those postcodes from our database.
Can you write Python code for it?

for entries in entries_with_unexpected_postcode():
    sydney.update_one(
        { '_id': entries['_id'] },
        { '$unset': {'properties.addr:postcode': ''} }
    )


I am working on the file "P3/P3.ipynb".
The first task is:
Those are actually the corridors inside the airport terminal.
Can you write Python code for it?


I am working on the file "P3/P3.ipynb".
The first task is:
With this list of road type appeared in the dataset, I then compared it against a list of standard highway types I pulled mannually from OSM documentation and print out a list of highway type that is not listed in OSM documentation.
Can you write Python code for it?

road_types = map(lambda x: x['_id'], roads)

# this is a list I pulled manually from OSM document on map feature
osm_highway_types = set([
    # way
    "motorway", "trunk", "primary", "secondary", "tertiary", "unclassified", "residential", "service",
    "motorway_link", "trunk_link", "primary_link", "secondary_link", "tertiary_link", "living_street", "pedestrian",
    "track", "bus_guideway", "escape", "raceway", "road", "footway", "bridleway", "steps", "path", "cycleway", "crossing"
    # area
    "rest_area", "services",
    # lifecycle
    "proposed", "construction"
])

set(road_types) - osm_highway_types # non-standard highway types


I am working on the file "P3/Wrangle OpenStreetMap Data.ipynb".
The first task is:
AKC seems to like coffee a lot. What else do they like?
Can you write Python code for it?

#Find nodes where user is AKC, then group and count the cuisines (where there is a user) then sort
pipeline = [{"$match" : {"created.user":"AKC"}},
            {"$match" : {"cuisine":{"$exists":{"$ne":"null"}}}},
            {"$group": { "_id": "$cuisine", "total" : {"$sum":1}}},
            {"$sort": { "total": -1}}]

results = list(db.maps.aggregate(pipeline))

for result in results:
     print '%s:%s' % (result['_id'],result['total'])


I am working on the file "P3/Wrangle OpenStreetMap Data.ipynb".
The first task is:
Coffee shops seem to be the most popular cuisine added.
What users have added the most kind of coffee shops?
Who likes coffee the most? Let's find out!
Can you write Python code for it?

#Find nodes where cuisine is coffee_shop, then group and count the users (where there is a user) then sort
pipeline = [{"$match" : {"cuisine":"coffee_shop"}},
            {"$match" : {"created.user":{"$exists":{"$ne":"null"}}}},
            {"$group": { "_id": "$created.user", "total" : {"$sum":1}}},
            {"$sort": { "total": -1}}]

results = list(db.maps.aggregate(pipeline))

print '\nTop 5 coffee lovers'

i = 0
for result in results:
    if i < 5:
        print '%s:%s' % (result['_id'],result['total'])
    else:
        break
    i += 1


I am working on the file "P3/Wrangle OpenStreetMap Data.ipynb".
The first task is:
From the first run of this, here are the first 5 results:
```
+#-###-###-#### : 135
(###) ###-#### : 47
+# ### ### #### : 25
-###-#### : 21
(###)###-#### : 10
```
I will use the first format as a template and put the rest into that format.
From my knowledge of Canadian numbers there are 10 digits, not including the country code. If the country code is not included, it is assumed to be 1.
All the digits will be extracted (in order), then put into the correct format.
The code for cleaning is:
```python
if('phone' in d and not phoneNumber.match(d['phone'])):
    #The phone number with nothing but digits
    stripped = re.sub('[^0-9]', '', d['phone'])

    #The number should be of length 10 or 11
    #If it is 10, add a 1 to the start
    if(len(stripped) == 10):
        stripped = '1' + stripped

    #Put into the correct format +#-###-###-####
    if(len(stripped) == 11):
        d['phone'] = '+' + stripped[0] \
                     + '-' + stripped[1:4] \
                     + '-' + stripped[4:7] \
                     + '-' + stripped[7:11]

```
and the regex phoneNumber is:
python
phoneNumber = re.compile(r'^[+][0-9][-][0-9]{3}[-][0-9]{3}[-][0-9]{4}$')
Now the results of the phone number query are:
```
+#-###-###-#### : 277
-###-#### ext ## : 1
```
Much better. For more improvement, figure out a format for extensions.
Postal Codes
Check the format of the postal code, and make sure all addresses are using postal codes of the same (most common) format.
Can you write Python code for it?

formatPipeline = [{"$match" : {"address.post_format":{"$exists":{"$ne":"null"}}}},
                  {"$group": { "_id": "$address.post_format", "total" : {"$sum":1}}},
                  {"$sort": { "total": -1}}]

results = list(db.maps.aggregate(formatPipeline))

for r in results:

    print '%s : %d' % (r['_id'],r['total'])


I am working on the file "P3/Wrangle OpenStreetMap Data.ipynb".
The first task is:
From the first run:
```
U#U #U# : 1112
U#U#U# : 48
UU U#U #U# : 2
U#U #U#  : 2
U#U-#U# : 1
U#U#U#;U#U #U# : 1
-###-#### : 1
: 1
U#U : 1
```
Not so bad. Some of these just need to be thrown away. The only thing acceptable is (without punctuation or spaces is) L#L#L#.
Anything without 3 numbers and 3 letters not in that order is not valid.
The following code was added to clean the postal codes:
```python
if ('address' in d \
        and 'postcode' in d['address'] \
        and not postCode.match(d['address']['postcode'])):
    #Remove everything but letter and numbers. Make it upper case
    stripped = re.sub('[^0-9A-Za-z]', '', d['address']['postcode']).upper()

    #Check if the stripped (only letters and numbers) post code is valid
    #Drop it if it isn't
    if(postCodeStripped.match(stripped)):
        d['address']['postcode'] = stripped[0:3] + " " + stripped[3:]
    else:
        d['address'].pop("postcode", None)

```
Using the regex's:
```python
postCode = re.compile(r'^[A-Z][0-9][A-Z][\s][0-9][A-Z][0-9]$')
postCodeStripped = re.compile(r'^[A-Z][0-9][A-Z][0-9][A-Z][0-9]$')
```
After the cleaning code was added this is the output of the audit:
U#U #U# : 1163
Good. That's better.
Analysis
Okay. The data is loaded into the database and cleaned (kind of). Now to run some queries to answer my original questions.
What does the map data look like? Size of file, number of elements, number of nodes, number of ways and number of unique users.
Answered using the following queries:
Can you write Python code for it?

#Use the collstats fucntion to get the file size and nunmber of elemements
stats = db.command("collstats", "maps")

total = stats['count']
size = stats['storageSize']

#Use the count function to get the amount of each type
nodes = db.maps.find({'type':'node'}).count()
ways = db.maps.find({'type':'way'}).count()

#Find the unique users query
#Group by users, then group and count the results.
pipeline = [{"$group": { "_id": "$created.user", "total" : {"$sum":1}}},
            {"$group": { "_id": "null", "total" : {"$sum":1}}}]

results = list(db.maps.aggregate(pipeline))

uniqueUsers = results[0]['total']

print "\nData Summary"
print "=================================="
print "Storage Size: %d bytes" % size
print "Total number of elements: %d " % total
print "Total number of nodes: %d " % nodes
print "Total number of ways: %d " % ways
print "Total number of unique users: %d " % uniqueUsers


I am working on the file "P3/Wrangle OpenStreetMap Data.ipynb".
The first task is:
To start with, running this code on the full dataset I got (the output changed after adding cleaning logic):
Places with cafe as amenity: 190
Places with coffee_shop as cuisine: 78
Places with cafe as amenity but not coffee_shop as cuisine: 121
Places with coffee_shop as cuisine but not cafe as amenity: 9
To clean this up, I want all coffee shops to look the same. To do this I decided that I would make it so all cafe's have the cuisine coffee_shop and from here on, to use the field, 'cuisine' to to the research.
The follow code is included in the code for cleaning a document:
```python
if('amenity' in d and d['amenity'] == 'cafe'):
        d['cuisine'] = 'coffee_shop'
```
After adding this code, the output for the 'Coffee Audit' was:
Places with cafe as amenity: 190
Places with coffee_shop as cuisine: 199
Places with cafe as amenity but not coffee_shop as cuisine: 0
Places with coffee_shop as cuisine but not cafe as amenity: 9
Now {'cuisine' : 'coffee_shop'} includes all the coffee shops.
Phone Numbers
Check the format of the phone numbers, pick a standard (the most used maybe) then convert all to that standard.
During the clean stage, add a field to all documents that has the format of the number. D is digit Eg (###) ###-####
Code in cleanDocument:
python
if('phone' in d):
        d['phone_format'] = re.sub('[0-9]', '#', d['phone'])
        print d['phone_format']
Now run a query to see the common formats.
Can you write Python code for it?

#Query all documents with something in the 'phone_format' field, group by the phone format,
#count and sort by the number in each format

formatPipeline = [{"$match" : {"phone_format":{"$exists":{"$ne":"null"}}}},
                  {"$group": { "_id": "$phone_format", "total" : {"$sum":1}}},
                  {"$sort": { "total": -1}}]

results = list(db.maps.aggregate(formatPipeline))

for r in results:

    print '%s : %d' % (r['_id'],r['total'])


I am working on the file "P3/Wrangle OpenStreetMap Data.ipynb".
The first task is:
Where does coffee shop rank as an cuisine?
Answered using the following query
Can you write Python code for it?

#Query how many non null cuisines sorted
pipeline = [{"$match" : {"cuisine":{"$exists":{"$ne":"null"}}}},
            {"$group": { "_id": "$cuisine", "total" : {"$sum":1}}},
            {"$sort": { "total": -1}}]

results = list(db.maps.aggregate(pipeline))

print '\nTop 10 cuisine types'
i = 0
for result in results:
    if i < 10:
        print '%s:%s' % (result['_id'],result['total'])
    else:
        break
    i += 1


I am working on the file "P3/Wrangle OpenStreetMap Data.ipynb".
The first task is:
While converting the data, 2 kinds of cleaning will happen.

Pre-processing: This happens before data is converted to json. Mainly, it is making consistent and valid keys.
Post-processing: Once an element is converted to JSON, it will be cleaned according to different rules.

Pre-processing Cleaning
To start, the following rules were applied.

'created' index for elements. Elements have attributes related to their creation ("version", "changeset", "timestamp", "user", "uid"). These are combined into a sub-document with the index 'created'

'pos' index. For elements with lat and lon (latitude and longitude) attributes, they will be combined into an float list (2 elements) and given the index of 'pos'.

Valid keys. Elements have sub elements called 'tags'. Each tag has a key (k) and value (v). Before changing to json the keys need to be checked. Keys can only have lowercase letter or an _. For tags with a single ':', a sub-document is created with the string before the ':' being the key in the main document and string after the ':' being the key in the sub-document. Address is a special. There are records with addr:<something>, and instead of using addr for the key, address will be used.

Applying these rules, 15289 tags were dropped (pre_process_log_v0.txt). Here is a sample:
Bad tag key, not added: geobase:datasetName
Bad tag key, not added: geobase:acquisitionTechnique
Bad tag key, not added: geobase:datasetName
Bad tag key, not added: geobase:acquisitionTechnique
Bad tag key, not added: catmp-RoadID
Bad tag key, not added: catmp-RoadID
Bad tag key, not added: catmp-RoadID
Tags with keys like geobase:acquisitionTechnique and geobase:datasetName were being dropped because they had a capital letter. These appear to be using a naming standard that separates words using capital letters to start words. I added logic to instead use _ to divide these words. For example, datasetName becomes dataset_name.
I added code to replace - with _ and to make all words lower cases (after _ were added where...
Can you write Python code for it?

client = MongoClient('localhost', 27017)
db = client.p3


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Answer:

 In the above table looking at **Segment 0** we can see from the median difference that this segment only majorly corresponds to buyers of larger than the meadian quantity of `"Fresh"` and `"Frozen"` products perhaps a **_Restaurants or Café_**. Although the mean difference table suggests otherwise, a closer look at the lower number of standard deviations the predicted value is from the mean when compared to the values of other features confirms our earlier intuition.

Similarly, **Segment 1** has significantly higher mean and median difference in `"`Milk"`, `"Grocery"` &  `"Detergents_paper"` perhaps representing **_Retailers_** regularly purchasing these goods in larger quantities when compared to the other goods.

Question 9
For each sample point, which customer segment from Question 8 best represents it? Are the predictions for each sample point consistent with this?
Run the code block below to find which cluster each sample point is predicted to be.
Can you write Python code for it?

# Display the predictions
for i, pred in enumerate(sample_preds):
    print "Sample point", i, "predicted to be in Cluster", pred


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Answer:

The above tables helps compare the samples with the descriptive statistics measures of mean and median of the data.

The first sample establishment buys significantly more `"Fresh"`,`"Grocery"`,and `"Detergent_Paper"` than the mean of the data, it also seems to be larger buyer than the median in all categories of products. Essentially it seems like large buyer of all kinds of products. These numbers suggest that the sample is perhaps a **_Retailer_**.

The second sample establishment buys significantly more `"Fresh"` products and lesser of all the other categories when compared to mean. Further, in comparison to the median it seems to buy more of `"Fresh"`, `"Frozen"` and `"Delicatessen"` products and less of `"Milk"`, `"Grocery"` and `"Detergent_papers"`. Using these observations it seems right to say that this sample is a **_Restarant_**

The third establishment buys more `"Frozen"` goods and lesser of all other products when compared to the mean. The same can be said in comparison with the `"median"` However, the spend on `"delicatessen"` goods is close to the median. These factors are indicative of a small scale **_Café_**.

The heatmap below of the percentile ranks of samples' category spending further helps visualize the amount of spending in each category in comparison with other
Implementation: Feature Relevance
One interesting thought to consider is if one (or more) of the six product categories is actually relevant for understanding customer purchasing. That is to say, is it possible to determine whether customers purchasing some amount of one category of products will necessarily purchase some proportional amount of another category of products? We can make this determination quite easily by training a supervised regression learner on a subset of the data with one feature removed, and then score how well that model can predict the removed feature.
In the code block below, you will need to implement the following:
 - Assign new_data a...
Can you write Python code for it?

# TODO: Make a copy of the DataFrame, using the 'drop' function to drop the given feature
new_data = data.drop('Fresh',axis=1)

# TODO: Split the data into training and testing sets using the given feature as the target
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(new_data,data['Fresh'],test_size=0.25,random_state=42)

# TODO: Create a decision tree regressor and fit it to the training set
from sklearn.tree import DecisionTreeRegressor
regressor = DecisionTreeRegressor(random_state=42)

# TODO: Report the score of the prediction using the testing set
regressor.fit(X_train,y_train)
score = regressor.score(X_test,y_test)

print score


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Answer:

The sample point 1 has features `"Milk"` `"Grocery"` and `"Detergents_paper"` with values above the mean and the median. The prediction of the segment according to our initial intuition that sample is likely a **Retailer**

The other two sample points have the features `"Fresh"` and `"Frozen"` with values above the mean and all other features with values below the mean. Predicting this as belonging to **Segment 0**, most likely a **Restaurant/Café** again is congruent with our the initial guess and consistent with the characteristics of the Segment.

Conclusion
Question 10
Companies often run A/B tests when making small changes to their products or services. If the wholesale distributor wanted to change its delivery service from 5 days a week to 3 days a week, how would you use the structure of the data to help them decide on a group of customers to test?
Hint: Would such a change in the delivery service affect all customers equally? How could the distributor identify who it affects the most?
Answer:

Since, the intention of A/B test here is to verify the effectiveness of changing the delivery service from 5 days a week to 3 days a week. The wholesaler must be able to explore, experiment and test the strategy without the risk of negatively impacting the entire business. Customers belonging to **Segment 0** are mostly large buyers of fresh products, the new strategy is likely to have a negative impact on this segment since fresh products are purchased more frequently and are perishable.

If one were to recommend a group to the distributor purely based on the segments derived above. It would be appropriate for the distributor to conduct the A/B test on **Segment 1**  and test out the effectiveness of the strategy on them. Essentilay split a subset of **Segment 1** customers into "A/control" and "B/treatment" groups to detect a difference without even disturbing the **Segment 0**. This is more likely to not have an negative impact on the customer base. Once proven to...
Can you write Python code for it?

# Display the clustering results based on 'Channel' data
rs.channel_results(reduced_data, outliers, pca_samples)


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Answer:

Yes, from the scatter matrix aand heatmap above we can observe that the following pair of features exhibit a greater that 0.5 correlation:

`"Milk"` - `"Detergents_Paper"`

`"Milk"` - `"Grocery"`

`"Grocery"` - `"Detergent_Paper"`

The lack of correlation with other features confirms our hypothesis that the feature `"Fresh"` is relevant in identifying a customer's spending habits. It's not significantly correlated to any if not most of the other features and provides new information to a learning algorithm that is otherwise not present in the other features. Hence it cannot be removed without affecting the performance of a learning algorithm.

Most of the data for the features lie in a (right) positively skewed distribution. From the distribution it is quite evident that lower values are more common than higher values. i.e. most customers purchase in the lower monetary units.

Data Preprocessing
In this section, you will preprocess the data to create a better representation of customers by performing a scaling on the data and detecting (and optionally removing) outliers. Preprocessing data is often times a critical step in assuring that results you obtain from your analysis are significant and meaningful.
Implementation: Feature Scaling
If data is not normally distributed, especially if the mean and median vary significantly (indicating a large skew), it is most often appropriate to apply a non-linear scaling — particularly for financial data. One way to achieve this scaling is by using a Box-Cox test, which calculates the best power transformation of the data that reduces skewness. A simpler approach which can work in most cases would be applying the natural logarithm.
In the code block below, you will need to implement the following:
 - Assign a copy of the data to log_data after applying a logarithm scaling. Use the np.log function for this.
 - Assign a copy of the sample data to log_samples after applying a logrithm scaling. Again, use np.log.
Can you write Python code for it?

# TODO: Scale the data using the natural logarithm
log_data = np.log(data)

# TODO: Scale the sample data using the natural logarithm
log_samples = np.log(samples)

# Produce a scatter matrix for each pair of newly-transformed features
pd.scatter_matrix(log_data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Data Exploration
In this section, you will begin exploring the data through visualizations and code to understand how each feature is related to the others. You will observe a statistical description of the dataset, consider the relevance of each feature, and select a few sample data points from the dataset which you will track through the course of this project.
Run the code block below to observe a statistical description of the dataset. Note that the dataset is composed of six important product categories: 'Fresh', 'Milk', 'Grocery', 'Frozen', 'Detergents_Paper', and 'Delicatessen'. Consider what each category represents in terms of products you could purchase.
Can you write Python code for it?

# Display a description of the dataset
display(data.describe())


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Implementation: Dimensionality Reduction
When using principal component analysis, one of the main goals is to reduce the dimensionality of the data — in effect, reducing the complexity of the problem. Dimensionality reduction comes at a cost: Fewer dimensions used implies less of the total variance in the data is being explained. Because of this, the cumulative explained variance ratio is extremely important for knowing how many dimensions are necessary for the problem. Additionally, if a signifiant amount of variance is explained by only two or three dimensions, the reduced data can be visualized afterwards.
In the code block below, you will need to implement the following:
 - Assign the results of fitting PCA in two dimensions with good_data to pca.
 - Apply a PCA transformation of good_data using pca.transform, and assign the reuslts to reduced_data.
 - Apply a PCA transformation of the sample log-data log_samples using pca.transform, and assign the results to pca_samples.
Can you write Python code for it?

# TODO: Fit PCA to the good data using only two dimensions
pca = PCA(n_components=2)
pca.fit(good_data)

# TODO: Apply a PCA transformation the good data
reduced_data = pca.transform(good_data)

# TODO: Apply a PCA transformation to the sample log-data
pca_samples = pca.transform(log_samples)

# Create a DataFrame for the reduced data
reduced_data = pd.DataFrame(reduced_data, columns = ['Dimension 1', 'Dimension 2'])


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Implementation: Outlier Detection
Detecting outliers in the data is extremely important in the data preprocessing step of any analysis. The presence of outliers can often skew results which take into consideration these data points. There are many "rules of thumb" for what constitutes an outlier in a dataset. Here, we will use Tukey's Method for identfying outliers: An outlier step is calculated as 1.5 times the interquartile range (IQR). A data point with a feature that is beyond an outlier step outside of the IQR for that feature is considered abnormal.
In the code block below, you will need to implement the following:
 - Assign the value of the 25th percentile for the given feature to Q1. Use np.percentile for this.
 - Assign the value of the 75th percentile for the given feature to Q3. Again, use np.percentile.
 - Assign the calculation of an outlier step for the given feature to step.
 - Optionally remove data points from the dataset by adding indices to the outliers list.
NOTE: If you choose to remove any outliers, ensure that the sample data does not contain any of these points!
Once you have performed this implementation, the dataset will be stored in the variable good_data.
Can you write Python code for it?

# For each feature find the data points with extreme high or low values
for feature in log_data.keys():

    # TODO: Calculate Q1 (25th percentile of the data) for the given feature
    Q1 = np.percentile(log_data[feature],q=25)

    # TODO: Calculate Q3 (75th percentile of the data) for the given feature
    Q3 = np.percentile(log_data[feature],q=75)

    # TODO: Use the interquartile range to calculate an outlier step (1.5 times the interquartile range)
    step = (Q3-Q1)*1.5

    # Display the outliers
    print "Data points considered outliers for the feature '{}':".format(feature)
    display(log_data[~((log_data[feature] >= Q1 - step) & (log_data[feature] <= Q3 + step))])

# OPTIONAL: Select the indices for data points you wish to remove [66,95,75,109,128,142,187,218,338]
outliers = [val for sublist in
            [log_data[~((log_data[feature] >= np.percentile(log_data[feature],q=25) -
                         ((np.percentile(log_data[feature],q=75) -
                           np.percentile(log_data[feature],q=25))*1.5)) &
                        (log_data[feature] <= np.percentile(log_data[feature],q=75) +
                         ((np.percentile(log_data[feature],q=75) -
                           np.percentile(log_data[feature],q=25))*1.5)))].index.values
             for feature in log_data.keys()] for val in sublist]
outliers = list(set([x for x in outliers if outliers.count(x)>1]))

print outliers

# Remove the outliers, if any were specified
good_data = log_data.drop(log_data.index[outliers]).reset_index(drop = True)


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Implementation: Selecting Samples
To get a better understanding of the customers and how their data will transform through the analysis, it would be best to select a few sample data points and explore them in more detail. In the code block below, add three indices of your choice to the indices list which will represent the customers to track. It is suggested to try different sets of samples until you obtain customers that vary significantly from one another.
Can you write Python code for it?

# TODO: Select three indices of your choice you wish to sample from the dataset
indices = [13,120,390]

# Create a DataFrame of the chosen samples
samples = pd.DataFrame(data.loc[indices], columns = data.keys()).reset_index(drop = True)
print "Chosen samples of wholesale customers dataset:"
display(samples)


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Observation
After applying a natural logarithm scaling to the data, the distribution of each feature should appear much more normal. For any pairs of features you may have identified earlier as being correlated, observe here whether that correlation is still present (and whether it is now stronger or weaker than before).
Run the code below to see how the sample data has changed after having the natural logarithm applied to it.
Can you write Python code for it?

# Display the log-transformed sample data
display(log_samples)


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Question 2
Which feature did you attempt to predict? What was the reported prediction score? Is this feature is necessary for identifying customers' spending habits?
Hint: The coefficient of determination, R^2, is scored between 0 and 1, with 1 being a perfect fit. A negative R^2 implies the model fails to fit the data.
Answer:
<p>
The Decision tree regressor reported a prediction score of **-0.38** for the `"Fresh"` goods. The negative coefficient of determination implies that there is no relevant information in the other features to replace the information present in `"Fresh"`. Since, it's not possible to predict that a customer will purchase `"Fresh"` goods given the other features it adds relevance to the problem of identifying customer spending habits. Hence, it is a relevant feature that adds new information and cannot be ignored while trying to group customers based on their spending habits.
</p>

Visualize Feature Distributions
To get a better understanding of the dataset, we can construct a scatter matrix of each of the six product features present in the data. If you found that the feature you attempted to predict above is relevant for identifying a specific customer, then the scatter matrix below may not show any correlation between that feature and the others. Conversely, if you believe that feature is not relevant for identifying a specific customer, the scatter matrix might show a correlation between that feature and another feature in the data. Run the code block below to produce a scatter matrix.
Can you write Python code for it?

# Produce a scatter matrix for each pair of features in the data
pd.scatter_matrix(data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Question 4
Are there any data points considered outliers for more than one feature? Should these data points be removed from the dataset? If any data points were added to the outliers list to be removed, explain why.
Answer:
<p> Yes, the data points with the indices [128, 65, 66, 75, 154] appear in more than one feature of the dataset. These data points are being removed from the dataset as they are likely to not belong to a segment of customers and might affect the performance of a clustering algorithm. This is because clusering algorithms such as K-Means work to optimize the [**sum of squares**]('http://stackoverflow.com/questions/13989419/removing-outliers-from-a-k-mean-cluster') thus assigning the outliers with more weight and moving the cluster closer toward them.
</p>

Feature Transformation
In this section you will use principal component analysis (PCA) to draw conclusions about the underlying structure of the wholesale customer data. Since using PCA on a dataset calculates the dimensions which best maximize variance, we will find which compound combinations of features best describe customers.
Implementation: PCA
Now that the data has been scaled to a more normal distribution and has had any necessary outliers removed, we can now apply PCA to the good_data to discover which dimensions about the data best maximize the variance of features involved. In addition to finding these dimensions, PCA will also report the explained variance ratio of each dimension — how much variance within the data is explained by that dimension alone. Note that a component (dimension) from PCA can be considered a new "feature" of the space, however it is a composition of the original features present in the data.
In the code block below, you will need to implement the following:
 - Import sklearn.decomposition.PCA and assign the results of fitting PCA in six dimensions with good_data to pca.
 - Apply a PCA transformation of the sample log-data log_samples using pca.transform, and assign the results to pca_samples.
Can you write Python code for it?

# TODO: Apply PCA to the good data with the same number of dimensions as features
from sklearn.decomposition import PCA
pca = PCA()
pca.fit(good_data)

# TODO: Apply a PCA transformation to the sample log-data
pca_samples = pca.transform(log_samples)

# Generate PCA results plot
pca_results = rs.pca_results(good_data, pca)


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Question 5
How much variance in the data is explained in total by the first and second principal component? What about the first four principal components? Using the visualization provided above, discuss what the first four dimensions best represent in terms of customer spending.
Hint: A positive increase in a specific dimension corresponds with an increase of the positive-weighted features and a decrease of the negative-weighted features. The rate of increase or decrease is based on the indivdual feature weights.
Answer:

The first and the second principal components together have an **explained variance ratio** of __0.7093__. This means that together the first and the second principal components contains 70.93% of the variance/information present in the 6 features. Similarly, the first four principal components have a **total explained variance ratio** of __0.9319__ i.e. together the first four principal components explain 93.19% of the variance/information present in the 6 features.

PCA helps transform a large set of features to a smaller subset of Principal Components(PCs). It achieves this be performing two important tasks that are as follows:

By capturing the variance in a feature:

Variance in data is synonymous to the presence of information from a wide variety of observations/samples. The weights of a feature in the above diagram are representative of the variance in the feature. Thus, larger weights for a feature in a (PC) represents more variance is captured for that feature i.e representative a large range of values in the sample of the feature.

By capturing the correlation between features:

As discussed earlier correlation among features represents relationships and patterns among them. The more two features are correlated the more likely it is that they capture similar information. While a strong positive correlation between two features suggests that it's likely for both features to increase or decrease...
Can you write Python code for it?

# Display sample log-data after having a PCA transformation applied
display(pd.DataFrame(np.round(pca_samples, 4), columns = pca_results.index.values))


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Question 6
What are the advantages to using a K-Means clustering algorithm? What are the advantages to using a Gaussian Mixture Model clustering algorithm? Given your observations about the wholesale customer data so far, which of the two algorithms will you use and why?
Answer:

The **K-Means** clustering is an algorithm that splits data into **K** distinct groups (clusters). It makes deterministic assignment of data points to clusters, essentially making a hard assignment. It's relatively simple, robust and fast for large feature sets. It gives best results when the data is distinctly separable.

**Gaussian Mixture Model** on the other hand probabilistically determines the **K** subpopulations without associating the samples to a cluster with certainty. Instead, the clusters are defined by gaussian distributions of unknown parameters. Expectation-Maximization is used to learn the parameters. Each sample is hence assigned a probability of belonging to a cluster.

Since, we would like to identify customer segments using clustering it would be simple to use K-Means clustering. However, since we don't want to make hard assumptions about which segment customers belong to and GMM presumes very little about the structure of the data GMM shall be used in this problem

Implementation: Creating Clusters
Depending on the problem, the number of clusters that you expect to be in the data may already be known. When the number of clusters is not known a priori, there is no guarantee that a given number of clusters best segments the data, since it is unclear what structure exists in the data — if any. However, we can quantify the "goodness" of a clustering by calculating each data point's silhouette coefficient. The silhouette coefficient for a data point measures how similar it is to its assigned cluster from -1 (dissimilar) to 1 (similar). Calculating the mean silhouette coefficient provides for a simple scoring method of a given clustering.
In the code block below, you will need to implement the...
Can you write Python code for it?

# TODO: Apply your clustering algorithm of choice to the reduced data
from sklearn.mixture import GMM
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

for i in range (6,1,-1):
    clusterer = GMM(n_components = i,random_state=42)
    clusterer.fit(reduced_data)

    # TODO: Predict the cluster for each data point
    preds = clusterer.predict(reduced_data)

    # TODO: Find the cluster centers
    centers = clusterer.means_

    # TODO: Predict the cluster for each transformed sample data point
    sample_preds = clusterer.predict(pca_samples)

    # TODO: Calculate the mean silhouette coefficient for the number of clusters chosen
    score = silhouette_score(reduced_data,preds)
    print "clusters = {} : score = {:.3f}\n".format(i,score),


I am working on the file "P3:_creating_customer_segments/.ipynb_checkpoints/customer_segments-checkpoint.ipynb".
The first task is:
Question 8
Consider the total purchase cost of each product category for the representative data points above, and reference the statistical description of the dataset at the beginning of this project. What set of establishments could each of the customer segments represent?
Hint: A customer who is assigned to 'Cluster X' should best identify with the establishments represented by the feature set of 'Segment X'.
Can you write Python code for it?

print "Mean Difference"
display(true_centers - np.round(data.mean()))
print "Median Differences"
display((true_centers - np.round(data.median())))
print "N-stds"
display((true_centers - np.round(data.mean()))/data.std())


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Answer:

Yes, from the scatter matrix aand heatmap above we can observe that the following pair of features exhibit a greater that 0.5 correlation:

`"Milk"` - `"Detergents_Paper"`

`"Milk"` - `"Grocery"`

`"Grocery"` - `"Detergent_Paper"`

The lack of correlation with other features confirms our hypothesis that the feature `"Fresh"` is relevant in identifying a customer's spending habits. It's not significantly correlated to any if not most of the other features and provides new information to a learning algorithm that is otherwise not present in the other features. Hence it cannot be removed without affecting the performance of a learning algorithm.

Most of the data for the features lie in a (right) positively skewed distribution. From the distribution it is quite evident that lower values are more common than higher values. i.e. most customers purchase in the lower monetary units.

Data Preprocessing
In this section, you will preprocess the data to create a better representation of customers by performing a scaling on the data and detecting (and optionally removing) outliers. Preprocessing data is often times a critical step in assuring that results you obtain from your analysis are significant and meaningful.
Implementation: Feature Scaling
If data is not normally distributed, especially if the mean and median vary significantly (indicating a large skew), it is most often appropriate to apply a non-linear scaling — particularly for financial data. One way to achieve this scaling is by using a Box-Cox test, which calculates the best power transformation of the data that reduces skewness. A simpler approach which can work in most cases would be applying the natural logarithm.
In the code block below, you will need to implement the following:
 - Assign a copy of the data to log_data after applying a logarithm scaling. Use the np.log function for this.
 - Assign a copy of the sample data to log_samples after applying a logrithm scaling. Again, use np.log.
Can you write Python code for it?

# TODO: Scale the data using the natural logarithm
log_data = np.log(data)

# TODO: Scale the sample data using the natural logarithm
log_samples = np.log(samples)

# Produce a scatter matrix for each pair of newly-transformed features
idx_reorder = ['Detergents_Paper', 'Grocery', 'Milk', 'Fresh', 'Frozen', 'Delicatessen']
axes = pd.scatter_matrix(log_data[idx_reorder], alpha = 0.3, figsize = (14,8), diagonal = 'kde')
corr = log_data[idx_reorder].corr().as_matrix()

for i, j in zip(*plt.np.triu_indices_from(axes, k=1)):
    axes[i, j].annotate("%+.3f" %corr[i,j], (0.8, 0.2), xycoords='axes fraction', ha='center', va='center',color="red", fontsize=14)


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Data Exploration
In this section, you will begin exploring the data through visualizations and code to understand how each feature is related to the others. You will observe a statistical description of the dataset, consider the relevance of each feature, and select a few sample data points from the dataset which you will track through the course of this project.
Run the code block below to observe a statistical description of the dataset. Note that the dataset is composed of six important product categories: 'Fresh', 'Milk', 'Grocery', 'Frozen', 'Detergents_Paper', and 'Delicatessen'. Consider what each category represents in terms of products you could purchase.
Can you write Python code for it?


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Implementation: Data Recovery
Each cluster present in the visualization above has a central point. These centers (or means) are not specifically data points from the data, but rather the averages of all the data points predicted in the respective clusters. For the problem of creating customer segments, a cluster's center point corresponds to the average customer of that segment. Since the data is currently reduced in dimension and scaled by a logarithm, we can recover the representative customer spending from these data points by applying the inverse transformations.
In the code block below, you will need to implement the following:
 - Apply the inverse transform to centers using pca.inverse_transform and assign the new centers to log_centers.
 - Apply the inverse function of np.log to log_centers using np.exp and assign the true centers to true_centers.
Can you write Python code for it?

# TODO: Inverse transform the centers
log_centers = pca.inverse_transform(centers)

# TODO: Exponentiate the centers
true_centers = np.exp(log_centers)

# Display the true centers
segments = ['Segment {}'.format(i) for i in range(0,len(centers))]
true_centers = pd.DataFrame(np.round(true_centers), columns = data.keys())
true_centers.index = segments
display(true_centers)


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Implementation: Outlier Detection
Detecting outliers in the data is extremely important in the data preprocessing step of any analysis. The presence of outliers can often skew results which take into consideration these data points. There are many "rules of thumb" for what constitutes an outlier in a dataset. Here, we will use Tukey's Method for identfying outliers: An outlier step is calculated as 1.5 times the interquartile range (IQR). A data point with a feature that is beyond an outlier step outside of the IQR for that feature is considered abnormal.
In the code block below, you will need to implement the following:
 - Assign the value of the 25th percentile for the given feature to Q1. Use np.percentile for this.
 - Assign the value of the 75th percentile for the given feature to Q3. Again, use np.percentile.
 - Assign the calculation of an outlier step for the given feature to step.
 - Optionally remove data points from the dataset by adding indices to the outliers list.
NOTE: If you choose to remove any outliers, ensure that the sample data does not contain any of these points!
Once you have performed this implementation, the dataset will be stored in the variable good_data.
Can you write Python code for it?

# For each feature find the data points with extreme high or low values
for feature in log_data.keys():

    # TODO: Calculate Q1 (25th percentile of the data) for the given feature
    Q1 = np.percentile(log_data[feature],q=25)

    # TODO: Calculate Q3 (75th percentile of the data) for the given feature
    Q3 = np.percentile(log_data[feature],q=75)

    # TODO: Use the interquartile range to calculate an outlier step (1.5 times the interquartile range)
    step = (Q3-Q1)*1.5

    # Display the outliers
    #print "Data points considered outliers for the feature '{}':".format(feature)
    #display(log_data[~((log_data[feature] >= Q1 - step) & (log_data[feature] <= Q3 + step))])

# OPTIONAL: Select the indices for data points you wish to remove [66,95,75,109,128,142,187,218,338]
outliers = [val for sublist in
            [log_data[~((log_data[feature] >= np.percentile(log_data[feature],q=25) -
                         ((np.percentile(log_data[feature],q=75) -
                           np.percentile(log_data[feature],q=25))*1.5)) &
                        (log_data[feature] <= np.percentile(log_data[feature],q=75) +
                         ((np.percentile(log_data[feature],q=75) -
                           np.percentile(log_data[feature],q=25))*1.5)))].index.values
             for feature in log_data.keys()] for val in sublist]
outliers = list(set([x for x in outliers if outliers.count(x)>1]))

# Remove the outliers, if any were specified
good_data = log_data.drop(log_data.index[outliers]).reset_index(drop = True)

#visualizing the outliers removed with box-plots.
ax = sns.boxplot(data=log_data)
ax = sns.swarmplot(data=log_data.iloc[outliers], color="red", size=8)


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Observation
After applying a natural logarithm scaling to the data, the distribution of each feature should appear much more normal. For any pairs of features you may have identified earlier as being correlated, observe here whether that correlation is still present (and whether it is now stronger or weaker than before).
Run the code below to see how the sample data has changed after having the natural logarithm applied to it.
Can you write Python code for it?


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Question 2
Which feature did you attempt to predict? What was the reported prediction score? Is this feature is necessary for identifying customers' spending habits?
Hint: The coefficient of determination, R^2, is scored between 0 and 1, with 1 being a perfect fit. A negative R^2 implies the model fails to fit the data.
Answer:
<p>
The Decision tree regressor reported a prediction score of **-0.38** for the `"Fresh"` goods. The negative coefficient of determination implies that there is no relevant information in the other features to replace the information present in `"Fresh"`. Since, it's not possible to predict that a customer will purchase `"Fresh"` goods given the other features it adds relevance to the problem of identifying customer spending habits. Hence, it is a relevant feature that adds new information and cannot be ignored while trying to group customers based on their spending habits.
</p>

Visualize Feature Distributions
To get a better understanding of the dataset, we can construct a scatter matrix of each of the six product features present in the data. If you found that the feature you attempted to predict above is relevant for identifying a specific customer, then the scatter matrix below may not show any correlation between that feature and the others. Conversely, if you believe that feature is not relevant for identifying a specific customer, the scatter matrix might show a correlation between that feature and another feature in the data. Run the code block below to produce a scatter matrix.
Can you write Python code for it?


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Question 3
Are there any pairs of features which exhibit some degree of correlation? Does this confirm or deny your suspicions about the relevance of the feature you attempted to predict? How is the data for those features distributed?
Hint: Is the data normally distributed? Where do most of the data points lie?
Can you write Python code for it?

import seaborn as sns
import pylab as plt
corr = data.corr()
mask = np.zeros_like(corr)
mask[np.triu_indices_from(mask, 1)] = True
with sns.axes_style("white"):
    ax = sns.heatmap(corr, mask=mask, square=True, annot=True, cmap='RdBu_r')
    plt.xticks(rotation=45, ha='center');


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Question 4
Are there any data points considered outliers for more than one feature? Should these data points be removed from the dataset? If any data points were added to the outliers list to be removed, explain why.
Answer:
<p> Yes, the data points with the indices [128, 65, 66, 75, 154] appear in more than one feature of the dataset. These data points are being removed from the dataset as they are likely to not belong to a segment of customers and might affect the performance of a clustering algorithm. This is because clusering algorithms such as K-Means work to optimize the [**sum of squares**]('http://stackoverflow.com/questions/13989419/removing-outliers-from-a-k-mean-cluster') thus assigning the outliers with more weight and moving the cluster closer toward them.
</p>

Feature Transformation
In this section you will use principal component analysis (PCA) to draw conclusions about the underlying structure of the wholesale customer data. Since using PCA on a dataset calculates the dimensions which best maximize variance, we will find which compound combinations of features best describe customers.
Implementation: PCA
Now that the data has been scaled to a more normal distribution and has had any necessary outliers removed, we can now apply PCA to the good_data to discover which dimensions about the data best maximize the variance of features involved. In addition to finding these dimensions, PCA will also report the explained variance ratio of each dimension — how much variance within the data is explained by that dimension alone. Note that a component (dimension) from PCA can be considered a new "feature" of the space, however it is a composition of the original features present in the data.
In the code block below, you will need to implement the following:
 - Import sklearn.decomposition.PCA and assign the results of fitting PCA in six dimensions with good_data to pca.
 - Apply a PCA transformation of the sample log-data log_samples using pca.transform, and assign the results to pca_samples.
Can you write Python code for it?


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Question 6
What are the advantages to using a K-Means clustering algorithm? What are the advantages to using a Gaussian Mixture Model clustering algorithm? Given your observations about the wholesale customer data so far, which of the two algorithms will you use and why?
Answer:

The **K-Means** clustering is an algorithm that splits data into **K** distinct groups (clusters). It makes deterministic assignment of data points to clusters, essentially making a hard assignment. It's relatively simple, robust and fast for large feature sets. It gives best results when the data is distinctly separable.

**Gaussian Mixture Model** on the other hand probabilistically determines the **K** subpopulations without associating the samples to a cluster with certainty. Instead, the clusters are defined by gaussian distributions of unknown parameters. Expectation-Maximization is used to learn the parameters. Each sample is hence assigned a probability of belonging to a cluster.

Since, we would like to identify customer segments using clustering it would be simple to use K-Means clustering. However, since we don't want to make hard assumptions about which segment customers belong to and GMM presumes very little about the structure of the data GMM shall be used in this problem

Implementation: Creating Clusters
Depending on the problem, the number of clusters that you expect to be in the data may already be known. When the number of clusters is not known a priori, there is no guarantee that a given number of clusters best segments the data, since it is unclear what structure exists in the data — if any. However, we can quantify the "goodness" of a clustering by calculating each data point's silhouette coefficient. The silhouette coefficient for a data point measures how similar it is to its assigned cluster from -1 (dissimilar) to 1 (similar). Calculating the mean silhouette coefficient provides for a simple scoring method of a given clustering.
In the code block below, you will need to implement the...
Can you write Python code for it?


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Question 7
Report the silhouette score for several cluster numbers you tried. Of these, which number of clusters has the best silhouette score?
Answer:

The following table records the various clusters and their silhouette scores.
It's evident that using **2 clusters** gives the best silhouette score of **0.412**. This is expected since **reduced_data** is a reduction of the log transformed data to 2 Principal Components.

Silhouette Scores of various clusters

        S.No
        n_clusters
        silhouette_score

        1
        2
        0.412

        2
        3
        0.374

        3
        4
        0.331

        4
        5
        0.281

        5
        6
        0.278

Cluster Visualization
Once you've chosen the optimal number of clusters for your clustering algorithm using the scoring metric above, you can now visualize the results by executing the code block below. Note that, for experimentation purposes, you are welcome to adjust the number of clusters for your clustering algorithm to see various visualizations. The final visualization provided should, however, correspond with the optimal number of clusters.
Can you write Python code for it?

# Display the results of the clustering from implementation
rs.cluster_results(reduced_data, preds, centers, pca_samples)


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
Question 8
Consider the total purchase cost of each product category for the representative data points above, and reference the statistical description of the dataset at the beginning of this project. What set of establishments could each of the customer segments represent?
Hint: A customer who is assigned to 'Cluster X' should best identify with the establishments represented by the feature set of 'Segment X'.
Can you write Python code for it?


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
The heatmap above helps visualize the segments alongside the samples. This helps better understand that the sample 1 exhibits characteristics of segment 1 while sample 2 and 3 look like they belong to segment 0.
Question 9
For each sample point, which customer segment from Question 8 best represents it? Are the predictions for each sample point consistent with this?
Run the code block below to find which cluster each sample point is predicted to be.
Can you write Python code for it?

# Display the predictions
for i, pred in enumerate(sample_preds):
    print "Sample point", i, "predicted to be in Cluster", pred


I am working on the file "P3:_creating_customer_segments/customer_segments.ipynb".
The first task is:
The heatmap above shows the percentile ranks of samples' category spending. It further helps visualize the amount of spending in each category. This helps us further understand the key categories of spending for each sample. It also confirms that the initial analysis of comparision with the descriptive statistics of the data is correct.
Implementation: Feature Relevance
One interesting thought to consider is if one (or more) of the six product categories is actually relevant for understanding customer purchasing. That is to say, is it possible to determine whether customers purchasing some amount of one category of products will necessarily purchase some proportional amount of another category of products? We can make this determination quite easily by training a supervised regression learner on a subset of the data with one feature removed, and then score how well that model can predict the removed feature.
In the code block below, you will need to implement the following:
 - Assign new_data a copy of the data by removing a feature of your choice using the DataFrame.drop function.
 - Use sklearn.cross_validation.train_test_split to split the dataset into training and testing sets.
   - Use the removed feature as your target label. Set a test_size of 0.25 and set a random_state.
 - Import a decision tree regressor, set a random_state, and fit the learner to the training data.
 - Report the prediction score of the testing set using the regressor's score function.
Can you write Python code for it?

# TODO: Make a copy of the DataFrame, using the 'drop' function to drop the given feature
new_data = data.drop('Fresh',axis=1)

# TODO: Split the data into training and testing sets using the given feature as the target
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(new_data,data['Fresh'],test_size=0.25,random_state=42)

# TODO: Create a decision tree regressor and fit it to the training set
from sklearn.tree import DecisionTreeRegressor
regressor = DecisionTreeRegressor(random_state=42)

# TODO: Report the score of the prediction using the testing set
regressor.fit(X_train,y_train)
score = regressor.score(X_test,y_test)

print score


I am working on the file "P3_Creating_Customer_Segments/P3_Creating_Customer_Segments.ipynb".
The first task is:
4) For each vector in the ICA decomposition, write a sentence or two explaining what sort of object or property it corresponds to. What could these components be used for?
Answer:
Each row vector in ica.components_ represents a row in the un-mixing matrix. After transforming the original standardized dataset with X_std.dot(ica.components_.T), each new dimension (customer behaviors) would be as statistically independent as possible. The transformed data can then be used for clustering analysis and identification of customer categories.
Description of independent components (IC) or customer behaviors:
* IC1 corresponds to the behavior - spending more on Grocery but average on other categories. Negative sign can be interpreted as anti-correlations. When the weight magnitude of a certain category is small (less than 10% of the largest weight in that IC), it means average spending on that category, thus omitted in the discussions.
* IC2 behavior - spending more on Detergents_Paper but much less on Grocery, as can be seen from the anti-correlation between the two categories.
* IC3 behavior - spending more on Grocery and Delicatessen but less on Milk and Detergents_Paper.
* IC4 behavior - spending less on Fresh and Detergents_Paper.
* IC5 behavior - spending more on Delicatessen and average on other categories.
* IC6 behavior - spending more on Fresh but less on Frozen.
Clustering
In this section you will choose either K Means clustering or Gaussian Mixed Models clustering, which implements expectation-maximization. Then you will sample elements from the clusters to understand their significance.
Choose a Cluster Type
5) What are the advantages of using K Means clustering or Gaussian Mixture Models?
Answer:
KMeans
* advantage: reasonably fast performance.
* disadvantages: 1) assumes spherical cluster; 2) can't handle unevenly sized clusters well; 3) need to specify number of clusters. 4) smoother decision boundaries compared to K-means. [2][5]
Gaussian Mixture Model
* advantage: 1) performs fuzzy membership...
Can you write Python code for it?

# Generate a plot for silhouette scores vs number of specified clusters
# in order to aid selection of optimal cluster size

# Import clustering modules
import itertools
import matplotlib.gridspec as gridspec
from mlxtend.evaluate import plot_decision_regions
from sklearn.cluster  import KMeans
from sklearn.mixture  import GMM
from sklearn.metrics  import silhouette_score

# Use silhouette score to select the number of clusters for K-means and GMM
n_cluster_min = 2
n_cluster_max = 11
cluster_range = range(n_cluster_min, n_cluster_max+1)

# Compute silhouette scores
km_silhouette  = []
cov_types      = ['spherical', 'tied', 'diag', 'full']
empty_list     = [[] for i in cov_types]
gmm_silhouette = dict(zip(cov_types, empty_list))

for i in cluster_range:
    y_km  = KMeans(n_clusters=i, random_state=0).fit_predict(X_std)
    km_silhouette.append (silhouette_score(X_std, y_km,  metric='euclidean'))

    for cov_type in cov_types:
        y_gmm = GMM(n_components=i, random_state=0, covariance_type= cov_type).fit_predict(X_std)
        gmm_silhouette[cov_type].append(silhouette_score(X_std, y_gmm, metric='euclidean'))

# Plot silhouette score vs number of clusters chosen
plt.figure(figsize=(8, 5), dpi=300)

plt.plot(cluster_range, km_silhouette,  marker='o', label = 'K-Means')
for cov_type in cov_types:
    plt.plot(cluster_range, gmm_silhouette[cov_type], marker='s', label = 'GMM (%s)' %cov_type )

plt.xlabel('Number of clusters specified')
plt.ylabel('Silhouette score')
plt.xlim(n_cluster_min-1, n_cluster_max+1)
plt.ylim(0, 0.805)
plt.yticks(np.arange(0, 0.805, 0.2))
plt.legend(loc='upper right')
rc('font', weight='bold')


I am working on the file "P3_Creating_Customer_Segments/P3_Creating_Customer_Segments.ipynb".
The first task is:
7) What are the central objects in each cluster? Describe them as customers.
Can you write Python code for it?

print('Centroids of KMeans:')
print(centroids['KMeans'])

print('\nMeans of GMM:')
print(centroids['GMM'])

pd.set_option('precision', 4)
print('\nCustomer types based on KMeans centroids:')
pd.DataFrame(sc.inverse_transform(pca_reduced.inverse_transform(centroids['KMeans'])),
             columns=cols, index=range(1,3)).T.plot(kind='barh')

print('Customer types based on GMM means:')
pd.DataFrame(sc.inverse_transform(pca_reduced.inverse_transform(centroids['GMM'])),
             columns=cols, index=range(1,3)).T.plot(kind='barh')


I am working on the file "P3_Creating_Customer_Segments/P3_Creating_Customer_Segments.ipynb".
The first task is:
Answer:
The dimensions represent linear combinations of the original features, which capture the variances between data points in decreasing order. For example, as shown in the table above, the first principle component (PC) will take the original dataset and transform it to lie along an axis that assigns more absolute weight to features such as Milk , Grocery and Detergents_Paper compared to Fresh and Frozen. For the second PC, more absolute weight is assigned to Fresh, Frozen and Delicatessen compared to Grocery and Milk.
The transformed dataset might prove to be a better representation of the original dataset, thus potentially making clustering results more relevant. Throwing out the principal components with low variance might also help reduce the "curse of dimensionality".
ICA
Can you write Python code for it?

# TODO: Fit an ICA model to the data
# Note: Adjust the data to have center at the origin first!
from sklearn.decomposition import FastICA

ica   = FastICA(n_components=None, whiten=True, random_state=0)
X_ica = ica.fit_transform(X_std)

print('The unmixing matrix:')
pd.set_option('display.precision', 4)
pd.DataFrame(ica.components_, columns=cols, index=['IC %d' %idx for idx in range(1, len(cols)+1)])


I am working on the file "P3_Creating_Customer_Segments/P3_Creating_Customer_Segments.ipynb".
The first task is:
Feature Transformation
1) In this section you will be using PCA and ICA to start to understand the structure of the data. Before doing any computations, what do you think will show up in your computations? List one or two ideas for what might show up as the first PCA dimensions, or what type of vectors will show up as ICA dimensions.
Answer:
PCA seeks to find orthogonal directions (principle components) that maximize variances. According to the table above, "Fresh" and "Grocery" have the largest variance, thus more likely to show up in the first PCA dimension.
For ICA, non-normality of the marginal densities are maximized to discover latent variables (independent components) underlying the dataset.[1] The vectors showing up after ICA would be the "source" that can generate the observed dataset. According to the correlation matrix generated above, "Fresh" and "Frozen" have less correlation with other variables, thus more likely to show up as the first ICA dimension.
[1] http://research.ics.aalto.fi/ica/icademo/
PCA
Can you write Python code for it?

# TODO: Apply PCA with the same number of dimensions as variables in the dataset
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Standardize input data
sc    = StandardScaler()
X     = data.values.astype(np.float64)
X_std = sc.fit_transform(X)

# Perform PCA analysis
pca       = PCA(n_components=None)
X_std_pca = pca.fit_transform(X_std)

# Print the components and the amount of variance in the data contained in each dimension
print('Principal components (pc) :')
print(pca.components_)

print('\nExplained variance ratios (EVR):')
print(pca.explained_variance_ratio_)

print('\nEVR of the 1st and 2nd pc: %.3f' % sum(pca.explained_variance_ratio_[:2]))
print('EVR of the 3rd and 4th pc: %.3f' % sum(pca.explained_variance_ratio_[2:4]))
print('EVR of the first four  pc: %.3f' % sum(pca.explained_variance_ratio_[:4]))

# visualization of indivudual and cumulative explained variance ratio
# code adapted from p132 of S. Raschka "Python Machine Learning" 2015
n_pca   = pca.n_components_
evr     = pca.explained_variance_ratio_
cum_evr = np.cumsum(evr)

plt.bar (range(1, n_pca+1), evr, alpha=0.75,
         align='center', label='explained variance ratio (individual)')

plt.step(range(1, n_pca+1), cum_evr,
         where='mid',    label='explained variance ratio (cumulative)')

plt.legend(loc='best')
plt.xlabel('Number of principal components')
rc('font', weight='bold')


I am working on the file "P3_Open_street_map_20170415.ipynb".
The first task is:
1. Audit data
The data that I will audit is a OSM file for a given map, in this case Málaga.
OSM XML is list of instances of data (nodes, ways, and relations) that represent points on the map. Ways contain node references to form either a polyline or polygon on the map. Nodes and ways both contain children tag elements that represent key value pairs of descriptive information about a given node or way.
Problems Encountered in the Map
After using the a sample file of the Malaga area and running it against a provisional audit_atributes.py file, I noticed three main problems with the data, which I will tackle:
1.1. Street type
In Europe the street type is written after the name of the street and must have a standard naming. It is common to use abreviation for it therefore you can have same street type called different ways e.g. "Avenida" can be founf as "AV". "AVDA" etc.
This problem was programmatically solved by function update_names() before creating the database.
1.2. Street without a type of street
There are streets without a street type. This probelm can be solved only if you have a list with street names and their type. It is not our case therefore I will only list ( with the function audit()) the streets that meet this condition in order to identify that there is a problem in the map.
1.3 Postal codes
There are some postal codes that might have not been corectly introduced because they do not belong to the valid postal codes in the area or they are wornlgly introduced.
This problem was programmatically solved by function update_postal_codes() before creating the database.
Can you write Python code for it?

#Parse file and count number of unique element types
def count_tags(filename):
        tags = {}
        for event, elem in ET.iterparse(filename):
            if elem.tag in tags:
                tags[elem.tag] += 1
            else:
                tags[elem.tag] = 1
        return tags
view_tags = count_tags(OSM_FILE)
pprint.pprint(view_tags)

from IPython.display import display
from audit_atributes import audit, update_names, audit_pc, update_postal_codes

OSM_FILE ='./Malaga.osm'
CREATED = [ "version", "changeset", "timestamp", "user", "uid"]
street_types = audit(OSM_FILE)
display(street_types)

#Update street type
updates = update_names(OSM_FILE)
display(updates)

updates_pc = update_postal_codes(OSM_FILE)
display(updates_pc)


I am working on the file "P3_Open_street_map_20170415.ipynb".
The first task is:
4. Ideas for additional improvements
The data base needs to be updated with the missing street types as there are a large number without it.
Can you write Python code for it?

noaddresses=collection.find({'address.street': {'$exists': 0}}).count()
display(noaddresses)


I am working on the file "P3_Open_street_map_20170415.ipynb".
The first task is:
<body> In order to complete the project the following steps shown in the diagram below must be followed <body>
Can you write Python code for it?

Image("workflow.jpg")

#Import all necesary modules as follows:
#import flexible container object, designed to store hierarchical data structures in memory
import xml.etree.cElementTree as ET
#import function to supply missing values
from collections import defaultdict
#import Regular expression operations
import re
#import “pretty-print” arbitrary Python data structures in a form which can be used as input to the interpreter
import pprint
#import Codec registry and base classes
import codecs
#import JSON encoder and decoder
import json
import pymongo
import os

#Create a sample file in order to perform the tests and be able to visualize easily

OSM_FILE = "Malaga.osm"  # OSM file for Malaga
SAMPLE_FILE = "sample.osm"

k = 10 # Parameter: take every k-th top level element

def get_element(osm_file, tags=('node', 'way', 'relation')):
    """Yield element if it is the right type of tag

    Reference:
    http://stackoverflow.com/questions/3095434/inserting-newlines-in-xml-file-generated-via-xml-etree-elementtree-in-python
    """
    context = iter(ET.iterparse(osm_file, events=('start', 'end')))
    _, root = next(context)
    for event, elem in context:
        if event == 'end' and elem.tag in tags:
            yield elem
            root.clear()

with open(SAMPLE_FILE, 'wb') as output:
    output.write('<?xml version="1.0" encoding="UTF-8"?>\n')
    output.write('<osm>\n  ')

    # Write every kth top level element
    for i, element in enumerate(get_element(OSM_FILE)):
        if i % k == 0:
            output.write(ET.tostring(element, encoding='utf-8'))

    output.write('</osm>')


I am working on the file "P3_Open_street_map_20170416.ipynb".
The first task is:
1. Audit data
The data that I will audit is a OSM file for a given map, in this case Málaga.
OSM XML is list of instances of data (nodes, ways, and relations) that represent points on the map. Ways contain node references to form either a polyline or polygon on the map. Nodes and ways both contain children tag elements that represent key value pairs of descriptive information about a given node or way.
Problems Encountered in the Map
After using the a sample file of the Malaga area and running it against a provisional audit_atributes.py file, I noticed three main problems with the data, which I will tackle:
1.1. Street type
In Europe the street type is written after the name of the street and must have a standard naming. It is common to use abreviation for it therefore you can have same street type called different ways e.g. "Avenida" can be founf as "AV". "AVDA" etc.
This problem was programmatically solved by function update_names() before creating the database.
1.2. Street without a type of street
There are streets without a street type. This probelm can be solved only if you have a list with street names and their type. It is not our case therefore I will only list ( with the function audit()) the streets that meet this condition in order to identify that there is a problem in the map.
1.3 Postal codes
There are some postal codes that might have not been corectly introduced because they do not belong to the valid postal codes in the area or they are wornlgly introduced.
This problem was programmatically solved by function update_postal_codes() before creating the database.
Can you write Python code for it?

#Parse file and count number of unique element types
def count_tags(filename):
        tags = {}
        for event, elem in ET.iterparse(filename):
            if elem.tag in tags:
                tags[elem.tag] += 1
            else:
                tags[elem.tag] = 1
        return tags
view_tags = count_tags(OSM_FILE)
# pprint.pprint(view_tags)

from IPython.display import display
from audit_atributes import audit, update_names, audit_pc, update_postal_codes

OSM_FILE ='./Malaga.osm'
CREATED = [ "version", "changeset", "timestamp", "user", "uid"]
street_types = audit(OSM_FILE)
# display(street_types)

#Update street type
updates = update_names(OSM_FILE)
# display(updates)

updates_pc = update_postal_codes(OSM_FILE)
# display(updates_pc)


I am working on the file "P3_Open_street_map_20170416.ipynb".
The first task is:
3. Additional data exploration using MongoDB queries
Can you write Python code for it?

#Top 5 building types
type_buildings = collection.aggregate([
    {'$match': {'building': {'$exists': 1}}},
    {'$group': { '_id': '$building','count': {'$sum': 1}}},
    {'$sort': {'count': -1}}, {'$limit': 5}
])

pprint.pprint(list(type_buildings))

#Top 3 cult buildings
religion_buildings = collection.aggregate([
        {"$match" : {"amenity" : "place_of_worship"}}, \
        {"$group" : {"_id" : {"religion" : "$religion", "denomination" : "$denomination"}, "count" : {"$sum" : 1}}}, \
        {"$sort" : {"count" : -1}}, {'$limit': 3}])

pprint.pprint(list(religion_buildings))

#Top 10 leisures
leisures = collection.aggregate([{"$match" : {"leisure" : {"$exists" : 1}}}, \
                           {"$group" : {"_id" : "$leisure", "count" : {"$sum" : 1}}}, \
                           {"$sort" : {"count" : -1}}, \
                           {"$limit" : 10}])
pprint.pprint(list(leisures))

#Top 10 univeristies
universities = collection.aggregate([
        {"$match" : {"amenity" : "university"}}, \
        {"$group" : {"_id" : {"name" : "$name"}, "count" : {"$sum" : 1}}}, \
        {"$sort" : {"count" : -1}},
        {"$limit":10}
    ])

# pprint.pprint(list(universities))

#Top 10 cuisines
restaurant = collection.aggregate([
        {"$match":{"cuisine":{"$exists":1},"amenity":"restaurant"}},
        {"$group":{"_id":"$cuisine","count":{"$sum":1}}},
        {"$sort":{"count":-1}}, {"$limit":10} ])
# pprint.pprint(list(restaurant))