I am working on the file "P3_Open_street_map_20170416.ipynb".
The first task is:
There is a code for running directly from python the mongoimport ( see https://github.com/bestkao/data-wrangling-with-openstreetmap-and-mongodb) but as I do not have administrator right for my computer I had to manually instroduce the command
In order to create the data base the for the openstreet execute in console cmd the following code:
mongoimport -h 127.0.0.1:27017 --db openstreetmap1 --collection Malaga --file C:\Users\c41237\Desktop\Udacity\P3\P3_OpenStreetMap\Malaga.osm.json
Can you write Python code for it?

# Number of documents
documents = collection.find().count()
display(documents)

#Number of unique users
users=len(collection.distinct('created.user'))
display(users)

#number of nodes & ways
nodes=collection.find({'type':'node'}).count()
ways=collection.find({'type':'way'}).count()
display(nodes)
display(ways)

#disply types and number of nodes & ways
node_way = collection.aggregate([
        {"$group" : {"_id" : "$type", "count" : {"$sum" : 1}}}])

pprint.pprint(list(node_way))

#top3 contributors to the map
top3 = collection.aggregate([{ '$group' : {'_id' : '$created.user',
                                           'count' : { '$sum' : 1}}},
                             { '$sort' : {'count' : -1}},
                             { '$limit' : 3 }])
display(list(top3))

#number of documents with street addresses
addresses=collection.find({'address.street': {'$exists': 1}}).count()
print(addresses)

#top3 postal codes
top10_pc = collection.aggregate([{ '$group' : {'_id' : '$address.postcode',
                                           'count' : { '$sum' : 1}}},
                             { '$sort' : {'count' : -1}},
                             { '$limit' : 3 }])
display(list(top10_pc))

#list of postal codes limited to 3
postal_codes = collection.aggregate([
        {"$match" : {"address.postcode" : {"$exists" : 1}}}, \
        {"$group" : {"_id" : "$address.postcode", "count" : {"$sum" : 1}}}, \
        {"$sort" : {"count" : -1}}])

# pprint.pprint(list(postal_codes))

#list top 10 street names in data base
streets = collection.aggregate([
        {"$match" : {"address.street" : {"$exists" : 1}}}, \
        {"$group" : {"_id" : "$address.street", "count" : {"$sum" : 1}}}, \
        {"$sort" : {"count" : -1}},
        {"$limit":10}])

pprint.pprint(list(streets))

#Top 10 amenities
top10_amenities = collection.aggregate([{"$match":{"amenity":{"$exists":1}}},
                             {"$group":{"_id":"$amenity","count":{"$sum":1}}},
                             {"$sort":{"count":-1}},
     ...


I am working on the file "P3_Open_street_map_20170416.ipynb".
The first task is:
This can be done either by having the information first hand, entering the data one by one or using other map data such as https://www.google.es/maps or https://www.viamichelin.es/. If you go to one of this websites and you tipe in the name ( e.g. Gustavo Pittaluga you will identify that is a street type "Calle".
One other option it will be to use the official governamental data base from the cadastral reference that can be found at this webpage https://www1.sedecatastro.gob.es/OVCFrames.aspx?TIPO=CONSULTA
and we have available at hand the help page: http://www.catastro.meh.es/ayuda/ayuda_rc.htm
The expected benefits are:
•the data comes form an official source (governement), therefore reliable and up to date. Any modification can be identifyed right ayway.
•the other two websites belong to the most used maps ( google and viamihelin) and you can also get a visual of their location on the map, street view etc.
As it can be observed that the maps have to be cleaned for any imput done by hand and not chosen from a list, therefore we might encounter some problems for this ideea.
The anticipated problems are:
•high volume of information therefore proun of type errors
•street name can be the same in two or more cities and the author shouldpay attention when looking for the street to introduce the right city.
Anyhow the task can be completed in this way with reliable information and in the case that the info for the street tyoe is not introduced corectly we can allways run the audit routine above to clean the data.
Can you write Python code for it?


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
Here is the final 52 param model that we are using for this project.

Through some experiments I know that once the loss reach around .03 we have a model that can drive in the simulator so we use a custom early termination callback to stop the training!
Can you write Python code for it?

class CustomEarlyStop(Callback):
    """
    Custom Callback that stops the epoch when val_loss reachs 0.3
    This callback assumes you are logging val loss
    """
    def __init__(self, monitor='val_loss'):
        super(CustomEarlyStop, self).__init__()
        self.monitor = monitor

    def on_epoch_end(self, epoch, logs=None):
        val_loss = logs.get(self.monitor)
        if val_loss <= 0.039:
            print("\nEarly Stop on Epoch {0} with Val_loss {1}\n".format(epoch,val_loss))
            self.model.stop_training = True


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
Last step is to create the shift image and  load image methods.
Notice the load_image() method takes a dataframe row and uses the assumed meta columns to make the modification to the image on the fly. This assumes the preprocessing stage has added the columns needed.
Can you write Python code for it?

def shift_img(image, random_shift):
    rows, cols = image.shape
    mat = np.float32([[1, 0, random_shift], [0, 1, 0]])
    return cv2.warpAffine(image, mat, (cols, rows))

def load_image(row):
    image = cv2.imread("./data/{0}".format(row.image.strip()))
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    image = crop_top_and_bottom(image)

    if(row.is_flipped):
        image = cv2.flip(image,1)
    if(row.is_shift):
        image = shift_img(image, row.random_shift)
    return image


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
Now the histogram is mostly evenly distributed and the question now becomes how much data can we trim and still train the model?!
After some experimenting we can trim this histogram with 100 bins by trimming the bins with over 500 count.
The hacky solution I created below basically generates a list of strings that are panda dataframe masking code using the ranges from np.histogram() to search; then I loop through the list of code and run them through the eval() method to execute the code!
 The end result is no bins will go past 400 examples
Can you write Python code for it?

hist, counts = np.histogram(processed_log.steering, bins=100)
upper_limit = 400
over = [(i, v) for i, v in enumerate(hist) if v > upper_limit ]
over_ranges = [(counts[i],counts[i+1]) for i,_ in over]

#loop through ranges and create a mask for each bin
masks = ["processed_log[(processed_log.steering >= {0}) & (processed_log.steering < {1})]".format(l,r) for l,r in over_ranges]

for mask in masks:
    selected = eval(mask)
    selected_length = len(selected)
    frac_to_drop = (selected_length-upper_limit)/selected_length
    samples_to_drop = selected.sample(frac=frac_to_drop)
    processed_log = processed_log.drop(samples_to_drop.index)

plt.figure(figsize=(10,4))
processed_log.steering.hist(bins=100, color='g')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(processed_log.steering))

processed_log.to_csv('preprocessed_driver_log.csv')


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
Now we start making changes to the driver log file and we create a new driver log file.
This new driver log file is created to later support larger datasets by adding more 'meta' columns with augmentation information. You'll notice that each augmentation method adds the columns it needs to later augment the image when loaded.
Can you write Python code for it?

def update_left_right_steering_correction(df):
    records = []
    for index, row in df.iterrows():
        left = row.left
        center = row.center
        right = row.right
        steering = row.steering

        records.append({
            'image': left,
            'steering': steering + 0.23
        })

        records.append({
            'image': right,
            'steering': steering - 0.23
        })

        records.append({
            'image': center,
            'steering': steering
        })

    return pd.DataFrame(data=records, columns=['image', 'steering'])

new_drive_log = update_left_right_steering_correction(drive_log_df)
new_drive_log.tail()


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
The histogram above shows the dataset is unbalanced.
We will now go ahead and drop most of the zero steering angles, which correspond to mostly straight driving.
Can you write Python code for it?

zero_steering = drive_log_df[drive_log_df.steering == 0].sample(frac=0.9)
drive_log_df = drive_log_df.drop(zero_steering.index)

plt.figure(figsize=(10,4))
drive_log_df.steering.hist(bins=100, color='r')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(drive_log_df.steering))


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
The histogram looks more balanced from the upsampling, but this caused us to created steering angles outside the bounds  of [-1,1]
The process_driver_log() method is used to pump out the final driver log file that we will use to generate the new driver log dataframe and also drops those examples.
Can you write Python code for it?

def process_driver_log(driver_log):
    update_log = update_left_right_steering_correction(driver_log)
    update_log = flip_images_augmentation(update_log)
    update_log = shift_img_augmentation(update_log)
    #update_log = change_brightness_augmentation(update_log)

    #reset index since we it's no longer good.
    update_log = update_log.reset_index(drop=True)

    #drop outbound steering examples to be between [-1,1] !
    outbound_steering = update_log[abs(update_log.steering) > 1]
    update_log = update_log.drop(outbound_steering.index)

    return update_log

processed_log = process_driver_log(drive_log_df)

plt.figure(figsize=(10,4))
processed_log.steering.hist(bins=100, color='r')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(processed_log.steering))


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
To run the model training below you'll need to download and run the vis tool
Simply do the following:

git clone https://github.com/fchollet/hualos
cd into the directory
run python api.py
visit localhost:9000
run the training below and it'll produce the loss plot you see below!

Click RemoteMonitor for more information on how this callback works!
Can you write Python code for it?

#This is a callback that works with https://github.com/fchollet/hualos and gives you simple vis of the training loss
remote = RemoteMonitor(headers=None)
early_stop = CustomEarlyStop(monitor='val_loss')

h = model.fit(x=features,y=labels,verbose=1,batch_size=128,nb_epoch=50,validation_split=0.3, callbacks=[remote, early_stop])


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
We have to upsample under represented example and we do this by shifting images and their corresponding angles
Can you write Python code for it?

def shift_img_augmentation(df):
    df.loc[:,'random_shift'] = 0
    new_df = df[df.steering != 0].copy()
    df.loc[:,'is_shift'] = False
    new_df.loc[:,'is_shift'] = True

    max_shift = 30
    max_ang = 0.17

    def row_shift_update(row):
        random_shift = np.random.randint(-max_shift, max_shift + 1)
        row.random_shift = random_shift
        updated_steer = row.steering + (random_shift / max_shift) * max_ang
        if abs(updated_steer) > 1:
            updated_steer = -1 if (updated_steer < 0) else 1

        row.steering = updated_steer
        return row

    new_df = new_df.apply(row_shift_update, axis=1)
    return pd.concat([df, new_df])

shifted = shift_img_augmentation(augmented)
plt.figure(figsize=(10,4))
shifted.steering.hist(bins=100, color='r')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(shifted.steering))

shifted.tail(1)
plt.show()


I am working on the file "P3_behavioral_cloning/playground.ipynb".
The first task is:
We sample (normal dist) 40% of example that are not zero angles; flip them and reverse steering angle.
Can you write Python code for it?

def flip_images_augmentation(df):
    new_df = df[df.steering != 0].sample(frac=0.4)
    df.loc[:,'is_flipped'] = False
    new_df.loc[:,'is_flipped'] = True
    left_rows = (new_df.steering < 0)
    right_rows = (new_df.steering > 0)
    new_df.loc[left_rows,'steering'] = new_df[left_rows].steering.abs()
    new_df.loc[right_rows, 'steering'] = new_df[right_rows].steering * -1
    return pd.concat([df, new_df])

augmented = flip_images_augmentation(new_drive_log)
plt.figure(figsize=(10,4))
augmented.steering.hist(bins=100, color='r')
plt.xlabel('steering angle bins')
plt.ylabel('counts')
plt.show()
print("Current Dataset Size: ", len(augmented.steering))


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Answer:
Sample points 1 and 2 fit into the descriptions of each cluster: point 1 buys a large amount of Milk, Grocery and Detergents_Paper while point 2 buys a lot of Fresh and Frozen. I would, judging by eye, think point 2 is misclassified. In fact, looking at the cell below, we see that the probabilities assigned to this point are not so clear cut: the algorithms estimates there to be a 30% chance it got it wrong.
Sample point 0, on the other hand, seems more unclear: it buys a lot of Milk and Detergents_Paper, suggesting cluster 0, but also a lot of Fresh and Frozen, suggesting cluster 1. Looking at the cluster visualization, above, we see that this point should still clearly be labeled cluster 0, which just goes to show how useful a visualization like that can be.
Can you write Python code for it?


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Answer:
The first buys more Fresh and Frozen (similar quantities to the dataset average), but little Milk,  Grocery and Detergents_Paper (half to less-than-half of the dataset average). That suggests some sort of restaurant / fast-food place which prepares some food from scratch but uses a lot of frozen products as well.
The second is the opposite: they buy a lot of Milk (near average of whole dataset), Groceries (much higher than average) and Detergents_Paper (twice as high as average) but very little Fresh or Frozen compared to the dataset average. This seems much closer to a hotel-type of business, where there is a lot of cleaning and washing involved, and a lot of breakfast preparation (which uses a lot of Milk and "Grocery" which I assume must mean non-fresh foods like cereal).
Question 9
For each sample point, which customer segment from Question 8 best represents it? Are the predictions for each sample point consistent with this?
Run the code block below to find which cluster each sample point is predicted to be.
Can you write Python code for it?

# Display the predictions
for i, pred in enumerate(sample_preds):
    print "Sample point", i, "predicted to be in Cluster", pred

samples


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Answer:
The first two principal components together account for 0.707 of the total variance. The first four account for 0.931 of the total variance.
The first PC mainly describes spending on Milk, Grocery and/or Detergents_Paper; in other words, consumers who spend on this sort of "retail goods" will have a large value on "Dimension 1". Our second sample customer is likely to have a large value on this first dimension.
The second PC mostly describes spending on food items, in particular Food, Frozen, and Delicatessen: customers who purchase a lot of this sort of food-based items will appear higher on Dimension 2. Our third sample customer is an example of this, as it bought a lot of Food and Frozen (but not Delicatessen).
The third PC breaks apart those customers who bought a lot of Fresh and a lot of Frozen/Delicatessen, from those who bought mostly Fresh or mostly Frozen/Delicatessen. In other words, here it is no longer equivalent which of the three the customer bought, and it breaks up the data depending on the balance between these categories. This must be because there is quite a lot of variety in those who bought Fresh-Frozen-Delicatessen, and this component tells them apart.
The fourth PC further refines the Fresh-Frozen-Delicatessen (i.e. food-item) customers, but additionally splitting up those who bough a lot of Frozen from those who bought a lot of Delicatessen. This fully breaks up the various types of food buyers.
Observation
Run the code below to see how the log-transformed sample data has changed after having a PCA transformation applied to it in six dimensions. Observe the numerical value for the first four dimensions of the sample points. Consider if this is consistent with your initial interpretation of the sample points.
Can you write Python code for it?

# Display sample log-data after having a PCA transformation applied
display(pd.DataFrame(np.round(pca_samples, 4), columns = pca_results.index.values))


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Answer:
There appears to be a correleation between "Grocery" and "Milk", as well as between "Detergents_Paper" and "Milk". This might seem surprising given how poorly the regression performed; this may have to do with Decision-tree-regressors not being very good, and a different regressor would be more suited. It may also be that for the vast majority of the data, there is no clear correlation at all, i.e. that most of the data is in some shapeless ball close to the origin, and there is a "linear-looking branch" shooting off from it, giving the impression of a clear linear dependence when in fact this is only true for very large x and y values.
The data for each feature is mostly on the "low" end of the spectrum, with a long tail---it looks much more similar to a binomial or a Poisson distribution than a Gaussian. This supports the idea of a "ball" near the origin, with linear dependences only clear when including the long tail in each distribution.
Data Preprocessing
In this section, you will preprocess the data to create a better representation of customers by performing a scaling on the data and detecting (and optionally removing) outliers. Preprocessing data is often times a critical step in assuring that results you obtain from your analysis are significant and meaningful.
Implementation: Feature Scaling
If data is not normally distributed, especially if the mean and median vary significantly (indicating a large skew), it is most often appropriate to apply a non-linear scaling — particularly for financial data. One way to achieve this scaling is by using a Box-Cox test, which calculates the best power transformation of the data that reduces skewness. A simpler approach which can work in most cases would be applying the natural logarithm.
In the code block below, you will need to implement the following:
 - Assign a copy of the data to log_data after applying logarithmic scaling. Use the np.log function for this.
 - Assign a copy of the sample data to log_samples after applying logarithmic scaling. Again,...
Can you write Python code for it?

# TODO: Scale the data using the natural logarithm
log_data = np.log(data)

# TODO: Scale the sample data using the natural logarithm
log_samples = np.log(samples)

# Produce a scatter matrix for each pair of newly-transformed features
pd.scatter_matrix(log_data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Conclusion
In this final section, you will investigate ways that you can make use of the clustered data. First, you will consider how the different groups of customers, the customer segments, may be affected differently by a specific delivery scheme. Next, you will consider how giving a label to each customer (which segment that customer belongs to) can provide for additional features about the customer data. Finally, you will compare the customer segments to a hidden variable present in the data, to see whether the clustering identified certain relationships.
Question 10
Companies will often run A/B tests when making small changes to their products or services to determine whether making that change will affect its customers positively or negatively. The wholesale distributor is considering changing its delivery service from currently 5 days a week to 3 days a week. However, the distributor will only make this change in delivery service for customers that react positively. How can the wholesale distributor use the customer segments to determine which customers, if any, would react positively to the change in delivery service?
Hint: Can we assume the change affects all customers equally? How can we determine which group of customers it affects the most?
Answer:
The group of customers who buy lot of detergents, milk and groceries can probably withstand a 3-days-a-week delivery service, since the products they purchase will probably not go bad within that time; this cluster is the one that is least likely to get affected. However, it will force them to be more careful in how much milk they buy, it cannot run out in that time, but also it shouldn't expire before the next delivery.
The other cluster, which buys a lot of fresh food and frozen food, will dislike getting fresh food that is actually several days older than it could have been (obvisouly the frozen food doesn't matter).
Probably it would make sense to only A/B test the change on (a small subset of) the cluster buying a lot of detergents.
Question...
Can you write Python code for it?

# Display the clustering results based on 'Channel' data
vs.channel_results(reduced_data, outliers, pca_samples)


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Data Exploration
In this section, you will begin exploring the data through visualizations and code to understand how each feature is related to the others. You will observe a statistical description of the dataset, consider the relevance of each feature, and select a few sample data points from the dataset which you will track through the course of this project.
Run the code block below to observe a statistical description of the dataset. Note that the dataset is composed of six important product categories: 'Fresh', 'Milk', 'Grocery', 'Frozen', 'Detergents_Paper', and 'Delicatessen'. Consider what each category represents in terms of products you could purchase.
Can you write Python code for it?

# Display a description of the dataset
display(data.describe())


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Implementation: Data Recovery
Each cluster present in the visualization above has a central point. These centers (or means) are not specifically data points from the data, but rather the averages of all the data points predicted in the respective clusters. For the problem of creating customer segments, a cluster's center point corresponds to the average customer of that segment. Since the data is currently reduced in dimension and scaled by a logarithm, we can recover the representative customer spending from these data points by applying the inverse transformations.
In the code block below, you will need to implement the following:
 - Apply the inverse transform to centers using pca.inverse_transform and assign the new centers to log_centers.
 - Apply the inverse function of np.log to log_centers using np.exp and assign the true centers to true_centers.
Can you write Python code for it?

# TODO: Inverse transform the centers
log_centers = pca.inverse_transform(centers)

# TODO: Exponentiate the centers
true_centers = np.exp(log_centers)

# Display the true centers
segments = ['Segment {}'.format(i) for i in range(0,len(centers))]
true_centers = pd.DataFrame(np.round(true_centers), columns = data.keys())
true_centers.index = segments
display(true_centers)


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Implementation: Dimensionality Reduction
When using principal component analysis, one of the main goals is to reduce the dimensionality of the data — in effect, reducing the complexity of the problem. Dimensionality reduction comes at a cost: Fewer dimensions used implies less of the total variance in the data is being explained. Because of this, the cumulative explained variance ratio is extremely important for knowing how many dimensions are necessary for the problem. Additionally, if a signifiant amount of variance is explained by only two or three dimensions, the reduced data can be visualized afterwards.
In the code block below, you will need to implement the following:
 - Assign the results of fitting PCA in two dimensions with good_data to pca.
 - Apply a PCA transformation of good_data using pca.transform, and assign the results to reduced_data.
 - Apply a PCA transformation of log_samples using pca.transform, and assign the results to pca_samples.
Can you write Python code for it?

# TODO: Apply PCA by fitting the good data with only two dimensions
pca = PCA(n_components=2)
pca.fit(good_data)

# TODO: Transform the good data using the PCA fit above
reduced_data = pca.transform(good_data)

# TODO: Transform log_samples using the PCA fit above
pca_samples = pca.transform(log_samples)

# Create a DataFrame for the reduced data
reduced_data = pd.DataFrame(reduced_data, columns = ['Dimension 1', 'Dimension 2'])


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Observation
Once we have the original feature projections (in red), it is easier to interpret the relative position of each data point in the scatterplot. For instance, a point the lower right corner of the figure will likely correspond to a customer that spends a lot on 'Milk', 'Grocery' and 'Detergents_Paper', but not so much on the other product categories.
From the biplot, which of the original features are most strongly correlated with the first component? What about those that are associated with the second component? Do these observations agree with the pca_results plot you obtained earlier?
ANSWER: The first PC is mostly correlated with Milk, Grocery and Detergents_Paper, while the second is mostly composed of Fresh and Frozen, but also Delicatessen. This was precisely my observation described eariler.
Clustering
In this section, you will choose to use either a K-Means clustering algorithm or a Gaussian Mixture Model clustering algorithm to identify the various customer segments hidden in the data. You will then recover specific data points from the clusters to understand their significance by transforming them back into their original dimension and scale.
Question 6
What are the advantages to using a K-Means clustering algorithm? What are the advantages to using a Gaussian Mixture Model clustering algorithm? Given your observations about the wholesale customer data so far, which of the two algorithms will you use and why?
Answer:
K-means is fast and is guaranteed to converge to an answer. It is also very simple: it makes no assumptions about the underlying distribution of the data. The advantages of using Gaussian Mixture Models are that we obtain probabilities for each data point to have a specific label, assuming the distribution of the various features is Gaussian. Since after the log transform our distributions look very similar to Gaussians (even though we do have a couple of multimodal distributions), Gaussian mixture models are likely to generate useful results which are more powerful than...
Can you write Python code for it?

from sklearn.mixture import GMM
from sklearn.metrics import silhouette_score

# TODO: Apply your clustering algorithm of choice to the reduced data
clusterer = GMM(n_components=2).fit(reduced_data)

# TODO: Predict the cluster for each data point
preds = clusterer.predict(reduced_data)

# TODO: Find the cluster centers
centers = clusterer.means_

# TODO: Predict the cluster for each transformed sample data point
sample_preds = clusterer.predict(pca_samples)

# TODO: Calculate the mean silhouette coefficient for the number of clusters chosen
score = silhouette_score(reduced_data, preds)
print score


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Observation
Run the code below to see how the log-transformed sample data has changed after having a PCA transformation applied to it using only two dimensions. Observe how the values for the first two dimensions remains unchanged when compared to a PCA transformation in six dimensions.
Can you write Python code for it?

# Display sample log-data after applying PCA transformation in two dimensions
display(pd.DataFrame(np.round(pca_samples, 4), columns = ['Dimension 1', 'Dimension 2']))


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Question 1
Consider the total purchase cost of each product category and the statistical description of the dataset above for your sample customers.
What kind of establishment (customer) could each of the three samples you've chosen represent?
Hint: Examples of establishments include places like markets, cafes, and retailers, among many others. Avoid using names for establishments, such as saying "McDonalds" when describing a sample customer as a restaurant.
Answer:
The first customer bought a lot (i.e. significantly above the mean) of several types of items, especially Milk and Delicatessen. This suggests that this user is running some sort of business, and judging by the Milk and Delicatessen, probably a cafe where those things are consumed in greater quantities.
The second customer bought a very large quantity, much greater than the mean plus standard deviation, of groceries and detergents / paper. This would suggest, possibly, a hotel or a bed and breakfast, where there is a lot of cleaning and toiletries involved but also a daily element of cooking "from scratch".
The third customer bought a normal amount of all items, all within the mean plus standard deviation, except for Fresh and Frozen, which are very high. While it's not clear how "Fresh" is different from "Groceries", since fresh food (at least in Italy where I live) is the vast majority of anyone's groceries, it's plausible that some sort of restaurant / fast-food business may have a large need for both of those types of items.
Implementation: Feature Relevance
One interesting thought to consider is if one (or more) of the six product categories is actually relevant for understanding customer purchasing. That is to say, is it possible to determine whether customers purchasing some amount of one category of products will necessarily purchase some proportional amount of another category of products? We can make this determination quite easily by training a supervised regression learner on a subset of the data with one feature removed, and then...
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
from sklearn.tree import DecisionTreeRegressor

# TODO: Make a copy of the DataFrame, using the 'drop' function to drop the given feature
new_data = data.drop(['Milk'], axis = 1)

# TODO: Split the data into training and testing sets using the given feature as the target
X_train, X_test, y_train, y_test = train_test_split(new_data, data["Milk"], test_size=0.25, random_state=42)

# TODO: Create a decision tree regressor and fit it to the training set
regressor = DecisionTreeRegressor(random_state=42)
regressor.fit(X_train, y_train)

# TODO: Report the score of the prediction using the testing set
score = regressor.score(X_test, y_test)
print score


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Question 2
Which feature did you attempt to predict? What was the reported prediction score? Is this feature is necessary for identifying customers' spending habits?
Hint: The coefficient of determination, R^2, is scored between 0 and 1, with 1 being a perfect fit. A negative R^2 implies the model fails to fit the data.
Answer:
I predicted the feature "Milk", which was predicted with score 0.156. This score is extremely poor; in other words, the other variables are unable to predict it well, and hence this variable is (as far as this regressor is concerned) quite indepedent from the others: it appears to be necessary to keep this variable.
Visualize Feature Distributions
To get a better understanding of the dataset, we can construct a scatter matrix of each of the six product features present in the data. If you found that the feature you attempted to predict above is relevant for identifying a specific customer, then the scatter matrix below may not show any correlation between that feature and the others. Conversely, if you believe that feature is not relevant for identifying a specific customer, the scatter matrix might show a correlation between that feature and another feature in the data. Run the code block below to produce a scatter matrix.
Can you write Python code for it?

# Produce a scatter matrix for each pair of features in the data
pd.scatter_matrix(data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Question 3
Are there any pairs of features which exhibit some degree of correlation? Does this confirm or deny your suspicions about the relevance of the feature you attempted to predict? How is the data for those features distributed?
Hint: Is the data normally distributed? Where do most of the data points lie?
Can you write Python code for it?

import seaborn as sns
sns.heatmap(data.corr(), annot=True);


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Question 4
Are there any data points considered outliers for more than one feature based on the definition above? Should these data points be removed from the dataset? If any data points were added to the outliers list to be removed, explain why.
Can you write Python code for it?

def howManyTimesOutlier(index):
    return np.sum([(index in outlierindices) for outlierindices in allindices])
more_than_1 = np.array([ind for ind in log_data.index.values if howManyTimesOutlier(ind)>1])
print "%d outliers in more than one feature:" % len(more_than_1)
print more_than_1

more_than_2 = np.array([ind for ind in log_data.index.values if howManyTimesOutlier(ind)>2])
print "\n%d Outliers in more than two features:" % len(more_than_2)
print more_than_2

more_than_3 = np.array([ind for ind in log_data.index.values if howManyTimesOutlier(ind)>3])
print "\n%d Outliers in more than three features:" % len(more_than_3)
print more_than_3


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Question 7
Report the silhouette score for several cluster numbers you tried. Of these, which number of clusters has the best silhouette score?
Answer:
I tried the silhouette scores for num_components ranging from 2 to 10. These are the results:
Can you write Python code for it?

allvariances = []
allscores = []
for compnum in range(1,10):
    clust = GMM(n_components=compnum).fit(reduced_data)
    allvariances.append(np.mean([np.mean(sig) for sig in clust.covars_]))
    thepreds = clust.predict(reduced_data)
    if compnum==1:
        thescore = 0
    else:
        thescore = silhouette_score(reduced_data, thepreds)
    allscores.append(thescore)

plt.plot(range(1,10), allscores)
plt.xlabel("num_components")
plt.ylabel("silhouette score");


I am working on the file "P3_customer_segments/customer_segments.ipynb".
The first task is:
Visualizing a Biplot
A biplot is a scatterplot where each data point is represented by its scores along the principal components. The axes are the principal components (in this case Dimension 1 and Dimension 2). In addition, the biplot shows the projection of the original features along the components. A biplot can help us interpret the reduced dimensions of the data, and discover relationships between the principal components and original features.
Run the code cell below to produce a biplot of the reduced-dimension data.
Can you write Python code for it?

# Create a biplot
vs.biplot(good_data, reduced_data, pca)


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:

Declaring variables
A central feature of programming is the declaration of variables.  When you declare a variable, you are storing data in the computer's memory and you are assigning a name to that data.  Both storage and name-assignment are carried out with the single equality symbol =.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
After you declare a variable, its value remains the same until it is changed.  You can change the value of a variable with a simple assignment.  After the above lines, the value of my_number is 17.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
All data in Python has a type, but a common complaint about Python is that types are a bit concealed "under the hood".  But they are not far under the hood.  Anyone can find out the type of some data with a single command.
Can you write Python code for it?

type(3)

type(3.0)

type('Hello')

type([1,2,3])


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
As before, Python interprets an operation (**) differently in different contexts.  When given integer input, Python evaluates 2**1000 exactly.  The result is a large integer.  A nice fact about Python, for number theorists, is that it handles exact integers of arbitrary length!  Many other programming languages (like C++) will give an error message if integers get too large in the midst of a computation.
New in version 3.x, Python implements long integers without giving signals to the programmer or changing types.  In Python 2.x, there were two types: int for somewhat small integers (e.g., up to $2^{31}$) and long type for all larger integers.  Python 2.x would signal which type of integer was being used, by placing the letter "L" at the end of a long integer.  Now, in Python 3.x, the programmer doesn't really see the difference.  There is only the int type.  But Python still optimizes computations, using hardware functionality for arithmetic of small integers and custom routines for large integers.  The programmer doesn't have to worry about it most of the time.
For scientific applications, one often wants to keep track of only a certain number of significant digits.  If one computes the floating point exponent 2.0**1000, the result is a decimal approximation.  It is still a float.  The expression "e+301" stands for "multiplied by 10 to the 301st power", i.e., Python uses scientific notation for large floats.
Can you write Python code for it?

type(2**1000)

type(2.0**1000)

#  An empty cell.  Have fun!


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
But the "for loop" is more efficient and more readable to programmers.  Indeed, it saves the repetition of writing the same command print n*n over and over again.  It also makes transparent, from the beginning, the range of values that n is assigned to.
When you read and write "for loops", you should consider how they look unravelled -- that is how Python will carry out the loop.  And when you find yourself faced with a repetetive task, you might consider whether it may be wrapped up in a for loop.
Try to unravel the loop below, and predict the result, before evaluating the code.
Can you write Python code for it?

P = 1
for n in range(1,6):
    P = P * n
print(P)


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
Can you create a string with 100 A's (like AAA...)?  Use an appropriate operation in the cell below.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
Experiment below to see how Python handles a double or triple negative, i.e., something with a not not.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
For number theory, division with remainder is an operation of central importance.  Integer division provides the quotient, and the operation % provides the remainder.  It's a bit strange that the percent symbol is used for the remainder, but this dates at least to the early 1970s and has become standard across computer languages.
Can you write Python code for it?

23 // 5  # Integer division

23 % 5  # The remainder after division


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
Here we have used the print command with strings and numbers together.  In Python 3.x, you can print multiple things on the same line by separating them by commas.  The "things" can be strings (enclosed by single or double-quotes) and numbers (int, float, etc.).
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
How might you create a nested loop which prints 1 a then 2 a then 3 a then 1 b then 2 b then 3 b?  Try it below.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
If we change the indentation, it changes the scope of the for loop.  Predict what the following loop will do, by unraveling, before evaluating it.
Can you write Python code for it?

P = 1
for n in range(1,6):
    P = P * n
print("n is",n,"and P is",P)
print(P)


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
If we unravel the loop above, the linear sequence of commands interpreted by Python is the following.
Can you write Python code for it?

P = 1
n = 1
P = P * n
print("n is",n,"and P is",P)
n = 2
P = P * n
print("n is",n,"and P is",P)
n = 3
P = P * n
print("n is",n,"and P is",P)
n = 4
P = P * n
print("n is",n,"and P is",P)
n = 5
P = P * n
print("n is",n,"and P is",P)
print (P)


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
Python is an interpreted language, which means (roughly) that Python carries out commands line-by-line from top to bottom.  So consider the three lines
python
my_number = 17
new_number = my_number + 1
my_number = 3.14
Line 1 sets the value of my_number to 17.  Line 2 sets the value of new_number to 18.  Line 3 sets the value of my_number to 3.14.  But Line 3 does not change the value of new_number at all.
(This will become confusing and complicated later, as we study mutable and immutable types.)
Exercises

What is the difference between = and == in the Python language?

If the variable x has value 3, and you then evaluate the Python command x = x * x, what will be the value of x after evaluation?

Imagine you have two variables a and b, and you want to switch their values.  How could you do this in Python?
Can you write Python code for it?

#  Use this space to work on the exercises.


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
Scopes can be nested by nesting indentation.  What do you think the following loop will do?  Can you unravel it?
Can you write Python code for it?

for x in [1,2,3]:
    for y in ['a', 'b']:
        print(x,y)


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
The expressions <, >, <=, >= are interpreted here as operations with numerical input and boolean output.  The symbol == (two equal symbols!) gives a True result if the numbers are equal, and False if the numbers are not equal.  An extremely common typo is to confuse = with ==.  But the single equality symbol = has an entirely different meaning, as we shall see.
Using the remainder operator % and equality, we obtain a divisibility test.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
The first loop, unraveled, carries out the following sequence of commands.
Can you write Python code for it?

n = 1
print(n*n)
n = 2
print(n*n)
n = 3
print(n*n)
n = 4
print(n*n)
n = 5
print(n*n)


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
The key to careful computation in Python is always being aware of the type of your data, and knowing how Python operates differently on data of different types.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
The results are probably not too surprising, though the last two require a bit of explanation.  Python interprets the input number 5 as an int (integer) and 5.0 as a float.  "Float" stands for "floating point number," which are decimal approximations to real numbers.  The word "float" refers to the fact that the decimal (or binary, for computers) point can float around (as in 1.2345 or 12.345 or 123.45 or 1234.5 or 0.00012345).  There are deep computational issues related to how computers handle decimal approximations, and you can read about the IEEE standards if you're interested.
Python enables different kinds of division.  The single-slash division in Python 3.x gives a floating point approximation of the quotient.  That's why 5 / 11 and 6 / 3 both output floats.  On the other hand, 5 // 11 and 6 // 3 yield integer outputs (rounding down) -- this is useful, but one has to be careful!
In fact the designers of Python changed their mind.  This tutorial assumes that you are using Python 3.x.  If you are using Python 2.x, the command 5 / 11 would output zero.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
There is another type called a tuple that we will rarely use.  Tuples use parentheses for enclosure instead of brackets.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
There's another list-like type in Python 3, called the range type.  Ranges are kind of like lists, but instead of plunking every item into a slot of memory, ranges just have to remember three integers:  their start, their stop, and their step.
The range command creates a range with a given start, stop, and step.  If you only input one number, the range will start at zero and use steps of one and will stop just before the given stop-number.
One can create a list from a range (plunking every term in the range into a slot of memory), by using the list command.  Here are a few examples.
Can you write Python code for it?

type(range(10)) # Ranges are their own type, in Python 3.x.  Not in Python 2.x!

list(range(10)) # Let's see what's in the range.  Note it starts at zero!  Where does it stop?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
This ability of Python to interpret operations based on context is a mixed blessing.  On one hand, it leads to handy shortcuts -- quick ways of writing complicated programs.  On the other hand, it can lead to code that is harder to read, especially for a Python novice.  Good programmers aim for code that is easy to read, not just short!
The Zen of Python is a series of 20 aphorisms for Python programmers.  The first seven are below.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.

Exercises

Did you look at the truth tables closely?  Can you remember, from memory, what True or False equals, or what True and False equals?

How might you easily remember the truth tables?  How do they resemble the standard English usage of the words "and" and "or"?

If you wanted to know whether a number, like 2349872348723, is a multiple of 7 but not a multiple of 11, how might you write this in one line of Python code?

You can chain together and commands, e.g., with an expression like True and True and True (which would evaluate to True).  You can also group booleans, e.g., with True and (True or False).  Experiment to figure out the order of operations (and, or, not) for booleans.

The operation xor means "exclusive or".  Its truth table is: True xor True = False and False xor False = False and True xor False = True and False xor True = True.  How might you implement xor in terms of the usual and, or, and not?
Can you write Python code for it?

#  Use this space to work on the exercises


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
This can be used for descending ranges too, and observe that the final number b in range(a,b,s) is not included.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
This command reassigns the value of my_number to 3.14.  Note that it changes the type too!  It effectively overrides the previous value and replaces it with the new value.
Often it is useful to change the value of a variable incrementally or recursively.  Python, like many programming languages, allows one to assign variables in a self-referential way.  What do you think the value of S will be after the following four lines?
Can you write Python code for it?

S = 0
S = S + 1
S = S + 2
S = S + 3
print(S)


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
This might have been difficult!  So what if you want to trace through the loop, as it goes?  Sometimes, especially when debugging, it's useful to inspect every step of the loop to see what Python is doing.  We can inspect the loop above, by inserting a print command within the scope of the loop.
Can you write Python code for it?

P = 1
for n in range(1,6):
    P = P * n
    print("n is",n,"and P is",P)
print(P)


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
Use the cell below to determine whether 1234567890 is divisible by 3.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
Use the truth tables to predict the result (True or False) of each of the following, before evaluating the code.
Can you write Python code for it?

(2 > 3) and (3 > 2)

(1 + 1 == 2) or (1 + 1 == 3)

not (-1 + 1 >= 0)

2 + 2 == 4

2 + 2 != 4  # For "not equal", Python uses the operation `!=`.

2 + 2 != 5  # Is 2+2 *not* equal to 5?

not (2 + 2 == 5)  # The same as above, but a bit longer to write.


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
What are the values of the variables my_number and new_number, after the execution of these three lines?
To access these values, you can use the print function.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 1.ipynb".
The first task is:
Why use integer division // and why use floating point division?  In practice, integer division is typically a faster operation.  So if you only need the rounded result (and that will often be the case), use integer division.  It will run much faster than carrying out floating point division then manually rounding down.
Observe that floating point operations involve approximation.  The result of 5.0/11.0 might not be what you expect in the last digit.  Over time, especially with repeated operations, floating point approximation errors can add up!
Python allows you to group expressions with parentheses, and follows the order of operations that you learn in school.
Can you write Python code for it?

(3 + 4) * 5

3 + (4 * 5)

3 + 4 * 5   #  What do you think will be the result?  Remember PEMDAS?


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:

Control statements
It is important for a computer program to behave differently under different circumstances.  The simplest control statements, if and its relative else, can be used to tell Python to carry out different actions depending on the value of a boolean variable.  The following function exhibits the syntax.
Can you write Python code for it?

def is_even(n):
    if n%2 == 0:
        print("{} is even.".format(n))
        return True
    else:
        print("{} is odd.".format(n))
        return False

is_even(17)

is_even(1000)


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:

Solving the linear Diophantine equation
In Chapter 1 of An Illustrated Theory of Numbers, we not only used the Euclidean algorithm to find the GCD of two integers, but also to solve the linear Diophantine equation $ax + by = c$.  On paper, this required us to perform the Euclidean algorithm, then "work backwards" to carefully solve a series of linear equations.  This process is repetetive and error-prone... perfect for a computer.
So here we develop a function solve_LDE(a,b,c) which will describe all integer solutions $x,y$ to the equation $ax + by = c$.
The idea of the algorithm is to keep track of "hops" and "skips" throughout the Euclidean algorithm.  A general step in the Euclidean algorithm looks like u = q(v) + r.  The remainder can then be expressed by the formula r = u - q(v).  If u and v can be built from hops and skips, then r can be built from hops and skips.  How many?  Just tally the hops and skips to find:

`r_hops = u_hops - q (v_hops)` and `r_skips = u_skips - q (v_skips)`.

This sort of tallying is what makes the algorithm below work.  The function below does not introduce any new programming concepts, but it assembles many ideas together.
Can you write Python code for it?

def hop_and_skip(a,b):
    '''
    Takes two integer arguments a,b, and prints a sentence of the form
    GCD(a,b) = x(a) + y(b).  The method is the Euclidean algorithm,
    tallying hops (units of a) and skips (units of b) along the way.
    '''
    u = a # We use u instead of dividend.
    v = b # We use v instead of divisor.
    u_hops, u_skips = 1,0 # u is built from one hop (a) and no skips, for now.
    v_hops, v_skips = 0,1 # v is built from no hops and one skip (b), for now.
    while v != 0:   # We could just write "while v:"
        q = u // v  # q stands for quotient.
        r = u % v  # r stands for remainder.  So u = q(v) + r.

        r_hops = u_hops - q * v_hops  # Tally hops
        r_skips = u_skips - q * v_skips  # Tally skips

        u,v = v,r  # The new dividend,divisor is the old divisor,remainder.
        u_hops, v_hops = v_hops, r_hops # The new u_hops, v_hops is the old v_hops, r_hops
        u_skips, v_skips = v_skips, r_skips # The new u_skips, v_skips is the old v_skips, r_skips

    print("{} = {}({}) + {}({})".format(u,u_hops,a,u_skips,b))

hop_and_skip(102,45)


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:

While loops and implementation of the Eucidean algorithm
We almost have all the tools we need to implement the Euclidean algorithm.  The last tool we will need is the while loop.  We have seen the for loop already, which is very useful for iterating over a range of numbers.  The Euclidean algorithm involves repetition, but there is no way to know in advance how many steps it will take.  The while loop allows us to repeat a process as long as a boolean value (sometimes called a flag) is True.  The following countdown example illustrates the structure of a while loop.
Can you write Python code for it?

def countdown(n):
    current_value = n
    while current_value > 0:  # The condition (current_value > 0) is checked before every instance of the scope!
        print(current_value)
        current_value = current_value - 1

countdown(10)


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
An important note:  the return keyword terminates the function.  So as soon as a factor is found, the function terminates and outputs False.  If no factor is found, then the function execution survives past the loop, and the line return True is executed to terminate the function.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
Exercises

Solve problems 4-7 of Chapter 1 of An Illustrated Theory of Numbers using the solve_LDE function.

Write an LCM function, using the previous GCD function and the GCD-LCM product formula, Theorem 1.23 of An Illustrated Theory of Numbers.

Sometimes it is important to find not the integer solutions, but the positive integer solutions to a Diophantine equation.  Modify the solve_LDE function to create a solve_LDE_positive(a,b,c) function.  The output of the function should be all pairs of positive integers $x$, $y$, such that $ax + by = c$ (if any pairs exist), and a helpful message if no pairs exist (and a return None should be used in this case).
Can you write Python code for it?


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
Let's break down the syntax of the function declaration, line by line.
python
def square(x):
    answer = x * x
    return answer
The first line begins with the Python reserved word def.  (So don't use def as a variable name!).  The word def stands for "define" and it defines a function called square.  After the function name square comes parentheses (x) containing the argument x.  The arguments or parameters of a function refer to the input data.  Even if your function has no arguments, you need parentheses.  The argument x is used to name whatever number is input into the square function.
At the end of the function declaration line is a colon : and the following two lines are indented.  As in the case of for loops, the colon and indentation are signals of scope.  Everything on the indented lines is considered the scope of the function and is carried out when the function is used later.
The second line answer = x * x is the beginning of the scope of the function.  It declares a variable answer and sets the value to be x * x.  So if the argument x is 12, then answer will be set to 144.  The variable answer, being declared within the scope of the function, will not be accessible outside the scope of the function.  It is called a local variable.
The last line return answer contains the Python reserved word return, which terminates the function and outputs the value of the variable answer.  So when you apply the function with the command square(1.5), the number 1.5 is the argument x, and answer is 2.25, and that number 2.25 becomes the output.
A function does not have to return a value.  Some functions might just provide some information.  Here is a function which displays the result of division with remainder as a sentence with addition and multiplication.
Can you write Python code for it?

def display_divmod(a,b):
    quotient = a // b # Integer division
    remainder = a % b #
    print("{} = {} ({}) + {}".format(a,quotient,b,remainder))

display_divmod(23,5)


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
Notice that this function has no return line.  The function terminates automatically at the end of its scope.
The function also uses Python's string formatting.  This has changed between Python 2.x and 3.x, and this notebook uses Python 3.x syntax.
String formatting allows you to insert placeholders like {} within a string, and later fill those places with a list of things.
Can you write Python code for it?

print("My favorite number is {}".format(17))  # The .format "method" substitutes 17 for {}

print("{} + {} = {}".format(13,12,13+12))


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
The format command is an example of a string method.  It has the effect of replacing all placeholders {} by the its inputs, in sequence.  There is an intricate syntax for these placeholders, to allow one to match placeholders with values in different orders, and to format different kinds of values.  Here is the official reference for string formatting in Python 3.x.  We will only use the most basic features, exhibited below.
Can you write Python code for it?

print ("The number {} comes before {}.".format(1,2)) # This should be familiar.
print ("The number {1} comes before {0}.".format(1,2)) # What happens?
print ("The number {1} comes before {1}.".format(1,2)) # Got it now?


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
The while loop syntax begins with while <boolean>: and the following indented lines comprise the scope of the loop.  If the boolean is True, then the scope of the loop is executed.  If the boolean is True again afterwards, then the scope of the loop is executed again.  And again and again and so on.
This can be a dangerous process!  For example, what would happen if you made a little typo and the last line of the while loop read current_value = current_value + 1?  The numbers would increase and increase... and the boolean current_value > 0 would always be True.  Therefore the loop would never end.  Bigger and bigger numbers would scroll down your computer screen.
You might panic under such a circumstance, and maybe turn your computer off to stop the loop.  Here is some advice for when your computer gets stuck in such a neverending loop:

Back up your work often.  When you're programming, make sure everything else is saved just in case.
Save your programming work (use "Save and checkpoint" under the "File" menu) often, especially before running a cell with a loop for the first time.
If you do get stuck in a neverending loop, click on "Kernel... Interrupt".  This will often unstick the loop and allow you to pick up where you left off.
On a Mac, you might try a "Force Quit" of the Python process, using the Activity Manager.

Now, if you're feeling brave, save your work, change the while loop so that it never ends, and try to recover where you left off.  But be aware that this could cause your computer to freeze or behave erratically, crashing your browser, etc.  Don't panic... it won't break your computer permanently.
The neverending loop causes two problems here.  One is with your computer processor, which will be essentially spinning its wheels.  This is called busy waiting, and your computer will essentially be busy waiting forever.  The other problem is that your loop is printing more and more lines of text into the notebook.  This could easily crash your web browser, which is trying to store...
Can you write Python code for it?

def Euclidean_algorithm(a,b):
    dividend = a
    divisor = b
    while divisor != 0:   # Recall that != means "is not equal to".
        quotient = dividend // divisor
        remainder = dividend % divisor
        print("{} = {} ({}) + {}".format(dividend, quotient, divisor, remainder))
        dividend = divisor
        divisor = remainder

Euclidean_algorithm(133, 58)

Euclidean_algorithm(1312331323, 58123123)


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
There are two limiting factors, which we study in more detail in the next lesson.  These are time and space (your computer's memory space).  As the loop of is_prime goes on and on, it might take your computer a long time!  If each step of the loop takes only a nanosecond (1 billionth of a second), the loop would take about a second when executing is_prime(1000000001).  If you tried is_prime on a much larger number, like is_prime(2**101 - 1), the loop would take longer than the lifetime of the Earth.
The other issue that can arise is a problem with space.  In Python 3.x, the range(2,n) cleverly avoids storing all the numbers between 2 and n-1 in memory.  It just remembers the endpoints, and how to proceed from one number to the next.  In the older version, Python 2.x, the range command range(2,n) would have tried to store the entire list of numbers [2,3,4,...,n-1] in the memory of your computer.  Your computer has some (4 or 8 or 16, perhaps) gigabytes of memory (RAM).  A gigabyte is a billion bytes, and a byte is enough memory to store a number between 0 and 255.  (More detail about this later!).  So a gigabyte will not even hold a billion numbers.  So our is_prime function would have led to memory problems in Python 2.x, but in Python 3.x we don't have to worry (for now) about space.
Exercises

Create a function my_abs(x) which outputs the absolute value of the argument x.  (Note that Python already has a built-in abs(x) function).

Modify the is_prime function so that it prints a message Number too big and returns None if the input argument is bigger than one million.  (Note that None is a Python reserved word.  You can use the one-line statement return None.)

Write a Python function thrarity which takes an argument n, and outputs the string threeven if n is a multiple of three, or throdd is n is one more than a multiple of three, or thrugly if n is one less than a multiple of three.  Example:  thrarity(31) should output throdd and thrarity(44) should output thrugly.  Hint:  study the if/elif syntax...
Can you write Python code for it?

#  Use this space for your solutions to the questions.


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
This code is essentially optimal, if one wishes to execute the Euclidean algorithm to find the GCD of two integers.  It almost matches the GCD code in a standard Python library.  It might be slightly faster than our original code -- but there is a tradeoff here between execution speed and readability of code.  In this and the following lessons, we often optimize enough for everyday purposes, but not so much that readability is lost.
Exercises

Modify the is_prime function by using a while loop instead of for j in range(2,n):.  Hint:  the function should contain the lines j = 2 and while j < n: and j = j + 1 in various places.  Why might this be an improvement from the for loop?

Modify the Euclidean_algorithm function to create a function which returns the number of steps that the Euclidean algorithm requires, i.e., the number of divisions-with-remainder.

Create a function which carries out division with minimal remainder.  In other words, given integers a,b, the function expresses a = q(b) + r, where r is a positive or negative integer of magnitude bounded by b/2.  Use such a function to create a new Euclidean algorithm function which uses minimal remainder.

What GCD(a,b) function do you think strikes the best balance between efficiency and readability?
Can you write Python code for it?

# Use this space to work on the exercises.


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
This is excellent if we want to know every step of the Euclidean algorithm.  If we just want to know the GCD of two numbers, we can be less verbose.  We carefully return the last nonzero remainder after the while loop is concluded.  This last nonzero remainder becomes the divisor when the remainder becomes zero, and then it would become the dividend in the next (unprinted) line.  That is why we return the (absolute value) of the dividend after the loop is concluded.  You might insert a line at the end of the loop, like print dividend, divisor, remainder to help you track the variables.
Can you write Python code for it?

def GCD(a,b):
    dividend = a # The first dividend is a.
    divisor = b # The first divisor is b.
    while divisor != 0:   # Recall that != means "not equal to".
        quotient = dividend // divisor
        remainder = dividend % divisor
        dividend = divisor
        divisor = remainder
    return abs(dividend)  #  abs() is used, since we like our GCDs to be positive.


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
To conclude this lesson, we put everything together to create a long(ish) function to solve linear Diophantine equations.  We want this function to be smart enough to respond when an equation has no solutions, and to describe all solutions when they exist.
The first part of the function solve_LDE is the same as the hop and skip function above.  But rather than expressing $GCD(a,b)$ as $ax + by$, the function uses the GCD to determine the existence and the general form of a solution to $ax + by = c$.  The formula for the general form comes from An Illustrated Theory of Numbers, Chapter 1, Corollary 1.25.
Can you write Python code for it?

def solve_LDE(a,b,c):
    '''
    Describes all of the solutions to the linear Diophantine equation
    ax + by = c.  There are either no solutions or infinitely many solutions.
    Prints a description of the solution set, and returns None if there are no solutions
    or returns a single solution if one exists.
    '''
    u = a # We use u instead of dividend.
    v = b # We use v instead of divisor.
    u_hops, u_skips = 1,0 # u is built from one hop (a) and no skips.
    v_hops, v_skips = 0,1 # v is built from no hops and one skip (b).
    while v != 0:   # We could just write while v:
        q = u // v  # q stands for quotient.
        r = u % v  # r stands for remainder.  So u = q(v) + r.

        r_hops = u_hops - q * v_hops  # Tally hops
        r_skips = u_skips - q * v_skips  # Tally skips

        u,v = v,r  # The new dividend,divisor is the old divisor,remainder.
        u_hops, v_hops = v_hops, r_hops # The new u_hops, v_hops is the old v_hops, r_hops
        u_skips, v_skips = v_skips, r_skips # The new u_skips, v_skips is the old v_skips, r_skips

    g = u # The variable g now describes the GCD of a and b.

    if c%g == 0:  # When GCD(a,b) divides c...
        d = c//g
        x = d * u_hops
        y = d * u_skips  # Now ax + by = c is a specific solution!
        print("{} x + {} y = {} if and only if ".format(a, b, c))
        print("x = {} + {} n and y = {} - {} n, for some integer n.".format(x,b//g,y,-a//g))
        return x,y
    else:  # When GCD(a,b) does not divide c...
        print("There are no solutions to {} x + {} y = {},".format(a,b,c))
        print("because GCD({}, {}) = {}, which does not divide {}.".format(a,b,g,c))
        return None

solve_LDE(102,45,3)

solve_LDE(72,100,17)


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
Try out the hop_and_skip code on some integers of your choice.  Does it behave correctly?  Check the results, using Python as a calculator.  Does it run quickly for large integers?
Can you write Python code for it?


I am working on the file "P3wNT Notebook 2.ipynb".
The first task is:
When you run the code block, you probably didn't see anything happen.  But you have effectively taught your computer a new trick, increasing the vocabulary of commands it understands through the Python interpreter.  Now you can use the square command as you wish.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Another modest improvement is the following.  In the code above, the program counts the terms in sequences like 9,15,21,27,..., in order to set them to False.  This is accomplished with the length command len(flags[p*p::2*p]).  But that length computation is a bit too intensive.  A bit of algebraic work shows that the length is given formulaically in terms of p and n by the formula:
$$len = \lfloor \frac{n - p^2 - 1}{2p} \rfloor + 1$$
(Here $\lfloor x \rfloor$ denotes the floor function, i.e., the result of rounding down.)  Putting this into the code yields the following.
Can you write Python code for it?

def isprime_list(n):
    '''
    Return a list of length n+1
    with Trues at prime indices and Falses at composite indices.
    '''
    flags = [True] * (n+1)  # A list [True, True, True,...] to start.
    flags[0] = False  # Zero is not prime.  So its flag is set to False.
    flags[1] = False  # One is not prime.  So its flag is set to False.
    flags[4::2] = [False] * ((n-2)//2)
    p = 3
    while p <= sqrt(n):  # We only need to sieve by p is p <= sqrt(n).
        if flags[p]:  # We sieve the multiples of p if flags[p]=True.
            flags[p*p::2*p] = [False] * ((n-p*p-1)//(2*p)+1) # Sieves out multiples of p, starting at p*p.
        p = p + 2 # Try the next value of p.

    return flags

%timeit sum(isprime_list(10**6))  # How much did this speed it up?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
As a last example of data visualization, we use matplotlib to produce a histogram of the prime gaps.
Can you write Python code for it?

max(primegaps)  # The largest prime gap that appears!

%matplotlib inline
plt.figure(figsize=(12, 5))  #  Makes the resulting figure 12in by 5in.
plt.hist(primegaps, bins=range(1,115)) #  Makes a histogram with one bin for each possible gap from 1 to 114.
plt.ylabel('Frequency')
plt.xlabel('Gap size')
plt.grid(True)
plt.title('The frequency of prime gaps, for primes up to 1 million')
plt.show()


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Changing list slices
Not only can we extract and study terms or slices of a list, we can change them by assignment.  The simplest case would be changing a single term of a list.
Can you write Python code for it?

print(L) # Start with the list L.

L[5] = 'Bacon!'

print(L)  # What do you think L is now?

print(L[2::3]) # What do you think this will do?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Combined with the isprime_list function, we can produce long lists of primes.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Comparing the graph to the theoretical result, we find that the ratio $\pi(x) / (x / \log(x))$ approaches $1$ (the theoretical result) but very slowly (see the graph above!).
A much stronger result relates $\pi(x)$ to the "logarithmic integral" $li(x)$.  The Riemann hypothesis is equivalent to the statement
$$\left\vert \pi(x) - li(x) \right\vert = O(\sqrt{x} \log(x)).$$
In other words, the error if one approximates $\pi(x)$ by $li(x)$ is bounded by a constant times $\sqrt{x} \log(x)$.  The logarithmic integral function isn't part of Python or numpy, but it is in the mpmath package.  If you have this package installed, then you can try the following.
Can you write Python code for it?

from mpmath import li

print(primes_upto(1000000))  # The number of primes up to 1 million.
print(li(1000000))  # The logarithmic integral of 1 million.


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Exercises

Create a list L with L = [1,2,3,...,100] (all the numbers from 1 to 100).  What is L[50]?

Take the same list L, and extract a slice of the form [5,10,15,...,95] with a command of the form L[a:b:c].

Take the same list L, and change all the even numbers to zeros, so that L looks like [1,0,3,0,5,0,...,99,0].  Hint:  You might wish to use the list [0]*50.

Try the command L[-1::-1] on a list.  What does it do?  Can you guess before executing it?  Can you understand why?  In fact, strings are lists too.  Try setting L = 'Hello' and the previous command.

Sieve of Eratosthenes
The Sieve of Eratosthenes (hereafter called "the sieve") is a very fast way of producing long lists of primes, without doing repeated primality checking.  It is described in more detail in Chapter 2 of An Illustrated Theory of Numbers.  The basic idea is to start with all of the natural numbers, and successively filter out, or sieve, the multiples of 2, then the multiples of 3, then the multiples of 5, etc., until only primes are left.
Using list slicing, we can carry out this sieving process efficiently.  And with a few more tricks we encounter here, we can carry out the Sieve very efficiently.
The basic sieve
The first approach we introduce is a bit naive, but is a good starting place.  We will begin with a list of numbers up to 100, and sieve out the appropriate multiples of 2,3,5,7.
Can you write Python code for it?

primes = list(range(100)) # Let's start with the numbers 0...99.


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
I've chosen function names with "fast" and "slow" in them.  But what makes them faster or slower?  Are they faster than the original?  And how can we tell?
Python comes with a great set of tools for these questions.  The simplest (for the user) are the time utilities.  By placing the magic %timeit before a command, Python does something like the following:

Python makes a little container in your computer devoted to the computations, to avoid interference from other running programs if possible.
Python executes the command lots and lots of times.
Python averages the amount of time taken for each execution.

Give it a try below, to compare the speed of the functions is_prime (the original) with the new is_prime_fast and is_prime_slow.  Note that the %timeit commands might take a little while.
Can you write Python code for it?

%timeit is_prime_fast(1000003)

%timeit is_prime_slow(1000003)

%timeit is_prime(1000003)


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
If $n$ is a prime number, then the is_prime(n) function will iterate through all the numbers between $2$ and $n-1$.  But this is overkill!  Indeed, if $n$ is not prime, it will have a factor between $2$ and the square root of $n$.  This is because factors come in pairs:  if $ab = n$, then one of the factors, $a$ or $b$, must be less than or equal to the square root of $n$.  So it suffices to search for factors up to (and including) the square root of $n$.
We haven't worked with square roots in Python yet.  But Python comes with a standard math package which enables square roots, trig functions, logs, and more.  Click the previous link for documentation.  This package doesn't load automatically when you start Python, so you have to load it with a little Python code.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
If you look carefully at the list of booleans, you will notice a True value at the 2nd index, the 3rd index, the 5th index, the 7th index, etc..  The indices where the values are True are precisely the prime indices.  Since booleans take the smallest amount of memory of any data type (one bit of memory per boolean), your computer can carry out the isprime_list(n) function even when n is very large.
To be more precise, there are 8 bits in a byte.  There are 1024 bytes (about 1000) in a kilobyte.  There are 1024 kilobytes in a megabyte.  There are 1024 megabytes in a gigabyte.  Therefore, a gigabyte of memory is enough to store about 8 billion bits.  That's enough to store the result of isprime_list(n) when n is about 8 billion.  Not bad!  And your computer probably has 4 or 8 or 12 or 16 gigabytes of memory to use.
To transform the list of booleans into a list of prime numbers, we create a function called where.  This function uses another Python technique called list comprehension.  We discuss this technique later in this lesson, so just use the where function as a tool for now, or read about list comprehension if you're curious.
Can you write Python code for it?

def where(L):
    '''
    Take a list of booleans as input and
    outputs the list of indices where True occurs.
    '''
    return [n for n in range(len(L)) if L[n]]


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Indeed, the is_prime_fast(n) function will go through a loop of length about sqrt(n) when n is prime.  But is_prime(n) will go through a loop of length about n.  Since sqrt(n) is much less than n, especially when n is large, the is_prime_fast(n) function is much faster.
Between is_prime_fast and is_prime_slow, the difference is that the fast version precomputes the square root sqrt(n) before going through the loop, where the slow version repeats the sqrt(n) every time the loop is repeated.  Indeed, writing while j <= sqrt(n): suggests that Python might execute sqrt(n) every time to check.  This might lead to Python computing the same square root a million times... unnecessarily!
A basic principle of programming is to avoid repetition.  If you have the memory space, just compute once and store the result.  It will probably be faster to pull the result out of memory than to compute it again.
Python does tend to be pretty smart, however.  It's possible that Python is precomputing sqrt(n) even in the slow loop, just because it's clever enough to tell in advance that the same thing is being computed over and over again.  This depends on your Python version and takes place behind the scenes.  If you want to figure it out, there's a whole set of tools (for advanced programmers) like the disassembler to figure out what Python is doing.
Can you write Python code for it?

is_prime_fast(10**14 + 37) # This might get a bit of delay.


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
It still looks almost linear, but there's a visible downward bend here.  How can we see this bend more clearly?  If the graph were linear, its equation would have the form $\pi(x) = mx$ for some fixed slope $m$ (since the graph does pass through the origin).  Therefore, the quantity $\pi(x)/x$ would be constant if the graph were linear.
Hence, if we graph $\pi(x) / x$ on the y-axis and $x$ on the x-axis, and the result is nonconstant, then the function $\pi(x)$ is nonlinear.
Can you write Python code for it?

m_values = pix_values[1:] / x_values[1:]  # We start at 1, to avoid a division by zero error.

%matplotlib inline
plt.plot(x_values[1:], m_values)
plt.title('The ratio $\pi(x) / x$ as $x$ varies.')
plt.xlabel('x')
plt.ylabel('$\pi(x) / x$')
plt.grid(True)
plt.show()


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Just like the range command, list slicing can take an optional third argument to give a step size.  To understand this, try the command below.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Let's push it a bit further.  How many primes are there between 1 and 1 million?  We can figure this out in three steps:

Create the isprime_list.
Use where to get the list of primes.
Find the length of the list of primes.

But it's better to do it in two steps.

Create the isprime_list.
Sum the list!  (Note that True is 1, for the purpose of summation!)
Can you write Python code for it?

sum(isprime_list(1000000))  # The number of primes up to a million!

%timeit isprime_list(10**6)  # 1000 ms = 1 second.

%timeit sum(isprime_list(10**6))


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Not too shabby!
Prime gaps
As a last bit of data analysis, we consider the prime gaps.  These are the numbers that occur as differences between consecutive primes.  Since all primes except 2 are odd, all prime gaps are even except for the 1-unit gap between 2 and 3.  There are many unsolved problems about prime gaps; the most famous might be that a gap of 2 occurs infinitely often (as in the gaps between 3,5 and between 11,13 and between 41,43, etc.).
Once we have our data set of prime numbers, it is not hard to create a data set of prime gaps.  Recall that primes is our list of prime numbers up to 1 million.
Can you write Python code for it?

len(primes) # The number of primes up to 1 million.

primes_allbutlast = primes[:-1]  # This excludes the last prime in the list.
primes_allbutfirst = primes[1:]  # This excludes the first (i.e., with index 0) prime in the list.

primegaps = numpy.array(primes_allbutfirst) - numpy.array(primes_allbutlast) # Numpy is fast!

print(primegaps[:100])  # The first hundred prime gaps!


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Now let's filter out the multiples of 2, starting at 4.  This is the slice primes[4::2]
Can you write Python code for it?

primes[4::2] = [None] * len(primes[4::2])  # The right side is a list of Nones, of the necessary length.
print(primes) # What have we done?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Now let's improve our is_prime(n) function by searching for factors only up to the square root of the number n.  We consider two options.
Can you write Python code for it?

def is_prime_slow(n):
    '''
    Checks whether the argument n is a prime number.
    Uses a brute force search for factors between 1 and n.
    '''
    j = 2
    while j <= sqrt(n):  # j will proceed through the list of numbers 2,3,... up to sqrt(n).
        if n%j == 0:  # is n divisible by j?
            print("{} is a factor of {}.".format(j,n))
            return False
        j = j + 1  # There's a Python abbreviation for this:  j += 1.
    return True

def is_prime_fast(n):
    '''
    Checks whether the argument n is a prime number.
    Uses a brute force search for factors between 1 and n.
    '''
    j = 2
    root_n = sqrt(n)
    while j <= root_n:  # j will proceed through the list of numbers 2,3,... up to sqrt(n).
        if n%j == 0:  # is n divisible by j?
            print("{} is a factor of {}.".format(j,n))
            return False
        j = j + 1  # There's a Python abbreviation for this:  j += 1.
    return True

is_prime_fast(1000003)

is_prime_slow(1000003)


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Now we filter out the multiples of 3, starting at 9.
Can you write Python code for it?

primes[9::3] = [None] * len(primes[9::3])  # The right side is a list of Nones, of the necessary length.
print(primes) # What have we done?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Now we graph the prime counting function.  To do this, we use a list comprehension, and the visualization library called matplotlib.  For graphing a function, the basic idea is to create a list of x-values, a list of corresponding y-values (so the lists have to be the same length!), and then we feed the two lists into matplotlib to make the graph.
We begin by loading the necessary packages.
Can you write Python code for it?

import matplotlib  #  A powerful graphics package.
import numpy  #  A math package
import matplotlib.pyplot as plt  # A plotting subpackage in matplotlib.


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Now we have two numpy arrays:  the array of x-values and the array of y-values.  We can make a plot with matplotlib.
Can you write Python code for it?

len(x_values) == len(pix_values)  # These better be the same, or else matplotlib will be unhappy.

%matplotlib inline
plt.plot(x_values, pix_values)
plt.title('The prime counting function')
plt.ylabel('$\pi(x)$')
plt.xlabel('x')
plt.grid(True)
plt.show()


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Now we use matplotlib to create a simple line graph.
Can you write Python code for it?

%matplotlib inline
plt.plot(x_values, y_values)
plt.title('The graph of $y = x^2$')  # The dollar signs surround the formula, in LaTeX format.
plt.ylabel('y')
plt.xlabel('x')
plt.grid(True)
plt.show()


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
Now, to "filter", i.e., to say that a number is not prime, let's just change the number to the value None.
Can you write Python code for it?

primes[0] = None # Zero is not prime.
primes[1] = None # One is not prime.
print(primes) # What have we done?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
There are a few different ways to import functions from packages.  The above syntax is a good starting point, but sometimes problems can arise if different packages have functions with the same name.  Here are a few methods of importing the sqrt function and how they differ.
from math import sqrt:  After this command, sqrt will refer to the function from the math package (overriding any previous definition).
import math:  After this command, all the functions from the math package will be imported.  But to call sqrt, you would type a command like math.sqrt(1000).  This is convenient if there are potential conflicts with other packages.
from math import *:  After this command, all the functions from the math package will be imported.  To call them, you can access them directly with a command like sqrt(1000).  This can easily cause conflicts with other packages, since packages can have hundreds of functions in them!
import math as mth:  Some people like abbreviations.  This imports all the functions from the math package.  To call one, you type a command like mth.sqrt(1000).
Can you write Python code for it?

import math

math.sqrt(1000)

factorial(10)  # This will cause an error!

math.factorial(10)  # This is ok, since the math package comes with a function called factorial.


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
This command imports the square root function (sqrt) from the package called math.  Now you can find square roots.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
This continues the strange (for beginners) Python convention of starting at the first number and ending just before the last number.  Compare to range(3,7), for example.
The command L[0:5] can be replaced by L[:5] to abbreviate.  The empty opening index tells Python to start at the beginning.  Similarly, the command L[5:11] can be replaced by L[5:].  The empty closing index tells Python to end the slice and the end.  This is helpful if one doesn't know where the list ends.
Can you write Python code for it?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
This is pretty close!  It seems like prime numbers are about evenly distributed between red and blue.  Their remainder after division by 4 is about as likely to be 1 as it is to be 3.  In fact, it is proven that asymptotically the ratio between the number of red primes and the number of blue primes approaches 1.  However, Chebyshev noticed a persistent slight bias towards blue primes along the way.
Some of the deepest conjectures in mathematics relate to the prime counting function $\pi(x)$.  Here $\pi(x)$ is the number of primes between 1 and $x$ (inclusive).  So $\pi(2) = 1$ and $\pi(3) = 2$ and $\pi(4) = 2$ and $\pi(5) = 3$.  One can compute a value of $\pi(x)$ pretty easily using a list comprehension.
Can you write Python code for it?

def primes_upto(x):
    return len([p for p in primes if p <= x]) # List comprehension recovers the primes up to x.

primes_upto(1000)  # There are 168 primes between 1 and 1000.


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
This isn't too bad!  It takes a fraction of a second to identify the primes up to a million, and a smaller fraction of a second to count them!  But we can do a little better.
The first improvement is to take care of the even numbers first.  If we count carefully, then the sequence 4,6,8,...,n (ending at n-1 if n is odd) has the floor of (n-2)/2 terms.  Thus the line flags[4::2] = [False] * ((n-2)//2) will set all the flags to False in the sequence 4,6,8,10,...  From there, we can begin sieving by odd primes starting with 3.
The next improvement is that, since we've already sieved out all the even numbers (except 2), we don't have to sieve out again by even multiples.  So when sieving by multiples of 3, we don't have to sieve out 9,12,15,18,21,etc..  We can just sieve out 9,15,21,etc..  When p is an odd prime, this can be taken care of with the code flags[p*p::2*p] = [False] * len(flags[p*p::2*p]).
Can you write Python code for it?

def isprime_list(n):
    '''
    Return a list of length n+1
    with Trues at prime indices and Falses at composite indices.
    '''
    flags = [True] * (n+1)  # A list [True, True, True,...] to start.
    flags[0] = False  # Zero is not prime.  So its flag is set to False.
    flags[1] = False  # One is not prime.  So its flag is set to False.
    flags[4::2] = [False] * ((n-2)//2)
    p = 3
    while p <= sqrt(n):  # We only need to sieve by p is p <= sqrt(n).
        if flags[p]:  # We sieve the multiples of p if flags[p]=True.
            flags[p*p::2*p] = [False] * len(flags[p*p::2*p]) # Sieves out multiples of p, starting at p*p.
        p = p + 2 # Try the next value of p.  Note that we can proceed only through odd p!

    return flags

%timeit sum(isprime_list(10**6))  # How much did this speed it up?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
To carry out serious analysis, we will use the method of list comprehension to place our population into "bins" for statistical analysis.  Our first type of list comprehension has the form [x for x in LIST if CONDITION].  This produces the list of all elements of LIST satisfying CONDITION.  It is similar to list slicing, except we pull out terms from the list according to whether a condition is true or false.
For example, let's divide the (odd) primes into two classes.  Red primes will be those of the form 4n+1.  Blue primes will be those of the form 4n+3.  In other words, a prime p is red if p%4 == 1 and blue if p%4 == 3.  And the prime 2 is neither red nor blue.
Can you write Python code for it?

redprimes = [p for p in primes if p%4 == 1] # Note the [x for x in LIST if CONDITION] syntax.
blueprimes = [p for p in primes if p%4 == 3]

print('Red primes:',redprimes[:20]) # The first 20 red primes.
print('Blue primes:',blueprimes[:20]) # The first 20 blue primes.

print("There are {} red primes and {} blue primes, up to 1 million.".format(len(redprimes), len(blueprimes)))


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
We can change a slice of a list with a single assignment, even when that slice does not consist of consecutive terms.  Try to predict what the following commands will do.
Can you write Python code for it?

print(L)  # Let's see what the list looks like before.

L[::2] = ['A','B','C','D','E','F']  # What was L[::2] before this assignment?

print(L)  # What do you predict?


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
We can change an entire slice of a list with a single assignment.  Let's change the first two terms of L in one line.
Can you write Python code for it?

L[:2] = ['Pancakes', 'Ham']  # What was L[:2] before?

print(L) # Oh... what have we done!

L[0]

L[1]

L[2]


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
What have we done?  It is useful to try out this method on a short list.
Can you write Python code for it?

L = [1,3,7,20]  # A nice short list.

print(L[:-1])
print(L[1:])


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
You can also use Python to find the sum of a list of numbers.
Can you write Python code for it?

sum([1,2,3,4,5])

sum(range(100))  # Be careful.  This is the sum of which numbers?  # The sum function can take lists or ranges.


I am working on the file "P3wNT Notebook 3.ipynb".
The first task is:
You might notice that the format looks a bit different from a list.  Indeed, if you check type(x_values), it's not a list but something else called a numpy array.  Numpy is a package that excels with computations on large arrays of data.  On the surface, it's not so different from a list.  The numpy.linspace command is a convenient way of producing an evenly spaced list of inputs.
The big difference is that operations on numpy arrays are interpreted differently than operations on ordinary Python lists.  Try the two commands for comparison.
Can you write Python code for it?

[1,2,3] + [1,2,3]

x_values + x_values

y_values = x_values * x_values  # How is multiplication interpreted on numpy arrays?
print(y_values)


I am working on the file "P3wNT Notebook 4.ipynb".
The first task is:

Multiplicative functions
A multiplicative function is a function $f(n)$ which takes positive integer input $n$, and which satisfies $f(1) = 1$ and $f(ab) = f(a) f(b)$ whenever $a$ and $b$ are coprime.  A good example is the divisor-sum function, implemented below.
Can you write Python code for it?

def divisor_sum(n):
    S = 0 # Start the sum at zero.
    for d in range(1,n+1):  # potential divisors between 1 and n.
        if n%d == 0:
            S = S + d
    return S

divisor_sum(100)  # The sum 1 + 2 + 4 + 5 + 10 + 20 + 25 + 50 + 100

%timeit divisor_sum(730) # Let's see how quickly this runs.


I am working on the file "P3wNT Notebook 4.ipynb".
The first task is:
A perfect number is a positive integer which equals the sum of its proper factors (its positive factors, not including itself).  Thus a number $n$ is perfect if its divisor sum equals $2n$.  This can be implemented in a very short function.
Can you write Python code for it?

def is_perfect(n):
    return divisor_sum(n) == 2*n

is_perfect(10)

is_perfect(28)


I am working on the file "P3wNT Notebook 4.ipynb".
The first task is:
By having each list entry point to the same location in memory, Python avoids having to fill three blocks of memory with the same number 19.
Python objects can have methods attached to them.  Methods are functions which can utilize and change the data within an object.  The basic syntax for using methods is <object>.<method>().  Here are two examples to get started:  The keys and values of a dictionary can be recovered using the keys() and values() methods.
Can you write Python code for it?

nemo.keys()  # What are the keys of nemo?

nemo.values()  # What are the values of nemo?


I am working on the file "P3wNT Notebook 4.ipynb".
The first task is:
Dictionaries can have values of any type, and their keys can be numbers or strings.  In this case, the keys are all strings, while the values include strings and integers.  In this way, dictionaries are useful for storing properties of different kinds -- they can be used to store [records](https://en.wikipedia.org/wiki/Record_(computer_science%29 ), as they are called in other programming languages.
An interlude on Python objects
We have discussed how Python stores data of various types:  int, bool, str, list, dict, among others.  But now seems like a good time to discuss the fundamental "units" which are stored:  these are called Python objects.  If you have executed the cells above, Python is currently storing a lot of objects in your computer's memory.  These objects include nemo and L.  Also L[0] is an object and nemo['age'] is an object.  Each of these objects are occupying a little space in memory.
We reference these objects by the names we created, like nemo and L.  But for internal purposes, Python assigns every object a unique ID number.  You can see an object's ID number with the id function.
Can you write Python code for it?