I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
In the first method plot_predicted_map() at line 3, we call the  predict() method of DNNClassifier class to get an array of prediction results (10,000 rows) like [1 0 0 1 ... 0 0 1 0] where 1 means that the neural network believes the geolocation is in Manhattan, and 0 means it's not. By using this array as an indexer for selecting lat and lng pairs in each class, the method plots geolocations predicted as Manhattan in blue dots and others in yellow dots.
In the second method print_accuracy() at line 9, we call the evaluate() method of DNNClassifier class to calculate the accuracy of the prediction using the test data latlng_test and is_mt_test and print it.
After defining these two methods, we call the fit() method of DNNClassifier class at line 14 to train the model for just one step. A step in the fit() method moves the weights and bias in the neural network only a little in the direction that reduces the network error. However, it usually it takes thousands of steps for neural networks to find the best weights and bias. So, what you are effectively seeing is that the neural network in the initial state (= before the training) acheives a very low accuracy and cannot classify the Manhattan locations properly.
Train the neural network
Finally, let's actually train the neural network! This time, we will train the network by calling fit() method for 500 steps with the training data latlng_train and is_mt_train. Every 100 steps, we will call plot_predicted_map() and print_accuracy() to show the current accuracy of the network. Run the cell below and wait for a while until the message "Finished" is printed. You will see the network continually tries to move the weights and bias in small steps to minimize the error and find the best position of the line for classifying geolocations in Manhattan. The final accuracy should be as high as 97%.
Can you write Python code for it?

steps = 100
for i in range (1, 6):
  dnnc.fit(x=latlng_train, y=is_mt_train, steps=steps)
  plot_predicted_map()
  print('Steps: ' + str(i * steps))
  print_accuracy()

print('\nTraining Finished.')


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
Lab: NumPy basics
(You can skip this lab if you know how to use NumPy)
You might notice that we just used NumPy for extracting the results. NumPy is the most popular Python library for numerical calculations. For machine learning with Python, many people use NumPy for wide variety of numerical operations, including the basic array operations such as reshaping, merging, splitting, filtering, slicing and indexing. Many of TensorFlow APIs are also influenced by NumPy and use similar concepts. If you want to learn machine learning and TensorFlow with Python, we recommend you also learn some of the basics of NumPy too.
In this lab, Let's try a few basic array operations with NumPy. Run the cell below and see what kind of numpy array will be created.
Can you write Python code for it?

# create an numpy array with numbers from 0 to 14
A = np.arange(15)
print(A)


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
Lab: Try training the neural network a couple of times

Go back to the section "Define a single layer neural network" and run the following cells again to train the network from scratch
Repeat the training a couple of times and confirm that the network's max accuracy is around 97%
Discuss with your buddy the key reason that the single layer network can't achieve accuracy higher than 97%. The single layer neural network is also known as Perceptron. You may refer to the Wikipedia page for Perceptron to learn more about its characteristics and limitations.

2-4. Train a Deep Neural Network with TensorFlow
You just saw that the network can only draw a straight line on the map and classify whether a location is in Manhattan or not. This is so-called Linear Classification. That is the limitation of the single layer neural network and you can only achieve around 97% accuracy because the straight line (linear classification) can't split the geolocation points between Manhattan and Brooklyn with the necessary curved boundary.
We must go deeper. Let's define a deep neural network (DNN). Run the cell below to define a new DNNClassifier.
Can you write Python code for it?

dnnc = tf.contrib.learn.DNNClassifier(
  feature_columns=feature_columns,
  hidden_units=[20, 20, 20, 20],
  n_classes=2)
dnnc


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
Lab: check the standardized feature values

Print mean and standard deviation values on both latitude and longitude of variable latlng_std by using NumPy and confirm the mean is almost 0 and standard deviation is 1.
Can you write Python code for it?


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
Let's take a look at what's inside the result. Run the cell below and check the variable is_mt has an array of 1s and 0s representing each geolocation is in Manhattan or not, and the variable latlng has an array of pairs of latitude and longitude.
Can you write Python code for it?

import numpy as np
is_mt = nyc_cols[:,0].astype(np.int32) # read the 0th column (is_mt) as int32
latlng = nyc_cols[:,1:3].astype(np.float32) # read the 1st and 2nd column (latitude and longitude) as float32
print("Is Manhattan: " + str(is_mt))
print("\nLat/Lng: \n\n" + str(latlng))


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
Now, add the necessary new code in the following cells, and run them, to get the result described in the comments with NumPy. You should refer to the NumPy Quickstart to learn how to get the results required.
Can you write Python code for it?

# reshape the array A into an array with shape in 3 rows and 5 columns,
# set it to variable A, and print it.

# *** ADD YOUR CODE HERE ***

print(A)

# expected result:
# [[ 0  1  2  3  4]
#  [ 5  6  7  8  9]
#  [10 11 12 13 14]]

# print() the shape, data type name, size (total number of elements) of the array A

# *** ADD YOUR CODE HERE ***

# expected result:
# (3, 5)
#  int64
# 15

# multiply the array A by the number 2 and print the result

# *** ADD YOUR CODE HERE ***

# expected result:
# [[ 0  2  4  6  8]
#  [10 12 14 16 18]
#  [20 22 24 26 28]]

# create a new array that has the same shape as the array A filled with zeros, and print it

# *** ADD YOUR CODE HERE ***

# expected result:
# [[ 0.  0.  0.  0.  0.]
#  [ 0.  0.  0.  0.  0.]
#  [ 0.  0.  0.  0.  0.]]

# create a new array that has the elements in the right-most column of the array A

# *** ADD YOUR CODE HERE ***

# expected result:
# [ 4  9 14]

# Collect elements in array B with an index "I % 2 == 0" and print it
B = np.arange(10)
I = np.arange(10)

# *** ADD YOUR CODE HERE ***

# expected result:
# [0 2 4 6 8]


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
Plot the training data with Matplotlib
Now, all the preprocessing on the training data have been done. Let's see what the data looks like by using Matplotlib, the popular visualization library for Python. In this case we will use scatter() method to plot dots with the pairs of latitude and longitude. Run the cell below and see the plot.
Can you write Python code for it?

import matplotlib.pyplot as plt
lat = latlng_std[:,0]
lng = latlng_std[:,1]
plt.scatter(lng[is_mt == 1], lat[is_mt == 1], c='b') # plot points in Manhattan in blue
plt.scatter(lng[is_mt == 0], lat[is_mt == 0], c='y') # plot points outside Manhattan in yellow
plt.show()


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
Preprocess the training data on BigQuery
In this codelab, we do not care about the car accidents. We just wanted to use the data for getting pairs of "latitude", "longitude" and "Is it Manhattan or not" values. So, we want to do the following preprocessing on this raw data:

Add a column "is_mt" that returns 1 or 0 to indicate if the borough is Manhattan or not
Remove rows without borough info
Remove rows without longitude/latitude info
Remove rows for Bronx (because it's too close to Manhattan and hard to classify with single layer neural network!)
Randomly shuffule all the rows (for making the training data even)
Select only the 10,000 rows

So, our SQL with the preprocessing will look like the following. Select the cell below and run it. Please note that this only defines the SQL module "nyc_collisions" that will be used later and does not output anything.
Can you write Python code for it?

%%sql --module nyc_collisions
SELECT
  IF(borough = 'MANHATTAN', 1, 0) AS is_mt,
  latitude,
  longitude
FROM
  `bigquery-public-data.new_york.nypd_mv_collisions`
WHERE
  LENGTH(borough) > 0
  AND latitude IS NOT NULL AND latitude != 0.0
  AND longitude IS NOT NULL AND longitude != 0.0
  AND borough != 'BRONX'
ORDER BY
  RAND()
LIMIT
  10000


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
The hidden layers give the power
The only difference from the last DNNClassifier definition is the hidden_units parameter which defines 4 hidden layers with 20 neurons each. As the network has total of 5 layers, we're now working with a deep neural network ("deep" means you have layers more than 2).
Let's see how the deep neural network works. Run the cell below and wait for a couple of minutes until it finishes training.
Can you write Python code for it?

steps = 30
for i in range (1, 6):
  dnnc.fit(x=latlng_train, y=is_mt_train, steps=steps)
  plot_predicted_map()
  print 'Steps: ' + str(i * steps)
  print_accuracy()

print('\nTraining Finished.')


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
This codelab requires TensorFlow 1.0 or above. If you see older versions such as 0.11.0rc0, please follow the instruction below to update your local Datalab.

Stop your Datalab by pressing Ctrl+C on the console
Run the following command on the console
```
docker pull gcr.io/cloud-datalab/datalab:local
```

Run the docker run command to restart local Datalab

2-1. Importing the training data from BigQuery
To prepare for the analysis, we will download a training data from BigQuery, a fully managed scalable data warehouse service on Google Cloud. BigQuery provides many kinds of public datasets which makes it a useful datasource for learning data analytics with TensorFlow.
One of the public datasets is NYPD Motor Vehicle Collisions Data which collects all the car accidents happened in NYC from 2012 to the present. In this codelab, we will use it for getting 10,000 pairs of "borough" column and "latitude/longitude" columns.
Let's take a look at the data by executing a BigQuery SQL query. In Cloud Datalab, you can execute BigQuery commands by using the "%%sql" command (see this doc to learn more about the BigQuery commands). Select the cell below and run the query by clicking "Run" on the menu.
Can you write Python code for it?

%%sql -d standard
SELECT
  timestamp,
  borough,
  latitude,
  longitude
FROM
  `bigquery-public-data.new_york.nypd_mv_collisions`
ORDER BY
  timestamp DESC
LIMIT
  15


I am working on the file "2. Classify Manhattan with TensorFlow.ipynb".
The first task is:
You can see that the geolocations in Manhattan are plotted as blue dots, and others are yellow dots. Also, latitudes and longitudes are scaled to have 0 as the center.
Split the data into "Training Data" and "Test Data"
Before start training the neural network model, we need to separate out a part of the training data as test data. The test data will be used for checking accuracy of classifications by the model after training.  This is common practice in machine learning, so that the performance of your model can be accurately evaluated.
Run the cell below and split the data into 8,000 pairs of training data and 2,000 pairs of test data.
Can you write Python code for it?

# 8,000 pairs for training
latlng_train = latlng_std[0:8000]
is_mt_train = is_mt[0:8000]

# 2,000 pairs for test
latlng_test = latlng_std[8000:10000]
is_mt_test = is_mt[8000:10000]

print("Split finished.")


I am working on the file "2. Data Analysis and Visualization/Analyzing Thanksgiving Dinner/Thanksgiving survey.ipynb".
The first task is:
Travel by income
Hypothesis that people with lower income travel more, because they might be younger does not seem to be valid (assumption that younger people have lower income may be wrong, we could use values from age instead).
Can you write Python code for it?

def thanksgiving_and_friends(data, aggregated_column):
    return data.pivot_table(index="Have you ever tried to meet up with hometown friends on Thanksgiving night?",
                     columns='Have you ever attended a "Friendsgiving?"',
                     values=aggregated_column)

print(thanksgiving_and_friends(celebrate, "age"))

print(thanksgiving_and_friends(celebrate, "income"))


I am working on the file "2. Data Analysis and Visualization/Visualizing Earnings Based On College Majors/Visualizing Earnings Based On College Majors.ipynb".
The first task is:
Questions

What penrcent of majors are predominantly male? Predominantly female?
This is rough estimate from ShareWomen figure - assuming predominant means is around 66% and above. Than 20% of majors are female predominant and 30% percent are male predominant.

What's the most common salary range?
30,000\$ - 35,000\$
Can you write Python code for it?

scatter_matrix(recent_grads[["Sample_size", "Median"]], figsize=(10,10))

scatter_matrix(recent_grads[["Sample_size", "Median", "Unemployment_rate"]], figsize=(10,10))

recent_grads.sort_values("ShareWomen", ascending=False)[0:10].plot.bar(x="Major", y="ShareWomen")

recent_grads.sort_values("ShareWomen", ascending=True)[0:10].plot.bar(x="Major", y="ShareWomen")

recent_grads.sort_values("Unemployment_rate", ascending=False)[0:10].plot.bar(x="Major", y="Unemployment_rate")

recent_grads.sort_values("Unemployment_rate", ascending=True)[0:10].plot.bar(x="Major", y="Unemployment_rate")


I am working on the file "2. Deep Learning Frameworks/2.2 Introduction - Tensorflow.ipynb".
The first task is:
Basic Operations (Examples)
Can you write Python code for it?

matrix1 = np.array([(2,2,2),(2,2,2),(2,2,2)],dtype='float32')
matrix2 = np.array([(1,1,1),(1,1,1),(1,1,1)],dtype='float32')

tf_mat1 = tf.constant(matrix1)
tf_mat2 = tf.constant(matrix2)

matrix_product = tf.matmul(tf_mat1, tf_mat2)
matrix_sum = tf.add(tf_mat1, tf_mat2)

matrix_det = tf.matrix_determinant(matrix2)

with tf.Session() as sess:
    prod_res = sess.run(matrix_product)
    sum_res = sess.run(matrix_sum)
    det_res = sess.run(matrix_det)

print("matrix1*matrix2 : \n", prod_res)
print("matrix1+matrix2 : \n", sum_res)
print("det(matrix2) : \n", det_res)


I am working on the file "2. Deep Learning Frameworks/2.2 Introduction - Tensorflow.ipynb".
The first task is:
Computing the Gradient

Gradients are free!
Can you write Python code for it?

x = tf.placeholder(tf.float32)
y = tf.log(x)
var_grad = tf.gradients(y, x)
with tf.Session() as session:
    var_grad_val = session.run(var_grad, feed_dict={x:2})
    print(var_grad_val)


I am working on the file "2. Deep Learning Frameworks/2.2 Introduction - Tensorflow.ipynb".
The first task is:
Data Flow Graph

(IDEA)
_A Machine Learning application is the result of the repeated computation of complex mathematical expressions, thus
we could describe this computation by using a Data Flow Graph

Data Flow Graph: a graph where:

each Node represents the instance of a mathematical operation
multiply, add, divide

each Edge is a multi-dimensional data set (tensors) on which the operations are performed.

Tensorflow Graph Model

Node: In TensorFlow, each node represents the instantion of an operation.

Each operation has inputs (>= 2) and outputs >= 0.

Edges: In TensorFlow, there are two types of edge:

Data Edges:
They are carriers of data structures (tensors), where an output of one operation (from one node) becomes the input for another operation.
Dependency Edges: These edges indicate a control dependency between two nodes (i.e. "happens before" relationship).
Let's suppose we have two nodes A and B and a dependency edge connecting A to B. This means that B will start its operation only when the operation in A ends.

Tensorflow Graph Model (cont.)

Operation: This represents an abstract computation, such as adding or multiplying matrices.

An operation manages tensors, and It can just be polymorphic: the same operation can manipulate different tensor element types.
For example, the addition of two int32 tensors, the addition of two float tensors, and so on.

Kernel: This represents the concrete implementation of that operation.

A kernel defines the implementation of the operation on a particular device.
For example, an add matrix operation can have a CPU implementation and a GPU one.

Tensorflow Graph Model Session
Session: When the client program has to establish communication with the TensorFlow runtime system, a session must be created.
As soon as the session is created for a client, an initial graph is created and is empty. It has two fundamental methods:

session.extend: To be used during a computation, requesting to add more operations (nodes) and edges...
Can you write Python code for it?

a = tf.constant(5, name="a")
b = tf.constant(45, name="b")
y = tf.Variable(a+b*2, name='y')
model = tf.global_variables_initializer()

with tf.Session() as session:
    # Merge all the summaries collected in the default graph.
    merged = tf.summary.merge_all()

    # Then we create `SummaryWriter`.
    # It will write all the summaries (in this case the execution graph)
    # obtained from the code's execution into the specified path”
    writer = tf.summary.FileWriter("tmp/tf_logs_simple", session.graph)
    session.run(model)
    print(session.run(y))


I am working on the file "2. Deep Learning Frameworks/2.2 Introduction - Tensorflow.ipynb".
The first task is:
Handling Tensors
Can you write Python code for it?

%matplotlib inline

import matplotlib.image as mp_image
filename = "../imgs/keras-logo-small.jpg"
input_image = mp_image.imread(filename)

#dimension
print('input dim = {}'.format(input_image.ndim))
#shape
print('input shape = {}'.format(input_image.shape))

import matplotlib.pyplot as plt
plt.imshow(input_image)
plt.show()


I am working on the file "2. Deep Learning Frameworks/2.2 Introduction - Tensorflow.ipynb".
The first task is:
Slicing
Can you write Python code for it?

my_image = tf.placeholder("uint8",[None,None,3])
slice = tf.slice(my_image,[10,0,0],[16,-1,-1])

with tf.Session() as session:
    result = session.run(slice,feed_dict={my_image: input_image})
    print(result.shape)

plt.imshow(result)
plt.show()


I am working on the file "2. Deep Learning Frameworks/2.2 Introduction - Tensorflow.ipynb".
The first task is:
Two Dimensional Tensor (Matrix)
Can you write Python code for it?

tensor_2d = np.arange(16).reshape(4, 4)
print(tensor_2d)
tf_tensor = tf.placeholder(tf.float32, shape=(4, 4))
with tf.Session() as sess:
    print(sess.run(tf_tensor, feed_dict={tf_tensor: tensor_2d}))


I am working on the file "2. Deep Learning Frameworks/2.3 Introduction to Keras.ipynb".
The first task is:
Learning in a TF Session
Can you write Python code for it?

LOGDIR = "/tmp/logistic_logs"
import os, shutil
if os.path.isdir(LOGDIR):
    shutil.rmtree(LOGDIR)
os.mkdir(LOGDIR)

# Plug TensorBoard Visualisation
writer = tf.summary.FileWriter(LOGDIR, graph=tf.get_default_graph())

for var in tf.get_collection(tf.GraphKeys.SUMMARIES):
    print(var.name)

summary_op = tf.summary.merge_all()
print('Summary Op: ' + summary_op)

# Launch the graph
with tf.Session() as session:
    # Initializing the variables
    session.run(tf.global_variables_initializer())

    cost_epochs = []
    # Training cycle
    for epoch in range(training_epochs):
        _, summary, c = session.run(fetches=[optimizer, summary_op, cost],
                                    feed_dict={x: X_train, y: Y_train})
        cost_epochs.append(c)
        writer.add_summary(summary=summary, global_step=epoch)
        print("accuracy epoch {}:{}".format(epoch, accuracy.eval({x: X_train, y: Y_train})))

    print("Training phase finished")

    #plotting
    plt.plot(range(len(cost_epochs)), cost_epochs, 'o', label='Logistic Regression Training phase')
    plt.ylabel('cost')
    plt.xlabel('epoch')
    plt.legend()
    plt.show()

    prediction = tf.argmax(activation, 1)
    print(prediction.eval({x: X_test}))

%%bash
python -m tensorflow.tensorboard --logdir=/tmp/logistic_logs


I am working on the file "2. Deep Learning Frameworks/2.3 Introduction to Keras.ipynb".
The first task is:
Model (Introducing Tensorboard)
Can you write Python code for it?

# Construct (linear) model
with tf.name_scope("model") as scope:
    # Set model weights
    W = tf.Variable(tf.zeros([dims, nb_classes]))
    b = tf.Variable(tf.zeros([nb_classes]))
    activation = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax

    # Add summary ops to collect data
    w_h = tf.summary.histogram("weights_histogram", W)
    b_h = tf.summary.histogram("biases_histograms", b)
    tf.summary.scalar('mean_weights', tf.reduce_mean(W))
    tf.summary.scalar('mean_bias', tf.reduce_mean(b))

# Minimize error using cross entropy
# Note: More name scopes will clean up graph representation
with tf.name_scope("cost_function") as scope:
    cross_entropy = y*tf.log(activation)
    cost = tf.reduce_mean(-tf.reduce_sum(cross_entropy,reduction_indices=1))
    # Create a summary to monitor the cost function
    tf.summary.scalar("cost_function", cost)
    tf.summary.histogram("cost_histogram", cost)

with tf.name_scope("train") as scope:
    # Set the Optimizer
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)


I am working on the file "2. Deep Learning Frameworks/2.3 Introduction to Keras.ipynb".
The first task is:
Multi-Layer Fully Connected Networks

Forward and Backward Propagation

Q: How hard can it be to build a Multi-Layer Fully-Connected Network with keras?
A: It is basically the same, just add more layers!
Can you write Python code for it?

model = Sequential()
model.add(Dense(100, input_shape=(dims,)))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.compile(optimizer='sgd', loss='categorical_crossentropy')
model.summary()

model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20,
          batch_size=128, verbose=True)


I am working on the file "2. Deep Learning Frameworks/2.3 Introduction to Keras.ipynb".
The first task is:
Now lets understand:

The core data structure of Keras is a model, a way to organize layers. The main type of model is the Sequential model, a linear stack of layers.

What we did here is stacking a Fully Connected (Dense) layer of trainable weights from the input to the output and an Activation layer on top of the weights layer.
Dense
```python
from keras.layers.core import Dense
Dense(units, activation=None, use_bias=True,
      kernel_initializer='glorot_uniform', bias_initializer='zeros',
      kernel_regularizer=None, bias_regularizer=None,
      activity_regularizer=None, kernel_constraint=None, bias_constraint=None)
```

units: int > 0.

init: name of initialization function for the weights of the layer (see initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.

activation: name of activation function to use (see activations), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).

weights: list of Numpy arrays to set as initial weights. The list should have 2 elements, of shape (input_dim, output_dim) and (output_dim,) for weights and biases respectively.

kernel_regularizer: instance of WeightRegularizer (eg. L1 or L2 regularization), applied to the main weights matrix.

bias_regularizer: instance of WeightRegularizer, applied to the bias.

activity_regularizer: instance of ActivityRegularizer, applied to the network output.

kernel_constraint: instance of the constraints module (eg. maxnorm, nonneg), applied to the main weights matrix.

bias_constraint: instance of the constraints module, applied to the bias.

use_bias: whether to include a bias (i.e. make the layer affine rather than linear).

(some) others keras.core.layers

keras.layers.core.Flatten()
keras.layers.core.Reshape(target_shape)
keras.layers.core.Permute(dims)

```python
model =...
Can you write Python code for it?

model.summary()

from sklearn.model_selection import train_test_split
from keras.callbacks import EarlyStopping, ModelCheckpoint

X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.15, random_state=42)

fBestModel = 'best_model.h5'
early_stop = EarlyStopping(monitor='val_loss', patience=2, verbose=1)
best_model = ModelCheckpoint(fBestModel, verbose=0, save_best_only=True)

model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=50,
          batch_size=128, verbose=True, callbacks=[best_model, early_stop])


I am working on the file "2. Deep Learning Frameworks/2.3 Introduction to Keras.ipynb".
The first task is:
Simplicity is pretty impressive right? :)
Theano:
shape = (channels, rows, cols)
Tensorflow:
shape = (rows, cols, channels)
image_data_format : channels_last | channels_first
Can you write Python code for it?


I am working on the file "2. Deep Learning Frameworks/2.3 Introduction to Keras.ipynb".
The first task is:
Using Tensorflow
Can you write Python code for it?

import tensorflow as tf

# Parameters
learning_rate = 0.01
training_epochs = 25
display_step = 1

# tf Graph Input
x = tf.placeholder("float", [None, dims])
y = tf.placeholder("float", [None, nb_classes])

x


I am working on the file "2. Deep Learning Frameworks/2.3 Introduction to Keras.ipynb".
The first task is:
Your Turn!
Hands On - Keras Fully Connected
Take couple of minutes and try to play with the number of layers and the number of parameters in the layers to get the best results.
Can you write Python code for it?

model = Sequential()
model.add(Dense(100, input_shape=(dims,)))

# ...
# ...
# Play with it! add as much layers as you want! try and get better results.

model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.compile(optimizer='sgd', loss='categorical_crossentropy')

model.summary()

model.fit(X_train, Y_train, validation_data = (X_val, Y_val), epochs=20,
          batch_size=128, verbose=True)


I am working on the file "2. Hashing Algorithms.ipynb".
The first task is:
*Exercise *

Let's explore what other hashing algorithms are available and use a different one.
Change "PyCon2017", to PyCon 2017" and see check the result

Python Builtin Support
"Additional algorithms may also be available depending upon the OpenSSL library that Python uses on your platform. On most platforms the sha3_224(), sha3_256(), sha3_384(), sha3_512(), shake_128(), shake_256() are also available."
source: https://docs.python.org/3/library/hashlib.html
Can you write Python code for it?

import hashlib
sha256 = hashlib.sha256()
sha256.update(b"PyCon2017")

msg_digest = sha256.digest()
# Notice the output size of the digest
print ("msg_digest:", len(msg_digest), len(msg_digest) * 8)
print ("base64 encoding:", base64.b64encode(msg_digest))


I am working on the file "2. Hashing Algorithms.ipynb".
The first task is:
*Exercise *

Let's see what other hashing algorithms our python supports.

Hash-based message authentication code (HMAC)
HMACs are used for message authentications combined with a secret key. This provide integrity checking and authentication. An attacker can not forge the digest without knowing the secret key.

image source: wikipedia
$ HMAC(K,m) = H((K\oplus opad) || H((K \oplus ipad) || m))$
HMAC Calculation
Can you write Python code for it?

from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hmac, hashes
import os
import base64

hmc_key = k = os.urandom(16)
hmc = hmac.HMAC(hmc_key, hashes.SHA1(), default_backend())
hmc.update(b"PyCon2017")
hmc_sig = hmc.finalize()
print (base64.b64encode(hmc_sig))


I am working on the file "2. Hashing Algorithms.ipynb".
The first task is:
HMAC Verification
Can you write Python code for it?

# Verification Successufl
hmc = hmac.HMAC(hmc_key, hashes.SHA1(), default_backend())
hmc.update(b"PyCon2017")
hmc.verify(hmc_sig)

# Verification Fails
hmc = hmac.HMAC(hmc_key, hashes.SHA1(), default_backend())
hmc.update(b"PyCon2017")
hmc.verify(hmc_sig+b"1")


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
Aquí hay unas cosas que notar:
1. La estructura inicial de una expresión condicional es if <condición> :
2. Después de esto hay un espacio que nos indica que el código que escribiremos forma parte del condicional
3. La expresión se puede terminar ahí o podemos darle otra instrucción con else: (nótese que else se encuentra al mismo nivel que if)
4. El código que forma parte de else lleva también un espacio (estos espacios son llamados indentación)

Pregunta 6
¿Qué pasa si tengo más de una condición que debe ser cumplida?
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
Como lo has de haber notado, el elemento nuevo que agregaste se va al final de la lista. Si quisieras que tu nuevo elemento estuviera en una posición específica tendrías que usar el método .insert(índice, elemento)
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
Expresiones condicionales
Imagina que estás en una calle a punto de cruzarla, ¿Qué es lo que te hace decidir si la cruzas o no?
Lo mismo sucede con python. En este caso le tienes que dar ciertas condiciones que deben ser cumplidas, esto es que al evaluar una expresión, esta resulte en un cierto valor o en True para que siga con una operación. Si es necesario también se le pueden dar instrucciones para cuando la condición no sea cumplida.
Veamos unos ejemplos
Can you write Python code for it?

#Crea un elemento nuevo en una lista si la longitud es menor a 7 elementos
if len(lista1) < 7:
    lista1.append('elemento')

#Veamos qué pasó con lista1
lista1

del(lista1[5])

lista1

#Si el primer valor de una lista es integer conviértelo a float, de lo contratio conviértelo a integer
if type(lista2[0]) is int:
    lista2[0] = float(lista2[0])
    lista2[1] = float(lista2[1])
elif lista2[2] == 10:
    print(lista2[2])
elif lista2[2] > 10:
    print("si")

#Veamos cómo quedó la lista 2
lista2


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
Hay ocasiones en las que no quieres agregar nuevos elementos, pero alterar el valor que tenías de los mismos, para esto puedes accesar por índice al elemento que quieras y directamente asignarle otro valor
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
La parte "mutable" de las listas significa que podemos alterar el contenidos de estas al agregar elementos a la colección y/o cambiando el valor de algunos elementos.
Veamos como agregar elementos a las listas. Para esto usaremos el método .append(elemento) que tienen las listas
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
Pregunta 2.
¿Es posible ir saltando un número en el rango con el que seleccionamos las listas? Esto es cómo pedirle a python (si es posible) que me de los elementos con índices pares del rango 1:6 (los índices 2, 4 y 6)
Can you write Python code for it?

#Si no sabemos un índice, pero sabemos que hay un valor en una lista podemos preguntarle a python cuál es ese índice
#esto nos dice el índice del primer elemento que se encuentre con ese valor
#(si hay valores repetidos nos da el de menor índice)
lista1.index(2.0)

lista2.index(5)


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
Pregunta 3.
¿Qué resultaría de la siguiente expresión? lista1[::-1]
La parte "mutable" de las listas significa que podemos alterar el contenidos de estas al agregar elementos a la colección y/o cambiando el valor de algunos elementos.
Veamos como agregar elementos a las listas. Para esto usaremos el método .append(elemento) que tienen las listas
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
Pregunta 5
Crea una lista nueva que se llame "copia" de la lista2
Can you write Python code for it?

lista3.append(True)

lista3.remove(True)

lista3


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
Vamos a explorar ahora la parte de los índices y mutabilidad en una lista
Para acceder a un valor en una lista podemos hacer uso de los índices. Los índices empiezan desde cero.
Can you write Python code for it?

#Para especificar que queremos acceder a un valor en python, el objeto es seguido de corchetes con el índice deseado
lista1[0]

lista2[2]

#¿Qué pasa si damos un índice que no tiene un valor?
lista1[4]


I am working on the file "2. Lists_and_conditionals/.ipynb_checkpoints/lists and conditionals-checkpoint.ipynb".
The first task is:
¡Obvio! nos da un error de índice... Y nos dice qué está fuera del rango de valores que tenemos guardados en la lista
También tenemos acceso a un rango de valores
Can you write Python code for it?

#Vamos a pedirle a python que nos muestre los primero tres elementos de la lista1
lista1[0:3]


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Aquí hay unas cosas que notar:
1. La estructura inicial de una expresión condicional es if <condición> :
2. Después de esto hay un espacio que nos indica que el código que escribiremos forma parte del condicional
3. La expresión se puede terminar ahí o podemos darle otra instrucción con else: (nótese que else se encuentra al mismo nivel que if)
4. El código que forma parte de else lleva también un espacio (estos espacios son llamados indentación)

Pregunta 6
¿Qué pasa si tengo más de una condición que debe ser cumplida?
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Como lo has de haber notado, el elemento nuevo que agregaste se va al final de la lista. Si quisieras que tu nuevo elemento estuviera en una posición específica tendrías que usar el método .insert(índice, elemento)
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Expresiones condicionales
Imagina que estás en una calle a punto de cruzarla, ¿Qué es lo que te hace decidir si la cruzas o no?
Lo mismo sucede con python. En este caso le tienes que dar ciertas condiciones que deben ser cumplidas, esto es que al evaluar una expresión, esta resulte en un cierto valor o en True para que siga con una operación. Si es necesario también se le pueden dar instrucciones para cuando la condición no sea cumplida.
Veamos unos ejemplos
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Hay ocasiones en las que no quieres agregar nuevos elementos, pero alterar el valor que tenías de los mismos, para esto puedes accesar por índice al elemento que quieras y directamente asignarle otro valor
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
La parte "mutable" de las listas significa que podemos alterar el contenidos de estas al agregar elementos a la colección y/o cambiando el valor de algunos elementos.
Veamos como agregar elementos a las listas. Para esto usaremos el método .append(elemento) que tienen las listas
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Pregunta 1.
Ya vimos que lista1 posee elementos con índices del 0 al 3, ¿por qué te imaginas que en la expresión anterior no nos mostró el último elemento si le pedimos ver del 0:3?
Can you write Python code for it?

#Veamos que pasa si le damos un índice mayor, a pesar de que sabemos que lista1[4] nos da un error
lista1[0:4]

#Otra forma de mostrar todos los elementos es con ":"
lista2[:]

lista2

copia = lista2

copia

copia *2

copia = copia *2

copia

lista2

copia = lista2[:]

copia

copia is lista2

copia.append(13)

copia

lista2

#Aquí le estamos pidiendo a python que nos de todos los elementos de la lista menos el primero
lista1[1:]

#Si queremos ver el último elemento pero no sabemos cuántos hay en una lista podemos usar el siguiente truco
lista1[:-1]


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Pregunta 3.
¿Qué resultaría de la siguiente expresión? lista1[::-1]
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Pregunta 4
¿Cómo le harías para borrar elementos de una lista?
Pista: del y remove
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Pregunta 5
Crea una lista nueva que se llame "copia" de la lista2
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
Vamos a explorar ahora la parte de los índices y mutabilidad en una lista
Para acceder a un valor en una lista podemos hacer uso de los índices. Los índices empiezan desde cero.
Can you write Python code for it?


I am working on the file "2. Lists_and_conditionals/lists and conditionals.ipynb".
The first task is:
¡Obvio! nos da un error de índice... Y nos dice qué está fuera del rango de valores que tenemos guardados en la lista
También tenemos acceso a un rango de valores
Can you write Python code for it?


I am working on the file "2. Merge_Invididual_to_Household.ipynb".
The first task is:
Create Pivot Dataset
We create a pivot dataset in order to merge it later with the household dataset. With this we will have all individual information relative to a particular household.
Can you write Python code for it?

cleanData = data.copy().loc[(headAndSpouse),
                              ['id',
                               'AGLOMERADO',
                            'familyRelation',
                              'age',
                              'age2',
                              'female',
                              'education',
                              'education2',
                              'primary',
                              'secondary',
                              'university',
                              'P21',
                              'P47T',
                              'lnIncome',
                               u'lnIncomeT',
                              'predictedLnIncome',
                              'job',
                              'DECCFR',
                               'DECIFR',
                               'maritalStatus',
                               'reading',
                               'placeOfBirth',
                              ]]

cleanData.head()

pivot = cleanData.pivot(index='id', columns='familyRelation')
pivot.head()


I am working on the file "2. Merge_Invididual_to_Household.ipynb".
The first task is:
DATA HANDLING
Can you write Python code for it?

data1 = pd.read_csv('data/cleanDatat310.csv')

data2 = categorize.categorize(data1)
data3 = schoolYears.schoolYears(data2)
data4 = createVariables.createVariables(data3)


I am working on the file "2. Merge_Invididual_to_Household.ipynb".
The first task is:
GET HOUSEHOLD DATA
 Variables description:
- HomeType
- FloorMaterial
- RoofMaterial
- RoofCoat
- Water
- WaterType
- Toilet
- ToletLocation
- ToiletType
Can you write Python code for it?

getEPH.getEPHdbf('t310')
hog = pd.read_csv('data/cleanDataHouseholdt310.csv')
print hog.shape
hog.head()


I am working on the file "2. Merge_Invididual_to_Household.ipynb".
The first task is:
LOAD INDIVIDUAL DATA
Can you write Python code for it?

# Load data previously exported

ind = pd.read_csv('data/pivotInd.csv')
ind['id'] = ind['id'].astype(str)

ind.drop(['AGLO2'],axis =1,inplace=True)
ind['sumPredicted'] = ind.headPredictedLnIncome + ind.spousePredictedLnIncome


I am working on the file "2. Merge_Invididual_to_Household.ipynb".
The first task is:
Merge subset of individuals (with job and income) with total individuals
We create a merged dataset combining a subset of individuals (with job and income) with the total individuals. We then account for people with no income (kids or unemployed) and assign them a "0" income. The resulting dataset will succesfully match every individual with their household.
Can you write Python code for it?

paraMerge = dataParaModelo.loc[:,['CODUSU', 'NRO_HOGAR', 'COMPONENTE','predictedLnIncome']]
paraMerge.head()

data = pd.merge(left = data4 , right = paraMerge, on = ['CODUSU', 'NRO_HOGAR', 'COMPONENTE'], how = 'left')

data.predictedLnIncome[data.predictedLnIncome.isnull()] = 0


I am working on the file "2. Merge_Invididual_to_Household.ipynb".
The first task is:
New variables for individuals
We create a set of new variables (re-arrangement of previous ones) set in a way we create will add value to our prediction
Can you write Python code for it?

# Variables related to Occupation
data['job'] = (data.activity==1).astype(int)
data['noJob'] = (data.activity!=1).astype(int)
data['schoolAndJob'] = data.job * data.education

cantidadActivos = data.job.groupby(by=data['id']).sum() # Number of people working in the household
cantidadInactivos = data.noJob.groupby(by=data['id']).sum() # Number of people not working in the household
schoolAndJob = data.schoolAndJob.groupby(by=data['id']).sum() # Sum of total schooling years in the household

dfJobsAndEduc = pd.merge(left = schoolAndJob.to_frame() ,
                         right = cantidadInactivos.to_frame(),
                         left_index = True,
                         right_index = True)
dfJobsAndEduc = pd.merge(left = dfJobsAndEduc ,
                         right = cantidadActivos.to_frame(),
                         left_index = True,
                         right_index = True)


I am working on the file "2. Merge_Invididual_to_Household.ipynb".
The first task is:
Ordinal Order Transformation
In order to have positive coefficients we changed the order in which ordinal variables were displayed. We only do this for 'CookingCombustible' (which shows what combustible is used in the household) and for the others we use dummy variables.
Can you write Python code for it?

variables = ['CookingCombustible']
for var in variables:
    print hog2[var].value_counts()
    plt.scatter(hog2[var], hog2.TotalHouseHoldIncome)
    plt.show()

hog2['CookingRec'] = np.nan
hog2['CookingRec'][hog2.CookingCombustible == 4] = 1
hog2['CookingRec'][hog2.CookingCombustible == 3] = 1
hog2['CookingRec'][hog2.CookingCombustible == 2] = 2
hog2['CookingRec'][hog2.CookingCombustible == 1] = 3

variables = ['CookingRec']
for var in variables:
    print hog2[var].value_counts()
    plt.scatter(hog2[var], hog2.TotalHouseHoldIncome)
    plt.show()

hog2['WaterRec'] = (hog2.Water == 1).astype(int)

hog2['OwnershipRec'] = ((hog2.Ownership == 1) | (hog2.Ownership == 3)).astype(int)
hog2['Hacinamiento'] = hog2.HouseMembers * 1.0 / hog2.SleepingRooms
hog2['id'] =  (hog2.CODUSU.astype(str) + hog2.NRO_HOGAR.astype(str))

hog2['TotalHouseHoldIncome'].replace(to_replace=[0], value=[1] , inplace=True, axis=None)
hog2['lnHouseIncome'] = np.log(hog2['TotalHouseHoldIncome'])
sinCuartosParaDormir = (hog2.SleepingRooms == 0)

hogReducida = hog2.copy().drop(['CODUSU','NRO_HOGAR','REGION','HomeTypeesp','FloorMaterialesp',
                              'WaterTypeesp','Ownershipesp','CookingCombustibleesp','DomesticService1',
                               'DomesticService2', 'DomesticService3','DomesticService4', 'DomesticService5',
                               'DomesticService6'],axis = 1)


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Basic Network Statistics
Let's first understand how many people and relationships are represented in the network.
Can you write Python code for it?

# Who are represented in the network?
G.nodes(data=True)


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Coding Patterns
These are some recommended coding patterns when doing network analysis using NetworkX, which stem from my roughly two years of experience with the package.
Iterating using List Comprehensions
I would recommend that you use the following for compactness:
[d['attr'] for n, d in G.nodes(data=True)]

And if the node is unimportant, you can do:
[d['attr'] for _, d in G.nodes(data=True)]

Iterating over Edges using List Comprehensions
A similar pattern can be used for edges:
[n2 for n1, n2, d in G.edges(data=True)]

or
[n2 for _, n2, d in G.edges(data=True)]

If the graph you are constructing is a directed graph, with a "source" and "sink" available, then I would recommend the following pattern:
[(sc, sk) for sc, sk, d in G.edges(data=True)]

or
[d['attr'] for sc, sk, d in G.edges(data=True)]

Drawing Graphs
As illustrated above, we can draw graphs using the nx.draw() function. The most popular format for drawing graphs is the node-link diagram.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Concept
A network, more technically known as a graph, is comprised of:

a set of nodes
joined by a set of edges

They can be represented as two lists:

A node list: a list of 2-tuples where the first element of each tuple is the representation of the node, and the second element is a dictionary of metadata associated with the node.
An edge list: a list of 3-tuples where the first two elements are the nodes that are connected together, and the third element is a dictionary of metadata associated with the edge.

Since this is a social network of people, there'll be attributes for each individual, such as a student's gender. We can grab that data off from the attributes that are stored with each node.
Can you write Python code for it?

# Let's get a list of nodes with their attributes.
G.nodes(data=True)

# NetworkX will return a list of tuples in the form (node_id, attribute_dictionary)


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Edges can also store attributes in their attribute dictionary.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Exercise
Can you count how many males and females are represented in the graph?
Hint: You may want to use the Counter object from the collections module.
Can you write Python code for it?

from collections import Counter
Counter([d['sex'] for n, d in G.nodes(data=True)])


from collections import Counter
mf_counts = Counter([d['gender'] for n, d in G.nodes(data=True)])

def test_answer(mf_counts):
    assert mf_counts['female'] == 17
    assert mf_counts['male'] == 12

test_answer(mf_counts)


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Exercise
Can you write a single line of code that returns the number of nodes in the graph?
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Exercise
Can you write a single line of code that returns the number of relationships represented?
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Exercise
We found out that there are two individuals that we left out of the network, individual no. 31 and 32. They are one male (31) and one female (32), their ages are 22 and 24 respectively, they knew each other on 2010-01-09, and together, they both knew individual 7, on 2009-12-11. Use the functions G.add_node() and G.add_edge() to introduce this data into the network.
If you need more help, check out https://networkx.github.io/documentation/latest/tutorial/tutorial.html
Can you write Python code for it?

# Answer
G.add_node(31, age=22, sex='Male')
G.add_node(32, age=24, sex='Female')
G.add_edge(31, 32, date=datetime(2010,1,9))
G.add_edge(31, 7, date=datetime(2009,12,11))
G.add_edge(32, 7, date=datetime(2009,12,11))

G.node[31]


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Exercise: Can you write a single line of code that returns the number of individuals represented?
Can you write Python code for it?

len(G.nodes())

# Who is connected to who in the network?
G.edges(data=True)


len(G.nodes())

# Who is connected to who in the network?
G.edges()


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
However, note that if the number of nodes in the graph gets really large, node-link diagrams can begin to look like massive hairballs. This is undesirable for graph visualization.
Instead, we can use a matrix to represent them. The nodes are on the x- and y- axes, and a filled square represent an edge between the nodes. This is done by using the nx.to_numpy_matrix(G) function.
We then use matplotlib's pcolor(numpy_array) function to plot. Because pcolor cannot take in numpy matrices, we will cast the matrix as an array of arrays, and then get pcolor to plot it.
Can you write Python code for it?

matrix = nx.to_numpy_matrix(G)

plt.pcolor(np.array(matrix))
plt.axes().set_aspect('equal') # set aspect ratio equal to get a square visualization
plt.xlim(min(G.nodes()), max(G.nodes())) # set x and y limits to the number of nodes present.
plt.ylim(min(G.nodes()), max(G.nodes()))
plt.title('Adjacency Matrix')
plt.show()


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
If the network is small enough to visualize, and the node labels are small enough to fit in a circle, then you can use the with_labels=True argument.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
In this synthetic social network, I have stored the date as a datetime object. Datetime objects have attributes, namely .year, .month, .day.
Exercise
Can you figure out the range of dates during which these relationships were forged?
Can you write Python code for it?

# Answer
dates = [d['date'] for _, _, d in G.edges(data=True)]
mindate = min(dates)
maxdate = max(dates)
print(mindate, maxdate)


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
It's pretty obvious in this visualization that there are nodes, such as node 5 and 18, that are not connected to any other node via an edge. There are other nodes, like node number 19, which is highly connected to other nodes.
Finally, let's try hive plots for the network. Two groups (male and female), and then edges drawn between them.
Can you write Python code for it?

nodes = dict()
nodes['male'] = [n for n,d in G.nodes(data=True) if d['sex'] == 'Male']
nodes['female'] = [n for n,d in G.nodes(data=True) if d['sex'] == 'Female']

edges = dict()
edges['group1'] = G.edges(data=True)

nodes_cmap = dict()
nodes_cmap['male'] = 'blue'
nodes_cmap['female'] = 'red'

edges_cmap = dict()
edges_cmap['group1'] = 'black'

h = HivePlot(nodes, edges, nodes_cmap, edges_cmap)
h.draw()


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Let's now figure out who is connected to who in the network
Can you write Python code for it?

# Who is connected to who in the network?
G.edges()


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Let's try another visualization, the Circos plot. We can order the nodes in the Circos plot according to the node ID, but any other ordering is possible as well. Edges are drawn between two nodes.
Credit goes to Justin Zabilansky (MIT) for the implementation, and Jon Charest for improvements.
Can you write Python code for it?

from circos import CircosPlot

fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)

nodes = sorted(G.nodes())
edges = G.edges()
node_cmap = {'male':'blue', 'female':'red'}
nodecolors = [node_cmap[G.node[n]['gender']] for n in G.nodes()]

c = CircosPlot(nodes, edges, radius=10, ax=ax, fig=fig, nodecolor=nodecolors)
c.draw()
plt.savefig('images/seventh.png', dpi=300)


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Let's try another visualization, the Circos plot. We can order the nodes in the Circos plot according to the node ID, but any other ordering is possible as well. Edges are drawn between two nodes.
Credit goes to Justin Zabilansky (MIT) for the implementation.
Can you write Python code for it?

from circos import CircosPlot

fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)

nodes = sorted(G.nodes())
edges = G.edges()

c = CircosPlot(nodes, edges, radius=10, ax=ax)
c.draw()


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Live Exercise
While we're on the matter of graph construction, let's take a look at our tutorial class. On your sheet of paper, you should have a list of names - these are people for which you knew their name prior to coming to class.
As we iterate over the class, I would like you to holler out your name, your nationality, and in a very slow fashion, the names of the people who you knew in the class.
Can you write Python code for it?

ptG = nx.DiGraph() #ptG stands for PyCon Tutorial Graph.

# Add in nodes and edges
ptG.add_node('Eric', nationality='Canada')
ptG.add_node('Paul', nationality='Canada') # (my own TextExpander shortcut is ;addnode)
ptG.add_node('Max', nationality='US')
ptG.add_node('Martin', nationality='Other')
ptG.add_node('Jim', nationality='US')
ptG.add_node('Lucas', nationality='US')
ptG.add_node('Thomas', nationality='US')
ptG.add_node('Brad', nationality='US')
ptG.add_node('Troy', nationality='Canada')
ptG.add_node('Cory', nationality='Canada')
ptG.add_node('Gokhan', nationality='US')
ptG.add_node('Riley', nationality='US')
ptG.add_node('Steve', nationality='US')
ptG.add_node('Ryan', nationality='US')
ptG.add_node('Andrew', nationality='US')
ptG.add_node('Ronan', nationality='Other')
ptG.add_node('Cody', nationality='Canada')
ptG.add_node('Jon', nationality='US')
ptG.add_node('Eric2', nationality='US')
ptG.add_node('William', nationality='US')
ptG.add_node('Tom', nationality='Other')
ptG.add_node('Chris', nationality='US')
ptG.add_node('Stu', nationality='US')
ptG.add_node('Zach', nationality='US')
ptG.add_node('Clint', nationality='Canada')
ptG.add_node('Aaron', nationality='US')
ptG.add_node('Vishal', nationality='US')
ptG.add_node('Federico', nationality='Other')

ptG.add_edge('Vishal', 'Aaron')
ptG.add_edge('Vishal', 'Eric')
ptG.add_edge('Aaron', 'Vishal')
ptG.add_edge('Aaron', 'Eric')
ptG.add_edge('Clint', 'Zach')
ptG.add_edge('Clint', 'Eric')
ptG.add_edge('Zach', 'Clint')
ptG.add_edge('Zach', 'Riley')
ptG.add_edge('Zach', 'Stu')
ptG.add_edge('Stu', 'Zach')
ptG.add_edge('Stu', 'Eric')
ptG.add_edge('Stu', 'Chris')
ptG.add_edge('Chris', 'Stu')
ptG.add_edge('Chris', 'Eric')
ptG.add_edge('Tom', 'Tom')
ptG.add_edge('William', 'Jon')
ptG.add_edge('William', 'Eric2')
ptG.add_edge('William', 'Eric')
ptG.add_edge('Eric2', 'William')
ptG.add_edge('Eric2', 'Jon')
ptG.add_edge('Jon', 'Eric2')
ptG.add_edge('Jon', 'William')
ptG.add_edge('Jon', 'Eric')
ptG.add_edge('Cody', 'Eric')
ptG.add_edge('Cody',...


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Nodes and Edges: How do we represent relationships between individuals using NetworkX?
As mentioned earlier, networks, also known as graphs, are comprised of individual entities and their representatives. The technical term for these are nodes and edges, and when we draw them we typically use circles (nodes) and lines (edges).
In this notebook, we will work with a synthetic (i.e. simulated) social network, in which nodes are individual people, and edges represent their relationships. If two nodes have an edge between them, then those two individauls know one another.
Data Representation
In the networkx implementation, graph objects store their data in dictionaries.
Nodes are part of the attribute Graph.node, which is a dictionary where the key is the node ID and the values are a dictionary of attributes.
Edges are part of the attribute Graph.edge, which is a nested dictionary. Data are accessed as such: G.edge[node1][node2]['attr_name'].
Because of the dictionary implementation of the graph, any hashable object can be a node. This means strings and tuples, but not lists and sets.
Synthetic Social Network
With this synthetic social network, we will attempt to answer the following basic questions using the NetworkX API:

How many people are present in the network?
What is the distribution of attributes of the people in this network?
How many relationships are represented in the network?
What is the distribution of the number of friends that each person has?

First off, let's load up the synthetic social network. This will show you through some of the basics of NetworkX.
For those who are interested, I simply created an Erdõs-Rényi graph with n=30 and p=0.1. I used randomized functions that I wrote to generate attributes and append them to each node and edge. I then pickled the graph to disk.
Can you write Python code for it?

G = nx.read_gpickle('Synthetic Social Network.pkl') #If you are Python 2.7, read in Synthetic Social Network 27.pkl
nx.draw(G)


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Since this is a social network of people, there'll be attributes for each individual, such as age, and sex. We can grab that data off from the attributes that are stored with each node.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Tests
A note about the tests: Testing is good practice when writing code. Well-crafted assertion statements help you program defensivel, by forcing you to explicitly state your assumptions about the code or data.
For more references on defensive programming, check out Software Carpentry's website: http://swcarpentry.github.io/python-novice-inflammation/08-defensive.html
For more information on writing tests for your data, check out these slides from a lightning talk I gave at Boston Python and SciPy 2015: http://j.mp/data-test
Coding Patterns
These are some recommended coding patterns when doing network analysis using NetworkX, which stem from my roughly two years of experience with the package.
Iterating using List Comprehensions
I would recommend that you use the following for compactness:
[d['attr'] for n, d in G.nodes(data=True)]

And if the node is unimportant, you can do:
[d['attr'] for _, d in G.nodes(data=True)]

Iterating over Edges using List Comprehensions
A similar pattern can be used for edges:
[n2 for n1, n2, d in G.edges(data=True)]

or
[n2 for _, n2, d in G.edges(data=True)]

If the graph you are constructing is a directed graph, with a "source" and "sink" available, then I would recommend the following pattern:
[(sc, sk) for sc, sk, d in G.edges(data=True)]

or
[d['attr'] for sc, sk, d in G.edges(data=True)]

Drawing Graphs
As illustrated above, we can draw graphs using the nx.draw() function. The most popular format for drawing graphs is the node-link diagram.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
This visualization helps us highlight nodes that there are poorly connected, and others that are strongly connected.
Next up, let's try Hive Plots.
Can you write Python code for it?

from hiveplot import HivePlot

nodes = dict()
nodes['male'] = [n for n,d in G.nodes(data=True) if d['gender'] == 'male']
nodes['female'] = [n for n,d in G.nodes(data=True) if d['gender'] == 'female']

edges = dict()
edges['group1'] = G.edges(data=True)

nodes_cmap = dict()
nodes_cmap['male'] = 'blue'
nodes_cmap['female'] = 'red'

edges_cmap = dict()
edges_cmap['group1'] = 'black'

h = HivePlot(nodes, edges, nodes_cmap, edges_cmap)
h.draw()


I am working on the file "2. Network(X) Basics (Instructor).ipynb".
The first task is:
Verify that you have added in the edges and nodes correctly by running the following cell.
Can you write Python code for it?

def test_graph_integrity(G):
    assert 30 in G.nodes()
    assert 31 in G.nodes()
    assert G.node[30]['gender'] == 'male'
    assert G.node[31]['gender'] == 'female'
    assert G.has_edge(30, 31)
    assert G.has_edge(30, 7)
    assert G.has_edge(31, 7)
    assert G.edge[30][7]['count'] == 3
    assert G.edge[7][30]['count'] == 3
    assert G.edge[31][7]['count'] == 3
    assert G.edge[7][31]['count'] == 3
    assert G.edge[30][31]['count'] == 3
    assert G.edge[31][30]['count'] == 3
    print('All tests passed.')

test_graph_integrity(G)


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Basic Network Statistics
Let's first understand how many people and relationships are represented in the network.
Can you write Python code for it?

# Who are represented in the network?
G.nodes()


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Coding Patterns
These are some recommended coding patterns when doing network analysis using NetworkX, which stem from my roughly two years of experience with the package.
Iterating using List Comprehensions
I would recommend that you use the following for compactness:
[d['attr'] for n, d in G.nodes(data=True)]

And if the node is unimportant, you can do:
[d['attr'] for _, d in G.nodes(data=True)]

Iterating over Edges using List Comprehensions
A similar pattern can be used for edges:
[n2 for n1, n2, d in G.edges(data=True)]

or
[n2 for _, n2, d in G.edges(data=True)]

If the graph you are constructing is a directed graph, with a "source" and "sink" available, then I would recommend the following pattern:
[(sc, sk) for sc, sk, d in G.edges(data=True)]

or
[d['attr'] for sc, sk, d in G.edges(data=True)]

Drawing Graphs
As illustrated above, we can draw graphs using the nx.draw() function. The most popular format for drawing graphs is the node-link diagram.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Concept
A network, more technically known as a graph, is comprised of:

a set of nodes
joined by a set of edges

They can be represented as two lists:

A node list: a list of 2-tuples where the first element of each tuple is the representation of the node, and the second element is a dictionary of metadata associated with the node.
An edge list: a list of 3-tuples where the first two elements are the nodes that are connected together, and the third element is a dictionary of metadata associated with the edge.

Since this is a social network of people, there'll be attributes for each individual, such as a student's gender. We can grab that data off from the attributes that are stored with each node.
Can you write Python code for it?

# Let's get a list of nodes with their attributes.
G.nodes(data=True)

# NetworkX will return a list of tuples in the form (node_id, attribute_dictionary)


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Edges can also store attributes in their attribute dictionary.
Can you write Python code for it?

G.edges(data=True)

lst = [c['date'] for a, b, c in G.edges(data=True)]

#def test_answer(mf_counts):
#    print(mf_counts['Female'])
#    print(mf_counts['Male'])

#test_answer(mf_counts)
print(min(lst), max(lst))


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Exercise
Can you count how many males and females are represented in the graph?
Hint: You may want to use the Counter object from the collections module.
Can you write Python code for it?

from collections import Counter
Counter([d['sex'] for n, d in G.nodes(data=True)])


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Exercise
Can you write a single line of code that returns the number of nodes in the graph?
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Exercise
Can you write a single line of code that returns the number of relationships represented?
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Exercise
We found out that there are two individuals that we left out of the network, individual no. 30 and 31. They are one male (30) and one female (31), and they are a pair that just love hanging out with one another and with individual 7 (count=3). Add this information to the graph.
If you need more help, check out https://networkx.github.io/documentation/latest/tutorial/tutorial.html
Can you write Python code for it?

# Answer: Follow the coding pattern.
G.add_node(30, {'age': 22, 'sex': 'Male'})
G.add_node(31, {'age': 24, 'sex': 'Female'})

G.add_edge(30, 31, date= datetime(2010, 1, 9))
G.add_edge(30, 7, date= datetime(2009, 12, 11))
G.add_edge(31, 7, date= datetime(2009, 12, 11))

G.edge[30]


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Exercise: Can you write a single line of code that returns the number of individuals represented?
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
If the network is small enough to visualize, and the node labels are small enough to fit in a circle, then you can use the with_labels=True argument.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
In this synthetic social network, I have stored the date as a datetime object. Datetime objects have attributes, namely .year, .month, .day.
Exercise
Can you figure out the range of dates during which these relationships were forged?
Exercise
We found out that there are two individuals that we left out of the network, individual no. 31 and 32. They are one male (31) and one female (32), their ages are 22 and 24 respectively, they knew each other on 2010-01-09, and together, they both known individual 7, on 2009-12-11. Use the functions G.add_node() and G.add_edge() to introduce this data into the network.
If you need more help, check out https://networkx.github.io/documentation/latest/tutorial/tutorial.html
Can you write Python code for it?

G.add_node(31, age=22, sex='Male')

G.add_edge(31, 32, datetime=datetime(2010, 1, 9))


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
In this synthetic social network, I have stored the date as a datetime object. Datetime objects have attributes, namely .year, .month, .day.
Exercise
Can you figure out the range of dates during which these relationships were forged?
Exercise
We found out that there are two individuals that we left out of the network, individual no. 31 and 32. They are one male (31) and one female (32), their ages are 22 and 24 respectively, they knew each other on 2010-01-09, and together, they both known individual 7, on 2009-12-11. Use the functions G.add_node() and G.add_edge() to introduce this data into the network.
If you need more help, check out https://networkx.github.io/documentation/latest/tutorial/tutorial.html
Live Exercise
While we're on the matter of graph construction, let's take a look at our tutorial class. On your sheet of paper, you should have a list of names - these are people for which you knew their name prior to coming to class.
As we iterate over the class, I would like you to holler out your name, your nationality, and in a very slow fashion, the names of the people who you knew in the class.
Can you write Python code for it?

## You may choose to join me in this endeavor together.

ptG = nx.DiGraph() #ptG stands for PyCon Tutorial Graph.

# Add in nodes and edges
ptG.add_node('', nationality='') # (my own TextExpander shortcut is ;addnode)
ptG.add_edge('', '') # (my own TextExpander shortcut is ;addedge)

# We are now going to draw the network using a hive plot, grouping the nodes by the top two nationality groups, and 'others'
# for the third group.

nodes = dict()
nodes['group1'] = [] #list comprehension here
nodes['group2'] = [] #list comprehension here
nodes['group3'] = [] #list comprehension here

edges = dict()
edges['group1'] = [] #list comprehension here

nodes_cmap = dict()
nodes_cmap['group1'] = 'blue'
nodes_cmap['group2'] = 'green'
nodes_cmap['group3'] = 'black'

edges_cmap = dict()
edges_['group1'] = 'black'

from hiveplot import HivePlot
h = HivePlot(nodes, edges, nodes_cmap, edges_cmap)
# h.set_minor_angle(np.pi / 32) #optional
h.draw()


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
In this synthetic social network, the number of times the left student indicated that the right student was their favourite is stored in the "count" variable.
Exercise
Can you figure out the maximum times any student rated another student as their favourite?
Can you write Python code for it?

# Answer
counts = [d['_____'] for _, _, _ in G._______(_________)]
maxcount = max(_________)

def test_maxcount(maxcount):
    assert maxcount == 3

test_maxcount(maxcount)


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Let's try another visualization, the Circos plot. We can order the nodes in the Circos plot according to the node ID, but any other ordering is possible as well. Edges are drawn between two nodes.
Credit goes to Justin Zabilansky (MIT) for the implementation, and Jon Charest for improvements.
Can you write Python code for it?

from circos import CircosPlot

fig = plt.figure(figsize=(6,6))
ax = fig.add_subplot(111)

nodes = sorted(G.nodes())
edges = G.edges()
node_cmap = {'Male':'blue', 'Female':'red'}
nodecolors = [node_cmap[G.node[n]['sex']] for n in G.nodes()]

c = CircosPlot(nodes, edges, radius=10, ax=ax, fig=fig, nodecolor=nodecolors)
c.draw()
plt.savefig('images/seventh.png', dpi=300)


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Let's try another visualization, the Circos plot. We can order the nodes in the Circos plot according to the node ID, but any other ordering is possible as well. Edges are drawn between two nodes.
Credit goes to Justin Zabilansky (MIT) for the implementation.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Nodes and Edges: How do we represent relationships between individuals using NetworkX?
As mentioned earlier, networks, also known as graphs, are comprised of individual entities and their representatives. The technical term for these are nodes and edges, and when we draw them we typically use circles (nodes) and lines (edges).
In this notebook, we will work with a synthetic (i.e. simulated) social network, in which nodes are individual people, and edges represent their relationships. If two nodes have an edge between them, then those two individauls know one another.
Data Representation
In the networkx implementation, graph objects store their data in dictionaries.
Nodes are part of the attribute Graph.node, which is a dictionary where the key is the node ID and the values are a dictionary of attributes.
Edges are part of the attribute Graph.edge, which is a nested dictionary. Data are accessed as such: G.edge[node1][node2]['attr_name'].
Because of the dictionary implementation of the graph, any hashable object can be a node. This means strings and tuples, but not lists and sets.
Synthetic Social Network
With this synthetic social network, we will attempt to answer the following basic questions using the NetworkX API:

How many people are present in the network?
What is the distribution of attributes of the people in this network?
How many relationships are represented in the network?
What is the distribution of the number of friends that each person has?

First off, let's load up the synthetic social network. This will show you through some of the basics of NetworkX.
For those who are interested, I simply created an Erdõs-Rényi graph with n=30 and p=0.1. I used randomized functions that I wrote to generate attributes and append them to each node and edge. I then pickled the graph to disk.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Since this is a social network of people, there'll be attributes for each individual, such as age, and sex. We can grab that data off from the attributes that are stored with each node.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Tests
A note about the tests: Testing is good practice when writing code. Well-crafted assertion statements help you program defensivel, by forcing you to explicitly state your assumptions about the code or data.
For more references on defensive programming, check out Software Carpentry's website: http://swcarpentry.github.io/python-novice-inflammation/08-defensive.html
For more information on writing tests for your data, check out these slides from a lightning talk I gave at Boston Python and SciPy 2015: http://j.mp/data-test
Coding Patterns
These are some recommended coding patterns when doing network analysis using NetworkX, which stem from my roughly two years of experience with the package.
Iterating using List Comprehensions
I would recommend that you use the following for compactness:
[d['attr'] for n, d in G.nodes(data=True)]

And if the node is unimportant, you can do:
[d['attr'] for _, d in G.nodes(data=True)]

Iterating over Edges using List Comprehensions
A similar pattern can be used for edges:
[n2 for n1, n2, d in G.edges(data=True)]

or
[n2 for _, n2, d in G.edges(data=True)]

If the graph you are constructing is a directed graph, with a "source" and "sink" available, then I would recommend the following pattern:
[(sc, sk) for sc, sk, d in G.edges(data=True)]

or
[d['attr'] for sc, sk, d in G.edges(data=True)]

Drawing Graphs
As illustrated above, we can draw graphs using the nx.draw() function. The most popular format for drawing graphs is the node-link diagram.
Can you write Python code for it?


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
This visualization helps us highlight nodes that there are poorly connected, and others that are strongly connected.
Next up, let's try Hive Plots.
Can you write Python code for it?

from hiveplot import HivePlot

nodes = dict()
nodes['Male'] = [n for n,d in G.nodes(data=True) if d['sex'] == 'Male']
nodes['Female'] = [n for n,d in G.nodes(data=True) if d['sex'] == 'Female']

edges = dict()
edges['group1'] = G.edges(data=True)

nodes_cmap = dict()
nodes_cmap['Male'] = 'blue'
nodes_cmap['Female'] = 'red'

edges_cmap = dict()
edges_cmap['group1'] = 'black'

h = HivePlot(nodes, edges, nodes_cmap, edges_cmap)
h.draw()


I am working on the file "2. Network(X) Basics (Student).ipynb".
The first task is:
Verify that you have added in the edges and nodes correctly by running the following cell.
Can you write Python code for it?

def test_graph_integrity(G):
    assert 30 in G.nodes()
    assert 31 in G.nodes()
    assert G.node[30]['sex'] == 'Male'
    assert G.node[31]['sex'] == 'Female'
    assert G.has_edge(30, 31)
    assert G.has_edge(30, 7)
    assert G.has_edge(31, 7)
    print('All tests passed.')

test_graph_integrity(G)


I am working on the file "2. Objects and types.ipynb".
The first task is:
2.2. Instance methods.
Instances (objects) of different classes have different built-in methods (another word for a function). An instance method is like a super power: think about it like an intrinsic ability of the object.
We can use instance methods by writing the name of the object, and then a period (.), and then the name of the method. For example, we can change a str to lower case using the lowercase() method:
Can you write Python code for it?

mystring = 'YYYYIKES'
print mystring.lower()    # lower() is an instance method for strings!


I am working on the file "2. Objects and types.ipynb".
The first task is:
2.3. Make your own types!
It's easy to define your own types/classes in Python. Just use the class declaration.
Let's create a Car class. The Car class will define what attributes and methods an instance of a car will have.
Can you write Python code for it?

class Car:
    """
    A small passenger automobile.
    """

    driver = None              # By default, there is no driver.
    number_of_passengers = 0   # By default, the car is empty.


I am working on the file "2. Objects and types.ipynb".
The first task is:
A function is also an object, for which there is a function type.
Can you write Python code for it?

def myfunction():
    return True

type(myfunction)


I am working on the file "2. Objects and types.ipynb".
The first task is:
Another word for type is "class". When I'm working with a particular string (str), for example, I'm working with an instance of the str class.
Can you write Python code for it?

print "I am an instance of a string"
print str                # <-- This is the str class, or type.
print type(str)          # <-- The str type is an instance of class `type`.


I am working on the file "2. Objects and types.ipynb".
The first task is:
But all is not lost! If I transform it into an int first, I'm golden.
Can you write Python code for it?

print type(mystring)    # mystring is a str
myint = int(mystring)   # But I can turn it into an int!

print type(myint)       # Yup, it's an int.
print myint + 3.4       # And now I can math!


I am working on the file "2. Objects and types.ipynb".
The first task is:
IPython: tab completion
In IPython, you can use tab completion! Just type the name of an object plus a period, and then press tab to see a list of attributes. You can start typing in part of an attribute name to limit the list.
Can you write Python code for it?

myCar.    # Put your cursor right after the dot, and press ``tab``.


I am working on the file "2. Objects and types.ipynb".
The first task is:
Notice that this includes methods, like count, title, and upper.
The __dict__ attribute.
Many objects can be represented as dictionaries (we haven't talked about that yet). For now, the important thing is that access the __dict__ attribute of most objects will display a list of non-function attributes as well as their values.
Can you write Python code for it?

myCar = Car()
myCar.driver = 'Fido'

print myCar.__dict__


I am working on the file "2. Objects and types.ipynb".
The first task is:
Now let's give our Car class some methods. We'll redeclare the class, this time with methods.
Can you write Python code for it?

class Car:
    """
    A small passenger automobile.
    """

    driver = None              # By default, there is no driver.
    number_of_passengers = 0   # By default, the car is empty.

    def honk(self):
        """
        What noise does a car make?
        """
        print 'beep beep!'

    def count_passengers(self):
        """
        Get the number of passengers in this car.
        """
        return self.number_of_passengers


I am working on the file "2. Objects and types.ipynb".
The first task is:
Pro tip: when defining a class, always include a docstring. That's the part with the triple open and close quotes and a descriptive message.
python
"""
I am a docstring. I describe what a class is all about.
"""
2.4. Changing the type of an object.
Sometimes it makes sense to transform an instance/object of one type into an instance/object of another type. For example, suppose I have a string that contains the character "5".
python
mystring = '5'
If I want to do math with that 5 (say, I want to add 3.4 to it), I'll run into trouble:
Can you write Python code for it?


I am working on the file "2. Objects and types.ipynb".
The first task is:
This looks a lot like the function declarations that you saw earlier, but with two important differences:

They are part of the class definition for Car, so they are indented one extra level.
Their first (and, in this case, only) parameter is something called self.

self is how we refer to the instance of a class. When we create an instance of car, and then use that car instance's number_of_passengers method, the self in that function is the instance of the car that we're interacting with.
So when I use the count_passengers method, and it returns self.number_of_passengers, it is retrieving the number_of_passengers attribute from the current car instance.
Can you write Python code for it?

mycar = Car()    # Putting the parens () after the class name instantiates the class.
mycar.driver = 'John'
mycar.number_of_passengers += 5

print mycar.count_passengers()


I am working on the file "2. Pandas. Analyzing Thanksgiving Dinner/Basics.ipynb".
The first task is:
Despite the fact that we have rounded off the age, the data reflects the present proportions of age groups (the same proportions as in [9])
Can you write Python code for it?

def income(string):
    if pd.isnull(string) == True:
        return None
    else:
        string = string.split(' ')[0]
        if string == 'Prefer':
            return None
        string = string.replace('$', '')
        string = string.replace(',', '')
        return int(string)
data['int_income'] = data['How much total combined money did all members of your HOUSEHOLD earn last year?'].apply(income)
data['int_income'].describe()

print(data['How much total combined money did all members of your HOUSEHOLD earn last year?'].value_counts())


I am working on the file "2. Pandas. Analyzing Thanksgiving Dinner/Basics.ipynb".
The first task is:
Only 38 percent of the respondents (with an income of 50000 thousand) celebrate at home. At the same time, 47 percent of respondents (with an income of 150 and above) celebrate at home. Despite not much difference, we can assume that students prefer to celebrate a holiday in the home of parents who earn more (and accordingly, celebrate in their own home).
Can you write Python code for it?

table = pd.pivot_table(data, values='int_age', index='Have you ever tried to meet up with hometown friends on Thanksgiving night?', columns='Have you ever attended a "Friendsgiving?"')
table

table_income = pd.pivot_table(data, values='int_income', index='Have you ever tried to meet up with hometown friends on Thanksgiving night?', columns='Have you ever attended a "Friendsgiving?"')
table_income


I am working on the file "2. Pandas. Analyzing Thanksgiving Dinner/Basics.ipynb".
The first task is:
When analyzing this result, it should be borne in mind that we took only the lower value of each group. Also, we have a very high standard deviation (59 thousand dollars) with the step size of each group of 25 thousand.
Can you write Python code for it?

data[data['int_income'] < 50000]['How far will you travel for Thanksgiving?'].value_counts(normalize=True)

data[data['int_income'] >= 150000]['How far will you travel for Thanksgiving?'].value_counts(normalize=True)


I am working on the file "2. Ploting my Caves of Qud data.ipynb".
The first task is:
A look at some of the remaining data
I took in a lot of data into the dataframe but have only looked at End Time, score, zone and turns. As time goes on and I get more entries I may be able to do more with the following bits of data
Can you write Python code for it?

#Each month mentioned in the Game End Time
game_months = qud["Game End Time"]
print np.unique(game_months)
print len(np.unique(game_months))

#Use groupby to find most mentioned month, ie the month I have died most in. Nivvun Ut is the very first month...
qud['Game End Time'].groupby(qud['Game End Time']).count().order(ascending = False)

#Use group by to find the most advanced artifact I held when I died. Lots of no artifacts and lots of artifacts awarded for finishing the first 2 missions in Joppa
qud['Artifact'].groupby(qud['Artifact']).count().order(ascending = False)


I am working on the file "2. Ploting my Caves of Qud data.ipynb".
The first task is:
Further Cleaning
With the data now pulled into a dataframe there is still a small bit of cleaning to do. Below are three functions to convert the End Time of each game to a datetime, to pull the game month from the Game End Time (I left the day in as at a later time I might check am I more likely to die early or late in the month) and to clean up the Artifact column
Can you write Python code for it?

import re
from datetime import datetime
from time import strptime

def convert_to_date(date_in):
    date_search = re.search("(\w{6,9}),\s*(\w{3,9})\s*(\d{2}),\s*(\d{4})\s*(\d{1,2}):(\d{2}):(\d{2})\s*(\w{2})", date_in)
    #date_search.group(1) = Day as word(ie Sunday), 2 = Month as word (ie August), 3 = day of month, 4 = year, 5 = hour, 6 = minute, 7 = second, 8 = AM or PM

    #In End Time hour is expressed from 1 to 12, ie 1 AM or 1 PM. The below code converts that to 0 to 23, ie 1 or 13
    hour = int(date_search.group(5))

    if date_search.group(8) == "PM":
        if hour == 12:
            pass
        else:
            hour += 12

    if date_search.group(8) == "AM":
        if hour == 12:
            hour = 0

    #Create a datetime. strptime is used to take the first 3 letters of the Month as word and get the int value for that month, ie August = Aug, is month 8 of 12
    new_date = datetime(int(date_search.group(4)), strptime(date_search.group(2)[:3], "%b").tm_mon, int(date_search.group(3)), hour, int(date_search.group(6)), int(date_search.group(7)))

    return new_date

qud["End Time"] = qud["End Time"].apply(convert_to_date)

#Pull the month out of Game Time

def convert_game_month(date_in):
    date_search = re.search("of\s*((\w*\s*)*)", date_in)
    return date_search.group(1)

qud["Game End Time"] = qud["Game End Time"].apply(convert_game_month)

def clean_artifacts(artifact):
    x_search = re.search("(x\d+)", artifact) #remove multipliers like "x2"
    if x_search != None:
        artifact = artifact.replace(x_search.group(0), "").strip()

    mul_search = re.search("((-?\d+\s*\d+d\d+)+)", artifact) #removes pv and possible weapon damage like "2 1d3"
    if mul_search != None:
        artifact = artifact.replace(mul_search.group(0), "").strip()

    artifact = artifact.replace("->", "").replace("<>", "").strip() #removes -> and <> which should be empty from previous cleaning

    cell_search = re.search("(\[(\w*\s*)*\])", artifact)...


I am working on the file "2. Ploting my Caves of Qud data.ipynb".
The first task is:
I have done slightly more work with the Enemy column. First I print it off as is but then make some changes.
Deaths from bleeding were always the result of young ivory. In my early games I took hemophilia as a defect which basicly meant instant death at low toughness levels and no bandages when I stared bleeding, so "bleeding" and "young ivory" are combined into a single group.
Deaths from scalding steam were always the result of fire ants setting the water around me on fire and I, like an idiot, taking a step forward. Only once was there another reason...I set the water on fire with my Flaming Hands and walked into it. Since then I have only used Freezing Hands.
All snapjaws, including the two faction leaders, are put into a single catagory.
"Wahmahcalcalit", "Umchuum", "Duhmahcaluhcal" are evidence my character was suffering confusion when he died, these were 3 wizard faction leaders.
I could do further grouping. Chute crab and king crab could be combined, the two named Snapjaws could be put in with the ogre instead to form a Named or Faction Leader catagory, dawnglider could be added to the fire ant/scalding steam catagory etc.
Can you write Python code for it?

qud['Enemy'].groupby(qud['Enemy']).count().order(ascending = False)

#create a list called enemies, add new values to it, convert to a dataframe and groupby name
enemies = qud["Enemy"].tolist()
for i in range(len(enemies)):
    name = enemies[i].strip()
    if name in ["Wahmahcalcalit", "Umchuum", "Duhmahcaluhcal"]:
        enemies[i] = "wizard"
    if name in ["snapjaw scavenger", "napjaw scavenger", "snapjaw hunter", "Groubuubu-wof-wofuz, the stalwart Snapjaw Tot-eater", "Ruf-ohoubub, the stalwart Snapjaw Bear-baiter"]:
        enemies[i] = "snapjaw"
    if name in ["young ivory", "bleeding"]:
        enemies[i] = "young ivory/bleeding"
    if name in ["scalding steam", "fire ant"]:
        enemies[i] = "fire ant/scalding steam"

enemy_df = pd.DataFrame(enemies, columns=["Name"])
enemy_df['Name'].groupby(enemy_df['Name']).count().order(ascending = False)


I am working on the file "2. Ploting my Caves of Qud data.ipynb".
The first task is:
Sorting by date and by score
With the End Time now cleaned up and converted to a datetime the entire dataframe can be sorted on this column, giving the dataframe in order of my earliest game to my most recent game. I can then print off my highscore progression and after sorting the dataframe by score I can print my 5 highest scores.
Can you write Python code for it?

sorted_qud = qud.sort(["End Time"]).reset_index(drop = True) #Sort by End Time, reset the index and drop the old index
highscore = -10000
print "Highscore Progression" #Game Number, Name, Date, Score
for score in sorted_qud["Score"]:
    if int(score) > highscore:
        highscore = int(score)
        print "%d %s %s %d" % (
        int(sorted_qud.index[sorted_qud["Score"] == score][0])+ 1, #the index value of the game + 1. My first game is at index 0 so add 1 and this becomes game 1
        sorted_qud["Name"][sorted_qud["Score"] == score].tolist()[0], #Character's name
        sorted_qud["End Time"][sorted_qud["Score"] == score].tolist()[0], #End Time of game
        int(score) #Score
        )
print "\n"
print "Highest Scores"
sorted_scores = qud.sort(["Score"], ascending = False).reset_index(drop = True) #sort by score
for i in range(5):
    print sorted_scores["Name"].iloc[i], sorted_scores["End Time"].iloc[i], sorted_scores["Score"].iloc[i] #print Name, End Time and Score for first 5 rows


I am working on the file "2. Ploting my Caves of Qud data.ipynb".
The first task is:
There are a number of highscores that have an empty line where the death description should be and these are marked unknown. Also, others will just read "died from bleeding". Here bleeding is added as both the enemy name and the weapon.
Below is a look at the weapons that have killed my characters. Rending mandibles. Giant pseudopod. The things I've seen.
Can you write Python code for it?


I am working on the file "2. Ploting my Caves of Qud data.ipynb".
The first task is:
Using the linear regression models I can now get the coefficient, intercept and root mean square error for the score per turn line and the score per zone line. Also, as the number of turns is displayed when a game is saved I can calculate how many points my current save game is worth. However, I would not expect this figure to be very accurate due to the small number of points available and I'm not willing to have my character die to check how right the figure is!
In linear regression y = a + Xb where a is the intercept and b is the coefficient. So using the below data:
score = -2171.04135919 + Turns(1.21948537419)
The root mean square error is found by taking the root of the mean_squared_error of the score compared to the predicted score
With a bit of moving around of figures I get to equation turns = (score + intercept)/coefficient which allows me to predict the number of turns needed for 100,000 points and 1,000,000 points. I have a bit of work to do yet! (I expect these numbers to change substantially as the model gets more data points in the high game points range, over half the data points are in the minus range at the moment)
Can you write Python code for it?

from sklearn.metrics import mean_squared_error
from math import sqrt
print "For Score Per Turn"
print "Total turns multiplied by the coefficient plus the intercept = my score"
print "Coefficient: ", turns_score.coef_[0][0]
print "Intercept: ", turns_score.intercept_[0]
print "RMSE: ", sqrt(mean_squared_error(y, turns_score.predict(np.array(qud["Turns"]).reshape(len(qud),1))))
print "Predicted score from my current game (59924 turns): ", int(turns_score.predict(59924)[0][0])
print "Turns needed for 100,000 points: ", int(math.ceil(((100000 + abs(turns_score.intercept_))/turns_score.coef_)[0][0]))
print "Turns needed for 1,000,000 points: ", int(math.ceil(((1000000 + abs(turns_score.intercept_))/turns_score.coef_)[0][0]))

print "For Score Per Zone"
print "Total zones visited multiplied by the coefficient plus the intercept = my score"
print "Coefficient: ", zones_score.coef_[0][0]
print "Intercept ", zones_score.intercept_[0]
print "RMSE: ", sqrt(mean_squared_error(y, zones_score.predict(np.array(qud["Zones"]).reshape(len(qud),1))))


I am working on the file "2. Python Advanced.ipynb".
The first task is:
Basics on Class and OOP
This section is built around the fundamental of Object Oriented Programming (OOP).
It aims strengthening basics but doesn't justify the broad topic itself. As OOP is a very important programming concept you should read further to better get a grip on python as well as go deep in understanding how it is useful and essential to programming.
Below are some essential resources :
- Improve Your Python: Python Classes and Object Oriented Programming
- Learn Python The Hard Way
- Python For Beginners
- A Byte Of Python
OOP
In all the code we wrote till now, we have designed our program around functions i.e. blocks of statements which manipulate data. This is called the procedure-oriented way of programming.
There is another way of organizing your program which is to combine data and functionality and wrap it inside something called an object. This is called the object oriented programming paradigm.
Most of the time you can use procedural programming, but when writing large programs or have a problem that is better suited to this method, you can use object oriented programming techniques.
Classes and Objects
Classes and objects are the two main aspects of object oriented programming. A class creates a new type where objects are instances of the class.
Objects can store data using ordinary variables that belong to the object. Variables that belong to an object or class are referred to as fields or attributes.
Objects can also have functionality by using functions that belong to a class. Such functions are called methods of the class.
The simplest class possible is shown in the following example :
Can you write Python code for it?

class Person:
    pass  # An empty block

p = Person()
print(p)


I am working on the file "2. Python Advanced.ipynb".
The first task is:
For more efficient Search Algorithms, look through the  Algorithm Implementation section of this repository

Sorting
Sorting is the process of placing elements from a collection in some kind of order.
For example, a list of words could be sorted alphabetically or by length.
A list of cities could be sorted by population, by area, or by zip code.
Python lists have a built-in sort() method that modifies the list in-place and a sorted() built-in function that builds a new sorted list from an iterable.

list.sort():  Modifies existing list and can be used only with lists.
sorted(list): Creates a new list when called and can be used with other iterables.

Basic sorting functions
The most basic use of the sorted function can be seen below :
Can you write Python code for it?

# Using sort() with a list.

values = [7, 4, 3, 6, 1, 2, 5]
print("Unsorted list :", values)     ## [7, 4, 3, 6, 1, 2, 5]
newValues = values.sort()
print("New list :", newValues)       ## None
print("Old list :", values)          ## [1, 2, 3, 4, 5, 6, 7]
print()
# Using sorted() with a list.

values = [7, 4, 3, 6, 1, 2, 5]
print("Unsorted list :", values)     ## [7, 4, 3, 6, 1, 2, 5]
newValues = sorted(values)
print("New list :", newValues)       ## [1, 2, 3, 4, 5, 6, 7]
print("Old list :", values)          ## [7, 4, 3, 6, 1, 2, 5]


I am working on the file "2. Python Advanced.ipynb".
The first task is:
In such situations its better to redefine the functions as below.
Can you write Python code for it?

egg_count = 0

def buy_eggs():
    return egg_count + 12

egg_count = buy_eggs()
print(egg_count)
egg_count = buy_eggs()
print(egg_count)


I am working on the file "2. Python Advanced.ipynb".
The first task is:
It is also possible for a programmer to cause a runtime exception by using the raise statement. For example, instead of calling the square root function with a negative number, we could have checked the value first and then raised our own exception. The code fragment below shows the result of creating a new RuntimeError exception. Note that the program would still terminate but now the exception that caused the termination is something explicitly created by the programmer.
Can you write Python code for it?

if anumber < 0:
    raise RuntimeError("You can't use a negative number")
else:
    print(math.sqrt(anumber))


I am working on the file "2. Python Advanced.ipynb".
The first task is:
List Methods
Besides simple accessing of values, lists have a large variety of methods that are used to performed different useful manipulations on them.
Some of them are:

list.append(element): adds a single element to the end of the list. Common error: does not return the new list, just modifies the original.
Can you write Python code for it?

# list.append example
names = ['Hermione Granger', 'Ronald Weasley']
names.append('Harry Potter')
print("New list :", names)  ## ['Hermione Granger', 'Ronald Weasley', 'Harry Potter']


I am working on the file "2. Python Advanced.ipynb".
The first task is:
Mutability
Strings are immutable and list are mutable.
For example :
Can you write Python code for it?

# Creating sentence and list form of sentence
name = "Welcome to coding with Python v3.6"
words = ["Welcome", "to", "coding", "with", "Python", "v3.6"]

print(name[4])
print(words[4])

# This is okay
words[5] = "v2.7"
print(words)

# This is not
# name[5] = "d"
# print(name)