I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Most passgeners came from Southampton, which number of males were double of female
Can you write Python code for it?

# quick look passenger embarktation

passenger_by_class_who = titanic.groupby(['Class','Embarked'])
print passenger_by_class_who.count()['Passenger']

# set plot title
ax = sns.plt.title('Embarkation by Class')

# show the counts of passengers
ax = sns.countplot(x = 'Class', hue = 'Embarked', data = titanic)

# add percentage for each group
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x(), height + 10, '%1.2f'%((height*100)/passengercount))


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Passenger class look not difference for each class
3.) Where are the passengers came from?
Can you write Python code for it?

# map Embarked port character to full name of each town
titanic['Embarked'] = titanic['Embarked'].dropna().map({'C' : 'Cherbourg', 'Q' : 'Queenstown', 'S': 'Southampton' })

# Check passenger who from

passenger_by_embark_who = titanic.groupby('Embarked')
print passenger_by_embark_who.count()['Passenger']

# set plot title
ax = sns.plt.title('Port of Embarkation')

# show the counts of passengers
ax = sns.countplot(x = 'Embarked', data = titanic)

# add percentage for each group
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x(), height + 10, '%1.2f'%((height*100)/passengercount))

# Check passenger who from

passenger_by_embark_who = titanic.groupby(['Embarked','Who'])
print passenger_by_embark_who.count()['Passenger']

# set plot title
ax = sns.plt.title('Port of Embarkation')

# show the counts of passengers
ax = sns.countplot(x = 'Embarked', hue = 'Who', data = titanic)

# add percentage for each group
for p in ax.patches:
    height = p.get_height()
    ax.text(p.get_x(), height + 10, '%1.2f'%((height*100)/passengercount))


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Question Phase
Here's questions we'll try to answer from the dataset
1.) Who are the passengers of Titanic?
2.) How old are the passengers?
3.) Where are the passengers came from?
4.) Who are purchase expensive ticket?
5.) Dose passenger class related with cabin position
6.) Who are the passengers travel alone or travel with family?
7.) What is the significant factors helped some passengers survive?

From dataset found 3 column had missing data "Age" -> missing 177 rows "Cabin" -> missing 687 rows "Embarked" -> missing 2 rows, let's start to fixed the missing data first
Data Wrangling Phase
Data cleaning
Can you write Python code for it?

## Find the total number of rows and the number of unique students (account keys)
## in each table.

def count_passenger(df):
    '''
    find the total number of row in dataframe

    Parameters:
    -----------
    df  : a dataframe.

    return:
    -----------
    len(df) : data lenght
    '''

    return len(df)

def remove_column(df, key):
    '''
    drop list of key from dataframe, return dataframe that exclude that list key

    Parameters:
    -----------
    df  : a dataframe.
    key : a key to remove from dataframe

    return:
    -----------
    new_df : data frame without key
    '''

    new_df = df.copy()

    return new_df.drop(key, axis=1)

def had_family(df):
    '''
    Check is passenger travel with family or alone

    Parameters:
    -----------
    df  : a dataframe.

    return:
    -----------
    'With family' : if travel with family
    'Alone'       : if travel alone
    '''

    if df > 0:
        return 'With family'
    else:
        return 'Alone'

def had_alive(df):
    '''
    Check is passenger who alived or died

    Parameters:
    -----------
    df  : a dataframe.

    return:
    -----------
    'Alive' : if Survied = 1
    'Died'  : if Survied = 0
    '''

    if df > 0:
        return "alive"
    else:
        return "died"

def had_class(df):
    '''
    Check is passenger class

    Parameters:
    -----------
    df  : a dataframe.

    return:
    -----------
    '1st class' : if Pclass = 1
    '2nd class' : if Pclass = 2
    '3rd class' : if Pclass = 3
    '''

    if df == 1:
        return 'First'
    elif df == 2 :
        return 'Second'
    else:
        return 'Third'

# First let's make a function to sort through the sex
def had_child(passenger):
    '''
    Treat passenger who age under 16 as a child

    Parameters:
    -----------
    passenger  : a objcect.

    return:
    -----------
    'child' : if age <= 15
    sex     : if age > 15

    '''

    #...


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
The survival rates for men are too lower than women and child for all class and embarked.
The 3rd class women passenger survival rates look like it lowest than other class.
The 2nd class men like survival rates very low thand other class.
From this data the result is show that being a male in any class dramatically decreases a chances of survival.
Can you write Python code for it?

# set range of age for linear plot
age_range = [10,20,40,60,80]

# looking age versus survival by linear plot
ax = sns.lmplot('Age','Survived',
               data = titanic, palette = 'winter',
               x_bins = age_range)


I am working on the file "P2-Investigate Titanic data/Investigate_titanic_data.ipynb".
The first task is:
Wow!, interesting result, faminly passenger show hight survived than who alone.
Can you write Python code for it?

# how about survival rate if relate with family or alone
ax = sns.factorplot(x = 'Family', y = 'Survived',
                   hue = 'Class', col = 'Embarked',
                   data = titanic, size = 5, aspect = .6)

# how about survival rate of each gender who with family or alone
ax = sns.factorplot(x = 'Family', y = 'Survived',
                   hue = 'Sex', col = 'Embarked',
                   data = titanic, size = 5, aspect = .6)


I am working on the file "P2-Investigate a Dataset/P2.ipynb".
The first task is:
1. Embark position
Can you write Python code for it?

# According to the embark poistion, 'gourpby' data
# And count the number of people survived or not embarked through different poistion
embarked_survived=survived.groupby('Embarked').count().Survived
embarked_totalcounts=data.groupby('Embarked').count().Survived

survived_total=len(survived)/len(data)

sum(pd.isnull(data.Embarked))


I am working on the file "P2-Investigate a Dataset/P2.ipynb".
The first task is:
From the graph, we can see that the survive rate for the people who embarked through C are obviously higher than Q or S models, meanwhile, Q model's survive rate is also higher than S'model. Therefore, I expect that embarking through different poistion might influence people's chance to survive.
Null hypothesis: There is no differences in terms of population mean of survive rate between passengers with different embark position, that is the survive rates for people who embarked through C,Q,S, these three variables are independent.
Alternative hypothesis: The population mean of survive rate for the passenger embarked through C,Q,S, these three variables are not independent.
Taking a Chi-Squared test, with $\alpha$level equals 0.05.
Can you write Python code for it?

from scipy.stats import chi2_contingency
pivot_embarked = pd.pivot_table(data = data[['Survived', 'Embarked']], index = 'Survived', columns = ['Embarked'], aggfunc = len)
chi2, p_value, dof, expected = chi2_contingency(pivot_embarked)
print ("Results of Chi-Squared test on Embarked to Survival.")
print ("Does Embark position have a significant effect on Survival?")
print ("Chi-Squared Score = %f"%chi2)
print ("Pvalue = %s"%str(p_value))


I am working on the file "P2-Investigate a Dataset/P2.ipynb".
The first task is:
From the graphs, we can see that the survive conditions for the people with different socio-economic class obviously vary with each other. Fisrt, we got much fewer class one and two passengers than class three originally before the accident, which also may symbolize that only wealthy people could afford upper class tickets. However, after the accident, the higher class, there were more people survived. Moveover, we can easily tell passengers with higher class have much higher survive rate through the third graphs. Therefore, I expect that people with higher socio-economic class (like class 1) have more chance to survive.
Null hypothesis: There is no differences in terms of population mean of survive rate between different socio-economic class, that is the survive rates for people with different class 1,2,3, these three variables are independent.
Alternative hypothesis: There is significant differences of survive rate between different socio-economic class, that is the survive rates for people with different class 1,2,3, these three variables are not independent.
Taking a Chi-Squared test, with $\alpha$level equals 0.05.
Can you write Python code for it?

from scipy.stats import chi2_contingency
pivot_pclass = pd.pivot_table(data = data[['Survived', 'Pclass']], index = 'Survived', columns = ['Pclass'], aggfunc = len)
chi2, p_value, dof, expected = chi2_contingency(pivot_pclass)
print ("Results of Chi-Squared test on Pclass to Survival.")
print ("Does Pclass have a significant effect on Survival?")
print ("Chi-Squared Score = %f"%chi2)
print ("Pvalue = %s"%str(p_value))


I am working on the file "P2-Investigate a Dataset/P2.ipynb".
The first task is:
The results are significant for all reasonable alphas, including 0.05, so we reject the null hypothesis, and accept the alternative hypothesis. That is the survive rates for people with different class 1,2,3, these three variables are not independent. Along with former graphs, I suppose that the population mean of survive rate for the passenger with higher class mode will be higher than the counterpart of lower class.
'Pclass' represent socio-economic status, and 1 is upper class while 3 is lower class. From the results we got, we can audaciously hypothesis that wealthy people were put on lifeboats first which were very limited. With the run out of lifeboats, poor people were only conuted on their own, which obviously resulted in much lower survive rate among them.
3. Male/Female
Can you write Python code for it?

sex_survived=survived.groupby('Sex').count().Survived
sex_totalcounts=data.groupby('Sex').count().Survived

sex_rate=np.array(sex_survived)/np.array(sex_totalcounts)

sum(pd.isnull(survived.Sex)) # Also no missing data in sex column


I am working on the file "P2-Investigate a Dataset/P2.ipynb".
The first task is:
The results are significant for all reasonable alphas, including 0.05, we reject the null hypothesis, and accept the alternative hypothesis. Survive rate for the passenger embarked through C,Q,S, these three variables are not independent. Combining with former graph, I deduced that embarking through C does help people get more chance to survive, comparing to the other two population.
C represents Cherbourg, so the people who embarked from Cherbourg might just are natives belong to this town. Therefore, we can propose that they might be stronger or more familiar with extreme situation and know more about how to safe themselve, and finally they got more chance to survive.
2. Passenger class
Can you write Python code for it?


I am working on the file "P2-Investigate a Dataset/P2.ipynb".
The first task is:
There are two embarked data(in 'Embarked column') are missed, and labeled as 'NaN'.  However, it doesn't influence how we count the number of people involved when we used 'gourpby.count()' function for a numpy object.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
# compare the surviving rate of different embark poistion, including the whole population.
embarked_survivedrate=np.array(embarked_survived)/np.array(embarked_totalcounts)
all_survivedrate=np.append(embarked_survivedrate,survived_total)
labels=['C','Q','S','Total']
plt.bar(np.arange(4),all_survivedrate,align='center',tick_label=labels)
plt.xlabel('Embark position')
plt.ylabel('Survived rate (ratio)')
plt.title('The difference of survive rate as function of embark position')
for a,b in zip(np.arange(4),all_survivedrate):
    c=str(b*100)[:5]+"%"
    plt.text(a,b+0.01,c,horizontalalignment='center')


I am working on the file "P2-Investigate a Dataset/P2.ipynb".
The first task is:
There is no missing data in passengers' sex (the 'Sex' column)
Can you write Python code for it?

plt.bar([1.2,3.2],sex_totalcounts,width=0.8,color='green',label='Original total counts')
plt.bar([2,4],sex_survived,width=0.8,color='orange',label='Survived counts')
plt.xlim(0,6)
plt.ylim(0,650)
plt.xticks([2,4],('female','male'))
plt.legend(loc='best')
plt.ylabel('people counts')
plt.title('The number of people comparison before and after accident within different sex')
for a,b in zip([1.6,3.6],sex_totalcounts):
    c=str(b)
    plt.text(a,b+5,c,horizontalalignment='center')
for a,b in zip([2.4,4.4],sex_survived):
    c=str(b)
    plt.text(a,b+5,c,horizontalalignment='center')

plt.barh([1,2],sex_rate,align='center',color=['green','orange'])
plt.yticks([1,2],['female','male'])
plt.xlabel('survive rate(ratio)')
plt.title('The difference of survive rate as function of sex')
for a,b in zip([1,2],sex_rate):
    c=str(b*100)[:5]+'%'
    plt.text(b+0.05,a,c,horizontalalignment='center')


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Accuracy on testing set
Can you write Python code for it?

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('model2/'))

    test_accuracy = evaluate(X_test, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Accuracy on training set
Can you write Python code for it?

BATCH_SIZE = 128
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('model2/'))

    train_accuracy = evaluate(X_train, y_train)
    print("Train Accuracy = {:.3f}".format(train_accuracy))


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Analyze Performance
Can you write Python code for it?

accuracy = np.sum(predictions == ground_truth) / new_test_set.shape[0]
print('Accuracy = {0}%'.format(accuracy * 100))


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Plotting the distribution of samples in the three datasets
How many examples of each class can be found on the training, validation and testing sets.
Can you write Python code for it?

count_train = []
count_valid = []
count_test = []
for i in range(n_classes):
    count_train.append(len(y_train[y_train==i]))
    count_valid.append(len(y_valid[y_valid==i]))
    count_test.append(len(y_test[y_test==i]))

plt.figure(figsize=(15,8))
plt.subplot(131)
plt.bar(np.arange(n_classes),height=count_train)
plt.title('Training distribution')

plt.subplot(132)
plt.bar(np.arange(n_classes),height=count_valid)
plt.title('Validation distribution')

plt.subplot(133)
plt.bar(np.arange(n_classes),height=count_test)
plt.title('Testing distribution')


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Predict the Sign Type for Each Image
Can you write Python code for it?

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('model2/'))

    out_logits = sess.run(logits, feed_dict={x: new_test_set, keep_prob: 1.0})

predictions = np.argmax(out_logits,1)

df = pd.read_csv('signnames.csv')

plt.figure(figsize=(20,8))
for i in range(new_test_set.shape[0]):
    image = new_test_set[i].squeeze()
    label = df.loc[predictions[i]]['SignName']
    plt.subplot(2,5,i+1)
    plt.imshow(image)
    plt.title(label)


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Project Writeup
Once you have completed the code implementation, document your results in a project writeup using this template as a guide. The writeup can be in a markdown or pdf file.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to  \n",
    "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

Step 4 (Optional): Visualize the Neural Network's State with Test Images
This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.
Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the LeNet lab's feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.
For an example of what feature map outputs look like, check out NVIDIA's results in their paper End-to-End...
Can you write Python code for it?

### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.

# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry

def outputFeatureMap(image_input, tf_activation, sess, activation_min=-1, activation_max=-1 ,plt_num=1):
    # Here make sure to preprocess your image_input in a way your network expects
    # with size, normalization, ect if needed
    # image_input =
    # Note: x should be the same name as your network's tensorflow data placeholder variable
    # If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
    activation = tf_activation.eval(session=sess,feed_dict={x : image_input})
    featuremaps = activation.shape[3]
    plt.figure(plt_num, figsize=(15,15))
    for featuremap in range(featuremaps):
        plt.subplot(6,8, featuremap+1) # sets the number of feature maps to show on each row and column
        plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
        if activation_min != -1 & activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
        elif activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
        elif activation_min !=-1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
        else:
           ...


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Step 2: Design and Test a Model Architecture
Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.
The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!
With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89. To meet specifications, the validation set accuracy will need to be at least 0.93. It is possible to get an even higher accuracy, but 0.93 is the minimum for a successful project submission.
There are various aspects to consider when thinking about this problem:

Neural network architecture (is the network over or underfitting?)
Play around preprocessing techniques (normalization, rgb to grayscale, etc)
Number of examples per label (some have more than others).
Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.
Pre-process the Data Set (normalization, grayscale, etc.)
Minimally, the image data should be normalized so that the data has mean zero and equal variance. For image data, (pixel - 128)/ 128 is a quick way to approximately normalize the data and can be used in this project.
Other pre-processing steps are optional. You can try different techniques to see if it improves performance.
Use the code cell (or multiple code cells, if necessary) to implement the first step of your project.
*Note:
All the preprocessing steps (Normalization and grayscale conversion) are performed directly on TensorFlow
Model Architecture
Can you write Python code for it?

from tensorflow.contrib.layers import flatten

num_channels = 1

def Traffic_Net(x, keep_prob):
    mu = 0
    sigma = 0.1

    # Layer 1: Convolutional. Input = 32x32xnum_channels. Output = 30x30x16.
    conv1_W = tf.Variable(tf.truncated_normal(shape=(3, 3, num_channels, 16), mean = mu, stddev = sigma))
    conv1_b = tf.Variable(tf.zeros(16))
    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b
    conv1   = tf.nn.relu(conv1)

    # Layer 2: Convolutional. Input = 30x30x16. Output = 28x28x16.
    conv2_W = tf.Variable(tf.truncated_normal(shape=(3, 3, 16, 16), mean = mu, stddev = sigma))
    conv2_b = tf.Variable(tf.zeros(16))
    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b
    conv2   = tf.nn.relu(conv1)

    # Pooling. Input = 28x28x16 Output = 14x14x16.
    conv2_pool = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # Layer 3: Convolutional. Output = 12x12x24.
    conv3_W = tf.Variable(tf.truncated_normal(shape=(3, 3, 16, 24), mean = mu, stddev = sigma))
    conv3_b = tf.Variable(tf.zeros(24))
    conv3   = tf.nn.conv2d(conv2_pool, conv3_W, strides=[1, 1, 1, 1], padding='VALID') + conv3_b
    conv3   = tf.nn.relu(conv3)

    # Layer 4: Convolutional. Output = 10x10x24.
    conv4_W = tf.Variable(tf.truncated_normal(shape=(3, 3, 24, 24), mean = mu, stddev = sigma))
    conv4_b = tf.Variable(tf.zeros(24))
    conv4   = tf.nn.conv2d(conv3, conv4_W, strides=[1, 1, 1, 1], padding='VALID') + conv4_b
    conv4   = tf.nn.relu(conv4)

    # Pooling. Input = 10x10x24. Output = 5x5x24.
    conv4_pool = tf.nn.max_pool(conv4, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # Flatten. Input = 5x5x16. Output = 600.
    hidden_layer_0 = flatten(conv4_pool)
    hidden_layer_0 = tf.nn.dropout(hidden_layer_0, keep_prob)

    # Fully Connected. Input = 600. Output = 600.
    fc1_W = tf.Variable(tf.truncated_normal(shape=(600, 400), mean = mu, stddev = sigma))
    fc1_b =...


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Train, Validate and Test the Model
A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation
sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.
Can you write Python code for it?

x = tf.placeholder(tf.float32, (None, 32, 32, 3))
y = tf.placeholder(tf.int32, (None))

# To Grayscale
x_gray = tf.image.rgb_to_grayscale(x)
# Normalization
x_norm = tf.map_fn(lambda img: tf.image.per_image_standardization(img), x_gray)

one_hot_y = tf.one_hot(y, n_classes)
keep_prob = tf.placeholder(tf.float32)

rate = 0.0001

logits, activation_layers = Traffic_Net(x_norm, keep_prob)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss)

correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()

def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, keep_prob: 1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples


I am working on the file "P2-Traffic-Sign-Recognition/Traffic_Sign_Classifier.ipynb".
The first task is:
Training the model
Can you write Python code for it?

from sklearn.utils import shuffle

EPOCHS = 30
BATCH_SIZE = 128
dropout = 0.8

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train)

    log_loss = []
    log_valid_acc = []

    print("Training...")
    print()
    for i in range(EPOCHS):
        X_train, y_train = shuffle(X_train, y_train)
        train_loss = []
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset:end], y_train[offset:end]
            _, loss_train = sess.run((training_operation, loss), feed_dict={x: batch_x, y: batch_y, keep_prob: dropout})
            train_loss.append(loss_train)

        validation_accuracy = evaluate(X_valid, y_valid)
        print("EPOCH {} ...".format(i+1))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print()

        log_loss.append(np.mean(train_loss))
        log_valid_acc.append(validation_accuracy)

    saver.save(sess, 'model2/trained_model')
    print("Model saved")


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
2. Numpy exercises
2.1. Create numpy arrays
The following code fragment defines variable x as a list of 4 integers, you can check that by printing the type of any element of x.
Can you write Python code for it?


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Another key difference with Matlab is that the "*" operator won't give us matrix multiplication. It will instead compute an element-wise multiplication. Again, numpy has a built-in function for this purpose that will guarantee propper handling of errors:
Can you write Python code for it?

# We can perform element-wise multiplication by using the * operator:
x1 = np.arange(9).reshape((3, 3))
x2 = np.ones((3, 3)) * 2 # a 3x3 array with 2s in every cell
result = x1 * x2
print('x1:\n', x1, '\n\nx2:\n', x2)
print('\nElement-wise multiplication of x1 and x2 using the * operator:\n', result)

# or by using the built-in numpy function:
result = np.multiply(x1, x2)
print('\nElement-wise multiplication of x1 and x2 using built-in functions:\n', result)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Effectively, array1 is a 1D array, whereas array2 is a 2D  array.  There are some methods that will let you modify the dimensions of an array. To go from a 2-D to 1-D array we have the methods flatten(), ravel() and reshape(). Check the result of the following code (you can use the help function to check the funcionalities of each method).
Can you write Python code for it?

x1 = np.arange(9).reshape((3, 3))
print('x1:\n', x1)
print('Its shape is: ', x1.shape)

print('\n Use the method flaten:')
print('x1.flatten(): ', x1.flatten())
print('Its shape is: ', x1.flatten().shape)

print('\n Use the method ravel:')
print('x1.ravel(): ', x1.ravel())
print('Its shape is:', x1.ravel().shape)

print('\n Use the method shape:')
print('x1.reshape(-1): ', x1.reshape(-1))
print('Its shape is: ', x1.reshape(-1).shape)

# Note that here the method shape is used to reorganize the array into a 1-D
# array. A more common use of reshape() is to simply redimension an array from
# shape (i, j) to shape (i', j') satisfying the condition i*j = i'*j'.
# For example:
print('\n A more common use of reshape():')
x1 = np.arange(12).reshape((4, 3))
print('x1:\n', x1)
print('Its shape is: ', x1.shape)

print('\nx1.reshape((2,6)):\n', x1.reshape((2,6)))
print('Its shape is: ', x1.reshape((2,6)).shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Exercise 2: Complete the following exercises:
Can you write Python code for it?

# 1. Define a new 3x2 array named my_array with [1, 2, 3] in the first row and
# [4, 5, 6] in the second. Check the dimension of the array.
# my_array = <FILL IN>
my_array = np.array([[1, 2, 3],[4, 5, 6]])
print(my_array)
print('Its shape is: ', np.shape(my_array))

#2. Define a new 3x4 array named my_zeros with all its elements to zero
# my_zeros = <FILL IN>
my_zeros = np.zeros((3,4))
print('A 3x4 vector of zeros:')
print(my_zeros)

#3. Define a new 4x2 array named my_ones with all its elements to one
# my_ones = <FILL IN>
my_ones = np.ones((4,2))
print('A 4x2 vector of ones:')
print(my_ones)

#4. Modify the dimensions of my_ones to a 2x4 array using command np.reshape()
# my_ones2 = <FILL IN>
my_ones2 = my_ones.reshape((2,4))
print('A 2x4 vector of ones:')
print(my_ones2)

#5. Define a new 4x4 identity array named my_eye
# my_ones2 = <FILL IN>
my_ones2 = np.eye((4))
print('A 4x4 identity vector:')
print(my_ones2)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Exercise 3: In the next cell you'll find a few exercises for you to practice these operations.
Can you write Python code for it?

# Complete the following exercises. Print the partial results to visualize them.

# Create a 3x4 array called `y`. It's up to you to decide what it contains.
#y = <FILL IN>
y = np.arange(12).reshape((3, 4))

# Create a column vector of length 3 called `x_col`.
#x_col = <FILL IN>
x_col = np.ones((3, 1)) * 2

# Multiply the 2-D array `y` by 2
#y_by2 = <FILL IN>
y_by2 = y * 2

# Multiply each of the columns in `y` by the column vector `x_col`
#z = <FILL IN>
z = x_col * y

# Obtain the matrix product of the transpose of x_col and y
#x_by_y = <FILL IN>
x_by_y = x_col.T.dot(y)

# Compute the sine of a vector that spans from -5 to 5 in increments of 0.5
#x = <FILL IN>
x = np.arange(-5, 5.5, 0.5)
#x_sin = <FILL IN>
x_sin = np.sin(x)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Exercise 4: Complete the following excercise:
Can you write Python code for it?

# Given the following matrix and vector:
vect = np.arange(3)[:, np.newaxis]
mat = np.arange(9).reshape((3, 3))

# Apply the necessary transformation to vect so that you can perform the matrix
# multiplication np.matmul(vect, mat)

# vect = <FILL IN>
vect = np.squeeze(vect)

print(np.matmul(vect, mat))


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Exercise 7: Complete the following excersises:
Can you write Python code for it?

X = np.arange(0,25).reshape((5,5))
print('X:\n',X)

# 1. Keep last row of matrix X
#X_sub1 = <FILL IN>
X_sub1 = X[-1,]
print('\nX_sub1: \n',X_sub1)

# 2. Keep first column of the three first rows of X
#X_sub2 = <FILL IN>
X_sub2 = X[:3,0]
print('\nX_sub2: \n',X_sub2)

# 3. Keep first two columns of the three first rows of X
#X_sub3 = <FILL IN>
X_sub3 = X[:3,:2]
print('\nX_sub3: \n',X_sub3)

# 4. Invert the order of the rows of X
#X_sub4 = <FILL IN>
X_sub4 = X[::-1,:]
print('\nX_sub4: \n',X_sub4)

# 5. Keep odd columns (first, third...) of X
#X_sub5 = <FILL IN>
X_sub5 = X[:,::2]
print('\nX_sub5: \n',X_sub5)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Exercise 8: Given array x below, select the subarray formed by the columns of x that add more than 4.:
Can you write Python code for it?

#

x = np.arange(12).reshape((3, 4))

# Write your code here
# <SOL>
idx = np.where(np.sum(x, axis=0) > 4)[0]
x_new = x[:, idx]
# </SOL>

print(x_new)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Finally, we can transpose a matrix by using the numpy.transpose() function, the ndarray.transpose() method or its abbreviated version, ndarray.T. We usually use the abbreviated version, but the other forms have their place in certain contexts. Check their documentations to see what options they offer:
Can you write Python code for it?

# Three different ways of transposing a matrix:
x1 = np.arange(9).reshape((3, 3))
print('x1:\n', x1)
print('\nTranspose of x1 using the numpy function:\n', np.transpose(x1))
print('\nTranspose of x1 using the ndarray method:\n', x1.transpose())
print('\nTranspose of x1 using the abbreviated form:\n', x1.T)
print('\nOddly enough, the three methods produce the same result!')


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
If you want to apply a transformation over each element of this list you have to build a loop and operate over each element.
Exercise 1: Complete the following code to create a new list with the same elements as x, but where each element of the list is a float (you can use the float() function).
Can you write Python code for it?

# Create a list of floats containing the same elements as in x

x_f = []
for element in x:
  # <FILL IN>
  x_f.append(float(element))

print(x_f)
print(type(x_f[0]))


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Note: Compare the shape of x and y with the shape of z and note the difference between 1-D and N-D numpy arrays (ndarrays). We will later review this issue in detail.
We can also convert a numpy array or matrix into a python list with the method np.tolist().
Can you write Python code for it?

my_array = np.arange(9).reshape((3, 3))
print(my_array)
print('the type is: ', type(my_array))

# Convert my_array to list

my_array_list =  my_array.tolist()
print('\n', my_array_list)
print('the type is: ',type(my_array_list))


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Numpy also gives us funtions to perform matrix multiplications and dot products.
Can you write Python code for it?

# We can perform matrix multiplication:
x1 = np.arange(9).reshape((3, 3))
x2 = np.ones((3, 3)) * 2 # a 3x3 array with 2s in every cell
result = np.matmul(x1, x2)
print('x1:\n', x1, '\n\nx2:\n', x2)
print('\nProduct of x1 and x2:\n', result)

# Or the dot product between vectors:
v1 = np.arange(4)
v2 = np.arange(3, 7)
result = np.dot(v1, v2)
print('\nv1:\n', v1, '\n\nv2:\n', v2)
print('\nDot product of v1 and v2:\n', result)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Other numpy methods where you can specify the axis along with a certain operation should be carried out are:

np.median()
np.std()
np.var()
np.percentile()
np.sort()
np.argsort()

If the axis argument is not provided, the array is flattened before carriying out the corresponding operation.
Exercise 5: Complete the following exercises:
Can you write Python code for it?

# Given the following list of heights:
heights = [1.60, 1.85, 1.68, 1.90, 1.78, 1.58, 1.62, 1.60, 1.70, 1.56]

# 1. Obtain a 2x5 array, called `h_array`, using the methods you learnt above.

# h_array = <FILL IN>
h_array = np.array(heights).reshape((2,5))
print('h_array: \n',h_array)
print('Its shape is: \n', h_array.shape)

# 2. Use method mean() to get the mean of each column, and the mean of each row.
# Store them in two vectors, named `mean_column` and `mean_row` respectively.

#mean_column = <FILL IN>
mean_column = np.mean(h_array,axis=0)
#mean_row = <FILL IN>
mean_row = np.mean(h_array,axis=1)

print('\nMean of each column: \n',mean_column)
print('Its shape is (it must coincide with number of columns):\n', mean_column.shape)

print('\nMean of each row: \n',mean_row)
print('Its shape is (it must coincide with number of rows):\n', mean_row.shape)

# 3. Obtain a 5x2 array by multiplying the mean vectors. You may need to create a
# new axis. The array name should be `new_array`

#new_array = <FILL IN>
new_array = mean_column[:,np.newaxis].dot(mean_row[:,np.newaxis].T)
print('\nNew array: \n',new_array)
print('New array shape: \n', new_array.shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
The output should be:
[5.0, 4.0, 3.0, 4.0]
<class 'float'>
Numpy is a python library that lets you work with data vectors and matrices (we will call them numpy arrays) and directly apply operations over these arrays without the need to operate element by element.
Numpy arrays can be defined directly using methods such as np.arange(), np.ones(), np.zeros(), np.eye(), as well as random number generators. Alternatively, you can easily generate them from python lists (or lists of lists) containing elements of a numerical type by using np.array(my_list).
You can easily check the shape of any numpy array with the property .shape.
Can you write Python code for it?

# Numpy arrays can be created from numeric lists or using different numpy methods
y = np.arange(8) + 1
x = np.array(x_f)
z = np.array([[1, 2], [2, 1]])

# The arange() function generates vectors of equally spaced numbers. We can
# specify start and stop positions as well as the step length (the steps don't
# need to be integers!):
print('A vector that goes from 2 to 8 in steps of 2: ', np.arange(2, 9, 2))

# Numpy also has a linspace() function that works exactly like its Matlab
# counterpart:
print('\nA vector of length 5 that spans from 0 to 1 in constant increments:\n',
      np.linspace(0, 1, 5))

# Check the different data types involved
print('\nThe type of variable x_f is ', type(x_f))
print('The type of variable x is ', type(x))

# Print the shapes of the numpy arrays
print('\nThe variable x has shape ', x.shape)
print('The variable y has shape ', y.shape)
print('The variable z has shape ', z.shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
The output should be:
`
[15 18 21]
2.3. Numpy methods that can be carried out along different dimensions
Compare the result of the following commands:
Can you write Python code for it?

x1 = np.arange(24).reshape((8, 3))

print(x1.shape)
print(np.mean(x1))
print(np.mean(x1,axis=0))
print(np.mean(x1,axis=1))


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
The output should be:
```
X:
 [[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]
 [15 16 17 18 19]
 [20 21 22 23 24]]
X_sub1:
 [20 21 22 23 24]
X_sub2:
 [ 0  5 10]
X_sub3:
 [[ 0  1]
 [ 5  6]
 [10 11]]
X_sub4:
 [[20 21 22 23 24]
 [15 16 17 18 19]
 [10 11 12 13 14]
 [ 5  6  7  8  9]
 [ 0  1  2  3  4]]
X_sub5:
 [[ 0  2  4]
 [ 5  7  9]
 [10 12 14]
 [15 17 19]
 [20 22 24]]
 ```
We have seen how slicing allows us to index over the different dimensions of a given array. In the previous examples we learned how to select the rows and columns we're interested in, but how can we select only the elements of an array that meet a specific condition?
Numpy provides us with the method np.where(condition). A common way of using this function is by setting a condition involving an array. For example, the condition x > 5 will give us the indexes in which x contains numbers higher than 5.
Can you write Python code for it?

x = np.array([-3,-2,-1,0,1,2,3])

# Create a new vector `y` with the elements of x, but replacing by 0 each number whose
# absolute value is 2 or less.

y = np.copy(x) # CAUTION! Doing y = x will create two pointers to the same array.
condition = np.abs(x)<=2
y[np.where(condition)[0]]=0
# Note that np.where() returns a tuple. In this case the second element of the
# tuple is empty. Read the np.where() docstring for more info.

print('Before conditioning: \n',x)
print('\nAfter conditioning: \n',y)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
The output should be:
```
[[1 2 3]
 [4 5 6]]
Its shape is:  (2, 3)
A 3x4 vector of zeros:
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]]
A 4x2 vector of ones:
[[1. 1.]
 [1. 1.]
 [1. 1.]
 [1. 1.]]
A 2x4 vector of ones:
[[1. 1. 1. 1.]
 [1. 1. 1. 1.]]
```
2.2 Numpy operations
We can perform all the usual numerical and matrix operations with numpy. In the case of matrix addition and subtraction, we can use the common "+" or "-" operators:
Can you write Python code for it?

x1 = np.arange(9).reshape((3, 3))
x2 = np.ones((3, 3))
result = x1 + x2
print('x1:\n', x1, '\n\nx2:\n', x2)
print('\nAddition of x1 and x2 using the + operator:\n', result)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
The output should be:
```
h_array:
 [[1.6  1.85 1.68 1.9  1.78]
 [1.58 1.62 1.6  1.7  1.56]]
Its shape is:
 (2, 5)
Mean of each column:
 [1.59  1.735 1.64  1.8   1.67 ]
Its shape is (it must coincide with number of columns):
 (5,)
Mean of each row:
 [1.762 1.612]
Its shape is (it must coincide with number of rows):
 (2,)
New array:
 [[2.80158 2.56308]
 [3.05707 2.79682]
 [2.88968 2.64368]
 [3.1716  2.9016 ]
 [2.94254 2.69204]]
New array shape:
 (5, 2)```
2.4. Concatenating arrays
Provided that the corresponding dimensions fit, horizontal and vertical stacking of matrices can be carried out with methods np.hstack() and np.vstack().
Exercise 6: Complete the following exercises to practice matrix concatenation:
Can you write Python code for it?

my_array = np.array([[1, -1, 3, 3],[2, 2, 4, 6]])
print('Array 1:')
print(my_array)
print(my_array.shape)

my_array2 = np.ones((2,3))
print('Array 2:')
print(my_array2)
print(my_array2.shape)

# Vertically stack matrix my_array with itself
#ex1_res = <FILL IN>
ex1_res = np.vstack((my_array,my_array))
print('Vertically stack:')
print(ex1_res)

# Horizontally stack matrix my_array and my_array2
#ex2_res = <FILL IN>
ex2_res = np.hstack((my_array,my_array2))
print('Horizontally stack:')
print(ex2_res)

# Transpose the vector `my_array`, and then stack a ones vector
# as the first column. Alternatively, you can stack a row, and then transpose.
# Just make sure that the final shape is (4,3). Name it `expanded`:

#ones_v = <FILL_IN>
ones_v = np.ones((4,1))
#expanded = <FILL IN>
expanded = np.hstack((ones_v,my_array.T))

print('Expanded array: \n',expanded)
print('Its shape is: \n', expanded.shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
We might also need to remove empty or unused dimensions. For this we have np.squeeze():
Can you write Python code for it?

array1_1D = np.squeeze(array1)
print('1D array:\n',array1_1D)
print('Its shape is: ', array1_1D.shape)

array2_1D = np.squeeze(array2)
print('\n1D array:\n',array2_1D)
print('Its shape is: ', array2_1D.shape)

array3_1D = np.squeeze(array3)
print('\n1D array:\n',array3_1D)
print('Its shape is: ', array3_1D.shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_professor.ipynb".
The first task is:
Whether you use the basic operators or the built-in functions will depend on the situation.
We can also add or subtract column or row vectors from arrays. Again, both the basic operators and the built in functions will perform the same operations. Unlike in Matlab, where this operation will raise an error, Python will automatically execute it row by row or column by column as appropriate:
Can you write Python code for it?

# We can add or subtract row and column vectors:
row_vect = np.ones((1, 3))
col_vect = np.ones((3, 1))
result = np.add(x1, row_vect)
print('x1:\n', x1, '\n\nrow_vect:\n', row_vect, '\n\ncol_vect:\n', col_vect)
print('\nAddition of a row vector:\n', result)
result = np.add(x1, col_vect)
print('\nAddition of a column vector:\n', result)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
2. Numpy exercises
2.1. Create numpy arrays
The following code fragment defines variable x as a list of 4 integers, you can check that by printing the type of any element of x.
Can you write Python code for it?


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
2.2. N-D numpy arrays
To correctly operate with numpy arrays we have to be aware of their dimensions. Are these two arrays  equal?
Can you write Python code for it?

array1 = np.array([1,1,1])
print('array1:\n', array1)

array2 = np.ones((3,1))
print('\narray2:\n', array2)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Effectively, array1 is a 1D array, whereas array2 is a 2D  array.  There are some methods that will let you modify the dimensions of an array. To go from a 2-D to 1-D array we have the methods flatten(), ravel() and reshape(). Check the result of the following code (you can use the help function to check the funcionalities of each method).
Can you write Python code for it?

x1 = np.arange(9).reshape((3, 3))
print('x1:\n', x1)
print('Its shape is: ', x1.shape)

print('\n Use the method flaten:')
print('x1.flatten(): ', x1.flatten())
print('Its shape is: ', x1.flatten().shape)

print('\n Use the method ravel:')
print('x1.ravel(): ', x1.ravel())
print('Its shape is:', x1.ravel().shape)

print('\n Use the method shape:')
print('x1.reshape(-1): ', x1.reshape(-1))
print('Its shape is: ', x1.reshape(-1).shape)

# Note that here the method shape is used to reorganize the array into a 1-D
# array. A more common use of reshape() is to simply redimension an array from
# shape (i, j) to shape (i', j') satisfying the condition i*j = i'*j'.
# For example:
print('\n A more common use of reshape():')
x1 = np.arange(12).reshape((4, 3))
print('x1:\n', x1)
print('Its shape is: ', x1.shape)

print('\nx1.reshape((2,6)):\n', x1.reshape((2,6)))
print('Its shape is: ', x1.reshape((2,6)).shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Element-wise division between matrices is performed using the ''/'' operator or the  divide() built-in function:
Can you write Python code for it?

# Element-wise division of two matrices:
x1 = np.arange(9).reshape((3, 3)) * 3
x2 = np.ones((3, 3)) * 3
result = np.divide(x1, x2)
print('x1:\n', x1, '\n\nx2:\n', x2)
print('\nElement-wise division of x1 and x2:\n', result)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Exercise 2: Complete the following exercises:
Can you write Python code for it?

# 1. Define a new 3x2 array named my_array with [1, 2, 3] in the first row and
# [4, 5, 6] in the second. Check the dimension of the array.
# my_array = <FILL IN>
print(my_array)
print('Its shape is: ', np.shape(my_array))

#2. Define a new 3x4 array named my_zeros with all its elements to zero
# my_zeros = <FILL IN>
print('A 3x4 vector of zeros:')
print(my_zeros)

#3. Define a new 4x2 array named my_ones with all its elements to one
# my_ones = <FILL IN>
print('A 4x2 vector of ones:')
print(my_ones)

#4. Modify the dimensions of my_ones to a 2x4 array using command np.reshape()
# my_ones2 = <FILL IN>
print('A 2x4 vector of ones:')
print(my_ones2)

#5. Define a new 4x4 identity array named my_eye
# my_ones2 = <FILL IN>
print('A 4x4 identity vector:')
print(my_ones2)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Exercise 4: Complete the following excercise:
Can you write Python code for it?

# Given the following matrix and vector:
vect = np.arange(3)[:, np.newaxis]
mat = np.arange(9).reshape((3, 3))

# Apply the necessary transformation to vect so that you can perform the matrix
# multiplication np.matmul(vect, mat)

# vect = <FILL IN>

print(np.matmul(vect, mat))


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Exercise 7: Complete the following excersises:
Can you write Python code for it?

X = np.arange(0,25).reshape((5,5))
print('X:\n',X)

# 1. Keep last row of matrix X
#X_sub1 = <FILL IN>
print('\nX_sub1: \n',X_sub1)

# 2. Keep first column of the three first rows of X
#X_sub2 = <FILL IN>
print('\nX_sub2: \n',X_sub2)

# 3. Keep first two columns of the three first rows of X
#X_sub3 = <FILL IN>
print('\nX_sub3: \n',X_sub3)

# 4. Invert the order of the rows of X
#X_sub4 = <FILL IN>
print('\nX_sub4: \n',X_sub4)

# 5. Keep odd columns (first, third...) of X
#X_sub5 = <FILL IN>
print('\nX_sub5: \n',X_sub5)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Exercise 8: Given array x below, select the subarray formed by the columns of x that add more than 4.:
Can you write Python code for it?

#

x = np.arange(12).reshape((3, 4))

# Write your code here
# <SOL>
# </SOL>

print(x_new)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Note: flatten() always returns a copy of the original vector, whereas ravel() and shape() returns a view of the original array whenever possible.
Sometimes we need to add a new dimension to an array, for example to turn a  1-D array into a 2-D column vector. For this we use np.newaxis.
Can you write Python code for it?

# Let's start with a 1-D array:
array1 = np.array([1,1,1])
print('1D array:\n',array1)
print('Its shape is: ', array1.shape)

# Let's turn it into a column vector (2-D array with dimension 1x3):
array2 = array1[:,np.newaxis]
print('\n2D array:\n',array2)
print('Its shape is: ', array2.shape)

# Let's turn it into a row vector (2-D array with dimension 3x1):
array3 = array1[np.newaxis,:]
print('\n2D array:\n',array3)
print('Its shape is: ', array3.shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Other numpy methods where you can specify the axis along with a certain operation should be carried out are:

np.median()
np.std()
np.var()
np.percentile()
np.sort()
np.argsort()

If the axis argument is not provided, the array is flattened before carriying out the corresponding operation.
Exercise 5: Complete the following exercises:
Can you write Python code for it?

# Given the following list of heights:
heights = [1.60, 1.85, 1.68, 1.90, 1.78, 1.58, 1.62, 1.60, 1.70, 1.56]

# 1. Obtain a 2x5 array, called `h_array`, using the methods you learnt above.

# h_array = <FILL IN>
print('h_array: \n',h_array)
print('Its shape is: \n', h_array.shape)

# 2. Use method mean() to get the mean of each column, and the mean of each row.
# Store them in two vectors, named `mean_column` and `mean_row` respectively.

#mean_column = <FILL IN>
#mean_row = <FILL IN>

print('\nMean of each column: \n',mean_column)
print('Its shape is (it must coincide with number of columns):\n', mean_column.shape)

print('\nMean of each row: \n',mean_row)
print('Its shape is (it must coincide with number of rows):\n', mean_row.shape)

# 3. Obtain a 5x2 array by multiplying the mean vectors. You may need to create a
# new axis. The array name should be `new_array`

#new_array = <FILL IN>
print('\nNew array: \n',new_array)
print('New array shape: \n', new_array.shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
The answer is no. We can easily check this by examining their shapes and dimensions:
Can you write Python code for it?

print('Shape of array1 :',array1.shape)
print('Number of dimensions of array1 :',array1.ndim)
print('Shape of array2 :',array2.shape)
print('Number of dimensions of array2 :',array2.ndim)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
The output should be:
`
[15 18 21]
2.3. Numpy methods that can be carried out along different dimensions
Compare the result of the following commands:
Can you write Python code for it?


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
The output should be:
```
[[1 2 3]
 [4 5 6]]
Its shape is:  (2, 3)
A 3x4 vector of zeros:
[[0. 0. 0. 0.]
 [0. 0. 0. 0.]
 [0. 0. 0. 0.]]
A 4x2 vector of ones:
[[1. 1.]
 [1. 1.]
 [1. 1.]
 [1. 1.]]
A 2x4 vector of ones:
[[1. 1. 1. 1.]
 [1. 1. 1. 1.]]
```
2.2 Numpy operations
We can perform all the usual numerical and matrix operations with numpy. In the case of matrix addition and subtraction, we can use the common "+" or "-" operators:
Can you write Python code for it?


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
The output should be:
```
h_array:
 [[1.6  1.85 1.68 1.9  1.78]
 [1.58 1.62 1.6  1.7  1.56]]
Its shape is:
 (2, 5)
Mean of each column:
 [1.59  1.735 1.64  1.8   1.67 ]
Its shape is (it must coincide with number of columns):
 (5,)
Mean of each row:
 [1.762 1.612]
Its shape is (it must coincide with number of rows):
 (2,)
New array:
 [[2.80158 2.56308]
 [3.05707 2.79682]
 [2.88968 2.64368]
 [3.1716  2.9016 ]
 [2.94254 2.69204]]
New array shape:
 (5, 2)```
2.4. Concatenating arrays
Provided that the corresponding dimensions fit, horizontal and vertical stacking of matrices can be carried out with methods np.hstack() and np.vstack().
Exercise 6: Complete the following exercises to practice matrix concatenation:
Can you write Python code for it?

my_array = np.array([[1, -1, 3, 3],[2, 2, 4, 6]])
print('Array 1:')
print(my_array)
print(my_array.shape)

my_array2 = np.ones((2,3))
print('Array 2:')
print(my_array2)
print(my_array2.shape)

# Vertically stack matrix my_array with itself
#ex1_res = <FILL IN>
print('Vertically stack:')
print(ex1_res)

# Horizontally stack matrix my_array and my_array2
#ex2_res = <FILL IN>
print('Horizontally stack:')
print(ex2_res)

# Transpose the vector `my_array`, and then stack a ones vector
# as the first column. Alternatively, you can stack a row, and then transpose.
# Just make sure that the final shape is (4,3). Name it `expanded`:

#ones_v = <FILL_IN>
ones_v = np.ones((4,1))
#expanded = <FILL IN>

print('Expanded array: \n',expanded)
print('Its shape is: \n', expanded.shape)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
We can use the "**" operator or the power() built-in function to raise elements from a matrix to a given power, or to raise elements of one matrix to positionally-corresponding powers in another matrix:
Can you write Python code for it?

# Performing power operations with the ** operator:
x1 = np.arange(9).reshape((3, 3))
result = x1**2
print('x1:\n', x1)
print('\nRaising all elements in x1 to the power of 2 using the ** operator:\n', result)

result = x1**x1
print('\nRaising all elements in x1 to themselves using the ** operator:\n', result)

# Performing power operations with the power() function:
result = np.power(x1, 2)
print('\nRaising all elements in x1 to the power of 2 using the power() function:\n', result)

result = np.power(x1, x1)
print('\nRaising all elements in x1 to themselves using the power() function:\n', result)


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
We might also need to remove empty or unused dimensions. For this we have np.squeeze():
Can you write Python code for it?


I am working on the file "P2.Numpy/P2_Numpy_basics_student.ipynb".
The first task is:
Whether you use the basic operators or the built-in functions will depend on the situation.
We can also add or subtract column or row vectors from arrays. Again, both the basic operators and the built in functions will perform the same operations. Unlike in Matlab, where this operation will raise an error, Python will automatically execute it row by row or column by column as appropriate:
Can you write Python code for it?


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
5. Slicing
Particular elements of numpy arrays (both unidimensional and multidimensional) can be accessed using standard python slicing. When working with multidimensional arrays, slicing can be carried out along the different dimensions at once
Can you write Python code for it?

# Keep last row of matrix X
# X_sub1 = <FILL IN>
X_sub1 = X[-1,]

# Keep first column of the three first rows of X
# X_sub2 = <FILL IN>
X_sub2 = X[:3,0]

# Keep first two columns of the three first rows of X
# X_sub3 = <FILL IN>
X_sub3 = X[:3,:2]

# Invert the order of the rows of X
# X_sub4 = <FILL IN>
X_sub4 = X[::-1,:]

test_hashedequal(X_sub1.tostring(),'51fb613567c9ef5fc33e7190c60ff37e0cd56706','Wrong value for variable X_sub1')
test_hashedequal(X_sub2.tostring(),'12a72e95677fc01de6b7bfb7f62d772d0bdb5b87','Wrong value for variable X_sub2')
test_hashedequal(X_sub3.tostring(),'f45247c6c31f9bcccfcb2a8dec9d288ea41e6acc','Wrong value for variable X_sub3')
test_hashedequal(X_sub4.tostring(),'1fd985c087ba518c6d040799e49a967e4b1d433a','Wrong value for variable X_sub4')


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
6. Matrix inversion
Non singular matrices can be inverted with method np.linalg.inv(). Invert square matrices $X\cdot X^\top$ and $X^\top \cdot X$, and see what happens when trying to invert a singular matrix. The rank of a matrix can be studied with method numpy.linalg.matrix_rank().
Can you write Python code for it?

print(X.shape)
print(X.dot(X.T))
print(X.T.dot(X))

print(np.linalg.inv(X.T.dot(X)))
#print np.linalg.inv(X.dot(X.T))


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
7. Exercises
In this section, you will complete three exercises where you will carry out some common operations when working with data structures. For this exercise you will work with the 2-D numpy array X, assuming that it contains the values of two different variables for 8 data patterns. A first column of ones has already been introduced in a previous exercise:
$${\bf X} = \left[ \begin{array}{ccc} 1 & x_1^{(1)} & x_2^{(1)} \ 1 & x_1^{(2)} & x_2^{(2)} \ \vdots & \vdots & \vdots \ 1 & x_1^{(8)} & x_2^{(8)}\end{array}\right]$$
First of all, let us check that you are working with the right matrix
Can you write Python code for it?

test_hashedequal(X.tostring(),'bdf94b49c2b7c6ae71a916beb647236918ead39f','Wrong value for variable X')


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
7.1. Non-linear transformations
Create a new matrix Z, where additional features are created by carrying out the following non-linear transformations:
$${\bf Z} = \left[ \begin{array}{ccc} 1 & x_1^{(1)} & x_2^{(1)} & \log\left(x_1^{(1)}\right) & \log\left(x_2^{(1)}\right)\ 1 & x_1^{(2)} & x_2^{(2)} & \log\left(x_1^{(2)}\right) & \log\left(x_2^{(2)}\right) \ \vdots & \vdots & \vdots \ 1 & x_1^{(8)} & x_2^{(8)} & \log\left(x_1^{(8)}\right) & \log\left(x_2^{(8)}\right)\end{array}\right] = \left[ \begin{array}{ccc} 1 & z_1^{(1)} & z_2^{(1)} & z_3^{(1)} & z_4^{(1)}\ 1 & z_1^{(2)} & z_2^{(2)} & z_3^{(1)} & z_4^{(1)} \ \vdots & \vdots & \vdots \ 1 & z_1^{(8)} & z_2^{(8)} & z_3^{(1)} & z_4^{(1)} \end{array}\right]$$
In other words, we are calculating the logarightmic values of the two original variables. From now on, any function involving linear transformations of the variables in Z, will be in fact a non-linear function of the original variables.
Can you write Python code for it?

# Obtain matrix Z using concatenation functions
# Z = np.hstack(<FILL IN>)
Z = np.hstack((X,np.log(X[:,1:])))

test_hashedequal(Z.tostring(),'737dee4c168c5ce8fc53a5ec5cad43b5a53c7656','Incorrect matrix Z')


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
7.3. Model evaluation
Finally, we can use previous data matrices Z and Z_poly to efficiently compute the output of the corresponding non-linear models over all the patterns in the data set. In this exercise, we consider the two following linear-in-the-parameters models to be evaluated:
$$f_\text{log}({\bf x}) = w_0 + w_1 \cdot x_1 + w_2 \cdot x_2 + w_3 \cdot \log(x_1) + w_4 \cdot \log(x_2)$$
$$f_\text{poly}({\bf x}) = w_0 + w_1 \cdot x_1 + w_2 \cdot x_1^2 + w_3 \cdot x_1^3$$
Compute the output of the two models for the particular weights that are defined in the code below. Your output variables f_log and f_poly should contain the outputs of the model for all eight patterns in the data set.
Note that for this task, you just need to implement appropriate matricial products among the extended data matrices, Z and Z_poly, and the provided weight vectors.
Can you write Python code for it?

w_log = np.array([3.3, 0.5, -2.4, 3.7, -2.9])
w_poly = np.array([3.2, 4.5, -3.2, 0.7])

# f_log = <FILL IN>
f_log = Z.dot(w_log)
# f_poly = <FILL IN>
f_poly = Z_poly.dot(w_poly)

test_hashedequal(f_log.tostring(),'d5801dfbd603f6db7010b9ef80fa48e351c0b38b','Incorrect evaluation of the logarithmic model')
test_hashedequal(f_poly.tostring(),'32abdcc0e32e76500947d0691cfa9917113d7019','Incorrect evaluation of the polynomial model')


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
If you wish that the extracted row or column is still a 2-D row or column vector, it is important to specify an interval instead of a single value, even if such interval consists of just one value.
Many numpy functions will also return 1-D vectors. It is important to be aware of such behavior to avoid and detect bugs in your code that may give place to undesired behaviors.
Can you write Python code for it?

X_col2 = X[:,1:2]
X_row3 = X[2:3,]

print('Second column of matrix X:', X_col2, '; Dimensions:', X_col2.shape)
print('Third row of matrix X:', X_row3, '; Dimensions:', X_row3.shape)


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
Other numpy methods where you can specify the axis along with a certain operation should be carried out are:

np.median()
np.std()
np.var()
np.percentile()
np.sort()
np.argsort()

If the axis argument is not provided, the array is flattened before carriying out the corresponding operation.
4. Concatenating matrices and vectors
Provided that the necessary dimensions fit, horizontal and vertical stacking of matrices can be carried out with methods np.hstack() and np.vstack().
Complete the following exercises to practice with matrix concatenation:
Can you write Python code for it?

# Previous check that you are working with the right matrices
test_hashedequal(z_4_2.tostring(),'0727ed01af0aa4175316d3916fd1c8fe2eb98f27','Incorrect result for variable z_4_2')
test_hashedequal(x_array.tostring(), '1215ced5d82501bf03e04b30f16c45a4bdcb8838', 'Incorrect variable x_array')

# Vertically stack matrix z_4_2 with itself
# ex1_res = <FILL IN>
ex1_res = np.vstack((z_4_2,z_4_2))

# Horizontally stack matrix z_4_2 and vector x_array
# ex2_res = <FILL IN>
ex2_res = np.hstack((z_4_2,x_array))

# Horizontally stack a column vector of ones with the result of the first exercise (variable ex1_res)
# X = <FILL IN>
X = np.hstack((np.ones((8,1)),ex1_res))

test_hashedequal(ex1_res.tostring(),'e740ea91c885cdae95499eaf53ec6f1429943d9c','Wrong value for variable ex1_res')
test_hashedequal(ex2_res.tostring(),'d5f18a630b2380fcae912f449b2a87766528e0f2','Wrong value for variable ex2_res')
test_hashedequal(X.tostring(),'bdf94b49c2b7c6ae71a916beb647236918ead39f','Wrong value for variable X')


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
Repeat the previous exercise once more. This time, define a lambda function for the task.
Can you write Python code for it?

# Z_lambda = np.array(list(map(lambda x: <FILL IN>,X)))
Z_lambda = np.array(list(map(lambda x: np.hstack((x,np.log(x[1]),np.log(x[2]))),X)))

test_hashedequal(Z_lambda.tostring(),'737dee4c168c5ce8fc53a5ec5cad43b5a53c7656','Incorrect matrix Z')


I am working on the file "P2.Numpy/old/numpy_professor.ipynb".
The first task is:
This notebook reviews some of the Python modules that make it possible to work with data structures in an easy an efficient manner. We will review Numpy arrays and matrices, and some of the common operations which are needed when working with these data structures in Machine Learning.
1. Create numpy arrays of different types
The following code fragment defines variable x as a list of 4 integers, you can check that by printing the type of any element of x. Use python command map() to create a new list with the same elements as x, but where each element of the list is a float. Note that, since in Python 3 map() returns an iterable object, you need to call function list() to populate the list.
Can you write Python code for it?

x = [5, 4, 3, 4]
print(type(x[0]))

# Create a list of floats containing the same elements as in x
# x_f = list(map(<FILL IN>))
x_f = list(map(float, x))

test_arrayequal(x, x_f, 'Elements of both lists are not the same')
if ((type(x[-2])==int) & (type(x_f[-2])==float)):
    print('Test passed')
else:
    print('Type conversion incorrect')


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
5. Slicing
Particular elements of numpy arrays (both unidimensional and multidimensional) can be accessed using standard python slicing. When working with multidimensional arrays, slicing can be carried out along the different dimensions at once
Can you write Python code for it?

# Keep last row of matrix X
# X_sub1 = <FILL IN>

# Keep first column of the three first rows of X
# X_sub2 = <FILL IN>

# Keep first two columns of the three first rows of X
# X_sub3 = <FILL IN>

# Invert the order of the rows of X
# X_sub4 = <FILL IN>

test_hashedequal(X_sub1.tostring(),'51fb613567c9ef5fc33e7190c60ff37e0cd56706','Wrong value for variable X_sub1')
test_hashedequal(X_sub2.tostring(),'12a72e95677fc01de6b7bfb7f62d772d0bdb5b87','Wrong value for variable X_sub2')
test_hashedequal(X_sub3.tostring(),'f45247c6c31f9bcccfcb2a8dec9d288ea41e6acc','Wrong value for variable X_sub3')
test_hashedequal(X_sub4.tostring(),'1fd985c087ba518c6d040799e49a967e4b1d433a','Wrong value for variable X_sub4')


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
7. Exercises
In this section, you will complete three exercises where you will carry out some common operations when working with data structures. For this exercise you will work with the 2-D numpy array X, assuming that it contains the values of two different variables for 8 data patterns. A first column of ones has already been introduced in a previous exercise:
$${\bf X} = \left[ \begin{array}{ccc} 1 & x_1^{(1)} & x_2^{(1)} \ 1 & x_1^{(2)} & x_2^{(2)} \ \vdots & \vdots & \vdots \ 1 & x_1^{(8)} & x_2^{(8)}\end{array}\right]$$
First of all, let us check that you are working with the right matrix
Can you write Python code for it?


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
7.1. Non-linear transformations
Create a new matrix Z, where additional features are created by carrying out the following non-linear transformations:
$${\bf Z} = \left[ \begin{array}{ccc} 1 & x_1^{(1)} & x_2^{(1)} & \log\left(x_1^{(1)}\right) & \log\left(x_2^{(1)}\right)\ 1 & x_1^{(2)} & x_2^{(2)} & \log\left(x_1^{(2)}\right) & \log\left(x_2^{(2)}\right) \ \vdots & \vdots & \vdots \ 1 & x_1^{(8)} & x_2^{(8)} & \log\left(x_1^{(8)}\right) & \log\left(x_2^{(8)}\right)\end{array}\right] = \left[ \begin{array}{ccc} 1 & z_1^{(1)} & z_2^{(1)} & z_3^{(1)} & z_4^{(1)}\ 1 & z_1^{(2)} & z_2^{(2)} & z_3^{(1)} & z_4^{(1)} \ \vdots & \vdots & \vdots \ 1 & z_1^{(8)} & z_2^{(8)} & z_3^{(1)} & z_4^{(1)} \end{array}\right]$$
In other words, we are calculating the logarightmic values of the two original variables. From now on, any function involving linear transformations of the variables in Z, will be in fact a non-linear function of the original variables.
Can you write Python code for it?

# Obtain matrix Z using concatenation functions
# Z = np.hstack(<FILL IN>)

test_hashedequal(Z.tostring(),'737dee4c168c5ce8fc53a5ec5cad43b5a53c7656','Incorrect matrix Z')


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
7.2. Polynomial transformations
Similarly to the previous exercise, now we are interested in obtaining another matrix that will be used to evaluate a polynomial model. In order to do so, compute matrix Z_poly as follows:
$$Z_\text{poly} = \left[ \begin{array}{cccc} 1 & x_1^{(1)} & (x_1^{(1)})^2 & (x_1^{(1)})^3 \ 1 & x_1^{(2)} & (x_1^{(2)})^2 & (x_1^{(2)})^3  \ \vdots & \vdots & \vdots \ 1 & x_1^{(8)} & (x_1^{(8)})^2 & (x_1^{(8)})^3 \end{array}\right]$$
Note that, in this case, only the first variable of each pattern is used.
Can you write Python code for it?

# Calculate variable Z_poly, using any method that you want
# Z_poly = <FILL IN>

test_hashedequal(Z_poly.tostring(),'7e025512fcee1c1db317a1a30f01a0d4b5e46e67','Wrong variable Z_poly')


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
Extracting columns and rows from multidimensional arrays
Something to be aware of when extracting rows or columns from numpy arrays is that if you specify just the index of the row or column you want to extract, the result will be a 1-D numpy array in any case. For instance, the following code prints the second column and third row of the numpy array X, and shows its dimensions. Notice that in both cases you get arrays with 1 dimension only.
Can you write Python code for it?

X_col2 = X[:,1]
X_row3 = X[2,]

print('Matrix X is\n', X)
print('Second column of matrix X:', X_col2, '; Dimensions:', X_col2.shape)
print('Third row of matrix X:', X_row3, '; Dimensions:', X_row3.shape)


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
If you wish that the extracted row or column is still a 2-D row or column vector, it is important to specify an interval instead of a single value, even if such interval consists of just one value.
Many numpy functions will also return 1-D vectors. It is important to be aware of such behavior to avoid and detect bugs in your code that may give place to undesired behaviors.
Can you write Python code for it?


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
Numpy arrays can be defined directly using methods such as np.arange(), np.ones(), np.zeros(), as well as random number generators. Alternatively, you can easily generate them from python lists (or lists of lists) containing elements of numeric type.
You can easily check the shape of any numpy vector with the property .shape, and reshape it with the method reshape(). Note the difference between 1-D and N-D numpy arrays (ndarrays). You should also be aware of the existence of another numpy data type: Numpy matrices (http://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.matrix.html) are inherently 2-D structures where operators * and ** have the meaning of matrix multiplication and matrix power.
In the code below, you can check the types and shapes of different numpy arrays. Complete also the exercise where you are asked to convert a unidimensional array into a vector of size $4\times2$.
Can you write Python code for it?

# Numpy arrays can be created from numeric lists or using different numpy methods
y = np.arange(8)+1
x = np.array(x_f)

# Check the different data types involved
print('Variable x_f is of type', type(x_f))
print('Variable x is of type ', type(x))
print('Variable y is of type', type(y))

# Print the shapes of the numpy arrays
print('Variable y has dimension', y.shape)
print('Variable x has dimension', x.shape)

#Complete the following exercises
# Convert x into a variable x_matrix, of type `numpy.matrixlib.defmatrix.matrix` using command
# np.matrix(). The resulting matrix should be of dimensions 4x1
# x_matrix = <FILL IN>

# Convert x into a variable x_array, of type `ndarray`, and shape (4,1)
# x_array = <FILL IN>

# Reshape array y into a numpy array of shape (4,2) using command np.reshape()
# y = <FILL IN>

test_strequal(str(type(x_matrix)), "<class 'numpy.matrixlib.defmatrix.matrix'>", 'x_matrix is not defined as a matrix')
test_hashedequal(x_matrix.tostring(), '1215ced5d82501bf03e04b30f16c45a4bdcb8838', 'Incorrect variable x_matrix')
test_strequal(str(type(x_array)), "<class 'numpy.ndarray'>", 'x_array is not defined as numpy ndarray')
test_hashedequal(x_array.tostring(), '1215ced5d82501bf03e04b30f16c45a4bdcb8838', 'Incorrect variable x_array')
test_strequal(str(type(y)), "<class 'numpy.ndarray'>", 'y is not defined as a numpy ndarray')
test_hashedequal(y.tostring(), '0b61a85386775357e0710800497771a34fdc8ae5', 'Incorrect variable y')


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
Some other useful Numpy methods are:

np.flatten(): converts a numpy array or matrix into a vector by concatenating the elements in the different dimension. Note that the result of the method keeps the type of the original variable, so the result is a 1-D ndarray when invoked on a numpy array, and a numpy matrix (and necessarily 2-D) when invoked on a matrix.
np.tolist(): converts a numpy array or matrix into a python list.

These uses are illustrated in the code fragment below.
Can you write Python code for it?

print('Applying flatten() to matrix x_matrix (of type matrix)')
print('x_matrix.flatten():', x_matrix.flatten())
print('Its type:', type(x_matrix.flatten()))
print('Its dimensions:', x_matrix.flatten().shape)

print('\nApplying flatten() to matrix y (of type ndarray)')
print('y.flatten():', y.flatten())
print('Its type:', type(y.flatten()))
print('Its dimensions:', y.flatten().shape)

print('\nApplying tolist() to x_matrix (of type matrix) and to the 2D vector y (of type ndarray)')
print('x_matrix.tolist():', x_matrix.tolist())
print('y.tolist():', y.tolist())


I am working on the file "P2.Numpy/old/numpy_student.ipynb".
The first task is:
This notebook reviews some of the Python modules that make it possible to work with data structures in an easy an efficient manner. We will review Numpy arrays and matrices, and some of the common operations which are needed when working with these data structures in Machine Learning.
1. Create numpy arrays of different types
The following code fragment defines variable x as a list of 4 integers, you can check that by printing the type of any element of x. Use python command map() to create a new list with the same elements as x, but where each element of the list is a float. Note that, since in Python 3 map() returns an iterable object, you need to call function list() to populate the list.
Can you write Python code for it?

x = [5, 4, 3, 4]
print(type(x[0]))

# Create a list of floats containing the same elements as in x
# x_f = list(map(<FILL IN>))

test_arrayequal(x, x_f, 'Elements of both lists are not the same')
if ((type(x[-2])==int) & (type(x_f[-2])==float)):
    print('Test passed')
else:
    print('Type conversion incorrect')


I am working on the file "P2/P2.ipynb".
The first task is:
<hr>
Data Exploration
Visualisation
We first use the pointplot to explore the effect of "Sex", "Pclass", "SibSp", and "Parch" on survival. The pointplot is great for categorical variable and numerical variable with small countable range. It provides the mean estimate and confidence interval for each possible value of the variable.
For age, we will compare the kdeplot for both who survived and those didn't survive. If the kde are similar, then it indicates the age have little correlation with the chance of survival. If one kde value for survival is higher than the other at a particular age range, then it suggests the chance of survival is higher for than age group.
Can you write Python code for it?

# using seaborn's pointplot to visually explore the relationship between individual categorical varibles
sns.set_style("whitegrid")
g = sns.PairGrid(titanic, x_vars=["Sex", "Pclass", "SibSp", "Parch"], y_vars=["Survived"], size=4)
g.map(sns.pointplot, ci=99)
g.axes[0,0].set_ylabel("survival rate")
g.fig.suptitle("Point Plots")


I am working on the file "P2/P2.ipynb".
The first task is:
Name and Embarked are dropped from the dataset because passenger name and embarking location shouldn't have any meaningful correlation with their chance of surviving. Arguably, the embarking locations might give some indication of passengers' social-economical background. However, I will be in favor of using "Pclass",  because it is specifically mentioned in the special notes in the data souce to be a proxy for social-economic status.
I dropped Fare in favor of Pclass for the same reason.
Cabin is an interesting feature. It is plausible the exact location of the passenger when the accident happens have an significant impact on the chance of survival. According to Wikipedia, the Titanic hits an iceburg at 11:40 pm. Given its already late in night, it is likely people will be sleeping at their cabin. However, there is only 204 data entries has a non-NaN value for this feature. Cabin is hence removed from the dataset as well.
Ticket number seems to be fairly random and not consistent across the dataset. It doesn't seems to contain any useful information. I have dropped from our dataset as well.
Other features like social-economic class (Pclass), gender (Sex), Age, and number of families onboard (Parch and SibSp) are all reasonable factors that could influence the chance of survival.
<hr />
Data Audit
Can you write Python code for it?

# helpers ----------------------------
def value_in_range(series, min, max):
    assert pd.to_numeric(series, errors="raise").all() # make sure those are numbers
    return min <= series.min() and series.max() <= max
# ------------------------------------

# Sex can either be male or female
assert titanic["Sex"].isin(["male", "female"]).all() == True

# Survived
assert titanic["Survived"].isin([0,1]).all() == True

# Pclass should be either 1, 2, or 3 and
assert titanic["Pclass"].isin([1, 2, 3]).all() == True

# Age should be sensible, say between 0 to 100
AgeSeries = titanic[titanic["Age"].isnull() == False]["Age"]
assert value_in_range(AgeSeries, 0, 100) == True


I am working on the file "P2/P2.ipynb".
The first task is:
Sex

The deep slope and small verticle bars (representing 99% confidence interval of mean) in the first visualisation suggests that "Sex" is strongly correlated with the chance of survival. Females have much higher chance of survival than male.

Social class (Pclass)

Similar to "Sex", the second visualisation shows people with higher social class are more likely to survive the disaster.

# Family members onboard (sibling, spouse, parents and children)

Overall, the pattern seems to be people with a few family member aboard has the highest chance of survival. Passengers with no family member or a lot of family aboard have a lower chance of survival.
The confidence intervals are relatively large at higher, suggesting the explanation power of those two various might be relatively weak.
Can you write Python code for it?

# ploting the kernal distribution for age
figure = plt.figure()

ax_top = figure.add_subplot(211)
ax_top.set_xlim(0,85)
ax_top.set_xlabel("Age")
ax_top.set_ylabel("Proportional of Population")
ax_top.set_title("Kernal Density Estimate for Age grouped by survival")

ax_bottom = figure.add_subplot(212)
ax_bottom.set_xlim(0,85)
ax_bottom.set_title("Boxplot for Age distribution grouped by survival")

x = titanic[titanic["Survived"] == 1]
y = titanic[titanic["Survived"] == 0]

_ = sns.kdeplot(x["Age"].dropna(),
                label="survived == True",
                cut= True, shade=True,
                ax=ax_top)

_ = sns.kdeplot(y["Age"].dropna(),
                label="survived == False",
                cut=True, shade=True,
                ax=ax_top)

_ = sns.boxplot(x="Age",
                y="Survived",
                data=titanic.dropna(subset = ["Age"]),
                orient="h",
                ax=ax_bottom)

plt.tight_layout()


I am working on the file "P2/UCDAND-P2.ipynb".
The first task is:
2. Question 1: How does sex effect passenger class?
2.1 Data Wrangling
Data loaded from trian.csv.
Can you write Python code for it?

# Open the csv and load into pandas dataframe

df = pd.read_csv('train.csv')


I am working on the file "P2/UCDAND-P2.ipynb".
The first task is:
3rd Class makes up the majority of the passengers. There are a similar number of 1st and 2nd class passengers.
It might be interesting to compare this with passenger data from other ships in that era, or ships/trains/planes today.
2.3 2D Investigation
Investigate the relationship between passenger class and sex.
2.3.1 Class Vs Sex on the Titanic
For this we can break the dataset into 2 groups (male and female) then look at how the makeup of the class is.
Can you write Python code for it?

# Group passenegers into male and female, and then group by class and count the number of passengers in the groups
femaleVsClass = df[df['Sex'] == 'female'].groupby(['Pclass'])['Pclass'].count()
maleVsClass = df[df['Sex'] == 'male'].groupby(['Pclass'])['Pclass'].count()

# Combine the male and female results (for better graphing)
sexVsClass = pd.concat([femaleVsClass, maleVsClass], axis=1, keys=['females','males'])

#Plot the results
sexVsClass.plot.pie(subplots=True,
                    figsize=(16, 8),
                    autopct='%.2f%%',
                    title='Passengers on Titanic Class Distribution for Males and Females',
                    labels = ['First Class', 'Second Class', 'Third Class'],
                    legend=None,
                    colors=classColors)


I am working on the file "P2/UCDAND-P2.ipynb".
The first task is:
For readability, add a column in the dataframe call 'Lived or Died' which has a string representation of whether a passenger survived.
Can you write Python code for it?

# Add a column called 'Live or Died' with a string representation of the Survivedd column (which is 0 or 1)
d = {0: 'Died', 1: 'Lived'}
df['Lived or Died'] = df['Survived'].map(d)


I am working on the file "P2/UCDAND-P2.ipynb".
The first task is:
It appears some rows with passenger age is missing. For this investigation, rows with missing age information will be discarded.
Can you write Python code for it?

# Remove rows with a null age (blank age) from the dataframe
df = df[pd.notnull(df['Age'])]

nullAge = df[pd.isnull(df['Age'])]['PassengerId'].count()
totalRows = df['PassengerId'].count()

print "Rows with no Age: %d\nTotal: %d" % (nullAge,totalRows)


I am working on the file "P2/UCDAND-P2.ipynb".
The first task is:
Now that the dataset is loaded, check if any rows contain bad data for the variables we are looking at.

Passenger sex
Passenger class
If the passenger has siblings or spouses (this comes later)
Can you write Python code for it?

#Use the pandas.isnull function to find any missing data
nullSex = df[pd.isnull(df['Sex'])]['PassengerId'].count()
nullClass = df[pd.isnull(df['Pclass'])]['PassengerId'].count()
nullSibSp = df[pd.isnull(df['SibSp'])]['PassengerId'].count()

print "Rows with no sex: %d\nRows with no pClass: %d\nRows with no SibSp: %d" % (nullSex,nullClass,nullSibSp)


I am working on the file "P2/UCDAND-P2.ipynb".
The first task is:
There are more males then females. The distribution looks pretty close with the average age of males is slightly higher, and there are a higher % of very young females then males.
3.3 2D Investigation
3.3.1 Passenger Age vs Survival
Once again we will took at the whole population and then separate into males and females.
Use logistic regression to estimate the survival chances of a person at a certain age.
Can you write Python code for it?

#Use seaborn to create graphs that use logistic regression to predict the survival % at different ages
#Show both population as a whole and split up males and females.
sns.set_context("notebook", font_scale=3)
sns.set_style("darkgrid")

g = sns.lmplot(x='Age',
               y='Survived',
               data=df,
               y_jitter=.02,
               logistic=True,
               size=6,
               aspect=4)

g.set(xlim=(0,80),title='Survival Rate of All Passengers Using Logistic Regression')

g = sns.lmplot(x='Age',
               y='Survived',
               hue="Sex",
               data=df,
               y_jitter=.02,
               logistic=True,
               size=6,
               aspect=4,)

g.set(xlim=(0,80),title='Survival Rate of Passengers, Seperated by Sex, Using Logistic Regression')


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
50 childs were into the titanic without any parents.
Can you write Python code for it?


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
First Look at Data & Questions

What factors made people more likely to survive?
Is the average age of survivors significantly different from the average age of non-survivors?
Is there a positive correlation between the fare you paid and the likelihood to survive?
Is there a positive correlation between a higher social economic status and the fare?
Is there a significant difference over the social/fare correlation between people coming from the 3 ports?
Is there any passenger whose age is under 20 who came alone on the titanic?
Is there someone who payed way more than the average to come on the titanic?

What factors made people more likely to survive?
Can you write Python code for it?

# Likelihood to survive if you are a male / female
def prob_to_survive(x,y):
    varx = titanic[titanic[x] == y]
    return varx[varx['Survived'] == 1].count() / varx.count()
print prob_to_survive('Sex','male')[0]
print prob_to_survive('Sex','female')[0]


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
H0: μs = μns - The average age of survivors is not significantly different from the average age of non-survivors.<br>
HA: μns < μs - The average age of survivors is significantly lower from the average age of non-survivors.
I will perform a independent-samples one tail t-test. I choose this test because:
- I’m working with two independent samples of data where independent variable is the survival of the individual.
- Because we don't have the age of every individual on the boat, population parameters like the population standard deviation are unknown.
- we assume that the two samples are selected from a normal population.
I’ll use a critical statistic value of 0.05%.
Can you write Python code for it?

print titanic_survivors['Age'].mean()
print titanic_non_survivors['Age'].mean()

t = (titanic_survivors['Age'].mean() - titanic_non_survivors['Age'].mean()) / m.sqrt((titanic_survivors['Age'].var() / titanic_survivors['Age'].count()) +  (titanic_non_survivors['Age'].var() / titanic_non_survivors['Age'].count()))
t


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
The violin plot chart shows us that the survived violin is larger on the bottom and on the top. It's hard to conclude / draw a strong hypothesis looking at this chart. But because I think childs have been prioritized in boats, my hypothesis will be that the average age of survivors is lower than the average age of non-survivors.
Can you write Python code for it?

#Data wrangling - creation and cleaning of my two samples
titanic_survivors = titanic[titanic['Survived'] == 1]
titanic_non_survivors = titanic[titanic['Survived'] == 0]
titanic_survivors = titanic_survivors[['Survived','Age']].dropna()
titanic_non_survivors = titanic_non_survivors[['Survived','Age']].dropna()

print titanic_survivors['Age'].count()
print titanic_non_survivors['Age'].count()


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
There is a weak correlation between the fare you paid and your likelihood to survive.
Is there a positive correlation between a higher social economic status and the fare?
Can you write Python code for it?

titanic_fare_norm
titanic_status_norm = (titanic['Pclass'] - titanic['Pclass'].mean()) / (titanic['Pclass'].std(ddof=0))
print pearsonr(titanic_status_norm,titanic_fare_norm)


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
Your likelihood to survive was way higher if you were a women.
Can you write Python code for it?

# Likelihood to survive according to your Pclass
print prob_to_survive('Pclass',1)[0]
print prob_to_survive('Pclass',2)[0]
print prob_to_survive('Pclass',3)[0]


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
Your likelihood to survive was way higher if you were coming from a higher socio-economic status.
Can you write Python code for it?

# Likelihood to survive if you were in family or not
titanic['inFamily'] = np.where(titanic['Parch'] > 0, 'Yes','No')
print prob_to_survive('inFamily','Yes')[0]
print prob_to_survive('inFamily','No')[0]


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
r2 indicates that even if the age of survivors is significantly lower than the age of non-survivors, that variable does not explained the variability of our independent variable, the survival of individuals in the ship.
We might then make the hypothesis that the sex was the variable having the higher impact on the survival of the individual, women might have been prioritized to take lifeboats, and women might be younger than man in the boat and might have take their child with them.
Is there a positive correlation between the fare you paid and the likelihood to survive?
Can you write Python code for it?

# Likelihood to survive according to your fare

titanic_fare_norm = (titanic['Fare'] - titanic['Fare'].mean()) / (titanic['Fare'].std(ddof=0))
titanic_survived_norm = (titanic['Survived'] - titanic['Survived'].mean()) / (titanic['Survived'].std(ddof=0))
print pearsonr(titanic_fare_norm,titanic_survived_norm)


I am working on the file "P2/titanic_nicolas_hemidy.ipynb".
The first task is:
t(712) = -2.0460, p<.05, one-tailed <br>
t-critical value = -1.646 with 712 degree of freedom.
Based on this t test, we can reject the null hypothesis and conclude that age of survivors is significantly lower than the age of non-survivors.
HA: μns < μs
Results match my expectations. Let's calculate r2 to see how much the age influenced the survival of an individual.
Can you write Python code for it?


I am working on the file "P22 Activity Recognition.ipynb".
The first task is:
Classification
The next step is the classification of the labels with the extracted features. Therefore the team used 2 different classifiers (Random Forest - as the main classifier build on the idea of the research paper by Pierluigi et al. {Pierluigi Casale, Oriol Pujol, and Petia Radeva. Human activity recognition from accelerometer data using a wearable device. Pattern Recognition and Image Analysis, pages 289–296, 2011} - and Support Vector machines as a comparison to the main classifier
Can you write Python code for it?

#function to classify without crossvalidaion:
#we created here funtion to predict with Random forest and SVM the values with a simple division of the dataset.
#we split infact the dataset in 2 part:Training and test set and we provide the follow estimators:
#1)F1score
#2)Accuracy
#3)Confusion Matrix

def classify(x_features, y_features):
    X_train, X_test, y_train, y_test = train_test_split(x_features, y_features.ravel(), test_size=0.2, random_state=0)
    X_train.shape, y_train.shape
    X_test.shape, y_test.shape
    forest= RandomForestClassifier(n_estimators=100, random_state=0)
    clf = svm.SVC(kernel='linear', C=1)
    model = forest.fit(X_train,y_train )
    modelSv=clf.fit(X_train,y_train )
    predicted_labels = model.predict(X_test)
    predicted_labelsSv = modelSv.predict(X_test)

    # Compute the F1 score, also known as balanced F-score or F-measure
    # The F1 score can be interpreted as a weighted average of the precision and recall,
    #  where an F1 score reaches its best value at 1 and worst score at 0.
    #  The relative contribution of precision and recall to the F1 score are equal.
    #  The formula for the F1 score is:
    # F1 = 2 * (precision * recall) / (precision + recall)
    # In the multi-class and multi-label case, this is the weighted average of the F1 score of each class.

    print(" F1 score Random forest: %f" % f1_score(y_test, predicted_labels, average='macro'))
    print(" F1 score precision with SV: %f" % f1_score(y_test, predicted_labels, average='macro'))

    print("Accuracy Random forest: %f" % metrics.accuracy_score(y_test, predicted_labels))
    print("Accuracy with SV: %f" % metrics.accuracy_score(y_test, predicted_labelsSv))
    #By definition a confusion matrix C is such that C_{i, j}
    # is equal to the number of observations known to be in group i but predicted to be in group j.
    print("Confusion Matrix Random forest: ")
    print(confusion_matrix(y_test, predicted_labels, labels=[1, 2, 3, 4, 5, 6, 7]))
    print("Confusion Matrix with SVM: ")
   ...


I am working on the file "P22 Activity Recognition.ipynb".
The first task is:
Code execution
Can you write Python code for it?

# Importing data
dataframe = read(1)
#print(dataframe)
# Delete incorrect Data
cleaned_data = zeroDet(dataframe, 0)

#Check how balanced data is
counts = count_labels(cleaned_data)
print("Instances of every label, starting by one to seven",counts)

#Feature Extraction
grouped_data = grouping(cleaned_data)
features = extract_features(grouped_data)

#we are calling the function for classify our data, first with a simple
#splitting of the dataset to divide test and training set, after
#using k-crossvalidation with two different train model: 1)RandomForest classifier , 2)Super Vector Machine
classify(features.__getitem__(0),features.__getitem__(1))
CrossValidation(features.__getitem__(0),features.__getitem__(1), 5)


I am working on the file "P22 Activity Recognition.ipynb".
The first task is:
Feature Extraction
2. Step we want to extract two feature types for each window (6 different features for each window - x-, y-, z- axis)
Can you write Python code for it?

#Feature Extraction

# For Feature Extraction we use a technique called window overlapping (Pierluigi Casale, Oriol Pujol, and Petia Radeva. Human activity recognition from accelerometer
# data using a wearable device. Pattern Recognition and Image Analysis, pages 289–296, 2011). It has an overlap of 50% between the different
# time series. As a time window 1 second is use --> corresponds to 52 samplings (52 Hz frequency)
# Then we start with the sequencing
# Slicing needs to be done as follows:
# - it is not possible that 2 activities are grouped in one sequence (would falsify the outcome of the mean value)
# - therefore only labels with the same value are grouped into one sequence
# @params: array_data is list of array that contains the grouped data
# @output: data_list which contains numpy arrays with the respective windows

def grouping(array_data):
    start = int(0)
    end = int(52)
    data_list = []
    length = np.size(array_data, 0)
    while start < length-52:
        if(array_data[start][4] != array_data[end-1][4]):        # this control sequence is necessary to ensure that not two of the same
            while(array_data[start][4] != array_data[end-1][4]): # labels are in one window
                end = end -1
            newArray = array_data[slice(start, end)]
            start = end
            end = end + 52
        else:
            newArray = array_data[slice(start, end)]
            start = start + 26
            end = end + 26
        data_list.append(newArray)
        if(end-52 > length - 1):
            end = length-1

    # Comment in to show the size and length of the data_list array
    # print(np.size(data_list))
    # print(len(data_list))
    return data_list

# This is an additional function which could be called to print a data list to a text file (e.g to examine it)
# Comment in for printing the data to a text file
# def sysout_to_text(dataList):
#     file = open("tempFile", "w")
#     for item in dataList:
#         file.write("%s\n" % item)
#     file.close()

#Now we...


I am working on the file "P22 Activity Recognition.ipynb".
The first task is:
Importing of the dataset
Method definition for reading one of the available datasets
Can you write Python code for it?

# Specifying engine = python because c engine can not handle 'sep'
# @params: dataNum
# @output: dataset (Dataframe)
def read(data_num):
    dataset = pd.read_csv('data/%d.csv' % (data_num), sep=',', header=None, engine='python', names=names_attributes)

    # Comment in for printing out the array data and the size of the array
    # print(array_data)
    # print(np.size(array_data, 0))

    return dataset


I am working on the file "P2:_student_intervention/P2 Building a Student Intervention System.ipynb".
The first task is:
Further Graphical Comparision of Classifiers
Can you write Python code for it?

%matplotlib inline
from matplotlib import pylab
def plotClfCompare(clfs,devs=None,**kwargs):
    if kwargs is not None:
        wmap =  {"test_score":"F1 Score on Testing Set",
                "train_score":"F1 Score on Training Set",
                "predict_time":"Prediction time (secs)",
                "train_time":"Training time (secs)"}
        for metric in [wmap[key] for key,value in kwargs.items() if value]:
            for key,value in clfs.items():
                pylab.plot(value.index,value[metric],label=key)
                pylab.xlabel(value.index.name)
                pylab.ylabel(value[metric].name)
                pylab.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
                pylab.grid()
                title = "".join([value.index.name," vs ",value[metric].name])
                pylab.title(title)
                if devs:
                    pylab.fill_between(value.index, value[metric] - devs[key][metric],
                                       value[metric] + devs[key][metric], alpha=0.1,color="r")
            pylab.show()

cv = StratifiedShuffleSplit(y_all, n_iter=30, test_size=0.24, random_state=42)
cvlist = [MultinomialNB(),LogisticRegression(),SVC()]
clfs,devs = classifier_perf(cvlist,X_all,y_all,cv=cv)
plotClfCompare(clfs,devs,test_score =True,train_score=True,
               predict_time=True,train_time=True)