I am working on the file "P5.Data preprocessing/Intro5_DataNormalization_professor.ipynb".
The first task is:
1. Data preprocessing
1.1. The dataset.
A key component of any data processing method or any machine learning algorithm is the dataset, i.e., the set of data that will be the input to the method or algorithm.
The dataset collects information extracted from a population (of objects, entities, individuals,...). For instance, we can measure the weight and height of students from a class and collect this information in a dataset ${\cal S} = {{\bf x}k, k=0, \ldots, K-1}$ where $K$ is the number of students, and each sample is a 2 dimensional vector, ${\bf x}_k= (x{k0}, x_{k1})$, with the height  and the weight in the first and the second component, respectively. These components are usually called features. In other datasets, the number of features can be arbitrarily large.
1.1. Data preprocessing
The aim of data preprocessing methods is to transform the data into a form that is ready to apply machine learning algorithms. This may include:

Data normalization: transform the individual features to ensure a proper range of variation
Data imputation: assign values to features that may be missed for some data samples
Feature extraction: transform the original data to compute new features that are more appropiate for a specific prediction task
Dimensionality reduction: remove features that are not relevant for the prediction task.
Outlier removal: remove samples that may contain errors and are not reliable for the prediction task.
Clustering: partition the data into smaller subsets, that could be easier to process.

In this notebook we will focus on data normalization.
2. Data normalization
All samples in the dataset can be arranged by rows in a $K \times m$ data matrix ${\bf X}$, where $m$ is the number of features (i.e. the dimension of the vector space containing the data). Each one of the $m$ data features may represent variables of very different nature (e.g. time, distance, price, volume, pixel intensity,...). Thus, the scale and the range of variation of each feature can be completely different.
As an...
Can you write Python code for it?

from sklearn.datasets import make_blobs
X, y = make_blobs(n_samples=300, centers=4, random_state=0, cluster_std=0.60)
X = X @ np.array([[30, 4], [-8, 1]]) + np.array([90, 10])

plt.figure(figsize=(12, 3))
plt.scatter(X[:, 0], X[:, 1], s=50);
plt.axis('equal')
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
plt.show()


I am working on the file "P5.Data preprocessing/Intro5_DataNormalization_professor.ipynb".
The first task is:
2.1.1. Implementation in sklearn
The sklearn package contains a method to perform the standard scaling over a given data matrix.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X)
print(f'The sample mean is m = {scaler.mean_}')

T2 = scaler.transform(X)
plt.figure(figsize=(4, 4))
plt.scatter(T2[:, 0], T2[:, 1], s=50);
plt.axis('equal')
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
plt.show()


I am working on the file "P5.Data preprocessing/Intro5_DataNormalization_professor.ipynb".
The first task is:
Note that, once we have defined the scaler object in Python, you can apply the scaling transformation to other datasets. This will be useful in further topics, when the dataset may be split in several matrices and we may be interested in defining the transformation using some matrix, and apply it to others
2.2. Other normalizations.
The are some alternatives to the standard scaling that may be interesting for some datasets. Here we show some of them, available at the preprocessing module in sklearn:

preprocessing.MaxAbsScaler: Scale each feature by its maximum absolute value. As a result, all feature values will lie in the interval [-1, 1].
preprocessing.MinMaxScaler: Transform features by scaling each feature to a given range. Also, all feature values will lie in the specified interval.
preprocessing.Normalizer: Normalize samples individually to unit norm. That is, it applies the transformation ${\bf t}_k = \frac{1}{\|{\bf x}_k\|} {\bf x}_k$
preprocessing.PowerTransformer: Apply a power transform featurewise to make data more Gaussian-like.
preprocessing.QuantileTransformer: Transform features using quantile information. The transformed features follow a specific target distribution (uniform or normal).
preprocessing.RobustScaler: Scale features using statistics that are robust to outliers. This way, anomalous values in one or very few samples cannot have a strong influence in the normalization.

You can find more detailed explanation of these transformations sklearn documentation.
Exercise 2: Use sklearn to transform the data matrix X into a matrix T24such that the minimum feature value is 2 and the maximum is 4.
(Hint: select and import the appropriate preprocessing module from sklearn an follow the same steps used in the code cell above for the scandard scaler)
Can you write Python code for it?

# Write your solution here
# <SOL>
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(2, 4))
scaler.fit(X)
T24 = scaler.transform(X)
# </SOL>

# We can visually check that the transformed data features lie in the selected range.
plt.figure(figsize=(4, 4))
plt.scatter(T24[:, 0], T24[:, 1], s=50);
plt.axis('equal')
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
plt.show()


I am working on the file "P5.Data preprocessing/Intro5_DataNormalization_professor.ipynb".
The first task is:
We can see that the first data feature ($x_0$) has a much large range of variation than the second ($x_1$). In practice, this may be problematic: the convergence properties of some machine learning algorithms may depend critically on the feature distributions and, in general, features sets ranging over similar scales use to offer a better performance.
For this reason, transforming the data in order to get similar range of variations for all features is desirable. This can be done in several ways.
2.1. Standard scaling.
A common normalization method consists on applying an affine transformation
$$
{\bf t}_k = {\bf D}({\bf x}_k - {\bf m})
$$
where ${\bf D}$ is a diagonal matrix, in such a way that the transformed dataset ${\cal S}' = {{\bf t}_k, k=0, \ldots, K-1}$ has zero sample mean, i.e.,
$$
\frac{1}{K} \sum_{k=0}^{K-1} {\bf t}_k = 0
$$
and unit sample variance, i.e.,
$$
\frac{1}{K} \sum_{k=0}^{K-1} t_{ki}^2 = 1
$$
It is not difficult to verify that this can be done by taking ${\bf m}$ equal to the sample mean
$$
{\bf m} = \frac{1}{K} \sum_{k=0}^{K-1} {\bf x}_k
$$
and taking the diagonal components of ${\bf D}$ equal to the inverse of the standard deviation of each feature, i.e.,
$$
d_{ii} = \frac{1}{\sqrt{\frac{1}{K} \sum_{k=0}^{K-1} (x_{ki} - m_i)^2}}
$$
Using the data matrix ${\bf X}$ and the broadcasting property of the basic mathematical operators in Python, the implementation of this normalization is straightforward.
Exercise 1: Apply a standard scaling to the data matrix. To do so:

Compute the mean, and store it in variable m (you can use method mean from numpy)
Compute the standard deviation of each feature, and store the result in variable s (you can use method std from numpy)
Take advangate of the broadcasting property to normalize the data matrix in a single line of code. Save the result in variable T.
Can you write Python code for it?

# Compute the sample mean
# m = <FILL IN>
m = np.mean(X, axis=0)     # Compute the sample mean
print(f'The sample mean is m = {m}')

# Compute the standard deviation of each feature
# s = <FILL IN>
s = np.std(X, axis=0)      # Compute the standard deviation of each feature

# Normalize de data matrix
# T = <FILL IN>
T = (X-m)/s                # Normalize


I am working on the file "P5.Data preprocessing/Intro5_DataNormalization_student.ipynb".
The first task is:
1. Data preprocessing
1.1. The dataset.
A key component of any data processing method or any machine learning algorithm is the dataset, i.e., the set of data that will be the input to the method or algorithm.
The dataset collects information extracted from a population (of objects, entities, individuals,...). For instance, we can measure the weight and height of students from a class and collect this information in a dataset ${\cal S} = {{\bf x}k, k=0, \ldots, K-1}$ where $K$ is the number of students, and each sample is a 2 dimensional vector, ${\bf x}_k= (x{k0}, x_{k1})$, with the height  and the weight in the first and the second component, respectively. These components are usually called features. In other datasets, the number of features can be arbitrarily large.
1.1. Data preprocessing
The aim of data preprocessing methods is to transform the data into a form that is ready to apply machine learning algorithms. This may include:

Data normalization: transform the individual features to ensure a proper range of variation
Data imputation: assign values to features that may be missed for some data samples
Feature extraction: transform the original data to compute new features that are more appropiate for a specific prediction task
Dimensionality reduction: remove features that are not relevant for the prediction task.
Outlier removal: remove samples that may contain errors and are not reliable for the prediction task.
Clustering: partition the data into smaller subsets, that could be easier to process.

In this notebook we will focus on data normalization.
2. Data normalization
All samples in the dataset can be arranged by rows in a $K \times m$ data matrix ${\bf X}$, where $m$ is the number of features (i.e. the dimension of the vector space containing the data). Each one of the $m$ data features may represent variables of very different nature (e.g. time, distance, price, volume, pixel intensity,...). Thus, the scale and the range of variation of each feature can be completely different.
As an...
Can you write Python code for it?


I am working on the file "P5.Data preprocessing/Intro5_DataNormalization_student.ipynb".
The first task is:
2.1.1. Implementation in sklearn
The sklearn package contains a method to perform the standard scaling over a given data matrix.
Can you write Python code for it?


I am working on the file "P5.Data preprocessing/Intro5_DataNormalization_student.ipynb".
The first task is:
Note that, once we have defined the scaler object in Python, you can apply the scaling transformation to other datasets. This will be useful in further topics, when the dataset may be split in several matrices and we may be interested in defining the transformation using some matrix, and apply it to others
2.2. Other normalizations.
The are some alternatives to the standard scaling that may be interesting for some datasets. Here we show some of them, available at the preprocessing module in sklearn:

preprocessing.MaxAbsScaler: Scale each feature by its maximum absolute value. As a result, all feature values will lie in the interval [-1, 1].
preprocessing.MinMaxScaler: Transform features by scaling each feature to a given range. Also, all feature values will lie in the specified interval.
preprocessing.Normalizer: Normalize samples individually to unit norm. That is, it applies the transformation ${\bf t}_k = \frac{1}{\|{\bf x}_k\|} {\bf x}_k$
preprocessing.PowerTransformer: Apply a power transform featurewise to make data more Gaussian-like.
preprocessing.QuantileTransformer: Transform features using quantile information. The transformed features follow a specific target distribution (uniform or normal).
preprocessing.RobustScaler: Scale features using statistics that are robust to outliers. This way, anomalous values in one or very few samples cannot have a strong influence in the normalization.

You can find more detailed explanation of these transformations sklearn documentation.
Exercise 2: Use sklearn to transform the data matrix X into a matrix T24such that the minimum feature value is 2 and the maximum is 4.
(Hint: select and import the appropriate preprocessing module from sklearn an follow the same steps used in the code cell above for the scandard scaler)
Can you write Python code for it?

# Write your solution here
# <SOL>
# </SOL>

# We can visually check that the transformed data features lie in the selected range.
plt.figure(figsize=(4, 4))
plt.scatter(T24[:, 0], T24[:, 1], s=50);
plt.axis('equal')
plt.xlabel('$x_0$')
plt.ylabel('$x_1$')
plt.show()


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
Adding the rate fields seems to make a difference, so I will keep it.
Feature Conclusion
For now I will keep all the email data and add the positive feature. These can be tunned!
Can you write Python code for it?


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
Classifier
Working with the modified set of features, naive bayes, decision tree and random forest stand out.
Even though nb seems to perform slightly better then random forrest, random forrest allows for more 'tuneabilty' so I will stick with it.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
from sklearn.cross_validation import cross_val_score
from sklearn.cross_validation import KFold

labelIndex = feature_list[0]
featureIndexes = feature_list[1:]

labels = df[labelIndex].values[:].astype(bool)
features = df[featureIndexes].values[:,:].astype(np.float64)

clf = RandomForestClassifier(n_estimators=10)

from sklearn.cross_validation import StratifiedShuffleSplit
kf = StratifiedShuffleSplit(labels, 100, random_state = 42)

recallScore = cross_val_score(clf, features, labels, scoring='recall',cv=kf).mean()
perScore = cross_val_score(clf, features, labels, scoring='precision',cv=kf).mean()
accuracyScore = cross_val_score(clf, features, labels, scoring='accuracy',cv=kf).mean()

print "Acc: %f, Precision: %f, Recall: %f" % (accuracyScore,perScore,recallScore)


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
Clearly SVM is just guessing false for everyone. That is why it's accuracy is high but it's recall is 0.

Using 0's gives the best recall in nb and dt, so lets go with that.
Can you write Python code for it?


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
Feature Testing
Test out how adding and removing  features makes a difference.
Can you write Python code for it?

gnb = GaussianNB()
dt = tree.DecisionTreeClassifier()
svc = svm.SVC()
ada = AdaBoostClassifier(n_estimators=10)
rf = RandomForestClassifier(n_estimators=10)

clfs = [gnb,dt,svc,ada,rf]

#Test each data set with some features removed
print "to_messages:"
ml_test.testMany(df,features[:1] + features[2:],clfs,0.3)

print "from_messages:"
ml_test.testMany(df,features[:2] + features[3:],clfs,0.3)

print "from_poi_to_this_person:"
ml_test.testMany(df,features[:3] + features[4:],clfs,0.3)

print "from_this_person_to_poi:"
ml_test.testMany(df,features[:4] + features[5:],clfs,0.3)

print "shared_receipt_with_poi:"
ml_test.testMany(df[features[:5]],features[:5],clfs,0.3)


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
First, I will autoscale the inputs. See if that helps.
Can you write Python code for it?

from sklearn import preprocessing

labelIndex = feature_list[0]
featureIndexes = feature_list[1:]

labels = df[labelIndex].values[:].astype(bool)
features = df[featureIndexes].values[:,:].astype(np.float64)

featuresScaled = preprocessing.scale(features)

clf = RandomForestClassifier(n_estimators=10)

recallScore = cross_val_score(clf, featuresScaled, labels, scoring='recall',cv=kf).mean()
perScore = cross_val_score(clf, featuresScaled, labels, scoring='precision',cv=kf).mean()
accuracyScore = cross_val_score(clf, featuresScaled, labels, scoring='accuracy',cv=kf).mean()

print "Acc: %f, Precision: %f, Recall: %f" % (accuracyScore,perScore,recallScore)


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
Missing Values
For the following I need to figure out what to do with null values. 0 or mean?
to_messages
from_messages
shared_receipt_with_poi
from_this_person_to_poi
from_poi_to_this_person
My thought is to_messages and from_messages should recieve the mean (everyone sends emails) while the poi emails with null get a 0.
Can you write Python code for it?

email_data = ['to_messages',
              'from_messages',
              'from_poi_to_this_person',
              'from_this_person_to_poi',
              'shared_receipt_with_poi']

#Fill all NAs with 0s
df_all_zero = df.copy()
df_all_zero[email_data] = df_all_zero[email_data].fillna(0)

#Fill all NAs with the mean of the column
df_all_mean = df.copy()
df_all_mean[email_data] = df_all_zero[email_data].fillna(df[email_data].mean())

#Fill some NAs with mean, and other with 9
df_mixed = df.copy()
df_mixed[email_data[:2]] = df_mixed[email_data[:2]].fillna(df[email_data[:2]].mean())
df_mixed[email_data[2:]] = df_mixed[email_data[2:]].fillna(0)

features = [('poi')] + email_data

import ml_test

#Import the classifiers
from sklearn.naive_bayes import GaussianNB
from sklearn import tree
from sklearn import svm
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import RandomForestClassifier

gnb = GaussianNB()
dt = tree.DecisionTreeClassifier()
svc = svm.SVC()
ada = AdaBoostClassifier(n_estimators=100)
rf = RandomForestClassifier(n_estimators=100)

clfs = [gnb,dt,svc,ada,rf]

#Test each data set with the classifiers
print "All zero:"
ml_test.testMany(df_all_zero,features,clfs,0.3)

print "All mean:"
ml_test.testMany(df_all_mean,features,clfs,0.3)

print "Mixed:"
ml_test.testMany(df_mixed,features,clfs,0.3)


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
PCA seemed a little inconsistent. At best it does as good. At worst it does much worse. Don't use in final script.
By adding rates for poi emails, I manually reduced the dimensions.
Feature Tunning
Tune for recall and precision (This take a while to run.... so watch out!)
Can you write Python code for it?

from sklearn.grid_search import GridSearchCV

param_grid = {'max_depth': np.arange(2, 10),
             'min_samples_split': np.arange(5, 20),
             'class_weight': [None,'balanced'],
             'max_features': [None,'auto','sqrt']}

clf = GridSearchCV(RandomForestClassifier(n_estimators=5), param_grid,scoring='recall',n_jobs=-1,cv=kf)

clf.fit(featuresScaled,labels)

print "Best score: " + str(clf.best_score_)
print(clf.best_params_)
print(featureIndexes)
print(clf.best_estimator_.feature_importances_)

from sklearn.grid_search import GridSearchCV

param_grid = {'max_depth': np.arange(2, 10),
             'min_samples_split': np.arange(5, 20),
             'class_weight': [None,'balanced','balanced'],
             'max_features': [None,'auto','sqrt']}

clf = GridSearchCV(RandomForestClassifier(n_estimators=5), param_grid,scoring='precision',n_jobs=-1,cv=kf)

clf.fit(featuresScaled,labels)

print "Best score: " + str(clf.best_score_)
print(clf.best_params_)
print(featureIndexes)
print(clf.best_estimator_.feature_importances_)


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
Scaling seems to help overall
Keep for now (can't hurt)
Feature reduction (PCA)
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA(n_components=len(featuresScaled[0])-1)

pcaFeatures = pca.fit_transform(featuresScaled)

clf = RandomForestClassifier(n_estimators=10)

recallScore = cross_val_score(clf, featuresScaled, labels, scoring='recall',cv=kf).mean()
perScore = cross_val_score(clf, featuresScaled, labels, scoring='precision',cv=kf).mean()
accuracyScore = cross_val_score(clf, featuresScaled, labels, scoring='accuracy',cv=kf).mean()

print "Acc: %f, Precision: %f, Recall: %f" % (accuracyScore,perScore,recallScore)


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
The classifier works better when adding the finance fields count
POI Email Rates
Make rates out of the to and from counters and the to and from poi counters.
Can you write Python code for it?

df.ix[df.to_messages==0, 'to_rate'] = 0
df.ix[df.to_messages!=0, 'to_rate'] = df[df.to_messages!=0]['from_poi_to_this_person'] / df[df.to_messages!=0]['to_messages']

df.ix[df.from_messages==0, 'from_rate'] = 0
df.ix[df.from_messages!=0, 'from_rate'] = df[df.from_messages!=0]['from_this_person_to_poi'] / df[df.from_messages!=0]['from_messages']

rateFeatures = ['poi',
                'to_rate',
              'from_rate',
              'shared_receipt_with_poi',
               'finance_field_count']

print "Current features"
ml_test.testMany(df[features],features,clfs,0.3)

print "Rate features"
ml_test.testMany(df[rateFeatures],rateFeatures,clfs,0.3)


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
The recall gets worse when any feature is removed, so for now keep all the features (or maybe combine them...)

Adding Features
Finance Fields Count
Count the number of none-null values each person has for financal data fields.
Can you write Python code for it?

finance_fields = ['salary',
                 'bonus',
                 'restricted_stock',
                 'expenses',
                 'other',
                 'deferred_income',
                 'long_term_incentive']

df['finance_field_count'] = df[finance_fields].notnull().sum(axis=1)

features_added_pos = features + ['finance_field_count']

print "Current features"
ml_test.testMany(df[features],features,clfs,0.3)

print "Added positive feature"
ml_test.testMany(df[features_added_pos],features_added_pos,clfs,0.3)


I am working on the file "P5/P5-Submission/P5 - Notebook.ipynb".
The first task is:
Total number of data points
Can you write Python code for it?

print "Total number of data points: " + str(len(df))


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Build the network
For the neural network, you'll build each layer into a function.  Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function.  This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.

Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section.  TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.
However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the conv2d class, tf.layers.conv2d, you would want to use the TF Neural Network version of conv2d, tf.nn.conv2d.

Let's begin!
Input
The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions
* Implement neural_net_image_input
 * Return a TF Placeholder
 * Set the shape using image_shape with batch size set to None.
 * Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_label_input
 * Return a TF Placeholder
 * Set the shape using n_classes with batch size set to None.
 * Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_keep_prob_input
 * Return a TF Placeholder for dropout keep probability.
 * Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in...
Can you write Python code for it?

import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a batch of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32, [None, image_shape[0], image_shape[1], image_shape[2]], name="x")

def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32, [None, n_classes], name="y")

def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    # TODO: Implement Function
    return tf.placeholder(tf.float32, name="keep_prob")

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Check Point
This is your first checkpoint.  If you ever decide to come back to this notebook or have to restart the notebook, you can start from here.  The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Checkpoint
The model has been saved to disk.
Test Model
Test your model against the test dataset.  This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import tensorflow as tf
import pickle
import helper
import random

# Set batch size if not already set
try:
    if batch_size:
        pass
except NameError:
    batch_size = 64

save_model_path = './image_classification'
n_samples = 4
top_n_predictions = 3

def test_model():
    """
    Test the saved model against the test dataset
    """

    test_features, test_labels = pickle.load(open('preprocess_training.p', mode='rb'))
    loaded_graph = tf.Graph()

    with tf.Session(graph=loaded_graph) as sess:
        # Load model
        loader = tf.train.import_meta_graph(save_model_path + '.meta')
        loader.restore(sess, save_model_path)

        # Get Tensors from loaded model
        loaded_x = loaded_graph.get_tensor_by_name('x:0')
        loaded_y = loaded_graph.get_tensor_by_name('y:0')
        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')
        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')
        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')

        # Get accuracy in batches for memory limitations
        test_batch_acc_total = 0
        test_batch_count = 0

        for train_feature_batch, train_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):
            test_batch_acc_total += sess.run(
                loaded_acc,
                feed_dict={loaded_x: train_feature_batch, loaded_y: train_label_batch, loaded_keep_prob: 1.0})
            test_batch_count += 1

        print('Testing Accuracy: {}\n'.format(test_batch_acc_total/test_batch_count))

        # Print Random Samples
        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))
        random_test_predictions = sess.run(
            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),
            feed_dict={loaded_x: random_test_features, loaded_y:...


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Create Convolutional Model
Implement the function conv_net to create a convolutional neural network model. The function takes in a batch of images, x, and outputs logits.  Use the layers you created above to create this model:

Apply 1, 2, or 3 Convolution and Max Pool layers
Apply a Flatten Layer
Apply 1, 2, or 3 Fully Connected Layers
Apply an Output Layer
Return the output
Apply TensorFlow's Dropout to one or more layers in the model using keep_prob.
Can you write Python code for it?

def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    # TODO: Apply 1, 2, or 3 Convolution and Max Pool layers
    #    Play around with different number of outputs, kernel size and stride
    # Function Definition from Above:
    #    conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    convlayer_1 = conv2d_maxpool(x, 20, (4, 4), (1, 1), (2, 2), (2, 2))

    convlayer_2 = conv2d_maxpool(convlayer_1, 40, (4, 4), (1, 1), (2, 2), (2, 2))

    convlayer_3 = conv2d_maxpool(convlayer_2, 100, (4, 4), (1, 1), (2, 2), (2, 2))

    # TODO: Apply a Flatten Layer
    # Function Definition from Above:
    #   flatten(x_tensor)
    flattened_tensor = flatten(convlayer_3)

    # TODO: Apply 1, 2, or 3 Fully Connected Layers
    #    Play around with different number of outputs
    # Function Definition from Above:
    #   fully_conn(x_tensor, num_outputs)
    connlayer_1 = fully_conn(flattened_tensor, 200)

    connlayer_2 = fully_conn(connlayer_1, 100)

    connlayer_3 = fully_conn(connlayer_2, 30)

    # APPLY DROPOUT NOW
    connlayer_3_dropout = tf.nn.dropout(connlayer_3, keep_prob)

    # TODO: Apply an Output Layer
    #    Set this to the number of classes
    # Function Definition from Above:
    #   output(x_tensor, num_outputs)
    outputlayer = output(connlayer_3_dropout, 10)

    # TODO: return output
    return outputlayer

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""

##############################
## Build the Neural Network ##
##############################

# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()

# Inputs
x = neural_net_image_input((32, 32, 3))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()

# Model
logits = conv_net(x, keep_prob)

# Name logits...


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Flatten Layer
Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor.  The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Can you write Python code for it?

def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    # TODO: Implement Function

    flat_dimension = np.prod(x_tensor.shape.as_list()[1:])
    x_flat = tf.reshape(x_tensor, [-1, flat_dimension])

    return x_flat

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Fully Train the Model
Now that you got a good accuracy with a single CIFAR-10 batch, try it with all five batches.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_model_path = './image_classification'

print('Training...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        # Loop over all batches
        n_batches = 5
        for batch_i in range(1, n_batches + 1):
            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
            print_stats(sess, batch_features, batch_labels, cost, accuracy)

    # Save Model
    saver = tf.train.Saver()
    save_path = saver.save(sess, save_model_path)


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Fully-Connected Layer
Implement the fully_conn function to apply a fully connected layer to x_tensor with the shape (Batch Size, num_outputs). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Can you write Python code for it?

def fully_conn(x_tensor, num_outputs):
    """
    Apply a fully connected layer to x_tensor using weight and bias
    : x_tensor: A 2-D tensor where the first dimension is batch size.
    : num_outputs: The number of output that the new tensor should be.
    : return: A 2-D tensor where the second dimension is num_outputs.
    """
    # TODO: Implement Function

    input_dimensions = x_tensor.shape.as_list()[1]
    W = tf.Variable(tf.truncated_normal([input_dimensions, num_outputs], stddev=0.1))
    b = tf.Variable(tf.constant(0.1, shape=[num_outputs]))

    h_connected = tf.nn.tanh(tf.matmul(x_tensor, W) + b)

    return h_connected

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_fully_conn(fully_conn)


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Implement Preprocess Functions
Normalize
In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive.  The return object should be the same shape as x.
Can you write Python code for it?

def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    # TODO: Implement Function
    return x/255

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
One-hot encode
Just like the previous code cell, you'll be implementing a function for preprocessing.  This time, you'll implement the one_hot_encode function. The input, x, are a list of labels.  Implement the function to return the list of labels as One-Hot encoded Numpy array.  The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode.  Make sure to save the map of encodings outside the function.
Hint:
Look into LabelBinarizer in the preprocessing module of sklearn.
Can you write Python code for it?

import numpy as np
from sklearn.preprocessing import LabelBinarizer

enc = LabelBinarizer()
enc.fit(range(10))

def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    # TODO: Implement Function
    return enc.transform(x)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Show Stats
Implement the function print_stats to print loss and validation accuracy.  Use the global variables valid_features and valid_labels to calculate validation accuracy.  Use a keep probability of 1.0 to calculate the loss and validation accuracy.
Can you write Python code for it?

def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    # TODO: Implement Function
    cost_value = session.run(cost, feed_dict={x: feature_batch, y: label_batch, keep_prob:1.0})
    accuracy_value = session.run(accuracy, feed_dict={x: valid_features, y: valid_labels, keep_prob:1.0})
    print("\nLoss: {}".format(cost_value))
    print("Accuracy (validation): {}".format(accuracy_value))


I am working on the file "P5_image_classification/image_classification.ipynb".
The first task is:
Train on a Single CIFAR-10 Batch
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy.  Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Build Convolutional Neural Network
Can you write Python code for it?

def conv_net(x, keep_prob):
    """
    Create a convolutional neural network model
    : x: Placeholder tensor that holds image data.
    : keep_prob: Placeholder tensor that hold dropout keep probability.
    : return: Tensor that represents logits
    """
    # Function Definition from Above:
    #    conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides)
    convlayer_1 = tf.nn.dropout(conv2d_maxpool(x, 20, (4, 4), (1, 1), (2, 2), (2, 2)), keep_prob)
    #convlayer_1b = tf.nn.dropout(conv2d_maxpool(convlayer_1, 10, (4, 4), (1, 1), (2, 2), (1, 1)), keep_prob)

    convlayer_2 = tf.nn.dropout(conv2d_maxpool(convlayer_1, 30, (4, 4), (1, 1), (2, 2), (2, 2)), keep_prob)
    #convlayer_2b = conv2d_maxpool(convlayer_2, 20, (1, 1), (1, 1), (1, 1), (1, 1))

    convlayer_3 = tf.nn.dropout(conv2d_maxpool(convlayer_2, 60, (4, 4), (1, 1), (2, 2), (2, 2)), keep_prob)
    #convlayer_3b = conv2d_maxpool(convlayer_3, 50, (1, 1), (1, 1), (1, 1), (1, 1))

    # Function Definition from Above:
    #   flatten(x_tensor)
    flattened_tensor = flatten(convlayer_3)

    # Function Definition from Above:
    #   fully_conn(x_tensor, num_outputs)
    connlayer_1 = tf.nn.dropout(fully_conn(flattened_tensor, 200), keep_prob)

    connlayer_2 = tf.nn.dropout(fully_conn(connlayer_1, 100), keep_prob)

    connlayer_3 = tf.nn.dropout(fully_conn(connlayer_2, 30), keep_prob)

    # Function Definition from Above:
    #   output(x_tensor, num_outputs)
    outputlayer = output(connlayer_3, 10)

    return outputlayer

#=============================
#  Build the Neural Network
#=============================

# Remove previous weights, bias, inputs, etc..
tf.reset_default_graph()

# Inputs
x = neural_net_image_input((28, 28, 1))
y = neural_net_label_input(10)
keep_prob = neural_net_keep_prob_input()

# Model
#logits = conv_net(x, keep_prob)
logits = conv_net(x, keep_prob)

# Name logits Tensor, so that is can be loaded from disk after training
logits = tf.identity(logits, name='logits')


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Choose which of the training or testing cells below to use
Train on a single batch
In order to pick the best hyperparameters, we begin by training on a single batch. This will tell us when to stop the learning and will help in choosing a learning rate.
Can you write Python code for it?

batch_i = 0

print('Checking the Training on a Single Batch, i.e. number {}'.format(batch_i))

accuracy_list = []
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        for batch_inputarrays, batch_labels in zip(batch_list(load_training_data(batch_i), size_of_minibatch),
                                                   batch_list(load_training_labels(batch_i), size_of_minibatch)):
            sess.run(optimizer, feed_dict={x: batch_inputarrays, y: batch_labels, keep_prob: keep_probability})
        cost_value, accuracy_value = get_stats(sess, batch_inputarrays, batch_labels, cost, accuracy,
                                               printout=False)
        #print('\nEpoch {:>2}, Batch {}: {} '.format(epoch + 1, batch_i, accuracy_value), end='')
        accuracy_list.append(accuracy_value)

    # Save the model
    #saver = tf.train.Saver()
    #save_path = saver.save(sess, "./trained_model", global_step=epoch)

plt.plot(accuracy_list)
plt.xlabel("Epoch")
plt.ylabel("Accuracy")


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Convolution and Max Pooling
Can you write Python code for it?

def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):
    """
    Apply convolution then max pooling to x_tensor
    :param x_tensor: TensorFlow Tensor
    :param conv_num_outputs: Number of outputs for the convolutional layer
    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer
    :param conv_strides: Stride 2-D Tuple for convolution
    :param pool_ksize: kernal size 2-D Tuple for pool
    :param pool_strides: Stride 2-D Tuple for pool
    : return: A tensor that represents convolution and max pooling of x_tensor
    """
    # Number of input colors
    num_inputcolors = x_tensor.shape.as_list()[3]

    # Convolutional filter
    W_conv= tf.Variable(tf.truncated_normal([conv_ksize[0], conv_ksize[1], num_inputcolors, conv_num_outputs], stddev=0.1))
    b_conv = tf.Variable(tf.constant(0.1, shape=[conv_num_outputs]))

    convolution = tf.nn.conv2d(x_tensor, W_conv, strides=[1, conv_strides[0], conv_strides[1], 1], padding='SAME')
    h_conv = tf.nn.relu(convolution + b_conv)

    h_pool = tf.nn.max_pool(h_conv, ksize=[1, pool_ksize[0], pool_ksize[1], 1],
                            strides=[1, pool_strides[0], pool_strides[1], 1], padding='SAME')

    return h_pool


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Find path names of files and prepare one-hot encoder
Can you write Python code for it?

# We first get all path-names for the training and testing images
training_pathnames = [[folderandfiles[0]+"/"+imname for imname in folderandfiles[2]]
                          for folderandfiles in os.walk("./notMNIST_large") if folderandfiles[2]!=[]]
testing_pathnames = [[folderandfiles[0]+"/"+imname for imname in folderandfiles[2]]
                          for folderandfiles in os.walk("./notMNIST_small") if folderandfiles[2]!=[]]
# training_pathnames has the structure:
#[[list of path-names in first folder], [list of paths in second folder], ...]

def get_letter(filepath):
    """
    Returns the letter corresponding to an image found in filepath
    """
    # The letter is given by the name of the folder in which we find the image
    indexname = filepath.rfind("/")
    letter = filepath[indexname-1:indexname]
    return letter

# In each folder all images depict the same letter
all_letters = np.sort([get_letter(pathname[0]) for pathname in training_pathnames])

# We may now make the function that one-hot-encodes letters into arrays
enc = LabelBinarizer()
enc.fit(all_letters)

def one_hot_encode(list_of_letters):
    """
    One hot encode a list of letters. Returns a one-hot encoded vector for each letter.
    """
    return enc.transform(list_of_letters)

# We now flatten the lists of path names
training_pathnames = np.array(sum(training_pathnames, []))
testing_pathnames = np.array(sum(testing_pathnames, []))

# When trainig, we don't want theimages to be ordered. Therefore, we take a
# random permutation of their order.
np.random.seed(42)
training_pathnames = np.random.permutation(training_pathnames)


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Flatten Layer
Can you write Python code for it?

def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """
    flat_dimension = np.prod(x_tensor.shape.as_list()[1:])
    x_flat = tf.reshape(x_tensor, [-1, flat_dimension])

    return x_flat


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Fully train the model
Can you write Python code for it?

print('Training...')

epochs = 30

accuracy_list = []
with tf.Session() as sess:
    # It is very important the saver is defined INSIDE the block "with tf.Session() as sess"
    # otherwise it will be very difficult to load the graph (unless we name all the variables etc)
    saver = tf.train.Saver()

    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        for batch_i in range(num_saved_batches - 1):
            for batch_inputarrays, batch_labels in zip(batch_list(load_training_data(batch_i),
                                                                  size_of_minibatch),
                                                       batch_list(load_training_labels(batch_i),
                                                                  size_of_minibatch)
                                                      ):
                sess.run(optimizer, feed_dict={x: batch_inputarrays, y: batch_labels, keep_prob: keep_probability})
            cost_value, accuracy_value = get_stats(sess, batch_inputarrays, batch_labels, cost, accuracy,
                                                   printout=False)
            print('Epoch {:>2}, Batch {}: {}'.format(epoch + 1, batch_i, accuracy_value))

        accuracy_list.append(accuracy_value)

        if epoch % 10 == 0:
            # Save the intermediate model
            save_path = saver.save(sess, "./trained_model", global_step=epoch)

    # Save the final model
    save_path = saver.save(sess, "./trained_model", global_step=epoch)

plt.plot(accuracy_list)
plt.xlabel("Epoch")
plt.ylabel("Accuracy")


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Load and display an image
Can you write Python code for it?

def load_training_data(batch_number, image_numbers=[]):
    """
    Loads the training data from files. It is possible to specify an interval
    of images to load, or by defauly load th entire batch.
    """
    if image_numbers == []:
        return np.load("./training_data/training_images_batch" + str(batch_number) + ".npy")
    else:
        return np.load("./training_data/training_images_batch" + str(batch_number) + ".npy")[image_numbers]

def load_training_labels(batch_number, image_numbers=[]):
    """
    Loads the training data from files. It is possible to specify an interval
    of images to load, or by defauly load th entire batch.
    """
    if image_numbers == []:
        return np.load("./training_data/training_labels_batch" + str(batch_number) + ".npy")
    else:
        return np.load("./training_data/training_labels_batch" + str(batch_number) + ".npy")[image_numbers]

def display_image(imagearray):
    array_to_plot = imagearray.reshape((imagearray.shape[0], imagearray.shape[1]))
    print("Image shape: {}".format(imagearray.shape))
    plt.imshow(array_to_plot, cmap="gray")
    plt.axis("off")
    plt.show()

display_image(load_training_data(0)[0])


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Load images and labels into arrays, save to disk
We first normalize the pixel-values to lie between 0 and 1. We also reshape each image to be a 3-dimensional array: (x_length, y_length, color_channels).
Finally, we save the arrays to disk.
Can you write Python code for it?

def load_normalize_image(path):
    """
    Takes the directory path of an image and returns a normalized
    3-dimensional array representing that image.
    """
    # First we load the image
    try:
        imagearray = scipy.ndimage.imread(path)
        # Now we normalize it
        imagearray = imagearray / 255
        # We reshape it to be 3-dimensional: x-dim, y-dim, num_colors
        imagearray = imagearray.reshape(imagearray.shape + (1,))
        return imagearray
    except:
        # Some images are broken in the database; these will raise errors.
        pass

def array_all_images(list_of_path_names):
    """
    Takes a list of directory paths to images and returns a 4-dimensional array
    containing the pixel-data of those images. The shape is:
    (num_images, x_dim, y_dim, num_colors)
    """
    all_images = [load_normalize_image(path) for path in list_of_path_names]
    # Some of these might be None since the function load_normalize_image
    # does not load broken images. We now remove these Nones.
    all_images = np.array(list(filter(None.__ne__, all_images)))
    return all_images

def load_letter(path):
    """
    Takes the directory path of an image and returns a the letter-label of the image.
    """
    # First we see if it is possible to load the image
    try:
        imagearray = scipy.ndimage.imread(path)
        # If this didn't give an error, we may get the letter
        return get_letter(path)
    except:
        # Some images are broken in the database; these will raise errors.
        pass

def array_all_labels(list_of_path_names):
    """
    Takes a list of directory paths to images and returns a 2-dimensional array
    containing the one-hot-encoded labels of those images
    """
    the_letters = [load_letter(path) for path in list_of_path_names]
    the_letters = list(filter(None.__ne__, the_letters))
    all_labels = one_hot_encode(the_letters)
    return all_labels

def batch_list(inputlist, batch_size):
    """
    Returns the inputlist split into batches of...


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Preprocess data
Can you write Python code for it?

import scipy.ndimage
import os
import numpy as np
from sklearn.preprocessing import LabelBinarizer
from tqdm import tqdm

import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Print cost and accuracy
Can you write Python code for it?

# The final batch will be our validation set
validation_inputarray = load_training_data(num_saved_batches - 1)
validation_labels = load_training_labels(num_saved_batches - 1)

def get_stats(session, feature_batch, label_batch, cost, accuracy, printout=True):
    """
    Obtain information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    cost_value = session.run(cost, feed_dict={x: feature_batch, y: label_batch, keep_prob:1.0})
    accuracy_value = session.run(accuracy, feed_dict={x: validation_inputarray,
                                                      y: validation_labels, keep_prob:1.0})
    if printout:
        print("\nLoss: {}".format(cost_value))
        print("Accuracy (validation): {}".format(accuracy_value))
    return cost_value, accuracy_value


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Test model
Can you write Python code for it?

load_model = "./trained_model-59"
testing_inputarray = np.load("./testing_data/testing_images.npy")
testing_labels = np.load("./testing_data/testing_labels.npy")

with tf.Session() as sess:

    saver = tf.train.Saver()

    saver.restore(sess, load_model)
    print("The model's test-set accuracty is {}%".format(np.round(sess.run(accuracy,
                                                                            feed_dict={x: testing_inputarray,
                                        y: testing_labels, keep_prob:1.0})*100, decimals=2)))


I am working on the file "P5b_optional_letter_recognition/digit_recognition.ipynb".
The first task is:
Train the neural network
Hyperparameters
Can you write Python code for it?

epochs = 30
keep_probability = 0.5
learning_rate = 0.0001 # default is 0.001 N.B. it is also possible to make this a placeholder object!
size_of_minibatch = 2**7

#===============================
# Don't need to edit below this
#===============================

# Loss and Optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

# Accuracy
correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')


I am working on the file "P6 data_viz/data/.ipynb_checkpoints/P2-checkpoint.ipynb".
The first task is:
Data Exploration and Visualization
Can you write Python code for it?

data_s=data
survival_group = data_s.groupby('Survived')
survival_group.describe()


I am working on the file "P6 data_viz/data/.ipynb_checkpoints/P2-checkpoint.ipynb".
The first task is:
From the above info(),We can see columns Age, Cabin and Embarked have missing values.
Handling the missing values:

Ignore the rows with missing data,
Exclude the variable at all or we might substite it with  mean or median.

Age  80% of the data is available,which seems a important variable so not to exclude.
Port of embarkation doesn't seem interesting.
cabin 23% of the data so decided to exclude.
PassengerId,Name,fare doesnt seem to contribute to any survival investigation
Can you write Python code for it?

#exculding some coloumns
for a in ['Ticket','Cabin','Embarked','Name','PassengerId','Fare']:
    if a in data.columns:
        del data[a]

print "Age median values by Age and Sex:"
#we are grouping by gender and class and taking median of age so we can replace with corrresponding values instead of NaN
print data.groupby(['Sex','Pclass'], as_index=False).median().loc[:, ['Sex','Pclass', 'Age']]
print "Age values for 5 first persons in dataset:"
print data.loc[data['Age'].isnull(),['Age','Sex','Pclass']].head(5)
# apply transformation: Age missing values are filled with regard to Pclass and Sex:
data.loc[:, 'Age'] = data.groupby(['Sex','Pclass']).transform(lambda x: x.fillna(x.median()))
print data.loc[[5,17,19,26,28],['Age','Sex','Pclass']].head(5)
data['Age'] = data['Age'].fillna(data['Age'].mean())


I am working on the file "P6 data_viz/data/.ipynb_checkpoints/P2-checkpoint.ipynb".
The first task is:
From the above statistics
- Youngest to survive: 0.42
- Youngest to die: 1.0
- Oldest to survive: 80.0
- Oldest to die: 74.0
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn as sns

# Set style for all graphs
#sns.set_style("light")
#sns.set_style("whitegrid")
sns.set_style("ticks", {"xtick.major.size": 8, "ytick.major.size": 8})


I am working on the file "PA1-Q3-1.ipynb".
The first task is:
Hessian function for marking corners
Can you write Python code for it?

# inputt -the input image name
# s -Standard Deviation value
# t -The threshold of eigen value to be considered as edge

def hessian(inputt,s,t):

    start = time.time()
    I = array(Image.open(inputt).convert('L')) # reads the input image into I
    G = []
    for i in range(-2,2+1):
        G.append(gfilter(i,0,s)) # equating y to 0 for a 1D matrix
    Gx = []  #gaussian in x direction
    for i in range(-size,size+1):
        Gx.append(gfilter1(i,0,s,'x'))

    Gy = []  #gaussian in y direction
    for i in range(-size,size+1):
        Gy.append([gfilter1(0,i,s,'y')])

    Gx2 = []
    for i in range(-size,size+1):
        Gx2.append(gfilter2(i,0,s,'x'))

    Gy2 = []
    for i in range(-size,size+1):
        Gy2.append([gfilter2(0,i,s,'y')])

    Ix = []
    for i in range(len(I[:,0])):
        Ix.extend([convolve(I[i,:],Gx)]) # I*G in x direction
    Ix = array(matrix(Ix))
    Iy = []
    for i in range(len(I[0,:])):
        Iy.extend([convolve(I[:,i],Gx)]) # I*G in y direction
    Iy = array(matrix(transpose(Iy)))

    Ixx = []
    for i in range(len(Ix[:,0])):
        Ixx.extend([convolve(Ix[i,:],Gx2)]) # Ix * Gx in x direction
    Ixx = array(matrix(Ixx))

    Iyy = []
    for i in range(len(Iy[0,:])):
        Iyy.extend([convolve(Ix[:,i],Gx2)]) # Iy * Gy in y direction
    Iyy = array(matrix(transpose(Iyy)))

    Ixy = []
    for i in range(len(Iy[0,:])):
        Ixy.extend([convolve(Ix[:,i],Gx2)]) # Iy * Gy in y direction
    Ixy = array(matrix(transpose(Ixy)))
    #store values in x,y to plot the corners
    x = [] # array x[] stores x coordinates of the corner
    y = [] # array y[] stores y coordinates of the corner
    for i in range(len(I[:,0])):
        for j in range(len(I[0,:])):
            H1 = linalg.eigvals(([Ixx[i,j],Ixy[i,j]],[Ixy[i,j],Iyy[i,j]]))
            if((abs(H1[0])>t) & (abs(H1[1])>t)): # if corner
                y.append(i-2) # appending y index to mark corners
                x.append(j-2) # appending y index to mark corners
   ...


I am working on the file "PAT MeOH Kinetik.ipynb".
The first task is:
Umformung auf Hysys-Form
Konventionen
| Froment etc. | Hysys |
| - | - |
| $r[=]\frac{mol}{kg_{Kat}\cdot s}$ | $r_{Hysys}[=]\frac{kgmol}{m^3_{Gas} \cdot s}$ |
$\begin{array}{ccl}
\phi &[=]& \frac{m^3_{Gas}}{m^3_{Kat, Schüttung}}=\frac{m^3_{Gas}}{m^3_{Kat}+m^3_{Gas}}\
1-\phi &[=]& \frac{m^3_{Kat}}{m^3_{Kat, Schüttung}}=\frac{m^3_{Kat}}{m^3_{Kat}+m^3_{Gas}}\
\rho_b &[=]& \frac{kg_{Kat}}{m^3_{Kat, Schüttung}}=\frac{kg_{Kat}}{m^3_{Kat}+m^3_{Gas}}\
\rho_c &[=]& \frac{kg_{Kat}}{m^3_{Kat}}\
\Rightarrow \rho_b &=&  (1-\phi)\rho_c\
\Rightarrow r_{Hysys} &=& r\cdot \left( \frac{1-\phi}{\phi}\right)\rho_c
\end{array}$
Kinetik-Ausdrücke, aus Van den Bussche-Froment, auf $Cu/ZnO/Al_2O_3$ Katalysator
$p_i$ hier eigentlich: $p_i/p^{\circ}$ , $p^{\circ}=1bar$. Es heißt, $r_{MeOH}$ hat die Einheiten $r_{MeOH}[=]k_{5a}'=k_{5a}c_t^2=\frac{mol}{kg_{Kat}\cdot s}$, und die Reaktionsquotienten haben keine Einheiten.
$r_{MeOH}=\frac{k_{5a}' K_2' K_3 K_4 K_{H_2} p_{CO_2}p_{H_2}\left(1-\frac{1}{K_1^*}\cdot\frac{p_{H_2O}\cdot p_{MeOH}}{p_{H_2}^3\cdot p_{CO_2}}\right)}{\left(1+\left(\frac{K_{H_2O}}{K_8 K_9 K_{H_2}}\right)\frac{p_{H_2O}}{p_{H_2}}+\sqrt{K_{H_2}\cdot p_{H2}}+K_{H_2O}\cdot p_{H_2O}\right)^3}$
$r_{RWGS}=\frac{k_1' p_{CO_2}\left(1-K_3^*\cdot\frac{p_{H_2O}\cdot p_{CO}}{p_{H_2}\cdot p_{CO_2}}\right)}{\left(1+\left(\frac{K_{H_2O}}{K_8 K_9 K_{H_2}}\right)\frac{p_{H_2O}}{p_{H_2}}+\sqrt{K_{H_2}\cdot p_{H2}}+K_{H_2O}\cdot p_{H_2O}\right)}$
Umformung der Geschwindigkeitskonstanten von der Form im Artikel auf die Form
in Hysys:
$k = A \cdot exp\left(\frac{-E}{R T}\right)*T^\beta$

$r_{MeOH}$, Vorwärtsreation

im Artikel (Tabelle 1):
$k(i) = A(i) \cdot exp\left(\frac{B(i)}{R T}\right)$
$\begin{align}
k_{5a}' K_2' K_3 K_4 K_{H_2} &=7070,34 \cdot e^{\frac{-36696}{8,3145\cdot501,57}} \frac{mol}{kg_{Kat}\cdot s} \cdot exp\left(\frac{\frac{36,696 J/mol}{8,3145 J/(mol K)}}{ T}\right) \
& = 1,066\frac{mol}{kg_{Kat}\cdot s} \cdot exp\left(\frac{4413,49K}{ T}\right)
\end{align}$
Da
$r_{MeOH}[=]k_{5a}'=k_{5a}c_t^2=\frac{mol}{kg_{Kat}\cdot s}...
Can you write Python code for it?


I am working on the file "PAT MeOH Kinetik.ipynb".
The first task is:
WGS auf ICI-Fe3O4-Cr2O3
Can you write Python code for it?

import locale
import numpy as np
from IPython.display import Latex
locale.setlocale(locale.LC_ALL, '')

epsilon_b = 0.457

table_latex_str = [
    r'\text{Hohlraumvolumen: } \phi=', locale.format('%4.8g',epsilon_b),
    r'\\',
    r'\begin{array}{lccccc}',
    '\hline ',
    'Var & A(i) & A(i)_{Hysys} & B(i) & E(i)_{Hysys} & ',
    r'\text{Hysys Param}', r'\\',
    r'\text{Form } K=A(i) \cdot exp\left(\frac{B(i)}{R T}\right) ','&',
    r' \frac{mol^{0,36} (m^3_{G})^{0,64}}{kg_{Kat}\cdot s}',
    r' & \left(\frac{kgmol}{m^3_{G}}\right)^{0,36} \cdot \frac{1}{s}',
    r' & \frac{J}{mol} & \frac{kJ}{kgmol} & ',
    r'\text{Form } K=A_{Hysys} \cdot exp\left(-\frac{E_{Hysys}}{R T}\right) ',
    r'\\',
    '\hline ',
    r'\\','r_{WGS}', r'\\', r'\\',
    "k_m", '&',
    locale.format('%4.8g', np.exp(8.22)), '&',
    locale.format('%4.8g', np.exp(8.22)*(1-epsilon_b)/epsilon_b*1960/1000.**0.36),
    '&',
    locale.format('%4.8g', -8008*8.3145), '&',
    locale.format('%4.8g', +8008*8.3145), '&', 'k_{Vorwärts}',
    r'\\',
    r"k_m\cdot \frac{1}{K_{eq}}", '&',
    locale.format('%4.8g', np.exp(8.22+4.27)),'&',
    locale.format('%4.8g', np.exp(8.22+4.27)*(1-epsilon_b)/epsilon_b*1960/1000.**0.36),'&',
    locale.format('%4.8g', (-8008-4483)*8.3145),'&',
    locale.format('%4.8g', -(-8008-4483)*8.3145),'&',
    "k'_{Rückwärts}", r'\\'
    r'\hline'
    r'\end{array}'
]
Latex('$'+''.join(table_latex_str)+'$')


I am working on the file "PAT MeOH Kinetik.ipynb".
The first task is:
Wedel 1988
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_2.ipynb".
The first task is:
Extract data
Can you write Python code for it?

X = training_data.drop(['Formation', 'Well Name', 'Depth', 'Facies', 'FaciesLabels'], axis=1).values
y = training_data['Facies'].values - 1

wells = training_data["Well Name"].values


I am working on the file "PA_Team/PA_Team_Submission_2.ipynb".
The first task is:
Feature Augmentation method from Bestagini
Can you write Python code for it?

# Feature windows concatenation function
def augment_features_window(X, N_neig):

    # Parameters
    N_row = X.shape[0]
    N_feat = X.shape[1]

    # Zero padding
    X = np.vstack((np.zeros((N_neig, N_feat)), X, (np.zeros((N_neig, N_feat)))))

    # Loop over windows
    X_aug = np.zeros((N_row, N_feat*(2*N_neig+1)))
    for r in np.arange(N_row)+N_neig:
        this_row = []
        for c in np.arange(-N_neig,N_neig+1):
            this_row = np.hstack((this_row, X[r+c]))
        X_aug[r-N_neig] = this_row

    return X_aug

# Feature gradient computation function
def augment_features_gradient(X, depth):

    # Compute features gradient
    d_diff = np.diff(depth).reshape((-1, 1))
    d_diff[d_diff==0] = 0.001
    X_diff = np.diff(X, axis=0)
    X_grad = X_diff / d_diff

    # Compensate for last missing value
    X_grad = np.concatenate((X_grad, np.zeros((1, X_grad.shape[1]))))

    return X_grad

# Feature augmentation function
def augment_features(X, well, depth, N_neig=1):

    # Augment features
    X_aug = np.zeros((X.shape[0], X.shape[1]*(N_neig*2+2)))
    for w in np.unique(well):
        w_idx = np.where(well == w)[0]
        X_aug_win = augment_features_window(X[w_idx, :], N_neig)
        X_aug_grad = augment_features_gradient(X[w_idx, :], depth[w_idx])
        X_aug[w_idx, :] = np.concatenate((X_aug_win, X_aug_grad), axis=1)

    # Find padded rows
    padded_rows = np.unique(np.where(X_aug[:, 0:7] == np.zeros((1, 7)))[0])

    return X_aug, padded_rows

well = training_data['Well Name'].values
depth = training_data['Depth'].values

X, padded_rows = augment_features(X, well, depth, N_neig=1)


I am working on the file "PA_Team/PA_Team_Submission_2.ipynb".
The first task is:
Load dataset
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_2.ipynb".
The first task is:
Neural Network
Can you write Python code for it?

def fDNN(in_dim, out_dim):

    # Model
    model = Sequential()
    model.add(Dense(152, input_dim=in_dim, activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(512, activation='relu'))
    model.add(Dropout(0.2))
    model.add(Dense(out_dim, activation='softmax'))

    # Compilation
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model


I am working on the file "PA_Team/PA_Team_Submission_2.ipynb".
The first task is:
Validation with Leave One Well Out on Training Dataset
Can you write Python code for it?

logo = LeaveOneGroupOut()

nb_classes = 9
epoch = 10
bats = 20

t0 = tm.time()

f1s_ls = []
acc_ls = []
adj_ls = []

from scipy.signal import medfilt

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]

    # Scaling
    scaler = preprocessing.MinMaxScaler().fit(X)
    X_tr = scaler.transform(X[train])
    X_te = scaler.transform(X[test])

    Y_tr = np_utils.to_categorical(y[train], nb_classes)

    in_dim = len(X_tr[0])

    # Method initialization
    mlp = fDNN(in_dim, nb_classes)

    # Training
    mlp.fit(X_tr, Y_tr, nb_epoch=epoch, batch_size=bats, verbose=0)

    # Predict
    y_hat = mlp.predict_classes(X_te, verbose=0)
    y_hat = medfilt(y_hat, kernel_size=5)

    try:
        f1s = f1_score(y[test], y_hat, average="weighted", labels=[0, 1, 2, 3, 4, 5, 6, 7, 8])
    except:
        f1s = 0

    try:
        conf = confusion_matrix(y[test], y_hat, labels=[0, 1, 2, 3, 4, 5, 6, 7, 8])
        acc = f1_score(y[test], y_hat, average="micro", labels=[0, 1, 2, 3, 4, 5, 6, 7, 8])
    except:
        acc = 0

    try:
        acc_adj = accuracy_adjacent(conf, adjacent_facies)
    except:
        acc_adj = 0

    f1s_ls += [f1s]
    acc_ls += [acc]
    adj_ls += [acc_adj]
    print("{:>20s} f1w:{:.3f} | f1m:{:.3f} | acc_adj:{:.3f}".format(well_name, f1s, acc, acc_adj))

t1 = tm.time()
print("Avg F1w", np.average(f1s_ls)*100, "Avg F1m", np.average(acc_ls)*100, "Avg Adj", np.average(adj_ls)*100)
print((t1-t0), "seconds")


I am working on the file "PA_Team/PA_Team_Submission_3.ipynb".
The first task is:
Extract data
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_3.ipynb".
The first task is:
Load dataset
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_3.ipynb".
The first task is:
Modified imputation method using MLPRegressor
Can you write Python code for it?

from sklearn.neural_network import MLPRegressor

reg = MLPRegressor()
DataImpAll = training_data.drop(['Formation', 'Well Name', 'Depth', 'FaciesLabels'], axis=1).copy()
DataImp = DataImpAll.dropna(axis = 0, inplace=False)
Ximp=DataImp.loc[:, DataImp.columns != 'PE']
Yimp=DataImp.loc[:, 'PE']
reg.fit(Ximp, Yimp)
X[np.array(DataImpAll.PE.isnull()),4] = reg.predict(DataImpAll.loc[DataImpAll.PE.isnull(),:].drop('PE',axis=1,inplace=False))


I am working on the file "PA_Team/PA_Team_Submission_3.ipynb".
The first task is:
Neural Network
Can you write Python code for it?

def fDNN(in_dim, out_dim):

    # Model
    model = Sequential()
    model.add(Dense(28, input_dim=in_dim, init='glorot_uniform', activation='relu'))
    model.add(Dense(28, init='glorot_uniform', activation='relu'))

    model.add(Dense(out_dim, activation='softmax'))

    # Compilation
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model


I am working on the file "PA_Team/PA_Team_Submission_3.ipynb".
The first task is:
Remove Outlier
Can you write Python code for it?

# Comment this block to delete outlier removal

[Scores,indices] = mad_based_outlier(training_data['GR'].values,3.5)
ind = np.where(indices==True)
training_data.drop(training_data.index[ind[0]],inplace=True)
[Scores,indices] = mad_based_outlier(training_data['ILD_log10'].values,3.5)
ind = np.where(indices==True)
training_data.drop(training_data.index[ind[0]],inplace=True)
[Scores,indices] = mad_based_outlier(training_data['DeltaPHI'].values,3.5)
ind = np.where(indices==True)
training_data.drop(training_data.index[ind[0]],inplace=True)


I am working on the file "PA_Team/PA_Team_Submission_3.ipynb".
The first task is:
Utilities function
Can you write Python code for it?

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc

adjacent_facies = np.array([[1], [0, 2], [1], [4], [3, 5], [4, 6, 7], [5, 7], [5, 6, 8], [6, 7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))

def mad_based_outlier(points, thresh=4.5):
    median = np.median(points, axis=0)
    diff = (points - median)**2
    diff = np.sqrt(diff)
    med_abs_deviation = np.median(diff)

    modified_z_score = 0.6745 * diff / med_abs_deviation

    return abs(modified_z_score),abs(modified_z_score) > thresh

# 1=sandstone  2=c_siltstone   3=f_siltstone
# 4=marine_silt_shale 5=mudstone 6=wackestone 7=dolomite
# 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00',
       '#1B4F72','#2E86C1', '#AED6F1', '#A569BD', '#196F3D']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors
facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

training_data.loc[:,'FaciesLabels'] = training_data.apply(lambda row: label_facies(row, facies_labels), axis=1)

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6,...


I am working on the file "PA_Team/PA_Team_Submission_4-revised.ipynb".
The first task is:
Applying to Test Dataset
Can you write Python code for it?

Org_blind_data = pd.read_csv('../data/nofacies_data.csv')
blind_data = Org_blind_data[Org_blind_data["NM_M"]==1]

X_blind = blind_data.drop(['Formation', 'Well Name', 'Depth'], axis=1).values
well_blind = blind_data['Well Name'].values
depth_blind = blind_data['Depth'].values

X_blind, padded_rows = augment_features(X_blind, well_blind, depth_blind, N_neig=1)

# Scaling

scl = preprocessing.MinMaxScaler().fit(X1org)
X_train = scl.transform(X1org)
X_blind = scl.transform(X_blind)
Y_train = np_utils.to_categorical(y1org, nb_classes)

# Method initialization
model = fDNN(in_dim, nb_classes)

# Training
model.fit(X_train, Y_train, nb_epoch=epoch, batch_size=bats, verbose=0)

# Predict
y_blind = model.predict_classes(X_blind, verbose=0)
y_blind = medfilt(y_blind, kernel_size=5)

Org_blind_data.ix[Org_blind_data["NM_M"]==1,"Facies"] = y_blind + 1  # return the original value (1-9)

blind_data = Org_blind_data[Org_blind_data["NM_M"]==2]
X_blind = blind_data.drop(['Formation', 'Well Name', 'Depth','Facies'], axis=1).values
well_blind = blind_data['Well Name'].values
depth_blind = blind_data['Depth'].values
X_blind, padded_rows = augment_features(X_blind, well_blind, depth_blind, N_neig=1)

# Scaling
scl = preprocessing.MinMaxScaler().fit(X2org)
X_train = scl.transform(X2org)
X_blind = scl.transform(X_blind)

Y_train = np_utils.to_categorical(y2org, nb_classes)

# Method initialization
model = fDNN(in_dim, nb_classes)

# Training
model.fit(X_train, Y_train, nb_epoch=epoch, batch_size=bats, verbose=0)

# Predict
y_blind = model.predict_classes(X_blind, verbose=0)
y_blind = medfilt(y_blind, kernel_size=5)

Org_blind_data.ix[Org_blind_data["NM_M"]==2,"Facies"] = y_blind + 1  # return the original value (1-9)

Org_blind_data.to_csv("PA_Team_Submission_4-revised.csv")

make_facies_log_plot(
    Org_blind_data[Org_blind_data['Well Name'] == 'STUART'],
    facies_colors)

make_facies_log_plot(
    Org_blind_data[Org_blind_data['Well Name'] == 'CRAWFORD'],
    facies_colors)


I am working on the file "PA_Team/PA_Team_Submission_4-revised.ipynb".
The first task is:
Extract data
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_4-revised.ipynb".
The first task is:
Feature Augmentation method from Bestagini
Can you write Python code for it?

# Feature windows concatenation function
def augment_features_window(X, N_neig):

    # Parameters
    N_row = X.shape[0]
    N_feat = X.shape[1]

    # Zero padding
    X = np.vstack((np.zeros((N_neig, N_feat)), X, (np.zeros((N_neig, N_feat)))))

    # Loop over windows
    X_aug = np.zeros((N_row, N_feat*(2*N_neig+1)))
    for r in np.arange(N_row)+N_neig:
        this_row = []
        for c in np.arange(-N_neig,N_neig+1):
            this_row = np.hstack((this_row, X[r+c]))
        X_aug[r-N_neig] = this_row

    return X_aug

# Feature gradient computation function
def augment_features_gradient(X, depth):

    # Compute features gradient
    d_diff = np.diff(depth).reshape((-1, 1))
    d_diff[d_diff==0] = 0.001
    X_diff = np.diff(X, axis=0)
    X_grad = X_diff / d_diff

    # Compensate for last missing value
    X_grad = np.concatenate((X_grad, np.zeros((1, X_grad.shape[1]))))

    return X_grad

# Feature augmentation function
def augment_features(X, well, depth, N_neig=1):

    # Augment features
    X_aug = np.zeros((X.shape[0], X.shape[1]*(N_neig*2+2)))
    for w in np.unique(well):
        w_idx = np.where(well == w)[0]
        X_aug_win = augment_features_window(X[w_idx, :], N_neig)
        X_aug_grad = augment_features_gradient(X[w_idx, :], depth[w_idx])
        X_aug[w_idx, :] = np.concatenate((X_aug_win, X_aug_grad), axis=1)

    # Find padded rows
    padded_rows = np.unique(np.where(X_aug[:, 0:7] == np.zeros((1, 7)))[0])

    return X_aug, padded_rows

# Marine Models

Org_data = training_data
training_data = training_data[training_data["NM_M"]==1]

X = training_data.drop(['Formation', 'Well Name', 'Depth', 'Facies', 'FaciesLabels'], axis=1).values
y = training_data['Facies'].values - 1
wells = training_data["Well Name"].values
well = training_data['Well Name'].values
depth = training_data['Depth'].values

X, padded_rows = augment_features(X, well, depth, N_neig=1)
X1org = X
y1org = y


I am working on the file "PA_Team/PA_Team_Submission_4-revised.ipynb".
The first task is:
Load dataset
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_4-revised.ipynb".
The first task is:
Modified imputation method using MLPRegressor
Can you write Python code for it?

from sklearn.neural_network import MLPRegressor

reg = MLPRegressor()
DataImpAll = training_data.drop(['Formation', 'Well Name', 'Depth', 'FaciesLabels'], axis=1).copy()
DataImp = DataImpAll.dropna(axis = 0, inplace=False)
Ximp=DataImp.loc[:, DataImp.columns != 'PE']
Yimp=DataImp.loc[:, 'PE']
reg.fit(Ximp, Yimp)
X[np.array(DataImpAll.PE.isnull()),4] = reg.predict(DataImpAll.loc[DataImpAll.PE.isnull(),:].drop('PE',axis=1,inplace=False))

training_data.ix[:,"PE"] = X[:,4]


I am working on the file "PA_Team/PA_Team_Submission_4-revised.ipynb".
The first task is:
Neural Network
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_4-revised.ipynb".
The first task is:
Utilities function
Can you write Python code for it?

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc

adjacent_facies = np.array([[1], [0, 2], [1], [4], [3, 5], [4, 6, 7], [5, 7], [5, 6, 8], [6, 7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))

# 1=sandstone  2=c_siltstone   3=f_siltstone
# 4=marine_silt_shale 5=mudstone 6=wackestone 7=dolomite
# 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00',
       '#1B4F72','#2E86C1', '#AED6F1', '#A569BD', '#196F3D']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors
facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

training_data.loc[:,'FaciesLabels'] = training_data.apply(lambda row: label_facies(row, facies_labels), axis=1)

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '.g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '.')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '.', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '.', color='r')
    ax[4].plot(logs.PE, logs.Depth, '.', color='black')
    im=ax[5].imshow(cluster,...


I am working on the file "PA_Team/PA_Team_Submission_4-revised.ipynb".
The first task is:
Validation with Leave One Well Out on Training Dataset
Can you write Python code for it?

logo = LeaveOneGroupOut()

nb_classes = 9
epoch = 10
bats = 20

t0 = tm.time()

f1s_ls = []
acc_ls = []
adj_ls = []

from scipy.signal import medfilt

for train, test in logo.split(X, y, groups=wells):
    well_name = wells[test[0]]
    # Scaling
    scaler = preprocessing.MinMaxScaler().fit(X)
    X_tr = scaler.transform(X[train])
    X_te = scaler.transform(X[test])

    Y_tr = np_utils.to_categorical(y[train], nb_classes)

    in_dim = len(X_tr[0])

    # Method initialization
    mlp = fDNN(in_dim, nb_classes)

    # Training
    mlp.fit(X_tr, Y_tr, nb_epoch=epoch, batch_size=bats, verbose=0)

    # Predict
    y_hat = mlp.predict_classes(X_te, verbose=0)
    y_hat = medfilt(y_hat, kernel_size=5)

    try:
        f1s = f1_score(y[test], y_hat, average="weighted", labels=[0, 1, 2, 3, 4, 5, 6, 7, 8])
    except:
        f1s = 0

    try:
        conf = confusion_matrix(y[test], y_hat, labels=[0, 1, 2, 3, 4, 5, 6, 7, 8])
        acc = f1_score(y[test], y_hat, average="micro", labels=[0, 1, 2, 3, 4, 5, 6, 7, 8])
    except:
        acc = 0

    try:
        acc_adj = accuracy_adjacent(conf, adjacent_facies)
    except:
        acc_adj = 0

    f1s_ls += [f1s]
    acc_ls += [acc]
    adj_ls += [acc_adj]
    print("{:>20s} f1w:{:.3f} | f1m:{:.3f} | acc_adj:{:.3f}".format(well_name, f1s, acc, acc_adj))

t1 = tm.time()
print("Avg F1w", np.average(f1s_ls)*100, "Avg F1m", np.average(acc_ls)*100, "Avg Adj", np.average(adj_ls)*100)
print((t1-t0), "seconds")

# Non - Marine

training_data = Org_data
training_data = training_data[training_data["NM_M"]==2]

X = training_data.drop(['Formation', 'Well Name', 'Depth', 'Facies', 'FaciesLabels'], axis=1).values
y = training_data['Facies'].values - 1
wells = training_data["Well Name"].values
well = training_data['Well Name'].values
depth = training_data['Depth'].values
X, padded_rows = augment_features(X, well, depth, N_neig=1)
X2org =X
y2org = y

f1s_ls = []
acc_ls = []
adj_ls = []

from scipy.signal import medfilt

for train, test in logo.split(X, y,...


I am working on the file "PA_Team/PA_Team_Submission_4.ipynb".
The first task is:
Applying to Test Dataset
Can you write Python code for it?

Org_blind_data = pd.read_csv('../data/nofacies_data.csv')
blind_data = Org_blind_data[Org_blind_data["NM_M"]==1]

X_blind = blind_data.drop(['Formation', 'Well Name', 'Depth'], axis=1).values
well_blind = blind_data['Well Name'].values
depth_blind = blind_data['Depth'].values

X_blind, padded_rows = augment_features(X_blind, well_blind, depth_blind, N_neig=1)

# Scaling

scl = preprocessing.MinMaxScaler().fit(X1org)
X_train = scl.transform(X1org)
X_blind = scl.transform(X_blind)
Y_train = np_utils.to_categorical(y1org, nb_classes)

# Method initialization
model = fDNN(in_dim, nb_classes)

# Training
model.fit(X_train, Y_train, nb_epoch=epoch, batch_size=bats, verbose=0)

# Predict
y_blind = model.predict_classes(X_blind, verbose=0)
y_blind = medfilt(y_blind, kernel_size=5)

Org_blind_data.ix[Org_blind_data["NM_M"]==1,"Facies"] = y_blind + 1  # return the original value (1-9)

blind_data = Org_blind_data[Org_blind_data["NM_M"]==2]
X_blind = blind_data.drop(['Formation', 'Well Name', 'Depth','Facies'], axis=1).values
well_blind = blind_data['Well Name'].values
depth_blind = blind_data['Depth'].values
X_blind, padded_rows = augment_features(X_blind, well_blind, depth_blind, N_neig=1)

# Scaling

scl = preprocessing.MinMaxScaler().fit(X2org)
X_train = scl.transform(X2org)
X_blind = scl.transform(X_blind)

Y_train = np_utils.to_categorical(y2org, nb_classes)

# Method initialization
model = fDNN(in_dim, nb_classes)

# Training
model.fit(X_train, Y_train, nb_epoch=epoch, batch_size=bats, verbose=0)

# Predict
y_blind = model.predict_classes(X_blind, verbose=0)
y_blind = medfilt(y_blind, kernel_size=5)

Org_blind_data.ix[Org_blind_data["NM_M"]==2,"Facies"] = y_blind + 1  # return the original value (1-9)

blind_data.to_csv("PA_Team_Submission_4.csv")

make_facies_log_plot(
    Org_blind_data[Org_blind_data['Well Name'] == 'STUART'],
    facies_colors)

make_facies_log_plot(
    Org_blind_data[Org_blind_data['Well Name'] == 'CRAWFORD'],
    facies_colors)


I am working on the file "PA_Team/PA_Team_Submission_4.ipynb".
The first task is:
Extract data
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_4.ipynb".
The first task is:
Feature Augmentation method from Bestagini
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_4.ipynb".
The first task is:
Load dataset
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_4.ipynb".
The first task is:
Neural Network
Can you write Python code for it?


I am working on the file "PA_Team/PA_Team_Submission_4.ipynb".
The first task is:
Validation with Leave One Well Out on Training Dataset
Can you write Python code for it?


I am working on the file "PCA Test.ipynb".
The first task is:
Do PCA
Can you write Python code for it?

plt.cla()
pca = decomposition.PCA(n_components=2)
pca.fit(matrix)
X = pca.transform(matrix)


I am working on the file "PCA Test.ipynb".
The first task is:
Prepare the plot
Can you write Python code for it?

fig = plt.figure(1, figsize=(10, 6))
plt.clf()


I am working on the file "PCA nie je vyber atributov.ipynb".
The first task is:
Mozeme skusit pouzit PCA na oznacenie tych atributov, ktore najviac prispievaju k variancii v datach
Zobrazime si heatmapu toho, ako silno prispievaju jednotlive vlastnosti k tvorbe komponentov a teda ako silno su v nich odrazene. To by nam malo vediet povedat, ktory atribut je vo vyslednych datach najviac odrazeny.
Can you write Python code for it?

import sklearn.datasets as ds
from sklearn.decomposition import PCA

pca_trafo = PCA()
data = ds.load_breast_cancer()['data']
pca_data = pca_trafo.fit_transform(data)

ax = seaborn.heatmap(np.log(pca_trafo.inverse_transform(np.eye(data.shape[1]))), cmap="hot", cbar=False)
ax.set_xlabel('features')
ax.set_ylabel('components')


I am working on the file "PCA nie je vyber atributov.ipynb".
The first task is:
Naozaj sa nam podarilo najst tie data, ktore suviseli s predikovanou premennou.
Da sa na nieco podobne pouzit PCA?
A case against PCA a.k.a. Nepouzivajte PCA na vyber atributov
PCA sa obycajne pouziva na redukciu atributov do mensieho poctu komponentov. Tu sa ale vyrabaju uplne nove komponenty (atributy), ktore su linearnymi kombinaciami tych povodnych
Can you write Python code for it?


I am working on the file "PCA nie je vyber atributov.ipynb".
The first task is:
Pouzijeme oblubeny dataset kvietkov ktory ma 150 pozorovani a 4 atributy
K nemu dogenerujeme 20 nahodnych atributov, ktore by mali mat len minimalny vplyv na predikciu zavyslej premennej
Can you write Python code for it?

# vygenerujme si 20 uplne nahodnych atributov a pricapme ich k povodnym datam
E = np.random.uniform(0, 0.1, size=(len(iris.data), 20))
X = np.hstack((iris.data, E))

y = iris.target
X_indices = np.arange(X.shape[-1])
X_indices

X.shape


I am working on the file "PCA nie je vyber atributov.ipynb".
The first task is:
Pre porovnanie sa pozireme na dva riadky povodnych a novych dat
Can you write Python code for it?

# povodne data
iris.data[:2]

# data rozsirene o dalsich 20 nahodnych atributov
# len tie prve by mali davat zmysel
X[:2]


I am working on the file "PCA nie je vyber atributov.ipynb".
The first task is:
Skusme pouzit PCA na zobrazenie toho, kolko potrebuje komponentov na vysvetlenie datasetu
pozor, hovorime o komponentoch ktore vznikli linearnou kombinaciou atributov a nie priamo o atributoch
Can you write Python code for it?

import sklearn.datasets as ds

data = ds.load_breast_cancer()['data']
pca_trafo = PCA().fit(data)

fig = plt.figure()
ax = fig.add_subplot(1,1,1)

line, = ax.plot(pca_trafo.explained_variance_ratio_, '--o')

ax.set_yscale('log') # skus si vyhodit logaritmicku mierku, uvidis, ze je tam asi problem
ax.set_title('Prispevok komponentov k vysvetleniu variancie datasetu')


I am working on the file "PCA nie je vyber atributov.ipynb".
The first task is:
Teraz potrebujem trochu viac komponentov na to aby som vysvetlil rovnake mnozstvo variancie
Atributy maju asi vyrovnanejsi prispevok
Ako bude vyzerat ta heatmapa?
Can you write Python code for it?

import sklearn.datasets as ds
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

z_scaler = StandardScaler()
data = ds.load_breast_cancer()['data']

pca_trafo = PCA()
z_data = z_scaler.fit_transform(data)
pca_data = pca_trafo.fit_transform(z_data)

ax = seaborn.heatmap(np.log(pca_trafo.inverse_transform(np.eye(data.shape[1]))), cmap="hot", cbar=False)
ax.set_xlabel('features')
ax.set_ylabel('components')


I am working on the file "PCA nie je vyber atributov.ipynb".
The first task is:
Z tejto heatmapy sa neda vycitat ziadny jasny trend. Skusme este tie priemery prispevkov do komponentov per atribut.
Can you write Python code for it?

means = np.mean(pca_trafo.inverse_transform(np.eye(X.shape[1])), axis=0)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.bar(X_indices, means)
ax.set_ylabel('mean contrib. in components')
ax.set_xlabel('feature #')


I am working on the file "PCA.ipynb".
The first task is:
Again, with the magic command time.
Can you write Python code for it?

%time PCA(Xbar, 2)
%time PCA_high_dim(Xbar, 2)
pass


I am working on the file "PCA.ipynb".
The first task is:
Exercise 1: dimensionality reduction on wine chemical data
In this tutorial we will use real experimental data, representing the chemical composition of about 178 wine samples, from three different cultivars in the same region in Italy.
For each wine, the analysis determined the quantities of 13 constituents, listed below:

    Feature No.
    Wine Chemical Composition

    1
    Alcohol

    2
    Malic acid

    3
    Ash

    4
    Alcalinity of ash

    5
    Magnesium

    6
    Total phenols

    7
    Flavanoids

    8
    Nonavanoid phenols

    9
    Proanthocyanins

    10
    Color intensity

    11
    Hue

    12
    OD280/OD315 of diluted wines

    13
    Proline

Wine chemical composition*

  *Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science

The data file ./data/wineInputs.txt contains 178 lines, each with the 13 entries representing the wine chemical composition (descriptor). The file  ./data/wineOutputs.txt contains the type of each wine: either 1, 2 or 3.
In this tutorial, the objective is to reduce the dimensionality of the descriptors by identifying and eliminating the redundant ones with PCA.
Fill in the missing code based on the previous examples.
This tutorial will be used later for the exercise on the following topics (Kmeans, KNN and Neural Network).
Can you write Python code for it?

# First, we load input/output data from the files
dataIn = np.genfromtxt('./data/wineInputs.txt', delimiter=',')
dataOut = np.genfromtxt('./data/wineOutputs.txt', delimiter=',')

## # Apply normalization
# ...

## # Apply PCA
# ...

# Here, we try to obtain the normalized cumulative sum (cumsum) of eigenvalues
# assuming D are the eigenvalues from PCA
idc = np.divide(np.cumsum(D),np.sum(D))

# We plot normalized cumulative sum to understand the contributions of the obtained PCs
plt.title('Normalized cumulative sum')
plt.ylim([0, 1.1])
plt.xlabel('# principal components')
plt.ylabel('cumulative sum')

plt.plot(range(1,len(D)+1),idc,'bo') # re-plot the data

plt.grid()
plt.show()


I am working on the file "PCA.ipynb".
The first task is:
Multicollinearity Check
Using $corr(X)$, check to see that we some level of multicollinearity in the data, enough to warrant using PCA of visualization in reduced-rank principal component space. As a rule of thumb, the off-diagonal correlation values (either in the upper- or lower-triangle) should have absolute values of around 0.30 or so.
Can you write Python code for it?

corr = X_zscaled.corr()

tmp = pd.np.triu(corr) - np.eye(corr.shape[0])
tmp = tmp.flatten()
tmp = tmp[np.nonzero(tmp)]
tmp = pd.Series(np.abs(tmp))

print('Correlation matrix:\n\n{}\n\n'.format(corr.values))

print('Multicollinearity check using off-diagonal values:\n\n{}'.format(tmp.describe()))


I am working on the file "PCA.ipynb".
The first task is:
Next, we create some toy data and normalize it using <i>zscore</i>.
For simplicity, we create a set of points in the plane, roughly arranged as an ellipse.
Can you write Python code for it?

# Make an array of 13 angle values, from 0 to 2PI, at regular intervals
t = np.linspace(0, 2*np.pi, 13)

# set some parameters of the ellipse
A = 1
B = 1.3
x0 = 1.5
y0 = 0.5
angle = -45

# compute x and y coordinates from the angles and parameters
x = A*np.cos(t+angle)+x0 + 0.05*np.random.uniform(-1.0,1.0,t.size)
y = B*np.sin(t)+y0 + 0.05*np.random.uniform(-1.0,1.0,t.size)
# put the x and y arrays into a matrix format
Xdata=np.column_stack([x,y])

# Next, we normalize the data
X = zscore(Xdata)

# Plot the original data first (normalized)
plt.figure(1)
plt.plot(Xdata[:,0],Xdata[:,1],'ro', label='original') # re-plot the data
plt.plot(X[:,0],X[:,1],'bo',label='zscored') # re-plot the data
plt.legend()
plt.xlabel('X1')
plt.ylabel('X2')

plt.title('Original data prior to PCA transformation')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.ylim(bottom=-2.0, top=3.0)
plt.xlim(left=-2, right=3)
plt.grid()
plt.show()


I am working on the file "PCA.ipynb".
The first task is:
Now let's compare the running time between PCA and PCA_high_dim.
Tips for running benchmarks or computationally expensive code:
When you have some computation that takes up a non-negligible amount of time. Try separating
the code that produces output from the code that analyzes the result (e.g. plot the results, comput statistics of the results). In this way, you don't have to recompute when you want to produce more analysis.
Can you write Python code for it?

def time(f, repeat=10):
    times = []
    for _ in range(repeat):
        start = timeit.default_timer()
        f()
        stop = timeit.default_timer()
        times.append(stop-start)
    return np.mean(times), np.std(times)

times_mm0 = []
times_mm1 = []

for datasetsize in np.arange(4, 784, step=20):
    XX = Xbar[:datasetsize]
    mu, sigma = time(lambda : XX.T @ XX)
    times_mm0.append((datasetsize, mu, sigma))

    mu, sigma = time(lambda : XX @ XX.T)
    times_mm1.append((datasetsize, mu, sigma))

times_mm0 = np.asarray(times_mm0)
times_mm1 = np.asarray(times_mm1)

fig, ax = plt.subplots()
ax.set(xlabel='size of dataset', ylabel='running time')
bar = ax.errorbar(times_mm0[:, 0], times_mm0[:, 1], times_mm0[:, 2], label="$X^T X$ (PCA)", linewidth=2)
ax.errorbar(times_mm1[:, 0], times_mm1[:, 1], times_mm1[:, 2], label="$X X^T$ (PCA_high_dim)", linewidth=2)
ax.legend();


I am working on the file "PCA.ipynb".
The first task is:
Now, let's plot a digit from the dataset:
Can you write Python code for it?

from ipywidgets import interact

from sklearn.datasets import fetch_mldata
MNIST = fetch_mldata('MNIST original', data_home='./MNIST')
%matplotlib inline

plt.figure(figsize=(4,4))
plt.imshow(MNIST.data[0].reshape(28,28), cmap='gray');


I am working on the file "PCA.ipynb".
The first task is:
Now, with the help of the functions you have implemented above, let's implement PCA! When you implement PCA, do take advantage of the functions that you have implemented above.
Can you write Python code for it?

# GRADED FUNCTION: DO NOT EDIT THIS LINE

# ===YOU SHOULD EDIT THIS FUNCTION===
def PCA(X, num_components):
    """
    Args:
        X: ndarray of size (N, D), where D is the dimension of the data,
           and N is the number of datapoints
        num_components: the number of principal components to use.
    Returns:
        X_reconstruct: ndarray of size (N, D) the reconstruction
        of X from the first `num_components` principal components.
    """
    # Compute the data covariance matrix S
    S = np.cov(X, rowvar=False, bias=True)

    # Next find eigenvalues and corresponding eigenvectors for S by implementing eig().
    eig_vals, eig_vecs = eig(S)

    # Reconstruct the images from the lowerdimensional representation
    # To do this, we first need to find the projection_matrix (which you implemented earlier)
    # which projects our input data onto the vector space spanned by the eigenvectors
    P = projection_matrix(eig_vecs[:,:num_components]) # projection matrix

    # Then for each data point x_i in the dataset X
    #   we can project the original x_i onto the eigenbasis.
    X_reconstruct = (P @ X.T).T
    return X_reconstruct

## Some preprocessing of the data
NUM_DATAPOINTS = 1000
X = (MNIST.data.reshape(-1, 28 * 28)[:NUM_DATAPOINTS]) / 255.
Xbar, mu, std = normalize(X)


I am working on the file "PCA.ipynb".
The first task is:
PCA transforms the original data space to PC space where the highest information is held by the first PC. PCA will be extremely beneficial when we deal with high dimensional data sets, so we can discard unneccesary PCs which contain very low variance.
This type of problem will be tested on real chemical data in the final tutorial.
The following code is to plot the original data together with the new transformation data as well
as their eigenvector (PCs) as the direction. The produced Figure demonstrates how the original data (blue
dots) rotate to form new transformation data (red dots).
Can you write Python code for it?

# We re-plot the original Figure
plt.figure(1)
plt.plot(X[:,0],X[:,1],'bo', label='zscored data') # re-plot the data
plt.xlabel('X1')
plt.ylabel('X2')

plt.plot(Ypca[:,0],Ypca[:,1],'ro', label='PCA transformed')
plt.title('After PCA transformation')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.ylim(bottom=-2.0, top=2.0)
plt.xlim(left=-2, right=2)
plt.legend()
plt.grid()
plt.show()


I am working on the file "PCA.ipynb".
The first task is:
PCA via Eigen-Decomposition
Obtain eigenvalues, eigenvectors of $cov(X)$ via eigen-decomposition
From the factorization of symmetric matrix $S$ into orthogonal matrix $Q$ of the eigenvectors and diagonal matrix $\Lambda$ of the eigenvalues, we can likewise decompose $cov(X)$ (or in our case $corr(X)$ since we standardized our data).
\begin{align}
  S &= Q \Lambda Q^\intercal \
  \Rightarrow X^\intercal X &= V \Lambda V^\intercal \
\end{align}
where $V$ are the orthonormal eigenvectors of $X X^\intercal$.
We can normalize the eigenvalues to see how much variance is captured per each respective principal component. We will also calculate the cumulative variance explained. This will help inform our decision of how many principal components to keep when reducing dimensions in visualizing the data.
Can you write Python code for it?

eigenvalues, eigenvectors = np.linalg.eig(X_zscaled.cov())

eigenvalues_normalized = eigenvalues / eigenvalues.sum()

cumvar_explained = np.cumsum(eigenvalues_normalized)


I am working on the file "PCA.ipynb".
The first task is:
Relative weights of the original features in principal component space

Each row of $V$ is a vector representing the relative weights for each of the original features for each principal component.
Given reduced-rank $k$, the columns of $V_k$ are the weights for principal components $1, \dots, k$.
Calculate the norm for each row of $V_k$, and normalize the results to obtain the relative weights for each original feature in principal component space.
Can you write Python code for it?

feature_norms = np.linalg.norm(eigenvectors[:, 0:k], axis=1)
feature_weights = feature_norms / feature_norms.sum()

msg = ('Using {} principal components, '
       'the original features are represented with the following weights:')
print(msg.format(k))
for feature, weight in zip(iris.feature_names, feature_weights):
    print('-  {}: {:0.3f}'.format(feature, weight))


I am working on the file "PCA.ipynb".
The first task is:
The greater number of of principal components we use, the smaller will our reconstruction
error be. Now, let's answer the following question:

How many principal components do we need
in order to reach a Mean Squared Error (MSE) of less than $100$ for our dataset?
Can you write Python code for it?

def mse(predict, actual):
    return np.square(predict - actual).sum(axis=1).mean()

loss = []
reconstructions = []
for num_component in range(1, 100):
    reconst = PCA(Xbar, num_component)
    reconst = np.real(reconst)
    error = mse(reconst, Xbar)
    reconstructions.append(reconst)
    # print('n = {:d}, reconstruction_error = {:f}'.format(num_component, error))
    loss.append((num_component, error))

reconstructions = np.asarray(reconstructions)
reconstructions = reconstructions * std + mu # "unnormalize" the reconstructed image
loss = np.asarray(loss)

loss