I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(2c) Automated creation of an OHE dictionary
Now use the code from Parts (2a) and (2b) to write a function that takes an input dataset and outputs an OHE dictionary.  Then use this function to create an OHE dictionary for the sample dataset, and verify that it matches the dictionary from Part (2b).
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def createOneHotDict(inputData):
    """Creates a one-hot-encoder dictionary based on the input data.

    Args:
        inputData (RDD of lists of (int, str)): An RDD of observations where each observation is
            made up of a list of (featureID, value) tuples.

    Returns:
        dict: A dictionary where the keys are (featureID, value) tuples and map to values that are
            unique integers.
    """
    return (inputData
            .flatMap(lambda x: x)
            .distinct()
            .zipWithIndex()
            .collectAsMap())

sampleOHEDictAuto = createOneHotDict(sampleDataRDD)
print sampleOHEDictAuto

# TEST Automated creation of an OHE dictionary (2c)
Test.assertEquals(sorted(sampleOHEDictAuto.keys()),
                  [(0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'),
                   (1, 'tabby'), (2, 'mouse'), (2, 'salmon')],
                  'sampleOHEDictAuto has unexpected keys')
Test.assertEquals(sorted(sampleOHEDictAuto.values()), range(7),
                  'sampleOHEDictAuto has unexpected values')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(3a) Loading and splitting the data
We are now ready to start working with the actual CTR data, and our first task involves splitting it into training, validation, and test sets.  Use the randomSplit method with the specified weights and seed to create RDDs storing each of these datasets, and then cache each of these RDDs, as we will be accessing them multiple times in the remainder of this lab. Finally, compute the size of each dataset.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
weights = [.8, .1, .1]
seed = 42
# Use randomSplit with weights and seed
rawTrainData, rawValidationData, rawTestData = rawData.randomSplit(weights, seed)
# Cache the data
rawTrainData.cache()
rawValidationData.cache()
rawTestData.cache()

nTrain = rawTrainData.count()
nVal = rawValidationData.count()
nTest = rawTestData.count()
print nTrain, nVal, nTest, nTrain + nVal + nTest
print rawData.take(1)

# TEST Loading and splitting the data (3a)
Test.assertTrue(all([rawTrainData.is_cached, rawValidationData.is_cached, rawTestData.is_cached]),
                'you must cache the split data')
Test.assertEquals(nTrain, 79911, 'incorrect value for nTrain')
Test.assertEquals(nVal, 10075, 'incorrect value for nVal')
Test.assertEquals(nTest, 10014, 'incorrect value for nTest')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(3c) Create an OHE dictionary from the dataset
Note that parsePoint returns a data point as a list of (featureID, category) tuples, which is the same format as the sample dataset studied in Parts 1 and 2 of this lab.  Using this observation, create an OHE dictionary using the function implemented in Part (2c). Note that we will assume for simplicity that all features in our CTR dataset are categorical.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
ctrOHEDict = createOneHotDict(parsedTrainFeat)
numCtrOHEFeats = len(ctrOHEDict.keys())
print numCtrOHEFeats
print ctrOHEDict[(0, '')]

# TEST Create an OHE dictionary from the dataset (3c)
Test.assertEquals(numCtrOHEFeats, 233286, 'incorrect number of features in ctrOHEDict')
Test.assertTrue((0, '') in ctrOHEDict, 'incorrect features in ctrOHEDict')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(3d) Apply OHE to the dataset
Now let's use this OHE dictionary by starting with the raw training data and creating an RDD of LabeledPoint objects using OHE features.  To do this, complete the implementation of the parseOHEPoint function. Hint: parseOHEPoint is an extension of the parsePoint function from Part (3b) and it uses the oneHotEncoding function from Part (1d).
Can you write Python code for it?

from pyspark.mllib.regression import LabeledPoint

# TODO: Replace <FILL IN> with appropriate code
def parseOHEPoint(point, OHEDict, numOHEFeats):
    """Obtain the label and feature vector for this raw observation.

    Note:
        You must use the function `oneHotEncoding` in this implementation or later portions
        of this lab may not function as expected.

    Args:
        point (str): A comma separated string where the first value is the label and the rest
            are features.
        OHEDict (dict of (int, str) to int): Mapping of (featureID, value) to unique integer.
        numOHEFeats (int): The number of unique features in the training dataset.

    Returns:
        LabeledPoint: Contains the label for the observation and the one-hot-encoding of the
            raw features based on the provided OHE dictionary.
    """
    label = point.split(',', 1)[0]
    features = parsePoint(point)
    OHEfeatures = oneHotEncoding(features, OHEDict, numOHEFeats)
    return LabeledPoint(label, OHEfeatures)

OHETrainData = rawTrainData.map(lambda point: parseOHEPoint(point, ctrOHEDict, numCtrOHEFeats))
OHETrainData.cache()
print OHETrainData.take(1)

# Check that oneHotEncoding function was used in parseOHEPoint
backupOneHot = oneHotEncoding
oneHotEncoding = None
withOneHot = False
try: parseOHEPoint(rawTrainData.take(1)[0], ctrOHEDict, numCtrOHEFeats)
except TypeError: withOneHot = True
oneHotEncoding = backupOneHot

# TEST Apply OHE to the dataset (3d)
numNZ = sum(parsedTrainFeat.map(lambda x: len(x)).take(5))
numNZAlt = sum(OHETrainData.map(lambda lp: len(lp.features.indices)).take(5))
Test.assertEquals(numNZ, numNZAlt, 'incorrect implementation of parseOHEPoint')
Test.assertTrue(withOneHot, 'oneHotEncoding not present in parseOHEPoint')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(3e) Handling unseen features
We naturally would like to repeat the process from Part (3d), e.g., to compute OHE features for the validation and test datasets.  However, we must be careful, as some categorical values will likely appear in new data that did not exist in the training data. To deal with this situation, update the oneHotEncoding() function from Part (1d) to ignore previously unseen categories, and then compute OHE features for the validation data.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def oneHotEncoding(rawFeats, OHEDict, numOHEFeats):
    """Produce a one-hot-encoding from a list of features and an OHE dictionary.

    Note:
        If a (featureID, value) tuple doesn't have a corresponding key in OHEDict it should be
        ignored.

    Args:
        rawFeats (list of (int, str)): The features corresponding to a single observation.  Each
            feature consists of a tuple of featureID and the feature's value. (e.g. sampleOne)
        OHEDict (dict): A mapping of (featureID, value) to unique integer.
        numOHEFeats (int): The total number of unique OHE features (combinations of featureID and
            value).

    Returns:
        SparseVector: A SparseVector of length numOHEFeats with indicies equal to the unique
            identifiers for the (featureID, value) combinations that occur in the observation and
            with values equal to 1.0.
    """
    OHE = []
    for rawFeat in rawFeats:
        if rawFeat in OHEDict:
            OHE.append((OHEDict[rawFeat], 1.0))
    return SparseVector(numOHEFeats, OHE)

OHEValidationData = rawValidationData.map(lambda point: parseOHEPoint(point, ctrOHEDict, numCtrOHEFeats))
OHEValidationData.cache()
print OHEValidationData.take(1)

# TEST Handling unseen features (3e)
numNZVal = (OHEValidationData
            .map(lambda lp: len(lp.features.indices))
            .sum())
Test.assertEquals(numNZVal, 372080, 'incorrect number of features')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(4b) Log loss
Throughout this lab, we will use log loss to evaluate the quality of models.  Log loss is defined as: $$  \begin{align} \scriptsize \ell_{log}(p, y) = \begin{cases} -\log (p) & \text{if } y = 1 \\ -\log(1-p) & \text{if } y = 0 \end{cases} \end{align} $$ where $ \scriptsize p$ is a probability between 0 and 1 and $ \scriptsize y$ is a label of either 0 or 1. Log loss is a standard evaluation criterion when predicting rare-events such as click-through rate prediction (it is also the criterion used in the Criteo Kaggle competition).  Write a function to compute log loss, and evaluate it on some sample inputs.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
from math import log

def computeLogLoss(p, y):
    """Calculates the value of log loss for a given probabilty and label.

    Note:
        log(0) is undefined, so when p is 0 we need to add a small value (epsilon) to it
        and when p is 1 we need to subtract a small value (epsilon) from it.

    Args:
        p (float): A probabilty between 0 and 1.
        y (int): A label.  Takes on the values 0 and 1.

    Returns:
        float: The log loss value.
    """
    epsilon = 10e-12
    if y == 1:
        logLoss = -log(p + epsilon)
    else:
        logLoss = -log(1 - p + epsilon)
    return logLoss

print computeLogLoss(.5, 1)
print computeLogLoss(.5, 0)
print computeLogLoss(.99, 1)
print computeLogLoss(.99, 0)
print computeLogLoss(.01, 1)
print computeLogLoss(.01, 0)
print computeLogLoss(0, 1)
print computeLogLoss(1, 1)
print computeLogLoss(1, 0)

# TEST Log loss (4b)
Test.assertTrue(np.allclose([computeLogLoss(.5, 1), computeLogLoss(.01, 0), computeLogLoss(.01, 1)],
                            [0.69314718056, 0.0100503358535, 4.60517018599]),
                'computeLogLoss is not correct')
Test.assertTrue(np.allclose([computeLogLoss(0, 1), computeLogLoss(1, 1), computeLogLoss(1, 0)],
                            [25.3284360229, 1.00000008275e-11, 25.3284360229]),
                'computeLogLoss needs to bound p away from 0 and 1 by epsilon')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(4c)  Baseline log loss
Next we will use the function we wrote in Part (4b) to compute the baseline log loss on the training data. A very simple yet natural baseline model is one where we always make the same prediction independent of the given datapoint, setting the predicted value equal to the fraction of training points that correspond to click-through events (i.e., where the label is one). Compute this value (which is simply the mean of the training labels), and then use it to compute the training log loss for the baseline model.  The log loss for multiple observations is the mean of the individual log loss values.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
# Note that our dataset has a very high click-through rate by design
# In practice click-through rate can be one to two orders of magnitude lower
classOneFracTrain = OHETrainData.map(lambda x: x.label).mean()
print classOneFracTrain

logLossTrBase = OHETrainData.map(lambda x: computeLogLoss(classOneFracTrain, x.label)).mean()
print 'Baseline Train Logloss = {0:.3f}\n'.format(logLossTrBase)

# TEST Baseline log loss (4c)
Test.assertTrue(np.allclose(classOneFracTrain, 0.22717773523), 'incorrect value for classOneFracTrain')
Test.assertTrue(np.allclose(logLossTrBase, 0.535844), 'incorrect value for logLossTrBase')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(4d) Predicted probability
In order to compute the log loss for the model we trained in Part (4a), we need to write code to generate predictions from this model. Write a function that computes the raw linear prediction from this logistic regression model and then passes it through a sigmoid function $ \scriptsize \sigma(t) = (1+ e^{-t})^{-1} $ to return the model's probabilistic prediction. Then compute probabilistic predictions on the training data.
Note that when incorporating an intercept into our predictions, we simply add the intercept to the value of the prediction obtained from the weights and features.  Alternatively, if the intercept was included as the first weight, we would need to add a corresponding feature to our data where the feature has the value one.  This is not the case here.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
from math import exp #  exp(-t) = e^-t

def getP(x, w, intercept):
    """Calculate the probability for an observation given a set of weights and intercept.

    Note:
        We'll bound our raw prediction between 20 and -20 for numerical purposes.

    Args:
        x (SparseVector): A vector with values of 1.0 for features that exist in this
            observation and 0.0 otherwise.
        w (DenseVector): A vector of weights (betas) for the model.
        intercept (float): The model's intercept.

    Returns:
        float: A probability between 0 and 1.
    """
    rawPrediction = x.dot(w) + intercept

    # Bound the raw prediction value
    rawPrediction = min(rawPrediction, 20)
    rawPrediction = max(rawPrediction, -20)
    return 1.0/(1.0 + exp(-rawPrediction))

trainingPredictions = OHETrainData.map(lambda x: getP(x.features, model0.weights, model0.intercept))

print trainingPredictions.take(5)

# TEST Predicted probability (4d)
Test.assertTrue(np.allclose(trainingPredictions.sum(), 18135.4834348),
                'incorrect value for trainingPredictions')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(4e) Evaluate the model
We are now ready to evaluate the quality of the model we trained in Part (4a). To do this, first write a general function that takes as input a model and data, and outputs the log loss.  Then run this function on the OHE training data, and compare the result with the baseline log loss.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def evaluateResults(model, data):
    """Calculates the log loss for the data given the model.

    Args:
        model (LogisticRegressionModel): A trained logistic regression model.
        data (RDD of LabeledPoint): Labels and features for each observation.

    Returns:
        float: Log loss for the data.
    """

    def evaluateResult(model, point):
    # Calculates the log loss for the individual data points given the model.
        prob = getP(point.features, model.weights, model.intercept)
        return computeLogLoss(prob, point.label)

    return data.map(lambda x: evaluateResult(model, x)).mean()

logLossTrLR0 = evaluateResults(model0, OHETrainData)
print ('OHE Features Train Logloss:\n\tBaseline = {0:.3f}\n\tLogReg = {1:.3f}'
       .format(logLossTrBase, logLossTrLR0))

# TEST Evaluate the model (4e)
Test.assertTrue(np.allclose(logLossTrLR0, 0.456903), 'incorrect value for logLossTrLR0')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(4f) Validation log loss
Next, following the same logic as in Parts (4c) and 4(e), compute the validation log loss for both the baseline and logistic regression models. Notably, the baseline model for the validation data should still be based on the label fraction from the training dataset.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
logLossValBase = OHEValidationData.map(lambda x: computeLogLoss(classOneFracTrain, x.label)).mean()

logLossValLR0 = evaluateResults(model0, OHEValidationData)
print ('OHE Features Validation Logloss:\n\tBaseline = {0:.3f}\n\tLogReg = {1:.3f}'
       .format(logLossValBase, logLossValLR0))

# TEST Validation log loss (4f)
Test.assertTrue(np.allclose(logLossValBase, 0.527603), 'incorrect value for logLossValBase')
Test.assertTrue(np.allclose(logLossValLR0, 0.456957), 'incorrect value for logLossValLR0')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(5b) Creating hashed features
Next we will use this hash function to create hashed features for our CTR datasets. First write a function that uses the hash function from Part (5a) with numBuckets = $ \scriptsize 2^{15} \approx 33K $ to create a LabeledPoint with hashed features stored as a SparseVector.  Then use this function to create new training, validation and test datasets with hashed features. Hint: parsedHashPoint is similar to parseOHEPoint from Part (3d).
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
def parseHashPoint(point, numBuckets):
    """Create a LabeledPoint for this observation using hashing.

    Args:
        point (str): A comma separated string where the first value is the label and the rest are
            features.
        numBuckets: The number of buckets to hash to.

    Returns:
        LabeledPoint: A LabeledPoint with a label (0.0 or 1.0) and a SparseVector of hashed
            features.
    """
    label = point.split(',', 1)[0]
    features = parsePoint(point)
    hashfeatures = SparseVector(numBuckets, hashFunction(numBuckets, features))
    return LabeledPoint(label, hashfeatures)

numBucketsCTR = 2 ** 15
hashTrainData = rawTrainData.map(lambda x: parseHashPoint(x, numBucketsCTR))
hashTrainData.cache()
hashValidationData = rawValidationData.map(lambda x: parseHashPoint(x, numBucketsCTR))
hashValidationData.cache()
hashTestData = rawTestData.map(lambda x: parseHashPoint(x, numBucketsCTR))
hashTestData.cache()

print hashTrainData.take(1)

# TEST Creating hashed features (5b)
hashTrainDataFeatureSum = sum(hashTrainData
                           .map(lambda lp: len(lp.features.indices))
                           .take(20))
hashTrainDataLabelSum = sum(hashTrainData
                         .map(lambda lp: lp.label)
                         .take(100))
hashValidationDataFeatureSum = sum(hashValidationData
                                .map(lambda lp: len(lp.features.indices))
                                .take(20))
hashValidationDataLabelSum = sum(hashValidationData
                              .map(lambda lp: lp.label)
                              .take(100))
hashTestDataFeatureSum = sum(hashTestData
                          .map(lambda lp: len(lp.features.indices))
                          .take(20))
hashTestDataLabelSum = sum(hashTestData
                        .map(lambda lp: lp.label)
                        .take(100))

Test.assertEquals(hashTrainDataFeatureSum, 772, 'incorrect number of features in...


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
(5d) Logistic model with hashed features
Now let's train a logistic regression model using the hashed features. Run a grid search to find suitable hyperparameters for the hashed features, evaluating via log loss on the validation data. Note: This may take a few minutes to run. Use 1 and 10 for stepSizes and 1e-6 and 1e-3 for regParams.
Can you write Python code for it?

numIters = 500
regType = 'l2'
includeIntercept = True

# Initialize variables using values from initial model training
bestModel = None
bestLogLoss = 1e10

# TODO: Replace <FILL IN> with appropriate code
stepSizes = [1, 10]
regParams = [1e-6, 1e-3]
for stepSize in stepSizes:
    for regParam in regParams:
        model = (LogisticRegressionWithSGD
                 .train(hashTrainData, numIters, stepSize, regParam=regParam, regType=regType,
                        intercept=includeIntercept))
        logLossVa = evaluateResults(model, hashValidationData)
        print ('\tstepSize = {0:.1f}, regParam = {1:.0e}: logloss = {2:.3f}'
               .format(stepSize, regParam, logLossVa))
        if (logLossVa < bestLogLoss):
            bestModel = model
            bestLogLoss = logLossVa

print ('Hashed Features Validation Logloss:\n\tBaseline = {0:.3f}\n\tLogReg = {1:.3f}'
       .format(logLossValBase, bestLogLoss))

# TEST Logistic model with hashed features (5d)
Test.assertTrue(np.allclose(bestLogLoss, 0.4481683608), 'incorrect value for bestLogLoss')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
Part 2: Construct an OHE dictionary
(2a) Pair RDD of (featureID, category)
To start, create an RDD of distinct (featureID, category) tuples. In our sample dataset, the 7 items in the resulting RDD are (0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'), (1, 'tabby'), (2, 'mouse'), (2, 'salmon'). Notably 'black' appears twice in the dataset but only contributes one item to the RDD: (1, 'black'), while 'mouse' also appears twice and contributes two items: (0, 'mouse') and (2, 'mouse').  Use flatMap and distinct.
Can you write Python code for it?

# TODO: Replace <FILL IN> with appropriate code
sampleDistinctFeats = (sampleDataRDD
                       .flatMap(lambda x: x)
                       .distinct())

# TEST Pair RDD of (featureID, category) (2a)
Test.assertEquals(sorted(sampleDistinctFeats.collect()),
                  [(0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'),
                   (1, 'tabby'), (2, 'mouse'), (2, 'salmon')],
                  'incorrect value for sampleDistinctFeats')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
Part 3: Parse CTR data and generate OHE features
Before we can proceed, you'll first need to obtain the data from Criteo.  If you have already completed this step in the setup lab, just run the cells below and the data will be loaded into the rawData variable.
Below is Criteo's data sharing agreement.  After you accept the agreement, you can obtain the download URL by right-clicking on the "Download Sample" button and clicking "Copy link address" or "Copy Link Location", depending on your browser.  Paste the URL into the # TODO cell below.  The file is 8.4 MB compressed.  The script below will download the file to the virtual machine (VM) and then extract the data.
If running the cell below does not render a webpage, open the Criteo agreement in a separate browser tab.  After you accept the agreement, you can obtain the download URL by right-clicking on the "Download Sample" button and clicking "Copy link address" or "Copy Link Location", depending on your browser.  Paste the URL into the # TODO cell below.
Note that the download could take a few minutes, depending upon your connection speed.
Can you write Python code for it?

# Run this code to view Criteo's agreement
from IPython.lib.display import IFrame

IFrame("http://labs.criteo.com/downloads/2014-kaggle-display-advertising-challenge-dataset/",
       600, 350)

# TODO: Replace <FILL IN> with appropriate code
# Just replace <FILL IN> with the url for dac_sample.tar.gz
import glob
import os.path
import tarfile
import urllib
import urlparse

# Paste url, url should end with: dac_sample.tar.gz
url = '<FILL IN>'

url = url.strip()
baseDir = os.path.join('data')
inputPath = os.path.join('cs190', 'dac_sample.txt')
fileName = os.path.join(baseDir, inputPath)
inputDir = os.path.split(fileName)[0]

def extractTar(check = False):
    # Find the zipped archive and extract the dataset
    tars = glob.glob('dac_sample*.tar.gz*')
    if check and len(tars) == 0:
      return False

    if len(tars) > 0:
        try:
            tarFile = tarfile.open(tars[0])
        except tarfile.ReadError:
            if not check:
                print 'Unable to open tar.gz file.  Check your URL.'
            return False

        tarFile.extract('dac_sample.txt', path=inputDir)
        print 'Successfully extracted: dac_sample.txt'
        return True
    else:
        print 'You need to retry the download with the correct url.'
        print ('Alternatively, you can upload the dac_sample.tar.gz file to your Jupyter root ' +
              'directory')
        return False

if os.path.isfile(fileName):
    print 'File is already available. Nothing to do.'
elif extractTar(check = True):
    print 'tar.gz file was already available.'
elif not url.endswith('dac_sample.tar.gz'):
    print 'Check your download url.  Are you downloading the Sample dataset?'
else:
    # Download the file and store it in the same directory as this notebook
    try:
        urllib.urlretrieve(url, os.path.basename(urlparse.urlsplit(url).path))
    except IOError:
        print 'Unable to download and store: {0}'.format(url)

    extractTar()

import os.path
baseDir = os.path.join('data')
inputPath = os.path.join('cs190',...


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
Part 4: CTR prediction and logloss evaluation
 (4a) Logistic regression
We are now ready to train our first CTR classifier.  A natural classifier to use in this setting is logistic regression, since it models the probability of a click-through event rather than returning a binary response, and when working with rare events, probabilistic predictions are useful.  First use LogisticRegressionWithSGD to train a model using OHETrainData with the given hyperparameter configuration.  LogisticRegressionWithSGD returns a LogisticRegressionModel.  Next, use the LogisticRegressionModel.weights and LogisticRegressionModel.intercept attributes to print out the model's parameters.  Note that these are the names of the object's attributes and should be called using a syntax like model.weights for a given model.
Can you write Python code for it?

from pyspark.mllib.classification import LogisticRegressionWithSGD

# fixed hyperparameters
numIters = 50
stepSize = 10.
regParam = 1e-6
regType = 'l2'
includeIntercept = True

# TODO: Replace <FILL IN> with appropriate code
model0 = LogisticRegressionWithSGD.train(OHETrainData, numIters, stepSize,
                                         regParam = regParam,
                                         regType = regType,
                                         intercept = includeIntercept)
sortedWeights = sorted(model0.weights)
print sortedWeights[:5], model0.intercept

# TEST Logistic regression (4a)
Test.assertTrue(np.allclose(model0.intercept,  0.56455084025), 'incorrect value for model0.intercept')
Test.assertTrue(np.allclose(sortedWeights[0:5],
                [-0.45899236853575609, -0.37973707648623956, -0.36996558266753304,
                 -0.36934962879928263, -0.32697945415010637]), 'incorrect value for model0.weights')


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
Visualization 1: Feature frequency
We will now visualize the number of times each of the 233,286 OHE features appears in the training data. We first compute the number of times each feature appears, then bucket the features by these counts.  The buckets are sized by powers of 2, so the first bucket corresponds to features that appear exactly once ( $ \scriptsize 2^0 $ ), the second to features that appear twice ( $ \scriptsize 2^1 $ ), the third to features that occur between three and four ( $ \scriptsize 2^2 $ ) times, the fifth bucket is five to eight ( $ \scriptsize 2^3 $ ) times and so on. The scatter plot below shows the logarithm of the bucket thresholds versus the logarithm of the number of features that have counts that fall in the buckets.
Can you write Python code for it?

def bucketFeatByCount(featCount):
    """Bucket the counts by powers of two."""
    for i in range(11):
        size = 2 ** i
        if featCount <= size:
            return size
    return -1

featCounts = (OHETrainData
              .flatMap(lambda lp: lp.features.indices)
              .map(lambda x: (x, 1))
              .reduceByKey(lambda x, y: x + y))
featCountsBuckets = (featCounts
                     .map(lambda x: (bucketFeatByCount(x[1]), 1))
                     .filter(lambda (k, v): k != -1)
                     .reduceByKey(lambda x, y: x + y)
                     .collect())
print featCountsBuckets

import matplotlib.pyplot as plt

x, y = zip(*featCountsBuckets)
x, y = np.log(x), np.log(y)

def preparePlot(xticks, yticks, figsize=(10.5, 6), hideLabels=False, gridColor='#999999',
                gridWidth=1.0):
    """Template for generating the plot layout."""
    plt.close()
    fig, ax = plt.subplots(figsize=figsize, facecolor='white', edgecolor='white')
    ax.axes.tick_params(labelcolor='#999999', labelsize='10')
    for axis, ticks in [(ax.get_xaxis(), xticks), (ax.get_yaxis(), yticks)]:
        axis.set_ticks_position('none')
        axis.set_ticks(ticks)
        axis.label.set_color('#999999')
        if hideLabels: axis.set_ticklabels([])
    plt.grid(color=gridColor, linewidth=gridWidth, linestyle='-')
    map(lambda position: ax.spines[position].set_visible(False), ['bottom', 'top', 'left', 'right'])
    return fig, ax

# generate layout and plot data
fig, ax = preparePlot(np.arange(0, 10, 1), np.arange(4, 14, 2))
ax.set_xlabel(r'$\log_e(bucketSize)$'), ax.set_ylabel(r'$\log_e(countInBucket)$')
plt.scatter(x, y, s=14**2, c='#d6ebf2', edgecolors='#8cbfd0', alpha=0.75)
pass


I am working on the file "ML_lab4_ctr_student.ipynb".
The first task is:
Visualization 3: Hyperparameter heat map
We will now perform a visualization of an extensive hyperparameter search.  Specifically, we will create a heat map where the brighter colors correspond to lower values of logLoss.
The search was run using six step sizes and six values for regularization, which required the training of thirty-six separate models.  We have included the results below, but omitted the actual search to save time.
Can you write Python code for it?


I am working on the file "ML_test.ipynb".
The first task is:
Methodology
As a start, we test all algorithms for applicability. Make an indicator for raise or fall. From one day to the next. In these cases the model is created on 0.8 of the previous years data and then tested on 20% of the data. A walk forward optimization is performed, using new train and test data. In the case of XGboost, walk forward optimization has not been performed as it is too computationally expensive.
The technical indicators which have been selected are:
'meanfractal'
'RStok0'
'mom'
'MACD_l'
These were chosen from running all indicators through xgboost and selecting those which contribute the most. See the file xgboost_test for the methodology.
Predicting the day thats been
Using the technical indicators given above, the following machine learning algorithms have been used. KNN, Linear regression, Gaussian Naive Bayes, Random forest and gradient boosting decision trees.
Can you write Python code for it?

# First try a knn model.
m = []
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].knn_learn(6, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH")
knn = ret.ACC.mean()
# KNN model accuracy: 0.82500665779

# Now try a linear classifier model
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].linear_learn(1, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH")
ll = ret.ACC.mean()
# Linear model accuracy: 0.910972037284

# Try a Gaussian interpretation of the Naive Byers
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].GaussianNB(252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH")
gnb = ret.ACC.mean()
# GaussianNB model accuracy: 0.89650244119

# Try a random forrest
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].randomForset_learn(3, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH")
rf = ret.ACC.mean()
# RandomForest model accuracy: 0.932871726587

# Try an xgboosted random forrest
m.append(models.ML(pred.props))
temp = make_logic()
sd = temp.index[252]
ret = m[-1].short_xgboost_model(sd,252*2,temp)
XB = ret.ACC.mean()
# Xgboost model accuracy: 0.9505

# The computation time for the SCV model is too great for my computer.
do_svc = False
if do_svc:
    kernels = {'linear', 'poly', 'rbf', 'sigmoid'}
    for kernel in kernels:
        m.append(models.ML(pred.props))
        m[-1].pred.PHH = ((1 - pred.data.Close.shift(1).div(pred.data.Close)) > 0) * 1
        ret = m[-1].SVC_learn(252, c=1, p={'meanfractal', 'Stok0', 'mom', 'MACD_l'}, y_colums = "PHH",kernel=kernel)


I am working on the file "ML_test.ipynb".
The first task is:
Now we have an array of back projected results for different price fluctuations. Lets find out the most probable. This should be fitted with a Heaviside function. But since we're backtesting over 3000 points, we'll let it slide and use the poor mans version. Note, this means that if one computed point is wrong we could get a drop when we don't mean to. Similarly, if it's all equal to (0, 1) we have an erroneous value.
Can you write Python code for it?

# This is what we are predicting.
change = pd.DataFrame({
    "Actual": actual,
    "Pred"  : 0})

for i in range(1,len(ret)):
    change["Pred"] = change["Pred"] + ((ret[i-1].PRED > ret[i].PRED)*fluc[i])

change.to_csv('ml_predictions_all.csv', sep=',')
change = pd.read_csv('ml_predictions_all.csv' ,index_col=0, parse_dates=True)

gs = gridspec.GridSpec(2, 1, height_ratios=[2, 1])
ax1 = plt.subplot(gs[0])
corr_data.Close.plot(label="SPY")
ax2 = plt.subplot(gs[1],sharex=ax1)
change["Actual"].plot(label="SPY Change")
change["Pred"].plot(label="Predicted Change")
ax1.set_ylabel("Share Price [$]")
ax2.set_xlabel("Time")
ax2.set_ylabel("Percenage Change")
plt.title("RandomForestClassifier")
plt.show()

display(Image(filename="Images/Full_prediction.png"))
display(Image(filename="Images/Cropped_prediction.png"))


I am working on the file "ML_test.ipynb".
The first task is:
Now, because of commission (C) just saying profit or loss is not enough. We need to say if tomorrow we can get above the break even point. This occurs when the percentage increase is greater than 1+C/1-C -1. So now lets test  the models against this criteria.
Can you write Python code for it?

mm = []
be = {"knn": [], "ll": [], "gnb": [], "rf": [],"XB": []}
for i in range(0, 3):
    mm.append(models.ML(pred.props))
    mm[-1].pred.PHH = make_logic(0.0051)
    ret = mm[-1].knn_learn(6, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=i)
    be['knn'].append(ret.ACC.mean())

    mm.append(models.ML(pred.props))
    mm[-1].pred.PHH = make_logic(0.0051)
    ret = mm[-1].linear_learn(1, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=i)
    be['ll'].append(ret.ACC.mean())

    mm.append(models.ML(pred.props))
    mm[-1].pred.PHH = make_logic(0.0051)
    ret = mm[-1].GaussianNB(252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=i)
    be['gnb'].append(ret.ACC.mean())

    mm.append(models.ML(pred.props))
    mm[-1].pred.PHH = make_logic(0.0051)
    ret = mm[-1].randomForset_learn(3, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=i)
    be['rf'].append(ret.ACC.mean())

    mm.append(models.ML(pred.props))
    temp = make_logic()
    sd = temp.index[252]
    ret = mm[-1].short_xgboost_model(sd,252*1,temp,forward_look=i)
    be['XB'].append(ret.ACC.mean())

# Now plot the accuracy. Well, this is surprising!
ModelsAccuracyP = pd.DataFrame(be)
ModelsAccuracyP = ModelsAccuracyP.transpose()
ModelsAccuracyP.columns = ["0 Day", '1 Day', '2 Day']
ax = ModelsAccuracyP.plot.bar()
ax.set_xticklabels(["KNN", "Lin", "GNB", "RF", "XGB"])
ax.set_ylabel("Accuracy")
plt.show()

display(Image(filename="Images/AccVDayF.png"))


I am working on the file "ML_test.ipynb".
The first task is:
OK, so Random Forrest seems to be the best. Now lets see if it can model the percentage change. We perform models for a -1.5% to 2% change as this accounts for most of the daily fluctuations.
Can you write Python code for it?

fluc_m = []
fluc = np.arange(-.015, 0.02, 0.001)
ret = []
acc = []
for x in np.nditer(fluc):
    fluc_m.append(models.ML(pred.props))
    fluc_m[-1].pred.PHH = make_logic(x)
    # ret.append(fluc_m[-1].GaussianNB(252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=1))
    ret.append(fluc_m[-1].randomForset_learn(3,252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=1))
    acc.append(ret[-1].ACC.mean())

actual = (1 - pred.data.Close.shift(1).div(pred.data.Close))
# Let's test this signal
i = 10 # A random date to inspect.
signal = [ret[j].PRED[i] for j in range(0,len(ret))]
plt.plot(fluc,signal,label='Signal')
plt.axvline(actual[i-1],color='r',label='Real')
ax = plt.gca()
ax.set_ylabel("Random Forest Prediction")
ax.set_xlabel('Price Change')
plt.show()


I am working on the file "ML_test.ipynb".
The first task is:
Predicting the future - 2 Days
Can you write Python code for it?

m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].knn_learn(6, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=2)
knnL2 = ret.ACC.mean()
# KNN model accuracy: 0.570910706353

# Now try a linear classifier model
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].linear_learn(1, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=2)
llL2 = ret.ACC.mean()
# Linear model accuracy: 0.625890715238

# Try a Gaussian interpretation of the Naive Byers
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].GaussianNB(252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=2)
gnbL2 = ret.ACC.mean()
# GaussianNB model accuracy: 0.628342958685

# Try a random forrest
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].randomForset_learn(3, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=2)
rfL2 = ret.ACC.mean()
# RandomForest model accuracy: 0.558960462017

# Try an xgboosted random forrest
m.append(models.ML(pred.props))
temp = make_logic()
sd = temp.index[252]
ret = m[-1].short_xgboost_model(sd,252*1,temp,forward_look=2)
XBL2 = ret.ACC.mean()
# Xgboost model accuracy: 0.58

# Lets visualise this
ModelsAccuracy = pd.DataFrame({"0 Day": [knn, ll, gnb, rf, XBL],
                               "1 Day": [knnL, llL, gnbL, rfL, XBL],
                               "2 Days": [knnL2, llL2, gnbL2, rfL2, XBL2]})
ax = ModelsAccuracy.plot.bar()
ax.set_xticklabels(["KNN", "Lin", "GNB", "RF", "XGB"])
ax.set_ylabel("Accuracy")
plt.show()

display(Image(filename="Images/AccVDay.png"))


I am working on the file "ML_test.ipynb".
The first task is:
So with all of these models, it looks like it's a Random Forest model or an xboost model.
Predicting the future - 1 day
Can you write Python code for it?

# Now lets try looking forward 1 day.
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].knn_learn(6, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=1)
knnL = ret.ACC.mean()
# KNN model accuracy: 0.666731793961

# Now try a linear classifier model
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].linear_learn(1, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=1)
llL = ret.ACC.mean()
# Linear model accuracy: 0.728925399645

# Try a Gaussian interpretation of the Naive Byers
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].GaussianNB(252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=1)
gnbL = ret.ACC.mean()
# GaussianNB model accuracy: 0.717744227353

# Try a random forrest
m.append(models.ML(pred.props))
m[-1].pred.PHH = make_logic()
ret = m[-1].randomForset_learn(3, 252, {'meanfractal', 'RStok0', 'mom', 'MACD_l'}, "PHH",forward_look=1)
rfL = ret.ACC.mean()
# RandomForest model accuracy: 0.655959147425

# Try an xgboosted random forrest
m.append(models.ML(pred.props))
temp = make_logic()
sd = temp.index[252]
ret = m[-1].short_xgboost_model(sd,252*1,temp,forward_look=1)
XBL = ret.ACC.mean()
# Predicted model accuracy: 0.699


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Computing the Forcing Functions for the MMS (balancing discrete residuals)
Can you write Python code for it?

S_inv = zeros(neq,1)
S_vis = zeros(neq,1)
S     = zeros(neq,1)

for i in range(0,neq):
    S_inv[i] = Div_F_inv[i]
    if NS>0:
        S_vis[i] = Div_F_vis[i]

    S[i] = S_inv[i]+S_vis[i]

    if NS==2 and i==neq-1:
        S[i] = S[i]+S_SA


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Computing the Viscous Flux divergence:
Can you write Python code for it?

#Compute the divergence the long way:
if NS>0:
    Div_F_vis    = zeros(neq,1)
    for i in range(0,neq):
        Div_F_vis[i] = diff(vis_F_orig[i,0],x)+diff(vis_F_orig[i,1],y)
    Div_F_vis    = simplify(Div_F_vis)

if NS>0:
    Div_F_vis = subs_derivs_by_d2wdxy(Div_F_vis)

    Div_F_vis = subs_derivs_by_dwdx(Div_F_vis)
    Div_F_vis = subs_derivs_by_dwdy(Div_F_vis)


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Computing the viscous Flux
Can you write Python code for it?


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Heat Flux
Can you write Python code for it?

if NS>0:
    q = simplify(k_eff*Matrix([dTdx, dTdy]))


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Inviscid Fluxes
Can you write Python code for it?

inv_F_orig=Matrix([ [rho*u,       rho*v       ],
                    [rho*u**2+p,  rho*u*v     ],
                    [rho*v*u,     rho*v**2+p  ],
                    [(rho*E+p)*u, (rho*E+p)*v ],
                    [(rho*ntl)*u, (rho*ntl)*v ],])
inv_F = inv_F_orig


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Manufactured Solutions
Trigonometric MS
Can you write Python code for it?

print ("The Manufactured Solution Number -",MS_no, "- is activated")

if MS_no in range(1,6):

    rho_0   = symbols('rho_0')
    rho_x   = symbols('rho_x')
    rho_y   = symbols('rho_y')
    rho_xy  = symbols('rho_xy')
    u_0     = symbols('u_0')
    u_x     = symbols('u_x')
    u_y     = symbols('u_y')
    u_xy    = symbols('u_xy')
    v_0     = symbols('v_0')
    v_x     = symbols('v_x')
    v_y     = symbols('v_y')
    v_xy    = symbols('v_xy')
    p_0     = symbols('p_0')
    p_x     = symbols('p_x')
    p_y     = symbols('p_y')
    p_xy    = symbols('p_xy')

    a_rhox  = symbols('a_rhox')
    a_rhoy  = symbols('a_rhoy')
    a_rhoxy = symbols('a_rhoxy')
    a_ux    = symbols('a_ux')
    a_uy    = symbols('a_uy')
    a_uxy   = symbols('a_uxy')
    a_vx    = symbols('a_vx')
    a_vy    = symbols('a_vy')
    a_vxy   = symbols('a_vxy')
    a_px    = symbols('a_px')
    a_py    = symbols('a_py')
    a_pxy   = symbols('a_pxy')
    L       = symbols('L')

    # SA
    nu_sa_0  =  symbols('nu_sa_0')
    nu_sa_x  =  symbols('nu_sa_x')
    nu_sa_y  =  symbols('nu_sa_y')
    nu_sa_xy =  symbols('nu_sa_xy')
    nu_sa_t  =  symbols('nu_sa_t')
    a_nusay  =  symbols('a_nusay')
    a_nusax  =  symbols('a_nusax')
    a_nusaxy =  symbols('a_nusaxy')
    a_nusat  =  symbols('a_nusat')

    MS_RHO = rho_0   + rho_x   * sin(a_rhox  * pi * x / L)  + rho_y   * cos(a_rhoy  * pi * y / L)   + rho_xy   * cos(a_rhoxy * pi * x / L)*cos(a_rhoxy * pi * y / L);
    MS_U   = u_0     + u_x     * sin(a_ux    * pi * x / L)  + u_y     * cos(a_uy    * pi * y / L)   + u_xy     * cos(a_uxy   * pi * x / L)*cos(a_uxy   * pi * y / L);
    MS_V   = v_0     + v_x     * cos(a_vx    * pi * x / L)  + v_y     * sin(a_vy    * pi * y / L)   + v_xy     * cos(a_vxy   * pi * x / L)*cos(a_vxy   * pi * y / L);
    MS_P   = p_0     + p_x     * cos(a_px    * pi * x / L)  + p_y     * sin(a_py    * pi * y / L)   + p_xy     * cos(a_pxy   * pi * x / L)*cos(a_pxy   * pi * y /...


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Printing code in C for the derivatives of the primitive variables
Can you write Python code for it?

#Compute the MS derivatives

dEdx     = diff(MS_E  ,x)
dEdy     = diff(MS_E  ,y)
d2Edxx   = diff(MS_E  ,x,x)
d2Edyy   = diff(MS_E  ,y,y)

dRHOdx   = diff(MS_RHO,x)
dRHOdy   = diff(MS_RHO,y)
d2RHOdxx = diff(MS_RHO,x,x)
d2RHOdyy = diff(MS_RHO,y,y)

dUdx     = diff(MS_U  ,x)
dUdy     = diff(MS_U  ,y)
d2Udxx   = diff(MS_U  ,x,x)
d2Udxy   = diff(MS_U  ,x,y)
d2Udyy   = diff(MS_U  ,y,y)

dVdx     = diff(MS_V  ,x)
dVdy     = diff(MS_V  ,y)
d2Vdxx   = diff(MS_V  ,x,x)
d2Vdxy   = diff(MS_V  ,x,y)
d2Vdyy   = diff(MS_V  ,y,y)

dPdx     = diff(MS_P  ,x)
dPdy     = diff(MS_P  ,y)
d2Pdxx   = diff(MS_P  ,x,x)
d2Pdyy   = diff(MS_P  ,y,y)

if NS==2:
    dNTLdx   = diff(MS_NTL,x)
    dNTLdy   = diff(MS_NTL,y)
    d2NTLdxx = diff(MS_NTL,x,x)
    d2NTLdyy = diff(MS_NTL,y,y)

# State variables
s = ccode(MS_RHO,assign_to='RHO')
s = Cout_cleanup(s)
print (s,"\n\n")

# First degree derivatives
s = ccode(dRHOdx,assign_to='dRHOdx')
s = Cout_cleanup(s)
print (s,"\n\n")

s = ccode(dRHOdy,assign_to='dRHOdy')
s = Cout_cleanup(s)
print (s,"\n\n")

if (NS>0):
    s = ccode(d2RHOdxx,assign_to='d2RHOdxx')
    s = Cout_cleanup(s)
    #print len(s),"\n"
    print (s,"\n\n")

    s = ccode(d2RHOdyy,assign_to='d2RHOdyy')
    s = Cout_cleanup(s)
    #print len(s),"\n"
    print (s,"\n\n")

s = ccode(MS_U,assign_to='U')
s = Cout_cleanup(s)
print (s,"\n\n")

s = ccode(dUdx,assign_to='dUdx')
s = Cout_cleanup(s)
print (s,"\n\n")

s = ccode(dUdy,assign_to='dUdy')
s = Cout_cleanup(s)
print (s,"\n\n")

if (NS>0):
    s = ccode(d2Udxx,assign_to='d2Udxx')
    s = Cout_cleanup(s)
    print (s,"\n\n")

    s = ccode(d2Udxy,assign_to='d2Udxy')
    s = Cout_cleanup(s)
    print (s,"\n\n")

    s = ccode(d2Udyy,assign_to='d2Udyy')
    s = Cout_cleanup(s)
    print (s,"\n\n")

s = ccode(MS_V,assign_to='V')
s = Cout_cleanup(s)
print (s,"\n\n")

s = ccode(dVdx,assign_to='dVdx')
s = Cout_cleanup(s)
print (s,"\n\n")

s = ccode(dVdy,assign_to='dVdy')
s = Cout_cleanup(s)
print (s,"\n\n")

if (NS>0):
    s = ccode(d2Vdxx,assign_to='d2Vdxx')
    s = Cout_cleanup(s)
 ...


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
SA source terms
Can you write Python code for it?

def SA_Sources(NS,SA,SA_Sbar,lam_sup):
    if NS==2:
        # interm vars
        dtp2   = d*d
        oosig  = 1/sig
        Ktp2   = K*K
        cw3tp6 = pow(cw3,6)
        chitp2 = pow(chi,2)
        chitp3 = pow(chi,3)

        fv2  = 1-(chi/(1+chi*fv1))
        S    = abs((dvdx-dudy))
        Sbar = fv2*ntl/((Ktp2)*(dtp2))
        #Laminar suppression and trip terms:
        if lam_sup!=0:
            ft2  = ct3*exp(-ct4*chitp2)
            Trp  = 0
            print ("Laminar suppression is considered, but no trip term")
        else:
            ft2  = 0
            Trp  = 0

    # The modified vorticity: Stl (S_tilda)
    # the condition on the S_bar (and modified vorticity) is not directly dependent on the sign of ntl
    # it is rather based on the numerical condition (Sbar>=-cv2*S).

        if SA_Sbar>0:
            Stl = S + Sbar;
        elif SA_Sbar<0:
            Stl = S + (S*(cv2*cv2*S+cv3*Sbar))/(S*(cv3-2*cv2)-Sbar)

        # The Production term
        if(SA>0):
            P = cb1*Stl*ntl*(1-ft2)
        elif SA<0:
            P = cb1*S*ntl*(1-ct3)

        # The Destruction term
        if SA>0:
            r    = Min(ntl/(Stl*Ktp2*dtp2),rlim)
            rtp6 = pow(r,6)
            g    = r + cw2*(rtp6-r)
            gtp6 = pow(g,6)
            fw   = g*pow((1+cw3tp6)/(gtp6+cw3tp6),1/6)
            D    = (cw1*fw-cb1*ft2/(Ktp2))*(ntl*ntl)/(dtp2)
        # Negative SA
        elif SA<0:
            D    = -cw1*(ntl*ntl)/(dtp2)

        # fn function is defined earlier

        # The "Distribution" term
        trm1 = cb2/sig*(dntldx*dntldx+dntldy*dntldy);

        # The density counterpart of the conservative form
        trm2 = oosig*(nu+ntl*fn)*(drhodx*dntldx+drhody*dntldy);

        #src  = rho*(-P+D-Trp)-rho*trm1+trm2;
        S_SA_src_product = nsimplify(  rho*(-P)   )
        S_SA_src_destruc = nsimplify(  rho*(+D)   )
        S_SA_src_distrib = nsimplify( -rho*(trm1) )
        S_SA_src_conserv = nsimplify(  trm2       )

       ...


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
SWITCHES
Choosing the MS
Can you write Python code for it?

MS_no = 5

#1 : Inviscid flows in subsonic regime - MS-1
if MS_no==1:
    NS = 0
#2 : Inviscid flows in supersonic regime - MS-2
if MS_no==2:
    NS = 0
#3 : Laminar flows - MS-3
if MS_no==3:
    NS = 1
#4 : Turbulent flows - MS-4 (original SA)
if MS_no==4:
    NS      =  2
    SA      = +1
    SA_Sbar = +1
    lam_sup =  1
#5 : Turbulent flows - MS-5 (negative SA)
if MS_no==5:
    NS      =  2
    SA      = -1
    SA_Sbar = +1
    lam_sup =  1

# Legend:
# ----------------------------------------------------------------------------
# NS::      (NS = 0) Euler || (NS = 1) NS || (NS = 2) RANS
# SA::      (SA = -1) Negative SA and ntl<0 || (SA = +1) Original SA and 0<=ntl
# lam_sup:: (lam_sup = 0) No laminar suppression term || (lam_sup = 1) Laminar suppression considered (ft2)
# SA_Sbar:: (SA_Sbar = -1) Negative SA  || (SA_Sbar = +1): Original SA

dim  = 2 # number of dimensions
if NS<=1:
    neq = dim+2
elif NS==2:
    neq = dim+3


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Spalart-Allmaras variables
Can you write Python code for it?

if (NS>=2):
    nu   = (1/W[0])*mu
    chi  = ntl/nu
    fv1  = pow(chi,3)/(pow(chi,3)+pow(cv1,3))
    mut  = rho*ntl*fv1
    kt   = Cp*mut/Prt


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
The Shear Stress definition
Can you write Python code for it?

Tau = zeros(dim,dim)
if NS>0:
    Tau = 2*mu_eff*Matrix([ [diff(u,x)-1/3*(diff(u,x)+diff(v,y)), 1/2*(diff(v,x)+diff(u,y))            ],
                          [1/2*(diff(u,y)+diff(v,x))            , diff(v,y)-1/3*(diff(u,x)+diff(v,y))  ]])
    Tau = simplify(Tau)
    Tau = nsimplify(Tau)
    Tau


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
The Temperature and its derivatives
Can you write Python code for it?

if NS>0:
    T = e/Cv
    dTdx = simplify(diff(T,x))
    dTdy = simplify(diff(T,y))


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Useful Functions
Can you write Python code for it?

# Function for cleaning up the C output
def Cout_cleanup(s):
    s = s.replace("rho(x, y)","RHO")
    s = s.replace("u(x, y)","U")
    s = s.replace("v(x, y)","V")
    s = s.replace("W(x, y)","W")
    s = s.replace("E(x, y)","E")
    s = s.replace("p(x, y)","P")
    s = s.replace("ntl(x, y)","NTL")

    s = s.replace("M_PI","piMS")
    s = s.replace("gamma","Gamma")
    s = s.replace("Min","MIN")

    s = s.replace("Derivative(MS_RHO(x, y), x)",   "dRHOdx")
    s = s.replace("Derivative(MS_RHO(x, y), x, x)","d2RHOdxx")
    s = s.replace("Derivative(MS_RHO(x, y), y)",   "dRHOdy")
    s = s.replace("Derivative(MS_RHO(x, y), y, y)","d2RHOdyy")
    s = s.replace("MS_RHO(x, y)","RHO")

    s = s.replace("Derivative(MS_P(x, y), x)",     "dPdx")
    s = s.replace("Derivative(MS_P(x, y), x, x)",  "d2Pdxx")
    s = s.replace("Derivative(MS_P(x, y), y)",     "dPdy")
    s = s.replace("Derivative(MS_P(x, y), y, y)",  "d2Pdyy")
    s = s.replace("MS_P(x, y)","P")

    s = s.replace("Derivative(MS_U(x, y), x)",     "dUdx")
    s = s.replace("Derivative(MS_U(x, y), x, x)",  "d2Udxx")
    s = s.replace("Derivative(MS_U(x, y), y)",     "dUdy")
    s = s.replace("Derivative(MS_U(x, y), y, y)",  "d2Udyy")
    s = s.replace("MS_U(x, y)","U")

    s = s.replace("Derivative(MS_V(x, y), x)",     "dVdx")
    s = s.replace("Derivative(MS_V(x, y), x, x)",  "d2Vdxx")
    s = s.replace("Derivative(MS_V(x, y), y)",     "dVdy")
    s = s.replace("Derivative(MS_V(x, y), y, y)",  "d2Vdyy")
    s = s.replace("MS_V(x, y)","V")
    return(s)

# Functions to substitute derivative of primitive variables by intermediate variable so the final expression is simpler
#                                                                         (broken to 2 expressions dwdx and S(w,dwdx))

def subs_derivs_by_dwdx(var):
    var = var.subs(diff(rho,x)  ,drhodx)
    var = var.subs(diff(u  ,x)  ,dudx)
    var =...


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
Viscous Fluxes
Can you write Python code for it?


I am working on the file "MMS_FreeFlow_Euler-NS-RANS-SA_arXiv_1712.09478.ipynb".
The first task is:
negative SA's 'Fn' function
Can you write Python code for it?

# Depending on the sign of ntl, there are two possibilities:
if (NS==2):
    if (SA>0):
        fn = 1
    else:
    #for ntl<0
        fn = (cn1+pow(chi,3))/(cn1-pow(chi,3))


I am working on the file "MMW_API_landproperties_demo.ipynb".
The first task is:
3. Issue job requests, fetch job results when done, then examine results. Repeat for each request type
Can you write Python code for it?

# convenience function, to simplify the request calls, below
def analyze_api_request(api_name, s, api_url, json_payload):
    post_url = "{}analyze/{}/".format(api_url, api_name)
    post_req = requests_retry_session(session=s).post(post_url, data=json_payload)
    jobrequest_json = json.loads(post_req.content)
    # Fetch and examine job result
    result = get_job_result(api_url, s, jobrequest_json)
    return result


I am working on the file "MMW_API_landproperties_demo.ipynb".
The first task is:
Issue job request: analyze/climate/
Can you write Python code for it?

result = analyze_api_request('climate', s, api_url, json_payload)


I am working on the file "MMW_API_landproperties_demo.ipynb".
The first task is:
Issue job request: analyze/land/
Can you write Python code for it?

result = analyze_api_request('land', s, api_url, json_payload)


I am working on the file "MMW_API_landproperties_demo.ipynb".
The first task is:
Issue job request: analyze/terrain/
Can you write Python code for it?

result = analyze_api_request('terrain', s, api_url, json_payload)


I am working on the file "MMW_API_landproperties_demo.ipynb".
The first task is:
MMW production API endpoint base url.
Can you write Python code for it?


I am working on the file "MMW_API_landproperties_demo.ipynb".
The first task is:
The job is not completed instantly and the results are not returned directly by the API request that initiated the job. The user must first issue an API request to confirm that the job is complete, then fetch the results. The demo presented here performs automated retries (checks) until the server confirms the job is completed, then requests the JSON results and converts (deserializes) them into a Python dictionary.
Can you write Python code for it?

def get_job_result(api_url, s, jobrequest):
    url_tmplt = api_url + "jobs/{job}/"
    get_url = url_tmplt.format

    result = ''
    while not result:
        get_req = requests_retry_session(session=s).get(get_url(job=jobrequest['job']))
        result = json.loads(get_req.content)['result']

    return result

s = requests.Session()

APIToken = 'Token 0501d9a98b8170a41d57df8ce82c000c477c621a'  # HIDE THE API TOKEN

s.headers.update({
    'Authorization': APIToken,
    'Content-Type': 'application/json'
})


I am working on the file "MMW_API_landproperties_demo.ipynb".
The first task is:
result is a dictionary with one item, survey. This item in turn is a dictionary with 3 items: displayName, name, categories. The first two are just labels. The data are in the categories item.
Can you write Python code for it?

categories = result['survey']['categories']

len(categories), categories

[d for d in categories if d['type'] == 'average']


result['survey'].keys()

categories = result['survey']['categories']

len(categories), categories[1]

land_categories_nonzero = [d for d in categories if d['coverage'] > 0]

land_categories_nonzero


I am working on the file "MMW_API_watershed_demo.ipynb".
The first task is:
2. Construct and issue the job request
Parameters passed to the RWD ("watershed") API request. This is a point in Salt Lake City
Can you write Python code for it?

payload = {
    'location': [40.746054, -111.847987], # [latitude, longitude]
    'snappingOn': True,
    'dataSource': 'nhd'}

json_dat = json.dumps(payload)

post_req = requests_retry_session(session=s).post(post_url, data=json_dat)

json_out = json.loads(post_req.content)
json_out


I am working on the file "MMW_API_watershed_demo.ipynb".
The first task is:
4. Examine the results
Everything below is just exploration of the results.
Examine the content of the results (as JSON, GeoJSON, and Python dictionaries)
Can you write Python code for it?


I am working on the file "MMW_API_watershed_demo.ipynb".
The first task is:
MMW staging (test) API Rapid Watershed Delineation (RWD) "watershed" endpoint:
Can you write Python code for it?


I am working on the file "MMW_API_watershed_demo.ipynb".
The first task is:
Render the geospatial results on an interactive map
Can you write Python code for it?

%matplotlib inline
import folium

# Initialize Folium map
m = folium.Map(location=[input_pt_geojson['properties']['Lat'], input_pt_geojson['properties']['Lon']],
               tiles='CartoDB positron', zoom_start=12)

# Add RWD watershed and drainage point onto map
folium.GeoJson(watershed_geojson).add_to(m);
folium.GeoJson(input_pt_geojson).add_to(m);

m


I am working on the file "MMW_API_watershed_demo.ipynb".
The first task is:
The results (result) are made up of two dictionary items: input_pt and watershed. Each one of those is a GeoJSON-like object already converted to a Python dictionary.
input_pt:
Can you write Python code for it?

input_pt_geojson = result['input_pt']

input_pt_geojson


I am working on the file "MNIST Classification.ipynb".
The first task is:
$$
 \ell(x, y) = L = {l_1,\dots,l_N}^\top, \quad
    l_n = - w_n \left[ y_n \cdot \log x_n + (1 - y_n) \cdot \log (1 - x_n) \right],
$$
Can you write Python code for it?

mlp_fun(torch.DoubleTensor(x.reshape(28*28,1)), Weight, Bias, Func)

# Softmax

import numpy as np

x = np.random.randn(2)

#x = np.array([-1, -1, 1])

sm = np.exp(x)
sm = sm/np.sum(sm)

print(x)
print(sm)

plt.imshow(sm.reshape(1,len(sm)), cmap='gray_r', vmin=0, vmax=1)
plt.show()


I am working on the file "MNIST Classification.ipynb".
The first task is:
Cross Entropy Loss
Target class labels $y$ in $[0, C-1]$
Can you write Python code for it?

# Number of data points
N = 4
# Number of data points
C = 3

CEloss = torch.nn.CrossEntropyLoss(reduce=True, size_average=False)
#X = np.array([[1,-1,-1],[2,1,0],[-1,3,-2],[1,2,3]])
X = np.random.randn(N, C)

x = torch.DoubleTensor(X)
print(x)
y = torch.LongTensor(N).random_(C)
print(y)

inp = Variable(x, requires_grad=True)
target = Variable(y)
out = CEloss(inp, target)
out

xx = x.numpy()
t = y.numpy()
E = 0
for n,i in enumerate(t):
    E += -xx[n,i] + np.log(np.sum(np.exp(xx[n,:])))

E

x.numpy()

x

A = torch.LongTensor(*[10,3])

A.random_(100)

f = mlp_fun(torch.DoubleTensor(x.reshape(28*28,1)), Weight, Bias, Func)

loss_fn(f, Variable(torch.DoubleTensor([[0]])))

loss_fn(Variable(torch.DoubleTensor([0])),Variable(torch.DoubleTensor([0.2])))

import torch

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 100, 10, 2

# Create random Tensors to hold inputs and outputs
x = Variable(torch.randn(N, D_in), requires_grad=False)
y = Variable(torch.randn(N, D_out), requires_grad=False)

# Use the nn package to define our model and loss function.
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)
loss_fn = torch.nn.MSELoss(size_average=True, reduce=True)

# Use the optim package to define an Optimizer that will update the weights of
# the model for us. Here we will use Adam; the optim package contains many other
# optimization algoriths. The first argument to the Adam constructor tells the
# optimizer which Tensors it should update.
learning_rate = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
for t in range(500):
    # Forward pass: compute predicted y by passing x to the model.
    y_pred = model(x)

    # Compute and print loss.
    loss = loss_fn(y_pred, y)
    print(t, loss.data[0])

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables it will update (which are the learnable
    # weights of the...


I am working on the file "MNIST Classification.ipynb".
The first task is:
Generate a Network
Can you write Python code for it?

import torch
import torch.autograd
from torch.autograd import Variable

# The sizes of layers from input to output
#sizes = [3,3,10,20,3,4]
sizes = [28*28, 100, 10]

# Generate the network
g = torch.tanh
sm = torch.nn.Softmax(dim=0)
identity = lambda x: x

D = len(sizes)
Weight = []
Bias = []
Func = []
for i in range(D-1):
    # For layer i, Weights are a S_{i+1} \times S_{i} matrix
    W = Variable(torch.randn(sizes[i+1],sizes[i]).double(), requires_grad=True)
    # For layer i, Biases are a S_{i+1} \times 1 matrix (a vector)
    b = Variable(torch.randn(sizes[i+1],1).double(), requires_grad=True)
    Weight.append(W)
    Bias.append(b)
    Func.append(g)

#Reset the final layer to sigmoid mapping
#Func[-1] = torch.sigmoid
Func[-1] = sm

# Define the exact functional form
# x: A S_0 \times B matrix where each column is a training example with S_0 features
def mlp_fun(x, Weight, Bias, Func):
    f = Variable(x, requires_grad=False)
    NumOfLayers = len(Weight)
    for i in range(NumOfLayers):
        #print(f)
        f = Func[i](torch.matmul(Weight[i], f) + Bias[i])
        #print(Func[i])
        #print(f)
    return f


I am working on the file "MNIST Classification.ipynb".
The first task is:
Training
Can you write Python code for it?

# Cross Entropy
Error = torch.nn.CrossEntropyLoss(size_average=True, reduce=True)

# Number of examples in the Training set
N = len(target)

eta = 1
MAX_ITER = 50000
BatchSize = min(1000, N)
EE = []
for epoch in range(MAX_ITER):
    idx = np.random.choice(N, size=BatchSize, replace=False)
    x = features[idx]
    #idx = [epoch%N]
    f = mlp_fun(torch.DoubleTensor(x.T), Weight, Bias, Func)
    y = Variable(torch.LongTensor(target[idx].reshape([len(idx)])), requires_grad=False)
    # Measure the error
    E = Error(f.transpose(0,1), y)
    EE.append(E.data.numpy())

    # Compute the derivative of the error with respect to Weights and Biases
    E.backward()

    # Take the step and reset weights
    for i in range(D-1):
        Weight[i].data.add_(-eta*Weight[i].grad.data)
        Bias[i].data.add_(-eta*Bias[i].grad.data)
        Weight[i].grad.zero_()
        Bias[i].grad.zero_()

    if epoch % 1000 == 0:
        print(epoch, E.data[0])

plt.plot(EE)
plt.show()

sm = torch.nn.Softmax(dim=0)
inp = Variable(torch.randn(2, 3))
print(inp)
print(sm(inp))


I am working on the file "MNIST Classification.ipynb".
The first task is:
Visualize the network
Can you write Python code for it?

%matplotlib inline
sizes_plot = sizes.copy()
sizes_plot[0] = 28
fig = plt.figure(figsize=(8, 8))
ax = fig.gca()
ax.axis('off')
draw_neural_net(ax, .1, .9, .1, .9, sizes_plot, bias=1, draw_edges=False)


I am working on the file "MNIST Competition/mnist_competition.ipynb".
The first task is:
Load MNIST Data
Can you write Python code for it?

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
# test data
test_images = mnist.test.images.reshape(10000, 28, 28, 1)
test_labels = mnist.test.labels[:]


I am working on the file "MNIST Competition/mnist_competition.ipynb".
The first task is:
Model 2 Architecture

Convolution * 2 + MaxPool + Dropout
Convolution * 2 + MaxPool + Dropout
Convolution + MaxPool + Dropout
Dense + Dropout
Dense + Dropout
Output
Can you write Python code for it?

model2 = Sequential([Convolution2D(filters=64, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu', input_shape=(28, 28, 1)),
                     Convolution2D(filters=128, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu'),
                     Convolution2D(filters=128, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Flatten(),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(10, activation='softmax'),
                     ])
model2.compile(optimizer=Adam(lr=0.0005), loss='categorical_crossentropy', metrics=['accuracy'])
model2.fit(images, labels, batch_size=256, epochs=epochs, shuffle=True, verbose=1, validation_data=(test_images, test_labels))

model_json = model2.to_json()
with open("model2.json", "w") as json_file:
    json_file.write(model_json)
model2.save_weights("model2.h5")
print("Saved model to disk")


I am working on the file "MNIST Competition/mnist_competition.ipynb".
The first task is:
Train Parameter
Can you write Python code for it?

model_1_filter_size = 3
model_2_filter_size = 5
model_3_filter_size = 7
epochs = 10


I am working on the file "MNIST Competiton_9980/mnist_competition_9980_Final.ipynb".
The first task is:
3. Evaluate
Can you write Python code for it?

from keras.models import model_from_json
from tensorflow.examples.tutorials.mnist import input_data
from keras.optimizers import Adam
import numpy as np

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
# test data
test_images = mnist.test.images.reshape(10000, 28, 28, 1)
test_labels = mnist.test.labels[:]

# load json and create model
def model_open(name, test_images, test_labels):
    json_file = open(name + '.json', 'r')
    loaded_model_json = json_file.read()
    json_file.close()
    loaded_model = model_from_json(loaded_model_json)
    # load weights into new model
    loaded_model.load_weights(name + '.h5')
    print("Loaded model from disk")
    loaded_model.compile(optimizer=Adam(lr=0.0001), loss='categorical_crossentropy', metrics=['acc'])
    prob = loaded_model.predict_proba(test_images)
    acc = np.mean(np.equal(np.argmax(prob, axis=1), np.argmax(test_labels, axis=1)))
    print('\nmodel : %s, test accuracy : %.4f\n' % (name, acc))
    return prob, acc

model_1_prob, model_1_acc = model_open('model1', test_images, test_labels)
model_2_prob, model_2_acc = model_open('model2', test_images, test_labels)
model_3_prob, model_3_acc = model_open('model3', test_images, test_labels)


I am working on the file "MNIST Competiton_9980/mnist_competition_9980_Final.ipynb".
The first task is:
4. Final Result(Ensemble)

This ensembe method gives higher weight to the model with higher performance. If the performances of two models were equal, the higher weight was given to the one with a bigger filter size
best accuracy with a single model is $0.9972$
Final Ensemble Accuracy is $0.9980$
Can you write Python code for it?

def model_ensemble(prob1, acc1, prob2, acc2, prob3, acc3):
    prob_list = [prob1, prob2, prob3]
    acc_list = [acc1, acc2, acc3]
    idx_acc_list = {idx: acc for idx, acc in enumerate(acc_list)}
    sorted_acc_list = [idx for idx, _ in sorted(idx_acc_list.items(), key=lambda value: (value[1], value[0]), reverse=True)]
    final_prob = 0
    for i in sorted_acc_list:
        final_prob += prob_list[i] * (i+1)
    final_score = np.mean(np.equal(np.argmax(final_prob, axis=1), np.argmax(test_labels, axis=1))) # Test
    print('Final test accuracy : %.4f' % final_score)

model_ensemble(model_1_prob, model_1_acc, model_2_prob, model_2_acc, model_3_prob, model_3_acc)


I am working on the file "MNIST Competiton_9980/mnist_competition_9980_Final.ipynb".
The first task is:
Model 3
Can you write Python code for it?

model3 = Sequential([Convolution2D(filters=64, kernel_size=(7, 7), padding='same', activation='elu', input_shape=(28, 28, 1)),
                     Convolution2D(filters=128, kernel_size=(7, 7), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(7, 7), padding='same', activation='elu'),
                     Convolution2D(filters=128, kernel_size=(7, 7), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(7, 7), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Flatten(),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(10, activation='softmax'),
                     ])
model3.compile(optimizer=Adam(lr=0.0001), loss='categorical_crossentropy', metrics=['accuracy'])
model3.fit(images, labels, batch_size=256, epochs=20, shuffle=True, verbose=1, validation_data=(test_images, test_labels))

model_json = model3.to_json()
with open("model3.json", "w") as json_file:
    json_file.write(model_json)
model3.save_weights("model3.h5")
print("Saved model to disk")


I am working on the file "MNIST Tutorial.ipynb".
The first task is:
A Deeper MLP
Next we build a two-layer MLP with the same number of hidden nodes, half in each layer.
Can you write Python code for it?

from keras.models import Sequential
nb_layers = 2
mlp2 = Sequential()
# add hidden layers
for i in range(nb_layers):
    mlp2.add(Dense(output_dim=nb_hidden/nb_layers, input_dim=nb_input if i==0 else nb_hidden/nb_layers, init='uniform'))
    mlp2.add(Activation('sigmoid'))
# add output layer
mlp2.add(Dense(output_dim=nb_output, input_dim=nb_hidden/nb_layers, init='uniform'))
mlp2.add(Activation('softmax'))

mlp2.compile(loss='categorical_crossentropy', optimizer='SGD')

mlp2.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, show_accuracy=True, verbose=1)

mlp2.evaluate(X_test, y_test, show_accuracy=True)


I am working on the file "MNIST Tutorial.ipynb".
The first task is:
For example, let's inspect the first 3 labels:
Can you write Python code for it?

print('labels', labels_train[:3])
print('y', y_train[:3])


I am working on the file "MNIST Tutorial.ipynb".
The first task is:
Inspecting the data
Let's load some data
Can you write Python code for it?

from keras.datasets import mnist
(images_train, labels_train), (images_test, labels_test) = mnist.load_data()
print('images',images_train.shape)
print('labels', labels_train.shape)


I am working on the file "MNIST Tutorial.ipynb".
The first task is:
Next we train the model on our training data. Watch the loss, which is the objective function which we are minimising, and the estimated accuracy of the model.
Can you write Python code for it?

mlp.fit(X_train, y_train,
        batch_size=batch_size, nb_epoch=nb_epoch,
        verbose=1, show_accuracy=True)


I am working on the file "MNIST Tutorial.ipynb".
The first task is:
Simple Multi-Layer Perceptron (MLP)
The simplest kind of Artificial Neural Network is as Multi-Layer Perceptron (MLP) with a single hidden layer.
Can you write Python code for it?

# Neural Network Architecture Parameters
nb_input = 784
nb_hidden = 512
nb_output = 10
# Training Parameters
nb_epoch = 1
batch_size = 128


I am working on the file "MNIST Tutorial.ipynb".
The first task is:
Stacked Autoencoder
Can you write Python code for it?

from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout

class StackedAutoencoder(object):

    def __init__(self, layers, mode='autoencoder',
                 activation='sigmoid', init='uniform', final_activation='softmax',
                 dropout=0.2, optimizer='SGD'):
        self.layers = layers
        self.mode = mode
        self.activation = activation
        self.final_activation = final_activation
        self.init = init
        self.dropout = dropout
        self.optimizer = optimizer
        self._model = None

        self.build()
        self.compile()

    def _add_layer(self, model, i, is_encoder):
        if is_encoder:
            input_dim, output_dim = self.layers[i], self.layers[i+1]
            activation = self.final_activation if i==len(self.layers)-2 else self.activation
        else:
            input_dim, output_dim = self.layers[i+1], self.layers[i]
            activation = self.activation
        model.add(Dense(output_dim=output_dim,
                        input_dim=input_dim,
                        init=self.init))
        model.add(Activation(activation))

    def build(self):
        self.encoder = Sequential()
        self.decoder = Sequential()
        self.autoencoder = Sequential()
        for i in range(len(self.layers)-1):
            self._add_layer(self.encoder, i, True)
            self._add_layer(self.autoencoder, i, True)
            #if i<len(self.layers)-2:
            #    self.autoencoder.add(Dropout(self.dropout))

        # Note that the decoder layers are in reverse order
        for i in reversed(range(len(self.layers)-1)):
            self._add_layer(self.decoder, i, False)
            self._add_layer(self.autoencoder, i, False)

    def compile(self):
        print("Compiling the encoder ...")
        self.encoder.compile(loss='categorical_crossentropy', optimizer=self.optimizer)
        print("Compiling the decoder ...")
        self.decoder.compile(loss='mse',...


I am working on the file "MNIST Tutorial.ipynb".
The first task is:
The labels we transform to a "one-hot" encoding
Can you write Python code for it?

# The labels need to be transformed into class indicators
from keras.utils import np_utils
y_train = np_utils.to_categorical(labels_train, nb_classes=10)
y_test = np_utils.to_categorical(labels_test, nb_classes=10)
print(y_train.shape, 'train labels')
print(y_test.shape, 'test labels')


I am working on the file "MNIST Tutorial.ipynb".
The first task is:
then we compile it. This takes the symbolic computational graph of the model and compiles it an efficient implementation which can then be used to train and evaluate the model.
Note that we have to specify what loss/objective function we want to use as well which optimisation algorithm to use. SGD stands for Stochastic Gradient Descent.
Can you write Python code for it?


I am working on the file "MNIST-Digit Recognition/Feedforward_Recog.ipynb".
The first task is:
'softmax' is a sigmoid shaped curve
'categorical_crossentropy' is the used loss-function or error-function
'adam' is the specified way of performing gradient descent
Run the model
Can you write Python code for it?

# Build the model
model = model()
# Fit the model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200, verbose=2)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)
print("Error: %.2f%%" % (100-scores[1]*100))


I am working on the file "MNIST-For-ML-Beginners.ipynb".
The first task is:
Each step of the loop, we get a "batch" of one hundred random data points from our training set. We run train_step feeding in the batches data to replace the placeholders.
Using small batches of random data is called stochastic training -- in this case, stochastic gradient descent. Ideally, we'd like to use all our data for every step of training because that would give us a better sense of what we should be doing, but that's expensive. So, instead, we use a different subset every time. Doing this is cheap and has much of the same benefit.
Evaluating Our Model

How well does our model do?
Well, first let's figure out where we predicted the correct label. tf.argmax is an extremely useful function which gives you the index of the highest entry in a tensor along some axis. For example, tf.argmax(y,1) is the label our model thinks is most likely for each input, while tf.argmax(y_,1) is the correct label. We can use tf.equal to check if our prediction matches the truth.
Can you write Python code for it?

correct_prediction = tf.equal(tf.arg_max(y,1), tf.argmax(y_,1))


I am working on the file "MNIST-For-ML-Beginners.ipynb".
The first task is:
First, tf.log computes the logarithm of each element of $\mathrm{y}$. Next, we multiply each element of y_ with the corresponding element of tf.log(y). Then tf.reduce_sum adds the elements in the second dimension of y, due to the reduction_indices=[1] parameter. Finally, tf.reduce_mean computes the mean over all the examples in the batch.
<mark>Note that in the source code, we don't use this formulation, because it is numerically unstable. Instead, we apply tf.nn.softmax_cross_entropy_with_logits on the unnormalized logits (e.g., we call softmax_cross_entropy_with_logits on tf.matmul(x, W) + b), because this more numerically stable function internally computes the softmax activation. In your code, consider using tf.nn.softmax_cross_entropy_with_logits instead.</mark>
Now that we know what we want our model to do, it's very easy to have TensorFlow train it to do so. Because TensorFlow knows the entire graph of your computations, it can automatically use the backpropagation algorithm to efficiently determine how your variables affect the loss you ask it to minimize. Then it can apply your choice of optimization algorithm to modify the variables and reduce the loss.
Can you write Python code for it?


I am working on the file "MNIST-For-ML-Beginners.ipynb".
The first task is:
In this case, we ask TensorFlow to minimize cross_entropy using the gradient descent algorithm with a learning rate of 0.5. Gradient descent is a simple procedure, where TensorFlow simply shifts each variable a little bit in the direction that reduces the cost. But TensorFlow also provides many other optimization algorithms: using one is as simple as tweaking one line.
What TensorFlow actually does here, behind the scenes, is to add new operations to your graph which implement backpropagation and gradient descent. Then it gives you back a single operation which, when run, does a step of gradient descent training, slightly tweaking your variables to reduce the loss.
We can now launch the model in an InteractiveSession:
Can you write Python code for it?


I am working on the file "MNIST-For-ML-Beginners.ipynb".
The first task is:
Softmax Regressions

We know that every image in MNIST is of a handwritten digit between zero and nine. So there are only ten possible things that a given image can be. We want to be able to look at an image and give the probabilities for it being each digit. For example, our model might look at a picture of a nine and be 80% sure it's a nine, but give a 5% chance to it being an eight (because of the top loop) and a bit of probability to all the others because it isn't 100% sure.
This is a classic case where a softmax regression is a natural, simple model. If you want to assign probabilities to an object being one of several different things, softmax is the thing to do, because softmax gives us a list of values between 0 and 1 that add up to 1. Even later on, when we train more sophisticated models, the final step will be a layer of softmax.
A softmax regression has two steps: first we add up the evidence of our input being in certain classes, and then we convert that evidence into probabilities.
To tally up the evidence that a given image is in a particular class, we do a weighted sum of the pixel intensities. The weight is negative if that pixel having a high intensity is evidence against the image being in that class, and positive if it is evidence in favor.
The following diagram shows the weights one model learned for each of these classes. Red represents negative weights, while blue represents positive weights.
<img src="images/image-08.png?raw=true" alt="Smiley face" height="300" width="300">
We also add some extra evidence called a bias. Basically, we want to be able to say that some things are more likely independent of the input. The result is that the evidence for a class i given an input x is:
$$\mathrm{evidence_{i} = \sum\limits_{j} {w_{i,j} \cdot x_j + b_i}}$$
where $W_i$ is the weights and $b_i$ is the bias for class $i$, and $j$ is an index for summing over the pixels in our input image $x$. We then convert the evidence tallies into our predicted probabilities $y$ using the "softmax"...
Can you write Python code for it?


I am working on the file "MNIST-For-ML-Beginners.ipynb".
The first task is:
That gives us a list of booleans. To determine what fraction are correct, we cast to floating point numbers and then take the mean. For example, [True, False, True, True] would become [1,0,1,1] which would become 0.75.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "MNIST-For-ML-Beginners.ipynb".
The first task is:
Then we can implement the cross-entropy function, $\mathrm{-\sum\limits_{i} {y'_{i}\space log(y_i)}}$:
Can you write Python code for it?

cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))


I am working on the file "MNIST-For-ML-Beginners.ipynb".
The first task is:
We create these Variables by giving tf.Variable the initial value of the Variable: in this case, we initialize both $\mathrm{W}$ and $\mathrm{b}$ as tensors full of zeros. Since we are going to learn $\mathrm{W}$ and $\mathrm{b}$, it doesn't matter very much what they initially are.
Notice that $\mathrm{W}$ has a shape of $\mathrm{[784, 10]}$ because we want to multiply the 784-dimensional image vectors by it to produce 10-dimensional vectors of evidence for the difference classes. $\mathrm{b}$ has a shape of $\mathrm{[10]}$ so we can add it to the output.

$$\mathrm{\begin{bmatrix}
       {y_1}\[0.3em]
       {y_2}\[0.3em]
       \cdots\
       {y_{10}}
     \end{bmatrix}
     = softmax \begin{bmatrix} \begin{bmatrix}
      W_{1,1} & W_{1,2} & \cdots & W_{1,784}\[0.3em]
      W_{2,1} & W_{2,2} & \cdots & W_{2,784}\[0.3em]
      \cdots\
      W_{10,1} & W_{10,2} & \cdots & W_{10,784}\[0.3em]
     \end{bmatrix}
     \cdot
     \begin{bmatrix}
       {x_1}\[0.3em]
       {x_2}\[0.3em]
       \cdots\
       {x_{784}}
     \end{bmatrix}
     +
     \begin{bmatrix}
       {b_1}\[0.3em]
       {b_2}\[0.3em]
       \cdots\
       {b_{10}}
     \end{bmatrix} \end{bmatrix}}$$

$$\mathrm{[10 \times 1] = [10 \times 784]\cdot[784 \times 1] + [10 \times 1]}$$

We can now implement our model. It only takes one line to define it!
Can you write Python code for it?


I am working on the file "MNIST-For-ML-Beginners.ipynb".
The first task is:
We first have to create an operation to initialize the variables we created:
Can you write Python code for it?


I am working on the file "MNIST-TFlearn/Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Retrieving training and test data
The MNIST data set already contains both training and test data. There are 55,000 data points of training data, and 10,000 points of test data.
Each MNIST data point has:
1. an image of a handwritten digit and
2. a corresponding label (a number 0-9 that identifies the image)
We'll call the images, which will be the input to our neural network, X and their corresponding labels Y.
We're going to want our labels as one-hot vectors, which are vectors that holds mostly 0's and one 1. It's easiest to see this in a example. As a one-hot vector, the number 0 is represented as [1, 0, 0, 0, 0, 0, 0, 0, 0, 0], and 4 is represented as [0, 0, 0, 0, 1, 0, 0, 0, 0, 0].
Flattened data
For this example, we'll be using flattened data or a representation of MNIST images in one dimension rather than two. So, each handwritten number image, which is 28x28 pixels, will be represented as a one dimensional array of 784 pixel values.
Flattening the data throws away information about the 2D structure of the image, but it simplifies our data so that all of the training data can be contained in one array whose shape is [55000, 784]; the first dimension is the number of training images and the second dimension is the number of pixels in each image. This is the kind of data that is easy to analyze using a simple neural network.
Can you write Python code for it?

# Retrieve the training and test data
trainX, trainY, testX, testY = mnist.load_data(one_hot=True)


I am working on the file "MNIST-TFlearn/Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Testing
After you're satisified with the training output and accuracy, you can then run the network on the test data set to measure it's performance! Remember, only do this after you've done the training and are satisfied with the results.
A good result will be higher than 95% accuracy. Some simple models have been known to get up to 99.7% accuracy!
Can you write Python code for it?

# Compare the labels that our model predicts with the actual labels

# Find the indices of the most confident prediction for each item. That tells us the predicted digit for that sample.
predictions = np.array(model.predict(testX)).argmax(axis=1)

# Calculate the accuracy, which is the percentage of times the predicated labels matched the actual labels
actual = testY.argmax(axis=1)
test_accuracy = np.mean(predictions == actual, axis=0)

# Print out the result
print("Test accuracy: ", test_accuracy)


I am working on the file "MNIST-TFlearn/Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Training the network
Now that we've constructed the network, saved as the variable model, we can fit it to the data. Here we use the model.fit method. You pass in the training features trainX and the training targets trainY. Below I set validation_set=0.1 which reserves 10% of the data set as the validation set. You can also set the batch size and number of epochs with the batch_size and n_epoch keywords, respectively.
Too few epochs don't effectively train your network, and too many take a long time to execute. Choose wisely!
Can you write Python code for it?

# Training
model.fit(trainX, trainY, validation_set=0.1, show_metric=True, batch_size=100, n_epoch=20)


I am working on the file "MNIST-TFlearn/Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Visualize the training data
Provided below is a function that will help you visualize the MNIST data. By passing in the index of a training example, the function show_digit will display that training image along with it's corresponding label in the title.
Can you write Python code for it?

# Visualizing the data
import matplotlib.pyplot as plt
%matplotlib inline

# Function for displaying a training image by it's index in the MNIST set
def show_digit(index):
    label = trainY[index].argmax(axis=0)
    # Reshape 784 array into 28x28 image
    image = trainX[index].reshape([28,28])
    plt.title('Training data, index: %d,  Label: %d' % (index, label))
    plt.imshow(image, cmap='gray_r')
    plt.show()

# Display the first (index 0) training image
show_digit(0)

trainY[0]


I am working on the file "MNIST-example.ipynb".
The first task is:
At this point we may perform some sanity checks to vaildate that the right tensors are cached and fed into training runs of different sections. For example, we expect the backward pass of section sec2 to depend on some output of sec1 as well as gradients computed in sec3:
Can you write Python code for it?


I am working on the file "MNIST-example.ipynb".
The first task is:
Construct Residual Network
partialflow allows us to split a tensorflow graph into several sections which can then be trained separately to lower the memory consumption. This means that the training graph of each section on its own has to fit into GPU memory, whereas the full network's training graph may not.
The graph sections are managed by a GraphSectionManager that orchestrates the data flow between the graph sections during training:
Can you write Python code for it?

from partialflow import GraphSectionManager

sm = GraphSectionManager()


I am working on the file "MNIST-example.ipynb".
The first task is:
This should log a single forward pass for each section into the console running this notebook:
Forward pass over section 0[[[[0 0 0]]]...]
Forward pass over section 1[[[[0.25482464 0.54249996 0.15713426]]]...]
Forward pass over section 2[[[[1.3474643 0.62452459 0.14982516]]]...]
Forward pass over section 3[[[[0.52292633 -0.39113081 0.74775648]]]...]
Training
Since intermediate results need to be cached for the backward pass, training operations need to be run using GraphSectionManager. The run_full_cycle method will run a forward pass, cache intermediate results as needed, and perform a backward pass over the training operations.
Forward passes are not performed section-wise, because tensorflow optimizes memory consumption by dropping intermediate results anyway. Hence the full forward pass graph is assumed to fit into GPU memory. In contrast, backward passes are performed section-wise. run_full_cycle takes care of evaluating the graph elements in fetches during the right phases of this procedure.
The following should log a full forward pass followed by interleaved forward and backward passes for each section. Note that the basic_feed parameter is used analogous to feed_dict in session.run:
Can you write Python code for it?


I am working on the file "MNIST-example.ipynb".
The first task is:
We can now use the GraphSectionManager to create new sections in which we define our network. partialflow automatically analyzes the tensorflow graph and keeps track of tensors flowing across section borders and variables defined in sections.
In the following, we define our CNN in four sections. This is mainly done for illustrative purposes since two sections might already suffice, depending on your GPU memory. We added some tf.Print statements to make tensorflow log forward passes for each section.
Can you write Python code for it?

from BasicNets import BatchnormNet

# flag for batch normalization layers
is_training = tf.placeholder(name='is_training', shape=[], dtype=tf.bool)
net = BatchnormNet(is_training, image_batch)

# first network section with initial convolution and three residual blocks
with sm.new_section() as sec0:
    with tf.variable_scope('initial_conv'):
        stream = net.add_conv(net._inputs, n_filters=16)
        stream = tf.Print(stream, [stream], 'Forward pass over section 0')
        stream = net.add_bn(stream)
        stream = tf.nn.relu(stream)

    with tf.variable_scope('scale0'):
        for i in range(3):
            with tf.variable_scope('block_%d' % i):
                stream = net.res_block(stream)

# second network section strided convolution to decrease the input resolution
with sm.new_section() as sec1:
    with tf.variable_scope('scale1'):
        stream = tf.Print(stream, [stream], 'Forward pass over section 1')
        stream = net.res_block(stream, filters_factor=2, first_stride=2)
        for i in range(2):
            with tf.variable_scope('block_%d' % i):
                stream = net.res_block(stream)

# third network section
with sm.new_section() as sec2:
    with tf.variable_scope('scale2'):
        stream = tf.Print(stream, [stream], 'Forward pass over section 2')
        stream = net.res_block(stream, filters_factor=2, first_stride=2)
        for i in range(4):
            with tf.variable_scope('block_%d' % i):
                stream = net.res_block(stream)

# fourth network section with final pooling and cross-entropy loss
with sm.new_section() as sec3:
    with tf.variable_scope('final_pool'):
        stream = tf.Print(stream, [stream], 'Forward pass over section 3')
        # global average pooling over image dimensions
        stream = tf.reduce_mean(stream, axis=2)
        stream = tf.reduce_mean(stream, axis=1)

        # final conv for classification
        stream = net.add_fc(stream, out_dims=10)

    with tf.variable_scope('loss'):
      ...


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
At this point we can pick any classifier and train it. This is the iterative part of choosing and testing all the classifiers and tuning the hyper parameters

SGDClassifier
Training
Can you write Python code for it?

from sklearn.linear_model import SGDClassifier

clf = SGDClassifier(random_state = 0)
clf.fit(X_train, y_train_0)


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Danger of Blindly Applying Evaluator As a Performance Measure
Let's check against a dumb classifier
Can you write Python code for it?


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Each row: actual class
Each column: predicted class
First row: Non-zero images, the negative class:
* 53360 were correctly classified as non-zeros. True negatives.
* Remaining 717 were wrongly classified as 0s. False positive
Second row: The images of zeros, the positive class:
* 395 were incorrectly classified as 0s. False negatives
* 5528 were correctly classified as 0s. True positives

Precision
Precision measures the accuracy of positive predictions. Also called the precision of the classifier
$$\textrm{precision} = \frac{\textrm{True Positives}}{\textrm{True Positives} + \textrm{False Positives}}$$

Can you write Python code for it?

from sklearn.metrics import precision_score, recall_score


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Exercise
High Recall Score. Recall score > 0.9
Can you write Python code for it?

idx = len(recalls[recalls > 0.9])

thresholds[idx]

y_train_pred_90 = (y_scores > thresholds[idx])

precision_score(y_train_0, y_train_pred_90)

recall_score(y_train_0, y_train_pred_90)


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Exercise:
What if you would like to perform 10-fold CV test? How would you do that
Can you write Python code for it?

cross_val_score(clf, X_train, y_train_0, cv=10, scoring='accuracy')


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Note the result here may vary from the video as the results from the confusion matrix are different each time you run it.
Can you write Python code for it?


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Performance Measures
Measuring Accuracy Using Cross-Validation
StratifiedKFold
Let's try with the StratifiedKFold stratified sampling to create multiple folds. At each iteration, the classifier was cloned and trained using the training folds and makes predictions on the test fold.
StratifiedKFold utilised the Stratified sampling concept

The population is divided into homogeneous subgroups called strata
The right number of instances is sampled from each stratum
To guarantee that the test set is representative of the population
Can you write Python code for it?

from sklearn.model_selection import StratifiedKFold
from sklearn.base import clone
clf = SGDClassifier(random_state=0)

skfolds = StratifiedKFold(n_splits=3, random_state=100)

for train_index, test_index in skfolds.split(X_train, y_train_0):
    clone_clf = clone(clf)
    X_train_fold = X_train[train_index]
    y_train_folds = (y_train_0[train_index])
    X_test_fold = X_train[test_index]
    y_test_fold = (y_train_0[test_index])

    clone_clf.fit(X_train_fold, y_train_folds)
    y_pred = clone_clf.predict(X_test_fold)
    n_correct = sum(y_pred == y_test_fold)
    print("{0:.4f}".format(n_correct / len(y_pred)))


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Precision / Recall Tradeoff
Increasing precision reduced recall and vice versa
<img src="img\precision-recall.png">
Our classifier is designed to pick up zeros.
12 observations

Central Arrow
Suppose the decision threshold is positioned at the central arrow:
* We get 4 true positives (We have 4 zeros to the right of the central arrow)
* 1 false positive which is actually seven.
At this threshold, the precision accuracy is $\frac{4}{5}=80\%$
However, out of the 6 zeros, the classifier only picked up 4. The recall accuracy is $\frac{4}{6}=67\%$

Right Arrow

We get 3 true positives
0 false positive

At this threshold, the precision accuracy is $\frac{3}{3}=100\%$
However, out of the 6 zeros, the classifier only picked up 3. The recall accuracy is $\frac{3}{6}=50\%$

Left Arrow

We get 6 true positives
2 false positive

At this threshold, the precision accuracy is $\frac{6}{8}=75\%$
Out of the 6 zeros, the classifier picked up all 6. The recall accuracy is $\frac{6}{6}=100\%$
Can you write Python code for it?

clf = SGDClassifier(random_state=0)
clf.fit(X_train, y_train_0)

y[1000]

y_scores = clf.decision_function(X[1000].reshape(1, -1))
y_scores

threshold = 0

y_some_digits_pred = (y_scores > threshold)

y_some_digits_pred

threshold = 40000
y_some_digits_pred = (y_scores > threshold)
y_some_digits_pred

y_scores = cross_val_predict(clf, X_train, y_train_0, cv=3, method='decision_function')

plt.figure(figsize=(12,8)); plt.hist(y_scores, bins=100);


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Prediction
Can you write Python code for it?


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Setting High Precisions
Let's aim for 99% precisions.
Can you write Python code for it?

idx = len(precisions[precisions < 0.99])


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
Splitting the train and test sets
Can you write Python code for it?

num_split = 60000

X_train, X_test, y_train, y_test = X[:num_split], X[num_split:], y[:num_split], y[num_split:]


I am working on the file "MNIST/0410 -  MNIST Project 6 - The ROC Curve/MNIST.ipynb".
The first task is:
The Receiver Operating Characteristics (ROC) Curve
Instead of plotting precision versus recall, the ROC curve plots the true positive rate (another name for recall) against the false positive rate. The false positive rate (FPR) is the ratio of negative instances that are incorrectly classified as positive. It is equal to one minus the true negative rate, which is the ratio of negative instances that are correctly classified as negative.
The TNR is also called specificity. Hence the ROC curve plots sensitivity (recall) versus 1 - specificity.
<img src="img\tnr_and_fpr.png">
Can you write Python code for it?

from sklearn.metrics import roc_curve

fpr, tpr, thresholds = roc_curve(y_train_0, y_scores)

def plot_roc_curve(fpr, tpr, label=None):
    plt.plot(fpr, tpr, linewidth=2, label=label)
    plt.plot([0,1], [0,1], 'k--')
    plt.axis([0, 1, 0, 1])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('ROC Curve')

plt.figure(figsize=(12,8));
plot_roc_curve(fpr, tpr)
plt.show();

from sklearn.metrics import roc_auc_score

roc_auc_score(y_train_0, y_scores)