I am working on the file "NCPSchool2021/Examples/TMVA_CNN_Classification_py.ipynb".
The first task is:
Booking Methods
Here we book the TMVA methods. We book a DNN and a CNN
Booking Deep Neural Network
Here we book the new DNN of TMVA. If using master version you can use the new DL method
Can you write Python code for it?

inputLayoutString = "InputLayout=1|1|64";
batchLayoutString= "BatchLayout=1|32|64";
layoutString = ("Layout=DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR")

training1  = "Optimizer=ADAM,LearningRate=1e-3,Momentum=0.,Regularization=None,WeightDecay=1e-4,"
training1 += "DropConfig=0.+0.+0.+0.,MaxEpochs=30,ConvergenceSteps=10,BatchSize=32,TestRepetitions=1"
trainingStrategyString = "TrainingStrategy=" + training1

dnnOptions = "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=G:WeightInitialization=XAVIER::Architecture=CPU"

dnnOptions +=  ":" + inputLayoutString
dnnOptions +=  ":" + batchLayoutString
dnnOptions +=  ":" + layoutString
dnnOptions +=  ":" + trainingStrategyString

#we can now book the method

factory.BookMethod(loader, ROOT.TMVA.Types.kDL, "DL_DENSE", dnnOptions)


I am working on the file "NCPSchool2021/Examples/TMVA_CNN_Classification_py.ipynb".
The first task is:
Plot ROC Curve
We enable JavaScript visualisation for the plots
Can you write Python code for it?

//%jsroot on

c1 = factory.GetROCCurve(loader)
c1.Draw()

## close outputfile to save output file
outputFile.Close()


I am working on the file "NCPSchool2021/Examples/TMVA_CNN_Classification_py.ipynb".
The first task is:
Setup Dataset(s)
Define input data file and signal and background trees
Can you write Python code for it?

## Apply additional cuts on the signal and background samples (can be different)
mycuts = ROOT.TCut("")   ## for example: TCut mycuts = "abs(var1)<0.5 && abs(var2-0.5)<1";
mycutb = ROOT.TCut("")   ## for example: TCut mycutb = "abs(var1)<0.5";

loader.PrepareTrainingAndTestTree( mycuts, mycutb,
                                  "nTrain_Signal=5000:nTrain_Background=5000:SplitMode=Random:"
                                   "NormMode=NumEvents:!V" )


I am working on the file "NCPSchool2021/Examples/TMVA_CNN_Classification_py.ipynb".
The first task is:
Test and Evaluate Methods
Can you write Python code for it?


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
2. Define Trainining Strategy
We define here the different training strategy for the DNN. One can concatenate different training strategy changing parameters like:
 - Optimizer
 - Learning rate
 - Momentum (valid for SGD and RMSPROP)
 - Regularization and Weight Decay
 - Dropout
 - Max number of epochs
 - Convergence steps. if the test error will not decrease after that value the training will stop
 - Batch size (This value must be the same specified in the input layout)
 - Test Repetitions (the interval when the test error will be computed)
Can you write Python code for it?

##Training strategies
## one can catenate several training strategies

training1  = "Optimizer=ADAM,LearningRate=1e-3,Momentum=0.,Regularization=None,WeightDecay=1e-4,"
training1 += "DropConfig=0.+0.+0.+0.,MaxEpochs=20,ConvergenceSteps=10,BatchSize=100,TestRepetitions=1"

# we add regularization in the second phase
training2  = "Optimizer=ADAM,LearningRate=1e-3,Momentum=0.,Regularization=L2,WeightDecay=1e-4,"
training2 += "DropConfig=0.0+0.0+0.0+0,MaxEpochs=20,ConvergenceSteps=10,BatchSize=100,TestRepetitions=1"

trainingStrategyString = "TrainingStrategy=" + training1 ## + training2


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
3. Define general options and book method
We define the general DNN options such as

Type of Loss function (e.g. cross entropy)
Weight Initizalization (e.g XAVIER, XAVIERUNIFORM, NORMAL )
Variable Transformation
Type of Architecture (e.g. CPU, GPU, Standard)

We add then also all the other options defined before
Can you write Python code for it?

## General Options.
dnnOptions = "!H:V:ErrorStrategy=CROSSENTROPY:WeightInitialization=XAVIER::Architecture=CPU"

dnnOptions +=  ":" + inputLayoutString
dnnOptions +=  ":" + batchLayoutString
dnnOptions +=  ":" + layoutString
dnnOptions +=  ":" + trainingStrategyString

#we can now book the method

factory.BookMethod(loader, ROOT.TMVA.Types.kDL, "DL_CPU", dnnOptions)

## to use tensorflow backend
import os
import tensorflow as tf

from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam, SGD
from tensorflow.keras.initializers import TruncatedNormal
from tensorflow.keras.layers import Input, Dense, Dropout, Flatten, Conv2D, MaxPooling2D, Reshape, BatchNormalization

# Define model
model = Sequential()
model.add(Dense(64, kernel_initializer='glorot_normal', activation='tanh', input_dim=7))
#model.add(Dropout(0.2))
model.add(Dense(64, kernel_initializer='glorot_normal', activation='tanh'))
#model.add(Dropout(0.2))
model.add(Dense(64, kernel_initializer='glorot_normal', activation='tanh'))
model.add(Dense(2, kernel_initializer='glorot_uniform', activation='softmax'))

# Set loss and optimizer
model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=['categorical_accuracy',])

# Store model to file
model.save('model_dense.h5')

# Print summary of model
model.summary()

factory.BookMethod(loader, ROOT.TMVA.Types.kPyKeras, 'Keras_Dense',
        'H:!V:FilenameModel=model_dense.h5:'+\
        'NumEpochs=20:BatchSize=100:TriesEarlyStopping=10')


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Booking Methods
Here we book the TMVA methods. We book a Likelihood based a BDT and a standard MLP (shallow NN)
Can you write Python code for it?

## Boosted Decision Trees
factory.BookMethod(loader,ROOT.TMVA.Types.kBDT, "BDT",
                   "!V:NTrees=200:MinNodeSize=2.5%:MaxDepth=2:BoostType=AdaBoost:AdaBoostBeta=0.5:UseBaggedBoost:"
                   "BaggedSampleFraction=0.5:SeparationType=GiniIndex:nCuts=20" )

## Multi-Layer Perceptron (Neural Network)
factory.BookMethod(loader, ROOT.TMVA.Types.kMLP, "MLP",
                   "!H:!V:NeuronType=tanh:VarTransform=N:NCycles=100:HiddenLayers=N+5:TestRate=5:!UseRegulator" );


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Close outputfile to save all output information (evaluation result of methods)
Can you write Python code for it?


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Declare DataLoader(s)
The next step is to declare the DataLoader class that deals with input data abd variables
We add first the signal and background trees in the data loader and then we
define the input variables that shall be used for the MVA training
note that you may also use variable expressions, which can be parsed by TTree::Draw( "expression" )]
Can you write Python code for it?

loader = ROOT.TMVA.DataLoader("dataset")

### global event weights per tree (see below for setting event-wise weights)
signalWeight     = 1.0
backgroundWeight = 1.0

### You can add an arbitrary number of signal or background trees
loader.AddSignalTree    ( signalTree,     signalWeight     )
loader.AddBackgroundTree( backgroundTree, backgroundWeight )

## Define input variables

loader.AddVariable("m_jj")
loader.AddVariable("m_jjj")
loader.AddVariable("m_lv")
loader.AddVariable("m_jlv")
loader.AddVariable("m_bb")
loader.AddVariable("m_wbb")
loader.AddVariable("m_wwbb")


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Evaluate all methods
Here we evaluate all methods and compare their performances, computing efficiencies, ROC curves etc.. using both training and tetsing data sets. Several histograms are produced which can be examined with the TMVAGui or directly using the output file
Can you write Python code for it?


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Input Data
We define now the input data file and we retrieve the ROOT TTree objects with the signal and background input events
Can you write Python code for it?

inputFileName = "Higgs_data.root"

inputFile = ROOT.TFile.Open( inputFileName )

# retrieve input trees

signalTree     = inputFile.Get("sig_tree")
backgroundTree = inputFile.Get("bkg_tree")

signalTree.Print()


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Plot ROC Curve
We enable JavaScript visualisation for the plots
Can you write Python code for it?

%jsroot on

c1 = factory.GetROCCurve(loader);
c1.Draw();


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Test  all methods
Here we test all methods using the test data set
Can you write Python code for it?


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Train Methods
Can you write Python code for it?


I am working on the file "NCPSchool2021/Examples/TMVA_Higgs_Classification_py.ipynb".
The first task is:
Using scikit-learn
here we book some scikit learn packages
Can you write Python code for it?

#factory.BookMethod(loader, ROOT.TMVA.Types.kPyGTB, "PyGTB","H:!V:VarTransform=G:NEstimators=400:LearningRate=0.1:"
#                                                  "MaxDepth=3")
#
#factory.BookMethod(loader, ROOT.TMVA.Types.kPyRandomForest, "PyRandomForest","!V:VarTransform=G:NEstimators=400:"
#                           "Criterion=gini:MaxFeatures=auto:MaxDepth=6:MinSamplesLeaf=3:MinWeightFractionLeaf=0:"
#                            "Bootstrap=kTRUE" )
#
#factory.BookMethod(loader, ROOT.TMVA.Types.kPyAdaBoost, "PyAdaBoost","!V:VarTransform=G:NEstimators=400" )


I am working on the file "NCPSchool2021/RDataFrame/02-rdataframe-collections.ipynb".
The first task is:
Define a new column with operations on RVecs
Can you write Python code for it?

df1 = df.Define("good_pt", "sqrt(px*px + py*py)[E>100]")


I am working on the file "NCPSchool2021/RDataFrame/02-rdataframe-collections.ipynb".
The first task is:
To quickly inspect the data we can export it as a dictionary of numpy arrays thanks to the AsNumpy RDataFrame method. Note that for each row, E is an array of values:
Can you write Python code for it?

npy_dict = df.AsNumpy(["E"])

for row, vec in enumerate(npy_dict["E"]):
    print(f"\nRow {row} contains:\n{vec}")


I am working on the file "NCPSchool2021/RDataFrame/03-rdataframe-features.ipynb".
The first task is:
Cutflow reports
Filters applied to the dataset can be given a name. The Report method will gather information about filter efficiency and show the data flow between subsequent cuts on the original dataset.
Can you write Python code for it?

df = ROOT.RDataFrame("sig_tree", "https://root.cern/files/Higgs_data.root")

filter1 = df.Filter("lepton_eta > 0", "Lepton eta cut")
filter2 = filter1.Filter("lepton_phi < 1", "Lepton phi cut")

rep = df.Report()
rep.Print()


I am working on the file "NCPSchool2021/RDataFrame/03-rdataframe-features.ipynb".
The first task is:
Then let's define another function that takes a float values and computes its square.
Can you write Python code for it?

%%cpp

float square(float val){
    return val * val;
}


I am working on the file "NCPSchool2021/RDataFrame/03-rdataframe-features.ipynb".
The first task is:
Using C++ functions in Python
Since we still want to perform complex operations in Python but plain Python code is prone to be slow and not thread-safe, you can inject C++ functions doing the work in your event loop during runtime. This mechanism uses the C++ interpreter cling shipped with ROOT, making this possible in a single line of code. Let's start by defining a function that will allow us to change the type of a the RDataFrame dataset entry numbers (stored in the special column "rdfentry") from unsigned long long to float.
Can you write Python code for it?

%%cpp

float asfloat(unsigned long long entrynumber){
    return entrynumber;
}


I am working on the file "NCPSchool2021/RDataFrame/03-rdataframe-features.ipynb".
The first task is:
Using all cores of your machine with multi-threaded RDataFrame
RDataFrame can transparently perform multi-threaded event loops to speed up the execution of its actions. Users have to call ROOT::EnableImplicitMT() before constructing the RDataFrame object to indicate that it should take advantage of a pool of worker threads. Each worker thread processes a distinct subset of entries, and their partial results are merged before returning the final values to the user.
RDataFrame operations such as Histo1D or Snapshot are guaranteed to work correctly in multi-thread event loops. User-defined expressions, such as strings or lambdas passed to Filter, Define, Foreach, Reduce or Aggregate will have to be thread-safe, i.e. it should be possible to call them concurrently from different threads.
Can you write Python code for it?

%%time
# Get a first baseline measurement

treename = "Events"
filename = "root://eospublic.cern.ch//eos/opendata/cms/derived-data/AOD2NanoAODOutreachTool/Run2012BC_DoubleMuParked_Muons.root"
df = ROOT.RDataFrame(treename, filename)

df.Sum("nMuon").GetValue()

%%time
# Activate multithreading capabilities
# By default takes all available cores on the machine
ROOT.EnableImplicitMT()

treename = "Events"
filename = "root://eospublic.cern.ch//eos/opendata/cms/derived-data/AOD2NanoAODOutreachTool/Run2012BC_DoubleMuParked_Muons.root"
df = ROOT.RDataFrame(treename, filename)

df.Sum("nMuon").GetValue()

# Disable implicit multithreading when done
ROOT.DisableImplicitMT()


I am working on the file "NCPSchool2021/RDataFrame/03-rdataframe-features.ipynb".
The first task is:
We can now plot the values of the columns in a graph:
Can you write Python code for it?

# Show the two columns created in a graph
c = ROOT.TCanvas()
graph = df2.Graph("a","b")
graph.SetMarkerStyle(20)
graph.SetMarkerSize(0.5)
graph.SetMarkerColor(ROOT.kBlue)
graph.SetTitle("My graph")
graph.Draw("AP")
c.Draw()


I am working on the file "NCPSchool2021/RDataFrame/04-rdataframe-advanced.ipynb".
The first task is:
Distributed RDataFrame
An RDataFrame analysis written in Python can be executed both locally - possibly in parallel on the cores of the machine - and distributedly by offloading computations to external resources, including Spark and Dask clusters. This feature is enabled by the architecture depicted below, which shows that RDataFrame computation graphs can be mapped to different kinds of resources via backends. In this notebook we will exercise the Spark backend, which divides an RDataFrame input dataset in logical ranges and submits computations for each of those ranges to Spark resources.

Create a Spark context
In order to work with a Spark cluster we need a SparkContext object, which represents the connection to that cluster and allows to configure execution-related parameters (e.g. number of cores, memory). When running this notebook from SWAN, a SparkContext object is already created for us when connecting to the selected cluster via the graphical interface. Alternatively, we could create a SparkContext as described in the Spark documentation.
Can you write Python code for it?

import pyspark
sc = pyspark.SparkContext.getOrCreate()


I am working on the file "NCPSchool2021/RDataFrame/04-rdataframe-advanced.ipynb".
The first task is:
Multiple concurrent RDataFrame runs
If your analysis needs multiple RDataFrames to run (for example multiple dataset samples, data vs simulation etc.), the ROOT.RDF.RunGraphs
Can you write Python code for it?

ROOT.EnableImplicitMT()
treename1 = "myDataset"
filename1 = "https://github.com/root-project/root/raw/master/tutorials/dataframe/df017_vecOpsHEP.root"
treename2 = "dataset"
filename2 = "data/example_file.root"

df1 = ROOT.RDataFrame(treename1, filename1)
df2 = ROOT.RDataFrame(treename2, filename2)
h1 = df1.Histo1D("px")
h2 = df2.Histo1D("a")

ROOT.RDF.RunGraphs((h1, h2))

c = ROOT.TCanvas()
h1.Draw()
c.Draw()

c = ROOT.TCanvas()
h2.Draw()
c.Draw()


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
Combining Python and C++ in the same notebook
As we saw in 02-root-python-cpp.ipynb, PyROOT lets us define C++ and call it from Python right away.
Can you write Python code for it?

ROOT.gInterpreter.ProcessLine("void print_integer(int i) { std::cout << i << std::endl; }")

ROOT.print_integer(7)


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
If we draw again the same canvas after activating JS graphics, this time the histogram plot is interactive! Try to hover over it with the mouse and zoom in and out.
Can you write Python code for it?


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
In a notebook, we can use the %%cpp magic to do the same. If %%cpp is present in a cell, its whole content is interpreted and executed as C++, and it has C++ syntax coloring!
Can you write Python code for it?

%%cpp
void print_integer_2(int i) {
   std::cout << i << std::endl;
}


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
Interactive graphics
As mentioned before, we can also display interactive ROOT graphics in a notebook. To do that, we can use the %jsroot magic and turn on the JavaScript mode.
Can you write Python code for it?


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
Note that, besides defining new C++ entities, we can also run C++ statements.
Can you write Python code for it?

%%cpp
std::cout << "Hello from C++!" << std::endl;

print_integer_2(7);


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
Now let's plot the histogram. For that purpose, we first need to create a canvas, then draw the histogram and finally draw the canvas. See how the output is a static image.
Can you write Python code for it?

c = ROOT.TCanvas()
h.Draw()
c.Draw() # don't forget to draw the canvas too!


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
Quickly checking the docs
To take a quick peek at a function signature, we can use the notebook's ? feature:
Can you write Python code for it?


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
Redefining C++ entities
In C++, redefinition of entities such as variables is not allowed. However, ROOT provides a cool redefinition feature that is especially useful in interactive interfaces like notebooks. This feature works for types, functions and variables defined in the global namespace.
You can check it by running the next cells, even multiple times!
Can you write Python code for it?

%%cpp
int foo = 0;

%%cpp
class foo {};

%%cpp
void foo() {}


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
The function we just defined in the previous (C++) cell can be now invoked from the next (Python) cell:
Can you write Python code for it?


I am working on the file "NCPSchool2021/introduction.ipynb".
The first task is:
Type conversions: In the cell above we have passed a Python integer 7 and it has been implicitly converted for us to a C++ integer to pass the function print_integer. This conversion is type-aware so that just the correct type is used in the correct context
Can you write Python code for it?

mylist = [1,2,3]
ROOT.print_integer(mylist)


I am working on the file "NCSS/NCSS_as_point_example.ipynb".
The first task is:
First generate a sample RESTful URL using the NCSS Web Form:
http://data.ncof.co.uk/thredds/ncss/METOFFICE-NWS-AF-WAV-HOURLY/pointDataset.html
Can you write Python code for it?


I am working on the file "NCSS/NCSS_as_point_example.ipynb".
The first task is:
Once you have a sample URL, it's easy to create a different URL programmatically
Can you write Python code for it?

latitude = 61.15
longitude = -9.5
time_start = '2017-06-02T00:00:00Z'
time_end =   '2017-06-17T23:00:00Z'

url = 'http://data.ncof.co.uk/thredds/ncss/METOFFICE-NWS-AF-WAV-HOURLY?var=VHM0&var=VHM0_SW1&var=VHM0_WW&latitude={}&longitude={}&time_start={}&time_end={}&accept=csv'.format(latitude,longitude,time_start,time_end)

df = pd.read_csv(url, parse_dates=True, index_col=0, na_values=[-32767.0],
        skiprows=[0], names=['Lon','Lat','SigHeight(m)','Swell(m)','Wind Waves(m)'])

# drop lon,lat columns
df = df.drop(df.columns[[0,1]], axis=1)

df.head()

# Ugh, NCSS apparently ignores the scale_factor. Shite!  I'll report this bug, but in the meantime...
df = df/100.

df.head()

df.plot(figsize=(12,4),grid='on');


I am working on the file "NCdemo/CombineRuns.ipynb".
The first task is:
In this example, we will only adjust the results for the ClimateChange variable. In practice it would be done to every variable and the associated p-values
Can you write Python code for it?

# Set the name of the var and the pvar
var  = "ClimateChange"
pvar = "ClimateChange.Pvalue"
# Calculate tthe adjusted
VARadj = Weighted_mean(ds_c3, ds_c4, C4frac, var)
VARadj_pvalues = Combinee_Pvalues(ds_c3, ds_c4, C4frac, pvar)
# make some plots
print(var)
plt.figure(var)
VARadj.plot()
plt.show()
print(pvar)
plt.figure(pvar)
VARadj_pvalues.plot()

plt.show()


I am working on the file "NCdemo/CombineRuns.ipynb".
The first task is:
Next step is to load in the relevant datasets.  There are three netcdf files: a C3 run, a C4 run and the fraction of C4 photosyenthetic plants.
Can you write Python code for it?

# The C3 run
ds_c3 = xr.open_dataset("./results/TSSRattribution_Results_C3_125km.nc")
# The C4 run
ds_c4 = xr.open_dataset("./results/TSSRattribution_Results_C3_125km.nc")
# The fraction of C4 plans
C4frac = xr.open_dataset("./data/AUSdemo_SYNMAP_C4Fraction_xrcoarsen_5win.nc")

# ===== print one of the datasets =====
print(ds_c3)


I am working on the file "NCdemo/CombineRuns.ipynb".
The first task is:
The next step is to define a couple of functions. The first function is called Weighted_mean and it will just get the mean of the two runs, weighted by the fraction of C4 plans in each pixel.  The second uses stouffer's method to combine the pvalues for that variable with the C4frac being as weights.
Can you write Python code for it?

def Weighted_mean(ds_c3, ds_c4, C4frac, var):
    # pull out the dataarrays
    da_c3 = ds_c3[var]
    da_c4 = ds_c4[var]
    C4f   = C4frac["C4frac"].values

    #Calculate the weighted mean
    c4_adj = (da_c4 * C4f) + (da_c3 * (1-C4f))

    #return the results
    return c4_adj

def Combinee_Pvalues(ds_c3, ds_c4, C4frac, pvar):
	# Pull out the p values

	# Stack long and land to make 1d array
	da_c3p = ds_c3[pvar].stack(cord=('longitude', 'latitude'))
	da_c4p = ds_c4[pvar].stack(cord=('longitude', 'latitude'))
	C4f_p  = C4frac["C4frac"].stack(cord=('longitude', 'latitude'))
	C3f_p  = 1-C4f_p

	# stack into a 2d array so i can use  np.apply_along_axis
	stacked = np.vstack([da_c3p.values, da_c4p.values, C4f_p.values, C3f_p.values])

	def _combine_2pvalue(array):
		pvals  = array[:2]
		wei    = array[2:]
		if bn.allnan(pvals):
			return np.NaN
		else:
			# Deal with pvalues that are= 0. It happens when the values are smaller
            # than the is possible with float 32.  If they are pased to
			# stats.combine_pvalues it will return nan
			pvals[pvals==0] = 0.000001
			#Deals with places where thing failed to get a p values
			pvals[np.isnan(pvals)] = 1

			# Adjust the pvalues
			sta, pv = stats.combine_pvalues(pvals, method="stouffer", weights=wei)
			return np.array(pv)

	# ===== Apply the 2d pvalue combination function to calculate the new pvalues =====
	res = np.apply_along_axis(_combine_2pvalue, 0, stacked).reshape([1,-1])

	# Convert back intoa a datarray
	pv_adj = da_c3p.copy(data=res)
	# Unstack the lats and lons then return the result
	return pv_adj.unstack().transpose('time', 'latitude', 'longitude').sortby("latitude", ascending=False)


I am working on the file "NERD/DecisionTree/.ipynb_checkpoints/Decision Tree NER Model Results-checkpoint.ipynb".
The first task is:
It appears adding in the morphological features greatly increased classifier performance.<br>
Below, find the underlying decision tree structure representing the classifier.
Can you write Python code for it?

import graphviz
from sklearn.externals import joblib
from Tools import arff_converter
from sklearn import tree

featureset = "o"
entity_name = "adversereaction"

fold = 3
training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
dataset = arff_converter.arff2df(training_data)
dtree = joblib.load('../Models/decisiontree/adversereaction_o/decisiontree_o_adversereaction_train-%i.arff.pkl' % fold)
tree.export_graphviz(dtree, out_file="visual/temptree.dot",
                                  feature_names=dataset.columns.values[:-1],
                                  class_names=["Entity", "Non-Entity"], label='all',
                               filled=True, rounded=True, proportion=False, leaves_parallel=True,
                                 special_characters=True)
with open("visual/temptree.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)


I am working on the file "NERD/DecisionTree/.ipynb_checkpoints/Decision Tree NER Model Results-checkpoint.ipynb".
The first task is:
Rather lackluster performance.
Orthographic + Morphological Features
Can you write Python code for it?

import subprocess

""" Creates models for each fold and runs evaluation with results """
featureset = "om"
entity_name = "adversereaction"

for fold in range(1,1): #training has already been done
    training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
    os.system("python3 decisiontree.py -tr %s" % (training_data))

for fold in range(1,11):
    testing_data = "../ARFF_Files/%s_ARFF/_%s/_test/%s_test-%i.arff" % (entity_name, featureset, entity_name, fold)
    output = subprocess.check_output("python3 evaluate_decisiontree.py -te %s" % (testing_data), shell=True)
    print(output.decode('utf-8'))


I am working on the file "NERD/DecisionTree/Decision Tree NER Model Results.ipynb".
The first task is:
Adverse Reaction Feature Set
Orthographic Features
Can you write Python code for it?

import subprocess

""" Creates models for each fold and runs evaluation with results """
featureset = "o"
entity_name = "adversereaction"

for fold in range(1,1): #training has already been done
    training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
    os.system("python3 decisiontree.py -tr %s" % (training_data))

for fold in range(1,11):
    testing_data = "../ARFF_Files/%s_ARFF/_%s/_test/%s_test-%i.arff" % (entity_name, featureset, entity_name, fold)
    output = subprocess.check_output("python3 evaluate_decisiontree.py -te %s" % (testing_data), shell=True)
    print(output.decode('utf-8'))


I am working on the file "NERD/DecisionTree/Decision Tree NER Model Results.ipynb".
The first task is:
Average Precision:  0.6329762
Average Recall   :  0.0080615

Average F-Measure:  0.0158644

Rather lackluster performance.
Can you write Python code for it?

import graphviz
from sklearn.externals import joblib
from Tools import arff_converter
from sklearn import tree

featureset = "o" #Careful with highly dimensional datasets
entity_name = "adversereaction"

fold = 1  #change this to display a graph of the decision tree structure for a fold
training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
dataset = arff_converter.arff2df(training_data)
dtree = joblib.load('../Models/decisiontree/%s_%s/decisiontree_%s_%s_train-%i.arff.pkl' % (entity_name, featureset, featureset, entity_name,fold))
tree.export_graphviz(dtree,
                     out_file="visual/temptree1.dot",
                     feature_names=dataset.columns.values[:-1],
                     class_names=["Entity", "Non-Entity"], label='all',
                     filled=True, rounded=True, proportion=False, leaves_parallel=True,
                     special_characters=True,
                     max_depth=3  #change for more detail, careful with large datasets
                    )
with open("visual/temptree1.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)
#graphviz.Source(dot_graph).view()
#the above line is a fullscreen alternative, also generates a temporary file that requires manual removal


I am working on the file "NERD/DecisionTree/Decision Tree NER Model Results.ipynb".
The first task is:
Average Precision:  0.6423055
Average Recall   :  0.4637322

Average F-Measure:  0.5329495

It appears adding in the morphological features greatly increased classifier performance.

Below, find the underlying decision tree structure representing the classifier.
Can you write Python code for it?

import graphviz
from sklearn.externals import joblib
from Tools import arff_converter
from sklearn import tree

featureset = "om" #Careful with highly dimensional datasets
entity_name = "adversereaction"

fold = 2  #change this to display a graph of the decision tree structure for a fold
training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
dataset = arff_converter.arff2df(training_data)
dtree = joblib.load('../Models/decisiontree/%s_%s/decisiontree_%s_%s_train-%i.arff.pkl' % (entity_name, featureset, featureset, entity_name,fold))
tree.export_graphviz(dtree,
                     out_file="visual/temptree.dot",
                     feature_names=dataset.columns.values[:-1],
                     class_names=["Entity", "Non-Entity"], label='all',
                     filled=True, rounded=True, proportion=False, leaves_parallel=True,
                     special_characters=True,
                     max_depth=3  #change for more detail, careful with large datasets
                    )
with open("visual/temptree.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)
#graphviz.Source(dot_graph).view()
#the above line is a fullscreen alternative, also generates a temporary file that requires manual removal


I am working on the file "NERD/DecisionTree/Decision Tree NER Model Results.ipynb".
The first task is:
Average Precision:  0.6639918
Average Recall   :  0.6856795

Average F-Measure:  0.6662661

The addition of lexical features clearly assists the classifiers recall of the minority class. It appears, however, that the inclusion of lexical features leads to lowering of classifier precision relative to recall. This suggests that the inclusion of lexical features introduces noise that skews the decision boundary towards the majority 'Non-entity' class but still is necessary to strengthen the boundary around the minority class as shown by the higher recall scores.
The undersampling of majority class instances coupled with feature selection may lead to favorable results on this set of combined features.
Can you write Python code for it?

import graphviz
from sklearn.externals import joblib
from Tools import arff_converter
from sklearn import tree

featureset = "omt" #Careful with large datasets
entity_name = "adversereaction"

fold = 10  #change this to display a graph of the decision tree structure for a fold
training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
dataset = arff_converter.arff2df(training_data)
dtree = joblib.load('../Models/decisiontree/%s_%s/decisiontree_%s_%s_train-%i.arff.pkl' % (entity_name, featureset, featureset, entity_name,fold))
tree.export_graphviz(dtree,
                     out_file="visual/temp1.dot",
                     feature_names=dataset.columns.values[:-1],
                     class_names=["Entity", "Non-Entity"], label='all',
                     filled=True, rounded=True, proportion=False, leaves_parallel=True,
                     special_characters=True,
                     max_depth=4  #change for more detail, careful with large datasets
                    )
with open("visual/temp1.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)
#graphviz.Source(dot_graph).view()
#the above line is a fullscreen alternative, also generates a temporary file that requires manual removal


I am working on the file "NERD/DecisionTree/Decision Tree NER Model Results.ipynb".
The first task is:
The above tree suggests that solely orthograhic information may not be enough to train the classifier. Notice left subtree of the root node splits on the 'single_character' feature. Clearly an adverse reaction would not be a single character yet the tree predicts that any instance where 'single_character' holds the value of 1 would in-fact be an adverse reaction.
<hr>
Orthographic + Morphological Features
Can you write Python code for it?


I am working on the file "NERD/DecisionTree/Decision Tree NER Model Results.ipynb".
The first task is:
The decision tree structure above confirms that the features maximizing split purity are dominantly morphological; that is, orthographic features may just be serving as noise if they are included.
<hr>
Orthographic + Morphological + Lexical Features
Can you write Python code for it?

import subprocess

""" Creates models for each fold and runs evaluation with results """
featureset = "omt"
entity_name = "adversereaction"

for fold in range(1,1): #training has already been done
    training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
    os.system("python3 decisiontree.py -tr %s" % (training_data))

for fold in range(1,11):
    testing_data = "../ARFF_Files/%s_ARFF/_%s/_test/%s_test-%i.arff" % (entity_name, featureset, entity_name, fold)
    output = subprocess.check_output("python3 evaluate_decisiontree.py -te %s" % (testing_data), shell=True)
    print(output.decode('utf-8'))


I am working on the file "NERD/DecisionTreeRandomForestEnsemble/.ipynb_checkpoints/Random Forest Ensemble NER Model Results-checkpoint.ipynb".
The first task is:
It appears adding in the morphological features greatly increased classifier performance.<br>
Below, find the underlying decision tree structure representing the classifier.
Can you write Python code for it?

import graphviz
from sklearn.externals import joblib
from Tools import arff_converter
from sklearn import tree

featureset = "o" #Careful with high dimensional datasets
entity_name = "adversereaction"

fold = 1  #change this to display a graph of the decision tree structure for a fold
training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
dataset = arff_converter.arff2df(training_data)
dtree = joblib.load('../Models/decisiontree/%s_%s/decisiontree_%s_%s_train-%i.arff.pkl' % (entity_name, featureset, featureset, entity_name,fold))
tree.export_graphviz(dtree,
                     out_file="visual/temptree.dot",
                     feature_names=dataset.columns.values[:-1],
                     class_names=["Entity", "Non-Entity"], label='all',
                     filled=True, rounded=True, proportion=False, leaves_parallel=True,
                     special_characters=True,
                     max_depth=3  #change for more detail, careful with large datasets
                    )
with open("visual/temptree.dot") as f:
    dot_graph = f.read()
graphviz.Source(dot_graph)
#graphviz.Source(dot_graph).view()
#the above line is a fullscreen alternative, also generates a temporary file that requires manual removal


I am working on the file "NERD/DecisionTreeRandomForestEnsemble/.ipynb_checkpoints/Random Forest Ensemble NER Model Results-checkpoint.ipynb".
The first task is:
Rather lackluster performance.
Orthographic + Morphological Features
Can you write Python code for it?


I am working on the file "NERD/DecisionTreeRandomForestEnsemble/Random Forest Ensemble NER Model Results.ipynb".
The first task is:
It appears adding in the morphological features greatly increased classifier performance.<br>
Below, find the underlying decision tree structure representing the classifier.
Orthographic + Morphological + Lexical Features
Can you write Python code for it?

import subprocess

""" Creates models for each fold and runs evaluation with results """
featureset = "omt"
entity_name = "adversereaction"

for fold in range(1,1): #training has already been done
    training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
    os.system("python3 decisiontree.py -tr %s" % (training_data))

for fold in range(1,11):
    testing_data = "../ARFF_Files/%s_ARFF/_%s/_test/%s_test-%i.arff" % (entity_name, featureset, entity_name, fold)
    output = subprocess.check_output("python3 evaluate_randomforest.py -te %s" % (testing_data), shell=True)
    print(output.decode('utf-8'))


I am working on the file "NERD/DecisionTreeRandomForestEnsemble/Random Forest Ensemble NER Model Results.ipynb".
The first task is:
Orthographic features alone contribute to a relatively high precision but very low recall. This implies that orthographic features alone are not enough to carve out the decision boundary for all the positive instances hence the low recall.However,the decision boundary created is very selective as indicated by the high precision.
Orthographic + Morphological Features
Can you write Python code for it?

import subprocess

""" Creates models for each fold and runs evaluation with results """
featureset = "om"
entity_name = "adversereaction"

for fold in range(1,1): #training has already been done
    training_data = "../ARFF_Files/%s_ARFF/_%s/_train/%s_train-%i.arff" % (entity_name, featureset, entity_name, fold)
    os.system("python3 decisiontree.py -tr %s" % (training_data))

for fold in range(1,11):
    testing_data = "../ARFF_Files/%s_ARFF/_%s/_test/%s_test-%i.arff" % (entity_name, featureset, entity_name, fold)
    output = subprocess.check_output("python3 evaluate_randomforest.py -te %s" % (testing_data), shell=True)
    print(output.decode('utf-8'))


I am working on the file "NERSCJupyterGuide.ipynb".
The first task is:
Head start for Developers
For developers, it is preferable to do updates with git pull
from a true terminal window, or follow other standard git practice of
merging in upstream changes. (e.g. git fetch)
The main development of imaginglss is done this way:

One terminal window connected to NERSC
One notebook connected to http://jupyter.nersc.gov
Write up some ideas in the notebook
git checkout -b mybigidea
implement it from the terminal window
Reload the notebook kernel and test the implementation
If it works, git add, git commit, git push, and file a Pull Request.
If it does not work, go to 'implement' step.
Wait till the feature is merged,
git checkout master; git branch -d mybigidea.

Reinstall
Usually not needed, but if things go severely wrong, try this.
Can you write Python code for it?

%%bash

cd ~/imaginglss
/anaconda/bin/pip uninstall imaginglss
/anaconda/bin/pip install --user -e .

import imaginglss
print imaginglss

decals = imaginglss.DECALS('/global/project/projectdirs/m779/imaginglss/dr2.conf.py')


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in source-code below. In computer programming, it is best to use variables and constants rather than having to hard-code specific numbers every time that number is used.
Can you write Python code for it?

# MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
ima_shape = (img_size, img_size)

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Finally we have the placeholder variable for the true classes of each image in the placeholder variable x. These are integers and the dimensionality of this placeholder variable is se to [None] which means the placeholder variable is a one-dimensional vector or arbitrary length.
Can you write Python code for it?

y_true_cls = tf.placeholder(tf.int64, [None])


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Function for performing a number of optimization iterations so as to gradually improve the weights and biases of the model. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.
Can you write Python code for it?

def optimize(num_iterations):
    for i in range(num_iterations):
        x_batch, y_true_batch = data.train.next_batch(batch_size)

        feed_dict_train = {x: x_batch,
                          y_true: y_true_batch}
        session.run(optimizer, feed_dict = feed_dict_train)


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Helper-functin for plotting images
Function used to plot 9 images in a 3*3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_image(images, cls_true, cls_pred=None):
    assert len(image) == len(cls_true) == 9

    #Create figure with 3*3 sub-plots.
    fig, axes = plt.subplots(3,3)
    fig.subplots_adjust(hspace = 0.3, wspace = 0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred:{1}".format(cls_true[i],cls_pred[i])

        ax.set_xlabel(xlabel)

        #Remove ticks from the plot
        ax.set_xticks([])
        ax.set_yticks([])


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Model
This simple mathematical model multiplies in the placeholder variable x with the weights and then holdsthe biases.
The result is a matrix of shape [num_images, num_classes] because x has shape [num_images, img_size_flat] and weights has shape [img_size_flat, num_classes], so the multiplication of those two matricies is a matrix shape [num_images, num_classes] and then the biases vector is added to each row of matrix.
Note that the name logits is typical TensorFlow terminology, but other people may call the variable something else.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Next we have the placeholder variable for the true labels associated with the images that were input in the placeholder variable x. The shape of this variable is [None, num_clssses] which means it may hold an arbitrary number of labels and each label is a vector of length num_classes which is 10 in this case.
Can you write Python code for it?

y_true = tf.placeholder(tf.float32, [None, num_classes])


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Now logits is a matrix with num_images rows and num_classes columns, where the element of the ith row and jth column is an estimate of how likely the ith input image is to be of jth class.
However, these estimates are a bit rouh and difficult to interpret because the numbers may be very small or large, so we want to normalize them so that each row of the logits matrix sums to one, and each element is limited between zero and one. This is calculated using the so-called softmax function and the result is stored in y_pred.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Optimization method
Now that we have a cost measure that must be minimized, we can then reduce an optimizer. in this case it is the basic form of Gradient Descent where the step-size is set to 0.5.
Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Performance measures
We need a few more performance measures to display the progress to the user.
This is a vector of booleans whether the predicted class equals the true class of each image.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

#Get the first images from the test-set.
images = data.test.images[0:9]

#Get the true classes for those images
cls_true = data.test.cls[0:9]

#Plot the images and labels using our helper-function above
plot_images(images = images, cls_true=cls_true)


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
TensorFlow Graph
The entir purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathmatical operation at a time.
TensorFlow can also take automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used to change the input to the graph
Model variables that are going to be optimized so as to make the model perform better
The model which is essentially just a mathematical function that calculates some output given the input in the placeholder variables and the model variables.
A cost measure that can be used to guide the optimization of the variables.
An optimization method which updates the variables of the model.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to displayed using TensorBoard, which is not covered in this tutorial.
Placeholder variables
Placeholder variables serve as the input to the graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means...
Can you write Python code for it?

x = tf.placeholder(tf.float32, [None, img_size_flat])


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
The predicted class can be calculated from the y_pred matrix by taking the index of the largest element in each row.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension = 1)


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
This calculates the classification accuracy by first type-casting the vector of booleans to floats, so that False becomes 0, and True becomes 1, and then calculating the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
We also need the classes as single numbers for various comparisons and performance measures, so we convert the One-Hot encoded vectors to a single number by taking the index of the highest element. Note that the word 'class' is a keyword used in Python so we need to use the name 'cls' instead.
Can you write Python code for it?

data.test.cls = np.array([label.argmax() for label in data.test.labels])


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
We can now see the class for the first five images in the test-set. Compare these to One-Hot encoded vector above. For example, the class for the first image is 7, which corresponds to a One-Hot encoded vector where all elements are zero except for the element with index 7.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/.ipynb_checkpoints/Tensorflow Tutorial_1_GY-checkpoint.ipynb".
The first task is:
We have now calculated the cross-entropy for each of the image classifications so we have a measure of how well the model performs on each image individually. But in order to use the cross-entropy to guide the optimization of the model's variables we need a single scalar value, so we simply take the average of the cross-entropy for all the entire classifications.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Data dimensions
The data dimensions are used in several places in the source-code below. In computer programming it is generally best to use variables and constants rather than having to hard-code specific numbers every time that number is used. This means the numbers only have to be changed in one single place. Ideally these would be inferred from the data that has been read, but here we just write the numbers.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Finally we have the placeholder variable for the true class of each image in the placeholder variable x. These are integers and the dimensionality of this placeholder variable is set to [None] which means the placeholder variable is a one-dimensional vector of arbitrary length.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors():
    # Use TensorFlow to get a list of boolean values
    # whether each test-image has been correctly classified,
    # and a list for the predicted class of each image.
    correct, cls_pred = session.run([correct_prediction, y_pred_cls],
                                    feed_dict=feed_dict_test)

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Function for printing and plotting the confusion matrix using scikit-learn.
Can you write Python code for it?

def print_confusion_matrix():
    # Get the true classifications for the test-set.
    cls_true = data.test.cls

    # Get the predicted classifications for the test-set.
    cls_pred = session.run(y_pred_cls, feed_dict=feed_dict_test)

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)

    # Make various adjustments to the plot.
    plt.tight_layout()
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Helper-function to plot the model weights
Function for plotting the weights of the model. 10 images are plotted, one for each digit that the model is trained to recognize.
Can you write Python code for it?

def plot_weights():
    # Get the values for the weights from the TensorFlow variable.
    w = session.run(weights)

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w)
    w_max = np.max(w)

    # Create figure with 3x4 sub-plots,
    # where the last 2 sub-plots are unused.
    fig, axes = plt.subplots(3, 4)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Only use the weights for the first 10 sub-plots.
        if i<10:
            # Get the weights for the i'th digit and reshape it.
            # Note that w.shape == (img_size_flat, 10)
            image = w[:, i].reshape(img_shape)

            # Set the label for the sub-plot.
            ax.set_xlabel("Weights: {0}".format(i))

            # Plot the image.
            ax.imshow(image, vmin=w_min, vmax=w_max, cmap='seismic')

        # Remove ticks from each sub-plot.
        ax.set_xticks([])
        ax.set_yticks([])


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Helper-functions to show performance
Dict with the test-set data to be used as input to the TensorFlow graph. Note that we must use the correct names for the placeholder variables in the TensorFlow graph.
Can you write Python code for it?

feed_dict_test = {x: data.test.images,
                  y_true: data.test.labels,
                  y_true_cls: data.test.cls}


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets("data/MNIST/", one_hot=True)


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Model
This simple mathematical model multiplies the images in the placeholder variable x with the weights and then adds the biases.
The result is a matrix of shape [num_images, num_classes] because x has shape [num_images, img_size_flat] and weights has shape [img_size_flat, num_classes], so the multiplication of those two matrices is a matrix with shape [num_images, num_classes] and then the biases vector is added to each row of that matrix.
Note that the name logits is typical TensorFlow terminology, but other people may call the variable something else.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Next we have the placeholder variable for the true labels associated with the images that were input in the placeholder variable x. The shape of this placeholder variable is [None, num_classes] which means it may hold an arbitrary number of labels and each label is a vector of length num_classes which is 10 in this case.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Now logits is a matrix with num_images rows and num_classes columns, where the element of the $i$'th row and $j$'th column is an estimate of how likely the $i$'th input image is to be of the $j$'th class.
However, these estimates are a bit rough and difficult to interpret because the numbers may be very small or large, so we want to normalize them so that each row of the logits matrix sums to one, and each element is limited between zero and one. This is calculated using the so-called softmax function and the result is stored in y_pred.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
One-Hot Encoding
The data-set has been loaded as so-called One-Hot encoding. This means the labels have been converted from a single number to a vector whose length equals the number of possible classes. All elements of the vector are zero except for the $i$'th element which is one and means the class is $i$. For example, the One-Hot encoded labels for the first 5 images in the test-set are:
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Performance after 1 optimization iteration
Already after a single optimization iteration, the model has increased its accuracy on the test-set to 40.7% up from 9.8%. This means that it mis-classifies the images about 6 out of 10 times, as demonstrated on a few examples below.
Can you write Python code for it?

optimize(num_iterations=1)

print_accuracy()

plot_example_errors()


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Performance after 10 optimization iterations
Can you write Python code for it?

# We have already performed 1 iteration.
optimize(num_iterations=9)

print_accuracy()

plot_example_errors()

plot_weights()


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is 9.8%. This is because the model has only been initialized and not optimized at all, so it always predicts that the image shows a zero digit, as demonstrated in the plot below, and it turns out that 9.8% of the images in the test-set happens to be zero digits.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Performance measures
We need a few more performance measures to display the progress to the user.
This is a vector of booleans whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Plot a few images to see if data is correct
I changed here from test plot to train set plot to see how the function works.
Can you write Python code for it?

# Get the first images from the test-set.
#images = data.test.images[0:9]
images = data.train.images[0:9]

# Get the true classes for those images.
#cls_true = data.test.cls[0:9]
cls_true = data.train.cls[0:9]

# Plot the images and labels using our helper-function above.
#plot_images(images=images, cls_true=cls_true)
plot_images(images=images, cls_true=cls_true)


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
TensorFlow Graph
The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.
TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used to change the input to the graph.
Model variables that are going to be optimized so as to make the model perform better.
The model which is essentially just a mathematical function that calculates some output given the input in the placeholder variables and the model variables.
A cost measure that can be used to guide the optimization of the variables.
An optimization method which updates the variables of the model.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial.
Placeholder variables
Placeholder variables serve as the input to the graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just...
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70.000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
The model has now been trained for 1000 optimization iterations, with each iteration using 100 images from the training-set. Because of the great variety of the images, the weights have now become difficult to interpret and we may doubt whether the model truly understands how digits are composed from lines, or whether the model has just memorized many different variations of pixels.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
The predicted class can be calculated from the y_pred matrix by taking the index of the largest element in each row.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
This calculates the classification accuracy by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then calculating the average of these numbers.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
Variables to be optimized
Apart from the placeholder variables that were defined above and which serve as feeding input data into the model, there are also some model variables that must be changed by TensorFlow so as to make the model perform better on the training data.
The first variable that must be optimized is called weights and is defined here as a TensorFlow variable that must be initialized with zeros and whose shape is [img_size_flat, num_classes], so it is a 2-dimensional tensor (or matrix) with img_size_flat rows and num_classes columns.
Can you write Python code for it?

weights = tf.Variable(tf.zeros([img_size_flat, num_classes]))


I am working on the file "NEU/Guowei_Yang_DL/01_Simple_Linear_Model.ipynb".
The first task is:
We also need the classes as single numbers for various comparisons and performance measures, so we convert the One-Hot encoded vectors to a single number by taking the index of the highest element. Note that the word 'class' is a keyword used in Python so we need to use the name 'cls' instead.
Can you write Python code for it?

data.test.cls = np.array([label.argmax() for label in data.test.labels])
data.train.cls = np.array([label.argmax() for label in data.train.labels])


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Cost-function to be optimized
To make the model better at classifying the input images, we must somehow change the variables for weights and biases. To do this we first need to know how well the model currently performs by comparing the predicted output of the model y_pred to the desired output y_true.
The cross-entropy is a performance measure used in classification. The cross-entropy is a continuous function that is always positive and if the predicted output of the model exactly matches the desired output the the cross-entropy equals zero. The goal of optimization if therefore to minimize the cross-entropy so it gets as close to zero as possible by changing the weights and biases of the model.
TensorFlow has a built-in function for calculating the cross-entropy. Note that it uses the values of the logits because it also calculates the softmax internally.
Can you write Python code for it?

cross-entropy = tf.nn.softmax_cross_entropy_logits(logits=logits,
                                                   labels=y_true)


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in source-code below. In computer programming, it is best to use variables and constants rather than having to hard-code specific numbers every time that number is used.
Can you write Python code for it?

# MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
ima_shape = (img_size, img_size)

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Function for performing a number of optimization iterations so as to gradually improve the weights and biases of the model. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.
Can you write Python code for it?

def optimize(num_iterations):
    for i in range(num_iterations):
        x_batch, y_true_batch = data.train.next_batch(batch_size)

        feed_dict_train = {x: x_batch,
                          y_true: y_true_batch}
        session.run(optimizer, feed_dict = feed_dict_train)


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Helper-functin for plotting images
Function used to plot 9 images in a 3*3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Initilize variables
The variables for weights and biases must be initilized before we start optimizing them.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Model
This simple mathematical model multiplies in the placeholder variable x with the weights and then holdsthe biases.
The result is a matrix of shape [num_images, num_classes] because x has shape [num_images, img_size_flat] and weights has shape [img_size_flat, num_classes], so the multiplication of those two matricies is a matrix shape [num_images, num_classes] and then the biases vector is added to each row of matrix.
Note that the name logits is typical TensorFlow terminology, but other people may call the variable something else.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Now logits is a matrix with num_images rows and num_classes columns, where the element of the ith row and jth column is an estimate of how likely the ith input image is to be of jth class.
However, these estimates are a bit rouh and difficult to interpret because the numbers may be very small or large, so we want to normalize them so that each row of the logits matrix sums to one, and each element is limited between zero and one. This is calculated using the so-called softmax function and the result is stored in y_pred.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
One-Hot Encoding
The data set has been loaded and as so-called One-Hot encoding. This means the labels have been converted to a single number to a vector whose length equals the number of possible classes. All elements of the vector are zero except for the i^th element which is one and means the class is i. For example, the One-Hot encoded for the first 5 images in the test-set are:
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Performance measures
We need a few more performance measures to display the progress to the user.
This is a vector of booleans whether the predicted class equals the true class of each image.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
TensorFlow Graph
The entir purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathmatical operation at a time.
TensorFlow can also take automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used to change the input to the graph
Model variables that are going to be optimized so as to make the model perform better
The model which is essentially just a mathematical function that calculates some output given the input in the placeholder variables and the model variables.
A cost measure that can be used to guide the optimization of the variables.
An optimization method which updates the variables of the model.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to displayed using TensorBoard, which is not covered in this tutorial.
Placeholder variables
Placeholder variables serve as the input to the graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means...
Can you write Python code for it?

x = tf.placeholder(tf.float32, [None, img_size_flat])


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
The MNIST data set has now been loaded and it consists of 70,000 images and associated labels.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validatin-set:\t{}".format(len(data.validation.labels)))


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
This calculates the classification accuracy by first type-casting the vector of booleans to floats, so that False becomes 0, and True becomes 1, and then calculating the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
We also need the classes as single numbers for various comparisons and performance measures, so we convert the One-Hot encoded vectors to a single number by taking the index of the highest element. Note that the word 'class' is a keyword used in Python so we need to use the name 'cls' instead.
Can you write Python code for it?

data.test.cls = np.array([label.argmax() for label in data.test.labels])


I am working on the file "NEU/Guowei_Yang_DL/Tensorflow Tutorial_1_GY.ipynb".
The first task is:
We have now calculated the cross-entropy for each of the image classifications so we have a measure of how well the model performs on each image individually. But in order to use the cross-entropy to guide the optimization of the model's variables we need a single scalar value, so we simply take the average of the cross-entropy for all the entire classifications.
Can you write Python code for it?


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Autoencoders.ipynb".
The first task is:
As well as the decoder model:
Can you write Python code for it?

# create a placeholder for an encoded (32-dimensional) input
encoded_input = Input(shape=(encoding_dim,))
# retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]
# create the decoder model
decoder = Model(encoded_input, decoder_layer(encoded_input))


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Autoencoders.ipynb".
The first task is:
Deep autoencoder
We do not have to limit ourselves to a single layer as encoder or decoder, we could instead use a stack of layers, such as:
Can you write Python code for it?

input_img = Input(shape=(784,))
encoded = Dense(128, activation='relu')(input_img)
encoded = Dense(64, activation='relu')(encoded)
encoded = Dense(32, activation='relu')(encoded)

decoded = Dense(64, activation='relu')(encoded)
decoded = Dense(128, activation='relu')(decoded)
decoded = Dense(784, activation='sigmoid')(decoded)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Autoencoders.ipynb".
The first task is:
Let's also create a separate encoder model:
Can you write Python code for it?

# this model maps an input to its encoded representation
encoder = Model(input_img, encoded)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Autoencoders.ipynb".
The first task is:
Let's train this model for 100 epochs (with the added regularization the model is less likely to overfit and can be trained longer).
Can you write Python code for it?

autoencoder.compile(optimizer='adagrad', loss='binary_crossentropy')
autoencoder.fit(x_train, x_train,
                epochs=250,
                batch_size=25,
                shuffle=True,
                validation_data=(x_test, x_test))


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Autoencoders.ipynb".
The first task is:
Let's try this:
Can you write Python code for it?

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

autoencoder.fit(x_train, x_train,
                epochs=100,
                batch_size=100,
                shuffle=True,
                validation_data=(x_test, x_test))


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Autoencoders.ipynb".
The first task is:
Now let's train our autoencoder to reconstruct MNIST digits.
First, we'll configure our model to use a per-pixel categorical crossentropy loss, and the Adagrad optimizer:
Can you write Python code for it?


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Autoencoders.ipynb".
The first task is:
The models ends with a train loss of 0.1887 and test loss of 0.876. The difference between the two is mostly due to the regularization term being added to the loss during training (worth about 0.01).
Here's a visualization of our new results:
Can you write Python code for it?

# encode and decode some digits
# note that we take them from the *test* set
encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)

# use Matplotlib
import matplotlib.pyplot as plt

n = 10  # how many digits we will display
plt.figure(figsize=(20, 4))
for i in range(n):
    # display original
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(x_test[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # display reconstruction
    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Autoencoders.ipynb".
The first task is:
We will normalize all values between 0 and 1 and we will flatten the 28x28 images into vectors of size 784.
Can you write Python code for it?

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
print(x_train.shape)
print(x_test.shape)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Basics_tf.ipynb".
The first task is:
See, the loss's function is lower than what we got before.
Can you write Python code for it?

optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss_function)

sess.run(init) # reset values to incorrect defaults.
for i in range(1000):
  sess.run(train, {x: [1, 2, 3, 4], B: [0,-5,-4,-3]})

print(sess.run([M, C]))


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Basics_tf.ipynb".
The first task is:
That is totally not okay as is nearly 80 and its too high. But if we want to lower our loss_function, we need to fix the input values.
Can you write Python code for it?

fixM = tf.assign(M,-1.)
fixC = tf.assign(C, 1)
sess.run([fixM,fixC])

print(sess.run(loss_function, {x : [1,2,3,4], B : [0,-5,-4,-3]}))


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Basics_tf.ipynb".
The first task is:
We werent even close to guessing these values, but we got the exact and accurate values of the paramters defined earlier through proper training of the model.
Lets sum up the steps of the linear model.
Can you write Python code for it?

import tensorflow as tf

# Model parameters
M = tf.Variable([.6], dtype=tf.float32)
C = tf.Variable([-.6], dtype=tf.float32)
# Model input and output
x = tf.placeholder(tf.float32)
y = M * x + C

B = tf.placeholder(tf.float32)

# loss_function
loss_function = tf.reduce_sum(tf.square(y - B)) # sum of the squares

# optimizer
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss_function)

# training data
x_train = [1, 2, 3, 4]
y_train = [0,-5,-4,-3]

# training loop
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init) # reset values to wrong
for i in range(1000):
  sess.run(train, {x: x_train, B : [0,-5,-4,-3]})

# evaluate training accuracy
curr_M, curr_C, curr_loss_f = sess.run([M, C, loss_function], {x: x_train, B: y_train})
print("W: %s b: %s loss: %s"%(curr_M, curr_C, curr_loss_f))

import tensorflow as tf
# NumPy is often used to load, manipulate and preprocess data.
import numpy as np

# Declare list of features. We only have one numeric feature. There are many
# other types of columns that are more complicated and useful.
feature_cols = [tf.feature_column.numeric_column("a", shape=[1])]

# An estimator is the front end to invoke training (fitting) and evaluation
# (inference). There are many predefined types like linear regression,
# linear classification, and many neural network classifiers and regressors.
# The following code provides an estimator that does linear regression.
estimator = tf.estimator.LinearRegressor(feature_columns=feature_cols)

# TensorFlow provides many helper methods to read and set up data sets.
# Here we use two data sets: one for training and one for evaluation
# We have to tell the function how many batches
# of data (num_epochs) we want and how big each batch should be.
a_train = np.array([0., 5., 3., 4.])
y_train = np.array([0., -6., -7., -3.])
a_eval = np.array([2., 5., 8., 10.])
y_eval = np.array([-1.01, -6.1, -7., 4.])
input_fun = tf.estimator.inputs.numpy_input_fn(
    {"a": a_train}, y_train, batch_size=4,...


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/CIFAR_10-Keras.ipynb".
The first task is:
As is good practice, we next initialize the random number seed with a constant to ensure the results are reproducible.
Can you write Python code for it?

# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/CIFAR_10-Keras.ipynb".
The first task is:
Let’s start off by defining a simple CNN structure as a baseline and evaluate how well it performs on the problem.
We will use a structure with two convolutional layers followed by max pooling and a flattening out of the network to fully connected layers to make predictions.
Our baseline network structure can be summarized as follows:

Convolutional input layer, 32 feature maps with a size of 3×3, a rectifier activation function and a weight constraint of max norm set to 3.
Dropout set to 20%.
Convolutional layer, 32 feature maps with a size of 3×3, a rectifier activation function and a weight constraint of max norm set to 3.
Max Pool layer with size 2×2.
Flatten layer.
Fully connected layer with 512 units and a rectifier activation function.
Dropout set to 50%.
Fully connected output layer with 10 units and a softmax activation function.

A logarithmic loss function is used with the stochastic gradient descent optimization algorithm configured with a large momentum and weight decay start with a learning rate of 0.01.
Can you write Python code for it?

# Create the model
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(3, 32, 32), padding='same', activation='relu', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same', kernel_constraint=maxnorm(3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(512, activation='relu', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
# Compile model
epochs = 10
lrate = 0.01
decay = lrate/epochs
sgd = SGD(lr=lrate, momentum=0.9, decay=decay, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer='adagrad', metrics=['categorical_accuracy'])
print(model.summary())


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/CIFAR_10-Keras.ipynb".
The first task is:
Next we can load the CIFAR-10 dataset.
Can you write Python code for it?

# load data
(X_train, y_train), (X_test, y_test) = cifar10.load_data()


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/CIFAR_10-Keras.ipynb".
The first task is:
Simple Convolutional Neural Network for CIFAR-10
The CIFAR-10 problem is best solved using a Convolutional Neural Network (CNN).
We can quickly start off by defining all of the classes and functions we will need in this example.
Can you write Python code for it?

# Simple CNN model for CIFAR-10
import numpy
from keras.datasets import cifar10
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import Flatten
from keras.constraints import maxnorm
from keras.optimizers import SGD
from keras.layers.convolutional import Conv2D
from keras.layers.convolutional import MaxPooling2D
from keras.utils import np_utils
from keras import backend as K
K.set_image_dim_ordering('th')


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/CIFAR_10-Keras.ipynb".
The first task is:
The pixel values are in the range of 0 to 255 for each of the red, green and blue channels.
It is good practice to work with normalized data. Because the input values are well understood, we can easily normalize to the range 0 to 1 by dividing each value by the maximum observation which is 255.
Note, the data is loaded as integers, so we must cast it to floating point values in order to perform the division.
Can you write Python code for it?

# normalize inputs from 0-255 to 0.0-1.0
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train = X_train / 255.0
X_test = X_test / 255.0


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/CIFAR_10-Keras.ipynb".
The first task is:
We can fit and evaluate this model using the same a procedure above and the same number of epochs but a larger batch size of 64, found through some minor experimentation.
Can you write Python code for it?

numpy.random.seed(seed)
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=epochs, batch_size=64)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/CIFAR_10-Keras.ipynb".
The first task is:
We can fit this model with 10 epochs and a batch size of 32.
A small number of epochs was chosen to help keep this tutorial moving. Normally the number of epochs would be one or two orders of magnitude larger for this problem.
Once the model is fit, we evaluate it on the test dataset and print out the classification accuracy.
Can you write Python code for it?

# Fit the model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=epochs, batch_size=32)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)
print("Accuracy: %.2f%%" % (scores[1]*100))


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/CIFAR_10-Keras.ipynb".
The first task is:
We can improve the accuracy significantly by creating a much deeper network.
Larger Convolutional Neural Network for CIFAR-10
We have seen that a simple CNN performs poorly on this complex problem. In this section we look at scaling up the size and complexity of our model.
Let’s design a deep version of the simple CNN above. We can introduce an additional round of convolutions with many more feature maps. We will use the same pattern of Convolutional, Dropout, Convolutional and Max Pooling layers.
This pattern will be repeated 3 times with 32, 64, and 128 feature maps. The effect be an increasing number of feature maps with a smaller and smaller size given the max pooling layers. Finally an additional and larger Dense layer will be used at the output end of the network in an attempt to better translate the large number feature maps to class values.
We can summarize a new network architecture as follows:
Convolutional input layer, 32 feature maps with a size of 3×3 and a rectifier activation function.
Dropout layer at 20%.
Convolutional layer, 32 feature maps with a size of 3×3 and a rectifier activation function.
Max Pool layer with size 2×2.
Convolutional layer, 64 feature maps with a size of 3×3 and a rectifier activation function.
Dropout layer at 20%.
Convolutional layer, 64 feature maps with a size of 3×3 and a rectifier activation function.
Max Pool layer with size 2×2.
Convolutional layer, 128 feature maps with a size of 3×3 and a rectifier activation function.
Dropout layer at 20%.
Convolutional layer,128 feature maps with a size of 3×3 and a rectifier activation function.
Max Pool layer with size 2×2.
Flatten layer.
Dropout layer at 20%.
Fully connected layer with 1024 units and a rectifier activation function.
Dropout layer at 20%.
Fully connected layer with 512 units and a rectifier activation function.
Dropout layer at 20%.
Fully connected output layer with 10 units and a softmax activation function.
We can very easily define this network topology in Keras, as follows:
Can you write Python code for it?

# Create the model

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(3, 32, 32), activation='relu', padding='same'))
model.add(Dropout(0.2))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(Dropout(0.2))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(Dropout(0.2))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dropout(0.2))
model.add(Dense(1024, activation='relu', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(num_classes, activation='softmax'))

# Compile model

epochs = 10
lrate = 0.01
decay = lrate/epochs
sgd = SGD(lr=lrate, momentum=0.9, decay=decay, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer='adagrad', metrics=['categorical_accuracy'])
print(model.summary())


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Creditcard Fraud detection using Autoencoders.ipynb".
The first task is:
Exploration
Can you write Python code for it?


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Creditcard Fraud detection using Autoencoders.ipynb".
The first task is:
Let's have a more graphical representation:
Can you write Python code for it?

f, axes = plt.subplots(nrows = 2, ncols = 1, sharex = True)

axes[0].hist(normal.Amount, bins = 100)
axes[0].set_xlim([0,20000])
axes[0].set_ylim([0,10000])
axes[0].set_title('Normal')

axes[1].hist(frauds.Amount, bins = 50)
axes[1].set_xlim([0,10000])
axes[1].set_ylim([0,200])
axes[1].set_title('Frauds')


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Creditcard Fraud detection using Autoencoders.ipynb".
The first task is:
Loading the data
The dataset we're going to use can be downloaded from Kaggle. It contains data about credit card transactions that occurred during a period of two days, with 492 frauds out of 284,807 transactions.
All variables in the dataset are numerical. The data has been transformed using PCA transformation(s) due to privacy reasons. The two features that haven't been changed are Time and Amount. Time contains the seconds elapsed between each transaction and the first transaction in the dataset.
Can you write Python code for it?


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Creditcard Fraud detection using Autoencoders.ipynb".
The first task is:
We have a highly imbalanced dataset on our hands. Normal transactions overwhelm the fraudulent ones by a large margin. Let's look at the two types of transactions:
Can you write Python code for it?

frauds = df[df.Class == 1]
normal = df[df.Class == 0]

frauds.shape

plt.hist(normal.Amount, bins = 100)
plt.xlim([0,20000])
plt.ylim([0,10000])
plt.tight_layout()


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Hands_on_Keras_Tutorial.ipynb".
The first task is:
Compilation
Before training a model, you need to configure the learning process, which is done via the compile method. It receives three arguments:

An optimizer. This could be the string identifier of an existing optimizer (such as rmsprop or adagrad), or an instance of the  Optimizer class. See: optimizers.
A loss function. This is the objective that the model will try to minimize. It can be the string identifier of an existing loss function (such as categorical_crossentropy or mse), or it can be an objective function. See: losses.
A list of metrics. For any classification problem you will want to set this to metrics=['accuracy']. A metric could be the string identifier of an existing metric or a custom metric function.
Can you write Python code for it?

# For a multi-class classification problem
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# For a binary classification problem
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# For a mean squared error regression problem
model.compile(optimizer='rmsprop',
              loss='mse')

# For custom metrics
import keras.backend as K

def mean_pred(y_true, y_pred):
    return K.mean(y_pred)

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy', mean_pred])


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Hands_on_Keras_Tutorial.ipynb".
The first task is:
Examples
Here are a few examples to get you started!
Multilayer Perceptron (MLP) for multi-class softmax classification:
Can you write Python code for it?

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD

# Generate dummy data
import numpy as np
x_train = np.random.random((1000, 20))
y_train = keras.utils.to_categorical(np.random.randint(10, size=(1000, 1)), num_classes=10)
x_test = np.random.random((100, 20))
y_test = keras.utils.to_categorical(np.random.randint(10, size=(100, 1)), num_classes=10)

model = Sequential()
# Dense(64) is a fully-connected layer with 64 hidden units.
# in the first layer, you must specify the expected input data shape:
# here, 20-dimensional vectors.
model.add(Dense(64, activation='relu', input_dim=20))
model.add(Dropout(0.5))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(10, activation='softmax'))

sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
              optimizer=sgd,
              metrics=['accuracy'])

model.fit(x_train, y_train,
          epochs=20,
          batch_size=128)
score = model.evaluate(x_test, y_test, batch_size=128)


I am working on the file "NEU/Sai_Raghuram_Kothapalli_DL/Hands_on_Keras_Tutorial.ipynb".
The first task is:
Specifying the input shape
The model needs to know what input shape it should expect. For this reason, the first layer in a Sequential model (and only the first, because following layers can do automatic shape inference) needs to receive information about its input shape. There are several possible ways to do this:
Pass an input_shape argument to the first layer. This is a shape tuple (a tuple of integers or None entries, where None indicates that any positive integer may be expected). In input_shape, the batch dimension is not included.
Some 2D layers, such as Dense, support the specification of their input shape via the argument input_dim, and some 3D temporal layers support the arguments input_dim and input_length.
If you ever need to specify a fixed batch size for your inputs (this is useful for stateful recurrent networks), you can pass a  batch_size argument to a layer. If you pass both batch_size=32 and input_shape=(6, 8) to a layer, it will then expect every batch of inputs to have the batch shape (32, 6, 8).
As such, the following snippets are strictly equivalent:
Can you write Python code for it?

model = Sequential()
model.add(Dense(32, input_shape=(784,)))
model = Sequential()
model.add(Dense(32, input_dim=784))