I am working on the file "171106_t-test_wilcoxon_comparison/Generic Comparison T-Test Wilcoxon-Rank-Sum Test.ipynb".
The first task is:
Create data.
Both sample names and variable names are simply intgers starting from 0.
Can you write Python code for it?

adata=AnnData(np.multiply(binomial(1,reference_zero_inflation,(n_simulated_cells,n_simulated_genes)),
                          negative_binomial(reference_n,reference_p,(n_simulated_cells,n_simulated_genes))))
# adapt marker_genes for cluster
adata.X[0:n_cluster,0:n_marker_genes]=np.multiply(binomial(1,cluster_zero_inflation,(n_cluster,n_marker_genes)),
                                                  negative_binomial(cluster_n,cluster_p,(n_cluster,n_marker_genes)))


I am working on the file "171106_t-test_wilcoxon_comparison/Generic Comparison T-Test Wilcoxon-Rank-Sum Test.ipynb".
The first task is:
Testing
Case 1: No mean difference, large variance difference.
Using the data created above, we get the following expectation and variance
$\mathbb{E}[Z_r]=\mathbb{E}[Z_c]=0.9$
$\mathbb{V}[Z_r]=8.19$
$\mathbb{V}[Z_c]=1.89$
Can you write Python code for it?

sc.tl.rank_genes_groups(adata, 'true_groups')
sc.pl.rank_genes_groups(adata, n_genes=20)

sc.tl.rank_genes_groups(adata, 'true_groups', test_type='wilcoxon')
sc.pl.rank_genes_groups(adata, n_genes=20)


I am working on the file "171106_t-test_wilcoxon_comparison/Generic Comparison T-Test Wilcoxon-Rank-Sum Test.ipynb".
The first task is:
This parameter initialization leads to the following expectations/ variances:
$\mathbb{E}[Z_r]=0.9$
$\mathbb{E}[Z_c]=1.47$
$\mathbb{V}[Z_r]=6.39$
$\mathbb{V}[Z_c]=2.92$
Can you write Python code for it?

adata=AnnData(np.multiply(binomial(1,reference_zero_inflation,(n_simulated_cells,n_simulated_genes)),
                          negative_binomial(reference_n,reference_p,(n_simulated_cells,n_simulated_genes))))
# adapt marker_genes for cluster
adata.X[0:n_cluster,0:n_marker_genes]=np.multiply(binomial(1,cluster_zero_inflation,(n_cluster,n_marker_genes)),
                                                  negative_binomial(cluster_n,cluster_p,(n_cluster,n_marker_genes)))

smp='true_groups'
true_groups_int=np.ones((n_simulated_cells,))
true_groups_int[0:n_cluster]=0
true_groups=list()
for i,j  in enumerate(true_groups_int):
    true_groups.append(str(j))
adata.smp['true_groups']=pd.Categorical(true_groups, dtype='category')
adata.uns[smp + '_order']=np.asarray(['0','1'])

sc.tl.rank_genes_groups(adata, 'true_groups')
sc.pl.rank_genes_groups(adata, n_genes=20)

sc.tl.rank_genes_groups(adata, 'true_groups', test_type='wilcoxon')
sc.pl.rank_genes_groups(adata, n_genes=20)


I am working on the file "171106_t-test_wilcoxon_comparison/Generic Comparison T-Test Wilcoxon-Rank-Sum Test.ipynb".
The first task is:
This parameter initialization leads to the following expectations/ variances:
$\mathbb{E}[Z_r]=\mathbb{E}[Z_c]=0.9$
$\mathbb{V}[Z_r]=6.39$
$\mathbb{V}[Z_c]=1.89$
Can you write Python code for it?

adata=AnnData(np.multiply(binomial(1,reference_zero_inflation,(n_simulated_cells,n_simulated_genes)),
                          negative_binomial(reference_n,reference_p,(n_simulated_cells,n_simulated_genes))))
# adapt marker_genes for cluster
adata.X[0:n_cluster,0:n_marker_genes]=np.multiply(binomial(1,cluster_zero_inflation,(n_cluster,n_marker_genes)),
                                                  negative_binomial(cluster_n,cluster_p,(n_cluster,n_marker_genes)))

import pandas as pd
smp='true_groups'
true_groups_int=np.ones((n_simulated_cells,))
true_groups_int[0:n_cluster]=0
true_groups=list()
for i,j  in enumerate(true_groups_int):
    true_groups.append(str(j))
adata.smp['true_groups']=pd.Categorical(true_groups, dtype='category')
adata.uns[smp + '_order']=np.asarray(['0','1'])

sc.tl.rank_genes_groups(adata, 'true_groups')
sc.pl.rank_genes_groups(adata, n_genes=20)

sc.tl.rank_genes_groups(adata, 'true_groups', test_type='wilcoxon')
sc.pl.rank_genes_groups(adata, n_genes=20)


I am working on the file "171106_t-test_wilcoxon_comparison/Generic Comparison T-Test Wilcoxon-Rank-Sum Test.ipynb".
The first task is:
With smaller difference in variance, still all marker genes are detected, but less clearly.
Case 3: Small difference in expectation, difference in variance
Can you write Python code for it?

# n_cluster needs to be smaller than n_simulated_cells, n_marker_genes needs to be smaller than n_simulated_genes
n_simulated_cells=1000
n_simulated_genes=100
n_cluster=100
n_marker_genes=10
# Specify parameter between 0 and 1 for zero_inflation and p, positive integer for r
# Differential gene expression is simulated using reference parameters for all cells/genes
# except for marker genes in the distinct cells.
reference_zero_inflation=0.15
reference_p=0.5
reference_n=6
cluster_zero_inflation=0.9
cluster_p=0.55
cluster_n=2


I am working on the file "171111_SPRING_export/SPRING_export.ipynb".
The first task is:
Exporting the results to a SPRING project directory.
Can you write Python code for it?

adata_krumsiek11 = sc.read('./write/krumsiek11.h5')

sc.export_to.spring_project(adata_krumsiek11, './spring_krumsiek11')


I am working on the file "171111_SPRING_export/SPRING_export.ipynb".
The first task is:
For background information, see here.
Can you write Python code for it?


I am working on the file "171111_SPRING_export/SPRING_export.ipynb".
The first task is:
Let us add some computational analysis results.
Can you write Python code for it?

sc.pp.neighbors(adata_krumsiek11)

sc.tl.louvain(adata_krumsiek11)
sc.tl.dpt(adata_krumsiek11, n_branchings=1)


I am working on the file "171111_SPRING_export/SPRING_export.ipynb".
The first task is:
Let's plot this.
Can you write Python code for it?

sc.pl.draw_graph(adata_krumsiek11, color=['cell_type', 'louvain', 'dpt_groups', 'dpt_pseudotime', 'Gata1'])

adata_krumsiek11.write('./write/krumsiek11.h5')


I am working on the file "171111_SPRING_export/SPRING_export.ipynb".
The first task is:
Now, after starting a server, open
http://localhost:8000/SPRING/springViewer.html?../spring_kurmsiek11
You might need to click 'center view' and 'deselect all' to see the data.
Example 2: 3k PBMCs data
This starts from the standard clustering tutorial: https://github.com/theislab/scanpy_usage/tree/master/170505_seurat.
Selecting genes: It is recommended that - especially for large datasets - only a preselected number of genes of interest is used for SPRING, so that these genes can be found more easily (in the dataset below, searching for a certain gene in a dropdown list of ca. 13000 entries can be hard ).
Can you write Python code for it?

adata = sc.read('../170505_seurat/write/pmbc3k.h5ad')

sc.tl.draw_graph(adata)


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
A brief word of warning, the pretty printing shown above will take longer than just printing out the values of an array.  This is unnoticable in this case, but is evident if you try to print a large array of quantities.
Can you write Python code for it?

mass = 1 * u.Msun
print(type(mass))
mass

# quantities subclass numpy ndarray, so you can make handle arrays of quantities like you would
# any other array object
mass.__class__.__bases__

# we can convert units to other equivalent units
mass.to(u.kg)

# there are shortcuts for converting to the relevent system, regardless of what type of quantity it is
mass.cgs

mass.si

# we can inspect their unit and their numeric value with that unit
print(mass.value, mass.unit)

# calculations with quantities can produce quantities with new units
average_density = mass / (4 / 3 * np.pi * u.Rearth ** 3)
average_density.cgs


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
Astropy quantities
Astropy quantitites are a great way to handle all sorts of messy unit conversions.  Careful unit conversions save lives!  https://en.wikipedia.org/wiki/Gimli_Glider
The simplest way to create a new quantity object is multiply or divide a number by a Unit instance.
Can you write Python code for it?


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
Calculations with quantity arrays
Can you write Python code for it?

# made into a quantity array by multiplying numeric array by unit
R = np.linspace(1, 5) * u.Rearth
v = np.sqrt(2 * c.G * u.Msun / R)
print(v)
v = v.to(u.km / u.s)
print(v)
plt.plot(R, v)
plt.xlabel(r"Radius  [R$_\oplus$]")
plt.ylabel(r"Escape velocity  [km s$^{-1}$]")


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
Constants
Physical constants are found in the astropy.constants module, and work just like units.
Can you write Python code for it?

# Newton's constant
c.G

# Planck's constant
c.h

# speed of light
c.c


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
Even when you think you have a dimensionless array, it can still be a dimensionless quantity.
Can you write Python code for it?


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
Quantities as sanity checks
Can you write Python code for it?

obscure_quantity = 42 * c.G * c.m_e ** 2 / c.k_B ** 2 * c.c ** 3 * (5700 * u.K) ** -2 * u.Msun / u.Mpc

obscure_quantity

# what the heck is a m^6 kg Msun / (Mpc J^2 s^5)??
obscure_quantity.decompose()

# will fail!
obscure_quantity.to(u.m)

# addition works for like units
(1 * u.m) + (1 * u.cm)

# and fails for the wrong dimensions
(1 * u.m) + (1 * u.s)


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
SkyCoord
The SkyCoord class, from astropy.coordinates, is a convenient way of dealing with astronomical coordinate systems.
http://docs.astropy.org/en/stable/coordinates/index.html
Can you write Python code for it?

coord = SkyCoord(45, 30, unit=u.deg)

# ICRS is the reference frame
coord

# we can transform between coordinate frames
coord.fk4

coord.fk5

coord.galactic

# latitude and longitude are accessed with ra and dec (when in icrs or fk frames)
coord.ra

coord.dec


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
Spectral energy density equivalencies
Can you write Python code for it?

intensity_unit = blackbody_nu(wavelengths[0], temperature=1e3 * u.K).unit
wavelengths = np.logspace(-1, 1, 100) * u.micron
temperatures = np.linspace(5e3, 1e4, 5) * u.K

for T in temperatures:
    plt.plot(wavelengths, blackbody_nu(wavelengths, temperature=T),
             label='{:.2e}'.format(T))

plt.xscale('log')
plt.legend(loc='best')
plt.xlabel(r'$\lambda$  [$\mu$m]')
plt.ylabel('$I_\\nu$  [{}]'.format(texify(intensity_unit)))

intensity_unit = blackbody_lambda(wavelengths[0], temperature=1e3 * u.K).unit
for T in temperatures:
    plt.plot(wavelengths, blackbody_lambda(wavelengths, temperature=T),
             label='{:.2e}'.format(T))

plt.xscale('log')
plt.legend(loc='best')
plt.xlabel(r'$\lambda$  [$\mu$m]')
plt.ylabel('$I_\\lambda$  [{}]'.format(texify(intensity_unit)))

T = 1e4 * u.K
solid_angle = ((1 * u.arcsec) ** 2).to(u.sr)
f_nu = blackbody_nu(wavelengths, temperature=T) * solid_angle
f_lambda = blackbody_lambda(wavelengths, temperature=T) * solid_angle

print(f_nu.unit)
print(f_lambda.unit)

# I_nu.to(I_lambda.unit) # would fail

# for conversion of spectral energy density, we need to specify what part of the spectra we're looking at
f_lambda_converted = f_nu.to(f_lambda.unit, equivalencies=u.spectral_density(wavelengths))

print(f_lambda_converted.unit)

# shouldn't raise any exceptions!
assert np.all(np.isclose(f_lambda.value, f_lambda_converted.value))


I am working on the file "17_02_23_astropy_quantities/astropy_quantities.ipynb".
The first task is:
Unit equivalencies
Equivalencies allow you to do unit conversions under certain physical assumptions.  For instance, it makes sense to talk about converting wavelength to frequency when you are discussing the properties of light waves in a vacuum.  See http://docs.astropy.org/en/stable/units/equivalencies.html#unit-equivalencies.
Spectral equivalence
Can you write Python code for it?

wavelengths = np.linspace(0.1, 1, 100) * u.micron

# will fail without the correct equivalency passed in!
frequencies = wavelengths.to(u.THz, equivalencies=u.spectral())

plt.plot(wavelengths, frequencies)
plt.xlabel(r"$\lambda$  [$\mu$m]")
plt.ylabel(r"$\nu$  [THz]")


I am working on the file "17_04_20_2to3/2to3.ipynb".
The first task is:
New string formatting
The old string formatting (with %) is depricated in favor of str.format().  A good comparison of the two can be found here:
https://pyformat.info/
Unicode
Dealing with unicode can be a pain when *nix doesn't give or expect unicode.  Sometimes importing data in python3 will give you strings with a weird b in front.  These are bytestrings, and they can usually be converted to unicode strings with bytestring.decode('utf-8').
Can you write Python code for it?

s = 'asdf'
b = s.encode('utf-8')
b

b.decode('utf-8')

# this will be problematic if other encodings are used...
s = 'asdf'
b = s.encode('utf-32')
b

b.decode('utf-8')


I am working on the file "17_04_20_2to3/2to3.ipynb".
The first task is:
Writing code for both Python 2 and Python 3
Ever wonder what those from __future__ import foo statements were doing?
http://python-future.org/quickstart.html

Using the future package, you can write code that works for either Python 2 or Python 3.  You'll still have to avoid using some Python 3 specific syntax.
Can you write Python code for it?

# shouldn't change anything in python3
from __future__ import print_function, division

print('non-truncated division in a print function: 2/3 =', 2/3)


I am working on the file "17_Estimator_API.ipynb".
The first task is:
An input-function is also needed for predicting the class of new data. As an example we just use a few images from the test-set.
Can you write Python code for it?

some_images = data.x_test[0:9]

predict_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"x": some_images},
    num_epochs=1,
    shuffle=False)


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Calling this function returns a tuple with TensorFlow ops for returning the input and output data:
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Copy some of the data-dimensions for convenience.
Can you write Python code for it?

# The number of pixels in each dimension of an image.
img_size = data.img_size

# The images are stored in one-dimensional arrays of this length.
img_size_flat = data.img_size_flat

# Tuple with height and width of images used to reshape arrays.
img_shape = data.img_shape

# Number of classes, one class for each of 10 digits.
num_classes = data.num_classes

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = data.num_channels


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Create an Instance of the Estimator
We can specify hyper-parameters e.g. for the learning-rate of the optimizer.
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Evaluation
Once the model has been trained, we can evaluate its performance on the test-set.
Can you write Python code for it?

result = model.evaluate(input_fn=test_input_fn)

result

print("Classification accuracy: {0:.2%}".format(result["accuracy"]))


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "17_Estimator_API.ipynb".
The first task is:
In this example we want to use a 3-layer DNN with 512, 256 and 128 units respectively.
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given dir.
Can you write Python code for it?

from mnist import MNIST
data = MNIST(data_dir="data/MNIST/")


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "17_Estimator_API.ipynb".
The first task is:
New Estimator
If you cannot use one of the built-in Estimators, then you can create an arbitrary TensorFlow model yourself. To do this, you first need to create a function which defines the following:

The TensorFlow model, e.g. a Convolutional Neural Network.
The output of the model.
The loss-function used to improve the model during optimization.
The optimization method.
Performance metrics.

The Estimator can be run in three modes: Training, Evaluation, or Prediction. The code is mostly the same, but in Prediction-mode we do not need to setup the loss-function and optimizer.
This is another aspect of the Estimator API that is poorly designed and resembles how we did ANSI C programming using structs in the old days. It would probably have been more elegant to split this into several functions and sub-classed the Estimator-class.
Can you write Python code for it?

def model_fn(features, labels, mode, params):
    # Args:
    #
    # features: This is the x-arg from the input_fn.
    # labels:   This is the y-arg from the input_fn,
    #           see e.g. train_input_fn for these two.
    # mode:     Either TRAIN, EVAL, or PREDICT
    # params:   User-defined hyper-parameters, e.g. learning-rate.

    # Reference to the tensor named "x" in the input-function.
    x = features["x"]

    # The convolutional layers expect 4-rank tensors
    # but x is a 2-rank tensor, so reshape it.
    net = tf.reshape(x, [-1, img_size, img_size, num_channels])

    # First convolutional layer.
    net = tf.layers.conv2d(inputs=net, name='layer_conv1',
                           filters=16, kernel_size=5,
                           padding='same', activation=tf.nn.relu)
    net = tf.layers.max_pooling2d(inputs=net, pool_size=2, strides=2)

    # Second convolutional layer.
    net = tf.layers.conv2d(inputs=net, name='layer_conv2',
                           filters=36, kernel_size=5,
                           padding='same', activation=tf.nn.relu)
    net = tf.layers.max_pooling2d(inputs=net, pool_size=2, strides=2)

    # Flatten to a 2-rank tensor.
    net = tf.contrib.layers.flatten(net)
    # Eventually this should be replaced with:
    # net = tf.layers.flatten(net)

    # First fully-connected / dense layer.
    # This uses the ReLU activation function.
    net = tf.layers.dense(inputs=net, name='layer_fc1',
                          units=128, activation=tf.nn.relu)

    # Second fully-connected / dense layer.
    # This is the last layer so it does not use an activation function.
    net = tf.layers.dense(inputs=net, name='layer_fc2',
                          units=10)

    # Logits output of the neural network.
    logits = net

    # Softmax output of the neural network.
    y_pred = tf.nn.softmax(logits=logits)

    # Classification output of the neural network.
    y_pred_cls = tf.argmax(y_pred, axis=1)

    if mode == tf.estimator.ModeKeys.PREDICT:
    ...


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.x_test[0:9]

# Get the true classes for those images.
cls_true = data.y_test_cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Pre-Made / Canned Estimator
When using a pre-made Estimator, we need to specify the input features for the data. In this case we want to input images from our data-set which are numeric arrays of the given shape.
Can you write Python code for it?

feature_x = tf.feature_column.numeric_column("x", shape=img_shape)


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Predictions
The model can also be used to make predictions on new data.
Can you write Python code for it?

predictions = model.predict(input_fn=predict_input_fn)

cls_pred = np.array(list(predictions))
cls_pred

plot_images(images=some_images,
            cls_true=some_images_cls,
            cls_pred=cls_pred)


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Predictions
The trained model can also be used to make predictions on new data.
Note that the TensorFlow graph is recreated and the checkpoint is reloaded every time we make predictions on new data. If the model is very large then this could add a significant overhead.
It is unclear why the Estimator is designed this way, possibly because it will always use the latest checkpoint and it can also be distributed easily for use on multiple computers.
Can you write Python code for it?

predictions = model.predict(input_fn=predict_input_fn)

cls = [p['classes'] for p in predictions]

cls_pred = np.array(cls, dtype='int').squeeze()
cls_pred

plot_images(images=some_images,
            cls_true=some_images_cls,
            cls_pred=cls_pred)


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Similarly we need to create a function for reading the data for the test-set. Note that we only want to process these images once so num_epochs=1 and we do not want the images shuffled so shuffle=False.
Can you write Python code for it?

test_input_fn = tf.estimator.inputs.numpy_input_fn(
    x={"x": np.array(data.test.images)},
    y=np.array(data.test.cls),
    num_epochs=1,
    shuffle=False)


I am working on the file "17_Estimator_API.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "17_Estimator_API.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70.000 images and class-numbers for the images. The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(data.num_train))
print("- Validation-set:\t{}".format(data.num_val))
print("- Test-set:\t\t{}".format(data.num_test))


I am working on the file "17_Estimator_API.ipynb".
The first task is:
The class-numbers are actually not used in the input-function as it is not needed for prediction. However, the true class-number is needed when we plot the images further below.
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
These are the corresponding class-numbers:
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
This actually returns a function:
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
This was developed using Python 3.6 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Training
Now that our new Estimator has been created, we can train it.
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
Training
We can now train the model for a given number of iterations. This automatically loads and saves checkpoints so we can continue the training later.
Note that the text INFO:tensorflow: is printed on every line and makes it harder to quickly read the actual progress. It should have been printed on a single line instead.
Can you write Python code for it?


I am working on the file "17_Estimator_API.ipynb".
The first task is:
We can then create an instance of the new Estimator.
Note that we don't provide feature-columns here as it is inferred automatically from the data-functions when model_fn() is called.
It is unclear from the TensorFlow documentation why it is necessary to specify the feature-columns when using DNNClassifier in the example above, when it is not needed here.
Can you write Python code for it?

model = tf.estimator.Estimator(model_fn=model_fn,
                               params=params,
                               model_dir="./checkpoints_tutorial17-2/")


I am working on the file "17_Estimator_API.ipynb".
The first task is:
You can have several input features which would then be combined in a list:
Can you write Python code for it?


I am working on the file "18--matrix-storage.ipynb".
The first task is:
Exercise. Complete the following routine, which takes a sparse matrix keyed by names and multiplies it by a vector. (See also: spmv() from Lab 14.)
Can you write Python code for it?

# Build a dense vector
x = cse6040.dense_vector (n)

def spmv_named_keys (n, A, x, name2id):
    y = cse6040.dense_vector (n)

    # @YOUSE: Complete this routine

    return y

%timeit spmv_named_keys (n, A_named_keys, x, name2id)

%timeit cse6040.spmv (n, A_numbered_keys, x)


I am working on the file "18--matrix-storage.ipynb".
The first task is:
Exercise. Create a coordinate (COO) matrix using SciPy, and measure the time to perform a matrix-vector product.
Can you write Python code for it?

import scipy.sparse as sp

rows = ... # @YOUSE: Complete this statement
cols = ... # @YOUSE: Complete this statement
assert len (rows) == len (cols)
vals = [1.] * len (rows)
A_coo = ... # @YOUSE: Complete this statement
x_np = np.array (x)

%timeit A_coo.dot (x_np)


I am working on the file "18--matrix-storage.ipynb".
The first task is:
Exercise. Implement a matrix-vector product that operates on native Python lists. Assume the 1-D storage of the matrix, as shown above
Can you write Python code for it?

def matvec_py (m, n, A, x):
    """
    Native Python-based matrix-vector multiply, using lists.
    The dimensions of the matrix A are m-by-n, and x is a
    vector of length n.
    """
    assert type (A) is list
    assert type (x) is list
    assert len (x) >= n
    assert len (A) >= (m*n)

    y = [0.] * m

    # @YOUSE: Fill in an implementation here
    for j in range (n):
        for i in range (m):
            y[i] += A[i + j*n] * x[j]

#[sum (i*j for i, j in zip (a_row, x) for a_row in A)]

    return y

# Estimate a bound on the difference between these two
EPS = np.finfo (float).eps # "machine epsilon"
CONST = 2.0 # Some constant for the error bound
dy_max = CONST * n * EPS

print ("""==> Error bound estimate:
         C*n*eps
         == %g*%g*%g
         == %g
""" % (CONST, n, EPS, dy_max))

# Run the Numpy version and your code
y_np = A_np.dot (x_np)
y_py = matvec_py (n, n, A_py, x_py)

# Compute the difference between these
dy = y_np - np.reshape (y_py, (n, 1), order='F')
dy_norm = np.linalg.norm (dy, ord=np.inf)

# Summarize the results
from IPython.display import display, Math

comparison = "\leq" if dy_norm <= dy_max else "\gt"
display (Math (
        r'||y_{\textrm{np}} - y_{\textrm{py}}||_{\infty}'
        r' = \textrm{%g} %s \textrm{%g}\ (\textrm{estimated bound})'
        % (dy_norm, comparison, dy_max)
    ))

if n <= 4: # Debug: Print all data for small inputs
    print "@A_np:\n" ; print A_np
    print "@x_np:\n" ; print x_np
    print "@y_np:\n" ; print y_np
    print "@A_py:\n" ; print A_py
    print "@x_py:\n" ; print x_py
    print "@y_py:\n" ; print y_py
    print "@dy:\n" ; print dy

# Trigger an error on likely failure
assert dy_norm <= dy_max

%timeit matvec_py (n, n, A_py, x_py)


I am working on the file "18--matrix-storage.ipynb".
The first task is:
Exercise. Write a function that iterates over the columns of a matrix and scales them by a given value. Then use that routine to measure the difference in time when the input is row-major versus column-major.
Can you write Python code for it?

def scale_colwise (A):
    """Given a Numpy matrix `A`, visits each column `A[:, j]`
    and scales it by `j`."""
    assert type (A) is np.ndarray

    n_cols = A.shape[1] # number of columns

    # @YOUSE: Fill in this code
    for j in range (n_cols):
        A[:, j] *= j

    return A

# Measure time to scale a row-major input column-wise
%timeit scale_colwise (A_rowmaj)

# Measure time to scale a column-major input column-wise
%timeit scale_colwise (A_colmaj)


I am working on the file "18--matrix-storage.ipynb".
The first task is:
Matrix storage: Column-major versus row-major layouts
For linear algebra, we will be especially interested in 2-D arrays, which we will use to store matrices. For this common case, there is a subtle performance issue related to how matrices are stored in memory.
By way of background, physical storage---whether it be memory or disk---is basically one big array. And because of how physical storage is implemented, it turns out that it is much faster to access consecutive elements in memory than, say, to jump around randomly.
A matrix is a two-dimensional object. Thus, when it is stored in memory, it must be mapped in some way to the one-dimensional physical array. There are many possible mappings, but the two most common conventions are known as the column-major and row-major layouts:
<img src="https://raw.githubusercontent.com/rvuduc/cse6040-ipynbs/master/16--numpy-matrix-layout.png" alt="Exercise: Extract these slices" width="640">
Exercise. Let $A$ be an $m \times n$ matrix stored in column-major format. Let $B$ be an $m \times n$ matrix stored in row-major format.
Based on the preceding discussion, recall that these objects will be mapped to 1-D arrays of length $mn$, behind the scenes. Let's call the 1-D array representations $\hat{A}$ and $\hat{B}$. Thus, the $(i, j)$ element of $a$, $a_{ij}$, will map to some element $\hat{a}u$ of $\hat{A}$; similarly, $b{ij}$ will map to some element $\hat{b}_v$ of $\hat{B}$.
Determine formulae to compute the 1-D index values, $u$ and $v$, in terms of ${i, j, m, n}$. Assume that all indices are 0-based, i.e., $0 \leq i \leq m-1$, $0 \leq j \leq n-1$, and $0 \leq u, v \leq mn-1$.
Can you write Python code for it?

def calc_u (i, j, m, n): # column major
    u = j*m + i # @YOUSE: Replace with the correct formula
    return u

def calc_v (i, j, m, n): # row major
    v = i*n + j # @YOUSE: Replace with the correct formula
    return v

# Quick check (not exhaustive):
print "==> Testing your implementations (not exhaustive)..."
assert calc_u (7, 4, 10, 20) == 47
assert calc_v (7, 4, 10, 20) == 144
print "Passed!"


I am working on the file "18--matrix-storage.ipynb".
The first task is:
Sparse matrix storage
Take a look at the slides from today's class, which cover the basics of sparse matrix storage formats: link
Let's implement and time some of these routines below. To help you get started with a realistic input, the following code loads the raw data from the Yelp! user-user connectivity subgraph (see top of this notebook to download).
Can you write Python code for it?

import cse6040utils as cse6040
import pandas as pd

edges = pd.read_csv ('UserEdges-1M.csv')
display (edges.head ())

# Build a set of unique vertex names
V_names = set (edges.Source)
V_names.update (set (edges.Target))

m = len (edges)
n = len (V_names)
print ("==> |V| == %d, |E| == %d" % (n, m))

# Create a mapping of names to IDs, and vice-versa
id2name = {}
name2id = {}
for (k, v) in enumerate (V_names):
    if k <= 5: print ("[%d] %s" % (k, v)) # for debugging
    id2name[k] = v
    name2id[v] = k

# Build two different kinds of sparse matrices using nested dictionaries
A_named_keys = cse6040.sparse_matrix ()
A_numbered_keys = cse6040.sparse_matrix ()

for (k, row) in edges.iterrows ():
    ni = row['Source']
    nj = row['Target']
    A_named_keys[ni][nj] = 1.
    A_named_keys[nj][ni] = 1.

    i = name2id[ni]
    j = name2id[nj]
    A_numbered_keys[i][j] = 1.
    A_numbered_keys[j][i] = 1.


I am working on the file "18-01-22-workcamp-ml/18-01-22-workcamp-ml-pandas-grundlagen-30.ipynb".
The first task is:

Aus panda datframes direkt plotten

Can you write Python code for it?

import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "18-01-22-workcamp-ml/18-01-22-workcamp-ml-pandas-grundlagen-30.ipynb".
The first task is:

Einlesen einer csv Datei: temperaturen-1300.csv sep=; decimal = ,

Can you write Python code for it?

temp=pd.read_csv('temperaturen-1300.csv',sep=(';'),decimal=(','))

temp.head()

print(temp)

plt.plot(temp)
plt.show()

# Create a plot with color='red'
temp.plot(color='r')

# Add a title
plt.title('Temperatur')

# Specify the x-axis label
plt.xlabel('Stunden seit Mitternacht 1. August 2015')

# Specify the y-axis label
plt.ylabel('Temperatur (Grad C)')

# Display the plot
plt.show()


I am working on the file "18-01-22-workcamp-ml/18-01-22-workcamp-ml-pandas-grundlagen-30.ipynb".
The first task is:

Einlesen einer csv-Datei: wetterdaten-1300.csv header=0 sep=; decimal = ,

Can you write Python code for it?

wett=pd.read_csv('wetterdaten-1300.csv',header=0, sep=(';'),decimal=(','))

wett.head()

wett.plot()
plt.show()

wett.plot(subplots=True)
plt.show()

# Plot all columns (default)
wett.plot()
plt.show()

# Plot all columns as subplots
wett.plot(subplots=True)
plt.show()
#
col_list1=['Temperatur']
wett[col_list1].plot()
plt.show()
col_list2=['Luftdruck ']
wett[col_list2].plot()
plt.show()

wett.info()

wett.columns


I am working on the file "18-01-22-workcamp-ml/18-01-22-workcamp-ml-pandas-grundlagen-30.ipynb".
The first task is:

Einlesen historischer Aktienwerte von Amazon. Quelle yahoo finance

Can you write Python code for it?

amzn = pd.read_csv('AMZN.csv', sep=',', index_col='Date', parse_dates=True)

close_arr=amzn['Close'].values

plt.plot(close_arr)
plt.show()

close_series = amzn['Close']

plt.plot(close_series)
plt.show()

#Serien direkt plotten
close_series.plot()
plt.show()

amzn.plot()
plt.show()

plt.plot(amzn)
plt.show()

amzn.plot()
#Logarithmische Skala auf der y-Achse
plt.yscale('log')
plt.show

amzn['Open'].plot(color='b',legend=True)
amzn['Close'].plot(color='r',legend=True)
#plt.axis('2014','2018',0,100)
plt.show()

amzn.info()

amzn.loc['2016':'2018',['Open']].plot()
plt.show()

amzn.loc['2015':'2018',['Close']].plot()
plt.savefig('amzn.png')
plt.savefig('amzn.jpg')
plt.savefig('amzn.pdf')
plt.show()


I am working on the file "18-01-22-workcamp-ml/18-01-22-workcamp-ml-pandas-grundlagen-30.ipynb".
The first task is:

Einlesen historischer Aktienwerte von Apple. Quelle yahoo finance

Can you write Python code for it?

aapl = pd.read_csv('AAPL.csv')

aapl.head(5)

aapl = pd.read_csv('AAPL.csv',sep=',', index_col='Date')

aapl.head(5)

aapl = pd.read_csv('AAPL.csv', sep=',', index_col='Date', parse_dates=True)

aapl.head(5)

close_arr=aapl['Close'].values

print(close_arr)

aapl.info()

type(close_arr)

plt.plot(close_arr)
plt.show()


I am working on the file "18-01-22-workcamp-ml/18-01-22-workcamp-ml-pandas-grundlagen-30.ipynb".
The first task is:
Dataframes kann man als csv und excel file im aktuellen Directory speichern mit:
out_csv = 'example.csv'
df.to_csv('out_csv', sep=(';'))
out_xlsx = 'example.xlsx')
df.to_excel('out_xlsx')

Geben Sie die Datenreihen 10:200 aus

Can you write Python code for it?


I am working on the file "18-01-22-workcamp-ml/18-01-22-workcamp-ml-pandas-grundlagen-30.ipynb".
The first task is:
Wir können jetzt einige Probleme erkennen. Die Spaltenüberschriften machten einfach keinen Sinn.

Man mussten mit header=None arbeiten.Und hier gibt es überraschend in Spalte 5 EInträge der Art

-1.0.
Die -1.0 Einträge bedeuten, das kein Wert vorhanden ist.
Die Datei enthält folgende Werte:

Column 1-3: Gregorian calendar date
- Year
- Month
- Day
Column 4: Date in fraction of year
Column 5: Daily total sunspot number. A value of -1 indicates that no number is available for that day (missing value).
Column 6: Daily standard deviation of the input sunspot numbers from individual stations.
Column 7: Number of observations used to compute the daily value.
Column 8: Definitive/provisional indicator. A blank indicates that the value is definitive. A '*' symbol indicates that the value is still provisional and is subject to a possible revision (Usually the last 3 to 6 months)
Wir wollen in der datei die Dpaltennamen ändern
Can you write Python code for it?

spalten_namen=['Jahr','Monat','Dez_Tag', 'Sunspot', 'Stabw','Definitv','Observationen']

sun_spots=pd.read_csv(dateipfad, header=None, sep =';', names=spalten_namen)

sun_spots.head(10)
sun_spots.info()


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 4.4 - Overfitting and underfitting.ipynb".
The first task is:
Comparison between the original and the small network
Can you write Python code for it?

# Initializing the original network
original_network = build_original_model()

# Training the original network
original_network_history = original_network.fit(x_train,
                     y_train,
                     epochs = 20,
                     batch_size = 512,
                     validation_data = (x_test, y_test))

small_network = build_small_model()

small_network_history = small_network.fit(x_train,
                                          y_train,
                                          epochs = 20,
                                          batch_size = 512,
                                          validation_data = (x_test, y_test))

epochs = range(1, 21)
original_val_loss = original_network_history.history['val_loss']
small_model_val_loss = small_network_history.history['val_loss']

import matplotlib.pyplot as plt
plt.figure(figsize = (10, 6))

# Plotting the validation loss of the original network
# b+ is for "blue cross"
plt.plot(epochs,
         original_val_loss,
         'b+',
         label='Original model')
# Plotting the validation loss of the small network
# "bo" is for "blue dot"
plt.plot(epochs,
         small_model_val_loss, 'bo', label='Small model')
plt.xlabel('Epochs')
plt.ylabel('Validation loss')
plt.legend()

plt.show()


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 4.4 - Overfitting and underfitting.ipynb".
The first task is:
Conclusions
The bigger network adpats faster, but it results in more severe overfitting issues.
Weight regularization
Can you write Python code for it?

from keras.regularizers import l2

# Building the original network with weight regularization
def build_original_model_with_l2():
    model = Sequential()
    model.add(Dense(units = 16,
                    activation = 'relu',
                    # Parameter l -> weight loss
                    kernel_regularizer = l2(l = 0.001),
                    input_shape = (10000,)))
    model.add(Dense(units = 16,
                    activation = 'relu',
                    kernel_regularizer = l2(l = 0.001)))
    model.add(Dense(units = 1,
                    activation = 'sigmoid'))
    model.compile(optimizer = 'rmsprop',
                       loss = 'binary_crossentropy',
                       metrics = ['acc'])
    return model

original_network_with_l2 = build_original_model_with_l2()

original_network_with_l2_history = original_network_with_l2.fit(x_train,
                                                                y_train,
                                                                epochs = 20,
                                                                batch_size = 512,
                                                                validation_data = (x_test, y_test))

original_network_with_l2_val_loss = original_network_with_l2_history.history['val_loss']

plt.figure(figsize = (10, 6))
plt.plot(epochs,
         original_val_loss,
         'b+',
         label = 'Original model')
plt.plot(epochs,
         original_network_with_l2_val_loss,
         'bo',
         label = 'L2-regularized model')
plt.xlabel('Epochs')
plt.ylabel('Validation loss')
plt.legend()

plt.show()


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 4.4 - Overfitting and underfitting.ipynb".
The first task is:
Conclusions
The smaller network overfits later than the bigger network.
Let's investigate other scenario.
Comparison between the original and the big network
Can you write Python code for it?

# Initializing the big network
big_network = build_big_model()

big_network_history = big_network.fit(x_train,
                                      y_train,
                                      epochs = 20,
                                      batch_size = 512,
                                      validation_data = (x_test, y_test))

# Comparison of validation losses
big_model_val_loss = big_network_history.history['val_loss']

plt.figure(figsize = (10, 6))
plt.plot(epochs,
         original_val_loss,
         'b+',
         label = 'Original model')
plt.plot(epochs,
         big_model_val_loss,
         'bo',
         label = 'Big model')
plt.xlabel('Epochs')
plt.ylabel('Validation loss')
plt.legend()

plt.show()

# Comparison of traning losses
original_train_loss = original_network_history.history['loss']
bigger_model_train_loss = big_network_history.history['loss']

plt.figure(figsize = (10, 6))
plt.plot(epochs,
         original_train_loss,
         'b+',
         label = 'Original model')
plt.plot(epochs,
         bigger_model_train_loss,
         'bo',
         label = 'Big model')
plt.xlabel('Epochs')
plt.ylabel('Training loss')
plt.legend()

plt.show()


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 4.4 - Overfitting and underfitting.ipynb".
The first task is:
Dropout
Can you write Python code for it?

from keras.layers import Dropout

# Defining the original model with dropout
def build_original_model_with_dropout():
    model = Sequential()
    model.add(Dense(units = 16,
                    activation = 'relu',
                    input_shape = (10000,)))
    # Parameter rate -> fraction of the input units to drop
    model.add(Dropout(rate = 0.5))
    model.add(Dense(units = 16,
                    activation = 'relu'))
    model.add(Dropout(rate = 0.5))
    model.add(Dense(units = 1,
                    activation = 'sigmoid'))
    model.compile(optimizer = 'rmsprop',
                       loss = 'binary_crossentropy',
                       metrics = ['acc'])
    return model

original_model_with_dropout = build_original_model_with_dropout()

original_model_with_dropout_history = original_model_with_dropout.fit(x_train,
                                                                      y_train,
                                                                      epochs = 20,
                                                                      batch_size = 512,
                                                                      validation_data = (x_test, y_test))

original_model_with_dropout_val_loss = original_model_with_dropout_history.history['val_loss']

plt.figure(figsize = (10, 6))
plt.plot(epochs, original_val_loss, 'b+', label='Original model')
plt.plot(epochs, original_model_with_dropout_val_loss, 'bo', label='Dropout-regularized model')
plt.xlabel('Epochs')
plt.ylabel('Validation loss')
plt.legend()

plt.show()


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.1.1 - One-hot encoding of words and characters.ipynb".
The first task is:
Initializing an example
Can you write Python code for it?

samples = ['The cat sat on the mat.', 'The dog ate my homework.']

# Intializing token index as an empty dictionary
token_index = {}


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.1.1 - One-hot encoding of words and characters.ipynb".
The first task is:
Keras Tokenizer
Can you write Python code for it?

# Importing Keras Tokenizer
from keras.preprocessing.text import Tokenizer

samples = ['The cat sat on the mat.', 'The dog ate my homework.']

# Intializing tokenizer, which will take into account only 1000 most commonly used words.
tokenizer = Tokenizer(num_words = 1000)

# Building the dictionary
tokenizer.fit_on_texts(samples)

# Turning the sequences to an array of integers corresponding to the unique words
sequences = tokenizer.texts_to_sequences(samples)

sequences

# Representing the data as one-hot encoded.
one_hot_results = tokenizer.texts_to_matrix(samples,
                                            mode = 'binary')

one_hot_results

# Retrieving the index
word_index = tokenizer.word_index

word_index


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.1.1 - One-hot encoding of words and characters.ipynb".
The first task is:
Vectoring the example
Can you write Python code for it?

# Taking into consideration only first 10 words in each sentence.
max_length = 10

# Initializing the result array with zeros
# It will be of shape (number_of_samples, max_length_taken_into_consideration, number_of_unique_words)
results = np.zeros((len(samples), max_length, max(token_index.values()) + 1))

# Enumerating through samples and words
# One-hot encoding
for i, sample in enumerate(samples):
    for j, word in list(enumerate(sample.split()))[:max_length]:
        index = token_index.get(word)
        results[i, j, index] = 1.

results


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.1.2 - Using word embeddings.ipynb".
The first task is:
GloVe Embedding
Download from: http://nlp.stanford.edu/data/glove.6B.zip
Can you write Python code for it?

# Importing tqdm to show a progress bar
from tqdm import tqdm

glove_dir = './data/Chapter 6.1.2 - Using word embeddings/glove.6B/'
embeddings_index = {}
f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'),
         encoding = 'utf-8')
for line in tqdm(f):
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:],
                       dtype = 'float32')
    embeddings_index[word] = coefs
f.close()

len(embeddings_index)

embedding_dim = 100
embedding_matrix = np.zeros((max_words, embedding_dim))

for word, i in word_index.items():
    if i < max_words:
        embedding_vector = embeddings_index.get(word)
        # Words not found in the embedding index will be represented as zeros
        if embedding_vector is not None:
            embedding_matrix[i] = embedding_vector

embedding_matrix


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.1.2 - Using word embeddings.ipynb".
The first task is:
Model
Can you write Python code for it?

from keras.models import Sequential
from keras.layers import Embedding, Flatten, Dense

model = Sequential()
model.add(Embedding(input_dim = max_words,
                    output_dim = embedding_dim,
                    input_length = maxlen))
model.add(Flatten())
model.add(Dense(units = 32,
                activation = 'relu'))
model.add(Dense(units = 1,
                activation = 'sigmoid'))
model.summary()

# Loading pretrained word embeddings
model.layers[0].set_weights([embedding_matrix])
# Freezing the layer
model.layers[0].trainable = False

model.compile(optimizer = 'rmsprop',
              loss = 'binary_crossentropy',
              metrics = ['acc'])

history = model.fit(x = x_train,
                    y = y_train,
                    epochs = 10,
                    batch_size = 32,
                    validation_data = (x_val, y_val))

model.save_weights('./saved_checkpoints/Chapter 6.1.2 - Using word embeddings/pre_trained_glove_model.h5')


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.1.2 - Using word embeddings.ipynb".
The first task is:
The layers transforms a 2D input tensor of integer of shape (number_of samples, sequence_length) into a 3D floating point tensor, of shape (number_of_samples, sequence_length, embedding_dimensionality.)
Such tensor can be processed a RNN layer of a 1D convolutional layer.
IMDB example
Can you write Python code for it?

from keras.datasets import imdb
from keras.preprocessing.sequence import pad_sequences

# Number of words considered as features
max_features = 10000

# Cutting reviews after only 20 words
sequence_max_length = 20

# Loading data
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words = max_features)

x_train.shape

x_train_sequence = pad_sequences(x_train, maxlen = sequence_max_length)

x_train_sequence.shape

x_train[0:2]

x_train[0].__getitem__(-20)

x_train_sequence[0, :]

x_train_sequence[0]

x_train_sequence[1]


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.1.2 - Using word embeddings.ipynb".
The first task is:
Tokenizing the data
Can you write Python code for it?

from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
import numpy as np

# Using only first 100 words of each review
maxlen = 100

# Number of training samples
training_samples = 200

# Number of validation samples
validation_samples = 10000

# Tokenizing only top 10 000 words in the dataset.
max_words = 10000

# Initializing Tokenizer
tokenizer = Tokenizer(num_words = max_words)

# Fitting the Tokenizer on the text
tokenizer.fit_on_texts(texts)

# Text to sequence
sequences = tokenizer.texts_to_sequences(texts)

sequences[0:2]

# Word index
word_index = tokenizer.word_index

type(word_index)

first10pairs = {k: word_index[k] for k in list(word_index)[:10]}

first10pairs

# Padding the sequence
data = pad_sequences(sequences, maxlen = maxlen)

data.shape

labels = np.asarray(labels)

labels.shape

indices = np.arange(data.shape[0])
np.random.shuffle(indices)
data = data[indices]
labels = labels[indices]

# Splitting the data into train and validation datasets
x_train = data[:training_samples]
y_train = labels[:training_samples]
x_val = data[training_samples: training_samples + validation_samples]
y_val = labels[training_samples: training_samples + validation_samples]

x_train.shape

x_val.shape


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.2 - Understanding recurrent neural networks.ipynb".
The first task is:
IMDB example
Can you write Python code for it?

from keras.datasets import imdb
from keras.preprocessing.sequence import pad_sequences

# Number of words to be used as features
max_features = 10000
# Cutting the review after this number of words
maxlen = 500
batch_size = 32

print('Loading data...')
(input_train, y_train), (input_test, y_test) = imdb.load_data(num_words=max_features)
print(len(input_train), 'train sequences')
print(len(input_test), 'test sequences')

print('Pad sequences (samples x time)')
input_train = pad_sequences(input_train, maxlen=maxlen)
input_test = pad_sequences(input_test, maxlen=maxlen)
print('input_train shape:', input_train.shape)
print('input_test shape:', input_test.shape)

from keras.layers import Dense

model = Sequential()
model.add(Embedding(input_dim = max_features,
                    output_dim = 32))
model.add(SimpleRNN(units = 32))
model.add(Dense(units = 1,
                activation='sigmoid'))

model.compile(optimizer = 'rmsprop',
              loss = 'binary_crossentropy',
              metrics = ['acc'])
history = model.fit(x = input_train,
                    y = y_train,
                    epochs = 10,
                    batch_size = 128,
                    validation_split = 0.2)


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.2 - Understanding recurrent neural networks.ipynb".
The first task is:
LSTM
Can you write Python code for it?

from keras.layers import LSTM

model = Sequential()
model.add(Embedding(input_dim = max_features,
                    output_dim = 32))
model.add(LSTM(units = 32))
model.add(Dense(units = 1,
                activation = 'sigmoid'))

model.compile(optimizer = 'rmsprop',
              loss = 'binary_crossentropy',
              metrics = ['acc'])
history = model.fit(x = input_train,
                    y = y_train,
                    epochs = 10,
                    batch_size = 128,
                    validation_split = 0.2)


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.2 - Understanding recurrent neural networks.ipynb".
The first task is:
Stacking multiple recurrent layers on top of each other can have benefits, like with convolutional neural networks.
Can you write Python code for it?

model = Sequential()
model.add(Embedding(input_dim = 10000,
                    output_dim = 32))
model.add(SimpleRNN(units = 32,
                    return_sequences = True))
model.add(SimpleRNN(units = 32,
                    return_sequences = True))
model.add(SimpleRNN(units = 32,
                    return_sequences = True))
# The last layer returns only the last outputs
model.add(SimpleRNN(32))
model.summary()


I am working on the file "18-03-07-Deep Learning With Python by François Chollet/Chapter 6.2 - Understanding recurrent neural networks.ipynb".
The first task is:
Visualizing the results
Can you write Python code for it?

acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()


import matplotlib.pyplot as plt

acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(acc))

plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensoardaten-Workcamp-ML-10.ipynb".
The first task is:

Visualisierung der Daten

Can you write Python code for it?

# Ausgabe Histogramm
pyplot.rcParams["figure.figsize"] = (15,12)
dataset.hist()
pyplot.show()

# Ausgabe der Dichtefunktion
pyplot.rcParams["figure.figsize"] = (15,12)
dataset.plot(kind='density', subplots=True, layout=(8,8), sharex=False, legend=False)
pyplot.show()

# Ausgabe scatter plot matrix
pyplot.rcParams["figure.figsize"] = (15,12)
scatter_matrix(dataset)
pyplot.show()

# Ausgabe correlation matrix
fig = pyplot.figure()
pyplot.rcParams["figure.figsize"] = (15,12)
ax = fig.add_subplot(111)
cax = ax.matshow(dataset.corr(), vmin=-1, vmax=1, interpolation='none')
fig.colorbar(cax)
pyplot.show()


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensoardaten-Workcamp-ML-10.ipynb".
The first task is:
<h3>Vorbereiten der Daten: Aufteilen in Test und Trainingsdaten </h3>
Can you write Python code for it?

# Split-out validation dataset
array = dataset.values
X = array[:,0:10].astype(float)
Y = array[:,10]
validation_size = 0.30
seed = 7
X_train, X_validation, Y_train, Y_validation = train_test_split(X, Y, test_size=validation_size, random_state=seed)

# Evaluate Algorithms

# Test options and evaluation metric
num_folds = 10
seed = 7
scoring = 'accuracy'

# Spot Check Algorithms
models = []
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))
results = []
names = []
for name, model in models:
	kfold = KFold(n_splits=num_folds, random_state=seed)
	cv_results = cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring)
	results.append(cv_results)
	names.append(name)
	msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
	print(msg)

# Compare Algorithms
fig = pyplot.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
pyplot.boxplot(results)
ax.set_xticklabels(names)
pyplot.show()

# Daten standardisieren
# Erstellen einer Lise fuer die Pipelines
pipelines = []
pipelines.append(('ScaledLR', Pipeline([('Scaler', StandardScaler()),('LR', LogisticRegression())])))
pipelines.append(('ScaledLDA', Pipeline([('Scaler', StandardScaler()),('LDA', LinearDiscriminantAnalysis())])))
pipelines.append(('ScaledKNN', Pipeline([('Scaler', StandardScaler()),('KNN', KNeighborsClassifier())])))
pipelines.append(('ScaledCART', Pipeline([('Scaler', StandardScaler()),('CART', DecisionTreeClassifier())])))
pipelines.append(('ScaledNB', Pipeline([('Scaler', StandardScaler()),('NB', GaussianNB())])))
pipelines.append(('ScaledSVM', Pipeline([('Scaler', StandardScaler()),('SVM', SVC())])))
results = []
names = []
for name, model in pipelines:
	kfold = KFold(n_splits=num_folds, random_state=seed)
	cv_results = cross_val_score(model, X_train, Y_train, cv=kfold,...


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Daten Skalieren-Workcamp-ML.ipynb".
The first task is:

Beschreibende Statistik

Can you write Python code for it?

# Ausgabe df.shape
print(datensatz.shape)

# Ausgabe df.dtypes
# Spalte enthält die Classifikation R oder M
set_option('display.max_rows', 50)
print(datensatz.dtypes)

# Ausgabe df.head mit vergösserter display width
set_option('display.width', 100)
print(datensatz.head(20))

# Ausgabe df.describe() mit 4 Nachkomma Stellen
set_option('precision', 4)
print(datensatz.describe())

# Ausgabe der Klassen Verteilung in der Spalte 60
print(datensatz.groupby('Label').size())


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Daten Skalieren-Workcamp-ML.ipynb".
The first task is:

Binärisierung der Daten

Wir können die Daten mit einer binären Schwelle (threshold) transformieren. Alle Werte oberhalb der Schwelle werden mit 1 und alle gleich oder kleiner mit 0 gekennzeichnet. Dies wird als Binärisierung der Daten. Es kann nützlich sein, wenn wir scharfe Trennungen haben wollen. Es ist auch im Feature Engineering nützlich, wenn neue Festures mit großer Bedeutung hinzugefügt werden sollen. Wir können neue binäre Attribute in Python in scikit-learn mit der Binarizer class erstellen.
Can you write Python code for it?

# Binärisierung der Daten
# Laden des Moduls Binarizer aus sklearn.preprocessing
from sklearn.preprocessing import Binarizer
# Übergabe der Werte in datensatz an ein array3
array3 = datensatz.values
# Aufteilen des arrays in abhängige Variable Y und unabhängige Variable X
X = array3[:,0:10]
Y = array3[:,10]
# Einstellen des threshold() für den Binarizer. Wir wählen 50
binarizer = Binarizer(threshold=50.0).fit(X)
binaryX = binarizer.transform(X)
# Ausgabe einer Kurzfassung der Daten 0:10
set_printoptions(precision=3)
print(binaryX[0:10,:])


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Daten Skalieren-Workcamp-ML.ipynb".
The first task is:

Normalisierung der Daten

Die Normalisierung in scikit-learn bezieht sich auf die Skalierung jeder Beobachtung (Zeile) auf eine Länge von 1 (eine Einheitsnorm oder ein Vektor mit der Länge von 1 in der linearen Algebra). Diese Vorverarbeitungsmethode kann für spärliche Datensätze (viele Nullen) mit Attributen unterschiedlicher Größenordnung nützlich sein, wenn Algorithmen verwendet werden, die Eingabewerte gewichten, wie Neuronale Netze und Algorithmen mit Entfernungsmessungen wie k-Nearest Neighbors. Wir können Daten in Python in scikit-learn mit der Normalizer class normalisieren.
Can you write Python code for it?

# Normalisierung der Daten (Vektorlänge 1)
# Laden des Moduls Normalizer
from sklearn.preprocessing import Normalizer
#Übergabe der Werte in datensatz an ein array2
array2 = datensatz.values
# Aufteilen des arrays in abhängige Variable Y und unabhängige Variable X
X = array2[:,0:10]
Y = array2[:,10]
scaler = Normalizer().fit(X)
normalizedX = scaler.transform(X)
# Ausgabe einer Kurzfassung der Daten 0:10
set_printoptions(precision=3)
print(normalizedX[0:10,:])


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Daten Skalieren-Workcamp-ML.ipynb".
The first task is:

Skalierung der Daten

Wenn die Daten aus Attributen mit unterschiedlichen Größenordnungen bestehen, soe wie hier im Datensatz der Sensordaten, können sich viele Algorithmen des maschinellen Lernens durch die Neuskalierung der Attribute im Ergebnis verbessern. Oft wird dies als Normalisierung bezeichnet. Die Attribute werden dann in den Bereich zwischen 0 und 1 skaliert. Dies ist nützlich zum Beispiel für Optimierungsalgorithmen, die im Kern von maschinellen Lernalgorithmen wie dem Gradientenabstieg (gradient descent) verwendet werden. Dies ist auch nützlich bei Algorithmen, wie Regression und Neuronale Netze, die Eingaben gewichten und Algorithmen, die Entfernungsmessungen verwenden, wie zum Beipiel k-Nearest Neighbors. Wir können die Daten in scikit-learn mit der MinMaxScaler-Klasse neu skalieren.
Can you write Python code for it?

# Skalierung der Daten
# Laden des Moduls MinMaxScaler aus sklearn.preprocessing
from sklearn.preprocessing import MinMaxScaler
# Übergabe der Werte in datensatz an ein array4
array4 = datensatz.values
# Aufteilen des arrays in abhängige Variable Y und unabhängige Variable X
X = array4[:,0:10]
Y = array4[:,10]
# Einstellen der feature_range() im MinMaxScaler
scaler = MinMaxScaler(feature_range=(0, 1))
rescaledX = scaler.fit_transform(X)
# Ausgabe einer Kurzfassung der Daten 0:10 mit precision=2
set_printoptions(precision=2)
print(rescaledX[0:10,:])


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Daten Skalieren-Workcamp-ML.ipynb".
The first task is:

Standardisierung der Daten

Standardisierung ist eine nützliche Technik, um Attribute mit einer beliebigen Gauss-Normalverteilung N( , ) in eine Standard-Gaußsche Verteilung N(0,1) mit einem Mittelwert von 0 und einer Standardabweichung von 1 zu transformieren. Die Standardisierung ist am besten geeignet für Algorithmen, die eine Gaußsche Verteilung in den Eingangsvariablen annehmen und besser mit skalierten Daten arbeiten, wie z.B. Lineare Regression, Logistische Regression und Lineare Diskriminanzanalyse. Wir können Daten mit scikit-learn mit der StandardScaler-Klasse standardisieren.
Can you write Python code for it?

# Standardisierung der Daten (Mittelwert = 0 , Standardabweichung = 1)
# mit from sklearn.preprocessing import StandardScaler wurde die Klasse in Zelle 1 bereits geladen
#Laden des Moduls set_printoptions
from numpy import set_printoptions
# Übergabe der Werte in datensatz an ein array
array = datensatz.values
# Aufteilen des arrays in abhängige Variable Y und unabhängige Variable X
X = array[:,0:10]
Y = array[:,10]
scaler = StandardScaler().fit(X)
rescaledX = scaler.transform(X)
# Ausgabe einer Kurzfassung der Daten 0:10
set_printoptions(precision=3)
print(rescaledX[0:10,:])


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Daten Skalieren-Workcamp-ML.ipynb".
The first task is:

Visualisierung der Daten

Can you write Python code for it?

# Ausgabe Histogramm
pyplot.rcParams["figure.figsize"] = (15,12)
datensatz.hist()
pyplot.show()


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Feature-Selektion-Workcamp-ML.ipynb".
The first task is:

Beschreibende Statistik

Can you write Python code for it?


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Feature-Selektion-Workcamp-ML.ipynb".
The first task is:

Univariate Feature Selektion

Uum die Merkmale auszuwählen, die am stärksten mit der Ausgabevariablen verknüpft sind, können Statistische Tests verwendet werden,. Die scikit-learn Bibliothek stellt die SelectKBest Klasse zur Verfügung, die mit einer Reihe von unterschiedlichen statistischen Tests zur Auswahl der relavantesten Features eingesetzt werden kann. Das folgende Beispiel verwendet den Chi-squared (Chi2) statistischen Test für nicht-negative Merkmale, um 5 der besten Merkmale aus den Sensordaten auszuwählen.
Can you write Python code for it?

from numpy import set_printoptions
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
# Übergabe der Dtaen
array = datensatz.values
X = array[:,0:10]
Y = array[:,10]
# Feature Extraktion
test = SelectKBest(score_func=chi2, k=5)
fit = test.fit(X, Y)
# Zusammenfassung der Ergebnisse
set_printoptions(precision=3)
print(fit.scores_)
features = fit.transform(X)
# Ausgewählte Features
print(features[0:9,:])


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Feature-Selektion-Workcamp-ML.ipynb".
The first task is:

Visualisierung der Daten

Can you write Python code for it?


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Feature-Selektion-Workcamp-ML.ipynb".
The first task is:
Die Abschätzung der Bedeutung von Features wählt die gleichen Sensoren aus wie bei der univariaten Auswahl.

Weiteres Beispiel

Can you write Python code for it?

# Feature Importance with Extra Trees Classifier
from sklearn.ensemble import ExtraTreesClassifier
# Übergabe des Dateinamens an die Variable dateiname
dateiname = 'pima-indians-diabetes.data.csv'
# Festlegen der Spalten Namen für den DataFrame
namen = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class']
# Einlesen der Daten in einen panda DataFrame mit read_csv()
df = read_csv(dateiname, names=namen)
# Übergabe der Werte in df an ein array5
array5 = df.values
# Aufteilen des arrays in abhängige Variable Y und unabhängige Variable X - hier steht die Klasse in Spalte 9
X = array5[:,0:8]
Y = array5[:,8]
# feature extraction
model = ExtraTreesClassifier()
model.fit(X, Y)
print(model.feature_importances_)


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Feature-Selektion-Workcamp-ML.ipynb".
The first task is:
Die rekursive Feature Elimination wählt die gleichen Sensoren aus wie bei der univariaten Auswahl.

Principal Component Analysis

Die Principal Component Analysis (oder PCA) verwendet lineare Algebra, um den Datensatz in eine komprimierte Form zu transformieren. Im Allgemeinen wird dies als Datenreduktionstechnik bezeichnet. Eine Eigenschaft von PCA ist, dass wir die Anzahl der Dimensionen oder Hauptkomponenten im transformierten Ergebnis wählen können. Im folgenden Beispiel verwenden wir PCA und wählen 3 Hauptkomponenten aus.
Can you write Python code for it?

# Binärisierung der Daten
# Laden des Moduls Binarizer aus sklearn.preprocessing
from sklearn.decomposition import PCA
# Übergabe der Werte in datensatz an ein array3
array3 = datensatz.values
# Aufteilen des arrays in abhängige Variable Y und unabhängige Variable X
X = array3[:,0:10]
Y = array3[:,10]
# feature extraction
pca = PCA(n_components=3)
fit = pca.fit(X)
# summarize components
print("Explained Variance: %s" % fit.explained_variance_ratio_)
print(fit.components_)


I am working on the file "18-05-14-ml-workcamp/sensor-daten-10/Projekt-Sensordaten-Feature-Selektion-Workcamp-ML.ipynb".
The first task is:
Problem Beschreibung:

Der Fokus dieses Projektes liegt auf dem Datensatz "sensordaten-10.csv". Das Problem ist die Vorhersage von guten und schlechten Werkstücken aus den 10 Sensordaten. Jedes Muster ist ein Satz von 10 Zahlen. Die Sensoren decken unterschiedliche Wertebereiche ab.Das Label, das jeder Datenreihe zugeordnet ist, enthält 0 oder 1. Wenn das Werkstück die Beurteilung gut hat steht eine 1 in der Spalte Label, sonst eine 0.

Aufgabe:

Laden Sie die Daten und verschaffen Sie sich einen ersten Überblick

Can you write Python code for it?

#Laden der Daten [12100 Datensätze mit 10 Sensoren und einer Spalte Label (12100x11)Matrix]
url = 'sensordaten-10.csv'
datensatz = read_csv(url, sep=';', header=0)


I am working on the file "18-05-14-ml-workcamp/sonar-daten/Projekt-Sonardaten-Workcamp-ML.ipynb".
The first task is:

Beschreibende Statistik

Can you write Python code for it?

# Ausgabe df.shape
print(dataset.shape)

# Ausgabe df.dtypes
# Spalte enthält die Classifikation R oder M
set_option('display.max_rows', 500)
print(dataset.dtypes)

# Ausgabe df.head mit vergösserter display width
set_option('display.width', 100)
print(dataset.head(20))

# Ausgabe df.describe() mit 4 Nachkomma Stellen
set_option('precision', 4)
print(dataset.describe())

# Ausgabe der Klassen Verteilung in der Spalte 60
print(dataset.groupby(60).size())


I am working on the file "18-05-14-ml-workcamp/sonar-daten/Projekt-Sonardaten-Workcamp-ML.ipynb".
The first task is:

Visualisierung der Daten

Can you write Python code for it?

# Ausgabe Histogramm
dataset.hist()
pyplot.show()

# Ausgabe der Dichtefunktion
dataset.plot(kind='density', subplots=True, layout=(8,8), sharex=False, legend=False)
pyplot.show()

# Ausgabe scatter plot matrix
scatter_matrix(dataset)
pyplot.show()

# Ausgabe correlation matrix
fig = pyplot.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(dataset.corr(), vmin=-1, vmax=1, interpolation='none')
fig.colorbar(cax)
pyplot.show()


I am working on the file "18-05-14-ml-workcamp/sonar-daten/Projekt-Sonardaten-Workcamp-ML.ipynb".
The first task is:
<h3>Vorbereiten der Daten: Aufteilen in Test und Trainingsdaten </h3>
Can you write Python code for it?

# Split-out validation dataset
array = dataset.values
X = array[:,0:60].astype(float)
Y = array[:,60]
validation_size = 0.20
seed = 7
X_train, X_validation, Y_train, Y_validation = train_test_split(X, Y, test_size=validation_size, random_state=seed)

# Evaluate Algorithms

# Test options and evaluation metric
num_folds = 10
seed = 7
scoring = 'accuracy'

# Spot Check Algorithms
models = []
models.append(('LR', LogisticRegression()))
models.append(('LDA', LinearDiscriminantAnalysis()))
models.append(('KNN', KNeighborsClassifier()))
models.append(('CART', DecisionTreeClassifier()))
models.append(('NB', GaussianNB()))
models.append(('SVM', SVC()))
results = []
names = []
for name, model in models:
	kfold = KFold(n_splits=num_folds, random_state=seed)
	cv_results = cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring)
	results.append(cv_results)
	names.append(name)
	msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
	print(msg)

# Compare Algorithms
fig = pyplot.figure()
fig.suptitle('Algorithm Comparison')
ax = fig.add_subplot(111)
pyplot.boxplot(results)
ax.set_xticklabels(names)
pyplot.show()

# Standardize the dataset
pipelines = []
pipelines.append(('ScaledLR', Pipeline([('Scaler', StandardScaler()),('LR', LogisticRegression())])))
pipelines.append(('ScaledLDA', Pipeline([('Scaler', StandardScaler()),('LDA', LinearDiscriminantAnalysis())])))
pipelines.append(('ScaledKNN', Pipeline([('Scaler', StandardScaler()),('KNN', KNeighborsClassifier())])))
pipelines.append(('ScaledCART', Pipeline([('Scaler', StandardScaler()),('CART', DecisionTreeClassifier())])))
pipelines.append(('ScaledNB', Pipeline([('Scaler', StandardScaler()),('NB', GaussianNB())])))
pipelines.append(('ScaledSVM', Pipeline([('Scaler', StandardScaler()),('SVM', SVC())])))
results = []
names = []
for name, model in pipelines:
	kfold = KFold(n_splits=num_folds, random_state=seed)
	cv_results = cross_val_score(model, X_train, Y_train, cv=kfold, scoring=scoring)
	results.append(cv_results)
	names.append(name)
	msg = "%s:...


I am working on the file "18-05-14-ml-workcamp/sonar-daten/Projekt-Sonardaten-Workcamp-ML.ipynb".
The first task is:
Im Ergebnis sind also K-NN und SCM die Algorithmen

die bessere Ergebnisse liefern

In die weiteren Betrachtungen und Optimierungen werden nur

noch diese Algorithmen einbezogen.

Can you write Python code for it?

# Tuning K-NN mit skalierten Daten - Die Anzahl der Nachbarn wird variiert
scaler = StandardScaler().fit(X_train)
rescaledX = scaler.transform(X_train)
neighbors = [1,3,5,7,9,11,13,15,17,19,21]
param_grid = dict(n_neighbors=neighbors)
model = KNeighborsClassifier()
kfold = KFold(n_splits=num_folds, random_state=seed)
grid = GridSearchCV(estimator=model, param_grid=param_grid, scoring=scoring, cv=kfold)
grid_result = grid.fit(rescaledX, Y_train)
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

# Tuning der SVM mit skalierten Daten über ein param_grid
scaler = StandardScaler().fit(X_train)
rescaledX = scaler.transform(X_train)
c_values = [0.1, 0.3, 0.5, 0.7, 0.9, 1.0, 1.3, 1.5, 1.7, 2.0]
kernel_values = ['linear', 'poly', 'rbf', 'sigmoid']
param_grid = dict(C=c_values, kernel=kernel_values)
model = SVC()
kfold = KFold(n_splits=num_folds, random_state=seed)
grid = GridSearchCV(estimator=model, param_grid=param_grid, scoring=scoring, cv=kfold)
grid_result = grid.fit(rescaledX, Y_train)
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))

# Aufbauen von Ensembles
ensembles = []
ensembles.append(('AB', AdaBoostClassifier()))
ensembles.append(('GBM', GradientBoostingClassifier()))
ensembles.append(('RF', RandomForestClassifier()))
ensembles.append(('ET', ExtraTreesClassifier()))
results = []
names = []
for name, model in ensembles:
	kfold = KFold(n_splits=num_folds, random_state=seed)
	cv_results = cross_val_score(model, X_train, Y_train, cv=kfold,...


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/00-Crash-Course-Basics/04-Crash-Course-Review-Exercises.ipynb".
The first task is:
Create a 2-D visualization using plt.imshow of the numpy matrix with a colorbar. Add a title to your plot. Bonus: Figure out how to change the aspect of the imshow() plot.
Can you write Python code for it?

fig = plt.figure(figsize = (12, 12))
plt.imshow(random_integers, aspect = 0.05)
plt.colorbar()
plt.title("2D visualisation")


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/00-Crash-Course-Basics/04-Crash-Course-Review-Exercises.ipynb".
The first task is:
Create a NumPy Matrix of 100 rows by 5 columns consisting of random integers from 1-100. (Keep in mind that the upper limit may be exclusive.
Can you write Python code for it?

random_integers = np.random.randint(low = 1,
                                    high = 101,
                                    size = (100, 5))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/00-Crash-Course-Basics/04-Crash-Course-Review-Exercises.ipynb".
The first task is:
Now use pd.DataFrame() to read in this numpy array as a dataframe. Simple pass in the numpy array into that function to get back a dataframe. Pandas will auto label the columns to 0-4
Can you write Python code for it?

df = pd.DataFrame(random_integers)
df.head()


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/00-Crash-Course-Basics/04-Crash-Course-Review-Exercises.ipynb".
The first task is:
Set Numpy's Random Seed to 101
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/00-Crash-Course-Basics/04-Crash-Course-Review-Exercises.ipynb".
The first task is:
Using your previously created DataFrame, use df.columns = [...] to rename the pandas columns to be ['f1','f2','f3','f4','label']. Then perform a train/test split with scikitlearn.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

df.columns = ['f1','f2','f3','f4','label']
df.head()

X = df.iloc[:, df.columns != 'label']
Y = df['label']

X.shape

Y.shape

X_train, X_test, Y_train, Y_test = train_test_split(X, Y,
                                                    test_size = 0.3,
                                                    random_state = 101)

X_train.shape

X_test.shape

Y_train.shape

Y_test.shape


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
Activation Function
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

# Defining sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

sample_z = np.linspace(-10, 10, 100)
sample_a = sigmoid(sample_z)

plt.figure(figsize = (8, 8))
plt.title("Sigmoid")
plt.plot(sample_z, sample_a)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
Classification Example
Can you write Python code for it?

from sklearn.datasets import make_blobs

# Creating 50 samples divided into 2 blobs with 2 features
data = make_blobs(n_samples = 50,
                  n_features = 2,
                  centers = 2,
                  random_state = 75)

data

features = data[0]
plt.scatter(features[:,0],features[:,1])

labels = data[1]
plt.scatter(x = features[:,0],
            y = features[:,1],
            c = labels,
            cmap = 'coolwarm')

# DRAW A LINE THAT SEPERATES CLASSES
x = np.linspace(0, 11 ,10)
y = -x + 5
plt.scatter(features[:,0],
            features[:,1],
            c = labels,
            cmap = 'coolwarm')
plt.plot(x,y)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
Graph
Can you write Python code for it?

class Graph():
    def __init__(self):
        self.operations = []
        self.placeholders = []
        self.variables = []

    def set_as_default(self):
        """
        Sets this Graph instance as the Global Default Graph
        """
        global _default_graph
        _default_graph = self


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
If we want to use initialization from the parent class, we can do that using:
python
super().__init__()
Can you write Python code for it?

class ExtendedClass(SimpleClass):
    def __init__(self):
        super().__init__(" My String")
        print('EXTENDED')

s = ExtendedClass()


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
Looks like we did it!
Can you write Python code for it?

g = Graph()

g.set_as_default()

A = Variable([[10, 20], [30, 40]])
b = Variable([1, 1])

x = Placeholder()

y = matmul(A,x)

z = add(y,b)

sess = Session()

result = sess.run(operation = z,
                  feed_dict = {x : 10})

result


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
Operation
Can you write Python code for it?

class Operation():
    """
    An Operation is a node in a "Graph". TensorFlow will also use this concept of a Graph.

    This Operation class will be inherited by other classes that actually compute the specific
    operation, such as adding or matrix multiplication.
    """
    def __init__(self, input_nodes = []):
        """
        Intialize an Operation
        """
        # The list of input nodes
        self.input_nodes = input_nodes

        # Initialize list of nodes consuming this node's output
        self.output_nodes = []

        # For every node in the input, we append this operation (self) to the list of
        # the consumers of the input nodes
        for node in input_nodes:
            node.output_nodes.append(self)

        # There will be a global default graph (TensorFlow works this way)
        # We will then append this particular operation
        # Append this operation to the list of operations in the currently active default graph
        _default_graph.operations.append(self)

    def compute(self):
        """
        This is a placeholder function. It will be overwritten by the actual specific operation
        that inherits from this class.
        """
        pass


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
Or if we have (4,-10)
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
Placeholders
Can you write Python code for it?

class Placeholder():
    """
    A placeholder is a node that needs to be provided a value for computing the output in the Graph.
    In case of supervised learning, X (input) and Y (output) will require placeholders.
    """
    def __init__(self):
        self.output_nodes = []
        _default_graph.placeholders.append(self)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
The child class will use its own initialization method, if not specified otherwise.
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
The result should look like:
Variable (A), Placeholder (x), Multiple operation (Ax), Variable (b), Add (Ax + b)
Can you write Python code for it?

result

10 * 10 + 1

# Running just y = Ax
# The post order should be only up to
result = sess.run(operation = y,
                  feed_dict = {x : 10})

result


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/01-Introduction-to-Neural-Networks/01_Neural Network From Scratch.ipynb".
The first task is:
Using an Example Session Graph
Can you write Python code for it?

g = Graph()

g.set_as_default()

x = Placeholder()

w = Variable([1,1])

b = Variable(-5)

z = add(matmul(w,x),b)

a = Sigmoid(z)

sess = Session()

sess.run(operation = a,
         feed_dict = {x : [8, 10]})

sess.run(operation = a,
         feed_dict = {x : [2, -10]})


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Categorical Features
If you know the set of all possible feature values of a column and there are only a few of them, you can use categorical_column_with_vocabulary_list. If you don't know the set of possible values in advance you can use categorical_column_with_hash_bucket
Can you write Python code for it?

assigned_group = tf.feature_column.categorical_column_with_vocabulary_list('Group',['A','B','C','D'])
# Alternative
# assigned_group = tf.feature_column.categorical_column_with_hash_bucket('Group', hash_bucket_size=10)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Clean the Data
Can you write Python code for it?

# Columns that will be normalized
cols_to_norm = ['Number_pregnant', 'Glucose_concentration', 'Blood_pressure', 'Triceps',
                'Insulin', 'BMI', 'Pedigree']

# Normalizing the columns
diabetes[cols_to_norm] = diabetes[cols_to_norm].apply(lambda x: (x - x.min()) / (x.max() - x.min()))

diabetes.head()


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Continuous Features

Number of times pregnant
Plasma glucose concentration a 2 hours in an oral glucose tolerance test
Diastolic blood pressure (mm Hg)
Triceps skin fold thickness (mm)
2-Hour serum insulin (mu U/ml)
Body mass index (weight in kg/(height in m)^2)
Diabetes pedigree function
Can you write Python code for it?

num_preg = tf.feature_column.numeric_column('Number_pregnant')
plasma_gluc = tf.feature_column.numeric_column('Glucose_concentration')
dias_press = tf.feature_column.numeric_column('Blood_pressure')
tricep = tf.feature_column.numeric_column('Triceps')
insulin = tf.feature_column.numeric_column('Insulin')
bmi = tf.feature_column.numeric_column('BMI')
diabetes_pedigree = tf.feature_column.numeric_column('Pedigree')
age = tf.feature_column.numeric_column('Age')


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Converting Continuous to Categorical
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

diabetes['Age'].hist(bins = 20)

age_buckets = tf.feature_column.bucketized_column(age, boundaries=[20, 30, 40, 50, 60, 70, 80])


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Creating the Model
Can you write Python code for it?

model = tf.estimator.LinearClassifier(feature_columns = feat_cols,
                                      n_classes = 2)

model.train(input_fn = input_func,
            steps = 1000)

# Useful link for your own data
# https://stackoverflow.com/questions/44664285/what-are-the-contraints-for-tensorflow-scope-names


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
DNN Classifier
Can you write Python code for it?

dnn_model = tf.estimator.DNNClassifier(hidden_units=[10, 10, 10],
                                       feature_columns = feat_cols,
                                       n_classes = 2)

# Creating an embedding columns with 4 groups (A, B, C, D)
embedded_group_column = tf.feature_column.embedding_column(assigned_group,
                                                           dimension = 4)

feat_cols = [num_preg, plasma_gluc, dias_press, tricep, insulin,
             bmi, diabetes_pedigree, embedded_group_column, age_buckets]

input_func = tf.estimator.inputs.pandas_input_fn(x = X_train,
                                                 y = y_train,
                                                 batch_size = 10,
                                                 num_epochs = 1000,
                                                 shuffle = True)

dnn_model = tf.estimator.DNNClassifier(hidden_units=[10, 10, 10],
                                       feature_columns = feat_cols,
                                       n_classes = 2)

dnn_model.train(input_fn = input_func,
                steps = 1000)

eval_input_func = tf.estimator.inputs.pandas_input_fn(
      x = X_test,
      y = y_test,
      batch_size = 10,
      num_epochs = 1,
      shuffle = False)

dnn_model.evaluate(eval_input_func)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Evaluation
Can you write Python code for it?

eval_input_func = tf.estimator.inputs.pandas_input_fn(
    x = X_test,
    y = y_test,
    batch_size = 10,
    num_epochs = 1,
    shuffle = False)

results = model.evaluate(eval_input_func)

results


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Input Function
Can you write Python code for it?

input_func = tf.estimator.inputs.pandas_input_fn(x = X_train,
                                                 y = y_train,
                                                 batch_size = 10,
                                                 num_epochs = 1000,
                                                 shuffle = True)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Predictions
Can you write Python code for it?

pred_input_func = tf.estimator.inputs.pandas_input_fn(
      x = X_test,
      batch_size = 10,
      num_epochs = 1,
      shuffle = False)

# Predictions is a generator!
predictions = model.predict(pred_input_func)

list(predictions)[0:5]


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Putting them together
Can you write Python code for it?

feat_cols = [num_preg, plasma_gluc, dias_press, tricep, insulin,
             bmi, diabetes_pedigree, assigned_group, age_buckets]


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/05-TensorFlow-Classification-Example.ipynb".
The first task is:
Train Test Split
Can you write Python code for it?

diabetes.head()

diabetes.info()

# Dropping 'Class' to exclude the column
x_data = diabetes.drop('Class',axis = 1)

labels = diabetes['Class']

from sklearn.model_selection import train_test_split

# Test train split
X_train, X_test, y_train, y_test = train_test_split(x_data,
                                                    labels,
                                                    test_size = 0.33,
                                                    random_state = 101)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/06-Regression-Exercise.ipynb".
The first task is:
Create Feature Columns
 Create the necessary tf.feature_column objects for the estimator. They should all be trated as continuous numeric_columns.
Can you write Python code for it?

df.columns

import tensorflow as tf

age = tf.feature_column.numeric_column('housingMedianAge')
rooms = tf.feature_column.numeric_column('totalRooms')
bedrooms = tf.feature_column.numeric_column('totalBedrooms')
pop = tf.feature_column.numeric_column('population')
households = tf.feature_column.numeric_column('households')
income = tf.feature_column.numeric_column('medianIncome')

feature_columns = [age, rooms, bedrooms, pop, households, income]


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/06-Regression-Exercise.ipynb".
The first task is:
Create the estimator model. Use a DNNRegressor. Play around with the hidden units!
Can you write Python code for it?

dnn_model = tf.estimator.DNNRegressor(hidden_units = [6, 5, 5], feature_columns = feature_columns)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/06-Regression-Exercise.ipynb".
The first task is:
Create the input function for the estimator object. (play around with batch_size and num_epochs)
Can you write Python code for it?

input_feature_func = tf.estimator.inputs.pandas_input_fn(x = X_train,
                                                         y = Y_train,
                                                         batch_size = 10,
                                                         num_epochs = 1000,
                                                         shuffle = True)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/06-Regression-Exercise.ipynb".
The first task is:
Scale the Feature Data
 Use sklearn preprocessing to create a MinMaxScaler for the feature data. Fit this scaler only to the training data. Then use it to transform X_test and X_train. Then use the scaled X_test and X_train along with pd.Dataframe to re-create two dataframes of scaled data.
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

scaler.fit(X_train)

# Keeping Pandas DataFrame format after re-scaling
X_train = pd.DataFrame(data = scaler.transform(X_train),
                       columns = X_train.columns,
                       index = X_train.index)

X_test = pd.DataFrame(data = scaler.transform(X_test),
                      columns = X_test.columns,
                      index = X_test.index)

X_train.head()


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/06-Regression-Exercise.ipynb".
The first task is:
Train the model for ~1,000 steps. (Later come back to this and train it for more and check for improvement)
Can you write Python code for it?

dnn_model.train(input_fn = input_feature_func, steps = 5000)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Create Input Function
 Batch_size is up to you. But do make sure to shuffle!
Can you write Python code for it?

input_func = tf.estimator.inputs.pandas_input_fn(x = X_train,
                                                 y = y_train,
                                                 batch_size = 100,
                                                 num_epochs = None,
                                                 shuffle = True)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Create a list of only the class_ids key values from the prediction list of dictionaries, these are the predictions you will use to compare against the real y_test values.
Can you write Python code for it?

final_preds = []
for pred in predictions:
    final_preds.append(pred['class_ids'][0])

final_preds[:10]


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Create the continuous feature_columns for the continuous values using numeric_column
Can you write Python code for it?

age = tf.feature_column.numeric_column("age")
education_num = tf.feature_column.numeric_column("education_num")
capital_gain = tf.feature_column.numeric_column("capital_gain")
capital_loss = tf.feature_column.numeric_column("capital_loss")
hours_per_week = tf.feature_column.numeric_column("hours_per_week")


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Create the tf.feature_columns for the categorical values. Use vocabulary lists or just use hash buckets.
Can you write Python code for it?

gender = tf.feature_column.categorical_column_with_vocabulary_list("gender", ["Female", "Male"])
occupation = tf.feature_column.categorical_column_with_hash_bucket("occupation", hash_bucket_size=1000)
marital_status = tf.feature_column.categorical_column_with_hash_bucket("marital_status", hash_bucket_size=1000)
relationship = tf.feature_column.categorical_column_with_hash_bucket("relationship", hash_bucket_size=1000)
education = tf.feature_column.categorical_column_with_hash_bucket("education", hash_bucket_size=1000)
workclass = tf.feature_column.categorical_column_with_hash_bucket("workclass", hash_bucket_size=1000)
native_country = tf.feature_column.categorical_column_with_hash_bucket("native_country", hash_bucket_size=1000)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Each item in your list will look like this:
Can you write Python code for it?


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Evaluation
 Create a prediction input function. Remember to only supprt X_test data and keep shuffle=False.
Can you write Python code for it?

pred_fn = tf.estimator.inputs.pandas_input_fn(x = X_test,
                                              batch_size = len(X_test),
                                              shuffle = False)


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Import classification_report from sklearn.metrics and then see if you can figure out how to use it to easily get a full report of your model's performance on the test data.
Can you write Python code for it?

from sklearn.metrics import classification_report

print(classification_report(y_test, final_preds))


I am working on the file "18-05-28-Complete-Guide-to-Tensorflow-for-Deep-Learning-with-Python/02-Tensorflow-Basics/07-Classification-Exercise.ipynb".
The first task is:
Perform a Train Test Split on the Data
Can you write Python code for it?

from sklearn.model_selection import train_test_split

x_data = census.drop('income_bracket', axis = 1)
y_labels = census['income_bracket']
X_train, X_test, y_train, y_test = train_test_split(x_data, y_labels, test_size = 0.3,random_state = 101)

x_data.head()

y_labels.head()