I am working on the file "ImageSimilarity_and_ClusterDemo.ipynb".
The first task is:
To check whether the file search has succeeded, display some found images to get a feeling of the data we are going to deal with.
Can you write Python code for it?

img1=imread(jpegFilePaths[0])
img2=imread(jpegFilePaths[1388])
img3=imread(jpegFilePaths[1389])

#Creates two subplots and unpacks the output array immediately
f, (ax1, ax2,ax3) = plt.subplots(1, 3, sharex='all', sharey='all')
ax1.axis('off')
ax2.axis('off')
ax3.axis('off')

ax1.set_title("Image #0")
ax1.imshow(img1)
ax2.set_title("Image #1388")
ax2.imshow(img2)
ax3.set_title("Image #1389")
ax3.imshow(img3)


I am working on the file "ImageSimilarity_and_ClusterDemo.ipynb".
The first task is:
To facilitate the assessment of the effectiveness of the different measures and metrics, the next cell creates an HTML overview document with the first found documents.
Sample results are available here.
Can you write Python code for it?

measures=["p1","p2","histdiff","histcut","emd","cosine","phash"]
ranks=dict()
printLog("Creating QBE report files...")

htmlFile=open(outputDir+"_qbe.html", "w")
printLog("HTML output will be saved to: %s"%outputDir+"_qbe.html")
htmlFile.write("\n")
htmlFile.write("<link href='../css/helvetica.css' rel='stylesheet' type='text/css'>\n")
#htmlFile.write("\n")

htmlFile.write("
mir comparison.
")
htmlFile.write("

\n")
for measureName in measures:
    typeOfMeasure="distance"
    # check whether the measure is a similarity or a distance measure
    # (assuming identity (i.e., identity of indiscernibles) of the measure)
    if df[df.index==qbeIndex][measureName].tolist()[0]>0:
        df2=df.sort_values(by=[measureName],ascending=False).head(20)
        typeOfMeasure="similarity"
    else:
        df2=df.sort_values(by=[measureName],ascending=True).head(20)
        typeOfMeasure="distance"

    htmlFile.write("\n")
    measureTitle=measureName
    if typeOfMeasure=="similarity":
        measureTitle=measureName.replace("dist_","sim_")
    htmlFile.write("
"+measureTitle+"
\n")
    htmlFile.write("
"+typeOfMeasure+"
\n")
    ranks[measureName]=df2.index.tolist()

    jpegFilePathsReport=[]
    # image directory must be relative to the directory of the html files
    imgBaseDir="./extracted_images/"
    for row in df2.itertuples(index=False):
        i=row.index
        score=getattr(row, measureName)
        # create JPEG copies if not available already
        tiffImage=imgBaseDir+ppnList[i]+"/"+nameList[i]
        jpgPath=tiffImage.replace(".tif",".jpg")
        if not...


I am working on the file "ImageSimilarity_and_ClusterDemo.ipynb".
The first task is:
To give an example, we will extract ORB features from the first two images and match them. The discovered matches will be illustrated below.
Can you write Python code for it?

# extract features
descriptor_extractor = ORB(n_keypoints=200)

descriptor_extractor.detect_and_extract(img1)
keypoints1 = descriptor_extractor.keypoints
descriptors1 = descriptor_extractor.descriptors

descriptor_extractor.detect_and_extract(img2)
keypoints2 = descriptor_extractor.keypoints
descriptors2 = descriptor_extractor.descriptors

# match features
matches12 = match_descriptors(descriptors1, descriptors2, cross_check=True)

# visualize the results
fig, ax = plt.subplots(nrows=1, ncols=1)

plt.gray()

plot_matches(ax, img1, img2, keypoints1, keypoints2, matches12)
ax.axis('off')
ax.set_title("Image 1 vs. Image 2")


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
Another Example
Can you write Python code for it?

from skimage import filters, data
xray = plt.imread('chestxray.jpg')

#apply sobel gradient
sobel_xray = filters.sobel(xray)

#apply prewitt gradient
prewitt_xray = filters.prewitt(xray)

figure(figsize=(15, 5))
subplot(1, 3, 1)
imshow(xray, cmap=cm.gray)
title('Original')
subplot(1, 3, 2)
imshow(sobel_xray, cmap=cm.gray)
title('Result of a Sobel gradient')
subplot(1, 3, 3)
imshow(prewitt_xray, cmap=cm.gray)
title('Result of a Prewitt Gradient')


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
As one can easily see, an image corrupted with speckle noise can be better denoised with mean filtering. Now check an image with salt and pepper noise.
Can you write Python code for it?

salt = plt.imread('saltandpepper.jpg')
mask = (1/9)*np.array([[1, 1, 1], [1, 1, 1], [1, 1, 1]])
salt_lp = ndimage.convolve(salt, mask, mode='constant', cval=0.0)
salt_med = median(salt, disk(2))
figure(figsize=(15, 5))
subplot(1, 3, 1)
imshow(salt, cmap=cm.gray)
title('Original')
subplot(1, 3, 2)
imshow(salt_lp, cmap=cm.gray)
title('Filtered with LP mask $\mathcal{L}_1$')
subplot(1, 3, 3)
imshow(salt_med, cmap=cm.gray)
title('Median filtering')


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
As we can see that the image has low contrast, we will use the following transformation function to stretch the contrast of the image.
$$s=\mathcal{W}(r)=\begin{cases}
0, & r<103\
\frac{255}{132}(r-103), & 103\le r\le235\
255, & r>235
\end{cases}$$
Can you write Python code for it?

r = np.arange(0, 256, 1)
s = np.zeros(shape(r))
s = (255/intensity_range)*(r - mini)
s[r<mini] = 0
s[r>maxi] = 255

plt.plot(r, s)
plt.axis([0, 260, -5, 260])
xlabel('r')
ylabel('s')
title('gray-level transformation function $\mathcal{W}$')
plt.grid()

girl_high = ((girl.astype('float64') - mini) * 255 / intensity_range).astype('uint8')

plt.imshow(girl_high, cmap=cm.gray)

plt.hist(girl_high.flatten(), 256, range=(0, 255), fc='k', ec='k');


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
Bit Plane Slicing

Sometimes it is desirable to highlight the contribution made by specific bits to the total image appearance.
The image can be imagined to be composed of Eight 1-bit planes -- Plane 0 for the LSB plane and Plane 7 for the MSB.
The higher order bits contain visually significant data, the lower order plane contain more subtle details.

This can be acomplished by doing a Bitwise AND operation. For example, say the intensity of a pixel is 246 in decimal. In binary this would be $(1111\,0110)_2$. So in order to find the value of the 6th bit, one has to siimply do this $(1111\,0110)_2 \odot  (0100\,0000)_2$. The result will simply produce the value of the 6th bit. In other words, $246 \odot 64$ will give you the value of the 6th bit.
Can you write Python code for it?

plane7 = A_gr & 128*np.ones(shape(A_gr)).astype('uint8')
plane6 = A_gr &  64*np.ones(shape(A_gr)).astype('uint8')
plane5 = A_gr &  32*np.ones(shape(A_gr)).astype('uint8')
plane4 = A_gr &  16*np.ones(shape(A_gr)).astype('uint8')
plane3 = A_gr &   8*np.ones(shape(A_gr)).astype('uint8')
plane2 = A_gr &   4*np.ones(shape(A_gr)).astype('uint8')
plane1 = A_gr &   2*np.ones(shape(A_gr)).astype('uint8')
plane0 = A_gr &   1*np.ones(shape(A_gr)).astype('uint8')

plt.figure(figsize=(20,7))
plt.subplot(2, 4, 1)
plt.imshow(plane7, cmap=cm.gray)
plt.title('Plane 7 (MSB)')
plt.subplot(2, 4, 2)
plt.imshow(plane6, cmap=cm.gray)
plt.title('Plane 6')
plt.subplot(2, 4, 3)
plt.imshow(plane5, cmap=cm.gray)
plt.title('Plane 5')
plt.subplot(2, 4, 4)
plt.imshow(plane4, cmap=cm.gray)
plt.title('Plane 4')
plt.subplot(2, 4, 5)
plt.imshow(plane3, cmap=cm.gray)
plt.title('Plane 3')
plt.subplot(2, 4, 6)
plt.imshow(plane2, cmap=cm.gray)
plt.title('Plane 2')
plt.subplot(2, 4, 7)
plt.imshow(plane1, cmap=cm.gray)
plt.title('Plane 1')
plt.subplot(2, 4, 8)
plt.imshow(plane0, cmap=cm.gray)
plt.title('Plane 0 (LSB)')


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
Change the value of the threshold in the above programme and see the changes!
Now consider gradual thresholding. We consider that the region of interest lies between intensity values of 100 and 170.
Can you write Python code for it?

thres1 = copy(scanned)
threshold_hi = 170
threshold_lo = 100
thres1[scanned<threshold_lo] = 0
thres1[scanned>threshold_hi] = 255
plt.imshow(thres1, cmap=cm.gray)


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
Check the data type and the size of the array $A$
Can you write Python code for it?


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
In case you didn't have any error so far, you're good to go!
What is an image anyway?
Short answer It is just a 2 dimensional array (grayscale image) or a set of Three 2 dimensional arrays (colour image).
Long answer
Loading an Image as a Numpy array
Can you write Python code for it?


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
Laplacian filter
A second order derivative filter can be implemented by employing a Laplacian mask. The Laplacian of an image function $f(x, y)$ of two variables is defined as $\nabla ^2 f(x, y) = \dfrac{\partial ^2f(x,y)}{\partial x^2} + \dfrac{\partial ^2f(x,y)}{\partial y^2}.$
Thus, $\nabla ^2 f(x, y) = f(x+1, y) + f(x-1, y) + f(x, y+1) + f(x, y-1) - 4f(x,y)$.
So, a Laplacian mask would look like
$$\nabla^2_\perp=\underset{\textrm{Laplacian mask}}{\underbrace{\left[\begin{array}{rrr}
0 & 1 & 0\
1 & -4 & 1\
0 & 1 & 0
\end{array}\right]}}; \quad
\nabla^2_\odot=\underset{\textrm{Omnidirectional Laplacian mask}}{\underbrace{\left[\begin{array}{rrr}
1 & 1 & 1\
1 & -8 & 1\
1 & 1 & 1
\end{array}\right]}}.$$
The second one, here, considers four directions 1. horizontal, 2. vertical, 3. +45$^\circ$ and 4. -45$^\circ$, whereas, the first one only considers the horizontal and vertical directions.
However, there is a problem regarding the direct implementation of a Laplacian mask. Being a second derivative operation, it highlights intensity discontinuities in an image, and in the process de-emphasizes image regions having slow variations in intensity profile. SO, in order to preserve the original background features and yet perform sharpening operation, the Laplacian operator is utilized in the following manner:
$$g(x,y) = f(x, y) + c\left[\nabla^2 f(x, y)\right],$$ where, $c=-1$ for the operators we considered.
Here is a comparison of Laplacian filter with Sobel filters. For the first time, we are going to use OpenCV.
Can you write Python code for it?

import cv2 #this is OpenCV
import numpy as np
from matplotlib import pyplot as plt

img = cv2.imread('dave.jpg',0) #import the image as grayscale

laplacian = cv2.Laplacian(img,cv2.CV_64F)
sobelx = cv2.Sobel(img,cv2.CV_64F,1,0,ksize=5)
sobely = cv2.Sobel(img,cv2.CV_64F,0,1,ksize=5)

plt.figure(figsize=(15, 10))
plt.subplot(2,2,1)
plt.imshow(img, cmap = 'gray')
plt.title('Original')
plt.subplot(2,2,2),plt.imshow(laplacian,cmap = 'gray')
plt.title('Laplacian')
plt.subplot(2,2,3)
plt.imshow(sobelx,cmap = 'gray')
plt.title('Sobel X')
plt.subplot(2,2,4)
plt.imshow(sobely,cmap = 'gray')
plt.title('Sobel Y')

plt.show()


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
Let us now narrow the contrast the image
Contrast Stretching
The possible causes for low contrast images are
1. poor illumination
2. lack of dynamic range in imaging sensor
3. wrong setting of the lens aperture during image acquisition.
Contrast stretching attempts to increase the dynamic range of the gray levels of the image being processed. For a neighbourhood of size $1 \times 1$, contast stretching is usually done by a gray level transformation of the form $s = \mathcal{W}[r]$, where $r$ is the gray level of $f(x, y)$ at $(x, y)$, $s$ is the gray level of $g(x, y)$ at $(x, y)$ and $\mathcal{W}$ is a graylevel transformation function.
Now, let us load a low contrast image.
Can you write Python code for it?

girl = plt.imread('low_contrast.jpg')
plt.imshow(girl)

plt.hist(girl.flatten(), 256, range=(0, 255), fc='k', ec='k');

maxi = np.amax(girl)
mini = np.amin(girl)
intensity_range = maxi - mini
print('lowest intensity:', mini, ', highest intensity:', maxi, ', spread:', intensity_range)


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
Now that we know how to read and display an image. Let us do this:
Can you write Python code for it?

#create a random uint8 array of size 300x300 with numbers from 0 to 255
x = np.random.randint(0, 256, (300, 400)).astype('uint8')
plt.imshow(x, cmap=cm.gray)


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
See that, now the pixel intensities are spread over wider range $[0, 255]$. This is known as linear contrast streatching.
Negative
If you subtract the pixel intensities of the original image from 255, what you get is essentially the negative of the image.
$$ X_{neg} = \left[\begin{array}{ccc}
255 & \cdots & 255\
\vdots & \ddots & \vdots\
255 & \cdots & 255
\end{array}\right]
 {m\times n} - X{m\times n}.$$
The transformation function in this case looks like this.
Can you write Python code for it?

r = np.arange(0, 256, 1)
s = np.zeros(shape(r))
s = 255 - r

plt.plot(r, s)
plt.axis([0, 260, -5, 260])
xlabel('r')
ylabel('s')
title('$\mathcal{W}$ for finding the negative of an image')
plt.grid()

girl_neg = (255*np.ones(shape(girl_high)) - girl_high).astype('uint8')

plt.imshow(girl_neg, cmap=cm.gray)

plt.hist(girl_neg.flatten(), 256, range=(0, 255), fc='k', ec='k');


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
See the difference!
Sharpening spatial filters
Derivative filters

The differentiation operation is expected to sharpen an image.
One can use either first derivative or second derivative information.

Digital approximation of first derivative : $\dfrac{\partial f(x,y)}{\partial x} = f(x+1, y) - f(x, y), $ and $\dfrac{\partial f(x,y)}{\partial y} = f(x, y+1) - f(x, y).$
Constraints : The response of a first derivative filter must be
1. zero in areas of constant intensity,
2. must be non-zero at the onset of an intensity step or ramp,
3. nonzero along ramps.
Digital approximation of second derivative : $\dfrac{\partial ^2f(x,y)}{\partial x^2} = f(x+1, y) - 2 f(x, y) +f(x-1, y),$ and $\dfrac{\partial ^2f(x,y)}{\partial y^2} = f(x, y+1) - 2 f(x, y) +f(x, y-1).$
Constraints : The response of a second order derivative filter must be
1. zero in areas of constant intensity,
2. non-zero at the onset and the end of an intensity step or ramp,
3. zero along ramps of constant slope.
Implementing a first derivative filter for image sharpening
A first derivative image sharpening filter can be implemented by by applying the Gradient function. The gradient of a function $f(x, y)$ at coordinates $(x, y)$ is defined as the 2D column vector
$$\nabla f(x,y)\equiv\mathrm{grad}(f)\equiv \mathbf{g} \equiv \left[\begin{array}{c}
g_{x}(x,y)\
g_{y}(x,y)
\end{array}\right]=\left[\begin{array}{c}
\frac{\partial f(x,y)}{\partial x}\
\frac{\partial f(x,y)}{\partial y}
\end{array}\right].$$
The magnitude (length) of vector $\nabla f$ is given by $M(x, y) = ||\nabla f||= \sqrt{\mathbf{g}^T \mathbf{g}} = \sqrt{g_x^2 + g_y^2}$. In image processing, this last expression is often approximated as $|g_x| + |g_y|$.
$M(x, y)$ is an image of the same size of the original and called the gradient image. The computation of this gradient is the basis of various approaches to develop first derivative filter.
If $\left[\begin{array}{ccc}
z_{1} & z_{2} & z_{3}\
z_{4} & z_{5} & z_{6}\
z_{7} & z_{8} & z_{9}
\end{array}\right]$ is the image section under...
Can you write Python code for it?

from skimage import filters, data
camera = data.camera()

#apply sobel gradient
sobel_camera = filters.sobel(camera)

#apply prewitt gradient
prewitt_camera = filters.prewitt(camera)

figure(figsize=(15, 5))
subplot(1, 3, 1)
imshow(camera, cmap=cm.gray)
title('Original')
subplot(1, 3, 2)
imshow(sobel_camera, cmap=cm.gray)
title('Result of a Sobel gradient')
subplot(1, 3, 3)
imshow(prewitt_camera, cmap=cm.gray)
title('Result of a Prewitt Gradient')


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
See! A random image! Now let us create a random color image!
Can you write Python code for it?

X = np.random.randint(0, 256, (300, 400, 3)).astype('uint8')
plt.imshow(X)


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
So, our image is a colour image and it has a resolution of $400 \times 267$. It is imported as an N-dimenstional array object available from numpy.
Can you write Python code for it?


I am working on the file "Image_Processing_in_Python.ipynb".
The first task is:
Spatial Filtering
Low pass filters attenuate or eliminate high frequncy components in the Fourier domain. Low pass filtering gives rise to image blurring.
High pass filters attenuate or eliminate low frequncy components in the Fourier domain. High pass filtering gives rise to sharpening of edges and other sharp details.
How to implement?
Utilize suitable 2D masks of suitable size, e.g. $3 \times 3$, $5 \times 5$ or $7 \times 7$.
In most of our use cases, we shall restrict ourselves to $3 \times 3$ masks. The mask is applied to certain pixels. Upon application, the mask calculates a wighted sum of the neighbourhood of the concerned pixel and the result substitutes the original pixel. Here is how it works.
Consider that there is a pixel of intensity $z_5$. The $3 \times 3$ neighbourhood of the pixel can be seen as $\left[\begin{array}{ccc}
z_{1} & z_{2} & z_{3}\
z_{4} & z_{5} & z_{6}\
z_{7} & z_{8} & z_{9}
\end{array}\right]$.
Now consider that the mask that is applied on the pixel with intensity $z_5$ looks like $\left[\begin{array}{ccc}
w_{1} & w_{2} & w_{3}\
w_{4} & w_{5} & w_{6}\
w_{7} & w_{8} & w_{9}
\end{array}\right]$.
Then, it will substitute $z_5$ by $w_1 z_1 + w_2 z_2 + \cdots + w_9 z_9$. Thus, $$z_{5{new}}=\sum_{i=1}^9 w_i z_i.$$
Thus,
$$\left[\begin{array}{ccc}
z_{1} & z_{2} & z_{3}\
z_{4} & z_{5} & z_{6}\
z_{7} & z_{8} & z_{9}
\end{array}\right] \otimes
\left[\begin{array}{ccc}
w_{1} & w_{2} & w_{3}\
w_{4} & w_{5} & w_{6}\
w_{7} & w_{8} & w_{9}
\end{array}\right] =
\left[\begin{array}{ccc}
z_{1} & z_{2} & z_{3}\
z_{4} & \sum_{i=1}^9 w_i z_i & z_{6}\
z_{7} & z_{8} & z_{9}
\end{array}\right].$$
It does not do any changes to any other pixel anyway. The mask is centred on the image pixel whose new intensity value is tobe calculated. This calculation is performed for each pixel seperately by moving the mask to centre it on the pixel under consideration.
Smoothig spatial filters
Low pass spatial filtering
Examples of low pass spatial filter masks are
$$\mathcal{L}_1 =...
Can you write Python code for it?

#mean filtering
from scipy import ndimage
lena_noisy = plt.imread('lena_noisy.png')
mask1 = (1/9)*np.array([[1, 1, 1], [1, 1, 1], [1, 1, 1]]) #LP mask
mask2 = (1/16)*np.array([[1, 2, 1], [2, 1, 2], [1, 2, 1]]) #LP mask
result1 = ndimage.convolve(lena_noisy, mask1, mode='constant', cval=0.0)
result2 = ndimage.convolve(lena_noisy, mask2, mode='constant', cval=0.0)
figure(figsize=(15, 5))
subplot(1, 3, 1)
imshow(lena_noisy, cmap=cm.gray)
title('Original')
subplot(1, 3, 2)
imshow(result1, cmap=cm.gray)
title('Filtered with LP mask $\mathcal{L}_1$')
subplot(1, 3, 3)
imshow(result2, cmap=cm.gray)
title('Filtered with LP mask $\mathcal{L}_2$')


I am working on the file "Images/music/samples/TensorFlow/1_hello_tensorflow.ipynb".
The first task is:
Above, we're taking a 1 x 4 vector [1 0 0 1] and multiplying it by a 4 by 2 matrix full of random values from a normal distribution (mean 0, stdev 1). The output is a 1 x 2 matrix.
You might try modifying this example. Running the cell multiple times will generate new random weights and a new output. Or, change the input, e.g., to [0 0 0 1]), and run the cell again. Or, try initializing the weights using the TensorFlow op, e.g., random_normal, instead of using numpy to generate the random weights.
What we have here is the basics of a simple neural network already. If we are reading in the input features, along with some expected output, and change the weights based on the error with the output each time, that's a neural network.
Use of variables
Let's look at adding two small matrices in a loop, not by creating new tensors every time, but by updating the existing values and then re-running the computation graph on the new data. This happens a lot with machine learning models, where we change some parameters each time such as gradient descent on some weights and then perform the same computations over and over again.
Can you write Python code for it?

#@test {"output": "ignore"}
import tensorflow as tf
import numpy as np

with tf.Session() as sess:
    # Set up two variables, total and weights, that we'll change repeatedly.
    total = tf.Variable(tf.zeros([1, 2]))
    weights = tf.Variable(tf.random_uniform([1,2]))

    # Initialize the variables we defined above.
    tf.global_variables_initializer().run()

    # This only adds the operators to the graph right now. The assignment
    # and addition operations are not performed yet.
    update_weights = tf.assign(weights, tf.random_uniform([1, 2], -1.0, 1.0))
    update_total = tf.assign(total, tf.add(total, weights))

    for _ in range(5):
        # Actually run the operation graph, so randomly generate weights and then
        # add them into the total. Order does matter here. We need to update
        # the weights before updating the total.
        sess.run(update_weights)
        sess.run(update_total)

        print(weights.eval(), total.eval())


I am working on the file "Images/music/samples/TensorFlow/1_hello_tensorflow.ipynb".
The first task is:
Adding two matrices
Next, let's do something very similar, adding two matrices:
$\begin{bmatrix}
  1. & 1. & 1. \
  1. & 1. & 1. \
\end{bmatrix} +
\begin{bmatrix}
  1. & 2. & 3. \
  4. & 5. & 6. \
\end{bmatrix} =
\begin{bmatrix}
  2. & 3. & 4. \
  5. & 6. & 7. \
\end{bmatrix}$
Can you write Python code for it?

import tensorflow as tf
import numpy as np

with tf.Session():
    input1 = tf.constant(1.0, shape=[2, 3])
    input2 = tf.constant(np.reshape(np.arange(1.0, 7.0, dtype=np.float32), (2, 3)))
    output = tf.add(input1, input2)
    print(output.eval())


I am working on the file "Images/music/samples/TensorFlow/1_hello_tensorflow.ipynb".
The first task is:
Details of adding two vectors in TensorFlow
The example above of adding two vectors involves a lot more than it seems, so let's look at it in more depth.

import tensorflow as tf

This import brings TensorFlow's public API into our IPython runtime environment.

with tf.Session():

When you run an operation in TensorFlow, you need to do it in the context of a Session. A session holds the computation graph, which contains the tensors and the operations. When you create tensors and operations, they are not executed immediately, but wait for other operations and tensors to be added to the graph, only executing when finally requested to produce the results of the session. Deferring the execution like this provides additional opportunities for parallelism and optimization, as TensorFlow can decide how to combine operations and where to run them after TensorFlow knows about all the operations.

input1 = tf.constant([1.0, 1.0, 1.0, 1.0])
input2 = tf.constant([2.0, 2.0, 2.0, 2.0])

The next two lines create tensors using a convenience function called constant, which is similar to numpy's array and numpy's full. If you look at the code for constant, you can see the details of what it is doing to create the tensor. In summary, it creates a tensor of the necessary shape and applies the constant operator to it to fill it with the provided values. The values to constant can be Python or numpy arrays. constant can take an optional shape parameter, which works similarly to numpy's fill if provided, and an optional name parameter, which can be used to put a more human-readable label on the operation in the TensorFlow operation graph.

output = tf.add(input1, input2)

You might think add just adds the two vectors now, but it doesn't quite do that. What it does is put the add operation into the computational graph. The results of the addition aren't available yet. They've been put in the computation graph, but the computation graph hasn't been executed yet.

result = output.eval()
print result

eval() is also slightly...
Can you write Python code for it?

import tensorflow as tf

with tf.Session():
    input1 = tf.constant(1.0, shape=[4])
    input2 = tf.constant(2.0, shape=[4])
    input3 = tf.constant(3.0, shape=[4])
    output = tf.add(tf.add(input1, input2), input3)
    result = output.eval()
    print(result)


I am working on the file "Images/music/samples/TensorFlow/1_hello_tensorflow.ipynb".
The first task is:
Recall that you can pass numpy or Python arrays into constant.
In this example, the matrix with values from 1 to 6 is created in numpy and passed into constant, but TensorFlow also has range, reshape, and tofloat operators. Doing this entirely within TensorFlow could be more efficient if this was a very large matrix.
Try experimenting with this code a bit -- maybe modifying some of the values, using the numpy version, doing this using, adding another operation, or doing this using TensorFlow's range function.
Multiplying matrices
Let's move on to matrix multiplication. This time, let's use a bit vector and some random values, which is a good step toward some of what we'll need to do for regression and neural networks.
Can you write Python code for it?

#@test {"output": "ignore"}
import tensorflow as tf
import numpy as np

with tf.Session():
    input_features = tf.constant(np.reshape([1, 0, 0, 1], (1, 4)).astype(np.float32))
    weights = tf.constant(np.random.randn(4, 2).astype(np.float32))
    output = tf.matmul(input_features, weights)
    print("Input:")
    print(input_features.eval())
    print("Weights:")
    print(weights.eval())
    print("Output:")
    print(output.eval())


I am working on the file "Images/music/samples/TensorFlow/1_hello_tensorflow.ipynb".
The first task is:
This version uses constant in a way similar to numpy's fill, specifying the optional shape and having the values copied out across it.
The add operator supports operator overloading, so you could try writing it inline as input1 + input2 instead as well as experimenting with other operators.
Can you write Python code for it?

with tf.Session():
    input1 = tf.constant(1.0, shape=[4])
    input2 = tf.constant(2.0, shape=[4])
    output = input1 + input2
    print(output.eval())


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
As expected without training, the predictions are all noise. Let's write a scoring function that picks the class with the maximum probability and compares with the example's label. We'll start by converting the probability vectors returned by the softmax into predictions we can match against the labels.
Can you write Python code for it?

# The highest probability in the first entry.
print('First prediction', numpy.argmax(predictions[0]))

# But, predictions is actually a list of BATCH_SIZE probability vectors.
print(predictions.shape)

# So, we'll take the highest probability for each vector.
print('All predictions', numpy.argmax(predictions, 1))


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
Great -- we've retained the correct image data while properly rescaling to the range [-0.5, 0.5].
Reading the labels
Let's next unpack the test label data. The format here is similar: a magic number followed by a count followed by the labels as uint8 values. In more detail:
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  10000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

As with the image data, let's read  the first test set value to sanity check our input path. We'll expect a 7.
Can you write Python code for it?

with gzip.open(test_labels_filename) as f:
    # Print the header fields.
    for field in ['magic number', 'label count']:
        print(field, struct.unpack('>i', f.read(4))[0])

    print('First label:', struct.unpack('B', f.read(1))[0])


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
Looks good. Now we know how to index our full set of training and test images.
Label data
Let's move on to loading the full set of labels. As is typical in classification problems, we'll convert our input labels into a 1-hot encoding over a length 10 vector corresponding to 10 digits. The vector [0, 1, 0, 0, 0, 0, 0, 0, 0, 0], for example, would correspond to the digit 1.
Can you write Python code for it?

NUM_LABELS = 10

def extract_labels(filename, num_images):
    """Extract the labels into a 1-hot matrix [image index, label index]."""
    print('Extracting', filename)
    with gzip.open(filename) as bytestream:
        # Skip the magic number and count; we know these values.
        bytestream.read(8)
        buf = bytestream.read(1 * num_images)
        labels = numpy.frombuffer(buf, dtype=numpy.uint8)
    # Convert to dense 1-hot representation.
    return (numpy.arange(NUM_LABELS) == labels[:, None]).astype(numpy.float32)

train_labels = extract_labels(train_labels_filename, 60000)
test_labels = extract_labels(test_labels_filename, 10000)


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
Now let's wrap this up into our scoring function.
Can you write Python code for it?

def error_rate(predictions, labels):
    """Return the error rate and confusions."""
    correct = numpy.sum(numpy.argmax(predictions, 1) == numpy.argmax(labels, 1))
    total = predictions.shape[0]

    error = 100.0 - (100 * float(correct) / float(total))

    confusions = numpy.zeros([10, 10], numpy.float32)
    bundled = zip(numpy.argmax(predictions, 1), numpy.argmax(labels, 1))
    for predicted, actual in bundled:
        confusions[predicted, actual] += 1

    return error, confusions

print('Done')


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
Now that we've defined the variables to be trained, we're ready to wire them together into a TensorFlow graph.
We'll define a helper to do this, model, which will return copies of the graph suitable for training and testing. Note the train argument, which controls whether or not dropout is used in the hidden layer. (We want to use dropout only during training.)
Can you write Python code for it?

def model(data, train=False):
    """The Model definition."""
    # 2D convolution, with 'SAME' padding (i.e. the output feature map has
    # the same size as the input). Note that {strides} is a 4D array whose
    # shape matches the data layout: [image index, y, x, depth].
    conv = tf.nn.conv2d(data,
                        conv1_weights,
                        strides=[1, 1, 1, 1],
                        padding='SAME')

    # Bias and rectified linear non-linearity.
    relu = tf.nn.relu(tf.nn.bias_add(conv, conv1_biases))

    # Max pooling. The kernel size spec ksize also follows the layout of
    # the data. Here we have a pooling window of 2, and a stride of 2.
    pool = tf.nn.max_pool(relu,
                          ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1],
                          padding='SAME')
    conv = tf.nn.conv2d(pool,
                        conv2_weights,
                        strides=[1, 1, 1, 1],
                        padding='SAME')
    relu = tf.nn.relu(tf.nn.bias_add(conv, conv2_biases))
    pool = tf.nn.max_pool(relu,
                          ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1],
                          padding='SAME')

    # Reshape the feature map cuboid into a 2D matrix to feed it to the
    # fully connected layers.
    pool_shape = pool.get_shape().as_list()
    reshape = tf.reshape(
        pool,
        [pool_shape[0], pool_shape[1] * pool_shape[2] * pool_shape[3]])

    # Fully connected layer. Note that the '+' operation automatically
    # broadcasts the biases.
    hidden = tf.nn.relu(tf.matmul(reshape, fc1_weights) + fc1_biases)

    # Add a 50% dropout during training only. Dropout also scales
    # activations such that no rescaling is needed at evaluation time.
    if train:
        hidden = tf.nn.dropout(hidden, 0.5, seed=SEED)
    return tf.matmul(hidden, fc2_weights) + fc2_biases

print('Done')


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
Now we can compare the predicted and label classes to compute the error rate and confusion matrix for this batch.
Can you write Python code for it?

correct = numpy.sum(numpy.argmax(predictions, 1) == numpy.argmax(batch_labels, 1))
total = predictions.shape[0]

print(float(correct) / float(total))

confusions = numpy.zeros([10, 10], numpy.float32)
bundled = zip(numpy.argmax(predictions, 1), numpy.argmax(batch_labels, 1))
for predicted, actual in bundled:
  confusions[predicted, actual] += 1

plt.grid(False)
plt.xticks(numpy.arange(NUM_LABELS))
plt.yticks(numpy.arange(NUM_LABELS))
plt.imshow(confusions, cmap=plt.cm.jet, interpolation='nearest');


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
Now we're ready to perform operations on the graph. Let's start with one round of training. We're going to organize our training steps into batches for efficiency; i.e., training using a small set of examples at each step rather than a single example.
Can you write Python code for it?

BATCH_SIZE = 60

# Grab the first BATCH_SIZE examples and labels.
batch_data = train_data[:BATCH_SIZE, :, :, :]
batch_labels = train_labels[:BATCH_SIZE]

# This dictionary maps the batch data (as a numpy array) to the
# node in the graph it should be fed to.
feed_dict = {train_data_node: batch_data,
             train_labels_node: batch_labels}

# Run the graph and fetch some of the nodes.
_, l, lr, predictions = s.run(
  [optimizer, loss, learning_rate, train_prediction],
  feed_dict=feed_dict)

print('Done')


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
The 1-hot encoding looks reasonable.
Segmenting data into training, test, and validation
The final step in preparing our data is to split it into three sets: training, test, and validation. This isn't the format of the original data set, so we'll take a small slice of the training data and treat that as our validation set.
Can you write Python code for it?

VALIDATION_SIZE = 5000

validation_data = train_data[:VALIDATION_SIZE, :, :, :]
validation_labels = train_labels[:VALIDATION_SIZE]
train_data = train_data[VALIDATION_SIZE:, :, :, :]
train_labels = train_labels[VALIDATION_SIZE:]

train_size = train_labels.shape[0]

print('Validation shape', validation_data.shape)
print('Train size', train_size)


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
The first 10 pixels are all 0 values. Not very interesting, but also unsurprising. We'd expect most of the pixel values to be the background color, 0.
We could print all 28 * 28 values, but what we really need to do to make sure we're reading our data properly is look at an image.
Can you write Python code for it?

%matplotlib inline

# We'll show the image and its pixel value histogram side-by-side.
_, (ax1, ax2) = plt.subplots(1, 2)

# To interpret the values as a 28x28 image, we need to reshape
# the numpy array, which is one dimensional.
ax1.imshow(image.reshape(28, 28), cmap=plt.cm.Greys);

ax2.hist(image, bins=20, range=[0,255]);


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
The large number of 0 values correspond to the background of the image, another large mass of value 255 is black, and a mix of grayscale transition values in between.
Both the image and histogram look sensible. But, it's good practice when training image models to normalize values to be centered around 0.
We'll do that next. The normalization code is fairly short, and it may be tempting to assume we haven't made mistakes, but we'll double-check by looking at the rendered input and histogram again. Malformed inputs are a surprisingly common source of errors when developing new models.
Can you write Python code for it?

# Let's convert the uint8 image to 32 bit floats and rescale
# the values to be centered around 0, between [-0.5, 0.5].
#
# We again plot the image and histogram to check that we
# haven't mangled the data.
scaled = image.astype(numpy.float32)
scaled = (scaled - (255 / 2.0)) / 255
_, (ax1, ax2) = plt.subplots(1, 2)
ax1.imshow(scaled.reshape(28, 28), cmap=plt.cm.Greys);
ax2.hist(scaled, bins=20, range=[-0.5, 0.5]);


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
We can see here that we're mostly accurate, with some errors you might expect, e.g., '9' is often confused as '4'.
Let's do another sanity check to make sure this matches roughly the distribution of our test set, e.g., it seems like we have fewer '5' values.
Can you write Python code for it?


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
We'll need to train for some time to actually see useful predicted values. Let's define a loop that will go through our data. We'll print the loss and error periodically.
Here, we want to iterate over the entire data set rather than just the first batch, so we'll need to slice the data to that end.
(One pass through our training set will take some time on a CPU, so be patient if you are executing this notebook.)
Can you write Python code for it?

# Train over the first 1/4th of our training set.
steps = train_size // BATCH_SIZE
for step in range(steps):
    # Compute the offset of the current minibatch in the data.
    # Note that we could use better randomization across epochs.
    offset = (step * BATCH_SIZE) % (train_size - BATCH_SIZE)
    batch_data = train_data[offset:(offset + BATCH_SIZE), :, :, :]
    batch_labels = train_labels[offset:(offset + BATCH_SIZE)]
    # This dictionary maps the batch data (as a numpy array) to the
    # node in the graph it should be fed to.
    feed_dict = {train_data_node: batch_data,
                 train_labels_node: batch_labels}
    # Run the graph and fetch some of the nodes.
    _, l, lr, predictions = s.run(
      [optimizer, loss, learning_rate, train_prediction],
      feed_dict=feed_dict)

    # Print out the loss periodically.
    if step % 100 == 0:
        error, _ = error_rate(predictions, batch_labels)
        print('Step %d of %d' % (step, steps))
        print('Mini-batch loss: %.5f Error: %.5f Learning rate: %.5f' % (l, error, lr))
        print('Validation error: %.1f%%' % error_rate(
              validation_prediction.eval(), validation_labels)[0])


I am working on the file "Images/music/samples/TensorFlow/3_hello_tensorflow.ipynb".
The first task is:
Working with the images
Now we have the files, but the format requires a bit of pre-processing before we can work with it. The data is gzipped, requiring us to decompress it. And, each of the images are grayscale-encoded with values from [0, 255]; we'll normalize these to [-0.5, 0.5].
Let's try to unpack the data using the documented format:
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000803(2051) magic number
0004     32 bit integer  60000            number of images
0008     32 bit integer  28               number of rows
0012     32 bit integer  28               number of columns
0016     unsigned byte   ??               pixel
0017     unsigned byte   ??               pixel
........
xxxx     unsigned byte   ??               pixel

Pixels are organized row-wise. Pixel values are 0 to 255. 0 means background (white), 255 means foreground (black).
We'll start by reading the first image from the test data as a sanity check.
Can you write Python code for it?

import gzip, binascii, struct, numpy
import matplotlib.pyplot as plt

with gzip.open(test_data_filename) as f:
    # Print the header fields.
    for field in ['magic number', 'image count', 'rows', 'columns']:
        # struct.unpack reads the binary data provided by f.read.
        # The format string '>i' decodes a big-endian integer, which
        # is the encoding of the data.
        print(field, struct.unpack('>i', f.read(4))[0])

    # Read the first 28x28 set of pixel values.
    # Each pixel is one byte, [0, 255], a uint8.
    buf = f.read(28 * 28)
    image = numpy.frombuffer(buf, dtype=numpy.uint8)

    # Print the first few values of image.
    print('First 10 pixels:', image[:10])


I am working on the file "Images/music/samples/libROSA/LibROSA_Demo.ipynb".
The first task is:
Beat tracking
The beat tracker returns an estimate of the tempo (in beats per minute) and frame indices of beat events.
The input can be either an audio time series (as we do below), or an onset strength envelope as calculated by librosa.onset.onset_strength().
Can you write Python code for it?

# Now, let's run the beat tracker.
# We'll use the percussive component for this part
plt.figure(figsize=(12, 6))
tempo, beats = librosa.beat.beat_track(y=y_percussive, sr=sr)

# Let's re-draw the spectrogram, but this time, overlay the detected beats
plt.figure(figsize=(12,4))
librosa.display.specshow(log_S, sr=sr, x_axis='time', y_axis='mel')

# Let's draw transparent lines over the beat frames
plt.vlines(librosa.frames_to_time(beats),
           1, 0.5 * sr,
           colors='w', linestyles='-', linewidth=2, alpha=0.5)

plt.axis('tight')

plt.colorbar(format='%+02.0f dB')

plt.tight_layout()


I am working on the file "Images/music/samples/libROSA/LibROSA_Demo.ipynb".
The first task is:
Beat-synchronous feature aggregation
Once we've located the beat events, we can use them to summarize the feature content of each beat.
This can be useful for reducing data dimensionality, and removing transient noise from the features.
Can you write Python code for it?

# feature.sync will summarize each beat event by the mean feature vector within that beat

M_sync = librosa.util.sync(M, beats)

plt.figure(figsize=(12,6))

# Let's plot the original and beat-synchronous features against each other
plt.subplot(2,1,1)
librosa.display.specshow(M)
plt.title('MFCC-$\Delta$-$\Delta^2$')

# We can also use pyplot *ticks directly
# Let's mark off the raw MFCC and the delta features
plt.yticks(np.arange(0, M.shape[0], 13), ['MFCC', '$\Delta$', '$\Delta^2$'])

plt.colorbar()

plt.subplot(2,1,2)
# librosa can generate axis ticks from arbitrary timestamps and beat events also
librosa.display.specshow(M_sync, x_axis='time',
                         x_coords=librosa.frames_to_time(librosa.util.fix_frames(beats)))

plt.yticks(np.arange(0, M_sync.shape[0], 13), ['MFCC', '$\Delta$', '$\Delta^2$'])
plt.title('Beat-synchronous MFCC-$\Delta$-$\Delta^2$')
plt.colorbar()

plt.tight_layout()

# Beat synchronization is flexible.
# Instead of computing the mean delta-MFCC within each beat, let's do beat-synchronous chroma
# We can replace the mean with any statistical aggregation function, such as min, max, or median.

C_sync = librosa.util.sync(C, beats, aggregate=np.median)

plt.figure(figsize=(12,6))

plt.subplot(2, 1, 1)
librosa.display.specshow(C, sr=sr, y_axis='chroma', vmin=0.0, vmax=1.0, x_axis='time')

plt.title('Chroma')
plt.colorbar()

plt.subplot(2, 1, 2)
librosa.display.specshow(C_sync, y_axis='chroma', vmin=0.0, vmax=1.0, x_axis='time',
                         x_coords=librosa.frames_to_time(librosa.util.fix_frames(beats)))

plt.title('Beat-synchronous Chroma (median aggregation)')

plt.colorbar()
plt.tight_layout()


I am working on the file "Images/music/samples/libROSA/LibROSA_Demo.ipynb".
The first task is:
By default, librosa will resample the signal to 22050Hz.
You can change this behavior by saying:
librosa.load(audio_path, sr=44100)
to resample at 44.1KHz, or
librosa.load(audio_path, sr=None)
to disable resampling.
Mel spectrogram
This first step will show how to compute a Mel spectrogram from an audio waveform.
Can you write Python code for it?

# Let's make and display a mel-scaled power (energy-squared) spectrogram
S = librosa.feature.melspectrogram(y, sr=sr, n_mels=128)

# Convert to log scale (dB). We'll use the peak power as reference.
log_S = librosa.logamplitude(S, ref_power=np.max)

# Make a new figure
plt.figure(figsize=(12,4))

# Display the spectrogram on a mel scale
# sample rate and hop length parameters are used to render the time axis
librosa.display.specshow(log_S, sr=sr, x_axis='time', y_axis='mel')

# Put a descriptive title on the plot
plt.title('mel power spectrogram')

# draw a color bar
plt.colorbar(format='%+02.0f dB')

# Make the figure layout compact
plt.tight_layout()


I am working on the file "Images/music/samples/libROSA/LibROSA_Demo.ipynb".
The first task is:
By default, the beat tracker will trim away any leading or trailing beats that don't appear strong enough.
To disable this behavior, call beat_track() with trim=False.
Can you write Python code for it?

print('Estimated tempo:        %.2f BPM' % tempo)

print('First 5 beat frames:   ', beats[:5])

# Frame numbers are great and all, but when do those beats occur?
print('First 5 beat times:    ', librosa.frames_to_time(beats[:5], sr=sr))

# We could also get frame numbers from times by librosa.time_to_frames()


I am working on the file "Images/music/samples/libROSA/LibROSA_Demo.ipynb".
The first task is:
Chromagram
Next, we'll extract Chroma features to represent pitch class information.
Can you write Python code for it?

# We'll use a CQT-based chromagram here.  An STFT-based implementation also exists in chroma_cqt()
# We'll use the harmonic component to avoid pollution from transients
C = librosa.feature.chroma_cqt(y=y_harmonic, sr=sr)

# Make a new figure
plt.figure(figsize=(12,4))

# Display the chromagram: the energy in each chromatic pitch class as a function of time
# To make sure that the colors span the full range of chroma values, set vmin and vmax
librosa.display.specshow(C, sr=sr, x_axis='time', y_axis='chroma', vmin=0, vmax=1)

plt.title('Chromagram')
plt.colorbar()

plt.tight_layout()


I am working on the file "Images/music/samples/libROSA/LibROSA_Demo.ipynb".
The first task is:
Harmonic-percussive source separation
Before doing any signal analysis, let's pull apart the harmonic and percussive components of the audio.  This is pretty easy to do with the effects module.
Can you write Python code for it?

y_harmonic, y_percussive = librosa.effects.hpss(y)

# What do the spectrograms look like?
# Let's make and display a mel-scaled power (energy-squared) spectrogram
S_harmonic   = librosa.feature.melspectrogram(y_harmonic, sr=sr)
S_percussive = librosa.feature.melspectrogram(y_percussive, sr=sr)

# Convert to log scale (dB). We'll use the peak power as reference.
log_Sh = librosa.logamplitude(S_harmonic, ref_power=np.max)
log_Sp = librosa.logamplitude(S_percussive, ref_power=np.max)

# Make a new figure
plt.figure(figsize=(12,6))

plt.subplot(2,1,1)
# Display the spectrogram on a mel scale
librosa.display.specshow(log_Sh, sr=sr, y_axis='mel')

# Put a descriptive title on the plot
plt.title('mel power spectrogram (Harmonic)')

# draw a color bar
plt.colorbar(format='%+02.0f dB')

plt.subplot(2,1,2)
librosa.display.specshow(log_Sp, sr=sr, x_axis='time', y_axis='mel')

# Put a descriptive title on the plot
plt.title('mel power spectrogram (Percussive)')

# draw a color bar
plt.colorbar(format='%+02.0f dB')

# Make the figure layout compact
plt.tight_layout()


I am working on the file "Images/music/samples/libROSA/LibROSA_Demo.ipynb".
The first task is:
MFCC
Mel-frequency cepstral coefficients are commonly used to represent texture or timbre of sound.
Can you write Python code for it?

# Next, we'll extract the top 13 Mel-frequency cepstral coefficients (MFCCs)
mfcc        = librosa.feature.mfcc(S=log_S, n_mfcc=13)

# Let's pad on the first and second deltas while we're at it
delta_mfcc  = librosa.feature.delta(mfcc)
delta2_mfcc = librosa.feature.delta(mfcc, order=2)

# How do they look?  We'll show each in its own subplot
plt.figure(figsize=(12, 6))

plt.subplot(3,1,1)
librosa.display.specshow(mfcc)
plt.ylabel('MFCC')
plt.colorbar()

plt.subplot(3,1,2)
librosa.display.specshow(delta_mfcc)
plt.ylabel('MFCC-$\Delta$')
plt.colorbar()

plt.subplot(3,1,3)
librosa.display.specshow(delta2_mfcc, sr=sr, x_axis='time')
plt.ylabel('MFCC-$\Delta^2$')
plt.colorbar()

plt.tight_layout()

# For future use, we'll stack these together into one matrix
M = np.vstack([mfcc, delta_mfcc, delta2_mfcc])


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:

Can you write Python code for it?

total = 0 # Takto by to bolo imperativne
for item in item_list:
    total = add(total, item)
total


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Ak predate immutable objekt funkcii, tak vam ho funkcia urcite nezmeni
String je imutable
Podobne ako vsetky zakladne typy
Can you write Python code for it?

a = 'text'
print(a)
print('Adresa je: {}'.format(id(a)))

# Znamena to, ze neviem menit hodnotu
a[0] = 'T'
print(a)
print('Adresa je: {}'.format(id(a)))


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Alebo este iny
Can you write Python code for it?

list(map(len, open('data/morho.txt')))

list(map(print, open('data/morho.txt')))


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Da sa spracovavat aj nieco ine ako cisla
Can you write Python code for it?

from functools import reduce
from operator import add

print(reduce(add, open('data/morho.txt')))


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Dalsi priklad - nasobenie prvkov zoznamu
Can you write Python code for it?

from functools import reduce
def mul(a,b):
    return a * b

reduce(mul, [1,2,3,4,5])


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Dalsi priklad pouzitia funkcie Filter
Can you write Python code for it?

fibonacci = [0,1,1,2,3,5,8,13,21,34,55]
def is_even(x):
    return x % 2 == 0

list(filter(is_even, fibonacci))


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Dalsi priklad pouzitia funkcie map
Can you write Python code for it?

def fahrenheit(T):
    return ((float(9)/5)*T + 32)

def celsius(T):
    return (float(5)/9)*(T-32)

temperatures = (36.5, 37, 37.5, 38, 39)
F = list(map(fahrenheit, temperatures))
C = list(map(celsius, F))
print(F)
print(C)


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Filter
Zo zoznamu sa vytvara novy zoznam s tymi prvkami, ktore splnaju podmienku
Can you write Python code for it?

item_list = [1,2,3,4,5,6]
def condition(x):
    return(x % 2 == 0)

collection = []
for item in item_list:
    if condition(item):
        collection.append(item)
collection


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Filter pomocou funkcie vyssej urovne
Can you write Python code for it?

item_list = [1,2,3,4,5,6]
def condition(x):
    return(x % 2 == 0)

collection = filter(condition, item_list)
list(collection)


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Funkcia map odstranuje potrebu udrzovat si stav

nepotrebujem ziadnu kolekciu, ktora je v nejakom case ciastocne naplnena
nepotrebujem ziadny index, ktory sa inkrementuje
nestaram sa o to, ako map funguje
iterativne, rekurziou, paralelne, distribuovane, pomocou indexu?
nestaram sa o vnutornu strukturu kolekcie
staci aby sa cez nu dalo iterovat
o tomto si povieme viac nabuduce

Funkcia map by mohla byt implementovana napriklad takto
Can you write Python code for it?

def my_map(f, seq): # Takto by to mohlo byt v pythone 2 a nie 3. Tam map vracia iterator.
    result = []
    for x in seq:
        result.append(f(x))
    return result


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Imutable znamena, ze hociaka operacia nad objektom vytvori novy objekt
Can you write Python code for it?

x = 'foo'
y = x
print(x) # foo
y += 'bar'
print(x) # foo
print(y)

x = [1, 2, 3]
y = x
print(x)
y += [3, 2, 1]
print(x)


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Lambda funkcia
anonymna funkcia
Can you write Python code for it?


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
List / Vektor
Can you write Python code for it?


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Map / dict
Can you write Python code for it?

m1 = ps.pmap({'a':1, 'b':2})
m1 == ps.m(a=1, b=2)

m1['a']

m1.b # toto s dict nejde

print(m1.set('a', 3))
print(m1)

print(id(m1), id(m1.set('a', 3)))


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Nemennost moze komplikovat pracu s objektami
Can you write Python code for it?


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Nie je to to iste ako klucove slovo final v Jave
Final premenna po vytvoreni nemoze smerovat na iny objekt
Objekt samtny ale moze byt zmeneny
Can you write Python code for it?

# -- JAVA --
final List<Integer> list = new ArrayList<Integer>();
list = new ArrayList<Integer>(); // toto sa neskompiluje

# -- JAVA --
final List<Integer> list = new ArrayList<Integer>();
list.add(1); //toto prejde bez problemov

# -- JAVA --
final List<Integer> list = Collections.unmodifiableList(new ArrayList<Integer>(...)); //toto je immutable list


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Preco je nemennost dobra
Netreba pocitat s tym, ze sa vam moze objekt zmenit

Je to bezpecnejsie.
vznika menej chyb
Lahsie sa debuguje

Lahsie sa testuje

staci test na jednu funkciu a nie celu skupinu objektov

Toto je dovod, preco ma Test Driven Development (TDD) taky usepch

Testy sa píšu ešte pred kódom
Zamýšľate sa ako napísať kód tak aby bol testovateľný
Bez toho aby ste o tom vedeli odstraňujete vedľajšie efekty
Nazite sa o to, aby na sebe funkcie co najmenej zavyseli
Pripravovanie objektov je pre vas zbytocnou komplikaciou
Zmena stavu objktu sposobuje, ze musite pisat velmi vela tetsov aby ste osetrili mnozstvo hranicnych stavov

Da sa lahsie zdielat medzi vlaknami a procesmi

netreba synchronizovat pristup k objektom

Da sa hashovat

ak pouzijete objekt ako kluc, tak sa urcite nezmeni a mozete
hashovacia funkcia nad nim vzdy vrati rovnaku hodnotu

Objekty mozu byt mensie. Zaberaju menej miesta v pamati a operacie nad  nimi su rychlejsie.
Ale

Je treba vytvarat velmi vela objektov.
Garbage collector sa narobi.
Can you write Python code for it?

# inspirovane https://www.youtube.com/watch?v=5qQQ3yzbKp8
employees = ['Jozo', 'Eva', 'Fero', 'Miro', 'Anna', 'Kristina']

output = '
\n'

for employee in employees:
    output += '\t
{}
\n'.format(employee)
#     print('Adresa outputu je: {}'.format(id(output)))

output += '
'

print(output)


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Redukcia
reduce(func, seq, init)
func(a, b)
Opakovane aplikuje funkciu na sekvenciu.
func prijma dva argumenty: hodnotu akumulatora a jeden prvok mnoziny
Atributom func moze byt prvok sekvencie alebo navratova hodnota inej func

Typicky priklad je suma prvkov zoznamu
Can you write Python code for it?

item_list = [47,11,42,13]
def add(a,b):
    return(a+b)

from functools import reduce

reduce(add, item_list)


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Transformacia mutable <=> immutable
Can you write Python code for it?


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Tuple je immutable
Can you write Python code for it?


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
Vela funkcii uz je predpripravenych
Can you write Python code for it?

from operator import add

from operator import mul


I am working on the file "Immutable a Higher order functions.ipynb".
The first task is:
obemdzenie na jediny riadok
nepotrebuje return

Lambda je celkom prakticka ako parameter funkcie
Can you write Python code for it?

item_list = [1,2,3,4,5]
print(list(map(lambda x: x**2, item_list)))

item_list = ["auto", "macka", "traktor"]
list(map(lambda x: x.upper(), item_list))


I am working on the file "Impala_SQL_Jupyter/Impala_Basic.ipynb".
The first task is:
2. Run a query and fetch the results
Can you write Python code for it?

cur = conn.cursor()

cur.execute('select * from test2.emp limit 2')

cur.fetchall()


I am working on the file "Impala_SQL_Jupyter/Impala_Basic.ipynb".
The first task is:
More examples of integration with IPython ecosystem
Can you write Python code for it?

cur = conn.cursor()
cur.execute('select ename, sal from test2.emp')
df = as_pandas(cur)

%matplotlib inline
import matplotlib
matplotlib.style.use('ggplot')

df.plot()


I am working on the file "Impala_SQL_Jupyter/Impala_Basic_Kerberos.ipynb".
The first task is:
2. Run a query and fetch the results
Can you write Python code for it?


I am working on the file "Impala_SQL_Jupyter/Impala_Basic_Kerberos.ipynb".
The first task is:
More examples of integration with IPython ecosystem
Can you write Python code for it?


I am working on the file "Impala_SQL_Jupyter/Impala_SQL_Magic.ipynb".
The first task is:
2. Connect to the target database (requires Cloudera impyla)
Can you write Python code for it?


I am working on the file "Impala_SQL_Jupyter/Impala_SQL_Magic.ipynb".
The first task is:
The integration with Pandas opens many additional possibilities for data analysis
Can you write Python code for it?


I am working on the file "Impala_SQL_Jupyter/Impala_SQL_Magic_Kerberos.ipynb".
The first task is:
3. Run SQL on the target using the %%sql cell magic or %sql line magic
Can you write Python code for it?


I am working on the file "Impala_SQL_Jupyter/Impala_SQL_Magic_Kerberos.ipynb".
The first task is:
Bind variables
Can you write Python code for it?

Employee_name="SCOTT"

%sql select * from emp where ename=:Employee_name


I am working on the file "Implementing PID in Python.ipynb".
The first task is:
define PID weights
PID controller minimizes error by adjusting a control variable (eg power supplied) to a new value determined by a weighted sum of present (P), past (I), and future (D) error values.
Can you write Python code for it?

P = 1.2 # weight current errors more
I = 1
D = 0.0 # ignore future potential errors

L = 50 # number of iterations

pid = PID.PID(P, I, D)

pid.SetPoint=0.0
pid.setSampleTime(0.01)

END = L
feedback = 0

feedback_list = []
time_list = []
setpoint_list = []

for i in range(1, END):
    pid.update(feedback)
    output = pid.output
    if pid.SetPoint > 0:
        feedback += (output - (1/i))
    if i>9:
        pid.SetPoint = 1
    time.sleep(0.02)

    feedback_list.append(feedback)
    setpoint_list.append(pid.SetPoint)
    time_list.append(i)

time_sm = np.array(time_list)
time_smooth = np.linspace(time_sm.min(), time_sm.max(), 300)
feedback_smooth = spline(time_list, feedback_list, time_smooth)


I am working on the file "Implementing PID in Python.ipynb".
The first task is:
how quickly does it converge?
green is desired value; blue is actual
Can you write Python code for it?

plt.plot(time_smooth, feedback_smooth)
plt.plot(time_list, setpoint_list)
plt.xlim((0, L))
plt.ylim((min(feedback_list)-0.5, max(feedback_list)+0.5))
plt.xlabel('time (s)')
plt.ylabel('PID (PV)')
plt.title('TEST PID')

plt.ylim((1-0.5, 1+0.5))


I am working on the file "Implementing a Braindrop-like architecture in Nengo.ipynb".
The first task is:
Building larger systems
Now that we can compute things, we can build larger systems by connecting together multiple Ensembles.  Here, we make three ensembles, two of which will do the same "compute the product" calculation, and a third will compare the two results and see which one is bigger.  That is, the overall system should output a 1 if a[0]*a[1] >b[0]*b[1], and otherwise output a -1.
Can you write Python code for it?

# define a helper function to make it easier to define an Ensemble
def make_brainsim_ensemble(shape, dimensions, rng, n_tap_points=4, spread=2):
    N = np.prod(shape)
    encoder = np.array([make_diffusor(shape=shape, n_tap_points=n_tap_points, spread=spread, rng=rng).flatten() for i in range(dimensions)])
    encoder_gain = np.linalg.norm(encoder, axis=0)
    encoder /= encoder_gain[None, :]

    gain = 10+rng.randn(N)
    bias = rng.randn(N)

    ens = nengo.Ensemble(n_neurons=N, dimensions=dimensions, encoders=encoder.T,
                         gain=gain, bias=bias)
    return ens

rng = np.random.RandomState(seed=3)
model = nengo.Network()
with model:
    stim_a = nengo.Node(nengo.processes.WhiteSignal(period=10, high=0.5), size_out=2)
    a = make_brainsim_ensemble((16,16), dimensions=2, rng=rng)
    nengo.Connection(stim_a, a)

    stim_b = nengo.Node(nengo.processes.WhiteSignal(period=10, high=0.5), size_out=2)
    b = make_brainsim_ensemble((16,16), dimensions=2, rng=rng)
    nengo.Connection(stim_b, b)

    compare = make_brainsim_ensemble((16,16), dimensions=1, rng=rng)
    nengo.Connection(a, compare, function=lambda x: x[0]*x[1])
    nengo.Connection(b, compare, function=lambda x: x[0]*x[1], transform=-1)

    output = nengo.Node(None, size_in=1)
    nengo.Connection(compare, output, function=lambda x: 1 if x>0 else -1)

    p_stim_a = nengo.Probe(stim_a)
    p_stim_b = nengo.Probe(stim_b)
    p_output = nengo.Probe(output)

sim = nengo.Simulator(model, seed=3)
with sim:
    sim.run(10)


I am working on the file "Implementing a Braindrop-like architecture in Nengo.ipynb".
The first task is:
Computing with neurons
We now have spiking activity that reflects the changing input.  Now let's try to compute something using that input.  For this example, we'll just compute the product between the two values.
To do this, Nengo will compute decoders.  These are connection weights that will decode the spikes into some desired output value.  With the addition of the decoders, we now have a standard single-hidden-layer network where the first set of weights are the encoders and the second set of weights are the decoders.  Importantly, we do not have a neural non-linearity at the input or output -- the only neural non-linearities are in the hidden layer.  But we can use this to approximate any function, as single-hidden-layer networks are universal function approximators.
Nengo automatically computes these weights for you when you make a Connection out of an Ensemble.  In a more detailed simulator, we might constrain these weights by discretizing them or bounding them using a custom nengo.Solver class, but we don't do that here.  Instead, we use the default where nengo will automatically feed random inputs into the network, record the resulting firing rates, and then use least-squares minimization to compute the optimal output weights.
Can you write Python code for it?

import nengo
model = nengo.Network()
with model:
    stim = nengo.Node(nengo.processes.WhiteSignal(period=10, high=0.5), size_out=D)
    ens = nengo.Ensemble(n_neurons=N, dimensions=D, encoders=encoder.T,
                         gain=gain, bias=bias)
    nengo.Connection(stim, ens)

    output = nengo.Node(None, size_in=1)
    def my_function(x):
        return x[0]*x[1]
    nengo.Connection(ens, output, function=my_function, synapse=None)

    p_stim = nengo.Probe(stim)
    p_spikes = nengo.Probe(ens.neurons)
    p_output = nengo.Probe(output)
with nengo.Simulator(model, seed=0) as sim:
    sim.run(10)

plt.figure(figsize=(12,4))
plt.plot(sim.trange(), sim.data[p_output], label='raw decoded output')
filt = nengo.synapses.Lowpass(0.01)   # for smoothing the output
plt.plot(sim.trange(), filt.filt(sim.data[p_output]), label='smoothed output')
ideal = [my_function(x) for x in sim.data[p_stim]]
plt.plot(sim.trange(), ideal, ls='--', lw=2, label='ideal output', c='k')
plt.legend()
plt.show()


I am working on the file "Implementing a Braindrop-like architecture in Nengo.ipynb".
The first task is:
Implementing Brainsim: A Braindrop-like simulation in Nengo
I've had a number of people ask about building an emulator for Braindrop, an analog neuromorphic chip I helped develop https://ieeexplore.ieee.org/document/8591981 .  Doing a "full" emulation is rather difficult, especially since you'd want to be able to adjust the level of detail on the emulation -- during development we created a variety of different emulators at different levels of abstraction, ranging from extremely low-level (nanosecond-scale simulation of individual transistors) to very high-level (just treating all the neurons as LIF and using floating-point multiplies instead of accumulators).
This notebook is an attempt at starting with the highest level of abstraction, and then adding a bit of detail.  The idea here is to show people how to use Nengo to build simulations based around different architectural assumptions, and that this might help people who are making their own chips that will probably have very different assumptions than those made in Braindrop.
Nengo as a generic neural simulator
The simplest answer for how to use Nengo to implement your particular neuromorphic architecture is to just use Nengo as a normal neural simulator.  You can define your own neuron class (see https://www.nengo.ai/nengo/examples/usage/rectified_linear.html for an example) and you can connect neurons however you want with the Connection class.
However, this is a somewhat unsatisfying answer.  Nengo also has a lot of built-in tools to help build up larger neural systems using separate sub-networks, each of which is optimized locally.  (These tools are known as the Neural Engineering Framework, or NEF). It would be nice to be able to leverage these tools to help build up larger systems.   Furthermore, Braindrop was specifically designed to be suitable for this sort of constructive approach to building systems out of neurons, so it seems like we should be able to make use of that when simulating Braindrop or any other similar system.
Diffusors and...
Can you write Python code for it?

def make_diffusor(shape, n_tap_points, spread, rng):
    mesh = np.zeros(shape)
    np.add.at(mesh, (rng.randint(shape[0], size=n_tap_points), rng.randint(shape[1], size=n_tap_points)), 1)
    np.add.at(mesh, (rng.randint(shape[0], size=n_tap_points), rng.randint(shape[1], size=n_tap_points)), -1)
    mesh = scipy.ndimage.gaussian_filter(mesh, sigma=spread)
    return mesh

rng = np.random.RandomState(seed=0)
plt.imshow(make_diffusor(shape=(16,16), n_tap_points=4, spread=2, rng=rng))
plt.colorbar()
plt.show()


I am working on the file "Implementing a Braindrop-like architecture in Nengo.ipynb".
The first task is:
You can interact with this model using the Nengo GUI and see the neural activity.  Right-click on the ensemble of neurons and choose 'Firing pattern' to see the neural activity.  Right-click on the stimulus and choose 'Sliders' to let you manually control the inputs.  Press the Play button in the bottom-right to run the simulation.
Can you write Python code for it?

import nengo_gui.jupyter
nengo_gui.jupyter.InlineGUI(model, cfg='model.cfg')


I am working on the file "Implicit Georeferencing.ipynb".
The first task is:
Spatial extent name-geometry lookup
The fully qualified names and GeoJSON geometries of relevant spatial areas are contained in our custom dataschema.
Can you write Python code for it?

# Getting the extent dictionary e
url = "https://raw.githubusercontent.com/datawagovau/ckanext-datawagovautheme/dpaw-internal/ckanext/datawagovautheme/datawagovau_dataset.json"
ds = json.loads(requests.get(url).content)
choice_dict = [x for x in ds["dataset_fields"] if x["field_name"] == "spatial"][0]["choices"]
e = dict([(x["label"], json.dumps(x["value"])) for x in choice_dict])
print("Extents: {0}".format(e.keys()))


I am working on the file "Import from Cloudant Python example.ipynb".
The first task is:
3. Work with a Dataframe
At this point, all transformations and functions should behave as specified with Spark SQL. (http://spark.apache.org/sql/)
Can you write Python code for it?

# This code prints the schema and a record count
cloudantdata.printSchema()
cloudantdata.count()

# This code displays the values of the naturecode field
cloudantdata.select("properties.naturecode").show()

# This code filters the data to just those records with a naturecode of "DISTRB", and then displays that data
disturbDF = cloudantdata.filter("properties.naturecode = 'DISTRB'")
disturbDF.show()

# This code writes the filtered data to a Cloudant database called crimes_filtered. If the Cloudant database exists, the documents will be added to the database.
# If the database does not exist, set the createDBOnSave option to 'true'.
disturbDF.select("properties").write.format("org.apache.bahir.cloudant")\
     .option("cloudant.host","xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix.cloudant.com")\
     .option("cloudant.username", "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix")\
     .option("cloudant.password","xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")\
     .option("createDBOnSave", "true")\
     .save("crimes_filtered")

# Next, you'll see how to create a visualization of the crimes data.
# First, this line creates a DataFrame containing all of the naturecodes and a count of the crime incidents for each code.
reducedValue = cloudantdata.groupBy("properties.naturecode").count()
reducedValue.printSchema()


I am working on the file "Import from Cloudant Python example.ipynb".
The first task is:
4. Generate visualizations
Can you write Python code for it?

# This line imports two Python modules. The pprint module helps to produce pretty representations of data structures,
# and the counter subclass from the collections module helps to count hashable objects.
import pprint
from collections import Counter

# This line imports PySpark classes for Spark SQL and DataFrames.
from pyspark.sql import *
from pyspark.sql.functions import udf, asc, desc
from pyspark import SparkContext, SparkConf
from pyspark.sql.types import IntegerType

# This line converts an Apache Spark DataFrame to a Panda DataFrame, and also sorts the DataFrame by count first,
# and then by naturecode second in order to produce a sorted graph later.
import pandas as pd
pandaDF = reducedValue.orderBy(desc("count"), asc("naturecode")).toPandas()
print(pandaDF)

# This is needed to actually see the plots
%matplotlib inline

# This line imports matplotlib.pyplot which is a collection of command style functions that make matplotlib work like MATLAB
import matplotlib.pyplot as plt

# These lines assign the data to the values and labels objects.
values = pandaDF['count']
labels = pandaDF['naturecode']

# These lines provide the format for the plot.
plt.gcf().set_size_inches(16, 12, forward=True)
plt.title('Number of crimes by type')

# These lines specify that the plot should display as a horizontal bar chart with values being for the x axis
# and labels for the y axis
plt.barh(range(len(values)), values)
plt.yticks(range(len(values)), labels)

# This last line displays the plot
plt.show()


I am working on the file "ImportAppleHealthXML.ipynb".
The first task is:
Authenticate with Google
This will open a browser to let you beging the process of authentication with an existing Google Drive account. This process will be separate from Python. For this to work, you will need to set up a Other Authentication OAuth credential at https://console.developers.google.com/apis/credentials, save the secret file in your root directory and a few other things that are detailed at https://pythonhosted.org/PyDrive/. The PyDrive instructions also show you how to set up your Google application. There are other methods for accessing the Google API from python, but this one seems pretty nice.
The first time through the process, regular sign in and two factor authentication is required (if you require two factor auth) but after that it is just a process of telling Google that it is ok for your Google application to access Drive.
Can you write Python code for it?

# Authenticate into Google Drive
from pydrive.auth import GoogleAuth

gauth = GoogleAuth()
gauth.LocalWebserverAuth()


I am working on the file "ImportAppleHealthXML.ipynb".
The first task is:
Boxplots Using Seaborn
Can you write Python code for it?

import seaborn as sns
sns.set(style="ticks", palette="muted", color_codes=True)

sns.boxplot(x="Month", y="HeartRate", data=HR_df,whis=np.inf, color="c")
# Add in points to show each observation
snsplot = sns.stripplot(x="Month", y="HeartRate", data=HR_df,jitter=True, size=1, alpha=.15, color=".3", linewidth=0)

hr_only = HR_df[['Create','HeartRate']]
hr_only.tail()

hr_only.to_csv('~/Downloads/stc_hr.csv')


I am working on the file "ImportAppleHealthXML.ipynb".
The first task is:
Download the most recent Apple Health export file
Now that we are authenticated into Google Drive, use PyDrive to access the API and get to files stored.
Google Drive allows multiple files with the same name, but it indexes them with the ID to keep them separate.
In this block, we make one pass of the file list where the file name is called export.zip, and save the row that corresponds with the most recent date. We will use that  file id later to download the correct file that corresponds  with the most recent date. Apple Health export names the file export.zip, and at the time this was written, there is no other option.
Can you write Python code for it?

from pydrive.drive import GoogleDrive
drive = GoogleDrive(gauth)

file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()

# Step through the file list and find the most current export.zip file id, then use
#      that later to download the file to the local machine.
# This may look a little old school, but these file lists will never be massive and
#     it is readable and easy one pass way to get the most current file using the
#     least (or low) amount of resouces
selection_dt = datetime.strptime("2000-01-01T01:01:01.001Z","%Y-%m-%dT%H:%M:%S.%fZ")
print("Matching Files")
for file1 in file_list:
    if re.search("^export-*\d*.zip",file1['title']):
        dt = datetime.strptime(file1['createdDate'],"%Y-%m-%dT%H:%M:%S.%fZ")
        if dt > selection_dt:
            selection_id = file1['id']
            selection_dt = dt
        print('    title: %s, id: %s createDate: %s' % (file1['title'], file1['id'], file1['createdDate']))

if not os.path.exists('healthextract'):
    os.mkdir('healthextract')


I am working on the file "ImportAppleHealthXML.ipynb".
The first task is:
List XML headers by element count
Can you write Python code for it?

pd.Series([el.tag for el in e.iter()]).value_counts()


I am working on the file "ImportAppleHealthXML.ipynb".
The first task is:
Parse Apple Health Export document
Can you write Python code for it?

path = "healthextract/apple_health_export/export.xml"
e = et.parse(path)
#this was from an older iPhone, to demonstrate how to join files
legacy = et.parse("healthextract/apple_health_legacy/export.xml")

#<<TODO: Automate this process

#legacyFilePath = "healthextract/apple_health_legacy/export.xml"
#if os.path.exists(legacyFilePath):
#    legacy = et.parse("healthextract/apple_health_legacy/export.xml")
#else:
#    os.mkdir('healthextract/apple_health_legacy')


I am working on the file "ImportAppleHealthXML.ipynb".
The first task is:
Unzip the most current file to a holding directory
Can you write Python code for it?

zip_ref = zipfile.ZipFile('healthextract/export.zip', 'r')
zip_ref.extractall('healthextract')
zip_ref.close()


I am working on the file "ImportAppleHealthXML.ipynb".
The first task is:
import calmap
ts = pd.Series(HR_df['HeartRate'].values, index=HR_df['Days'])
ts.index = pd.to_datetime(ts.index)
tstot = ts.groupby(ts.index).median()
plt.rcParams['figure.figsize'] = 16, 8
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
calmap.yearplot(data=tstot,year=2017)
Flag Chemotherapy Days for specific analysis
The next two cells provide the ability to introduce cycles that start on specific days and include this data in the datasets so that they can be overlaid in graphics. In the example below, there are three cycles of 21 days. The getDelta function returns the cycle number when tpp == 0 and the days since day 0 when tpp == 2. This allows the overlaying of the cycles, with the days since day 0 being overlaid.
Can you write Python code for it?

# This isnt efficient yet, just a first swipe. It functions as intended.
def getDelta(res,ttp,cyclelength):
    mz = [x if (x >= 0) & (x < cyclelength) else 999 for x in res]
    if ttp == 0:
        return(mz.index(min(mz))+1)
    else:
        return(mz[mz.index(min(mz))])

#chemodays = np.array([date(2017,4,24),date(2017,5,16),date(2017,6,6),date(2017,8,14)])
chemodays = np.array([date(2018,1,26),date(2018,2,2),date(2018,2,9),date(2018,2,16),date(2018,2,26),date(2018,3,2),date(2018,3,19),date(2018,4,9),date(2018,5,1),date(2018,5,14),date(2018,6,18),date(2018,7,10),date(2018,8,6)])

HR_df = xmltodf(e,"HKQuantityTypeIdentifierHeartRate","HeartRate")
#I dont think this is efficient yet...
a = HR_df['Create'].apply(lambda x: [x.days for x in x.date()-chemodays])
HR_df['ChemoCycle'] = a.apply(lambda x: getDelta(x,0,21))
HR_df['ChemoDays'] = a.apply(lambda x: getDelta(x,1,21))

import seaborn as sns
plotx = HR_df[HR_df['ChemoDays']<=21]
plt.rcParams['figure.figsize'] = 24, 8
ax = sns.boxplot(x="ChemoDays", y="HeartRate", hue="ChemoCycle", data=plotx, palette="Set2",notch=1,whis=0,width=0.75,showfliers=False)
plt.ylim(65,130)
#the next statement puts the chemodays variable as a rowname, we need to fix that
plotx_med = plotx.groupby('ChemoDays').median()
#this puts chemodays back as a column in the frame. I need to see if there is a way to prevent the effect
plotx_med.index.name = 'ChemoDays'
plotx_med.reset_index(inplace=True)

snsplot = sns.pointplot(x='ChemoDays', y="HeartRate", data=plotx_med,color='Gray')


I am working on the file "Importing Modules.ipynb".
The first task is:
Alternatively, we can chose to import all symbols (functions and variables) in a module to the current namespace (so that we don't need to use the prefix "math." every time we use something from the math module:
Can you write Python code for it?

from math import cos, pi

x = cos(2 * pi)

print(x)


I am working on the file "Importing Modules.ipynb".
The first task is:
And using the function help we can get a description of each function (almost .. not all functions have docstrings, as they are technically called, but the vast majority of functions are documented this way).
Can you write Python code for it?


I am working on the file "Importing Modules.ipynb".
The first task is:
Looking at what a module contains, and its documentation
Once a module is imported, we can list the symbols it provides using the dir function:
Can you write Python code for it?


I am working on the file "Importing Modules.ipynb".
The first task is:
This includes the whole module and makes it available for use later in the program.
For example, we can do:
Can you write Python code for it?

import math

x = math.cos(2 * math.pi)

print(x)


I am working on the file "Importing Modules.ipynb".
The first task is:
This is called selective Import
This pattern can be very convenient, but in large programs that include many modules it is often a good idea to keep the symbols from each module in their own namespaces, by using the import math pattern. This would elminate potentially confusing problems with name space collisions.
Btw, in case of namespace collisions (or to avoid namespace pollution) we may use the as keyword
Can you write Python code for it?

from math import cos as cosine  # Now the `cos` function can be referenced as `cosine`

cosine(pi/2)


I am working on the file "Importing Notebooks.ipynb".
The first task is:
After this point, my notebooks should be importable.
Let's look at what we have in the CWD:
Can you write Python code for it?


I am working on the file "Importing Notebooks.ipynb".
The first task is:
Aside: displaying notebooks
Here is some simple code to display the contents of a notebook
with syntax highlighting, etc.
Can you write Python code for it?

from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

from IPython.display import display, HTML

formatter = HtmlFormatter()
lexer = PythonLexer()

# publish the CSS for pygments highlighting
display(HTML("""

""" % formatter.get_style_defs()
))

def show_notebook(fname):
    """display a short summary of the cells of a notebook"""
    with io.open(fname, 'r', encoding='utf-8') as f:
        nb = read(f, 4)
    html = []
    for cell in nb.cells:
        html.append("
%s cell
" % cell.cell_type)
        if cell.cell_type == 'code':
            html.append(highlight(cell.source, lexer, formatter))
        else:
            html.append("
%s
" % cell.source)
    display(HTML('\n'.join(html)))

show_notebook(os.path.join("nbpackage", "mynotebook.ipynb"))


I am working on the file "Importing Notebooks.ipynb".
The first task is:
Hooray again!
Even the function that contains IPython syntax works:
Can you write Python code for it?


I am working on the file "Importing Notebooks.ipynb".
The first task is:
Note that the __init__.py is necessary for nb to be considered a package,
just like usual.
Can you write Python code for it?

show_notebook(os.path.join("nbpackage", "nbs", "other.ipynb"))

from nbpackage.nbs import other
other.bar(5)


I am working on the file "Importing Notebooks.ipynb".
The first task is:
Notebook Loader
Here we have our Notebook Loader.
It's actually quite simple - once we figure out the filename of the module,
all it does is:

load the notebook document into memory
create an empty Module
execute every cell in the Module namespace

Since IPython cells can have extended syntax,
the IPython transform is applied to turn each of these cells into their pure-Python counterparts before executing them.
If all of your notebook cells are pure-Python,
this step is unnecessary.
Can you write Python code for it?

class NotebookLoader(object):
    """Module Loader for Jupyter Notebooks"""
    def __init__(self, path=None):
        self.shell = InteractiveShell.instance()
        self.path = path

    def load_module(self, fullname):
        """import a notebook as a module"""
        path = find_notebook(fullname, self.path)

        print ("importing Jupyter notebook from %s" % path)

        # load the notebook object
        with io.open(path, 'r', encoding='utf-8') as f:
            nb = read(f, 4)

        # create the module and add it to sys.modules
        # if name in sys.modules:
        #    return sys.modules[name]
        mod = types.ModuleType(fullname)
        mod.__file__ = path
        mod.__loader__ = self
        mod.__dict__['get_ipython'] = get_ipython
        sys.modules[fullname] = mod

        # extra work to ensure that magics that would affect the user_ns
        # actually affect the notebook module's ns
        save_user_ns = self.shell.user_ns
        self.shell.user_ns = mod.__dict__

        try:
          for cell in nb.cells:
            if cell.cell_type == 'code':
                # transform the input to executable Python
                code = self.shell.input_transformer_manager.transform_cell(cell.source)
                # run the code in themodule
                exec(code, mod.__dict__)
        finally:
            self.shell.user_ns = save_user_ns
        return mod


I am working on the file "Importing Notebooks.ipynb".
The first task is:
Notebooks in packages
We also have a notebook inside the nb package,
so let's make sure that works as well.
Can you write Python code for it?


I am working on the file "Importing Notebooks.ipynb".
The first task is:
Register the hook
Now we register the NotebookFinder with sys.meta_path
Can you write Python code for it?

sys.meta_path.append(NotebookFinder())

ls

from linear_algebra import sum_of_squares, dot


I am working on the file "Importing Notebooks.ipynb".
The first task is:
So I should be able to import nbpackage.mynotebook.
Can you write Python code for it?


I am working on the file "Importing Notebooks.ipynb".
The first task is:
and import the notebook from IPython.utils
Can you write Python code for it?

from IPython.utils import inside_ipython
inside_ipython.whatsmyname()


I am working on the file "Importing the large data sets to psql and computing their metrics.ipynb".
The first task is:
Importing the data in psql
To import the data in psql we create a table with the appropriate shape and import form the csv files generated above.
Some preparation to run psql transactions and queries in python
Can you write Python code for it?

import psycopg2 as pg
import pandas as pd

db_conf = {
    'user': 'mariosk',
    'database': 'amazon_reviews'
}

connection_factory = lambda: pg.connect(user=db_conf['user'], database=db_conf['database'])

def transaction(*statements):
    try:
        connection = connection_factory()
        cursor = connection.cursor()
        for statement in statements:
            cursor.execute(statement)
        connection.commit()
        cursor.close()
    except pg.DatabaseError as error:
        print(error)
    finally:
        if connection is not None:
            connection.close()

def query(statement):
    try:
        connection = connection_factory()
        cursor = connection.cursor()
        cursor.execute(statement)

        header = [ description[0] for description in cursor.description ]
        rows = cursor.fetchall()

        cursor.close()
        return pd.DataFrame.from_records(rows, columns=header)
    except (Exception, pg.DatabaseError) as error:
        print(error)
        return None
    finally:
        if connection is not None:
            connection.close()


I am working on the file "Importing the large data sets to psql and computing their metrics.ipynb".
The first task is:
Querying the metrics
Can you write Python code for it?

def average_reviews_per_product(table_name):
    return (query('''
        with distinct_products as (select count(distinct asin) as products from {0}),
             reviews_count as (select cast(count(*) as double precision) as reviews from {0})
        select reviews / products as reviews_per_product
        from distinct_products cross join reviews_count
    '''.format(table_name))
    .rename(index={0: table_name.replace('_', ' ')}))

def average_reviews_per_reviewer(table_name):
    return (query('''
        with distinct_reviewers as (select count(distinct reviewerID) as reviewers from {0}),
             reviews_count as (select cast(count(*) as double precision) as reviews from {0})
        select reviews / reviewers as reviews_per_reviewer
        from distinct_reviewers cross join reviews_count
    '''.format(table_name))
    .rename(index={ 0: table_name.replace('_', ' ')}))

def percentages_per_rating(table_name):
    return (query('''
            with rating_counts as (select overall, count(overall) as rating_count from {0} group by overall),
                 reviews_count as (select cast(count(*) as double precision) as reviews from {0})
            select cast(overall as int) as dataset_name, rating_count / reviews as row
            from rating_counts cross join reviews_count
        '''.format(table_name))
        .set_index('dataset_name')
        .sort_index()
        .transpose()
        .rename(index={'row': table_name.replace('_', ' ')}))

def number_of_reviews(table_name):
    return (query('''
        select count(*) as number_of_reviews from {0}
    '''.format(table_name))
    .rename(index={ 0: table_name.replace('_', ' ') }))

def all_metrics(table_name):
    print(table_name)

    return pd.concat(
        [ f(table_name)
            for f
            in [ percentages_per_rating, number_of_reviews, average_reviews_per_product, average_reviews_per_reviewer ]],
        axis=1)

metrics = stopwatch(lambda: pd.concat([ all_metrics(table) for table in table_names...


I am working on the file "Importing the large data sets to psql and computing their metrics.ipynb".
The first task is:
Unzipping the data and converting it to csv format
Unfortunately psql does not support an import of record json files therefore we need to convert the data sets to csv. We use here the command line tool json2csv.
WARNING: The following two commands will run for a while, especially the second one. You can expect approximately 1 minute per GB of unzipped data.
Can you write Python code for it?

start_time = timeit.default_timer()

!ls ./data/large-datasets/*.gz | grep -Po '.*(?=.gz)' | xargs -I {} gunzip {}.gz

print('Elapsed time: %i sec' % int(timeit.default_timer() - start_time))

start_time = timeit.default_timer()

!ls ./data/large-datasets/*.json | xargs sed -i 's/|/?/g;s/\u0000/?/g'

print('Elapsed time: %i sec' % int(timeit.default_timer() - start_time))

start_time = timeit.default_timer()

!ls ./data/large-datasets/*.json | grep -Po '.*(?=.json)' | xargs -I {} json2csv -p -d '|' -k asin,helpful,overall,reviewText,reviewTime,reviewerID,reviewerName,summary,unixReviewTime -i {}.json -o {}.csv
!rm ./data/large-datasets/*.json

print('Elapsed time: %i sec' % int(timeit.default_timer() - start_time))


I am working on the file "Improving Deep Neural Networks/Gradient+Checking.ipynb".
The first task is:
Expected Output:

          ** J **
         8

Exercise: Now, implement the backward propagation step (derivative computation) of Figure 1. That is, compute the derivative of $J(\theta) = \theta x$ with respect to $\theta$. To save you from doing the calculus, you should get $dtheta = \frac { \partial J }{ \partial \theta} = x$.
Can you write Python code for it?

# GRADED FUNCTION: backward_propagation

def backward_propagation(x, theta):
    """
    Computes the derivative of J with respect to theta (see Figure 1).

    Arguments:
    x -- a real-valued input
    theta -- our parameter, a real number as well

    Returns:
    dtheta -- the gradient of the cost with respect to theta
    """

    ### START CODE HERE ### (approx. 1 line)
    dtheta = x
    ### END CODE HERE ###

    return dtheta

x, theta = 2, 4
dtheta = backward_propagation(x, theta)
print ("dtheta = " + str(dtheta))


I am working on the file "Improving Deep Neural Networks/Gradient+Checking.ipynb".
The first task is:
Expected Output:

          ** dtheta **
         2

Exercise: To show that the backward_propagation() function is correctly computing the gradient $\frac{\partial J}{\partial \theta}$, let's implement gradient checking.
Instructions:
- First compute "gradapprox" using the formula above (1) and a small value of $\varepsilon$. Here are the Steps to follow:
    1. $\theta^{+} = \theta + \varepsilon$
    2. $\theta^{-} = \theta - \varepsilon$
    3. $J^{+} = J(\theta^{+})$
    4. $J^{-} = J(\theta^{-})$
    5. $gradapprox = \frac{J^{+} - J^{-}}{2  \varepsilon}$
- Then compute the gradient using backward propagation, and store the result in a variable "grad"
- Finally, compute the relative difference between "gradapprox" and the "grad" using the following formula:
$$ difference = \frac {\mid\mid grad - gradapprox \mid\mid_2}{\mid\mid grad \mid\mid_2 + \mid\mid gradapprox \mid\mid_2} \tag{2}$$
You will need 3 Steps to compute this formula:
   - 1'. compute the numerator using np.linalg.norm(...)
   - 2'. compute the denominator. You will need to call np.linalg.norm(...) twice.
   - 3'. divide them.
- If this difference is small (say less than $10^{-7}$), you can be quite confident that you have computed your gradient correctly. Otherwise, there may be a mistake in the gradient computation.
Can you write Python code for it?

# GRADED FUNCTION: gradient_check

def gradient_check(x, theta, epsilon = 1e-7):
    """
    Implement the backward propagation presented in Figure 1.

    Arguments:
    x -- a real-valued input
    theta -- our parameter, a real number as well
    epsilon -- tiny shift to the input to compute approximated gradient with formula(1)

    Returns:
    difference -- difference (2) between the approximated gradient and the backward propagation gradient
    """

    # Compute gradapprox using left side of formula (1). epsilon is small enough, you don't need to worry about the limit.
    ### START CODE HERE ### (approx. 5 lines)
    thetaplus = theta+epsilon                               # Step 1
    thetaminus = theta-epsilon                      # Step 2
    J_plus = forward_propagation(x,thetaplus)                                  # Step 3
    J_minus = forward_propagation(x,thetaminus)                                 # Step 4
    gradapprox = (J_plus-J_minus)/(2*epsilon  )                            # Step 5
    ### END CODE HERE ###

    # Check if gradapprox is close enough to the output of backward_propagation()
    ### START CODE HERE ### (approx. 1 line)
    grad = backward_propagation(x, theta)
    ### END CODE HERE ###

    ### START CODE HERE ### (approx. 1 line)
    numerator = np.linalg.norm(gradapprox-grad)                               # Step 1'
    denominator =np.linalg.norm(gradapprox)+np.linalg.norm(grad)                             # Step 2'
    difference = numerator/denominator                              # Step 3'
    ### END CODE HERE ###

    if difference < 1e-7:
        print ("The gradient is correct!")
    else:
        print ("The gradient is wrong!")

    return difference

x, theta = 2, 4
difference = gradient_check(x, theta)
print("difference = " + str(difference))


I am working on the file "Improving Deep Neural Networks/Gradient+Checking.ipynb".
The first task is:
Expected Output:
The gradient is correct!

          ** difference **
         2.9193358103083e-10

Congrats, the difference is smaller than the $10^{-7}$ threshold. So you can have high confidence that you've correctly computed the gradient in backward_propagation().
Now, in the more general case, your cost function $J$ has more than a single 1D input. When you are training a neural network, $\theta$ actually consists of multiple matrices $W^{[l]}$ and biases $b^{[l]}$! It is important to know how to do a gradient check with higher-dimensional inputs. Let's do it!
3) N-dimensional gradient checking
The following figure describes the forward and backward propagation of your fraud detection model.

  Figure 2 : deep neural network
LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID

Let's look at your implementations for forward propagation and backward propagation.
Can you write Python code for it?

def forward_propagation_n(X, Y, parameters):
    """
    Implements the forward propagation (and computes the cost) presented in Figure 3.

    Arguments:
    X -- training set for m examples
    Y -- labels for m examples
    parameters -- python dictionary containing your parameters "W1", "b1", "W2", "b2", "W3", "b3":
                    W1 -- weight matrix of shape (5, 4)
                    b1 -- bias vector of shape (5, 1)
                    W2 -- weight matrix of shape (3, 5)
                    b2 -- bias vector of shape (3, 1)
                    W3 -- weight matrix of shape (1, 3)
                    b3 -- bias vector of shape (1, 1)

    Returns:
    cost -- the cost function (logistic cost for one example)
    """

    # retrieve parameters
    m = X.shape[1]
    W1 = parameters["W1"]
    b1 = parameters["b1"]
    W2 = parameters["W2"]
    b2 = parameters["b2"]
    W3 = parameters["W3"]
    b3 = parameters["b3"]

    # LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID
    Z1 = np.dot(W1, X) + b1
    A1 = relu(Z1)
    Z2 = np.dot(W2, A1) + b2
    A2 = relu(Z2)
    Z3 = np.dot(W3, A2) + b3
    A3 = sigmoid(Z3)

    # Cost
    logprobs = np.multiply(-np.log(A3),Y) + np.multiply(-np.log(1 - A3), 1 - Y)
    cost = 1./m * np.sum(logprobs)

    cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)

    return cost, cache


I am working on the file "Improving Deep Neural Networks/Gradient+Checking.ipynb".
The first task is:
Now, run backward propagation.
Can you write Python code for it?

def backward_propagation_n(X, Y, cache):
    """
    Implement the backward propagation presented in figure 2.

    Arguments:
    X -- input datapoint, of shape (input size, 1)
    Y -- true "label"
    cache -- cache output from forward_propagation_n()

    Returns:
    gradients -- A dictionary with the gradients of the cost with respect to each parameter, activation and pre-activation variables.
    """

    m = X.shape[1]
    (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache

    dZ3 = A3 - Y
    dW3 = 1./m * np.dot(dZ3, A2.T)
    db3 = 1./m * np.sum(dZ3, axis=1, keepdims = True)

    dA2 = np.dot(W3.T, dZ3)
    dZ2 = np.multiply(dA2, np.int64(A2 > 0))
    dW2 = 1./m * np.dot(dZ2, A1.T) * 2
    db2 = 1./m * np.sum(dZ2, axis=1, keepdims = True)

    dA1 = np.dot(W2.T, dZ2)
    dZ1 = np.multiply(dA1, np.int64(A1 > 0))
    dW1 = 1./m * np.dot(dZ1, X.T)
    db1 = 4./m * np.sum(dZ1, axis=1, keepdims = True)

    gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3,
                 "dA2": dA2, "dZ2": dZ2, "dW2": dW2, "db2": db2,
                 "dA1": dA1, "dZ1": dZ1, "dW1": dW1, "db1": db1}

    return gradients