I am working on the file "Lesson14-Sentiment Analysis/sentiment-network/Sentiment_Classification_Projects.ipynb".
The first task is:
With a learning rate of 0.001, the network should finall have started to improve during training. It's still not very good, but it shows that this solution has potential. We will improve it in the next lesson.
End of Project 3.
Watch the next video to see Andrew's solution, then continue on to the next lesson.
Understanding Neural Noise
The following cells include includes the code Andrew shows in the next video. We've included it here so you can run the cells along with the video without having to type in everything.
Can you write Python code for it?

from IPython.display import Image
Image(filename='sentiment_network.png')

def update_input_layer(review):

    global layer_0

    # clear out previous state, reset the layer to be all 0s
    layer_0 *= 0
    for word in review.lower().replace(",", " ").replace(".", " ").split(" "):
        layer_0[0][word2index[word]] += 1

update_input_layer(reviews[0])

layer_0

review_counter = Counter()

for word in reviews[0].lower().replace(",", " ").replace(".", " ").split(" "):
    review_counter[word] += 1

review_counter.most_common()


I am working on the file "Lesson14-Sentiment Analysis/sentiment-network/Sentiment_Classification_Projects.ipynb".
The first task is:
layer_0 contains one entry for every word in the vocabulary, as shown in the above image. We need to make sure we know the index of each word, so run the following cell to create a lookup table that stores the index of every word.
Can you write Python code for it?

# Create a dictionary of words in the vocabulary mapped to index positions
# (to be used in layer_0)
word2index = {}
for i,word in enumerate(vocab):
    word2index[word] = i

# display the map of words to indices
word2index


I am working on the file "Lesson14/Twitter assignment.ipynb".
The first task is:
Bounding box to WKT
Calculates the centroid of a bounding box and returns a Well Known Text of that point. Only polygonal bounding boxes are supported (but are there any other kind?)
Can you write Python code for it?

def bbox_to_wkt(bbox):
    if bbox.coordinates == None:
        return ""
    if bbox.type == "Polygon":
        centroid = [0, 0]
        centroid[0] = (bbox.coordinates[0][2][0] + bbox.coordinates[0][0][0]) / 2
        centroid[1] = (bbox.coordinates[0][2][1] + bbox.coordinates[0][0][1]) / 2
        return "POINT("+str(centroid[0])+" "+str(centroid[1])+")"
    print "Unknown place type!"
    return ""


I am working on the file "Lesson14/Twitter assignment.ipynb".
The first task is:
Close the database
Can you write Python code for it?


I am working on the file "Lesson14/Twitter assignment.ipynb".
The first task is:
Coordinates to WKT
Converts Twitter coordinates (two points) into a Well Known Text.
Can you write Python code for it?

def coordinates_to_wkt(coords):
    if coords == None:
        return ""
    return "POINT("+str(coords["coordinates"][0])+" "+str(coords["coordinates"][1])+")"


I am working on the file "Lesson14/Twitter assignment.ipynb".
The first task is:
Process query: main function of the script
Does the query parsing and output to SpatiaLite. Pass the result of api.search() to it.
Can you write Python code for it?

def process_query(search_results):
    for tweet in search_results:
        full_place_name = ""
        place_type = ""
        location = ""
        username =  tweet.user.screen_name
        followers_count =  tweet.user.followers_count
        tweettext = tweet.text.encode("utf-8")
        if tweet.place != None:
            full_place_name = tweet.place.full_name
            place_type =  tweet.place.place_type
        coordinates = tweet.coordinates
        if (coordinates != None) or (tweet.place != None):
            print 'Found a geolocated tweet! By:'
            print username
            print '==========================='
            if coordinates != None:
                location = coordinates_to_wkt(coordinates)
            else:
                if tweet.place != None:
                    location = bbox_to_wkt(tweet.place.bounding_box)
            curs.execute("insert into tweets (username, followers_count, tweettext, full_place_name, place_type, coordinates, geometry) values (?, ?, ?, ?, ?, ?, ST_GeomFromText( ? , 4326));", \
                (username, followers_count, tweettext.decode('utf-8'), full_place_name, place_type, location, location))
            conn.commit()


I am working on the file "Lesson14/Twitter assignment.ipynb".
The first task is:
SQLite opening
Make sure you load pysqlite2, and give a path for mod_spatialite for SpatiaLite support! This may be distribution-specific! You might need to install libspatialite!
Can you write Python code for it?

conn = sqlite3.connect(databasefile)
conn.enable_load_extension(True)
conn.execute('SELECT load_extension("/usr/lib64/mod_spatialite.so.7")')
curs = conn.cursor()


I am working on the file "Lesson14/Twitter assignment.ipynb".
The first task is:
Visualise data
Opens QGIS with the database passed as an argument. It should show you all the points. Add a layer of OpenStreetMap or such for a nice visualisation of the points.
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
2. Creating Figures
There are two major challenges with creating figures. First is understanding the
syntax to actually make the basic plot appear. Second is formatting the basic plot to look
exactly how you would like it to look. In general, the formatting will probably take you
longer...
Within pyplot (currently imported as 'plt'), there are two basic ways to go about making
plots - using the Matlab-like clone, and using the object-oriented approach. The latter
provides better control over plot features, while only requiring slightly more typing. It's
easy to quickly outgrow the Matlab clone, so we'll go right to the object-oriented syntax.
2.1 A first plot
In simple matplotlib plotting, there are two concepts to distinguish:

Figure - the entire figure, like what you might see in a journal, including all
subplots, axes, lines, labels, etc. The whole enchilada.

Subplot/Axes - one of the sub-sections of the figure, labeled (a), (b), etc. in
articles. Each subplot will contain one Axes object, which is the container where all of the
useful stuff, such as actual lines, legends, labels, etc., are actually housed.

For example, here's how to make one figure with two subplots, the second of which contains
two lines.
Can you write Python code for it?

# First we make some data to plot
x = np.linspace(-2*np.pi, 2*np.pi)
y1 = np.sin(x)
y2 = np.cos(x)


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
3.2 Tick marks
You can set the values where the ticks are located using the xticks and yticks methods.
They take a list of values
plt.xticks([val1, val2, val3])
plt.yticks([yval1, yval2, yval3])
Can you write Python code for it?

x = np.linspace(-2*np.pi, 2*np.pi)
y1 = np.sin(x)
y2 = np.cos(x)

fig, axes = plt.subplots(1,2)
axes[0].plot(x, y1, color='r', linewidth=5)
axes[1].plot(x, y1, color='#ffaa00', linewidth=0.5, linestyle='--')
axes[1].plot(x, y2, color='green', linestyle='-.')
plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi])
plt.yticks([-1, 0, 1]);


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
3.4 Setting tick labels
To set the tick labels, you pass a second parameter to the xticks and yticks methods with a list of labels for that axis.
plt.sca(axis)
plt.xticks([tickvalues], [ticklabels])
plt.yticks([tickvalues], [ticklabels])
Can you write Python code for it?

x = np.linspace(-2*np.pi, 2*np.pi)
y1 = np.sin(x)
y2 = np.cos(x)

fig, axes = plt.subplots(1,2)
axes[0].plot(x, y1, color='r', linewidth=5)
axes[1].plot(x, y1, color='#ffaa00', linewidth=0.5, linestyle='--')
axes[1].plot(x, y2, color='green', linestyle='-.')

# Set the current axis to the first subplot
fig.sca(axes[0])
# set x and y ticks
plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi], ['-pi', '-pi/2', '0', 'pi/2', 'pi'])
# You probably don't want to set the labels when you just want the exact numbers.
plt.yticks([-1, 0, 1], ['-1', '0', '1'])

# set x and y limits
plt.xlim(-np.pi, np.pi)
plt.ylim(-1, 1)

# Set the current axis to the second subplot
fig.sca(axes[1])
# set x and y ticks
plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi], ['-pi', '-pi/2', '0', 'pi/2', 'pi'])
plt.yticks([-1, 0, 1], ['-1', '0', '1'])

# set x and y limits
plt.xlim(-np.pi, np.pi)
plt.ylim(-1, 1);


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
3.5 Legend
When you create a line on a plot, you can pass it a keyword argument label and then create a legend that will use that label using the legend method. The legend method takes an optional parameter of loc for the location of the legend. You can see the values allowed here: http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.legend This is another one of those commands that you need to set the current axis if you have more than one subplot.
plt.plot(x,y, label='my_label')
plt.legend(loc='best')
Can you write Python code for it?

x = np.linspace(-2*np.pi, 2*np.pi)
y1 = np.sin(x)
y2 = np.cos(x)

fig, axes = plt.subplots(1,2)
# Let's set labels here
axes[0].plot(x, y1, color='r', linewidth=5, label='sin(x)')
axes[1].plot(x, y1, color='#ffaa00', linewidth=0.5, linestyle='--', label='sin(x)')
axes[1].plot(x, y2, color='green', linestyle='-.', label='cos(x)')

# Set the current axis to the first subplot
fig.sca(axes[0])
# set x and y ticks
plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi], ['-pi', '-pi/2', '0', 'pi/2', 'pi'])
# You probably don't want to set the labels when you just want the exact numbers.
plt.yticks([-1, 0, 1], ['-1', '0', '1'])
plt.legend(loc='best')

# set x and y limits
plt.xlim(-np.pi, np.pi)
plt.ylim(-1, 1)

# Set the current axis to the second subplot
fig.sca(axes[1])
# set x and y ticks
plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi], ['-pi', '-pi/2', '0', 'pi/2', 'pi'])
plt.yticks([-1, 0, 1], ['-1', '0', '1'])
plt.legend(loc='upper right')

# set x and y limits
plt.xlim(-np.pi, np.pi)
plt.ylim(-1, 1);


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
First, create an empty figure with 2 subplots
using the subplots method
figure, axes = plt.subplots(rows, columns)

The arguments (1, 2) indicate 1 row and 2 cols
The function plt.subplots returns an object for the figure and for each axes
There are multiple ways to accomplish this same goal, but this is probably the simplest - notice that each subplot is associated with one of the axes objects.
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
Many of the basic formatting problems you have will be solved by the magic of tight_layout. Before you start tweaking how you figure looks, try it out.
plt.tight_layout()
Can you write Python code for it?

fig, axes = plt.subplots(1,2)
axes[0].plot(x, y1)
axes[1].plot(x, y1)
axes[1].plot(x, y2)

plt.tight_layout();


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
Now let's actually plot the data using the plot method on an axis
axis.plot(x, y)
You can plot multiple lines on an axis
Can you write Python code for it?

fig, axes = plt.subplots(1,2)

# We plot one line on the first axis
axes[0].plot(x, y1)
# and both lines on the second axis
axes[1].plot(x, y1)
axes[1].plot(x, y2);


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
Oh no! That changed it for the last plot but not for the first plot.
To set each plot individually, you need to set the current axis to the subplot you are interested in using the method sca. Then you can use xticks and yticks.
plt.sca(axis)
plt.xticks([val1, val2, val3])
plt.yticks([yval1, yval2, yval3])
Can you write Python code for it?

x = np.linspace(-2*np.pi, 2*np.pi)
y1 = np.sin(x)
y2 = np.cos(x)

fig, axes = plt.subplots(1,2)
axes[0].plot(x, y1, color='r', linewidth=5)
axes[1].plot(x, y1, color='#ffaa00', linewidth=0.5, linestyle='--')
axes[1].plot(x, y2, color='green', linestyle='-.')

# Set the current axis to the first subplot
fig.sca(axes[0])
# set x and y ticks
plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi])
plt.yticks([-1, 0, 1])

# Set the current axis to the second subplot
fig.sca(axes[1])
# set x and y ticks
plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi])
plt.yticks([-1, 0, 1]);


I am working on the file "Lesson14_NumpyAndMatplotlib/matplotlib.ipynb".
The first task is:
To save your figure you can use the savefig command:
fig.savefig('fileanme', format='png')

Format options include png, pdf, ps, eps and svg
Can you write Python code for it?

fig, axes = plt.subplots(1,2)
axes[0].plot(x, y1)
axes[1].plot(x, y1)
axes[1].plot(x, y2)

fig.savefig('first_plot.png', format='png');


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
Boolean operators work on arrays too, and they return boolean arrays
Much like the basic arithmetic operations we discussed above, comparison operations are performed element-wise. That is, rather than returning a
single boolean, comparison operators compare each element in both arrays pairwise, and return an array of booleans (if the sizes of the input
arrays are incompatible, the comparison will simply return False). For example:
Can you write Python code for it?

arr1 = np.array([1, 2, 3, 4, 5])
arr2 = np.array([1, 1, 3, 3, 5])
print((arr1 == arr2))
c = (arr1 == arr2)
print((type(c)))
print((c.dtype))


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
How can we index if the array has more than one dimension?
Can you write Python code for it?

c = np.random.rand(3,3)
print(c)
print((c[1:3,0:2]))
print(a)
c[0,:] = a
print(c)


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
If you are planning on doing serious linear algebra, you might be better off using the np.matrix object instead of np.array.
Numpy 'gotchas'
Multiplication and Addition
As you may have noticed above, since NumPy arrays are modeled more closely after vectors and matrices, multiplying by a scalar will multiply each element of the array, whereas multiplying a list by a scalar will repeat that list N times.
Can you write Python code for it?

# Numpy arrays
A = np.arange(5)*2
print(A)
# Lists
B = list(range(5))*2
print(B)


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
In addition, if one of the arguments is a scalar, that value will be applied to all the elements of the array.
scalar - a quantity possessing only magnitude. (In this case we mean a single number either an int or a float)
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
Indexing arrays
In addition to the usual methods of indexing lists with an integer (or with a series of colon-separated integers for a slice), numpy allows you
to index arrays in a wide variety of different ways for more advanced operations.
First, the simple way:
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
Let's see what numpy can do.
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
Linear algebra with arrays
You can use arrays as vectors and matrices in linear algebra operations
Specifically, you can perform matrix/vector multiplication between arrays, by using the .dot method, or the np.dot function
dot product - the dot product between two vectors is based on the projection of one vector onto another.
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
Note: You can use the methods .any() and .all() or the functions np.any and np.all to return a single boolean indicating whether any or all values in the array are True, respectively.
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
NumPy has its own range() function, np.arange() (stands for array-range), that is more efficient for building larger arrays.  It functions in much the same way as range().
NumPy also has linspace() and logspace(), that can generate equally spaced samples between a start-point and an end-point.  Find out more with np.linspace?.
Can you write Python code for it?

print("Arange")
print((np.arange(5)))

# Args: start, stop, number of elements
print("Linspace")
print((np.linspace(5, 10, 5)))

# logspace can also take a base argument, by default it is 10
print("Logspace")
print((np.logspace(0, 1, 5)))
print((np.logspace(0, 1, 5, base=2)))


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
Similarly, when adding two numpy arrays together, we get the vector sum back, whereas when adding two lists together, we get the concatenation back.
Can you write Python code for it?

# Numpy arrays
A = np.arange(5) + np.arange(5)
print(A)
# Lists
B = list(range(5)) + list(range(5))
print(B)


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
TRY IT
Create a boolean array for arr1 for where values are >= 3
Views vs. Copies
In order to be as efficient as possible, numpy uses "views" instead of copies wherever possible. That is, numpy arrays derived from another base array generally refer to the ''exact same data'' as the base array. The consequence of this is that modification of these derived arrays will also modify the base array. The result of an array indexed by an array of indices is a ''copy'', but an array indexed by an array of booleans is a ''view''.
Specifically, slices of arrays are always views, unlike slices of lists or tuples, which are always copies.
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
TRY IT
Create an array from the list [0,1,2] and print out it's dtype
Datatype options
Choose your datatype based on how large the largest values could be, and how much memory you expect to use

bool_ - Boolean (True or False) stored as a byte
int_ - Default integer type (same as C long; normally either int64 or int32)
int8 - Byte (-128 to 127)
int16 - Integer (-32768 to 32767)
int32 - Integer (-2147483648 to 2147483647)
int64 - Integer (-9223372036854775808 to 9223372036854775807)
uint8 - Unsigned integer (0 to 255)
uint16 - Unsigned integer (0 to 65535)
uint32 - Unsigned integer (0 to 4294967295)
uint64 - Unsigned integer (0 to 18446744073709551615)
float_ - Shorthand for float64.
float16 - Half precision float: sign bit, 5 bits exponent, 10 bits mantissa
float32 - Single precision float: sign bit, 8 bits exponent, 23 bits mantissa
float64 - Double precision float: sign bit, 11 bits exponent, 52 bits mantissa
complex_ - Shorthand for complex128.
complex64 - Complex number, represented by two 32-bit floats (real and imaginary components)
complex128 - Complex number, represented by two 64-bit floats (real and imaginary components)

Creating Arrays
There are many other ways to create NumPy arrays, such as np.identity, np.zeros, np.zeros_like, np.ones, np.ones_like
Can you write Python code for it?

print(('2 rows, 3 columns of zeros:\n', np.zeros((2,3))))
print(('4x4 identity matrix:\n', np.identity(4)))
squared = []
for x in range(5):
    squared.append(x**2)
print(squared)
a = np.array(squared)
b = np.zeros_like(a)

print(('a:\n', a))
print(('b:\n', b))


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
TRY IT
Load the file 'example.tsv' a tab delimited file. Once you have that working, only load the odd numbered columns (1,3,5).
Arithmetic with ndarrays
Standard arithmetic operators perform element-wise operations on arrays of the same size.
Can you write Python code for it?

A = np.arange(5)
B = np.arange(5, 10)

print(('A', A))
print(('B', B))

print(('A+B', A+B))
print(('B-A', B-A))
print(('A*B', A*B))


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
The simplest way to use it is to just give it a file name. By default, your data will be loaded as floats with whitespace being the delimiter
my_arr = np.loadtxt('myfile.txt')
More likely you will need to use some of the keyword arguments. like dtype, delimiter, skiprows, or usecols Docs available here: http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html
my_array = loadtxt('myfile.csv', usecols=[1,2,3,4,5,6,7,8,9,10,11,12], delimiter=',')
Can you write Python code for it?


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
These arrays have attributes, like .ndim and .shape that tell us about the number and length of the dimensions.
The dimension of an array is the number of indices needed to select an element. Thus, if the array is seen as a function on a set of possible index combinations, it is the dimension of the space of which its domain is a discrete subset. Thus a one-dimensional array is a list of data, a two-dimensional array a rectangle of data, a three-dimensional array a block of data, etc.
The shape is the number of elements in each dimension of data
Can you write Python code for it?

c = np.ones((15, 30))
print(('number of dimensions of c:', c.ndim))
print(('length of c in each dimension:', c.shape))

x = np.array([[[1,2,3],[4,5,6],[7,8,9]] , [[0,0,0],[0,0,0],[0,0,0]]])
print(('number of dimensions of x:', x.ndim))
print(('length of x in each dimension:', x.shape))


I am working on the file "Lesson14_NumpyAndMatplotlib/numpy.ipynb".
The first task is:
We can try out some of those constants and functions:
Can you write Python code for it?

print((np.sqrt(4)))
print((np.pi))         # a constant
print((np.sin(np.pi)))


I am working on the file "Lesson15-TFLearn/Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Testing
After you're satisified with the training output and accuracy, you can then run the network on the test data set to measure it's performance! Remember, only do this after you've done the training and are satisfied with the results.
A good result will be higher than 95% accuracy. Some simple models have been known to get up to 99.7% accuracy!
Can you write Python code for it?

# Compare the labels that our model predicts with the actual labels

# Find the indices of the most confident prediction for each item. That tells us the predicted digit for that sample.
predictions = np.array(model.predict(testX)).argmax(axis=1)

# Calculate the accuracy, which is the percentage of times the predicated labels matched the actual labels
actual = testY.argmax(axis=1)
test_accuracy = np.mean(predictions == actual, axis=0)

# Print out the result
print("Test accuracy: ", test_accuracy)


I am working on the file "Lesson15-TFLearn/Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Training the network
Now that we've constructed the network, saved as the variable model, we can fit it to the data. Here we use the model.fit method. You pass in the training features trainX and the training targets trainY. Below I set validation_set=0.1 which reserves 10% of the data set as the validation set. You can also set the batch size and number of epochs with the batch_size and n_epoch keywords, respectively.
Too few epochs don't effectively train your network, and too many take a long time to execute. Choose wisely!
Can you write Python code for it?

# Training
model.fit(trainX, trainY, validation_set=0.1, show_metric=True, batch_size=100, n_epoch=20)


I am working on the file "Lesson15-TFLearn/Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Visualize the training data
Provided below is a function that will help you visualize the MNIST data. By passing in the index of a training example, the function show_digit will display that training image along with it's corresponding label in the title.
Can you write Python code for it?

# Visualizing the data
import matplotlib.pyplot as plt
%matplotlib inline

# Function for displaying a training image by it's index in the MNIST set
def show_digit(index):
    label = trainY[index].argmax(axis=0)
    # Reshape 784 array into 28x28 image
    image = trainX[index].reshape([28,28])
    plt.title('Training data, index: %d,  Label: %d' % (index, label))
    plt.imshow(image, cmap='gray_r')
    plt.show()

# Display the first (index 0) training image
show_digit(0)


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn - Solution.ipynb".
The first task is:
Building the network
TFLearn lets you build the network by defining the layers.
Input layer
For the input layer, you just need to tell it how many units you have. For example,
net = tflearn.input_data([None, 100])
would create a network with 100 input units. The first element in the list, None in this case, sets the batch size. Setting it to None here leaves it at the default batch size.
The number of inputs to your network needs to match the size of your data. For this example, we're using 10000 element long vectors to encode our input data, so we need 10000 input units.
Adding layers
To add new hidden layers, you use
net = tflearn.fully_connected(net, n_units, activation='ReLU')
This adds a fully connected layer where every unit in the previous layer is connected to every unit in this layer. The first argument net is the network you created in the tflearn.input_data call. It's telling the network to use the output of the previous layer as the input to this layer. You can set the number of units in the layer with n_units, and set the activation function with the activation keyword. You can keep adding layers to your network by repeated calling net = tflearn.fully_connected(net, n_units).
Output layer
The last layer you add is used as the output layer. There for, you need to set the number of units to match the target data. In this case we are predicting two classes, positive or negative sentiment. You also need to set the activation function so it's appropriate for your model. Again, we're trying to predict if some input data belongs to one of two classes, so we should use softmax.
net = tflearn.fully_connected(net, 2, activation='softmax')
Training
To set how you train the network, use
net = tflearn.regression(net, optimizer='sgd', learning_rate=0.1, loss='categorical_crossentropy')
Again, this is passing in the network you've been building. The keywords:

optimizer sets the training method, here stochastic gradient descent
learning_rate is the learning rate
loss determines how the network error is...
Can you write Python code for it?

# Network building
def build_model():
    # This resets all parameters and variables, leave this here
    tf.reset_default_graph()

    # Inputs
    net = tflearn.input_data([None, 10000])

    # Hidden layer(s)
    net = tflearn.fully_connected(net, 200, activation='ReLU')
    net = tflearn.fully_connected(net, 25, activation='ReLU')

    # Output layer
    net = tflearn.fully_connected(net, 2, activation='softmax')
    net = tflearn.regression(net, optimizer='sgd',
                             learning_rate=0.1,
                             loss='categorical_crossentropy')

    model = tflearn.DNN(net)
    return model


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn - Solution.ipynb".
The first task is:
If you do this right, the following code should return
```
text_to_vector('The tea is for a party to celebrate '
               'the movie so she has no time for a cake')[:65]
array([0, 1, 0, 0, 2, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 1, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0])
```
Can you write Python code for it?


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn - Solution.ipynb".
The first task is:
Intializing the model
Next we need to call the build_model() function to actually build the model. In my solution I haven't included any arguments to the function, but you can add arguments so you can change parameters in the model if you want.

Note: You might get a bunch of warnings here. TFLearn uses a lot of deprecated code in TensorFlow. Hopefully it gets updated to the new TensorFlow version soon.
Can you write Python code for it?


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn - Solution.ipynb".
The first task is:
Let's keep the first 10000 most frequent words. As Andrew noted, most of the words in the vocabulary are rarely used so they will have little effect on our predictions. Below, we'll sort vocab by the count value and keep the 10000 most frequent words.
Can you write Python code for it?

vocab = sorted(total_counts, key=total_counts.get, reverse=True)[:10000]
print(vocab[:60])


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn - Solution.ipynb".
The first task is:
Now, run through our entire review data set and convert each review to a word vector.
Can you write Python code for it?

word_vectors = np.zeros((len(reviews), len(vocab)), dtype=np.int_)
for ii, (_, text) in enumerate(reviews.iterrows()):
    word_vectors[ii] = text_to_vector(text[0])

# Printing out the first 5 word vectors
word_vectors[:5, :23]


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn - Solution.ipynb".
The first task is:
Preparing the data
Following along with Andrew, our goal here is to convert our reviews into word vectors. The word vectors will have elements representing words in the total vocabulary. If the second position represents the word 'the', for each review we'll count up the number of times 'the' appears in the text and set the second position to that count. I'll show you examples as we build the input data from the reviews data. Check out Andrew's notebook and video for more about this.
Read the data
Use the pandas library to read the reviews and postive/negative labels from comma-separated files. The data we're using has already been preprocessed a bit and we know it uses only lower case characters. If we were working from raw data, where we didn't know it was all lower case, we would want to add a step here to convert it. That's so we treat different variations of the same word, like The, the, and THE, all the same way.
Can you write Python code for it?

reviews = pd.read_csv('reviews.txt', header=None)
labels = pd.read_csv('labels.txt', header=None)


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn - Solution.ipynb".
The first task is:
The last word in our vocabulary shows up 30 times in 25000 reviews. I think it's fair to say this is a tiny proportion. We are probably fine with this number of words.
Note: When you run, you may see a different word from the one shown above, but it will also have the value 30. That's because there are many words tied for that number of counts, and the Counter class does not guarantee which one will be returned in the case of a tie.
Now for each review in the data, we'll make a word vector. First we need to make a mapping of word to index, pretty easy to do with a dictionary comprehension.

Exercise: Create a dictionary called word2idx that maps each word in the vocabulary to an index. The first word in vocab has index 0, the second word has index 1, and so on.
Can you write Python code for it?

word2idx = {word: i for i, word in enumerate(vocab)}


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Building the network
TFLearn lets you build the network by defining the layers.
Input layer
For the input layer, you just need to tell it how many units you have. For example,
net = tflearn.input_data([None, 100])
would create a network with 100 input units. The first element in the list, None in this case, sets the batch size. Setting it to None here leaves it at the default batch size.
The number of inputs to your network needs to match the size of your data. For this example, we're using 10000 element long vectors to encode our input data, so we need 10000 input units.
Adding layers
To add new hidden layers, you use
net = tflearn.fully_connected(net, n_units, activation='ReLU')
This adds a fully connected layer where every unit in the previous layer is connected to every unit in this layer. The first argument net is the network you created in the tflearn.input_data call. It's telling the network to use the output of the previous layer as the input to this layer. You can set the number of units in the layer with n_units, and set the activation function with the activation keyword. You can keep adding layers to your network by repeated calling net = tflearn.fully_connected(net, n_units).
Output layer
The last layer you add is used as the output layer. Therefore, you need to set the number of units to match the target data. In this case we are predicting two classes, positive or negative sentiment. You also need to set the activation function so it's appropriate for your model. Again, we're trying to predict if some input data belongs to one of two classes, so we should use softmax.
net = tflearn.fully_connected(net, 2, activation='softmax')
Training
To set how you train the network, use
net = tflearn.regression(net, optimizer='sgd', learning_rate=0.1, loss='categorical_crossentropy')
Again, this is passing in the network you've been building. The keywords:

optimizer sets the training method, here stochastic gradient descent
learning_rate is the learning rate
loss determines how the network error is...
Can you write Python code for it?

# Network building
def build_model():
    # This resets all parameters and variables, leave this here
    tf.reset_default_graph()

    #### Your code ####
    net = tflearn.input_data([None, trainX.shape[1]])
    net = tflearn.fully_connected(net, 200 , activation='ReLU')
    net = tflearn.fully_connected(net, 25 , activation='ReLU')
    net = tflearn.fully_connected(net, 2, activation='softmax')
    net = tflearn.regression(net, optimizer='sgd', learning_rate=0.1, loss='categorical_crossentropy')

    model = tflearn.DNN(net)
    return model


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Counting word frequency
To start off we'll need to count how often each word appears in the data. We'll use this count to create a vocabulary we'll use to encode the review data. This resulting count is known as a bag of words. We'll use it to select our vocabulary and build the word vectors. You should have seen how to do this in Andrew's lesson. Try to implement it here using the Counter class.

Exercise: Create the bag of words from the reviews data and assign it to total_counts. The reviews are stores in the reviews Pandas DataFrame. If you want the reviews as a Numpy array, use reviews.values. You can iterate through the rows in the DataFrame with for idx, row in reviews.iterrows(): (documentation). When you break up the reviews into words, use .split(' ') instead of .split() so your results match ours.
Can you write Python code for it?

from collections import Counter

total_counts = Counter()# bag of words here
for idx, row in reviews.iterrows():
    total_counts.update(row[0].lower().replace(",", " ").replace(".", " ").split(" "))

print("Total words in data set: ", len(total_counts))


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Let's keep the first 10000 most frequent words. As Andrew noted, most of the words in the vocabulary are rarely used so they will have little effect on our predictions. Below, we'll sort vocab by the count value and keep the 10000 most frequent words.
Can you write Python code for it?


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Now, run through our entire review data set and convert each review to a word vector.
Can you write Python code for it?


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Preparing the data
Following along with Andrew, our goal here is to convert our reviews into word vectors. The word vectors will have elements representing words in the total vocabulary. If the second position represents the word 'the', for each review we'll count up the number of times 'the' appears in the text and set the second position to that count. I'll show you examples as we build the input data from the reviews data. Check out Andrew's notebook and video for more about this.
Read the data
Use the pandas library to read the reviews and postive/negative labels from comma-separated files. The data we're using has already been preprocessed a bit and we know it uses only lower case characters. If we were working from raw data, where we didn't know it was all lower case, we would want to add a step here to convert it. That's so we treat different variations of the same word, like The, the, and THE, all the same way.
Can you write Python code for it?


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Testing
After you're satisified with your hyperparameters, you can run the network on the test set to measure its performance. Remember, only do this after finalizing the hyperparameters.
Can you write Python code for it?

predictions = (np.array(model.predict(testX))[:,0] >= 0.5).astype(np.int_)
test_accuracy = np.mean(predictions == testY[:,0], axis=0)
print("Test accuracy: ", test_accuracy)


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Text to vector function
Now we can write a function that converts a some text to a word vector. The function will take a string of words as input and return a vector with the words counted up. Here's the general algorithm to do this:

Initialize the word vector with np.zeros, it should be the length of the vocabulary.
Split the input string of text into a list of words with .split(' '). Again, if you call .split() instead, you'll get slightly different results than what we show here.
For each word in that list, increment the element in the index associated with that word, which you get from word2idx.

Note: Since all words aren't in the vocab dictionary, you'll get a key error if you run into one of those words. You can use the .get method of the word2idx dictionary to specify a default returned value when you make a key error. For example, word2idx.get(word, None) returns None if word doesn't exist in the dictionary.
Can you write Python code for it?

# Slow....
def text_to_vector(text):
    words_vector = np.zeros(len(vocab))
    words = text.lower().replace(",", " ").replace(".", " ").split(" ")
    keys = list(word2idx)
    for key in words:
        if key in keys:
            words_vector[word2idx[key]] += 1
    return words_vector

# Mat's Fast solution
def text_to_vector(text):
    word_vector = np.zeros(len(vocab), dtype=np.int_)
    for word in text.replace(",", " ").replace(".", " ").split(" "):
        idx = word2idx.get(word, None)
        if idx is None:
            continue
        else:
            word_vector[idx] = 1
    return np.array(word_vector)


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Training the network
Now that we've constructed the network, saved as the variable model, we can fit it to the data. Here we use the model.fit method. You pass in the training features trainX and the training targets trainY. Below I set validation_set=0.1 which reserves 10% of the data set as the validation set. You can also set the batch size and number of epochs with the batch_size and n_epoch keywords, respectively. Below is the code to fit our the network to our word vectors.
You can rerun model.fit to train the network further if you think you can increase the validation accuracy. Remember, all hyperparameter adjustments must be done using the validation set. Only use the test set after you're completely done training the network.
Can you write Python code for it?

# Training
model.fit(trainX, trainY, validation_set=0.1, show_metric=True, batch_size=128, n_epoch=10)


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
Try out your own text!
Can you write Python code for it?

# Helper function that uses your model to predict sentiment
def test_sentence(sentence):
    positive_prob = model.predict([text_to_vector(sentence.lower())])[0][1]
    print('Sentence: {}'.format(sentence))
    print('P(positive) = {:.3f} :'.format(positive_prob),
          'Positive' if positive_prob > 0.5 else 'Negative')

sentence = "Moonlight is by far the best movie of 2016."
test_sentence(sentence)

sentence = "It's amazing anyone could be talented enough to make something this spectacularly awful"
test_sentence(sentence)


I am working on the file "Lesson15-TFLearn/Sentiment Analysis with TFLearn.ipynb".
The first task is:
What's the last word in our vocabulary? We can use this to judge if 10000 is too few. If the last word is pretty common, we probably need to keep more words.
Can you write Python code for it?

print(vocab[-1], ': ', total_counts[vocab[-1]])


I am working on the file "Lesson16-Intro to TensorFlow/intro_to_tensorflow.ipynb".
The first task is:
<img src="image/Learn Rate Tune - Image.png" style="height: 70%;width: 70%">
Problem 3
Below are 2 parameter configurations for training the neural network. In each configuration, one of the parameters has multiple options. For each configuration, choose the option that gives the best acccuracy.
Parameter configurations:
Configuration 1
* Epochs: 1
* Learning Rate:
  * 0.8
  * 0.5
  * 0.1
  * 0.05
  * 0.01
Configuration 2
* Epochs:
  * 1
  * 2
  * 3
  * 4
  * 5
* Learning Rate: 0.2
The code will print out a Loss and Accuracy graph, so you can see how well the neural network performed.
If you're having trouble solving problem 3, you can view the solution here.
Can you write Python code for it?

# Change if you have memory restrictions
batch_size = 128

# TODO: Find the best parameters for each configuration
epochs = 5
learning_rate = 0.1

### DON'T MODIFY ANYTHING BELOW ###
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

# The accuracy measured against the validation set
validation_accuracy = 0.0

# Measurements use for graphing loss and accuracy
log_batch_step = 50
batches = []
loss_batch = []
train_acc_batch = []
valid_acc_batch = []

with tf.Session() as session:
    session.run(init)
    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        # Progress bar
        batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(epoch_i+1, epochs), unit='batches')

        # The training cycle
        for batch_i in batches_pbar:
            # Get a batch of training features and labels
            batch_start = batch_i*batch_size
            batch_features = train_features[batch_start:batch_start + batch_size]
            batch_labels = train_labels[batch_start:batch_start + batch_size]

            # Run optimizer and get loss
            _, l = session.run(
                [optimizer, loss],
                feed_dict={features: batch_features, labels: batch_labels})

            # Log every 50 batches
            if not batch_i % log_batch_step:
                # Calculate Training and Validation accuracy
                training_accuracy = session.run(accuracy, feed_dict=train_feed_dict)
                validation_accuracy = session.run(accuracy, feed_dict=valid_feed_dict)

                # Log batches
                previous_batch = batches[-1] if batches else 0
                batches.append(log_batch_step + previous_batch)
                loss_batch.append(l)
                train_acc_batch.append(training_accuracy)
                valid_acc_batch.append(validation_accuracy)

        # Check accuracy against Validation data
        validation_accuracy =...


I am working on the file "Lesson16-Intro to TensorFlow/intro_to_tensorflow.ipynb".
The first task is:
<img src="image/Mean Variance - Image.png" style="height: 75%;width: 75%; position: relative; right: 5%">
Problem 1
The first problem involves normalizing the features for your training and test data.
Implement Min-Max scaling in the normalize_grayscale() function to a range of a=0.1 and b=0.9. After scaling, the values of the pixels in the input data should range from 0.1 to 0.9.
Since the raw notMNIST image data is in grayscale, the current values range from a min of 0 to a max of 255.
Min-Max Scaling:
$
X'=a+{\frac {\left(X-X_{\min }\right)\left(b-a\right)}{X_{\max }-X_{\min }}}
$
If you're having trouble solving problem 1, you can view the solution here.
Can you write Python code for it?

train_features.min()

# Problem 1 - Implement Min-Max scaling for grayscale image data
def normalize_grayscale(image_data):
    """
    Normalize the image data with Min-Max scaling to a range of [0.1, 0.9]
    :param image_data: The image data to be normalized
    :return: Normalized image data
    """
    # TODO: Implement Min-Max scaling for grayscsale image data
    a = 0.1
    b = 0.9
    return (a + ((image_data-image_data.min())*(b-a))/(image_data.max()-image_data.min()))

### DON'T MODIFY ANYTHING BELOW ###
# Test Cases
np.testing.assert_array_almost_equal(
    normalize_grayscale(np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 255])),
    [0.1, 0.103137254902, 0.106274509804, 0.109411764706, 0.112549019608, 0.11568627451, 0.118823529412, 0.121960784314,
     0.125098039216, 0.128235294118, 0.13137254902, 0.9],
    decimal=3)
np.testing.assert_array_almost_equal(
    normalize_grayscale(np.array([0, 1, 10, 20, 30, 40, 233, 244, 254,255])),
    [0.1, 0.103137254902, 0.13137254902, 0.162745098039, 0.194117647059, 0.225490196078, 0.830980392157, 0.865490196078,
     0.896862745098, 0.9])

if not is_features_normal:
    train_features = normalize_grayscale(train_features)
    test_features = normalize_grayscale(test_features)
    is_features_normal = True

print('Tests Passed!')

if not is_labels_encod:
    # Turn labels into numbers and apply One-Hot Encoding
    encoder = LabelBinarizer()
    encoder.fit(train_labels)
    train_labels = encoder.transform(train_labels)
    test_labels = encoder.transform(test_labels)

    # Change to float32, so it can be multiplied against the features in TensorFlow, which are float32
    train_labels = train_labels.astype(np.float32)
    test_labels = test_labels.astype(np.float32)
    is_labels_encod = True

print('Labels One-Hot Encoded')

assert is_features_normal, 'You skipped the step to normalize the features'
assert is_labels_encod, 'You skipped the step to One-Hot Encode the labels'

# Get randomized datasets for training and validation
train_features,...


I am working on the file "Lesson16-Intro to TensorFlow/intro_to_tensorflow.ipynb".
The first task is:
Problem 2
Now it's time to build a simple neural network using TensorFlow. Here, your network will be just an input layer and an output layer.

For the input here the images have been flattened into a vector of $28 \times 28 = 784$ features. Then, we're trying to predict the image digit so there are 10 output units, one for each label. Of course, feel free to add hidden layers if you want, but this notebook is built to guide you through a single layer network.
For the neural network to train on your data, you need the following float32 tensors:
 - features
  - Placeholder tensor for feature data (train_features/valid_features/test_features)
 - labels
  - Placeholder tensor for label data (train_labels/valid_labels/test_labels)
 - weights
  - Variable Tensor with random numbers from a truncated normal distribution.
    - See tf.truncated_normal() documentation for help.
 - biases
  - Variable Tensor with all zeros.
    - See  tf.zeros() documentation for help.
If you're having trouble solving problem 2, review "TensorFlow Linear Function" section of the class.  If that doesn't help, the solution for this problem is available here.
Can you write Python code for it?

# All the pixels in the image (28 * 28 = 784)
features_count = 784
# All the labels
labels_count = 10

# TODO: Set the features and labels tensors
features = tf.placeholder(tf.float32)
labels = tf.placeholder(tf.float32)

# TODO: Set the weights and biases tensors
weights = tf.Variable(tf.truncated_normal((features_count, labels_count)))
biases = tf.Variable(tf.zeros(labels_count ))

### DON'T MODIFY ANYTHING BELOW ###

#Test Cases
from tensorflow.python.ops.variables import Variable

assert features._op.name.startswith('Placeholder'), 'features must be a placeholder'
assert labels._op.name.startswith('Placeholder'), 'labels must be a placeholder'
assert isinstance(weights, Variable), 'weights must be a TensorFlow variable'
assert isinstance(biases, Variable), 'biases must be a TensorFlow variable'

assert features._shape == None or (\
    features._shape.dims[0].value is None and\
    features._shape.dims[1].value in [None, 784]), 'The shape of features is incorrect'
assert labels._shape  == None or (\
    labels._shape.dims[0].value is None and\
    labels._shape.dims[1].value in [None, 10]), 'The shape of labels is incorrect'
assert weights._variable._shape == (784, 10), 'The shape of weights is incorrect'
assert biases._variable._shape == (10), 'The shape of biases is incorrect'

assert features._dtype == tf.float32, 'features must be type float32'
assert labels._dtype == tf.float32, 'labels must be type float32'

# Feed dicts for training, validation, and test session
train_feed_dict = {features: train_features, labels: train_labels}
valid_feed_dict = {features: valid_features, labels: valid_labels}
test_feed_dict = {features: test_features, labels: test_labels}

# Linear Function WX + b
logits = tf.matmul(features, weights) + biases

prediction = tf.nn.softmax(logits)

# Cross entropy
cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)

# Training loss
loss = tf.reduce_mean(cross_entropy)

# Create an operation that initializes all variables
init =...


I am working on the file "Lesson16-Intro to TensorFlow/intro_to_tensorflow.ipynb".
The first task is:
Test
You're going to test your model against your hold out dataset/testing data.  This will give you a good indicator of how well the model will do in the real world.  You should have a test accuracy of at least 80%.
Can you write Python code for it?

### DON'T MODIFY ANYTHING BELOW ###
# The accuracy measured against the test set
test_accuracy = 0.0

with tf.Session() as session:

    session.run(init)
    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        # Progress bar
        batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(epoch_i+1, epochs), unit='batches')

        # The training cycle
        for batch_i in batches_pbar:
            # Get a batch of training features and labels
            batch_start = batch_i*batch_size
            batch_features = train_features[batch_start:batch_start + batch_size]
            batch_labels = train_labels[batch_start:batch_start + batch_size]

            # Run optimizer
            _ = session.run(optimizer, feed_dict={features: batch_features, labels: batch_labels})

        # Check accuracy against Test data
        test_accuracy = session.run(accuracy, feed_dict=test_feed_dict)

assert test_accuracy >= 0.80, 'Test accuracy at {}, should be equal to or greater than 0.80'.format(test_accuracy)
print('Nice Job! Test Accuracy is {}'.format(test_accuracy))


I am working on the file "Lesson16-Intro to TensorFlow/intro_to_tensorflow.ipynb".
The first task is:
The notMNIST dataset is too large for many computers to handle.  It contains 500,000 images for just training.  You'll be using a subset of this data, 15,000 images for each label (A-J).
Can you write Python code for it?

def download(url, file):
    """
    Download file from <url>
    :param url: URL to file
    :param file: Local file path
    """
    if not os.path.isfile(file):
        print('Downloading ' + file + '...')
        urlretrieve(url, file)
        print('Download Finished')

# Download the training and test dataset.
download('https://s3.amazonaws.com/udacity-sdc/notMNIST_train.zip', 'notMNIST_train.zip')
download('https://s3.amazonaws.com/udacity-sdc/notMNIST_test.zip', 'notMNIST_test.zip')

# Make sure the files aren't corrupted
assert hashlib.md5(open('notMNIST_train.zip', 'rb').read()).hexdigest() == 'c8673b3f28f489e9cdf3a3d74e2ac8fa',\
        'notMNIST_train.zip file is corrupted.  Remove the file and try again.'
assert hashlib.md5(open('notMNIST_test.zip', 'rb').read()).hexdigest() == '5d3c7e653e63471c88df796156a9dfa9',\
        'notMNIST_test.zip file is corrupted.  Remove the file and try again.'

# Wait until you see that all files have been downloaded.
print('All files downloaded.')

def uncompress_features_labels(file):
    """
    Uncompress features and labels from a zip file
    :param file: The zip file to extract the data from
    """
    features = []
    labels = []

    with ZipFile(file) as zipf:
        # Progress Bar
        filenames_pbar = tqdm(zipf.namelist(), unit='files')

        # Get features and labels from all files
        for filename in filenames_pbar:
            # Check if the file is a directory
            if not filename.endswith('/'):
                with zipf.open(filename) as image_file:
                    image = Image.open(image_file)
                    image.load()
                    # Load image data as 1 dimensional array
                    # We're using float32 to save on memory space
                    feature = np.array(image, dtype=np.float32).flatten()

                # Get the the letter from the filename.  This is the letter of the image.
                label = os.path.split(filename)[1][0]

                features.append(feature)
        ...


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
If you want to understand what is batch and epoch, you can read here.
Not that bad in performance since we have limited data. :) If you curious, you can use all dogs vs cats data (approx. 25000 train data vs 15000 test data) and get 97% in accuracy with only 1 epoch.
Breakdown the Process, Step by Step
For Basic Image Recognition
We will try to use VGG16 to recognise categories from ImageNet from each image. Means, we just want to know each type of breed, not necessesarily guess dog or cat (since the model are not capable to do that because ImageNet don't have 'cat' or 'dog' category). First, we load the object.
Can you write Python code for it?


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
In 2014, the creators of VGG substracted the average of each of the three (R,G,B) channel first, so that the data for each channel had a mean of zero.The problem is, while their original software expected the channels to be in B,G,R order,  Python uses R,G,B. Now, we need to preprocess our data to make these two changes, so that it is compatible with the VGG model.
Can you write Python code for it?

# Mean of each channel as provided by VGG researchers
vgg_mean = np.array([123.68,116.779,103.939]).reshape((3,1,1))

def vgg_preprocess(x):
    x=x - vgg_mean      # substract mean
    return x[:, ::-1]   # reverse axis bgr -> rgb


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
Next, we import libraries that we will use later.
Can you write Python code for it?

from __future__ import division,print_function

import os,json
from glob import glob
import numpy as np
np.set_printoptions(precision=4,linewidth=100)
from matplotlib import pyplot as plt


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
Now we learned basic steps on how to use VGG16 for image recognition task. (ex: disease, paintings, etc.
Next, we will try to understand what's inside VGG16.
Create VGG16 model from scratch with Keras (TODO: Complete this section)
Model Setup
First, we import all the libraries that we need.
Can you write Python code for it?

from numpy.random import random, permutation
from scipy import ndimage, misc
from scipy.ndimage.interpolation import zoom

import keras
from keras import backend as K
from keras.utils.data_utils import get_file
from keras.models import Sequential, Model
from keras.layers.core import Flatten, Dense, Dropout, Lambda
from keras.layers import Input
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.optimizers import SGD, RMSprop
from keras.preprocessing import image


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
Now, import ImageNet category ids and descriptions. It's not necessarily important for our model, but we will use this for plotting purpose.
Can you write Python code for it?

FILES_PATH = 'http://files.fast.ai/models'; CLASS_FILE= 'imagenet_class_index.json'
fpath=get_file(CLASS_FILE,FILES_PATH+CLASS_FILE,cache_subdir='models')
with open(fpath) as f: class_dict=json.load(f)
# Convert dictionaries to an array
classes= [class_dict[str(i)][1] for i in range(len(class_dict))]


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
Now, we configure Keras to use Theano as default configuration. We can change it to Tensorflow if we want.
Can you write Python code for it?

!mkdir -p ~/.keras
!echo '{"epsilon":1e-07,"floatx":"float32","backend":"theano","img_dim_ordering":"th"}' > ~/.keras/keras.json


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
Now, we grab batches from our training data.
Can you write Python code for it?

path = "D:/datasets/dogscats/"

batches = vgg.get_batches(path+'train',batch_size=4)

imgs, labels = next(batches)


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
The category indexes are based on VGG model from ImageNet. If we want to know the first four of that categories:
Can you write Python code for it?


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
Use VGG16 to finetune Cats vs Dogs Model
Now, we will try to specifically using VGG16 for classify dogs and cats images. Because by default VGG16 was trained using ImageNet, we will use a technique called "finetuning". Finetuning is a process that use a pretrained model for second similiar task. Source
First, we setup batches. A batch is a subset of the training data. We need this to speed up training time and to control gpu memory usage.
Can you write Python code for it?

batch_size=16

batches = vgg.get_batches(path+'train',batch_size=batch_size)
val_batches = vgg.get_batches(path+'valid',batch_size=batch_size)


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
Using Pretrained VGG Model
Now, for simplification, we just need 7 lines of code to classify cats and dogs photos. For using VGG, we already have pretrained model. So now we just load the model.
Can you write Python code for it?

batch_size = 16

import vgg16; reload(vgg16)
from vgg16 import Vgg16

vgg = Vgg16()

batches = vgg.get_batches(path+'train',batch_size=batch_size)
val_batches = vgg.get_batches(path+'valid', batch_size=batch_size)
vgg.finetune(batches)
vgg.fit(batches,val_batches,nb_epoch=10)


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
We also need the weights that the VGG creators trained. The weights are the part of the model that is learnt from the data, whereas the architecture is pre-defined based on the nature of the problem.
It's preferred to download pre-trained weights than train the model by ourselves, since it's really resources demanding (time, size of dataset, etc.)
Can you write Python code for it?

#fpath = get_file('vgg16.h5', FILES_PATH+'vgg16.hdf',cache_subdir='models')
#model.load_weights(fpath)
model.load_weights('vgg16.hdf')


I am working on the file "Lesson1_Aris.ipynb".
The first task is:
We'll know what these blocks do later. For now:

Convolution layers are for finding patterns in images.
Dense (fully connected) layers are for combining patterns across an image.

Now that we've defined the architecture, we can create the model like any python object.
Can you write Python code for it?


I am working on the file "Lesson_00_algorithms.ipynb".
The first task is:
- Take the supplies
Seems like a similar task to "mapping" tools. Let's extend the action list:
Can you write Python code for it?

# Chair leg, screwdriver, 30mm screw
# Char seat, hammer, wooden peg
# Char leg, screwdriver, 30mm screw
# ...


I am working on the file "Lesson_00_algorithms.ipynb".
The first task is:
But what if we no longer have any screws? We should stop the program. It also means, somewhere else in program, we should keep count of screws, and each time one is used, decrease the count by 1.
- Perform instruction
Now, everyone knows this is the hardest part. We need to tell the computer to do something with many elements. "Doing stuff" in programming is performed by instructions, or a sequence of instructions (called a procedure, function, method).
Lets say we define how to use a screw and screwdriver:
Can you write Python code for it?

def screw_together( furniture_piece, supplies, tool):
    connected_furniture = tool.screw(furniture_piece, supplies)
    return connected_furniture


I am working on the file "Lesson_00_algorithms.ipynb".
The first task is:
Example algorithm - IKEA furniture assembly
Now, let's complicate things a bit.
Given a list of pieces, each with information about what tool to use, and needed supplies like screws, how will you do it step by step?
Simple description
We can just take a new piece, get the tool, get the supplies, and do the instruction. Then, take the next one.
How to model this information?
Now, imagine that you need to explain this to the computer.
- Take a new piece
From where? We need a thing with all pieces. IKEA instructions tell you to do things in a sequence, so let's try a list. We need a list with all pieces.
- Take the required tool
How do we know which one? It seems like, for each piece, we need to know this. We need to connect this information. To keep things simple, let's keep this information in a list. So for example:
Can you write Python code for it?

# Chair leg, screwdriver
# Chair seat, hammer
# Chair leg, screwdriver


I am working on the file "Lesson_00_algorithms.ipynb".
The first task is:
How does the algorithm look like?
Generally speaking, for each furniture element, perform the instruction using a tool and some supplies.
In pseudo code:
Can you write Python code for it?

for action in list_of_actions:
    action()


I am working on the file "Lesson_00_algorithms.ipynb".
The first task is:
We managed to at least model the behavior. We know the minimum amount of information an algorithm needs to work its way to the end.
One additional thing is, actually it's really hard to write this down. And a single row of our action list has some duplicated information. Maybe we can write it easier:
Can you write Python code for it?

# screw_together( Chair leg, 30mm screw, screwdriver)
# hammer_together( Chair seat,  wooden peg, hammer)
# screw_together( Chair leg, 30mm screw, screwdriver)
# ...


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Actually, strings are 'lists' of characters. We will explore lists in just a moment, but I want you to become familiar with a new notation. It is based on the order of sequence. When I say, "Give me the second character of this string", I can write is as such:
Can you write Python code for it?

print 'Second character is:  ' + name[1]


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Adding and removing from a list
Can you write Python code for it?

l = []
l.append(1)
print l
l[0] = 222
print l

l.remove(1)
print l

l = [1,2,3,3,4,5,3,2,3,2]

# Make a new list from a part of that list
new = l[4:7]
print new


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Boolean
It represents the True and False values. Variables of this type, can be only True or False.
It is useful to know, that in Python we can check any variable to be True or False, even lists!
We use the bool() function.
Can you write Python code for it?

a = True
b = False

print("Is a equal to b ?")
print(a==b)

print("Logical AND")
print(a and b)

print("Logical OR")
print(a or b)

print("Logical value of True")
print( bool(a) )

print("Logical value of an empty list")
print( bool([]) )

print("Logical value of an empty string")
print( bool("") )

print("Logical value of integer 0")
print( bool(0) )


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Dictionary
This data structure is very useful. In essence, it stores pairs of values, first of which is always a "key", a unique identifier, and the "value", which is the connected object.
A dictionary performs a mapping between keys and values. Because the key is always unique (has to be, we will find out in a minute), there is always exactly one key with specific content.
A dictionary is also very efficient - finding a value in a dictionary takes only one operation, whereas searching through a list one by one could require going through the whole list.
This means that for any situation, where you need to store lot's of values, that will be often used, it is much better to store them in a dictionary.
Also, I recommend to read on Wikipedia on "hash maps".
Creating dictionaries
Can you write Python code for it?

empty_dictionary = {}
print empty_dictionary
print type(empty_dictionary)

dictionary_from_direct_definition = {"key1": 1, "key2": 33}
print dictionary_from_direct_definition

# Let's create a dictionary from a list of tuples
dictionary_from_a_collection = dict([("a", 1), ("b", 2)])
print dictionary_from_a_collection

# Let's create a dictionary from two lists
some_list_with_strings = ["a", "b", "c"]
some_list_with_numbers = [1,2,3]
dictionary_from_two_lists = dict(zip(some_list_with_strings, some_list_with_numbers))
print dictionary_from_two_lists
print type(dictionary_from_two_lists)

# Let's create a dictionary from a dictionary comprehension
dict_from_comprehension = {key:value for key, value in zip(some_list_with_strings, some_list_with_numbers)}
print dict_from_comprehension


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Even though strings are not numbers, you can do a lot of operations on them using the usual operators.
Can you write Python code for it?

name = 'Adam'
print name + name
print name * 3


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Example of looking for a specific thing in a list, and in a dictionary:
Can you write Python code for it?

l = ["r", "p", "s", "t"]
d = {a: a for a in l}

# Find "t" in list.
for letter in l:
    if letter == "t":
        print "Found it!"
    else:
        print "Not yet!"

# Find "t" in dictionary keys.
print "In dictionary - found it! " + d["t"]


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Inspect a dictionary
Can you write Python code for it?

# How many keys?
print d.keys()
print len(d)
print len(d.keys())

# How many values?
print d.values()
print len(d.values())


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Iterate over dictionary
Can you write Python code for it?

for key, value in d.items():
    print key, value


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Iterating over a list
But lists are not only used to hold some sequences! You can iterate over a list. This means no more, no less, then doing something for each of the elements in a given range, or for all of them. We will cover the so-called 'for' loop in next lessons, but I guess you can easily imagine what this minimal example would do.
Can you write Python code for it?

# Do something for all of elements.
for element in [1, 2, 3]:
    print element + 20

# Do something for numbers coming from a range of numbers.
for number in range(0,3):
    print number + 20

# Do something for all of elements, but written in a short way.
some_list = ['a', 'b', 'c']
print [element*2 for element in some_list]


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
List
Prepare to use this data type A LOT. Lists can store any objects, and have as many elements, as you like. The most important thing about lists, is that their elements are ordered. You can create a list by making an empty list, converting something else to a list, or defining elements of a list right there, when you declare it.
Creating lists.
Can you write Python code for it?

empty_list = []
list_from_something_else = list('I feel like Im going to explode')
list_elements_defined_when_list_is_created = [1, 2, 3, 4]
print empty_list
print list_from_something_else
print list_elements_defined_when_list_is_created


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Multiple assignment
Can you write Python code for it?

a, b, c = 1, 2, 3
print a, b, c

a = b = c = d = "The same string"
print a, b, c, d


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Now, let's change this name to something else.
Can you write Python code for it?

some_person = "Jamie"
person_age = 24
print some_person, type(some_person), hex(id(some_person))
print person_age, type(person_age), hex(id(person_age))


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Now, when we modify the binding of 'shared_list' variable, both of our variables will change also!
Can you write Python code for it?

shared_list.append(33)
print shared_list, my_list, your_list


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Remove items
Can you write Python code for it?


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Sets
A set behaves pretty much like a mixture of a dictionary and a list. It has two features:
- it only has unique values
- it does not respect order of things - it has no order, like a dictionary
Can you write Python code for it?

some_sequence = [1,1,1,1,2,2,2,3,3,3]
some_set = set(some_sequence)
print some_set

some_string = "What's going ooooon?"
another_set = set(some_string)
print another_set

some_dictionary = {"a": 2, "b": 2}
print some_dictionary

yet_another_set = set(some_dictionary)
print yet_another_set

print set(some_dictionary.values())


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Since we are counting from 0, the second character has index = 1.
Now, say I want characters from second, to fourth.
Can you write Python code for it?

print 'From second to fourth: ' + name[1:4]

print 'The last character (or first counting from the end) is: ' + name[-1]

print 'All characters, but skip every second: ' + name[0:4:2]


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Strings
Represents text, or to be more specific, sequences of 'Unicode' characters. To let Python know we are using strings, put them in quotes, either single, or double.
Can you write Python code for it?

a = "Something"
b = 'Something else'
print type(a), type(b)


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
The important bit is that, even though we use the same variable "person_age", the memory address changed. The object holding integer '22' is still living somewhere on the process heap, but is no longer bound to any name, and probably will be deleted by the "Garbage Collector". The binding that exists now, if from name "person_age" to the int object "24".
The same can be said about variable 'some_person'.
Mutability and immutability
The reason we need to talk about this, is that when you use variables in Python, you have to understand that such a "binding" can be shared! When you modify one, the other shared bindings will be modified as well! This is true for "mutable" objects. There are also "immutable" objects, that behave in a standard, standalone, not-changeable way.
Immutable types: int, float, decimal, complex, bool, string, tuple, range, frozenset, bytes
Mutable types: list, dict, set, bytearray, user-defined classes
Can you write Python code for it?

shared_list = [11,22]
my_list = shared_list
your_list = shared_list
print shared_list, my_list, your_list


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
These operations are called 'slicing'.
We can also find substrings in other substrings. THe result is the index, at which this substring occurs.
Can you write Python code for it?

some_string = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAxxAAAAAAAAAAAAAAAAAAAA"
substring = "xx"
location = some_string.find(substring)

print("Lets see what we found:")
print(some_string[location:location+len(substring)])


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
This can be very confusing later on, if you do not grasp this right now. Feel free to play around :)
Data types
What is a data type? It is a way of telling our computer, that we want to store a specific kind of information in a particular variable. This allows us to access tools and mechanisms that are allowed for that type.
We already mentioned that actually every time we create a variable, we create a complex type variable, or an object.
This is called creating an object, or instantiating an object. Each object comes from a specific template, or how we call it in Object Oriented Programming, from a class.
So when you assign a variable, you instantiate an object from a class.
In Python, every data type is a class!
Also, we will use some built-in tools for inspection - type()  and  isinstance() functions. The function type() will just say from which class does this object come from. THe function isinstance() will take an object reference, and then a class name, and will tell you if this is an instance of this class.
Let's review data types used in Python (most of them).
Numeric types
These types allow you to store numbers. Easy.
int
Integers. If you create a really big integer, it will become a 'long integer', or 'long'.
Can you write Python code for it?


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
To be completely precise, let's look at creating two variables that store some names. To see where in memory does the object go, we can use method "id". To see the hex representation of this memory, as you will usually see, we can use the method "id".
Can you write Python code for it?

some_person = "Andrew"
person_age = 22
print some_person, type(some_person), hex(id(some_person))
print person_age, type(person_age), hex(id(person_age))


I am working on the file "Lesson_01_variables_and_data_types.ipynb".
The first task is:
Using dictionaries
Add key-value pairs
Can you write Python code for it?


I am working on the file "Lesson_02_classes_object_oriented.ipynb".
The first task is:
Accessing objects 'fields'
Can you write Python code for it?


I am working on the file "Lesson_02_classes_object_oriented.ipynb".
The first task is:
What are class variables and instance variables?
Class variables are variables attached to the definition of a class. Simply, they are just regular variable definitions inside a class
Instance variables are variables created for each instance of a class. We denote them by adding "self." in front of them.
Examples:
Can you write Python code for it?

class ClassI:
    # Define instance variables in a special method, called a "constructor", that defines what happens when an object is created.
    def __init__(self):
        self.a = 1
        self.b = 2

class ClassC:
    # Define class variables normally. They are here, whether you create an object or not.
    a = 3
    b = 44

instance_of_ClassC = ClassC()
print instance_of_ClassC.a, instance_of_ClassC.b

print ClassC.a

instance_of_ClassI = ClassI()
print instance_of_ClassI.a, instance_of_ClassI.b

# This will cause an error, because to access instance variables, you need an instance of class!
print ClassI.a


I am working on the file "Lesson_03_Characteristics.ipynb".
The first task is:
Alternatively the characteristics could separate. In this case the red shaded region is not determined by the initial data at all.
Try modifying the initial data and see what the results are.
Can you write Python code for it?

from ipywidgets import interactive, FloatSlider
def interactive_characteristics(ql, qr):
    return plot_characteristics(df_burgers, ql, qr)
interactive(interactive_characteristics,
            ql = FloatSlider(min=-2.0, max=2.0, step=0.1, value=-0.1),
            qr = FloatSlider(min=-2.0, max=2.0, step=0.1, value=0.2))


I am working on the file "Lesson_03_Characteristics.ipynb".
The first task is:
As we move from the left state $q_l = 1$ to smaller values of $q$ we see that the characteristic coordinate $\xi = f'(q)$ increases, as it should. Therefore the first section of the compound wave could be a rarefaction. However, we cannot get all the way to $q_r = 0$, as after the maximum in $f'$ the characteristic coordinate would start to decrease again.
So, at some point $q_$ the wave must stop being a rarefaction and join to a discontinuity. In order for the characteristics to make sense, they must join, not intersect (as at a shock). That is, the shock speed must match the characteristic speed at $q_$.
So our first attempt at a solution must try to find a state $q_$ such that, when connected to the right state $q_r = 0$ using a discontinuity satisfying the Rankine-Hugoniot conditions, the shock speed is $s = f'(q_)$. This requires that
$$
\begin{align}
  && \frac{q_r^2}{q_r^2 + a (1 - q_r)^2} - \frac{q_^2}{q_^2 + a (1 - q_)^2} & = \frac{2 a q_ (1 - q_)}{\left[ q_^2 + a (1 - q_)^2 \right]^2} \left( q_r - q_ \right) \
  \implies && \frac{q_^2}{q_^2 + a (1 - q_)^2} & = \frac{2 a q_^2 (1 - q_)}{\left[ q_^2 + a (1 - q_)^2 \right]^2} \
  \implies && q_^2 + a (1 - q_)^2 & = 2 a  (1 - q_) \
  \implies && q_* & = \pm \sqrt{\frac{a}{1+a}}.
\end{align}
$$
The characteristic picture for this shows a rarefaction attached to a non-classical shock. The shock is not the standard shock, as the characteristics only converge on it from one side.
Can you write Python code for it?

def flux_buckley_leverett(q):
    return q**2 / (q**2 + a * (1.0 - q)**2)

def df_buckley_leverett(q):
    return 2.0 * a * q * (1.0 - q) / (q**2 + a * (1.0 - q)**2)

fig = plt.figure(figsize=(12,6))
ax = fig.add_subplot(111)

a = 0.5
ql = 1.0
qr = 0.0
qs = np.sqrt(a / (1.0 + a))
s = df_buckley_leverett(qs) # Shock speed matches characteristic speed on left!

slope_l = df_buckley_leverett(ql)
slope_r = df_buckley_leverett(qr)

left_edge = np.min([-1.0, -1.0 - slope_l])
right_edge = np.max([1.0, 1.0 - slope_r])
x_start_points_l = np.linspace(left_edge, 0.0, 20)
x_start_points_r = np.linspace(0.0, right_edge, 20)
x_end_points_l = x_start_points_l + slope_l
x_end_points_r = x_start_points_r + slope_r

# Rarefaction wave
xi_l = df_buckley_leverett(ql)
xi_r = df_buckley_leverett(qs)
xi = np.linspace(xi_l, xi_r, 7)
x_end_rarefaction = xi

# Shock intersections
t_end_points_r = np.ones_like(x_start_points_r)
t_end_points_r = np.minimum(t_end_points_r, x_start_points_r / (s - slope_r))

for xs, xe in zip(x_start_points_l, x_end_points_l):
    ax.plot([xs, xe], [0.0, 1.0], 'b-')
for xs, xe, te in zip(x_start_points_r, x_end_points_r, t_end_points_r):
    ax.plot([xs, xe], [0.0, te], 'g-')
for xe in x_end_rarefaction:
    ax.plot([0.0, xe], [0.0, 1.0], 'r--')
# Shock
ax.plot([0.0, s], [0.0, 1.0], 'k-', linewidth=3)

ax.set_xbound(-1.0, 1.0)
ax.set_ybound(0.0, 1.0)
ax.set_xlabel(r"$x$")
ax.set_ylabel(r"$t$")

fig.tight_layout()


I am working on the file "Lesson_03_Characteristics.ipynb".
The first task is:
Exercise
Again, repeat this calculation for the LWR traffic equation.
Solution to the Riemann Problem
For Burger's equation, these two cases are sufficient for a complete solution of the Riemann problem. If $q_l < q_r$ then we have a rarefaction. If $q_l > q_r$ then we have a shock. If $q_l = q_r$ then the solution is trivial (although care should be taken in the code!).
Can you write Python code for it?

def plot_solution_burgers(ql, qr):
    """Plot the characteristic lines for the full solution to Burger's equation."""

    fig = plt.figure(figsize=(12,6))
    ax1 = fig.add_subplot(121)

    dq = np.max([abs(qr-ql), 0.1])
    qmin = np.min([ql, qr])
    qmax = np.max([ql, qr])
    ax1.plot([-1.0, 0.0], [ql, ql], 'b-')
    ax1.plot([0.0, 1.0], [qr, qr], 'g-')
    ax1.set_xbound(-1.0, 1.0)
    ax1.set_ybound(qmin - 0.1*dq, qmax + 0.1*dq)
    ax1.set_xlabel(r"$x$")
    ax1.set_ylabel(r"$q(x, t=0)$")

    ax2 = fig.add_subplot(122)

    slope_l = ql
    slope_r = qr

    if (ql <= qr):
        # Rarefaction case

        left_edge = np.min([-1.0, -1.0 - slope_l])
        right_edge = np.max([1.0, 1.0 - slope_r])
        x_start_points_l = np.linspace(left_edge, 0.0, 20)
        x_start_points_r = np.linspace(0.0, right_edge, 20)
        x_end_points_l = x_start_points_l + slope_l
        x_end_points_r = x_start_points_r + slope_r

        # Rarefaction wave
        xi_l = ql
        xi_r = qr
        xi = np.linspace(xi_l, xi_r, 7)
        x_end_rarefaction = xi

        for xs, xe in zip(x_start_points_l, x_end_points_l):
            ax2.plot([xs, xe], [0.0, 1.0], 'b-')
        for xs, xe in zip(x_start_points_r, x_end_points_r):
            ax2.plot([xs, xe], [0.0, 1.0], 'g-')
        for xe in x_end_rarefaction:
            ax2.plot([0.0, xe], [0.0, 1.0], 'r--')

    else:
        # Shock case
        s = 0.5 * (qr + ql)
        ax2.plot([0.0, s], [0.0, 1.0], 'k-', linewidth=3)

        left_edge = np.min([-1.0, -1.0 - slope_l])
        right_edge = np.max([1.0, 1.0 - slope_r])
        x_start_points_l = np.linspace(left_edge, 0.0, 20)
        x_start_points_r = np.linspace(0.0, right_edge, 20)
        t_end_points_l = np.ones_like(x_start_points_l)
        t_end_points_r = np.ones_like(x_start_points_r)

        # Look for intersections
        t_end_points_l = np.minimum(t_end_points_l, x_start_points_l / (s - slope_l))
        t_end_points_r =...


I am working on the file "Lesson_03_Characteristics.ipynb".
The first task is:
Exercise
Try repeating this characteristic analysis using different scalar problems, such as the LWR traffic equation
\begin{equation}
  \partial_t q + \partial_x \left( q (1 - q) \right) = 0.
\end{equation}
Shocks
When the characteristics cross, we have to find a weak solution that gives a unique self-similar solution. A shock satisfying the Rankine-Hugoniot conditions is such a solution.
For Burger's equation we can solve this explicitly. We have
$$
\begin{align}
  && {\bf f}({\bf q}_r) - {\bf f}({\bf q}_l) & = s \left( {\bf q}_r - {\bf q}_l \right) \
  \implies && \frac{1}{2} \left( q_r^2 - q_l^2 \right) & = s \left( q_r - q_l \right) \
  \implies && s & = \frac{1}{2} \frac{ q_r^2 - q_l^2 }{ q_r - q_l } \
  && & = \frac{q_r + q_l}{2}.
\end{align}
$$
Given the shock speed, we then have the explicit solution to the Riemann problem for Burger's equation in the case of a shock, and when it occurs:
$$
  q_l > q_r \quad \implies \quad q(x, t) = \begin{cases} q_l & x < s t \ q_r & x > s t \end{cases}
$$
where $s = (q_r + q_l) / 2$.
Can you write Python code for it?

def plot_shock_burgers(ql, qr):
    """Plot the characteristic lines for the shock solution to Burger's equation."""

    assert(ql > qr)

    fig = plt.figure(figsize=(12,6))
    ax1 = fig.add_subplot(121)

    dq = np.max([abs(qr-ql), 0.1])
    qmin = np.min([ql, qr])
    qmax = np.max([ql, qr])
    ax1.plot([-1.0, 0.0], [ql, ql], 'b-')
    ax1.plot([0.0, 1.0], [qr, qr], 'g-')
    ax1.set_xbound(-1.0, 1.0)
    ax1.set_ybound(qmin - 0.1*dq, qmax + 0.1*dq)
    ax1.set_xlabel(r"$x$")
    ax1.set_ylabel(r"$q(x, t=0)$")

    ax2 = fig.add_subplot(122)

    slope_l = ql
    slope_r = qr
    s = 0.5 * (qr + ql)
    ax2.plot([0.0, s], [0.0, 1.0], 'k-', linewidth=3)

    left_edge = np.min([-1.0, -1.0 - slope_l])
    right_edge = np.max([1.0, 1.0 - slope_r])
    x_start_points_l = np.linspace(left_edge, 0.0, 20)
    x_start_points_r = np.linspace(0.0, right_edge, 20)
    t_end_points_l = np.ones_like(x_start_points_l)
    t_end_points_r = np.ones_like(x_start_points_r)

    # Look for intersections
    t_end_points_l = np.minimum(t_end_points_l, x_start_points_l / (s - slope_l))
    t_end_points_r = np.minimum(t_end_points_r, x_start_points_r / (s - slope_r))
    x_end_points_l = x_start_points_l + slope_l * t_end_points_l
    x_end_points_r = x_start_points_r + slope_r * t_end_points_r

    for xs, xe, te in zip(x_start_points_l, x_end_points_l, t_end_points_l):
        ax2.plot([xs, xe], [0.0, te], 'b-')
    for xs, xe, te in zip(x_start_points_r, x_end_points_r, t_end_points_r):
        ax2.plot([xs, xe], [0.0, te], 'g-')

    ax2.set_xbound(-1.0, 1.0)
    ax2.set_ybound(0.0, 1.0)
    ax2.set_xlabel(r"$x$")
    ax2.set_ylabel(r"$t$")

    fig.tight_layout()

interactive(plot_shock_burgers,
            ql = FloatSlider(min=-2.0, max=2.0, step=0.1, value=0.1),
            qr = FloatSlider(min=-2.0, max=2.0, step=0.1, value=-0.2))


I am working on the file "Lesson_03_Characteristics.ipynb".
The first task is:
In the red shaded area the characteristics cross - there is characteristic focusing. The solution is not determined here as it could take the value of the initial data either from the left or from the right, which would give different values.
Can you write Python code for it?


I am working on the file "Lesson_03_Characteristics.ipynb".
The first task is:
This is a problem, as within a rarefaction wave we have to solve $\xi = f'(q)$. More formally we invert the derivative so that, given the characteristic coordinate $\xi$ we have the solution
\begin{equation}
  q = \left( f' \right)^{-1} (\xi)
\end{equation}
But clearly here there are two possible solutions!
This will be a problem for any nonlinear problem where the second derivative, $f''$, vanishes. For problems such as these, we need to look again at the characteristic structure to see what an allowable solution should be.
Compound waves
We'll consider the Riemann problem for the Buckley-Leverett problem with $q_l = 1$ and $q_r = 0$.
First let's look at the characteristic speeds of the left and right states. Given $f'$ as above we see that $\xi_l = 0 = \xi_r$. So the characteristics are parallel. At first glance we would not expect either a shock or a rarefaction, but a linear discontinuity. However, such a discontinuity would satisfy the Rankine-Hugoniot conditions with the same speed as the neighbouring characteristics - here 0. Let us check:
$$
\begin{align}
  && f_r - f_l & = s \left( q_r - q_l \right) \
  \implies && s & = \frac{0 - 1}{0 - 1} \
  && & = 1.
\end{align}
$$
This cannot be right. If the left and right states were separated by a single discontinuity then it would have to travel to the right with speed 1, and points in the $x-t$ plane with $0 < x < t$ would therefore not be determined.
However, neither can we have a single rarefaction. The left and right characteristics speeds would give the boundaries of such a rarefaction, and here they are both $0$.
The alternative is a compound wave: one that is made up of more than one piece, separating rarefactions with discontinuities satisfying the Rankine-Hugoniot conditions.
To construct a solution, let us start from the left state $q_l = 1$. We know that we want to connect to the state at $q_r = 0$, so we want $q$ to decrease as we move along the wave curve.
Recall the behaviour of the flux and the characteristic speed $\xi = f'(q)$:
Can you write Python code for it?

a = 0.5
q = np.linspace(0.0, 1.0)
f = q**2 / (q**2 + a * (1.0 - q)**2)
df = 2.0 * a * q * (1.0 - q) / (q**2 + a * (1.0 - q)**2)
fig = plt.figure(figsize=(12,6))
ax1 = fig.add_subplot(121)
ax1.plot(q, f)
ax1.set_xlabel(r"$q$")
ax1.set_ylabel(r"$f$")
ax2 = fig.add_subplot(122)
ax2.plot(q, df)
ax2.set_xlabel(r"$q$")
ax2.set_ylabel(r"$\partial f / \partial q$")
fig.tight_layout()
plt.show()


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
- While
Will run until the condition is satisfied. Will stop on exception, or when loop is exited.
Can you write Python code for it?


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
Example interview task - folder crawler
Task is to create a program, that will print out contents of a folder, recursively down the folder structure.
Requirements

use Python module "os"
go down N floors of folders - it is an input parameter
must accept absolute paths
Can you write Python code for it?

import os

def print_current_path(path, level):
    print("level: {0}, path: {1}".format(level, path))

def folder_crawler(starting_path, level_limit, current_level=0):
        if current_level > level_limit:
            return

        print_current_path(starting_path, current_level)

        try:
            contents = os.listdir(starting_path)

            for item in contents:
                item_path = os.path.join(starting_path, item)
                print_current_path(item_path, current_level)
                if os.path.isdir(item_path):
                    folder_crawler(item_path, level_limit, current_level + 1)
        except PermissionError:
            print("Permission denied. Skipping")

folder_crawler(r"C:\Users", 3)


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
Exceptions general purpose and definition
An exception is an event, which occurs during the execution of a program, that disrupts the normal flow of the program's instructions.
Anomalous or exceptional conditions requiring special processing – often changing the normal flow of program execution
Exception handling - the process of responding to occurence of exceptions
Handled exception - exception, that arose during normal flow of program, but was caught - the program went on
Unhandled exception - exception, that arose during normal flow of program, but was uncaught - it crashed the whole process
Types of exceptions
Exceptions come in hundreds of flavours, and you can also write your own kinds.
Exceptions main role is to signal that a terrible or unexpected situation happened, and there is just no way of going on with program flow.
Most popular Python exception types:
Can you write Python code for it?

# IndexError
a = [1,2,3]
print(a[4])

# KeyError
a = {"something": 1}
a["something_else"]

# ModuleNotFoundError

import whatever

# NameError
print(for_sure_I_dont_exist)

# TypeError
int([a,a,a])

# ValueError
int("a")


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
Extend example by yourself - if it's a text file, print out the first line of this file to the screen.
folder_crawler evolves into folder_spy!
Can you write Python code for it?

a = ["a", "d", "b", "c"]

for letter in a:
    if letter == "d":
        print("d was the number: " + str(a.index(letter)))

enumerate(a)

for index, letter in enumerate(a):
    if letter == "d":
        print(str(index))

a = ["a", "b", "c"]
b = [3342,4554,334]
dict(zip(a,b))

dict(enumerate(a))

def bla():
    return [1,2,3,4]

x, y, z, w = bla()
x

x, y = (1,3)

print(y)

a = 7
b = 4

c = a
a = b
b = c

a, b = b, (a+12)

range(0,88)

a = [1,2,3,4,5,6,7,8]
sum(a)

def add_number(number):
    return number + 30

new_list = map(add_number, a)

print(new_list)

for i in new_list:
    print(i)

def some_filter(x):
    if x > 4:
        return True
    return False

new_list = filter(some_filter, a)

print(new_list)

for i in new_list:
    print(i)

new_list = filter((lambda x: x > 4), a)

another_list = map((lambda  x: x + 30), a)

reduced_list = reduce((lambda x,a: a + x), a)

sum = 0
[x+1 for x in [1,2,1,1]  if x < 2]

sum


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
Flow control
Sometimes, we do not want to do all loop iterations. Sometimes, we want to:
continue - Skip this whole loop iteration, from this moment, and go to next loop iteration
Can you write Python code for it?

for i in range(0,4):
    if i == 2:
        continue
    print(i)


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
It's not exactly right, isn't it? Why are there 4 repetitions? It's because we start from 0. 0 divided by integer>0 gives always 0 remainder.
Can you write Python code for it?

0 % 3

# The fixed smart way.
for i in range(1,11):
    if i % 3 == 0:
        print("We did something every 3-rd time")

# The FizzBuzz
for i in range(1,101):
    if i % 15 == 0:
        print("FizzBuzz")
    elif i % 5 == 0:
        print("Buzz")
    elif i % 3 == 0:
        print("Fizz")


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
break - Skip this whole loop iteration, from this moment, and do not do any more loop iterations
Can you write Python code for it?

for i in range(0,100):
    if i == 2:
        break
    print(i)


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
catch Exception - ignore it, and go on as if nothing happened
Can you write Python code for it?

for i in range(0,10):
    try:
        if i == 5:
            raise Exception("I hate fives.")
    except Exception as error_message:
        print("Stop hate! Just go on. Details: " + str(error_message))
    print(i)


I am working on the file "Lesson_03_loops_flow_control_exceptions.ipynb".
The first task is:
return - skip this whole loop iteration, and exit this scope (for example, method), not doing any more iterations
Can you write Python code for it?

def print_a_lot_of_numbers_but_exit_on(number):
    N = 1000
    for i in range(0,N):
        if i == number:
            return i
        print(i)
    # Notice, how we return the last number. Otherwise, on 1000, it would return None automatically - Python feature.
    return N

print_a_lot_of_numbers_but_exit_on(1)

print_a_lot_of_numbers_but_exit_on(1000)


I am working on the file "Lesson_04_comprehensions.ipynb".
The first task is:
Dictionary comprehensions
This patterns allow for creating dictionaries using other lists or dictionaries or to transform a dictionary.
Can you write Python code for it?

# Create dictionary from two lists.

d = {a:b for a,b in zip(['a', 'b', 'c'], range(0,3))}
print(d)

# Transform dictionary.

transformed = {key: value*2 for (key,value) in d.items()}
print(transformed)

transformed_keys = {key*2: value for (key,value) in d.items()}
print(transformed_keys)

# Conditions
d = {a:b for a,b
         in zip(['a', 'b', 'c'], range(0,3))
         if a != 'c'
         and b > 0}
print(d)

# Do you have a complex condition using if-else? No problem!

d = {a:(str(b) + " is even"
        if b%2==0
        else str(b) + " is odd")
     for a,b
     in zip(['a', 'b', 'c', 'd'], range(0,4))}
print(d)

# Using just one list
d = {a:a**2 for a in range(0,3)}
print(d)

# Nested comprehensions
# Transform a nested dictionary into a different nested dictionary.

nested_dict = {'something':{'a':1}, 'something_else':{'b':2}}
squared = {k: {k2 + " " + str(v2**2)
               for (k2, v2)
               in v.items()}
           for (k, v)
           in nested_dict.items()}
print(squared)


I am working on the file "Lesson_04_comprehensions.ipynb".
The first task is:
Examples from real life
Can you write Python code for it?

# Flatten a matrix

matrix = [range(0,5), range(5,10), range(10,15)]

flatten = [column for row in matrix for column in row]
print(flatten)

# Removing a set of characters from a string

forbidden_chars = ["x", "z", "u"]
some_string = "Mr Garlax Underwood was a good man, but underpaid"
removed_list = [char for char in some_string if char.lower() not in forbidden_chars]
removed_as_string = "".join(removed_list)
print(removed_as_string)


I am working on the file "Lesson_05_python_idioms_and_style.ipynb".
The first task is:
Basics of itertools
These tools provide some nice operations you can do on collections.
Can you write Python code for it?

import itertools

# List all the different sequences of a starting list

permutations = itertools.permutations([1,2,3])
print(list(permutations))

# Cycle constantly through a short sequence
from itertools import cycle

counter = 20
for item in cycle('Adamek'):
    if counter > 0:
        print(item)
        counter -= 1


I am working on the file "Lesson_05_python_idioms_and_style.ipynb".
The first task is:
Calling a function until something happens
Can you write Python code for it?

# Lets write a simple file
import os

filename = 'example.txt'

try:
    os.remove(filename)
except OSError:
    pass

with open('example.txt', 'w+') as f:
    [f.write(str(x) + "\n") for x in range(0,20)]

# Bad way
with open('example.txt', 'r') as f:
    while True:
        line = f.readline()
        if line == '':
            break
        print(line)

# Better way
with open('example.txt', 'r') as f:
    for line in iter(f.readline, ''):
        print(line)


I am working on the file "Lesson_05_python_idioms_and_style.ipynb".
The first task is:
Loop forwards and backwards through a list
Can you write Python code for it?

cars = ['ford', 'volvo', 'chevrolet']

# This is bad
for i in range(len(cars)): print(cars[i])

# This is better
for car in cars: print(car)

# Reversed
for car in reversed(cars): print(car)


I am working on the file "Lesson_05_python_idioms_and_style.ipynb".
The first task is:
Loop over a list AND the indexes at the same time
Can you write Python code for it?

# I want to know the index of an item inside iteration

# This is bad
for i in range(len(cars)):
    print(str(i) + " " + cars[i])

# This is better
for i, car in enumerate(cars): print(str(i) + " " + car)


I am working on the file "Lesson_05_python_idioms_and_style.ipynb".
The first task is:
Looping over dictionary keys and values at the same time
Can you write Python code for it?

dictionary = {k:v for k,v in zip(range(0,3), range(0,3))}

# Bad Way
for k in dictionary.keys():
    print(k, dictionary[k])

# Much better way
for k, v in dictionary.items():
    print(k, v)