I am working on the file "Author_Classification/bag.ipynb".
The first task is:
The dataset contains 1028 entries (tuples) and 2 columns as described above.
Splitting Training and testing sets
Spliting the dataset into a training and testing set by using the train_test_split method in sklearn.
Spliting the data by using the following variables:

-> X_train is our training data for the 'textData' column.

-> y_train is our training data for the 'rollNo' column

-> X_test is our testing data for the 'textData' column.

-> y_test is our testing data for the 'rollNo' column.

Can you write Python code for it?

# split into training and testing sets
# USE from sklearn.model_selection import train_test_split to avoid seeing deprecation warning.
from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(df['textData'],
                                                    df['rollNo'],
                                                    random_state=1)

# Printing out the number of rows we have in each our training and testing data.
print('Number of rows in the total set: {}'.format(df.shape[0]))
print('Number of rows in the training set: {}'.format(X_train.shape[0]))
print('Number of rows in the test set: {}'.format(X_test.shape[0]))


I am working on the file "AutoCorrelation.ipynb".
The first task is:
DW=1.094となり、上限分布において有意水準5%でも帰無仮説を棄却することができ、自己相関が存在すると結論することができる。
例題5-3
変数変換によって自己相関に対処する方法を考える。
Can you write Python code for it?

data['dX'] = np.nan
data['dY'] = np.nan
for i in range(len(data)):
    if i == 0:
        data['dX'][i] = np.nan
        data['dY'][i] = np.nan
    else:
        data['dX'][i] = data['X'][i] - data['X'][i-1]
        data['dY'][i] = data['Y'][i] - data['Y'][i-1]
data

# 説明変数設定
X = data['dX'][1:]
X = sm.add_constant(X)
# 被説明変数設定
Y = data['dY'][1:]
# OLSの実行(Ordinary Least Squares: 最小二乗法)
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())
# グラフ生成
plt.plot(data["dX"], data["dY"], 'o', label="data")
plt.plot(data["dX"][1:], results.fittedvalues, label="OLS")
plt.xlim(min(data["dX"][1:])-0.5, max(data["dX"][1:])+0.5)
plt.ylim(min(data["dY"][1:])-0.5, max(data["dY"][1:])+0.5)
plt.title('5-3: Auto Correlation')
plt.legend(loc=2)
plt.show()


I am working on the file "AutoCorrelation.ipynb".
The first task is:
Durbin-Watson: 2.029となり、2に近い値であり、自己相関がないと結論出来る。
DW統計量に対応するP値を求めることがPythonではおそらくできない(Excelでも出来ない)ため、ダービン=ワトソン統計量の分布表を利用して有意性の検定を行います。
この場合、標本の大きさ=10, 定数項を除いた説明変数の数=1なので、有意水準1%において、下限分布ではDW=0.604, 上限分布ではDW=1.001が有意となる境界、有意水準5%において、下限分布ではDW=0.879, 上限分布ではDW=1.320が有意となる境界です。(詳しくは本書の表を参考にしてください。)
例題5-2
Can you write Python code for it?

# データ読み込み
data = pd.read_csv('example/k0502.csv')
data

# 説明変数設定
X = data[['X']]
X = sm.add_constant(X)
# 被説明変数設定
Y = data['Y']
# OLSの実行(Ordinary Least Squares: 最小二乗法)
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())
# グラフ生成
plt.plot(data["X"], data["Y"], 'o', label="data")
plt.plot(data["X"], results.fittedvalues, label="OLS")
plt.xlim(min(data["X"])-1, max(data["X"])+1)
plt.ylim(min(data["Y"])-1, max(data["Y"])+1)
plt.title('5-2: Auto Correlation')
plt.legend(loc=2)
plt.show()


I am working on the file "AutoGIS/02-Geometric-Objects-Spatial-Data-Model.ipynb".
The first task is:
.xy는 x, y를 tuple로 리턴하고 .x는 x 리턴

.distance로 거리를 구할 수 있음

이 거리는 좌표계에 기반한 거리임
Can you write Python code for it?

point_dist = point1.distance(point2)

print("Distance between the points is {0:.2f} decimal degrees".format(point_dist))


I am working on the file "AutoGIS/02-Geometric-Objects-Spatial-Data-Model.ipynb".
The first task is:
Geometry Collections

Y모양의 선 형상(도로) 또는 다중 다각형(섬)은 MultiLineString 또는 MultiPolygon을 적절하게 사용해 표현할 수 있음
Can you write Python code for it?

from shapely.geometry import MultiPoint, MultiLineString, MultiPolygon, box

# Create a MultiPoint object of our points 1,2 and 3
multi_point = MultiPoint([point1, point2, point3])

# It is also possible to pass coordinate tuples inside
multi_point2 = MultiPoint([(2.2, 4.2), (7.2, -25.1), (9.26, -2.456)])

# We can also create a MultiLineString with two lines
line1 = LineString([point1, point2])
line2 = LineString([point2, point3])
multi_line = MultiLineString([line1, line2])

# MultiPolygon can be done in a similar manner
# Let's divide our world into western and eastern hemispheres with a hole on the western hemisphere
# --------------------------------------------------------------------------------------------------

# Let's create the exterior of the western part of the world
west_exterior = [(-180, 90), (-180, -90), (0, -90), (0, 90)]

# Let's create a hole --> remember there can be multiple holes, thus we need to have a list of hole(s).
# Here we have just one.
west_hole = [[(-170, 80), (-170, -80), (-10, -80), (-10, 80)]]

# Create the Polygon
west_poly = Polygon(shell=west_exterior, holes=west_hole)

# Let's create the Polygon of our Eastern hemisphere polygon using bounding box
# For bounding box we need to specify the lower-left corner coordinates and upper-right coordinates
min_x, min_y = 0, -90
max_x, max_y = 180, 90

# Create the polygon using box() function
east_poly_box = box(minx=min_x, miny=min_y, maxx=max_x, maxy=max_y)

# Let's create our MultiPolygon. We can pass multiple Polygon -objects into our MultiPolygon as a list
multi_poly = MultiPolygon([west_poly, east_poly_box])

# Print outputs
print("MultiPoint:", multi_point)
print("MultiLine: ", multi_line)
print("Bounding box: ", east_poly_box)
print("MultiPoly: ", multi_poly)


I am working on the file "AutoGIS/02-Geometric-Objects-Spatial-Data-Model.ipynb".
The first task is:
LineString
Can you write Python code for it?

# Create a LineString from our Point objects
line = LineString([point1, point2, point3])

# It is also possible to use coordinate tuples having the same outcome
line2 = LineString([(2.2, 4.2), (7.2, -25.1), (9.26, -2.456)])

# Print the results
print("line variable: \n", line, "\n")
print("line2 variable: \n", line2, "\n")
print("type of the line: \n", type(line))


I am working on the file "AutoGIS/02-Geometric-Objects-Spatial-Data-Model.ipynb".
The first task is:
Polygon 속성과 함수

.area, .centroid, .bounds, .exteror, .length 등
Can you write Python code for it?

# Get the centroid of the Polygon
world_centroid = world.centroid

# Get the area of the Polygon
world_area = world.area

# Get the bounds of the Polygon (i.e. bounding box)
world_bbox = world.bounds

# Get the exterior of the Polygon
world_ext = world.exterior

# Get the length of the exterior
world_ext_length = world_ext.length

# Print the outputs
print("Poly centroid: ", world_centroid)
print("Poly Area: ", world_area)
print("Poly Bounding Box: ", world_bbox)
print("Poly Exterior: ", world_ext)
print("Poly Exterior Length: ", world_ext_length)


I am working on the file "AutoGIS/02-Geometric-Objects-Spatial-Data-Model.ipynb".
The first task is:
Polyon
Can you write Python code for it?

# Create a Polygon from the coordinates
poly = Polygon([(2.2, 4.2), (7.2, -25.1), (9.26, -2.456)])

# We can also use our previously created Point objects (same outcome)
# --> notice that Polygon object requires x,y coordinates as input
poly2 = Polygon([[p.x, p.y] for p in [point1, point2, point3]])

# Geometry type can be accessed as a String
poly_type = poly.geom_type

# Using the Python's type function gives the type in a different format
poly_type2 = type(poly)

# Let's see how our Polygon looks like
print('poly:', poly)
print('poly2:', poly2)
print("Geometry type as text:", poly_type)
print("Geometry how Python shows it:", poly_type2)

help(Polygon)


I am working on the file "AutoGIS/02-Geometric-Objects-Spatial-Data-Model.ipynb".
The first task is:
length나 중심점을 추출할 수 있음
Can you write Python code for it?

# Get the lenght of the line
l_length = line.length

# Get the centroid of the line
l_centroid = line.centroid

# What type is the centroid?
centroid_type = type(l_centroid)

# Print the outputs
print("Length of our line: {0:.2f}".format(l_length))
print("Centroid of our line: ", l_centroid)
print("Type of the centroid:", centroid_type)


I am working on the file "AutoGIS/02-Geometric-Objects-Spatial-Data-Model.ipynb".
The first task is:
line은 multiple 좌표 쌍을 가지고 있음

LineString 속성과 함수

.xy로 x, y를 리턴
Can you write Python code for it?

lxy = line.xy

print(lxy)

# Extract x coordinates
line_x = lxy[0]

# Extract y coordinates straight from the LineObject by referring to a array at index 1
line_y = line.xy[1]

print('line_x:\n', line_x, '\n')

print('line_y:\n', line_y)


I am working on the file "AutoMPG/AutoMPG.ipynb".
The first task is:

The variance score is very low due to we use all the features.

For a better solution we aplied a feature selection.

From the pair graph, we can see that the features 'cylinders', 'model year' and 'origin' dont'show a clear correlation with the 'mpg' variable, so those are rejected.

Can you write Python code for it?

c = ['mpg','horsepower', "weight",'acceleration']
limit = int(3*cars.shape[0]/4)
X_train = cars[c].iloc[:limit,1:]
y_train = cars.iloc[:limit,0]
X_test = cars[c].iloc[limit:,1:]
y_test = cars.iloc[limit:,0]

lr2 = LinearRegression()

#Remember the [[]], is the data in 1 column
lr2.fit(X_train, y_train)
predictions = lr2.predict(X_test)

#mean_squared_error
mse = mean_squared_error(y_test, predictions)

# The coefficients
print('Coefficients: ', lr2.coef_)
# The mean square error
print("Residual sum of squares: %.2f" % mse)
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % lr2.score(X_test, y_test))

feature = "weight"

plt.scatter(X_test[feature], y_test, c='b')
plt.scatter(X_test[feature], prediction, c='r')

plt.ylim([10,45])
plt.xlim([1500,4000])
plt.ylabel('MPG')
plt.xlabel(feature)
plt.show()


I am working on the file "Autoencoder/VAE.ipynb".
The first task is:
Generate digits from two random numbers:
Can you write Python code for it?

#Pick two random values (rather small)
z1=1
z2=3

z_sample = np.array([[z1, z2]])

#Let the decoder-network output
x_decoded = decoder.predict(z_sample)

#Display it
digit = x_decoded[0].reshape(28, 28)
plt.imshow(digit)
plt.gray()
plt.show()


I am working on the file "Autoencoder/VAE.ipynb".
The first task is:
How are the inputs encoded?
This piece of code takes the coordinates z1 and z2 of all 10000 values of x_test and plots them
Can you write Python code for it?

#Let the encoder evaluate all z's at once
x_test_encoded = encoder.predict(x_test, batch_size=m)

#Scatter them
plt.rcParams.update(plt.rcParamsDefault) #restore default setting (else it is gray)
plt.figure(figsize=(10, 10))
plt.scatter(x_test_encoded[:, 0], x_test_encoded[:, 1], c=y_test[:]) #y_test is needed for the colormap
plt.colorbar()
plt.show()


I am working on the file "Autoencoder/VAE.ipynb".
The first task is:
To what outputs do which z's lead?
This piece of code composes an image out of the outputs of equally spaced values of z between -4 and 4
Can you write Python code for it?

# sample n points within [-4, 4] standard deviations, because as seen above most of the points lie there
z1_array = np.linspace(-4, 4, n)
z2_array = np.linspace(-4, 4, n)

n = 15# figure with 15x15 digits
digit_size = 28
figure = np.zeros((digit_size * n, digit_size * n))#init figure with zeros

for i, yi in enumerate(z2_array):
    for j, xi in enumerate(z1_array):
        z_sample = np.array([[xi, -yi]]) # (-1)*yi that the axes of the manifold are the same as the scatter
        x_decoded = decoder.predict(z_sample)
        digit = x_decoded[0].reshape(digit_size, digit_size)
        figure[i * digit_size : (i + 1) * digit_size,
               j * digit_size : (j + 1) * digit_size] = digit # add the digit a the right place to the figure

plt.figure(figsize=(10, 10))
plt.imshow(figure)
plt.gray()
plt.show()


I am working on the file "Autoencoder/VAE.ipynb".
The first task is:
Train VAE on MNIST digits
Can you write Python code for it?

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))

vae.compile(optimizer='adam', loss=vae_loss)
vae.fit(x_train, x_train, batch_size=m, epochs=n_epoch)


I am working on the file "Autoencoders/Simple_Autoencoder.ipynb".
The first task is:
Below I'm plotting an example image from the MNIST dataset. These are 28x28 grayscale images of handwritten digits.
Can you write Python code for it?

img = mnist.train.images[2]
plt.imshow(img.reshape((28, 28)), cmap='Greys_r')


I am working on the file "Autoencoders/Simple_Autoencoder.ipynb".
The first task is:
Training
Can you write Python code for it?

# Create the session
sess = tf.Session()


I am working on the file "Autoencoders/Simple_Autoencoder.ipynb".
The first task is:
We'll train an autoencoder with these images by flattening them into 784 length vectors. The images from this dataset are already normalized such that the values are between 0 and 1. Let's start by building basically the simplest autoencoder with a single ReLU hidden layer. This layer will be used as the compressed representation. Then, the encoder is the input layer and the hidden layer. The decoder is the hidden layer and the output layer. Since the images are normalized between 0 and 1, we need to use a sigmoid activation on the output layer to get values matching the input.

Exercise: Build the graph for the autoencoder in the cell below. The input images will be flattened into 784 length vectors. The targets are the same as the inputs. And there should be one hidden layer with a ReLU activation and an output layer with a sigmoid activation. The loss should be calculated with the cross-entropy loss, there is a convenient TensorFlow function for this tf.nn.sigmoid_cross_entropy_with_logits (documentation). You should note that tf.nn.sigmoid_cross_entropy_with_logits takes the logits, but to get the reconstructed images you'll need to pass the logits through the sigmoid function.
Can you write Python code for it?

# Size of the encoding layer (the hidden layer)
encoding_dim = 32 # feel free to change this value

inputs_ =
targets_ =
# Output of hidden layer
encoded =

# Output layer logits
logits =
# Sigmoid output from logits
decoded =

# Sigmoid cross-entropy loss
loss =
# Mean of the loss
cost =

# Adam optimizer
opt =


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Autoencoder
Can you write Python code for it?

##
tf.reset_default_graph()

n_inputs = 3
n_hidden = 2  # codings
n_outputs = n_inputs

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
hidden = fully_connected(X, n_hidden, activation_fn=None)
outputs = fully_connected(hidden, n_outputs, activation_fn=None)

mse = tf.reduce_mean(tf.square(outputs - X))
optimizer = tf.train.AdamOptimizer(learning_rate=0.01)
training_op = optimizer.minimize(mse)

init = tf.global_variables_initializer()

##
n_iterations = 10000
codings = hidden

with tf.Session() as sess:
    init.run()
    for iteration in range(n_iterations):
        training_op.run(feed_dict={X: X_train})
    codings_val = codings.eval(feed_dict={X: X_train})

fig = plt.figure(figsize=(4,3))
plt.plot(codings_val[:,0], codings_val[:, 1], "b.")
plt.xlabel("$z_1$", fontsize=18)
plt.ylabel("$z_2$", fontsize=18, rotation=0)
plt.show()


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Finally, we can create a Stacked Autoencoder by simply reusing the weights and biases from the Autoencoders we just trained:
Can you write Python code for it?

tf.reset_default_graph()

n_inputs = 28*28

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
hidden1 = tf.nn.elu(tf.matmul(X, W1) + b1)
hidden2 = tf.nn.elu(tf.matmul(hidden1, W2) + b2)
hidden3 = tf.nn.elu(tf.matmul(hidden2, W3) + b3)
outputs = tf.matmul(hidden3, W4) + b4

show_reconstructed_digits(X, outputs)


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Highlights:

weight3 and weights4 are not variables, they are respectively the transpose of weights2 and weights1 (they are “tied” to them).
Since they are not variables, it’s no use regularizing them: we only regularize weights1 and weights2.
Biases are never tied, and never regularized.

Training one Autoencoder at a time in multiple graphs
Rather than training the whole stacked autoencoder in one go like we just did, it is often much faster to train one shallow autoencoder at a time, then stack all of them into a single stacked autoencoder. This is especially useful for very deep autoencoders.
There are many ways to train one Autoencoder at a time. The first approach it to train each Autoencoder using a different graph, then we create the Stacked Autoencoder by simply initializing it with the weights and biases copied from these Autoencoders.Let's create a function that will train one autoencoder and return the transformed training set (ie. the output of the hidden layer) and the model parameters.
Can you write Python code for it?

def train_autoencoder(X_train, n_neurons, n_epochs, batch_size, learning_rate = 0.01, l2_reg = 0.0005, activation_fn=tf.nn.elu):
    graph = tf.Graph()
    with graph.as_default():
        n_inputs = X_train.shape[1]

        X = tf.placeholder(tf.float32, shape=[None, n_inputs])
        with tf.contrib.framework.arg_scope(
                [fully_connected],
                activation_fn=activation_fn,
                weights_initializer=tf.contrib.layers.variance_scaling_initializer(),
                weights_regularizer=tf.contrib.layers.l2_regularizer(l2_reg)):
            hidden = fully_connected(X, n_neurons, scope="hidden")
            outputs = fully_connected(hidden, n_inputs, activation_fn=None, scope="outputs")

        mse = tf.reduce_mean(tf.square(outputs - X))

        reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
        loss = tf.add_n([mse] + reg_losses)

        optimizer = tf.train.AdamOptimizer(learning_rate)
        training_op = optimizer.minimize(loss)

        init = tf.global_variables_initializer()

    with tf.Session(graph=graph) as sess:
        init.run()
        for epoch in range(n_epochs):
            n_batches = len(X_train) // batch_size
            for iteration in range(n_batches):
                print("\r{}%".format(100 * iteration // n_batches), end="")
                sys.stdout.flush()
                indices = rnd.permutation(len(X_train))[:batch_size]
                X_batch = X_train[indices]
                sess.run(training_op, feed_dict={X: X_batch})
            mse_train = mse.eval(feed_dict={X: X_batch})
            print("\r{}".format(epoch), "Train MSE:", mse_train)
        params = dict([(var.name, var.eval()) for var in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)])
        hidden_val = hidden.eval(feed_dict={X: X_train})
        return hidden_val, params["hidden/weights:0"], params["hidden/biases:0"], params["outputs/weights:0"], params["outputs/biases:0"]


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Let's plot the reconstructions:
Can you write Python code for it?

fig = plt.figure(figsize=(8, 2.5 * n_digits))
for iteration in range(n_digits):
    plt.subplot(n_digits, 2, 1 + 2 * iteration)
    plot_image(X_test[iteration])
    plt.subplot(n_digits, 2, 2 + 2 * iteration)
    plot_image(outputs_val[iteration])


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Now let's train it! Note that we don't feed target values (y_batch is not used). This is unsupervised training.
Can you write Python code for it?

n_epochs = 4
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        mse_train = mse.eval(feed_dict={X: X_batch})
        print("\r{}".format(epoch), "Train MSE:", mse_train)
        saver.save(sess, "./my_model_all_layers.ckpt")


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Now let's train two Autoencoders. The first one is trained on the training data, and the second is trained on the previous Autoencoder's hidden layer output:
Can you write Python code for it?

hidden_output, W1, b1, W4, b4 = train_autoencoder(mnist.train.images, n_neurons=300, n_epochs=4, batch_size=150)
_, W2, b2, W3, b3 = train_autoencoder(hidden_output, n_neurons=150, n_epochs=4, batch_size=150)


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
One of the triggers of the current Deep Learning tsunami is the discovery in (Hinton et al., 2006) that deep neural networks can be pretrained in an unsupervised fashion. They used restricted Boltzmann machines for that, but in (Bengio et al., 2006) showed that autoencoders worked just as well. There is nothing special about the TensorFlow implementation: just train an autoencoder using all the training data, then reuse its encoder layers to create a new neural network. Up to now, in order to force the autoencoder to learn interesting features, we have limited the size of the coding layer, making it undercomplete.
Denoising Autoencoders
Another way to force the autoencoder to learn useful features is to add noise to its inputs, training it to recover the original, noise-free inputs. This prevents the autoencoder from trivially copying its inputs to its outputs, so it ends up having to find patterns in the data. The idea of using autoencoders to remove noise has been around since the 1980s (e.g., it is mentioned in Yann LeCun’s 1987 master’s thesis). In a 2008 paper, 3 Pascal Vincent et al. showed that autoencoders could also be used for feature extraction. The idea of using autoencoders to remove noise has been around since the 1980s (e.g., it is mentioned in Yann LeCun’s 1987 master’s thesis). In (Pascal Vincent et al., 2008) showed that autoencoders could also be used for feature extraction. In (Vincent et al.,2010) it is introduced stacked denoising autoencoders.
Denoising Autoencoders with Gaussian noise
Can you write Python code for it?

import math

tf.reset_default_graph()

from tensorflow.contrib.layers import dropout

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01
l2_reg = 0.00001
keep_prob = 0.7

activation = tf.nn.elu
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
initializer = tf.contrib.layers.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
is_training = tf.placeholder_with_default(False, shape=(), name='is_training')

X_noisy = tf.cond(is_training,
                  lambda: X + tf.random_normal(shape=tf.shape(X),mean=0,stddev=1/math.sqrt(n_hidden1)), ## Gaussian noise
                  lambda: X)

weights1_init = initializer([n_inputs, n_hidden1])
weights2_init = initializer([n_hidden1, n_hidden2])

weights1 = tf.Variable(weights1_init, dtype=tf.float32, name="weights1")
weights2 = tf.Variable(weights2_init, dtype=tf.float32, name="weights2")
weights3 = tf.transpose(weights2, name="weights3")  # tied weights
weights4 = tf.transpose(weights1, name="weights4")  # tied weights

biases1 = tf.Variable(tf.zeros(n_hidden1), name="biases1")
biases2 = tf.Variable(tf.zeros(n_hidden2), name="biases2")
biases3 = tf.Variable(tf.zeros(n_hidden3), name="biases3")
biases4 = tf.Variable(tf.zeros(n_outputs), name="biases4")

hidden1 = activation(tf.matmul(X_noisy, weights1) + biases1)
hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)
hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)
outputs = tf.matmul(hidden3, weights4) + biases4

optimizer = tf.train.AdamOptimizer(learning_rate)
mse = tf.reduce_mean(tf.square(outputs - X))
reg_loss = regularizer(weights1) + regularizer(weights2)
loss = mse + reg_loss
training_op = optimizer.minimize(loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

n_epochs = 10
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration...


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Sparse Autoencoders
Another kind of constraint that often leads to good feature extraction is sparsity: by adding an appropriate term to the cost function, the autoencoder is pushed to reduce the number of active neurons in the coding layer. For example, it may be pushed to have on average only 5% significantly active neurons in the coding layer. This forces the autoencoder to represent each input as a combination of a small number of activations. As a result, each neuron in the coding layer typically ends up representing a useful feature (if you could speak only a few words per month, you would probably try to make them worth listening to). In order to favor sparse models, we must first measure the actual sparsity of the coding layer at each training iteration. We do so by computing the average activation of each neuron in the coding layer, over the whole training batch. The batch size must not be too small, or else the mean will not be accurate. Once we have the mean activation per neuron, we want to penalize the neurons that are too active by adding a sparsity loss to the cost function. For example, if we measure that a neuron has an average activation of 0.3, but the target sparsity is 0.1, it must be penalized to activate less. One approach could be simply adding the squared error (0.3 – 0.1) 2 to the cost function, but in practice a better approach is to use the Kullback– Leibler divergence which has much stronger gradients than the Mean Squared Error.
Can you write Python code for it?

p = 0.1
q = np.linspace(0, 1, 500)
kl_div = p * np.log(p / q) + (1 - p) * np.log((1 - p) / (1 - q))
mse = (p - q)**2
plt.plot([p, p], [0, 0.3], "k:")
plt.text(0.05, 0.32, "Target\nsparsity", fontsize=14)
plt.plot(q, kl_div, "b-", label="KL divergence")
plt.plot(q, mse, "r--", label="MSE")
plt.legend(loc="upper left")
plt.xlabel("Actual sparsity")
plt.ylabel("Cost", rotation=0)
plt.axis([0, 1, 0, 0.95])

def kl_divergence(p, q):
    """Kullback Leibler divergence"""
    return p * tf.log(p / q) + (1 - p) * tf.log((1 - p) / (1 - q))

tf.reset_default_graph()

n_inputs = 28 * 28
n_hidden1 = 1000  # sparse codings
n_outputs = n_inputs

learning_rate = 0.01
sparsity_target = 0.1
sparsity_weight = 0.2

#activation = tf.nn.softplus # soft variant of ReLU
activation = tf.nn.sigmoid
initializer = tf.contrib.layers.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])

weights1_init = initializer([n_inputs, n_hidden1])
weights2_init = initializer([n_hidden1, n_outputs])

weights1 = tf.Variable(weights1_init, dtype=tf.float32, name="weights1")
weights2 = tf.Variable(weights2_init, dtype=tf.float32, name="weights2")

biases1 = tf.Variable(tf.zeros(n_hidden1), name="biases1")
biases2 = tf.Variable(tf.zeros(n_outputs), name="biases2")

hidden1 = activation(tf.matmul(X, weights1) + biases1)
outputs = tf.matmul(hidden1, weights2) + biases2

optimizer = tf.train.AdamOptimizer(learning_rate)
mse = tf.reduce_mean(tf.square(outputs - X))

hidden1_mean = tf.reduce_mean(hidden1, axis=0) # batch mean
sparsity_loss = tf.reduce_sum(kl_divergence(sparsity_target, hidden1_mean))
loss = mse + sparsity_weight * sparsity_loss
training_op = optimizer.minimize(loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

n_epochs = 100
batch_size = 1000

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches),...


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
This function loads the model, evaluates it on the test set (it measures the reconstruction error), then it displays the original image and its reconstruction:
Can you write Python code for it?

def plot_image(image, shape=[28, 28]):
    plt.imshow(image.reshape(shape), cmap="Greys", interpolation="nearest")
    plt.axis("off")

def show_reconstructed_digits(X, outputs, model_path = None, n_test_digits = 2):
    with tf.Session() as sess:
        if model_path:
            saver.restore(sess, model_path)
        X_test = mnist.test.images[:n_test_digits]
        outputs_val = outputs.eval(feed_dict={X: X_test})

    fig = plt.figure(figsize=(8, 3 * n_test_digits))
    for digit_index in range(n_test_digits):
        plt.subplot(n_test_digits, 2, digit_index * 2 + 1)
        plot_image(X_test[digit_index])
        plt.subplot(n_test_digits, 2, digit_index * 2 + 2)
        plot_image(outputs_val[digit_index])

show_reconstructed_digits(X, outputs, "./my_model_all_layers.ckpt",n_test_digits = 2)


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Train all layers at once
Let's build a stacked Autoencoder with 3 hidden layers and 1 output layer (ie. 2 stacked Autoencoders). We will use ELU activation, He initialization and L2 regularization.
Can you write Python code for it?

tf.reset_default_graph()

from tensorflow.contrib.layers import fully_connected

n_inputs = 28*28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01
l2_reg = 0.0001

initializer = tf.contrib.layers.variance_scaling_initializer() # initialization

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
with tf.contrib.framework.arg_scope([fully_connected],
                                    activation_fn=tf.nn.elu,
                                    weights_initializer=initializer,
                                    weights_regularizer=tf.contrib.layers.l2_regularizer(l2_reg)):
    hidden1 = fully_connected(X, n_hidden1)
    hidden2 = fully_connected(hidden1, n_hidden2)
    hidden3 = fully_connected(hidden2, n_hidden3)
    outputs = fully_connected(hidden3, n_outputs, activation_fn=None)

mse = tf.reduce_mean(tf.square(outputs - X))

reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
loss = tf.add_n([mse] + reg_losses)

optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver()


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Training one Autoencoder at a time in a single graph
Another approach is to use a single graph. To do this, we create the graph for the full Stacked Autoencoder, but then we also add operations to train each Autoencoder independently: phase 1 trains the bottom and top layer (ie. the first Autoencoder) and phase 2 trains the two middle layers (ie. the second Autoencoder).
Can you write Python code for it?

tf.reset_default_graph()

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01
l2_reg = 0.0001

activation = tf.nn.elu
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
initializer = tf.contrib.layers.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])

weights1_init = initializer([n_inputs, n_hidden1])
weights2_init = initializer([n_hidden1, n_hidden2])
weights3_init = initializer([n_hidden2, n_hidden3])
weights4_init = initializer([n_hidden3, n_outputs])

weights1 = tf.Variable(weights1_init, dtype=tf.float32, name="weights1")
weights2 = tf.Variable(weights2_init, dtype=tf.float32, name="weights2")
weights3 = tf.Variable(weights3_init, dtype=tf.float32, name="weights3")
weights4 = tf.Variable(weights4_init, dtype=tf.float32, name="weights4")

biases1 = tf.Variable(tf.zeros(n_hidden1), name="biases1")
biases2 = tf.Variable(tf.zeros(n_hidden2), name="biases2")
biases3 = tf.Variable(tf.zeros(n_hidden3), name="biases3")
biases4 = tf.Variable(tf.zeros(n_outputs), name="biases4")

hidden1 = activation(tf.matmul(X, weights1) + biases1)
hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)
hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)
outputs = tf.matmul(hidden3, weights4) + biases4

with tf.name_scope("phase1"):
    optimizer = tf.train.AdamOptimizer(learning_rate)
    phase1_outputs = tf.matmul(hidden1, weights4) + biases4  # bypass hidden2 and hidden3
    phase1_mse = tf.reduce_mean(tf.square(phase1_outputs - X))
    phase1_reg_loss = regularizer(weights1) + regularizer(weights4)
    phase1_loss = phase1_mse + phase1_reg_loss
    phase1_training_op = optimizer.minimize(phase1_loss)

with tf.name_scope("phase2"):
    optimizer = tf.train.AdamOptimizer(learning_rate)
    phase2_mse = tf.reduce_mean(tf.square(hidden3 - hidden1))
    phase2_reg_loss = regularizer(weights2) + regularizer(weights3)
    phase2_loss = phase2_mse + phase2_reg_loss
    phase2_training_op =...


I am working on the file "Autoencoders_1.1.0.ipynb".
The first task is:
Tying Weights
When an autoencoder is neatly symmetrical, like the one we just built, a common technique is to tie the weights of the decoder layers to the weights of the encoder layers. This halves the number of weights in the model, speeding up training and limiting the risk of overfitting.
Can you write Python code for it?

tf.reset_default_graph()

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01
l2_reg = 0.0001

activation = tf.nn.elu
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
initializer = tf.contrib.layers.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])

weights1_init = initializer([n_inputs, n_hidden1])
weights2_init = initializer([n_hidden1, n_hidden2])

weights1 = tf.Variable(weights1_init, dtype=tf.float32, name="weights1")
weights2 = tf.Variable(weights2_init, dtype=tf.float32, name="weights2")
weights3 = tf.transpose( weights2, name ="weights3") # tied weights
weights4 = tf.transpose( weights1, name ="weights4") # tied weights

biases1 = tf.Variable(tf.zeros(n_hidden1), name="biases1")
biases2 = tf.Variable(tf.zeros(n_hidden2), name="biases2")
biases3 = tf.Variable(tf.zeros(n_hidden3), name="biases3")
biases4 = tf.Variable(tf.zeros(n_outputs), name="biases4")

hidden1 = activation(tf.matmul(X, weights1) + biases1)
hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)
hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)
outputs = tf.matmul(hidden3, weights4) + biases4

reconstruction_loss = tf.reduce_mean( tf.square( outputs - X))
reg_loss = regularizer(weights1) + regularizer(weights2)
loss = reconstruction_loss + reg_loss

optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)
init = tf.global_variables_initializer()
saver = tf.train.Saver()

n_epochs = 4
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        mse_train = reconstruction_loss.eval(feed_dict={X:...


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
A Dictionary cannot lookup a value that is not available.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
Another useful command is
   dictionary.setdefault('key', 'value)

Which sets a default value in a dictionary faster than looping through the dictionary.
Can you write Python code for it?

eggs = {
    'name': 'Zophie',
    'species': 'cat',
    'age': 8
    }

print(eggs)

if 'color' not in eggs:
    eggs['color'] = 'black'

print(eggs)

eggs = {
    'name': 'Zophie',
    'species': 'cat',
    'age': 8
    }

print(eggs)

eggs.setdefault('color','black')

print(eggs)


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
Dictionaries are equivalent.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
Now hand that to to the Character Count Program to see if it works, and use pprint to do a 'pretty print'.
Can you write Python code for it?

import pprint

count = {}

for character in rj.text.upper(): # Use upper case method to avoid counting lower and upper case letters seperately
    count.setdefault(character, 0) # Start the counter at 0 so theres no error on the next line
    count[character] = count[character] + 1 # Now add a value for that character everytime you run into that character

pprint.pprint(count)


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
This is useful to avoid errors when keys aren't populated:
Can you write Python code for it?

print('I am bringing ' + str(picnicItems['napkins']) + ' napkins to the picnic.')

print('I am bringing ' + str(picnicItems.get('napkins', 0)) + ' napkins to the picnic.')

eggs =


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
This method would work for a string of any conceivable length. For example, the entire text of Romeo and Juliet.
Can you write Python code for it?

import requests # get the Requests library

link = "https://automatetheboringstuff.com/files/rj.txt" # store the link where the file is at
rj = requests.get(link) # use Requests to get the link

print(rj.text) # Print out the text from Requests, and check that its stored


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
To check quickly if a key exists in a Dictionary, a useful command is:
    get('key','return this value if key isn't there.)
Can you write Python code for it?

eggs.get('age','0')

eggs.get('color','')

picnicItems = {'apples': 5, 'cups': 2}


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
Without .setdefault() we would get an error message because there is no default character for those values:
Can you write Python code for it?

count = {}

for character in message.upper(): # Use upper case method to avoid counting lower and upper case letters seperately
    #count.setdefault(character, 0) # Start the counter at 0 so theres no error on the next line
    count[character] = count[character] + 1 # Now add a value for that character everytime you run into that character

print(count)


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
You can use multiple commands to query dictionaries. Some of the basic ones are:
    list(dictionary.keys()) # lists keys
    list(dictionary.values()) # lists values
    list(dictionary.items()) # lists keys and values as tuples

Use list() to get actual list values, otherwise you get dictionary 'list-like' data types.
Can you write Python code for it?

list(eggs.keys())

list(eggs.values())

list(eggs.items())


I am working on the file "AutomateTheBoringStuffWithPython/lesson17.ipynb".
The first task is:
You can use the values in for loops.
Can you write Python code for it?

for v in eggs.values():
    print(v)

for k, v in eggs.items():
    print(k,v)

for i in eggs.items():
    print(i)


I am working on the file "AutomateTheBoringStuffWithPython/lesson18.ipynb".
The first task is:
Dictionaries, lists, and strings can be used in combination to simulate real world things, translating machine responses to human responses.
If confused about what type of data you're dealing with, use the type() function.
Can you write Python code for it?

print('type(42)')
print(type(42))

print('type(hello)')
print(type('hello'))

print('type(3.14)')
print(type(3.14))

print('type(theBoard)')
print(type(theBoard))

print('type(theBoard[top-R])')
print(type(theBoard['top-R']))


I am working on the file "AutomateTheBoringStuffWithPython/lesson18.ipynb".
The first task is:
Human version:
Can you write Python code for it?

from IPython.display import Image

Image(url='https://automatetheboringstuff.com/images/000008.png')


I am working on the file "AutomateTheBoringStuffWithPython/lesson18.ipynb".
The first task is:
The Tic-Tac-Toe Game Program
We can use data structures to represent values in Python that can be understood. For example, using key-value pairs and strings to represent regions of a tic-tac-toe board.
Can you write Python code for it?

from IPython.display import Image

Image(url='https://automatetheboringstuff.com/images/000003.png')


I am working on the file "AutomateTheBoringStuffWithPython/lesson18.ipynb".
The first task is:
The keys are arbitrary; they are just used to store and change values.
Machine version:
Can you write Python code for it?

theBoard['mid-M'] = 'X'

pprint.pprint(theBoard)


I am working on the file "AutomateTheBoringStuffWithPython/lesson19.ipynb".
The first task is:
Functions can now be run on it:
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson19.ipynb".
The first task is:
Large strings are treated in this way.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson19.ipynb".
The first task is:
Many escape characters are available:
Can you write Python code for it?

print('\'') # Single quote
print('\"') # Double quote
print('\t') # Tab
print('\n') # New line
print('\\') # Back slash


I am working on the file "AutomateTheBoringStuffWithPython/lesson19.ipynb".
The first task is:
Multi line strings (with '''' or """ are often easier.)
Can you write Python code for it?

print("""Dear Alice,
Eve's cat has been arrested for catnapping, cat burglarly, and extortion.
Sincerly, Bob.""")


I am working on the file "AutomateTheBoringStuffWithPython/lesson19.ipynb".
The first task is:
Use r'' to print raw strings, where all characters are treated as strings.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson20.ipynb".
The first task is:
.islower() and .isupper() return boolean responses about the case of the string. There must be at least one upper or lower character to function.
Can you write Python code for it?

print(spam.islower())

print(spam.upper().isupper()) # Double methods; works because spam.upper() is its own string.


I am working on the file "AutomateTheBoringStuffWithPython/lesson20.ipynb".
The first task is:
The .join() method joins strings/lists.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson20.ipynb".
The first task is:
The Pyperclip Module can be used to copy text to and from the keyboard.
Can you write Python code for it?

import pyperclip

# Use pip to install packages into the Python3.5/scripts directory

pyperclip.copy('Hello!') # Copies to clipboard
pyperclip.paste() # Pastes whatever is on clipboard


I am working on the file "AutomateTheBoringStuffWithPython/lesson20.ipynb".
The first task is:
There are also .startwith() and .endwith() methods:
Can you write Python code for it?

print('Hello world!'.startswith('Hello')) # True
print('Hello world!'.startswith('H')) # True
print('Hello world!'.startswith('ello')) # False
print('Hello world!'.endswith('world!')) # True
print('Hello world!'.endswith('world')) # False


I am working on the file "AutomateTheBoringStuffWithPython/lesson20.ipynb".
The first task is:
There are other 'checking' methods that can be used:
Can you write Python code for it?

print('hello'.isalpha()) # letters only
print('hello123'.isalnum()) # alphanumeric only
print('123'.isdecimal()) # numbers only
print('   '.isspace()) # whitespace only
print('Hello world!'[5].isspace()) # This index of this string is whitespace
print('This Is Title Case'.istitle()) # Titlecase only


I am working on the file "AutomateTheBoringStuffWithPython/lesson20.ipynb".
The first task is:
This fails if we use capitals.
Instead, we can make the input insensitive.
Can you write Python code for it?

answer = input()

if answer.lower() == 'yes':
    print('Playing again.')


I am working on the file "AutomateTheBoringStuffWithPython/lesson21.ipynb".
The first task is:
Python has string interpolation, which uses %s to insert other strings into placeholders.
Can you write Python code for it?

print(' Hello %s, you are invited to a party at %s at %s. Please bring %s.' % (name, place, time, food))


I am working on the file "AutomateTheBoringStuffWithPython/lesson21.ipynb".
The first task is:
This gets harder with more variables.
Can you write Python code for it?

name = 'Alice'
place = 'Main Street'
time = '6 pm'
food = 'turnips'

print('Hello ' + name + ', you are invited to a party at ' + place + ' at ' + time + '. Please bring ' + food + '.')


I am working on the file "AutomateTheBoringStuffWithPython/lesson22.ipynb".
The first task is:
You can then use CMD/Terminal to run this script
$sh /path/to/shellscript.sh

You can skip adding the absolute path to scripts by adding folders to the PATH environment variable.
This lets the operating system check these folders before looking anywhere else. Editing the PATH gives the entire OS access to those folders from any location.
Temporarily edit the PATH in OSX
Show PATH in Terminal
$ echo $PATH

Temporarily add to $PATH for that terminal session
$ PATH=/usr/bin:/bin:/usr/sbin:/newpathtofolder/

Permanently edit the PATH in OSX
Move to home folder
$ cd

Edit the .bash_profile (use whatever editor you want instead of 'nano')
$nano .bash_profile #

Add the new value to the PATH and include exiting folders (with :PATH)
$ export PATH="/usr/local/mysql/bin:$PATH"

Should now show the new PATH with the added folder.
$ echo $PATH

Scripts can also take command line arguments:
sh script.sh arg1 arg2

These are called system arguments, and can be accessed in a Python program via the sys.argv command.
Can you write Python code for it?

import sys
print('Hello world')
print(sys.argv)


I am working on the file "AutomateTheBoringStuffWithPython/lesson23.ipynb".
The first task is:
This is 3 lines of code in place of 30, which is significantly more effecient.
You can also use the .findall() method to find all RegEx matches, not just the first.
Can you write Python code for it?

mo = phoneNumRegex.findall(message) #re.findall() returns a Match Object List
print(mo) # mo.findall() returns a list, so it doesn't need .group()


I am working on the file "AutomateTheBoringStuffWithPython/lesson23.ipynb".
The first task is:
This is a lot of code for text pattern matching; which is very common activity in most programming. Therefore, regular expressions are used to simplify this process.
Phone Number Matching with RegEx
The re module stores RegEx functions. It usually takes raw strings (r'').
Can you write Python code for it?

import re

print(message)

phoneNumRegex =  re.compile(r'\d\d\d-\d\d\d-\d\d\d\d') # Defines the pattern and converts to Regex
mo = phoneNumRegex.search(message) #re.search() searches string and  returns a Match Object
print(mo.group()) # the .group() method contains the actual text in the Match Object


I am working on the file "AutomateTheBoringStuffWithPython/lesson23.ipynb".
The first task is:
Use to check if strings contain phone numbers:
Can you write Python code for it?

message = 'Call me at 415-555-1011 tomorrow, or at 415-555-9999 any other day.'
message2 = 'There are no phone numbers in this message.'

def findNumber(message):
    foundNumber = False # set False to start|
    for i in range(len(message)):
        chunk = message[i:i+12] # Take a phone number size 'chunk' of the string, character by character
        #print(chunk) # debug
        if isPhoneNumber(chunk):
            print('Phone number found: ' + chunk)
            foundNumber = True
    if not foundNumber: # Run after loop, not during loop
            print('Could not find any phone numbers.')

findNumber(message)
findNumber(message2)


I am working on the file "AutomateTheBoringStuffWithPython/lesson23.ipynb".
The first task is:
You can then test what strings count as phone numbers using this program.
Can you write Python code for it?

print(isPhoneNumber('415-555-1234')) # False
print(isPhoneNumber('Hello')) # False
print(isPhoneNumber('415551234')) # False


I am working on the file "AutomateTheBoringStuffWithPython/lesson24.ipynb".
The first task is:
The '|' RegEx Operater
The pipe character, |, matches one of several patterns in a group.
Can you write Python code for it?

batRegex = re.compile(r'Bat(man|mobile|copter|cat)') # The pipe character seperates the suffixes, and allows any match.

mo = batRegex.search('Batmobile lost a wheel.')
print(mo.group()) # Print matching string

print(mo.group(1)) # Pass variable 1 into the group function to find which suffix group actually matched

#mo2 = batRegex.search('Batmotorcycle lost a wheel.')
#mo.group() # will return error because no match


I am working on the file "AutomateTheBoringStuffWithPython/lesson25.ipynb".
The first task is:
However, it cannot match multiple repititions:
Can you write Python code for it?

mo = batRegex.search('The Adventures of Batwowowowoman')
print(mo.group())


I am working on the file "AutomateTheBoringStuffWithPython/lesson25.ipynb".
The first task is:
RegEx does greedy matches, which means it will try to find the longest string that matches, not the shortest.
Can you write Python code for it?

haRegex = re.compile(r'(Ha){1,6}') # at least 1, or 6
print(haRegex.search('HaHaHaHaHaHaHaHa').group()) # Matches longest string; 6


I am working on the file "AutomateTheBoringStuffWithPython/lesson25.ipynb".
The first task is:
The '*' Regex Operater
The * character can be used to match many (0 or n)  times.
Can you write Python code for it?

import re

batRegex = re.compile(r'Bat(wo)*man') # The ()* says this group can appear 0 or n times to match

print(batRegex.search('The Adventures of Batwoman').group())

print(batRegex.search('The Adventures of Batwowowowoman').group())


I am working on the file "AutomateTheBoringStuffWithPython/lesson25.ipynb".
The first task is:
The '+' Regex Operater
The + character can match one or more (1 or n) times.
Can you write Python code for it?

import re

batRegex = re.compile(r'Bat(wo)+man') # The ()+ says this group can appear 1 or n times; it is NOT optional

print(batRegex.search('The Adventures of Batwoman').group())

print(batRegex.search('The Adventures of Batwowowowoman').group())

print(batRegex.search('The Adventures of Batman').group())


I am working on the file "AutomateTheBoringStuffWithPython/lesson25.ipynb".
The first task is:
The '{}' Regex Operater
The {x} character can match x times.
Can you write Python code for it?

haRegex = re.compile(r'(Ha){3}')

print(haRegex.search('HaHaHa').group())
print(haRegex.search('HaHaHaHa').group())  # Matches only three times, so returns only 3
#print(haRegex.search('HaHa').group()) # No Match

phoneRegex = re.compile(r'((\d)?\d\d\d(\d)?){3}') # Useful to avoid repitition
phoneNumRegex.search('My number is 415-555-4242').group()


I am working on the file "AutomateTheBoringStuffWithPython/lesson25.ipynb".
The first task is:
This operator can also take the {x,y} argument to create a minimum or maximum number of repititions.
Can you write Python code for it?

haRegex = re.compile(r'(Ha){3,5}')

print(haRegex.search('HaHaHa').group())
print(haRegex.search('HaHaHaHa').group())
print(haRegex.search('HaHaHaHaHa').group())
print(haRegex.search('HaHaHaHaHaHaHaHa').group()) # Matches max of 5

haRegex = re.compile(r'(Ha){,5}') # Can drop one or the other for unbounded matches
print(haRegex.search('Ha').group())
print(haRegex.search('HaHa').group())
print(haRegex.search('HaHaHa').group())
print(haRegex.search('HaHaHaHa').group())
print(haRegex.search('HaHaHaHaHa').group())
print(haRegex.search('HaHaHaHaHaHaHaHa').group()) # Matches max of 5


I am working on the file "AutomateTheBoringStuffWithPython/lesson25.ipynb".
The first task is:
We can use this to find strings that may or may not include elements, like phone numbers with and without area codes.
Can you write Python code for it?

phoneNumRegex = re.compile(r'\d\d\d\-\d\d\d-\d\d\d\d') # this requires an area code.

mo = phoneNumRegex.search('My number is 415-555-4242') # matches
print(mo.group())

mo2 = phoneNumRegex.search('My number is 555-4242') # will not match
print(mo2)

phoneNumRegex = re.compile(r'(\d\d\d\-)?\d\d\d-\d\d\d\d') # Make first three digits and dash optional

mo = phoneNumRegex.search('My number is 415-555-4242') # matches
print(mo.group())
mo2 = phoneNumRegex.search('My number is 555-4242') # matches
print(mo2.group())


I am working on the file "AutomateTheBoringStuffWithPython/lesson26.ipynb".
The first task is:
It is possible to create your own character classes, outside of these shorthand classes, using []:
Can you write Python code for it?

vowelRegex = re.compile(r'[aeiouAEIOU]') # RegEx for lowercase and uppercase vowels
alphabetRegex = re.compile(r'[a-zA-Z]') # RegEx for lowercase and uppercase alphabet using ranges

print(vowelRegex.findall('Robocop eats baby food.')) # Finds a list of all vowels in string

doublevowelRegex = re.compile(r'[aeiouAEIOU]{2}') # RegEx for two lowercase and uppercase vowels in a row; {2} repeats.
print(doublevowelRegex.findall('Robocop eats baby food.')) # Finds a list of all vowels in string


I am working on the file "AutomateTheBoringStuffWithPython/lesson26.ipynb".
The first task is:
Other regex characters are:

\D Any character that is NOT a numeric digit from 0 to 9.
\w Any letter, numeric digit, punctuation, or the underscore character (word characters.)
\W Any character that is NOT a letter, numeric digit, or the underscore character.

\s Any space, tab, or newline character (space characters.)

\S Any character that is NOT a space character.
Can you write Python code for it?

# Example using lyrics from The Twelve Days of Christmas

lyrics = '''
12 Drummers Drumming
11 Pipers Piping
10 Lords a Leaping
9 Ladies Dancing
8 Maids a Milking
7 Swans a Swimming
6 Geese a Laying
5 Golden Rings
4 Calling Birds
3 French Hens
2 Turtle Doves
and 1 Partridge in a Pear Tree
'''

xmasRegex = re.compile(r'\d+\s\w+') # 1 or more digits, space, 1 or more words

xmasRegex.findall(lyrics) # Returns all 'x gift', but stops at space because \w+ does not include spaces


I am working on the file "AutomateTheBoringStuffWithPython/lesson26.ipynb".
The first task is:
To get the total string, just wrap the total regex in its own group, so you get [(totalstring, group1, group2),...].
RegEx Character Classes
\d\ is the RegEx character for digits.
Can you write Python code for it?

#digitRegex = re.compile(r'(1|2|3|4...|n)`) is equivalent to

#digitRegex = re.compile(r'\d\')


I am working on the file "AutomateTheBoringStuffWithPython/lesson26.ipynb".
The first task is:
find.all() returns a list of strings.
It behaves differently with groups.
Can you write Python code for it?

import re

phoneRegex = re.compile(r'(/d/d/d)-(/d/d/d-/d/d/d/d)') # Two groups, so returns tuples

#phoneRegex.findall() # finds all matches in pairs; [('group1', 'group2'),...]


I am working on the file "AutomateTheBoringStuffWithPython/lesson27.ipynb".
The first task is:
.* is greedy by default, but you can activate non-greedy mode with .*?
Can you write Python code for it?

serve = '<To serve humans> for dinner.>'

greedyRegex = re.compile(r'<(.*)>') # Looking for any length match, between brackets.
nongreedyRegex = re.compile(r'<(.*?)>') # Looking for any length match, between brackets.

print(greedyRegex.findall(serve)) # Matches the longest string
print(nongreedyRegex.findall(serve)) # Matches the shortest string


I am working on the file "AutomateTheBoringStuffWithPython/lesson27.ipynb".
The first task is:
.* matches any character except the newline (\n) character.
Can you write Python code for it?

primeDirectives = 'Serve the public trust.\nProtect the innocent.\nUphold the law.'

print(primeDirectives)

dotStar = re.compile(r'.*')
print(dotStar.findall(primeDirectives))


I am working on the file "AutomateTheBoringStuffWithPython/lesson27.ipynb".
The first task is:
Similiarily, re.IGNORECASE or re.I to ignore case:
Can you write Python code for it?

vowelRegex = re.compile(r'[aeiou]', re.I) # Match any vowel, regardless of case

print(vowelRegex.findall('Al, why does your programming book talk about RoboCop so much?'))


I am working on the file "AutomateTheBoringStuffWithPython/lesson27.ipynb".
The first task is:
The .* is therefore used to match anything, any number of any character:
Can you write Python code for it?

name = 'First Name: Al, Last Name: Sweigart' # To pull names from this string would require a lot of indexing code
name2 = 'First Name: Vivek, Last Name: Menon' # To pull names from this string would require a lot of indexing code

nameRegex = re.compile(r'First Name: (.*), Last Name: (.*)') # Matches anything in this groups formatted exactly like this

print(nameRegex.findall(name))
print(nameRegex.findall(name2))


I am working on the file "AutomateTheBoringStuffWithPython/lesson28.ipynb".
The first task is:
This is basically a find and replace feature with regex.
Regex objects also have a re.verbose argument, to allow multline line comments for complicated regex patterns, helping readabilitiy.
Can you write Python code for it?

phoneRegex = re.compile(r'''
(\d\d\d-)|(\(d\d\d\) )   # area code (without parenthesis with dash, with parenthesis without dash )
-                        # first dash
\d\d\d                   # first 3 digits
-                        # second dash
\d\d\d\d                 # last 4 digits
\sx\d{2,4}               # Extension, like x1234, with at least 2 and at most 4 digits
'''
, re.VERBOSE) # Allows multiline regex strings that ignore newlines, allowing for new comments/documentation on every line.


I am working on the file "AutomateTheBoringStuffWithPython/lesson28.ipynb".
The first task is:
You can also partially replace a match using a group, using placeholders like \1.
Can you write Python code for it?

import re

namesRegex = re.compile(r'Agent (\w)\w*') # Seperate the first letter into its own group, and match 0 or more words

print(namesRegex.findall('Agent Alice gave the secret documents to Agent Bob.')) # Will only return group 1 matches, not searching for entire strings

print(namesRegex.sub(r'Agent \1***', 'Agent Alice gave the secret documents to Agent Bob.')) # Replace matches with group 1 matches


I am working on the file "AutomateTheBoringStuffWithPython/lesson30.ipynb".
The first task is:
But this string only works on Windows; to create an OS insensitive path, using the os module:
Can you write Python code for it?

import os # contains many file path related functions

print(os.path.join('folder1','folder2','folder3','file.png')) # takes string arguments and returns OS-appropriate path
print(os.sep) # show the seperator currently being used.


I am working on the file "AutomateTheBoringStuffWithPython/lesson30.ipynb".
The first task is:
If no explicit path is specified, Python will look for files in the current working directory.
You can find out what the current working directory is with os.getcwd(), and change it is with os.chdir().
Can you write Python code for it?

# Start at current directory
defaultpath = os.path.expanduser('~/Dropbox/learn/books/Python/AutomateTheBoringStuffWithPython/')
os.chdir(defaultpath)
print(os.getcwd())

# Change path to /files folder
os.chdir('files') # changes current working directory, if not currently in it (to /files)
print(os.getcwd()) # prints the current working directory (should be /files)

# Reset back to notebook directory
os.chdir(defaultpath)


I am working on the file "AutomateTheBoringStuffWithPython/lesson30.ipynb".
The first task is:
There are two kinds of file paths, relative and absolute.
Absolute file paths are the full address, while relative paths are related to the current working directory; they start at the cwd, not the root folder.
There are also . and .. operators:
* .\ refers to the cwd, and .\path will look at any folders below this folder.
* ..\ refers to the folder above the cwd, and will look at any folders below the parent folder of the cwd.

os.path.abspath returns the absolute path of whatever relative path is passed to it.
os.path.relpath() returns the relative path from an absolute path to a relative location.
os.path.isabs() returns a boolean if the path is a true path.
Can you write Python code for it?

print(os.path.abspath('files')) # print absolute path of the files subdirectory
print(os.path.isabs(os.path.abspath('files'))) # Is the absolute path to files an absolute path (True)
print(os.path.relpath('../..', 'files')) # print the relative file path of a folder two folders up relative to subfolder (3 folders; ../../..)


I am working on the file "AutomateTheBoringStuffWithPython/lesson30.ipynb".
The first task is:
os.makedirs() creates new directories at the location.
os.removedirs() removes folders in an absolute location.
Can you write Python code for it?

# clear the folders if the exist already
if os.path.exists(os.path.abspath('files/newfolder/anotherone')) == True:
    os.removedirs(os.path.abspath('files/newfolder/anotherone')) # clear folders if they exist

# create new folders at an absolute path
os.makedirs(os.path.abspath('files/newfolder/anotherone')) # create new folders

# check if they exist
if os.path.exists(os.path.abspath('files/newfolder/anotherone')) == True:
    print('\'files/newfolder/anotherone\' exists.')


I am working on the file "AutomateTheBoringStuffWithPython/lesson30.ipynb".
The first task is:
os.path.dirname() pulls out just the directory path component above a filepath.
os.path.basename() pulls out whats past the last slash.
Can you write Python code for it?

print(os.path.dirname(os.path.abspath('files'))) # outputs absolute path above 'files'
print(os.path.basename('files/26645.pdf')) # outputs just 'files'


I am working on the file "AutomateTheBoringStuffWithPython/lesson30.ipynb".
The first task is:
os.path.exists() can check if a path exists.
os.path.isfile() checks if the path ends at a file
os.path.isdir() checks if the path ends at a folder.
Can you write Python code for it?

# Reset back to notebook directory
os.chdir(defaultpath)

print(os.path.exists(os.path.abspath('files'))) # checks if 'files' exists (True)
print(os.path.isfile('files')) # checks if 'files' is a file (False)
print(os.path.isdir('files')) # checks if 'files' is a folder (False)


I am working on the file "AutomateTheBoringStuffWithPython/lesson30.ipynb".
The first task is:
os.path.getsize() returns the size of a file (bytes)
os.path.listdir() returns a list of all the files at the path.
File Size Finder
Can you write Python code for it?

"""
A simple program to loop through a folder and find the size of all files in bytes, and the total size of the folder.
"""

import os

# starting size
totalSize = 0

# for the fileName in the 'files' directory
for fileName in os.listdir('files'):
    # generate filePaths
    filePath = os.path.join('files',fileName)
    # check if filePath is a file
    if os.path.isfile(filePath) == True:
    # if True, increase totalSize by the size of fileName
        totalSize += os.path.getsize(filePath)
    # also print what the file was and the size
        print('%s is %d bytes.'%(filePath, os.path.getsize(filePath)))
    # otherwise keep looping
    else:
        continue

# print the size of the folder at the end
print('\n\nThe \'%s\' folder contains %s bytes in total.'%('files',str(totalSize)))


I am working on the file "AutomateTheBoringStuffWithPython/lesson31.ipynb".
The first task is:
To handle more data types than strings, the shelve module is often used:
Can you write Python code for it?

# Define base directory
defaultpath = os.path.expanduser('~/Dropbox/learn/books/Python/AutomateTheBoringStuffWithPython')

#Change directory to files directory if set in default
if (os.getcwd() == defaultpath):
       os.chdir('/files')
else:
    os.chdir(defaultpath + '/files')

print(os.getcwd())

import shelve # import the shelve module

shelfFile = shelve.open('mycatdata')

# Create a shelf file that works like a dictionary

shelfFile['cats'] = ['Zophie', 'Pooka', 'Simon', 'Fat-tail', 'Cleo']

shelfFile.close()


I am working on the file "AutomateTheBoringStuffWithPython/lesson31.ipynb".
The first task is:
Unless otherwise specified, all new files are created in the current working directory.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson31.ipynb".
The first task is:
shelve files work like dictionaries, but are stored in binary files, which cannot be easily understood in text editors
Can you write Python code for it?

shelfFile = shelve.open('mycatdata') # Open shelve file

list(shelfFile.keys())

list(shelfFile.values())


I am working on the file "AutomateTheBoringStuffWithPython/lesson32.ipynb".
The first task is:
copytree() can  be used to copy entire directories:
Can you write Python code for it?

shutil.copytree('newfiles', 'newerfiles') # Copy a directory and rename it


I am working on the file "AutomateTheBoringStuffWithPython/lesson32.ipynb".
The first task is:
import shutil and use copy() to copy a file:
Can you write Python code for it?

import shutil # import shutil

shutil.copy('bacon.txt', 'newfiles') # Copy file to a new directory

shutil.copy('bacon.txt', 'bacon2.txt') # Copy file to with a new name


I am working on the file "AutomateTheBoringStuffWithPython/lesson32.ipynb".
The first task is:
move() can move files and also rename them, and copy them to the same directory with a new name.
Can you write Python code for it?

shutil.move('bacon2.txt', 'bacon3.txt') # Rename file without moving them

shutil.move('bacon3.txt', 'newfiles/bacon3.txt') # Move and rename file


I am working on the file "AutomateTheBoringStuffWithPython/lesson33.ipynb".
The first task is:
All these deletions are permanent, but the send2trash module can be used to send deletions to the trash.
Can you write Python code for it?

import send2trash # install via pip3

send2trash.send2trash(os.path.abspath('newfiles/bacon3.txt')) # Delete by snding to trash


I am working on the file "AutomateTheBoringStuffWithPython/lesson33.ipynb".
The first task is:
Since these deletions are permanent, it is useful to run these programs in 'dry-run' mode; where the deletions/functions are commented, and print() is used instead:
Can you write Python code for it?

import os

#Change directory to files directory if set in default
if (os.getcwd() == defaultpath):
       os.chdir('/files')
else:
    os.chdir(defaultpath + '/files')

# Move into the newfiles directory
os.chdir('newfiles')

for filename in os.listdir():
    if filename.endswith('.txt'):
        #os.unlink(filename)
        print(filename)


I am working on the file "AutomateTheBoringStuffWithPython/lesson35.ipynb".
The first task is:
Box Priting Program
Can you write Python code for it?

def boxPrint(symbol, width, height):
"""

A program to build boxes like this:

***************
*             *
*             *
***************

"""
    # print top part
    print(symbol * width)
    # print the sides (excluding top and bottom)
    for i in range(height-2):
        # empty spaces in the middle except the last two spots
        print(symbol + (' ' * (width - 2) + symbol))
    # print the bottom
    print(symbol * width)

# Working functions
boxPrint('*', 15,5)
boxPrint('O', 10,8)

# Not working
boxPrint('**', 5,16)


I am working on the file "AutomateTheBoringStuffWithPython/lesson35.ipynb".
The first task is:
Formerly, the try and except statements were used to handle exceptions, but you can also create your own exceptions with raise.
Can you write Python code for it?

raise Exception('This is the error message.')


I am working on the file "AutomateTheBoringStuffWithPython/lesson35.ipynb".
The first task is:
Stoplight Program
Can you write Python code for it?

market_2nd = {'ns':'green', 'ew':'red'} #ns: north south, ew: east west

def switchLights(intersection):
    for key in intersection.keys():
        if intersection[key] == 'green':
            intersection[key] = 'yellow'
        elif intersection[key] == 'yellow':
            intersection[key] = 'red'
        elif intersection[key] == 'red':
            intersection[key] = 'green'

print(market_2nd)
switchLights(market_2nd) # Run function in data structure
print(market_2nd)


I am working on the file "AutomateTheBoringStuffWithPython/lesson35.ipynb".
The first task is:
This code is buggy, because the 'north-south' direction is green, while the 'east-west' direction is yellow. The cars crash into each other at intersections.
We need to create a sanity check with an assert statement to make sure illogical things throw errors.
Can you write Python code for it?

market_2nd = {'ns':'green', 'ew':'red'} #ns: north south, ew: east west

def switchLights(intersection):
    for key in intersection.keys():
        if intersection[key] == 'green':
            intersection[key] = 'yellow'
        elif intersection[key] == 'yellow':
            intersection[key] = 'red'
        elif intersection[key] == 'red':
            intersection[key] = 'green'
        # if false, raise assert statement
        assert 'red' in intersection.values(), 'Neither light is red at ' + str(intersection) + '!'

print(market_2nd)
switchLights(market_2nd) # Run function in data structure
print(market_2nd)


I am working on the file "AutomateTheBoringStuffWithPython/lesson36.ipynb".
The first task is:
The factorial program is returning a 0 output, which is invalid. We can use a log to find the error(s).
Build a log framerwork:
Can you write Python code for it?

import logging

# For iPython/Jupyter, logging is handled at the root level. There are two ways to reacces logging

# Way 1: (Easy way)
import imp # Import the 'import' internals module
imp.reload(logging) # Reload the logging module for this specific instance

# Way 2: (Less Easy Way)
# Must define a logging object for this instance:
# logger = logging.getLogger() # Logging object
# logger.setLevel(logging.DEBUG) # Setting logging level to the level defined in the program
# This can be used by passing the normal logging.debug('message') as usual

logging.basicConfig(level=logging.DEBUG, format = '%(asctime)s - %(levelname)s - %(message)s') # Format for basic logging

logging.debug('Start of Program') # Start of program in log

def factorial(n):
    """
    A function for finding factorials.
    """
    logging.debug('Start of the factorial(%s)' % (n)) # Start the program and pass the argument to the log
    total = 1
    for i in range(n+1):
        total *= i # Total = total multiplied by ever element i added to itself
        logging.debug('i is the %s, total is %s' % (i, total)) # Pass the actual function arguments to the log

    logging.debug('Return value is %s' % (total)) # Return the total argument to the log
    return total

print(factorial(5))
logging.debug('End of Program') # End of program in log


I am working on the file "AutomateTheBoringStuffWithPython/lesson36.ipynb".
The first task is:
There are 5 log levels defined in Python:
* Debug: The lowest level, for testing, accessed via logging.debug().
* Info: The information level, for informing the programmer, accessed via logging.info().
* Warning: The warning level, where something could go wrong, accessed via logging.warnig().
* Error: The error level, where something has gone wrong, accessed via logging.error().
* Critical: The highest level, where something has gone wrong, and might stop the program, accessed via logging.critical().
logging.disable() disables that level and every element below it.
logging.disable(logging.WARNING) would therefore disable the Warning, Info and Debug levels.
logging.disable(logging.CRITICAL) would disable all levels except Critical.
To put the log outputs in a text file, you can modify the logging.basicConfig() to take a filename parameter.
Formerly, the try and except statements were used to handle exceptions, but you can also create your own exceptions with raise.
Can you write Python code for it?

import logging
import os

import imp
imp.reload(logging)

#logging.disable(logging.CRITICAL) # Switch the log level from 'DEBUG' to 'CRITICAL'; only show 'critical' level debugs

logging.basicConfig(filename = os.path.abspath('files/Factoriallog.txt'), level=logging.DEBUG, format = '%(asctime)s - %(levelname)s - %(message)s') # Format for basic logging

logging.debug('Start of Program') # Start of program in log

def factorial(n):
    """
    A function for finding factorials.
    """
    logging.debug('Start of the factorial(%s)' % (n)) # Start the program and pass the argument to the log
    total = 1
    for i in range(1, n+1):
        total *= i # Total = total multiplied by ever element i added to itself
        logging.debug('i is the %s, total is %s' % (i, total)) # Pass the actual function arguments to the log

    logging.debug('Return value is %s' % (total)) # Return the total argument to the log
    return total

print(factorial(5))
logging.debug('End of Program') # End of program in log

print(open(os.path.abspath('files/Factoriallog.txt'), 'r').read()) # Open and read the created log file


I am working on the file "AutomateTheBoringStuffWithPython/lesson37.ipynb".
The first task is:
Factorial Program
Create a program that can return the n! of a number (5! = 1 * 2 * 3 * 4 * 5 = 120).
Can you write Python code for it?

import ipdb # Import iPython Debugger

import random

heads = 0

for i in range(1,1001):
    ipdb.set_trace() # Invoke the iPython debugger
    if random.randint(0, 1) == 1:
        heads = heads + 1
    if i == 500:
        print('Halfway done!')

print('Heads came up ' + str(heads) + ' times.')


I am working on the file "AutomateTheBoringStuffWithPython/lesson37.ipynb".
The first task is:
The problem with the program was that input() stores its responses as 'strings', and therefore concatenates as a string, not an integer. Assing an int() fixes this problem:
Can you write Python code for it?

#import ipdb # Import the iPython debugger

def simpleAdd():
    #ipdb.set_trace() # Invoke the iPython debugger
    print('Enter the first number to add:')
    first = int(input())
    print('Enter the second number to add:')
    second = int(input())
    print('Enter the third number to add:')
    third = int(input())
    print('The sum is ' + str(first + second + third) +'.')
simpleAdd()


I am working on the file "AutomateTheBoringStuffWithPython/lesson37.ipynb".
The first task is:
Tracing the function gains the following output:
Can you write Python code for it?

import ipdb # Import iPython Debugger

def blah():
    ipdb.set_trace() # Invoke the iPython debugger
    print('blah')
    print('blah')
    print('blah')
    moreblah()
    print('blah')
    print('blah')
    print('blah')
    evenmoreblah()

def moreblah():
    print('more blah')
    print('more blah')
    print('more blah')
    evenmoreblah()

def evenmoreblah():
    print('even more blah')

print(blah())


I am working on the file "AutomateTheBoringStuffWithPython/lesson38.ipynb".
The first task is:
To run this is a script, we would need to the full path to python, followed by the script, followed by the address.
An easy way to skip this process is to create a shell script holding the script addresses, and passing arguments to it:
Can you write Python code for it?

#! usr/bin/env bash

#python3 mapit.py %*


I am working on the file "AutomateTheBoringStuffWithPython/lesson39.ipynb".
The first task is:
Response objects can be checked via status codes:

'404' is the typical 'file not found' code.
'200' is the typical 'success' code.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson40.ipynb".
The first task is:
No errors were raised, we can now parse the webpage text:
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson40.ipynb".
The first task is:
The element was sucessful imported and stored.
Amazon Price Scraper Program:
If we tie this all together, we can create a simple program to perform these steps.
Can you write Python code for it?

import bs4, requests

def getAmazonPrice(productUrl):
    # Use requests to download a URL
    res = requests.get(productUrl)
    # Raise for status to check for errors and crash if there are issues
    res.raise_for_status()

    # Pass the HTML to Beautiful Soup
    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    # Pass the price CSS selector, and store into a list of all matching elements
    elems = soup.select('#buyNewSection > div > div > span > span')
    # Examine and return the text of the first (and only) element in the list
    return elems[0].text.strip()

price = getAmazonPrice('http://www.amazon.ca/Automate-Boring-Stuff-Python-Programming/dp/1593275994')
print('The current price of \'Automate the Boring Stuff with Python\' on Amazon is ' + price)


I am working on the file "AutomateTheBoringStuffWithPython/lesson40.ipynb".
The first task is:
This typically raises a warning, but it is not an exception. It can be solved by using bs4.BeautifulSoup(res.text, 'html.paser') as described.
We can now find elements in the page, and we can do this via CSS selections and the .select method.
In Chrome, we can copy the CSS Path via 'Inspect > Copy >  Copy selector'.
Can you write Python code for it?

# Store this element in that list at index 0
elems = soup.select('#buyNewSection > div > div > span > span')

print(elems[0])
print(elems[0].text)


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
All web elements have a .text() variable that stores all text of the element.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
Once selected, we can a run a .click() method on that element to interact with the selected web element.
Can you write Python code for it?

# Clicks on that first link

elem.click()


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
The webdriver.Chrome() command launches an interactive browser using selenium. All commands referencing this new browser variable will use this specific browser window. There are a variety of browsers available, they just need to be installed.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
There are many ways to use the webdriver to find elements, and they are summarized here.
There are also ways to interact with a webpage, and input our own text as necessary. Let use the search box of the Automate website as the example:
Can you write Python code for it?

# Element by class name example

searchElem = browser.find_element_by_class_name('search-field')


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
We can also quit the entire browser with .quit().
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
We can also select multiple elements on a webpage and storing them in a list using the find_elements_by method instead of find_element_by.
Can you write Python code for it?

# Find all paragraph elements on the current page in the browser (Introduction)

elems = browser.find_elements_by_css_selector('p')

elems


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
We can also use Python scripts to read the content via the selenium browser.
Can you write Python code for it?

browser = webdriver.Chrome()
browser.get('https://automatetheboringstuff.com')

elem = browser.find_element_by_css_selector('#post-6 > div > p:nth-child(6)')


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
We can also use browser functions themselves programmatically, including functions like 'back', 'forward', 'refresh', etc.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
We can now use CSS selectors to interact with specific elements on this page; you can find them in Chrome using the Developer tools:

This web element is then referenced using the browser method find_element_by_css_selector. There are a variety of other find methods, which can reference names, classes, tags, etc.
Can you write Python code for it?

elem = browser.find_element_by_css_selector('#post-6 > div > ol:nth-child(20) > li:nth-child(1) > a')


I am working on the file "AutomateTheBoringStuffWithPython/lesson41.ipynb".
The first task is:
We can then use commands like browser.get() to load urls:
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson42.ipynb".
The first task is:
All cell values inherit their data types from Excel.
Can you write Python code for it?

print("The value in cell %s is '%s' and is type %s." %('A1', sheet['A1'].value, type(sheet['A1'].value)))
print("The value in cell %s is '%s' and is type %s." %('B1', sheet['B1'].value, type(sheet['B1'].value)))
print("The value in cell %s is '%s' and is type %s." %('C1', sheet['C1'].value, type(sheet['C1'].value)))


I am working on the file "AutomateTheBoringStuffWithPython/lesson42.ipynb".
The first task is:
Once the workbook is loaded, we can interact with specific sheets by loading them via workbook methods.
Can you write Python code for it?

sheet = workbook.get_sheet_by_name('Sheet1')
type(sheet)


I am working on the file "AutomateTheBoringStuffWithPython/lesson42.ipynb".
The first task is:
This particular cell returns a datetime reference from Excel via Python's own datetime module. A string value is available by passing into the str() function:
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson42.ipynb".
The first task is:
We can also use the .get_sheet_names() method to print all sheet names, in case we aren't sure.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson42.ipynb".
The first task is:
We can now interact with specific cells by creating cell objects, referenced via a sheet method.
Can you write Python code for it?

# Just references an object exists; requires an additional method to interact with
sheet['A1']


I am working on the file "AutomateTheBoringStuffWithPython/lesson42.ipynb".
The first task is:
We must first nagivate to the directory containing the spreadsheets, which for this notebook is the subdirectory 'files'.
Can you write Python code for it?

# Import OS module to navigate directories
import os

# Change the directory to the excel file location, using relative and absolute paths as previously discussed.
os.chdir('files')

os.listdir()


I am working on the file "AutomateTheBoringStuffWithPython/lesson42.ipynb".
The first task is:
You can also reference cells via rows and columns. Excel rows start at 1 and columns at A.
Can you write Python code for it?

# B1 Cell
sheet.cell(row = 1, column = 2)


I am working on the file "AutomateTheBoringStuffWithPython/lesson44.ipynb".
The first task is:
Despite its functionality, there are some PDFs that even this module can't perfectly handle. This is simply the nature of PDF files.
We must first change the directory to the folder holding PDF files, to avoid having to reference full paths.
Can you write Python code for it?

import os

os.chdir('files')

os.listdir()


I am working on the file "AutomateTheBoringStuffWithPython/lesson44.ipynb".
The first task is:
Note that while this translation is not perfect, it is sufficient for understanding the content of the PDF File.

To extract all the text from this PDF document, we can loop this method over all pages:
Can you write Python code for it?

for pageNum in range(reader.numPages):
    print(reader.getPage(pageNum).extractText())


I am working on the file "AutomateTheBoringStuffWithPython/lesson44.ipynb".
The first task is:
Now the writer object contains all the pages, and we must save  this object (at the location defined earlier, or the absolute path provided).
We need to  first'open' a file we can save these 'reader' object to.
Can you write Python code for it?

outputFile = open('combinedminutes2.pdf', 'wb')

writer.write(outputFile)


I am working on the file "AutomateTheBoringStuffWithPython/lesson44.ipynb".
The first task is:
Once created, we can use methods like .extractText to retrieve the text of that page object.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson44.ipynb".
The first task is:
The .getPage() method loads a specific page, and turns it into a 'page' object for even more interaction.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson44.ipynb".
The first task is:
The PyPDF2 module also includes a 'writer' as well as a 'reader', which can create new PDF files. However, it cannot write arbitrary text to PDFs, due to the binary nature. It can therefore only add, remove, or reorder pages, not the text or layout itself.
We can use this function to manage PDF pages across multiple documents in a batch approach.
Can you write Python code for it?

import PyPDF2

# Open up both example PDFs in read-binary mode
pdf1File = open('meetingminutes.pdf', 'rb')
pdf2File = open('meetingminutes2.pdf', 'rb')

# Now generate readers for both of these functions
reader1 = PyPDF2.PdfFileReader(pdf1File)
reader2 = PyPDF2.PdfFileReader(pdf2File)

# Now create a writer object that will merge these PDFs into one pdf.
# this object only exists in memory at the moment, and will need to be saved later.
writer = PyPDF2.PdfFileWriter()


I am working on the file "AutomateTheBoringStuffWithPython/lesson44.ipynb".
The first task is:
The newly created file is now available as combinedminutes2.pdf.
We can now close all the open files, to make sure no further edits are done.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson44.ipynb".
The first task is:
We can now loop through all the pages in the 'reader' objects and add them to the 'writer object'.
Can you write Python code for it?

# For every page in reader 1
for pageNum in range(reader1.numPages):
    # Create a page object at that page
    page = reader1.getPage(pageNum)
    # Add that page object to the writer defined earlier
    writer.addPage(page)

# For every page in reader 2
for pageNum in range(reader2.numPages):
    # Create a page object at that page
    page = reader2.getPage(pageNum)
    # Add that page object to the writer defined earlier
    writer.addPage(page)


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
Documents have a few more features than plaintext files. They have the following objects in this module:
* The entire document is contained within the document object.
* The document object contains paragraph objects, which is each paragraph in the document.
* The paragraph object contains one or more run objects.
We first navigate into the subdirectory containing the Word documents, using relative or absolute paths.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
Every paragraph object includes a .text variable containing the string of the document.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
Run objects have a .bold or .italics variable, identifying whether that run has that style property.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
We can also create runs within these paragraph objects.
Can you write Python code for it?

# Select the first paragraph we created.
p = d.paragraphs[0]

# Add a new run to that paragraph.
p.add_run('This is a new run')

# Set the newly added run (Index 1) to a True bold variable
p.runs[1].bold = True

# Save changes
d.save('demo4.docx')


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
We can list the paragraphs within this document object.
A paragraph object is any text seperated by a newline ('Enter').
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
We can now add paragraphs to this document object.
Can you write Python code for it?

d.add_paragraph('This is a paragraph.')

d.add_paragraph('This is another paragraph.')


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
We can save style attributes using this approach.
Can you write Python code for it?

p.runs[3].text
p.runs[3].underline = True


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
We can save this created document to the hard drive using the .save() method again. It will be saved here as demo4.docx.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
We then use the Document() function to open the word document as a document object.
For this example, we will be using 'demo.docx'.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson45.ipynb".
The first task is:
We will test it on demo.docx and demo4.docx.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson46.ipynb".
The first task is:
Once we get a good response, we can now send emails via the .sendmail() method, which takes a variety of parameters, but requires a 'from' email, a 'to' email, and some content.
Most modern email services include protections to stop users from spamming other users, so they cap the number of emails that can be sent per day (Gmail is roughly 150 emails).
Can you write Python code for it?

# For this exercise, we will just send an email to and from the same address; be sure to update these values with your
# own email addresses

from_email = 'youremail@gmail.com'
to_email = 'theiremail@gmail.com'

conn.sendmail(from_email, to_email, 'Subject: Python Test Email \n\n Look,\n script text!')


I am working on the file "AutomateTheBoringStuffWithPython/lesson46.ipynb".
The first task is:
The first step in setting up email is creating a connection object to interact with an email/SMTP server.
This function gets a domain name and a port number. The port number for SMTP is typically 587; it is part of the internet address you are sending data to. The domain name is the address at which an SMTP server is hosted (typically the domain name with 'smtp' appended to it). Here are some additional examples.
Can you write Python code for it?

conn = smtplib.SMTP('smtp.gmail.com', 587)


I am working on the file "AutomateTheBoringStuffWithPython/lesson46.ipynb".
The first task is:
The returned variable is the response code (anything starting with a 2 is good), followed by a byte sting.
Since we have a positive response, we can now start an encrypted TLS connection (which is the newer version of SSL technology, both of which are based on public cryptography). Most modern email services will require TLS to begin interaction.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson46.ipynb".
The first task is:
We will now use the .ehlo() method to test and connect.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
Because this value is returned as a collections.defaultdict, we must use a series of keys to parse it and explore its values. This process is explained more thoroughly on Mr. Sweigart's blog..
Below is a rough explanation of the key approach, as initially  explain the process behind keys.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
Once this value has been stored, we can use the .get() method to pull out various variables from the email.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
The first step in setting up email is creating a connection object again to interact with an email, but this time for the IMAP server.
This function takes a domain name, and an optional parameter asking if we are using SSL encrpytion (which is a variant of TLS discussed in Lesson 45). This domain name again depends on your email provider, an example list is available on Mr. Sweigarts blog in 16-2.
Can you write Python code for it?

conn = imapclient.IMAPClient('imap.gmail.com', ssl=True)


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
We can end our session using the .logout method on the connection object.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
We can now access different elements in this message object.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
We can now pass in our login paramaters via the .login() method.
Can you write Python code for it?

# Real values were used in testing, and removed for Github
# Due to the nature of Gmail's security, you may have to allow access from 'less secure apps' (like this script)
# The setting can be changed here: https://www.google.com/settings/u/2/security/lesssecureapps

conn.login('youremail@gmail.com','yourpassword')


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
We get a byte string with a response if we login correctly.
Next we choose which email folder to access with the .select_folder method. You will typically want the 'Inbox' folder. We will also pass a readonly=True parameter to make sure we don't accidentally delete something. We can use the .list_folders() method to list all available folders, and switch the readonly parameter to false to actually edit messages.
Can you write Python code for it?

conn.list_folders()

conn.select_folder('INBOX', readonly=True)


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
We get a series of byte strings, if we have connected correctly.
We can now search for specific emails. However, IMAP has its own search syntax, and requires special operators, and will return a list of UIDs (Unique IDs) describing each email. There are many available search keys, depending on the service, with more examples here in 16-3.
Can you write Python code for it?

UIDs = conn.search('SINCE 26-May-2016')
print(UIDs)


I am working on the file "AutomateTheBoringStuffWithPython/lesson47.ipynb".
The first task is:
We now have to translate these UIDs into the actual emails, and we can do that using the .fetch() method.
It requires a list of UIDs to retrive, as well as list of what parts of the email you need; typically ['BODY[]','FLAGS'] which contains most of the information an average user might need.
Below is the email we sent as part of Lesson 46, which we will store in a variable.
Can you write Python code for it?

rawMessage = conn.fetch([88177], ['BODY[]','FLAGS'])
print(rawMessage)

type(rawMessage)


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
Now that we have mastered movement, we can now use click() functions to interact with objects.
Can you write Python code for it?

# Find the 'Help' button in the top Jupyter Navigation
# helpCoordinates = pyautogui.position()

helpCoordinates = (637, 126)

# Click at those coordinates
pyautogui.click(helpCoordinates)


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
Similarly, the position() function returns the current position of the mouse cursor.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
The first function to control the mouse is the moveTo() function, which moves the mouse immediately to an absolute location.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
The left and right most corners are 1 less than the maximum.
Can you write Python code for it?

# left corner
print(pyautogui.position())

# right corner
print(pyautogui.position())


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
This lesson will control all the mouse controlling functions in this module.
The module treats the screen as cartesian coordinates of pixels, with x referencing a point on the horizontal line, and y referencing a point on the vertical line.

We can examine the available screen size using the size() function, with variables returned as an (x,y) pair.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
To make pyautogui useful, you need to know the coordinates on the screen at any given time, and interact with them. The module contains a useful sub program, called displayMousePosition, which can be run via the terminal to track the position of the real-time mouse at any given time (it also returns RGB values.)
Can you write Python code for it?

# Not run here, just keeps spitting out print functions. Much more useful in the terminal.
pyautogui.displayMousePosition()


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
We can also pass in y coordinates to move up or down, but we have to use negative values to move 'up' in a relative way.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
We can pass the duration parameter to this function slow down this process to simulate human activity over the duration defined.
Can you write Python code for it?


I am working on the file "AutomateTheBoringStuffWithPython/lesson48.ipynb".
The first task is:
We can store this tuple in two variables, width and height:
Can you write Python code for it?