I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Generator network

Here we'll build the generator network. To make this network a universal function approximator, we'll need at least one hidden layer. We should use a leaky ReLU to allow gradients to flow backwards through the layer unimpeded. A leaky ReLU is like a normal ReLU, except that there is a small non-zero output for negative input values.
Variable Scope
Here we need to use tf.variable_scope for two reasons. Firstly, we're going to make sure all the variable names start with generator. Similarly, we'll prepend discriminator to the discriminator variables. This will help out later when we're training the separate networks.
We could just use tf.name_scope to set the names, but we also want to reuse these networks with different inputs. For the generator, we're going to train it, but also sample from it as we're training and after training. The discriminator will need to share variables between the fake and real input images. So, we can use the reuse keyword for tf.variable_scope to tell TensorFlow to reuse the variables instead of creating new ones if we build the graph again.
To use tf.variable_scope, you use a with statement:
python
with tf.variable_scope('scope_name', reuse=False):
    # code here
Here's more from the TensorFlow documentation to get another look at using tf.variable_scope.
Leaky ReLU
TensorFlow doesn't provide an operation for leaky ReLUs, so we'll need to make one . For this you can just take the outputs from a linear fully connected layer and pass them to tf.maximum. Typically, a parameter alpha sets the magnitude of the output for negative values. So, the output for negative input (x) values is alpha*x, and the output for positive x is x:
$$
f(x) = max(\alpha * x, x)
$$
Tanh Output
The generator has been found to perform the best with $tanh$ for the generator output. This means that we'll have to rescale the MNIST images to be between -1 and 1, instead of 0 and 1.
Can you write Python code for it?

def generator(z, out_dim, n_units=128, reuse=False,  alpha=0.01):
    ''' Build the generator network.

        Arguments
        ---------
        z : Input tensor for the generator
        out_dim : Shape of the generator output
        n_units : Number of units in hidden layer
        reuse : Reuse the variables with tf.variable_scope
        alpha : leak parameter for leaky ReLU

        Returns
        -------
        out, logits: generated outputs after and before activation

    '''
    with tf.variable_scope('generator', reuse=reuse):
        # Hidden layer
        #h1 = tf.contrib.layers.fully_connected(z, n_units, activation_fn=None)
        # shorter
        h1 = tf.layers.dense(z, n_units)

        # Leaky ReLU
        h1 = tf.maximum(alpha * h1, h1)

        # Logits and tanh output
        #logits =  tf.contrib.layers.fully_connected(h1, out_dim, activation_fn=None)
        # shorter
        logits =  tf.layers.dense(h1, out_dim)
        out = tf.tanh(logits)

        return out, logits


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Generator samples from training
Here we can view samples of images from the generator. First we'll look at images taken while training.
Can you write Python code for it?

def view_samples(epoch, samples):
    print(len(samples[0][0]),len(samples[0][1]))
    fig, axes = plt.subplots(figsize=(7,7), nrows=4, ncols=4, sharey=True, sharex=True)
    for ax, img in zip(axes.flatten(), samples[epoch][0]):
        ax.xaxis.set_visible(False)
        ax.yaxis.set_visible(False)
        im = ax.imshow(img.reshape((28,28)), cmap='Greys_r')

    return fig, axes

# Load samples from generator taken while training
with open('train_samples.pkl', 'rb') as f:
    samples = pkl.load(f)


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
It starts out as all noise. Then it learns to make only the center white and the rest black. You can start to see some number like structures appear out of the noise. Looks like 1, 9, and 8 show up first. Then, it learns 5 and 3.
Sampling from the generator
We can also get completely new images from the generator by using the checkpoint we saved after training. We just need to pass in a new latent vector $z$ and we'll get new samples!
Can you write Python code for it?

saver = tf.train.Saver(var_list=g_vars)
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))
    sample_z = np.random.uniform(-1, 1, size=(16, z_size))
    gen_samples = sess.run(
                   generator(input_z, input_size, n_units=g_hidden_size, reuse=True, alpha=alpha),
                   feed_dict={input_z: sample_z})
view_samples(0, [gen_samples])


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Model Inputs
First we need to create the inputs for our graph. We need two inputs, one for the discriminator and one for the generator. Here we'll call the discriminator input inputs_real and the generator input inputs_z. We'll assign them the appropriate sizes for each of the networks.

Exercise: Finish the model_inputs function below. Create the placeholders for inputs_real and inputs_z using the input sizes real_dim and z_dim respectively.
Can you write Python code for it?

def model_inputs(real_dim, z_dim):
    inputs_real = tf.placeholder(tf.float32, (None, real_dim), 'input_real')
    inputs_z = tf.placeholder(tf.float32, (None, z_dim), 'input_z')
    return inputs_real, inputs_z


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Optimizers
We want to update the generator and discriminator variables separately. So we need to get the variables for each part and build optimizers for the two parts. To get all the trainable variables, we use tf.trainable_variables(). This creates a list of all the variables we've defined in our graph.
For the generator optimizer, we only want to generator variables. Our past selves were nice and used a variable scope to start all of our generator variable names with generator. So, we just need to iterate through the list from tf.trainable_variables() and keep variables that start with generator. Each variable object has an attribute name which holds the name of the variable as a string (var.name == 'weights_0' for instance).
We can do something similar with the discriminator. All the variables in the discriminator start with discriminator.
Then, in the optimizer we pass the variable lists to the var_list keyword argument of the minimize method. This tells the optimizer to only update the listed variables. Something like tf.train.AdamOptimizer().minimize(loss, var_list=var_list) will only train the variables in var_list.

Exercise:  Below, implement the optimizers for the generator and discriminator. First you'll need to get a list of trainable variables, then split that list into two lists, one for the generator variables and another for the discriminator variables. Finally, using AdamOptimizer, create an optimizer for each network that update the network variables separately.
Can you write Python code for it?

# Optimizers
learning_rate = 0.002

# Get the trainable_variables, split into G and D parts
t_vars = tf.trainable_variables()
g_vars = [x for x in tf.trainable_variables() if 'generator' in x.name]
d_vars = [x for x in tf.trainable_variables() if x.name.startswith('discriminator')]

d_train_opt = tf.train.AdamOptimizer(learning_rate).minimize(d_loss, var_list=d_vars)
g_train_opt = tf.train.AdamOptimizer(learning_rate).minimize(g_loss, var_list=g_vars)


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Training loss
Here we'll check out the training losses for the generator and discriminator.
Can you write Python code for it?

%matplotlib inline

import matplotlib.pyplot as plt

fig, ax = plt.subplots()
losses = np.array(losses)
plt.plot(losses.T[0], label='Discriminator')
plt.plot(losses.T[1], label='Generator')
plt.title("Training Losses")
plt.legend()


I am working on the file "16--numpy.ipynb".
The first task is:
And to check whether two Numpy array variables point to the same object, you can use the numpy.may_share_memory() function:
Can you write Python code for it?

print ("C and C_view share memory: %s" % np.may_share_memory (C, C_view))
print ("C and C_copy share memory: %s" % np.may_share_memory (C, C_copy))


I am working on the file "16--numpy.ipynb".
The first task is:
Besides arange(), you can also define an interval and a number of points. What does the following code do?
Can you write Python code for it?

print np.linspace (0, 1, 10)

print np.linspace (0, 1, 10, endpoint=False)


I am working on the file "16--numpy.ipynb".
The first task is:
Exercise. Complete the prime number sieve algorithm, which is illustrated below.
<img src="https://raw.githubusercontent.com/rvuduc/cse6040-ipynbs/master/16--numpy-prime-sieve.png" alt="Exercise: Extract these slices" width="480">
That is, given a positive integer $n$, the algorithm iterates from $i \in {2, 3, 4, \ldots, \left\lfloor\sqrt{n}\right\rfloor}$, repeatedly "crossing out" values that are strict multiples of $i$. "Crossing out" means maintaining an array of, say, booleans, and setting values that are multiples of $i$ to False.
Can you write Python code for it?

from math import sqrt

n = 20
is_prime = np.ones (n+1, dtype=bool) # the "sieve"

# Initial values
is_prime[0:2] = False
is_prime[2:] = True

# Sieving loop
for i in range (2, int (sqrt (n))):
    # Fill in your code here
    is_prime[2*i::i] = False

# Prints your primes
print ("==> Primes through %d:" % n)
print np.nonzero (is_prime)[0]


I am working on the file "16--numpy.ipynb".
The first task is:
Exercise. Consider the following $6 \times 6$ matrix, which has 4 different subsets highlighted.
<img src="https://raw.githubusercontent.com/rvuduc/cse6040-ipynbs/master/16--numpy-slicing-exercise.png" alt="Exercise: Extract these slices" width="240">
Write some code to generate this matrix, named Z. Then, for each subset illustrated above, write an indexing or slicing expression that extracts the subset.
Can you write Python code for it?

Z = ... # INSERT YOUR CODE HERE

print ("\n==> Orange slice:")
print Z[...] # Edit to print the orange-ish subset

print ("\n==> Red slice:")
print Z[...] # Edit to print the red-ish subset

print ("\n==> Blue slice:")
print Z[...] # Edit to print the blue-ish subset

print ("\n==> Green slice:")
print Z[...] # Edit to print the green-ish subset


I am working on the file "16--numpy.ipynb".
The first task is:
Exercise. Write a function that iterates over the columns of a matrix and scales each column by its index. That is, column $A(:, j)$ should be replaced by $j*A(:, j)$.
Then use that routine to measure the difference in time when the input is row-major versus column-major.

Note: In Numpy, this operation can be easily done just by multiplying
Can you write Python code for it?

def scale_colwise (A):
    """Given a matrix `A`, visits each column `A[:, j]` and scales it by `j`."""
    # Fill in this code
    pass

# Measure time to scale a row-major input column-wise
%timeit scale_colwise (A_rowmaj)

# Measure time to scale a column-major input column-wise
%timeit scale_colwise (A_colmaj)


I am working on the file "16--numpy.ipynb".
The first task is:
Getting help. By the way, if you need help getting documentation from within this notebook, here are some handy shortcuts.
Can you write Python code for it?

# Append '?' to get help on a specific routine
np.array?

# Search for key text
np.lookfor ("creating array")

# Wildcard search
np.con*?


I am working on the file "16--numpy.ipynb".
The first task is:
Incidentally, there is a very cute way to create the above matrix. I would have never guessed this method, so I've put some print statements so you can see what it is doing.
Can you write Python code for it?


I am working on the file "16--numpy.ipynb".
The first task is:
Indexing and slicing
The usual 0-based slicing and indexing notation you know and love from lists is also supported for Numpy arrays. In the multidimensional case,  including their natural multidimensional analogues with index ranges separated by commas.
Can you write Python code for it?


I am working on the file "16--numpy.ipynb".
The first task is:
Indirect addressing
Another common indexing operation is using either a boolean mask or a set of integer indices.
Can you write Python code for it?

np.random.seed(3)
x = np.random.random_integers(0, 20, 15) # 15 random ints in [0, 20)
print x

# Find all positive multiples of 3
mask_mult_3 = (x % 3 == 0) & (x > 0)
print mask_mult_3
print x[mask_mult_3]

# Pull out an arbitrary subset of elements
inds = np.array ([3, 7, 8, 12])
print x[inds]


I am working on the file "16--numpy.ipynb".
The first task is:
You can also create empty (uninitialized) arrays. What does the following produce?
Can you write Python code for it?

A = np.empty ((3, 4)) # An empty 3 x 4 matrix
print A


I am working on the file "16--numpy.ipynb".
The first task is:
You can force a copy using the .copy() method:
Can you write Python code for it?


I am working on the file "16. 과최적화와 정규화/01. 다항 회귀와 과최적화.ipynb".
The first task is:
Scikit-Learn을 이용한 다항 회귀
Scikit-Learn에서는 preprocessing 서브 패키지의 PolynomialFeatures 클래스를 사용하여 다항 회귀를 할 수 있다.
Can you write Python code for it?

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline

def polyreg(degree, seed=0, plot=True):
    polynomial_features = PolynomialFeatures(degree=degree)
    linear_regression = LinearRegression()
    model = Pipeline([("polynomial_features", polynomial_features),
                      ("linear_regression", linear_regression)])

    np.random.seed(seed)
    n_samples = 30
    X = np.sort(np.random.rand(n_samples))
    y = np.cos(1.5 * np.pi * X) + np.random.randn(n_samples) * 0.1
    X = X[:, np.newaxis]

    model.fit(X, y)

    if plot:
        plt.scatter(X, y)
        xx = np.linspace(0, 1, 1000)
        plt.plot(xx, model.predict(xx[:, np.newaxis]))
        plt.ylim(-2, 2)
        plt.show()

    reg = model.named_steps["linear_regression"]
    return reg.coef_, reg.intercept_

polyreg(1)

polyreg(2)

polyreg(3)

polyreg(4)

polyreg(5)

polyreg(50)


I am working on the file "16. 과최적화와 정규화/01. 다항 회귀와 과최적화.ipynb".
The first task is:
과최적화
모형을 특정 샘플 데이터에 대해 과도하게 최적화하는 것을 과최적화(overfitting)이라고 한다.
과최적화는
* 독립 변수 데이터 갯수에 비해 모형 모수의 수가 과도하게 크거나
* 독립 변수 데이터가 서로 독립이 아닌 경우에 발생한다.
이러한 상황에서는 같은 조건에 대해 답이 복수개 존재할 수 있기 때문이다.
과최적화가 문제가 되는 이유는 다음과 같다.

샘플이 조금만 변화해도 fitting 결과가 크게 달라지며
트레이닝에 사용되지 않은 새로운 독립 변수 값을 입력하면 오차가 커진다. (cross-validation 오차)
Can you write Python code for it?

polyreg(2, 0)

polyreg(2, 1)

polyreg(2, 2)

polyreg(50, 1)

polyreg(50, 2)

polyreg(50, 3)


I am working on the file "16. 과최적화와 정규화/02. 교차 검증.ipynb".
The first task is:
Label K-Fold

같은 label이 test와 train에 동시에 들어가지 않게 조절
label에 의한 영향을 최소화
Can you write Python code for it?

from sklearn.cross_validation import LabelKFold
cv = LabelKFold(y, n_folds=3)
for train_index, test_index in cv:
    print("test  y:", y[test_index])
    print("." * 80 )
    print("train y:", y[train_index])
    print("=" * 80 )


I am working on the file "16. 과최적화와 정규화/02. 교차 검증.ipynb".
The first task is:
ShuffleSplit

중복된 데이터를 허용
Can you write Python code for it?

from sklearn.cross_validation import ShuffleSplit
cv = ShuffleSplit(5)
for train_index, test_index in cv:
    print("test X:", X[test_index])
    print("=" * 20 )


I am working on the file "16. 과최적화와 정규화/02. 교차 검증.ipynb".
The first task is:
교차 평가 시행
CV는 단순히 데이터 셋을 나누는 역할을 수행할 뿐이다. 실제로 모형의 성능(편향 오차 및 분산)을 구하려면 이렇게 나누어진 데이터셋을 사용하여 평가를 반복하여야 한다. 이 과정을 자동화하는 명령이 cross_val_score() 이다.

cross_val_score(estimator, X, y=None, scoring=None, cv=None)

cross validation iterator cv를 이용하여 X, y data 를 분할하고 estimator에 넣어서 scoring metric을 구하는 과정을 반복

인수

estimator : ‘fit’메서드가 제공되는 모형
X : 배열
독립 변수 데이터
y : 배열
종속 변수 데이터
scoring : 문자열
성능 검증에 사용할 함수
cv : Cross Validator
None 이면 디폴트인  3-폴드 CV
숫자 K  이면 K-폴드 CV

Cross Validator 클래스 객체

반환값

scores
계산된 성능 값의 리스트
Can you write Python code for it?

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
X, y, coef = make_regression(n_samples=1000, n_features=1, noise=20, coef=True, random_state=0)
model = LinearRegression()
cv = KFold(1000, 10)

scores = np.zeros(10)
for i, (train_index, test_index) in enumerate(cv):
    X_train = X[train_index]
    y_train = y[train_index]
    X_test = X[test_index]
    y_test = y[test_index]
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    scores[i] = mean_squared_error(y_test, y_pred)

scores

from sklearn.cross_validation import cross_val_score
cross_val_score(model, X, y, "mean_squared_error", cv)


I am working on the file "16. 과최적화와 정규화/03. 정규화 선형 회귀.ipynb".
The first task is:
Ridge 모형과 Lasso 모형의 차이
Ridge 모형은 가중치 계수를 한꺼번에 축소시키는데 반해 Lasso 모형은 일부 가중치 계수가 먼저 0으로 수렴하는 특성이 있다.
<img src="https://datascienceschool.net/upfiles/10a19727037b4898984a4330c1285486.png">
Can you write Python code for it?

from sklearn.datasets import load_diabetes
diabetes = load_diabetes()
X = diabetes.data
y = diabetes.target

ridge0 = Ridge(alpha=0).fit(X, y)
p0 = pd.Series(np.hstack([ridge0.intercept_, ridge0.coef_]))
ridge1 = Ridge(alpha=1).fit(X, y)
p1 = pd.Series(np.hstack([ridge1.intercept_, ridge1.coef_]))
ridge2 = Ridge(alpha=2).fit(X, y)
p2 = pd.Series(np.hstack([ridge2.intercept_, ridge2.coef_]))
pd.DataFrame([p0, p1, p2]).T

lasso0 = Lasso(alpha=0.0001).fit(X, y)
p0 = pd.Series(np.hstack([lasso0.intercept_, lasso0.coef_]))
lasso1 = Lasso(alpha=0.1).fit(X, y)
p1 = pd.Series(np.hstack([lasso1.intercept_, lasso1.coef_]))
lasso2 = Lasso(alpha=10).fit(X, y)
p2 = pd.Series(np.hstack([lasso2.intercept_, lasso2.coef_]))
pd.DataFrame([p0, p1, p2]).T


I am working on the file "16. 과최적화와 정규화/03. 정규화 선형 회귀.ipynb".
The first task is:
Scikit-Learn의 정규화 회귀 모형
Scikit-Learn 패키지에서는 정규화 회귀 모형을 위한 Ridge, Lasso, ElasticNet 이라는 별도의 클래스를 제공한다. 각 모형에 대한 최적화 목적 함수는 다음과 같다.

http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html

$$
\text{RSS} + \text{alpha} \sum w_i^2
$$

http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html

$$
0.5 \times \text{RSS}/N + \text{alpha} \sum |w_i|
$$

http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html

$$
0.5 \times \text{RSS}/N + 0.5 \times \text{alpha} \times \big(0.5 \times  (1-\text{l1_ratio})\sum w_i^2 + \text{l1_ratio} \sum |w_i| \big)
$$
Can you write Python code for it?

def plot_sklearn(model):
    plt.scatter(X, y)
    xx = np.linspace(0, 1, 1000)
    plt.plot(xx, model.predict(xx[:, np.newaxis]))
    plt.show()

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet

poly = PolynomialFeatures(9)
model = make_pipeline(poly, LinearRegression()).fit(X[:, np.newaxis], y)
print(model.steps[1][1].coef_)
plot_sklearn(model)

model = make_pipeline(poly, Ridge(alpha=0.01)).fit(X[:, np.newaxis], y)
print(model.steps[1][1].coef_)
plot_sklearn(model)

model = make_pipeline(poly, Lasso(alpha=0.01)).fit(X[:, np.newaxis], y)
print(model.steps[1][1].coef_)
plot_sklearn(model)

model = make_pipeline(poly, ElasticNet(alpha=0.01, l1_ratio=0.5)).fit(X[:, np.newaxis], y)
print(model.steps[1][1].coef_)
plot_sklearn(model)


I am working on the file "16. 과최적화와 정규화/03. 정규화 선형 회귀.ipynb".
The first task is:
path 메서드
Lasso 와 ElasticNet  클래스는 하이퍼 모수 alpha 값의 변화에 따른 계수의 변화를 자동으로 계산하는 path 메서드를 제공한다.
lasso_path(), enet_path() 명령어도 path 메서드와 동일한 기능을 수행한다.
Can you write Python code for it?

lasso = Lasso()
alphas, coefs, _ = lasso.path(X, y, alphas=np.logspace(-6, 1, 8))
df = pd.DataFrame(coefs, columns=alphas)
df

df.T.plot(logx=True)
plt.show()


I am working on the file "16. 과최적화와 정규화/03. 정규화 선형 회귀.ipynb".
The first task is:
모수 L1_wt가 0과 1 사이이면 Elastic Net 모형이다.
Can you write Python code for it?

result4 = model.fit_regularized(alpha=0.01, L1_wt=0.5)
print(result4.params)
plot_statsmodels(result4)


I am working on the file "16. 과최적화와 정규화/03. 정규화 선형 회귀.ipynb".
The first task is:
반대로 모수 L1_wt가 1 이면 순수 Lasso 모형이 된다.
Can you write Python code for it?

result3 = model.fit_regularized(alpha=0.01, L1_wt=1)
print(result3.params)
plot_statsmodels(result3)


I am working on the file "16. 과최적화와 정규화/03. 정규화 선형 회귀.ipynb".
The first task is:
정규화 모형의 장점
정규화 모형은 회귀 분석에 사용된 데이터가 달라져도 계수가 크게 달라지지 않도록 한다.
Can you write Python code for it?

X_train = np.c_[.5, 1].T
y_train = [.5, 1]
X_test = np.c_[-1, 3].T
np.random.seed(0)

models = {"LinearRegression": LinearRegression(),
          "Ridge": Ridge(alpha=0.1)}

for i, (name, model) in enumerate(models.iteritems()):
    ax = plt.subplot(1, 2, i+1)
    for _ in range(10):
        this_X = .1 * np.random.normal(size=(2, 1)) + X_train
        model.fit(this_X, y_train)
        ax.plot(X_test, model.predict(X_test), color='.5')
        ax.scatter(this_X, y_train, s=100, c='.5', marker='o', zorder=10)
        model.fit(X_train, y_train)
        ax.plot(X_test, model.predict(X_test), linewidth=3, color='blue', alpha=0.5)
        ax.scatter(X_train, y_train, s=100, c='r', marker='D', zorder=10)
        plt.title(name)
        ax.set_xlim(-0.5, 2)
        ax.set_ylim(0, 1.6)


I am working on the file "16. 과최적화와 정규화/04. 최적 정규화.ipynb".
The first task is:
One Standard Error Rule
성능 오차를 고려해서 어느 정도 오차는 감수하더라도 더 단순한(정규화 모형에서는 더 제약 조건이 강한) 모형을 선택하는 것이 실용적이다. 예를 들어 One Standard Error Rule 은 최적 모형상에서 교차 검증을 통해 성능 오차의 분산값을 구하고 이 값 정도의 성능 감소가 있더라도 최적 모형보다 더 단순하거나 제약조건이 더 강한 모형을 선택한다.
Can you write Python code for it?

from sklearn.linear_model import LassoCV

alphas = np.logspace(-4, -.5, 50)
lasso_cv = LassoCV(alphas=alphas, cv=5)
lasso_cv.fit(X, y)

scores = -lasso_cv.mse_path_.mean(axis=1)
scores_std = lasso_cv.mse_path_.std(axis=1)
scores_std1 = scores + scores_std / np.sqrt(len(lasso_cv.mse_path_))
scores_std2 = scores - scores_std / np.sqrt(len(lasso_cv.mse_path_))

alpha_1se = lasso_cv.alphas_[np.argmax(scores_std1 > np.max(scores))]
print(alpha_1se)

plt.semilogx(lasso_cv.alphas_, scores)
plt.semilogx(lasso_cv.alphas_, scores_std1, 'o-')
plt.semilogx(lasso_cv.alphas_, scores_std2, 'o-')
plt.axhline(np.max(scores), linestyle=':')
plt.axvline(lasso_cv.alpha_ , linestyle=':')
plt.axvline(alpha_1se, linestyle=':')
plt.ylabel('CV score')
plt.xlabel('alpha')
plt.show()


I am working on the file "16. 과최적화와 정규화/04. 최적 정규화.ipynb".
The first task is:
정규화 하이퍼 모수 최적화
정규화 모형에서 하이퍼 모수가 바뀌는 경우에도 마찬가지로 최적의 하이퍼 모수를 찾는 작업이 필요하다.
Can you write Python code for it?

from sklearn.datasets import load_diabetes
from sklearn.linear_model import Lasso
from sklearn.cross_validation import cross_val_score

data = load_diabetes()
X = data.data[:200]
y = data.target[:200]

model = Lasso()
alphas = np.logspace(-4, -.5, 50)

scores = list()

for alpha in alphas:
    model.alpha = alpha
    this_scores = cross_val_score(model, X, y, "mean_squared_error", cv=5)
    scores.append(np.mean(this_scores))

plt.semilogx(alphas, scores)
plt.ylabel('CV score')
plt.xlabel('alpha')
plt.axhline(np.max(scores), linestyle=':')
plt.show()

from sklearn.linear_model import LassoCV

alphas = np.logspace(-4, -.5, 50)
lasso_cv = LassoCV(alphas=alphas, cv=5)
lasso_cv.fit(X, y)

print(lasso_cv.alpha_ )

scores = -lasso_cv.mse_path_.mean(axis=1)
plt.semilogx(lasso_cv.alphas_, scores)
plt.axhline(np.max(scores), linestyle=':')
plt.axvline(lasso_cv.alpha_ , linestyle=':')
plt.ylabel('CV score')
plt.xlabel('alpha')
plt.show()


I am working on the file "160002934JuanVargas.ipynb".
The first task is:
Pregunta 3:
Sea X_n = ( aX_{n-1} + b ) mod m un generador congruencial con los siguientes parámetros: m=271, a=139, b=17 y X_0=9, generar los primeros 100 valores X_i y U_i. ¿Cúales son los primeros cinco numeros aleatorios U_i?
Respuesta: a. 0.6790, 0.4391, 0.0996, 0.9114, 0.7528
Can you write Python code for it?

def generadorCongruencial(n):

    m = 271
    a = 139
    b = 17
    Xo = 9
    lastXn = Xo
    Xi = []
    Ui = []
    print "Used parameters: "
    print "a = "+str(a)+" b = "+str(b)+" m = "+str(m)
    print "Random numbers generated: "
    for i in range(n):
        Xn = float((a*lastXn + b)% m)
        lastXn = Xn
        Xi.append(Xn)
        random = float(Xn/m)
        Ui.append(random)
    print 'Xi: ', Xi
    print ""
    print 'Ui: ', Ui
    print ""
    print 'Los primeros cinco numeros aleatorios son: ',Ui[0:5]

def main():
    generadorCongruencial(100)
main()


I am working on the file "160002934JuanVargas.ipynb".
The first task is:
Pregunta 5:
¿Cúal es el valor del estadístico de Chi-cuadrado para una secuencia de 2000 numeros aleatorios y un k=100 de un generador congruencial mixto con a = 16645, b = 13467, m = 317, X_0 = 999?
Respuesta: d. 12330
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

def generadorCongruencial(n):

    m = 317
    a = 16645
    b = 13467
    Xo = 999
    lastXn = Xo
    Xi = []
    Ui = []
    print "Used parameters: "
    print "a = "+str(a)+" b = "+str(b)+" m = "+str(m)
    print "Random numbers generated: "
    for i in range(n):
        Xn = float((a*lastXn + b)% m)
        lastXn = Xn
        Xi.append(Xn)
        random = float(Xn/m)
        Ui.append(random)
    return Ui

def chicuadrado():

    n = 2000
    x = generadorCongruencial(n)
    k = 100
    fi, clases, nadaimportante = plt.hist(x, k)
    # fi = observaciones encontradas, ei= observaciones esperadas
    ei = np.ones_like(clases)*(n/k)
    plt.plot(clases, ei, linewidth=3, color = '#FA58F4')
    plt.show()
    chi2 = 0
    ei = n/k
    for i in fi:
        chi2 += (i-ei)**2
    print "El estadistico de Chi-cuadrado para este generador es: ", chi2

def main():
    chicuadrado()
main()


I am working on the file "160002934JuanVargas.ipynb".
The first task is:
Pregunta 6:
Dados los siguientes valores de la distribución de probabilidad discreta de la variable aleatoria X de un dado: X = {1,2,3,4,5,6}, p(X) = {0.2,0.1,0.05,0.2,0.05,0.4}, ¿Qué valores retornaría el generador de variables aleatorias por el método de la inversión si los números aleatorios generados son: U = {0.4280, 0.5609, 0.0258, 0.6531, 0.8487, 0.0332, 0.6790, 0.4391}?
Respuesta: a. [4, 5, 1, 6, 6, 1, 6, 4]
Can you write Python code for it?

def genVAD(u, x, px):

    dAcum = distAcum (x,px)
    rList = u
    xList = []
    for i in rList:
        for j in range(len(dAcum)):
            if(i <= dAcum[j]):
                xList.append(x[j])
                break
    print "Generated numbers: "
    print xList
    plt.hist(xList)

def distAcum(x, px):

    Px = []
    suma = 0
    for i in range(len(x)):
        Px.append(suma + px[i])
        suma = suma + px[i]
    return Px

def main():
    x = [1,2,3,4,5,6]
    px = [0.2,0.1,0.05,0.2,0.05,0.4]
    u = [0.4280, 0.5609, 0.0258, 0.6531, 0.8487, 0.0332, 0.6790, 0.4391]
    genVAD(u,x,px)

main()


I am working on the file "161209-aws/aws_lecture_161209.ipynb".
The first task is:
Snapshots
A snapshot is a copy of a volume. Snapshots are extremely useful.
You can use them to:
- Create backups for your instances
- Create AMIs
You can create them from the command line on your instance. However, it's easier to do it from the console. Note that when creating snapshots, you must specify the id or name of the <b>VOLUME</b>, not the instance
Tags
On creation, each resource is assigned a unique ID. You can also tag your resources with custom values. This can help you organize and identify them - for example, the "Name" tag is useful for labeling your individual EC2 instances and snapshots.
Let's add a "Name" tag to the instance we just created.
Can you write Python code for it?

# convert response to json dict
data = json.loads(''.join(list(response)))

# get instance id
instance_id = data["Instances"][0]["InstanceId"]

print instance_id

# add name tag
!aws ec2 create-tags --resources {instance_id} --tags Key=Name,Value=awsTestInstance


I am working on the file "161209-aws/aws_lecture_161209.ipynb".
The first task is:
You can specify a script to run on startup with the --user-data option. This enables you to do things like automatically install anaconda and nltk when the instance is created, or to configure a server before you launch it.
Volumes
Volumes are blocks of memory. When you create an instance, that instance is assigned a root volume - the place where its operating information is stored. This root volume is required for your instance to function as a computer.
You can assign more than one volume to an instance. Databases and large datsets should be stored in separate volumes.
Root Volume Size
The default size for this root volume is 8GB - this is too small and you should resize it. You MUST do this when you create the instance.
You can do this in the console. Or, if you're using the command line, you create a block device mapping file - a JSON file that contains the information for how you want your volumes to be configured.
Can you write Python code for it?

# view block device mappings
with open('examples/blockDeviceMappings.json') as f:
    block_device_mappings = json.load(f)

print json.dumps(block_device_mappings[0])+"\n"+json.dumps(block_device_mappings[1])


I am working on the file "16_02_03_intro_to_pandas/intro_to_pandas.ipynb".
The first task is:
But what happens if we try to add new columns? It'd be nice if it behaved like a dict
Can you write Python code for it?

data["price_per_carat"] = data["price"] / data["carat"]


I am working on the file "16_02_03_intro_to_pandas/intro_to_pandas.ipynb".
The first task is:
It also provides nice wrapper functions for visualizing an entire dataset:
Can you write Python code for it?


I am working on the file "16_02_03_intro_to_pandas/intro_to_pandas.ipynb".
The first task is:
That's an ugly plot. It'd be nicer if we just smooth the plot, instead of down-sampling
Can you write Python code for it?


I am working on the file "16_02_03_intro_to_pandas/intro_to_pandas.ipynb".
The first task is:
Time Domain
Pandas was also developed with time domain applications in mind, so it includes some extra power for dealing with DateTimes.
Can you write Python code for it?

url = "http://mesonet.agron.iastate.edu/cgi-bin/request/asos.py?station=WVI&data=tmpf&year1=2014&month1=1&day1=1&year2=2014&month2=12&day2=31&tz=America%2FLos_Angeles&format=comma&latlon=no&direct=no"

df = pd.read_csv(url,
            comment="#",
            names=["Station", "Time", "Temp"],
            parse_dates=True,
            header=1,
            na_values="M",
            index_col="Time")

df.head()

df.plot()


I am working on the file "16_02_03_intro_to_pandas/intro_to_pandas.ipynb".
The first task is:
We can access columns through their names (like with structured arrays) or we can access columns as an object attribute
Can you write Python code for it?


I am working on the file "16_02_03_intro_to_pandas/intro_to_pandas.ipynb".
The first task is:
We can add new columns:
Can you write Python code for it?

df["price_per_carat"] = df["price"] / df.carat


I am working on the file "16_02_03_intro_to_pandas/intro_to_pandas.ipynb".
The first task is:
We can create groups based on the values of a column:
Can you write Python code for it?


I am working on the file "16_MNIST-RNN(LSTM).ipynb".
The first task is:
1. 입력DataLoader 설정

train 데이터로 loader를 지정 (dataset은 Minist, batch 사이즈 50, shuffle를 실행)
test 데이터로 loader를 지정 (dataset은 Minist, batch 사이즈 1000)
Can you write Python code for it?

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from torchvision import datasets, transforms
from torch.autograd import Variable
import matplotlib.pyplot as plt
import numpy as np

is_cuda = torch.cuda.is_available() # cuda 사용가능시, True

batch_size = 50
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=True, download=True, transform=transforms.ToTensor()),
    batch_size=batch_size, shuffle=True)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=True, transform=transforms.ToTensor()),
    batch_size=1000)


I am working on the file "16_MNIST-RNN(LSTM).ipynb".
The first task is:
3. Trainning loop
* (입력 생성)
* model 생성
* loss 생성
* zeroGrad
* backpropagation
* optimizer step (update model parameter)
Can you write Python code for it?

# trainning
model.train()
train_loss = []
train_accu = []

for epoch in range(3):
    for i, (image, target) in enumerate(train_loader):
        image = image.squeeze()
        image, target = Variable(image), Variable(target) # 입력image Target 설정
        if is_cuda :  image, target = image.cuda(), target.cuda()
        output = model(image) # model 생성
        loss = loss_fn(output, target) #loss 생성
        optimizer.zero_grad() # zero_grad
        loss.backward() # calc backward grad
        optimizer.step() # update parameter

        pred = output.data.max(1)[1]
        accuracy = pred.eq(target.data).sum()/batch_size

        train_loss.append(loss.data[0])
        train_accu.append(accuracy)

        if i % 300 == 0:
            print(i, loss.data[0])

plt.plot(train_accu)

plt.plot(train_loss)


I am working on the file "16_MNIST-RNN(LSTM).ipynb".
The first task is:
4. Predict & Evaluate
Can you write Python code for it?

model.eval()
correct = 0
for image, target in test_loader:
    image = image.squeeze()
    if is_cuda :  image, target = image.cuda(), target.cuda()
    image, target = Variable(image, volatile=True), Variable(target)
    output = model(image)
    prediction = output.data.max(1)[1]
    correct += prediction.eq(target.data).sum()

print('\nTest set: Accuracy: {:.2f}%'.format(100. * correct / len(test_loader.dataset)))


I am working on the file "16_MNIST-RNN(LSTM).ipynb".
The first task is:
6. plot images which failed to predict
Can you write Python code for it?

model.eval()
image, target = iter(test_loader).next() #test_loader로 부터 한번만 dataset을 호출

if is_cuda :  image, target = image.cuda(), target.cuda()
image, target = Variable(image, volatile=True), Variable(target)
output = model(image.squeeze())

## 이미지, 참값, 예측값을 numpy array로 변환
images = image.data.cpu().numpy()
cls_true = target.data.cpu().numpy().squeeze()
prediction = output.data.max(1)[1].cpu().numpy().squeeze()

# 예측값이 참값과 틀린것을 확인
incorrect = (prediction != cls_true)

# 예측이 틀린 것만을 추출
images = images[incorrect]
cls_true = cls_true[incorrect]
prediction = prediction[incorrect]

# 에러율을 표지
print('error : {:.1%}, number ={:}'.format(incorrect.sum()/len(incorrect), incorrect.sum()))

# 틀린 것들의 이미지를 표시
tensorImg = torch.Tensor(images)
plt.imshow(torchvision.utils.make_grid(tensorImg).numpy().transpose((1,2,0)))
plt.show()

# 틀린 것들의 예측치를 표시
print('prediction :')
pred_resized = np.pad(prediction, (0, 8 - len(prediction)%8), 'constant', constant_values=(0, 0))
print(pred_resized.reshape(-1,8))
print('\n')

# 틀린 것들의 참값을 표시
print('True :')
true_resized = np.pad(cls_true, (0, 8 - len(cls_true)%8), 'constant', constant_values=(0, 0))
print(true_resized.reshape(-1,8))


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Download Pre-Trained Model
The original version of this tutorial provided some TensorFlow checkpoints with pre-trained models for download. But due to changes in both TensorFlow and OpenAI Gym, these pre-trained models cannot be loaded anymore so they have been deleted from the web-server. You will therefore have to train your own model further below.
Create Agent
The Agent-class implements the main loop for playing the game, recording data and optimizing the Neural Network. We create an object-instance and need to set training=True because we want to use the replay-memory to record states and Q-values for plotting further below. We disable logging so this does not corrupt the logs from the actual training that was done previously. We can also set render=True but it will have no effect as long as training==True.
Can you write Python code for it?

agent = rl.Agent(env_name=env_name,
                 training=True,
                 render=True,
                 use_logging=False)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Download Pre-Trained Model
You can download a TensorFlow checkpoint which holds all the pre-trained variables for the Neural Network. Two checkpoints are provided, one for Breakout and one for Space Invaders. They were both trained for about 150 hours on a laptop with 2.6 GHz CPU and a GTX 1070 GPU.
COMPATIBILITY ISSUES
These TensorFlow checkpoints were developed with OpenAI gym v. 0.8.1 and atari-py v. 0.0.19 which had unused / redundant actions as noted above. There appears to have been a change in the gym API since then, as the unused actions are no longer present. This means the vectors with actions and Q-values now only contain 4 elements instead of the 6 shown here. This also means that the TensorFlow checkpoints cannot be used with newer versions of gym and atari-py, so in order to use these pre-trained checkpoints you need to install the older versions of gym and atari-py - or you can just train a new model yourself so you get a new TensorFlow checkpoint.
WARNING!
These checkpoints are 280-360 MB each. They are currently hosted on the webserver I use for www.hvass-labs.org because it is awkward to automatically download large files on Google Drive. To lower the traffic on my webserver, this line has been commented out, so you have to activate it manually. You are welcome to download it, I just don't want it to download automatically for everyone who only  wants to run this Notebook briefly.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Example: Highest Q-Value
This example shows the states surrounding the one with the highest Q-values. This means that the agent has high expectation that several points will be scored in the following steps. Note that the Q-values decrease significantly after the points have been scored.
Can you write Python code for it?

idx = np.argmax(q_values_max)
idx

for i in range(0, 5):
    plot_state(idx=idx+i)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Example: Loss of Life
This example shows the states leading up to a loss of life for the agent.
Can you write Python code for it?

idx = np.argmax(replay_memory.end_life)
idx

for i in range(-10, 0):
    plot_state(idx=idx+i)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Example: Smallest Difference in Q-Values
This example shows the state where there is the smallest difference in Q-values, which means that the agent believes it does not really matter which action it selects, as they all have roughly the same expectations for future rewards.
The Neural Network estimates these Q-values and they are not precise. The differences in Q-values may be so small that they fall within the error-range of the estimates.
Can you write Python code for it?

idx = np.argmin(q_values_dif)
idx

for i in range(0, 5):
    plot_state(idx=idx+i)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
For each state, calculate the min / max Q-values and their difference. This will be used to lookup interesting states in the following sections.
Can you write Python code for it?

q_values_min = q_values.min(axis=1)
q_values_max = q_values.max(axis=1)
q_values_dif = q_values_max - q_values_min


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Game Environment
This is the name of the game-environment that we want to use in OpenAI Gym.
Can you write Python code for it?

env_name = 'Breakout-v0'
# env_name = 'SpaceInvaders-v0'


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Game State
This is the state that is being input to the Neural Network. The image on the left is the last image from the game-environment. The image on the right is the processed motion-trace that shows the trajectories of objects in the game-environment.
Can you write Python code for it?

idx = np.argmax(q_values_max)
plot_state(idx=idx, print_q=False)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Get the Q-values from the replay-memory that are actually used.
Can you write Python code for it?

q_values = replay_memory.q_values[0:num_used, :]


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
In training-mode, this function will output a line for each episode. The first counter is for the number of episodes that have been processed. The second counter is for the number of states that have been processed. These two counters are stored in the TensorFlow checkpoint along with the weights of the Neural Network, so you can restart the training e.g. if you only have one computer and need to train during the night.
Note that the number of episodes is almost 90k. It is impractical to print that many lines in this Notebook, so the training is better done in a terminal window by running the following commands:
source activate tf-gpu-gym  # Activate your Python environment with TF and Gym.
python reinforcement_learning.py --env Breakout-v0 --training
Training Progress
Data is being logged during training so we can plot the progress afterwards. The reward for each episode and a running mean of the last 30 episodes are logged to file. Basic statistics for the Q-values in the replay-memory are also logged to file before each optimization run.
This could be logged using TensorFlow and TensorBoard, but they were designed for logging variables of the TensorFlow graph and data that flows through the graph. In this case the data we want logged does not reside in the graph, so it becomes a bit awkward to use TensorFlow to log this data.
We have therefore implemented a few small classes that can write and read these logs.
Can you write Python code for it?

log_q_values = rl.LogQValues()
log_reward = rl.LogReward()


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Mean Reward
The game-play is slightly random, both with regard to selecting actions using the epsilon-greedy policy, but also because the OpenAI Gym environment will repeat any action between 2-4 times, with the number chosen at random. So the reward of one episode is not an accurate estimate of the reward that can be expected in general from this agent.
We need to run 30 or even 50 episodes to get a more accurate estimate of the reward that can be expected.
We will first reset the previous episode rewards.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Once the base-dir has been set, you need to call this function to set all the paths that will be used. This will also create the checkpoint-dir if it does not already exist.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Output of Convolutional Layer 2
These are the images output by the 2nd convolutional layer, when inputting the above state to the Neural Network. There are 32 output channels of this convolutional layer.
Can you write Python code for it?

plot_layer_output(model=model, layer_name='layer_conv2', state_index=idx, inverse_cmap=False)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Output of Convolutional Layer 3
These are the images output by the 3rd convolutional layer, when inputting the above state to the Neural Network. There are 64 output channels of this convolutional layer.
All these images are flattened to a one-dimensional array (or tensor) which is then used as the input to a fully-connected layer in the Neural Network.
During the training-process, the Neural Network has learnt what convolutional filters to apply to the images from the game-environment so as to produce these images, because they have proven to be useful when estimating Q-values.
Can you see what it is that the Neural Network has learned to detect in these images?
Can you write Python code for it?

plot_layer_output(model=model, layer_name='layer_conv3', state_index=idx, inverse_cmap=False)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Output of Convolutional Layers
The outputs of the convolutional layers can be plotted so we can see how the images from the game-environment are being processed by the Neural Network.
This is the helper-function for plotting the output of the convolutional layer with the given name, when inputting the given state from the replay-memory.
Can you write Python code for it?

def plot_layer_output(model, layer_name, state_index, inverse_cmap=False):
    """
    Plot the output of a convolutional layer.

    :param model: An instance of the NeuralNetwork-class.
    :param layer_name: Name of the convolutional layer.
    :param state_index: Index into the replay-memory for a state that
                        will be input to the Neural Network.
    :param inverse_cmap: Boolean whether to inverse the color-map.
    """

    # Get the given state-array from the replay-memory.
    state = replay_memory.states[state_index]

    # Get the output tensor for the given layer inside the TensorFlow graph.
    # This is not the value-contents but merely a reference to the tensor.
    layer_tensor = model.get_layer_tensor(layer_name=layer_name)

    # Get the actual value of the tensor by feeding the state-data
    # to the TensorFlow graph and calculating the value of the tensor.
    values = model.get_tensor_value(tensor=layer_tensor, state=state)

    # Number of image channels output by the convolutional layer.
    num_images = values.shape[3]

    # Number of grid-cells to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_images))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids, figsize=(10, 10))

    print("Dim. of each image:", values.shape)

    if inverse_cmap:
        cmap = 'gray_r'
    else:
        cmap = 'gray'

    # Plot the outputs of all the channels in the conv-layer.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid image-channels.
        if i < num_images:
            # Get the image for the i'th output channel.
            img = values[0, :, :, i]

            # Plot image.
            ax.imshow(img, interpolation='nearest', cmap=cmap)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Similarly, the Agent-class also allocates the replay-memory when training==True. The replay-memory will require more than 3 GB of RAM, so it should only be allocated when needed. We will need the replay-memory in this Notebook to record the states and Q-values we observe, so they can be plotted further below.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
The Neural Network is automatically instantiated by the Agent-class. We will create a direct reference for convenience.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
The main source-code for Reinforcement Learning is located in the following module:
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
The replay-memory has room for 200k states but it is only partially full from the above call to agent.run(num_episodes=1). This is how many states are actually used.
Can you write Python code for it?

num_used = replay_memory.num_used
num_used


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
This helper-function plots a state from the replay-memory and optionally prints the Q-values.
Can you write Python code for it?

def plot_state(idx, print_q=True):
    """Plot the state in the replay-memory with the given index."""

    # Get the state from the replay-memory.
    state = replay_memory.states[idx]

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(1, 2)

    # Plot the image from the game-environment.
    ax = axes.flat[0]
    ax.imshow(state[:, :, 0], vmin=0, vmax=255,
              interpolation='lanczos', cmap='gray')

    # Plot the motion-trace.
    ax = axes.flat[1]
    ax.imshow(state[:, :, 1], vmin=0, vmax=255,
              interpolation='lanczos', cmap='gray')

    # This is necessary if we show more than one plot in a single Notebook cell.
    plt.show()

    # Print the Q-values.
    if print_q:
        print_q_values(idx=idx)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
This is the base-directory for the TensorFlow checkpoints as well as various log-files.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
This state is where the ball hits the wall so the agent scores a point.
We can show the surrounding states leading up to and following this state. Note how the Q-values are very close for the different actions, because at this point it really does not matter what the agent does as the reward is already guaranteed. But note how the Q-values decrease significantly after the ball has hit the wall and a point has been scored.
Also note that the agent uses the Epsilon-greedy policy for taking actions, so there is a small probability that a random action is taken instead of the action with the highest Q-value.
Can you write Python code for it?

for i in range(-5, 3):
    plot_state(idx=idx+i)


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Training Progress: Q-Values
The following plot shows the mean Q-values from the replay-memory prior to each run of the optimizer for the Neural Network. Note how the mean Q-values increase rapidly in the beginning and then they increase fairly steadily for 40 million states, after which they still trend upwards but somewhat more irregularly.
The fast improvement in the beginning is probably due to (1) the use of a smaller replay-memory early in training so the Neural Network is optimized more often and the new information is used faster, (2) the backwards-sweeping of the replay-memory so the rewards are used to update the Q-values for many of the states, instead of just updating the Q-values for a single state, and (3) the replay-memory is balanced so at least half of each mini-batch contains states whose Q-values have high estimation-errors for the Neural Network.
The original paper from DeepMind showed much slower progress in the first phase of training, see Figure 2 in that paper but note that the Q-values are not directly comparable, possibly because they used a higher discount factor of 0.99 while we only used 0.97 here.
Can you write Python code for it?

plt.plot(log_q_values.count_states, log_q_values.mean, label='Q-Value Mean')
plt.xlabel('State-Count for Game Environment')
plt.legend()
plt.show()


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Training Progress: Reward
This plot shows the reward for each episode during training, as well as the running mean of the last 30 episodes. Note how the reward varies greatly from one episode to the next, so it is difficult to say from this plot alone whether the agent is really improving during the training, although the running mean does appear to trend upwards slightly.
Can you write Python code for it?

plt.plot(log_reward.count_states, log_reward.episode, label='Episode Reward')
plt.plot(log_reward.count_states, log_reward.mean, label='Mean of 30 episodes')
plt.xlabel('State-Count for Game Environment')
plt.legend()
plt.show()


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We also reset the previous episode rewards.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We can also plot a histogram with the episode rewards.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We can also plot the convolutional weights for the second input channel, that is, the motion-trace of the game-environment. Once again we see that the negative weights (blue) have a much greater magnitude than the positive weights (red).
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We can now print some statistics for the episode rewards, which vary greatly from one episode to the next.
Can you write Python code for it?

rewards = agent.episode_rewards
print("Rewards for {0} episodes:".format(len(rewards)))
print("- Min:   ", np.min(rewards))
print("- Mean:  ", np.mean(rewards))
print("- Max:   ", np.max(rewards))
print("- Stdev: ", np.std(rewards))


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We can now read the logs from file:
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We can now run 30 episodes. This records the rewards for each episode. It might have been a good idea to disable the output so it does not print all these lines - you can do this as an exercise.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We can now run a single episode by calling the run() function again. This should open a new window that shows the game being played by the agent. At the time of this writing, it was not possible to resize this tiny window, and the developers at OpenAI did not seem to care about this feature which should obviously be there.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We can render the game-environment to screen so we can see the agent playing the game, by setting this boolean:
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We disable the screen-rendering so the game-environment runs much faster.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
We will now instruct the agent that it should no longer perform training by setting this boolean:
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Weights for Convolutional Layer 1
These are the weights of the first convolutional layer of the Neural Network, with respect to the first input channel of the state. That is, these are the weights that are used on the image from the game-environment. Some basic statistics are also shown.
Note how the weights are more negative (blue) than positive (red). It is unclear why this happens as these weights are found through optimization. It is apparently beneficial for the following layers to have this processing with more negative weights in the first convolutional layer.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Weights for Convolutional Layer 2
These are the weights of the 2nd convolutional layer in the Neural Network. There are 16 input channels and 32 output channels of this layer. You can change the number for the input-channel to see the associated weights.
Note how the weights are more balanced between positive (red) and negative (blue) compared to the weights for the 1st convolutional layer above.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Weights for Convolutional Layer 3
These are the weights of the 3rd convolutional layer in the Neural Network. There are 32 input channels and 64 output channels of this layer. You can change the number for the input-channel to see the associated weights.
Note again how the weights are more balanced between positive (red) and negative (blue) compared to the weights for the 1st convolutional layer above.
Can you write Python code for it?


I am working on the file "16_Reinforcement_Learning.ipynb".
The first task is:
Weights for Convolutional Layers
We can also plot the weights of the convolutional layers in the Neural Network. These are the weights that are being optimized so as to improve the ability of the Neural Network to estimate Q-values. Tutorial #02 explains in greater detail what convolutional weights are.
There are also weights for the fully-connected layers but they are not shown here.
This is the helper-function for plotting the weights of a convoluational layer.
Can you write Python code for it?

def plot_conv_weights(model, layer_name, input_channel=0):
    """
    Plot the weights for a convolutional layer.

    :param model: An instance of the NeuralNetwork-class.
    :param layer_name: Name of the convolutional layer.
    :param input_channel: Plot the weights for this input-channel.
    """

    # Get the variable for the weights of the given layer.
    # This is a reference to the variable inside TensorFlow,
    # not its actual value.
    weights_variable = model.get_weights_variable(layer_name=layer_name)

    # Retrieve the values of the weight-variable from TensorFlow.
    # The format of this 4-dim tensor is determined by the
    # TensorFlow API. See Tutorial #02 for more details.
    w = model.get_variable_value(variable=weights_variable)

    # Get the weights for the given input-channel.
    w_channel = w[:, :, input_channel, :]

    # Number of output-channels for the conv. layer.
    num_output_channels = w_channel.shape[2]

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w_channel)
    w_max = np.max(w_channel)

    # This is used to center the colour intensity at zero.
    abs_max = max(abs(w_min), abs(w_max))

    # Print statistics for the weights.
    print("Min:  {0:.5f}, Max:   {1:.5f}".format(w_min, w_max))
    print("Mean: {0:.5f}, Stdev: {1:.5f}".format(w_channel.mean(),
                                                 w_channel.std()))

    # Number of grids to plot.
    # Rounded-up, square-root of the number of output-channels.
    num_grids = math.ceil(math.sqrt(num_output_channels))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot all the filter-weights.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid filter-weights.
        if i < num_output_channels:
            # Get the weights for the i'th filter of this input-channel.
           ...


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
A few variables for tracking progress:
Can you write Python code for it?

loss_val = np.infty
game_length = 0
total_max_q = 0
mean_max_q = 0.0


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
An environment can be visualized by calling its render() method, and you can pick the rendering mode (the rendering options depend on the environment).
Warning: some environments require access to your display, which opens up a separate window, even if you specify mode="rgb_array". In general you can safely ignore that window. However, if Jupyter is running on a headless server (ie. without a screen) it will raise an exception. One way to avoid this is to install a fake X server like Xvfb. On Debian or Ubuntu:
bash
$ apt update
$ apt install -y xvfb
You can then start Jupyter using the xvfb-run command:
bash
$ xvfb-run -s "-screen 0 1400x900x24" jupyter notebook
Alternatively, you can install the pyvirtualdisplay Python library which wraps Xvfb:
bash
python3 -m pip install -U pyvirtualdisplay
And run the following code:
Can you write Python code for it?

try:
    import pyvirtualdisplay
    display = pyvirtualdisplay.Display(visible=0, size=(1400, 900)).start()
except ImportError:
    pass

env.render()


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
An environment can be visualized by calling its render() method, and you can pick the rendering mode (the rendering options depend on the environment). In this example we will set mode="rgb_array" to get an image of the environment as a NumPy array:
Can you write Python code for it?


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
And now the main training loop!
Can you write Python code for it?

with tf.Session() as sess:
    if os.path.isfile(checkpoint_path + ".index"):
        saver.restore(sess, checkpoint_path)
    else:
        init.run()
        copy_online_to_target.run()
    while True:
        step = global_step.eval()
        if step >= n_steps:
            break
        iteration += 1
        print("\rIteration {}\tTraining step {}/{} ({:.1f})%\tLoss {:5f}\tMean Max-Q {:5f}   ".format(
            iteration, step, n_steps, step * 100 / n_steps, loss_val, mean_max_q), end="")
        if done: # game over, start again
            obs = env.reset()
            for skip in range(skip_start): # skip the start of each game
                obs, reward, done, info = env.step(0)
            state = preprocess_observation(obs)

        # Online DQN evaluates what to do
        q_values = online_q_values.eval(feed_dict={X_state: [state]})
        action = epsilon_greedy(q_values, step)

        # Online DQN plays
        obs, reward, done, info = env.step(action)
        next_state = preprocess_observation(obs)

        # Let's memorize what happened
        replay_memory.append((state, action, reward, next_state, 1.0 - done))
        state = next_state

        # Compute statistics for tracking progress (not shown in the book)
        total_max_q += q_values.max()
        game_length += 1
        if done:
            mean_max_q = total_max_q / game_length
            total_max_q = 0.0
            game_length = 0

        if iteration < training_start or iteration % training_interval != 0:
            continue # only train after warmup period and at regular intervals

        # Sample memories and use the target DQN to produce the target Q-Value
        X_state_val, X_action_val, rewards, X_next_state_val, continues = (
            sample_memories(batch_size))
        next_q_values = target_q_values.eval(
            feed_dict={X_state: X_next_state_val})
        max_next_q_values = np.max(next_q_values, axis=1, keepdims=True)
        y_val = rewards + continues * discount_rate *...


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
As you can see, a single image does not give you the direction and speed of the ball, which are crucial informations for playing this game. For this reason, it is best to actually combine several consecutive observations to create the environment's state representation. One way to do that is to create a multi-channel image, with one channel per recent observation. Another is to merge all recent observations into a single-channel image, using np.max(). In this case, we need to dim the older images so that the DQN can distinguish the past from the present.
Can you write Python code for it?

from collections import deque

def combine_observations_multichannel(preprocessed_observations):
    return np.array(preprocessed_observations).transpose([1, 2, 0])

def combine_observations_singlechannel(preprocessed_observations, dim_factor=0.5):
    dimmed_observations = [obs * dim_factor**index
                           for index, obs in enumerate(reversed(preprocessed_observations))]
    return np.max(np.array(dimmed_observations), axis=0)

n_observations_per_state = 3
preprocessed_observations = deque([], maxlen=n_observations_per_state)

obs = env.reset()
for step in range(10):
    obs, _, _, _ = env.step(1)
    preprocessed_observations.append(preprocess_observation(obs))

img1 = combine_observations_multichannel(preprocessed_observations)
img2 = combine_observations_singlechannel(preprocessed_observations)

plt.figure(figsize=(11, 7))
plt.subplot(121)
plt.title("Multichannel state")
plt.imshow(img1, interpolation="nearest")
plt.axis("off")
plt.subplot(122)
plt.title("Singlechannel state")
plt.imshow(img2, interpolation="nearest", cmap="gray")
plt.axis("off")
plt.show()


def combine_observations_multichannel(preprocessed_observations):
    return np.array(preprocessed_observations).transpose([1, 2, 0])

def combine_observations_singlechannel(preprocessed_observations, dim_factor=0.5):
    dimmed_observations = [obs * dim_factor**index
                           for index, obs in enumerate(reversed(preprocessed_observations))]
    return np.max(np.array(dimmed_observations), axis=0)

n_observations_per_state = 3
preprocessed_observations = deque([], maxlen=n_observations_per_state)

obs = env.reset()
for step in range(10):
    obs, _, _, _ = env.step(1)
    preprocessed_observations.append(preprocess_observation(obs))

img1 = combine_observations_multichannel(preprocessed_observations)
img2 = combine_observations_singlechannel(preprocessed_observations)

plt.figure(figsize=(11, 7))
plt.subplot(121)
plt.title("Multichannel state")
plt.imshow(img1, interpolation="nearest")
plt.axis("off")
plt.subplot(122)
plt.title("Singlechannel state")
plt.imshow(img2, interpolation="nearest", cmap="gray")
plt.axis("off")
plt.show()


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Build DQN
Note: instead of using tf.contrib.layers.convolution2d() or tf.contrib.layers.conv2d() (as in the book), we now use the tf.layers.conv2d(), which did not exist when this chapter was written. This is preferable because anything in contrib may change or be deleted without notice, while tf.layers is part of the official API. As you will see, the code is mostly the same, except that the parameter names have changed slightly:
* the num_outputs parameter was renamed to filters,
* the stride parameter was renamed to strides,
* the _fn suffix was removed from parameter names that had it (e.g., activation_fn was renamed to activation),
* the weights_initializer parameter was renamed to kernel_initializer,
* the weights variable was renamed to "kernel" (instead of "weights"), and the biases variable was renamed from "biases" to "bias",
* and the default activation is now None instead of tf.nn.relu.
Can you write Python code for it?

reset_graph()

input_height = 88
input_width = 80
input_channels = 1
conv_n_maps = [32, 64, 64]
conv_kernel_sizes = [(8,8), (4,4), (3,3)]
conv_strides = [4, 2, 1]
conv_paddings = ["SAME"]*3
conv_activation = [tf.nn.relu]*3
n_hidden_inputs = 64 * 11 * 10  # conv3 has 64 maps of 11x10 each
n_hidden = 512
hidden_activation = tf.nn.relu
n_outputs = env.action_space.n
initializer = tf.contrib.layers.variance_scaling_initializer()

learning_rate = 0.01

def q_network(X_state, scope):
    prev_layer = X_state
    conv_layers = []
    with tf.variable_scope(scope) as scope:
        for n_maps, kernel_size, strides, padding, activation in zip(conv_n_maps, conv_kernel_sizes, conv_strides, conv_paddings, conv_activation):
            prev_layer = tf.layers.conv2d(prev_layer, filters=n_maps, kernel_size=kernel_size, strides=strides, padding=padding, activation=activation, kernel_initializer=initializer)
            conv_layers.append(prev_layer)
        last_conv_layer_flat = tf.reshape(prev_layer, shape=[-1, n_hidden_inputs])
        hidden = tf.layers.dense(last_conv_layer_flat, n_hidden, activation=hidden_activation, kernel_initializer=initializer)
        outputs = tf.layers.dense(hidden, n_outputs)
    trainable_vars = {var.name[len(scope.name):]: var for var in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope=scope.name)}
    return outputs, trainable_vars

X_state = tf.placeholder(tf.float32, shape=[None, input_height, input_width, input_channels])
actor_q_values, actor_vars = q_network(X_state, scope="q_networks/actor")    # acts
critic_q_values, critic_vars = q_network(X_state, scope="q_networks/critic") # learns

copy_ops = [actor_var.assign(critic_vars[var_name])
            for var_name, actor_var in actor_vars.items()]
copy_critic_to_actor = tf.group(*copy_ops)

with tf.variable_scope("train"):
    X_action = tf.placeholder(tf.int32, shape=[None])
    y = tf.placeholder(tf.float32, shape=[None, 1])
    q_value = tf.reduce_sum(critic_q_values * tf.one_hot(X_action, n_outputs),
                      ...


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Build DQN
Note: instead of using tf.contrib.layers.convolution2d() or tf.contrib.layers.conv2d() (as in the first version of the book), we now use the tf.layers.conv2d(), which did not exist when this chapter was written. This is preferable because anything in contrib may change or be deleted without notice, while tf.layers is part of the official API. As you will see, the code is mostly the same, except that the parameter names have changed slightly:
* the num_outputs parameter was renamed to filters,
* the stride parameter was renamed to strides,
* the _fn suffix was removed from parameter names that had it (e.g., activation_fn was renamed to activation),
* the weights_initializer parameter was renamed to kernel_initializer,
* the weights variable was renamed to "kernel" (instead of "weights"), and the biases variable was renamed from "biases" to "bias",
* and the default activation is now None instead of tf.nn.relu.
Can you write Python code for it?

reset_graph()

input_height = 88
input_width = 80
input_channels = 1
conv_n_maps = [32, 64, 64]
conv_kernel_sizes = [(8,8), (4,4), (3,3)]
conv_strides = [4, 2, 1]
conv_paddings = ["SAME"] * 3
conv_activation = [tf.nn.relu] * 3
n_hidden_in = 64 * 11 * 10  # conv3 has 64 maps of 11x10 each
n_hidden = 512
hidden_activation = tf.nn.relu
n_outputs = env.action_space.n  # 9 discrete actions are available
initializer = tf.variance_scaling_initializer()

def q_network(X_state, name):
    prev_layer = X_state / 128.0 # scale pixel intensities to the [-1.0, 1.0] range.
    with tf.variable_scope(name) as scope:
        for n_maps, kernel_size, strides, padding, activation in zip(
                conv_n_maps, conv_kernel_sizes, conv_strides,
                conv_paddings, conv_activation):
            prev_layer = tf.layers.conv2d(
                prev_layer, filters=n_maps, kernel_size=kernel_size,
                strides=strides, padding=padding, activation=activation,
                kernel_initializer=initializer)
        last_conv_layer_flat = tf.reshape(prev_layer, shape=[-1, n_hidden_in])
        hidden = tf.layers.dense(last_conv_layer_flat, n_hidden,
                                 activation=hidden_activation,
                                 kernel_initializer=initializer)
        outputs = tf.layers.dense(hidden, n_outputs,
                                  kernel_initializer=initializer)
    trainable_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                                       scope=scope.name)
    trainable_vars_by_name = {var.name[len(scope.name):]: var
                              for var in trainable_vars}
    return outputs, trainable_vars_by_name

X_state = tf.placeholder(tf.float32, shape=[None, input_height, input_width,
                                            input_channels])
online_q_values, online_vars = q_network(X_state, name="q_networks/online")
target_q_values, target_vars = q_network(X_state, name="q_networks/target")

copy_ops =...


reset_graph()

input_height = 88
input_width = 80
input_channels = 1
conv_n_maps = [32, 64, 64]
conv_kernel_sizes = [(8,8), (4,4), (3,3)]
conv_strides = [4, 2, 1]
conv_paddings = ["SAME"] * 3
conv_activation = [tf.nn.relu] * 3
n_hidden_in = 64 * 11 * 10  # conv3 has 64 maps of 11x10 each
n_hidden = 512
hidden_activation = tf.nn.relu
n_outputs = env.action_space.n  # 9 discrete actions are available
initializer = tf.contrib.layers.variance_scaling_initializer()

def q_network(X_state, name):
    prev_layer = X_state
    with tf.variable_scope(name) as scope:
        for n_maps, kernel_size, strides, padding, activation in zip(
                conv_n_maps, conv_kernel_sizes, conv_strides,
                conv_paddings, conv_activation):
            prev_layer = tf.layers.conv2d(
                prev_layer, filters=n_maps, kernel_size=kernel_size,
                strides=strides, padding=padding, activation=activation,
                kernel_initializer=initializer)
        last_conv_layer_flat = tf.reshape(prev_layer, shape=[-1, n_hidden_in])
        hidden = tf.layers.dense(last_conv_layer_flat, n_hidden,
                                 activation=hidden_activation,
                                 kernel_initializer=initializer)
        outputs = tf.layers.dense(hidden, n_outputs,
                                  kernel_initializer=initializer)
    trainable_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                                       scope=scope.name)
    trainable_vars_by_name = {var.name[len(scope.name):]: var
                              for var in trainable_vars}
    return outputs, trainable_vars_by_name

X_state = tf.placeholder(tf.float32, shape=[None, input_height, input_width,
                                            input_channels])
online_q_values, online_vars = q_network(X_state, name="q_networks/online")
target_q_values, target_vars = q_network(X_state, name="q_networks/target")

copy_ops = [target_var.assign(online_vars[var_name])
            for var_name, target_var...


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Discrete(9) means that the possible actions are integers 0 through 8, which represents the 9 possible positions of the joystick (0=center, 1=up, 2=right, 3=left, 4=down, 5=upper-right, 6=upper-left, 7=lower-right, 8=lower-left).
Next we need to tell the environment which action to play, and it will compute the next step of the game. Let's go left for 110 steps, then lower left for 40 steps:
Can you write Python code for it?

env.reset()
for step in range(110):
    env.step(3) #left
for step in range(40):
    env.step(8) #lower-left


env.seed(42)
env.reset()
for step in range(110):
    env.step(3) #left
for step in range(40):
    env.step(8) #lower-left


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Extra material
Preprocessing for Breakout
Here is a preprocessing function you can use to train a DQN for the Breakout-v0 Atari game:
Can you write Python code for it?

def preprocess_observation(obs):
    img = obs[34:194:2, ::2] # crop and downsize
    return np.mean(img, axis=2).reshape(80, 80) / 255.0

env = gym.make("Breakout-v0")
obs = env.reset()
for step in range(10):
    obs, _, _, _ = env.step(1)

img = preprocess_observation(obs)

plt.figure(figsize=(11, 7))
plt.subplot(121)
plt.title("Original observation (160×210 RGB)")
plt.imshow(obs)
plt.axis("off")
plt.subplot(122)
plt.title("Preprocessed observation (80×80 grayscale)")
plt.imshow(img, interpolation="nearest", cmap="gray")
plt.axis("off")
plt.show()


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
In this example we will set mode="rgb_array" to get an image of the environment as a NumPy array:
Can you write Python code for it?

img = env.render(mode="rgb_array")
img.shape


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
In this particular environment, the past actions and observations can safely be ignored, since each observation contains the environment's full state. If there were some hidden state then you may need to consider past actions and observations in order to try to infer the hidden state of the environment. For example, if the environment only revealed the position of the cart but not its velocity, you would have to consider not only the current observation but also the previous observation in order to estimate the current velocity. Another example is if the observations are noisy: you may want to use the past few observations to estimate the most likely current state. Our problem is thus as simple as can be: the current observation is noise-free and contains the environment's full state.
You may wonder why we are picking a random action based on the probability given by the policy network, rather than just picking the action with the highest probability. This approach lets the agent find the right balance between exploring new actions and exploiting the actions that are known to work well. Here's an analogy: suppose you go to a restaurant for the first time, and all the dishes look equally appealing so you randomly pick one. If it turns out to be good, you can increase the probability to order it next time, but you shouldn't increase that probability to 100%, or else you will never try out the other dishes, some of which may be even better than the one you tried.
Let's randomly initialize this policy neural network and use it to play one game:
Can you write Python code for it?

n_max_steps = 1000
frames = []

with tf.Session() as sess:
    init.run()
    obs = env.reset()
    for step in range(n_max_steps):
        img = render_cart_pole(env, obs)
        frames.append(img)
        action_val = action.eval(feed_dict={X: obs.reshape(1, n_inputs)})
        obs, reward, done, info = env.step(action_val[0][0])
        if done:
            break

env.close()


n_max_steps = 1000
frames = []

with tf.Session() as sess:
    init.run()
    env.seed(42)
    obs = env.reset()
    for step in range(n_max_steps):
        img = env.render(mode="rgb_array")
        frames.append(img)
        action_val = action.eval(feed_dict={X: obs.reshape(1, n_inputs)})
        obs, reward, done, info = env.step(action_val[0][0])
        if done:
            break

env.close()


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Learning to Play MsPacman Using the DQN Algorithm
Warning: Unfortunately, the first version of the book contained two important errors in this section.

The actor DQN and critic DQN should have been named online DQN and target DQN respectively. Actor-critic algorithms are a distinct class of algorithms.
The online DQN is the one that learns and is copied to the target DQN at regular intervals. The target DQN's only role is to estimate the next state's Q-Values for each possible action. This is needed to compute the target Q-Values for training the online DQN, as shown in this equation:

$y(s,a) = \text{r} + \gamma . \underset{a'}{\max} \, Q_\text{target}(s', a')$

$y(s,a)$ is the target Q-Value to train the online DQN for the state-action pair $(s, a)$.
$r$ is the reward actually collected after playing action $a$ in state $s$.
$\gamma$ is the discount rate.
$s'$ is the state actually reached after played action $a$ in state $s$.
$a'$ is one of the possible actions in state $s'$.
$Q_\text{target}(s', a')$ is the target DQN's estimate of the Q-Value of playing action $a'$ while in state $s'$.

I hope these errors did not affect you, and if they did, I sincerely apologize.
Creating the MsPacman environment
Can you write Python code for it?

env = gym.make("MsPacman-v0")
obs = env.reset()
obs.shape

env.action_space


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Let's create a little helper function to plot an environment:
Can you write Python code for it?

def plot_environment(env, figsize=(5,4)):
    plt.close()  # or else nbagg sometimes plots in the previous cell
    plt.figure(figsize=figsize)
    img = env.render(mode="rgb_array")
    plt.imshow(img)
    plt.axis("off")
    plt.show()


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Let's initialize the environment by calling is reset() method. This returns an observation:
Can you write Python code for it?


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Let's play one full game (with 3 lives), by moving in random directions for 10 steps at a time, recording each frame:
Can you write Python code for it?

frames = []

n_max_steps = 1000
n_change_steps = 10

obs = env.reset()
for step in range(n_max_steps):
    img = env.render(mode="rgb_array")
    frames.append(img)
    if step % n_change_steps == 0:
        action = env.action_space.sample() # play randomly
    obs, reward, done, info = env.step(action)
    if done:
        break


frames = []

n_max_steps = 1000
n_change_steps = 10

env.seed(42)
obs = env.reset()
for step in range(n_max_steps):
    img = env.render(mode="rgb_array")
    frames.append(img)
    if step % n_change_steps == 0:
        action = env.action_space.sample() # play randomly
    obs, reward, done, info = env.step(action)
    if done:
        break


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Let's plot this image:
Can you write Python code for it?

plt.figure(figsize=(5,4))
plt.imshow(img)
plt.axis("off")
save_fig("MsPacman")
plt.show()


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Let's see how to interact with an environment. Your agent will need to select an action from an "action space" (the set of possible actions). Let's see what this environment's action space looks like:
Can you write Python code for it?


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Looks like it learned the policy correctly. Now let's see if it can learn a better policy on its own.
Policy Gradients
To train this neural network we will need to define the target probabilities y. If an action is good we should increase its probability, and conversely if it is bad we should reduce it. But how do we know whether an action is good or bad? The problem is that most actions have delayed effects, so when you win or lose points in a game, it is not clear which actions contributed to this result: was it just the last action? Or the last 10? Or just one action 50 steps earlier? This is called the credit assignment problem.
The Policy Gradients algorithm tackles this problem by first playing multiple games, then making the actions in good games slightly more likely, while actions in bad games are made slightly less likely. First we play, then we go back and think about what we did.
Can you write Python code for it?

import tensorflow as tf

reset_graph()

n_inputs = 4
n_hidden = 4
n_outputs = 1

learning_rate = 0.01

initializer = tf.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])

hidden = tf.layers.dense(X, n_hidden, activation=tf.nn.elu, kernel_initializer=initializer)
logits = tf.layers.dense(hidden, n_outputs)
outputs = tf.nn.sigmoid(logits)  # probability of action 0 (left)
p_left_and_right = tf.concat(axis=1, values=[outputs, 1 - outputs])
action = tf.multinomial(tf.log(p_left_and_right), num_samples=1)

y = 1. - tf.to_float(action)
cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=y, logits=logits)
optimizer = tf.train.AdamOptimizer(learning_rate)
grads_and_vars = optimizer.compute_gradients(cross_entropy)
gradients = [grad for grad, variable in grads_and_vars]
gradient_placeholders = []
grads_and_vars_feed = []
for grad, variable in grads_and_vars:
    gradient_placeholder = tf.placeholder(tf.float32, shape=grad.get_shape())
    gradient_placeholders.append(gradient_placeholder)
    grads_and_vars_feed.append((gradient_placeholder, variable))
training_op = optimizer.apply_gradients(grads_and_vars_feed)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

def discount_rewards(rewards, discount_rate):
    discounted_rewards = np.zeros(len(rewards))
    cumulative_rewards = 0
    for step in reversed(range(len(rewards))):
        cumulative_rewards = rewards[step] + cumulative_rewards * discount_rate
        discounted_rewards[step] = cumulative_rewards
    return discounted_rewards

def discount_and_normalize_rewards(all_rewards, discount_rate):
    all_discounted_rewards = [discount_rewards(rewards, discount_rate) for rewards in all_rewards]
    flat_rewards = np.concatenate(all_discounted_rewards)
    reward_mean = flat_rewards.mean()
    reward_std = flat_rewards.std()
    return [(discounted_rewards - reward_mean)/reward_std for discounted_rewards in all_discounted_rewards]

discount_rewards([10, 0, -50],...


import tensorflow as tf

reset_graph()

n_inputs = 4
n_hidden = 4
n_outputs = 1

learning_rate = 0.01

initializer = tf.contrib.layers.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])

hidden = tf.layers.dense(X, n_hidden, activation=tf.nn.elu, kernel_initializer=initializer)
logits = tf.layers.dense(hidden, n_outputs)
outputs = tf.nn.sigmoid(logits)  # probability of action 0 (left)
p_left_and_right = tf.concat(axis=1, values=[outputs, 1 - outputs])
action = tf.multinomial(tf.log(p_left_and_right), num_samples=1)

y = 1. - tf.to_float(action)
cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=y, logits=logits)
optimizer = tf.train.AdamOptimizer(learning_rate)
grads_and_vars = optimizer.compute_gradients(cross_entropy)
gradients = [grad for grad, variable in grads_and_vars]
gradient_placeholders = []
grads_and_vars_feed = []
for grad, variable in grads_and_vars:
    gradient_placeholder = tf.placeholder(tf.float32, shape=grad.get_shape())
    gradient_placeholders.append(gradient_placeholder)
    grads_and_vars_feed.append((gradient_placeholder, variable))
training_op = optimizer.apply_gradients(grads_and_vars_feed)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

def discount_rewards(rewards, discount_rate):
    discounted_rewards = np.zeros(len(rewards))
    cumulative_rewards = 0
    for step in reversed(range(len(rewards))):
        cumulative_rewards = rewards[step] + cumulative_rewards * discount_rate
        discounted_rewards[step] = cumulative_rewards
    return discounted_rewards

def discount_and_normalize_rewards(all_rewards, discount_rate):
    all_discounted_rewards = [discount_rewards(rewards, discount_rate) for rewards in all_rewards]
    flat_rewards = np.concatenate(all_discounted_rewards)
    reward_mean = flat_rewards.mean()
    reward_std = flat_rewards.std()
    return [(discounted_rewards - reward_mean)/reward_std for discounted_rewards in all_discounted_rewards]

discount_rewards([10, 0, -50],...


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Looks like it's doing what we're telling it to do. Now how can we make the poll remain upright? We will need to define a policy for that. This is the strategy that the agent will use to select an action at each step. It can use all the past actions and observations to decide what to do.
A simple hard-coded policy
Let's hard code a simple strategy: if the pole is tilting to the left, then push the cart to the left, and vice versa. Let's see if that works:
Can you write Python code for it?

frames = []

n_max_steps = 1000
n_change_steps = 10

obs = env.reset()
for step in range(n_max_steps):
    img = render_cart_pole(env, obs)
    frames.append(img)

    # hard-coded policy
    position, velocity, angle, angular_velocity = obs
    if angle < 0:
        action = 0
    else:
        action = 1

    obs, reward, done, info = env.step(action)
    if done:
        break

video = plot_animation(frames)
plt.show()


frames = []

n_max_steps = 1000
n_change_steps = 10

env.seed(42)
obs = env.reset()
for step in range(n_max_steps):
    img = env.render(mode="rgb_array")
    frames.append(img)

    # hard-coded policy
    position, velocity, angle, angular_velocity = obs
    if angle < 0:
        action = 0
    else:
        action = 1

    obs, reward, done, info = env.step(action)
    if done:
        break

plot_animation(frames)


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Markov Chains
Can you write Python code for it?

transition_probabilities = [
        [0.7, 0.2, 0.0, 0.1],  # from s0 to s0, s1, s2, s3
        [0.0, 0.0, 0.9, 0.1],  # from s1 to ...
        [0.0, 1.0, 0.0, 0.0],  # from s2 to ...
        [0.0, 0.0, 0.0, 1.0],  # from s3 to ...
    ]

n_max_steps = 50

def print_sequence(start_state=0):
    current_state = start_state
    print("States:", end=" ")
    for step in range(n_max_steps):
        print(current_state, end=" ")
        if current_state == 3:
            break
        current_state = rnd.choice(range(4), p=transition_probabilities[current_state])
    else:
        print("...", end="")
    print()

for _ in range(10):
    print_sequence()


transition_probabilities = [
        [0.7, 0.2, 0.0, 0.1],  # from s0 to s0, s1, s2, s3
        [0.0, 0.0, 0.9, 0.1],  # from s1 to ...
        [0.0, 1.0, 0.0, 0.0],  # from s2 to ...
        [0.0, 0.0, 0.0, 1.0],  # from s3 to ...
    ]

n_max_steps = 50

def print_sequence(start_state=0):
    current_state = start_state
    print("States:", end=" ")
    for step in range(n_max_steps):
        print(current_state, end=" ")
        if current_state == 3:
            break
        current_state = np.random.choice(range(4), p=transition_probabilities[current_state])
    else:
        print("...", end="")
    print()

for _ in range(10):
    print_sequence()


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Markov Decision Process
Can you write Python code for it?

transition_probabilities = [
        [[0.7, 0.3, 0.0], [1.0, 0.0, 0.0], [0.8, 0.2, 0.0]], # in s0, if action a0 then proba 0.7 to state s0 and 0.3 to state s1, etc.
        [[0.0, 1.0, 0.0], None, [0.0, 0.0, 1.0]],
        [None, [0.8, 0.1, 0.1], None],
    ]

rewards = [
        [[+10, 0, 0], [0, 0, 0], [0, 0, 0]],
        [[0, 0, 0], [0, 0, 0], [0, 0, -50]],
        [[0, 0, 0], [+40, 0, 0], [0, 0, 0]],
    ]

possible_actions = [[0, 1, 2], [0, 2], [1]]

def policy_fire(state):
    return [0, 2, 1][state]

def policy_random(state):
    return rnd.choice(possible_actions[state])

def policy_safe(state):
    return [0, 0, 1][state]

class MDPEnvironment(object):
    def __init__(self, start_state=0):
        self.start_state=start_state
        self.reset()
    def reset(self):
        self.total_rewards = 0
        self.state = self.start_state
    def step(self, action):
        next_state = rnd.choice(range(3), p=transition_probabilities[self.state][action])
        reward = rewards[self.state][action][next_state]
        self.state = next_state
        self.total_rewards += reward
        return self.state, reward

def run_episode(policy, n_steps, start_state=0, display=True):
    env = MDPEnvironment()
    if display:
        print("States (+rewards):", end=" ")
    for step in range(n_steps):
        if display:
            if step == 10:
                print("...", end=" ")
            elif step < 10:
                print(env.state, end=" ")
        action = policy(env.state)
        state, reward = env.step(action)
        if display and step < 10:
            if reward:
                print("({})".format(reward), end=" ")
    if display:
        print("Total rewards =", env.total_rewards)
    return env.total_rewards

for policy in (policy_fire, policy_random, policy_safe):
    all_totals = []
    print(policy.__name__)
    for episode in range(1000):
        all_totals.append(run_episode(policy, n_steps=100, display=(episode<5)))
    print("Summary: mean={:.1f}, std={:1f}, min={},...


transition_probabilities = [
        [[0.7, 0.3, 0.0], [1.0, 0.0, 0.0], [0.8, 0.2, 0.0]], # in s0, if action a0 then proba 0.7 to state s0 and 0.3 to state s1, etc.
        [[0.0, 1.0, 0.0], None, [0.0, 0.0, 1.0]],
        [None, [0.8, 0.1, 0.1], None],
    ]

rewards = [
        [[+10, 0, 0], [0, 0, 0], [0, 0, 0]],
        [[0, 0, 0], [0, 0, 0], [0, 0, -50]],
        [[0, 0, 0], [+40, 0, 0], [0, 0, 0]],
    ]

possible_actions = [[0, 1, 2], [0, 2], [1]]

def policy_fire(state):
    return [0, 2, 1][state]

def policy_random(state):
    return np.random.choice(possible_actions[state])

def policy_safe(state):
    return [0, 0, 1][state]

class MDPEnvironment(object):
    def __init__(self, start_state=0):
        self.start_state=start_state
        self.reset()
    def reset(self):
        self.total_rewards = 0
        self.state = self.start_state
    def step(self, action):
        next_state = np.random.choice(range(3), p=transition_probabilities[self.state][action])
        reward = rewards[self.state][action][next_state]
        self.state = next_state
        self.total_rewards += reward
        return self.state, reward

def run_episode(policy, n_steps, start_state=0, display=True):
    env = MDPEnvironment()
    if display:
        print("States (+rewards):", end=" ")
    for step in range(n_steps):
        if display:
            if step == 10:
                print("...", end=" ")
            elif step < 10:
                print(env.state, end=" ")
        action = policy(env.state)
        state, reward = env.step(action)
        if display and step < 10:
            if reward:
                print("({})".format(reward), end=" ")
    if display:
        print("Total rewards =", env.total_rewards)
    return env.total_rewards

for policy in (policy_fire, policy_random, policy_safe):
    all_totals = []
    print(policy.__name__)
    for episode in range(1000):
        all_totals.append(run_episode(policy, n_steps=100, display=(episode<5)))
    print("Summary: mean={:.1f}, std={:1f}, min={},...


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Next we will load the MsPacman environment, version 0.
Can you write Python code for it?


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Nope, it really can't walk. So let's train it!
Can you write Python code for it?

n_games_per_update = 10
n_max_steps = 1000
n_iterations = 1000
save_iterations = 10
discount_rate = 0.95

with tf.Session() as sess:
    init.run()
    for iteration in range(n_iterations):
        print("\rIteration: {}/{}".format(iteration + 1, n_iterations), end="")
        all_rewards = []
        all_gradients = []
        for game in range(n_games_per_update):
            current_rewards = []
            current_gradients = []
            obs = env.reset()
            for step in range(n_max_steps):
                action_index_val, gradients_val = sess.run([action_index, gradients],
                                                           feed_dict={X: obs.reshape(1, n_inputs)})
                action = possible_actions[action_index_val]
                obs, reward, done, info = env.step(action[0])
                current_rewards.append(reward)
                current_gradients.append(gradients_val)
                if done:
                    break
            all_rewards.append(current_rewards)
            all_gradients.append(current_gradients)

        all_rewards = discount_and_normalize_rewards(all_rewards, discount_rate=discount_rate)
        feed_dict = {}
        for var_index, gradient_placeholder in enumerate(gradient_placeholders):
            mean_gradients = np.mean([reward * all_gradients[game_index][step][var_index]
                                      for game_index, rewards in enumerate(all_rewards)
                                          for step, reward in enumerate(rewards)], axis=0)
            feed_dict[gradient_placeholder] = mean_gradients
        sess.run(training_op, feed_dict=feed_dict)
        if iteration % save_iterations == 0:
            saver.save(sess, "./my_bipedal_walker_pg.ckpt")

frames = run_bipedal_walker("./my_bipedal_walker_pg.ckpt")
plot_animation(frames)


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Nope, the system is unstable and after just a few wobbles, the pole ends up too tilted: game over. We will need to be smarter than that!
Neural Network Policies
Let's create a neural network that will take observations as inputs, and output the action to take for each observation. To choose an action, the network will first estimate a probability for each action, then select an action randomly according to the estimated probabilities. In the case of the Cart-Pole environment, there are just two possible actions (left or right), so we only need one output neuron: it will output the probability p of the action 0 (left), and of course the probability of action 1 (right) will be 1 - p.
Note: instead of using the fully_connected() function from the tensorflow.contrib.layers module (as in the book), we now use the dense() function from the tf.layers module, which did not exist when this chapter was written. This is preferable because anything in contrib may change or be deleted without notice, while tf.layers is part of the official API. As you will see, the code is mostly the same.
The main differences relevant to this chapter are:
* the _fn suffix was removed in all the parameters that had it (for example the activation_fn parameter was renamed to activation).
* the weights parameter was renamed to kernel,
* the default activation is None instead of tf.nn.relu
Can you write Python code for it?

import tensorflow as tf

# 1. Specify the network architecture
n_inputs = 4  # == env.observation_space.shape[0]
n_hidden = 4  # it's a simple task, we don't need more than this
n_outputs = 1 # only outputs the probability of accelerating left
initializer = tf.contrib.layers.variance_scaling_initializer()

# 2. Build the neural network
X = tf.placeholder(tf.float32, shape=[None, n_inputs])
hidden = tf.layers.dense(X, n_hidden, activation=tf.nn.elu,
                         kernel_initializer=initializer)
outputs = tf.layers.dense(hidden, n_outputs, activation=tf.nn.sigmoid,
                          kernel_initializer=initializer)

# 3. Select a random action based on the estimated probabilities
p_left_and_right = tf.concat(axis=1, values=[outputs, 1 - outputs])
action = tf.multinomial(tf.log(p_left_and_right), num_samples=1)

init = tf.global_variables_initializer()


I am working on the file "16_reinforcement_learning.ipynb".
The first task is:
Not the best walker, but at least it stays up and makes (slow) progress to the right.
A better solution for this problem is to use an actor-critic algorithm, as it does not require discretizing the action space, and it converges much faster. Check out this nice blog post by Yash Patel for more details.
9. Pong DQN
Let's explore the Pong-v0 OpenAI Gym environment.
Can you write Python code for it?

import gym

env = gym.make('Pong-v0')
obs = env.reset()

obs.shape

env.action_space