I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
Experience replay
Reinforcement learning algorithms can have stability issues due to correlations between states. To reduce correlations when training, we can store the agent's experiences and later draw a random mini-batch of those experiences to train on.
Here, we'll create a Memory object that will store our experiences, our transitions $<s, a, r, s'>$. This memory will have a maxmium capacity, so we can keep newer experiences in memory while getting rid of older experiences. Then, we'll sample a random mini-batch of transitions $<s, a, r, s'>$ and train on those.
Below, I've implemented a Memory object. If you're unfamiliar with deque, this is a double-ended queue. You can think of it like a tube open on both sides. You can put objects in either side of the tube. But if it's full, adding anything more will push an object out the other side. This is a great data structure to use for the memory buffer.
Can you write Python code for it?

from collections import deque
class Memory():
    def __init__(self, max_size = 1000):
        self.buffer = deque(maxlen=max_size)

    def add(self, experience):
        self.buffer.append(experience)

    def sample(self, batch_size):
        idx = np.random.choice(np.arange(len(self.buffer)),
                               size=batch_size,
                               replace=False)
        return [self.buffer[ii] for ii in idx]


I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
Exploration - Exploitation
To learn about the environment and rules of the game, the agent needs to explore by taking random actions. We'll do this by choosing a random action with some probability $\epsilon$ (epsilon).  That is, with some probability $\epsilon$ the agent will make a random action and with probability $1 - \epsilon$, the agent will choose an action from $Q(s,a)$. This is called an $\epsilon$-greedy policy.
At first, the agent needs to do a lot of exploring. Later when it has learned more, the agent can favor choosing actions based on what it has learned. This is called exploitation. We'll set it up so the agent is more likely to explore early in training, then more likely to exploit later in training.
Q-Learning training algorithm
Putting all this together, we can list out the algorithm we'll use to train the network. We'll train the network in episodes. One episode is one simulation of the game. For this game, the goal is to keep the pole upright for 195 frames. So we can start a new episode once meeting that goal. The game ends if the pole tilts over too far, or if the cart moves too far the left or right. When a game ends, we'll start a new episode. Now, to train the agent:

Initialize the memory $D$
Initialize the action-value network $Q$ with random weights
For episode = 1, $M$ do
For $t$, $T$ do
With probability $\epsilon$ select a random action $a_t$, otherwise select $a_t = \mathrm{argmax}_a Q(s,a)$
Execute action $a_t$ in simulator and observe reward $r_{t+1}$ and new state $s_{t+1}$
Store transition $<s_t, a_t, r_{t+1}, s_{t+1}>$ in memory $D$
Sample random mini-batch from $D$: $<s_j, a_j, r_j, s'_j>$
Set $\hat{Q}j = r_j$ if the episode ends at $j+1$, otherwise set $\hat{Q}_j = r_j + \gamma \max{a'}{Q(s'_j, a')}$
Make a gradient descent step with loss $(\hat{Q}_j - Q(s_j, a_j))^2$

endfor
endfor

Hyperparameters
One of the more difficult aspects of reinforcememt learning are the large number of hyperparameters. Not only are we tuning the network, but we're tuning the simulation.
Can you write Python code for it?

train_episodes = 1000          # max number of episodes to learn from
max_steps = 200                # max steps in an episode
gamma = 0.99                   # future reward discount

# Exploration parameters
explore_start = 1.0            # exploration probability at start
explore_stop = 0.01            # minimum exploration probability
decay_rate = 0.0001            # exponential decay rate for exploration prob

# Network parameters
hidden_size = 64               # number of units in each Q-network hidden layer
learning_rate = 0.0001         # Q-network learning rate

# Memory parameters
memory_size = 10000            # memory capacity
batch_size = 20                # experience mini-batch size
pretrain_length = batch_size   # number experiences to pretrain the memory

tf.reset_default_graph()
mainQN = QNetwork(name='main', hidden_size=hidden_size, learning_rate=learning_rate)


I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
If you ran the simulation above, we can look at the rewards:
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
Note: Make sure you have OpenAI Gym cloned into the same directory with this notebook. I've included gym as a submodule, so you can run git submodule --init --recursive to pull the contents into the gym repo.
Can you write Python code for it?

# Create the Cart-Pole game environment
env = gym.make('CartPole-v0')


I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
Populate the experience memory
Here I'm re-initializing the simulation and pre-populating the memory. The agent is taking random actions and storing the transitions in memory. This will help the agent with exploring the game.
Can you write Python code for it?

# Initialize the simulation
env.reset()
# Take one random step to get the pole and cart moving
state, reward, done, _ = env.step(env.action_space.sample())

memory = Memory(max_size=memory_size)

# Make a bunch of random actions and store the experiences
for ii in range(pretrain_length):
    # Uncomment the line below to watch the simulation
    # env.render()

    # Make a random action
    action = env.action_space.sample()
    next_state, reward, done, _ = env.step(action)

    if done:
        # The simulation fails so no next state
        next_state = np.zeros(state.shape)
        # Add experience to memory
        memory.add((state, action, reward, next_state))

        # Start new episode
        env.reset()
        # Take one random step to get the pole and cart moving
        state, reward, done, _ = env.step(env.action_space.sample())
    else:
        # Add experience to memory
        memory.add((state, action, reward, next_state))
        state = next_state


I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
Testing
Let's checkout how our trained agent plays the game.
Can you write Python code for it?

test_episodes = 10
test_max_steps = 400
env.reset()
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))

    for ep in range(1, test_episodes):
        t = 0
        while t < test_max_steps:
            env.render()

            # Get action from Q-network
            feed = {mainQN.inputs_: state.reshape((1, *state.shape))}
            Qs = sess.run(mainQN.output, feed_dict=feed)
            action = np.argmax(Qs)

            # Take action, get new state and reward
            next_state, reward, done, _ = env.step(action)

            if done:
                t = test_max_steps
                env.reset()
                # Take one random step to get the pole and cart moving
                state, reward, done, _ = env.step(env.action_space.sample())

            else:
                state = next_state
                t += 1

env.close()


I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
The game resets after the pole has fallen past a certain angle. For each frame while the simulation is running, it returns a reward of 1.0. The longer the game runs, the more reward we get. Then, our network's goal is to maximize the reward by keeping the pole vertical. It will do this by moving the cart to the left and the right.
Q-Network
We train our Q-learning agent using the Bellman Equation:
$$
Q(s, a) = r + \gamma \max{Q(s', a')}
$$
where $s$ is a state, $a$ is an action, and $s'$ is the next state from state $s$ and action $a$.
Before we used this equation to learn values for a Q-table. However, for this game there are a huge number of states available. The state has four values: the position and velocity of the cart, and the position and velocity of the pole. These are all real-valued numbers, so ignoring floating point precisions, you practically have infinite states. Instead of using a table then, we'll replace it with a neural network that will approximate the Q-table lookup function.

Now, our Q value, $Q(s, a)$ is calculated by passing in a state to the network. The output will be Q-values for each available action, with fully connected hidden layers.

As I showed before, we can define our targets for training as $\hat{Q}(s,a) = r + \gamma \max{Q(s', a')}$. Then we update the weights by minimizing $(\hat{Q}(s,a) - Q(s,a))^2$.
For this Cart-Pole game, we have four inputs, one for each value in the state, and two outputs, one for each action. To get $\hat{Q}$, we'll first choose an action, then simulate the game using that action. This will get us the next state, $s'$, and the reward. With that, we can calculate $\hat{Q}$ then pass it back into the $Q$ network to run the optimizer and update the weights.
Below is my implementation of the Q-network. I used two fully connected layers with ReLU activations. Two seems to be good enough, three might be better. Feel free to try it out.
Can you write Python code for it?

class QNetwork:
    def __init__(self, learning_rate=0.01, state_size=4,
                 action_size=2, hidden_size=10,
                 name='QNetwork'):
        # state inputs to the Q-network
        with tf.variable_scope(name):
            self.inputs_ = tf.placeholder(tf.float32, [None, state_size], name='inputs')

            # One hot encode the actions to later choose the Q-value for the action
            self.actions_ = tf.placeholder(tf.int32, [None], name='actions')
            one_hot_actions = tf.one_hot(self.actions_, action_size)

            # Target Q values for training
            self.targetQs_ = tf.placeholder(tf.float32, [None], name='target')

            # ReLU hidden layers
            self.fc1 = tf.contrib.layers.fully_connected(self.inputs_, hidden_size)
            self.fc2 = tf.contrib.layers.fully_connected(self.fc1, hidden_size)

            # Linear output layer
            self.output = tf.contrib.layers.fully_connected(self.fc2, action_size,
                                                            activation_fn=None)

            ### Train with loss (targetQ - Q)^2
            # output has length 2, for two actions. This next line chooses
            # one value from output (per row) according to the one-hot encoded actions.
            self.Q = tf.reduce_sum(tf.multiply(self.output, one_hot_actions), axis=1)

            self.loss = tf.reduce_mean(tf.square(self.targetQs_ - self.Q))
            self.opt = tf.train.AdamOptimizer(learning_rate).minimize(self.loss)


I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
To shut the window showing the simulation, use env.close().
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/Q-learning-cart.ipynb".
The first task is:
Visualizing training
Below I'll plot the total rewards for each episode. I'm plotting the rolling average too, in blue.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

def running_mean(x, N):
    cumsum = np.cumsum(np.insert(x, 0, 0))
    return (cumsum[N:] - cumsum[:-N]) / N

eps, rews = np.array(rewards_list).T
smoothed_rews = running_mean(rews, 10)
plt.plot(eps[-len(smoothed_rews):], smoothed_rews)
plt.plot(eps, rews, color='grey', alpha=0.3)
plt.xlabel('Episode')
plt.ylabel('Total Reward')


I am working on the file "Reinforcement-Learning/TD0-models/01.TaxiProblem.ipynb".
The first task is:
2. RL-Algorithms based on Temporal Difference - TD(0)
2a. Load the "PlotUtils" Python class
Load the Python class PlotUtils() which provides various plotting utilities and start a new instance.
Can you write Python code for it?

%run ../PlotUtils.py

plotutls = PlotUtils()


I am working on the file "Reinforcement-Learning/TD0-models/01.TaxiProblem.ipynb".
The first task is:
2b. Load the "Temporal Difference" Python class
Load the Temporal Difference Python class, TemporalDifferenceUtils() and start a new instance for the Taxi-v3 OpenAI Gym environment.
Can you write Python code for it?

%run ../TD0_Utils.py

TD0 = TemporalDifferenceUtils(env)


I am working on the file "Reinforcement-Learning/TD0-models/01.TaxiProblem.ipynb".
The first task is:
3. Model Training
3a. SARSA on-Policy TD(0) Control
Notes on the RL trials tested below:
* discount factor (or gamma): It is generally inrtrinsic property of the model. Having tried a gamma= 0.7 for the taxi problem we discuss render our SARSA on-policy solution quite unstable. For this reason we have fixed it at "1" in all the models we are testing below.
<img src="./sarsa_taxi_problem-low_discount.jpg">
Train some candidates RL-models of SARSA on-policy TD(0) Control:
Can you write Python code for it?

# Define Number of Episodes
n_episodes = 3e+3

# e-greedy parameters to investigate
print('Determine the epsilon parameters for the epsilon-greedy policy...\n')
epsilons = np.arange(0.01, 0.05, 0.01)
print('epsilons: {}'.format(epsilons), '\n')

# various step-sizes (alpha) to try
print('Determine the step-sizes parameters (alphas) for the TD(0)...\n')
step_sizes = np.array(0.4)
print('step_sizes: {}'.format(step_sizes), '\n')

# Fixed discount
discount_fixed = 1

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
epsilons, step_sizes = np.meshgrid(epsilons, step_sizes)

# Create a dictionary of the RL-trials of interest
RL_trials = {"baseline":
             {'epsilon': 0.017,
              'step_size': 0.4, 'discount': 1}}

for n, trial in enumerate(list(zip(*epsilons, *step_sizes))):
    key = 'trial_' + str(n+1)
    RL_trials[key] = {'epsilon': trial[0],
                      'step_size': trial[1], 'discount': discount_fixed}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes...\n'.format(int(n_episodes)))

rewards_per_trial_SARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_SARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']

    # Apply SARSA [on-policy TD(0) Control]
    q_values, tot_rewards = TD0.sarsa_on_policy_control(env, n_episodes=n_episodes,
                                                        step_size=step_size, discount=discount, epsilon=epsilon)

    # Update "rewards_per_trial" and "q_values_per_trial" OrderedDicts
    rewards_per_trial_SARSA[trial] = tot_rewards
    q_values_per_trial_SARSA[trial] = q_values

title = 'Efficiency of the RL Method\n[SARSA on-policy TD(0)...


I am working on the file "Reinforcement-Learning/TD0-models/01.TaxiProblem.ipynb".
The first task is:
3b. Q-Learning Off-Policy TD(0) Control
Can you write Python code for it?

# Define Number of Episodes
n_episodes = 3e+3

# e-greedy parameters to investigate
print('Determine the epsilon parameters for the epsilon-greedy policy...\n')
epsilons = np.arange(0.01, 0.05, 0.01)
print('epsilons: {}'.format(epsilons), '\n')

# various step-sizes (alpha) to try
print('Determine the step-sizes parameters (alphas) for the TD(0)...\n')
step_sizes = np.array(0.4)
print('step_sizes: {}'.format(step_sizes), '\n')

# Fixed discount
discount_fixed = 1

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
epsilons, step_sizes = np.meshgrid(epsilons, step_sizes)

# Create a dictionary of the RL-trials of interest
RL_trials = {"baseline":
             {'epsilon': 0.017,
              'step_size': 0.4, 'discount': 1}}

for n, trial in enumerate(list(zip(*epsilons, *step_sizes))):
    key = 'trial_' + str(n+1)
    RL_trials[key] = {'epsilon': trial[0],
                      'step_size': trial[1], 'discount': discount_fixed}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes...\n'.format(int(n_episodes)))

rewards_per_trial_QL = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_QL = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']

    # Apply SARSA [on-policy TD(0) Control]
    q_values, tot_rewards = TD0.q_learning_off_policy(env, n_episodes=n_episodes,
                                                      step_size=step_size, discount=discount, epsilon=epsilon)

    # Update "rewards_per_trial" and "q_values_per_trial" OrderedDicts
    rewards_per_trial_QL[trial] = tot_rewards
    q_values_per_trial_QL[trial] = q_values

title = 'Efficiency of the RL Method\n[Q-Learning off-policy TD(0)...


I am working on the file "Reinforcement-Learning/TD0-models/01.TaxiProblem.ipynb".
The first task is:
Again, the best RL-model was the so-called "trial_1":
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/01.TaxiProblem.ipynb".
The first task is:
For details on the notation followed for state-actions and rewards inside the "Taxi-v3" environment:
Can you write Python code for it?

# Note: this requires a better understanding
tmp = pd.DataFrame.from_dict(env.P[93], orient='index')
tmp = pd.DataFrame(tmp[0].tolist(), index=tmp.index, columns=['A', 'State_no', 'Reward', 'Illegal_Passenger'])
tmp

help(env)


I am working on the file "Reinforcement-Learning/TD0-models/01.TaxiProblem.ipynb".
The first task is:
It turns out that the best one is the so-called "trial_1":
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/01.TaxiProblem.ipynb".
The first task is:
The environment is shown below, where the letters (R, G, Y, B) represents the different locations and a tiny yellow colored rectangle is the taxi driving by our agent.
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
2. RL-Algorithms based on Temporal Difference TD(0)
2a. Load the "Temporal Difference" Python class
Load the Python class PlotUtils() which provides various plotting utilities and start a new instance.
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
2b. SARSA: On-Policy TD(0) Control
Can you write Python code for it?

# Define Number of Episodes
n_episodes = 3e+3

# e-greedy parameters to investigate
print('Determine the epsilon parameters for the epsilon-greedy policy...\n')
epsilons = np.array([0.1, 0.13, 0.16])
print('epsilons: '.format(epsilons), '\n')

# various step-sizes (alpha) to try
print('Determine the step-sizes parameters (alphas) for the TD(0)...\n')
step_sizes = np.array([0.4])
print('step_sizes: {}'.format(step_sizes), '\n')

# Fixed discount
discount_fixed = 1

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
epsilons, step_sizes = np.meshgrid(epsilons, step_sizes)
epsilons = epsilons.flatten()
step_sizes = step_sizes.flatten()

# Create a dictionary of the RL-trials of interest
RL_trials = {"baseline":
             {'epsilon': 0.1,
              'step_size': 0.5, 'discount': 1}}

for n, trial in enumerate(list(zip(epsilons, step_sizes))):
    key = 'trial_' + str(n+1)
    RL_trials[key] = {'epsilon': trial[0],
                      'step_size': trial[1], 'discount': discount_fixed}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes...\n'.format(int(n_episodes)))

rewards_per_trial_SARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_SARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']

    # Apply SARSA [on-policy TD(0) Control]
    q_values, tot_rewards = TD0.sarsa_on_policy_control(env,
                                                        n_episodes=n_episodes,
                                                        step_size=step_size, discount=discount, epsilon=epsilon)

    # Update "rewards_per_trial" and "q_values_per_trial" OrderedDicts
    rewards_per_trial_SARSA[trial] = tot_rewards
   ...


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
2c. Q-Learning: Off-Policy TD(0) Control
Can you write Python code for it?

# Define Number of Episodes
# n_episodes = 2e+3

# e-greedy parameters to investigate
print('Determine the epsilon parameters for the epsilon-greedy policy...\n')
epsilons = np.array([0.1, 0.13, 0.16])
print('epsilons: '.format(epsilons), '\n')

# various step-sizes (alpha) to try
print('Determine the step-sizes parameters (alphas) for the TD(0)...\n')
step_sizes = np.array([0.4])
print('step_sizes: {}'.format(step_sizes), '\n')

# Fixed discount
discount_fixed = 1

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
epsilons, step_sizes = np.meshgrid(epsilons, step_sizes)
epsilons = epsilons.flatten()
step_sizes = step_sizes.flatten()

# Create a dictionary of the RL-trials of interest
RL_trials = {"baseline":
             {'epsilon': 0.1,
              'step_size': 0.5, 'discount': 1}}

for n, trial in enumerate(list(zip(epsilons, step_sizes))):
    key = 'trial_' + str(n+1)
    RL_trials[key] = {'epsilon': trial[0],
                      'step_size': trial[1], 'discount': discount_fixed}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes...\n'.format(int(n_episodes)))

rewards_per_trial_QL = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_QL = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']

    # Apply SARSA [on-policy TD(0) Control]
    q_values, tot_rewards = TD0.q_learning_off_policy(env,
                                                      n_episodes=n_episodes,
                                                      step_size=step_size, discount=discount, epsilon=epsilon)

    # Update "rewards_per_trial" and "q_values_per_trial" OrderedDicts
    rewards_per_trial_QL[trial] = tot_rewards
   ...


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
3. Double Learning: a method to mitigate maximization bias
3a. Double SARSA: On-Policy TD(0) Control
Can you write Python code for it?

# Define Number of Episodes
# n_episodes = 2e+3

# e-greedy parameters to investigate
print('Determine the epsilon parameters for the epsilon-greedy policy...\n')
epsilons = np.array([0.1, 0.13, 0.16])
print('epsilons: '.format(epsilons), '\n')

# various step-sizes (alpha) to try
print('Determine the step-sizes parameters (alphas) for the TD(0)...\n')
step_sizes = np.array([0.4])
print('step_sizes: {}'.format(step_sizes), '\n')

# Fixed discount
discount_fixed = 1

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
epsilons, step_sizes = np.meshgrid(epsilons, step_sizes)
epsilons = epsilons.flatten()
step_sizes = step_sizes.flatten()

# Create a dictionary of the RL-trials of interest
RL_trials = {"baseline":
             {'epsilon': 0.1,
              'step_size': 0.5, 'discount': 1}}

for n, trial in enumerate(list(zip(epsilons, step_sizes))):
    key = 'trial_' + str(n+1)
    RL_trials[key] = {'epsilon': trial[0],
                      'step_size': trial[1], 'discount': discount_fixed}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes...\n'.format(int(n_episodes)))

rewards_per_trial_DSARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_DSARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']

    # Apply SARSA [on-policy TD(0) Control]
    q_values_1, tot_rewards = TD0.sarsa_on_policy_control(env,
                                                        n_episodes=n_episodes,
                                                        step_size=step_size, discount=discount, epsilon=epsilon,
                                                        double_learning=True)

    # Update "rewards_per_trial" and...


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
3c. Double Expected SARSA: On-Policy TD(0) Control
Can you write Python code for it?

# Define Number of Episodes
# n_episodes = 2e+3

# e-greedy parameters to investigate
print('Determine the epsilon parameters for the epsilon-greedy policy...\n')
epsilons = np.array([0.1, 0.13, 0.16])
print('epsilons: '.format(epsilons), '\n')

# various step-sizes (alpha) to try
print('Determine the step-sizes parameters (alphas) for the TD(0)...\n')
step_sizes = np.array([0.4])
print('step_sizes: {}'.format(step_sizes), '\n')

# Fixed discount
discount_fixed = 1

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
epsilons, step_sizes = np.meshgrid(epsilons, step_sizes)
epsilons = epsilons.flatten()
step_sizes = step_sizes.flatten()

# Create a dictionary of the RL-trials of interest
RL_trials = {"baseline":
             {'epsilon': 0.1,
              'step_size': 0.5, 'discount': 1}}

for n, trial in enumerate(list(zip(epsilons, step_sizes))):
    key = 'trial_' + str(n+1)
    RL_trials[key] = {'epsilon': trial[0],
                      'step_size': trial[1], 'discount': discount_fixed}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes...\n'.format(int(n_episodes)))

rewards_per_trial_DExpSARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_DExpSARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']

    # Apply SARSA [on-policy TD(0) Control]
    q_values, tot_rewards = TD0.expected_sarsa_on_policy(env,
                                                         n_episodes=n_episodes,
                                                         step_size=step_size, discount=discount, epsilon=epsilon,
                                                         double_learning=True)

    # Update "rewards_per_trial"...


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
4. Comparison of SARSA, Q-Learning and Expected SARSA best models
After an initial transient, Q-learning learns values for the optimal policy, the ones that travels right along the edge of the cliff. Unfortunately, this results the traveller fall-off the cliff ocasionally, because of the $\varepsilon$-greedy selection. SARSA, on the other hand, takes the action selection into account and learns the longer but safer path, through the upper part of the grid. Although, Q-learning actually learns the values of the optimal policy, its online performance is worse than that of SARSA, which learns the roundabout policy.
Can you write Python code for it?

winning_trial = 'baseline'
rewards_per_trial_best_models = OrderedDict([('Model_SARSA', np.array([])),
                                             ('Model_DSARSA', np.array([])),
                                             ('Model_QL', np.array([])),
                                             ('Model_DQL', np.array([])),
                                             ('Model_ExpSARSA', np.array([])),
                                             ('Model_DExpSARSA', np.array([]))])

rewards_per_trial_best_models['Model_SARSA'] = rewards_per_trial_SARSA[winning_trial]
rewards_per_trial_best_models['Model_DSARSA'] = rewards_per_trial_DSARSA[winning_trial]
rewards_per_trial_best_models['Model_QL'] = rewards_per_trial_QL[winning_trial]
rewards_per_trial_best_models['Model_DQL'] = rewards_per_trial_DQL[winning_trial]
rewards_per_trial_best_models['Model_ExpSARSA'] = rewards_per_trial_ExpSARSA[winning_trial]
rewards_per_trial_best_models['Model_DExpSARSA'] = rewards_per_trial_DExpSARSA[winning_trial]

title = 'Efficiency of the RL Method\n[SARSA vs Q-Learning and Expected SARSA Winning Models]'
plotutls.plot_learning_curve(rewards_per_trial_best_models, title=title, lower_reward_ratio=-100)

title = 'Efficiency of the RL Method\n[SARSA vs Q-Learning and Expected SARSA Winning Models]'
plotutls.plot_learning_curve(rewards_per_trial_best_models, title=title, lower_reward_ratio=-35)


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
Double Expected SARSA on-Policy TD(0) Control:
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
Double SARSA on-Policy TD(0) Control:
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
Instantiate the class for the environment of interest:
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
Load the Cliff-Walking environment:
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
Load the Temporal Difference Python class, TemporalDifferenceUtils():
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
Possible traveller's actions are of course her movements in this grid:
- "UP":  denoted by 0
- "RIGHT": denoted by 1
- "DOWN": denoted by 2
- "LEFT": denoted by 3
To get the new state at every next step of an episode, you may pass the current action into the .step() method of the environment. The environment then will return a tuple (observation, reward, done, info) each of which are explained as below:
- observation (object): agent's observation of the current environment
- reward (float): amount of reward returned after previous action
- done (bool): whether the episode has ended, in which case further step() calls will return undefined results
- info (dict): contains auxiliary diagnostic information (helpful for debugging, and sometimes learning)
Note: At termination of each episode, the programmer is responsible to reset the environment.
For further details concerning the CliffWalking-v0 environment of OpenAI Gym toolkit consult the docstring below.
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
Q-Learning off-Policy TD(0) Control:
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
The OpenAI Gym toolkit includes the below environment for the "Cliff-Walking" problem:
Can you write Python code for it?

print('OpenAI Gym environments for Cliff Walking Problem:')
[k for k in gym.envs.registry.env_specs.keys() if k.find('Cliff' , 0) >=0]


I am working on the file "Reinforcement-Learning/TD0-models/02.CliffWalking.ipynb".
The first task is:
This environment has to do about gridworld shown below, where the traveller initial position (x) and the target to achieve (reach T) has been flagged appropriately. In addition in a one of the edge of this gridwordld example there is a "Cliff" denoted with C. Reward is $-1$ on all transitions except those into the cliff region. Steppping into this region incurs a reward of $-100$ and sends the agent instantly back to the start.
Once the environment is initialized you get the situation below. This is an episodic (undiscounted) task with start at traveller's starting point, and it is completed either when the goal is achieved, that is the traveller manage to reach the target location, T, or she may happen to step into the cliff. In this case the environment is reseted in each initial state.
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
2b. n-step TD Prediction (estimating $V \approx v_{\pi}$)
We define the functions below to help:
1. compute the optimal state-action values of this problem,
2. provide the optimal policy which is expected to be learned by the agent, and
3. visualize the result to verify that everything have been configured correctly.
Can you write Python code for it?

def cliff_walking_optimal_q_values(grid_height=4, grid_width=12, cliff_index = np.s_[:,:,:]):

    # Define the start position
    start = [grid_height - 1, 0]

    # Define the position of target dstination
    goal = [grid_height - 1, grid_width - 1]

    # Define a dictionary of possible actions
    actions_dict = {}
    actions = ['UP', 'RIGHT', 'DOWN', 'LEFT']
    for k, v in zip(actions, range(0, len(actions))):
        actions_dict[k] = v

    # Define a "q_values" array for grid-world of interest
    n_states = grid_height * grid_width
    n_actions = len(actions_dict)
    q_values =  np.full((grid_height, grid_width, n_actions), fill_value=-100.)

    # Determine "q_values" of optimal policy
    n_steps = grid_width
    q_values[:cliff_index[0],:,actions_dict['RIGHT']] = np.arange(0, n_steps, 1)
    q_values[start[0], start[1], actions_dict['UP']] = 0.5
    q_values[:goal[0],goal[1],actions_dict['DOWN']] = n_steps

    return q_values

def cliff_walking_optimal_policy(env, state):
    active_q = cliff_walking_optimal_q_values(grid_height=4, grid_width=12, cliff_index = np.s_[3,1:12,:])
    active_q = active_q.reshape((env.observation_space.n, env.action_space.n))
    return TD.epsilon_greedy_policy(env, active_q, state, epsilon=0.)

# print optimal policy
def print_optimal_policy(q_values, grid_height=4, grid_width=12):
    # Define a helper dictionary of actions
    actions_dict = {}
    actions = ['UP', 'RIGHT', 'DOWN', 'LEFT']
    for k, v in zip(actions, range(0, len(actions))):
        actions_dict[k] = v

    # Define the position of target dstination
    GOAL = [3, 11]

    # Reshape the "q_values" table to follow grid-world dimensionality
    q_values = q_values.reshape((grid_height, grid_width, len(actions)))

    optimal_policy = []
    for i in range(0, grid_height):
        optimal_policy.append([])
        for j in range(0, grid_width):
            if [i, j] == GOAL:
                optimal_policy[-1].append('G')
                continue
           ...


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
3. RL-Algorithms based on Temporal Difference: On-Policy TD(n) Control
3a. SARSA: On-Policy TD(n) Control
Can you write Python code for it?

# Define TD(n) execution parameters
runs = 10         # Number of Independent Runs
n_episodes = 100   # Number of Episodes

# Various n-steps SARSA algorithms to try
print('Determine the n-steps your are interested to explore...\n')
n_step_min = 2; n_step_max = 6
n_steps = np.arange(n_step_min, n_step_max + 1)
print('n_steps: {}'.format(n_steps), '\n')

# various discount factors to try
discount_fixed = 1.
print('Determine a fixed discount factor: {}\n'.format(discount_fixed))

# various step size parameters to try
step_size_fixed = 0.3
print('Determine a fixed step-size: {}\n'.format(step_size_fixed))

# various epsilon parameters to try
epsilon_fixed = 0.1
print('Determine a fixed epsilon: {}\n'.format(epsilon_fixed))

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
n_steps, discounts = np.meshgrid(n_steps, discount_fixed)
n_steps = n_steps.flatten()
discounts = discounts.flatten()

# Create a dictionary of the RL-trials of interest
RL_trials = {"sarsa(0)":
             {'epsilon': epsilon_fixed,
              'step_size': step_size_fixed, 'discount': discount_fixed, 'n_step': 1}}

for n, trial in enumerate(list(zip(n_steps, discounts))):
    key = 'sarsa({})'.format(trial[0]-1)
    RL_trials[key] = {'epsilon': epsilon_fixed,
                      'step_size': step_size_fixed, 'discount': trial[1], 'n_step': trial[0]}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes and {1:,} independent runs...\n'.format(int(n_episodes), int(runs)))

rewards_per_trial_On_Policy_SARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_On_Policy_SARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']
    n_step = params_dict['n_step']
...


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
3b. Expected SARSA: On-Policy TD(n) Control
Can you write Python code for it?

# Define TD(n) execution parameters
runs = 10         # Number of Independent Runs
n_episodes = 100   # Number of Episodes

# Various n-steps SARSA algorithms to try
print('Determine the n-steps your are interested to explore...\n')
n_step_min = 2; n_step_max = 6
n_steps = np.arange(n_step_min, n_step_max + 1)
print('n_steps: {}'.format(n_steps), '\n')

# various discount factors to try
discount_fixed = 1.
print('Determine a fixed discount factor: {}\n'.format(discount_fixed))

# various step size parameters to try
step_size_fixed = 0.3
print('Determine a fixed step-size: {}\n'.format(step_size_fixed))

# various epsilon parameters to try
epsilon_fixed = 0.1
print('Determine a fixed epsilon: {}\n'.format(epsilon_fixed))

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
n_steps, discounts = np.meshgrid(n_steps, discount_fixed)
n_steps = n_steps.flatten()
discounts = discounts.flatten()

# Create a dictionary of the RL-trials of interest
RL_trials = {"sarsa(0)":
             {'epsilon': epsilon_fixed,
              'step_size': step_size_fixed, 'discount': discount_fixed, 'n_step': 1}}

for n, trial in enumerate(list(zip(n_steps, discounts))):
    key = 'sarsa({})'.format(trial[0]-1)
    RL_trials[key] = {'epsilon': epsilon_fixed,
                      'step_size': step_size_fixed, 'discount': trial[1], 'n_step': trial[0]}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes and {1:,} independent runs...\n'.format(int(n_episodes), int(runs)))

rewards_per_trial_On_Policy_ExpSARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_On_Policy_ExpSARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']
    n_step =...


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
4b. Expected SARSA: Off-Policy TD(n) Control
Can you write Python code for it?

# Define TD(n) execution parameters
runs = 10          # Number of Independent Runs
n_episodes = 500   # Number of Episodes

# Various n-steps SARSA algorithms to try
print('Determine the n-steps your are interested to explore...\n')
n_step_min = 2; n_step_max = 6
n_steps = np.arange(n_step_min, n_step_max + 1)
print('n_steps: {}'.format(n_steps), '\n')

# various discount factors to try
discount_fixed = 1.
print('Determine a fixed discount factor: {}\n'.format(discount_fixed))

# various step size parameters to try
step_size_fixed = 0.8
print('Determine a fixed step-size: {}\n'.format(step_size_fixed))

# various epsilon parameters to try
epsilon_fixed = 0.1
print('Determine a fixed epsilon: {}\n'.format(epsilon_fixed))

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
n_steps, discounts = np.meshgrid(n_steps, discount_fixed)
n_steps = n_steps.flatten()
discounts = discounts.flatten()

# Create a dictionary of the RL-trials of interest
RL_trials = {"sarsa(0)":
             {'epsilon': epsilon_fixed,
              'step_size': step_size_fixed, 'discount': discount_fixed, 'n_step': 1}}

for n, trial in enumerate(list(zip(n_steps, discounts))):
    key = 'sarsa({})'.format(trial[0]-1)
    RL_trials[key] = {'epsilon': epsilon_fixed,
                      'step_size': step_size_fixed, 'discount': trial[1], 'n_step': trial[0]}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes and {1:,} independent runs...\n'.format(int(n_episodes), int(runs)))

rewards_per_trial_Off_Policy_ExpSARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_Off_Policy_ExpSARSA = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())

for trial, params_dict in RL_trials.items():

    # Read out parameters from "params_dict"
    epsilon = params_dict['epsilon']
    step_size = params_dict['step_size']
    discount = params_dict['discount']
    n_step =...


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
5. A Unifying Algorithm: Off-Policy n-step $Q(\sigma)$, dynamic $\sigma$
We train below a $Q(\sigma)$ algorithm following a 0 to 12-step temporal difference updates.
The discount factor of the MDP, the step size between the updates and the probability of the epsilon-greedy policy that this agent learns has been set as below:

discount factor of MDP: $\gamma=1$
step size between the updates: $\alpha = 0.3$
epsilon of epsilon-greedy policy: $\varepsilon = 0.1$

The $\sigma$ parameter which controls the degree of sampling followed in each update, and more specifically:

Once $\sigma = 0$, results in a pure expectation without sampling, whereas
Once $\sigma = 1$, results in the other extreme in full sampling,

has been set initially at $\sigma = 0.5$, but with the option "sigma = dynamic" we asked from the .off_policy_q_sigma() function of the TDn_Utils class to change $\sigma$ dynamically, towards larger/smaller values depending on the improvement achieved in terms of "Commulative Mean Reward / Number of episodes". The corrections in the parameter $\sigma$ have been set to occur every 10 next episodes.
Can you write Python code for it?

# Define TD(n) execution parameters
runs = 10          # Number of Independent Runs
n_episodes = 300   # Number of Episodes

# Various n-steps SARSA algorithms to try
print('Determine the n-steps your are interested to explore...\n')
n_step_min = 2; n_step_max = 13
n_steps = np.arange(n_step_min, n_step_max + 1)
print('n_steps: {}'.format(n_steps), '\n')

# various discount factors to try
discount_fixed = 1.
print('Determine a fixed discount factor: {}\n'.format(discount_fixed))

# various step size parameters to try
step_size_fixed = 0.3
print('Determine a fixed step-size: {}\n'.format(step_size_fixed))

# various epsilon parameters to try
epsilon_fixed = 0.1
print('Determine a fixed epsilon: {}\n'.format(epsilon_fixed))

# Determine a sigma prameter, controlling the degree of sampling at each step of the TD-n algorithm
sigma = None
if not sigma:
    sigma = 'Random Variable in [0,1] range'
print('Determine sigma: {}'.format(sigma))
print('[Note: Controls the degree of sampling at each step of the TD(n) algorithm]\n')

# Create a mesh-grid of trials
print('Create a dictionary of the RL-models of interest...\n')
n_steps, discounts = np.meshgrid(n_steps, discount_fixed)
n_steps = n_steps.flatten()
discounts = discounts.flatten()

# Create a dictionary of the RL-trials of interest
RL_trials = {"0-step Q(σ)":
             {'epsilon': epsilon_fixed,
              'step_size': step_size_fixed, 'discount': discount_fixed, 'n_step': 1}}

for n, trial in enumerate(list(zip(n_steps, discounts))):
    key = '{}-step Q(σ)'.format(trial[0]-1)
    RL_trials[key] = {'epsilon': epsilon_fixed,
                      'step_size': step_size_fixed, 'discount': trial[1], 'n_step': trial[0]}
print('Number of RL-models to try: {}\n'.format(len(RL_trials)))

print('Let all RL-models to be trained for {0:,} episodes and {1:,} independent runs...\n'.format(int(n_episodes), int(runs)))

rewards_per_trial_Off_Policy_QSigma = OrderedDict((label, np.array([])) for label, _ in RL_trials.items())
q_values_per_trial_Off_Policy_QSigma =...


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
Instantiate the class for the environment of interest:
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
Possible traveller's actions are of course her movements in this grid:
- "UP":  denoted by 0
- "RIGHT": denoted by 1
- "DOWN": denoted by 2
- "LEFT": denoted by 3
To get the new state at every next step of an episode, you may pass the current action into the .step() method of the environment. The environment then will return a tuple (observation, reward, done, info) each of which are explained as below:
- observation (object): agent's observation of the current environment
- reward (float): amount of reward returned after previous action
- done (bool): whether the episode has ended, in which case further step() calls will return undefined results
- info (dict): contains auxiliary diagnostic information (helpful for debugging, and sometimes learning)
Note: At termination of each episode, the programmer is responsible to reset the environment.
For further details concerning the CliffWalking-v0 environment of OpenAI Gym toolkit consult the docstring below.
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
Reward achieved with the number of episodes: remaining six trials, 6 to 12-step $Q(\sigma)$
Can you write Python code for it?

rest_RL_trials =  list(RL_trials.keys())[5:] #+ [first6_RL_trials[0]]
rewards_per_trial = OrderedDict((label, rewards_per_trial_Off_Policy_QSigma[label]) for label in rest_RL_trials)
title = 'Efficiency of the RL Method\n[n-step $\mathbf{Q(\sigma)}$ (Off-Policy TD(n) Control, rest 6 trials)]'
plotutls.plot_learning_curve(rewards_per_trial, title=title,
                             cumulative_reward=True, lower_reward_ratio=None)

for trial in rest_RL_trials:
    print('\n', trial, ':')
    q_vals = q_values_per_trial_Off_Policy_QSigma[trial]
    print_optimal_policy(q_vals, grid_height=4, grid_width=12)


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
The OpenAI Gym toolkit includes the below environment for the "Cliff-Walking" problem:
Can you write Python code for it?


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
Use the temporal_difference_prediction() method to predict the state values if the agent let to follow a 5-step TD learning path.
Can you write Python code for it?

runs=10; n_episodes = 100
s_values = TD.temporal_difference_prediction(env, cliff_walking_optimal_policy,
                                             runs=runs, n_episodes=n_episodes, decimals=2,
                                             n_step=6, step_size=0.3, discount=1., epsilon=0.1)

title = 'State-value Predictions\n[Cliff-Walking task]'
plotutls.plot_state_values(s_values, grid_height=4, grid_width=12, title=title)


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
Verify that the optimal policy has been configured correctly.
Can you write Python code for it?

active_q = cliff_walking_optimal_q_values(grid_height=4, grid_width=12, cliff_index = np.s_[3,1:12,:])
print_optimal_policy(active_q, grid_height=4, grid_width=12)


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
Verify the learning curves of the RL-models we trained.
Can you write Python code for it?

title = 'Efficiency of the RL Method\n[Expected SARSA Οn-policy TD(n) Control]'
plotutls.plot_learning_curve(rewards_per_trial_On_Policy_ExpSARSA,title=title,
                             cumulative_reward=True, lower_reward_ratio=None)


title = 'Efficiency of the RL Method\n[Expected SARSA Off-Policy TD(n) Control]'
plotutls.plot_learning_curve(rewards_per_trial_Off_Policy_ExpSARSA,title=title,
                             cumulative_reward=True, lower_reward_ratio=None)


title = 'Efficiency of the RL Method\n[SARSA Off-Policy TD(n) Control]'
plotutls.plot_learning_curve(rewards_per_trial_Off_Policy_SARSA,title=title,
                             cumulative_reward=True, lower_reward_ratio=None)


I am working on the file "Reinforcement-Learning/TDn-models/01.CliffWalking_with_TD(n)-models.ipynb".
The first task is:
Visualize agent's move which is suggested by the solutions.
Can you write Python code for it?

for trial in list(RL_trials.keys()):
    print('\n', trial, ':')
    q_vals = q_values_per_trial_Off_Policy_SARSA[trial]
    print_optimal_policy(q_vals, grid_height=4, grid_width=12)


for trial in list(RL_trials.keys()):
    print('\n', trial, ':')
    q_vals = q_values_per_trial_Off_Policy_ExpSARSA[trial]
    print_optimal_policy(q_vals, grid_height=4, grid_width=12)


I am working on the file "Reinforcement_Learing_An_Introduction/Dynamic_Programming/note.ipynb".
The first task is:
4.2 Policy Improvement
\begin{align}
    q_\pi(s, a) = \sum_{s', r} p(s', r \mid s, a) \left [ r + \gamma v_\pi(s') \right ]
\end{align}
policy improvement theorem: For all $s \in \mathcal{S}$, $q_\pi(s, \pi'(s)) \geq v_\pi(s)$, then $v_{\pi'}(s) \geq v_\pi(s)$.
=> new greedy policy: $\pi'(s) = \operatorname{arg \, max}a q\pi(s, a)$, policy imporvement.
If there are ties in policy improvement step, each maximizing action can be given a portion of the probability of being selected in the new greedy policy.
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Dynamic_Programming/note.ipynb".
The first task is:
4.3 Policy Iteration
policy iteration:
\begin{align}
    \pi_0 \overset{E}{\longrightarrow} v_{\pi_0} \overset{I}{\longrightarrow} \pi_1 \overset{E}{\longrightarrow} v_{\pi_1} \overset{I}{\longrightarrow} \pi_2 \overset{E}{\longrightarrow} \cdots \overset{I}{\longrightarrow} \pi_\ast \overset{E}{\longrightarrow} v_{\pi_\ast}
\end{align}
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Dynamic_Programming/note.ipynb".
The first task is:
4.4 Value Iteration
truncate policy evaluation
=> value iteration: policy evaluation is stopped after just one sweep (one update of each state).
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Dynamic_Programming/note.ipynb".
The first task is:
4.5 Asynchronouse Dynamic Programming
avoiding sweeps
focus the DP algorithm's update onto parts of the state set that are most relevant to the agent.
4.6 Generalized Policy Iteration
Almost all reinforcement learning methods are well described as GPI:
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Finite_Markov_Decision_Processes/note.ipynb".
The first task is:
Chapter 3: Finite Markov Decision Processes
MDP(Markov Decision Processes): actions influence not just immediate rewards, but also subsequential situations.
3.1 The Agent-Environment Interface
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Monte_Carlo_Methods/note.ipynb".
The first task is:
5.5 Off-policy Prediction via Importances Sampling
use two policies:
+ target policy $\pi$: the potimal policy that is learned about.
+ behavior policy $b$: more exploratory policy that is used to generate behavior.
assumption of coverage: $\pi(a \mid s) > 0$ implies $b(a \mid s) > 0$.
we wish to estimate $v_\pi$ or $q_\pi$, but all we have are episodes following another policy $b$, where $b \neq \pi$.
=> importance sampling: a general technique for estimating expected values under one distribution given samples from another.
=> importance-sampling ratio $\rho_{t:T-1}$:
Given a starting state $S_t$, we have: $\operatorname{Pr}{A_t, S_{t+1}, A_{t+1}, \cdots, S_T \mid S_t, A_{t:T-1} \sim \pi} = \prod_{K=t}^{T-1} \pi(A_k \mid S_k) p(S_{k+1} \mid S_k, A_k)$
\begin{equation}
    \rho_{t:T-1} \doteq \prod_{K=t}^{T-1} \frac{\pi(A_k \mid S_k)}{b(A_k \mid S_k)}
\end{equation}
So we can have the right expected value by:
\begin{align}
    \mathbb{E}[G_t \mid S_t] &= v_b(S_t) \
    \mathbb{E}[\rho_{t:T-1} G_t \mid S_t] &= v_\pi(S_t)
\end{align}
To estimate $v_\pi(s)$, we simply scale the returns by the ratios and average the results:
+ ordinary importance sampling: $V(s) \doteq \frac{\sum_{t \in \mathcal{J}(s)} \rho_{t:T(t)-1} G_t}{|\mathcal{J}(s)|}$: unbiased, but it can be extreme.
+ weighted importance sampling: $V(s) \doteq \frac{\sum_{t \in \mathcal{J}(s)} \rho_{t:T(t)-1} G_t}{\sum_{t \in \mathcal{J}(s)} \rho_{t:T(t)-1}}$: biased, but its variance is bounded.
5.6 Incremental Implementation
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Monte_Carlo_Methods/note.ipynb".
The first task is:
Chapter 5 Monte Carlo Methods
Monte Carlo methods require only experience: sample sequences of states, actions, and rewards from actual or simulated interaction with an environment.
requirements: averaging sample returns => we define Monte Carlo methods only for episodic tasks.
5.1 Monte Carlo Prediction
the value of a state: expected return starting from that state.
An obvious way to estimate it from experience: simply to average the returns observed after visits to that state.
$s$ may be visited multiple times in the same episode:
+ first-visit MC method
+ every-visit MC method
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Multi-armed_Bandits/note.ipynb".
The first task is:
2.6 Optimistic Initial Values
All the methods we have discussed are biased by their initial estimates.
如2.6式红色部份：
\begin{equation}
    Q_{n+1} = \color{red}{(1 - \alpha)^n Q_1} + \sum_{i=1}^n \alpha (1 - \alpha)^{n-1} R_i
\end{equation}
We call this technique for encouraging exploration optimistic initial values: say, using $Q_1(a) = 5$.
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Multi-armed_Bandits/note.ipynb".
The first task is:
2.8 Gradient Bandit Algorithms
Learning a numerical preference $H_t(a)$ for each action.
Only the relative preference of one action over another is important, and the action probabilities are determined according to a soft-max distribution:
\begin{equation}
    \operatorname{Pr}{A_t = a} \doteq \frac{e^{H_t(a)}}{\sum_{b=1}^k e_{H_t(b)}} = \pi_t(a)
\end{equation}
update policy is based on the idea of stochastic gradient ascent:
\begin{align}
    H_{t+1}(A_t) & \doteq H_t(A_t) + \alpha (R_t - \bar{R}t) (1 - \pi_t(A_t)) & \text{ and } \
    H{t+1}(a) & \doteq H_t(a) - \alpha (R_t - \bar{R}_t) \pi_t(a) & \text{ for all $a \neq A_t$ } \
\end{align}
$\bar{R}_t$ is the average of all the rewards up through, which serves as a baseline.
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Multi-armed_Bandits/note.ipynb".
The first task is:
2.9 Associative Search (Contextual Bandits)

nonassociative task: stationary task. 没有场景
associative task: the goal is to learn a policy: a mapping from situations to the actions that are best in those situations. 有场景的区分
full reinforcement learning problem: action are allowed to effec the next situation as well as the reward. 行为可以影响场景。

Exercise 2.10:

cannot tell which case: 0.5 (random)
can tell: A - 2, B - 1: 0.55

2.10 Summary
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Multi-armed_Bandits/note.ipynb".
The first task is:
Chapter 2: Multi-armed Bandits
2.1 A k-armed Bandit Problem
nonassociative, evaluative feedback problem:
reward = sample(P(action))
+ P(action) is a stationary probability distribution.
+ reward: $R$
+ $k$ different actions $a$.
+ goal: maximize the expected total reward over some time period.
The value $q$ of $a$ is the expected reward:
\begin{align}
    q(a) &= \mathbb{E} \left [ R_t \mid A_t = a \right ] \
         &\approx \frac1{N} \sum_t^N R_t \mid A_t = a \quad \text{law of large numbers} \
         &= Q_t(a)
\end{align}
2.2 Action-value Methods
one natural way to estimate the value of $a$ is by averaging the rewards actually received: (sample-average method)
\begin{equation}
    Q_t(a) = \frac{\sum_{i=1}^{t-1} R_i \cdot \mathbb{1}(A_i = a)}{\sum_{i=1}^{t-1} \cdot \mathbb{1}(A_i = a)}
\end{equation}
strategies:
+ greedy action: $A_t = \operatorname{argmax}_a Q_t(a)$
+ $\epsilon$-greedy actions:
\begin{equation}
    A_t =
    \begin{cases}
        \operatorname{argmax}_a Q_t(a) & \quad \text{ with probability 1 - $\epsilon$} \
        \text{a random action} & \quad \text{ with probability $\epsilon$}
    \end{cases}
\end{equation}
Exercise
2.1:
$0.5 * 1 + 0.5 * 1/2 = 0.75$
2.2:

definitely occur: steps = 2, 5
possiblly occur: steps= 1, 3, 4

2.3 The 10-armed Testbed
Can you write Python code for it?

reward_funcs = [
    lambda size: np.random.normal(0.2, 1, size),
    lambda size: np.random.normal(-0.8, 1, size),
    lambda size: np.random.normal(1.5, 1, size),
    lambda size: np.random.normal(0.4, 1, size),
    lambda size: np.random.normal(1.2, 1, size),
    lambda size: np.random.normal(-1.3, 1, size),
    lambda size: np.random.normal(-0.2, 1, size),
    lambda size: np.random.normal(-1.0, 1, size),
    lambda size: np.random.normal(1.0, 1, size),
    lambda size: np.random.normal(-0.4, 1, size),
]

examples = pd.DataFrame([func(2000) for func in reward_funcs]).T
examples.index.name = "Reward distribution"
plt.figure(figsize=(15, 7))
sns.violinplot(data=examples)
del examples

class GreedyAction(object):
    def __init__(self, k, varepsilon=0):
        self.varepsilon = varepsilon
        self.k = k
        # sum of rewards, number of action
        self._records = np.zeros((k, 2))

    def _Q_t(self):
        r = self._records[:, 0]
        n = self._records[:, 1]
        res = np.divide(r, n, out=np.zeros_like(r), where=n!=0)
        logger.info("Q_t: {}".format(res))
        return res

    def action(self):
        if np.random.random() < self.varepsilon:
            a = np.random.randint(low=0, high=self.k)
            logger.info("random action: {}".format(a))
            return a
        else:
            a = np.argmax(self._Q_t())
            logger.info("greedy action: {}".format(a))
            return a

    def update_reward(self, action, reward):
        self._records[action, 0] += reward
        self._records[action, 1] += 1

class Environment(object):
    def __init__(self, reward_funcs):
        self.reward_funcs = reward_funcs
        self.k = len(reward_funcs)

    def feedback(self, action):
        assert 0 <= action < self.k, "action: {} must be in [0, {}]".format(action, self.k)
        return self.reward_funcs[action](1)[0]

def run_10_armed(varepsilon):
    opt_action = 2
    env = Environment(reward_funcs)
    actioner = GreedyAction(env.k,...


I am working on the file "Reinforcement_Learing_An_Introduction/Multi-armed_Bandits/note.ipynb".
The first task is:
General form:
\begin{equation}
    \text{NewEstimate} \leftarrow \text{OldEstimate} + \text{StepSize} \left [ \text{Target} - \text{OldEstimate} \right ]
\end{equation}
2.5 Tracking a Nonstationary Problem
it makes sense to give more weight to recent rewards than to long-past rewards: $Q_{n+1} = \alpha R_n + (1 - \alpha) Q_n$ (exponential recency-weighted average)
the conditions required to assure convergence with probability 1:
\begin{align}
    \sum_{n=1}^\infty \alpha_n(a) &= \infty &
    \sum_{n=1}^\infty \alpha_n^2(a) &< \infty &
\end{align}
exercise 2.5:
Can you write Python code for it?

class NonstationaryEnvironment(object):
    def __init__(self, reward_funcs):
        self.reward_funcs = reward_funcs
        self.k = len(reward_funcs)

        self.iteration = 0

    def feedback(self, action):
        assert 0 <= action < self.k, "action: {} must be in [0, {}]".format(action, self.k)
        self.iteration += 1
        return self.reward_funcs[action](1)[0] + np.random.normal(0, self.iteration / 100, 1)[0]

def run_nonstationary_10_armed(varepsilon):
    env = NonstationaryEnvironment(reward_funcs)
    actioner = GreedyAction(env.k, varepsilon)
    step_records = []

    def step():
        history = []
        for _ in range(100):
            a = actioner.action()
            r = env.feedback(a)
            actioner.update_reward(a, r)
            logger.info("action: {}, reward: {}".format(a, r))
            history.append([a, r])
        history = np.array(history)
        return np.sum(history[:, 1]), history.shape[0]

    for _ in range(100):
        step_records.append(step())

    result = pd.DataFrame(step_records, columns=["total_rewards", "iterate_nums"])
    result = result.cumsum()
    result["average_rewards"] = result["total_rewards"] / result["iterate_nums"]
    return result

run_nonstationary_10_armed(0.01).plot(x='iterate_nums', y='average_rewards')

class ExpGreedyAction(object):
    def __init__(self, k, varepsilon=0, alpha=0.1):
        self.varepsilon = varepsilon
        self.alpha = alpha
        self.k = k
        # sum of rewards, number of action
        self._records = np.zeros((k, 2))

    def _Q_t(self):
        return self._records[:, 0]

    def action(self):
        if np.random.random() < self.varepsilon:
            a = np.random.randint(low=0, high=self.k)
            logger.info("random action: {}".format(a))
            return a
        else:
            a = np.argmax(self._Q_t())
            logger.info("greedy action: {}".format(a))
            return a

    def update_reward(self, action, reward):
       ...


I am working on the file "Reinforcement_Learing_An_Introduction/Planning_and_Learning_with_Tabular/note.ipynb".
The first task is:
8.4 Prioritized Sweeping
uniform selection is usually not the best; planning can be much more efficient if simulated transitions and updates are focused on particular state-action pairs.
backward focusing of planning computations: work backward from aribitary states that have changed in value. (propagation)
prioritized sweeping: prioritize the updates according to a measure of their urgency, and perform them in order of priority.
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Planning_and_Learning_with_Tabular/note.ipynb".
The first task is:
8.5 Expected vs. Sample Updates
Three questions:

Whether the algorithm updates state values $v$, or action values $q$?
Whether the algorithm estimates the value for the optimal policy $\ast$, or for an arbitrary given policy $\pi$?
Whether the updates are expected updates, or sample updates?
expected updates: better estimate (uncorrupted by sampling error), but more computation.
sample updates: superior on problems with large stochastic branching factors and too many states to be solved exactly.
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Planning_and_Learning_with_Tabular/note.ipynb".
The first task is:
Learning and planning are deeply integrated in the sense that they share almost all the same machinery, differing only in the source of their experience.

Indirect methods: make fuller use of a limited amount of experiences.
Direct methods: much simpler and are not effected by biases in the design of the model.

8.3 When the Model Is Wrong
Models may be incorrect because:

Environment is stochastic and only a limited number of samples have been observed;
The model was learned using function approximation that has generalized imperfectly;
Then environment has changed and its new behavior has not yet been observed:
original optimial solution doesn't work any more.
better solution exists after environemt changed.

=> conflict between exploration and exploitation => simple heuristics are often effective.
Dyna-Q+ agen: keeps track for each state-action pair. The more time that has elapsed, the greater the chance to be picked next time => special "bonus reward": $r + k \sqrt{\tau}$
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Temporal_Difference_Learning/note.ipynb".
The first task is:
6.2 Advantages of TD Prediction Methods
TD: they learn a guess from a guess - they boostrap.

advantage:
over DP: TD do not require a model of the environment, of its reward and next-state probability distributions.
over Monte Carlo: TD are naturally implemented in an online, fully incremental fashion.

TD: guarantee convergence.
In practice, TD methods have usually been found that converge faster than constant-$\alpha$ MC methods on stochastic tasks.
6.3 Optimality of TD(0)
batch updating: updates are made only after processing each complete batch of training data until the value function converges.

Batch Monte Carlo methods: always find the estimates that minimize mean-squared error on the training set.
Batch TD(0): always find the estimates that would be exactly correct for the maximum-likelihood model of the Markov process.

6.4 Sarsa: On-policy TD Control
$Q(S_t, A_t) \gets Q(S_t, A_t) + \alpha \left [ R_{t+1} + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_t,, A_t) \right ]$
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Temporal_Difference_Learning/note.ipynb".
The first task is:
6.5 Q-learning: Off-policy TD Control
$Q(S_t, A_t) \gets Q(S_t, A_t) + \alpha \left [ R_{t+1} + \gamma \color{blue}{\max_a Q(S_{t+1}, a)} - Q(S_t,, A_t) \right ]$
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Temporal_Difference_Learning/note.ipynb".
The first task is:
6.6 Expected Sarsa
use expeteced value, how likely each action is under the current policy.
\begin{align}
    Q(S_t, A_t) & \gets Q(S_t, A_t) + \alpha \left [ R_{t+1} + \gamma \color{blue}{\mathbb{E}[Q(S_{t+1}, A_{t+1}) \mid S_{t+1}]} - Q(S_t,, A_t) \right ] \
                & \gets Q(S_t, A_t) + \alpha \left [ R_{t+1} + \gamma \color{blue}{\sum_a \pi(a \mid S_{t+1}) Q(S_{t+1}, a)} - Q(S_t,, A_t) \right ]
\end{align}

con: additional computational cost.
pro: eliminate the variance due to the random seleciton of $A_{t+1}$.

6.7 Maximization Bias and Double Learning
maximization bias:
+ a maximum over estimated values => an estimate of the maximum value => significant positive bias.
root of problem: using the same samples (plays) both to determine the maximizing action and to estimate its value. => divide the plays in two sets ($Q_1, Q_2$) and use them to learn two indepedent estimates. (double learning)
$Q_1(S_t, A_t) \gets Q_1(S_t, A_t) + \alpha \left [ R_{t+1} + \gamma Q_2 \left( S_{t+1}, \operatorname{argmax}a Q_1(S{t+1}, a) \right) - Q_1(S_t,, A_t) \right ]$
Can you write Python code for it?


I am working on the file "Reinforcement_Learing_An_Introduction/Temporal_Difference_Learning/note.ipynb".
The first task is:
Chapter 6 Temporal-Difference Learning
DP, TD, and Monte Carlo methods all use some variation of generalized policy iteration: primarily differences in their approaches to the prediction problem.
6.1 TD Prediction
constant-$\alpha$ MC: $V(S_t) \gets V(S_t) + \alpha \underbrace{\left [ G_t - V(S_t) \right ]}{= \sum{k=t}^{T-1} \gamma^{k-1} \theta_k}$
\begin{align}
    v_\pi(s) &\doteq \mathbb{E}\pi [ G_t \mid S_s = s] \qquad \text{Monte Carlo} \
             &= \mathbb{E}\pi [ R_{t+1} + \gamma \color{blue}{v_\pi(S_{t+1})} \mid S_t = s ] \quad \text{DP}
\end{align}
one-step TD, or TD(0): $V(S_t) \gets V(S_t) + \alpha \left [ \underbrace{R_{t+1} + \gamma \color{blue}{V(S_{t+1})} - V(S_t)}_{\text{TD error: } \theta_t} \right ]$
TD: samples the expected values and uses the current estimate $V$ instead of the true $v_\pi$.
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Ahora damos un ejemplo que sí es complementado, de hecho un álgebra de Boole
Can you write Python code for it?

u = set(a for a in range(1,31) if 30%a==0)

u

p = set((a,b) for a in u for b in u if b%a ==0)

r = relacion(p)

r.es_distributivo()

r.atomos()

r.es_algebra_Boole()

r.pinta(r.atomos())


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Claramente esta relación es de orden, y no es un orden total
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Diagramas de Hasse
Veamos ahora como ejemplo el conjunto de los divisores de 12 con la relación de divisibilidad
Can you write Python code for it?

d = set(a for a in range(1,13) if 12%a ==0)

rd =relacion(set((a,b) for a in d for b in d if b%a ==0))


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
El diamante
Can you write Python code for it?

r = divisores({0,1,2,3,5})

r.pinta()

r.hasse()

r.es_orden()

r.es_orden_total()

r.es_reticulo()

r.es_subreticulo({0,1,2,3})

r.es_distributivo()

r.es_complementado()


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
El pentágono
Can you write Python code for it?

r = divisores({1,0,2,4,3})

r.hasse()

r.es_distributivo()

r.es_complementado()


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
El universo es un atributo de la clase, y se puede acceder a él de la siguiente forma
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Los elementos notables se calculan con los métodos que tienen nombres acordes a ellos
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
O bien pintar todas las relaciones
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
O incluso podemos destacar esos elementos en el dibujo de relaciones
Can you write Python code for it?

rd.pinta(rd.mayorantes(set({2,3})))

rd.minimales(set({2,3}))

rd.minimo(set({2,4}))


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Otros ejemplos
Retículo que no es distributivo ni complementado
Can you write Python code for it?

r=divisores({1,0,2,18,9,3})

r.hasse()

r.es_complementado()

r.es_distributivo()

[s for s in r.universo if r.complemento(s)==None]

r.es_reticulo()


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Para definir una relación, podemos o bien dar un conjunto de parejas, o bien un conjunto de parejas y el universo donde están los elementos de los pares. En caso de no proporcionar el universo, se tomará como universo el conjunto de todos los valores que aparecen en las parejas.
Por ejemplo, definamos en el conjunto ${0,1,\ldots,11}$ la relación de ser congruentes módulo 5
Can you write Python code for it?

u =set(range(12))
rl = set((a,b) for a in u for b in u if (a-b)%5 ==0)

r =relacion(rl,u)


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Podemos dibujar el diagrama de Hasse como sigue
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Pues no es antisimétrica
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Sabemos que esta relación es de equivalencia
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
Si dibujamos las relaciones entre los elementos, aparecerán claramente las clases de equivalencia
Can you write Python code for it?


I am working on the file "Relaciones y Algebras de Boole/Tutorial relaciones.ipynb".
The first task is:
También podemos determinar si dos elementos están relacionados, bien usando el método rel o llamando a la relación con los dos elementos
Can you write Python code for it?


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
6) Gere gráficos para visualizar o histórico de treinamento. Em particular, gere curvas da precisão e da função de custo contra a quantidade de épocas de treinamento.
Gráfico agrupado com as Curvas de Treinamento: binary_accuracy e loss
Can you write Python code for it?

# Obtendo as métricas para exibir o Gráfico
metrics = history.history
metrics.keys()

plt.figure(figsize=(14,7))
plt.plot(metrics['binary_accuracy'])
plt.plot(metrics['loss'])
plt.title('Curvas de Treinamento')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['Precisão', 'Função de Custo'], loc='upper left')
plt.show();


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Caso 1: Rede Completamente  Conectada: única camada oculta e saída com softmax
Can you write Python code for it?

flatten_x_train = treino_X.reshape((treino_X.shape[0], -1))
flatten_x_test = teste_X.reshape((teste_X.shape[0], -1))

# Convertendo os alvos em matriz com dimensão de número de classes
train_set_y = keras.utils.to_categorical(treino_Y, classes.size)
test_set_y = keras.utils.to_categorical(teste_Y, classes.size)

flatten_x_train.shape , flatten_x_test.shape , train_set_y.shape , test_set_y.shape

RANDOM = keras.initializers.RandomNormal(seed = 656)
ADADELTA = keras.optimizers.Adadelta()
ERRO = keras.losses.categorical_crossentropy

model = Sequential()
model.add(Dense(3, input_shape=flatten_x_train.shape[1:], kernel_initializer=RANDOM, activation='relu'))
model.add(Dense(5, activation='sigmoid'))
model.add(Dense(2, activation='softmax'))

t1 = time.process_time()
model.compile(loss=ERRO, optimizer=ADADELTA, metrics=['binary_accuracy'])
history = model.fit(flatten_x_train, train_set_y, epochs=200, verbose=0)
t2 = time.process_time()

score = model.evaluate(flatten_x_test, test_set_y, verbose=0)
print('Custo do Teste:', score[0])
print('Precisão do Teste:', score[1])

print("\n ########## Tempo de Treinamento da Rede Completamente Conectada (CatvsNocat) ##########\n")
print ("\n\t ----- Computation time = " + str(t2 - t1) + "s -----")
print("\n #######################################################################################\n")

# Obtendo as métricas para exibir o Gráfico
metrics = history.history
metrics.keys()

plt.figure(figsize=(14,7))
plt.plot(metrics['binary_accuracy'])
plt.plot(metrics['loss'])
plt.title('Curvas de Treinamento')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['Precisão', 'Função de Custo'], loc='upper left')
plt.show();

PredicTest = model.predict(flatten_x_test).round()
print('Acurácia: ', accuracy_score(test_set_y, PredicTest))


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Configuração da Rede Neural ${feed-forward}$ with Epochs = 2000
Can you write Python code for it?

t1 = time.process_time()
model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['binary_accuracy'])
history = model.fit(PTrain, Train, epochs=2000, verbose=1)
t2 = time.process_time()

print("\n ########## Tempo de Treinamento da Rede Feed-Forward ##########\n")
print ("\n\t ----- Computation time = " + str(t2 - t1) + "s -----")
print("\n ####################################################################\n")

# Obtendo as métricas para exibir o Gráfico
metrics = history.history
metrics.keys()

plt.figure(figsize=(14,7))
plt.plot(metrics['binary_accuracy'])
plt.plot(metrics['loss'])
plt.title('Curvas de Treinamento')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['Precisão', 'Função de Custo'], loc='upper left')
plt.show();


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Construção da Rede Neural ${feed-forward}$ with Neurons = 3 and Activation = sigmoid
Can you write Python code for it?

model = Sequential()
L1 = Dense(3, input_dim=11, activation='sigmoid')
model.add(L1)
L2 = Dense(1, activation='sigmoid')
model.add(L2)


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Criando as matrizes ${P}$ e ${T}$
Can you write Python code for it?

P = dataTest.iloc[:,:11].as_matrix()
Test = dataTest.iloc[:, 11:].as_matrix()

P.shape , Test.shape

PTest = normalizaFeatures(P)
PTest


P = dataTrain.iloc[:,:11].as_matrix()
Train = dataTrain.iloc[:, 11:].as_matrix()

P.shape , Train.shape

PTrain = normalizaFeatures(P)
PTrain


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Criação de Modelo de Predição de Crédito
Após realizar os passos descritos acima, você deve ter condições de repetí-los para o conjunto de dados de
créditos (credtrain). Uma vez treinada, essa rede poderá inferir se um novo cliente irá ou não honrar um eventual
empréstimo.
Inicialmente, você deve carregar o arquivo credtrain.txt. Uma vez que os dados estão carregados, você deve criar
duas matrizes. A primeira matriz deve ter ordem ${1500}$ ${x}$ ${11}$ e deve conter os padrões de entrada (valores das variáveis ESTC, NDEP, RENDA, TIPOR, VBEM, NPARC, VPARC, TEL, IDADE, RESMS e ENTRADA). A outra matriz deve ter ordem ${1500}$ ${x}$ ${1}$ e deve conter os valores do atributo CLASSE. Vamos chamar essas duas matrizes de ${P}$ e ${T}$, respectivamente. A matriz ${P}$ é chamada a matriz de treinamento. Já a matriz T é chamada a matriz de alvo. Diferente do exemplo introdutório, um cuidado adicional que você deve ter é com relação à normalização dos dados
de entrada da rede, posto que algumas variáveis possuem ordens de grandeza diferentes.
Modelo de Classificação ${TREINAMENTO}$
Can you write Python code for it?

dataTrain = pd.read_csv("am-T3-dados/credtrain.txt", sep='\t',header=None)
dataTrain.head()


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Crie uma rede neural artificial completamente conectada de uma única camada oculta, com 3 neurônios nessa camada oculta e 1 neurônio na camada de saída. Em ambas as camadas, a função de ativação deve ser a função sigmóide logística.
Can you write Python code for it?

model = Sequential()
L1 = Dense(3, input_dim=4, activation='sigmoid')
model.add(L1)
L2 = Dense(1, activation='sigmoid')
model.add(L2)


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Declaração das Funcões utilizadas
Can you write Python code for it?

def normalizaFeatures(X):
    '''
    Função que normaliza cada elemento, dado um vetor.
    '''

    mu = np.mean(X, axis=0)
    sigma = np.std(X, axis=0)
    X = np.subtract(X, mu)
    X = np.divide(X,sigma)

    return X

# Função trazida dos exemplos apresentados no scikit-learn
def plot_confusion_matrix(matriz, classes, normalize=False, title='Matriz Confusão'):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """

    cmap = plt.cm.Blues
    CWH = "white"
    BLK = "black"
    CT = "center"

    if normalize:
        matriz = matriz.astype('float') / matriz.sum(axis=1)[:, np.newaxis]
        print("Matriz Confusão Normalizada")
    else:
        print('Matriz Confusão')

    print(matriz)

    plt.imshow(matriz, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    FORMAT = '.2f' if normalize else 'd'
    TRHSD = matriz.max() / 2.

    A = matriz.shape[0]
    B = matriz.shape[1]

    for i, j in itertools.product(range(A), range(B)):
        plt.text(j,i,format(matriz[i,j],FORMAT), horizontalalignment=CT, color=CWH if matriz[i,j] > TRHSD else BLK)

    plt.tight_layout()
    plt.ylabel('Verdadeiro')
    plt.xlabel('Predito')


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Iniciando Treinamento
Can you write Python code for it?

BTHSIZE = 64                                        # Definindo um valor de batch_size
CROSSENTROPY = keras.losses.categorical_crossentropy # Função de custo
ADADELTA = keras.optimizers.Adadelta()               # Função de minimização

model = Sequential()
model.add(Conv2D(32, kernel_size=(4, 4), activation='relu',input_shape=input_shape))
model.add(Conv2D(64, (4, 4), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(2, activation='softmax'))

t1 = time.process_time()
model.compile(loss=CROSSENTROPY, optimizer=ADADELTA, metrics=['accuracy'])
history = model.fit(x_train, y_train, batch_size=BTHSIZE, epochs=80, verbose=1, validation_data=(x_test, y_test))
t2 = time.process_time()

score = model.evaluate(x_test, y_test, verbose=0)
print('Custo do Teste:', score[0])
print('Precisão do Teste:', score[1])

print("\n ########## Tempo de Treinamento da Rede Convolucional (CatvsNocat) ##########\n")
print ("\n\t ----- Computation time = " + str(t2 - t1) + "s -----")
print("\n #######################################################################################\n")

# Obtendo as métricas para exibir o Gráfico
metrics = history.history
metrics.keys()

plt.figure(figsize=(14,7))
plt.plot(metrics['acc'])
plt.plot(metrics['val_acc'])
plt.title('Modelo de Acurácia')
plt.ylabel('acurácia')
plt.xlabel('Época')
plt.legend(['Treino', 'Teste'], loc='upper left')
plt.show();

plt.figure(figsize=(14,7))
plt.plot(metrics['loss'])
plt.plot(metrics['val_loss'])
plt.title('Modelo de Erro')
plt.ylabel('Erro')
plt.xlabel('Época')
plt.legend(['Treino', 'Teste'], loc='upper left')
plt.show();


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Procedimento de criação do ${Modelo}$ ${de}$ ${Classificação}$:
Defina a matriz de padrões. Note que os padrões são organizados em colunas. Por exemplo, a matriz de padrões abaixo contém cinco padrões de entrada (cada um com quatro características), e o primeiro padrão armazenado na matriz P abaixo é (0.4046, 0.9974, 0.3764, 0.6043).
$$
\left[\begin{array}{cc}
0.4046 & 0.3786 & 0.7010 & 0.8608 & 0.5947\
0.9974 & 0.8479 & 0.6201 & 0.4031 & 0.9653\
0.3764 & 0.9214 & 0.9331 & 0.7514 & 0.6914\
0.6043 & 0.3494 & 0.1438 & 0.6035 & 0.4111\
\end{array}\right]
$$
Can you write Python code for it?


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Realize a configuração da rede recém-criada, conforme instruções a seguir:
- Quantidade de épocas igual a 100000;

- Função de perda;

- Soma dos erros quadrados;

- Método de otimização: Gradiente Descendente Estocástico.
Can you write Python code for it?


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Redes Convolucionais
Nesta parte, você irá treinar modelos de redes neurais artificiais para classificar imagens. Em particular, dada uma imagem, seus modelos deverão indicar se a imagem contém um gato ou não. São fornecidos os conjuntos de imagens para
treinamento e para teste (arquivos train_catvnoncat.h5 e test_catvnoncat.h5). É também fornecido o código para a leitura dos arquivos correspondentes (veja listagem a seguir; veja também a seção de créditos no fim deste documento).
Can you write Python code for it?

import numpy as np
import h5py

def load_dataset():
    train_dataset = h5py.File('am-T3-dados/train_catvnoncat.h5', "r")
    train_set_x_orig = np.array(train_dataset["train_set_x"][:])
    train_set_y_orig = np.array(train_dataset["train_set_y"][:])

    test_dataset = h5py.File('am-T3-dados/test_catvnoncat.h5', "r")
    test_set_x_orig = np.array(test_dataset["test_set_x"][:])
    test_set_y_orig = np.array(test_dataset["test_set_y"][:])

    classes = np.array(test_dataset["list_classes"][:])
    train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))
    test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))

    return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes


I am working on the file "Relatorio-Aprendizado-Maquina-Trabalho-03.ipynb".
The first task is:
Treine a rede neural. Esse é o passo em que os pesos e os limiares da rede são ajustados. Esse passo deve ser realizado com a função fit da classe Model. Para chamar essa função, devemos passar a rede, a matriz de padrões e a matriz de alvos.
Nessa etapa vamos Treinar a rede após os passos descritos acima (item 3 e item 4), representando as etapas de Configuração e Treinamento da ${Rede}$ ${Neural}$
Can you write Python code for it?


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Ajustando os parâmetros da regressão linear
Nesta parte, use a função linearRegCostFunction para computar os valores ótimos para $\theta$, mas sem usar regularização, i.e., defina $\lambda$ = 0. Após isso, construa um gráfico para visualiar o modelo construído. Seu gráfico deve ser similar ao apresentado na Figura abaixo.
Can you write Python code for it?

def trainLinearReg(theta, X, y, Lambda):
    '''
    Função para mininzar a Função de Custo 'linearRegCostFunction' com Gradiente 'linearRegGradient'.
        theta = conjunto de pesos
        X = características (features)
        y = alvo (target)
        Lambda = Taxa de regularização
    '''

    M = minimize(linearRegCostFunction, theta, args=(X,y,Lambda), jac=linearRegGradient, options={'maxiter':5000})

    return M


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Caso 1:  Função de Minimização não convergiu com $\large \theta$ = array([[ 1.],[1.]])

Quando inicializamos o theta com valores 1, não houve sucesso na minimização.
Can you write Python code for it?

theta0 = np.ones((X_treino.shape[1],1))
fit = trainLinearReg(theta0, X_treino, y_treino, 0)
fit


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Caso 2: Função de Minimização somente convergiu com  $\large \theta$ = array([[15],[15]])

Quando inicializamos o theta com valores 15, houve sucesso na minimização.
Can you write Python code for it?

theta0 = np.array([[15],[15]])
fit = trainLinearReg(theta0, X_treino, y_treino, 0)
fit


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Esboço da fronteira de decisão
Nessa parte, você deve esboçar (plotar) a fronteira de decisão que foi aprendida
para separar os exemplos positivos dos negativos. Crie uma arquivo de nome
plotDecisionBoundary.py, para criar esse gráfico que traça o limite da decisão
não-linear. Seu gráfico deve ser semelhante ao apresentado na Figura abaixo.
Can you write Python code for it?

def plotDecisionBoundary(data, grad, LAMBDA):
    '''
    Função que deve esboçar um Gráfico de Dispersão com a sua Fronteira de Decisão.
    '''

    X = data[:,0:2]
    y = np.c_[data[:,2]]
    #theta = np.zeros(X_feature.shape[1])

    X_poli = PolynomialFeatures(6)
    X_fit = X_poli.fit_transform(X)
    #FMIN = minFunction(theta, 1, X_fit, y)

    # Plot decisionboundary
    X0_min, X0_max = X[:,0].min(), X[:,0].max()
    X1_min, X1_max = X[:,1].min(), X[:,1].max()
    X0, X1 = np.meshgrid(np.linspace(X0_min, X0_max), np.linspace(X1_min, X1_max))

    h = sigmoid(X_poli.fit_transform(np.c_[X0.ravel(), X1.ravel()]).dot(grad))
    #h = sigmoid((np.c_[X0.ravel(), X1.ravel()]).dot(grad))
    h = h.reshape(X0.shape)

    # Scatter plot of X,y and Get indexes for class 0 and class 1
    neg = data[:,2] == 0
    pos = data[:,2] == 1

    # If no specific axes object has been passed, get the current axes
    plt.figure(figsize=(10,5))
    plt.contour(X0, X1, h, [0.5], linewidths=3, colors='g', s=60, label='Decision boundary');
    plt.scatter(data[pos][:,0], data[pos][:,1], marker='+', c='k', s=60, linewidth=2, label='y=1')
    plt.scatter(data[neg][:,0], data[neg][:,1], marker='o', c='y', s=60, linewidth=2, label='y=0')
    plt.xlabel('Microchip Test 1')
    plt.ylabel('Microchip Test 2')
    plt.title('Lambda = '+str(LAMBDA))
    plt.legend(loc=1);


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Função de custo e gradiente
Agora, você deverá implementar o código para calcular a função de custo e
o gradiente para a regressão logística regularizada. Crie um arquivo de nome
costFunctionReg.py que contém uma função de nome costFunctionReg.py
e que computa o custo e o gradiente. Lembre-se de que a função de custo
regularizada na regressão logística é dada por:
$$J_{regularizado} = \small \underbrace{-\frac{1}{m} \sum\limits_{i = 1}^{m} \large{(}\small y^{(i)}\log\left(x^{(i)}\right) + (1-y^{(i)})\log\left(1- x^{(i)}\right) \large{)} }\text{Função de Custo} + \underbrace{\frac{\lambda}{2m} \sum\limits{j = 1}^{n}\ {\theta}{j}^{2} }\text{Fator Regularização} $$
Depois de concluir a implementação da função costFunctionReg, você deve
testar a corretude dela usando o valor inicial de ${\theta}$ (inicializado todo com zeros).
Você deve ver que o custo é de cerca de 0.693.
Porém, usando a função costFunctionReg, você agora deve computar os valores ótimos para ${\theta}$.
Can you write Python code for it?

def sigmoid(x):
    '''
    A função sigmoid
    '''
    g = np.array([x]).flatten()
    s = 1 / (1 + np.exp(-x))
    return s

def minFunction(theta, Lambda, X, y):
    '''
    Função de minimização sobrescrita  p/ atender à Função Objetivo
    'costFunctionReg' e com o a aplicação de gradiente 'gradientReg'
    '''

    M = minimize(costFunctionReg, theta, args=(Lambda, X, y), jac=gradientReg, options={'maxiter':3000})
    print('Custo de Minimização da Função: ', np.round(M.fun, 4))
    print('Custo da Função: ',M.status)
    print('Nº de Iterações necessárias: ',M.nit)
    print('Êxito de Otimização: ',M.message)
    return M

def costFunctionReg(theta, X, y, alpha):
    '''
    A versão do GD que acabamos de estudar é denominada *Batch gradienteient Descent*.
    Em cada iteração do algoritmo, todo o conjunto de treinamento é utilizado.

        X = características (features)
        y = alvo (target)
        Lambda = Taxa de regularização
    '''

    m = y.size
    h = sigmoid(X.dot(theta))
    Reg = (alpha/(2*m))*np.sum(np.square(theta[1:]))
    J = -1*(1/m)*(np.log(h).T.dot(y)+np.log(1-h).T.dot(1-y)) + Reg

    if np.isnan(J[0]): return(np.inf)

    return(J[0])

def gradientReg(theta, X, y, alpha):

    m = y.size
    h = sigmoid(X.dot(theta.reshape(-1,1)))
    Reg = (alpha/m)*np.r_[[[0]],theta[1:].reshape(-1,1)]
    grad = (1/m)*X.T.dot(h-y) + Reg

    return np.c_[grad]


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Gradiente na regressão linear regularizada
A derivada parcial do gradiente da função de custo da regressão linear regularizada é um vetor no qual o j-ésimo elemento.
No arquivo $linearRegCostFunction.py$, adicione código para calcular o gradiente. Quando você tiver completado essa implementação, teste a corretude usando $theta$ inicializado em $(1, 1)$. Você deve esperar ver um gradiente de $(-15.30, 598.250)$.
Can you write Python code for it?

theta0 = np.ones((X_treino.shape[1],1))
gradient = linearRegGradient(theta0, X_treino, y_treino, 0)
print(gradient)


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Gráfico Modelado com Regressão Linear Regularizada
Can you write Python code for it?

plt.figure(figsize=(10,5))
plt.plot(np.linspace(-50,40), (fit.x[0]+ (fit.x[1]*np.linspace(-50,40))), label='trainLinearReg')
plt.scatter(X_treino[:,1], y_treino, s=50, c='r', marker='x', linewidths=2, label='Water flow')
plt.xlabel('Change in water level (x)')
plt.ylabel('Water flowing out of the dam (y)')
plt.ylim(ymin=-5)
plt.xlim(xmin=-50)
plt.legend(loc=4);


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Gráfico de Dispersão para Visualização dos dados
Can you write Python code for it?

positive = ( data[:, 2] == 1 )
negative = ( data[:, 2] == 0 )

figura, grafico = plt.subplots(figsize=(10,5))
grafico.scatter(data[positive][:,0], data[positive][:,1], c='xkcd:black', marker='+', label='y=1')
grafico.scatter(data[negative][:,0], data[negative][:,1], c='xkcd:yellow', marker='o', label='y=0')
grafico.legend()
grafico.set_xlabel('Microchip Test1')
grafico.set_ylabel('Microchip Test2');


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Mapeamento de características (feature mapping)
Uma maneira de tornar os dados mais apropriados para a classificação é criar
mais características a partir das já existentes. Para isso, você deve criar uma
função mapFeature. Essa função deve ser implementada em um arquivo de
nome mapFeature.py, que irá mapear as características para todos os termos
polinomiais de x1 e x2, até a sexta potência. Como resultado desse mapeamento, nosso
vetor de duas características (os escores nos dois testes de QA) será transformado
em um vetor de 28 dimensões.
Um classificador que usa regressão logística treinado nesse vetor de características
de maior dimensão terá uma fronteira de decisão mais complexa e parecerá não-linear
quando desenhado em um gráfico bidimensional.
Embora o mapeamento de características nos permita construir um classificador mais expressivo,
também é mais suscetível a sobreajuste (overfitting). Desse modo, será implementada a
Regressão Logística Regularizada sobre os dados fornecidos e também verá como a regularização pode
ajudar a combater o problema do sobreajuste.
Can you write Python code for it?

def mapFeature(X1, X2):
    '''
    Função que mapeia características p/ os termos polinomiais X1 e X2 até a 6ª potência.
    Retorna um novo conjunto com mais características, através do algoritmo de mapping
    X1, X2, X1 ** 2, X2 ** 2, X1*X2, X1*X2 ** 2, etc...
    Os parâmetros X1, X2 devem ser do mesmo tamanho
    '''
    # Potência padrão para o mapeamento
    potencia = 6

    X1.shape = (X1.size, 1)
    X2.shape = (X2.size, 1)

    features = np.ones(shape=(X1.size, 1))

    for i in range(1, potencia + 1):
        for j in range(i + 1):
            r = (X1 ** (i - j)) * (X2 ** j)
            features = np.append(features, r, axis=1)

    return features


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Preparação para a $Tarefa$ $de$ $Aprendizado$
Can you write Python code for it?

# Obtendo os conjuntos de Teste, Treino e Validação para a Tarefa de Aprendizado
y = dataMatlab['y']
X = dataMatlab['X']

ytest = dataMatlab['ytest']
Xtest = dataMatlab['Xtest']

yval = dataMatlab['yval']
Xval = dataMatlab['Xval']

print('X:', X.shape)
print('y:', y.shape)
print('Xtest:', Xtest.shape)
print('ytest:', ytest.shape)
print('Xval:', Xval.shape)
print('yval:', yval.shape)


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Regressão Linear com Regularização
Na primeira metade desta parte, você implementará a regressão linear com regularização para prever a quantidade de água fluindo de uma barragem usando a mudança do nível da água em um reservatório. Na próxima metade, você realizará diagnósticos dos algoritmos de aprendizado por meio de depuração e irá examinar os efeitos de viés (bias) e da variância (variance).
Você irá usar o dataset fornecido no arquivo ex5data1.mat$^1$. Nesse arquivo há registros históricos na mudança no nível da água, x, e da quantidade de água que sai da barragem, y. Este conjunto de dados é dividido em três partes:
 - Um conjunto de treinamento que seu modelo aprenderá em: X, y
 - Um conjunto de validação cruzada para determinar o parâmetro de regularizacão: Xval, yval
 - Um conjunto de testes para avaliar o desempenho. Estes são exemplos que seu modelo não irá usar durante o treino: Xtest, ytest
Os nomes das variáveis contidas no arquivo ex5data1.mat são os seguintes: X, Xtest, Xval, y, ytest, yval. Você irá precisar desses nomes para carregar os dados do arquivo para usar em seus scripts em Python.

$^1$ Arquivos com a extensão .mat são normalmente criados no Octave ou no Matlab. Para
carregar esse arquivo no Python, você pode usar o procedimento descrito em http://www.
blogforbrains.com/blog/2014/9/6/loading-matlab-mat-data-in-python
Can you write Python code for it?

dataMatlab = loadmat('am-T2-dados/ex5data1.mat')
dataMatlab.keys()