Nagoya University, Fall 2023

Lecture 10
Reinforcement Learning: Deep Q Network
This week Tutorial: Wednesday 13th Dec. 5th period
https://www.henrikbachmann.com/mml2023.html

Deep reinforcement learning: Milestones
Q-Learning(Chris Watkins, 1989)

Deep Learning(AlexNet, 2012)

Deep Q-Network(DeepMind, 2015)

Deep reinforcement learning: Milestones
Q-Learning(Chris Watkins, 1989)

Deep Learning(AlexNet, 2012)

Sven Behnke, CC BY-SA 4.0, via Wikimedia Commons

Deep Q-Network(DeepMind, 2015)

Deep Reinforcement Learning: Milestones
Q-Learning(Chris Watkins, 1989)

Deep Learning(AlexNet, 2012)https://github.com/Farama-Foundation/Gymnasium

Deep Q-Network(DeepMind, 2015)

Recall reinforcement learning: Basic idea
Action(move a chess piece, take a step, turn the wheel,…)Agent(the program,machine, robot, …)

Reward(win/lose the game, achieving a goal,
falling over, crashing against a wall, …)

Observation (new state)(new position, new chessboard state, …)

Environment(the opponent,another agent, thereal world, …)

Recall Q-learning

Recall Q-learning: Policy and Q function

Recall Q-Learning: Basic idea

actionstateQ tableAgentEnv

Observation (state)Recall Q-learning: Basic idea

actionstate
Q table

Agent

Env

Observation (state)Recall Q-learning: Basic idea

Actionactionstate
Q table

Agent

Env

Observation (state)Recall Q-learning: Basic idea

Actionactionstate
Q tableAgent

Env
RewardObservation (state)Recall Q-learning + epsilon-greedy

Recall: Convolutional neural network

Demo: https://tensorspace.org/html/playground/alexnet.html

Deep Q-network

Mnih, Volodymyr; et al. (2015)"Human-level control through deep reinforcement learning"
Nature. 518 (7540): 529–533https://github.com/Farama-Foundation/Gymnasium

Deep Q-networkhttps://www.youtube.com/watch?v=TmPfTpjtdgg

Deep Q-network: Q-learning limitation

Actionactionstate
Q tableAgent

Env
RewardShould be finite and small

Observation (state)

Deep Q-network: How many states?

https://github.com/Farama-Foundation/Gymnasium

Deep Q-network: Solution?

Actionactionstate
Q tableAgent

Env
RewardShould be finite and small

Observation (state)

Deep Q-network: Neural network

ActionAgent
Q(s,a)Can be very complex

Env
RewardObservation (state)

Deep Q-network: Architecture (DeepMind, 2015)210 x 160

Gray scale,max of 2frames,resolution--,skip frames,stack 4frames,
etc

Outputdense layer

84 x 84 x 432 filters,8x8,stride=4,relu

64 filters,
3x3,stride=1,relu

https://github.com/Farama-Foundation /Gymn asium

𝐿𝐿𝑖𝑖 𝜃𝜃𝑖𝑖 = 𝐸𝐸𝑠𝑠,𝑎𝑎,𝑟𝑟,𝑠𝑠𝑠

𝜕𝜕𝐿𝐿𝑖𝑖 (𝜃𝜃𝑖𝑖 )= 𝐸𝐸𝑠𝑠,𝑎𝑎,𝑟𝑟,𝑠𝑠′𝜕𝜕𝜃𝜃𝑖𝑖𝑟𝑟 + 𝛾𝛾 max 𝑄𝑄(𝑠𝑠 , 𝑎𝑎 ; 𝜃𝜃𝑖𝑖− ) − 𝑄𝑄 𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖
𝑎𝑎𝑎′′

𝑟𝑟 + 𝛾𝛾 max 𝑄𝑄(𝑠𝑠 ′ , 𝑎𝑎 ′ ; 𝜃𝜃𝑖𝑖− ) − 𝑄𝑄 𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖𝑎𝑎𝑎2

𝜕𝜕𝑄𝑄(𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖 )𝜕𝜕𝜃𝜃𝑖𝑖Dense

Conv layer

Conv layer

Conv layer

Preprocess

64 filters,4x4,stride=2,relu

512 neurons,relu

Deep Q-network: Problem• Replacing Q-table with a neural network was not revolutionary.

Let’s sketch our algorithm so far.
for each episode:while not done:batch = []for several frames:
reward, next_frame = env.step(action)
batch.append(current_frame, reward, action, next_frame)
QNN.train(batch)

Deep Q-network: Problem• Replacing Q-table with a neural network was not revolutionary.
• Being able to train the neural network stably and at scale was the revolutionary idea.

Deep Q-network: Problem• Replacing Q-table with a neural network was not revolutionary.
• Being able to train the neural network stably and at scale was the revolutionary idea.

Spot a problem…
for each episode:while not done:batch = []current_frame = env.reset()for several frames:
action = QNN.predict(current_frame)
reward, next_frame = env.step(action)
batch.append(current_frame, reward, action, next_frame)
current_frame = next_frameQNN.train(batch)

Deep Q-network: Problem• Replacing Q-table with a neural network was not revolutionary.
• Being able to train the neural network stably and at scale was the revolutionary idea.

Spot a problem…
for each episode:while not done:batch = []current_frame = env.reset()for several frames:
action = QNN.predict(current_frame)
reward, next_frame = env.step(action)
batch.append(current_frame, reward, action, next_frame)
current_frame = next_frame
QNN.train(batch)! Our QNN is trained with highly correlated batch.
! A neural network should be trained with training data that represents the actual data.

Deep Q-network: Experience Replay
Replay buffer: all actions from all episodes
stateAction

Next state Reward

Batch: small sample for QNN training.
Sampl
e

state

ActionNext state Reward

Deep Q-network: Experience Replay

replay_buffer = []for each episode:while not done:
current_frame = env.reset()for several frames:
action = QNN.predict(current_frame)
reward, next_frame = env.step(action)
replay_buffer.append(current_frame, reward, action, next_frame)
current_frame = next_framebatch = replay_buffer.sample(batch_size)
QNN.train(batch)

Deep Q-network: Chasing a moving target
The target value in the loss function depends on QNN’s weight and keep changing.
𝐿𝐿𝑖𝑖 𝜃𝜃𝑖𝑖 = 𝐸𝐸𝑠𝑠,𝑎𝑎,𝑟𝑟,𝑠𝑠𝑠

𝜕𝜕𝐿𝐿𝑖𝑖 (𝜃𝜃𝑖𝑖 )= 𝐸𝐸𝑠𝑠,𝑎𝑎,𝑟𝑟,𝑠𝑠′𝜕𝜕𝜃𝜃𝑖𝑖𝑟𝑟 + 𝛾𝛾 max 𝑄𝑄(𝑠𝑠 ′ , 𝑎𝑎 ′ ; 𝜃𝜃𝑖𝑖− ) − 𝑄𝑄 𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖𝑎𝑎𝑎

𝑟𝑟 + 𝛾𝛾 max 𝑄𝑄(𝑠𝑠 ′ , 𝑎𝑎 ′ ; 𝜃𝜃𝑖𝑖− ) − 𝑄𝑄 𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖𝑎𝑎𝑎2

𝜕𝜕𝑄𝑄(𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖 )𝜕𝜕𝜃𝜃𝑖𝑖Deep Q-network: Chasing a moving target
The target value in the loss function depends on QNN’s weight and keep changing.
𝐿𝐿𝑖𝑖 𝜃𝜃𝑖𝑖 ≈ 𝐸𝐸𝑠𝑠,𝑎𝑎,𝑟𝑟,𝑠𝑠𝑠

𝜕𝜕𝐿𝐿𝑖𝑖 (𝜃𝜃𝑖𝑖 )≈ 𝐸𝐸𝑠𝑠,𝑎𝑎,𝑟𝑟,𝑠𝑠′𝜕𝜕𝜃𝜃𝑖𝑖� ′ , 𝑎𝑎 ′ ; 𝜃𝜃𝑖𝑖− ) − 𝑄𝑄 𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖
𝑟𝑟 + 𝛾𝛾 max 𝑄𝑄(𝑠𝑠𝑎𝑎𝑎

� ′ , 𝑎𝑎 ′ ; 𝜃𝜃𝑖𝑖− ) − 𝑄𝑄 𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖
𝑟𝑟 + 𝛾𝛾 max 𝑄𝑄(𝑠𝑠𝑎𝑎𝑎

https://github.com/FaramaFoundation/Gymnasium

𝑄𝑄(𝑠𝑠, 𝑎𝑎; 𝜃𝜃)

2𝜕𝜕𝑄𝑄(𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖 )𝜕𝜕𝜃𝜃𝑖𝑖Deep Q-network: Chasing a moving target
The target value in the loss function depends on QNN’s weight and keep changing.
𝐿𝐿𝑖𝑖 𝜃𝜃𝑖𝑖 ≈ 𝐸𝐸𝑠𝑠,𝑎𝑎,𝑟𝑟,𝑠𝑠𝑠

𝜕𝜕𝐿𝐿𝑖𝑖 (𝜃𝜃𝑖𝑖 )≈ 𝐸𝐸𝑠𝑠,𝑎𝑎,𝑟𝑟,𝑠𝑠′𝜕𝜕𝜃𝜃𝑖𝑖� ′ , 𝑎𝑎 ′ ; 𝜃𝜃𝑖𝑖− ) − 𝑄𝑄 𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖
𝑟𝑟 + 𝛾𝛾 max 𝑄𝑄(𝑠𝑠𝑎𝑎𝑎

� ′ , 𝑎𝑎 ′ ; 𝜃𝜃𝑖𝑖− ) − 𝑄𝑄 𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖
𝑟𝑟 + 𝛾𝛾 max 𝑄𝑄(𝑠𝑠𝑎𝑎𝑎2

𝜕𝜕𝑄𝑄(𝑠𝑠, 𝑎𝑎; 𝜃𝜃𝑖𝑖 )𝜕𝜕𝜃𝜃𝑖𝑖https://github.com/FaramaFoundation/Gymnasium

https://github.com/FaramaFoundation/Gymnasium

𝑄𝑄(𝑠𝑠, 𝑎𝑎; 𝜃𝜃)� 𝑎𝑎; 𝜃𝜃 − )𝑄𝑄(𝑠𝑠,

Deep Q-network: Implementations
Implementations:• https://keras.io/examples/rl/deep_q_network_breakout/,• Lecture 10 Colab (trained on a non-Atari game).