[major=AI | course_id=REL301m | syllabus_id=11256 | semester=8 | combo=core | title=Reinforcement Learning]
Course ID: REL301m
Title EN: Reinforcement Learning
Title VI: Học tăng cường
Credits: 3.0
Prerequisites: AIL303m, DPL302m
Description: This course helps students understand the concepts of Reinforcement Learning including the space of RL algorithms (Temporal- Difference learning, Monte Carlo, Sarsa, Q-learning, Policy Gradients, Dyna, and more). Students will understand how to formalize these tasks as a Reinforcement Learning problem. Students will learn how RL fits under the broader umbrella of machine learning, and how it complements deep learning, supervised and unsupervised learning. Students can implement a complete RL solution and understand how to apply AI tools to solve real-world problems.
Student tasks: '- Students must attend at least 80% of contact slots in order to be accepted to the final examination.
- Student is responsible to do all exercises, assignments and labs given by instructor in class or at home and submit on time
- Use laptop in class only for learning purpose
- Promptly access to the FU FLM at https://flm.fpt.edu.vn/ for up-to-date course information
Tools: Jupyter Notebooks, google Colab,
Course type: coursera_hybrid
Decision: 358/QĐ-ĐHFPT dated 04/03/2024
Approved date: 2024-04-03
CLO1: Understand the concepts of Reinforcement Learning
[1] category=Assignment; type=on-going; weight=30.0; duration=2 weeks; question_type=CLO1, CLO2, CLO3, CLO4, CLO5, CLO6, CLO7; grading_guide=- cover chapter 1, 2, 3; completion_criteria=; clos=>0
[2] category=Lab1; type=on-going; weight=10.0; duration=225'; question_type=CLO1, CLO2; grading_guide=- cover chapter 1; completion_criteria=; clos=>0
[3] category=lab2; type=on-going; weight=10.0; duration=225'; question_type=CLO3, CLO4; grading_guide=- cover chapter 2; completion_criteria=; clos=>0
[4] category=lab3; type=on-going; weight=10.0; duration=225'; question_type=CLO5, CLO6; grading_guide=- cover chapter 3; completion_criteria=; clos=>0
[5] category=Progress test; type=on-going; weight=10.0; duration=45'; question_type=CLO1, CLO2, CLO3, CLO4, CLO5, CLO6; grading_guide=- cover chapter 1,2,3; completion_criteria=; clos=>0
[6] category=Final exam; type=Final exam; weight=30.0; duration=60'; question_type=CLO1, CLO2, CLO3, CLO4, CLO5, CLO6, CLO7; grading_guide=- cover chapter 1, 2, 3; completion_criteria=; clos=4
[1] desc=Fundamentals of Reinforcement Learning-; author=University of Alberta; publisher=Coursera; edition=; isbn=; main=True; online=True; note=https://www.coursera.org/learn/fundamentals-of-reinforcement-learning
[2] desc=Sample-based Learning Methods; author=University of Alberta; publisher=Coursera; edition=; isbn=; main=True; online=True; note=https://www.coursera.org/learn/sample-based-learning-methods
[3] desc=Prediction and Control with Function Approximation; author=University of Alberta; publisher=Coursera; edition=; isbn=; main=True; online=True; note=https://www.coursera.org/learn/prediction-control-function-approximation
[4] desc=A Complete Reinforcement Learning System (Capstone); author=University of Alberta; publisher=Coursera; edition=; isbn=; main=True; online=True; note=https://www.coursera.org/learn/complete-reinforcement-learning-system
[5] desc=Guided Project
Create a Tic-Tac-Toe game in Python; author=Coursera Project Network; publisher=Coursera; edition=; isbn=; main=; online=True; note=https://www.coursera.org/projects/tic-tac-toe-game-python
Session 1: Course Introduction
Topic: Offline; Teaching type: CLO1; LO: T
ITU: - slides: Introduction
Materials: mat1
Download: Read slide introduction
Syllabus REL301m

Session 2: 1.1. The K-Armed Bandit Problem
Topic: Offline; Teaching type: CLO1; LO: T
ITU: - slides: chapter 1.1
Materials: mat2
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.1

Session 3: 1.2. Estimating Action Values
Topic: Offline; Teaching type: CLO1; LO: T
ITU: - slides: chapter 1.2
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.2

Session 4: 1.3 Exploration vs. Exploitation Tradeoff
Topic: Offline; Teaching type: CLO1; LO: T
ITU: - slide chapter 1.3
Download: https://flm.fpt.edu.vn/gui/role/student/Read chapter 1.3
CLO2: Know how to implement dynamic programming as an efficient solution approach to an industrial control problem
Session 1: 1.4 Introduction to Markov Decision Processes
Topic: Offline; Teaching type: CLO1; LO: T
ITU: - slide chapter 1.4
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.4

Session 2: 1.5 Goal of Reinforcement Learning
Topic: Offline; Teaching type: CLO1; LO: T
ITU: - slide chapter 1.5
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.5

Session 3: 1.6 Continuing Tasks
Topic: Offline; Teaching type: CLO2; LO: T
ITU: - slide chapter 1.6
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.6

Session 4: 1.7 Policies and Value Functions
Topic: Offline; Teaching type: CLO2; LO: T
ITU: '- slide chapter 1.7
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.7
CLO3: Understand several algorithms for estimating value: Temporal-Difference learning, Monte Carlo
Session 1: 1.8 Bellman Equations
Topic: Offline; Teaching type: CLO2; LO: T
ITU: '- slide chapter 1.8
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.8

Session 2: 1.9 Optimality (Optimal Policies & Value Functions)
Topic: Offline; Teaching type: CLO2; LO: T
ITU: '- slide chapter 1.9
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.9

Session 3: 1.10 Policy Evaluation (Prediction)
Topic: Offline; Teaching type: CLO2; LO: T
ITU: '- slide chapter 1.10
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.10

Session 4: 1.11 Policy Iteration (Control)
Topic: Offline; Teaching type: CLO2; LO: T
ITU: '- slide chapter 1.11
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.11
CLO4: Undersatnd and implement a model-based approach to RL: Temporal-Difference, Expected Sarsa, Q-learning
Session 1: 1.12 Generalized Policy Iteration
Topic: Offline; Teaching type: CLO2; LO: T
ITU: '- slide chapter 1.12
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 1.12

Session 2: Lab 1
Topic: Offline; Teaching type: CLO1, CLO2; LO: U
ITU: - slide lab 1
Download: Read Lab 1

Session 3: Lab 1
Topic: Offline; Teaching type: CLO1, CLO2; LO: U
ITU: - slide lab 1
Download: Read Lab 1

Session 4: Lab 1
Topic: Offline; Teaching type: CLO1, CLO2; LO: U
ITU: - slide lab 1
Download: Read Lab 1
CLO5: Understand several approaches that affect approximate value functions: supervised learning, fixed basis, neural network.
Session 1: Lab 1
Topic: Offline; Teaching type: CLO1, CLO2; LO: U
ITU: - slide lab 1
Download: Read Lab 1

Session 2: 4.1 Assignment introduction
Topic: Offline; Teaching type: CLO7; LO: 
ITU: - slide chapter 4.1
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 4.1

Session 3: 2.1 Introduction to Monte-Carlo Methods
Topic: Offline; Teaching type: CLO3; LO: T
ITU: - slide chapter 2.1
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.1

Session 4: 2.2 Monte-Carlo for Control
Topic: Offline; Teaching type: CLO3; LO: T
ITU: - slide chapter 2.2
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.2
CLO6: Understand how function approximation is used to predict and control in Reinforcement Learning
Session 1: 2.3 Exploration Methods for Monte-Carlo
Topic: Offline; Teaching type: CLO3; LO: T
ITU: - slide chapter 2.3
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.3

Session 2: 2.4 Off-policy learning for prediction
Topic: Offline; Teaching type: CLO3; LO: T
ITU: - slide chapter 2.4
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.4

Session 3: 2.5 Introduction to Temporal Difference Learning
Topic: Offline; Teaching type: CLO3; LO: T
ITU: - slide chapter 2.5
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.5

Session 4: 2.6 Advantages of Temporal Difference
Topic: Offline; Teaching type: CLO3; LO: T
ITU: - slide chapter 2.6
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.6
CLO7: Understand and implement an RL solution to a problem step by step: problem formulation, appropriate algorithm selection, implementation, and empirical study into the effectiveness of the solution.
Session 1: 2.7 Temporal Difference for Control
Topic: Offline; Teaching type: CLO4; LO: T
ITU: - slide chapter 2.7
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.7

Session 2: 2.8 Off-policy Temporal Difference Control: Q-learning
Topic: Offline; Teaching type: CLO4; LO: T
ITU: - slide chapter 2.8
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.8

Session 3: 2.9 Expected Sarsa
Topic: Offline; Teaching type: CLO4; LO: T
ITU: - slide chapter 2.9
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.9

Session 4: 2.10 Define model in Reinforcement Learning
Topic: Offline; Teaching type: CLO4; LO: T
ITU: - slide chapter 2.10
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.10
Session 1: 2.11 Define Planning in Reinforcement Learning
Topic: Offline; Teaching type: CLO4; LO: T
ITU: - slide chapter 2.11
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.11

Session 2: 2.12 Dyna as a formalism for planning
Topic: Offline; Teaching type: CLO4; LO: T
ITU: - slide chapter 2.12
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.12

Session 3: 2.13 Dealing with inaccurate models
Topic: Offline; Teaching type: CLO4; LO: T
ITU: - slide chapter 2.13
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 2.13

Session 4: Lab 2
Topic: Offline; Teaching type: CLO3, CLO4; LO: U
ITU: - slide chapter 2, lab 2
Download: Read chapter 2, lab 2
Session 1: Lab 2
Topic: Offline; Teaching type: CLO3, CLO4; LO: U
ITU: - slide chapter 2, lab 2
Download: Read chapter 2, lab 2

Session 2: Lab 2
Topic: Offline; Teaching type: CLO3, CLO4; LO: U
ITU: - slide chapter 2, lab 2
Download: Read chapter 2, lab 2

Session 3: Lab 2
Topic: Offline; Teaching type: CLO3, CLO4; LO: U
ITU: - slide chapter 2, lab 2
Download: Read chapter 2, lab 2

Session 4: 4.2 Assignment guide 1
Topic: Offline; Teaching type: CLO7; LO: 
ITU: - slide chapter 4.2
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 4.2
Session 1: 3.1 Estimating Value Functions as Supervised Learning
Topic: Offline; Teaching type: CLO5; LO: T
ITU: - slide chapter 3.1
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.1

Session 2: 3.2 The Objective for On-policy Prediction
Topic: Offline; Teaching type: CLO5; LO: T
ITU: - slide chapter 3.2
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.2

Session 3: 3.3 The Objective for Temporal Difference
Topic: Offline; Teaching type: CLO5; LO: T
ITU: - slide chapter 3.3
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.3

Session 4: 3.4 Linear Temporal Difference
Topic: Offline; Teaching type: CLO5; LO: T
ITU: - slide chapter 3.4
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.4
Session 1: 3.5 Feature Construction for Linear Methods
Topic: Offline; Teaching type: CLO5; LO: T
ITU: - slide chapter 3.5
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.5

Session 2: 3.6 Episodic Sarsa with Function Approximation
Topic: Offline; Teaching type: CLO6; LO: T
ITU: - slide chapter 3.6
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.6

Session 3: 3.7 Exploration under Function Approximation
Topic: Offline; Teaching type: CLO6; LO: T
ITU: - slide chapter 3.7
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.7

Session 4: 3.8 Understand Average Reward
Topic: Offline; Teaching type: CLO6; LO: T
ITU: - slide chapter 3.8
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.8
Session 1: 3.9 Learning Parameterized Policies
Topic: Offline; Teaching type: CLO6; LO: T
ITU: - slide chapter 3.9
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.9

Session 2: 3.10 Policy Gradient for Continuing Tasks
Topic: Offline; Teaching type: CLO6; LO: T
ITU: - slide chapter 3.10
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.10

Session 3: 3.11 Actor-Critic for Continuing Tasks
Topic: Offline; Teaching type: CLO6; LO: T
ITU: - slide chapter 3.11
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.11

Session 4: 3.12 Policy Parameterizations
Topic: Offline; Teaching type: CLO6; LO: T
ITU: - slide chapter 3.12
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 3.12
Session 1: Lab 3
Topic: Offline; Teaching type: CLO5, CLO6; LO: U
ITU: - slide chapter 3, lab 3
Download: Read chapter 3, lab 3

Session 2: Lab 3
Topic: Offline; Teaching type: CLO5, CLO6; LO: U
ITU: - slide chapter 3, lab 3
Download: Read chapter 3, lab 3

Session 3: Lab 3
Topic: Offline; Teaching type: CLO5, CLO6; LO: U
ITU: - slide chapter 3, lab 3
Download: Read chapter 3, lab 3

Session 4: Lab 3
Topic: Offline; Teaching type: CLO5, CLO6; LO: U
ITU: - slide chapter 3, lab 3
Download: Read chapter 3, lab 3
Session 1: 4.3 Assignment guide 2
Topic: Offline; Teaching type: CLO7; LO: 
ITU: - slide chapter 4.3
Download: https://flm.fpt.edu.vn/gui/role/student/Read Chapter 4.3

Session 2: Guided Project: Create a Tic-Tac-Toe game in Python
Topic: Offline; Teaching type: CLO1, CLO2, CLO3, CLO4, CLO5, CLO6; LO: U
ITU: Link in the Materials: https://www.coursera.org/projects/tic-tac-toe-game-python
Download: Do Guided Project

Session 3: Guided Project: Create a Tic-Tac-Toe game in Python
Topic: Offline; Teaching type: CLO1, CLO2, CLO3, CLO4, CLO5, CLO6; LO: U
ITU: Link in the Materials: https://www.coursera.org/projects/tic-tac-toe-game-python
Download: Do Guided Project

Session 4: Quiz
Topic: Offline; Teaching type: CLO1, CLO2, CLO3, CLO4, CLO5, CLO6; LO: T
ITU: - slide chapter 1, 2,3
Download: - slide chapter 1, 2,3
Session 1: Assignment- Assignment grading
Topic: Offline; Teaching type: CLO1, CLO2, CLO3, CLO4, CLO5, CLO6, CLO7; LO: U
ITU: - slide chapter 1, 2, 3
Download: - Review chapter 1, 2, 3
- Do assignment

Session 2: Assignment- Assignment grading
Topic: Offline; Teaching type: CLO1, CLO2, CLO3, CLO4, CLO5, CLO6, CLO7; LO: U
ITU: - slide chapter 1, 2, 3
Download: - Review chapter 1, 2, 3
- Do assignment

Session 3: Assignment- Assignment grading
Topic: Offline; Teaching type: CLO1, CLO2, CLO3, CLO4, CLO5, CLO6, CLO7; LO: U
ITU: - slide chapter 1, 2, 3
Download: - Review chapter 1, 2, 3
- Do assignment

Session 4: Review course
Results
Topic: Offline; Teaching type: CLO1, CLO2, CLO3, CLO4, CLO5, CLO6, CLO7; LO: U
ITU: Syllabus REL301m
Slide chapter 1,2,3
Download: Syllabus REL301m
Slide chapter 1,2,3