# Artificial Intelligence in Adaptive and Intelligent Educational System: A Review
## 2 Reinforcement Learning
### Q-Learning  
The quintessential reinforcement learning algorithm is Q-learning, a model-free, off-policy algorithm, which is also one of the most commonly used in many reinforcement-learning-related studies.  
The essence of Q-learning is derived from the Bellman equation [31], which is formulated as shown below.  
\[V_{\pi}(s)=\sum_{a}\pi(s,a)\sum_{s^{\prime}}p\left(s^{\prime}\mid s,a\right) \left(\mathbb{W}_{s\to s^{\prime}|a}+\gamma V_{\pi}\big{(}s^{\prime} \big{)}\right) \tag{3}\]  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**MDP Versions** & **System Form** & **State Characteristic** & **Reference** \\ \hline Fully Observable MDP (POMDP) & Discrete & All observable & [29] \\ Partially Observable MDP (POMDP) & Discrete & Partially observable & [16] \\ Semi-MDP(SMDP) & Continuous & All observable & [30] \\ \hline \hline \end{tabular}
\end{table}
Table 2: Different versions of [PERSON] decision processes.  
Figure 4: Model-based reinforcement learning algorithms.  
Figure 3: Model-free reinforcement learning algorithms.  
\[Q_{\pi}(s,a)= \sum_{s^{\prime}}p\left(s^{\prime}\mid s,a\right)\left(\delta\mathbb{W }_{s\to s^{\prime}\mid a}+\gamma\sum_{s^{\prime}}\pi\left(s^{\prime},a^{ \prime}\right)Q_{\pi}\left(s^{\prime},a^{\prime}\right)\right) \tag{4}\]  
In the Bellman equation,  
\[\mathbb{W}_{s\to s^{\prime}\mid a}=\mathbb{E}\big{[}r_{t+1}\mid s_{t}=s,a_{t} =a,s_{t+1}=s^{\prime}\big{]} \tag{5}\]  
V\({}_{\pi}\) (s) is the state value function, and Q\({}_{\pi}\) (\(s\), \(a\)) is the action value function. In the equation above, \(s\) represents the state, as the state \(s\) to the next state \(s+1\) is uncertain; hence, the expectation E needs to be added to the equation, and \(r\) represents the reward.  
For the Q-learning algorithm, the policy is selected based on the Q-table, which is structured as S*A, where S represents the state, and A represents the action. Based on the given Q-table, the next action can be determined based on the current state of the environment. After deciding which action to perform, the agent proceeds with the next action accordingly, and after performing the action, the agent receives a reward from the environment. After each action, the Q-table in the environment is updated, and the adjustment of the Q-table is carried out according to the following formula [32]:  
\[Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\alpha\bigg{[}r+\gamma\underset{a_{t} }{max}Q(s_{t+1},a_{t})-Q(s_{t},a_{t})\bigg{]} \tag{6}\]  
In this formula, \(s\) represents the state, a represents the action, \(r\) represents the reward, \(\alpha\) represents the learning rate, and \(\gamma\) represents the discount factor. \(\alpha\) and \(\gamma\) both range on a scale from 0 to 1.  
However, in 1993, [PERSON] and [PERSON] [33] suggested that, since random errors may uniformly occur in the action values, there may be a bias in the path of progressively seeking the optimal, consequently resulting in a sub-optimal solution rather than the optimal solution. Seventeen years later, [PERSON]'s paper [34] showed that environmental noise might also cause overestimation problems in classical Q-learning algorithms.  
In [PERSON]'s paper, a Double Q-learning algorithm was proposed to solve the overestimation problem, and later, [PERSON], [PERSON], and [PERSON] [12] also proposed an algorithm called Double DQN to solve this problem; the details of which are presented in the next section.