# Deep Reinforcement Learning for Selection of Dispatch Rules for Scheduling of Production Systems
## 3 Method
### Reinforcement Learning Approach
#### 3.2.1 Q-Learning and Deep Q-Network  
Q-learning is a fundamental reinforcement learning algorithm in the field of AI and ML, used to solve problems where an agent interacts with an environment to learn an optimal policy for sequential decision-making tasks. In reinforcement learning, an agent learns to make a sequence of decisions (actions) in an environment to maximize a cumulative reward. Q-learning is typically applied to problems modeled as MDPs, where the agent interacts with an environment composed of states, actions, transition probabilities and rewards. MDPs have the following components. * State space (S): A set of all possible states that the agent can be in, where \(s\) represents the current situation in the environment. Action space (\(A\)): A set of all possible actions that the agent can take, where \(a\) denotes the decision that the agent makes in a given state \(s\). * Transition function (\(P\)): This describes the probability of transitioning from one state to another after taking a specific action. * Reward function (\(R\)): This defines the immediate reward that the agent receives for taking an action in a particular state. * Q-value (action value) function: This represents the expected cumulative reward that an agent can obtain by taking action \(a\) in state \(s\) and following an optimal policy thereafter. The key insight behind Q-learning is the Bellman equation, which expresses the relationship between the Q-values of the current and future states:  
\[Q(s,a)=R(s,a)+\gamma*\max\bigl{[}Q(s^{\prime},a^{\prime})\bigr{]} \tag{1}\]  
where \(Q(s,\,a)\) is the Q-value for state-action pair (\(s,\,a\)), \(R(s,\,a)\) is the immediate reward when taking action \(a\) in state \(s\), \(\gamma\) (gamma) is the discount factor that represents the agent's preference for immediate rewards over future rewards and \(max\bigl{[}Q(s^{\prime},a^{\prime})\bigr{]}\) represents the maximum Q-value over all possible actions in the next state, denoted as \(s^{\prime}\). The idea of the Q-learning is to use the Bellman equation as an iterative update. The agent, at each decision point t in a state \(s_{t}\in S\), selects an action \(a_{t}\in A\) according to a policy \(\pi\). After the agent takes the action \(a_{t}\), it enters a new state \(s_{t+1}\) with the transition probability \(p(s_{t+1}|s_{t},a_{t})\in P(S\times A\to S)\) and reward \(r_{t}\in R\). The agent's objective is to find the optimal policy \(\pi*\) that maximizes the expected sum of rewards. A policy is a rule that an agent follows when selecting an action by considering the given state that the agent is in. The model consists of an agent, a state space \(S\) and a set of actions for each state \(A\) (action space). All the information is stored in a Q-table, which consists of the Q-values between different states. The goal of the agent is to maximize the sum of reward \(r_{t}\) discounted by \(\gamma\) at each time step \(t\), based on \(\pi=P(a|s)\), after making an observation \(s\) and taking action \(\alpha\). \[Q^{\pi}(s_{t},a_{t})=\max_{\pi}\mathbb{E}\Bigl{[}r_{t+1}+\gamma r_{t+2}+\gamma ^{2}r_{t+3}+\ldots\Bigl{|}s_{t}=s,a_{t}=a,\pi\Bigr{]} \tag{2}\]  
where \(\alpha\) is the learning \(0<\alpha\leq 1\) and \(\gamma\in(0,1]\). In a further analysis, when \(\gamma\to 0\), this means that there are near-term goals, but when \(\gamma\to 1\), this means that there are long-term rewards. Based on the Bellman optimality equation, the standard Q-learning algorithm can be derived. However, Q-learning has some limitations when the environment becomes more complicated. There are problems with the storage capacity when the state space is large. To solve this issue, neural networks are used and we implement the Q-learning method. In order to overcome the limitations mentioned in the previous section regarding Q-learning's inability to deal with unknown states, the concept of the DQN is used. The DQN is an RL algorithm that combines Q-learning and DNNs to approximate the Q-values for high-dimensional state spaces.