# Transfer Learning for Operator Selection: A Reinforcement Learning Approach
## 2 Background and Related Work
### Reinforcement Learning  
Reinforcement Learning (RL) is a type of machine learning technique for solving sequential decision-making problems. In this technique, a learning agent interacts with the environment to improve its performance through trial and error [9]. As with any other learning techniques, it is all about mapping situations to behaviours in order to optimise some rewards. However, unlike other machine learning techniques, the main challenge in RL is that the learning agent has to discover by itself the best action to take in a given situation. That is, in RL, the agent learns by itself without the intervention of a human. Dynamic programming is often used in this technique to find the optimum strategy to maximise reward in a given situation. The following are some key terms that describe the fundamental parts of an RL problem: Environment (\(E\)) -- the physical world in which the agent acts, States (\(S\)) -- the situations of the agent (what is the agent's current situation in a given state? ), Actions (\(A\))--the set of actions available to the agent, Reward (\(R:S\times A\mapsto\mathbb{R}\)) -- the feedback from the environment (good or bad), Policy (\(\Pi\)) -- a strategy to map the agent's state to actions (it is a strategy that an agent uses in pursuit of goals), and Value (\(V\)) -- the future reward that an agent will receive by taking an action in a particular state. The RL techniques can be implemented using various approaches, including Q-learning [9]. In this approach, the agent learns an optimal policy based on previous experience in the form of sample sequences of states, actions, and rewards. Therefore, each learning step consists of a state-transition tuple \((s_{i},a_{i},r_{i+1},s_{i+1})\), where \(s_{i}\in S\) is the current state of the agent, \(a_{i}\in A\) denotes the chosen action in the current state, \(r_{i+1}\in R\) specifies the immediate reward received after transitioning from the current state to the next state, and \(s_{i+1}\in S\) represents the next state. There are different ways we can formulate any problem in RL mathematically; one of them is Markov Decision Process (MDP). In many applications, it is assumed that the agent is unaware of anything in the environment. However, in some other applications, it can be assumed that not everything in the environment is unknown to the agent; for example, reward calculation is considered to be part of the environment even though the agent has some knowledge of how its reward is calculated as a function of its actions and states. An MDP can be represented as a tuple \((S,A,T,\gamma,R)\), where \(S,A\), and \(R\) are defined above, \(\gamma\in[0,1]\) is called the discount factor, and \(T:S\times A\times S\mapsto[0,1]\) is called the probabilistic transition relation such that for a given state \(s\) and an action \(a\), \(\sum_{s^{\prime}\in S}T(s,a,s^{\prime})=1\). The system being modelled is _Markovian_ if the result of an action does not depend on the previous actions and visited states (history) but only depends on the current state, i.e., \(P(s_{t+1}\mid s_{t},a_{t},s_{t-1},a_{t-1},\ldots)=P(s_{t+1}\mid s_{t},a_{t})= T(s_{t},a_{t},s_{t+1})\). This implies that the current state \(s\) gives enough information to the agent to make an optimal decision. That is, if the agent selects an action \(a\), the probability distribution over the next states is the same as the last time the agent tried this action in the same state. Once an MDP is defined, we can define policies, optimality criteria, and value functions to compute optimal policies. Solving a given MDP means computing an optimal policy. More detailed discussion on this can be found elsewhere [10]. The RL techniques have been successfully used to train robotic and/or software agents for a variety of purposes, including games, in a variety of situations ranging from simple to complex problems [11]. In particular, Deep RL has recently been developed and made available for dealing with and solving complex, dynamic, online, and real-time problems.