## 2 Foundational Concepts and Technologies
### Deep Reinforcement Learning (DRL)  
Deep reinforcement learning represents a fusion of deep learning and reinforcement learning (RL) principles within machine learning. As illustrated in Figure 11, DRL follows a cyclical interaction pattern where an agent engages with its environment through a continuous feedback loop [42]. In this process, the agent observes the current state, takes an action at based on that state, and receives both a reward and transitions to a new state from the environment. The agent's goal is to learn an optimal policy that maximizes cumulative rewards through these repeated interactions. Unlike traditional machine learning approaches, DRL enables agents to learn complex decision-making strategies through direct environmental interaction, making it particularly well suited for sequential decision tasks. Deep learning, utilizing neural networks with several layers [42], enhances RL by effectively managing complex state and action spaces. Prominent DRL approaches include Deep Q-Networks (DQN), which leverage neural networks to approximate Q-values (the expected rewards associated with specific actions), and Policy Gradient methods, which focus on directly optimizing the policy [38]. DRL employs techniques such as experience replay, where previously encountered experiences are stored and reused to improve training efficiency, and target networks, which ensure training stability by maintaining consistent target values. At the core of DRL is the concept of reinforcement learning, a type of machine learning where an agent learns to make decisions by performing actions in an environment to maximize some notion of cumulative reward. The agent observes the state of the environment, takes actions, and receives feedback in the form of rewards or penalties. Over time, the agent learns to take actions that maximize the total reward, which is formally defined as the expected sum of discounted future rewards:  
\[G_{t}=R_{t+1}+\gamma R_{t+2}+\gamma^{2}R_{t+3}+\dots=\sum_{k=0}^{\infty}\gamma ^{k}R_{t+k+1} \tag{1}\]  
where \(R_{t}\) is the reward at time step \(t\) and \(\gamma\) is the discount factor that determines the present value of future rewards. One of the seminal works in DRL is the development of the Deep Q-Network (DQN) [42]. The DQN algorithm combines Q-learning, a popular RL algorithm, with deep neural networks. Figure 11: The agent–environment interaction in the Markov Decision Process (MPD) for ranking information with reinforcement learning, reprinted from Ref. [43]. In Q-learning, the agent learns a Q-function, which estimates the expected cumulative reward for taking a given action in a given state and following the optimal policy thereafter. The Q-function is updated using the Bellman equation:  
\[Q(s_{t},a_{t})=R_{t+1}+\gamma\max_{a^{\prime}}Q(s_{t+1},a^{\prime}) \tag{2}\]  
where \(Q(s_{t},a_{t})\) is the action-value function, representing the expected return (reward) of taking action \(a_{t}\) in state \(s_{t}\). \(R_{t+1}\) is the reward received after taking action \(a_{t}\) at time step \(t\). \(\gamma\) is the discount factor, which determines the present value of future rewards. \(\max_{a^{\prime}}Q(s_{t+1},a^{\prime})\) represents the maximum expected future reward, starting from the next state \(s_{t+1}\) and considering all possible actions \(a^{\prime}\). Traditional Q-learning stores this Q-function in a table, which becomes impractical for large state spaces. Instead, DQN uses a neural network to approximate the Q-function, where the Q-values for all possible actions are output for a given state. The training of a DQN involves two main components: the neural network and an experience replay buffer [44]. The neural network takes the state as input and outputs Q-values for all possible actions. The experience replay buffer stores the agent's experiences, which are tuples of the form (state \(s_{t}\), action \(a_{t}\), reward \(r_{t+1}\), next state \(s_{t+1}\)). During training, random batches of experiences are sampled from the replay buffer to update the neural network.