## 1 Introduction  
In general, sensors are fundamental modules for device control, and their use in calculating paths and trajectories is a key approach to robotic control. Traditional control methods, such as Proportional-Integral-Derivative (PID) control and Model Predictive Control (MPC), are widely used; however, they can be challenging to model and susceptible to environmental changes, resulting in reduced performance in nonlinear systems. Recentresearch has explored various Machine Learning (ML) methods to achieve optimal behavior and overcome this limitation. In particular, Reinforcement Learning (RL) is one of the ML-based methods proposed to learn effective behavioral strategies by interaction with the environment, even if prior environmental knowledge is insufficient. Therefore, RL is actively being explored in research fields such as robot control systems [1; 2] autonomous driving [3; 4], and game theory [5; 6]. The goal of RL is to determine optimal policies that minimize the number of interactions an agent requires with the environment to achieve high rewards. To this end, the interaction of the RL-based agent with the environment can be processed by adopting the Markov Decision Process (MDP) framework [7]. The elements of an MDP for RL are defined as a tuple \(<\)\(S\), \(A\), \(P\), \(R\), \(\gamma\)\(>\), for a state (\(S\)), action (\(A\)), state transition probability (\(P\)), reward (\(R\)), and discount factor (\(\gamma\)). Once the initial state is set to the training environment, the agent performs the action defined in A that returns the following states \(S_{t+1}\) and \(R_{t+1}\) based on the state transition (\(P\)). In addition, the agent can generate a trajectory as it progresses by episode, learning a policy that maximizes cumulative rewards by computing the value at each time step and applying a discount factor. The learning process is schematically represented in Figure 1.  
The RL-based agent starts its search from state \(S_{0}\) to \(S_{5}\) and learns the policy for the optimal path that consistently leads to the reward at state \(S_{2}\). MDP based on RL can be divided into model-based and model-free approaches. In a model-based environment, agents need to engage in a learning process that explores it to develop optimal behavior that maximizes rewards, despite having no prior knowledge of the environment. Model-based RL, where agents learn environmental dynamics models, has limitations, as it is primarily applicable to specialized environments. Model-free RL has become increasingly useful in various domains and is generally categorized into value-based, policy-based, and actor-critic methods, based on differences in the learning process. RL is consistently developing diverse model-free approaches less affected by environmental limitations. Model-free exploration involves interacting with the environment and learning from experience that can operate effectively in diverse environments. The overall evolution of RL is shown in Figure 2. The figure describes model-free methods such as value-based, policy-based, and actor-critic [8]. In addition, RL needs to combine both value-based and policy-based features for optimal learning processes. Q-learning [9], which is a value-based RL, typically experiences exponential growth in the size of the Q-table as the environment increases in size. Deep Learning (DL) has been implemented to solve this challenge; however, it has still problems due to continuous and large-scale environments [10]. The policy-based RL has a policy gradient [11] to estimate policies, which we model and update as a probability distribution to proceed with the learning process. However, it has also limitations due to the instability  
Figure 1: [PERSON] decision process workflow overview. of policy updates. To overcome these problems, the actor-critic method has been developed. This approach typically adopts a state value function that updates the policy as the target and can be divided by the roles of the actor and the critic.