## 1 Introduction
However, actors are often unstable and lack convergence guarantees, as their policy gradients do not specify a clear direction for policy updates. To address this, the critic typically evaluates the action values for a given state throughout the learning process. This approach focuses on computing values for individual state-action pairs rather than for all states, which enhances computational efficiency and may help reduce instability. Nevertheless, actor-critic methods still face instabilities that depend on the learning environment, even though they attempt to mitigate these issues through integration. In this paper, we propose the Extended Maximum Actor-Critic (EMAC) method to effectively control systems and optimize their behavior. The EMAC is designed with a structure in which the learning process of the conventional actor-critic framework is directed toward maximizing cumulative rewards. It has a policy that achieves the highest reward as optimal behavior and iteratively refines this policy through the repeated learning process. Furthermore, the proposed enhancements optimize learning by more efficiently utilizing the policy that achieves the highest reward within the environment. To achieve this, it integrates both the On-Policy and Off-Policy structures, allowing the actor to reliably acquire optimal reward values during the learning process and evaluate the performance of the policy effectively. In addition, the replay buffer has been extended to effectively utilize diverse experience data generated during the exploration process, enabling learning even under challenging conditions. To evaluate the effectiveness of the proposed EMAC framework and the improved structure that incorporated the replay buffer, we performed experiments in the following sequence. We also enhanced performance by optimizing behavior and maximizing rewards simultaneously in terms of structural improvements in the control system environment. The replay buffer structure was expanded to facilitate a more efficient learning process. Since applying both proposed methods simultaneously may not provide reliable metrics for performance evaluation, we tried to experiment separately using two distinct approaches. First, we used the Mujoco environment in Gym to evaluate the structural performance enhancements of the proposed EMAC using an actor-critic framework. This environment is widely used as a standard benchmark for evaluating the control performance of RL-based algorithms. This experiment was performed using environments that have been widely employed in previous works as a foundation for evaluation. Secondly, the extended replay buffer was specifically developed to optimize the use of experience data in multi-goal  
Figure 2: Advancements in Reinforcement Learning algorithms. domains. To ensure an accurate performance evaluation, experiments were conducted using the Fetch environment. The paper is as follows: Section 2 discusses related work, including ML techniques for robotic control, RL approaches for control systems, and the use of replay buffers. Sections 3 and 4 detail the proposed method, its architecture, and the experimental results. Lastly, the conclusion and directions for future work are discussed in the final section.