## 3 Materials and Methods
### Extended Maximum Actor-Critic via Policy-Gradient Theorem  
In this paper, we propose the EMAC to build on the Policy-Gradient framework. The proposed approach improves stability and performance by using the maximized actor parameters, \(\theta^{max}\), obtained by achieving maximum reward, to enable a more efficient learning process. Equation (1) represents the parameters \(\theta\) of the Actor, which defines the objective function of the actor-critic framework. In the equation, \(\pi(a|s_{t};\theta)\) denotes the policy generated by the actor, representing the outcome of executing action \(a_{t}\) in state space \(s_{t}\). The critic evaluates the policy's value using \(Q_{v}(s_{t},a_{t})\), and the actor's policy parameters \(\theta\) are updated in proportion to the learning rate \(\alpha\). \[\theta\leftarrow\alpha\bigtriangledown_{\theta}\log(\pi_{\theta}(a_{t}|s_{t} ))(Q_{v}(s_{t},a_{t})) \tag{1}\]  
In general, the goal is to facilitate an efficient learning process and explore optimal actions by reusing \(\theta^{max}\), which was previously utilized to generate the maximum policy in the standard actor-critic framework. However, as shown in the equation, the value of \(Q_{v}(s_{t},a_{t})\) continues to increase as training progresses over an extended period. Therefore, an effective solution is needed to address and mitigate this issue. Therefore, to address these limitations, we attempted structural improvements based on the approach proposed in TD3 [29]. In the traditional approach, two critic networks are used to calculate \(Q_{v}\). To mitigate overestimation, we implemented an underestimation strategy by selecting the lower value generated by the two critic networks. In the proposed EMAC framework, the On-Policy approach is employed to generate the optimal policy. As On-Policy methods use the same policy for both decision-making and learning, they are susceptible to overestimation. To address this, the existing structure was preserved as much as possible. Moreover, the TD error (temporal difference error), which is used to update the critic network, is determined by the difference between the current and estimated values. In cases where the maximum reward is achieved, the learning process follows the structure defined in Equation (2). As shown in the equation, the TD error \(y\) is updated using the reward (\(r\)) and the discount factor (\(\gamma\)), with the minimum value selected from the two critic networks. By maintaining this structure, we were able to address the issues of over-fitting and overestimation that can occur when utilizing the maximum policy. \[y\leftarrow \begin{cases}r+\gamma\min_{i=1,2}Q_{\theta_{i}}(s^{\prime},\vec{a})& \text{if not Max Reward}\\ r+\gamma\min_{i=1,2}Q_{\theta^{max}_{i}}(s^{\prime},\vec{a})&\text{if Max Reward} \end{cases}  \tag{2}\]  
Figure 3: RL-based device sensor control process. In the update process, the next action \(\overline{\pi}\) is derived from the max actor, which generates the action based on the point at which the maximum reward is achieved. To address the issue of policy overestimation, we adopted a method that introduces noise using the structure \(\text{clip}(\text{Noise}(0,\overline{\sigma}),-c,c)\). This approach expands the policy distribution while reducing the variance, effectively addressing the overestimation issue. This method applies Gaussian noise (\(\epsilon\)) to the policy \(\pi_{\phi}\), effectively performing policy smoothing regularization. In EMAC, the parameter values were kept consistent with those of previous studies, with \(\overline{\sigma}=0.2\) and \(c=0.5\), to ensure consistent results in performance analysis. Mathematically, this can be represented by the structure shown in Equation (3). \[\overline{a}\leftarrow \begin{cases}\pi_{\phi}(s^{\prime})+\epsilon,&\epsilon \sim\text{clip}(\text{Noise}(0,\overline{\sigma}),-c,c)&\text{if not Max Reward}\\ \pi_{\phi^{max}}(s^{\prime})+\epsilon,&\epsilon\sim\text{clip}(\text{Noise}(0, \overline{\sigma}),-c,c)&\text{if Max Reward}\end{cases}