## 2 System Model and Problem Formulation
### Proposed 6 DChOA-DRL
The algorithm dynamically updates the hyper-parameters as the optimization process continues, ensuring the agent learns efficiently while avoiding local optima. The interaction between the 6 DChOA and DRL ensures that the DRL agent not only learns to solve the task at hand but does so with the most optimal hyper-parameter configuration. By automating the tuning process, the 6 DChOA-DRL approach reduces the need for manual trial and error and accelerates convergence to an optimal or near-optimal solution. The 6 DChOA provides an intelligent search mechanism for hyper-parameter tuning, which is essential for enhancing the performance of the DRL framework, making the learning process faster, more stable, and ultimately more effective. The proposed DRL framework is designed to operate in a simulated environment where the agent interacts with the system to optimize the RIS phase shifts for enhanced energy harvesting and communication performance. The environment is defined by three key components, states, actions, and rewards, described as follows:  
* States: The state space provides the DRL agent with real-time information about the current system conditions, which includes the following:
* The received signal strength at the IoT device, representing the total power received from both the direct and RIS-assisted paths. * The SINR at the IoT receiver, which reflects the quality of the received signal after accounting for interference and noise. * The current RIS phase configurations, indicating the passive reflection coefficients \(\Psi=\text{diag}(\psi_{1},\psi_{2},\dots,\psi_{n})\) of the RIS elements. These states collectively allow the agent to evaluate the effectiveness of the current RIS configuration and determine necessary adjustments. * Actions: The action space is defined as the possible adjustments the DRL agent can make to the RIS phase shifts. At each time step, the agent selects a new configuration for the RIS by modifying the phase shifts of its N elements. Each phase shift \(\phi_{n}\in[-\pi,\ \pi]\) is dynamically updated to optimize the reflection and focus RF signals toward the desired directions. This action space ensures that the agent
can explore a wide range of configurations to identify the optimal settings for the given environment. 3. Rewards: The reward function incentivizes the DRL agent to achieve the dual objectives of maximizing the energy harvested by the IoT device and ensuring reliable communication at the IoT receiver. Specifically, rewards are granted as follows: Positive rewards are granted for the following: * Maximizing the harvested power at the IoT device, \(P_{h}=\delta(1-\alpha)P_{T}\), where \(P_{T}\) is the total received power, \(\alpha\) is the reflection coefficient, and \(\delta\) is the energy conversion efficiency. * Increasing the SINR at the IoT receiver, as shown in Equation (8). Penalties are applied for the following: * Configurations that fail to meet the energy harvesting constraint at the IoT device: \(\delta(1-\alpha)P_{T}\geq P_{b}\), where \(P_{b}\) is the minimum power threshold. * Excessive interference at the IoT receiver, leading to degraded SINR values below the desired threshold. It is worth mentioning that at the beginning of training, the agent primarily explored the action space, resulting in widely varying rewards due to suboptimal RIS phase configurations. As training progressed, the agent began to identify patterns in the environment and optimize its actions, leading to a gradual increase in the average reward values. Convergence was observed after approximately 120 episodes, where the reward stabilized, indicating that the agent had successfully learned an effective policy for configuring the RIS. The final rewards demonstrated consistent alignment with the dual objectives of the system: achieving high energy harvesting efficiency while maintaining robust communication performance.