## 6 Simulation  
In this segment, we showcase the simulation outcomes of our novel optimization algorithm, addressing joint RIS-RB selection and resource allocation for a multi-RIS-assisted MANET. At the outset, we compared the performance of the U-DCB algorithm with that of the conventional MAB approach. Subsequently, we conducted a comparative analysis involving the TD3 algorithm and two alternative reinforcement learning techniques: Q-learning and the deep Q-network (DQN). In this simulation scenario, we configured the number of reflecting intelligent surfaces (RIS) and resource blocks (RB) as (10, 20), respectively, with 10 transmitters (Tx) and 10 receivers (Rx) randomly positioned in a 1000 m \(\times\) 1000 m map. The channel matrices, \(\mathbf{H}_{BR}\) and \(\mathbf{H}_{RR}\), adhered to a dynamic Rayleigh distribution. Each device-to-device (D2D) user pair was assigned one RB and one RIS, and both the RB and RIS were potentially shared among multiple D2D pairs. To ensure sufficient resources, the number of RB and RIS was set equal to or greater than the number of D2D user pairs. The experience replay buffer had a capacity of 1,000,000. The spatial distribution of RIS and D2D pairs was randomized within the cell, and additional parameters are detailed in Table 1. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Number of D2D pairs & 10 \\ \hline Number of RIS & (10, 20) \\ \hline Number of RB & (10, 20) \\ \hline Tx transmission power & 20 dBm \\ \hline Rx hardware cost power & 10 dBm \\ \hline RIS hardware cost power & 10 dBm \\ \hline Path loss in reference distance (1 m) & \(-\)30 dBm \\ \hline Target SINR threshold & 20 dBm \\ \hline Power of noise & \(-\)80 dBm \\ \hline D-UCB time steps & 500 \\ \hline D-UCB exploration parameter C & 2 \\ \hline TD3 time steps & 1000 \\ \hline Reward discount factor \(\gamma\) & 0.99 \\ \hline Network update learning rate \(\tau\) & 0.005 \\ \hline Target network update frequency \(T_{d}\) & 2 \\ \hline Policy noise clip \(c\) & 0.5 \\ \hline Max replay buffer size & 100,000 \\ \hline Batch size & 256 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters. The efficacy of the inner-outer actor-critic-based reinforcement learning (RL) algorithm is demonstrated through the following performance metrics. _(1) RIS selection_  
In Figure 4, the D-UCB algorithm demonstrates its effectiveness as agents dynamically choose the most suitable RIS over time. This strategic selection significantly contributes to enhancing the overall quality of the RIS-assisted wireless ad-hoc network. The adaptability of the online learning algorithm proves invaluable in efficiently capturing temporal variations in the wireless environment. Through dynamic RIS selection, the algorithm effectively ensures the continual maintenance of high-quality network performance. _(2) Regret of D-UCB algorithm vs. MAB algorithm with different number of arms_  
In Figure 5, a comparison of the network regret under various scenarios and methodologies is presented. The observed trend indicates that the control policy performs admirably, with the regret converging as training steps increase. Notably, the D-UCB algorithm outperforms the conventional multi-armed bandit (MAB) algorithm, showcasing its superior properties in optimizing the system performance over time. Figure 4: (**a**) RIS selection of agent 1. (**b**) RIS selection of agent 2. (**c**) RIS selection of agent 3. (**d**) RIS selection of agent 4. To conduct the hypothesis testing in this simulation, we focus on the comparison between the D-UCB algorithm and the conventional multi-armed bandit (MAB) algorithm in terms of network regret. The null hypothesis (H0) is that there is no significant difference between the two algorithms, while the alternative hypothesis (H1) is that the D-UCB algorithm outperforms the MAB algorithm. Below is the formulation.