# Channel-Hopping Using Reinforcement Learning for Rendezvous in Asymmetric Cognitive Radio Networks
## 4 Simulation Results  
### Actor and Critic Networks  
Figure 4 provides a simplified representation of the policy and critic networks used in the experiments, where \(S_{N}\) and \(H_{dim}\) denote the input and output dimensions of the fully connected network (FCN), respectively. The policy network exclusively takes the previously described states as inputs. Specifically, it receives two states, \(State_{R_{t}}\) and \(State_{W_{t}}\), which are processed by separate FCN layers internally before being merged to infer the final output. The critic network is similar in structure to the policy network, but with the distinction that it also takes the action as an input in addition to the states. This action is merged with \(State_{R_{t}}\) and \(State_{W_{t}}\), processed by individual FCN layers, and then combined to produce the final output.  
Additionally, since each model follows a sequential decision-making process where the next rendezvous channel is selected based on previously attempted channels and transformers have demonstrated the highest performance in obtaining representations of sequential data, we designed and utilized an actor-critic network structure based on transformers which is represented in Figure 5[17]. The input and output of the transformer-based model are the same as those of the network using FCN. However, the structure employs the short-term state as the input to the transformer, while embedding the long-term state using an FCN network, and finally combining them. For the positional embedding, we use the learnable embedding matrix instead of a sinusoidal one in [18]. By comparing the performance of the actor and critic models using a simple FCN and a more complex transformer, insights can be gained for optimization that considers the hardware level of the SU during channel exploration for rendezvous. Generally, if the SU is not a smartphone-level device,it will have relatively lower hardware complexity. Therefore, if the performance of the two models is similar, a reinforcement learning framework can be designed using a model similar to the FCN. Therefore, in subsequent experiments, comparing the performance of the models from this perspective could be an important factor.  
Figure 6 shows the trend of average reward when training the models of each proposed structure for the asymmetric cases; for each model, 3000 episodes are used for training. The reward was set to \(-1\) for each attempt, and 10 was given to the agent upon a successful rendezvous in the experiment. Therefore, if a rendezvous is not achieved within 10 channel explorations, the average reward can become negative. Since TTR generally exceeds 10, the average reward across the entire experiment tends to be negative. It can be observed that the average reward converges after around 500 episodes. In the case where the model is simplified using an FCN, it can be observed that the oscillation in the average reward becomes larger compared to the model utilizing a transformer as the number of channels, \(M\), increases. In the reward design, a negative reward is given when rendezvous fails. Therefore, if the oscillation of the average reward significantly occurs toward a lower range, it indicates that the TTR (time-to-rendezvous) was high in that episode, which naturally degrades the overall ETTR performance. Hence, efforts are needed to stabilize the convergence of the average reward in reinforcement learning while maintaining a high level of reward.  
Figure 4: Actor and critic networks using fully connected layers.  
Figure 5: Actor and critic networks using transformer.