# Channel-Hopping Using Reinforcement Learning for Rendezvous in Asymmetric Cognitive Radio Networks
## 1 Introduction  
The rapid expansion of wireless communication technology has led to an unprecedented increase in demand for the finite and valuable resource of radio spectrum [1]. Traditional spectrum allocation policies, where spectrum bands are statically assigned to licensed users (primary users, PUs), have resulted in significant inefficiencies. Some portions of the spectrum remain unused, while other parts become congested, preventing the optimal utilization of this critical resource. To address these issues, cognitive radio networks (CRNs) have emerged as a promising solution, allowing secondary users (SUs) to dynamically access underutilized spectrum without causing interference to PUs [2, 3, 4, 5, 6, 7, 8]. One of the most critical challenges in CRNs is the rendezvous problem, which refers to the requirement for two or more SUs to find a common communication channel to exchange data. Successful rendezvous is a prerequisite for establishing a communication link, making it a fundamental operation in CRNs. The rendezvous problem becomes particularly complex in asynchronous environments where SUs operate without global time synchronization, leading to mismatches in channel availability [2, 3, 4]. To address these challenges, various channel-hopping (CH) schemes have been developed to enable SUs to rendezvous efficiently and reliably by periodically hopping across different channels according to predefined sequences [3, 4, 8, 9]. However, many existing CH schemes assume that SUs share a common set of available channels, which is rarely the case in real-world scenarios. In practice, the set of available channels can differ among SUs due to factors such as geographic location, interference patterns, and varying regulatory constraints [5, 9, 10]. This heterogeneity in channel availability complicates the rendezvous process, making it challenging for SUs to find a common channel. An early solution to this problem is the jump-stay (JS) algorithm, which is designed tofacilitate rendezvous even when the set of shared channels differs among SUs. While the JS algorithm represents significant progress, it has the drawback of requiring a large number of attempts to achieve successful rendezvous, resulting in increased time-to-rendezvous (TTR) and making it less suitable for real-time applications [9]. To address the limitations of traditional CH schemes, recent research has explored the application of reinforcement learning (RL) techniques to the rendezvous problem in CRNs [5]. Reinforcement learning is well suited to environments where agents must learn optimal strategies through interaction with a dynamic environment. For example, [5] models the rendezvous problem as a multi-armed bandit (MAB) problem, where each SU independently learns to select the optimal channel based on past experiences. This approach leverages the adaptability of RL, allowing SUs to improve their channel selection strategies over time. However, the MAB approach has a limitation in that it does not consider state, which can be problematic when trying to learn the optimal policy in reinforcement learning. In this paper, we propose a new RL-based approach that designs the state using the experience of the SUs' previous attempts to learn the optimal policy. Additionally, we modify the existing method by applying a negative reward for each failed attempt, aiming to achieve rendezvous with the fewest possible attempts. Finally, we propose an architecture that embeds and learns each state to allow the use of deep learning-based reinforcement learning, and we train it using the actor-critic policy gradient technique. In summary, the key contributions of this paper are as follows:  
* We propose an RL-based rendezvous algorithm that effectively handles scenarios where SUs have different sets of available channels, addressing a significant limitation of existing methods [2; 3; 4; 9]. An actor-critic-based algorithm is utilized to train the agent, and the state and reward are designed considering the rendezvous scenario in a cognitive radio environment.