## 1 Introduction  
Spectrum shortage has been one of the major bottlenecks for the development of wireless communication systems [1]. As a promising solution to the frequency scarcity, Cognitive Radio (CR) is presented to allow Secondary Users (SUs) to exploit licensed spectrum opportunistically, which breaks the fixed frequency allocation regulation. Great developments have been achieved in some crucial CR technologies such as spectrum sensing, spectrum access and power allocation [2; 3; 4]; however, there has been little study on cross-layer routing in the research community. Routing plays a very important role in the overall performance of any networks [5]. Although the CR technique improves spectrum utilization efficiency, it raises significant challenges to route selection in distributed Cognitive Radio Networks (CRN) [6]. Firstly, spectrum allocation should be considered as selecting the next hop in routing [7]. In traditional wireless communication, all of the nodes share the same frequency bands, whereas in multi-hop CRN, each node may work with different available channel sets based on the uncertainty of Primary Users' (PUs') activities. Thus, both the geographic location and spectrum bands have an effect on the connectivity of network topology. Secondly, it is challenging to develop an efficient and robust routing protocol in multi-hop CRN due to the spectrum uncertainty feature and the unreliable characteristic of the wireless medium [8]. Conventional routing protocols are used to apply explicit and rigid programming to opportunistic routing design. In [9], the virtual coordinates were introduced to reflect both geographic distance and spectrum availability, and then the geographic routing scheme was developed to cut through the area with lower frequency occupancy. However, it did not consider the interference issue among secondary users. In [10], a novel routing metric was designed to estimate both the future spectrum availability and the average transmission time. Then, two routing schemes were proposed to reduce the probability of spectrum handoff and rerouting. Nevertheless, power adaptation was not taken into consideration, which affected its energy efficiency. The author in [11] presented an on-demand routing protocol for a cluster-based cognitive radio ad-hoc network. The network was divided into several clusters, and then the opportunistic routing was formulated as a weighted graph problem. However, the rendezvous algorithm for neighbor discovery remained to be studied. With transcendental knowledge such as network topology and channel characteristics, these works attempted to search a fixed path. This was not practical, owing to the distributed network architecture and uncertainty nature of CRN. To be truly intelligent and cognitive, the cognitive radio system should have the ability to learn and reason [12]. In recent years, the development of Deep Reinforcement Learning (DRL) has attracted great attention and has been widely used in the decision-making and control field. To resolve the curse of dimensionality in the problem with large state space, the Deep Q-Network (DQN) was proposed in [13]. It applied neural network to approximating the Q-function and improved the generalization ability of the algorithm. Simultaneously, the bias was introduced, and it was more difficult to guarantee the convergence. Due to the one-step update framework in DQN, its learning efficiency was relatively high. Furthermore, Policy Gradient (PG) was able to parameterize the strategy and learn the strategy directly [14]. It had better convergence properties and was effective in the continuous action space. However, PG typically converged to a local rather than global optimum, and it was inefficient since the strategy was updated every learning episode. To combine the advantages of DQN and PG, the Deep Deterministic Policy Gradient (DDPG) was presented in [15], introducing two independent neural networks called actor and critic, respectively.