## 1 Introduction
The spectrum aware clustering scheme in [14] is mainly driven by an event which requires a temporal cluster. The scheme uses nodes' local position in respect to the event and sinks to select eligible nodes for clustering. It then elects a clusterhead among the eligible nodes based on channel availability, node degree and distance to sinks in the neighborhood. The authors in [17] proposed a centralized Groupwise Spectrum-Aware (GWSA) clustering algorithm; it first creates a proximity matrix for all nodes, determines from the matrix the global minimum distance between pairs and then merges the nearest clusters that satisfy the Groupwise constraints in each iteration until an optimal number of clusters that minimizes network-wide energy consumption is achieved. However, this approach suffers from network instability because PU arrival may cause re-clustering of the whole network and also its high computational complexity which increases proportionately along with the size of the proximity matrix limits its practical implementation in a large-scale network. The algorithm described in [18] minimizes the network instability problem such that re-clustering involves only nodes that detect the PU arrival while network topology of other nodes remains intact. Furthermore, it determines local minimum distance between neighboring cluster pair and merges multiple nearest cluster pair at a single time to increase the convergence rate. Although the algorithm achieves relatively much less computation complexity, its performance is relatively inferior to the GWSA. Unlike [17,18] our proposed algorithm not only tends to minimize network energy consumption, but also improves spectrum hole detection, by way of exploring multi-user sensing diversity through cooperative spectrum sensing which is essential for dynamic spectrum access in cognitive radio. The reinforcement learning (RL) technique has been applied to many existing works related to cognitive radio, but they were mainly applied to spectrum sensing [2], cooperative sensing [3,19], spectrum sharing [20], channel sensing [21,22] and dynamic channel access [23,24,25]. For example, the authors in [3,19] applied RL to enhance cooperative gain and mitigate cooperative overhead in cognitive radio. The approach eliminates correlated and unreliable cooperative neighboring SUs from cooperation and determines optimal set of cooperative SUs that minimize spectrum sensing delays and control channel traffic. In a bid to improve channel sensing performance, the authors in [26] used channel achievable and channel availability as the basis for determining optimal channel sensing order by applying a low complexity RL algorithm. The approach in [2] achieved energy efficient spectrum sensing by exploration of sensing assignment and exploitation of high throughput frequency bands. Eventhough the existing RL approaches in the literature made valuable contributions to CR-WSN, none of them considered RL for network clustering  
In this paper, we propose an Energy Efficient Spectrum Aware clustering algorithm based on Reinforcement Learning (EESA-RLC) to enhance spectrum hole detection and minimize network energy consumption in CR-WSNs. Reinforcement learning is a machine learning technique that allows an agent to interact with its operating environment and learn an optimal policy that maximizes cumulative rewards [27]. The operating environment can be formulated as a Markov Decision Process (MDP) comprising actions, state of the system, transition rewards, transition probability, performance metric and policy [28]. The agent, which in this case is the SU, detects vacant licensed channels through channel sensing, imposes pairwise constraints to select a clusterhead among the neighboring clusterheads, cooperates with other member nodes in the cluster to determine channel availability, and then chooses an optimal policy that enhances spectrum hole detection and minimizes network energy consumption.