# A QoS Optimization Approach in Cognitive Body Area Networks for Healthcare Applications
## 3 System Model
### Channel Allocations
#### 3.5.1 Reinforcement Learning--Channel Assignment Algorithm (RL-CAA)  
The RL-CAA is based on reinforcement learning, which itself comes from the field of artificial intelligence and machine learning. The RL-CAA builds upon a Bernoulli distribution and its internal architecture is based on the Bernoulli random variables and probabilities. This internal part of the algorithm is called the Bernoulli logic unit [32; 33]. The algorithm works following an unsupervised learning approach and interacts with the target environment and considers the feedback. The algorithm interaction with the environment is based on feedback and M input signals. An input signal x is biased by the weighting factor w, which are real vectors containing input variables for the algorithm and their corresponding weights. The inputs and their weights are combined as per Equation (2) in a scalar quantity z:  
\[\mathrm{z}=\sum_{i=1}^{M}w_{i}x_{i} \tag{2}\] \begin{table}
\begin{tabular}{c c c} \hline \hline
**Constants** & **LOS** & **NLOS** \\ \hline a & 2.45 & 3.673 \\ b & \(-\)17.99 & \(-\)30.61 \\ c & 43.95 & 76.94 \\ d & 16.05 & 9.48 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Channel model values of the polynomial constants. where M is total number of channels; z is then subjected to a Bernoulli logistic function given in Equation (3):  
\[\mathrm{p}=\mathrm{f}(\mathrm{z})=\frac{1}{1+\mathrm{e}^{-\mathrm{z}}} \tag{3}\]  
Then the algorithm updates its internal probabilities and context variables (i.e., x and w) based on that feedback. A probability bias \(\sigma\) is added to p so that it may not converge to 0. Based on the feedback signal, a decision is made according to the context in which RL algorithm is being used. The feedback is termed reward signal r(t) in the RL terminology. The instantaneous reward signal in our simulation domain is given in Equation (4):  
\[\mathrm{r}(\mathrm{t})=\left\{ \begin{array}{c}0,\ \mathrm{if\ Throughput<TH\ or\ SNR<0\ or\ interference>I_{Th}}\\ \overline{\SNR},\ \mathrm{otherwise}\end{array} \right. \tag{4}\]  
where the throughput threshold, i.e., TH, depends upon the total number of nodes at the gateway (Table 4) and ITh is the interference threshold. The reward value is the average signal to noise ratio (SNR) of the channel. The output of the algorithm is a Bernoulli random variable (Equation (5)). At the start of the simulations, each gateway executes the RL-CAA in order to find the optimal channel allocation for the data transmission. The RL-CAA traverses all the available channels; the channel(s) which meet the criteria given in Equation (5) are then selected:  
\[\mathrm{y}=\left\{ \begin{array}{c}0,\ \mathrm{if\ r(t)\leq 0}\\ 1,\ \mathrm{if\ r(t)>0}\end{array} \right. \tag{5}\]  
The probability mass function representing the probability p of output y is described in Equation (6):  
\[\mathrm{g}(\mathrm{y},\mathrm{p})=\left\{ \begin{array}{c}1-\mathrm{p},\ \mathrm{if\ y=0}\\ \mathrm{p},\ \mathrm{if\ y=1}\end{array} \right. \tag{6}\]  
To simplify, y is a two action learning outcome of the RL process. The learning of the agent can be condensed in the weighting vector so that at each time step t, the agent learns by updating its weighting vector using Equation (7) and any instantaneous change in the individual weight is calculated by Equation (8):  
\[\mathrm{w}(\mathrm{t})=\mathrm{w}(\mathrm{t}-1)+\Delta\mathrm{w}(\mathrm{t}) \tag{7}\]  
\[\Delta\mathrm{w}_{\mathrm{i}}(\mathrm{t})=\alpha(\mathrm{t})[\mathrm{r}( \mathrm{t})-\bar{\mathrm{r}}(\mathrm{t}-1)][\mathrm{y}(\mathrm{t}-1)-\mathrm{ p}(\mathrm{t}-1)]\mathrm{x}_{\mathrm{i}}(\mathrm{t}-1) \tag{8}\]  
where \(\alpha(\mathrm{t})>0\) is called the learning rate, r(t) is the reward returned by the environment at any instant (step) of time t, and \(\bar{\mathrm{r}}(\mathrm{t})\) is the average reward which is obtained as shown in Equation (9):  
\[\bar{\mathrm{r}}(\mathrm{t})=\beta\mathrm{r}(\mathrm{t})+(1-\beta)\bar{ \mathrm{r}}(\mathrm{t}-1) \tag{9}\]  
where \(0<\beta\leq 1\) is called the reward memory factor of the RL.