# Interest Forwarding in Named Data Networking Using Reinforcement Learning
## 3 Reinforcement Learning and the Random Neural Network  
The RNN is a neural network model inspired by queueing networks [29] where neurons send and receive both positive and negative signals. The positive signals are called _excitatory_, while the negative signals are called _inhibitory_. It has been used in many applications, including image and video compression [30; 31], the recognition of tumours from Magnetic Resonance Images of the human brain [32], toxicity prediction of chemical compounds [33], web search [34], and network routing and cloud management [35; 36; 37; 38; 39]. It is a special instance of the family of stochastic networks known as G-Networks [40; 41; 42; 43; 44; 45] which have many different areas of application. Table 1 lists the notation for the RNN model. The state of a neuron \(i\) in the network at any time is represented by a non-negative potential value \(k_{i}(t)\), so that the vector:  
\[k(t)=(k_{1}(t),k_{2}(t),...,k_{n}(t))\] \begin{table}
\begin{tabular}{c c} \hline \hline
**Notation** & **Definition** \\ \hline \(k(t)\) & State vector of the RNN at time \(t\) where \(k_{i}(t)\geq 0\) is the potential level of a neuron \(i\) in the network \\ \(r_{i}\) & Firing rate at neuron \(i\) \\ \(\Lambda_{i}\) & Arrival rate of positive exogenous signals at neuron \(i\) \\ \(\lambda_{i}\) & Arrival rate of negative exogenous signals at neuron \(i\) \\ \(p^{+}(i,j)\) & Probability that a signal leaving neuron \(i\) is excitatory and heads for neuron \(j\) \\ \(p^{-}(i,j)\) & Probability that a signal leaving neuron \(i\) is inhibitory and heads for neuron \(j\) \\ \(d(i)\) & Probability that a signal leaving neuron \(i\) departs the network \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Notation for the RNN model. describes the state of an RNN at any time \(t\). An incoming signal either increases a neuron's potential by one if it is excitatory or decreases it by the same amount if it is inhibitory. An excited neuron \(i\), that is, one with non-zero potential (\(k_{i}(t)>0\)), fires randomly by sending out excitatory or inhibitory signals to other neurons or to the outside of the network. Whenever a neuron fires, its potential reduces by one. Signals can also arrive at a neuron from outside the RNN. For each neuron \(i\) in an \(n\)-neuron network:  
\[\sum_{j}^{n}[p^{+}(i,j)+p^{-}(i,j)]+d(i)=1. \tag{1}\]  
It was shown in [46] that the RNN with exponential firing intervals and Poisson exogenous signal arrivals has the product form solution:  
\[p(k)=\prod_{i=1}^{n}(1-q_{i})q_{i}^{k_{i}},\ q_{i}=\frac{\lambda_{i}^{+}}{(r_{ i}+\lambda_{i}^{-})}, \tag{2}\]  
where \(p(k)=\lim_{t\rightarrow\infty}Pr[k(t)=k]\), \(k=(k_{1},k_{2},...,k_{n})\), is the network's stationary probability distribution; \(q_{i}\) is the steady state probability that neuron \(i\) is excited. \(\lambda_{i}^{+}\) and \(\lambda_{i}^{-}\) represent the overall flow of positive and negative signals, respectively, at neuron \(i\). For \(i=1,2,...,n\), \(\lambda_{i}^{+}\) and \(\lambda_{i}^{-}\) satisfy the following system of non-linear equations whose solution exists and is unique [47]:  
\[\lambda_{i}^{+}=\sum_{j}[q_{i}r_{j}p^{+}(j,i)]+\Lambda_{i} \tag{3}\]  
\[\lambda_{i}^{-}=\sum_{j}[q_{i}r_{j}p^{-}(j,i)]+\lambda_{i} \tag{4}\]  
For adaptive routing applications, the recurrent RNN model is normally used. This RNN is fully connected and has as many neurons as there are possible outgoing links at the node it resides, that is, a neuron represents a possible forwarding decision. The weight of each RNN connection \((i,j)\) is the emission rate of positive or negative signals from neuron \(i\) to neuron \(j\).