# Interest Forwarding in Named Data Networking Using Reinforcement Learning
## 3 Reinforcement Learning and the Random Neural Network
That is,  
\[w^{+}(i,j)=r_{i}p^{+}(i,j),\quad w^{-}(i,j)=r_{i}p^{-}(i,j)\]  
Introducing the above expressions and assuming signals do not leave the network (\(d(i)=0\)), we can rewrite Equation (1) as:  
\[r_{i}=\sum_{j=1}^{n}[w^{+}(i,j)+w^{-}(i,j)]\quad i=1,2,...,n \tag{5}\]  
The matrices of the weights of the connections: \(W^{+}=\{w^{+}(i,j)\}\), \(W^{-}=\{w^{-}(i,j)\}\), are important parameters whose values are continuously modified during the learning process. The RNN is initialised with all the weights equal. This is done by choosing a firing rate \(r_{i}\) and using Equation (5) to solve for the initial weight value. That is, \(w=\frac{r_{i}}{2(n-1)}\), where \(w\) is the initial value of all weights; remember that \(w^{+}(i,i)=w^{-}(i,i)=0\ \forall i\). The pseudocode in Algorithm 1 illustrates how reinforcement learning can be implemented using the RNN. The reinforcement learning algorithm used is inspired from the _E-rule_ scheme in [17], and it assumes that a reward value can be estimated for every decision. In this approach, the outcome of a previous decision, \(R_{l}\), is compared with an internal expectation, \(T_{l-1}\), of the neural network and the RNN receives reinforcement based on the result of this comparison. The reinforcement can either be _positive_ if the outcome is considered a success, or _negative_ otherwise. In the illustration in Algorithm 1, the \(l\)-th decision is considered to be a success if \(R_{l}\geq T_{l-1}\); otherwise it is a failure. A success leads to a significant increase in the excitatory weights going into the corresponding neuron and a small increase in the inhibitory weights leading to the other neurons. Otherwise, the inhibitory weights of the neuron assigned to the decision is significantly increased and the excitatory weights into the remaining neurons are slightly increased. ```
1:\(T_{l-1}\), \(W^{+}\), \(W^{-}\), internal expectation and weight matrices before the \(l\)-th reward is received; \(0<\alpha<1\). 2:\(R^{i}_{l}\), the \(l\)-th received reward obtained when the \(l\)th decision was to select output link \(i\). 3:\(q_{j}\), \(\forall\)\(i=1,2,...,n\), the excitation probabilities for an RNN with \(n\) neurons if\(R^{i}_{l}\geq T_{l-1}\)then /* \(l\)-th decision is a success*/ for each neuron \(j\) in the RNN, \(j\
eq i\)do \(w^{+}(j,i)\gets w^{+}(j,i)+R^{i}_{l}\) for each neuron \(k\) in the RNN, \(k\
eq j\)do \(w^{-}(k,j)\gets w^{-}(k,j)+\frac{R^{i}_{l}}{n-2}\) else /* \(l\)-th decision is not a success*/ for each neuron \(j\) in the RNN, \(j\
eq i\)do \(w^{-}(j,i)\gets w^{-}(j,i)+R^{i}_{l}\) for each neuron \(k\) in the RNN, \(k\
eq j\)do \(w^{+}(k,j)\gets w^{+}(k,j)+\frac{R^{i}_{l}}{n-2}\) /* Normalizing the weights */ for each neuron \(j\) in the RNN do \(r^{*}_{j}\leftarrow\sum_{m}^{n}[w^{+}(j,m)+w^{-}(j,m)]\) for each neuron \(k\) in the RNN, \(k\
eq j\)do \(w^{+}(j,k)\gets w^{+}(j,k)*\frac{r_{j}}{r_{j}}\) \(w^{-}(j,k)\gets w^{-}(j,k)*\frac{r_{j}}{r_{j}}\) /* solve the \(n\) non-linear simultaneous equations in (2) for each \(0\leq q_{j}\leq 1\)*/ /* We use a fixed-point iteration, starting with \(q^{0}_{j}=0.5\)\(\forall\)\(j=1,2,...,n\)*/ \(q^{k+1}_{j}\leftarrow min[1,\frac{\sum_{m}[q^{k}_{m}w^{+}(m,j)]+\Lambda_{j}}{r _{j}+\sum_{m}[q^{k}_{m}w^{-}(m,j)]+\Lambda_{j}}]\) /* Updating the internal expectation */ \(T_{l}\leftarrow\alpha T_{l-1}+(1-\alpha)R_{l}\)/* \(\alpha\) is to be chosen closer to \(1\)*/
```  
**Algorithm 1** The RNN training process  
After updating the weights, Algorithm 1 also shows a renormalisation of the weights. This is to prevent ever-increasing weights, and the fact that it is the relative magnitudes of the weights that determine the state of the network justifies this step. The new firing rate \(r^{*}_{i}\) for each neuron is first evaluated using the updated weights in Equation (5).