# Pseudo Random Number Generation through Reinforcement Learning and Recurrent Neural Networks
## 3 Results
Figure 2: Experiment on BF with \(B=80\). The learning curve is different and the average total reward is better with \(\hat{\pi}_{BF}\). Volleys are composed of 1000 episodes each, and the fixed length of each trajectory is \(T=40\) steps. (**a**) \(\pi_{BF}\) with \(B=80\). (**b**) \(\hat{\pi}_{BF}\) with \(B=80\). For \(N=2\), training was not successful. We do not have a clear explanation for this fact. A possible reason is that we were trying to represent the non-observable portion of the state with a very low-dimensional input. However, this is not completely true, because in theory, the inputs from every time-step were preserved in the hidden state of the LSTM. This issue could be related to the difficulties that recurrent neural networks have shown with gradient descent optimizers--see [16]. For \(N=5\), that is, sequences of 500 bits, \(\pi_{RF}\) vastly outperforms \(\pi_{BF}\), and albeit by a narrower margin, \(\hat{\pi}_{BF}\) with sequences of 200 bits. Thus, we had better performance with 150% additional bits in the sequence. For \(N=10\), that is, sequences of 1000 bits, \(\pi_{RF}\) vastly outperforms \(\pi_{BF}\) with \(B=400\). The wanderer agent in this case performs similarly, but \(\pi_{RF}\) still produces sequences that are 150% longer than the ones produced by \(\hat{\pi}_{BF}\). Figure 4: Experiment on BF with \(B=400\). The difference in the achieved average total reward per episode between \(\pi_{BF}\) and \(\pi_{BF}\) is similar to the case with \(B=200\), while the learning curve is different. Volleys are composed of 1000 episodes each, and the fixed length of each trajectory is \(T=200\) steps. (**a**) \(\pi_{BF}\) with \(B=400\). (**b**) \(\hat{\pi}_{BF}\) with \(B=400\). Figure 5: Average total rewards during training of \(\pi_{RF}\) with \(N=2\) and \(T=100\). Volleys are composed of 2000 episodes each. We now want to test the hypothesis that, in order to generate PRNGs by DRL, modeling the MDP as in RF is much better than modeling it as in BF. To this aim, we compare the average total rewards per episode of random agents operating in RF and in the baseline BF. We call these agents \(\rho_{RF}\) and \(\rho_{BF}\), respectively. This comparison is performed to exclude the possibility that PPO is performing better with RF, but maybe different algorithms would perform better with BF. Using a random agent means removing the algorithm from the equation. Figures 8a,b-10a,b describe three comparisons between the random agents producing short, medium, and long sequences, respectively. As before, short means 80 bits in BF and 200 bits in RF, medium means 200 bits in BF and 500 bits in RF, and long means 400 bits in BF and 1000 bits in RF. In each comparison, the average total rewards of \(\rho_{RF}\) is higher than \(\rho_{BF}\). From these results, we can assess that RF is a better formulation than BF for the PRNG task. Figure 6: Average total rewards during training of \(\pi_{RF}\) with \(N=5\) and \(T=100\) steps. Volleys are composed of 2000 episodes each. Figure 7: Average total rewards during training of \(\pi_{RF}\) with \(N=10\) and \(T=100\) steps. Volleys are composed of 2000 episodes each. Figure 8: Average total rewards of a random agent on BF and RF for short sequences. (**a**) \(\rho_{BF}\) with \(B=80\), episodes length \(T=40\). (**b**) \(\rho_{RF}\) with \(N=2\), episodes length \(T=100\). Figure 10: Average total rewards of a random agent on BF and RF for long sequences. (**a**) \(\rho_{BF}\) with \(B=400\), episodes length \(T=200\). (**b**) \(\rho_{RF}\) with \(N=10\), episodes length \(T=100\). Figure 9: Average total rewards of a random agent on BF and RF for medium sequences. (**a**) \(\rho_{BF}\) with \(B=200\), episodes length \(T=100\). (**b**) \(\rho_{RF}\) with \(N=5\), episodes length \(T=100\). Finally, Figure 11a-c graphically represents three sequences of 1000 bits generated by the same agent \(\pi_{RF}\) after training, with NIST scores of 0.4, 0.43, and 0.53, respectively.