# Pseudo Random Number Generation through Reinforcement Learning and Recurrent Neural Networks
## 3 Results  
Our experiments consist of multiple sets of training processes of various BF and RF agents. The goal of these experiments was to measure the performance of the new RF agents and compare it with the results achieved by BF agents. We consider three different agents, all trained by PPO-Clip described in Section 2.1, with an actor-critic neural network described in Section 2.4. The agent based on the formulation RF introduced in this paper is called \(\pi_{RF}\), and similarly, we denote by \(\pi_{BF}\) the agent based on BF, which we use as a baseline. We also introduce a third agent, a variation of \(\pi_{BF}\) denoted by \(\hat{\pi}_{BF}\) and called \"wanderer\": this agent is forced to move as much as possible within the environment by masking out all actions that would keep the agent in the same state. In other words, at time-step \(t\), the agent \(\hat{\pi}_{BF}\) is forbidden from setting a certain bit to the same value it has at the previous time-step, \(t-1\). In the experiments, we optimize a PPO-Clip loss with GAE advantage estimation. The two separated policy and value losses are estimated by [PERSON] with mini-batch samples of 32 experiences drawn randomly from a buffer. The buffer is filled by 500 episodes for \(\pi_{BF}\) and \(\hat{\pi}_{BF}\), and by 1000 episodes for \(\pi_{RF}\). Once the buffer is full, a training epoch is performed: thus, for instance, an agent \(\pi_{BF}\) building sequences of length \(B=200\) by episodes of length \(T=100\) will start training when the buffer is filled with \(500\times 100=50\),000 experiences, and the epoch will end after [50,000/32] \(=1562\) training steps. Learning rates are \(3\times 10^{-4}\) for the policy and \(1\times 10^{-3}\) for the value. At the end of the epoch, the buffer is emptied, and the RL pipeline goes on by experiencing new episodes. We present experimental results in the form of plots. Like in [6], the performance metric used is the average total reward across sets of epochs called _volley_. In this paper, a volley is made from two epochs--that is, the plots represent a moving average over a window of two epochs. Figures 2a,b-4a,b describe three experiments with \(\pi_{BF}\) and \(\hat{\pi}_{BF}\) over sequences of different lengths: \(B=80\) bits, \(B=200\) bits and \(B=400\) bits, respectively. The length of episodes is \(T=40\), \(T=100\) and \(T=200\), respectively. For very short sequences of 80 bits, \(\hat{\pi}_{BF}\) has better performance at the end, yet very similar performance at the beginning of training. Making the sequence longer, \(B=200\) bits, allows the wanderer agent to perform much better than the baseline--starting from a slightly different performance at the beginning, \(\hat{\pi}_{BF}\) has a much higher average total reward at the end of the training process. This trend is confirmed with sequences of \(B=400\) bits. Figures 5-7 describe three experiments with \(\pi_{RF}\). In this case, the length of the trajectory \(T\) directly influences the length of the sequence generated by \(\pi_{RF}\). We keep \(T=100\) constant across \(\pi_{RF}\) experiments. This value is chosen experimentally as giving the best performance while keeping the inference time acceptable. To obtain sequences that are comparable with the ones generated by BF agents, we chose to append \(N=2\), \(N=5\) and \(N=10\) bits, respectively, resulting in final sequences of length \(2\times 100=200\) bits, \(5\times 100=500\) bits, and \(10\times 100=1000\) bits, respectively. The seed \(S_{0}\) of the PRNG was drawn from a standard multivariate normal distribution of dimensions \(N=2\), \(N=5\), and \(N=10\), respectively. Figure 3: Experiment on BF with \(B=200\). Despite the similar learning curve, there is a huge difference in the achieved average total reward per episode between \(\hat{\pi}_{BF}\) and \(\pi_{BF}\) at the end of the training process. Volleys are composed of 1000 episodes each and the fixed length of each trajectory is \(T=100\) steps. (**a**) \(\pi_{BF}\) with \(B=200\). (**b**) \(\hat{\pi}_{BF}\) with \(B=200\).