# Pseudo Random Number Generation through Reinforcement Learning and Recurrent Neural Networks
## 4 Discussion  
In this paper, we introduced a novel, partially observable MDP modeling the task of generating PRNGs from scratch using DRL, denoted by RF for recurrent formulation. RF improves our previous MDP modeling in [6] because it makes the action space size independent of the length of the generated sequence. Experiments show that RF agents trained with PPO simultaneously obtains a higher average NIST score and longer sequences, thus improving [6] in two different ways. We used a PPO instance with a hidden state of an LSTM to encode significant features of the non-observed portion of the sequence--as far as we know, this is an original idea for PRNG. Experiments with a random agent show that RF is a better form of MDP modeling when compared with the binary formulation BF of [6]--that is, when RF is compared with BF, it obtains a higher average NIST score with longer sequences. All this means that RF scales better to PRNG with longer periods.  
However, while RF is a significant improvement over BF, the action space size grows as \(2^{N}\), where \(N\) is the bit-length of the appended sequence. Since DRL does not scale well to discrete and large action sets (see, for instance, [17]), this is a limitation for RF. In our experiments, we have found that \(N>10\) is not feasible for RF. Moreover, the vanishing gradient is an obstacle to increasing the episodes' length \(T\) with recurrent neural networks, as shown in [16]. In our experiments, we have been unable to train RF with \(T=200\), so we consider \(T=100\) an upper bound for RF with PPO and the LSTM architecture described in Section 2.4. Since the PRNG period is \(T\cdot N\), we can say that RF, with all the hyperparameters described in this paper, does not scale well over 1000 bits. Another problem of this approach is the sparse reward, which in general makes it difficult for a RL agent to be trained. The above remarks takes to the following list of possible future improvements/research topics:  
* Devise a different, less sparse, reward function.
* Devise an algorithm using RF modified with a continuous action space. In this way, we can append sequences of any length without increasing the size of \(\mathcal{A}\). Some preliminary experiments show  
Figure 11: A graphical representation of three sequences of 1000 bits generated by the same trained \(\pi_{RF}\) with **(a)** NIST score 0.4, **(b)** NIST score 0.43, and **(c)** NIST score 0.53. Images were obtained by stacking the 1000 bits in 40 rows and 25 columns, then ones were converted to \(10\times 10\) white squares and zeros to \(10\times 10\) black squares. The resulting image was smoothed.  
that this approach could work, but at the time of writing, it still presents some performance issues, and is a work in progress.
* RF could be used to intelligently stack generated periods. For example, one could train a policy to stack sequences generated by one or multiple PRNGs, even with different sequence lengths. This would generate PRNGs with very long periods without a sensible drop in quality. This approach could use a mixture of state-of-the-art PRNGs, DRL generators like the ones seen in this paper, and so on. A challenge of this approach is how to measure the change in the NIST score when appending a random sequence to another.
* The upper bound given by the episodes' length \(T\) could be overcome, or at least mitigated, by other architectures capable of maintaining memory for a time longer than recurrent neural networks, like attention models.