# Pseudo Random Number Generation through Reinforcement Learning and Recurrent Neural Networks
## 1 Introduction  
Generating random numbers is an important task in cryptography, and more generally in computer science. Random numbers are used in several applications, whenever producing an unpredictable result is desirable--for instance, in games, gambling, encryption algorithms, statistical sampling, computer simulation, and modeling.  
An algorithm generating a sequence of numbers approximating properties of random numbers is called a Pseudo-Random Number Generator (PRNG). A sequence generated by a PRNG is \"pseudo-random\" in the sense that it is generated by a deterministic function: the function can be extremely complex, but the same input will give the same sequence. The input of a PRNG is called the seed, and to improve the randomness, the seed itself can be drawn from a probability distribution. Of course, assuming that drawing from the probability distribution is implemented with an algorithm, everything is still deterministic at a lower level, and the sequence will repeat after a fixed, unknown number of digits, called the _period_ of the PRNG.  
While true random number sequences are more fit to certain applications where true randomness is necessary--for instance, cryptography--they can be very expensive to generate. In most applications, a pseudo-random number sequence is good enough.  
The quality of a PRNG is measured by the randomness of the generated sequences. The randomness of a specific sequence can be estimated by running some kind of statistical test suite.  
In this paper, the National Institute of Standards and Technology (NIST) statistical test suite for random and pseudo-random number generators [1] is used to validate the PRNG.  
Neural networks have been used for predicting the output of an existing generator, that is, to break the key of a cryptography system. There have also been limited attempts at generating PRNGs using neural networks by exploiting their structure and internal dynamics. For example, the authors of [2] use recurrent neural networks dynamics to generate pseudo-random numbers. In [3], the authors use the dynamics of a feedforward neural network with random orthogonal weight matrices to generate pseudo-random numbers. Neuronal plasticity is used in [4] instead. In [5], a generative adversarial network approach to the task is presented, exploiting an input source of randomness, like an existing PRNG or a true random number generator.  
A PRNG usually generates the sequence incrementally, that is, it starts from the seed at time \(t=0\) to generate the first number of the sequence at time \(t=1\), then the second at \(t=2\), and so on. Thus, it is naturally modeled by a deterministic Markov Decision Process (MDP), where state space, action space, and rewards can be chosen in several ways. A Deep Reinforcement Learning (DRL) pipeline can then be used on this MDP to train a PRNG agent. This DRL approach has been used for the first time in [6] with promising results, see modeling details in Section 2.2. This is a probabilistic approach that generates pseudo-random numbers with a \"variable period\", because the learned policy will generally be stochastic. This is a feature of this RL approach.  
However, the MDP formulation in [6] has an action set where the size grows linearly with the length of the sequence. This is a severe limiting factor, because when the action set is above a certain size it becomes very difficult, if not impossible, for an agent to explore the action space within a reasonable time.  
In this paper, we overcome the above limitation with a different MDP formulation, using a partially observable state. By observing only the last part of the sequence, and using the hidden state of a Long-Short Term Memory (LSTM) neural network to extract important features of the full state, we significantly improve the results in [6], see Section 3. The code for this article can be found at GitHub repository [7].