# Pseudo Random Number Generation through Reinforcement Learning and Recurrent Neural Networks
## 2 Materials and Methods
### Reinforcement Learning (RL)
The average return from a state \(s\), that is, the average  
Figure 1: The agent–environment interaction is made at discrete time-steps \(t=0,1,2,\ldots\). At each time-step \(t\), the agent uses the state \(S_{t}\in\mathcal{S}\) given by the environment to select an action \(A_{t}\in\mathcal{A}\). The environment answers with a real number \(R_{t+1}\in\mathcal{R}\subset\mathbb{R}\), called a reward, as well as a next state \(S_{t+1}\). Going on, we obtain a trajectory \(S_{0},A_{0},R_{1},S_{1},A_{1},R_{2},\ldots\) Ref. [8] (Figure 3.1). total reward the agent can accumulate starting from \(s\), represents how good the state \(s\) is for the agent _following the policy \(\pi\)_, and it is called a _state-value_ function:  
\[v_{\pi}(s):=E_{\pi}[G_{t}|S_{t}=s].\]  
Likewise, one can define the _action-value_ function (known also as the _quality_ or _q-value_), encoding how good _choosing an action a from \(s\) and then following the policy \(\pi\)_ is:  
\[q_{\pi}(s,a):=E_{\pi}[G_{t}|S_{t}=s,A_{t}=a].\]  
Since the return \(G_{t}\) is recursively given by \(R_{t+1}+\gamma G_{t+1}\), the RL problem has an optimal substructure, expressed by recursive equations for \(v_{*}\) and \(q_{*}\). If an accurate description of the dynamic \(p\) of the environment is available and if one can store all states into memory, then dynamic programming iterative techniques can be used, and an approximate solution \(v_{*}\) or \(q_{*}\) to the Bellman optimality equations can be found. From \(v_{*}\) or \(q_{*}\), one can then easily recover an optimal policy--for instance, \(\pi_{*}(s):=\operatorname*{argmax}_{a\in\mathcal{A}}q_{*}(s,a)\) is a deterministic optimal policy. However, in most problems we have only _partial knowledge of the dynamics_, if any. This can be overcome by _sampling trajectories_\(S_{t}=s,A_{t}=a,R_{t+1},S_{t+1},A_{t+1},R_{t+2},\ldots\) to _estimate_ the \(q\)-value \(q_{\pi}(s,a)=\mathbb{E}_{\pi}[G_{t}|S_{t}=s,A_{t}=a]\), instead of computing a true expectation. Moreover, in most problems _there are way too many states to store them in memory_, or we just have to go through every state once. In this case, the estimate of \(q_{\pi}(s,a)\) must be stored in a parametric function approximator \(q_{\pi}(s,a;w)\), where \(w\) is a parameter vector living in a dimension much lower than \(|\mathcal{S}\times\mathcal{A}|\). Due to their high representational power, deep neural networks are nowadays widely used as approximators in RL, where the combination of deep neural networks with RL is called Deep Reinforcement Learning (DRL). Iterative dynamic programming techniques can then be approximated, giving a family of algorithms known as _Generalized Policy Iteration_ algorithms. They work by sampling trajectories to obtain estimates of the true values \(\mathbb{E}_{\pi}[G_{t}|S_{t}=s,A_{t}=a]\), and use supervised learning to find the optimal parameters vector \(w\) for \(q_{\pi}(s,a;w)\). This estimated \(q\)-value is used to find a policy \(\pi^{\prime}\) better than \(\pi\), and iterating over this evaluation-improvement loop usually gives an approximate solution to the RL problem. Generalized Policy Iteration is _value-based_, because it uses a value function as a proxy for the optimal policy. A completely different approach to the RL problem is given by Policy Gradient (PG) algorithms. They directly estimate the policy \(\pi(a|s;\theta)\), without using a value function. The parameters vector \(\theta_{t}\) at time \(t\) is modified to maximize a suitable scalar performance function \(J(\theta)\), with the gradient ascent update rule:  
\[\theta_{t+1}:=\theta_{t}+\widehat{\
abla J(\theta_{t})}.\]  
Here, the _learning rate_\(\alpha\) is the step size of the gradient ascent algorithm, determining how much we are trying to improve the policy at each update, and \(\widehat{\
abla J(\theta_{t})}\) is any estimate of the performance gradient \(\
abla J(\theta)\) of the policy. Different choices for the estimator corresponds to different PG algorithms.