# Reinforcement Learning in a New Keynesian Model +
## 5 Heterogeneous Expectations across Agents
### Endogenous Proportions of RE and BR Agents with Reinforcement Learning  
Proportions of rational households (\(n_{h,t}\)) and firms (\(n_{f,t}\)) are given by (8)  
\[n_{j,t}=\frac{\exp(-\gamma\Phi_{j,t}^{RE})}{\exp(-\gamma\Phi_{j,t})^{RE}+\exp( -\gamma\Phi_{j,t}^{BR})}\,;\quad j=h,f\]  
where fitness for households and firms \(j=h,f\) is given by  
\[\Phi_{j,t}^{RE} = \mu_{j}^{RE}\Phi_{j,t-1}^{RE}+(1-\mu_{j}^{RE})\Big{(}\text{weighted sum of forecast errors}+C_{j}\Big{)}\] \[\Phi_{j,t}^{BR} = \mu_{j}^{BR}\Phi_{j,t-1}^{BR}+(1-\mu_{j}^{BR})\Big{(}\text{weighted sum of forecast errors}\Big{)}\]  
Table 1 provides a third-order perturbation solution of the non-linear NK RE-BR model. We use the Bayesian estimation of the model in Reference [30] where the model is linearised and the proportions \(n_{h,t}\) and \(n_{f,t}\) are fixed. Non-linear estimation would be  
required to pin down the parameters \(n_{h}\), \(n_{f}\) in the steady state in the BR scenarios and \(\mu_{h}^{RE,BR}\), \(\mu_{f}^{RE,BR}\) and \(\gamma\) in the reinforcement learning process, which goes beyond the scope of this paper. Thus, here we impose them as reported in the table (\(n_{h,t}=n_{f,t}=0.1\)). We also scale the estimated standard deviations of the shocks using a parameter \(\sigma=1,2\). For the robustness of our results, we perform additional simulations, for different choice of the memory parameters, and present the results with \(\mu_{h}^{RE}=\mu_{h}^{\text{BR}}=\mu_{f}^{RE}=\mu_{f}^{BR}=0.5\) and \(=0.75\) in Appendix G. The robustness exercise assumes instead that agents have some memory of past observations.  
The main results from these simulations are as follows. First, reinforcement learning introduces _high kurtosis and skewness_ in macroeconomic variables, the absence of kurtosis in the standard NK model, often highlighted in the literature (see, for example, Reference [3]), is in part simply the consequence of linearisation, and non-normality is a feature of higher order approximations. Second, reinforcement learning with stronger switching processes (i.e., \(\gamma=100,1000\)) coupled with higher volatility of exogenous shocks results in the numbers of rational agents increasing from the estimated deterministic steady state value of 0.1 to 0.13 and 0.15 for households and firms, respectively, in the stochastic steady state. Third, given that BR is a welfare-reducing friction in these models, it follows that volatility can actually be welfare-increasing in our heterogeneous expectations setting. Furthermore, when we assume that agents have some memory of past observations when revising their expectations given their forecast performances, the simulated skewness and kurtosis are lower compared to the case when no memory is assumed in the learning process.  
Our main results clearly suggest that, when the switching process between groups of heterogeneous agents becomes more deterministic depending on agents' willingness to learn from the past performance when predicting future outcomes, this leads to an increase in the level of rationality in the BR macroeconomy. This result is in line with the finding in Reference [3]. The cognitive effect of this selection mechanism is much stronger with the occurrence of large exogenous shocks. This group behaviour not only plays a key role in explaining the dynamic properties of the data, revaluating the importance of expectations in driving economic fluctuations in the spirit of [PERSON] concept of animal spirits, but has important implications for the optimal control of policy in the spirit of the [PERSON] critique. Depending on intentions on the part of policymakers, the model suggests that different versions of policy can be designed and devised in a game between policymakers and the economy, with uncertainty as to which expectation formation is selected.