# Reinforcement Learning in a New Keynesian Model +
## 4 AU Learning and Market-Consistent Information  
With anticipated utility (AU) learning, our learning model is one where agents make fully optimal decisions, given their individual specification of beliefs, but have no macroeconomic model to form expectations of aggregate variables. We draw a clear distinction between aggregate and internal quantities so that identical agents in our model are not aware of this equilibrium property (nor any others). To close the model, we need to specify the manner in which households and firms form their expectations. To do so, we assume that variables which are local to the agents, in a geographical sense, are observable within the period, whereas variables that are strictly macroeconomic are only observable with a lag. This categorisation regarding information about the current state of the economy follows Reference [29], which distinguishes between the local information that agents acquire directly through their interactions in markets and statistics that are collected and summarised, usually by governments, and are made available to the wider public. (This paper actually focuses on a third category, information provided by the news media, and allows for imperfect information in the form of noisy signals, issues which go beyond the scope of our paper.) The policy rate is announced by the central bank; thus, it is observed without a lag, and it is common knowledge. Given this, we assume an adaptive expectations forecasting rule given below by (38) and (39) about variables external to agents' decisions. Let \(x_{t}=r_{t},r_{n,t},\pi_{t},w_{t},\gamma_{t}\), then household expectations are given by  
\[\mathbb{E}_{t}^{*}x_{t+i}=\mathbb{E}_{t}^{*}x_{t+1}\,;\quad i\geq 1 \tag{37}\]  
Expressing \(\mathbb{E}_{t}\omega_{1,t+1}\) and \(\mathbb{E}_{t}\omega_{2,t+1}\) in (29) as forward-looking summations and using (37), we arrive at the individual learning consumption equation  
\[\alpha_{1}c_{t} = \alpha_{2}w_{t}+\alpha_{3}(\omega_{2,t}+r_{t})+\alpha_{4}\omega_{ 1,t}\] \[\omega_{1,t} = \frac{1}{1-\beta}\Big{[}\alpha_{5}\mathbb{E}_{t}^{*}w_{t+1}- \alpha_{6}(\beta\mathbb{E}_{t}^{*}r_{n,t+1}-\mathbb{E}_{k,t}^{*}\pi_{t+1}) \Big{]}-\alpha_{6}r_{n,t}\] \[\omega_{2,t} = (1-\beta)(\gamma_{t}-g_{t})-r_{t}+\frac{\beta}{1-\beta}((1-\beta )(\mathbb{E}_{t}^{*}\gamma_{t+1}-\mathbb{E}_{t}^{*}\delta_{t+1})-\mathbb{E}_{ t}^{*}r_{t+1})\]  
which is now expressed in terms of one-step ahead forecasts by  
\[\mathbb{E}_{t}^{*}x_{t+1}=\mathbb{E}_{t}^{*}x_{t}+\lambda_{x}(x_{t-j}- \mathbb{E}_{t}^{*}x_{t})\,;\quad x=w,\,r_{n},\,\pi,\,\gamma\,;\quad j=0,1 \tag{38}\]  
Households make inter-temporal decisions for their consumption and hours supplied given adaptive expectations of the wage rate, the nominal interest rate, inflation and profits. These macro-variables may in principle be observed with or without a one-period lag (\(j=1,0\)), but as stated earlier, we assume \(j=0\) for market-specific variables \(w_{t},\,\gamma_{t}\), and \(j=1\) for aggregate inflation \(\pi_{t}\). However, we assume that the current nominal interest rate, \(r_{n,t}\), is announced and therefore is observed without a lag. We distinguish household and firm expectations \(\mathbb{E}_{h,t}^{*}\pi_{t+1},\,\mathbb{E}_{f,t}^{*}\pi_{t+1}\). Then, for retail firm \(m\)  
\[\mathbb{E}_{t}^{*}\pi_{t+i+1} = \mathbb{E}_{t}^{*}\pi_{t+1}\,;\quad i\geq 0\] \[\mathbb{E}_{t}^{*}(mc_{t+i}+ms_{t+i}) = \mathbb{E}_{t}^{*}(mc_{t+1}+ms_{t+1})\,;\quad i\geq 1\] \[p_{t}^{o}(m)-p_{t} = \frac{\beta\xi}{1-\beta}\mathbb{E}_{f,t}^{*}\pi_{t+1}+(1-\beta \xi)(mc_{t}+ms_{t})+\frac{\beta}{1-\beta}\mathbb{E}_{t}^{*}(mc_{t+1}+ms_{t+1})\]  
where one-step ahead forecasts are given by the adaptive expectations rule  
\[\mathbb{E}_{t}^{*}x_{t+1}=\mathbb{E}_{t}^{*}x_{t}+\lambda_{x}(x_{t-j}- \mathbb{E}_{t}^{*}x_{t})\,;\quad x=\pi,\,(mc+ms);\quad j=0,1 \tag{39}\]  
Retail firms make inter-temporal decisions for their price and output given adaptive expectations of the aggregate inflation rate and their post-shock real marginal shock wagerate.