# Reinforcement Learning in a New Keynesian Model +
## 3 The Non-Linear NK Model
### Households  
Household \(j\) chooses savings between work and labour supply. Let \(C_{t}(j)\) be consumption and \(H_{t}(j)\) be the proportion of available work or leisure spent at the former. The single-period utility we choose, compatible with a balanced growth steady state, is  
\[U_{t}(j)=U(C_{t}(j),H_{t}(j))=\log(C_{t}(j))-\frac{H_{t}(j)^{1+\phi}}{1+\phi}\]  
and the value function of the representative household at time \(t\) dependent on its assets \(B\) is  
\[V_{t}(j)=V_{t}(B_{t-1}(j))=\mathbb{E}_{t}\left[\sum_{s=0}^{\infty}\beta^{s}U( C_{t+s}(j),H_{t+s}(j))\right] \tag{11}\]  
The household's problem at time \(t\) is to choose paths for consumption \(\{C_{t}(j)\}\), labour supply \(\{H_{t}(j)\}\) and holdings of financial savings to maximise \(V_{t}(j)\), given by (11), given its budget constraint in period \(t\)  
\[B_{t}(j)=R_{t}B_{t-1}(j)+W_{t}H_{t}(j)+\Gamma_{t}-C_{t}(j)-T_{t}-\frac{\varpi} {2}(B_{t-1}(j)-B)^{2} \tag{12}\]  
where \(B_{t}(j)\) is the given net stock of real financial assets at the end of period \(t\), \(W_{t}\) is the wage rate, \(T_{t}\) are lump-sum taxes, and \(\Gamma_{t}\) are profits from wholesale and retail firms owned by households. In order to allow for a wealth distribution by heterogenous agents introduced later and to achieve a stationary path for bond holdings, we introduce a portfolio adjustment cost (this as a modelling device similar to that used in open economies with home and foreign household is pioneered by Reference [26]. We examine the limit as \(\varpi\) becomes very small so that our choice of real rather than nominal bond holding costs is immaterial. The wealth distribution effect does not significantly change the equilibrium). \(R_{t}\) is the real interest rate paid on assets held at the beginning of period \(t\) given by \(R_{t}=\frac{R_{t-1}}{\Pi_{t}}RS_{t}\), where \(R_{n,t}\) and \(\Pi_{t}\) are the nominal interest and inflation rates, respectively, and \(RS_{t}\) is a risk premium shock. \(W_{t}\), \(R_{n,t}\), \(\Pi_{t}\) and \(\Gamma_{t}\) are all exogenous to household \(j\). As usual, all real variables are expressed relative to the price of the final output. The standard first-order conditions are  
\[\mathbb{E}_{t}[\Lambda_{t,t+1}(j)R_{t+1}] = 1+\varpi(B_{t}(j)-B)\] \[\frac{U_{H,t}(j)}{U_{C,t}(j)} = -W_{t}\]  
where \(\Lambda_{t,t+1}(j)\equiv\beta\frac{U_{C,t+1}(j)}{U_{C,t}(j)}\) is the stochastic discount factor for household \(j\), over the interval \([t,t+1]\). For our choice of utility function \(U_{C,t}=\frac{1}{C_{t}}\) and \(U_{H,t}=-H_{t}^{\phi}\), and these become  
\[\beta\mathbb{E}_{t}\bigg{[}\frac{C_{t}(j)R_{t+1}}{C_{t+1}(j)} \bigg{]} = 1+\varpi(B_{t}(j)-B) \tag{13}\] \[C_{t}(j)H_{t}(j)^{\phi} = W_{t}\Rightarrow H_{t}(j)=\bigg{(}\frac{W_{t}}{C_{t}(j)}\bigg{)} ^{\frac{1}{\phi}} \tag{14}\]  
The first-order conditions up to now are suitable for the RE solution. We now express the solution in a form suitable for moving from an RE to a learning equilibrium. We consider the limit as \(\varpi\to 0\). Solving (12) forward in time and imposing the transversality condition on debt, we can write  
\[B_{t-1}(j)=\mathrm{PV}_{t}(C_{t}(j))-\mathrm{PV}_{t}(W_{t}H_{t}(j))-\mathrm{ PV}_{t}(\Gamma_{t})+\mathrm{PV}_{t}(T_{t}) \tag{15}\]  
where the present (expected) value of a series \(X\equiv\{X_{t+i}\}_{i=0}^{\infty}\) at time \(t\) is defined by  
\[\mathrm{PV}_{t}(X_{t})\equiv\mathbb{E}_{t}\sum_{i=0}^{\infty}\frac{X_{t+i}}{R _{t,t+i}}=\frac{X_{t}}{R_{t}}+\frac{1}{R_{t}}\mathrm{PV}_{t}(X_{t+1}) \tag{16}\]  
writing \(R_{t,t+i}\equiv R_{t}R_{t+1}R_{t+2}\cdots R_{t+i}\) as the real interest rate over the interval \([t-1,t+i]\). The forward-looking budget constraint (15) holds for the representative household. If we allow RE and BR agents to borrow from or lend to one another, we must allow for \(B_{t-1}\
eq 0\).