# A Parallel Monte Carlo Algorithm for the Life Cycle Asset Allocation Problem
## 2 Materials and Methods
### Description of Monte Carlo-Based Algorithm for Solving the Bellman Equation  
We first come up with this Monte Carlo-based algorithm to solve this pension problem, making use of the idea of reinforcement learning and making it possible and applicable to make the solving process to parallelization. We consider the proportion of risky asset \(\alpha_{t}\) as the rows, and therefore we discrete it as 20 intervals. In addition, we assume the consumption rate \(C_{t}\) as the columns, and discrete it as 20 intervals. The state \(s\) at age \(t\) is (\(\alpha_{t}\), \(C_{t}\)). The action \(a\) represents the movement (right, left, up, and down). To make the process more robust, we make the movement continuously with one interval a time, the next state at age \(t+1\) is (\(\alpha_{t+1}\), \(C_{t+1}\)). Each reward, where the utility \(U_{t}\) comes from, is the function of \(C_{t}\). When a state is encountered, choosing a certain behavior with a certain probability to reach the next state and obtaining the corresponding reward is a randomness strategy. There must be at least one optimal strategy among all the strategies. The sum of the multiple subsequent rewards is the payoff. It is not appropriate enough to measure the goodness of a strategy in terms of rewards. Specifying a state, taking a stochastic strategy, and then weighting the rewards by an average--which is the expectation--gives the state value function. In addition to this, there is another type of value function, which is the state action value function. Our solution algorithm is formulated in Algorithm 1:  
```
1: Initialization \(Q(s=(\alpha,C),a)\gets 0,\;\forall s\in S,a\in A\)
2: Set \(N(s=(\alpha,C),a)\gets 0,\;\forall s\in S,a\in A\)
3: Set \(\epsilon\gets 1\), \(k\gets 1\)
4:\(\pi_{k}\leftarrow\epsilon\text{-greedy}(Q)\)
5: For \(k<\;1000\) do
6: Generate episode \(E_{k}=(s_{1},a_{1},r_{2},\ldots s_{T})\) following \(\pi_{k}\), while for \(start\leq t\leq retirement\;C_{t}<Y_{t}\), for \(retirement\leq t\leq T,C_{t}\leq\
icefrac{{(1-\alpha_{t})\times W_{t}/\alpha_{t} }}{{\alpha_{t}W_{t}}}+\alpha_{t}W_{t}\)
7: for each \((s_{t},a_{t})\in E_{k}\) do
8: \(N(s_{t},a_{t})\gets N(s_{t},a_{t})+1\)
9: \(G_{t}=r_{t+1}+\tau\tau_{t+2}+\ldots+\gamma^{k}r_{t+1+k}\)
10: \(Q(s_{t},a_{t})\gets Q(s_{t},a_{t})+\frac{1}{N(s_{t},a_{t})}(G_{t}-Q(s_{t},a _{t}))\)
11: end for
12: \(k\gets k+1\)
13: \(\epsilon\leftarrow\frac{1}{k}\)
14: \(\pi_{k}\leftarrow\epsilon\text{-greedy}(Q)\)
15: end for
```  
**Algorithm 1**Monte Carlo Method for Solving the Bellman Equation  
To address high-dimensional and iterative problems, we used several methods. State space discretization was our first approach. We divided the state space (pension wealth) into a finite number of points, transforming a continuous problem into a discrete one. Thisreduces computational complexity. The Monte Carlo method simulated many future state paths. It estimates expectations in high-dimensional spaces. Dynamic programming was also applied, which iterates backward from the endpoint to the starting point. This progressively solves the value function, ensuring the optimal strategy at each step. We used interpolation methods, like linear interpolation, to smooth transitions between state points. This maintains accuracy in the value function calculations. Consistent results were ensured by setting the same random seed for Monte Carlo simulations. In setting the state space, we avoided computational issues by starting from 0.01 instead of 0. This prevents problems like division by zero. The range ends at 1000, making calculations and interpretations easier. We generated 100 state points to capture wealth changes accurately. This balance ensures sufficient granularity without overwhelming computational resources. We chose pension wealth for the state space because it influences consumption and investment decisions, which determines how much members can consume and invest. Pension wealth also affect the trade-offs between risk and return.