## 3 Bridge Element Maintenance Decision Optimization Model
### Sequential Decision Making and Solving  
The reward function represents the cumulative discounted rewards starting from time \(t\) (the value of all rewards at the current time). In the context of bridge maintenance decision making, this corresponds to the present value of all maintenance plan costs. \[G_{t}=R_{t}+\gamma R_{t+1}+\ldots=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k}, \tag{12}\]  
where \(R_{t}\) represents the immediate reward obtained by taking a maintenance plan at time \(t\), which corresponds to the cost incurred by the maintenance plan in that year. By substituting the formula of the state value function into the reward function, we can derive the Bellman equation and solve for the state value function:  
\[v(s)=\mathrm{E}[G_{t}\mid S_{t}=s]\] \[=\mathrm{E}\big{[}R_{t+1}+\gamma R_{t+2}+\gamma^{2}R_{t+3}+\ldots \mid S_{t}=s\big{]}=\mathrm{E}[R_{t+1}+\gamma G_{t+1}\mid S_{t}=s]. \tag{13}\] \[=\mathrm{E}[R_{t+1}+\gamma v(S_{t+1})\mid S_{t}=s]\]  
As per the Bellman equation, the value of a state is composed of the expected immediate reward and the expected value of the next state. Therefore, by maximizing all action value functions, we can obtain the optimal strategy:  
\[\pi_{*}(a\mid s)= \begin{cases}1&\text{ if }a=\underset{a\in A}{\mathrm{argmax}}q_{*}(s,a )\\ 0&\text{ otherwise}\end{cases} . \tag{14}\]  
Dynamic programming is an algorithm used to solve SMDP problems when the model is known. It can be used to evaluate a given policy and iteratively obtain the optimal value function. The specific approaches include policy iteration and value iteration. Compared to value iteration, policy iteration tends to converge faster in large-scale problems. This paper adopts a policy iteration method to solve for the optimal maintenance strategy, as illustrated in Figure 2. The solution process mainly consists of two steps: policy evaluation and policy improvement. In the stage of policy evaluation, the method of value function iteration can be used to simplify the computation. Firstly, the value function of each bridge state under the current maintenance strategy is estimated. The state value function is used to evaluate the superiority of the policy, where a larger value indicates a better policy. Specifically, the value function \(v^{\pi}(s,t)\) for each bridge state is defined as the expected cumulative reward that can be obtained starting from state \(s\) and after time \(t\). The value function of bridge states is recursively calculated using the Bellman equation:  
\[v^{\pi}(s,t)=R(s)+D_{T}\cdot\pi(a|s)\cdot T(s,s^{\prime},t^{\prime})\cdot\sum \gamma\cdot v^{\pi}(s^{\prime},t^{\prime}+t), \tag{15}\]  
where \(R(s)\) represents the immediate maintenance cost starting from bridge state \(s\); \(D_{T}\) is the state distribution vector; \(\pi(a|S)\) represents the probability of taking action a given the input state \(s\), where action a represents a maintenance strategy \(a\); \(T(s,s^{\prime},t^{\prime})\) represents the time distribution required to transition from state \(s\) to state \(s^{\prime}\); and \(\gamma\) is the discount factor that controls the importance of future rewards. The value is equal to the ratio of the existing maintenance funds to the estimated cost of repairing all diseases, and it also represents the adequacy of current maintenance funds. \(v(s,t^{\prime}+t)\) represents the value function of state \(s^{\prime}\), which is reached by state \(s\) after time \(t^{\prime}\). The above equation is a recursive equation that requires starting with an initial estimate of the value function for each state and gradually updating it until convergence is achieved. In the stage of policy improvement, the current maintenance strategy \(\pi\) should be improved based on the current state value function \(v^{\pi}\)(s,t) in order to make the maintenance strategy \(\pi\) more optimal.