## 3 Bridge Element Maintenance Decision Optimization Model
### Sequential Decision Making Model for Bridge Project-Level Maintenance Decisions
#### 3.2.4 Reward Set (Cost Matrix) and Discount Factor  
The reward set corresponds to the costs incurred by executing maintenance plans on bridge elements. In this study, reference is made to the \"Regulations for Budgeting of Highway Maintenance\" (JTG 5610-2020) [27] and \"Budget Quota for Highway Bridge Maintenance Engineering\" (JTG/T 5612-2020) [28]. The cost of maintenance measures should include labor costs, material costs, other direct and indirect costs, the management fees of the construction unit, preliminary work expenses for the construction project, as well as reserved costs and other initial construction costs. In this paper, the budget unit price method based on the work process of maintenance measures is used to determine the direct cost unit price of each maintenance measure. This method involves calculating the unit prices of materials and the labor involved in the maintenance process to estimate the direct costs of the maintenance measures. \[UP_{x}=\sum_{j=1}^{n}UP_{j}, \tag{3}\]  
where \(UP_{x}\) represents the cost unit price of maintenance measure \(x\) and \(UP_{j}\) denotes the base price of the \(j\)th work process in the description of maintenance measure \(x\). The specific base prices for the maintenance work processes can be referred to in the \"Budget Quota for Highway Bridge Maintenance Engineering\" (JTG/T 5612-2020), where \(n\) represents the total number of work processes included in the description of maintenance measure \(x\), as shown in Table 2.  
Due to the fact that the maintenance plans defined in this paper are composed of a series of maintenance measures, it is necessary to develop specific maintenance plans that are based on the actual condition of the bridge and the specific defects present. By determining the corresponding maintenance measures for each identified defect and substituting them into Equation (3), we can estimate the unit cost \(UP_{x}\) of the maintenance measures. This will provide an estimation of the maintenance costs for each identified defect location. \[c_{l_{k},m}=Q_{l_{k},m}\times UP_{x,m}, \tag{4}\]  
where \(c_{lk,m}\) represents the maintenance cost required for repairing the defect m on element \(k\), \(Q_{lk,m}\) represents the total number of defect m on element \(k\), \(UP_{x,m}\) represents taking action \(x\) for defect m, and the unit cost of action \(x\) is calculated according to Equation (3). The specific cost incurred by element \(k\) when executing maintenance plan \(A_{i}\) is calculated according to the following equation:  
\[C_{l_{k},A_{i}}=\sum_{m=1}^{r}c_{l_{k},m}, \tag{5}\]  
where \(C_{l_{k},A_{i}}\) represents the cost incurred by element \(k\) when executing maintenance plan \(A_{i}\), which includes repairing r defects on that element. After calculating the required costs for different maintenance plans on each element, a maintenance cost matrix, also known as the reward matrix \(R\), can be formed. In the context of SDPs, the objective is to find a maintenance strategy that maximizes the reward (and minimizes the cost). In this case, the reward can be defined as the negative value of the maintenance plan cost, and this allows us to solve for the optimal strategy that minimizes costs or maximizes the reward. The calculation formula is as follows:  
\[C_{A_{i},s}=\sum_{k=1}^{Q_{s}}C_{I_{k},A_{i}} \tag{6}\]  
\[R= \begin{bmatrix}-\frac{C_{A_{0},1}}{Q_{1}}&-\frac{C_{A_{0},2}}{Q_{2}}&-\frac{C_ {A_{0},3}}{Q_{3}}&-\frac{C_{A_{0},4}}{Q_{4}}&-\frac{C_{A_{0},5}}{Q_{5}}\\ -\frac{C_{A_{1},1}}{Q_{1}}&-\frac{C_{A_{1},2}}{Q_{2}}&-\frac{C_{A_{1},3}}{Q_{3} }&-\frac{C_{A_{1},4}}{Q_{4}}&-\frac{C_{A_{0},5}}{Q_{5}}\\ \vdots&\vdots&\vdots&\vdots&\vdots\\ -\frac{C_{A_{1},1}}{Q_{1}}&-\frac{C_{A_{2},2}}{Q_{2}}&-\frac{C_{A_{2},3}}{Q_{ 3}}&-\frac{C_{A_{1},4}}{Q_{4}}&-\frac{C_{A_{1},5}}{Q_{5}}\end{bmatrix} , \tag{7}\]  
where \(Q_{s}\) represents the quantity of elements in state \(s\).