## 4 Approach to Perishable Inventory Management Problem
### Action Reduction for Value Iteration  
In this study, we use value iteration, a classic DP method, to solve MDP problems. Value iteration defines the value function \(V(s)\) for all states, representing the expected value of the state when following a given policy. First, the value function \(V(s)\) is updated for all states, as shown in Equation (14).  
\[V^{\prime}(s_{t})=\max_{a_{t}}\left[r(s_{t},a_{t})+\gamma\sum_{s_{t+1}}P(s_{t+ 1}\mid s_{t},a_{t})V(s_{t+1})\right] \tag{14}\]  
Here, \(  
continues until the difference between the old value function \(V(s_{t})\) and the new value function \(V^{\prime}(s_{t})\) becomes less than \(\theta\).  
\[|V^{\prime}(s_{t})-V(s_{t})|<\theta,\ \forall s_{t}\in S \tag{15}\]  
Once the updates are complete, the value function \(V(s_{t})\) is used to update the policy \(\pi(s_{t})\), as shown in Equation (16), to obtain the optimal policy.  
\[\pi(s_{t})=\arg\max_{a_{t}}\left[\sum_{s_{t+1}}P(s_{t+1}\mid s_{t},a_{t})(r(s_ {t},a_{t})+\gamma V(s_{t+1}))\right] \tag{16}\]  
In the value iteration method of DP, computation time increases as the size of the state-action space grows because each action's value should be evaluated. Therefore, this study reduces computation time by eliminating irrational actions. Irrational actions refer to pricing strategies where fresher items with longer shelf lives are priced lower than older items with shorter shelf lives. In other words, actions where the discount for items with longer shelf lives is larger, as shown in Equation (17), are targeted for reduction.  
\[P^{-}=\{p\in[0,1]^{m-1}\mid(p_{1}>p_{2}>\cdots>p_{m-1})\} \tag{17}\]  
Let \(A\) be the set of all actions and \(A(s)\) be the set of actions available in state \(s\). An action \(a=(q,p_{1},\ldots,p_{m-1})\) is an element of \(A\). We define \(A^{-}\) as the set of irrational actions to be eliminated, where \(A^{-}=\{a\in A\mid p_{1}>p_{2}>\cdots>p_{m-1}\}\).  
The algorithm for value iteration with action reduction is presented in Algorithm 2.  
```
Input:\(S,s_{0},A,P(s_{t+1}\mid s_{t},a_{t}),R(s_{t},a_{t}),\gamma\) Output: Optimal value function \(V(s)\) and policy \(\pi(s)\)  initialize \(V(s)\) for all \(s\); set \(\Delta\) to a large value; for\(s\in S\)do \(A(s)\gets A(s)\setminus A^{-}\);  end while \(\Delta\geq\theta\)do \(\Delta\gets 0\); for\(s\in S\)do  update \(V(s)\) with Equation (14); \(\Delta\leftarrow\max(\Delta,|V^{\prime}(s)-V(s)|)\);  end for for\(s\in S\)do  update \(\pi(s)\) with Equation (16);  end for  end
```  
**Algorithm 2**Value Iteration with Action Reduction  
This study refers to this method as ar-DP. As shown in Algorithm 2, when the action space is given, irrational actions within the action space are reduced using Equation (17). The actions that are eliminated, as mentioned earlier, are those where products with longer shelf lives are given a larger discount than those with shorter shelf lives. When such actions are selected, demand is heavily skewed toward the longer-shelf-life products, leading to an increase in unsold shorter-shelf-life products and a rise in waste. Thus, the optimal actions exclude these, as they result in a decrease in sales due to price reductions on fresh products that are easier to sell due to their longer shelf lives, along with the increased waste costs.  
As a result, this method is expected to reduce computation time while maintaining the optimality of the solution.