# Resilience Analysis for Double Spending via Sequential Decision Optimization
## 5 Decisions under Uncertainty: Optimal Switching and Stopping  
Sequential decision-making arises in many applications and is usually addressed under the framework of discrete-time _Stochastic Control_. The theory of _Markov Decision Processes/Dynamic Programming_ provides a variety of methods to deal with such questions. In generic situations, approaching solutions even for simplest decision processes may be a cumbersome process (ref. [18; 19; 20]). However, for the questions formulated in the present work, a specific truncation technique will be applied to state the problem on a finite space within a finite time horizon, which makes all results obtainable by finite number of algebraic operations at machine precision. Let us introduce a particular Markov decision problem class: The _optimal stochastic switching_ (see [21]). On a finite time horizon \(\{0,1,\ldots,T\}\) consider an agent concerned with the problem of sequential decision-making: At any time \(t=0,1,\ldots,T-1\) an action \(a\in A\) from a finite set \(A\) of all available actions must be chosen. This decision returns an immediate reward/costs but also influences the future state evolution, i.e., at any time, an action optimally balances between the current rewards/costs of control and all future situations. In the framework of optimal stochastic switching, the decision variable has two components \((p,z)\in E=P\times\mathbb{R}^{d}\) consisting of operation mode \(p\) and environment state \(z\), thus the state space \(E\) is a Cartesian product of a finite set \(P\) all operation modes and the Euclidean space \(\mathbb{R}^{d}\). Therefore, the evolution \((Z_{t})_{t=0}^{T}\) of the second component is supposed to follow a Markov process with the interpretation that \(Z_{t}=z\) is the situation in the global environment at time \(t\) which is relevant for making decisions but cannot be changed by an agent's actions. Contrary to this, the current operation mode \(p\in P\) is under full deterministic control of the agent at any time. This aspect is modeled in terms of a function \(\alpha:P\times A\to P\), \((p,a)\rightarrow\alpha(p,a)\), which describes a deterministic change of the operation mode by the agent's actions with the interpretation that \(\alpha(p,a)\in P\) is the new mode if the action \(a\in A\) was taken in the previous mode \(p\in P\). Now, let us specify the control costs. Assume that taking an action \(a\in A\) causes an immediate reward \(r_{t}(p,z,a)\) which depends on the state \((p,z)\in E\) and on the action \(a\in A\) through given reward functions \(r_{t}:E\times A\rightarrow\mathbb{R}\) which may be time dependent. When the system arrives at the last time step \(t=T\) in the state \((p,z)\in E\), the agent collects the _scrap value_\(r_{T}(p,z)\), described by a pre-specified scrap function \(r_{T}:E\rightarrow\mathbb{R}\). At each time \(t=0,\ldots,T-1\) the _decision rule_\(\pi_{t}\) is given by a mapping \(\pi_{t}:E\to A\), prescribing at time \(t\) in the state \((p,z)\in E\) the action \(\pi_{t}(p,z)\in A\). A sequence \(\pi=(\pi_{t})_{t=0}^{T-1}\) of decision rules is called a _policy_. When controlling the system by the policy \(\pi=(\pi_{t})_{t=0}^{T-1}\), the positions \((p_{t}^{\pi})_{t=0}^{T}\) and the actions \((a_{t}^{\pi})_{t=0}^{T-1}\) evolve recursively  
\[a_{t}^{\pi}=\pi_{t}(p_{t}^{\pi},Z_{t}),\qquad p_{t+1}^{\pi}=\alpha(p_{t}^{\pi},a _{t}^{\pi}),\qquad t=0,\ldots,T-1.\]  
Having started at initial values \(p_{0}^{\pi}=p_{0}\) and \(Z_{0}=z_{0}\), the goal of the controller is to maximize (over all possible policies) the expectation of the total reward  
\[v_{0}^{\pi}(p_{0},z_{0})=\mathbb{E}\left(\sum_{t=0}^{T-1}r_{t}(p_{t}^{\pi},Z_{ t},a_{t}^{\pi})+r_{T}(p_{T}^{\pi},Z_{T})\right).\]  
The function \((p,z)\mapsto v_{0}^{\pi}(p,z)\) is called the value of the policy \(\pi\) and represents the total reward accumulated within the entire time. For technical details and solution algorithms to switching systems, we refer the interested reader to [22].