## 3 Methods and Materials
### Markov Decision Process  
Markov decision processes are a long-studied formalism for decision making [41]. In an MDP, the agent solves several decision problems in sequence, where each current decision influences the solution of the following problem [41; 42]. This sequential nature of the decisions is typically found in probabilistic planning problems, which generalize shortest-path algorithms in a stochastic environment. The objective of solving an MDP is to find an optimal policy that associates an optimal action with any state [42]. Solving a Markov decision problem can be thought of as controlling the agent to behave optimally, maximizing an average accumulated income in the long run. The solutions are policies that specify the action to be taken in each possible state, assuming that the agent possesses a perfect knowledge of the process and its state at all times. MDP has been long used in robotics for:  
* Task planning in a probabilistic environment, as analysed in [42]. * Allocating tasks between different robots, as investigated in [43]. * Managing failure scenarios with multiple robots. For example, in [44], failure reconfigurability comes with re-allocating tasks between robotic platforms. If a robot fails, another is ready to take on the task. MDP follows the Markov property: all information relevant to the prediction of the future is contained in the present state. This property is referred to as memorylessness by experts: the action leading to the next state can be decided based on the system's current state, as if the process's complete history is known. For the application of this paper, MDP is used instead of a partially observable Markov decision process (POMDP). The assumption is that during the initial design studies, complete knowledge is given to the agent about its state and environment. For the planetary rover, the environment is less dynamic and changing than autonomous systems on Earth. This assumption can stand during the initial conception phases. However, when more dynamic robotic systems, such as flying bots, are considered, looking toward POMDP can be a good solution. At the same time, failure models would benefit from a POMDP model as soon as sensors and monitoring equipment are defined. Most of the process in robotics falls into this category, like the work in [45]. An MDP problem can be easily solved if the state transition probabilities are known. The state transition functions depend only on the current state, not on how that state was reached. The solution of an MDP is to find an optimal policy. A policy is a mapping from states to actions. An optimal policy should ensure that, for a given state, no other action would provide a higher sum of discounted future rewards. Beyond classical MDP-solving techniques in planning, such as policy or value iteration, it is possible to define an optimal policy leveraging reinforcement learning. These techniques are used if the problem transition matrix is complex to compute. These strategies can be considered as planning with a simulator-type model. A simulator samples the following possible states starting from a current state and a reward function [46]. Those methods are called off-policy, as the values of the state transition probabilities are learnt with many trials and errors. Hence, the optimal policy can be learnt by the reinforcement learning algorithm. In [46], one of the identified drawbacks of this solving method is the significant degree of knowledge engineering spent in tweaking various parameters and somehow suggesting the solver of some optimal decision strategies. In the application of this study, this drawback is tackled by defining the set of parameters that influence the functional architecture and the system operations from the MBSE model. The model can include flight rules to decrease the algorithm's search space, such as communicating data only in specific time slots or when the memory is full of a defined amount. Suppose the application is a movable planetary system exploring a particular site of interest.