## 3 Self-Recovery Method
### Self-Learning Mudskipper-Inspired Crawling Algorithm (SLMIC)
#### 3.3.1 Q-Learning Algorithm  
Reinforcement learning can be simply explained as the process of training pets, such as dogs and cats, to perform certain actions. However, this process can be described as a trial-and-error problem as well. In the beginning of a dog's training process, it seems impossible for a dog to perform a requested task without error. The dog will try to perform some action randomly, such as walking, sitting, or jumping, to get a snack. After it gets the snack, the dog will remember the action that can possibly get it a snack again. By contrast, the dog will not do any action that would lead to an adverse outcome such as a scolding or a beating. Accordingly, the process of reinforcement learning can be thought to be based on reward and punishment. In this study, the quadruped robot is controlled using Q-learning and mudskipper-inspired behavior. The control policy will be adapted with the current state, provided by the surrounding environment, following the Q-learning approach. For example, if the robot is in a state in which it is heading north, but we command it to go to the east, the control policy will be changed to make the robot go to the east. The three main parameters of reinforcement learning are as follows:  
* State (\(S\)): It is defined as the current scenario of a system, for instance, the position of the robot.
* Action (\(A\)): In one system, several actions would be required to be conducted in each state. In wheel-based robots, the actions can be moving forward, turning left, and turning right.
* Reward (\(R\)): It depends on the current state and action. It can be positive, negative, or zero for the win, lose, and draw scenarios, respectively. For example, when a robot encounters an obstacle, it needs to avoid the obstacle. If the robot decides to move forward and hit an obstacle, it will get a negative outcome in the form of a punishment. On the contrary, if the robot avoids an obstacle properly, it will receive a positive reward.  
Finally, the objective of learning, which is known as policy (\(\pi\)), will be achieved. The best policy is the selection of actions that provide the highest reward, the so-called \"Maximum Sum of Expected Rewards\".  
Q-learning is a reinforcement learning approach with non-model requirements. It employs the concept of Markov decision process (MDP) with finite state arrays to arrive at the optimal policy [27]. The expected reward will be stored in a \(d\)-dimensional state-action array. The number of \(d\) can be set manually by the user. Q-learning is one algorithm among the various algorithms associated with the temporal-difference method. The Q-values are acquired as follows:  
\[V^{*}(s)=\max_{a}Q(s,a). \tag{19}\]  
Because the Q-function does not need a model for learning and selecting actions, no model is required for state transitions. \(Q\) is updated by using the new information obtained after performing an action to correct the old policy. The Q-values are updated numerically based on the temporal-difference concept using Equation (20) [28]:  
\[Q(s,a)=Q(s,a)+\alpha(R(s)+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime})-Q(s, a)), \tag{20}\]  
where \(\alpha\) and \(\gamma\) are the learning rate and the discount factor, respectively. The procedure of the Q-algorithm is summarized in Table 2. The proposed method employs both Q-learning and mudskipper-inspired behavior to control the damaged robot. The controller structure is illustrated in Figure 13.