## 2 Background Concept and Problem Modeling
### Reinforcement Learning  
Reinforcement learning aims to find an answer to a given MDP (Markov decision process) problem. MDP refers to the process by which an agent interacts with the environment through actions using the current state, and obtains a reward. As a basic method of solving such an MDP, an optimization-based method is used in which an agent interacts with a probabilistic environment, selects an action, and maximizes the sum of rewards [32]. MDP is usually defined in the form of \(\{S,A,P,r,\gamma\}\). \(S\) is the state set, \(A\) is the action set, \(P\) denotes the state transition probability, \(r(s,a)\) is the reward function, and \(\gamma\) is the discount factor [33]. The state transition probability \(P(s^{\prime}|s,a)\) represents the probability of transition to the next state \(s^{\prime}\) when the action \(a\in A\) is taken from the current state \(s\). The policy \(\pi(a|s)\) is the rule that determines how the agent interacts with the environment when it is in the current state \(s\). A policy can be expressed as a probability distribution or a deterministic function. The agent interacts with the environment at every time step. The agent selects the action \(a_{t}\in A\) according to the policy \(\pi:S\to A\) and the state \(s_{t}\in S\) and injects it into the environment. The environment returns the next state \(s_{t+1}\in S\) and the reward \(r_{t+1}\) to the agent according to the state transition probability \(P:S\times A\rightarrow[0,1]\) based on the current state and injected action. The purpose of reinforcement learning is to use this procedure to iteratively update the policy \(\pi(a|s)\) so that the agent can receive the maximum reward sum until it approaches the optimal policy \(\pi^{*}\). That is, if the policy is updated with the optimal policy, \(E_{\pi(a|s)}[\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1}]\)\(=E_{\pi^{*}}[\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1}]\) is established. There are two main ways to update the optimal policy. The first is value-based methods that update the optimal value function (i.e., the function that estimates the expected value of the reward sum) such as DQN (deep Q-network), DRQN (deep recurrent Q-learning for partially observable MDPs) using a neural network-like function approximation [34; 35]. The action is chosen based on the value function. The second method, called the policy gradient method, learns the optimal policy directly from the agent's experience. Policy-based learning is a method of determining the action \(a\) directly using the state \(s\), rather than using a value function to determine the action. Representative methods of policy-based learning are REINFORCE, the actor-critic method, DPG (deterministic policy gradient), DDPG (deep DPG), A3C (asynchronous advantage actor-critic), TRPO (trust region policy optimization), PPO (proximal policy optimization), MPO (maximum a posteriori policy optimization), D4 PG (distributed distributional DDPG), etc. [36; 37; 38; 39; 40; 41; 42; 43]. Reinforcement learning mostly uses a mini-batch method for learning. In order to perform batch learning, we need to collect many data samples. A sample usually consists of a state \(s_{t}\), an action \(a_{t}\), a reward \(r_{t+1}\), and the next state \(s_{t+1}\). Moreover, many reinforcement learning algorithms rely on techniques that use samples in various specific ways for efficient learning. Typically, the replay buffer, PER (prioritized experience replay), and HER are mainly used [34,44,45]. This paper employs SAC as a reinforcement learning algorithm with HER for path planning of a multi-arm manipulator with both fixed and moving obstacles [46].