## 3 Soft Actor-Critic Based Path Planning for Moving Obstacles with LSTM
### Path Planning for the Multi-Arm Manipulator and Augmented Configuration Space  
For the purpose of establishing MDP, the configuration space is defined first. Since there are \(n\) joints of each robot arm, the degree of freedom (DOF) of the robot arm manipulator is \(n\), and it is assumed that there are \(m\) robot arm manipulators. Therefore, as defined above, \(q_{t}^{i}\in\mathbb{R}^{n}\) means the set of \(n\) joint angle values of the \(i\)th robot arm at the \(t\)th time step of the SAC training samples. Expanding this, \(q_{t,j}^{i}\in\mathbb{R}^{n}\) is a component of \(q_{t}^{i}\in\mathbb{R}^{n}\), which means the \(j\)th joint angle value of the \(i\)th robot arm at the \(t\)th time step. It is difficult to define the configuration space for the path planning algorithm of the multi-arm manipulator in an environment with time-varying obstacles. This is because the manipulators influence each other on the definitions of \(\mathbf{Q}_{\text{sim}}^{i}\) and \(\mathbf{Q}_{\text{collide}}^{i}\) that define the configuration space of the manipulator because \(\mathbf{Q}_{\text{sim}}^{i}\) changes every moment while the manipulator is operating. In other words, it can be viewed as an environment in which there are more obstacles that do not know how to move in addition to the fixed and moving obstacles. Therefore, it is recommended to define multiple manipulators as one virtual manipulator in order to make the definition of the configuration space easier [51,52]. Namely, we define a multi-arm manipulator as a single virtual manipulator with \(n\times m\) joints. The new configuration space is as follows:  
\[q_{t}=\left[\begin{array}{c}q_{t}^{1}\\ \vec{q}_{t}^{2}\\ \vdots\\ q_{t}^{m}\end{array}\right]\in\mathbb{R}^{nm},\;\;q_{t}^{i}=\left[\begin{array} []{c}q_{t,1}^{i}\\ \vec{q}_{t,2}^{i}\\ \vdots\\ \vec{q}_{t,m}^{i}\end{array}\right]\in\mathbb{R}^{n},\;\;i\in\{1,2,\cdots,m\}. \tag{1}\]Therefore, the newly defined augmented configuration space is expressed as \(\mathbf{Q}^{a}:=\mathbf{Q}^{a}_{\text{two}}\bigcup\mathbf{Q}^{a}_{\text{single}}\) and is as follows.  
\[\mathbf{Q}^{a}_{\text{two}}=\underbrace{\mathbf{Q}^{1}_{\text{two}}\times\cdots \times\mathbf{Q}^{a}_{\text{two}}}_{\text{m times}}\subset\mathbb{R}^{nm}\;\text{ and}\;\;\mathbf{Q}^{a}_{\text{single}}=\underbrace{\mathbf{Q}^{1}_{\text{single}}\times\cdots\times \mathbf{Q}^{n}_{\text{single}}}_{\text{m times}}\subset\mathbb{R}^{nm},\]  
In the expression, \(\mathbf{Q}^{a}_{\text{two}}\) means the space where there is no collision in the augmented configuration space \(\mathbf{Q}^{a}\), and \(\mathbf{Q}^{a}_{\text{single}}\) means the space where the collision occurs in augmented the configuration space \(\mathbf{Q}^{a}\). In this case, it is necessary to reinterpret the path planning problem of the multi-arm manipulator using an augmented configuration space. We redefine it as a path planning problem for a single virtual manipulator has the start point \(q_{0}\in\mathbf{Q}^{a}_{\text{two}}\) and the goal point \(q_{\text{goal}}\in\mathbf{Q}^{a}_{\text{two}}\) in the augmented configuration space.  
\[q_{0}=\left[\begin{array}{c}q^{1}_{\text{two}}\\ q^{2}_{\text{two}}\\ \vdots\\ q^{m}_{\text{two}}\end{array}\right],\quad q_{\text{goal}}=\left[\begin{array} []{c}q^{1}_{\text{goal}}\\ q^{2}_{\text{goal}}\\ \vdots\\ q^{m}_{\text{goal}}\end{array}\right].\]  
In the expression, \(q^{i}_{\text{two}}\) and \(q^{i}_{\text{goal}}\) mean the starting point and goal point of the \(i\)th robot manipulator.