## 4 Problem Formulation  
Formally, the skeleton-format input to the action recognition and motion prediction modules is represented by 3D joint positions of \(m\) joints for a sequence of frames recorded over time, which is the same format as in [35]. The action pose at time (frame) \(t\) is \(\mathbf{p}(t)\in\mathbb{R}^{3m}\). The observed time length is \(t\in[1,\tau_{o}]\) and the unobserved time length is \(t\in[\tau_{o}+1,\tau_{o}+\tau_{p}]\) for prediction. For a single human action sequence belonging to a set of \(N_{a}\) possible actions, \(\mathcal{A}=\{1,\dots,N_{a}\}\), we have the observed and future motion matrices, \(\mathbf{P}_{\text{prev}}\) and \(\mathbf{P}_{\text{fut}}\), where \(\mathbf{P}_{\text{prev}}=[\mathbf{p}(1),\cdots,\mathbf{p}(\tau_{o})]\in\mathbb{R}^{\tau_ {o}\times 3m}\) and \(\mathbf{P}_{\text{fut}}=[\hat{\mathbf{p}}(\tau_{o}+1),\cdots,\hat{\mathbf{p}}(\tau_{o}+ \tau_{p})]\in\mathbb{R}^{\tau_{p}\times 3m}\) and the action classification label encoded as a one-hot vector \(\mathbf{y}\in\{0,1\}^{N_{a}}\). The predicted human pose at time \(t\) is denoted as \(\hat{\mathbf{p}}(t)\in\mathbb{R}^{3m}\), which can be divided into the right hand's position \(\hat{\mathbf{p}}_{rh}(t)\in\mathbb{R}^{3}\) as the target position for the robot and the remaining body joints' positions \(\hat{\mathbf{p}}_{o}(t)\in\mathbb{R}^{3(m-1)}\) to identify the positions of obstacles. Human action recognition and motion prediction are achieved using the two-stage deep learning computer vision approach introduced in [35], as depicted in Figure 2. The 3D-skeleton data stream provided by the OptiTrack motion capture system is first processed by a bidirectional LSTM and a GCN architecture incorporating the attention mechanism to extract spatial-temporal features of the human motion dynamics. The generated feature map, denoted as \(\mathbf{\
abla}\), is then fed simultaneously to two modules, one that performs prediction and one that performs recognition. The motion prediction task is achieved using an LSTM-based decoder, while the action recognition task is performed using a CNN followed by conditional random fields (CRF). Initially, the models are trained offline using collected HRC task samples. Following training, streaming data are fed to the model to generate the predicted future pose sequence \(\hat{\mathbf{P}}_{\text{fut}}\) and predicted action category \(\hat{y}\). Denoting the feature extraction, motion prediction and action recognition functions within the Deep Neural Network model as \(\mathbf{f}_{\text{ext}}\), \(\mathbf{f}_{\text{pred}}\) and \(\mathbf{f}_{\text{recg}}\), respectively, and \(\mathbf{\theta}_{\text{ext}}\), \(\mathbf{\theta}_{\text{pred}}\) and \(\mathbf{\theta}_{\text{recg}}\) as their trainable parameters, the extracted feature map \(\mathbf{\Psi}\), the estimated future motion \(\hat{\mathbf{P}}_{\text{fut}}\) and one-hot vector class category \(\hat{y}\) can be expressed as  
\[\mathbf{\Psi} =f_{\text{ext}}(\mathbf{P}_{\text{prev}};\mathbf{\theta}_{\text{ext}}), \tag{1a}\] \[\hat{\mathbf{P}}_{\text{fut}} =f_{\text{pred}}(\mathbf{\Psi};\mathbf{\theta}_{\text{pred}}),\] (1b) \[\hat{y} =f_{\text{recg}}(\mathbf{\Psi};\mathbf{\theta}_{\text{recg}}). \tag{1c}\]  
If the predicted action is one that requires robot interaction, the predicted final position of the right hand (denoted \(\hat{p}_{\text{fut}}(\tau_{p})\)) is set as the target interactive position for the robot end-effector and the predicted final position of the other joints are set as the center of capsule-format obstacles.