# Motion Capture Research: 3D Human Pose Recovery Based on RGB Video Sequences
## 2 Proposed Method
### Human-Body Pose Generation Based on Video Streams  
The end-to-end pose generation method is only for the input of a single image. When the input is a video sequence, this method cannot consider the correlation between the connected frames, resulting in a large jitter in the generated 3D model sequence. In addition, the video sequence can solve the problem that human-body parts are occluded in a certain frame. So taking advantage of video sequences during the inference phase will be able to generate a smoother and jitter-free parameterized SMPL model. The process of human-body pose generation based on a video stream is shown in Figure 5. The encoding network is known, and the three-iteration regression networks in the generated network (G) are referred to as the decoding network. In the smooth solution, we use the method of optimizing the 2048-dimensional feature space obtained by the encoding network \(\mathbf{Z}=\left\{\vec{z}_{1},\vec{z}_{2},\ldots,\vec{z}_{N}\right\}\). Specifically, the network parameters after the pre-training are kept constant, and the feature space \(\mathbf{Z}\) obtained by the encoding network is directly optimized. In this way, the feature of the video sequences can be obtained by using forward propagation, and then the feature space of the entire video sequence is used as the parameter to be optimized, and the loss function is inversely propagated for optimization. So that this solution process only needs to be imported into the decoder network, which is small in scale and can optimize the feature space of about 1000 pictures at a time. First, Openpose is used to extract 2D key points in each frame \(\left\{\hat{\mathbf{\pi}}_{1},\hat{\mathbf{\pi}}_{2},\ldots,\hat{\mathbf{\pi}} _{N}\right\}\), \(N\) is the number of video frames. The most intuitive 2D skeleton point constraint is calculated as follows:  
\[loss_{2d}=\sum_{t}\sum_{j}c_{t,j}||\hat{\mathbf{\pi}}_{t,j}-\Pi\left[J_{j} \left(f_{d}\left(\vec{z}_{t}\right)\right)\right]||_{1} \tag{11}\]  
where \(t\) is the serial number of each frame, \(j\) is the serial number of each key point, \(f_{d}\) is the decoding function, \(J\) is the human-body key-point prediction function, which can get the 3D skeleton point coordinates through the 3D human-body parameters, \(\Pi\) is the orthogonal projection, which uses  
Figure 5: The process of human-body pose generation based on video sequences. camera parameters to project 3D key points onto a 2D plane. The 2D key points extracted by Openpose have been very accurate to some extent. By adding this 2D skeleton constraint, the feature space can be further optimized to produce a more accurate 3D human model. Then, the end-to-end pre-training model is trained with a large amount of marked data, and the obtained 3D human-body sequence has jitter, but basically conforms to the normal human-body distribution. Therefore, when further optimizing the solution, some useful feature information of the 3D skeleton points in the initial state is retained, and only the inaccurate part is optimized. The initial 3D skeleton point constraint is calculated as follows:  
\[\textit{loss}_{\text{3D init}}=\sum_{t}w_{t}\left(\left\|\hat{\vec{\theta}}- \left[f_{d}\left(\vec{z}_{t}\right)\right]_{\theta}\right\|_{2}^{2}\right) \tag{12}\]  
where \(\hat{\vec{\theta}}\) is the 3D pose model of the initial state obtained by the pre-training model, and is also the converted rotation matrix \(9\times 9\). \(w_{t}\) is the weight used to maintain the useful information in \(\vec{z}_{t}\), and optimize the useless information, the calculation process is as follows:  
\[w_{t}=\exp\left(-\delta_{t}\right) \tag{13}\]  
\[\delta_{t}=\sum_{j}c_{t,j}\left\|\left(\vec{x}_{t,j}-\Pi J_{j}\left(f_{d} \left(\vec{z}_{t}\right)\right)\right)\right\|_{2}^{2}. \tag{14}\]  
Here, the validity of the information in \(\vec{z}_{t}\) is measured by the 2D key-point error. The smaller the 2D key-point error, the larger the weight \(w_{t}\), so the decoded feature space should be closer to the initial state.