# Automatic Scaffolding Workface Assessment for Activity Analysis through Machine Learning
## 3 Research Design
### 3D Pose Estimation  
Under the feature extraction section, we first extract 2D key joints using the OpenPose model from every image frame and subsequently lift 2D key joints into 3D. Then the raw coordinates of the 3D joints are converted into relative coordinates by subtracting the x, y, z values of a central point under raw coordinates. Each joint point is separated into x, y, z three features and 42 features are used for machine learning classification. As shown in Figure 6, the 3D pose estimation stands on the basis of 2D human joint prediction. The belief maps produced by the OpenPose system are used as one of the inputs at each stage. In each stage the structure of 3D pose estimation merges (1) the belief maps from a 2D joint predictor (OpenPose system) and (2) projected belief maps generated by a 3D pose projection as the inputs. The process undergoes six stages. The 3D projection is designed to lift the 2D coordinates into 3D and project 2D point locations into 3D and then the fusion layer combines the 2D joint belief maps and 3D projected belief maps and propagates them into a set of 2D point landmarks in order to iteratively reinforce the 2D joint prediction as well as 3D projection. At Stage 6, the fused belief map becomes the final output and is projected to 3D models eventually. The architecture of 3D estimation is using backpropagation and trains the model end to end [47].  
Inspired both by the approach [87] that represents the space of human poses as a mixture of Principal Components Analysis (PCA) and the idea that identifies poses as an  
Figure 6: The structure of the 3D pose estimation model.  
interpolation between pose categories, the probabilistic 3D model consists of a mixture of probabilistic PCA models with a number of clusters and the PCA models are trained by an Expectation-Maximization (EM) algorithm.  
First, poses are subdivided into several pose categories C and the Euclidean distance d between pairs are computed. The objective is to look for a set of samples \(S\) that keeps the distance between joint points and their nearest sample minimized. This searching process is iterative and applies greedy selection that retains the previous \(S\) until another \(s\) additionally minimizes the Euclidean distance is found. The process stops when one's candidate becomes close enough to the existing candidate within little discrimination and the aligned points of a certain pose category are assigned to its closest candidate \(s\) and then the EM algorithm is executed and a mixture of probabilistic PCA bases is built accordingly.  
\[\underset{s}{argmin}\sum_{p\in P}\underset{s\in S}{min}(s,p) \tag{1}\]  
A unimodal Gaussian 3D pose model is implemented to estimate the 3D human pose of a single frame. This model in practice takes 20 samples and optimizes the 3D pose reconstruction for every single rotation to achieve rotational invariance on the ground. A non-linear least squares solver is applied to find the basis coefficients of the best-found solution and this approach provides results close to the global optima with the same average accuracy but less computational cost.  
\[Y_{p}=argmax\;b_{p}[u,v] \tag{2}\]  
Then the generated 3D joint key points are projected onto a new set of 2D surface to form new 2D belief maps and, at the final layer in each stage, these new 2D belief maps, together with the belief maps from the 2D convolutional architecture, are fused into a single according to the equation below and the fused passed to the next stage as one of the input to refine the 2D joint prediction. For the final estimation of the pose, the 2D belief maps produced at Stage 6 are lifted into 3D space by the probabilistic 3D model [88].  
\[f_{t}^{p}=w_{t}\times b_{t}^{p}+(1-w_{t})\times\hat{b}_{t}^{p} \tag{3}\]