## 2 Full-Body Pose Recovery
### Face Fitting  
Once we have properly segmented the body region, the next step consists of fitting the face and the body limbs. For the case of face recovery, we base our procedure on mesh fitting using AAM, combining Active Shape Models and color and texture information [13]. AAM is generated by combining a model of shape and texture variation. First, a set of points are marked on the face of the training images that are aligned, and a statistical shape model is build [14]. Each training image is warped so the points match those of the mean shape. This is raster scanned into a texture vector, **g**, which is normalized by applying a linear transformation, \(\textbf{g}\mapsto(\textbf{g}-\mu_{g}\textbf{1})/\sigma_{g}\), where **1** is a vector of ones, and \(\mu_{g}\) and \(\sigma_{g}^{2}\) are the mean and variance of elements of **g**. After normalization, \(\textbf{g}^{T}\textbf{1}=0\) and \(|\textbf{g}|=1\). Then, principal component analysis is applied to build a texture model. Finally, the correlations between shape and texture are learnt to generate a combined appearance model. The appearance model has parameter **c** controlling the shape and texture according to  
\[x =\overline{x}+\textbf{Q}_{s}\textbf{c} \tag{8}\] \[g =\overline{g}+\textbf{Q}_{g}\textbf{c} \tag{9}\]  
where \(\overline{x}\) is the mean shape, \(\overline{g}\) the mean texture in a mean shaped patch, and \(\textbf{Q}_{s}\), \(\textbf{Q}_{g}\) are matrices designing the modes of variation derived from the training set. A shape **X** in the image frame can be generated by applying a suitable transformation to the points, \(\textbf{x}:\textbf{X}~{}=~{}S_{t}(\textbf{x})\). Typically, \(S_{t}\) will be a similarity transformation described by a scaling \(s\), an in-plane rotation, \(\theta\), and a translation \((t_{x},t_{y})\). Once constructed the AAM, it is deformed on the image to detect and segment the face appearance as follows. During matching, we sample the pixels in the region of interest \(\textbf{g}_{im}=T_{u}(\textbf{g})=(u_{1}+1)\textbf{g}_{im}+u_{2}\textbf{1}\), where **u** is the vector of transformation parameters, and project into the texture model frame, \(\textbf{g}_{s}=T_{u}^{-1}(\textbf{g}_{im})\). The current model texture is given by \(\textbf{g}_{m}=\overline{g}+\textbf{Q}_{g}\textbf{c}\), and the difference between model and image (measured in the normalized texture frame) is as follows  
\[\textbf{r}(\textbf{p})=\textbf{g}_{s}-\textbf{g}_{m} \tag{10}\]  
Given the error \(E=|\textbf{r}|^{2}\), we compute the predicted displacements \(\delta\textbf{p}=-\textbf{R}\textbf{r}(\textbf{p})\), where \(\textbf{R}=\left(\frac{\partial\textbf{r}^{T}}{\partial\textbf{p}}\frac{ \partial\textbf{r}}{\partial\textbf{p}}\right)^{-1}\frac{\partial\textbf{r}^ {T}}{\partial\textbf{p}}\). The model parameters are updated \(\textbf{p}~{}\mapsto~{}\textbf{p}+k\delta\textbf{p}\), where initially \(k~{}=~{}1\). The new points \(\textbf{X}^{\prime}\) and model frame texture \(\textbf{g}_{m}^{\prime}\) are estimated, and the image is sampled at the new points to obtain \(\textbf{g}_{mi}^{\prime}\) and the new error vector \(\textbf{r}^{\prime}=T_{u^{\prime}}^{-1}(g_{im}^{\prime})-g_{m}^{\prime}\). A final condition guides the end of each iteration: if \(|\textbf{r}^{\prime}|^{2}<E\), then we accept the new estimate, otherwise, we set to \(k=0.5\), \(k=0.25\), and so on. The procedure is repeated until no improvement is made to the error. With the purpose to discretize the head pose between frontal face and profile face, we create three AAM models corresponding to the frontal, right, and left view. Aligning every mesh of the model, we obtain the mean of the model. Finally, to determine the class of a fitted face by AAM models, that is given by its proximity to the closest mean model. Taking into account the discontinuity that appears when a face moves from frontal to profile view, we use three different AAM corresponding to three meshes of 21 points: frontal view \(\Im_{F}\), right lateral view \(\Im_{R}\), and left lateral view \(\Im_{L}\).