# Learning Robust Shape-Indexed Features for Facial Landmark Detection
## 3 Methods
### Relation Reasoning Stage  
An observation of unconstrained faces shows that the eyes and the eyebrows are often occluded by hair or sunglasses, the mouth may be occluded by food or microphone, and the cheek may be self-occluded due to large pose variations. Only the nose area is rarely  
Figure 3: Sparselandmark detection diagram. completely occluded, as it is located in the central region of the face. Considering all the above, we argue that the nose can be used as an anchor to constrain other facial components. The facial components are divided into six groups based on the inherent structure: the left group, including the left eyebrow and the left eye; the right group, including the right eyebrow and the right eye; the nose group; the mouth group the left cheek group; and the right cheek group, as shown in Figure 4.  
The relation reasoning stage employs a dense landmark detection network to exploit the shape constraint against occlusion, as shown in Figure 5. In this stage, the dense landmark detection network predicts the offset value at each iteration so that the initial shape will be gradually updated through multiple iterations. At the first iteration, the dense landmark detection network uses two candidate shapes to initialize: the reinitialized shape or the mean shape. The mean shape is used to avoid overfitting caused by heavily relying on the reinitialized shape. The probability of the reinitialized shape and the mean shape being chosen is \(\epsilon\) and \(1-\epsilon\), respectively. At the subsequent iterations, the prediction from the previous iteration is used for initialization. After obtaining the dense initial landmarks, we feed the cropped small-sized patches into the dense landmark detection network. This network uses three convolutional layers to extract shape-indexed features from small-sized patches. A fully connected layer is employed to transform shape-indexed features into whole features. The whole features are used to directly predict all landmarks; they are also used to combine with the shape-indexed features as the fusion features, which are used as the input of the group-relational module for grouping prediction. The group-relational module first uses the fusion features to extract features of the anchor group (i.e., nose) and predict the landmarks of the nose. It then uses the features of the anchor group and the fusion features to deduce the features of other groups and predict their landmarks. This can make full use of the structural relationship between facial components, so that the network can predict the landmarks in the case of severe occlusion. Specifically, the three convolutional layers extract shape-indexed features \(F\) based on local patches of \(34\times 34\) size that are cropped surrounding initial landmarks from the image of \(384\times 456\) size. These shape-indexed features are first mapped into the whole features \(f_{g}\) and then predict a global offset value as follows:  
\[ \begin{split}& f_{g}=\mathbf{tanh}(W_{g}^{1}*F+b_{g}^{1})\\ &\Delta y^{g}=W_{g}^{2}*f_{g}+b_{g}^{2},\end{split}  \tag{6}\]  
where \(\Delta y^{g}\) denotes the global offset value, \(\mathbf{tanh}\) denotes a nonlinear activation function, \(W_{g}^{1}\) denotes the weights of the input-to-hidden fully connected layers, and \(b_{g}^{1}\) denotes the biases. \(W_{g}^{2}\) and \(b_{g}^{2}\) are the weights and biases of the prediction layer. In the group-relational module, we introduce the shape-indexed features \(F\) into each facial group to supplement the contextual information when occlusion occurs, and the whole features \(f_{g}\) are also introduced to provide the global shape constraint. For the anchor group (i.e., nose), its offset value can be formulated as follows:  
\[ \begin{split}& f_{n}=\mathbf{tanh}(W_{n}^{1}*(F\oplus f_{g})+b_{n}^{1}) \\ &\Delta y^{n}=W_{n}^{2}*f_{n}+b_{n}^{2},\end{split}  \tag{7}\]  
Figure 4: Diagram of face division. Landmarks of the same color are divided into a group.