# Learning Robust Shape-Indexed Features for Facial Landmark Detection
## 3 Methods
### Patch Resampling Stage  
As shown in the first row of Figure 2, according to the predefined indexes, except for the cheek, the rest of the mean shape is downsampled into six landmarks corresponding to six facial parts: left eyebrow, right eyebrow, left eye, right eye, nose and mouth. This operation does not include the cheek because the landmarks of the cheek are distributed along the entire edge of the face and cannot be represented by a single point. These sparse landmarks allow us to crop large local patches that are fed into a lightweight network to acquire the offset values. Subsequently, all offset values are applied to the original mean shape to form a reinitialized shape, which is used to crop small shape-indexed patches. This resampling operation allows the STDN to improve the quality of shape-indexed patches, and thus, STDN extracts more robust shape-indexed features compared with existing methods.  
Figure 3 depicts the sparse landmark detection diagram. The mean face shape is downsampled into six sparse landmarks as the initial shape \(S_{0}\), and each landmark in this shape represents a facial part. Actually, it is the average of all landmarks in the specific  
Figure 2: Overall architecture of the proposed network. It consists of two stages: the patch resampling stage (**the first row**) and the relation reasoning stage (**the second row**). The downsample operation in the first row averages the coordinates of the landmarks in each facial part according to the predefined indexes. The \(+\) operation in the first row adds the offset values of the corresponding sparse landmarks to the coordinates of all landmarks in each facial part of the mean shape.  
facial part. The offset values \(\Delta S_{n}\) are predicted based on the shape-indexed patches of \(100\times 100\) size that are cropped surrounding the initial landmark. It is expressed as:  
\[\Delta S_{n}=f(I,S_{n-1}), \tag{1}\]  
where \(I\) is the input face image of size \(384\times 456\), \(S_{n-1}\) is the output shape of the last iteration, and \(f(\cdot)\) represents the regression function of the lightweight network. The current shape \(S_{n}\) is obtained by updating \(S_{n-1}\) and used as the initial shape for the next iteration:  
\[S_{n}=S_{n-1}+\Delta S_{n}. \tag{2}\]  
Take the sum of the offset of each iteration as the final offset values:  
\[\Delta S=\sum_{n=1}^{N}\Delta S_{n}, \tag{3}\]  
where \(N\) denotes the maximum number of iterations and is set to 2 in our implementation.  
The parameters of this network are updated by minimizing the following loss function:  
\[Loss=\sum_{n=1}^{N}\|\Delta S_{n}-(S^{*}-S_{n-1})\|_{2}^{2}, \tag{4}\]  
where \(S^{*}\) represents the ground truth of sparse landmarks generated after the downsampling operation.  
The reinitialized shape \(\bar{S}\) is derived by applying the offset value of the corresponding sparse landmark to all landmarks in the facial part of the mean shape:  
\[\bar{S}=\bigcup_{j=1}^{J}(S_{0}^{j}+\Delta S^{j}), \tag{5}\]  
\(S_{0}^{j}\) is the \(j\)-th facial part of the mean shape, and \(\bigcup\) denotes the operation of concatenating all six facial parts (i.e., both eyebrows, both eyes, nose and mouth). The landmarks of the cheek are taken directly from the mean shape. The reinitialized shape, which is used as the initial shape of the second stage is more similar to the ground truth than the mean shape.