# Automatic Recognition of Mexican Sign Language Using a Depth Camera and Recurrent Neural Networks
## 3 Materials and Methods  
### Data Acquisition  
We collected a dataset of 30 different signs, each one performed 25 times by four different people at different speeds, starting and ending times. The full dataset consists of 3000 samples. Each sign is composed of 20 consecutive frames containing hands, body, and facial 3D keypoint coordinates. We used an OAK-D camera for the data acquisition. This device consists of three cameras: a central camera to obtain RGB information and a stereo rig to get depth information from the disparity between images (see Figure 1). We used the DepthAI [29] and the MediaPipe [30; 31; 32] libraries to detect the face, body, and hands keypoints. From the total set of 543 keypoints, we selected a subset 67 distributed as follows: 20 for the face, 5 for the body, and 21 for each hand, as shown in Figure 2.  
The original facial keypoints include a dense face mesh containing 468 landmarks. Given that most of the face landmarks have a high correlation, we employ a feature selection approach to reduce the face landmarks down to 20, including four points for each eyebrow, four points around each eye, and four points around the mouth, as shown in Figure 2. Next, we selected the upper body points from the original body landmarks, including the chest, shoulders, and elbows. Since the MediaPipe library does not provide a chest point, we obtained it as the middle point between the shoulders. Finally, for the hands, we use all the landmarks that include four points on each finger and five points around each hand palm for a total of 21 points on each hand. Figure 3 shows some examples of keypoints captured for the words: (a) thank you, (b) explain, (c) where? and (d) why? As part of the dataset, we stored the keypoints coordinated and the sequence of images for each sign. Figure 1: OAK-D Camera. Figure 2: Face, body, and hand selected keypoints. The numbers indicate the landmark indices used in the MediaPipe library. We detect each keypoint in the RGB image and represent it by its \(P^{\prime}[X^{\prime},Y^{\prime}]\) coordinates. For these coordinates we obtain its depth value \(Z\) from the depth image and compute the 3D space coordinates \(P[X,Y,Z]\) using Equations (1) and (2). \[X=\left(\frac{X^{\prime}Z}{f}\right), \tag{1}\]  
\[Y=\left(\frac{Y^{\prime}Z}{f}\right), \tag{2}\]  
where \(X^{\prime}\) and \(Y^{\prime}\) are the pixel coordinates in the image, \(f\) is the focal length, and \(Z\) the depth. We collected samples for 30 different signs shown in Table 2. Out of these, four are static, and 26 are dynamic. In terms of the hands used, 17 are one-handed and 13 two-handed. They can also be classified into four subgroups: 8 are letters of the sign-language alphabet, 8 are questions, 7 are days of the week, and 7 are common phrases. The captured data is stored in comma-separated values (CSV) structured in 20 rows and 201 columns. Each file represents a single repetition of an individual sign, and each row represents the information obtained in a single image. Each row contains the keypoints (X, Y, Z) coordinates in the following order: five body keypoints, 20 facial keypoints, 21 left-hand keypoints, and 21 right-hand keypoints. The coordinates are in meters and normalized with respect to the chest to make the signs invariant to camera distance.