## 3 Datasets and Methods
### Methodology
#### 3.2.1 Feature Extraction
\begin{table}
\begin{tabular}{l l} \hline \hline
**Feature** & **Extraction Steps** \\ \hline Hand coordinates & **1:** Initialize Mediapipe [19] hands object; \\  & **2:** Pass the hand gesture image to the object; \\  & **3:** The location of 21 hand joints in a 3D coordinate space are returned with each of them in the form of a 3-tuple as follows: \\  & \[(x_{i},y_{i},z_{i})\] (1) \\  & where (\(i=0\)) to (20). \\ \hline Convolutional features & **1:** Feed the sign gesture image to the CNN described in Table 4; \\  & **2:** The convolutional features are obtained as a flattened 1D feature vector after passing through convolutional layers and pooling layers. \\ \hline Convolutional & features **1:** Obtain the hand coordinates 0, 3, 5, 8, 12, 16 and 20 using Mediapipe [19] as shown in Figure 2 to calculate the finger angles; \\  & **2:** The Euclidean distances [20]\(a_{i},b_{i},c_{i}\), where \(i=1\) to 4 are calculated in three dimensions; \\  & **3:** The four finger angles as depicted in Figure 2 are calculated using the law of cosines for triangles which is as follows: \\  & \[\cos\ \theta_{1}=\frac{a_{1}^{2}+b_{1}^{2}-c_{1}^{2}}{2a_{1}b_{1}}\] (2) \\  & \[\cos\ \theta_{2}=\frac{a_{2}^{2}+b_{2}^{2}-c_{2}^{2}}{2a_{2}b_{2}}\] (3) \\  & \[\cos\ \theta_{3}=\frac{a_{3}^{2}+b_{3}^{2}-c_{3}^{2}}{2a_{3}b_{3}}\] (4) \\  & \[\cos\ \theta_{4}=\frac{a_{4}^{2}+b_{4}^{2}-c_{4}^{2}}{2a_{4}b_{4}}\] (5) \\  & **4:** These finger angles are combined with Convolutional features to give a hybrid feature. \\ \hline CNN features on hand edges & **1:** Initialize a skin-mask with the real skin tone color scheme [21] but decrease the lower bound to capture darker skin tones as well as skin tones under insufficient lighting; \\  & **2:** Perform a bitwise AND between the skin mask and the input image to segment the hand region from the background; \\  & **3:** Denoise the hand-segmented image using median blur; \\  & **4:** Apply Canny filter [22] on the denoised image to retain only the hand edges; \\  & **5:** The CNN described in Table 4 is used to extract features from the edge image. \\ \hline CNN features on BRISK image & **1:** Initialize a BRISK object; \\  & **2:** Using the object, detect BRISK keypoints in the edge image obtained in the previous feature; \\  & **3:** Pass this keypoint-infused image through the CNN described in Table 4. \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Features extracted. Figure 2: Finger angles. Moving on to finger angles, these are proposed to capture the variations between signs of the same language such as the letters 'a' and 'e' in the American Sign Language as depicted through the two images in Figure 3. These variations are captured in terms of the relative position of the fingers with respect to each other. Similar to hand coordinates, finger angles also remain constant despite hand depth or orientation in the input image for the same class label. For a sign image taken in a cluttered environment, the resulting convolution process may be unable to extract features specific to the sign. CNN features on hand edges alleviate this problem by segmenting the hand region in the original input image with the help of an approximate skin mask so that the neighboring regions become black. A skin mask indicates a range of RGB values that encompasses the majority of skin tones that can be observed in sign images. This is followed by obtaining the edges in the hand-segmented image. For edge detection, a Canny filter is preferred over a Sobel filter [23] because the edges retained are smooth in nature. Figure 4 outlines the images obtained during the extraction of this feature. This feature extraction method is applied on the original input image without calculating the finger angles. Once the edge image has been obtained, the CNN described in Table 4 would be able to extract features corresponding to only the hand region without including any of the complex background since it has been removed when finding hand edges.