# Movement-in-a-Video Detection Scheme for Sign Language Gesture Recognition Using Neural Network
## 2 Sign Language Recognition Method  
The proposed sign language recognition method is shown in Figure 1. The input to the system is a dataset of videos containing sign language gestures. The video sequences are processed to extract the information in each frame, and that information is stored in an image. Each sign language gesture renders a distinct image representing the gesture. These images are used to train an NN. Finally, the performance of the NN in solving sign language recognition tasks is evaluated.  
A flow chart of the proposed sign language recognition method is shown in Figure 1a. As shown in the figure, the blocks containing the processes are labeled from one to five. The recognition method can be described as follows:  
1. In the first block, a series of frames are extracted from each video. The number of frames per video depends on the duration of each gesture. The input to the first block is a video of sign language, as shown in Figure 1b. Meanwhile, the output is a series of frames, as shown in Figure 1c.
2. In the second block, each extracted frame is subtracted from its preceding frame. The input to the second block is a series of extracted frames, as shown in Figure 1c. Meanwhile, the output is a series of frame differences, as shown in Figure 1d. Frame difference is the difference between two consecutive frames.
3. In the third block, the differences between consecutive frames obtained in the second block are added. The input to the third block is an image representing frame differences, as shown in Figure 1d. Meanwhile, the output is an image representing the sum of frame differences, as shown in Figure 1e.  
In the fourth and fifth blocks, the images obtained in the third block are used to train and test the NN for recognizing sign language gestures.  
The concept of obtaining the sum of frame differences is adopted from the sum of absolute difference (SAD) architecture used in block-based motion estimation (ME). [17; 18; 19] Block-based ME is widely used in video coding standards owing to its advantage of efficient removal of temporal redundant frames. [18] In block-based ME, each frame is divided into equal-sized M \(\times\) N blocks. Each block in the current frame is compared with the blocks in the reference frame. The best matching blocks in the current and reference frames are obtained using SAD. The displacement between the best matching current block and reference block is then stored in a motion vector (MV). SAD involves two calculations: first, the calculation of the absolute difference and then, the accumulation of absolute differences. This is expressed as:  
\[SAD(x,y,i,j)=\sum_{l=0}^{M-1}\sum_{k=0}^{N-1}\Big{|}A_{(x+l,y+k)}-B_{(x+i+l,y+j+ k)}\Big{|} \tag{1}\]  
where \((x,y)\) are the coordinates of the block in the reference frame; \((i,j)\) are the x and y distances, respectively, of the block in the current frame with the block in the reference frame; \((l,k)\) are the coordinates of the pixels in the equal-sized \(M\times N\) blocks; and \(A_{(x+l,y+k)}\) and \(B_{(x+i+l,y+j+k)}\) are the pixels in the blocks of the current and reference frame, respectively [17; 19].  
Figure 1: Proposed sign language recognition method. (**a**) Flow chart of proposed sign language recognition method. (**b**) Videos of sign language gestures. (**c**) Frames extracted from videos of sign language gestures. (**d**) An image representing the difference between two consecutive frames. (**e**) Sum of all frame differences.