# Content-Adaptive and Attention-Based Network for Hand Gesture Recognition
## 3 Proposed Method
### Content-Adaptive Sampler  
Gestures can be roughly categorized into two components: appearances and motion. Understanding the motion in gesture videos relies heavily on long-range temporal information. Learning video representations that capture long-range temporal information is a pivotal challenge for gesture recognition. To use the dynamic information from the whole video for video-level prediction, a sparse temporal sampling strategy is currently the dominant approach. The main idea is to divide a video into k segments evenly and then randomly select n frames from each segment. We visualized the sampling results of the sparse temporal sampling strategy, as shown in Figure 2. Figure 2 presents the sampling results in a NVIDIA dynamic hand gestures (NVGesture) dataset when \(k=16\) and \(n=1\). Figure 2 indicates the results when \(k=32\) and \(n=1\). We found that when \(k\) is set too low, the sampling results do not effectively represent the whole video; when \(k\) is set too high, too many frames without gestures are sampled, increasing the input length of the model.  
Figure 1: The architecture of the gesture recognition network based on content adaptation and attention.  
To sample the video sequence efficiently, we proposed a dynamic sampling strategy based on content adaptation. We use a sliding window to perform gesture detection over the incoming video frames. As shown in Figure 1, frames without gestures will not be sampled by the sampler, i.e., they cannot pass the sampler (shown as a light-colored crossed-out arrow). When the sampler detects a gesture, it starts sampling evenly (no more gesture detection) according to the time strategy until it reaches the established maximum number of frames. If the frame count of a video is less than the maximum frame count set by the sampler, we will pad the input sequence with zeros. The pseudo-code of our proposed content-based adaptive sampling algorithm can be found in Algorithm 1. Algorithm 1 can be briefly described as follows: first perform continuous gesture detection on the incoming frames and record the index of the current frames; when a gesture is detected, then stop gesture detection; the index of the gesture-containing frame and the time sampling'step' are used to generate the final list of sampled frames. The input of Algorithm 1 is a gesture video and the output is a list of sampled frames. Figure 2c presents the sampling results of our adaptive sampler. Another role of the sampler is to act as a switch for the encoder. When a gesture is detected in the video sequence, the encoder is activated and starts receiving the input from the sampler.  
Being able to detect hand gestures of various sizes is a challenge for hand detectors. In practice, we have selected the single-shot multibox detector [38] as our hand detector module because of its accuracy and real-time performance. Before feeding the sampled frames into the encoder, they need to be mapped into a sequence of tokens. In our work, we use a pre-trained DenseNet121 [10] as a 2D spatial feature extractor to map the extracted frames into meaningful features.  
Figure 2: Comparison of results using different sampling strategies: (a) sparse temporal sampling strategy (\(k=16,n=1\)); (b) sparse temporal sampling strategy (\(k=32,n=1\)); (c) our proposed content-adaptive sampling strategy (\(n=16,step=3\)).  
```
0: gesture video data
0: frame index list
1: initialization [frame list], counter=0, flag=True
2:while cap is opened & flag do
3: if a gesture is not detected then
4: counter \(\leftarrow\) counter + 1
5: continue
6: else
7: flag \(\leftarrow\) False
8:endif
9:endwhile
10:for each 'index' i in range(length n) do
11: [frame list] \(\leftarrow\) i*step + counter
12: i \(\leftarrow\) i + 1
13:endfor
14:return [frame list]
```  
**Algorithm 1** Content-adaptive sampler.