HandReader: Advanced Techniques for Efficient Fingerspelling Recognition

## Abstract

Abstract Fingerspelling is a significant component of Sign Language (SL), allowing the interpretation of proper names, characterized by fast hand movements during signing. Although previous works on fingerspelling recognition have focused on processing the temporal dimension of videos, there remains room for improving the accuracy of these approaches. This paper introduces HandReader, a group of three architectures designed to address the fingerspelling recognition task. HandReader RGB employs the novel Temporal Shift-Adaptive Module (TSAM) to process RGB features from videos of varying lengths while preserving important sequential information. HandReader KP is built on the proposed Temporal Pose Encoder (TPE) operated on keypoints as tensors. Such keypoints composition in a batch allows the encoder to pass them through 2D and 3D convolution layers, utilizing temporal and spatial information and accumulating keypoints coordinates. We also introduce HandReader RGB+KP – architecture with a joint encoder to benefit from RGB and keypoint modalities. Each HandReader model possesses distinct advantages and achieves state-of-the-art results on the ChicagoFSWild and ChicagoFSWild+ datasets. Moreover, the models demonstrate high performance on the first open dataset for Russian fingerspelling, Znaki, presented in this paper. The Znaki dataset and HandReader pre-trained models are publicly available 1 1 1 https://github.com/ai-forever/handreader .

## 1 Introduction

Figure: Figure 1: Three HandReader architectures. a) The videos with different lengths b) are processed to get crops and keypoints for HandReaderRGB and HandReaderKP, respectively, c) form the batches, and d) pass through the encoders. Further, e) the GRU decoder is applied to the encoder outputs. The final output of the HandReaderRGB+KP is obtained by summing the outputs of TSAM and TPE. Red-colored encoders are the paper’s contribution.
Refer to caption: extracted/6442147/images/overall.png

Sign Language (SL) facilitates communication among more than 430 million deaf and hard-of-hearing individuals worldwide(^2^22[https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss](https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss)). Due to the limited quantity of SL translators, an automatic SL Recognition (SLR) system can bridge the communication barrier between deaf individuals and people without SL knowledge [3, 2, 1]. Such systems find application in schools, hospitals, and other social environments.

Fingerspelling is an integral component of any sign language, used for parts of speech, proper names, and instances without specific signs, such as addresses, abbreviations, and surnames. For example, to sign the metro station’s name, the SL user should sign it letter by letter, using the corresponding signs for each letter from the SL alphabet. Therefore, fingerspelling is characterized by rapid and swift movements, complicating its recognition by blurry video frames. The motivation to tackle the fingerspelling recognition task is supported by interest from major companies, such as Google, which organized a Kaggle competition [5] to address it.

Many prior approaches to fingerspelling recognition are based on neural networks capable of processing the temporal component. To convert a video which consists of a frame sequence $I=\{I_{1},I_{2},...,I_{n}\}$ to a letter sequence $W=\{W_{1},W_{2},..W_{k}\}$, encoder-decoder architectures are typically utilized [6, 12, 19]. However, fingerspelling recognition, similar to Automatic Speech Recognition (ASR) and Optical Character Recognition (OCR), is complicated by the lack of alignment between $W$ and $I$, as multiple frames can interpret a single letter. The Connectionist Temporal Classification (CTC) loss [7] function is widely used in ASR and OCR to address this challenge by facilitating the learning of alignment between input frames and their corresponding output sequences, without the need for explicitly annotated segmentations.
So, there is an option to simplify the task and enhance methods’ performance by employing not only RGB frames but also such modalities as depth maps, keypoints, or combinations of video frames and keypoints [11, 20, 6, 18].

Despite multiple attempts to enhance fingerspelling recognition, the results of prior methods  [11, 20, 6, 18, 19, 17, 12] on two American fingerspelling datasets – ChicagoFSWild [19] and ChicagoFSWild+ [20] highlight the need for improved accuracy to ensure the applicability of existing approaches in real-world scenarios. Our work aims to enhance letter recognition accuracy while increasing frames-per-second (FPS) throughput.
Such advancements can be achieved by developing novel techniques to process input data more effectively and enrich the SL domain with data annotated more precisely. However, creating fingerspelling datasets is complicated by the scarcity of people with SL knowledge and the communication barriers between them and non-SL-speaking people. To the best of our knowledge, some sign languages, such as Russian, struggle with a lack of accessible fingerspelling datasets. The Russian fingerspelling dataset is important because it is used in multinational Russia and the CIS(^3^33CIS – The Commonwealth of Independent States.) countries, such as Belarus, Ukraine, Kazakhstan, etc. This widespread comprehensibility can be explained by the historical presence of a unified sign language in these regions.

We aimed to develop an efficient approach for fingerspelling recognition and facilitate this task in the Russian SL domain with a new dataset. The contributions of this paper are summarized as follows:

- •
We introduce the Temporal Shift-Adaptive Module (TSAM), extending the original Temporal Shift Module (TSM) [14] to effectively process RGB image features from videos of varying lengths without trimming or padding.
- •
The Temporal Pose Encoder (TPE) to operate keypoints as three-channel tensors, capturing spatial and temporal information.
- •
HandReader, a group of three architectures (see  [Fig. 1](https://arxiv.org/html/2505.10267v1#S1.F1)) that achieved state-of-the-art results on ChicagoFSWild and ChicagoFSWild+ test sets.
- •
An efficient methodology of collecting & labeling fingerspelling datasets in various languages oriented to limited signers.
- •
The first and the one open dataset for Russian fingerspelling, contained 1,593 annotated phrases and over 37 thousand HD+ videos.

We release the pre-trained models, code, and dataset(^4^44[https://github.com/ai-forever/handreader](https://github.com/ai-forever/handreader)).

## 2 Related Work

This overview excludes Continuous Sign Language Recognition (CSLR) research, as methods for such a task are complicated by the need to recognize the independent relationships between signs and letters within words. In contrast, fingerspelling recognition presents a more straightforward one-to-one correspondence.

Recently, the fingerspelling recognition task has been explored using various data modalities, including RGB imagery [20, 19, 17, 12], keypoints [6], and depth maps [11]. Moreover, [18] utilized data obtained by combining different modalities. Researchers are motivated to focus on the RGB modality because a fingerspelling recognition system must apply in real-world conditions and resist external changes. Meanwhile, acquiring data from other modalities requires additional equipment, such as extra cameras like Kinect for depth maps or an extra data processing stage for keypoints.

Figure: Figure 2: Comparison of the TSAM architecture (outlined in red at the top) with the original TSM (outlined in gray at the bottom). The input batch includes two videos of different lengths, with specific frames highlighted in blue and yellow. While the TSM model requires fixed-length inputs utilizing padding (shown in white) or trimming frames (shown in red) when necessary, the TSAM model employs individual shift counters to manage variable-length sequences without additional padding or trimming. The shift counter displayed at the top of the TSAM block shows how the shifts are activated individually for each video, incrementing the shift counter until it reaches the length of that specific video.
Refer to caption: extracted/6442147/images/tsm.png

Typical videos for the fingerspelling task contain irrelevant information, such as backgrounds and foreign objects. Therefore, only RGB data requires the model to identify the most relevant areas in the frames, specifically the hands, before recognizing a sign sequence. To simplify locating the signing hand, the authors in [19] developed a hand detection method using Faster R-CNN, then trained a CNN-LSTM model for the recognition task. In their subsequent work [20], they replaced the explicit hand detection method with an iterative attention mechanism. The authors in [17, 12] employed representation learning to improve the model’s ability to distinguish different signs by alignment from CTC and incorporated the method from [20] as a preprocessing stage. Research [17] acquired state-of-the-art results on ChicagoFsWild and ChicagoFsWild+.

Since most RGB modality methods are centered around the detection hands problem, they are limited to exploring different architectures or decoding methods on the final results. In contrast, the keypoint modality avoids this issue because it includes relevant information about hand placement. The authors [6] trained a transformer encoder-decoder architecture with a new decoding approach, advancing performance on ChicagoFSWild and ChicagoFSWild+ test sets. Keypoints are usually acquired using MediaPipe [16] or another neural network trained for pose estimation. The authors [18] combined RGB and keypoints modalities employing linear layers to project the 2D coordinates into 3D space. After extracting all features, they concatenated them and used a decoder based on an attention mechanism. Despite the promise shown by the methods above, recognition accuracy still has potential for improvement. Therefore, our primary objective is to enhance the methodologies employed without compromising the model’s weight and inference speed.

RGB Feature Extraction in the Fingerspelling Task. There are two main ways to extract features in the fingerspelling task. (1) Spatial features can be received by applying a 2D CNN to each frame, then extracting temporal features using 1D convolutions or RNNs. (2) Spatio-temporal features can be obtained with 3D convolution. However, processing frames as a sequence of independent frames lacks temporal information. The TSM [14] was developed to address this issue by employing 2D convolutions and shifting information along the temporal dimension. However, TSM processes fixed-size videos, which leads to the loss of valuable information in long videos or negatively affects training time and memory allocation due to the padding in shorter videos. Therefore, this paper presents a modified version of the original TSM to handle videos of varying lengths.

Figure: Figure 3: Demonstration of the Temporal Pose Encoder operating on a single batch. a) The encoder receives an input tensor containing $N$ frames and $K$ keypoints. All videos are padded to match the length of the most extended video clip in the batch, allowing them to be stacked. b) 2D convolutions process multiple keypoints across several frames, extracting temporal features. c) Subsequently, a 3D convolution in the form of a “tube” aggregates the extracted motion information for each body part into final feature representations $F$. d) After applying TPE, features are processed with linear layer, e) followed by MLP with convolution modules from [9].
Refer to caption: extracted/6442147/images/kp_arch.png

## 3 Approach

We present a group of three architectures – HandReader – designed to recognize sign sequences using various modalities and differ in the encoder stage only (see  [Fig. 1](https://arxiv.org/html/2505.10267v1#S1.F1)). Thus, HandReaderRGB solely utilizes RGB features, HandReaderKP relies on keypoints, and HandReaderRGB+KP combines both. Each was trained directly using CTC loss, eliminating the need for frame-level character alignment. A two-layer bidirectional Gated Recurrent Unit (GRU) [4] was chosen as a decoder to generate a sequence of characters, enhancing the model’s capability to capture temporal dynamics. We apply a linear layer with SoftMax to the GRU output to generate posterior probabilities for each predicted character at every frame. Encoders for each architecture are described below.

### 3.1 HandReader RGB RGB {}_{\textbf{RGB}} start_FLOATSUBSCRIPT RGB end_FLOATSUBSCRIPT

The original TSM is a video understanding approach that enables temporal modeling in 2D convolutional networks by shifting portions of channel features along the temporal dimension. However, TSM operates constrained to processing a fixed number of frames. Therefore, videos that exceed the frame limit require trimming, while shorter videos need additional empty frames (i.e., padding). Furthermore, the number of shifts is determined by the fixed frame limit. As a result, videos with extra empty frames lose critical sequential information during processing (see  [Fig. 2](https://arxiv.org/html/2505.10267v1#S2.F2)). We modified the TSM to manage a dynamic number of input frames, avoiding padding and enabling frame-by-frame processing for each video in the batch. Besides, a shift counter is introduced to preserve important sequential information, bypassing shifts on short videos.

Temporal Shift-Adaptive Module. The TSAM was designed to effectively process videos of various lengths. Figure [2](https://arxiv.org/html/2505.10267v1#S2.F2) illustrates the distinction between TSAM and TSM operations. The TSAM is incorporated into the ResNet-34 model and operates an input tensor with dimension $(bs,C,H,W)$, where $bs$ indicates batch size, equal to the sum of the video lengths, $C$ is the number of channels, $H$ and $W$ are the height and width of the frames, respectively. The TSAM also requires an auxiliary list of video lengths, denoted as $L=[L_{S_{1}},L_{S_{2}},\dots,L_{S_{n}}]$, where $S_{i}$ signifies $i$th video in the batch, and $bs=\sum_{i}L_{S_{i}}$. We iterate through $L$, extracting $L_{S_{i}}$ frames from the batch to obtain $S_{i}$ video sequence and apply shifts individually, as detailed in  [Algorithm 1](https://arxiv.org/html/2505.10267v1#alg1). Such a method reduces VRAM usage by up to $50\%$ during batch creation, thereby facilitating the processing of longer videos.

The number of shifts is determined by the number of convolutional blocks within the backbone architecture, leading to overwrite temporal features similar to the original TSM. To prevent this loss of information, we introduce a shift counter that stops temporal shifts at the final frame, i.e., no shifting is applied if a video’s length is smaller than the current shift count. Such a method ensures that each video in a batch is treated individually, resulting in more accurate feature extraction.

Each sequence of the received tensor with dimension $(bs,H,W,F)$, where $F$ represents the number of features, was subsequently padded with zeros to align with the maximum sequence length within the batch. All features are aggregated using 1D convolution, and the resulting tensor is transformed into the format $(bs,N,F)$, where $F$ is the flattened features across all channels, and $N$ is calculated as $H*W$.

Figure: Algorithm 1 Applying temporal shifts in TSAM.

### 3.2 HandReader KP KP {}_{\textbf{KP}} start_FLOATSUBSCRIPT KP end_FLOATSUBSCRIPT

We utilized MediaPipe [16] to extract body keypoints from every frame. Face keypoints were excluded as stated in [Sec. 6.2](https://arxiv.org/html/2505.10267v1#S6.SS2), resulting in 54 $(x,y,z)$ coordinates for the pose and hand body parts per frame. The proposed feature encoder is built on a new way to structure keypoints explicitly developed for it, which organizes keypoints within tensors. Each video is treated as a tensor with dimension $(3,N,54)$, where 54 $(x,y,z)$ vectors are stacked into matrices for each of $N$ video frames. Such a configuration enables the spatial representation of temporal data, allowing the application of processing techniques typically used for standard images. We passed the received tensor to the feature extractor described below.

Temporal Pose Encoder. We present a Temporal Pose Encoder based on the convolution layers. TPE operates an input tensor of dimension $(bs,3,N,K)$, where $bs$ is the batch size, indicating the number of videos in the batch, $N$ denotes the maximum length of videos in the batch, while shorter videos were padded, and $K$ – is the amount of coordinates of keypoints. The tensor is processed by two 2D convolution layers, followed by a 3D one (see  [Fig. 3](https://arxiv.org/html/2505.10267v1#S2.F3)). 2D convolutions serve to extract both temporal and spatial features, withal transforming keypoints coordinates, enhancing the model’s understanding of dynamic finger movements. At the same time, 3D convolution tries to accumulate coordinates information within a single video with the kernel-tube size of $(5,1,1)$, treating each coordinate independently. In this context, the hyperparameter of 5 defines the size of the receptive window along the coordinate dimension, which captures local patterns across five neighboring coordinate features. We insert a dummy dimension in the second position, enabling 3D convolution to treat the original channels as temporal dimensions. The process can be represented as follows: $(bs,32,N,K)$ is unsqueezed to $(bs,1,32,N,K)$ and processed by 3D convolution to $(bs,1,10,N,K)$. A linear layer is then applied to project the features of the keypoints $(bs,N,F)$, where $F$ is the final encoder’s features. We additionally use multi-layer perceptron (MLP) with convolution modules from [9] as a neck between the encoder and the GRU decoder.

### 3.3 HandReader KP+RGB KP+RGB {}_{\textbf{KP+RGB}} start_FLOATSUBSCRIPT KP+RGB end_FLOATSUBSCRIPT

We combine features from the HandReaderRGB and HandReaderKP encoders to take advantage of both modalities. Two output tensors of features from TSAM and TPE are summed and passed to the GRU (see  [Fig. 1](https://arxiv.org/html/2505.10267v1#S1.F1)).

## 4 Znaki Dataset

**Table 1: The number of unique samples contained within each category in the Znaki dataset.**
| Categories | Count | Categories | Count |
| --- | --- | --- | --- |
| Moscow metro stations | 231 | Cities world | 93 |
| Social terms | 121 | Movie titles | 92 |
| Banking terms | 107 | Sights of the world | 91 |
| Countries | 99 | Name of diseases | 90 |
| Russian Cities | 99 | Sights of the Russia | 84 |
| Authors | 98 | Professions | 75 |
| Book titles | 97 | Geographical names | 70 |
| Government institutions | 96 | Person’s names | 50 |

While datasets containing video spelling of single cyrillic letters [8, 13] exist, they are are unsuitable for fingerspelling recognition of Russian Sign Language (RSL) due to the lack of varied transitions between letters, the speed of signing and the naturalness & variability of surroundings during execution. Creating the dataset for this purpose involves one of three methods: recording in the studio, gathering online and TV videos, and utilizing crowdsourcing platforms. The first two methods result in the same background, low signer diversity, and inappropriate video quality, while the last requires a selection of signers and a careful record check. Using the pipeline described below, we created the dataset with the crowdsourcing platform TagMe(^5^55[https://tagme.sberdevices.ru/](https://tagme.sberdevices.ru/)).

### 4.1 Phrases Selection

Since fingerspelling is primarily used for signing proper names, only words and phrases included in this group were determined for the dataset. We consulted with deaf experts and identified the 16 most common word categories (see  [Tab. 1](https://arxiv.org/html/2505.10267v1#S4.T1)), with about 100 well-known words and phrases for each category. The phrases contain 1 to 4 proper names, with an average of 1.4 words.

### 4.2 Signers Selection

We have exclusively recorded individuals who are deaf or hard-of-hearing and who are members of the VOG (All-Russian Society of the Deaf)(^6^66All-Russian Society of the Deaf, [https://voginfo.ru/](https://voginfo.ru/)) community.

RSL knowledge test. Candidates were required to pass the knowledge test of Russian Sign Language (RSL), which involved recording a video with a corresponding translation of Russian annotations into RSL. Deaf experts and RSL teachers checked the assignment’s correctness and decided on the candidate’s admission to the project.

Fingerspelling exam. The candidates who passed the exam were permitted to register on the crowdsourcing platform and had to pass an additional fingerspelling exam. It included a video with RSL fingerspelling and the task of finding the correct translation into Russian from a list of words. 68 candidates out of 100 were accepted to the main tasks, passing the exam with a score of at least 85% correct answers.

### 4.3 Video Collection

The task of video collection was accompanied by written instructions that contained the recording rules and requirements for video quality. In addition, the signers were familiarized with video, where RSL teachers demonstrate the same instruction via signs. The rules for video recording were formulated as follows: only one person in the video with the gesture-performing hand entirely in the frame is permitted, and no shaking or visual effects are allowed. Besides, the platform’s verification process approved only videos with a minimum HD resolution, duration over 1 second, and frame rate over 15 frames per second. Also, the system does not pass duplicate videos. Signers have the option to re-record the video. They were allowed to record videos anywhere, making the dataset heterogeneous regarding external conditions.

### 4.4 Video Validation

Each video was checked for the correctness of the translation from Russian to RSL signs, as well as the compliance of the video with the requirements when recording the video. At least three validators checked each video, and the result relied on the majority’s opinion. There are 82% of samples aggregated with an entire agreement of each validator, reflecting a high level of reliability in the annotations.

Figure: Figure 4: Znaki dataset. a) video lengths distribution; b) total number of letters distribution; c) distance distribution: the distance is approximately estimated in meters by computed the length between the signers’ left and right shoulders using MediaPipe [16]; d) letter frequency: comparison of letters frequency in the dataset Znaki and the Russian language corpus.
Refer to caption: extracted/6442147/images/chars.png

Figure: Figure 5: Sample frames from the Znaki dataset.
Refer to caption: extracted/6442147/images/samples_4x2.png

### 4.5 Time Interval Annotation

Time interval annotation was required to separate the sign sequence from other activity periods. Validated videos were processed to 30 frames per second (FPS) to bind subsequent time interval annotations before this stage. Each video was annotated by three users, further aggregating to one result by open-source AggMe framework(^7^77[https://github.com/ai-forever/aggme](https://github.com/ai-forever/aggme)), resulting in 95% videos achieved with consistency of 80%.

### 4.6 Dataset Characteristics

Signers. The dataset was recorded by 37 deaf signers, 29 hard-of-hearing, and two sign language translators. The signers are aged from 20 to 54 years and belong to the Caucasian race, with 52 women and 16 men.  [Fig. 5](https://arxiv.org/html/2505.10267v1#S4.F5) provides samples from the Znaki dataset.

Sequences. The dataset contains 1,593 unique proper names, covering all 33 letters of the Russian alphabet and 696 pairs of different letter combinations. The distribution of letter frequency in Znaki corresponds to such distribution in the Russian language national corpus(^8^88[https://en.wikipedia.org/wiki/Russian_alphabet](https://en.wikipedia.org/wiki/Russian_alphabet)) (see  [Fig. 4](https://arxiv.org/html/2505.10267v1#S4.F4)b).

Videos. The Znaki includes 37,252 videos with a minimum frame side of 720 pixels trimmed according to time interval annotations, resulting in about 42 GB.
Each sequence was recorded by between 8 and 41 different signers, averaging 23 videos per phrase. The average length of each sequence was 102 frames (see  [Fig. 4](https://arxiv.org/html/2505.10267v1#S4.F4)a). More than 80% of the dataset’s video is in Full HD format, and the other is in HD. About 88% of samples were recorded on a phone camera, and the rest on a laptop.

Split. We split the dataset into training (58%), validation (23%), and test (19%) sets, each containing unique phrases only: 743, 300, and 550, respectively. The test set is not intersected in signers with training and validation sets.

## 5 Experiments

This section outlines the experimental setup and data preprocessing procedures used to assess the proposed HandReader architectures and compare them with other approaches.  [Tab. 2](https://arxiv.org/html/2505.10267v1#S5.T2) shows the open datasets used in this work.

### 5.1 Experimental Setup

All three architectures were trained from scratch using an AdamW optimizer[15] with an initial learning rate of 0.0001. The MultiStepLR modifies the learning rate with a gamma of 0.1 using milestones at 20 and 40 for the HandReaderRGB and HandReaderRGB+KP models and 25 and 40 for the HandReaderKP. We trained the HandReaderRGB for 60 epochs, while the other two models were trained for 100 epochs each. A single Nvidia H100 with 80GB RAM was utilized for training. All models are tested on a MacBook M1 Pro using a single CPU core.

**Table 2: Open fingerspelling recognition datasets used in this work.**
| Dataset | Lang | Sequences | Chars | Hours | Signers | Source |
| --- | --- | --- | --- | --- | --- | --- |
| ChicagoFSWild | ASL | 7K | 38K | 2 | 160 | Web |
| ChicagoFSWild+ | ASL | 55K | 0.3M | 14 | 260 | Web |
| Znaki | RSL | 37K | 0.37M | 35 | 68 | Crowd |

### 5.2 Data Preprocessing

The ChicagoFSWild and ChicagoFSWild+ datasets are commonly used to evaluate approaches to the fingerspelling task. Since we created a new fingerspelling dataset, the evaluation results of the models trained on it are also provided.

The RGB frames for each dataset were cropped using the model described in [20], augmented with horizontal flipping and random rotation (see parameters in [Tab. 3](https://arxiv.org/html/2505.10267v1#S5.T3)), then resized to 244 and normalized. Note that, each sign in Russian fingerspelling is invariant to horizontal flip operation. Keypoints were extracted by MediaPipe from raw frames rather than cropped images. We utilized hands and pose keypoints only, resulting in 54 keypoints. We replaced the NaN values with zeros, normalized the rest, and applied augmentations specified in [Tab. 3](https://arxiv.org/html/2505.10267v1#S5.T3). Only resampling, horizontal flipping, and random rotation augmentations are applied when combining two modalities in HandReaderRGB+KP.

**Table 3: Augmentations used during training.**
| Augmentatoin | Parameters | Probability | Modality |
| --- | --- | --- | --- |
| Resample | rate=(0.5, 1.5) | 0.8 | KP, RGB+KP |
| SpatialAffine | scale=(0.8, 1.2), shear=(-0.15, 0.15),<br>shift=(-0.1, 0.1), degree=(-30, 30) | 0.75 | KP |
| TemporalMask | size=(0.2, 0.4), value=0.0 | 0.5 | KP |
| SpatialMask | size=(0.05, 0.1),<br>value=0.0, mode=”relative” | 0.5 | KP |
| HoriznotalFlip | — | 0.5 | RGB, RGB+KP, KP |
| Rotation | angle=(-10, 10) | 0.5 | RGB, RGB+KP |

### 5.3 Results

We assessed HandReader architectures using the letter accuracy metric: $1-\frac{S+D+I}{N}$, where $S$, $D$, and $I$ denote substitutions, deletions, and insertions, respectively, $N$ is the length of ground-truth sequence. Metrics values for other methods were obtained from the original papers for comparison purposes. Each HandReader version achieved state-of-the-art results on ChicagoFSWild and ChicagoFSWild+ test sets (see  [Tab. 4](https://arxiv.org/html/2505.10267v1#S5.T4)). Also  [Tab. 6](https://arxiv.org/html/2505.10267v1#S5.T6) demonstrates the evaluation of presented methods on the Znaki dataset. The Znaki dataset has fewer videos than ChicagoFSWild+ but includes longer, higher-resolution videos with cleaner annotations, resulting in massive margins between metrics. We also present prediction comparison between PoseBased Transformer [6] and our models in  [Tab. 4](https://arxiv.org/html/2505.10267v1#S5.T4).

**Table 4: Results comparison on ChicagoFSWild and ChicagoFSWild+ test sets. * indicates the model was trained on both ChicagoFSWild and ChicagoFSWild+ training sets. ** means two convolution modules were used for HandreaderKP encoder, while only one was employed for other metrics.**
| Model | ChicagoFSWild | ChicagoFSWild+ | Modality |
| --- | --- | --- | --- |
| Hand Det + CNN + RNN [19] | 41.9 | 41.2 | RGB |
| Iterative Attention [20] | 45.1 | 46.7 | RGB |
| 3d hand pose + RGB [18] | 47.9 | – | RGB + KP |
| SiamseNet+Iteraive model [12] | 49.6 | – | RGB |
| Iter train + SupCon [17] | 64.9 | 71.7* | RGB |
| Pose-based transformer [6] | 66.3 | 71.1 | KP |
| HandReader${}_{\textbf{KP}}$ | 69.3 | 72.4 | KP |
| HandReader${}_{\textbf{RGB}}$ | 72.0 | 73.8 | RGB |
| HandReader${}_{\textbf{RGB+KP}}$ | 72.9** | 75.6 | RGB + KP |

**Table 5: Results comparison of predictions between previous SotA solution and our models, predictions for PoseBased model were taken from the original paper.**
| PoseBased [6]* | KP | KP+RGB | RGB | Reference |
| --- | --- | --- | --- | --- |
| macah | mach | mach | mach | mach |
| meach | meah | beack | fealc | beach |
| hospital | hospital | hospital | hospital | hospital |
| - | outube | iutube | iutube | youtube |
| - | organic | organic | orgadic | organic |

**Table 6: The results of HandReaderRGB, HandReaderKP, and HandReaderRGB+KP on the Znaki test set.**
| Model | Model<br>Size (MB) | Params. (M) | Inference<br>Time (ms) | Letter accuracy |
| --- | --- | --- | --- | --- |
| HandReaderKP | 34.7 | 9.1 | 3.9 | 92.65 |
| HandReaderRGB | 207.7 | 54.4 | 51.4 | 92.39 |
| HandReaderRGB+KP | 222.3 | 58.2 | 55.2 | 94.94 |

## 6 Ablation

This section provides an ablation study to demonstrate the impact of each component on the HandReader’s overall performance. Note that HandReaderRGB+KP was excluded from the study since it relies solely on the heuristics of the other two architectures. We maintained a consistent experimental setup described in [Sec. 5](https://arxiv.org/html/2505.10267v1#S5) and changed only one component at a time to analyze its influence. The ChicagoFSWild dataset was utilized in all experiments.

### 6.1 HandReader RGB RGB {}_{\textbf{RGB}} start_FLOATSUBSCRIPT RGB end_FLOATSUBSCRIPT

Encoder.  [Tab. 7](https://arxiv.org/html/2505.10267v1#S6.T7) indicates that integrating TSM into the baseline model improves the performance metric by 1.69. Furthermore, replacing TSM with TSAM leads to an additional 1.4 improvement. To isolate the effect of the count shift mechanism, we implemented a variant of TSAM without the timely stopping of shifts, which led to a 1.49 performance degradation.

Decoder. We experimented with the LSTM [10] decoder instead of the GRU [4] one, which resulted in a difference of 1.69. Replacing RNN with a linear layer reduces the metric by 11.0. Such observations underlined that high-level features of each frame cannot be used as independent features for final predictions.

**Table 7: Ablation study of HandReaderRGB’s components. * denotes baseline setup as ResNet34 configuration (without TSM or TSAM shift operations), which beats previous state-of-the-art solutions.**
| Experiment | Packed sequence | RNN | Count shift | Shift type | Image<br>Augs. | BS | Letter accuracy |
| --- | --- | --- | --- | --- | --- | --- | --- |
| best | ✓ | GRU | ✓ | TSAM | ✓ | 4 | 72.0 |
| Packed sequence | ✗ | GRU | ✗ | TSM | ✓ | 4 | $70.6_{\color[rgb]{1,0,0}-1.4}$ |
| RNN | ✓ | LSTM | ✓ | TSAM | ✓ | 4 | $70.31_{\color[rgb]{1,0,0}-1.69}$ |
| ✓ | ✗ | ✓ | TSAM | ✓ | 4 | $61.0_{\color[rgb]{1,0,0}-11.0}$ |  |
| Count shift | ✓ | GRU | ✗ | TSAM | ✓ | 4 | $70.51_{\color[rgb]{1,0,0}-1.49}$ |
| Image Augs. | ✓ | GRU | ✓ | TSAM | ✗ | 4 | $70.49_{\color[rgb]{1,0,0}-1.51}$ |
| BS | ✓ | GRU | ✓ | TSAM | ✓ | 2 | $71.46_{\color[rgb]{1,0,0}-0.54}$ |
| ✓ | GRU | ✓ | TSAM | ✓ | 6 | $70.1_{\color[rgb]{1,0,0}-1.9}$ |  |
| Baseline* | ✗ | GRU | ✗ | None | ✓ | 4 | $69.63_{\color[rgb]{1,0,0}-2.37}$ |

**Table 8: Ablation study of HandReaderKP’s components.**
| Experiment | Keypoints | RNN | Conv.<br>modules | Image Augs. | BS | Letter accuracy |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| hand | pose | face |  |  |  |  |  |  |
| best | ✓ | ✓ | ✗ | GRU | 1 | ✓ | 4 | 69.31 |
| Keypoints | ✓ | ✗ | ✗ | GRU | 1 | ✓ | 4 | $68.54_{\color[rgb]{1,0,0}-0.77}$ |
| ✓ | ✓ | ✓ | ✓ | $69.08_{\color[rgb]{1,0,0}-0.23}$ |  |  |  |  |
| ✓ | ✗ | ✓ | ✓ | $67.59_{\color[rgb]{1,0,0}-1.72}$ |  |  |  |  |
| ✗ | ✓ | ✓ | ✓ | $10.02_{\color[rgb]{1,0,0}-59.29}$ |  |  |  |  |
| ✗ | ✓ | ✗ | ✓ | $10.36_{\color[rgb]{1,0,0}-58.95}$ |  |  |  |  |
| ✗ | ✗ | ✓ | ✓ | $11.08_{\color[rgb]{1,0,0}-58.23}$ |  |  |  |  |
| RNN | ✓ | ✓ | ✗ | LSTM | 1 | ✓ | 4 | $67.79_{\color[rgb]{1,0,0}-1.52}$ |
| ✓ | ✓ | ✗ | ✗ | ✓ | $63.88_{\color[rgb]{1,0,0}-5.43}$ |  |  |  |
| Conv. modules | ✓ | ✓ | ✗ | GRU | 2 | ✓ | 4 | $68.09_{\color[rgb]{1,0,0}-1.22}$ |
| ✓ | ✓ | ✗ | 3 | ✓ | $68.11_{\color[rgb]{1,0,0}-1.2}$ |  |  |  |
| Image Augs | ✓ | ✓ | ✗ | GRU | 1 | ✗ | 4 | $62.43_{\color[rgb]{1,0,0}-6.88}$ |
| BS | ✓ | ✓ | ✗ | GRU | 1 | ✓ | 2 | $69.04_{\color[rgb]{1,0,0}-0.27}$ |
| ✓ | ✓ | ✗ | ✓ | 6 | $69.26_{\color[rgb]{1,0,0}-0.05}$ |  |  |  |
| ✓ | ✓ | ✗ | ✓ | 8 | $68.45_{\color[rgb]{1,0,0}-0.86}$ |  |  |  |

Others.  [Tab. 7](https://arxiv.org/html/2505.10267v1#S6.T7) highlights the significant impact of augmentations, as a model trained without rotations and horizontal flips reached a final metric of 70.49. Such influence is explained by the dataset containing signers with different dominant hands and videos where hands are misaligned due to camera angles or the signer’s position. Additionally, we tested different batch sizes and showed that a bigger batch size is not always optimal. It is especially noticeable in ablation studies with RGB. This may happen because the batch contains videos of varying lengths, complicating the model’s decoding process.

### 6.2 HandReader KP KP {}_{\textbf{KP}} start_FLOATSUBSCRIPT KP end_FLOATSUBSCRIPT

Keypoints. We tested possible combinations of different parts of keypoints and provided results in [Tab. 8](https://arxiv.org/html/2505.10267v1#S6.T8). The metric was reduced by adding face keypoints since sign language users can express the same phrases differently without verbal language skills. The model performs better with pose and hands keypoints than with just hands keypoints, as the pose coordinates carry information about the position of the whole hand, which may help the model better understand the hand’s relative position throughout the video.

Encoder and Decoder. The same procedure of RNN impact assessment is provided in [Tab. 8](https://arxiv.org/html/2505.10267v1#S6.T8). In contrast to [Sec. 6.1](https://arxiv.org/html/2505.10267v1#S6.SS1), the usage of the linear layer does not show a significant metric drop, as the HandReaderKP encoder contains more modules for processing temporal dimensions. We studied the impact of the number of convolution modules utilized in the encoder. As the number of convolution blocks increases, the final metric declines.

Others. The augmentations have a crucial effect on model performance. We also examined various batch sizes: 4, 6, and 8. As the batch size increases, there are noticeable metric drops.

### 6.3 HandReader KP+RGB KP+RGB {}_{\textbf{KP+RGB}} start_FLOATSUBSCRIPT KP+RGB end_FLOATSUBSCRIPT

Feature reduction.
We evaluated various feature reduction techniques, including simple summation, feature concatenation, element-wise multiplication, and weighted summation, obtaining 72.95, 72.95, 72.55, and 72.57 letter accuracy, respectively. We preferred the simple summation due to the smaller dimension of the sum output than concatenation one. The evaluation was conducted on the test set of the ChicagoFSWild dataset.

### 6.4 Dataset’s heterogeneity

We also assess the impact of user heterogeneity on the quality of the models in the Znaki dataset. We limited the training set of 743 videos in two configurations, ensuring that each phrase was represented without overlap. The first is a homogeneous training set consisting of only two signers, and the second is a heterogeneous training set – 53 signers recorded one video each, the test set was not changed. The results in  [Tab. 9](https://arxiv.org/html/2505.10267v1#S6.T9) show that the models significantly improve on the heterogeneous set, even for the KP modality.

**Table 9: Evaluating the impact of signers heterogeneity in the training part of the Znaki dataset.**
| Modilaty | Signers | Training setup | Letter accuracy |
| --- | --- | --- | --- |
| RGB | 2 | homogenous | 63,6 |
| RGB | 53 | heterogenous | $\textbf{81,8}_{\color[rgb]{0,1,0}+18,2}$ |
| KP | 2 | homogenous | 73,3 |
| KP | 53 | heterogenous | $\textbf{83,3}_{\color[rgb]{0,1,0}+10,1}$ |

## 7 Limitations

Architectures. Although the HandReaderKP and HandReaderRGB effectively recognize fingerspelling, extra models are required to extract keypoints and hand regions crops, respectively. Such necessities add computational overhead, decreasing inference speed and limiting the model’s real-time usability, particularly on devices with limited processing power. There is an opportunity to replace an additional model with center cropping augmentation in the HandReaderRGB, provided the input focuses on the hands’ region in the center of the frame. The HandReaderRGB+KP is subject to the limits of other HandReader architectures, but it can utilize only the keypoint prediction model and obtain crops operating received points.

Znaki Dataset. The proposed dataset is inappropriate for fully addressing the CSLR problem since fingerspelling mostly represents proper names. Despite the historical commonality of sign languages and similarity of individual signs across CIS countries, the dynamic evolution of these languages and influence of national characteristics have resulted in Russian SL fingerspelling being used primarily within Russia, while maintaining full comprehensibility mostly among the older generation. Since the dataset was collected through crowdsourcing, all participants were positioned strictly facing the camera, challenging recognition from different angles. The dataset primarily reflects the demographic composition of native RSL users, which may result in limited ethnic diversity.

## 8 Ethical Considerations

All signers provided informed consent, allowing for the processing and publication of their data in compliance with the Civil Code of the Russian Federation(^9^99[https://ihl-databases.icrc.org/en/national-practice/federal-law-no-152-fz-personal-data-2006](https://ihl-databases.icrc.org/en/national-practice/federal-law-no-152-fz-personal-data-2006)). We fully anonymized dataset annotations, protecting the identities of contributors and ensuring confidentiality. The payment for completed tasks to crowdworkers was equivalent to the average salary of a sign language interpreter relative to the time spent. Although the Znaki is intended solely for research purposes, we acknowledge the risk of misuse, which could involve identifying individuals or enabling large-scale surveillance.

## 9 Conclusion

This paper presents HandReader, a suite of three distinct architectures that significantly advance the fingerspelling recognition task. The Temporal Shift-Adaptive Module and Temporal Pose Encoder were developed to capture RGB and keypoint features effectively. We achieved state-of-the-art performance on ChicagoFSWild and ChicagoFSWild+ datasets. The ablation study revealed that the proposed techniques and training pipelines are essential for achieving optimal results. We also created Znaki, the first and the one open-source heterogeneous dataset for Russian fingerspelling recognition. The presented findings offer valuable insights for enhancing fingerspelling recognition systems.

## References

- [1]
Why are asl interpreters so hard to find?
[https://www.helloglobo.com/blog/why-are-asl-interpreters-so-hard-to-find](https://www.helloglobo.com/blog/why-are-asl-interpreters-so-hard-to-find).
- bar [2021]
American sign language interpreting: The size and state of the market.
[https://www.nimdzi.com/asl-interpreting/](https://www.nimdzi.com/asl-interpreting/), 2021.
- bar [2022]
The asl interpreter shortage and its impact on accessibility in college settings.
[https://nationaldeafcenter.org/news-items/the-asl-interpreter-shortage-and-its-impact-on-accessibility-in-college-settings/](https://nationaldeafcenter.org/news-items/the-asl-interpreter-shortage-and-its-impact-on-accessibility-in-college-settings/), 2022.
- Cho et al. [2014]
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014.
- Chow et al. [2023]
Ashley Chow, Glenn Cameron, Manfred Georg, Mark Sherwood, Phil Culliton, Sam Sepah, Sohier Dane, and Thad Starner.
Google – american sign language fingerspelling recognition.
[https://kaggle.com/competitions/asl-fingerspelling](https://kaggle.com/competitions/asl-fingerspelling), 2023.
Kaggle.
- Fayyazsanavi et al. [2024]
Pooya Fayyazsanavi, Negar Nejatishahidin, and Jana Košecká.
Fingerspelling posenet: Enhancing fingerspelling translation with pose-based transformer models.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pages 1120–1130, 2024.
- Graves [2012]
Alex Graves.
Connectionist temporal classification.
*Supervised sequence labelling with recurrent neural networks*, pages 61–93, 2012.
- Grif and Kondratenko [2021]
M. G. Grif and Y. K. Kondratenko.
Development of a software module for recognizing the fingerspelling of the russian sign language based on lstm.
*Journal of Physics: Conference Series*, (1), 2021.
- Gulati et al. [2020]
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang.
Conformer: Convolution-augmented transformer for speech recognition, 2020.
- Hochreiter and Schmidhuber [1997]
Sepp Hochreiter and Jürgen Schmidhuber.
Long short-term memory.
*Neural computation*, 9:1735–80, 1997.
- Kang et al. [2015]
Byeongkeun Kang, Subarna Tripathi, and Truong Q Nguyen.
Real-time sign language fingerspelling recognition using convolutional neural networks from depth map.
In *2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)*, pages 136–140. IEEE, 2015.
- Kumwilaisak et al. [2022]
Wuttipong Kumwilaisak, Peerawat Pannattee, Chatchawarn Hansakunbuntheung, Nattanun Thatphithakkul, et al.
American sign language fingerspelling recognition in the wild with iterative language model construction.
*APSIPA Transactions on Signal and Information Processing*, 11(1), 2022.
- Kvanchiani et al. [2024]
Karina Kvanchiani, Petr Surovtsev, Alexander Nagaev, Elizaveta Petrova, and Alexander Kapitanov.
Bukva: Russian sign language alphabet, 2024.
- Lin et al. [2019]
Ji Lin, Chuang Gan, and Song Han.
Tsm: Temporal shift module for efficient video understanding.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 7083–7093, 2019.
- Loshchilov and Hutter [2019]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization, 2019.
- Lugaresi et al. [2019]
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al.
Mediapipe: A framework for building perception pipelines.
*arXiv preprint arXiv:1906.08172*, 2019.
- Pannattee et al. [2024]
Peerawat Pannattee, Wuttipong Kumwilaisak, Chatchawarn Hansakunbuntheung, Nattanun Thatphithakkul, and C-C Jay Kuo.
American sign language fingerspelling recognition in the wild with spatio temporal feature extraction and multi-task learning.
*Expert Systems with Applications*, 243:122901, 2024.
- Parelli et al. [2020]
Maria Parelli, Katerina Papadimitriou, Gerasimos Potamianos, Georgios Pavlakos, and Petros Maragos.
Exploiting 3d hand pose estimation in deep learning-based sign language recognition from rgb videos.
In *Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16*, pages 249–263. Springer, 2020.
- Shi et al. [2018]
Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Jonathan Michaux, Diane Brentari, Greg Shakhnarovich, and Karen Livescu.
American sign language fingerspelling recognition in the wild.
In *2018 IEEE Spoken Language Technology Workshop (SLT)*, pages 145–152, 2018.
- Shi et al. [2019]
Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane, Diane Brentari, Greg Shakhnarovich, and Karen Livescu.
Fingerspelling recognition in the wild with iterative visual attention.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2019.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: Please list up and carefully describe any questions and suggestions for the authors. Think of the things where a response from the author can change your opinion, clarify a confusion or address a limitation. This is important for a productive rebuttal and discussion phase with the authors.

1. What is the computational efficiency trade-off of using TSAM versus the original TSM, given the less parallelizable, per-video logic?
2. The reliance on external preprocessing models is noted as a limitation. Have you explored any end-to-end alternatives that could integrate hand-localization or keypoint estimation directly into the architecture?
soundness: 2
strengths: A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results.

1. The paper introduces TSAM, a novel module that effectively handles variable-length video inputs by adapting the TSM mechanism. It also proposes TPE, a new encoder that processes keypoints as tensors using 2D and 3D convolutions to capture spatio-temporal dynamics.
2. The HandReader models achieve new state-of-the-art results on both the ChicagoFSWild and ChicagoFSWild+ test sets, outperforming prior work in RGB-only, KP-only, and combined modalities.
3. The paper introduces "Znaki," a large-scale (37k videos) and rigorously validated dataset for Russian fingerspelling.
summary: This paper addresses fingerspelling recognition by introducing "HandReader," a set of three architectures. The core contributions are two novel encoders: 1) the Temporal Shift-Adaptive Module (TSAM), which extends TSM to effectively process variable-length RGB videos, and 2) the Temporal Pose Encoder (TPE), which uses 2D/3D convolutions to process keypoints. These models achieve state-of-the-art results on the ChicagoFSWild and ChicagoFSWild+ benchmarks. The paper also introduces "Znaki," a new, large-scale 37k-video dataset for Russian fingerspelling, detailing its rigorous collection and validation methodology.
weaknesses: A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.

1. While the HandReader_KP model is very fast on a CPU (3.9ms, Table 5), the best-performing RGB (51.4ms) and RGB+KP (55.2ms) models are considerably slower. This could limit real-time applicability on low-power devices.
2. The architectures are not fully end-to-end and depend on separate, pre-trained models for hand-cropping and keypoint extraction. This adds computational overhead and complexity.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. It is somewhat surprising that the keypoint-only model performs similarly to the RGB-based model, as prior work typically reports stronger performance from RGB features. Could the authors explain this behavior or discuss possible reasons ?
2. Did the authors investigate how the quality or accuracy of the MediaPipe keypoint predictions affects overall model performance?
soundness: 2
strengths: 1. The authors achieve state-of-the-art results on both ChicagoFSWild and ChicagoFSWild+ benchmarks, which demonstrates the effectiveness of their input processing and pipeline. 
2. They also introduce a new dataset, containing a large number of high-resolution videos, which is helpful to the sign language community.
summary: The authors propose a compositional pipeline to solve the task of fingerspelling recognition. They introduce an architecture, consisting of a modified Temporal Shift Adaptive Module and a Temporal Pose Encoder to process input RGB and keypoint sequences both separately and jointly. They validate their method on the ChicagoFSWild and ChicagoFSWild+ datasets and introduce a new Russian Fingerspelling dataset.
weaknesses: 1.Some sections of the paper are not fully clear and easy to follow. 

2. The proposed method mainly leverages existing architectures to solve the task without significant modifications. For the RGB video processing the extension of the Temporal Shift-Adaptive Module (TSAM) for varying video lengths is a useful addition but is not in my opinion a major modification of the existing (TSAM) architecture. Also, organizing key points as tensors and using them as input to a convolutional encoder, to the best of my knowledge, has been extensively used in action and sign language recognition. 
2. The proposed approach is evaluated on a limited number of datasets. Its effectiveness could be evaluated further by extending it to more benchmarks.
3. Evaluating only on fingerspelling benchmarks limits the scope of this work. A more general approach that targets both widely used sign language datasets and fingerspelling might be more appropriate for this conference.

---

confidence: 3
contribution: 1
initial_rating: 0
presentation: 3
questions: 1) What are the results of a conformer using your exact pipeline for training ?
2) Did the authors try any other fusion methods apart from feature summation ? 
3) Regarding the claim "Such a method reduces VRAM usage by up to 50% during batch creation" we do not see any actual numbers or evidence to support this. 
4) What are the performance metrics of other methods ? Showcase a proper comparison.

In general, at its current state I would urge the authors to more restructure their paper around the new dataset and the results of previous sota methods on it. If they are going to claim contributions w.r.t. a new method more results/metrics are needed and clear comparisons with the state of the art. Currently TSAM only offers a slight 1% increase and we are not sure it is not noise -> maybe test on more datasets ?
soundness: 2
strengths: The main strength of the paper seems to be the Znaki dataset for Russian fingerspelling. The collection of the dataset is well documented but it is not clear if the dataset will be made available to researchers.  
Also another small strength is the engineering behind making the TSM module adaptable to variable length sequences, although the actual gains are minor.
summary: In this paper the authors present a new dataset for fingerspelling and russian and two architectures for fingerspeeling, relying on RGB or keypoint as input. The presentation is clear enough and the Znaki dataset is useful for the community of fingerspelling. The experimental results of the authors show that adding TSAM instead of TSM slightly improves the results on ChicagoFSWild dataset in terms of letter accuracy, and that the fusion of both models achieves competitive results w.r.t. previous methods such as Conformers. 
Overall apart from the Znaki dataset the paper is mostly engineering and reads more like a technical report.
weaknesses: Overall the novelty of the paper is very weak. TSAM seems a small engineering trick behind making TSM adapt to variable length sequences and the ablation result in Table 7 of the supplementary seems a very small increase in terms of letter accuracy. Also we only have letter accuracy as metrics - no CER/WER, word accuracy, S/D/I breakdown, or confusion matrices.

Furthermore, another contribution stated is the combination of the keypoints as a three-dimension tensor, however this has been done multiple times before and is the standard approach for inputting keypoints in deep networks. 

W.r.t. the Znaki dataset, the authors did only train their own methods and did not show any comparisons with previous state of the art. Also in table 5 the authors added model size and performance metrics however this was not presented for any other previous method.

In Table 3, the results are confusion. Conformers were trained on both train datasets as it is said - what about HandReader? It is not clear. Also, for fair comparison the authors should pretrain the conformer using their pipeline.

There is not intuition w.r.t. to the previous methods -> what is the trick of handreader that really enabled it to obtain such performance ? It seems that even standard TSNs in Table 7 are able to beat a lot of previous sota methods.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: Please provide channel split ratios, shift directions/steps, per-layer shift counts, and the algorithmic box (or pseudocode) for the shift counter; report boundary handling (zero/replicate/circular).
soundness: 2
strengths: Practical, simple solutions (TSAM/TPE) for real deployment pain points (variable length, low latency) with low engineering overhead.

Three tracks (RGB / KP / fusion) under a unified BiGRU+CTC training/decoding setup.

Znaki dataset shows strong quality control (multi-annotator verification, signer-disjoint splits), which can foster cross-language evaluation.
summary: The paper proposes three architectures for fingerspelling recognition, HandReader:

(1) HandReaderRGB introduces TSAM (Temporal Shift-Adaptive Module) into ResNet-34 to handle variable-length videos without cropping/padding, and uses a shift counter to avoid temporal overwriting on short sequences;

(2) HandReaderKP proposes TPE (Temporal Pose Encoder), which rearranges 54 hand/body keypoints per frame (x,y,z) into a (3,T,54) tensor and performs 2D→3D convolutions for “coordinate accumulation”;

(3) HandReaderRGB+KP sums the two feature streams and feeds them to BiGRU + CTC.
The authors also release Znaki (a Russian fingerspelling dataset) and report SOTA on ChicagoFSWild/+ and Znaki. The overall approach is engineering-oriented with simple and efficient modules.
weaknesses: 1) TSAM: No ablation for shift counter on/off, per-layer number of shifts, or other key hyperparameters; no direct comparison to “fixed-length TSM + mask/padding.”

2) Only “sum” is reported; no comparisons to concat/gating/cross-attention.

3) Efficiency and deployability: Missing a unified comparison against 3D-CNN, TimeSformer, TSM (fixed-clip) on FLOPs/Params/end-to-end latency/peak VRAM; preprocessing costs (hand-crop, MediaPipe) are not included.

4) Fairness of pipeline: RGB uses Shi (2019) cropping while keypoints are extracted from raw frames; the field-of-view mismatch may confound results.


Decision: Reject