Scaling and Distilling Transformer Models for sEMG
Authors: Nick Mehlman
Jean-Christophe Gagnon-Audet
Michael Shvartsman
Kelvin Niu
Alexander H Miller
Shagun Sodhani
Venue: Accepted by TMLR ()

Abstract: Surface electromyography (sEMG) signals offer a promising avenue for developing innovative human-computer interfaces by providing insights into muscular activity. However, limited available training data and computational constraints during deployment have restricted the use of state-of-the-art machine learning models, such as transformers, in challenging sEMG tasks. In this paper, we demonstrate that transformer models can learn effective and generalizable representations from sEMG datasets that are small by modern deep learning standards (approximately 100 users), surpassing the performance of classical machine learning methods and older neural network architectures. Additionally, by leveraging model distillation techniques, we reduce parameter counts by up to 50x with minimal loss of performance. This results in efficient and expressive models suitable for complex real-time sEMG tasks in dynamic real-world environments.

Published in Transactions on Machine Learning Research (07/2025)
Scaling and Distilling Transformer Models for sEMG
Nicholas Mchlman' Viterbi School of Engineering, University of Southern California
umchhmmnn eduu
Jean-Christone Christophe Gagnon-Audet"
jooudel@metta com
Meta FAIR
Michael Shvartsman Meta FAIR
mshvarts@mella. com
Kelvin Niu
kniu@meta..com com
Meta FAIR
Alexander H. Miller
ahmo@eela com
Meta FAIR
Shagun Sodhani
sodhani@metta, com
Meta FAIR
Reviewed OpenReview: https: openreview. net/ forum? id= hFPhThUUsz
Abstract
Surface electromyography (sEMG) signals offer promising avenue for developing innova- tive human-computer interfaces by providing insights into muscular activity. However, the limited volume of training data and computationnl constraints during deployment have re- stricted the investigation scaling up the model size for solving sEMG tasks. In this paper, we demonstrate that vanilla transformer models can be effectively scaled up on sEMG data and yield improved cross-user performance up to 110M parameters, surpassing the model size regime investigated in other sEMG research (usually <10M parameters). We show that 100M-parameter models can be effectively distilled into models 50x smaller with mini- mal loss of performance (< 1.5% absolute). This results in efficient and expressive models suitable for complex real-time sEMG tasks in real-world environments.
1 Introduction
Recently, there has been growing interest in using surface elettromyography (sEMG) in conjunction with
powerful deep learning techniques to decode human muscle activity (c.g. Di Nardo al. 2022 Gaso ct al,
2021 Wimalasena el al. 2022 Buongiorno et al. 2021 Ozdemir et al. 2020). sEMG olfers the potential for
novel human-commee computer interfaces (HCIs), where user gestures or movements can serve as direct control input
[CTRL-labs at Reality Labs 2024 Advances hardware (c.g. Lu et al. 2024 CTRL-labs at Reality Labs,
2024) have now made it feasible to reliably capture sEMG outside of a controlled clinical setting. Supported by these developments, deep learning methods have been applied to a variety FMG tasks, including muscle
activation detection (Di Nardo al.) 2022), /Wimalasena et al. 2022). gesture classification (Atzori al.
2016), (He et al.) 2018), (Zhang et al. 2023b), and speech recognition (Wand & Schmidhuber 2016).
However, there are few limitations in previous works. First, many introduce significant. complexity such
as bespoke deep learning architectures Wanga et al. 2024 Zabihi et al. 2023 Putro et al.
2024
Chen
"equal contribution
equal contribution

---

Published in Transactions on Machine Learning Research (07 2025))
et al.
2023a b Liu et al. 2024 additional training objectives (Dai et al 2023 2023 Zhang al. 2022
Zeng et al., 2022). This can act as barrier for usability by practitioners, as departing from simple and battle-tested recipes reduces both the robustness of available implementations and support available from the community. This is especially true for in-the-wild practitioners, typically neuroscientists, who may lack lamiliarity with deep learning systems. This limitation may be due to the restricted quantity and/or diversity of available training data (Li ct al. 2021), as large-scale data is often viewed as prerequisite for applying contemporary vanilla deep-learning methods to complex tasks. Lacking large EMG corpora, researchers may historically have relied on exploiting neuroscience domain knowledge (e.g., hand-designed features auxiliary losses) to achieve good performance. A second consideration commonly unaddressed in previous works is the computational challenge associated with running an HCI in the wild. In particular, there are likely to be substantial constraints on the model size (small enough to run on an edge device) and inference time (fast enough for the system to be responsive). Finally, the ability to effectively generalize to unseen users is critical to successful deployment in real-world applications, yet cvaluation on unscon users is often neglected in the existing lirerature.
In this paper, we take steps towards addressing these challenges:
1. We start by applying a simple convolution and transformer architecture (Schncider et al. 2019) on the emg2qwerly task (Sivakumar et al. 2024 and outperform previous CNN-based SOTA by 20% (absolute). Then, we show that the performance of the transformer can be further improved with model scale, enabling us to improve over the SOTA performance by an additional 5% (absolute) by increasing the number of parameters from 2.2M to 109M. This differs from other works, which
mainly focus on smaller model scales <10M parameters (angaa et al.
2024
Rahimian et al. 2021
Montazerin et al.
2023 Zabihi et al. 2023 Zhang et al. 2023a
Yang et al.
2024] with complex
non non-standard deep learning methodology (Wanga et al.]
2024
Dai et al. 2023
Zabihi et al.) 2023
Putro et al. 2024 Chen et al. 2023 Zhang et al. 2022
2023a b Liu et al.
2024
Zeng et al.
2022
2.
We analyze simple logil-based distillation (Hinton et al. 2015 for various model compression factors ranging from no reduction to 180x parameter reduction. We show that training larger transformer models followed by distillation into smaller models substantially outperforms direct training of the small-sized transformer without distillation across all compression factors. Furthermore, we find that we can reduce the parameter count of the transformer model by up to 50x before observing significant performance degradation (< 1.5% absolute).
We focus on performance on cross-user generalization, unlike most previous works in sEMG process- ing which often focus on reporting performance on cross-session generalization for the 'seen' (during training) users. Additionally, these works often use the Ninapro databases evaluation, which may have insufficient data coantity/qualityy for contermporary deep learning, and/or insufficient task complexity to serve as represcntative measure of real-world performance (CTRI.-labs at Reality Saponas et al. 2008 Yang et al. 2024).
Labs 2024
Put together, these contributions provide a simple and pragmatic roadmap for practitioners to apply real- world scttings. First, vanilla transformers can be scaled up beyond the sizes explored in other works. This is crucial to maximize cross-user performance, which is a prerequisite for the wider adoption of EMG models. Second, the simplest form of distillation is effective for producing scaled models that fit into specific compute constraints with minimal performance degradation (up to 50x). Thus, we can train models that maximize the tradeoff between capacity and performance. We believe that these findings represent an important step towards practical, efficient, and deployable EMG models amenable to in-the-wild usage.
The code used for training and distilling the models available at https:///itthb. com/ facebokresordd/Eaaeee We hope that it will make it casier for the scientific community to repro- duce our results and extend this work.
2

---

Published in Transactions on Machine Learning Research (07/2025)
Reference
Task
Number of participants
Model
She al.
2010
Lower-limb movt.
3
SVM
Alkan & Gunay
2012)
Upper-arm movt
Not Reported
SVM
Atzori et al.
2016
Hand gesture
78
CNN
Wand Schmidhuber
2016
Speoch recog.
4 dev, eval
DNN HMM
He et al.
2018
Hand gesture
27
LSTM + MLP
Cai et al.
2018
Facial expr.
7
SVM
Xia al. 2018
3D limb motion est.
8
CNN RNN
Shioji ct al. 2018
Auth., hand gesture
8
CNN
Morikawa et al.
2018
Authentication
6
CNN
Ozdemir al. 2020
Hand gesture
30
CNN (ResNet-50)
Rahimian et al.
2021
Hand gesture
40
Transformer
Gaso et al. 2021
Myopathy, ALS det.
25
FC-DNN
Godoy al. 2022
Hand gesture
10
VIT
Di Nardo et al. 2022
Muscle activation
18 + 30
FC-DNN
Chen al.
2023
Finger joint est.
12
Transformer
Zabihi et al. 2023
Hand gesture
40
Transformer
Zhang ct al. 2023b
Hand gesture
20
LSTM + Transf.
Liu et al.
2024
Hand gesture
50
CNN VIT
Rani et al. 2024
Hand gesture
8 + Not Reported + 6
RF, KNN, LDA
Putro et al. 2024
Finger joint est.
5
Transformer
Eddy et al.
2024
Hand gesture Typing
612
RNN TDS-ConvNet
Sivakumar et al.
2024
108
Table 1: Prior work: most datasets are relatively small, and often classical MI. approaches older neural network architectures are used. When the number of participants is of the format X + Y, different devices or protocols were used so that they cannot be trivially combined into one dataset. In some cases, the number of participants is not reported the papers, which we denote as 'Not Reported', Most these prior works have evaluated their models on unseen users.
2
Background
This section summarizes related works, including sEMG modeling rescarch in Scction 2.1 and deep learning knowledge distillation Section 2.2 For a tabular breakdown related techniques vs our own, see the supplement (Table
2.1 Surface Electromyography
The human central nervous system initiates muscular activity by transmitting an electrical impulse along a
nerve bundle (Plonscy & Barr 2007 CTRI-labs Roality Labs 2024). Surface clectromyography (sEMG) external electrodes measure these electrical action polentials as they propagate from the nerve fiber
to the motor unit walkkhlssaaddifarallll
& Gunjan 2013 CTRL-labs at Reality Labs 2024). sEMG data
is noisy and non-stationary (Chowdhury et al. 2013: Cochrane-Snyman et al. 2016), making it difficult signal modality for machine learning tasks.
Due to the difficulty collecting EMG data, most open-source EMG datasets are small in terms users and total recording time. Furthermore, most focus on capturing isolated movements that are relatively distinct from one another, such as the flexion of different fingers. For example, despite being some of the largest and most popular sEMG datasets, the Ninapro Atzzri et al. 2014 corpus is predominantly focused
3

---

Published in Transactions on Machine Learning Research (07 (07/2025)
recognizing isolated gestures and contains only 77 subjects in its largest dataset. The EPN dataset [Benalcazar et al. 2020) is much larger but still focuses on gesture recognition, task in which relatively simple models (LSTM) seem to saturate performance (Eddy et al.) 2024), reaching 98% accuracy on cross- session gencralization and 98% on cross-user generalization. Most other datasets, such as the oncs from Amma et al. (2015 5 subjects) and (Ortiz-Catalan et al. (2013 17 subjects), are even smaller and still primarily based on single-gesture, movement recognition. In contrast, an effective human-computer interface (HCI) should have the ability disentangle multiple sequential gestures (that may overlap with each other)
and generalize across diverse set of users.
t h
u
i
C
k
b
r
W
n
X
Right hand
Left hand
0.0
0.5
1.0
1.5
2.0
2.5
time (s)
The emg2querly lask: participants lype on keyboard while sEMG activity is recorded from both hands. The goal is to map from sequences of sEMG signals to sequences of characters. Figure cropped from licensed CC BY-NC-SA.
Figure 1:
Improving on the above, Sivakumar et al. (2024) relcased a dataset that represents a significant advancement over existing sEMG benchmarks terms of its scale, lask complexity, and real-world applicability. As described in Figure []] the task is to predict key presses while touch typing on keyboard using sEMG activity alone. The dataset captures dynamic typing behavior across 108 users and 1, 135 sessions totaling 346 hours of high-resolution wrist- based sEMG recordings. The naturalistie, high-dimensional output space (key pressed a keyboard) and the larger data scale make it suitable for studying both cross-user zero-shot generalization and personalized finetuning on unseen (during training) users. For these reasons, we focus on this dataset in our experiments.
As result of both limited data availability and the inherent challenges of the modality (e.g., noise, subject- based variance), prior works have historically tended to utilize classical approaches for sEMG tasks. In some, simple machine learning methods such support vector machines, random forests, K-nearest-neighhee
linear discriminant analysis are used [Rani et al.) 2024 Atzori et al.)
2014). In others, models such as CNNs 2018 have been applied, primarily for have begun to explore the application
Ozdemir et al. 2020 Atzori et al. 2016) and LSTMs (He et al.
gesture classification on a restricted set of users. Some recent work
of transformer- based models (Lai et al. 2023 Wanga et al.] 2024 Dai et al., 2023 Zabihi et al.) 2023)
Putro al. 2024 Chen al. 2023 Zhang et al. 2022 2023a b) and vision transformers (ViTs) sEMG
data (Scheck & Schultz, 2023 Rahimian et al.) 2021 Yang et al. 2024 Godoy
2022
Montazerin al.
2023
Liu et al. 2024), but these efforts also mostly limited to small-scale datasets
50 participants,
4

---

Published in Transactions on Machine Learning Research (07/2025)
many of them on subset of Ninapro databases) and model sizes (< 10M parameters). Additionally, many introduce significant complexity such as bespoke deep learning architectures (Wagaa et al. 2024 Zabihi
et al. 2023 Putro et al., 2024 Chen et al. 2023 Zhang et al.) 2022: 2023a bj Liu et al. 2024) or use manual
feature extraction techniques on sliding windows of the sEMG signals, c.g., spectrograms (Sivakumar et al.
2024),
multivariate power frequency features (CTRI~labs at Reality Labs 2024), Hjorth parameters (Rani
ct al. 2024) and others (Zhang et al. 2022; 2023a Putro et al., [024]. We instead focus on learning vanilla
transformer-basee models that use learned sEMG featurization directly from the raw data.
Finally, the existing literature often falls short in addressing critical challenge of sEMG data: cross-user generalization. It is well-ostablished that sEMG signals, like other bio-signals, exhibit high inter-individual variability (Chowdhury et al., 2013: CTRL-labs at Reality Labs] 2024), making it essential to evaluate the robustness of models on "unseen" (heldout during training) users. However, many prior works neglect this crucial consideration, instead opting to test. their models solely on held-out trials from the same individuals contained within the training set. This does not provide a meaningful assessment of the model's ability to generalize across diverse users. We focus our evaluation on new participants unseen in the training set.
Table D reviews recent work in the domain of sEMG decoding, showing that for the most part, datasets used are small (< 100 participants, or even < 10) and modeling techniques used are often classical.
2.2 Knowledge distillation
Knowledge distillation was popularized by Hinton et al. (2015 who proposed training a small 'student" model on the outputs (logits) of a larger pretrained 'teacher' model, along with the ground-truth labels from the training data. This improved the performance of the smaller 'student' model compared to the case where the smaller model was trained from scratch. Subsequent work hypothesizod that distillation helps because the 'teacher' model's logits provide information about interclass relationships (Tang el al., 2020) as well sample difficulty (Zhao et al. 2022).
Other approaches go beyond using outputs or logits alone for distillation, especially for deeper models. These approaches push the 'student" model's intermediate layer representations "close' to the intermediate layer reprosentations in the 'teacher' model (Romero et al. 2015). In practice, this is achieved by regularizing the distance (e.g. E,, f2) between the activations of the 'teacher' and the 'student" model for pre-determined pairs of layers. These feature distillation methods have been successfully used to distill large foundation models like HuBERT with minimal performance degradation (Lee et al., 2022, Peng et al. 2023 Wang et al., 2023 Komodakis & Zagoruyko (2017) proposed applying function that maps hidden representations of CNN to 2-D allention map and training the 'student" model to imitate the allention map from the 'leacher" instead of the features from the 'teacher' Chen et al. (2021) eliminates the need for ad-hoc mapping functions by learning the optimal sttudent-to-teacher mapping layer. In contrast to directly minimizing some form of distance measure, Xu et al.] (2018) proposed training an adversarial discriminator network to distinguish between the representations from the 'teacher' and the 'student".
Even more sophisticatod approaches encode 'teacher' knowledge in higher-order relationships between mul- tiple samples. For example, Tung & Mori (2019) proposed computing an inner- product- similarity matrix between a batch of samples for both the 'teacher' and 'student' models and then training the student to match the teacher's 's matrix. Park et al.] (2019b) presented a similar approach, exploring both Euclidean and angular similarity metrics. However, the benefits of these relational distillation methods can be marginal compared to feature-based approaches, especially considering the added computational complexity.
Distillation has additionally been explored for models trained on sEMG datasets. For instance, Lai et al. (2023) distilled ResNet model ensemble into a single model instance with good success. However, their evaluation is limited to intra-user and intra-session generalization across only five users. In contrast, our work: (i) focuses on cross-user evaluation, (ii) trains on a significantly larger dataset (346 hours compared to 10 hours) and model (~130M parameters compared to ~6x6M parameter ensemble), (iii) distills the model extensively (up to 180x size reduction range compared to up to 10x range) which contributes to a better understanding of different regimes where the distillation works effectively and where it yields dimin- ishing returns. Another example is work from Wanga et al. (2024), which modifies the transformer attention with
convolutional feature projections, replaces the MLP part of the teacher
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
model with variational information bottleneck layer (with removed skip connection) and performs simulta- neous logit and feature distillation. This complexity hinders the broad applicability of the approach, which furthermore does not investigate (a) how the teacher models perform at larger scales or (b) how the distilla- tion procedure behaves when the teasher-to-student parameter reduction is larger. In contrast, we show that the simplest distillation approach works and scales, making usable for in-the-wild sEMG praclitioners.
Other research works have investigated cross-modal knowledge distillation in the context of sEMG tasks. Notably, Dai et al. (023) examined distillation between sEMG and hand gestures, while Zeng et al. (2022) explored distillation between sEMG and ultrasound. In contrast, our work concentrates on distilling knowl- edge from models trained exclusively on sEMG signals.
3 Experiments
We demonstrate that:
1. Transformer models can be elfective for sEMG tasks even when considering datasets and models that are small by modern deep learning standards.
The performance of the transformer models further improves with scale, resulting in improvements to SOTA performance.
3. Transformer models can be distilled into smaller-sized models, recovering most of the large-model performance with 50x fewer parameters.
We primarily focus on zero-shot performance on held-out users, and additionally report personalization performance (where model is fine-tuned small amount of data from held-out single user, then evaluated on held-out sessions from that user). Both cross-user generalization and personalization are key challenges for real-world sEMG tasks (CTRL-labs at Reality Labs 2024 Saponas et al. 2008 Yang et al. 2024).
3.1 Dataset
We use the emg2querty datasct (Sivakumar al. 2024) in experiments. The dataset consists of two- handed sEMG recordings from users typing on computer keyboard. The data. is labeled with the corre- sponding keystrokes, and the task is to map from sEMG sequences to character sequences. Figure 1 shows a representative example. In total, the dataset contains 346 hours of sEMG recordings across 108 unique users. The dataset is split into 100 users for training and validation and 8 held-out users for testing. For each user, we hold out validation sessions and testing sessions, then use the rest for training. In the generic setting, we train on the 100 user training set, validate on the 100 user validation set and evaluate the user testing set. In the personalization setting, for each of the users, we train on their individual training set, then validate and test on their respective validation and testing set. Sessions are windowed to form 4 second samples, padded with an additional 900 ms of past context and 100 ms of future context.
3.2 Models
We compare the performance of our transformer model against the Timo- Timo-Depth Separable Convolutional Network (TDS-Con<Net) model introduced in Hammun et al. (2019) and used by /ivakunaa et al.) (2024), which reports that the paramneter-fflciency of CDS-ConvNet allows for wider receptive fields which have proven important in emg2qwerty modeling. We use the same train, validation, and test splits as used in Sivakumar et al. 20044 and report the performance of the baseline models from that paper.
Our model architecture consists of a convolutional featurizer followed by a transformer encoder and a linear decoder. The fcaturizer always uses convolutional layers, with instance norm applied along the time axis after the first convolutional layer, and downsamples the input sEMG data (which is sampled at 2kHz) to a sequence of uras (sampled at 100 Hz). The encoder consists of series of Transformer blocks (Vaswani

---

Published in Transactions on Machine Learning Research (07 (025))
et al. 2017 whose number and width we manipulate to create larger smaller models. The transformer blocks use causal attention so that they can be used in an online streaming setup. The linear decoder converts the transformer's output to a sequence of logits. Note that unlike Sivakumar et al. (2024) which uses log- spectrograms as input features, we train our model end-to-end using raw sEMG data directly, without using any hand-designed feature engineering pipeline. Following /ivakumar et al. (2024), during training we apply channel rotation, which randomly shifts the sEMG channels by as a data augmentation technique simulate different spatial orientations of the device.
We have trained 20 different architectures, generated by permuting [2, 4, 6, 8, 10] layers and inner dimension of [128, 256, 512, 1024]. The ratio of the transformer inner dimension and the transformer foed-forward dimension is fixed at four. While we report on performance of all models in the supplement, for ease of exposition in the main text, we are concerned with three 'reference' architectures: the TINY architecture consisting of 10 layers of inner dimension 128 (about 2.2M parameters); the SMALL architecture consisting of layers of inner dimension 256 (about 5.4M parameters, close to the 5.3M TDS- Nos-ConvNNet bascline); and the LARGE architecture consisting of 8 layers of inner dimension 1024 and about 109M parameters. We use the AdamW optimizer (Loshchilov] 2017 for training all models. In the figures we additionally include other models along the size-performance Pareto frontier (i.c. ones which perform better than any model of the same or lesser parameter count). For all the experiments, we report standard deviation across multiple seeds (6 seeds for supervised training of transformer models, 3 seeds for personalization experiments, and 3 seeds for distillation experiments).
3.2.1 Supervised training of transformer models
Following Sivakumar et al. [022)) we use the connection- ist temporal classification (CTC) loss (Graves et al. 2006
to train the transformer models on the emg2qwerty task. We train the transformer models on single HPC node containing 8 32GB V100 GPUs, in Distributed Data Parallel (DDP) training scheme. We use cosine learning- rate scheduling [Loshchilov & Hutter 2017) with linear warmup for first 5% of updates. We document all the hyperparameters in the Appendix in Section
Task Loss
Logit Distill Loss
Logits
Logits
Tesch.
Stu.
3.2.2 Model distillation
We use logit distillation to distill the pretrained LARGE teacher model into a set of smaller student models of dif- ferent sizes. For the student models, we include all 20 architectures discussed above (including ones of the same size the teacher). This enables us to demonstrate the benefits of distillation at varying student model sizes. The process is depicted in Figure 2
The distillation loss Laistill is the cross-entropy loss where the student's 's output probabilities at each time step are ex- pectod match the soft targets provided by the tcacher's per- per-timestep output probabilities. We use temperature scaling factor of when computing the probabilities. The distillation loss is then combined with the task loss (i.e. CTC) to give the final training loss for the student model:
EMG
Figure An illustration of the distillation pro- cess: the smaller student model receives training signal from the logits of the larger teacher model in addition to the regular supervised loss.
1
L =
(aLaistill + 8 Brrsa)
(1)
7

---

Published in Transactions on Machine Learning Research (07 (075))
Here, and 3 are hyperparameters. We set / 1.0 and experimented with values in [0.1, 2] and found the best values to be in the [0.3, 0.5] range, we use = 0.5 throughout our distillation experiments. The rest of the distillation hyperparameters used in our experimentation are documented in Appendix B.3
Benchmark
Architecture
Parameters
CER (1%)
TDS-ConvNet
5.3M
55.57
TINY TRANSFORMER
2.2M
35.9 0.9
Generic
SMALL TRANSFORMER
5.4M
35.21 1.1
LARGE TRANSFORMER
109M
30.5 0.6
TDS
5.3M
11.39
Personalized
TINY TRANSFORMER
2.2M
9.74 0.13
SMALL TRANSFORMER
5.4M
7.9 0.06
LARGE TRANSFORMER
109M
6.8 0.07
Table 2: Cross-user performance of transformer models trained on the emg2querty dataset, showing that [a] even the TINY transformer models substantially outperform the CDS-ConvNet baseline in spite of having lewer parameters, and [b] the performance of the transformer model keeps improving as we increase the number of parameters in the model. For the transformer models, we report standard deviation across 6 sceds for Generic benchmark and across 3 seeds for the Personalized benchmark. The standard deviation for the baseline models is not reported in Sivakumar et al. (2024).
3.3 Metrics
Following (Sivakumar al. 2024), evaluate the emg2qwerty models using Character Error Rate (CER), defined the Levenshtein edit-distance between the predicted and the ground-truth sequence. It can be expressed as (S+D+1)+100 where, given the predicted and the ground-truth sequences, S is the number of character substitutions, D is the number of deletions, and I is the number of insertions between them and N is the total number of characters in the ground-truth sequence.
4 Results
4.1 Parameter-matchee comparison of transformer with TDS baseline
In Table 2 we compare the performance of our transformer models with the baseline TDS-ConwVle model. On the generic (zero-shot, cross-user) benchmark, even the TINY model outperforms the Nes-ConwNet model baseline by margin of about 20% absolute CER, in spite of having fewer than half the number of parameters the bascline. This indicates that the performance gain from the transformer model arises from the specifics of the architecture (e.g,, self and cross attention) and not merely from higher parameter counts. While the performance of the SMALL (which is about the same size of the baseline) is not very different from that of the TINY model, scaling the model more aggressively to the LARCE size does yield a further 5% improvement. This is surprising since common deep learning wisdom indicates that this level of increase in model capacity with fixed dataset size would result increased overfitting the training users, thus hurting unseen user performance.
On the personalized benchmark, where the trained model is fine-tuned are much lower across the board and therefore the absolute gains of the transformer are more modest (1.7%, 1.8% and 4.5% CER for TINY, SMALL and LARGE models respectively). However, the relative magnitude of improvement. is similar to that seen on the generic benchmark, especially for the largest model.
single heldout user's data, CERs
These performance gains over the baseline across all model sizes and benchmark support the usage of transformer architecture for teanssrannninttoon based sEMG tasks. We compare our model architecture with other common ones from the sEMG field (Time-CNN, LSTM and Vision Transformers) in Appendix C.1
8

---

Published in Transactions on Machine Learning Research (07/2025)
Model
CER (4%)
Abs. Gain (1%)
Benchmark
Architecture
Params
Standard
Distilled
TINY
2.2M
35.9 0.9
31.94 0.4
4.0
Generic
SMALL
5.4M
35.2+ 1.1
32.7# 0.5
2.5
TINY
2.2M
9.7 0.1
8.6 0.04
1.1
Personalization
SMALI
5.4M
7.9 0.06
7.1+ 0.06
0.8
Table 3: Cross-user performance of small student models on the emg2querty dataset with and without rate (CER). The 'Abs. Gain' column reflects the
distillation. Performance measured by character absolute improvement in performance from using distillation as opposed to standard supervised training for given architecture. Personalized models are personalized from the distilled student. All models see a substantial benefit (7-11% relative improvement) from the use of the distillation loss. We report standard deviation across seeds for the distillation results. Here TINY and SMALL refer to TINY TRANSFORMER and SMALL TRANSFORMER in Table 2 respectively.
40-
Generic CER
35
30
1
10
100
Number of parameters (Millions)
Supervised learning Distilled
Figure 3: Scaling curve of transformers on the emg2qwerly dataset, showing the benefit of model size across orders of magnitude from <1M to over 100M paramoters, and the benefit of distillation. A few things are of note: [a] even the smallest transformer we consider here (about 600K parameters) outperforms the 5.3M parameter
baseline (55.57 CER), in spite of having almost an order of magnitude fewer parameters; and [b] the majority of distillation benefit seen for small-but--no--oo-smmall models, with 2.2M parameter distilled model getting within 1.5% CER (absolute) of the top-performing model in spite of having 50x fewer parameters. Figure includes only models along the pareto front, i.e. ones which outperform others at the same or smaller parameter count. The vertical bars denote standard deviation across 6 seeds.
4.2 The performance of transformer models improves with scale
In Table 2 we observe that as we scale the size of the transformer model, the performance of the model continues improve even with the same amount of data. An alternate view of this is shown in Figure 3 (red line) where report on the 10 architectures that the Pareto front w.r.t. scale and performance.
9

---

Published in Transactions on Machine Learning Research (07/2025)
i.c they perform at least as well any model of the same smaller size. These models show nearly log- linear scaling curve, validating that there are performance benefits to be realized by using larger transformer models for sEMG data.
One obvious challenge with the use of these larger transformer models is that they increase the computa- tional overhead during inference, thus limiting the potential for real-time usage in an HCT (e.g. to control a prosthetic, for computer text entry, or control cursor). Distillation techniques mitigate this limitation, which we investigate in the following section.
4.3 Model distillation can be applied effectively to models for sEMG
Table shows the results of distilling small-capacity student models on the emg2qwerty dataset, using the best-performing LARGE model as a teacher. An alternate view is in Figure 3 (turquoise line), where we see that the benefit of distillation is largest for satall-btt-nttooo-mmall models. That is, for the very tiniest models, they seem unable to fully benefit, from distillation (perhaps too few layers, or too few parameters to model the teacher's distribution), while for bigger models they have enough capacity their own. Notably, the 2.2M parameter TINY achieves performance within 1.5% of the LARGE teacher in spile of having about 50x fewer parameters. We also report that while the LARGE teacher model takes 26.966 ms for inference over a single sample, the SMALL model takes 5.7002 ms only, thus providing 4.7x speedup during inference. These numbers are obtained by running inference over a single sample 1000 times to get an average runtime and then further repeating this protocol 3 times and reporting the median of the three numbers. Additional details around benchmarking are provided in the Appendix. While the specific size cutoff for real-time model may vary, distillation provided benefit across many small model sizes. These result. indicate that, for student models within the appropriate size range, the benefits of distillation are substantial in maximizing performance for given model size. Obtaining such large (50x) parameter reductions at relatively minimal performance cost represents an important step towards models that can be used for practical BCIs outside of clinical or laboralory environments.
learn the task
5 Analysis
In Section 4.1 wC report considerable improvement of our modcling approach over results reported in (Siaau- mar al. (2024). In this section we investigate which differences contributed to these performance gains. We identify and ablate two core differences between our approach and Sivakumaa et al. (2024): first the featurizer and second the encoder type. Additionally, we investigate the contribution of data augmentation techniques to the training procedure. The technical details of these experiments are outlined in Appendix B.5
We refer to the "featurizer" as all modeling parts between the raw sEMG signal and the core encoder architecture, i.c., the Transformer in our approach or the TDS model in the case of Sivaku- mar et al. 22044). A featurizer is commonly employed in the deep learning literature as way to create mcaningful vector representations of the data for the encoder to process. In the case of sEMG, part of this featurization procedure has historically often been handcrafted using transformations of the data that are considered useful for the task, (e.g. Mean Absolute Value (MAV), Variance (VAR), Root Mean Square (RMS), spectrograms, etc). In this work, we adopt data-driven approach to featurization through a CNN architecture applied directly to the raw sEMG signal, popularized by Schneider et al. 2019)) We per- form the ablation experiment between our raw sEMG signal + CNN -enncode featurization strategy to the strategy from Sivakumar et al. (024) with the TDS and Transformer encoders in Table g We see that TDS and Transformer encoders benefit 10.94 and 8.19 absolute CER improvement, respectively, from using a dala-driven fealurization approach. In other words, empirically derived features are better able to capture and encode the low-level information required for the emg2qwerty task. Importantly, this is true not only for Transformer encoders, but even for the baseline TDS encoder.
Featurizer type
A Architecture ablation In addition to the fcaturization strategy, we investigate the contribution of using a transformer encoder instead of a TDS encoder. In Table 4 we show the performance of cach encoder type. We see that the transformer outperforms the TDS encoder by 8.88 and 11.63 CER, depending on the
10

---

Published in Transactions on Machine Learning Research (07/2025)
Featurizer type
sEMG Cignal+CNN Spectrogram+ MLP
TDS Transformer
45.84 0.06
56.78 0.17
Encoder type
36.96+t 0.95
45.15 1.17
Table 4: Featurizer and Encoder type ablation on cross-user performance when trained from scratch without distillation signal. Performance is measured by character rate (CER) on the emg2querly dataset. We report standard deviation across seeds.
Input masking
None
Length-7
Length=15
None
36 + 1.04
33.96 1.27
33.05+ 0.24
Channel rotation
+
3 - 0.82
3 0.97
34.77#: 1.59
Table 5: Data augmentation ablation on cross-user performance when trained from scratch without distil- lation signal. We investigate the cross-product 3 input masking and 2 channel rotation configurations. Performance is measured by character error rate (CER) on the emg2qwerly dataset. We report standard deviation across 3 seeds.
featurizer type. We hypothesize that the transformer better able capture the sequential nature of the typing signal by encoding information both local and global timescales. The attention mechanism in the transformer may also enhance the model's ability to discard noise and other irrelevant features, known challenge in sEMG data (Chowdhury et al., 2013 Cochrane-Snyman et al. 2016).
Data augmentation ablation We also investigate contribution of data. augmentation techniques used in this work, specifically input masking, also called SpecAugment (Park et al. 2019a) used by Sivakumar ctt al. (2024)), and input channel rotation. For both of these augmentations, we evaluate the Transformer with raw sEMG signal+Conv input feature lype shown in Table 4 and show the results in Table 5 We see that using imput masking improves performance over not using the augmentation; however, there does not seem to be a clear optimal masking length between and 15. Adding channel rotation augmentation doesn't 't appear to provide substantive performance improvements over not using We see an exception that when using input masking with mask lengths of 15, in which case not having rotation augmentation improves performance by approxinately 1.7 CER, indicating a potential interaction between the two augmentation techniques.
6 Conclusion
Our work provides new state-of-the-art on the emg2querty task without increasing the parameter count over the baseline. We achieved this with simple and pragmatic recipe that can easily be followed by practitioners in other task settings. First, we scale up vanilla. Transformer architecture, which yields improved performance at all sizes investigated in this work (up to 110M parameters). Second, if bigger models cannot be deployed because of compute knowledge distillation works to bring scaled models back into a usable regime (up to 50x reduction in size) with minimal performance degradation. Third, we provide an empirical limit where this approach seems yicld diminishing returns (>50x size reduction). We perform all of the evaluations in the challenging cross-user generalization setting. We see our work as providing a pragmatic recipe applying standard and battle-tested tools to deep learning for sEMG scale.
other constraints, we show that the simplest form of
11

---

Published in Transactions on Machine Learning Research (07 (025))
Broader Impact Statement
Beyond this paper, the broader usage of sEMG, and the specific development sEMG-based textual input models, pose novel ethical and societal considerations. There are numerous societal benefits for the devel- opment of sEMG models for textual input. sEMG allows one to directly interface a person's ncuromotor intent with a computing device, which can be used to, for example, develop adaptive controllers for those who struggle to use existing computer interfaces.
Acknowledgments
Thank you to Paul Parayil Varkey, Ishita Mediratta, Brett Nelson, Roman Rädle, Daniel Licht, and Andy Chung for extensive contributions to related code and rescarch that directly helped enable this work. For discussions on techniques for modeling sEMG, thank you to Alex Gramfort, Eftychios Pneviatikakis,, Nariman Farsad, Ali Farschian, Na Young Jun, Rick Warren, Tiberiu Tesileamu, Adam Calhoun, and Alan Du. Thanks to Viswanath Sivakumar for emg2qwerty support, and to Viswanath, Dan Wetmore, and Patrick Kaifosh for reviewing the paper. Thank you to Michael Mandel and Sean Bittner for reviewing the code.
References
Ahmet Alkan and Mücahid Günay. Identification of emg signals using discriminant analysis and svm classifier. Expert systems with Applications, 39(1):44-45 47, 2012.
Christoph Amma, Thomas Krings, Jonas Böer, and Tanja Schultz. Advancing muscle-computer inter- faces with high-density electromyography. In Proceedings of the 33rd Annual ACM Conference on Human Factors Computing Systems, CHI '15, pp. 929-938, New York, NY, USA, 2015. Associa- URL https:
tion for Computing Machincry. ISBN 9781450831456. doi: 10.1145/2702123 2700250..
//doi.org/10. 1141//10222222 2702501
M Atzori, A Gijsberts, C Castellini, B Caputo, AG Hager, S Elsig, G Giatsidis, F Bassctto, and H Müller, romyogrhhyy data for non-invasive naturalll-coottrolled robotie hand prostheses. sci data. 2014; 1: 140053, 2014.
Manfredo Atzori, Matteo Cognolato, and Henning Müller. Deep learning with convolutional neural networks applied to electromyography data: A resource for the classification of movements for prosthetic hands. Frontiers in Neurorobotics, 10, 2016. ISSN 1662-5218. doi: 00.3389//nbot..216..o 00009
URL https:
//www frontiersin.
Marco E. Benalcazar, Lorena Barona, Leonardo Valdivieso, Xavier Aguas, and Jonathan Zea. Emg-cpn-612 dataset, November 2020. URL https://doi. httg///0.22//zeoodooo 4421500
Domenico Buongiorno, Giacomo Donato Cascarano, Irio De Feudis, Antonio Brunctti, Leonarda Carnimco, Giovanni Dimauro, and Vitoantonio Bevilacqua. Deep learning for processing deeyroomygga sig- nals: A taxonomy-based survey. Neurocomputing, 452:549-55 -655 2021. ISSN 0925-2312. doi: https:// doi.org/
URL
S0925231220319022
Yi Cai, Yifan Guo, Haotian Jiang, and Ming-Chun Chun Huang. Machine-learning approaches for recognizing musele activities involved in facial expressions captured by multi-channels surface electromyogram. Smart Health, 5:15 25, 2018.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 7028-7036, 2021.
Zhouping Chen, Hong Wang, Haonan Chen, and Tao Wei. Contimuous motion finger joint angle estimation utilizing hybrid somg-fig modality driven transformer-based deep learning model. Biomedical Signal Processing and Control, 85:105030, 2023. ISSN 1744-8094. doi: https://doo.orr//1.0006////
105030. URL http:://wwwsstpeeecoocoo//ccttta
12

---

Published in Transactions on Machine Learning Research (07/2025)
Reaz H. Chowdhury, Mamun Bin Ibne Reaz, Md. Alauddin Ali, Ahmad Abu Bakar, Kasturi Chellappan, and Teck Guan Chang. Surface electromyography signal processing and classification techniques. Sensors (Basel), 13(9):12431 12466, Sep 17 2013. doi: 10.3390/s13091220
K. C. Cochrane-Snyman, T. J. Housh, C. M. Smith, E. C. Hill, N. D. Jenkins, R. J. Schmidt, and G. O. Johnson. inter-individual variability in the patterns of responses for electromyography and mechanomyog- raphy during cycle ergometry using an rpe-clamp model. Eur J Appl Physiol, 116(9): 1649, Sep 2016. Epub 2016 Jun 20.
doi: 10.1007 101007//002211
CTRL-labs at Reality Labs. A generic noninvasive neuromotor interface for human-computer interaction. Riov 2024. doi: 10.1101 10.1101/2024.00 222/2224..222.58817777 URL https://wwwbborrttt hwwpwv/.o///ommeettee/////22222
07/23/2024.02 02. 23. 581779
Qingfeng Dai, Yongkang Wong, Mohan Kankanhali, Xiangdong Li, and Weidong Geng. Improved network and training scheme for cross-trial surface electromyography (semg)-based gesture recognition. Bioengi- neering, 10(1))1101, 2023.
Francesco Di Nardo, Alessandro Nocera, Alessandro Cucchiarelli, Sandro Fioretti, and Chiara Morbidoni. Machine learning for detection of muscular activity from surface emg signals. Sensors (Basel), 22(9))3393, Apr 2022. doi: 10.3390/s22093338
Ethan Eddy, Evan Campbell, Scott Bateman, and Erik Scheme. Big data in myoelectric control: large multi-user models enable robust zero-shot emg-based discrete gesture recognition. Frontiers in Bioengineering and Biotechnology, 12, 2024. ISSN 2296-4185. doi: 10.3389/fbioe.2222
URL https://wwe wwwntiersin.cnoomm http://hooooooooaaaaaooooooaonnn http:///eriiiaaaooooooocooooo/rr 3389/fbioe. 2wt.fbieee222....... 1463377
Mekia Shigute Gaso, Selcuk Cankurt, and Abdulhamit Subasi. Electromyography signal classification us- ing deep learning. In 2021 16th International Conference Electronics Computer and Computation (ICECCO), pp. 1 1-6, 2021.
doi: 10. 10.110//CCCCCC00333000 /T/////CCCC33.3.222.0000 20219999000.0112968800
Ricardo V. Godoy, Gustavo J. G. Lahr, Anany Dwivedi, Tharik J. S. Reis, Paulo H. Polegato, Marcelo Becker, Glauco A. P. Caurin, and Minas Liarokapis. Ellecrroyygraphyy-beee robust hand motion classification employing temporal multi-channel vision transformers. IEEE Robotics and Automation Letters, 7(4):
10200-10207, 2022. doi: 10.1109/T.R, R..100//.AA.222231092223
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal clas- sification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pp. 369-376, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832, doi: 10. 1143944 URL
https://aoi.org: 10//g///..00 1111//111114//1484444 11433999 1143891
Awni Hanmun, Ann Lee, Qiantong Xu, and Ronan Collobert. time-depth separable convolutions. arXio preprint 19X4 (al::9944.22111 02611, 2019.
speech recognition with
Yunan He, Osamu Fukuda, Nan Bu, Hiroshi Okumura, and Nobuhiko Yamaguchi. Surface emg pattern recognition using long short-term memory combined with multilayer perceptron. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5636- 5639, 2018. doi: 10.1109/ BBW.://...........11....
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. URL http:://gg..oaaaattc.cacoarrrrrp
Nikos Komodakis and Sergey Zagoruyko. Paying more attention attention: improving the performance of convolutional neural networks via attention transfer. In ICLR, Paris, France, June 2017. URL https: 7/enpc. haa.....ccicccc//l111188777)9
13

---

Published in Transactions on Machine Learning Research (07/2025)
Wenqiang Lai, Qihan Yang, Ye Mao, Endong Sun, and Jiangnan Ye. Knowledge distilled ensemble model for semg-based silent speech interface. In TEEE EUROCON 2023-20th International Conference on Smart Technologies, pp. 117 122. LEEE, 2023.
Yeonghyeon Lee, Kangwook Jang, Jahyun Goo, Youngmoon Jung, and Hoirin Kim. Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning, 2022. URL https://arxiv.org
Wei Li, Ping Shi, and Hongliu Yu. Gesture recognition using surface electromyographyy and deep learning for prostheses hand: state-of-the-art, challenges, and future. Frontiers in neuroscience, 15:621885, 2021.
Xiaoguang Liu, Lijian Hu, Liang Tie, Li Jun, Xiaodong Wang, and Xiuling Liu. Integration of convolutional neural network and vision transformer for gesture recognition using semg. Biomedical Signal Processing and Control, 98: 18:1066,6 2024. ISSN 1746-8094 doi: http::/wwwwscceeceeeeee htt.a//ccc...cc..ctttttete/////.
URL
Loshchilov. Decoupled weight docay regularization. arXiv preprint arXiv:1111.0541 2017.
llya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International
Conference on Learning Representations, 2017. URL https://openrevies http://www....e.ooooooooodoa//e
Changjia Lu, Xin Xu, Yingjie Liu, Dan Li, Yue Wang, Wenhao Xian, Changbing Chen, Baichun Wei, and Jin Tian. An embedded electromyogram signal acquisition device. Sensors, 24(13), 2024. ISSN 1424-8220. doi: 10.3390/S241341066 URL https::////e http::://wwwmmm...... cctm://2222-222/22//333/43212
Behnam Mokhlesabadiffrrhaan and V. K. Gunjan. EMG Signals Characterization in Three States of Con- traction by Fuzzy Network and Feature Patraction. Springer Singapore Pte. Limited, 2015th edition, 2015. doi: 10.1007/975-991-287-9200
Mansooreh Montazerin, Elahe Rahimian, Farnoosh Naderkhani, S Farokh Atashzar, Svetlana Yanushkevich, and Arash Mohammadi. Transformer-basee hand gesture recognition from instantaneous to fused neural decompasition of high-density emg signals. Scienlific reports, 13(1): 10000, 2023.
Shion Morikawa, Shin-ichi Ito, Momoyo Ito, and Minoru Fukumi. Personal authentication by lips emg using dry electrode and cnn. In 2018 IEEE International Conference on Internet of Things and Intelligence 101.:/AAAASI......1..11
System (IOTAIS), pp. 180 183, 2018 doi:
Max CatilaCatalan, Rickard Bránemark, and Bo Hākansson. BioPatRec: A modular research platform for the control of artificial limbs based on pattern recognition algorithms. Source Code for Biology and Medicine, 8:11, 2013. doi: 10.1186 1758//1111--4738-11111
Mehmet Akif Ozdemir, Deniz Hande Kisa, Onan Guren, Aytug Onan, and Aydin Akan. Emg based hand gesture recognition using deep learning. In 2020 Medical Technologies Congress (TIPTEKNO), pp. 1-4, 2020. doi: TT100///TTTETNO00000.... 222009990002222099999964444
Daniel S Park, William Chan, Yu Zhang, Chung-Cheee Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904, 08779, 2019a.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVE conference on computer vision and paltern recoguition, pp. 3967 3976, 2019b.
Yifan Peng, Yui Sudo, Shakeel Muhammad, and Shinji Watanabe. Dphubert: Joint distillation and pruning of self-supervised speech models, 2023.
/ttp///rriivv.orga///////// 17651
Robert Plonsey and Robert C. Barr. Bioelectricity: A quantitative approach. Springer, 2007.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Hock, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606 624, 2023.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Nur Achmad Sulistyo Putro, Cries Avian, Setya Widyawan Prakosa, Muhammad Izzuddin Mahali, and Jenq-Shiou Leu. Estimating finger joint angles by surface emg signal using feature extraction and transformer-base based deep learning model. Biomedical Signal Processing and Control, 87:105447, 2024. ISSN com/
1746-8094. doi: htttps://doi.orr///o//
wwwp::/oooe............... 20.3..0022.1154., URL
Elahe Rahimian, Soheil Zabihi, Amir Asif, Dario Farina, S. Farokh Atashzar, and Arash Mohammadi. Temgnet: Deep transformer-basee decoding of upperlimb seing for hand gestures recognition, 2021. URL
http:://aaxiiv.org/ab////// 12379
Parul Rani, Sidharth Pancholi, Vikash Shaw, Manfredo Atzori, and Sanjcev Kumar. Enhancing gesture classification using active emg band and advanced feature extraction technique. IEEE Sensors Journal, 24(4):5246-5255 2024. doi: (0000/JSEE..023334441
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Ben- gio. Fitnets: Hints for thin deep nets. ICLR, abs/ (bs/1412.6550, 2015. URL http:::://pppi.mmmttiiiaaaaeoeie
htg/CorpusID:27220 2723173
T Scott Saponas, Desney Tan, Dan Morris, and Ravin Balakrishnan. Demonstrating the feasibility of using forearm clectromyography for muscle-cmpputer interfaces. In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 515 524, 2008.
Kevin Scheck and Tanja Schultz. Multi-speaker speech synthesis from electromyographic signals by soft speech unit prediction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for spooch recognition. arXiv preprint 19X4:1904005800 2019.
Qingshan She, Zhizeng Luo, Ming Meng, and Ping Xu. Multiple kernel learning svm-based ein pattern classification for lower limb control. In 2010 11th International Conference on Control Aulomation Robotics & Vision, pp. 2109 2113. IEEE, 2010.
Ryohei Shioji, Shin ichi Ito, Momoyo Ito, and Minoru Fukumi. Personal authentication and hand motion recognition based on wrist cmg analysis by a convolutional neural network. 2018 IEEE International Conference on Internet of Things and Intelligence System (IOTAIS), pp. 184 184-188, 2018. URL https: //api. www.ssmmencicccolar.....m arrg/CorpusID:577555
Viswanath Sivakumar, Jeffrey Seely, Alan Du, Sean Bittner, Adam Berenzweig, Amuoluwapo Bolarinwa, Alex Gramfort, and Michael Mandel. EMG2QWERTY: A Large Dataset with Basclines for Touch Typing using Surface Electromyygraphy In Proceedings of the Neural Information Processing Systems (NewrIPS), 2024.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXie preprint 00X5:22002.0353, 2020.
Frederick Tung and Greg Mori. Similarity-preservvi knowledge distillation. In Proceedings of the JEEE/CT /CF international conference on computer vision, pp. 1374-1374, 2019.
A Vaswani, N Shazeer, N Parmar, Uszkoreit, Jones, Gomez, L Kaiser, and I Polosukhin. Atlention is all you need. In NIPS, 2017.
Michael Wand and Jürgen Schmidhuber. Deep neural network frontend for continuous emg- based speech recognition. In Interspeech, pp. 3032 3036, 2016.
Yujin Wang, Changli Tang, Ziyang Ma, Zhishong Zheng, Xie Chen, and Wei-Qiang Zhang. Exploring effective distillation of self-supervised speech models for automatic speech recognition, 2023. URL https: 7/arxiv.org. wtwp://wwwrr/ab//2....om
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Zefeng Wanga, Bingbing Hub, Junfeng Yaoa, and Jinsong Sua. Lightweight transformer for seig gesture recognition with feature distilled variational information bottleneck. 2024.
L. N. Wimalasena, J. Braun, M. R. Keshtkaran, D. Hofmann, J. Á. Gallego, C. Alessandro, M. C. Tresch, T. E. Miller, and C. Pandarinath. Estimating muscle activation from emg using deep learning- based dynamical systems models. J Neural Eng, 19(3), May 2022. doi:
/ac6669..
Peng Xia, Jie Hu, and Yinghong Peng. Emg- based estimation of limb movement using deep learning with recurrent convolutional neural networks. Artificial Organs, E270))E67- 2018. doi: https:/ https:///ooo.ooo /tt.//oooo.o00.....////
1111 /all///rr.000 URL https://oonlimellbbrayy wiley. com hww.d///abbbsa 1101////000111/0111......1111110 13004
Zheng Xu, Yen-Chang Hsu, and Jiawei Huang. Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks, 2018. URL
00513
Jehan Yang, Maxwell Soh, Vivianna Lieu, Douglas Weber, and Zackory Erickson. Emgbench: Benchmarking out-of-distribution generalization and adaptation for electromyography, Advances in Neural Information Processing Systems, 37:50313 17:44333-5034,, 2024.
Soheil Zabihi, Elahe Rahimian, Amir Asif, and Arash Mohammadi. Trahgr: Translormer for hand gesture recognition via electromyography. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 31:4211 4224, 2023. doi:
Jia Zeng, Yixuan Sheng, Yicheng Yang. Ziliang Zhou, and Honghai Liu. Cross modality knowledge distillation between a-mode ultrasound and surface electromyography. IEEE Transactions Instrumentation and Measurement, 71:1-9, 2022.
Jiaxuan Zhang, Yuki Matsuda, Manato Fujimoto, Hirohiko Suwa, and Keiichi Yasumoto. Feasibility analysis of semg recognition via channel-wise transformer. Tn 2022 IEFE 11th Global Conference on Consumer Electronics (GCCE), pp. 105-066.0 LEEE, 2022.
Jiaxuan Zhang, Yuki Matsuda, Manato Fujimoto, Hirohiko Suwa, and Keiichi Yasumoto. Movement recog- nition via chamnolactivvtiion-iies semg attention. Methods, 218:30-47, 2023a.
Wenli Zhang, Tingsong Zhao, Jianyi Zhang, and Yufei Wang. Lst-emg-net: Long short-term trans- former feature fusion network for semg gesture recognition. Frontiers in Neurorobotics, 17, 2023b. URL
ISSN 1662-5218. doi: 0033389/nbbo..222311121
wwt.rrantterriroooororrrraaaaaa. org/journals/
patrrorobotics/artticee https//0.....//oooooooooo 2023. 1127338
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 11953-11962, 2022.
16

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Appendix
In Section B we lay out all of the experimental details of the results presented in this work and in Section we show additional results which were not presented in the main text.
The appendix organized follows:
A Extended related works
Table 6: Overview of related works to this paper along our contribution axes. For the model axis, we distinguish 4 subcategories: Architecture, 'Vanilla" (whether it the architecture is used as-is or modified specifically for sEMG), Raw features (whether the model processes sEMG signal a handcrafted represen- tation of it) and the number parameters. For the distillation axis, we distinguish between subcategories: whether does distilaation at all, 'Vanilla' (again, used as-is or modified specifically for sEMG) and the model size reduction achieved. Along the evaluation axis, we distinguish between 2 subcategories: Whether is it evaluating cross-user generalization and whether the core evaluation of the approach done Ni- napro databases.
Model
Distillation
Evaluation
Paper
Arch
Vanilla
Features
Does KD?
Vonilla
Stse reduction
Croes-user?
Ninapro?
(ensemblee)*
[0-10'x
specifie
<100k
YTS
VIT
6M
NO
Toundorase
specilied
NO
NO
VIT
specilied
NO
NO
YES
<500k
NO
NO
NO
Toanforacce
YES
<IM
NO
NO
YES
Toanforasee
NO
NO
Not specified
NO
NO
YIS
Tranffornce
NO
Y1S
Not spectfied
NO
NO
Transformer
NO
NO
spectfied
NO
NO
Y1S
Tranformee
NO
NO
<500k
NO
NO
YIS
Trandhemeer
NO
V18
Not specifie
NO
NO
V=S+-abbrr
VIT
NO
YES
Not specified
NO
NO
YES+athors
The model size reduction specified Lai
2023
infer approximation the reduction from the
model memory footprint (assuming FP32 weights).
B Experimental details
We describe the hyperparameters and model details in the following subsections. We provide constant hyper- parameters in one table and hyperparameters that are subject to vary in separate table. In Section B.2 we
list the hyperparameters of the supervised learning models presented in this paper, ie., without distillation signal; in Section B.3 we list the hyperparamotors the distilled models; and in Scction B.4 we describe the hyperparameters the personalization experiment.
B.1
Dataset details
In Table we detail the aggregated statistics of the emg2qwerty dataset.
B.2 Supervised learning details
In the supervised learning experiments, train a grid of model with sizes [128, 256, 512, 1024] transformer hidden representation size and [2, 4, 6. 8, 10] layers. For each of the hidden representation sizes, set the transformer feed-forward dimension such that the food-forward ratio (dg/dhsddem) is maintained at 4. For cach of these configuration, we launch multiple learning rales ((3--3, le 3, 3e-4, 4, le-4]) across 6 different seeds. Seeding is used to determine dataloading order and model initialization. We train all models to completion (200 epochs) and evaluate the model on the training and validation at the end of every epoch. For each model, we artificially carly-stop the model post-hoc by choosing the epoch with the lowest validation CER and record the test set CER of the model for this epoch. The rest of the training-related hyperparameters
17

---

Published in Transactions on Machine Learning Research (07/2025)
Table emg2qwerty dataset statistics
Total subjects
108
Total sessions
1, 135
Avg sessions per subject
10
Max sessions per subject
18
Min sessions per subject
Total duration
346.4 hours
Avg duration per subject
3.2 hours
Max duration per subject
6.5 hours
Min duration per subject
15.3 minutes
Avg duration per session Max duration per session
18.0 minutes
47.5 minutes
Min duration per session
9.5 minutes
Avg typing rale per subject
265 keys/min
Max typing rate per subject
439 keys/min
Min typing rate per subject
130 keys/min
Total keystrokes
5, 262, 671
(specifically dropout probabilities, weight decay and learning rate schedule) were chosen according to best values in prior experimentation with a fixed model size.
We aggregate the results by first taking the validation and test CER averages across seeds, then pick the best aggregated validation learning rate value and report the average and standard deviation of the test results. The chosen hyperparameters are reported in Table 9 and the test CER reported in Table 21
B.3 Distillation details
In the distillation experiments, we train the same grid of model width and depth as in the supervisod learning experiments (Section B.2 to use as the student model. The teacher model is chosen by picking the best performing model from the best model configuration from the supervised learning experiments in Table 21 For cach of the hidden representation sizes of the student model, we set the transformer feed-forward dimension such that the feed-forward ratio (dr/dhideen) is maintained at 4. For each of these 3,3e - 4)) across 3 different seeds. Seeding is
configurations, we launch multiple learning rates (Bee - 3, le -,-3,3e used to determine dataloading order and model weight initialization. We train all models to completion (200 epochs) and evaluate the model on the training and validation at the end of every epoch. For each model, we select the checkpoint with the lowest validation CER and record the test set CER of this checkpoint. The rest of the training-related hyperparameters (specifically distillation penalty weight, student dropout probabilities, weight decay and learning rate schedule) were chosen according to best validation metrics in prior experimentation with a fixed model size.
We aggregate the results by first taking the validation and test CER averages across seeds, then pick the best aggregated validation learning rate value and report the average and standard deviation on the test results. The chosen hyperparameters are reported in Table [ and the test CER reported in Table 22
B.4 Personalization details
In the personalization experiments, we focus on our three highlighted architecture configurations, i.e., TINY, SMALL and LARGE. For each architecture, we pick three models from the supervised learning experiments, irrespective of hyperparameters and scods, according to the best validation performance. We refer to these three models as seed A, B and C. For the TINY and SMALL architecture, we repeat this procedure with the dist illation set of experiments. We refer to these models being from the "Distillation" origin, as opposed
18

---

Published in Transactions on Machine Learning Research (07/2025)
Table Supervised learning task details and hyperparameters held constant for models from Table
Input sEMG channels
32
Data
Window length Padding
8000
[800, 200)
Featurizer channels
128, 64, 64
Featurizer kernels
[11, 3,
Featurizer strides
[5, 2,
Encoder feed-forward ratio
Encoder attentions heads
16
Architecture
Tokenizer
Character-level
Vocab size
99
Encoder hidden, attention and activation dropout
0.2
Encoder feature projection dropout
0.2
Encoder final layer dropout
0.2
Epochs
200.0
Training
Effective batch size
640
Encoder causal altention
True
Learning rate schedule Learning rate warmup ratio
linear warmup + cosine decay
0.05
Optimizer
Weight decay
0.2
CTC zero infinity
True
Gradient clipping
0.1
Torch version
2.3.1 cul211
Software
Transformers version
4.36.2
to the "Supervised" origin for the models taken from the supervised learning experiments. For each of those models, initialize the personalization models from the chosen chockpoint and we launch multiple learning rales (Be-- 3, le 3, 3e 4f) across different seeds for each of the 8 personalization users. Seeding is used to determine dataloading order and model weight initialization. We train all models to completion (100 epochs) and evaluate the model on the training and validation sets at the end of every epoch. For cach model, we select the checkpoint with the lowest validation CER and record the test set CER of this checkpoint. The rest of the training-related hyperparameters (specifically student dropout probabilities, weight decay and learning rate schedule) were chosen according best validation metrics in prior experimentation with fixed model size.
We aggregate the results by first taking the validation and test CER averages across the personalized users and seeds. We then pick the best aggregated validation learning rate value and report the average and standard deviation on the test results. The chosen hyperparameters are reported in Table [3] and the test CER reported in Table 23 Table 13 also reports the supervised or distillation learning rate used by the seed generic model the personalized was initialized from.
B.5 Ablation experiments
In the ablation experiments, we focus single architecture configurations for each investigated variations, ie., Raw sEMG/spectrogram,, Transformer/TDD
With/ without channel rotation and with/withoo input masking. All of the configuration for these experiments can be found in Table 14 for the architecture ablation and Table 16 for augmentation. For each of these configuration, we launch two learning rales ([le-3, 3e 3e-4]) across 3 different seeds. Seeding is used to determine dataloading order and model weight initialization. We train all models to completion (number of epochs mentioned the tables) and evaluate the model on the training and validation sets the end of every epoch. For each model, we select the checkpoint with the lowest validation CER and record the test set CER of this checkpoint. The rest of the training-related
19

---

Published in Transactions on Machine Learning Research (07/2025)
Table 9: Supervised learning task hyperparameters that vary for models from Table
Architecture
Encoder hidden size
Encoder layers
Optimizer learning rate
128
2
le-03
128
4
1e-03
128
6
1e-03
128
8
1c-03
Tiny
128
10
1e-03
256
2
1e-03
256
1c-03
Small
256
6
1e-03
256
8
1e-03
256
10
3e-04
512
2
1e-03
512
4
3e-04
512
6
3e-04
512
8
3e-04
512
3e-04
1024
2
1e-03
1024
4
3e-04
1024
6
3e-04
Large
1024
8
3e-04
1024
3c-04
hyperparameterss (specifically student dropout probabilities, weight decay and learning rale schedule) were chosen according to best validation metrics in prior experimentation with fixed model size.
We aggregate the results by first taking the validation and test CER averages across seeds, then pick the best aggregated validation learning rate value and report the average and standard deviation of the test results. The chosen hyperparameters are reported in Table 15 and 17 while the test CERs are reported in Table and
C
Additional results
we show a comparison with the transformer architecture used in this work with other common architectures the sEMG field. In Section C.2 we show training curve samples from our experiments.
In Scction C.1
Section C.3 show the results for the full model scaling grid explored. In Section C.4 we present measures of the inference speed of our models.
C.1 Additional baselines
In this section, we compare the TRANSFORMER architecture used throughout this work other architecture common in the sEMG literature. We investigate three models: Time Convolutional Neural Network (CNN), a Long Short-Term Memory (LSTM) network, and Visual Transformer (VTT). All architectures are constructed as to have about the same number of parameters as the
baseline from Sivakumar ct al. (2024). We provide high level descriptions of each in the following paragraphs, to get more details see Table and Table 20
Time-CNN
This model uses a raw LEMG+CNY featurizer (same as our TRANSFORMER architecture) and time-wise convolutional layers as encoder, each with a kernel size of and stride of with channel dimensions of [512.512.512,236] Decoding done with linear layer to make predictions over the dictrynary.
20

---

Published in Transactions on Machine Learning Research (07/2025)
Table 10: Distillation task details and hyperparameters held constant for models from Table
22
Input sEMG channels
32
Data
Window length
8000
Padding
[1 [1800, 200]
Featurizer channels
[ 1128, 64, 64
Featurizer kernels
11, 3, 3
Fealurizer strides
[,, 2, 2
Encoder attentions heads
Text Tokenizer
Character-level
Student arch.
Vocab size
99
Encoder hidden, attention and activation dropout
0.2
Encoder feature projection dropout
0.2
Encoder final layer dropout
0.2
Featurizer channels
[128, 64, 64]
Featurizer kernels
[11, 3,
Featurizer strides
[5, 2,
Encoder hidden size
1024
Encoder feed-forward ratio
4
Teacher arch.
Encoder layers
8
Encoder convolutional dim
[64]
Encoder attentions heads
Text Tokenizer Vocab size
Character-level
99
Encoder hidden, attention and activation dropout
0.0
Encoder feature projection dropout
0.0
Encoder final layer dropout.
0.0
Epochs
200
Training
Effective batch size
640
Encoder causal attention
True
Learning rate schedule Learning rate warmup ratio
linear warmup + cosine decay
0.05
Optimizer
Weight decay
0.1
CTC zero infinity
True
Gradient clipping
0.1
Distillation loss weight
0.5
Distillation loss (logits)
Cross Entropy
Torch version
2.3.1 13311ccu121 4.36.2
Software
Transformers version
LSTM
This model uses sEMG+CNN featurizer (same TRANSFORMER architecture) and a uni-directional LSTM network as encoder with 10 layers and dimension of 256. Decoding is done with linear layer to make predictions over the dictionary.
VIT
This model employs a transformer encoder architecture (same our TRANSFORMER architecture). Similar to the TRANSFORMER architecture, the featurization is CNN-based applied on raw sEMG signal, but it uses non-overlapping 10ms patches that include all EMG channels. We use 8 layer encoder, and we do a simple sweep across the dimension of the transformer to peek into the scalability of the model. Decoding done with linear layer make predictions over the dictionary.
We show the results of these experiments in Table 18
21

---

Published in Transactions on Machine Learning Research (07/2025)
Table 11: Distillation hyperparameters that vary for models from Table 22
Student arch.
Student encoder hidden size
Student encoder layers
Optimizer learning rate
128
2
1e-03
128
4
1e-03
128
6
1e-03
128
8
1c-03
Tiny
128
10
1e-03
256
2
1e-03
256
1c-03
Small
256
6
1e-03
256
8
1e-03
256
10
1e-03
512
2
3e-03
512
4
1e-03
512
6
1e-03
512
8
le-03
512
10
3e-04
1024
2
1e-03
1024
4
1e-03
1024
6
3e-04
Large
1024
8
3e-04
1024
10
3c-04
In Table 1, we see that the transformer model demonstrates similar performance to Vision Transformer (ViT) architecture. In contrast, the Time-CNN model exhibits significantly poorer performance compared to the other architectures. We attribute this to the Time-CNN' finite (and short) receptive field, which limits its ability to utilize past context from the sequence for predictions. Although IDS-ConvNet is also based on CNN layers, it is less affected by this limitation due its design, which is tailored to maintain larger receptive fields (Hannun al. 2019).
The LSTM (with Raw sEMG+CNN featurizer) model performs reasonably well, surpassing the previous baseline (with Spectrogram+MLLP featurizer) from Sivakumar et al. (2024). However, this improvement may largely be due to the enhanced featurizer (Raw 3EMG+CNT vs Spectrogram From the ablation experiments with TDS encoder performs about the same as the Raw sEMG+CNN featurizer with LSTM encoder model investigated here (i.e., about 45 CER).
Table g of the Section 5 we see that Raw sEMG+CNT featurizer
C.2 Sample training metrics
We show training curves from some of experiments assist others to more easily reproduce results. In Figure and we show training loss along with validation (cross-session) and test (cross-user) generie CER for the supervised and distillation tasks respectively. In Figure 0 show personalization training loss and personalized CER for three of the eight personalization users.
C.3 Full model grid scaling results
Model architecture exploration In Figure we show the results for all model shapes investigated in this work. We use this grid of shapes to compute the parelo front, i.e. ones which out perperform others at the same or smaller parameter count, presented in Figure 3 Our grid extends from an transformer hidden representation size of 128 1024 and a number of transformer layer from 10. The fcaturization module which first converts raw EMG into features feed into the transformer kept fixed for all models (see Table Slan and Table 10 for the exact configuration). The CER results for generic models (from Figure 7 along
22

---

Published in Transactions on Machine Learning Research (07/2025)
Table 12: Personalization task details and hyperparameters held constant for models from Table
Input sEMG channels
32
Data
Window length Padding
8000
[1800, 200]
Featurizer channels
128,, 64, 64
Featurizer kernels
[11, 3, 3
Featurizer strides
5, 2, 2
Encoder feed-forward ratio
Encoder attentions heads
16
Architecture
Tokenizer
Character-level
Vocab size
99
Encoder hidden, attention and activation dropout
0.2
Encoder feature projection dropout
0.2
Encoder final layer dropout
0.2
Epochs
100.0
Training
Effective batch size
640
Encoder causal altention
True
Learning rate schedule Learning rate warmup ratio
linear warmup + cosine decay
0.05
Optimizer
Weight decay
0.0
Training warmup ratio
0.05
CTC zero infinity
Truc
Gradient clipping
0.1
Torch version
2.3.1 2.3.1+cu121
Software
Transformers version
4.36.2
Table 13: Personalization hyperparameterss that vary for models from Table 23
Architecture
Tnit. origin
Init. seed
Generic training Ir
Personalization training Ir
Tiny
Distillation
seed A
1e-03
3e-04
Tiny
Distillation
seed B
le-03
3e-04
Tiny
Distillation
seed C
1e-03
3e-04
Tiny
Supervised
seed A
1e-03
3e-04
Tiny
Supervised seed B
1c-03
3e-04
Tiny
Supervised
seed C
le-03
3e-04
Small
Distillation
seed A
1e-03
3e-04
Small
Distillation
seed B
1c-03
3c-04
Small
Distillation
seed C
1e-03
3e-04
Small
Supervised seed A
1e-03
3e-04
Small
Supervised seed B
3e-04
3e-04
Small
Supervised seed C
3c-04
3c-04
Large
Supervised
seed A
3e-04
3e-05
Large
Supervised
seed B
3e-04
1e-04
Large
Supervised seed C
3e-04
1e-04
with their standard deviation can be found in Table 21 for supervised learning and Table models.
22 for distilled
Distillation improvement In Figure & we show the generic CER improvement observed through distilla- tion across transformer model width and depth.
23

---

Published in Transactions on Machine Learning Research (07 (025))25)
Table 14: Constant hyperpameter for the featurizer and encoder architecture ablation experiment Table
Input sEMG channels
32
Data
Window length Padding
8000
[1800, 200]
CNN channels
1288, 256, 384]
Raw SEMG+CNN featurizer
CNN kernels
[11, 3,
CNN strides
[5, 2, 2]
MTT featurizer
ML.P dims
[884]
feod-forward ratio
Num. layers
hidden size
384
Transformer encoder
attentions heads
12
hidden, attention and activation dropout
0.2
feature projection dropout
0.2
final layer dropout
0.2
Encoder block channels
[24, 24, 24, 24]
TDS encoder
Encoder kernel width
32
Tokenizer
Character-level
Task
Vocab size
99
Epochs
200.0
Training
Effective batch size
640
Encoder causal attention
True
Learning rate schedule
linear warmup cosine decay
Learning rate warmup ratio
0.05
Optimizer
Weight decay
0.2
CTC zero infinity
True
Gradient clipping
0.1
Torch version Transformers version
2.3. cc1111
Software
4.36.2
Table 15: Varying hyperpameter for the featurizer and encoder architecture ablation experiment in Table
Featurizer
Encoder
Optimizer learning rate
Raw sEMG+CNN featurizer
Transformer
0.001
Spectrogram+ featurizer
Transformer
0.001
Raw CEMG+CNN featurizer
TDS
0.0003
Spectrogram++MLP featurizer
TDS
0.001
Numerical results Following the full set of numerical performance reported in this paper. In Table 21 and 22 annotate the TINY, SMALL and LARGE architecture which corresponds to the archilecture high- lighted in Table 3 of the main paper.
C.4 Inference speed
We measured the inference speed of the highlighted TINY, SMALL and LARGE in Table Aableemoder model architecture assess their viability be run in real time and see how much model scale influences inference speeds. We present the inference speeds in Table 24
24

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 16: Hyperpameter for data augmentation ablation experiment in Table 5
Input sEMG channels
32
Data
Window length Padding
8000
[1800, 200)
Featurizer channels
[ [128, 256, 256]
Featurizer kernels
[11, 3, 3]
Featurizer strides
[5, 2, 2
Encoder feed-forward ratio
Num. Encoder layers
Architecture
Encoder hidden size
256
Encoder attentions heads
Tokenizer
Character-
Vocab size
99
Encoder hidden, attention and activation dropout
0.2
Encoder feature projection dropout
0.2
Encoder final layer dropout
0.2
Epochs
200.0
Training
Effective batch size
640
Encoder causal attention
True
Learning rate schedule Learning rate warmup ratio
linear warmup cosine decay
0.05
Optimizer
Weight decay
0.2
CTC zero infinily
True
Gradient clipping
0.1
Torch version
2.3. 2.3.1 c.l121
Software
Transformers version
4.36.2
Table 17: Varying hyperpameter for the featurizer and encoder architecture ablation experiment in Table
Channel rotation
Input masking
Optimizer learning rate
None
None
0.001
+ 1
None
0.001
None
Length=15
0.001
+ 1
Length=15
0.001
None
Length=7
0.001
+ 1
Length=7
0.001
Naive streaming inference In the most naive case of streaming inference, we pass full window length of data (4 second in emg2qwerty) to the model and exclusively the final prediction, then repeat this process for the next prediction.
Applying this naive streaming inference paradigm to single-window inference speed in Table 24 tells us that the LARGE architecture could not be inferable in real-time due to the maximum inference frequency being approximately / 27ms = 37Hx.
Note that under this setting one can trade off latency with inference speod, for example by using the last n predictions from window predictions, which alleviates the requirements on inference speed by nx at the cost of increasing latency of the output by nx.
25

---

Published in Transactions on Machine Learning Research (07/2025)
Architecture
Parameters
CER (4%)
TDS-ConvNet
5.3M
55.57
Time-CNN
5.9M
70.14 +0.35
LSTM
5.3M
45.96 #1.02
VIT (d-128)
1.8M
37.29 #1.6
VIT (d=256)
6.9M
35.29 +0.7
VIT (d=512)
27M
34.73 #1.0
TINY TRANSFORMER
2.2M
35.9 +0.9
SMALL TRANSFORMER
5.4M
35.2 #1.1
LARGE TRANSFORMER
109M
30.5 +0.6
Table 18: Cross-user generic performance of common architectural baselines on the emg2qwerty dataset. The baselines are compared to the TDS model from Sivakumar et al. (2024) and the TRANSFORMER model from the rest of our analysis. We report the standard deviation across seeds for the Simple CNN, LSTM and ViT baselines, and 6 seeds for the TRANSFORMER architecture. The standard deviation for the baseline models not reported in Sivakumar al. 2024
Training
Validation (Cross-session)
Testing (Cress-user)
tecture
Epochs
(a) Loss
Validation (Cross-session)
Testing (Cross-user)
75
ConcicC CER
Large
50-
Small
100
150
200
50
100
150
200
Epachs
(b) Generic CER
Figure 4: Supervised training sample training curves for the TINY, SMALL, LARGE architecture. Validation is done with unseen sessions from the training users while lesting is done across unseen sessions from the unseen users.
Accelerated strcaming inference In a more sophisticated implementation of streaming inference, one could further improve the performance of the models by using optimizations like KV Caching (Pope el. al. 2023 or designing custom kernels. These techniques are outside the scope of our work.
26

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Table 19: Hyperpameter for baseline experiment in Table 18
Input sEMG channels
32
Data
Window length
8000
Padding
[1800, 200]
Featurizer channels
[384, 384, 384]
Featurizer kernels
[11, 5, 5]
Featurizer strides
[5, 2,
CNN Encoder input channels
384
Time-CNN
CNN Encoder channels
512, 512, 512, 256
CNN Encoder kernels
[5 5, 5, 5]
CNN Encoder strides
[1, 1, 1,
Linear Decoder input size
256
Featurizer channels
[128, 128, 128)
Featurizer kernels
[11, 3,
Featurizer strides
[5, 2, 2]
LSTM
LSTM Encoder input channels
128
LSTM Encoder hidden dim
128
LSTM Encoder num. layers
10
Linear Decoder input size
256
Featurizer channels
[128, 64, 64]
Featurizer kernels
[5, 2, 2)
Featurizer strides
[5. 2, 2
Encoder feed-forward ratio
VIT Encoder hidden size
depends (sce name
VIT (d=*)
Num. Encoder layers
8
ViT Encoder attentions heads
16
VIT Encoder hidden, attention and activation dropout
0.2
VIT Encoder feature projection dropout
0.2
VIT Encoder final layer dropout.
0.2
VIT Encoder causal attention
True
Linear Decoder input size
256
Tokenizer
Character-
Task
Vocab size
99
Epochs
200.0
Training
Effective batch size
640
Learning rate schedule
linear warmup cosine decay
Learning rate warmup ratio CTC zero infinity Gradient clipping
0.05
Optimizer
True
0.1
Torch version Transformers version
2.3. 1tu111
Software
4.36.2
27

---

Published in Transactions on Machine Learning Research (07/2025)
Table 20: Varying hyperpameter for the bascline experiment in Table 18
Architecture
Optimizer weight decay
Optimizer learning rate
Timc-CNN
0.0
0.003
LSTM
0.2
0.0003
ViT (d=128)
0.2
0.001
VIT (d-256)
0.2
0.001
VIT (d=512)
0.2
0.0003
Training
Validation (Cross-session)
Testing (Cross-user)
tectue
50
130
200
100
150
200
50
100
200
Epochs
(a) Loss
Validation (Cross-aession)
Testing (Cross-user)
75
Gononc CER
Large
50-
Small
Try
100
150
200
50
100
150
200
Epachs
(b) Generic CER
Figure 5: Distillation training sample training curves for the TINY, SMALL, LARGE architecture. Validation is done with unseen sessions from the training users while testing is done across unseen sessions from the unseen users.
28

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Training
Validation (Cross-session)
(Cross-user)
1-
0-
Architecture
Large
2-
Z J950
Smal
LoSS
Trny
Init. arigin
Distiliatio
0-
Supervised
2-
E 2099
1-
25
100
75
50
100
Epochs
(a) Loss
Validation (Cross-session)
Testing (Cross-session)
30
User
10-
Architecture
Personalized CER
Large
Small
User 2
20
Tiny
origin
Diviitation in
Supervised
30
User
10
25
50
75
100 0
25
500
75
100
Epochs
(b) Generic CER
Figure 6: Personalization training sample training curves for the TINY, SMALL, LARGE architecture. For the TINY and SMALL architectures we show the separate curves for initializing the model coming from supervisod training and distillation training. We show the training curves for three of the eight personalization users. Both validation and testing done with unseen sessions from the (single) training user.
29

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Encoder Hidden Size: 128
Encoder Hidden Size: 256
Encoder Hidden Size: 512
Encoder Hidden Size: 1024
U.MM
Generic CER
40
1.000
1.50
2224
8.8M
31.0M
5
35-
7,00
0.5H
21.551
33.8M
8 10
2
4
10
10
2
4
6
10
Number of transformer layers
Supervised learning
Distilled
Figure 7: Supervised learning and distilled results the emg2querly dataset across multiple model hidden representation size and number of layers. The total number of parameters of the networks are annotated for each configuration. The performance of the teacher model (LARGE architecture) for the distillation training is highlighted by the horizontal line. The vertical bars denote standard deviation across seeds.
4
Distillation gen. CER Improvement
3+
Encoder hidden size
128
512
1024
1
01
Number transformer layers
Figure 8: Performance improvement from distillation over supervised learning the emg2querty dataset across multiple model hidden representation size and number of layers.
30

---

Published in Transactions on Machine Learning Research (07 (025)
Table 21: Numerical generic CER and their standard deviation for supervised learning analysis. Hyperpa-
rameters provided in Section B.2 bold.
The specifie named architectures from the main text highlighted in
Architecture
Encoder hidden size
Encoder layers
Model params
Generic CER
128
2
631523
42.19 +0.85
128
1028067
38.73 #0.64
128
1424611
37.76 +1.20
128
8
1821155
36.46 #0.92
Tiny
128
10
2217699
35.88 #0.91
256
2
2229091
37.81 #0.60
256
4
3808611
34.89 +0.67
Small
256
6
5388131
35.18 #1.11
256
8
6967651
34.14 #1.37
256
8547171
34.15 #0.42
512
8569955
36.36 #0.44
512
14874723
36.65 +0.65
512
6
21179491
34.89 #0.54
512
8
27484259
33.56 #0.75
512
10
33789027
32.34 #0.62
1024
2
33834595
37.00 +0.36
1024
59027043
35.04 #0.19
1024
84219491
33.10 #0.85
Large
1024
8
109411939
30.47 #0.61
1024
10
134604387
30.89 +0.60
Table 22: Numerical generic CER and their standard deviation for distillation analysis. Encoder hidden
sizes and layers represents the student encoder sizes. The teacher follows the LARGE architecture. Hyper- parameters are provided in Section B.3
Student arch. Student encoder hidden size
Student encoder layers
Model params
Generic CER
128
2
631523
41.56 =0.45
128
4
1028067
37.57 +0.72
128
6
1424611
34.89 10.76
128
8
1821155
33.32 =0.67
Tiny
128
10
2217699
31.93 +0.39
256
2229091
36.31 =0.60
256
4
3808611
33.29 +0.98
Small
256
6
5388131
32.67 +0.48
256
8
6967651
32.89 #1.11
256
10
8547171
32.47 #1.45
512
2
8569955
35.42 +0.48
512
14874723
34.70 +1.32
512
21179491
33.19 #0.91
512
8
27484259
32.24 +1.34
512
10
33789027
31.52 +0.23
1024
2
33834595
36.52 +0.19
1024
4
59027043
33.50 +0.75
1024
6
84219491
32.06 +0.22
Large
1024
8
109411939
30.38 10.46
1024
134604387
30.63 +0.56
31

---

Published in Transactions on Machine Learning Research (07/2025)
Table 23: Numerical generic CER and their standard deviation for the personalization results with varying model initialization. The initialization origin column represents whether the generic model used as initializa- tion for the personalized model has been trained through Supercised learning (i.e., without distillation loss) or through Distillation. The initialization seed column represents distinction between different models with the same architecture, but trained on different. seeds different hyperparameters, selected by choosing the best validation performance among the all the models trained. Hyperparameters provided in Section B.4
Architecture
Init. origin
Init. seed
Model params
Generic CER
Tiny
Distillation
seed A
2217699
8.64 +0.04
Tiny
Distillation
seed B
2217699
8.65 #0.02
Tiny
Distillation
seed C
2217699
8.91 #0.06
Tiny
Supervised seed A
2217699
9.72 #0.13
Tiny
Supervised
seed B
2217699
9.91 #0.13
Tiny
Supervised
seed C
2217699
10.36 =0.02
Small
Distillation
seed A
5388131
7.07 #0.06
Small
Distillation seed B
5388131
7.02 #0.03
Small
Distillation seed C
5388131
7.16 +0.01
Small
Supervised seed A
5388131
7.94 #0.06
Small
Supervised seed B
5388131
9.78 #0.07
Small
Supervised seed C
5388131
9.20 +0.06
Large
Supervised seed A
109411939
6.81 +0.07
Large
Supervised
seed B
109411939
6.48 #0.02
Large
Supervised seed C
109411939
6.54 #0.08
Table 24: Inference speed of highlighted model sizes (per 4 second window).
Architecture
Encoder hidden size
Encoder layers
Params
Inference speed (ms)
TINY TRANSFORMER
1 188
10
2.2M
6.1
SMALL TRANSFORMER
256
5.4M
5.7
LARGE TRANSFORMER
1024
8
109M
27.0
32


Reviews:
[{"review_id": "x6KcQehu0v", "reviewer": "Reviewer_QySn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While the results presented in the paper seem correct, the claims are not fully substantiated by it. There is plenty of work on transformers, compact transformers and even distilled transformers on sEMG, none of which appear in the baselines that the authors compare against [1,2,3,4 to name a few but there are plenty]. The only addition that I see in the proposed work is benchmarking on the emg2qwerty dataset but because the dataset was released in December last year (Neurips), it is understandable that there wont be any methods benchmarking on it. There is no innovation on learning, dataset, evaluation, etc. and while novelty isn't a criteria for TMLR, I dont see why the paper would be of interest to the readers when there are already so many works in the same space. \n\n[1] Wanga, Zefeng, et al. \"Lightweight Transformer for sEMG Gesture Recognition with Feature Distilled Variational Information Bottleneck.\" (2024).\n[2] Zhang, Jiaxuan, et al. \"Feasibility analysis of semg recognition via channel-wise transformer.\" 2022 IEEE 11th Global Conference on Consumer Electronics (GCCE). IEEE, 2022.\n[3] Yang, Jehan, et al. \"EMGBench: Benchmarking Out-of-Distribution Generalization and Adaptation for Electromyography.\" Advances in Neural Information Processing Systems 37 (2024): 50313-50342.\n[4] Zhang, Jiaxuan, et al. \"Movement recognition via channel-activation-wise sEMG attention.\" Methods 218 (2023): 39-47.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a transformer network for training on sEMG dataset. Benchmarking on emg2qwerty results in favorable comparisons against baseline methods. The paper shows how distillation can further reduce the parameter count of the network.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- Comparison against the real state-of-the-art methods\n- Clear positioning in the literature\n- Empirical or theoretical justification for why the proposed method is of interest and be used by practitioners\n- Justifications for the claimed contributions- distillation, validation via held-out set have all been done before and cannot be claimed as new contributions. \n- Any new insights that the model uncovers regarding the data", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lfTAaHsbjH", "reviewer": "Reviewer_gqjk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Results in subject-independent settings are interesting and relevant, since sEMG data differs quite a lot between subjects.\n- Improvement over the baseline (table 2) is substantial.\n- Decent writing, no major technical concerns.\n\nWeaknesses:\n- In my opinion the major weakness is the flawed initial claim that training transformer-based models on sEMG data is new. A quick internet search produces, for example:\n* Montazerin et al, Transformer‑based hand gesture recognition from instantaneous to fused neural decomposition of high‑density EMG signals\n* Scheck et al, Multi-Speaker Speech Synthesis from Electromyographic Signals by Soft Speech Unit Prediction\n\nall using some kind of transformer model and much less data than the present study. Both works are not cited.\n- Also, there is no real technical contribution: the techniques (transformer, distillation) are well-known, just the data is new.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents experiments and results on a) training transformer models on sEMG (Surface Electromyographic) datasets, b) performing model distillation on such data and models. Experiments are performed on the emg2qwerty dataset, with around 346 hours of data of more than 100 users. The non-transfomer baseline is taken from the original emg2qwerty paper (Sivakumar et al); settings for the transformer include CTC loss, the AdamW optimizer, and a grid search on architectural hyperparameters (section 3.2). Results are reported in subject-independent settings (i.e. the set of subjects whose data are used for training/testing are disjoint).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Since this paper provides a substantial improvement over the baseline, I think it could be interesting, but it needs to be rewritten regarding the premise (critical request for acceptance). The mere fact that a transformer works on this kind of data is not really interesting.", "broader_impact_concerns": "Given the major weakness described above, the impact of this paper would be quite limited.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0hr6ofhGRm", "reviewer": "Reviewer_nGrz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper shows that Transformer models are effective for sEMG, and show that better performance can be achieved by scaling up the transformer, even though the data scale remains small. \n- The authors distill the transformer into a smaller size, making it applicable for on-edge devices which is important for such settings,\n- Evaluation is performed on a more realistic setting, which I appreciate. \n\nWeaknesses:\n\n- [W1] One of the contributions is unclear to me. Transformers were successful on small-scale sEMG datasets. The authors mention in the related work: \"Some recent work has begun to explore the application of transformer-based models [cited works] and vision transformers (ViTs) to sEMG data [cited works] but these efforts are also limited to small-scale datasets and simpler tasks\". This means that the contribution is probably investigating transformer models on larger scale datasets (which should perform better since the data scale is larger)? I am confused about this contribution where the authors show that transformer models can be effective for sEMG tasks even\nwhen considering small-scale datasets (compared to standard deep learning datasets). \n- [W2] While the authors show that Transformer models are effective for sEMG data, It is unclear to me *why* do they work. Is it because of the hybrid transformer architecture (the 3 CNN layers the authors use at the beginning), or the alleviation of hand-crafted features (which I assume is the case), or the data augmentation...etc. I would appreciate ablation studies on this. \n- [W3] The authors mention that prior works (Table 1) fail. However, this is not shown quantitatively for the same dataset and experimental setting that the authors consider. There are no comparisons with respect to other works show in Table 1, in the results in Tables 2 and 3. Instead, the authors show results for their experiments only. I understand that replicating results of other works might be difficult, especially if the code is not provided. Therefore, I recommend the authors to perform their own replication of some prior works, as additional baselines, with exactly the same dataset, experimental and evaluation setting. I would like that the authors focus on Transformer-based and hybrid-based models (for example, Rahimian et al., 2021, Godoy et al., 2022, Chen et al., 2023, Zabihi et al., 2023, Zhang et al., 2023, (Liu et al., 2024, Putro et al., 2024) as this is the focus of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the use of small transformer models (2.2M params) on small-scale datasets such as sEMG, which previously failed on sEMG due to the lack of data quantity and diversity, which limited sEMG to old machine learning or deep learning models. They show that with a small amount of data (100 users), transformer models can learn effective representations from the raw sEMG data directly (without the need for hand-crafted feature extraction) and perform well on tasks. They also show, that scaling the transformer model size achieves better results (+20%), a finding consistent with the current literature of transformer models. Based on this finding, they scale the transformer in size, train on sEMG, and then distill a smaller version with 50x less parameters (with only a small drop in performance), which makes it fast and ready to be deployed on edge devices. Finally, they evaluate on more realistic settings from held-out unseen users (zero-shot setting) rather than held-out seen trials from users (where the trial setting is different), and from the (original) noisy, subject-variant sEMG signals. They also promise to release code such that future works can build on it.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would request the authors to address W2 and W3, as they are the most important to me. These are core experiments that should be conducted.", "broader_impact_concerns": "No issues", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)