Fast and Expressive Gesture Recognition using a Combination-Homomorphic Electromyogram Encoder
Authors: Niklas Smedemark-Margulies
Yunus Bicer
Elifnur Sunger
Tales Imbiriba
Eugene Tunik
Deniz Erdogmus
Mathew Yarossi
Robin Walters
Venue: Accepted by TMLR ()

Abstract: We study the task of gesture recognition from electromyography (EMG), with the goal of enabling expressive human-computer interaction at high accuracy, while minimizing the time required for new subjects to provide calibration data.
To fulfill these goals, we define combination gestures consisting of a direction component and a modifier component.
New subjects only demonstrate the single component gestures and we seek to extrapolate from these to all possible single or combination gestures.
We extrapolate to unseen combination gestures by combining the feature vectors of real single gestures to produce synthetic training data.
This strategy allows us to provide a large and flexible gesture vocabulary, while not requiring new subjects to demonstrate combinatorially many example gestures.
We pre-train an encoder and a combination operator using self-supervision, so that we can produce useful synthetic training data for unseen test subjects. 
To evaluate the proposed method, we collect a real-world EMG dataset, and measure the effect of augmented supervision against two baselines: a partially-supervised model trained with only single gesture data from the unseen subject, and a fully-supervised model trained with real single and real combination gesture data from the unseen subject.
We find that the proposed method provides a dramatic improvement over the partially-supervised model, and achieves a useful classification accuracy that in some cases approaches the performance of the fully-supervised model.

Published in Transactions on Machine Learning Research (04/2024)
Fast and Expressive Combination-Homomrrhiic Electromyogram Encoder
Gesture
Recognition using
Niklas Smedemark-Margulies
amddenatt-mammmiieennnttttttttt
Yunus Bicer
bicer. dicer.yy@noortheasteee edu
Elifnur Sunger?
sunger.c@hnorthhssleen. edu
Tales Imbiriba
Limbirrha&hnorrhhssttee edu
Eugene Tunik
e.tunkk@nnorthessterr.com edu
Deniz Erdogmus²
d. derdggmmsShoorthcssseen. edu
Mathew Yarossi2.3
ww.yarossiSnnorrhhastten.. edu
Robin Walters!
edu
Khoury College of Computer Sciences, Northeastern University 2 Department of Electrical and Computer Engineering, Northeastern, University
Department of Physical Therapy, Movement, and Rehabiltatiion Sciences, Northeastern, University Reviewed on OpenReview: https: // oppnreveeee ewe.w. normm? id- j5T4pcLbeY
Abstract
We study the task of gesture recognition from electromyography (EMG), with the goal of enabling expressive human-computer interaction at high accuracy, while minimizing the time required for new subjects to provide calibration data. To fulfiil these goals, we define combination gestures consisting of direction component and modifier component. New subjects only demonstrate the single component gestures and we seek to extrapolate from these to all possible single or combination gestures. We extrapolate to unseen combination gestures by combining the feature vectors of real single gestures to produce synthetic training data. This strategy allows us to provide a large and flexible gesture vocabulary, while not requiring new subjects to demonstrate combinatorially many example gestures. We pre-train an encoder and combination operator using self-supervision, synthetic training data for unseen test subjects. To evaluate the proposed method, we collect and release real-world EMG dataset, and measure the effect of augmented supervision against two basclines: a partialy--uperriseed model trained with only single gesture data from the unseen subject, and a fully-supervised model trained with real single and real combination gesture data from the unseen subject. We find that the proposed method provides dramatic improvement over the partially-supervisedd model, and achieves useful classification accuracy that in some cases approaches the performance of the fully-supervised model.
that we can produce useful
1 Introduction
Subject transfer learning describes the technique of using pretrained models on unsoen test subjects, sometimes with adaptation, and challenging and active topic of research for EMG and related biosignals (Hoshino et al., 2022; Bird et al., 2020; Yu et al., 2021). The difficulty of subject transfer arises from numerous sources.

---

Published in Transactions on Machine Learning Research (04/2024)
Noise in EMG data can stem from the stochastic nature of the EMG signals, variability in EMG sensor placement and skin conductance, and inter-individual differences in anatomy and behaviors such as the selection, timing, and intensity of actions. Labels for supervised EMG tasks may also be noisy due to variation in task adherence (Samadani and Kulic, 2014). As a result of the performance gap between train and test subjects, is lypical in tasks like gesture recognition that unseen lest subjects must provide supervised calibration data for training or fine-tuning models. This produces an undesirable trade-off; increasing the gesture vocabulary increases the expressiveness of the trained system but also increases subject time spent during calibration.
In this work, we propose thrce-part strategy to address the trade-off' between expressiveness and cal- ibration time. First, we define a large gesture vocab- ulary as the product of two smaller, biomechanically independent subsets. Subjects are given a set of direction gestures (Up, Down, Left, and Right) and 4 modifier gestures (Thumb, Pinch, Fist, and Open), and may either perform a single gesture chosen from the 4+4 individual options, or a combination gesture from the x 4 options consisting of one direction and one modifier.
TUP
FØr (Zup)
Encode
Tpinch
Fgg (Tpinch) )
Combine Data
Combine
Features
PUphPinch
Encode
Fer (TipaPinca)
Using a gesture vocabulary with this combinatorial structure makes the system expressive, but tradi- tional machine learning approaches cannot naively generalize to unseen combinations, and would there- fore require correspondingly large training dataset. For example, given only examples of single gestures Up and Pinch, traditional models would not naively be able to classify examples of the unseen combina- tion Up&Pinch gesturos. Therefore, the sccond key component is to define a training scheme using partial supervision with synthetic data augmentation. A new subject only needs to demonstrate the 4+44 single gesture classes, and a classifier can then be trained using these real single gesture examples plus synthetic combination gesture examples created by combining gestures in some feature space.
Figure 1: Overview of dcsired commutativity. A combination--oomonoopiic encoder Fap commutes with gesture combination: given data from real single gostures Tup and Tpinch and real gesture combina- lion Tupkpinch- the real combination gesture's 's encoding Fop (MipaPinch) should equal the combined encodings of the single gestures Combine (Zup), Fe (xrinca)).
In order enact this partially supervised training, we must be able to generate useful synthetic combination gestures. Thus, the third element of our strategy is to use contrastive learning to pre-train an EMG encoder and a combination operator, such that can combine real single gesture examples into realistic synthetic examples of combination gestures. The goal of this contrastive learning is that the encoder should be approximately homomorphic to combination of gestures, as shown in Figure 1. In other words, the property we desire is that combining two gestures in data space (e.g. when a subject perform a simultaneous Up&Pinch gesture) gives approximately the same features as combining two gestures in feature space (e.g. the features of an Up gesture combined with the features of a Pinch gesture). Given this property, the synthetic gesture combinations we produce can serve as useful training data.
We evaluate our method by collecting supervised dataset containing single and combination gestures, and performing computational experiments to understand the effect of various model design choices and hyperparameters. Note that our approach to extrapolating from partial supervision can be applied to any task in which data have multiple independent. labels, but where the combination classes are not easily predicted from the single classes. Classical machine- learning methods for biosignals modeling tasks often use hand-selected features; by learning our feature space entirely from data, we remove this reliance on domain expertise. In general, hand-selected features can be a useful way to enforce strong prior assumptions about a particular domain, and may be useful in the case of limited data, whereas data-driven approach may be beneficial the amount of available data increases in cases when domain assumptions may be violated.
The main contributions of our work are as follows:
2

---

Published in Transactions on Machine Learning Research (04/2024)
We propose two-stage classification approach that allows us to recognize large combinatorial set of gestures from sEMG after only collecting examples of the single component gestures. Our method uses a contrastive pre-training stage, followed by new-subject training stage with partial-supervision and synthotic data.
We collect and release a public dataset of real-world single and combination gestures to assess our proposed method on this novel generalization task!
We conduct thorough post-hoc computational experiments to demonstrate the benefit of using syuthetic supervision during the calibration stage, and to explore the sensitivity of our method to many design choices and hyperparameters'.
2 Related Work
Applications of EMG. The broad task of gesture recognition from EMG has been well studied. Research varies widely in terms of the specific application, and thus the exact sensing modality, the set of gestures used, and the classification techniques applied. Typical applications include augmented reality and human-computer computer interface, as well as prosthetic control and motor rehabilitation rescarch (Wang et al., 2017; Sarasola-San et al., 2017; Pilkar et al., 2014; Bicer et al., 2023). EMG data has also been used extensively for regression tasks beyond prosthetic applications, such as hand pose estimation (Yoshikawa et al., 2007; Quivira et al., 2018; Liu et al., 2021). These applications are well-reviewed elsewhere (Geethanjali, 2016; Yasen and Jusoh, 2019; Jaramillo- Yáncz et al., 2020; Li et al,, 2021).
Feature Extraction. One area of focus has been the effect of feature extraction on classifier performance. Most traditional approaches to EMG gesture recognition transform raw EMG signals into low-dimensional feature vectors before classification (Nazmi et al., 2016; Phinyomark et al., 2012; 2018). Feature extraction methods can be broadly characterized into those using time-domain features, frequency-domain features features based on the method of extraction (Nazmi et al., 2016). Numerous
and time- fainue-frqquecc--oonme investigations have evaluated the best set of features to maximize classifier performance, but the hardware, tasks, and findings vary between different studies, highlighting the potential issues of using hand-engineered features (Nazmi et al., 2016). Our approach differs in that we use contrastive learning to learn a fcature space with the correct structure for our classification task. This approach can be re-used more easily across different experimental setups and tasks.
Recognizing Combination Gestures. Previous work on gesture combinations is relatively limited. Some work has shown high accuracy in classifying movement along multiple axes with application to prosthetic control, though this was performed using fully-supervised data (Young et al., 2012). This application has been extended by using simultaneous linear regression models for controlling prostheses along multiple degrees of freedom, again using fully-supervised training data (Hahne et al., 2018). Other research has provided models for classifying single gestures from mixed vocabulary containing both static and dynamic gestures (Stoica et al., 2012), and models to simultancously classify a gesture and estimate force vectors (Leone et al., 2019); by contrast, we focus only on classifying static gestures. To our knowledge, only one previous article has attempted to classify combinations of discrete gestures from training data that contains only single gestures (Smedennark-Marguuee et al., 2023).
Semi-Supervised, Contrastive, and Few-Shot Learning. We propose a novel approach to simultanc- ously resolve issues with the classification of combination gestures and provide means for subject. transfer learning. We formulate the task of calibrating models for an unseen test subject semi-supervised learning problem, and use a contrastive loss function to ensure that items with matching labels have similar feature vectors.
Briefly, semi-supervised learning describes the setting in which labels are available for only a subset of training examples. Models are trained with a supervised loss function for labeled examples, and an unsupervised loss function on the unlabeled examples. A wide variety of unsupervised loss functions have been studied, such as
Dataset can be downloaded at https:/ /wwp//eeodoooooooooooooo/o wwwenddooorg/do///e http/////11...2//eeeeeee 10291624
2Code to reproduce all experiments can be found https://github. wwtps/..oooooooo//eeeaaeeooe.a wm/c:mm/ccmmmmeee hom-hon-eagg eng
3

---

Published in Transactions on Machine Learning Research (04/2024)
entropy minimization (encouraging the model to make confident predictions) (Grandvalet and Bengio, 2004; Lee et al., 2013), and consistency regularization (where the model must make consistent predictions despite small perturbations on the imput data) (Sohn et al., 2020). These and other techniques are well-reviewed clsowhere (Weng, 2019; Ouali ct al., 2020).
Contrastive learning describes techniques for learning useful representations by mapping similar items to similar feature vectors. This can be applied in tasks with full or partial labels, or even unsupervised contexts. Items may be identified as similar based on their label (Schroff ct al., 2015; Khosla et al., 2020), by applying small perturbations to input data that are known to leave the class label unchanged (Chen et al,, 2020; Grill et al., 2020). This area is also well-reviewed elsewhere (Weng, 2021; Jaiswal et al., 2020).
Our approach uses a combination of ideas from the literature on semi-supervised and cont learning. The key step that reduces subject calibration time is to collect only partial labels, thereby defining a semi- supervised learning problem. In order to allow models to successfully extrapolate from this partial supervision, we pretrain our models using a classic contrastive learning method called the triplet loss (Schroff el al., 2015).
Our problem statement is similar to the problem statement of standard few-shot learning setting (see Wang et al. (2020) for review), but has several additional propertics. First, in few-shot learning, models are evaluated on different "tasks," where the relationship between different tasks can be quite loose (such as image classification research in which the original dataset, and even the number and identity of class labels can vary between tasks). We are interested in the variation of behavior between subjects, so each "task" in our work represents applying a pretrained model on a certain unseen test subject. This leads to a particular relationship between different "tasks"; all the same classes are present when comparing two tasks, but data is obtained from different subject. Second, in typical few-shot learning, the labeled items available are typically randomly sampled from the full set of possible classes. By contrast, consider a problem with two-part labels, and consider that our partial supervision consists of only single gestures (itemss that use only one label component) and does not contain any combination gestures.
Subject Transfer Learning. Note that there are a variety of other approaches to the subject transfer learning problem, such as techniques for regularized pre-training (Smedemark- et al., 2022), domain adaptation techniques based on Riemannian geometry (Rodrigues al., 2018), and weighted model ensembling using unsupervised similarity (Guney and Ozkan, 2023). Techniques for subject transfer in EMG and related data types such electroencephalograppy (EEG) are well-reviewed elsewhere (Wan et al., 2021; Wu et al., 2023).
3 Methods
In Section 3.1, we define the subject transfer learning problem and provide notation including model components. In Section 3.2, we describe how feature vectors from single gestures will be combined to create synthotic features for combination gesturcs. In Scction 3.3, we describe the first stage of our proposed method in which we pretrain an encoder and a combination operator. In Section 3.4, we describe the second stage of our method, in which we collect calibration data from an unseen subject, then use the pretrained encoder and combination operator to generate synthetic combination gesture examples and train personalized classifier for that subject.
3.1 Problem Statement and Notation
In standard classification (ask, we seek to learn a function that predicts the correct label y for a given data example z, based on labeled examples provided during training. Recall that the motivation for our work is to design model that makes use of population information collected from a set of pre-training subjects, and can be quickly calibrated on new subjects. To that end, we modify this problem formulation as follows.
Consider a dataset D = {(x, y, })} consisting of triples of data x e RP, two-part gesture labels y (yavr Bmod), and subject identifiers s E (1.... S}. The components of the gesture label y take values as follows: your is in (Up, Down, Left, Right, NoDir), where NoDir indicates that no direction was performed, and Ymod is in (Thumb, Pinch, Fist, Open, NoMod), where NoMod indicates that modifier was performed.
4

---

Published in Transactions on Machine Learning Research (04/2024)
We partition the subject ID values into disjoint sets Pre and F Using these sets, we divide the overall dataset into three segments:
A pre-training segment DPre, where E SPre, This is used to learn population model.
A calibration segment DCalib, where E SEval, and containing only single gestures (y has the form (i, NoMod) or (NoDir, This represents the small amount of single-gesture calibration data we can obtain for new unseen subject and is used for further model training.
A test. segment Dlest, where E &Eval and containing all classes y- This used to measure final model performance.
We define two-stage architecture, consisting of an encoder For (:: - RK producing K-dimensional feature vectors 2, and classifier Goo (+) R + (55) x A(5) mapping feature vectors to pairs of 5- dimensional probability vectors. This classifier consists of two independent components, one that predicts the distribution over direction gestures, and one that predicts the distribution over modifier gestures. This "parallel" classification strategy adopted as the simplest way of producing a two-part label prediction, though in principle one could soek to capture correlations between the two label components. Note that there will be one classifier Go used during pretraining (which provides an auxiliary loss term), and it will be replaced by a fresh classifier Gl'est trained from scratch for the unseen test subject.
For convenience, we define several subscripts to refer data from single gestures or combination gestures. Let
Zair represent data, labels, and features, respectively, from a set of single gestures with only direction component. Such gesture has a label of the form (i, NoMod). Likewise define Xmnds M Zmod for set of modifier gestures, which have labels of the form (NoDir,j). More generally, let Esing represent data from a single gesture (either direction-only gesture, or a modifier-only). Finally, define Xcomb: Veomd, Zeomb for a set of combination gestures with labels of the form (i where + NoDir,j j + NoMod, and let Feomb represent data from one such combination gesture.
3.2 Homomorphisms and Combining Feature Vectors
Given two algebraic structures G and H, and a binary operator o defined on both sets, a homomorphism is a map f : G H satisfying
fgr o 92) = f(1)) f92), Vgi, 92 G.
(1)
This property can also be described by noting that the map f commutes with the operator o (Bronshtein and Semendyayov, 2013). As mentioned previously, we design a partially-superviedd calibration in which a new subject will only provide single gesture examples and we will seek to extrapolate to the unseen combination gestures. To facilitate this extrapolation, we seek to learn an encoder and combination operator to achieve this commutativity, as shown in Figure 1. Note that there are two types of combination to consider; a physiological process that allows subjects to plan and perform simultancous combination of gestures, and explicit parametric function that we will use to combine the feature vectors of two single gest ures. The first type of combination (which occurs in a subject's brain and musculature) is quite complex, and for the purposes of this work is inaccessible.
Define a combination operator (Ecomss Dcoms) Coe ((Eatrs Mair), (Emods Vmod)) that takes as input the features and label of real dircction gesture, and the features and label of a real modifier gesture, and produces as output synthetic feature vector and label. The output feature vector represents an estimate of what the corresponding simultaneous gesture combination should look like. Note that all feature vectors (from single gestures, real combination gestures, or synthetic combination gestures) have the same dimension K. Furthermore, note that the input label Udir has the form (i, NoMod), and Ymod has the form (NoDir, the output label simply uses the active component from the two inputs: Geoms = (i.j)-
Figure describes the desired relationship between the features of real and synthetic combination gestures. If we achieve the desired homomorphism property, a synthetic feature vector created by this combination operator should be good approximation for the features from corresponding real, simultaneous gesture.
5

---

Published in Transactions on Machine Learning Research (04/2024)
LTRIPLET
T Fcomb
Encoder FoF
Reamb
Combination
Zoom
Classifier Gerr
Lee
Esing
Zaing
Coc
Live
(a)
Triplet Loss
CTRPLET
T Tcomb
Encoder Fop
Reomb
Combination
eoomb
Classifier GBe
Lce
Coc
Ziing
Zaing
Lee
(b) Fake Cross- Entropy
CrmmLrr
T Foomb
Encoder
Zeomb
Combination
Zeomb
Classifier
For
Cae
GPre
Lixe
Fzing
Zaing
Lee
(c) Real Cross-Entrope
Figure 2: Fully-supervised pretraining for cumbination-homomrrpa model. Encoder parameters OF and combination opcrator parameters @g are optimized using sum of three loss terms. The classifier 0g is only used for auxiliary losses, and its parameters are discarded after pretraining. Colored boxes are models; white boxes are variables; orange boxes are loss terms; inactive components are faded out. (a): The contrastive loss term Lrrreer compares features of real and synthetic gesture combinations. Single gestures Zsing are encoded features ssing and combined into synthetic features &comb. Real combination gestures Icomb are encoded to features Zcomb- (b): Classifier GPre is applied to synthetic features Ecomb to compute fer- (c): Classifier Gife is applied to real features zsing and ccomb to compute Les-
For example, given features from single gestures 21 with label yı (Up, NoMod) and another single gesture 22 with label 1/2 (NoDir, Pinch,, the combination operator outputs synthetic feature vector &comb, (Up, Pinch) Ca ((21, 21), (22, 12)). In the successful case, this synthetic feature vector Ecomb should resemble feature vector from real Up&Pinch gesture.
Note that the combination operator C corresponds to the operator mentioned above for a general homomor- phism. We consider two functional forms for C. one case, we consider learning feature space in which combination can be performed trivially. Here, C' has zero learnable parameters (10cl = 0) and computes the average of the two input feature vectors Zcomb = (Zair + 2mod)/2, which we refer to as Cang, This approach requires the encoder to learn feature space in which combination occurs homogeneously for all pairs of input classes, In the other case, the combination operator C produces synthetic feature output using class-conditioned multi-layer perceptron (MLP) Æcomb = MLP( i, &mod; j). This allows the possibility that flexible combination operator may still be required, and that the process of combining gestures may depend on the particular input classes.
For convenience, define another function (2oomb Dcoms) CombineAllPairs ((Zirr Vair ), (2mod: Vmod)) that accepts two sets of feature vectors with labels, and applies Co. all pairs of direction and modifier gesture, producing set of synthetic feature vectors with labels.

---

Published in Transactions on Machine Learning Research (04/2024)
3.3 Pretraining the Encoder and the Combination Operator
Figure 2 describes our pretraining procedure. The goal of pretraining is to learn an encoder and feature combination operator that allows us to extrapolate from the features of two single gestures to the estimated features of their combination, as shown in Figure 1. We use contrastive learning to approximately achieve this homomorphism property. Let -( represent & notion of distance in feature space. Consider the feature vector of a real combination gesture, such as Zeomh = For (Zupkpanca), and the feature vector of a synthetic combination gesture &comb = Coc (Feg (Zup); For (Tpinet)). We seek parameters that minimize the distance between these matching items: arg ming, d(2comb; &comb). However, note that this does not yet provide a well-posed optimization lask, since a degenerale encoder may minimize this objective by mapping all input. feature vectors to single fixed point. Thus, we must simultaneously ensure that the distance between non-matching pairs of items remains large; for example, the feature vector of real combination gesture Reomb Fop (Zuphpinch) and the feature vector of a synthetic combination &comb = Cac (Fer (Tup), Fer (Frthmb))
We use a triplet loss function to achieve this property. Given a batch of labeled data Xair U Kmod U Noomb and Vair U Xmodd U &oomb, we use the encoder Føp to obtain features Zdir U Zmod U Zeomb. We can then create feature vectors and labels for synthetic combination gestures (Zcomb.ecmm) = CombineAllPaire ((Zdir, Vdir), ( (modd) Finally, we can use triplet loss to compare the distance between matching and non- non-matching combination gestures.
Consider a real combination gesture Ecomb E Zoomb and its label Veomb = (Vi, By); this item is the "anchor." A "positive" item is a synthetic gesture & whose label completely matches Toomb = (yi, Uj)- A "negative" item is a synthotic gesture Zcoms whose label differs on at Icast one component: Feomb = (vh. v/y) with either V # yo: or V/ # y/ or both. We also consider synthetic anchor items Æcomb E Zeombi then, positive items are real combination gestures with matching label, and negative items are real combination gestures with a non-matching label.
Given an anchor item a, positive item p. and negative item n selected as described above, and for some choice of margin paramcter y, we compute a triplct loss using
LTRILET = max (d(a. p) d(a, n) + 7).
Intuitively, we want the negative item to be at least y units farther away from the anchor than the positive item.
We consider several slight variations on the standard triplet loss. In the basic version, for each possible anchor item in a batch (i.e. cach real and cach synthetic combination gesture), we select N random pairs of positive and negative item, without replacement. In the hard version, for each possible anchor item, we form a single triplet using the farthest positive item and the closest negative item. This approach is often referred to as "hard-cxamplc mining" (Schroff et al., 2015), since these are the items whose encodings must be movod the most to achieve zero loss. In the centroids version, for each possible anchor item, we form a single triplet by comparing to the centroid of the positive class and the centroid of a randomly chosen negative class. At iteration 4, given data X, from one class and a momentum parameter M E [0, 1], the centroid c(t) is computed using an exponential moving average as cx²² MC +(1 M[]]X.].
Note that for a fixed-sized training dataset and a sufficiently flexible encoder, there may still exist degenerate solutions in which real combinations and synthetic combinations are mapped to the same locations, but these locations are not separated in a way that permits learning a classifier with smooth decision boundarics, or does not generalize well to new subjects. Therefore, along with the encoder, we simultaneously train a small classifier network G.pee Using this classifier, we compute two additional loss terms; a cross-entropy on real feature vectors Len, and a cross enty on synthetic feature combinations Len- As mentioned above, this classifier network has two independent and identical components; one is used to classify the direction part of the label, while the other classifies the modifier part. After the pretraining stage is finishod, we discard the classifier; as described in the next section, we will train a fresh classifier GTest from scratch for the unseen subject.
7

---

Published in Transactions on Machine Learning Research (04/2024)
Tranub
Encoder FøF
Combination Coc
Zcomb
Classifier
Grest
Cee
Esing
Eeing
Figure 3: Partially-Superviiedd Calibration for Unseen Subjects. We exploit combinatioon-oooooorpphhe to train a fresh classifier calibration on unseen test subjects using only single gesture data Iiing (neither real combination gestures Teomb nor their features 2comb are available). Data Tsing are encoded to features Eeing and combined into synthetic features Æcomb These real and synthetic gestures are pooled, and a freshly initialized classifier GTest is trained on these items using Ler.
Our overall training objective C is a mixture of the triplet loss and these two cross-entropy terms:
L= Crurer + hoe Leu-
3.4 Training Classifiers with Augmented Supervision
In Figure 3, we describe our proposed method of training fresh classifier on unseen test subject using synthetic data augmentation. For this new test subject, we collect real labeled single-gesture data (Rairy Vair) and (Amod; Mmod). We encode these items to obtain real feature vectors Zair and 2mod- We then oblain syuthetic features and labels (Zcomb, Yeomb) = CombineAl1Pairs ((Zar; (Zmod. Vmod)). Finally, we merge the real single gesture features and synthetic combination gesture features Zdir U Zmod U Zeomb. along with their labels Vair U Umad U.oom into a single dataset. to train a randomly ialized classifier Gifeet for this subject. As in the pretraining stage, this classifier consists of two separate, independent copies of the same model architecture order to classify the direction and modificr parts of the label.
4 Experimental Design
In this section and Scction 5, we focus measuring the effect of using synthetic combination gestures during the calibration stage of our method (Figure 3). In the Appendix, we include other experiments the effect of hyperparameters and model design choices (Section A.2) and the effect of various model ablations (Section A.3).
4.1 Evaluation and Baselines
For cach choice of model hyperparameters, we repeat model protraining and evaluation 50 times; this includes 10-fold cross-valdattion and 5 random seeds. During the calibration and test phase, the encoder is frozen, as shown in Figure 3.
As described in Scction 3.1, we consider three segments of our dataset; Drre, DCalib and Dest Here, we explain how the full dataset is divided into these segments. In each cross-validation fold, the pretraining dataset Pre contains data from 9 subjects; of these 9 is used for early stopping based on validation performance, Data from a 10th subject is divided in a stratified 80/20 split to form the calibration and test datasets Catib and Test, Specifically, this 80/20 split is stratified as follows. We use the encoder to obtain real feature vectors from all of the test subject's data Zai U 2mod U Zcomb: and divide each of these portions of data to obtain calibration datasct and a test dataset. The calibration datasct contains features and labels from direction gestures (ZCalib, ) CCCllb) from modifier gestures (2Caléb, yCClle), and from combination gestures (z comb. D)Cafib). ). The test dataset also contains features and labels from direction gestures (Z yTTest), ) from modifier gestures (2Teat, yTTeat) ) and from combination gestures (2Test. yTest)
Given this split of data, we can perform our augmented training scheme as described in Sec 3.4. Augmentation is performed using CombineAllPairs with all real single features Z ZCalib mod and their labels yCCatib UypColib In summary, this model is trained using real single gesture items (2 ( yCalib) and (2Calib, "'Cali)), as well as synthetic combination gesture items (2 DCCath)) We refer to this model as the "augmented supervision" model. Note that this model does not real combination gestures (2Cali). yCCld))
mod
8

---

Published in Transactions on Machine Learning Research (04/2024)
To measure the effect of adding the synthetic gestures created using the combination operator, compare this augmented supervision model against two baselines with different calibration data setups:
"partial supervision" model, which the calibration set only includes single-gesture items Calib, )dr D(rhbb and (Zoodh )
2. A "full supervision" model, in which the calibration set includes single-gesture items (Z Dorrithh and (oodd, )oodd)) ) as well as real combination items (Z Veomit))
To ensure that our experiments can clearly show the effect of adding synthetic gesture combinations to the model calibration data, the test data is the same for all three models.
Note that these baselines are designed to measure the effect of synthetic data model calibration. The partially-supervised baseline shows what performance we can expect if the subject demonstrates only single gestures and we naively train a classifier. It is conceivable that extrapolation from single gestures to combinations could be trivial, especially since our modols contain one classifier for the direction component and one for the modifier component. In that case, the partially-superve combination gestures, despite not having examples of them its calibration set.
baseline would perform well on
The full supervision model shows how well the classifier would have performed if we required the subject to exhaustively demonstrate all possible gesture classes (despite the burden this places the subject). The full supervision model also demonstrates the cumulative effect of other sources of error in our experiments, such as label noise in our dataset and differences in the signal characteristics between subjects that may limit unseen subject generalization.
Note that all three models (augmented supervision, partial supervision, and full supervision) use the encoder For at test time to obtain features from the unseen test subject. The baseline models do not directly use the combination operator during this calibration stage; however, the combination operator's presence during pretraining still affects the encoder's parameters, and thus indirectly affects these baselines as well.
4.2 Metrics: Balanced Accuracy and Feature-Space Similarity
Model performance mcasured using two metrics. First, we measure the performance of the final test classifier using balanced accuracy, which is the arithmetic mean of accuracy on each class. Balanced accuracy is used to ensure that overall performance is not affected by any issues of class imbalance. We compute balanced accuracy on the subset of 8 single gesture classes (Accsingte), the subset of 16 combination gesture classes (Acccomb). and the full set of 24 gesture classes (Accau).
Second, we characterize the feature-space similarity between real and synthetic gesture combinations. Recall that we pretrain the encoder F'g and feature combination Cag with a contrastive loss function as shown in Figure 2 in order to achieve the commutativity property shown in Figure To evaluate the quality of the learned feature space, we encode all data from the unseen test subject and use similarity metric to check whether matching classes end up being more similar to each other than non-matching classes, We use similarity measure based on the radial basis function (RBF) kernel.
Given a pair of feature vectors ø1, 22 and a lengthscale 6, the RBF kernel (also called Gaussian kernel) similarity is defined as
K KRBF 22) = -xx(-881a1- 22213).
(4)
The lengthscale hyperparameter d strongly affects the ability to detect structure using the RBF similarity metric. Using an adaptive length scale such as the median heuristic (Garreau et al., 2017) can help reveal structure within each run, but prevents the comparison of similarity values across runs (analogous to comparing pixel brightness values between photographs taken with different contrast levels). Therefore, we use fixed value of 1/128 for all experiments. This corresponds to the expected squared L2 distance between vectors drawn from standard Gaussian in 64 dimensions (the latent dimension of our models), which may be reasonable highly-simplified approximation for the typical distribution of feature vectors.
9

---

Published in Transactions on Machine Learning Research (04/2024)
To describe the similarity between two sets of items, such as items from two classes, compute the average RBF kernel similarity for all pairs of points. Given sets of items 21, 22, we define the set similarity metric SetSim(Z1, 22)
SetSim(21, = 21 Z2
Σ 21 P2
Σ
(21
(5)
This definition allows compare two different sets of items when 21 * Z2. To compute similarity of items in the same set when 2 22, we use the additional constraint that the sampled items are distinct 21 # 22- Note that the RBF kernel similarity takes values between O (for items very far apart in fcature space) and (for items with identical feature vectors); the set similarity SetSim takes values in the same range.
If the encoder and combination operator provide synthetic features that are a close approximation for real features, the similarity between two matching classes, such as real ZppaPinch and synthctic ZopePinch should be high. Likewise, if the triplet loss successfully separates non-matching classes, then the similarity between non non-matching classes, such as real ZupaPinch and synthetic ZopaTiunb; should be low.
To describe the structure of classes in feature space, we use SetSim to compute the entries of a symmetric 32 x 32 matrix Msim; whose j.j entry is the similarity between classes SetSim( 2, 2,). The first 16 classes are the real combination gestures, while the next 16 classes are the synthetic combination gestures. We also summarize several key regions of Msim The average of the first 16 diagonal elements represents the similarity between matching real items, i.e. the compactness of these real classes. Likewise, the average of the next 16 diagonal elements represents the compactness of the synthetic classes. The average of the 16th sub-diagonal represents the similarity between real and synthetic items that were considered matching in the castrastive learning objective (such as real Up&Pinch and synthetic Up&Pinch). Finally, the average of all other below-diagonal elements represent the similarity between non-matching items.
4.3 Hyperparameters and Experiment Details
Encoder Architecture. The encoder model For implemented in PyTorch (Paszke et al., 2019). The model architecture is a 1D convolutional network with residual connections and has 105K trainable parameters. The encoder is pre-trained for 300 epochs of gradient descent using the AdamW optimizer (Loshchilov and Hutter, 2017) with default values of B1 = 0.9 and $2 = 0.999 and a fixed learning rate of 0.0003. The encoder's output feature dimension K is set to 64.
Combination Operator. As mentioned in Section 3.2, we consider two functional forms for the feature combination operator Cec. In one form, two input feature vectors are simply averaged, and there are no parameters in 9c In the other, C is an MLP with 17K trainable parameters.
Classifier Architecture and Algorithms. We consider two options for the classifier model Gf that is used during the pretraining stage. In the "small" case, G has a single lincar layer to classify direction and a single linear layer to classifier modifier, totaling 650 trainable parameters. In the "large" case, each of these components has multiple layers, totaling 34K trainable parameters.
When training fresh classifier Gl'est for the unseen test subject, we use Random Forest implemented in Scikit-Learn (Pedregosa et al., 2011), since this is very simple parametric model that also trains very fast. For comparison, we also show the performance of several other simple classification strategies for subset of encoder hyperparameter settings: k-nearest neighbors (kNN), decision trees (DT), linear discriminant analysis (LDA), and logistic regression (LogR).
When performing augmented training, we first create all possible synthetic items using CombineAllPaire, and then we select a random N 500 items for each of the 16 combination gesture classes. The same subset procedure is used when computing feature-space similarities.
Hyperparameters.
In all experiments, we set the triplet margin parameter ? = 1.0. For our main experiments, we include all three loss terms. For ablation experiments, described below, we consider all subsets of loss terms.
10

---

Published in Transactions on Machine Learning Research (04/2024)
We consider three variations of a triplet loss as described in Section 3.3. When using the "basic" triplet loss strategy, we sample N = 3 random triplets without replacement for each item. When using the "centroids" triplet loss strategy, we use a momentum value of M = 0.9 to update the centroids.
Ablation Experiments and Varying Hyperparameters. The results of our experiments with varying model hyperparameters are shown in Appendix A.2. The results of our experiments with ablated models are shown in Appendix A.3.
4.4 Gesture Combinations Dataset
EMG data during gesture formation were collected from 10 subjects (6 female, 4 male, mean age 22.6 + 3.5 years). All protocols were conducted in conformance with the Declaration of Helsinki and were approved by the Institutional Review Board of Northeastern University (IRB number 15-10-22). All subjects provided informed written consent before participating.
Briefly, subjects were seated comfortably front of a computer screen with arms supported on arm rest, and the right forearm resting in an arm trough. Surface electromyography (sEMG, Trigno, Delsys Inc., 99.99% Ag electrodes, 1926 Hz sampling frequency, common mode rejection ratio: 80 dB, built-in 20 450 Hz bandpass filter) was recorded from 8 electrodes attached to the right forearm with adhesive tape. The eight electrodes were positioned with equidistant spacing around the circumference of the forearm at four-finger-idttl distance (using the subject's 's left hand) from the ulnar styloid. The first electrode was placed mid-line on the dorsal aspect of the forearm mid-line ween the ulnar and radial styloid.
Custom software for data acquisition was developed using the LabGraph (Feng et al., 2021) Python package. A custom user interface instructed participants in how to perform each of 4 direction gestures and 4 modifier gestures. Subjects were then prompled with timing cues on a computer screen, instructing them to perform multiple repetitions for each of the 4+4 single gestures, and multiple repetitions of the 4 x combination gestures (cach combination gesture consists of one direction and one modifier simultancously). From cach gest ure trial, non-overlapping windows of data were extracted to form supervised examples in the dataset; one window contains 500ms of raw sensor data at a sampling frequency of 1926Hz. After windowing trials, each subject provided a total of 584 single gesture examples (73 examples of each class) and 640 combination gesture examples (40 examples of each class). See Appendix A.1 for additional information on the dataset and collection procedure.
To help prevent overfitting, we add noise to the data during training, In each training batch, consider each class of data present, and add freshly sampled white Gaussian noise such that the signal-to-noise (SNR) ratio of the data is roughly 20 decibels. Given the raw data items from a single class X with dimension D, we produce noisy data X with SNR of decibels as follows:
x = X + TBE, =
σx (108/20)) ox = E - E(8)) (0,(0)
(6)
No other filtering or pre- pre-processing steps are performed. Pre-processing may provide additional benefit to the absolute performance of gesture recognition model, though this is orthogonal to the focus of the proposed work.
5 Results
In our experiments, we focus on examining the effect of adding augmented supervision during the calibration stage (See Figure 3). As described previously, we compare a model that receives augmented supervision, with a baseline model that receives only partial supervision (i.e. single gesture examples only) and another bascline that receives full supervision (i.e. real single and real combination gestures).
5.1 Balanced Accuracy and Confusion Matrices
Balanced Accuracy. In Table 1, we compare the balanced accuracy of the proposed augmented training scheme to the partially-supervieel and fully-supervised baseline methods. This table shows results for single
11

---

Published in Transactions on Machine Learning Research (04/2024)
Table I: Balanced accuracy, for the 8 single gesture classes (AcCsingle); the 16 combination gesture classes (AcCromb), and all 24 classes (Accau). Entries show mean and standard deviation across 50 trials (10 data splits and 5 random seeds). The proposed augmented training scheme greatly increases overall model accuracy by trading a modest reduction in performance single gestures for large increase in performance on combination gestures. See Section 4.1 for baseline model details.
Model
ACCsingle
ACCcomb
Accatl
Partial Supervision
0.90 + 0.05
0.01 + 0.01
0.31 +: 0.02
Full Supervision
0.80 + 0.07
0.58 + 0.09
0.65 + 0.07
Augmented Supervision
0.77 + 0.11
0.32 E 0.08
0.47 + 0.06
choice of hyperparameters that gave the highest overall accuracy and a good balance between performance on single gestures and combination gestures. For data shown here, the auxiliary classifier during pretraining was the "small" single-layer architecture; the combination operator was an MLP, and the "basic" triplet loss was used; results from all hyperparameters are shown in the A Appendix in Table 3. The encoder Fop and combination operator Ce. were pretrained as described in Section 3.3, and then a fresh classifier G was trained using different forms of supervision on the unseen test subject. In the augmented supervision case (bold row), the classifier was trained as in Section 3.4; the partially-supervised and fully-supervieed baselines were trained as in Section 4.1.
Comparing the performance of the partially-superviseed bascline to the augmented supervision model, we observe that adding synthetic combination gestures leads to a large improvement of about +16% overall classification performance on the unscon test subject. The overall gain is achieved by trading a small decrease in performance on the single gesture classes (-13%) for relatively larger increase in performance on the combination gesture classes (+31%). The resulting model achieves performance much better than random chance on all classes, whereas the partial supervision model achieves essentially zero percent accuracy on its unseen combination gesture classes. Recall that all three models (augmented supervision, partial supervision, and full supervision) use the same parallel approach lo classification; for a given gesture, one model classifies the direction component, and another independent model classifies the modifier component. If extrapolating to gesture combinations was trivial, then the partial supervision model would have enough information to predict combination gestures, since it has already seen all the individual gesture components. The fact that. this partial supervision is not enough demonstrates that gestures compose in a highly non-trivial manner. The fact that gestures compose in a non-trivial manner is not surprising from a biological perspective (Scott and Kalaska, 1997; Liu et al., 2014); for example, the set of muscle activations required to form Pinch gesture are different in a neutral wrist angle, than while simultaneously performing an Up or Down gesture.
Trade-off of Single and Combination Performance The trade-off in performance between single and combination gestures is noteworthy, and is fundamental element of the problem setting and our approach. As mentioned in Section 3.1, all models consist of two independent classifier copies, with one responsible for predicting direction and one for predicting modifier. Consider the direction classifier as an example.
In the partially-supervised case, the direction classifier is trained to predict one of 5 labels, and its training data came from 5 relatively compact classes: data that should be classified as Up came from gestures whose full label was (Up, NoMod); data that should be classified as Down came from (Down, NoMod) gestures, etc. As result, this classifier learns specialized decision boundaries that perform well for this particular subset of the data distribution (i.e. single gesture data).
In the fully-supervised or synthetic supervision case, the direction classifier is still trained to predict one of 5 labels, but its training data come from 5 heterogeneous classes: training data for the Up label came from the classes (Up, (Up, Thumb), .... NoMod), and likewise for other direction labels. In this case, the classifier is forcod to learn decision boundaries for a broader and more heterogencous data distribution. The decision boundaries that capture this greater variability are worse when focusing on only the single gesture portion of the data distribution, but better when considering the full data distribution.
12

---

Published in Transactions on Machine Learning Research (04/2024)
To explore this phenomenon, use the performance of a small classifier as empirical measurement of separation between different groups of items, since groups of data that are well-separated can be classified with high accuracy. For each subject, we consider three scenarios, and train a Random Forest classifier using an 80 20 train:test split. First, we compare single and combination data by splitting cach subject's data into two classes: those with labels (D, NoMod) and NoDir, M, and those with labels (D, M). Next, we compare the variation of direction data by splitting cach subject's data into four classes: (Up, M), (Down, M), (Left, M), and (Right, M). Lastly, we compare the variation of modifier data by splitting each subject's data into four classes: (D, Pinch), (D, Thumb), (D, Fist), and (D, Open). Table 2 shows the results of this experiment, where balanced accuracy is computed within each subject, and then averaged across subjects. We see that it is roughly as easy to distinguish single data from combination data (73.2%), as it is to distinguish between the four active direction gestures (74.7%), or to distinguish between the four active modifier gestures (66.7%). This leads to the phenomenon described above.
Table 2: Within-subject performance of Random Forest classifier on raw sEMG data for various classification tasks. Mean and standard deviation of balanced test accuracies averaged across 10 subjects and 5 random data splits. The similar classification accuracy indicates that separation of single vs combination data is roughly similar to the scparation of direction or modifier classes themselves. Due to this separation, the Partial Supervision model, which is trained with only single-gesture data, becomes overly-specialived for that subset of the data distribution, and performs better on that subset of items but worse overall (See Table 1).
Task
Balanced Test Acc (mean std)
Single vs Comb. Which Direction
0.732 + 0.060
0.747 = 0.069
Which Modifier
0.667 = 0.090
Confusion Matrices. Figure 4 presents the performance of the same three models in the form of confusion matrices. Here, the striking gain of performance on the combination gesture classes can be seen more clearly, as well several noteworthy patterns of errors. The dotted lines show the boundary between the single gesture classes, the combination gesture classes, and a (NoDir, NoMod) class. Items the bottom-left region of a plot are combination gestures that were incorrectly classified as single gestures.
Figure 4(a) shows the performance of the partial supervision model. This model performs well on the 8 single gesture classes the diagonal; however, almost none of the combination gesture classes are correctly classified. Instead, one of the two components is often classified as NoDir/NoMod /NMoo resulting in an entry in the lower-left region of the matrix, or both components are classified as NoDir/ prediction in the right-most column. As an example, this shows that seeing single gesture training data for (Down, NoMod) does not prepare the model to understand the direction component of a (Down, Pinch) gesture. This martiall-supeervieed model often correctly identifies the dircetion component, but prodicts that the modifier component is unlike its training data and belongs in the NoMod class.
resulting in a
Figure 4(b) shows the performance of the full supervision model, which achieves strong performance all classes. Even in this fully supervised case, the strongest pattern of crrors appears when the model sees a real combination gesture, and successfully identifies the direction component, but fails to identify the modifier component.
Finally, Figure 4(c) shows the performance of our proposed model trained with augmented supervision. There is a striking gain of performance on the combination gesture classes, with individual classes gaining between 9% and 55% balanced accuracy. This improvement is visible as the appearance of strong diagonal line in the lower right region on the confusion matrix.
5.2 Latent Feature- Space Similarity
The core idea behind our approach is that we can extrapolate to unseen gesture combinations by using a rastive objective to ensure that synthetic combination gestures are placed appropriately in feature space
13

---

Published in Transactions on Machine Learning Research (04/2024)
10
50
05
c:1 80
3 21
500
Cark Hone
20
90
90
08
30
911 11 00
PRck
020
10
Cane Thunkre 00
30
20
00
535
06
50
Prime Deeratit 05 on 30
08
30
80
30
53 0r 30 08
Phom Cood 0.0 of 83 of
50
83
00
ao
(og tt o2t
50
20
Thung 5o8 0z 0o 200 655
Rix Twoe LEC 642636 1000
30
(1 PuC Ca1 50 03
for Openi (.34
100
30
000000000
No.Cort 545 00
00
(ronny waree 23 2.0 oz1 liz 5A7 wwe.stae
30
63 531
30
20 0. 10g017
pour Heor 2o 21 ao 200 532 SEE
20
Cowns Shn 00 Au Ca 00 642
00
30 30
00
052
00
30
80
08
80
30 08 odgi0n
Soow For 03 030 00 83 03 OEL 03500
10
800
08
000
08 0.2
30
08
08 0.00 63300
sperie
bog www.ccannccaoooo
631
50
3 Poin o5 633 6-5 5.30 021
Det Thantada Do 00 060 11
30
00
00
83
30
03
C8 085
10
30
ao
on
pare Thuntre ao oce aou oe one 200 a.
08
30
30
00
80
06
10
08
8o o0 ao inee
sagen thang bo oo oe 111 0 00 10 105500
0.04
E01 08 000 800 00
04
10
EM4
oo
your to
(hapk Open 1
n0 5og 05 Rattto
08
50 30 63
no
oe s0 83 of
Caee big do oo
8:30 2o
plght Pholi
rin tnd
don mesedl
thatl
Gree 2nd cnt seeit giati cnl
(a) Partial Supervision Model
(b) Full Supervision Model
Pregiced
000 DO0
312
000
pown Home
001
30
Pight noreross Aot
90
305
30
80
63
00
10
10
02
30 63
00 900 300 c8s 00
Okhee Tweete oo no
sre
30
5/0
Done, Reg 800 204
30
30
(lew, 03
315
E.l Paace 53 aec
504
050 03 oE angen
EUB Ripozo 5.10 05 066 300 01
300 031 022
5:0 00
5.0 510 5:00 sages
(bevs, Prete omt 312 0.0 WR.CESEEE
100
Dover Toorboo 300
30
530
Doome Oper 30 A 05 80 08 50
30
53
300
031 02
10
50 03 or nuger
340 rnorteoes co 300
03 305
000
his
Bint comnoot co
00
80
09 o0 1
(lix Pacrouze co 6a1 201 01 041
(herk The to aog or ooe 000
30
30
806
83
08
50 008
0z
30 83
00 033 tr
Higs
prom
nel
(c) Augmented Supervision Model
Predcied
Figure 4: Confusion matrices corresponding to Table 1. The entry at row i column shows the fraction of items known to belong to class i that were predicted to belong to class j. (a) Model calibration performed using partial supervision (only single gesture examples). (b) Model calibration performed using full supervision (real single and combination gesture examples). (c) Model calibration performed using proposed method of augmented supervision. Dotied lines show the boundary between the single gest classes, the 16 combination gesture classes, and the class for outliers. Whereas the partial supervision model achieves nearly zero performance on combination gestures, the augmented supervision model shows strong performance improvement for combination gestures.
during training. In Section 5.1, we examined how using these synthetic combination gestures as training data during the calibration stage affects the performance of classifier. Here, we directly examine whether the
14

---

Published in Transactions on Machine Learning Research (04/2024)
Mial)) Roal 4.x
(Up, Thumb) Roal 13
(Up. Fed) Rani LD eu
(Mp, Open) Rewl 1
(Doom, Pinch)
(Down, Thumb) Real
(Down. Fle) Real
-000 546
(Down, Oper) Roal--
(Lon, (inor) Roal
(Le0) Thumb)
Quil: Fis)) Riadd
1:00
300
(Lert Open)- Reel
1:04 806
0.0
082 300 304
031
[Right, Pinch) Real-
65. 5:8
0a 200
(Riht, Thums)
(Right Real
(righs, open) Moall
0:0 sae
300
(Lap, Phrdd Fale
1:20
300
(Up, Thumb) Fake
505
(Up.
(Up, Open) Fale
0.11
300
(Down, Pinch) Fake
0:08 6ox
213
(Jown, hhmbb) Hake
318
(Down, Fis)
(Down, Openi-
(L400, Pincc) Fake-
0.0
506
(Le0,, Thurbb) Fake
8.00
808
0.0
634
036
300
300
3.1
000
[Lett Fast Fake
1:05
0.0
(Lot, Open)
(Right, Fo
(Right, Thumb) Fake-
0.1 c. 508
(tight, Fae) Fake
1.0 RM4 busl 6a0 0.9 2os aoe
prighs, Open)- Fale
(Up, Perdt) Perel
(ip. Thammit- Feed
Fien FPad
(Doun Pircky Real
cpesy Real
(Rown, Thuran- Feal (Down, Paaa
(Doon, Chart Riaal
(Loe, Pardly Rieal
(Lat, Trannlyy Fiees
migh Picolyy Fieal (Raghs Tturnlyy Riaal
Peno
(righe, Five) Reaal
(Righs, Opemt Faae) (up. (Ul Truml)) Fale
(Up, Fest Faaro
Piace
(Up cjan) Flaer
(Down, Peoli) Fakke
(Down, Thunt)) Fate
(Dom.. Fiu) Frace
(Down Open Fikke
Pertty Fiake
Ratt Foet Fiae
Rell Gront) Fiato
Fiae
(riigta Pinot) Fiake
(figie. Thamm)) Plake
progen Oyeern Frae
Fis) pake
(Lets,
Figure 5: Featuro-space similarity between real and synthetic ("fake") combination gestures. We show the lower-triangular portion of the symmetric matrix Msam as described in Section 4.2. The entry at row column j shows average RBF kernel similarity between items of class and class j (with values ranging in (0, 1]. Dotted lines separate the real and synthetic classes. Similarity matrix was averaged across 50 independent runs. The main diagonal and 16th subdiagonal are clearly visible, indicating that items with matching label are more similar than items with non-matching label.
contrastive learning procedure results in synthetic items being similar to matching real items and dissimilar from non-matching items.
Figure 5 shows the similarity matrix Msim defined in Section 4.2. Using the same model whose results are shown in Table 1 and Figure 4, we extract the features of all items from the unseen test subject, and construct 500 synthetic combination gestures per class. We compare each pair of classes using the set similarity metric SetSim, and fill in the entries of the similarity matrix Msim as described in Section 4.2. Finally, this matrix is averaged across 50 independent re-runs of the training procedure (comprising 10 data splits and random seeds).
In the upper-lest triangle, we see the similarity between pairs of real gesture classes. The strongest entries are on the diagonal, indicating that items are more similar to other items of the same class than to items from non-matching classes. The entries in the 1st 2nd, and 3rd sub-diagonal show that items with the same direction component but different modifier component are also sometines similar.
In the nower-right triangle, we see the similarity between pairs of synthetic gesture classes. The diagonal clements are again the strongest, showing that classes are relatively compact and well-soparated. The range of values in the lower-right region (both diagonal and off-diagonal) is higher than values in the upper-left region; this may indicate that the synthetic combination items are more compact overall (inereasing all
15

---

Published in Transactions on Machine Learning Research (04/2024)
similarity values). Since the combination operator here was MLP, it is not necessary for synthetic items to be closer to each other than real items; however, it is possible that this structure arose during pretraining. Tn particular, it is possible that the combination operator learns to place synthetic gestures close to the mean of the corresponding class, since this would lead to a better triplct loss on average.
In the bottom-left square region, we see the comparison between real and synthetic items. The diagonal is clearly visible, indicating that items with exactly matching label are most similar to each other, as desired. There is also a noteworthy block-diagonal pattern, indicating of similarity between items whose direction label matches, but whose modifier label does not match. For items whose label is totally non-matching, we see very low similarity values.
6 Discussion
6.1 Summary and Key Takeaways
We study the problem of gesture recognition from EMG, with the goal of maximizing expressiveness while minimizing the calibration time required for unseen test subject. Typical approaches to modeling biosignals require new subjects to perform burdensome calibration sessions and exhaustively demonstrate all regions of input space, duc to the large differences in signal characteristics between individuals. This exhaustive calibration scales with the nunber of classes and results in a large up-front time cost.
We construct a large gesture vocabulary as the product of two smaller sets; subjects may perform a single gesture or combination gesture. We propose two-stage method for performing classification while keeping calibration short. During fully-supervised contrastive pre-training, we train an encoder and combination function to produce useful synthetic gesture combinations from real single gesture examples. Then for an unseen test subject, collect only single gesture examples and extrapolate to their synthetic combinations. This proposed method simultancously keeps calibration brief, since subjects only demonstrate a linearly-sized set of single classes, while also keeping model expressiveness high, since subjects can use a quadratically-sizes set of combination classes.
To evaluate the proposed method, we collect a real-worll dataset of gesture combinations. Subjects follow visual prompts to perform repetitions of 4 direction and 4 modifier gestures, well the full set of 16 combinations; disjoint 500ms windows of 8-channel sEMG are recorded during gesture production. Our main experimenls are designed to measure the effect of the proposed synthetic supervision at calibration time. We therefore compare the proposed method against two baselines: a partial supervision case in which the unseen subject's model is trained using real single gestures with no augmentation, and a full supervision case in which the unseen subject's model is provided real supervision for both single and combination gesture classes. The partial supervision model demonstrates the performance decrease we would naïvely suffer by collecting only single gesture examples from the unseen subject. The full supervision model demonstrates the maximal performance that could achieve on particular unseen test subject using a particular encoder if did not atlempt to reduce time spent collecting calibration data, considering factors such as classifier algorithm and various sources of noise in the dataset. We measure model performance using balanced accuracy on single gestures, combination gestures, and all gestures together. To evaluate the structure of the learned feat space, we also use a set-similarity metric based on an RBF kernel to measure the similarity between matching and non-matching items. We repeat all experiments across multiple random seeds and data splits.
In our main experiments, presented in Section 5.1 and 5.2, we find that the proposed synthetic supervision greatly improves classification of unseen gesture combinations (+31%) at a cost of performance on single gesture classes (-13%), with an overall improvement across all classes (+16%). We explore this trade-off between single and combination performance further. We also find a clear pattern of feature-space similarity indicating that the pre-trained encoder and combination function are able to produce synthetic data that are well-clustered by class, well-separated, and highly similar to the corresponding real classes.
In supplementary experiments, presented in Appendices A.2 and A.3, we measure the effect of: different versions of contrastive triplet loss, auxiliary loss terms used in pre-training, the architecture of an auxiliary
16

---

Published in Transactions on Machine Learning Research (04/2024)
classifier used during pre-training, the classifier algorithm used during calibration, the form of the learned composition function, and the amount of noise added for regularization during training.
6.2 Future Directions
There are several sources of crror that limit the genoralization performance of our method in the current study. As mentioned, an unseen test subject may dilfer from training subjects due to measurement noise, differences in signal characteristics, or differences in motor behavior. The dataset also contains label noise, since labels are applied based on a visual task instruction, but subjects may perform the wrong gest or with inaccurate timing. Furthermore, the proposed contrastive learning approach forces the model to adjust the structure of its feature space; it is conceivable that this could impede the ability to train a classifier on the unseen subject (or that the effect of the auxiliary cross-entropy loss terms during pretraining was not sufficiently strong). There are other types of modeling changes that could help increase absolute performance for the full supervision, such as feature engineering and neural architecture search; these techniques are orthogonal to the proposed strategy for augmented supervision. Future experiments using ground-truth labels of hand position may further improve the absolute performance of our method by reducing label noise.
We briefly discuss the model accuracies achieved in our experiments from the perspective of real-world user-facing application. Note that the final balanced accuracy of our proposed model is 77% for single gestures and 32% for combination gestures (47% over all classes). For a user-facing application where very high accuracy is required, a number of standard techniques can be applied on top of such a model to achieve higher final accuracies. For example, Xu et al. (2022) develop a model for gesture recognition using incrtial mcasurement unit signals measured at the wrist. They develop a model whose accuracy on 4 active gesture classes ranges from 62% to 74% when classifying individual data windows ("window-level," Fig 3 (a) of Xu et al. (2022)). The same model achieves accuracy between 93% and 97% when aggregating predictions over 3 to 4 consecutive windows ("gesturre-evell"" " Fig 3 (b) of Xu et al. (2022)). Other techniques include aggregating predictions across an ensemble of independent model copies (e.g. see Du et al. (2022) for discussion of the mixture-of-experts technique), or incorporating contextual information in the form of a Bayesian prior, though this requires application-speecfic modeling (e.g. see Speier et al. (2016) and Orhan et al. (2012) for applications to assistive typing via eletroencehhalogappy))
One of the key design considerations of our method is the structure of the combination operator. In our main experiments, we considered operator may be an avenue for future innovation, such as summary statistics that describe the feature-space structure of the test subject's gestures.
condicoodd MLP. Providing additional information to the combination
For the purpose of clcarly measuring the effect of the proposed method, we used a simple downstream classification rategy. The relative changes in accuracy that we observe in our experiments clearly demonstrate the benefit of the proposed method, Further performance gains could be achieved by using more sophisticated downstream classification methods and other transfer learning techniques such as model ensembling regularized pre-training. The time required for demonstrating examples could also be further reduced by using active or continual learning techniques to reduce the number of examples provided for each gesture class. We study an extreme point on the spectrum of supervision, with zero real combination gesture examples during calibration; our method could be supplemented with a few real combination examples (e.g. a linearly-sized subset of combinations) to obtain a different trade-off between performance and calibration time.
While generalizing unseen combinations is a topic of broad interest in machine learning (e.g. Nikolaus et al. (2019); Madan et al. (2020); Wiedemer et al. (2024)), we emphasize an important aspect of our application that differs from the standard setting. In typical multi-label settings where we may study combinatorial gencralization, data examples often have a structured multi-part label. Depending on the representation of input data, it is possible for one label to be unobserved and thus absent from our datasel, but that label still exists in the underlying physical model. For example, consider recognizing objects with multiple attributes such as shape and color from a text description. For some inputs, we may be missing information about shape, but that object's shape still exists. In our setting, a single gesture has a partial label, not because of missing information, but because the other label component is truly inactive.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Future experiments may also consider expanded dataset from a similar setting. One of the simplest ways to expand the current work would be to increase the number of items in each of our single gesture sets. In our experiments, the gain for combination gesture accuracy (+31%) was much larger than the loss for single gestures (-13%), and thus we observed an overall benefit (+16%). If this approach gave similar increases and decreases on a dalaset with more single gestures, the net improvement in performance across all classes would grow, since a larger fraction of the total labels would be combinations (for N direction and M modifier NM fraction of all classes).
gestures, combination gestures constitute N+M+NM
Our proposed method may be modified for other applications. One area is tasks with multiple rare labels, such as classifying biomedical signals where rare discase states can co-occur. For example, consider electrocardiogram (ECG) measurement and multiple abnormal cardiac states that may be visible in ECG (e.g. see Houssein et al. (2017) for review). We may want to classify ECG signals as containing either one or multiple disease signatures, but there may be insufficient real combination training data due to the rarity of each single discase state. This application may require adjusting our approach, since we used a fully-supervised pre-training stage with real combination data.
Another area of possible application is tasks with many-part labels where exploring all combinations is not feasible, such as robotic learning and world modeling (e.g. see Muratore et al. (2022) for review). In such applications, the space of possible combinations is too large to feasibly collect real training data from all combinations. Thus, our proposed method of using syuthetic supervision from unseen combinations may be useful to supplement real training data.
Broader Impact Statement
Gesture recognition from EMG may increase the ability to interact expressively with computer systems, but the failure modes of these models may not be well-definod. Thus, gesture recognition systems should be used with care in sensitive production environments.
Some research has been done in the area of biometrics and de-anonymizing EMG and related biosignals. Extracting subject's identity from anonymized recordings of biosignals is only possible if a reference dataset is available from that individual. If reference signal for certain person is available, and if they use a gesture recognition system with an expectation of anonymity, then the possibility that they may be identified from their anonymized data may lead to a violation of privacy.
Acknowledgments
Funding for this research provided by Meta. Reality Labs Research.
References
Takayuki Hoshino, Suguru Kanoga, Masashi Tsubaki, and Atsushi Aoyama. Comparing subject-to-subiee transfer learning methods in surface clectromyogram-base motion recognition with shallow and deep classifiers. Neurocomputing, 489:599-613 612,, 2022.
Jordan J Bird, Jhonatan Kobylarz, Diego R Faria, Anikó Ekárt, and Eduardo P Ribeiro. Cross-domain MLP and CNN transfer learning for biological signal processing: EEG and EMG. IEEE Access, 8:54789 -4808,, 2020.
Zhipeng Yu, Jianghai Zhao, Yucheng Wang, Linglong He, and Shaonan Wang. Surface EMG-based instan- taneous hand gesture recognition using convolutional neural network with the transfer learning method. Sensors, 21(7):2540, 2021.
Ali-Akbar Samadani and Dana Kulic. Hand gesture recognition based on surface electromyograply. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages IEEE, 2014.
Nianfeng Wang, Kunyi Lao, and Xianmin Zhang. Design and myoelectric control of an anthropomorphic prosthetic hand. Journal of Bionic Engineering, 14(1):47-55) (7,49,9999 2017.
18

---

Published in Transactions on Machine Learning Research (04/2024)
Andrea Sarasola-Sanz, Nerca Irastorza-Landa, Eduardo López-Larraz, Carlos Bibián, Florian Helmhold, Doris Broetz, Niels Birbaumer, and Ander Ramos- Margos-urguaallaay A hybrid brain-machine interface based on EEG and EMG activity for the motor rehabilitation of stroke patients. In 2017 International conference on rehabilitation robotics (ICORR), pages 895-900. IEEE, 2017.
Rakesh Pilkar, Mathew Yarossi, and Karen Nolan. EMG of the libialis anterior demonstrates a training effect after utilization of a foot drop stimulator. NeuroRehhailitation, 35(2):299 305, 2014.
Yunus Bicer, Niklas Smnedemark-Marguliees, Basak Celik, Elifnur Sunger, Ryan Orendorff, Stephanie Naufel, Tales Imbiriba, Deniz Erdogmus, Eugene Tunik, and Mathew Yarossi. User Training with Error Augmenta- tion for Electromyogram baased Gesture Classification. arXiv preprint arziv:2309.0723 2023.
Masairo Yoshikawa, Masahiko Mikawa, and Kazuyo Tanaka. Hand pose estimation using EMG signals. 2007 29th Annual International Conference of the TEEE Engineering Medicine and Biology Society, pages 4830 4833. IEEE, 2007.
Fernando Quivira, Toshiaki Koiko-Akino, Ye Wang, and Deniz Erdogmus. Translating sEMG signals to coutinuous hand poses using recurrent neural networks. In 2018 IEEE EMBS International Conference Biomedical & Health Informatics (BHI), pages 166-169. IEEE, 2018.
Yilin Liu, Shijia Zhang, and Mahanth Gowda. NeuroPose: 3D hand pose tracking using EMG wearables. Proceedings of the Web Conference 2021, pages 1471 1482, 2021.
Purushothaman Geethanjali. Myoelectric control of prosthetic hands: state-of-the-art review. Medical Devices: Evidence and Research, pages 247-255, 2016.
Mais Yasen and Shaidah Jusoh. A systematic review on hand gesture recognition techniques, challenges and applications. PeerJ Computer Science, 5xe218, 2019.
Andrés Jaramillo- Yánez, Marco E Benalcázar, and Elisa Mala-naldooddo. Real-time hand gesture recognition using surface electromyograply and machine learning: systematic literature review. Sensors, 20(9):2467, 2020.
Wei Li, Ping Shi, and Hongliu Yu. Gesture recognition using surface electromyography and deep learning for prostheses hand: state-of-the-art, challenges, and future. Frontiers in neuroscience, 15:6218855 885, 2021.
Nurhazimah Nazmi, Mohd Azizi Abdul Rahman, Shin-Ichirol Yamamoto, Siti Anom Ahmad, Hairi Zamzuri, and Saiful Amri Mazlan. review of classification techniques of EMG signals during isotonic and isomctric contractions. Sensors, 16(8) 2016.
Angkoon Phinyomark, Pornchai Phukpattaranoni and Chusak Limsakul. Feature reduction and selection for EMG signal classification. Expert systems with applications, 3943):7400-7431, 2012.
Angkoon Phinyomark, Rami N. Khushaba, and Erik Scheme. Feature extraction and selection for myoclectric control based on wearable EMG sensors. Sensors, 18(5):1615, 2018.
Aaron J Young, Lauren H Smith, Elliott J Rouse, and Levi J Hargrove. Classification of simultancous movements using surface FMG pattern recognition. JFFE Transactions on Biomedionl Engineering, 60(5): 1250 1258, 2012.
Janne M Hahne, Meike A Schweisfurth, Mario Koppe, and Dario Farina. Simultaneous control of multiple functions of bionic hand prostheses: Performance and robustness in end users. Science Robolács, 3(19): eaat3630, 2018.
Adrian Stoica, Chris Assad, Michael Wolf, Ki Sung You, Marco Pavone, Terry Huntsberger, and Yumi Twashita. Using arm and hand gestures to command robots during stealth operations. In Multisensor, multisource information fusion: architectures, algorithms, and applications 2012, volume 8407, pages 138 146. SPIE, 2012.
19

---

Published in Transactions on Machine Learning Research (04/2024)
Francesca Leone, Cosimo Gentile, Anna Lisa Ciancio, Emanuele Gruppioni, Angelo Davalli, Rinaldo Sacchetti, Eugenio Guglielmelli, and Loredana Zollo. Simultaneous sEMG classification of hand/wrist gestures and forces. Frontiers in newrorobotics, 13:42, 2019.
Niklas Smnedemark-Mrrgulies,, Yunus Bicer, Elifnur Sunger, Stephanie Naufel, Tales Imbiriba, Eugene Tunik, Deniz Erdogmus, and Mathew Yarossi. A Multi-label Classification Approach to Increase Expressivity of EMG-based Gesture Recognition. arXiv preprint arziv:2309. 12217, 2023.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Dong-Hyun Lee et al. Pseudo-label:
The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML. Atlanta, 2013.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances neural information processing systems, 33:596-608, 2020.
Lilian Weng, Superviee Representation Learning. lilianweng.githie github.io, io, 2019. URL https:
github. io.. htt.hubb.oo/oott/201111---- 11----1------------ -tt---selsssssssisedd.eeoo
Yassine Ouali, Céline Hudelot, and Myriam Tami. An overview of deep semi-supervised learning, arXiv preprint Xrv::0006 05278, 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 815-880 115-823,40 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Aduances in neural information processing systems, 33:18661 18673, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597 1607. PMLR, 2020.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemoud, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 03:21271-21284 2020.
Bann-Battien
Lilian Weng. Contrastive Representation Learning, lilianwenggitthubii May 2021. URL https://llllii
github.io/poseeee httpaa/...//osst//221----00 05- 31ccontrastive/
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive supfrvupe learning. Technologies, 9(1::2 2020.
Yaqing Wang, Quamming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34 2020.
Niklas Smedemark- Margulles, Ye Wang, Toshiaki Koike-Akino, and Deniz Erdogmus. AutoTransfer: Subject transfer learning with censored representations on biosignals data. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 3159-3165. IEEE, 2022.
Pedro Luiz Coelho Rodrigues, Christian Jutten, and Marco Congedo. Riemannian procrustes analysis: Transfer learning for brain-computer interfaces. IEEE Transactions Biomedical Engineering, 66(8): 2390-244 24012 2018.
Osman Berke Guney and Huseyin Ozkan. Transfer learning of an ensemble of DNNs for SSVEP BCI spellers without user-specific training. Journal of Neural Engineering, 20(1):016013, 2023.
20

---

Published in Transactions on Machine Learning Research (04/2024)
Zitong Wan, Rui Yang, Mengjie Huang, Nianyin Zeng, and Xiaohui Liu. A review transfer learning in E.GG signal analysis. Neurocompuling, 421:1-14, 2021.
Di Wu, Jie Yang, and Mohamad Sawan. Transfer Learning on Electromyography (EMG) Tasks: Approaches and Beyond. IEEE Transactions on Newral Systems and Rehabilitation Engineering, 2023.
Mia Nikolaevich Bronshtein and Konstantin A Semendyayev. Handbook of malhemalics. Springer Science Business Media, 2013.
Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median hcuristic. arXiu preprint arxiv: 707.07 707.07269, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in newral information processing syslems, 32, 2019.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint ar arXv::1111.0110. 2017.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Reseorch, 12:2825 2830, 2011.
Jimmy Feng, Pradeep Damodara, George Gensure, Ryan Catoen, and Allen Yin. LabGraph. https: 2021.
//github. http://///ccccoooeeeee//////ggge
Stephen H Scott and John F Kalaska. Reaching movements with similar hand paths but different arm orientations. I. Activity of individual cells in motor cortex. Journal of newrophysiolog,, 77(2):826 852, 1997.
Jianwei Liu, Dingguo Zhang, Xinjun Sheng, and Xiangyang Zhu. Quantification and solutions of arm movements effect on sEMG pattern recognition. Biomedical Signal Processing and Control, 13:189-199 197, 2014.
Xuhai Xu, Jun Gong, Carolina Brum, Lilian Liang, Bongsoo Suh, Shivam Kumar Gupta, Yash Agarwal, Laurence Lindscy, Runchang Kang, Bchrooz Shahsavari, et al. Enabling hand gesture customization on wrist-worn devices. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-19, 2022.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547 5569. PMLR, 2022.
William Speier, C Arnold, and Nader Pouratian. Integrating language models into classifiers for bci communication: review. Journal of neural engineering, 1333)0030002 2016.
Umut Orhan, Deniz Erdogmus, Brian Roark, Barry Oken, Shalini Purwar, KE Hild, Andrew Fowler, and Melanie Fried-Oken. Improved accuracy using recursive bayesian estimation based language model fusion in erp-based bci typing systems. In 2012 annual international conference of the ieee engineering in medicine and biology society, pages 2497-2500. IEFE, 2012.
Mitja Nikolaus, Mostafa Abdou, Matthew Lamm, Rahul Aralikatte, and Desmond Elliott. Compositional generalization in image captioning. arXiv preprint arXiv:1909.0444 2019.
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. On the capability of neural networks to generalize to unseen calegory-pose combinations. Technical report, Center for Brains, Minds and Machines (CBMM), 2020.
Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36, 2024.
21

---

Published in Transactions on Machine Learning Research (04/2024)
Essam H Houssein, Moataz Kilany, and Aboul Ella Hassanien. Eeg signals classification: review. International Journal of Intelligent Engineering Informatics, 5(4):376-38) 2017.
Fabio Muratore, Fabio Ramos, Greg Turk, Wenhao Yu, Michael Gienger, and Jan Peters. Robot learning from randomized sinulations: A review. Prontiers in Robolics and AI, page 31, 2022.
A Appendix
In Section A.1, give additional details on the paradigm used when collecting dataset, In Section A.2, we conduct hyperparameter search experiments in which we vary model architecture, try learned and a fixed feature combination operator, and vary the contrastive loss function. These experiments show that the proposed method is robust to these choices. The primary effect of varying these hyperparameters is to change the trade-off between performance on single and combination gestures; however the proposed augmented supervision method yields large overall benefit in all cases. Section A.3, conduct ablation experiments to check the importance of each term in our pre-training objective, well the effect of adding noise to the raw input data during pre-training, We find that all three terms in our loss function are necessary to achieve maximal performance, and that the chosen level of inpul noise yields the best performance.
A.1 Dataset Details
Our supervised dataset of single and combination gestures was obtained by recording surface EMG while subjects performed gestures according to visual cues. EMG was recorded from electrodes (Trigno, Delsys Inc., 99.99 99.99% Ag electrodes, 1926 Hz sampling frequency, common mode rejection ratio: 80 dB, built-in 20 450 Hz bandpass filter), spaced uniformly around the mid forearm as shown in Figure 6.
e DA
Figure 6: Surface EMG Recording setup. Electrodes were placed on the of the subject starting from mid-line on the dorsal aspect and continuing towards the thumb.
Subjects followed visual prompts to perform each gesture, as shown in Figure 7. The experimental paradigm consisted of 6 blocks.
In the first block, subjects were shown the UI in Figure 7a and performed single gesture trials. Each single gesture trial consisted of 3s of preparation time (indicated by yellow screen border), followed by 2s of active time holding the gesture (green screen border), and finally 3s of rest time (red screen border). From each single gesture trial, we extracted single 500ms window of data centered within the active period.
In the next five blocks, subjects were shown the UI in Figure 7b and performed combination gesture trials. Fach combination gesture trial consisted of of preparation time (yellow screen border), followed by 8s of active time (green screen border), and 2s of rest time (red screen border). The structure of each combination gesture trial was defined by two horizontal line segments. One segment (the "held" gesture) spanned the full 8s of active time, while the other (the "pulsed" gesture) was present for intervals (alternating between is interval, and 0.5s interval). The vertical gray cursor gradually moved from left-to-right during the trial; subjects were instructed to follow the cursor and perform gestures as the cursor intersected with the horizontal line segments. In some trials, one the horizontal lines was omitted; in this case, single gesture was held for 8s, or a single gesture was pulsed times.
From each combination gesture trial, we extracted 500ms windows of data as follows. Consider the arrangement of the line segments shown in Figure 7b; there are 9 intervals of interest during the active period (5
22

---

Published in Transactions on Machine Learning Research (04/2024)
Extension
Extension
Extension
(a) Example Single Gesture Trial
(b) Example Combination Gesture Trial
Figure 7: User interface (UT) shown to subjects during supervised data collection. (a): UI during a single gesture trial. The screen border was yellow for 3s of preparation, then green for 2s of active time, then red for 3s of rest time. (b): UI during combination gesture trial. The gray vertical cursor scrolled from left right; when it intersected horizontal line segment, the subject performed that gesture. The screen border color also indicated when the subject should be active (yellow for 2s of preparation, then green for 8s of activity during the trial, then red for 2 of rest).
intervals when only the held gesture is active, and 4 intervals when both gestures are active). We extracted a window of data centered each interval, with labels decided as follows:
In trials that contained a held gesture and pulsed gesture (the majority of trials), we extracted 4 combination gesture windows.
In trials that contained only held gesture, extracted 9 single gesture windows.
In trials that contained only a pulsed gesture, we extracted single gesture windows.
Single gesture data from all blocks was pooled, giving a total of 584 single gesture example per subject (73 examples for each of the single gesture classes). Likewise, combination gesture from all blocks was pooled, giving a total of 640 combination gesture examples per subject (40 examples for each of the 16 combination gesture classes). When constructing data splits during model training and evaluation, we used stratified split described in Section 4.1.
23

---

Published in Transactions on Machine Learning Research (04/2024)
A.2 Varying Hyperparameters
A.2.1 Effect on balanced accuracy
Table 3: Effect of varying model hyperparameters on balanced accuracy, for the 8 single gesture classes (AcCsingle), the 16 combination gesture classes (AcCeomb), and all 24 classes (Accau). Augmented supervision: classifier for unscen subject trained according to Section 3.4. Partial Supervision and Full Supervision; baseline models described in Section 4.1. Gpre lype; structure of auxiliary classifier used during pretraining. Coc type; structure of combination operator. Triplet type; loss function for pretraining. See Section 4.3 for details on hyperparameter values. All models used random forest algorithm for unseen test subject. Entries show the mean and standard deviation across 50 independent trials. Bold row represents model shown in Table and Figure 4.
Hyperparameters
Auguieuted Supcrvision
Partial Supcrvision
Supcrvision
Other type
Cgp Sype
Triplet lype
Antuinh
Abtuna
Aetut
Alouingh
Ant
Aetai
Artzing
Artuut
Ancal
large
69g
basic
0.69
0.15
0.30
0.05
0.43 1 0.06
0,89 1 0.06
0.01 1 0.01
0,30
0.02
0.79 0,08
0.57 1 0.10
0.65 L 0.00
conlonils
0.05
0.89 000
1.08
0.00 0.08
0.60 008
Weye
Taril
0n8
0.31 0.07
0.34 0.06
0.80
000
0000
0.30- 0.72
0.77 0:07
0:50 + 0.10
038 000
largo
20lp
bastc
0.5
0.66
0.22 0.06
0.45 1 0.05
0,00
1 0.05
0.01 1 0.01
0.31 0.02
0.79 0.07
0.58 1 0.10
0.68 L. 0.06
contzoids
0.06
0.04
0,90
0.55
0.07
0.06
0.64 0.07
Serge
mlp
land
0.4
021 0.06
0.42 0.01
0.38 0004
0011 0.00
0.30-
0.77 0.00
0:54 008
0,22 000
small
bastc
067 1 0.14
0.38 0.07
0,87 1 6.05
0,00
1 604
0.01 1: 0.01
0.31
0.02
0.80 1 0.06
0.50 1: 0.00
0.00 = 0006
ccutzoids
0.05
0.45 0.05
0,90 0.05
0.55 0.10
0.60 0.08
wnulll
lind
0007
0.45 + 0:07
0.39 0000
0.30-
0:07
0:07 0.0
0.00 0007
emmil
mlp
hanic
0.77 T 0.11
0.32 1 0.08
0.47 1 0.06
0.30
1: 0.05
0.01
2: 0.01
0.31
0.02
0.80 0.07
0.58 1: 0.09
0.85
0.07
ceutzooids
0.33 0.06
0.45 0.05
0,30
1 0.05
0.06
0.54 0.09
0.68 000
anuil
alp
lard
0.8
0.22 0.07
0.81 a 0.05
0330 0006
0.0 0.01
0002
0.79 0.08
0.08 = 000
0.05 0007
In the previous sections, we showed results using a particular model setup. However, as described in Section 4.3, we also experimented with varying several hyperparameters in our model. Table 3 shows the effect of varying model hyperparameters on the performance of the proposed augmented training scheme as well as the partially- and fully-supervised baseline models. The bold row indicates the model hyperparameters used in Table and Figure 4.
The performance of the baseline models is relatively constant across these hyperparameters. Although the baseline models do not make use of the combination operator directly when during the calibration stage with the unseen test subject, the hyperparameters included here influence how the encoder For is trained, and therefore may have an effect on the baselines. The partially-supervised model's constant performance indicates that, despite changes in the pretraining procedure, extrapolating from single gestures to combination gestures remains challenging. The sully-sue model's constant performance indicates that the feature space continues to be amenable to training fresh classifier for the unseen subject.
Examining the performance of the model with augmented supervision, there is a clear trade-off between performance on single gestures and performance on combination gestures. The choice of architecture for the auxiliary classifier Gere and the choice of feature combination operator Ca affect this tradc-off strongly, while the loss function type has relatively less effect on performance. In all cases, however, the overall accuracy of the augmented model is much greater than the partial supervision model.
A.2.2 Effect on Feature- Spacee Similarity
In Table 4, we show how changing model hyperparameters afTects the structure of the learned feature space, measured using the similarity matrix Msim. As described in Section 4.2, perform pretraining, then extract fcatures of real and synthetic gesture combinations for an unseen test subject, analyze their structure by constructing Msim-
In order to show key information from this large matrix for many scenarios, we summarize Msi with four key values of interest:
within-class similarity for real combination gestures (SetSim( Zcomb, Zoomb)),
within-class similarity for synthetic combination gestures (SetSim( Bcomb))
24

---

Published in Transactions on Machine Learning Research (04/2024)
Table 4: Effect of varying model hyperparameters on feature-space similarity of combination gestures. See Section 4.2 for description of similarity metrics. Ger type; structure of auxiliary classifier used during pretraining, Csc type; structure of combination operator. Triplet type; loss function for pretraining. SetSim( Zcomb;
GetSim(Zcomb: Zromb); average similarity between real items of same class (first 16 diagonal elements of Msim). SetSim SetSim(Zcomb: Zcomb: Zoomb); average similarity between synthetic items of same class (last 16 diagonal elements of Ms SetSim( Zcomb: Zcomb); average similarity between real and synthetic items of matching label (16th sub-diagonal elements of Msmm). SetSim(Z (oomb,, comb); average similarity between items of non-matching label (all other below-diagonal elements of Msim: note this includes real and synthetic items). Entries show the mean and standard deviation across 50 independent trials. Bold row represents model shown in Table 1 and Figure
G type
Cso type
Triplet type SetSin(2camm Zeamb)
SetSim( camb. Zeoma)
SetSim Zeomb, Zooms)
SetSin Z 2 Z(am))
large
avg
basic
0.88 0.03
0.95 + 0.01
0.86 + 0.02
0.71 + 0.02
large
avg
centroids
0.12 0.17
0.20 0.23
0.06 + 0.14
0.02 + 0.06
large
evg
bard
0.92 0.06
0.96 0.03
0.91 + 0.06
0.82 + 0.13
large
mlp
basic
0.71 0.08
0.82 0.05
0.62 + 0.08
0.38 +: 0.09
large
mlp
centroids
0.10 0.12
0.17 0.16
0.03 + 0.10
0.01 +: 0.05
large
mlp
hard
0.83 0.07
0.89 0.04
0.77 + 0.07
0.59 + 0.13
small II
avg
basic
0.36 0.14
0.56 + 0.13
0.27 +: 0.12
0.10 + 0.07
small
BVg
centroids
0.19 0.15
0.35 0.21
0.13 + 0.14
0.04 + 0.06
small
svg
bard
0.64 0.13
0.79 0.10
0.58 +: 0.15
0.34 +: 0.15
small
mlp basic
0,33 0.14
0.51 0.14
0,22 + 0.12
0.07 + 0.07
small mlp
ccutroids
0.17 0.15
0.31 0.19
0.10 + 0.11
0.03 + 0.05
small mlp hard
0.60 0.15
0.74 0.10
0.49 + 0.16
0.28 +: 0.18
similarity between matching real and synthetic combinations (SetSim( Zeomb, Zoomb)), and
similarity between combinations with non-matching labels (SetSim 2lmmb))
The row in bold represents the model hyperparameters used in the main experiments (Section 5).
Note that these feature similarity values are difficult to interpret and compare. The key information comes from comparing the magnitude of similarities for "matching" items against the magnitude of similarities for "non-matching" items; model whose feature space brings matching items closer together than non-matching items may allow a downstream classifier to more easily learn decision boundaries. A well-structured feature space should have relatively larger values in the first three values (similarity of matching real items, matching synthetic items, and pairs of matching real/ henthttt items), and a relatively smaller value the final column (similarity between non-matching items). However, the absolute magnitude of similarity values on their own is not meaningful, since this only reflects the relative scale of distances in feature space to the kernel lengthscale 8 used in Equation 4. For example, keeping ó fixed while isotropically shrinking all feature vectors (multiplying all feature vectors by matrix of the form al, << 1) would drive all similarity values very close to 1.
Generally, we see that the similarity of non- non-matching items lower than for matching items; for most models the non-matching similarity values are many times smaller than the matching similarity values. This is unsurprising, given that we were able to train a downstream classifier successfully for all models. It also gives some circumstantial evidence for the hypothesis that the contrastive learning objective used in pretraining is effective at structuring the feature space, and that this structure correlates with downstream classifier performance.
A.2.3 Choice of Classifier Algorithm
In Table 5, we show how the balanced accuracy of the unseen subject's final classifier model is affected by the choice classifier algorithm. In this comparison, all other model hyperparameters were kept constant. We report the mcan and standard deviation of balancod accuracy on single gestures, combination gestures, or all gestures, for the model with augmented supervision, the partial supervision baseline, and the full supervision baseline. The first algorithms shown in the table learn non-linear decision boundary, while
25

---

Published in Transactions on Machine Learning Research (04/2024)
Table 5: Effect of classifier algorithm on balanced accuracy, for the 8 single gesture classes (AcCsingte). ) the 16 combination gesture classes (Accromb); and all 24 classes (Accau). Augmented supervision; classifier for unseen subject trained according to Section 3.4. Partial Supervision and Full Supervision: baseline models described in Section 4.1. Geeat Alg; classifier algorithm used for unseen test subject's model. RF; random forest. KNN; k-Nearest Neighbors. LDA; Linear Discriminant Analysis. DT; Decision Tree. LogR; Logistic Regression. All classifiers implemented in Scikit-Learn using default hyperparameters. All models used "small" auxiliary classifier during pretraining, MLP combination operator, and "basic" triplet loss. Entries show the mean and standard deviation across 50 independent trials. Bold row represents model shown in Table 1 and Figure 4.
Augmented Supervision Accons
Partial Supervisod
Full Supervision
Go Alg
Antsoogle
Accuz
Actsingle
Antcoud
Aocua
Acceingle
Actromb
Accun
0.32
0.08
0.47 + 0.06
0.90
0.05
0.31
0.02
0.80
0.07
0.65
0.07
KNN
0.74
0.12
0,33 1 0.08
0.47 1 0.06
0.91 1 0.05
0.00
0.00
0.30 1 0.02
0.76 4 0.07
0.37 1 0.00
0.64 0.08
DT
0.00 = 0.10
0.23 = 0.06
0.38 # 0.06
0.83 = 0.08
0.01 = 0.03
0.31 # 0.03
0.61 4 0.09
0.48 2 0.07
0.53 # 0.07
LDA
0.87
0.07
0,08
= 0.05
0,35 2 0.04
0.91 = 0.04
0.05
0.03
0.35
= 0.02
0.82 1 0.06
0.57 i 0.09
0.06 i: 0.07
Logit
0.88 0.07
0.13 0.06
0.91
0.05
0.10
0.04
0.37
0.07
0.08
0.66 0.07
the the final 2 algorithms can only learn linear decision boundaries. Two main patterns stand out from these rosults. First, most classification algorithms show the same trend, where the augmented training results in a small decrease in accuracy on single gesture classification, and a large increase in accuracy on combination gesture classification, resulting a net benefit overall as compared the partial supervision model. Second, it appears that classification algorithms with a non-linear decision boundary benefit much more from the augmented training, and achieve higher overall accuracy result.
A.3 Ablation Experiments
We conduct ablation experiments on several elements of our pretraining procedure. Specifically, we train models in which one more of the three loss terms in Equation 3 are removed. We also vary the amount of noise added to the raw imput data, described in Equation 6. In these experiments, use the same model hyperparameterss as in Table 1 and Figure 4; specifically, the auxiliary classifier Gpre used during pretraining is the "small" architecture, the combination operator Co is the MLP architecture, and the triplet loss function used is the "basic" version.
A.3.1 Effect on balanced accuracy
Table 6: Effect of ablations on balanced accuracy, for the 8 single gesture classes (AcCsingte), the 16 combination gesture classes (Accooms), and all 24 classes (Accat). Augmented supervision; classifier for unseen subject trained according to Section 3.4. Partial Supervised and Full Supervision; baseline models described in Section 4.1. LTRPLET, LeR, LeR; loss terms in Equation 3. SN R; signal-to-noise ratio (dB) after adding noise to raw input data. All models used random forest algorithm for unseen test subject, "small" architecture for auxiliary classifier GPre, MLP combination operator Co and "basic" triplet loss. Entries show the mean and standard deviation across 50 independent. trials. Bold row represents non-ablated model used in Table and Figure 4.
Hyperparamater
Augimented Supervision
Partial Supervision
Full Supervision Acc,.ui
Cuiple
C.
Aocuingl
Acc..m
Areuc
Accangtt
Apc.uus
Acc.a
Accusugt
Aocuu
/
20.0
0.71 +: 0.11
0.19 0.04
0.36 +: 0..4
0.86 +: 0.06
0.00 4: 0.00
0.29 4: 0.02
0.71 4: 0.08
0.45 4 0.08
0.54 0.06
0.14
0.41
1
0.30
0.09
7
20.0
0.69
4 0.13
0.33 0.06
0.45 1 0.06
0.88 4 0.08
0101 4 0.01
0.30
4: 0.02
0.18 4: 0.07
0.54 2 0.10
0.62 0.08
0.06
0.39
1:
0.29
7
20.0
0.86
:: 0.06
0:23 0.07
0.44 + 0.05
0.91 4: 0.04
0.01
4 0.02
0.31
4 0.02
0.80 =: 0.06
0.62 2: 0.00
0,88 0.07
0.45
0.90 1
0.01
0.30
0.80
7
/
10.0
0.81 :: 0.08
0.30 0.06
0.47 + 0.05
0.91
4: 0.06
0.01
4 0.01
0.31
4: 0.02
0.81 4: 0.07
0.61 2: 0.08
0:67 0.07
0.11
0.47
0.90 1
0.31 0.02
0.79
7
7
0.74 4: 0.12
0.34 2: 0.06
0.47 4 0.05
0.50 4 0.05
0.01
4 0.01
0.31 4: 0.02
0.80 4: 0.07
0.57 2: 0.00
0:65 0.07
0.77
0.32 0.08
0.47 0.06
0.01
0.01
0.02
0.68
0.09
0.65 0.07
26

---

Published in Transactions on Machine Learning Research (04/2024)
In Table 6, we show the balanced accuracy of various ablated models. We find that the original, non-ablated model (shown in bold) performs best. Of the three loss terms, the best single term to include is the real cross-entropy Los; surprisingly, including only the triplet loss Crercer results in model that performs better on single gestures than on combination gestures.
A.3.2 Effect on Feature-Spant Space Similarity
Table 7: Effect model ablations on feature-space similarity of combination gestures. See Section 4.2 for description of similarity metrics. Lrmrue, Lous Lous loss terms in Equation 3. SN Rt; signal-to-noist ratio (dB) after adding noise to raw imput data. SetSim( Zeomb, Zeomb); average similarity between real items of same class (first 16 diagonal elements of Msam). SetSim Zcomb); average similarity wetween synthetic items of same class (last 16 diagonal elements of Msim). SetSim( Zeomb; Zeamb); average similarity between real and synthetic items of matching label (16th sub-diagonal elements of M SetSim( SetSim(Zoamh, Zoomb) average similarity between items of non-matching label (all other below- dinggnal elements of Msim: note this includes real and synthetic items). Entries show the mean and standard deviation across 50 independent. trials. Bold row represents model shown in Table 1 and Figure 4.
Hyperparameters
Ltripltt feo Lee
SNR
SetSim Zoomb, Zcomb)
SetSim( Zcomb; Zoomb)
SetSim Zeomb Zcomb)
SetSim SetSim(Zcombr Brumch)
20.0
0.28 + 0.11
0.41 t 0.13
0.15 = 0.08
0.05 t 0.04
/
20.0
0.21 + 0.18
0.39 1 0.20
0.12
0.12
0.06 1 0.07
/
20.0
0.19 1 0.19
0.36 0.21
0.13
0.15
0.06 1 0.09
20.0
0.92 = 0.03
0.94 + 0.01
0.85 = 0.03
0.72 + 0.07
/
20.0
0.65 = 0.09
0.78 + 0.05
0.53 0.08
0.26 + 0.08
20.0
0.36 1 0.14
0.52 0.13
0.23
0.09 1 0.06
/
10.0
0.42 1 0.09
0.60 1 0.08
0.28 1 0.08
0.11 1 0.05
/
x
30.0
0.31 1 0.18
0.48 1 0.17
0.21 : 0.16
0.08 1 0.09
0.30 = 0.16
0.47 0.16
0..00 0.13
0.07 = 0.06
20
0.33 = 0.14
0.51 0.14
0.22 0.12
0.07 + 0.07
As mentioned previously, we used fixed kernel lengthscale 8 for all experiments. When interpreting similarity values, the key outcome we desire is that matching classes should have higher similarity values than non-matching classes. In general, we observe that this remains true across all model ablations, though the degree of contrast varies. Note that due to the dynamics of gradient descent during the pretraining stage, it is possible that some model's feature spaces have a generally shorter or longer lengthscale, and this may affect the apparent contrast in similarity values.
A.3.3 Relative Loss Scaling
In Equation 3, we define multi-objective optimization. The magnitudes of the triplet loss and cross-entropy losses may not be on the same scale, since the triplet loss depends on a margin parameter, and on the scaling of the feature space. Thus, we consider a modified objective with an additional coefficient to scale the relative size of these two types of losses:
CTRILE + ax(CE Lex).
We vary the coefficient a and measure the resulting balanced accuracy of the model as before. We find that varying the relative magnitude of triplct and cross-entropy losses slightly changes the trade-off between single and combination gesture performance. Specifically, we observe that increasing in Eq. (7) causes decrease in Accsingle and an increase in AcCcomb- However, this change in the trade-off does not provide an increase in Accau, indicating other techniques may be required to achieve better performance on both single and combination classes simultaneously.
27

---

Published in Transactions on Machine Learning Research (04/2024)
Table 8: Effect of varying relative loss scaling on balanced accuracy for the 8 single gesture classes (AcCeingle) ) the 16 combination gesture classes (Acceomb), and all 24 classes (Accau). Augmented supervision; classifier for unseen subject trained according Section 3.4. Partial Supervised and Full Supervision; baseline models described in Section 4.1. A single coefficient. a scales the relative magnitude of triplet and cross-eutropy losses in Eq. (7). Changing a changes the trade-off in performance on single and combination gestures, but does not affect overall accuracy across all classes. Entrics show the mean and standard deviation across 50 independent trials. Bold row represents the model shown in Table I and Figure 4.
Hyperparameter in Eq, (7)
Augmented Supervision
Partial Supervision
Full Supervision
Accsingic
Actoool
Abcul
Actsingls
Accuuesi
Actai
Antumple
Acccomb
Aotuli
0.07
0.05
0.02
0.09
0.5
0,00 + 0.30
0.31
0106
0:47 + 0.06
0.30 + 0.06
0.01 + 0.01
0.30
0.02
0.80 + 0.07
0.61 + 0.08
0.67 + 0.06
1.0
0.17 = 0.11
0.32
0.08
0.47 1 0.06
0.90 2: 0.05
0.01 1: 0.01
0.31
0.02
0,80 4: 0.07
0.58 0.09
0.65 4: 0.07
2.0
0.73
0.12
0.7
0.47
0.06
0.30
0.02
0.57
0.10
+
10.0
0.71
= 0.13
0.36
0.06
0.47 1 0.05
0.89
0.06
0.01 1 0.01
0.30 0.02
0.79 1: 0.07
0.57 0.10
0.64 1: 0.09
28


Reviews:
[{"review_id": "b8sbaU43Xl", "reviewer": "Reviewer_aZhy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- The task and problem definition of subject transfer learning is well-motivated and described.\n- The proposed pre-training method of an EMG encoder is well-designed. The use of triplet loss is convincing.\n\n[Weaknesses]\n- The experimental evaluation is conducted on only one dataset collected by the authors.\n- The baseline methods compared to the proposed method might be too simple.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper targets the task of gesture recognition from electromyography (EMG) as a machine learning application and to improve the accuracy while minimizing the calibration cost for adapting the model to a new subject. The authors consider the combination gestures consisting of direction and modifier components and the situation where only the single component gestures are given for a new subject. To extrapolate unseen combination gestures, the authors generate synthetic training data from real single gestures. The authors propose a novel encoder pre-trining strategy to produce useful synthetic training data for unseen test subjects and to possess the homomorphisms of combining feature vectors. In the experiment, the effectiveness of the proposed method is evaluated using a real EMG dataset. The experimental result demonstrates that the proposed method can improve the recognition accuracy compared to the partially-supervised model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Does the proposed method work well on other datasets or tasks other than that used in this paper? Could you discuss the generality of the proposed method and the possible applications other than gesture recognition from EMG?\n1. I am not sure whether the accuracies (e.g., $Acc_{single}=0.77$ and $Acc_{comb}=0.32$) of the proposed method are sufficient for a target application. It would be better to explain the accuracy requirements from the viewpoint of the target application.\n1. I am not sure whether the considered baseline methods are appropriate. Can we consider the domain adaptation and few-shot learning methods as baseline methods? Could you elaborate on the justification for the selection of the baseline methods?\n\n[Minor comments]\n- In Section 1: \"Noise in EMG data can stem from from the...\" -> \"Noise in EMG data can stem from the...\"", "broader_impact_concerns": "I do not have an additional concern on the ethical implications of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kRYPvQYlHV", "reviewer": "Reviewer_oc36", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper does a good job of introducing the problem and motivating why learning a combination-homomorphic encoder is important and useful.\n- The method design makes sense, i.e., contrastive learning to encourage encoded features to be close to real gesture pairs and far away from fake pairs. This also looks novel to me, although I am not familiar with self-supervised/contrastive learning literature.\n- Experiments show that the proposed method can largely increase the model performance on real pairs of gestures compared to the baselines.\n- The paper is well-written and easy to follow.\n\n**Weaknesses:**\n\n- The partial supervision baseline seems too simple and easy to beat. I’m sure there are self-supervised methods to learn better representations using real paired data. E.g., training a VAE w/ an auxiliary classification loss.\n- Currently, the method is only validated on a single task and single datasets. Since the method is general for combination labels, I’d like to see it being applied to more tasks (even if it’s toy/synthetic examples).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on the problem of efficient training of EEG-based gesture recognition models on unseen subjects. The gestures are defines as combinations of “directions” and “modifiers”, which makes collecting all pairs of data time-consuming and labor-intensive. Therefore, this paper aims to only collect either direction or modifier labels to train the gesture recogntion model. To achieve this, it proposes a self-supervised contrastive learning approach, where synthetic combinations of gesture labels are created, and a triplet loss is used to learn a feature encoder that maps EEG of real gesture pairs to features that are far away from fake gesture pairs. Once the encoder is learned, it is used to train a gesture classifier, which is expected to have higher accuracy since it can combine the “direction” and “modifier” features better. Experiments show that the proposed augmented supervision approach outperforms baselines that only use partial supervision w/ either gesture direction or modifier for training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- More/stronger baselines to show the method advances the state of the art.\n- More tasks to demonstrate the generality of the proposed method.", "broader_impact_concerns": "- The paper follows good IRB protocols and the research like doesn't present specific broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nfoqa4Sy35", "reviewer": "Reviewer_Um9S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is nicely written, most details are clear. Figures are high quality and convey the complexity of the problem and the solution.\n- The problem description is clear, it is a complicated problem, due to having multiple settings (single and combined), not common in a machine learning problem, where classes are known a priori and there is no expectation of generalization to combinations of classes not seen during training.\n- The problem is very interesting for the machine learning community, generalizing for partial data is an extremely interesting problem as it can impact/lessen the amount of data required for generalization, and overall generalizing from single classes to combination of classes is an extremely interesting problem from a supervision perspective. From this perspective the treatment of this problem is a contribution to the state of the art.\n- I believe the evaluation is correct, there is a good selection of metrics, only a single dataset due to the uniqueness of the problem, and three different supervision settings, with progressively more supervision added to the problem.\n- The overall result is that the proposed augmented supervision method improves on partial supervision (training on single settings while evaluating on the combined setting), with balanced accuracy improving from 1% to 32% (over 16+1 classes), while reducing accuracy in the single setting from 90% to 77%.\n- Overall I believe this paper proposes a novel method for a novel problem, and is a good contribution to the state of the art that is significant for the field and audience of machine learning.\n\nWeaknesses\n- Something a bit concerning is that the proposed method produces a reduction in balanced accuracy in the single setting, together with an improvement in the combined setting. This is fine, but the paper does not dive deeper on why there is a reduction in the single setting, at least as validation this should be done, as it might point to future research or at least could be an actionable weakness.\n- I believe some ablation studies could have been performed, for example, in Equation 3, weights should been added to this multi-task loss as the triplet and cross-entropy losses are not in the same scale, and an ablation study could be made for the loss weights. Maybe this way some trade-off between single and combined case performance can be found (this is just speculation).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is about training a classifier with individual gesture samples (gesture + modifier), and basically being able to generalize to combinations of those gestures (cross product of gesture and modifier, gesture x modifier) , by performing contrastive learning to learn features that are similar in both setups.\n\nThe paper designs and evaluates this method specifically for EMG gesture recognition, where the authors captured EMG data with users performing certain gestures, both in single and combined settings, and the proposed method is trained on the single setting, while evaluated on the combined setting, allowing a degree of generalization outside of the training set.\n\nResults show that the proposed method using a contrastive learning pipeline, outperforms the baseline (which does not generalize in the combined setting) while at the same time having a reduction in performance in the single setting.\n\nThe contributions of this paper are:\n- A method for generalization for gesture recognition, where classes can be divided into single class, or combinations of classes (similar to a multi-label setting).\n- Using partial supervision for the problem of gesture recognition in single/combined settings.\n- Results that indicate that the proposed method works better than naive solutions, better generalizing in the combined setting, while reducing performance in the single setting.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be very nice if the claims in the paper are explicitly written at the end of the introduction, same for the contributions, in order to make sure authors and reviewers are on the same page about what is being claimed.\n- I would expect that the authors make a deeper dive into why the single setting performance reduces when using the triplet loss and partial supervision, while I do not expect or require new experiments, this can probably be answered with some experimentation, but I am not sure.\n- Also please consider adding loss weights for Eq 3 and ablation experiments for those loss weights.", "broader_impact_concerns": "Broader impact looks good t o me, no additional ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on the problem of efficient training of EEG-based gesture recognition models on unseen subjects. The task of subject transfer learning is well-motivated and has good applications. After the first round of reviews, there are some major concerns on analysis of the single case and ablation studies, the generality of the proposed method and experimental comparisons.  These issues have been well addressed in the revision and all reviewers are satisfied with the revision.
Decision: Accept (poster)